Reconstruction uncertainty from digitized Kaplan-Meier curves
No, the refuting sentence fails: neither ensemble behaved as a multiple imputation. For RMST to 24 months at a 6-month risk table the variance calibration ratio (one for an imputation) was 31.8 to 44.3 for six analyst variants and 2.3 to 11.2 for 20 observation-model draws, the registered ‘both far from one’ outcome. Across cells the draws overstated the error of RMST (2.1 to 13.1) and 12-month survival (1.1 to 3.5) and understated it for Weibull survival at 48 months (0.15 to 0.77; 90% intervals covered the true-data value in 42% to 77%). Reconstruction error was small against sampling error (single-reconstruction RMSE 2% to 8% of the sampling SD for RMST, 11% to 26% for 48-month survival), so population coverage stayed at 0.91 to 0.98. Read the study below.
Statement
Survival indirect comparisons routinely digitize published Kaplan-Meier curves into pseudo individual patient data and carry the reconstruction downstream as if it were observed, and population-adjusted intervals do not propagate the reconstruction error. Part of the proposed remedy has been built: RESOLVE-IPD (arxiv:2511.01785) generates ensembles of mathematically valid labelings so that reconstruction uncertainty propagates into the downstream meta-analysis, and Rogula et al. (doi:10.1177/23814683221077643) reconstruct from marked censoring times and report curve-fit uncertainty the standard algorithm does not provide. Multiple reconstruction is therefore no longer merely promising, though it is a valid analogue of multiple imputation only where the repeated reconstructions are proper draws from a calibrated distribution conditional on the image, the risk table and the reported totals, rather than repeated runs of a deterministic algorithm or arbitrary tuning variants. Defining a Bayesian likelihood directly on the reported Kaplan-Meier ordinates and numbers-at-risk table remains unbuilt.
Why it is open
Reconstruction is deterministic given the image, so every standard method runs without complaint and repeating it reveals nothing; the error is silent while the reported interval ignores the extra variability. The published Kaplan-Meier ordinates are dependent derived statistics, so a likelihood defined on them needs an explicit digitization, rounding, censoring and risk-table observation model, which nobody has written. Population-adjusted implementations take reconstructed event and censoring rows straight into the likelihood as data: at commit 9d150e9, cpaic, the unreleased development package on GitHub that is not installable from CRAN, has no observation model for digitization, risk-table interpolation, event allocation or alternative compatible reconstructions. How much this matters depends on image quality and on downstream sensitivity, so it is context-dependent rather than uniformly decisive.
What has been tried
| Work | What it contributes |
|---|---|
| Lang, Zhao, Gao and Xu 2025, RESOLVE-IPD: High-Fidelity Individual Patient Data Reconstruction and Uncertainty-Aware Subgroup Meta-Analysis (arxiv:2511.01785) | its MAPLE component generates ensembles of mathematically valid labelings, facilitating a propagating meta-analysis that quantifies and reflects uncertainty from subgroup reconstruction, with VEC-KM extracting coordinates and explicit censoring marks from vectorized figures; it is the multiple-imputation analogue the entry proposed |
| Rogula, Lozano-Ortega and Johnston 2022, A Method for Reconstructing Individual Patient Data From Kaplan-Meier Survival Curves That Incorporate Marked Censoring Times, MDM Policy and Practice | reconstruction method using marked censoring times that yields estimates of uncertainty in the curve fit, which the standard Guyot-style algorithm does not provide |
Probable solution or research direction
Connect the existing ensemble reconstruction machinery to population-adjusted analysis rather than reinventing it, and state the calibration condition explicitly so the ensemble is a set of draws conditional on the image, risk table and reported totals rather than a set of arbitrary tuning variants. Build the Bayesian likelihood on the reported curve and numbers-at-risk table, which requires an explicit observation model for digitization, rounding, censoring and risk-table reporting. Report how far RMST and extrapolated survival move across reconstructions rather than assuming a uniform inflation, since curve error can partly cancel under integration and can be amplified or damped by the tail model.
Our study
Reconstruction ensembles were not imputations: their spread overstated the error in restricted mean survival and understated it in the extrapolated tail
The question put to it. When individual data are reconstructed repeatedly from a published Kaplan-Meier curve, does the spread of the ensemble match the true reconstruction error, so that pooling it like multiple imputation is valid?
What it found. No, the refuting sentence fails: neither ensemble behaved as a multiple imputation. For RMST to 24 months at a 6-month risk table the variance calibration ratio (one for an imputation) was 31.8 to 44.3 for six analyst variants and 2.3 to 11.2 for 20 observation-model draws, the registered ‘both far from one’ outcome. Across cells the draws overstated the error of RMST (2.1 to 13.1) and 12-month survival (1.1 to 3.5) and understated it for Weibull survival at 48 months (0.15 to 0.77; 90% intervals covered the true-data value in 42% to 77%). Reconstruction error was small against sampling error (single-reconstruction RMSE 2% to 8% of the sampling SD for RMST, 11% to 26% for 48-month survival), so population coverage stayed at 0.91 to 0.98.
Results.
- The analyst variants’ mean overstated RMST by 0.17 to 0.19 months in every cell; a post hoc rerun traced it to sparse digitization (error 0.29 to 0.34 months with 30 points, 0.15 to 0.16 with 60, 0.06 with 120), and their spread is that ordering, not uncertainty.
- Rubin pooling widened intervals by at most 1% with the draws; the variants widened them by up to 23% only for 48-month survival without a risk table.
- Positive control failed (12-month survival RMSE 13% of the sampling SD in the coarse, no-table, clustered cell against a registered 25%): milestone survival was not materially affected in this design.
- The null control (monthly table) passed on 161 of 300 replicates; 139 were lost to a 20-second wall-clock guard that fired under a machine load average of 500 to 900.
Design.
- Estimand
- Variance calibration ratio of a reconstruction ensemble: mean between-reconstruction variance times (1 + 1/M) over the variance of the ensemble mean’s error against the true data
- Methods compared
- single Guyot reconstruction, analyst variants (30, 60, 120 points; event total given or not), observation-model draws (pixel and censoring positions redrawn)
- Data-generating mechanism
- Simulated trial data rendered as a pixel-rounded Kaplan-Meier figure with or without a risk table, reconstructed by Guyot’s algorithm; analyst-variant and observation-model ensembles scored by the ratio of their spread to the true reconstruction error for RMST, milestone and extrapolated survival. 13 cells, 300 replicates each.
- Replicates
- 300 per cell, 13 cells
What this does not answer. The draws are not a posterior: Guyot’s event-count rounding is held fixed and there is no likelihood for the published ordinates. One arm and no downstream comparison, so propagation into a population-adjusted or component comparison (CMP-17) is not measured; no human digitization error; one sample size. The mechanism analyses are post hoc. Peer review has not been done.
Peer review. Not yet done. The result stands on the study alone until it is.
Read it: PDF · Markdown · OpenDocument · Code and data: studies/OUT-13-an-ensemble-that-is-not-an-imputation · Protocol: registered before the run
Verification
Verdict. Partially addressed. Part of this problem is addressed by existing work; the entry names which part.
The source said. “Survival indirect comparisons routinely digitize published Kaplan-Meier curves to build pseudo individual patient data, then carry that reconstruction into every downstream analysis as if it were observed data, with no propagation of reconstruction uncertainty.”
What was wrong with it. The source listed multiple reconstruction as an unbuilt promising direction; the literature auditor found it implemented in RESOLVE-IPD (arxiv:2511.01785) and partially in Rogula et al. (doi:10.1177/23814683221077643). The code auditor added two qualifications: repeatedly applying a deterministic reconstruction or varying arbitrary tuning choices is not a valid multiple-imputation analogue, and reconstruction error does not monotonically compound, since RMST integrates curve error that can partly cancel while extrapolation can amplify or damp it depending on the tail model.
- Literature and prior-art check
- partially-solved / supported-with-caveat. Routine practice does treat digitised pseudo-IPD as observed data, and I found no standard tool that propagates reconstruction error, so the description of practice holds. But the proposed direction has begun to be built. Rogula, Lozano-Ortega and Johnston (MDM Policy & Practice 2022, doi 10.1177/23814683221077643) reconstruct IPD using marked censoring times and report that the method yields estimates of the uncertainty in curve fits that traditional methods do not provide. More directly, RESOLVE-IPD (arXiv 2511.01785, Lang, Zhao, Gao and Xu) implements the multiple-reconstruction idea: its MAPLE component ‘generates ensembles of mathematically valid labelings, facilitating a propagating meta-analysis that quantifies and reflects uncertainty from subgroup reconstruction’, with VEC-KM extracting coordinates and explicit censoring marks from vectorised figures to minimise digitisation error. That is the multiple-imputation analogue named in the proposed direction, so it is no longer merely ‘promising’. The Bayesian-likelihood-on-reported-summaries route remains undeveloped; I found nothing implementing it. Cites arxiv:2511.01785: Implements ensemble (multiple-reconstruction) uncertainty propagation from KM digitisation into downstream meta-analysis, the analogue of multiple imputation the entry proposes.. Cites doi:10.1177/23814683221077643: Reconstruction method that yields estimates of uncertainty in the curve fit, unavailable from the standard Guyot-style algorithm..
- GPT-5.6 Sol, technical and source-code lens
- open / supported-with-caveat. At the requested cpaic revision, aggregate survival input consists of reconstructed event and censoring rows that enter the likelihood as data. There is no observation model for digitization, risk-table interpolation, event allocation, or alternative compatible reconstructions, so ordinary cpaic posterior intervals do not propagate reconstruction uncertainty. C3 is directionally correct, although reconstruction errors need not monotonically compound: RMST integrates curve error, which can partly cancel, while extrapolation can amplify or damp it depending on the tail model. C4 is technically promising only if repeated reconstructions are proper draws from a calibrated distribution conditional on the image, risk table, and reported totals; repeatedly applying a deterministic reconstruction or varying arbitrary tuning choices is not analogous to valid multiple imputation. C5 is possible in principle, but the reported Kaplan-Meier ordinates are dependent derived statistics and require an explicit digitization, rounding, censoring, and risk-table observation model. The field-wide words ‘routinely’ and ‘standard’ are empirical claims outside this audit’s source-code scope. The priority is context-dependent because reconstruction uncertainty may be negligible or decision-changing depending on image quality and downstream sensitivity.
- Decision path
-
R3:partially-solved(n=1) → R11:caveated(n=2)
How the evidence falls in time
21 papers in the reviewed corpus bear on this problem, published 2011 to 2026. Partial progress is on the record, and later work still calls this open. Both belong in view.
This problem is recurrent: three or more papers spanning at least eight years assert it independently. Sustained restatement by authors who mostly do not cite each other is the strongest evidence this reading can offer that a gap is real rather than one group’s framing.
| Year | Says | Paper |
|---|---|---|
| 2011 | confirms it is open | Network meta-analysis of survival data with fractional polynomials |
| 2012 | confirms it is open | Meta-regression models to address heterogeneity and inconsistency in network meta-analysis of survival outcomes |
| 2014 | partly addresses it | Network meta-analysis of (individual patient) time to event data alongside (aggregate) count data |
| 2014 | confirms it is open | A process for assessing the feasibility of a network meta-analysis: a case study of everolimus in combination with hormonal therapy versus chemotherapy for advanced breast cancer |
| 2015 | confirms it is open | Systematic review and meta-analysis: techniques and a guide for the academic surgeon |
| 2016 | confirms it is open | Live cumulative network meta-analysis: protocol for second-line treatments in advanced non-small-cell lung cancer with wild-type or unknown status for epidermal growth factor receptor |
| 2016 | confirms it is open | Review of the Reporting of Survival Analyses within Randomised Controlled Trials and the Implications for Meta-Analysis |
| 2017 | confirms it is open | Second-line Treatments for Advanced Gastric Cancer: A Network Meta-Analysis of Overall Survival Using Parametric Modelling Methods |
| 2020 | confirms it is open | Performance of unanchored matching-adjusted indirect comparison (MAIC) for the evidence synthesis of single-arm trials with time-to-event outcomes |
| 2020 | confirms it is open | Outcomes at 5 years in patients with severe aortic stenosis: reviewing current information using the restricted mean survival time |
| 2022 | confirms it is open | Challenges of modelling approaches for network meta-analysis of time-to-event outcomes in the presence of non-proportional hazards to aid decision making: Application to a melanoma network |
| 2022 | confirms it is open | Medical Therapy, Radiofrequency Ablation, or Cryoballoon Ablation as First-Line Treatment for Paroxysmal Atrial Fibrillation: Interpreting Efficacy Through the Shiny Method |
| 2023 | confirms it is open | Individual participant data from digital sources informed and improved precision in the evaluation of predictive biomarkers in Bayesian network meta-analysis |
| 2024 | confirms it is open | Overall survival with non-proportional hazards in first-line treatment for patients with metastatic colorectal cancer: Systematic review and network meta-analysis |
| 2025 | confirms it is open | Unanchored simulated treatment comparison on survival outcomes using parametric and Royston-Parmar models with application to lenvatinib plus pembrolizumab in renal cell carcinoma |
| 2025 | confirms it is open | Addressing challenges with Matching-Adjusted Indirect Comparisons to demonstrate the comparative effectiveness of entrectinib in metastatic ROS-1 positive Non-Small Cell Lung Cancer |
| 2025 | confirms it is open | NMAsurv: An R Shiny application for network meta-analysis based on survival data |
| 2025 | confirms it is open | One-step parametric network meta-analysis models using the exact likelihood that allow for time-varying treatment effects |
| 2025 | confirms it is open | Simplifying fractional polynomials in Bayesian network meta-analysis via variable powers |
| 2026 | confirms it is open | Anchors Away: Navigating Unanchored Indirect Comparisons with Multilevel Unanchored Meta-Regression (ML-UMR) (Chandler & Ishak; arXiv:2606.20341v1). The batch file points at media/media/Table1.pdf; the full LaTeX source mlumr.tex sits in the same arXiv package and was read. |
| 2026 | confirms it is open | Re-evaluating Treatments for Advanced Urothelial Carcinoma Using Restricted Mean Survival Time: A Systematic Review and Network Meta-analysis |
Source
Derived from C:L214-215, C:L245, C:L1578-1591 of the reviewed corpus.