Data access is simulated as benign rather than informative
Yes, for RMST within follow-up the omission is harmless. Reconstructing the comparator arm from a pixel-rounded Kaplan-Meier figure changed neither the ranking of Kaplan-Meier and Weibull RMST (same in all 8 cells) nor any method’s RMSE by more than 1%: it added 0.003 to 0.040 months of bias to Kaplan-Meier RMST and 0.006 to 0.059 to Weibull RMST against a sampling SD of about 0.5 months. The likelihood-based method was more exposed, as predicted, but immaterially. Read the study below.
Statement
No PAIC simulation varies informative IPD availability, selective baseline or outcome reporting, MNAR covariate or outcome missingness, informative censoring, differential loss to follow-up, or sampling error on published target summaries, so access-driven error goes unmeasured and is attributed to method failure by default. Two parts of the original claim need narrowing. A simulate-digitize-reconstruct evaluation design for digitization error already exists in the reconstruction literature (doi:10.1177/23814683221077643); what is missing is its propagation into PAIC estimator performance. And a sponsor holding IPD for its own trial makes access treatment-dependent without making it depend on the study-specific treatment effect, so an explicit availability model is needed when inference generalizes over studies and availability depends on effect-relevant study characteristics, not automatically.
Why it is open
Simulating informative availability requires an explicit model coupling data access to effect-relevant study characteristics, which no PAIC generator contains. Treating published target moments as fixed removes a genuine uncertainty component whenever the estimand is a superpopulation target, and no simulation can show the consequence if the generator never adds sampling error to them. Selective reporting mechanisms are simply absent from the design space, and digitization error, although it has a standard evaluation design of its own in the reconstruction literature, has never been carried through into PAIC estimator performance.
What has been tried
| Work | What it contributes |
|---|---|
| A Method for Reconstructing Individual Patient Data From Kaplan-Meier Survival Curves That Incorporate Marked Censoring Times, 2022 | Illustrates the existing simulate-digitize-reconstruct evaluation design for digitization error, narrowing the claim that digitization error has no standard generator. Retrieved at title, venue and summary level only. |
Probable solution or research direction
Add data-access mechanisms to the generator: reconstruction error using the existing simulate-digitize-reconstruct design, sampling error on published target summaries wherever the estimand is a superpopulation target, selective reporting of baseline characteristics and outcomes, MNAR covariate and outcome missingness, informative censoring, and differential loss to follow-up. Model informative IPD availability explicitly where availability depends on effect-relevant study characteristics rather than assuming sponsor ownership makes it informative. Report how each mechanism degrades estimator and interval performance separately from the primary causal assumptions, so that access-driven error is not attributed to method failure.
Our study
Reconstruction error from a digitized Kaplan-Meier curve was too small to change which method ranks best for RMST
The question put to it. When the comparator arm must be reconstructed from a published Kaplan-Meier figure, does the reconstruction error change which method performs best, as it would if it penalized likelihood-based methods more, or does it only shift every method’s error?
What it found. Yes, for RMST within follow-up the omission is harmless. Reconstructing the comparator arm from a pixel-rounded Kaplan-Meier figure changed neither the ranking of Kaplan-Meier and Weibull RMST (same in all 8 cells) nor any method’s RMSE by more than 1%: it added 0.003 to 0.040 months of bias to Kaplan-Meier RMST and 0.006 to 0.059 to Weibull RMST against a sampling SD of about 0.5 months. The likelihood-based method was more exposed, as predicted, but immaterially.
Results.
- Share of mean squared error attributable to reconstruction: -0.6% to 0.1% (KM), -0.4% to 0.9% (Weibull), zero within Monte Carlo error.
- Reconstruction bias was upward and largest at coarse resolution, from reading drops at pixel-column centers.
- Integrating the digitized curve directly performed as well as reconstruction.
Design.
- Estimand
- Comparator arm RMST to 24 months
- Methods compared
- Kaplan-Meier RMST, Weibull RMST, digitized curve integrated
- Data-generating mechanism
- OUT-13’s publication and reconstruction model; Kaplan-Meier and Weibull RMST on true and reconstructed data and the digitized curve integrated directly; fine or coarse figures, 6-monthly risk table or none, spread or clustered censoring. 8 cells, 1000 replicates each.
- Replicates
- 1000 per cell, 8 cells
What this does not answer. Reconstruction error only; RMST within follow-up only (extrapolated survival and hazard ratios not ranked); no human digitization variability; other access mechanisms not simulated. Peer review has not been done.
Peer review. Not yet done. The result stands on the study alone until it is.
Read it: PDF · Markdown · OpenDocument · Code and data: studies/DIA-10-access-driven-error-attributed-to-the-method · Protocol: registered before the run
Verification
Verdict. Partially addressed. Auditors agreed the problem is open and no counterevidence surfaced. Reopened by the full-text reading: 1 finding from 1 paper survived both the reader and an independent second reviewer, so the verdict moves from confirmed-open to partially-addressed, on a single finding.
The source said. “Simulation designs treat data access as benign: IPD is assumed available at random, published target summaries are treated as fixed and known, and pseudo-IPD is treated as exact. MAR and MNAR covariate missingness, outcome missingness and informative censoring, selective baseline and outcome reporting, informative IPD availability, digitization error, sampling error in target summaries and differential loss to follow-up are not varied.”
What was wrong with it. The source treated digitization error as having no standard generator and sponsor-held IPD as automatically informative; the literature auditor found an established simulate-digitize-reconstruct evaluation design (doi:10.1177/23814683221077643), and GPT-5.6 Sol separated structurally asymmetric access from an effect-informative availability mechanism and noted that fixed target summaries are appropriate when the target is the realized comparator-trial sample rather than a superpopulation.
Revised by the full-text reading. Applied 2026-07-26 full-text reading of the consolidated library (687 papers).
- Verdict changed
- confirmed-open -> partially-addressed 1 surviving finding from: A Bayesian hierarchical model for network meta-analysis of m. New citations: 10.1093/biostatistics/kxx025
- Literature and prior-art check
- open / supported-with-caveat. I found no PAIC simulation that varies informative IPD availability, selective baseline or outcome reporting, MNAR covariate or outcome missingness, informative censoring, differential loss to follow-up, or sampling error on published target summaries, so the core of the entry stands. The digitization sub-claim needs narrowing: the pseudo-IPD reconstruction literature does have a standard evaluation design in which curves are simulated, digitized and reconstructed and hazard ratios and survival quantiles are compared with the originals, and there is recent work propagating reconstruction uncertainty into downstream synthesis by bootstrapping pseudo-IPD. So a digitization-error generator is not absent from evidence synthesis generally; what is absent is its propagation into PAIC estimator performance. I retrieved these reconstruction items at title/summary level only, hence the moderate confidence. Cites 10.1177/23814683221077643: Illustrates the existing simulate-digitize-reconstruct evaluation design for digitization error, narrowing the claim that digitization error has no standard generator. Retrieved at title/venue and summary level only..
- GPT-5.6 Sol, technical and source-code lens
- open / overstated. C2, C4, and C5 identify legitimate and distinct mechanisms. C1, however, confuses a structurally asymmetric data configuration with an informative missing-data mechanism. A sponsor possessing IPD for its own treatment makes access treatment-dependent, but it does not by itself make access depend on the study-specific treatment effect or induce selection bias. An explicit availability model is needed only when inference generalizes over studies and availability depends on effect-relevant study characteristics. C3 also needs an estimand condition: target summaries have sampling uncertainty for inference to a target superpopulation, but they may properly be conditioned on as fixed when the target is the realized comparator-trial sample. Consequently, treating them as fixed does not remove uncertainty from every valid downstream interval. Claims about what simulations usually assume are outside this audit. The access and missingness extensions are valuable, but their necessity and severity depend on the inferential target and actual evidence-acquisition process.
- Decision path
-
R5:default-open → R9:substantiated-single-overstatement:codex
How the evidence falls in time
6 papers in the reviewed corpus bear on this problem, published 2018 to 2021. Partial progress is on the record, and later work still calls this open. Both belong in view.
| Year | Says | Paper |
|---|---|---|
| 2018 | partly addresses it | A Bayesian hierarchical model for network meta-analysis of multiple diagnostic tests |
| 2019 | confirms it is open | Participants’ outcomes gone missing within a network of interventions: Bayesian modeling strategies |
| 2019 | confirms it is open | Selective reporting bias in randomised controlled trials from two network meta-analyses: comparison of clinical trial registrations and their respective publications |
| 2020 | confirms it is open | Assessing the performance of population adjustment methods for anchored indirect comparisons: A simulation study |
| 2020 | confirms it is open | Performance of unanchored matching-adjusted indirect comparison (MAIC) for the evidence synthesis of single-arm trials with time-to-event outcomes |
| 2021 | confirms it is open | Individual Participant Data Meta-Analysis: A Handbook for Healthcare Research (Riley, Tierney, Stewart, eds.; Wiley Series in Statistics in Practice, 2021) |
Source
Derived from C:L1897-1908 of the reviewed corpus.