Held-out IPD is not used to benchmark transport against truth
Statement
No study masks selected trials of a multi-trial IPD dataset down to aggregate data and scores PAIC estimates against the full-IPD target-standardized analysis, and PAIC validation concentrates on relative effects rather than on the absolute arm-level predictions a transport model actually makes; the closest multi-IPD study, Phillippo et al. 2022 (doi:10.1177/0272989X221117162), validates through residual heterogeneity and inconsistency and modifier relaxation, with no held-out trial and no predicted-versus-observed arm check. Real-data benchmarking against randomized evidence does exist as one-off case studies, which the source does not acknowledge: Signorovitch et al. 2023 (doi:10.1080/09546634.2023.2169574) compares previously published guselkumab MAIC estimates against the subsequently completed ECLIPSE and IXORA-R head-to-head trials. The benchmark is a prespecified reference analysis carrying its own uncertainty rather than known truth, and it must match the estimand: a marginal PAIC estimate needs a marginal target-standardized comparator while conditional STC or conditional ML-NMR output needs a conditional one, a distinction that is material on non-collapsible scales.
Why it is open
Both designs require multi-trial IPD, which sits with sponsors and consortia rather than with methodologists, and data-sharing agreements rarely permit the repeated re-analysis a validation study needs. Population-shift validation additionally requires an identified absolute-risk transport model, compatible outcome definitions and follow-up, adequate overlap and preferably a shared arm for calibration, a rarer configuration than it appears. Masked-IPD validation is uniquely capable of embarrassing a method, so incentives run against it and the real-data checks that exist are sponsor-authored one-offs rather than standing benchmarks with reusable protocols.
What has been tried
| Work | What it contributes |
|---|---|
| Signorovitch et al. 2023, Matching-adjusted indirect comparison (MAIC) results confirmed by head-to-head trials: a case study in psoriasis | Benchmarks published MAIC estimates against subsequently completed head-to-head randomized trials on PASI 90 risk differences; a real-data check with a known answer, though a single sponsor-authored case study rather than a standing benchmark. |
| Phillippo et al. 2023, Validating the Assumptions of Population Adjustment: Application of Multilevel Network Meta-regression to a Network of Treatments for Plaque Psoriasis | The closest existing multi-IPD validation of population adjustment; validates through residual heterogeneity and inconsistency checks and shared-modifier relaxation, with no held-out trial and no predicted-versus-observed absolute arm outcome check. |
| cpaic (development version), commit 9d150e9 | Unreleased development package on GitHub, not available from CRAN. Cited by GPT-5.6 Sol to show the estimand distinction that determines the correct benchmark: cstc() returns a conditional effect at the target covariate values while cmaic() returns a marginal weighted effect in the target population. |
Probable solution or research direction
Establish standing IPD-masking benchmarks on shared multi-trial datasets with a fixed protocol: mask selected trials to aggregate data, run the comparator method set, and report discrepancy from a prespecified full-IPD analysis of the same estimand, marginal for marginal methods and conditional for conditional ones, with uncertainty in both estimates carried through, alongside decision agreement. Where trial pairs with IPD from distinct populations and a shared arm exist, transport in both directions and compare uncertainty-aware predicted arm outcomes with observed ones, treating a failed arm-level prediction as evidence against the transport model. Report arm-level calibration as a routine validation output rather than relative-effect agreement alone.
Our study
Question. On masked individual data, does agreement of an anchored population-adjusted estimate with the masked trial’s randomized result imply that the transport model’s arm-level predictions agree, and how much of any cross-method difference is the estimand rather than performance?
Design. multinma’s simulated UNCOVER IPD; each masked trial reduced to published etanercept versus placebo summaries with its ixekizumab arm withheld; Bucher, MAIC and conditional and marginal STC scored against estimand-matched references and at the arm level, with a joint bootstrap. 14 benchmark units (8 main, 4 self, 2 semi-synthetic positive controls), 1000 bootstrap draws each.
The protocol is registered before any result is seen: read it.
Verification
Verdict. Confirmed open. Auditors agreed the problem is open and no counterevidence surfaced.
The source said. “The two real-data designs with an unambiguous benchmark are not standard practice: IPD masking, in which selected trials of a multi-trial IPD dataset are reduced to aggregate data and the PAIC estimate is compared with the full-IPD target-standardized analysis it is supposed to approximate; and population-shift validation, in which transport between IPD populations is checked by comparing predicted with observed absolute arm outcomes as well as contrasts.”
What was wrong with it. The source called masked IPD the one design with an unambiguous benchmark and did not mention that MAIC estimates have already been checked against subsequent head-to-head trials (Signorovitch et al. 2023, doi:10.1080/09546634.2023.2169574). GPT-5.6 Sol added that a full-IPD benchmark is a prespecified reference analysis with its own sampling uncertainty and must be marginal or conditional to match the estimand of the method being scored.
Dissent. codex dissented (overstated): C1 is sound only if ‘known target’ means a prespecified reference analysis, not known causal truth. A full-IPD estimate remains finite-sample uncertain and may depend on its outcome and transport models. C3 must be estimand-specific. Marginal target standardization is the appropriate comparator for a marginal PAIC estimand, but conditional STC or cML-NMR output requires a corresponding conditional benchmark. This distinction is material on non-collapsible scales. The cpaic source confirms that cstc() is conditional and cmaic() is marginal, while cML-NMR’s current effect path is conditional rather than marginally standardized. C4 requires more than two IPD populations: absolute arm prediction needs an identified baseline-risk model, compatible outcomes and follow-up, positivity, and cross-trial transportability of absolute risks; a shared arm can provide calibration. Observed arm outcomes are noisy benchmarks and should be compared using uncertainty-aware predictions. C6 is correct because common baseline-risk errors can cancel in a contrast. C2 and C5 are literature-practice assertions outside this audit. The validation agenda remains valuable, but its benchmark is neither universal nor unambiguous.
Revised by the full-text reading. Applied 2026-07-26 full-text reading of the consolidated library (687 papers).
- Verdict left unchanged
- verdict left unchanged Later work reports progress, but on the chronology review the second reviewer judged it ‘different-question’ at high confidence: The later study uses leave-one-registry-out validation of patient-level prediction models and compares predictions with observed outcomes. That is useful external-validation methodology, but it neither creates the aggregate-data information constraint nor evaluates population-adjusted indirect-comparison transport estimators.
- Citation corrected
- normalized a citation year: ‘Phillippo et al. 2022, Validating the Assumptions of Population Adjustment: Application of Multilevel Network Meta-regression to a Network of Treatments for Plaque Psoriasis’ -> ‘Phillippo et al. 2023, Validating the Assumptions of Population Adjustment: Application of Multilevel Network Meta-regression to a Network of Treatments for Plaque Psoriasis’ 10.1177/0272989X221117162 is cited with two different years across the registry. CrossRef gives 2022 online and 2023 in print; the issue year is used throughout.
- Literature and prior-art check
- open / supported-with-caveat. I found no study that masks selected trials of a multi-trial IPD dataset down to aggregate data and scores PAIC estimates against the full-IPD target-standardized analysis, so the core claim holds. Two caveats. First, real-data benchmarking of PAIC against randomized evidence does exist and the entry does not acknowledge it: Signorovitch et al. 2023 compared previously published guselkumab MAIC estimates against the subsequently completed ECLIPSE and IXORA-R head-to-head trials on PASI 90 risk differences (14.4% in ECLIPSE vs 9.4% in the MAIC at week 48; 0.0% vs 0.7% at week 24). That is a one-off sponsor-authored case study, not a standing benchmark, which is consistent with the entry’s ‘not standard practice’ framing. Second, the population-shift design is not entirely unexercised: Phillippo et al. 2022 fit ML-NMR to a plaque psoriasis network with IPD from several trials and produce absolute target-population predictions, but explicitly do not hold out a trial or compare predicted with observed absolute arm outcomes, which confirms the entry’s arm-level calibration claim. Cites 10.1080/09546634.2023.2169574: A real-data benchmark of PAIC against randomized evidence with a known answer, which the entry does not acknowledge; it is a single sponsor-authored case study rather than the standing masked-IPD benchmark the entry calls for.. Cites 10.1177/0272989X221117162: The closest existing multi-IPD validation of population adjustment; confirms the entry’s claim by validating only via residual heterogeneity/inconsistency and shared-modifier relaxation, with no held-out trial and no predicted-vs-observed absolute arm outcome check..
- GPT-5.6 Sol, technical and source-code lens
- open / overstated. C1 is sound only if ‘known target’ means a prespecified reference analysis, not known causal truth. A full-IPD estimate remains finite-sample uncertain and may depend on its outcome and transport models. C3 must be estimand-specific. Marginal target standardization is the appropriate comparator for a marginal PAIC estimand, but conditional STC or cML-NMR output requires a corresponding conditional benchmark. This distinction is material on non-collapsible scales. The cpaic source confirms that cstc() is conditional and cmaic() is marginal, while cML-NMR’s current effect path is conditional rather than marginally standardized. C4 requires more than two IPD populations: absolute arm prediction needs an identified baseline-risk model, compatible outcomes and follow-up, positivity, and cross-trial transportability of absolute risks; a shared arm can provide calibration. Observed arm outcomes are noisy benchmarks and should be compared using uncertainty-aware predictions. C6 is correct because common baseline-risk errors can cancel in a contrast. C2 and C5 are literature-practice assertions outside this audit. The validation agenda remains valuable, but its benchmark is neither universal nor unambiguous.
- Decision path
-
R5:default-open → R9:substantiated-single-overstatement:codex
How the evidence falls in time
2 papers in the reviewed corpus bear on this problem, published 2019 to 2022. The most recent paper to touch this reports progress on it, and every paper calling it open is older.
Does the later work answer the earlier work? Put to an independent reviewer with the older and newer claims side by side, it judged that the later work is about a different question and leaves the earlier gap untouched (high confidence). The later study uses leave-one-registry-out validation of patient-level prediction models and compares predictions with observed outcomes. That is useful external-validation methodology, but it neither creates the aggregate-data information constraint nor evaluates population-adjusted indirect-comparison transport estimators. What remains: A PAIC-specific benchmark that masks trials from multi-trial IPD to aggregate summaries and compares PAIC estimates with an estimand-matched full-IPD target-standardized reference remains unperformed.
| Year | Says | Paper |
|---|---|---|
| 2019 | confirms it is open | Creating historical controls using data from a previous line of treatment - two non-standard approaches |
| 2022 | partly addresses it | Combining individual patient data from randomized and non-randomized studies to predict real-world effectiveness of interventions |
Source
Derived from C:L2099-2101, C:L2107-2109 of the reviewed corpus.