Narrow outcome families and hazard-ratio-only survival scoring
For competing risks, population adjustment transports the cause-specific hazard ratio (unbiased in every scenario) but not the cumulative incidence a decision model uses: when the target’s competing hazard differed from the source’s beyond the measured covariate, A’s absolute incidence at 2 years was biased by up to 0.20 (true value 0.25) with coverage near zero. The between-arm difference was affected less. Recalibrating the competing hazard to the target’s reported competing-event proportion removed the bias. The registered rule (bias beyond 0.05 wherever the competing hazard differed) was met in 20 of 24 method-scenarios; the exceptions were at half the competing hazard, with bias 0.041 to 0.049. Read the study below.
Statement
Published PAIC simulations score survival on the hazard ratio: Remiro-Azocar et al. 2021 on the log hazard ratio, Liu et al. 2025 on marginal log hazard ratios from Cox models, Hatswell et al. 2020 on mean error of the effect measure, and none reports time-indexed survival-probability error, RMST bias, integrated Brier score or extrapolation error. Under delayed, waning or crossing effects a hazard-ratio bias summary can be small while the survival curves the economic model consumes are badly wrong. Cure fractions, competing risks, recurrent events, multistate processes, proportional-odds violations for ordinal outcomes and jointly generated efficacy, safety, quality-of-life and cost outcomes are absent from this simulation literature. The tooling is no longer the obstacle: ML-NMR has been extended to general individual-level likelihoods with flexible M-spline baseline hazards (arxiv:2401.12640), so absolute-scale target-population survival quantities are computable even though simulations do not score on them.
Why it is open
PAIC theory is cleanest for collapsible linear-scale outcomes, so simulation practice has concentrated where the theory is tidy while HTA submissions are dominated by survival endpoints with non-proportional hazards. Generating crossing hazards, cure fractions or competing risks with a known target-population marginal estimand requires deriving that estimand analytically or by large-sample Monte Carlo, which is more work than a Gaussian scenario and is routinely skipped. Correlated multi-outcome generation is needed to evaluate net benefit and other nonlinear decision functionals, yet simulations generate one outcome at a time.
What has been tried
| Work | What it contributes |
|---|---|
| Multilevel network meta-regression for general likelihoods: synthesis of individual and aggregate data with applications to survival analysis, 2024 | Extends population-adjusted synthesis to arbitrary individual-level likelihoods with flexible M-spline baseline hazards and survival applications, so absolute-scale target-population survival quantities are computable. Retrieved at abstract level. |
| Remiro-Azocar et al. 2021, Methods for population adjustment with limited access to individual patient data: A review and simulation study | Scores survival performance on the log hazard ratio, one of the studies the literature auditor cites as evidence of hazard-ratio-only survival scoring. |
| Liu et al. 2025, Is the Use of Unanchored Matching-Adjusted Indirect Comparison Always Superior to Naive Indirect Comparison on Survival Outcomes? A Simulation Study | Scores survival performance on marginal log hazard ratios from Cox models, with no absolute-scale or time-indexed measures. |
| Hatswell et al. 2020, The Effects of Model Misspecification in Unanchored Matching-Adjusted Indirect Comparison: Results of a Simulation Study | Scores on mean error and mean absolute error of the effect measure, again with no survival-probability, RMST or extrapolation measures. |
Probable solution or research direction
Extend simulation generators to the missing outcome families with explicitly derived target-population marginal estimands (survival probability, RMST, cumulative incidence, cure fraction, event burden), using the general-likelihood ML-NMR machinery that already computes absolute-scale target-population quantities, so that bias is computed against the quantity the decision uses. Score survival PAIC on time-indexed survival-probability error and RMST bias by default, and add the remaining measures where they apply rather than universally: integrated Brier score where survival predictions and censoring handling are defined, extrapolation error where the pipeline extrapolates beyond observed follow-up, recurrent-event calibration where event burden is the estimand. Generate efficacy, safety, quality-of-life and cost outcomes jointly where the decision functional is nonlinear or joint uncertainty matters.
Our study
Competing risks: a correct hazard ratio, a wrong cumulative incidence
The question put to it. When population adjustment transports a time-to-event outcome with a competing cause, does a correctly transported cause-specific hazard ratio give the right cumulative incidence in the target, and what repairs it?
What it found. For competing risks, population adjustment transports the cause-specific hazard ratio (unbiased in every scenario) but not the cumulative incidence a decision model uses: when the target’s competing hazard differed from the source’s beyond the measured covariate, A’s absolute incidence at 2 years was biased by up to 0.20 (true value 0.25) with coverage near zero. The between-arm difference was affected less. Recalibrating the competing hazard to the target’s reported competing-event proportion removed the bias. The registered rule (bias beyond 0.05 wherever the competing hazard differed) was met in 20 of 24 method-scenarios; the exceptions were at half the competing hazard, with bias 0.041 to 0.049.
Results.
- Cause-1 log hazard ratio: bias within 3 MCSE in 16 of 16 scenarios, at most 0.013.
- A’s cumulative incidence from MAIC or STC: bias 0.13 to 0.20 with coverage at most 0.03 at four times the competing hazard; -0.04 to -0.06 at half; registered verdict NOT CONFIRMED (20 of 24 method-scenarios beyond 0.05).
- A-minus-C difference: bias at most 0.049.
- Recalibrated competing hazard: bias at most 0.004; coverage 0.851 to 0.943 for the absolute incidence and 0.943 to 0.972 for the difference.
Design.
- Estimand
- Target cumulative incidence of the event of interest at 2 years under A, and its A-minus-C difference; cause-1 log hazard ratio
- Methods compared
- MAIC, cause-specific STC, cause-specific STC with the competing hazard recalibrated to target data
- Data-generating mechanism
- Anchored transport with exponential cause-specific hazards, the target’s competing hazard differing from the source’s beyond the measured covariate; 16 cells crossing that difference, effect modification and overlap; MAIC, cause-specific STC and STC with the competing hazard recalibrated to the target’s reported competing-event proportion. 1000 replicates per cell.
- Replicates
- 1000 per scenario, 16,000 in total
What this does not answer. Only the competing-risks family; cure fractions, recurrent events, ordinal and joint net-benefit outcomes not run. Exponential hazards, one covariate, no ML-NMR. Peer review has not been done.
Peer review. Not yet done. The result stands on the study alone until it is.
Read it: PDF · Markdown · OpenDocument · Code and data: studies/DIA-09-outcome-families-beyond-the-hazard-ratio · Protocol: registered before the run
Verification
Verdict. Confirmed open. Auditors agreed the problem is open and no counterevidence surfaced.
The source said. “The simulation evidence base does not span the outcome families decision problems involve (rare and very rare binary events, overdispersed, zero-inflated and exposure-dependent counts, ordinal outcomes violating proportional odds, delayed, waning and crossing hazards, non-Weibull mixtures, cure fractions, competing risks, recurrent events, multistate disease, and correlated efficacy, safety, quality-of-life and cost outcomes), and survival performance is scored by hazard-ratio bias rather than by the quantities decisions use.”
What was wrong with it. Both auditors judged the problem open, and the correction is a caveat the source did not acknowledge: the literature auditor notes that survival PAIC methodology has moved, with ML-NMR extended to general likelihoods and flexible baseline hazards (arxiv:2401.12640), so the gap is in what simulations score rather than in what methods can produce. GPT-5.6 Sol also narrowed the proposed measures, since integrated Brier score, extrapolation error, joint outcome generation and recurrent-event calibration apply to specific pipelines and estimands rather than universally.
- Literature and prior-art check
- open / supported-with-caveat. The survival-scoring claim is broadly right in the PAIC simulation literature I retrieved: Remiro-Azocar 2021 scores on the log hazard ratio, Liu et al. 2025 on marginal log hazard ratios from Cox models, Hatswell 2020 on mean error/mean absolute error of the effect measure. None reports time-indexed survival-probability error, integrated Brier score, RMST bias or extrapolation error. The caveat is that survival PAIC methodology has moved: ML-NMR has been extended to general likelihoods with M-spline baseline hazards and survival applications, and unanchored STC has been implemented with parametric and Royston-Parmar models explicitly to handle non-proportional hazards and extrapolation, so the tooling for absolute-scale survival quantities exists even though simulations do not score on them. I found no PAIC simulation work at all on cure fractions, competing risks, recurrent events, multistate processes, ordinal proportional-odds violation, or jointly generated efficacy/safety/QoL/cost outcomes, so those parts of the claim stand. Cites arXiv:2401.12640: Extends population-adjusted synthesis to arbitrary individual-level likelihoods with flexible (M-spline) baseline hazards and survival applications, so absolute-scale target-population survival quantities are computable even though simulations still score on the hazard ratio. Retrieved at abstract/summary level..
- GPT-5.6 Sol, technical and source-code lens
- open / supported-with-caveat. The technical core is sound. Under non-proportional hazards, bias against a single constant hazard ratio does not characterize errors in survival probabilities, RMST, cumulative incidence, or downstream economic quantities. Delayed, waning, and crossing effects can have materially different decision consequences and merit distinguishable scenarios. Rare events and proportional-odds violations are also genuine factors. The proposed measures are not all universal requirements: integrated Brier score is a predictive-distribution score requiring defined survival predictions and censoring handling, not a generic measure of causal contrast estimation; joint outcome generation is essential for joint uncertainty and nonlinear decision functionals but not for every marginal-effect evaluation; extrapolation error belongs only to pipelines that extrapolate; and recurrent-event calibration is relevant only when recurrent burden is an estimand. The claims about which outcome families and measures appear in published studies are literature-coverage claims and were not adjudicated. Very high priority is proportionate for methods making survival decision claims, while the full outcome catalogue is a broader roadmap.
- Decision path
-
R5:default-open → R11:caveated(n=2)
How the evidence falls in time
7 papers in the reviewed corpus bear on this problem, published 2020 to 2025. Every paper found here asserts the problem is open; none reports progress.
| Year | Says | Paper |
|---|---|---|
| 2020 | confirms it is open | Assessing the performance of population adjustment methods for anchored indirect comparisons: A simulation study |
| 2020 | confirms it is open | Performance of unanchored matching-adjusted indirect comparison (MAIC) for the evidence synthesis of single-arm trials with time-to-event outcomes |
| 2021 | confirms it is open | Methods for population adjustment with limited access to individual patient data: A review and simulation study |
| 2024 | confirms it is open | Implementing Multilevel Network Meta-Regression for Time-To-Event Outcomes: A Case Study in Relapsed Refractory Multiple Myeloma |
| 2024 | confirms it is open | Advancing unanchored simulated treatment comparisons: A novel implementation and simulation study |
| 2025 | confirms it is open | Unanchored simulated treatment comparison on survival outcomes using parametric and Royston-Parmar models with application to lenvatinib plus pembrolizumab in renal cell carcinoma |
| 2025 | confirms it is open | The hazards of using hazard ratios from proportional hazard models in indirect treatment comparisons |
Source
Derived from C:L1880-1895, C:L2036-2044 of the reviewed corpus.