Design-dependent roles of prognostic covariates
Yes. On the log odds ratio scale a difference in the spread of prognosis between trials biases an anchored MAIC that balances only means by up to 0.11, even with no effect modification. Balancing variances or the variance of the prognostic index removes the bias, but when the source is less dispersed than the target the loss of effective sample size makes RMSE worse than not balancing. G-computation was unbiased with the lowest RMSE throughout. Read the study below.
Statement
Whether a strong prognostic non-modifier should be adjusted for depends on the estimand and the design, not on the variable alone. For a conditional or collapsible effect in an anchored comparison, balancing such a variable spends effective sample size without removing bias, although including it in an outcome model can improve precision. For non-collapsible measures such as the odds ratio or hazard ratio, a variable that is purely prognostic on the conditional scale can still change the marginal effect, so cross-study imbalance in it can bias an anchored marginal comparison even when marginal covariate moments are balanced (Remiro-Azocar 2024, doi:10.1002/sim.10111). In unanchored comparisons, omitting prognostic variables biases the cross-study contrast on any scale.
Why it is open
Covariate role is a function of the estimand, the effect scale and the evidence structure, and the clean split between anchored and unanchored designs holds only on collapsible or conditional scales; once the target is a marginal odds ratio or hazard ratio, prognostic distributions matter for transport too. Applied analyses and guidance still carry one covariate list across both designs, so either the effective-sample-size cost or the bias is paid without being noticed. Software support is uneven rather than absent: the cSTC interface in the development package cpaic already separates prognostic main effects from treatment interactions while its cML-NMR path has no prognostic-only role, and cpaic is an unreleased GitHub development version, commit 9d150e9, not installable from CRAN. The numbers that exist cover a transported risk difference, so the trade-off has not been quantified where non-collapsibility makes it hardest.
What has been tried
| Work | What it contributes |
|---|---|
| Webster-Clark, 2024 | Variable selection when estimating effects in external target populations; quantifies in one simulation the precision penalty of adjusting for non-effect-measure-modifiers and the bias from omitting modifiers when transporting a risk difference. |
| Campbell and Remiro-Azocar, 2025 | Doubly robust augmented weighting estimators for externally controlled single-arm trials and unanchored indirect comparisons; uses prognostic covariates through the outcome model rather than through balancing constraints, with higher precision than non-augmented weighting. |
| Remiro-Azocar, 2024 | Counter-evidence to the source’s first claim: unadjusted anchored indirect comparisons can be biased even when marginal covariate moments are balanced, so adjustment for purely prognostic variables may be necessary for non-collapsible marginal estimands. |
| cpaic, commit 9d150e9 | Unreleased development R package on GitHub, not on CRAN; its cSTC function already separates prognostic main effects from treatment interactions, while its cML-NMR function has no prognostic-only covariate role, so the proposed software separation is partly present already. |
Probable solution or research direction
Index covariate role by estimand and scale in guidance and in software, keeping the modifier set required for bias removal separate from the prognostic set used for precision, and complete that separation in the code paths that lack it rather than designing it from scratch. Extend the existing simulation evidence from a transported risk difference (doi:10.1093/aje/kwae048) to non-collapsible measures, so the anchored penalty and the unanchored bias are quantified where the design-based rule breaks down. Extend the augmented weighting approach (arxiv:2505.00113), which already uses prognostic covariates without balancing constraints, from the single-source unanchored case to anchored comparisons and to networks.
Our study
What an anchored comparison must balance on a non-collapsible scale
The question put to it. On the log odds ratio scale a purely prognostic covariate changes the marginal effect. Does an anchored MAIC that balances only covariate means, as practice does, leave material bias, and does balancing the variance of the prognostic index remove it at acceptable cost in effective sample size?
What it found. Yes. On the log odds ratio scale a difference in the spread of prognosis between trials biases an anchored MAIC that balances only means by up to 0.11, even with no effect modification. Balancing variances or the variance of the prognostic index removes the bias, but when the source is less dispersed than the target the loss of effective sample size makes RMSE worse than not balancing. G-computation was unbiased with the lowest RMSE throughout.
Results.
- MAIC on means biased by 0.05 to 0.11 in the 12 primary anchored scenarios, beyond 0.05 (95% Monte Carlo interval) in 9; registered verdict CONFIRMED.
- Unadjusted anchored bias on the exact population bias: slope 0.981 (SE 0.018).
- Balancing variances cut the bias to at most 0.08 (0.02 in all but one scenario) but dropped effective sample size to 44 to 284 of 1000 when the source index variance was half the target’s.
- G-computation: largest absolute bias 0.024 anchored and 0.020 unanchored, lowest RMSE in every primary scenario.
Design.
- Estimand
- Marginal log odds ratio, B versus A, in the target population
- Methods compared
- unadjusted, MAIC on means, MAIC on means and variances, MAIC on means and the prognostic index variance, G-computation
- Data-generating mechanism
- Anchored and unanchored binary-outcome comparisons, 108 cells crossing prognostic strength, the source-to-target ratio of prognostic index variance, target mean shift and effect modification; five methods including MAIC on means, on means and variances, on the prognostic index variance, and G-computation. 1000 replicates per cell.
- Replicates
- 1000 per scenario, 108,000 in total
What this does not answer. Independent normal covariates and a correctly specified outcome model favor G-computation. Hazard ratios, skewed covariates and augmented estimators not run; MAIC intervals treat weights as fixed. Peer review has not been done.
Peer review. Not yet done. The result stands on the study alone until it is.
Read it: PDF · Markdown · OpenDocument · Code and data: studies/COV-03-prognostic-index-variance · Protocol: registered before the run
Verification
Verdict. Overstated. 2 independent auditors judged the claim as stated too strong. On a collapsible effect scale, or for a conditional estimand, balancing strong prognostic variables that are not effect modifiers in an anchored comparison reduces effective sample size without removing bias; for non-collapsible measures such as the odds ratio or hazard ratio, purely prognostic variables can modify the marginal effect, so cross-trial imbalance in them can bias even an anchored marginal comparison and adjustment may be warranted. In unanchored comparisons, omitting prognostic variables biases the cross-study contrast on any scale.
The source said. “Whether a strong prognostic variable that is not an effect modifier should be adjusted for depends on the design: in anchored analyses balancing it can reduce effective sample size without reducing bias, whereas in unanchored analyses omitting prognostic variables can directly bias cross-study contrasts. Precision-oriented methods may use such variables differently again.”
What was wrong with it. Both auditors judged the source’s first claim too strong: it holds on a collapsible scale or for a conditional estimand, but under non-collapsibility purely prognostic variables can modify the marginal effect, so an anchored comparison can be biased even with balanced marginal moments and adjustment may be warranted (doi:10.1002/sim.10111). The source also said no guidance covers the precision-oriented intermediate case; Campbell and Remiro-Azocar 2025 (arxiv:2505.00113) give an augmented estimator that uses prognostic covariates through an outcome model without imposing balance constraints on them, and Webster-Clark 2024 (doi:10.1093/aje/kwae048) already quantifies the precision penalty of adjusting for non-modifiers and the bias from omitting modifiers in one simulation.
- Literature and prior-art check
- partially-solved / overstated. The first and fifth claims are false as stated for non-collapsible measures. Remiro-Azocar 2024 (Stat Med, read verbatim) concludes that ‘Unadjusted anchored indirect comparisons can be biased in the absence of individual-level treatment effect heterogeneity, or when marginal covariate moments are balanced across studies’, and that adjustment ‘may be necessary to account for cross-study imbalances in joint covariate distributions involving purely prognostic variables’. So balancing a purely prognostic variable in an anchored comparison does not merely cost effective sample size; on the odds-ratio or hazard-ratio scale it can remove real bias in the marginal estimand. Separately, the precision half of the problem is partly answered: Webster-Clark 2024 quantifies in one simulation both the precision cost of including non-modifiers and the bias from omitting modifiers, and Campbell & Remiro-Azocar 2025 give an augmented estimator that uses prognostic covariates in an outcome model for precision without imposing balance constraints on them, reporting ‘higher precision than all non-augmented weighting estimators’. Cites doi:10.1093/aje/kwae048: Quantifies, in a single simulation, the precision penalty of adjusting for non-effect-measure-modifiers and the bias from omitting modifiers when transporting a risk difference. Cites arxiv:2505.00113: Shows how prognostic covariates can be used for precision through the outcome model rather than through balancing constraints, i.e. the ‘intermediate case’ the entry says has no guidance.
- GPT-5.6 Sol, technical and source-code lens
- partially-solved / overstated. C1 is correct for a conditional or collapsible target contrast when the variable truly is not an effect modifier on that scale: balancing it can spend ESS without removing bias, while including it as an outcome-model main effect may improve precision. It is not generally correct for a marginal odds ratio or hazard ratio, because a purely prognostic variable on the conditional scale can change the marginal contrast through non-collapsibility. C2-C4 are broadly correct for unanchored absolute-outcome comparisons, provided the prognostic factor differs between studies and is otherwise uncontrolled. C5 confuses modifier selection with inclusion as a prognostic nuisance variable and its word ‘only’ misses non-collapsibility and proxy-variable cases. cpaic@9d150e9 already lets cstc() separate prognostic main effects from treatment interactions, while cmlnmr() explicitly lacks a prognostic-only role. Thus the software-design part is only partially open; the unanchored identification issue is decision-critical, while the anchored efficiency issue is a roadmap item. Cites choxos/cpaic@9d150e9: cstc() already implements separate treatment-interaction and prognostic-main-effect roles; cmlnmr() does not, so only part of the proposed software separation is present..
- Decision path
-
R3:partially-solved(n=2) → R8:overstated-majority(n=2)
How the evidence falls in time
6 papers in the reviewed corpus bear on this problem, published 2017 to 2022. Partial progress is on the record, and later work still calls this open. Both belong in view.
It is also being closed in pieces: partial results come from more than one paper in more than one year, each covering a different part.
| Year | Says | Paper |
|---|---|---|
| 2017 | confirms it is open | Methods for Population-Adjusted Indirect Comparisons in Health Technology Appraisal |
| 2019 | partly addresses it | Alternative Weighting Approaches for Anchored Matching-Adjusted Indirect Comparisons via a Common Comparator |
| 2020 | partly addresses it | Comparison of Methods for Estimating Therapy Effects by Indirect Comparisons: A Simulation Study |
| 2020 | partly addresses it | Extending inferences from a randomized trial to a new target population (Dahabreh, Robertson, Steingrimsson, Stuart, Hernan; tutorial, arXiv:1805.00550v3) |
| 2021 | partly addresses it | Methods for population adjustment with limited access to individual patient data: A review and simulation study |
| 2022 | confirms it is open | A Response to: Letter to the Editor Regarding “Network Meta-analysis of Food and Drug Administration-approved Treatment Options for Adults with Aquaporin-4 Immunoglobulin G-positive Neuromyelitis Optica Spectrum Disorder” |
Source
Derived from C:L924, C:L930 of the reviewed corpus.