Per-comparator adjustment sets yield non-comparable PAIC contrasts
Neither adjustment rule protects the ranking. The common (intersection) set ranked better in 15 of 36 scenarios and the maximal sets in 19; which wins depends on whether the omission bias pushes a lower-ranked comparator past a higher one, which the data do not show. The registered inversion was not confirmed in the primary scenarios, whose pattern turned out to widen the top comparator’s lead, but held in all six Fawsitt-like scenarios with similar populations. The intersection’s per-contrast bias was a median 2.4 times the maximal. A bounded analysis over the other comparators’ reported means covered at least 0.93 with similar comparator populations at little extra width. Read the study below.
Statement
When one individual participant data trial is compared against several aggregate-data comparators, the covariates that can enter the weighting or outcome model are fixed separately for each comparator by what that comparator’s publication happened to report. The resulting contrasts condition on different covariate sets, and therefore rest on different residual-confounding assumptions, yet they are printed side by side and read as a ranking. Fawsitt et al. 2025 document the mechanism in an unanchored simulated treatment comparison of lenvatinib plus pembrolizumab against four first-line regimens in renal cell carcinoma: the base-case overall-survival model nominated nine covariates, but favorable MSKCC risk was not reported by two comparator trials and the four lesion-location proportions were not reported by a third, and the analysis states that a selected covariate not reported by a comparator study was not included in the regression model, so different models can be used for each comparison. That the choice matters is measurable within the same analysis: refitting every comparison on a common all-covariates model left three of the four overall-survival comparisons and two of the four progression-free-survival comparisons no longer statistically significant. No convention requires a common minimum adjustment set across comparators, no diagnostic separates variation across comparators that reflects genuine efficacy differences from variation that reflects differential adjustment, and no reporting rule requires the per-comparator adjustment sets to be displayed together.
Why it is open
The analyst does not control which covariates a competitor publication prints, so a common adjustment set can only be built by intersecting across comparators, which discards information the sponsor’s own trial holds and can drop a modifier that at least one comparator did report; refitting each comparison on its own maximal set is therefore the defensible default, and it silently varies the conditioning set across contrasts. Nothing in the output signals the incoherence, because each contrast is separately correct: a reader sees four effect estimates against four comparators and cannot tell that they adjust for four different things. Transporting every comparison to one declared target population does not repair it, because the target fixes which population the contrast refers to and not which variables it conditions on, and on non-collapsible scales two contrasts conditioning on different sets are not estimands of the same kind even in the same population. A diagnostic would have to separate efficacy differences from adjustment differences using only the aggregate summaries that created the constraint, which is exactly the information that is missing; the sponsor can bound one comparison’s sensitivity by refitting its own trial, but cannot recover what the comparator did not publish.
What has been tried
| Work | What it contributes |
|---|---|
| Fawsitt et al. 2025, Unanchored simulated treatment comparison on survival outcomes using parametric and Royston-Parmar models with application to lenvatinib plus pembrolizumab in renal cell carcinoma, BMC Medical Research Methodology 25:26 | States the mechanism explicitly, that a selected covariate not reported by a comparator study is dropped so different models can be used for each comparison, and supplies the consequence: under a common all-covariates model three of four overall-survival comparisons and two of four progression-free-survival comparisons lost significance. |
| Macabeo et al. 2024, Methods for Indirect Treatment Comparison: Results from a Systematic Literature Review, Journal of Market Access and Health Policy 12:58-80 | Systematic review of 73 methods articles across seven indirect comparison techniques. States the population half of the incoherence, that population adjustment differs for each study to which the analysis is being adjusted, without treating the covariate-set half as a defect requiring a diagnostic or a reporting rule. |
| Jiang et al. 2025, A critical assessment of matching-adjusted indirect comparisons in relation to target populations, Research Synthesis Methods 16:569-574 | Records a real appraisal in which two matching-adjusted indirect comparisons used different sets of covariates, which complicated understanding of the cause of the inconsistency between them. The phenomenon is observed in practice, not only in a single analysis. |
| Phillippo, Ades, Dias, Palmer, Abrams and Welton 2018, Methods for Population-Adjusted Indirect Comparisons in Health Technology Appraisal, Medical Decision Making 38:200-211 (the peer-reviewed account of NICE DSU TSD 18) | Specifies matching-adjusted indirect comparison and simulated treatment comparison for a single pairwise comparison against a single aggregate-data comparator. The coherence of several such comparisons reported together is outside its scope, which is why nothing in current guidance forbids the practice. |
| Phillippo et al. 2020, Multilevel Network Meta-Regression for Population-Adjusted Treatment Comparisons, Journal of the Royal Statistical Society Series A 183:1189-1210 | Fits all comparisons in one model with a single covariate specification, so the conditioning set is common by construction. It requires covariate summaries from every aggregate study, which is the constraint that pushes analysts back to per-comparator models in the first place. |
Probable solution or research direction
Report the adjustment sets as a matrix with one row per comparator and one column per candidate covariate, marking for each cell whether the covariate was nominated, whether that comparator reported it, and whether it entered the model; this converts an invisible incoherence into a visible one and costs nothing. Require an intersection analysis alongside the maximal-set base case, so every published ranking is accompanied by one set of contrasts that do condition on the same variables, and report the difference between the two as an adjustment-heterogeneity diagnostic rather than burying it among sensitivity analyses. Where a comparator fails to report a nominated modifier, treat that contrast as partially identified in the missing variable and report bounds over a plausible range of its comparator-arm mean, rather than dropping the term and proceeding as if the variable were balanced. Prefer a single network model with one covariate specification whenever the aggregate summaries permit it, and name the comparators that forced the fallback when they do not. Then measure how much this matters across submissions: recompute maximal-set and intersection contrasts for a set of published multi-comparator analyses and report how often the ordering of comparators changes.
Our study
Covariate sets that differ across comparators, and the ranking read from them
The question put to it. When one individual-data trial is compared with several aggregate comparators that report different covariates, does adjusting each contrast for everything it reports give a worse ranking than adjusting all contrasts for the common set, even though each contrast is less biased?
What it found. Neither adjustment rule protects the ranking. The common (intersection) set ranked better in 15 of 36 scenarios and the maximal sets in 19; which wins depends on whether the omission bias pushes a lower-ranked comparator past a higher one, which the data do not show. The registered inversion was not confirmed in the primary scenarios, whose pattern turned out to widen the top comparator’s lead, but held in all six Fawsitt-like scenarios with similar populations. The intersection’s per-contrast bias was a median 2.4 times the maximal. A bounded analysis over the other comparators’ reported means covered at least 0.93 with similar comparator populations at little extra width.
Results.
- Registered primary (strongest covariate missing for the top comparator, similar populations, tied effects): maximal sets lower on both bias and wrong-top-rank probability in 3 of 3 scenarios; verdict MAXIMAL VINDICATED.
- Fawsitt-like pattern, similar populations: intersection lower wrong-top-rank probability in 6 of 6 scenarios, up to 0.94 against 0.31.
- The difference between the two analyses did not predict a wrong top rank (AUROC 0.567).
- Bounded interval coverage at least 0.929 with similar comparator populations at 1.00 to 1.99 times the maximal width; 0.678 at worst with dispersed populations. Maximal-set coverage fell to 0.086.
Design.
- Estimand
- B_k versus A mean difference in each comparator’s population, and the ranking of the four
- Methods compared
- MAIC with maximal reported sets, MAIC with the intersection set, bounded interval over other comparators’ reported means
- Data-generating mechanism
- Unanchored MAIC of one trial against four comparators with nine covariates, 36 cells crossing reporting pattern, prognostic strength, similarity of comparator populations and separation of true effects, plus 8 control cells; maximal, intersection and bounded analyses. 1000 replicates per cell.
- Replicates
- 1000 per scenario, 44,000 in total
What this does not answer. Continuous outcome and independent covariates; non-collapsible outcomes, correlated omissions, a single ML-NMR network fit and overlap as a factor were not run. Three reporting patterns sample but do not exhaust where omissions can fall. Peer review has not been done.
Peer review. Not yet done. The result stands on the study alone until it is.
Read it: PDF · Markdown · OpenDocument · Code and data: studies/COV-14-adjustment-sets-across-comparators · Protocol: registered before the run
Verification
Verdict. Confirmed open. One reader proposed this and the reviewer accepted it as distinct from covariate selection for a single comparison, while asking that the ranking consequence be stated as a risk rather than as an inevitable result, which this entry does. The evidence is a single applied analysis, though the same phenomenon is separately recorded in a systematic methods review and in a published critique of an appraisal.
Source
Derived from reading:core-02 of the reviewed corpus.