The MAIC paradox: swapping the IPD trial can reverse the comparison
Reversal is mostly a real target difference and it is common: with the modifier’s zero-effect point between the two populations, sponsors disagreed in sign in 23% to 56% of analyses at separations up to 1 SD and modification up to 0.4, and moving the point outside the populations cut that by a median 0.31. The two sponsors’ estimates are correlated, which a reversal calculation must use. Under linear modification the plain average of the two analyses is as accurate for the midpoint population as transporting both trials there. Read the study below.
Statement
Reversing which trial supplies the IPD can reverse the direction of the apparent active-treatment comparison while each analysis remains internally coherent, because the two analyses estimate effects in different target populations. This is not new with Jiang et al. 2025 (doi:10.1017/rsm.2025.10), who credit NICE DSU TSD18: Phillippo et al. 2018 (doi:10.1177/0272989X17725740) documented the same reversal in a real appraisal, two competing manufacturers’ MAICs of secukinumab and adalimumab, and prescribed the remedy this entry proposes, namely deciding which population represents the decision target and transporting the comparison to it under the shared effect modifier assumption. Jiang et al.’s increment is that the paradox persists even when both analyses use the same covariate set. A target-population difference is also not the only way two MAICs can disagree; sampling error, different adjustment sets, model misspecification and measurement differences can each do it.
Why it is open
Pairwise MAIC contains no mechanism forcing both directions of an analysis to answer the same question, because the target is set by which trial supplies IPD rather than by the decision. The remedy has been on the books since 2018 and has not changed sponsor behavior, since each sponsor’s own trial is the natural source of its IPD and nothing obliges two competing submissions to share a target. Transporting both effects to one declared target also needs common support, compatible outcome definitions and enough information to carry both transports, which is often exactly what is missing when the two populations differ enough to produce the reversal. How often the reversal arises and under what covariate configurations is unmeasured, so it is still discovered after submission rather than anticipated.
What has been tried
| Work | What it contributes |
|---|---|
| Jiang et al. 2025, A critical assessment of matching-adjusted indirect comparisons in relation to target populations, Research Synthesis Methods | Describes the reversal, its internal coherence and its population explanation, stating that both conclusions are potentially valid within the context of the specific populations considered, and shows the paradox persists with a common covariate set. It credits TSD18 for the original example and recommends IPD sharing rather than ruling on averaging. |
| Phillippo, Ades, Dias, Palmer, Abrams and Welton 2018, Methods for Population-Adjusted Indirect Comparisons in Health Technology Appraisal, Medical Decision Making (NICE DSU TSD18) | Documents the two-way MAIC reversal in practice, with two MAIC analyses from competing manufacturers comparing treatments for ankylosing spondylitis, and prescribes the fix the entry proposes: decide which population represents the decision target and transport the comparison to it under the shared effect modifier assumption. |
| Fang and He 2026, Arbitrated Indirect Treatment Comparisons | A dedicated methodological response to the MAIC paradox that fixes a single common target, the overlap population, rather than a sponsor-specific one. It does not establish how often the reversal occurs or which target a committee should prefer. |
Probable solution or research direction
Do not average conflicting MAICs unless the average is deliberately defined as an effect in a specified mixture population and the conditions for that definition hold. The remaining work is empirical and procedural rather than conceptual: characterize how often and under which covariate configurations the reversal arises, and give appraisal committees a rule for what to do when two sponsor submissions target different populations. Fang and He’s arbitrated indirect comparison (arxiv:2510.18071) is an existing methodological response that fixes a single common overlap target instead of a sponsor-specific one, so the task is to evaluate it against transporting both effects to an external decision target, not to reinvent it.
Our study
When two sponsors’ MAICs disagree: how often, why, and which common target resolves it
The question put to it. When two sponsors run MAIC on the same two trials, each weighting its own data to the other trial, how often do their conclusions reverse, how much of that is a real difference in target population rather than sampling error or differing adjustment sets, and which common-target remedy recovers the decision population’s effect?
What it found. Reversal is mostly a real target difference and it is common: with the modifier’s zero-effect point between the two populations, sponsors disagreed in sign in 23% to 56% of analyses at separations up to 1 SD and modification up to 0.4, and moving the point outside the populations cut that by a median 0.31. The two sponsors’ estimates are correlated, which a reversal calculation must use. Under linear modification the plain average of the two analyses is as accurate for the midpoint population as transporting both trials there.
Results.
- Between-minus-outside reversal probability exceeded 0.20 in 11 of 18 common-set configurations (median 0.31); the refuting sentence, that reversals need extreme configurations, fails.
- Correlation between the two sponsors’ estimates 0.14 to 0.99; a bivariate normal with that correlation predicted the reversal probability within 0.036, and ignoring it erred by up to 0.44.
- A sponsor that left the modifier out of its weights was biased by 0.10 to 0.91 against its own target.
- Transport of both trials to a declared population and the average of the two analyses: largest bias 0.024 and 0.027, median RMSE 0.231 and 0.239.
Design.
- Estimand
- B-versus-A mean difference in each sponsor’s target population and in a declared decision population
- Methods compared
- sponsor A MAIC, sponsor B MAIC, transport of both trials to a declared population, average of the two analyses
- Data-generating mechanism
- Two trials with linear modification of one treatment, analyzed by two sponsors each weighting its own trial to the other’s population; 108 scenarios crossing separation, modification strength, the position of the zero-effect point, trial sizes and adjustment sets, plus controls; transport of both trials to a declared population and the average of the two analyses as remedies.
- Replicates
- 1000 per scenario, 111,000 in total
What this does not answer. Linear modification and a continuous outcome; with nonlinear modification or a non-collapsible scale the average is not the midpoint effect. The arbitrated indirect comparison and measurement differences between trials were not run. Peer review has not been done.
Peer review. Not yet done. The result stands on the study alone until it is.
Read it: PDF · Markdown · OpenDocument · Code and data: studies/EST-06-maic-reversal-region · Protocol: registered before the run
Verification
Verdict. Partially addressed. Part of this problem is addressed by existing work; the entry names which part.
The source said. “Jiang et al. describe a target-population problem in which reversing which trial supplies the IPD can reverse the apparent active-treatment comparison, even when each analysis is internally coherent. The two analyses are not contradictory implementations; they estimate effects in different target populations.”
What was wrong with it. The source read as though Jiang et al. 2025 discovered the reversal and as though the ban on averaging came from them; the literature auditor found the reversal documented with an applied case in NICE DSU TSD18 nine years earlier (doi:10.1177/0272989X17725740), along with the same prescribed fix, and found no recommendation against averaging in Jiang et al., who instead call for IPD sharing with HTA agencies. GPT-5.6 Sol added that a target-population difference is not the only cause of discordant MAICs and that a weighted average is not invalid in principle if it is deliberately defined as an effect in a specified mixture population.
- Literature and prior-art check
- partially-solved / supported-with-caveat. I read the full text of Jiang et al. 2025 (PMC12527493). The reversal, its internal coherence and its population explanation are all verbatim there (‘both conclusions … are potentially valid within the context of the specific populations considered’). But the entry reads as if Jiang et al. discovered the phenomenon. They did not, and they say so, crediting NICE DSU TSD18. Phillippo et al. 2018 Med Decis Making documented the same reversal with a real applied case nine years earlier and prescribed the entry’s own proposed direction: ‘the real conflict lies not in the different results produced by the 2 MAICs but in deciding which of the 2 study populations better represents the decision target population … the shared effect modifier assumption, if justified, may be used to transport indirect comparisons into the target population for the decision.’ Jiang et al.’s genuine increment is that the paradox persists with the same covariate set. Separately, the claim that averaging is invalid is not in Jiang et al.; they recommend IPD sharing to HTA agencies. Fang & He (arXiv:2510.18071, v2 May 2026) is an explicit methodological response targeting the overlap population. Cites doi:10.1177/0272989X17725740: Documents the two-way MAIC reversal in practice and prescribes the fix the entry proposes: declare the decision target and transport both effects to it under the shared effect modifier assumption.. Cites arxiv:2510.18071: A dedicated methodological response to the MAIC paradox that fixes a single common target (the overlap population) rather than a sponsor-specific one..
- GPT-5.6 Sol, technical and source-code lens
- partially-solved / supported-with-caveat. C1 and C2 are technically possible: if swapping the IPD trial swaps the target population and the marginal treatment contrast varies across populations, the sign can reverse without a software error. C3 explains the constructed target-population paradox but is not the only possible cause of discordant MAICs; sampling error, different adjustment sets, model misspecification, and measurement differences can also reverse results. C4 is right for an unprincipled average of different estimands. A weighted average can be valid only if it is deliberately defined as an effect in a specified mixture population and the required linearity or standardization conditions hold. C5 states the correct design principle, subject to common support, compatible outcome definitions, and sufficient data or assumptions to transport both effects. C6 is acceptable as saying Jiang et al. describes the phenomenon, not that it uniquely discovered it. The conceptual cause and common-target remedy are already available; feasibility and empirical frequency remain open, so very-high severity is conditional rather than automatic. Cites doi:10.1017/rsm.2025.10: Demonstrates and explains the reversal as comparison of population-specific estimands rather than necessarily a software contradiction.. Cites doi:10.1177/0272989X17725740: Already prescribes choosing the decision target and transporting effects to it under stated assumptions.. Cites arxiv:2510.18071: Implements a common overlap-population target as one concrete response to sponsor-specific target reversal..
- Decision path
-
R3:partially-solved(n=2) → R11:caveated(n=2)
How the evidence falls in time
6 papers in the reviewed corpus bear on this problem, published 2017 to 2026. The most recent paper to touch this reports progress on it, and every paper calling it open is older.
It is also being closed in pieces: partial results come from more than one paper in more than one year, each covering a different part.
Does the later work answer the earlier work? Put to an independent reviewer with the older and newer claims side by side, it judged that the later work answers a component of it, leaving a named part (high confidence). Later work identifies the estimand mismatch mechanism, demonstrates that reversal can occur even with the same correct modifier set, and supplies an overlap-target estimator that removes the reciprocal discrepancy for that target. It does not establish that the overlap population is the appropriate decision target or solve the governance problem required to impose it. What remains: Selecting a decision-relevant common target population and establishing an institutional way to arbitrate it remain unresolved; disagreement can also arise from sampling, measurement, adjustment-set, and model failures.
| Year | Says | Paper |
|---|---|---|
| 2017 | partly addresses it | Methods for Population-Adjusted Indirect Comparisons in Health Technology Appraisal |
| 2018 | confirms it is open | Comparing pharmacological treatments for cocaine dependence: Incorporation of methods for enhancing generalizability in meta-analytic studies |
| 2022 | partly addresses it | Geometric approaches to assessing the numerical feasibility for conducting matching-adjusted indirect comparisons |
| 2025 | partly addresses it | A critical assessment of matching-adjusted indirect comparisons in relation to target populations |
| 2026 | partly addresses it | Arbitrated Indirect Treatment Comparisons |
| 2026 | partly addresses it | Reframing Population-Adjusted Indirect Comparisons as a Transportability Problem: An Estimand-Based Perspective and Implications for Health Technology Assessment (Chandler & Ishak; arXiv:2602.17041v3). The batch file points at media/media/Table_1_v2.pdf; the full LaTeX source Transportability_PAIC.tex sits in the same arXiv package and was read. |
Source
Derived from C:L308-310, C:L46 of the reviewed corpus.