No factorial study ranks which choice most often flips a decision
The choices are ordered rather than dominated by one. Near a decision threshold, adjusting or not flipped a population-adjusted decision in 36% of analyses, weighting against an outcome model in 20%, linear against quadratic STC in 13% and MAIC on means against means and SDs in 7%. Changing the target population moved the true log odds ratio by 0.39 to 0.59, more than any estimator choice moved the estimate, so the target belongs to the question, not the sensitivity analysis. Read the study below.
Statement
No study has crossed bridge type, prior specification, target-population definition and outcome model in a factorial design scored on decision reversals, so there is no ranking of which structural choice most often flips a recommendation. Individual factors have been examined: Jiang et al. 2025 (10.1017/rsm.2025.10) construct a MAIC paradox in which switching which trial supplies IPD reverses which drug is judged superior, and Cassidy et al. 2023 (10.1016/j.jval.2023.07.001) show across 80 comparisons that MAICs shift estimates toward the treatment whose IPD is held. The four choices are also not equivalent in kind: changing the target population changes the estimand and changing the bridge changes the identifying assumptions, whereas a prior changes regularization and the posterior and an outcome-model specification usually changes the estimator, not the estimand.
Why it is open
The four choices are made at different points in the workflow by different people, so no single analysis varies them jointly and their interactions are never observed. Conventional sensitivity analysis varies a parameter inside a fixed structure and so does not reach choices that change the identifying assumptions or the target population. The factorial study is feasible with ordinary scenario or simulation methods, so what is missing is that nobody has run it, not a missing identification principle; the design must hold a common decision target and use only coherent combinations, because a reversal between two target populations can mean two different decision questions rather than sensitivity of one.
What has been tried
| Work | What it contributes |
|---|---|
| Jiang et al. 2025 | Shows that the target-population choice alone can reverse which treatment is judged superior in MAIC, quantifying one of the four factors. |
| Cassidy et al. 2023 | Compares 80 MAIC estimates with the corresponding standard anchored ITCs across 17 publications and finds MAICs systematically favor the IPD holder’s treatment. |
Probable solution or research direction
Run the factorial simulation with a common decision target held fixed and only coherent factor combinations crossed, reporting the frequency and direction of decision reversals attributable to each factor and to their interactions. Separate factors that redefine the estimand, meaning target population and bridge, from those that change identification or estimation, meaning prior and outcome model, and report them on that footing rather than as one undifferentiated sensitivity. Use the resulting ranking to decide which structural choices must be prespecified in a protocol.
Our study
Which analysis choice flips a decision
The question put to it. Within one decision question, which analysis choices most often flip a population-adjusted recommendation, and how does that compare with the change in the question when the target population changes?
What it found. The choices are ordered rather than dominated by one. Near a decision threshold, adjusting or not flipped a population-adjusted decision in 36% of analyses, weighting against an outcome model in 20%, linear against quadratic STC in 13% and MAIC on means against means and SDs in 7%. Changing the target population moved the true log odds ratio by 0.39 to 0.59, more than any estimator choice moved the estimate, so the target belongs to the question, not the sensitivity analysis.
Results.
- Refuting sentence fails: the top flip rate (0.363) was less than three times the next (0.202).
- Far from the threshold (0.3) flip rates fell by more than half for every choice.
- True log odds ratio across target covariate means 0.3 to 1.2: changes of 0.39 to 0.59 across modification patterns.
Design.
- Estimand
- Decision to adopt B (estimated log odds ratio below a threshold) against the true decision
- Methods compared
- unadjusted, MAIC on means, MAIC on means and SDs, linear STC, quadratic STC
- Data-generating mechanism
- Re-scoring of DIA-08’s stored replicates: decision flips between the two options of four analysis choices at thresholds near and far from the truth, and the true effect across three target populations.
- Replicates
- 32,000 stored replicates from DIA-08 re-scored at four thresholds
What this does not answer. No prior or bridge factor; decision on the relative effect only; one outcome type. Peer review has not been done.
Peer review. Not yet done. The result stands on the study alone until it is.
Read it: PDF · Markdown · OpenDocument · Code and data: studies/DEC-02-which-structural-choice-flips-a-decision · Protocol: registered before the run
Verification
Verdict. Overstated. 2 independent auditors judged the claim as stated too strong. No study has jointly varied bridge type, prior specification, target-population definition and outcome model in a factorial design scored on decision reversals, so there is no ranking of which structural choice most often flips a recommendation. Individual factors have been examined: target-population choice has been shown to reverse MAIC conclusions, and estimator choice (MAIC vs standard anchored ITC) has been shown empirically to shift estimates in a consistent direction.
The source titled this. “Decision sensitivity to bridge, prior, target and model unmeasured” The title says this sensitivity is unmeasured, but the audit found target-population choice and estimator choice each already shown to change conclusions.
The source said. “How sensitive a reimbursement decision is to the analyst’s choice of bridge, prior, target population and outcome model has never been quantified. There is therefore no evidence about which structural modeling choice most often flips a recommendation, and no basis for ranking which choices must be pre-specified.”
What was wrong with it. The source said this sensitivity has never been quantified and that all four choices change the estimand. The literature auditor found two of the four already quantified (Jiang et al. 2025 for target population, Cassidy et al. 2023 for estimator choice), and GPT-5.6 Sol found the estimand claim false for prior and outcome-model specification, which change estimation rather than the estimand.
Revised by the full-text reading. Applied 2026-07-26 full-text reading of the consolidated library (687 papers).
- closure-upheld
- 7 findings from the reading assert this problem is open; the closure was re-examined and stands Put to an independent reviewer with the closing auditors’ reasoning, the work they relied on, and the new findings in view together. Every new finding is an isolated case study, correspondence dispute, or observational comparison showing that one or more modeling choices can change conclusions. That is already conceded by the closure, which specifically recognizes evidence for target-population and estimator-choice effects while retaining the absence of a coherent factorial study ranked by decision reversals. The 2026 paper postdates the closure but does not supersede its factual basis because it neither crosses the four factors nor estimates their relative frequency or interaction. The new papers reinforce the motivation for the proposed study, not a claim outside the closure’s residual. (reviewer confidence: high)
- Flagged for reopening
-
9 findings from the reading assert this problem is open while the registry carries
overstatedFlagged, not applied: reversing an earlier audit’s closure needs the closure’s own evidence re-examined alongside this, which the reading did not do. See documentation/audit/reading/PROPOSED.md. - Literature and prior-art check
- open / overstated. No factorial study crossing bridge type, prior, target population and outcome model and scoring decision reversals exists, so the proposed direction is genuinely unoccupied. But ‘has never been quantified’ is too strong for one of the four factors. Jiang et al. 2025 (Res Synth Methods, doi 10.1017/rsm.2025.10, full text read) constructs an explicit ‘MAIC paradox’ in which the comparative-effectiveness conclusion reverses purely by switching which trial supplies IPD, i.e. by changing the target population, and notes NICE DSU TSD 18 already documented a real secukinumab/adalimumab case where two MAICs reached opposing conclusions. Cassidy et al. 2023 (doi 10.1016/j.jval.2023.07.001) empirically compared 80 MAIC estimates against the corresponding standard anchored ITCs and found MAICs systematically favor the IPD-holder’s treatment, quantifying the estimator-choice factor. Phillippo et al. 2018 threshold analysis quantifies how far evidence must move before the recommendation flips, though for data perturbations rather than structural choices. So target population and estimator choice have each been shown to flip or shift conclusions; what is missing is the joint factorial evaluation and the prior and outcome-model arms. Cites 10.1017/rsm.2025.10: Demonstrates that the target-population choice alone can reverse which treatment is judged superior, partially quantifying one of the four factors.. Cites 10.1016/j.jval.2023.07.001: Empirically quantifies the estimator-choice factor across 80 comparisons from 17 publications..
- GPT-5.6 Sol, technical and source-code lens
- not-a-problem / overstated. The proposed sensitivity exercise is feasible using ordinary scenario, posterior-predictive, or factorial simulation, so no new statistical identification principle is missing. More importantly, C3 is false as a blanket statement. Changing the target population changes the estimand; changing a bridge usually changes the identifying assumptions or information path; changing a prior changes regularization and the posterior, not the estimand; and changing an outcome-model specification ordinarily changes the estimator or model, not the estimand, unless the outcome, horizon, or effect measure also changes. Factors cannot always be crossed independently because some combinations are incoherent. A decision reversal between different target populations may reflect a different decision question rather than sensitivity of one estimand. The claim that this has never been quantified is an uncited literature claim outside the technical evidence supplied.
- Decision path
-
R5:default-open → R8:overstated-majority(n=2)
How the evidence falls in time
9 papers in the reviewed corpus bear on this problem, published 2014 to 2026. Partial progress is on the record, and later work still calls this open. Both belong in view.
This problem is recurrent: three or more papers spanning at least eight years assert it independently. Sustained restatement by authors who mostly do not cite each other is the strongest evidence this reading can offer that a gap is real rather than one group’s framing.
It is also being closed in pieces: partial results come from more than one paper in more than one year, each covering a different part.
| Year | Says | Paper |
|---|---|---|
| 2014 | partly addresses it | The impact of incorporating Bayesian network meta-analysis in cost-effectiveness analysis - a case study of pharmacotherapies for moderate to severe COPD |
| 2015 | confirms it is open | Accounting for Heterogeneity in Relative Treatment Effects for Use in Cost-Effectiveness Models and Value-of-Information Analyses |
| 2017 | confirms it is open | Pharmacological interventions for alcoholic liver disease (alcohol-related liver disease): an attempted network meta-analysis |
| 2022 | confirms it is open | Challenges of modelling approaches for network meta-analysis of time-to-event outcomes in the presence of non-proportional hazards to aid decision making: Application to a melanoma network |
| 2022 | confirms it is open | The Value of Indirect Comparisons of Systemic Biologics for Psoriasis: Interpretation of Efficacy Findings |
| 2024 | confirms it is open | Letter to the Editor Regarding ‘Matching-Adjusted Indirect Comparison of the Long-Term Efficacy Maintenance and Adverse Event Rates of Lebrikizumab Versus Dupilumab in Moderate-to-Severe Atopic Dermatitis’ |
| 2025 | partly addresses it | Methods for information-sharing in network meta-analysis: Implications for inference and policy |
| 2025 | confirms it is open | Letter to the Editor Regarding “Dupilumab Versus Lebrikizumab Demonstrates Greater Likelihood of Achieving and Maintaining Improvements in Efficacy Outcomes Using a Placebo Adjusted Indirect Treatment Comparison” |
| 2026 | confirms it is open | Population-adjusted network meta-analyses provide new insights into the efficacy of treatment alternatives for metastatic castration-sensitive prostate cancer |
Source
Derived from C:L1784 of the reviewed corpus.