Nothing requires showing how far a conclusion is assumption-driven
Statement
No broadly applicable rule or agreed minimum stress-test set requires quantitative assessment of assumption-driven conclusions for decision-grade indirect comparisons, and EU HTA guidance now recommends shifted-null testing for population-adjusted comparisons without extending a comparable requirement to standard network meta-analysis or Bucher comparisons, and there is no agreed minimum set, so any single sensitivity analysis can be presented as sufficient. The methods themselves are largely available: decision tipping points, proposed by Caldwell et al. 2016 and given an algebraic form by Phillippo et al. 2018, are computable and on CRAN (nmathresh, implementing 10.1111/rssa.12341), quantitative bias analysis for unmeasured confounding in unanchored PAIC was published in 2025 (10.1017/rsm.2025.13) with deterministic, probabilistic and tipping-point variants, and component-drift tipping plus prior sensitivity are implemented in cpaic, an unreleased development package on GitHub that is not on CRAN. What is not computable from current tools is support and effective sample size across a whole bias region, joint evaluation of multiple interacting biases, and interaction-level drift sensitivity. A probability of recommendation reversal also requires a defensible joint distribution for the bias parameters and a fully specified decision rule before it can be reported.
Why it is open
Stress tests and bias analyses can only weaken a submission’s apparent strength, so a submitting party has no incentive to run them unless they are required, and appraisal templates ask for scenario analyses of economic inputs rather than of synthesis assumptions. Because there is no agreed minimum set, any single sensitivity analysis can be presented as sufficient, and the tools that exist are scattered across a CRAN package for network meta-analysis, a 2025 PAIC method paper and an unreleased development package rather than assembled into a standard. Support and effective sample size across a bias region and joint evaluation of interacting biases remain genuinely uncomputed, though support and ESS need re-evaluation only for scenarios that change the target distribution or the weighting problem.
What has been tried
| Work | What it contributes |
|---|---|
| Caldwell et al. 2016, Journal of Clinical Epidemiology | proposes threshold analysis for network meta-analysis and computes, by a two-stage numerical procedure, the smallest changes to the evidence that would change a recommendation; Phillippo et al. credit this paper as the origin of the method and describe their own contribution as replacing the numerical procedure with an algebraic one |
| Phillippo et al. 2018 | Derives decision-invariant bias-adjustment thresholds, the smallest changes to the data that change the treatment decision, which is the tipping-point computation the source described as unavailable. |
| nmathresh 0.1.6 (R package) | Released CRAN implementation of decision-invariant bias-adjustment thresholds and invariant intervals for network meta-analysis, showing tipping-point computation is standard software rather than custom work. |
| Ren et al. 2025 | Quantitative bias analysis for unmeasured confounding in unanchored population-adjusted indirect comparisons, with deterministic and probabilistic variants and tipping-point analysis. |
| Phillippo et al. 2019 (threshold analysis for guideline developers) | Presents threshold analysis to decision makers and guideline developers as a direct statement of how robust a treatment recommendation is to plausible changes in the evidence. |
| cpaic (development package, audited at commit 9d150e9) | Unreleased development package on GitHub, confirmed by an auditor not to be on CRAN. Its bridge_fragility() computes the component-main-effect drift needed to cross a specified decision threshold, with an optional posterior robustness probability, and it provides tighter and looser prior refits reporting contrast movement; it excludes interaction drift, joint biases, general quantitative bias analysis and reversal probability. |
Probable solution or research direction
Define a minimum stress-test set for decision-grade ITC and assemble it from existing components: threshold analysis for the decision tipping point, the published unanchored-PAIC quantitative bias analysis for unmeasured confounding, and drift sensitivity for cross-subnetwork bridges. Require the disclosures that accompany a bias analysis, namely separated bias mechanisms, stated bias parameters with ranges and evidence sources, dependence assumptions, deterministic scenarios alongside probabilistic ones, and labeling of which quantities are data-identified, externally informed, elicited or purely bounded. Develop what is genuinely missing: joint evaluation of interacting biases, interaction-level drift, and support and ESS re-evaluated for the scenarios that change the weighting problem.
Verification
Verdict. Overstated. 2 independent auditors judged the claim as stated too strong. No rule requires a formal quantitative bias analysis or a minimum stress-test set for a decision-grade unanchored analysis, and there is no agreed minimum set, so any single sensitivity analysis can be presented as sufficient. The methods themselves are not all missing: decision tipping points are computable and on CRAN for network meta-analysis (nmathresh), and quantitative bias analysis for unmeasured confounding in unanchored PAIC was published in 2025. What remains uncomputed is support and effective sample size across an entire bias region, joint evaluation of multiple interacting biases, and cross-subnetwork drift sensitivity.
The source said. “The analyses that would show how much of a conclusion rests on assumptions are optional: omitted modifier and prognostic sensitivity, cross-subnetwork drift, component-interaction and target-distribution sensitivity, MNAR missingness, alternative outcome and effect-scale analyses, bridge-deletion or held-out validation, and the decision tipping point with the probability of recommendation reversal. No rule requires a formal quantitative bias analysis for a decision-grade unanchored analysis, nor the disclosures that accompany one: separated bias mechanisms, stated bias parameters with ranges, dependence assumptions and evidence sources, deterministic scenarios alongside probabilistic ones, support, feasibility and ESS across the whole bias region, joint evaluation of multiple biases, tipping surfaces, and labeling of which quantities are data-identified, externally informed, elicited or purely bounded.”
What was wrong with it. The source said tipping surfaces and drift sensitivity do not exist as standard software features. Tipping-point computation has been on CRAN since nmathresh 0.1.6 (2020) and quantitative bias analysis for unanchored PAIC was published in 2025 (10.1017/rsm.2025.13); GPT-5.6 Sol found component-drift tipping already implemented in cpaic and rejected the claim that omitting a bias analysis asserts residual bias is exactly zero, since it only leaves inference conditional on assumptions that fix modeled residual bias at zero.
Revised by the full-text reading. Applied 2026-07-26 full-text reading of the consolidated library (687 papers).
- Citation corrected
- corrected who is credited with threshold analysis Phillippo et al. (doi:10.1111/rssa.12341) state in their own text that ‘Threshold analysis has previously been proposed by Caldwell et al. (2016)’ and that their contribution avoids the limitations of that approach. Caldwell et al. 2016 is now cited as the origin; the later work remains cited for the algebraic procedure. Previously: “No rule requires a formal quantitative bias analysis or a minimum stress-test set for a decision-grade unanchored analysis, and there is no agreed minimum set, so any single sensitivity analysis can be presented as sufficient. The methods themselves are largely available: decision tipping points are computable and on CRAN (nmathresh, implementing 10.1111/rssa.12341), quantitative bias analysis for unmeasured confounding in unanchored PAIC was published in 2025 (10.1017/rsm.2025.13) with deterministic, probabilistic and tipping-point variants, and component-drift tipping plus prior sensitivity”
- scope-narrowed
- statement: the closure stands, but its boundary was drawn in the wrong place A 2026 review (doi:10.3390/jmahp14020030) reports that EU guidance recommends shifted-null hypothesis testing for PAICs to account for unmeasured effect modifiers. That postdates the closure and defeats the categorical form of the claim, though it supplies no agreed minimum set and does not reach other indirect-comparison methods. Replacement wording reviewer; reviewer confidence high. Previously: “No rule requires a formal quantitative bias analysis or a minimum stress-test set for a decision-grade unanchored analysis, and there is no agreed minimum set, so any single sensitivity analysis can be presented as sufficient. The methods themselves are largely available: decision tipping points, proposed by Caldwell et al. 2016 and given an algebraic form by Phillippo et al. 2018, are computable and on CRAN (nmathresh, implementing 10.1111/rssa.12341), quantitative bias analysis for unmeasured confounding in unanchored PAIC was published in 2025 (10.1017/rsm.2025.13) with deterministic, proba”
- Flagged for reopening
-
9 findings from the reading assert this problem is open while the registry carries
overstatedFlagged, not applied: reversing an earlier audit’s closure needs the closure’s own evidence re-examined alongside this, which the reading did not do. See documentation/audit/reading/PROPOSED.md. - Literature and prior-art check
- partially-solved / overstated. The why_open text asserts that ‘several of the required computations do not exist as standard software features, including … tipping surfaces’, and that any single sensitivity analysis can be presented as sufficient. The first half is wrong. Phillippo et al. 2018 (doi 10.1111/rssa.12341) derive decision-invariant bias-adjustment thresholds, ‘the smallest changes to the data that result in a change of treatment decision’, shipped as the CRAN package nmathresh 0.1.6 (2020-06-11): ‘Calculation and presentation of decision-invariant bias adjustment thresholds and intervals for Network Meta-Analysis’. Phillippo et al. 2019 (doi 10.7326/M18-3542) proposed it to NICE guideline developers as an alternative to GRADE, and Holper 2020 (doi 10.1002/jrsm.1393) applied it with GRADE to 21 antidepressants. Separately, Ren et al. 2025 (doi 10.1017/rsm.2025.13, full text read) supplies formal QBA for unmeasured confounding in unanchored PAIC, covering multiple unmeasured covariates, deterministic and probabilistic variants, and tipping-point analysis. What is genuinely open: an agreed minimum stress-test set for PAIC, support and ESS evaluated across a whole QBA region, and joint evaluation of multiple biases. Cites 10.1111/rssa.12341: Provides decision tipping points and the revised recommendation, the exact computation the entry says is unavailable.. Cites cran:nmathresh 0.1.6: Shows tipping-point computation is a released, standard software feature, not custom work.. Cites 10.1017/rsm.2025.13: Supplies a published QBA method specific to unanchored PAIC, with deterministic and probabilistic variants and tipping-point analysis, and argues such analysis is necessary..
- GPT-5.6 Sol, technical and source-code lens
- partially-solved / overstated. C1-C2 are empirical practice claims not established by the supplied material. C3 is only partly right: tipping points are broadly interpretable, but a probability of recommendation reversal is unidentified without a defensible joint distribution for bias parameters and a fully specified decision rule. C4 requires care because changing effect scale generally changes the estimand; alternatives should be clinically meaningful, not presented as interchangeable answers. C5 is useful where deletion leaves the target contrast estimable, but held-out performance cannot validate an unobserved cross-gap assumption. C6 is proportionate for genuinely unanchored decision-grade analyses. C7 is too strong: omitting QBA leaves inference conditional on identifying assumptions that fix modeled residual bias at zero; it does not logically assert that actual residual bias is zero. C8-C10, C13, and C14 are sound transparency requirements. C11 is a category error for bias parameters that do not alter target moments or weights, because support and ESS then do not vary. C12 also overreaches: validation-data uncertainty may already be incorporated into sensitivity-parameter uncertainty, while numerical error should usually be bounded and reduced rather than probabilistically propagated. cpaic already implements component-main-effect drift tipping through bridge_fragility() and prior sensitivity, contradicting the claimed complete absence of drift sensitivity. It does not implement a general QBA, interaction drift, joint biases, or recommendation-reversal probability. Cites choxos/cpaic@9d150e9cad88f2e6fbcd94dbb92c3dee3b7c2dca: Partially resolves cross-subnetwork drift sensitivity and decision tipping by calculating the component-main-effect drift needed to cross a specified decision threshold, with an optional posterior robustness probability at a fixed drift bound. It excludes interaction drift and is not a general QBA.. Cites choxos/cpaic@9d150e9cad88f2e6fbcd94dbb92c3dee3b7c2dca: Provides tighter and looser prior refits and reports contrast movement, partially addressing assumption-driven conclusions and prior dependence..
- Decision path
-
R3:partially-solved(n=2) → R8:overstated-majority(n=2)
How the evidence falls in time
9 papers in the reviewed corpus bear on this problem, published 2016 to 2026. Partial progress is on the record, and later work still calls this open. Both belong in view.
This problem is recurrent: three or more papers spanning at least eight years assert it independently. Sustained restatement by authors who mostly do not cite each other is the strongest evidence this reading can offer that a gap is real rather than one group’s framing.
It is also being closed in pieces: partial results come from more than one paper in more than one year, each covering a different part.
| Year | Says | Paper |
|---|---|---|
| 2016 | partly addresses it | A threshold analysis assessed the credibility of conclusions from network meta-analysis |
| 2016 | confirms it is open | The Impact of Excluding Trials from Network Meta-Analyses - An Empirical Study |
| 2017 | partly addresses it | Sensitivity of treatment recommendations to bias in network meta-analysis |
| 2017 | confirms it is open | Methods for Population-Adjusted Indirect Comparisons in Health Technology Appraisal |
| 2019 | partly addresses it | Threshold Analysis as an Alternative to GRADE for Assessing Confidence in Guideline Recommendations Based on Network Meta-analyses |
| 2021 | confirms it is open | How robust are findings of pairwise and network meta-analysis in the presence of missing participant outcome data? |
| 2024 | confirms it is open | A systematic survey of 200 systematic reviews with network meta-analysis (published 2020-2021) reveals that few reviews report structured evidence summaries |
| 2025 | confirms it is open | Addressing challenges with Matching-Adjusted Indirect Comparisons to demonstrate the comparative effectiveness of entrectinib in metastatic ROS-1 positive Non-Small Cell Lung Cancer |
| 2026 | confirms it is open | Between Rigor and Relevance: Why the EU HTA Guidelines on Indirect Comparisons Miss the Mark |
Source
Derived from C:L2417-2426, C:L2428-2439 of the reviewed corpus.