Evidence-set assembly can favor the sponsor before modeling begins
Statement
Eligibility and comparator decisions taken while assembling the evidence base can materially change an indirect comparison by changing which studies, populations and contrasts are represented, and this is demonstrated rather than hypothetical: a multiverse of 9216 indirect comparison meta-analyses of nalmefene versus naltrexone reported a Janus effect (Palpacuer et al. 2019, doi:10.1186/s12916-019-1409-3), and a multiverse over all possible antidepressant networks found 172 of 231 pairwise comparisons showing opposite effects between the 1st and 99th percentiles (Vinatier et al. 2024, doi:10.1136/bmjebm-2024-112848). What remains open is that multiverse reporting is not required or routinely performed in sponsor submissions, and there is no accepted criterion for which alternative networks count as defensible. Selection changes how relevant the assembled evidence is to an externally defined target population; it does not by itself define the target estimand. The same discretion operates on the boundary of the network and not only on which studies pass eligibility. Admitting a non-focal treatment can connect the network, sharpen the focal contrasts and stabilize the heterogeneity estimate while importing populations whose effect-modifier distributions threaten transitivity, and because a rank is defined relative to the treatments in the network, scope is part of the estimand rather than a nuisance choice. No criterion prices that trade-off in a common currency, the search and extraction cost of extension falls on the reviewer rather than on the decision maker, and only 40 percent of 365 published drug network meta-analyses reported any objective criterion for which drugs or classes were admitted. Ranking output is affected more sharply than effect estimates, because SUCRA and P-score are functionals of the whole admitted node set, so admitting regimens that no decision-maker can choose shifts every other treatment’s position in the hierarchy. Nothing in current reporting requires the node-admission rule to be stated, requires the hierarchy to be recomputed over a decision-relevant set of available and viable regimens, or reports any measure of how far a treatment’s rank depends on inadmissible nodes.
Why it is open
The evidence set is assembled by the party that reports the result, and every downstream analysis conditions on the assembled network, so no covariate adjustment recovers the comparisons that were excluded. The tools to expose this now exist and their published applications show inclusion choices can flip the direction of an estimate, but nothing obliges a sponsor submission to report one, and the demonstrations to date sit in the general synthesis literature rather than in HTA practice. An undefined multiverse over all defensible networks also reintroduces analyst discretion and multiplicity, so the missing piece is a transparent rule for what counts as defensible rather than the technique itself.
What has been tried
| Work | What it contributes |
|---|---|
| Vinatier et al. 2024, BMJ Evidence-Based Medicine | Multiverse analysis covering all possible network meta-analyses over treatment combinations across 522 antidepressant trials and 116,477 patients, quantifying how far conclusions move with the assembly step. |
| Palpacuer et al. 2019, BMC Medicine | Enumerates 9216 indirect comparison meta-analyses of nalmefene versus naltrexone by varying eligibility criteria and analytic model, showing that inclusion decisions alone can reverse the direction of the comparison. |
| Guelimi et al. 2025, BMC Medical Research Methodology | Quantifies how inclusion criteria and other methodological choices change network meta-analysis conclusions in psoriasis; shows sensitivity over alternative networks is operational. |
| Tonin et al. 2018, PLOS ONE | Maps 365 drug network meta-analyses and finds that fewer than half, 146 or 40 percent, reported objective criteria for selecting the drugs or classes included, while 87 or 23.8 percent gave explicitly non-objective reasons such as most commonly used drugs, with no improvement over the period studied. Supplies the base rate for undisclosed node-admission rules. |
| Batson et al. 2016, PLOS ONE | Extends an atrial fibrillation network from 4 trials to 19 trials and 15 treatments solely to make covariate and baseline-risk models fittable, finds no interaction signal, and states that there is an unmet practical need to know how far to extend a network in the search for indirect evidence in health technology assessment. |
| Caldwell et al. 2015, Value in Health | Supplies the precision half of the scope decision and concludes that reviewers should include only those additional non-focal treatments compared with at least two focal treatments. Names value-of-information analysis as the tool for the cost and bias halves without applying it, and leaves the bias consequence of extension as an important question to address. |
| Caldwell 2014, Systematic Reviews | Overview of conducting systematic reviews with network meta-analysis, which records that network size has been called an unsolved issue in network meta-analysis and an area of developing interest, and reduces practice to prespecifying a population, intervention, comparator and outcome plus an extension strategy. |
| Babul et al. 2026, Cureus | Editorial examining how dose and regimen nodes were admitted into three alopecia areata evidence syntheses of oral JAK inhibitors. In a network of placebo against 22 active interventions, only six corresponded to approved drugs at approved doses, ritlecitinib entered at five regimens although one is approved, and deuruxolitinib 12 mg twice daily, never approved and placed under an FDA clinical hold in May 2023 for unfavorable benefit-risk, was reported in the abstract as the most efficacious intervention for both the primary and the aspirational endpoint. Its remedy is procedural: prespecify exclusion of doses never advanced to pivotal trials or analyze them only in prespecified sensitivity analyses, and report approval status and regulatory actions alongside pooled estimates. |
Probable solution or research direction
The multiverse machinery exists and the work is to require and discipline it rather than to invent it: specify transparent rules for which alternative evidence sets are defensible, then report the estimate across them as a matter of course. Protocol-first feasibility assessment that fixes eligibility before results are known addresses the outcome-aware part of the problem that a multiverse cannot, since a multiverse run after the fact still starts from a chosen frame. Applying target-trial emulation to the construction of the evidence set rather than to a single analysis remains unevidenced; the literature auditor could not verify any published instance and Grok also left it open.
Verification
Verdict. Overstated. 2 independent auditors judged the claim as stated too strong. Multiverse analysis over the assembled evidence set has been demonstrated for indirect comparisons and for network meta-analysis and shows that inclusion decisions can flip the direction of the estimate; what remains open is that it is not routinely required or performed in sponsor submissions, and there is no accepted criterion for which alternative networks count as defensible.
The source said. “Eligibility and comparator decisions taken while assembling the evidence base can induce a favorable network or a favorable target population before any modeling begins, and only one of the defensible networks is ever shown.”
What was wrong with it. The source presented multiverse analysis over defensible networks as an emerging direction and said alternative networks are never reported; the literature auditor found two published multiverses of exactly that object (Palpacuer et al. 2019, doi:10.1186/s12916-019-1409-3; Vinatier et al. 2024, doi:10.1136/bmjebm-2024-112848), with a third for inclusion criteria in psoriasis (Guelimi et al. 2025, doi:10.1186/s12874-025-02558-6). GPT-5.6 Sol separately rejected the conflation of the population represented by the selected studies with an externally defined target population, and struck the universal claims that every inclusion decision is outcome-aware and that alternatives are never shown.
Dissent. grok dissented (supported-with-caveat): The claim that eligibility and comparator choices can favor a network or target population before modeling, and that only one defensible network is typically shown, remains faithful for HTA/ITC practice. Recent work does, however, implement the multiverse response for treatment/network assembly in evidence synthesis, so the problem is not open as if no method existed. Vinatier et al. (2024) ran a multiverse covering all possible NMAs on different treatment combinations in antidepressants and showed large vibration of effects, especially for indirect comparisons. Guelimi et al. (2025) quantified how inclusion criteria and other methodological choices change NMA conclusions in psoriasis and explicitly contrasted their design with prior all-possible-networks work. These papers substantiate C1/C4 as live, demonstrated issues and give a usable multiverse template. They do not solve sponsor-side selective assembly, protocol-first eligibility fixed before knowing results, or target-trial emulation applied to the evidence set itself (C3); those remain open. RoB-NMA (2025) and PRISMA-NMA update work improve transparency of synthesis bias but do not replace multiverse reporting of defensible networks.
Revised by the full-text reading. Applied full-text reading.
- merged-in
- absorbed a proposal the reading raised separately: No decision framework for network scope and treatment admission 4 citation(s) added
- merged-in
- absorbed a proposal the reading raised separately: Treatment rankings lack a decision-relevant node-admission rule 1 citation(s) added
- closure-upheld
- 46 findings from the reading assert this problem is open; the closure was re-examined and stands Put to an independent reviewer with the closing auditors’ reasoning, the work they relied on, and the new findings in view together. The new findings repeatedly document that evidence-set and comparator choices can change results, sometimes in sponsor-authored analyses, but that mechanism was already conceded by the closure. The closure also expressly retained the practical residual that multiverse reporting is not routinely required and that no accepted rule defines defensible alternative networks. The 2026 Pompe analyses are strong applied examples, but they show adoption and governance gaps rather than contradicting Palpacuer or Vinatier’s demonstrations that the sensitivity method exists. None establishes that the cited methods fail to apply or that a principled defensibility criterion has since displaced the closure’s factual basis. (reviewer confidence: high)
- Flagged for reopening
-
53 findings from the reading assert this problem is open while the registry carries
overstatedFlagged, not applied: reversing an earlier audit’s closure needs the closure’s own evidence re-examined alongside this, which the reading did not do. See documentation/audit/reading/PROPOSED.md. - Literature and prior-art check
- partially-solved / overstated. I retrieved two published multiverse analyses of exactly the object this entry proposes as future work, both in indirect comparison settings. Palpacuer 2019 BMC Medicine enumerated 9216 indirect comparison meta-analyses of nalmefene vs naltrexone by varying eligibility (condition, abstinence, sex, comorbidity, psychological support, dose, duration, outcome, publication status, risk of bias) and analytic model, and reported a ‘Janus effect’. Vinatier 2024 BMJ EBM ran the multiverse over network composition itself: all possible NMAs over treatment combinations across 522 antidepressant trials and 116,477 patients. Claim 1 and claim 2 are therefore not merely plausible but empirically demonstrated. Claim 4, however, is written as if multiverse analysis over defensible networks were a promising future direction (‘Emerging’ maturity); it has been executed at scale and published in general medical journals. The why_open sentence ‘Defensible alternative networks usually exist and are never reported’ is contradicted by these two papers. Claim 3 (target-trial emulation applied to evidence-set construction) I could not verify; the only candidate I found was in an obscure venue I will not cite. Cites doi:10.1136/bmjebm-2024-112848: Executes the entry’s proposed direction: a multiverse over all defensible network compositions, quantifying how far conclusions move with the assembly step.. Cites doi:10.1186/s12916-019-1409-3: Demonstrates claims 1 and 2 for an indirect comparison specifically, varying eligibility criteria that define both the network and the target population..
- GPT-5.6 Sol, technical and source-code lens
- open / overstated. Evidence-set selection can materially change network connectivity, available contrasts, precision, and conclusions, so C1 and the underlying robustness problem are valid. C2 conflates the population represented by the selected studies with the externally defined target population; study selection changes evidential relevance to a target but should not itself define the target estimand. Target-trial principles can help prespecify eligibility and estimand attributes, but calling evidence-set assembly target-trial emulation does not recover excluded information or identify selection effects. C4 is a reasonable sensitivity strategy if defensible networks are specified by transparent rules, although an undefined multiverse of all defensible networks invites further researcher discretion and multiplicity. The assertions that every inclusion decision is outcome-aware and that alternatives are usually or never reported are unsupported universal claims. This can be critical when eligibility was selected after results were known, but it is not automatically a modeling failure in every ITC.
- Grok 4.5, recency lens
- partially-solved / supported-with-caveat. The claim that eligibility and comparator choices can favor a network or target population before modeling, and that only one defensible network is typically shown, remains faithful for HTA/ITC practice. Recent work does, however, implement the multiverse response for treatment/network assembly in evidence synthesis, so the problem is not open as if no method existed. Vinatier et al. (2024) ran a multiverse covering all possible NMAs on different treatment combinations in antidepressants and showed large vibration of effects, especially for indirect comparisons. Guelimi et al. (2025) quantified how inclusion criteria and other methodological choices change NMA conclusions in psoriasis and explicitly contrasted their design with prior all-possible-networks work. These papers substantiate C1/C4 as live, demonstrated issues and give a usable multiverse template. They do not solve sponsor-side selective assembly, protocol-first eligibility fixed before knowing results, or target-trial emulation applied to the evidence set itself (C3); those remain open. RoB-NMA (2025) and PRISMA-NMA update work improve transparency of synthesis bias but do not replace multiverse reporting of defensible networks. Cites doi:10.1136/bmjebm-2024-112848: Demonstrates multiverse/all-possible-NMA analysis over treatment selection and shows large vibration of effects; partially resolves the proposed multiverse response to selective network assembly (C4) for treatment inclusion.. Cites doi:10.1186/s12874-025-02558-6: Shows inclusion criteria and methodological choices alter NMA conclusions; supports that assembly decisions matter before modeling (C1) and that sensitivity over alternative networks is operational..
- Decision path
-
R3:partially-solved(n=2) → R8:overstated-majority(n=2)
How the evidence falls in time
52 papers in the reviewed corpus bear on this problem, published 2010 to 2026. Partial progress is on the record, and later work still calls this open. Both belong in view.
This problem is recurrent: three or more papers spanning at least eight years assert it independently. Sustained restatement by authors who mostly do not cite each other is the strongest evidence this reading can offer that a gap is real rather than one group’s framing.
It is also being closed in pieces: partial results come from more than one paper in more than one year, each covering a different part.
| Year | Says | Paper |
|---|---|---|
| 2010 | confirms it is open | Network meta-analysis on the log-hazard scale, combining count and hazard ratio statistics accounting for multi-arm trials: a tutorial |
| 2011 | confirms it is open | Conducting indirect-treatment-comparison and network-meta-analysis studies: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 2 |
| 2011 | confirms it is open | Interpreting indirect treatment comparisons and network meta-analysis for health-care decision making: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 1 |
| 2011 | confirms it is open | Network meta-analysis-highly attractive but more methodological research is needed |
| 2012 | confirms it is open | Adjustment for reporting bias in network meta-analysis of antidepressant trials |
| 2012 | confirms it is open | A Primer on Network Meta-Analysis for Dental Research |
| 2012 | confirms it is open | Impact of Reporting Bias in Network Meta-Analysis of Antidepressant Placebo-Controlled Trials |
| 2013 | partly addresses it | Evidence synthesis for decision making 7: a reviewer’s checklist |
| 2013 | partly addresses it | The effects of excluding treatments from network meta-analyses: survey |
| 2013 | confirms it is open | Demystifying trial networks and network meta-analysis |
| 2014 | partly addresses it | Industry sponsorship bias in research findings: a network meta-analysis of LDL cholesterol reduction in randomised trials of statins |
| 2014 | confirms it is open | A process for assessing the feasibility of a network meta-analysis: a case study of everolimus in combination with hormonal therapy versus chemotherapy for advanced breast cancer |
| 2014 | confirms it is open | An overview of conducting systematic reviews with network meta-analysis |
| 2014 | confirms it is open | Industry sponsorship bias in research findings: a network meta-analysis of LDL cholesterol reduction in randomised trials of statins |
| 2014 | confirms it is open | The importance of baseline viral load when assessing relative efficacy in treatment-naive HBeAg-positive chronic hepatitis B: a systematic review and network meta-analysis |
| 2015 | partly addresses it | Extending Treatment Networks in Health Technology Assessment: How Far Should We Go? |
| 2015 | confirms it is open | An Assessment of the Methodological Quality of Published Network Meta-Analyses: A Systematic Review |
| 2015 | confirms it is open | From Randomized Controlled Trials of Antidepressant Drugs to the Meta-Analytic Synthesis of Evidence: Methodological Aspects Lead to Discrepant Findings |
| 2015 | confirms it is open | Indirect Comparisons and Network Meta-Analyses: Estimation of Effects in the Absence of Head-to-Head Trials (Part 22 of a Series on Evaluation of Scientific Publications) |
| 2016 | confirms it is open | Live cumulative network meta-analysis: protocol for second-line treatments in advanced non-small-cell lung cancer with wild-type or unknown status for epidermal growth factor receptor |
| 2016 | confirms it is open | Meta-Analysis of a Complex Network of Non-Pharmacological Interventions: The Example of Femoral Neck Fracture |
| 2016 | confirms it is open | Network Meta-analysis for Clinical Practice Guidelines: A Case Study on First-Line Medical Therapies for Primary Open-Angle Glaucoma |
| 2016 | confirms it is open | The Impact of Excluding Trials from Network Meta-Analyses - An Empirical Study |
| 2016 | confirms it is open | Network meta-analyses performed by contracting companies and commissioned by industry |
| 2016 | confirms it is open | The Global Research Collaboration of Network Meta-Analysis: A Social Network Analysis |
| 2017 | confirms it is open | Characteristics and knowledge synthesis approach for 456 network meta-analyses: a scoping review |
| 2017 | confirms it is open | Common pitfalls and mistakes in the set-up, analysis and interpretation of results in network meta-analysis: what clinicians should look for in a published article |
| 2017 | confirms it is open | Network meta-analysis: drawing conclusions regarding trials of psychosocial interventions for bipolar disorder |
| 2017 | confirms it is open | Same family, different species: methodological conduct and quality varies according to purpose for five types of knowledge synthesis |
| 2018 | confirms it is open | Evidence synthesis, practice guidelines and real-world prescriptions of new generation antidepressants in the treatment of depression: a protocol for cumulative network meta-analyses and meta-epidemiological study |
| 2018 | confirms it is open | Rapid methods including network meta-analysis to produce evidence in clinical decision support: a decision analysis |
| 2018 | confirms it is open | The use of single armed observational data to closing the gap in otherwise disconnected evidence networks: a network meta-analysis in multiple myeloma |
| 2019 | confirms it is open | Choice of Methodology Impacts Outcome in Indirect Comparisons of Drugs for Idiopathic Pulmonary Fibrosis |
| 2020 | confirms it is open | A scoping review of network meta-analyses assessing the efficacy and safety of complementary and alternative medicine interventions |
| 2020 | confirms it is open | Assessing the Quality and Coherence of Network Meta-Analyses of Biologics in Plaque Psoriasis: What Does All This Evidence Synthesis Tell Us? |
| 2021 | confirms it is open | Reporting of Cochrane systematic review protocols with network meta-analyses-A scoping review |
| 2022 | partly addresses it | Feasibility study for interactive reporting of network meta-analysis: experiences from the development of the MetaInsight COVID-19 app for stakeholder exploration, re-analysis and sensitivity analysis from living systematic reviews |
| 2022 | confirms it is open | Duplicated network meta-analysis in advanced prostate cancer: a case study and recommendations for change |
| 2022 | confirms it is open | Systemic treatments for psoriasis: not another network meta-analysis! |
| 2023 | partly addresses it | Handling related publications reporting real-world evidence in network meta-analysis: a case study in multiple sclerosis |
| 2023 | confirms it is open | The Acceptance of Indirect Treatment Comparison Methods in Oncology by Health Technology Assessment Agencies in England, France, Germany, Italy, and Spain |
| 2023 | confirms it is open | Diversity when interpreting evidence in network meta-analyses (NMAs) on similar topics: an example case of NMAs on diabetic macular oedema |
| 2023 | confirms it is open | Does type of funding affect reporting in network meta-analysis? A scoping review of network meta-analyses |
| 2024 | confirms it is open | PD39 First Approach For Assessing Statistical Significance In Industry Funded Matching-Adjusted Indirect Comparison Studies: A Scoping Review |
| 2024 | confirms it is open | Comparing the efficacy of cipaglucosidase alfa plus miglustat with other enzyme replacement therapies for late-onset Pompe disease: a network meta-analysis utilizing patient-level and aggregate data |
| 2024 | confirms it is open | Funding and conclusions of network meta-analyses on targeted therapies in inflammatory diseases: an overview |
| 2025 | partly addresses it | Effect of methodological choices and inclusion criteria on network meta-analysis results in psoriasis |
| 2025 | confirms it is open | Letter to the Editor Regarding “Dupilumab Versus Lebrikizumab Demonstrates Greater Likelihood of Achieving and Maintaining Improvements in Efficacy Outcomes Using a Placebo Adjusted Indirect Treatment Comparison” |
| 2025 | confirms it is open | Exploring the Moderating Effect of Control Group Type on Intervention Effectiveness in School-Based Anxiety and Depression Prevention: Findings from a Rapid Review and Network Meta-analysis |
| 2026 | confirms it is open | Comparing the efficacy of cipaglucosidase alfa plus miglustat with alglucosidase alfa for late-onset Pompe disease: an expanded network meta-analysis utilizing patient-level and aggregate data |
| 2026 | confirms it is open | A tale of two meta-analyses: Similarity, consistency and avoiding overinterpretation in network meta-analysis |
| 2026 | confirms it is open | Evaluating search approaches for a network meta-analysis: a case study of Epistemonikos Database of Trials, CENTRAL, and other sources. |
| 2026 | confirms it is open | Evidence Synthesis Gone Awry: The Perils of Aggregating Ineffective or Unsafe Doses in Alopecia Areata Reviews. |
Source
Derived from C:L214-215, C:L216 of the reviewed corpus.