Trials combined under one outcome label are not one estimand
Evidence-base construction & data access
Very high priority
Partially addressed
Trials grouped under one outcome label may estimate non-equivalent quantities because they differ in follow-up, endpoint definition, analysis population or intercurrent-event handling; simulation of treatment switching in oncology shows that pooling estimates targeting different estimands yields a pooled estimator that targets no estimand of interest, even under a random-effects model (Metcalfe et al. 2025, doi:10.1017/rsm.2025.10039). Estimand mapping is the response and it is no longer a proposal: it has been argued into the evidence-synthesis literature (doi:10.1002/jrsm.1741), given a pragmatic framework with a worked NMA case study varying intercurrent-event strategies (arxiv:2510.15762), and codified as step 1 of a population-adjusted indirect comparison framework (doi:10.1007/s40273-025-01503-1). What remains open is per-trial estimand mapping from published reports, where the needed detail is often not reported, and structured encoding of intercurrent-event handling and analysis population.
Statement
Trials grouped under one outcome label may estimate non-equivalent quantities because they differ in follow-up, endpoint definition, analysis population or intercurrent-event handling; simulation of treatment switching in oncology shows that pooling estimates targeting different estimands yields a pooled estimator that targets no estimand of interest, even under a random-effects model (Metcalfe et al. 2025, doi:10.1017/rsm.2025.10039). Estimand mapping is the response and it is no longer a proposal: it has been argued into the evidence-synthesis literature (doi:10.1002/jrsm.1741), given a pragmatic framework with a worked NMA case study varying intercurrent-event strategies (arxiv:2510.15762), and codified as step 1 of a population-adjusted indirect comparison framework (doi:10.1007/s40273-025-01503-1). What remains open is per-trial estimand mapping from published reports, where the needed detail is often not reported, and structured encoding of intercurrent-event handling and analysis population.
Why it is open
Indirect comparison arithmetic assumes the contrasts refer to a common estimand, but conformity is usually checked at the level of the outcome label rather than the outcome definition. Differences in intercurrent-event strategy and analysis population change the causal question each trial answers, and no covariate adjustment repairs that mismatch. Published reports frequently do not carry the detail needed to do the mapping trial by trial, and uptake of estimand-aware synthesis remains modest, particularly without IPD. An outcome taxonomy exists for classifying outcomes, but nothing machine-readable encodes intercurrent-event strategy or analysis population.
What has been tried
| Work | What it contributes |
|---|---|
| Metcalfe et al. 2025, Research Synthesis Methods | Simulation of treatment switching in oncology showing that pooling treatment-policy and hypothetical estimands produces a pooled estimator targeting no estimand of interest, and that this can mislead even under random effects. |
| Remiro-Azocar and Gorst-Rasmussen 2024, Research Synthesis Methods | Position paper arguing that the language of estimands belongs in network meta-analysis and HTA, especially under the narrow treatment and target-population perspective HTA takes. |
| Remiro-Azocar et al., Incorporating estimands into meta-analyses of clinical trials (preprint) | Pragmatic framework for estimands in meta-analysis and NMA for HTA, applied to an NMA of semaglutide versus dulaglutide contrasting treatment-policy and hypothetical strategies for discontinuation and rescue. |
| Ishak et al. 2025, PharmacoEconomics | Six-element framework for reliable, transparent and reproducible population-adjusted indirect comparisons in which definition of the comparison as an estimand is element 1. |
| Ades et al. 2024, Research Synthesis Methods | Names integration of the estimand framework with evidence synthesis as an explicit agenda item of the core NMA methods group. |
| Analysis of opinions from studies citing the COMET outcomes taxonomy, 2025 | Shows an outcome taxonomy for evidence synthesis already exists and is widely cited, narrowing the gap to estimand-level, machine-readable detail rather than outcome classification as such. |
Probable solution or research direction
The direction is to make the existing estimand frameworks routine rather than to invent them: apply the published meta-analysis and PAIC estimand frameworks to each contributing trial and report the mapping alongside the synthesis, so mismatches are visible before pooling rather than after. Extend outcome taxonomies such as COMET beyond outcome classification to encode intercurrent-event strategy and analysis population in structured form. Latent harmonization and multivariate endpoint models are appropriate only where a defensible linking model connects the endpoints and the available data identify its parameters; they cannot generally reconcile different treatment-policy questions, intercurrent-event strategies, populations or time points.
Verification
Verdict. Partially addressed. Part of this problem is addressed by existing work; the entry names which part.
The source said. “Trials in the same comparison differ in follow-up duration, endpoint definitions, handling of intercurrent events, and analysis populations, so the quantities being combined are not the same estimand even before population adjustment.”
What was wrong with it. The source presented ICH E9(R1)-style estimand mapping and outcome ontologies as promising future directions; the literature auditor and Grok both found an active published programme already doing the mapping (doi:10.1002/jrsm.1741, arxiv:2510.15762, doi:10.1007/s40273-025-01503-1) and a widely used outcome taxonomy already in routine use (COMET, pmid:40754038). GPT-5.6 Sol rejected the claim that endpoint definitions live in prose no tool can read: the obstacle is incomplete reporting and unreliable structured interpretation, not literal unreadability.
- Literature and prior-art check
- partially-solved / supported-with-caveat. prior_work is empty, yet an active published programme addresses exactly claim 2. Metcalfe et al. 2025 Res Synth Methods simulated pooling of treatment-policy and hypothetical estimands across oncology RCTs allowing treatment switching, and showed the pooled estimator targets nothing of interest; that converts the entry’s assertion from conjecture into a quantified result. Remiro-Azocar and Gorst-Rasmussen 2024 RSM argue the estimand language belongs in NMA and HTA; Ades et al. 2024 RSM call for integration of the estimand framework with evidence synthesis; Ishak et al. 2025 PharmacoEconomics already makes estimand definition element 1 of a six-element PAIC framework. Claim 1 is uncontroversial and is documented by the COMET outcome-heterogeneity literature. Claim 3 needs softening: the COMET outcome taxonomy exists and is in routine use for classifying trial outcomes, so an outcome classification standard is not absent, though I found nothing machine-readable that encodes intercurrent-event handling or analysis population. Claim 4 I could not verify with a retrieved source this session. Status is partially-solved: the diagnosis and the framework are published; per-trial estimand mapping from published reports is not. Cites doi:10.1017/rsm.2025.10039: Quantifies the bias from combining trials whose estimands differ, which the entry states only qualitatively. Also confirms the gap is real.. Cites doi:10.1002/jrsm.1741: Claim 2 is already a published position paper in the NMA/HTA literature, not a novel proposed direction.. Cites doi:10.1002/jrsm.1745: Establishes that estimand-framework integration is an explicitly stated agenda item of the core NMA methods group.. Cites doi:10.1007/s40273-025-01503-1: Shows estimand-first definition is already codified as step 1 of a PAIC analysis framework, so the ITC-specific version of claim 2 is partly operationalized.. Cites pmid:40754038: Narrows claim 3: an outcome taxonomy for evidence synthesis already exists and is widely cited, so what is missing is estimand-level, machine-readable detail rather than outcome classification as such..
- GPT-5.6 Sol, technical and source-code lens
- open / overstated. The central estimand warning is correct and can be decision-critical. Differences in follow-up, outcome variable, analysis population, summary measure, or intercurrent-event strategy produce different estimands when they change the causal quantity being estimated; a common outcome label is insufficient. C2 is technically sound because structured estimand mapping exposes these differences before synthesis. C3 describes useful infrastructure, not a mathematical necessity for valid ITC. C4 holds only when a defensible measurement or linking model connects the endpoints and the available data identify its parameters. Latent or multivariate models cannot generally reconcile different treatment-policy questions, intercurrent-event strategies, populations, or time points. The assertion that no tool can read endpoint definitions in prose is false; the problem is incomplete reporting and unreliable structured interpretation, not literal unreadability.
- Grok 4.5, recency lens
- partially-solved / supported-with-caveat. That trials sharing an outcome label often target different estimands (follow-up, endpoint definition, intercurrent events, analysis population) remains correct and is now a central theme in synthesis methodology rather than a neglected gap. Remiro-Azócar and Gorst-Rasmussen (2024) argue for estimand thinking as a parallel to narrow research questions in evidence synthesis. Remiro-Azócar et al. (arXiv:2510.15762, versions through 2026) propose a pragmatic framework for estimands in meta-analyses for HTA, apply it to an NMA of semaglutide versus dulaglutide, and contrast treatment-policy versus hypothetical strategies for discontinuation/rescue. EU HTA quantitative evidence-synthesis guidance (2024) also stresses that a common estimand should apply across studies in a synthesis. These works partially resolve C2: ICH E9(R1)-style mapping is no longer only a vague proposal; it is formalized for synthesis and illustrated. They do not close the problem: authors still note modest uptake, difficulty without IPD, and residual cross-trial misalignment. C3 (machine-readable outcome ontology standards for ITC evidence bases) and C4 (latent outcome harmonization models as deployed standards) were not found as solved; semantic-variability/LLM and specialized outcome ontologies exist outside routine ITC pipelines. Mismatched estimands therefore remain open as an operational constraint; the mapping response is partially developed. Cites doi:10.1002/jrsm.1741: Places estimands at the center of evidence-synthesis question formulation; partially resolves the claim that estimand mapping is only aspirational (C2).. Cites arxiv:2510.15762: Provides a pragmatic estimand framework for meta-analysis/NMA in HTA and a worked NMA case study varying intercurrent-event strategies; partially operationalizes ICH E9(R1)-style mapping for synthesis (C2)..
- Decision path
-
R3:partially-solved(n=2) → R9:substantiated-single-overstatement:codex
How the evidence falls in time
54 papers in the reviewed corpus bear on this problem, published 2011 to 2026. Progress and a fresh assertion of openness land in the same year.
This problem is recurrent: three or more papers spanning at least eight years assert it independently. Sustained restatement by authors who mostly do not cite each other is the strongest evidence this reading can offer that a gap is real rather than one group’s framing.
It is also being closed in pieces: partial results come from more than one paper in more than one year, each covering a different part.
| Year | Says | Paper |
|---|---|---|
| 2011 | confirms it is open | Conducting indirect-treatment-comparison and network-meta-analysis studies: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 2 |
| 2012 | confirms it is open | A systematic review with attempted network meta-analysis of asthma therapy recommended for five to eighteen year olds in GINA steps three and four |
| 2013 | partly addresses it | The use of individual patient-level data (IPD) to quantify the impact of pretreatment predictors of response to treatment in chronic hepatitis B patients |
| 2014 | partly addresses it | A process for assessing the feasibility of a network meta-analysis: a case study of everolimus in combination with hormonal therapy versus chemotherapy for advanced breast cancer |
| 2014 | confirms it is open | Bayesian network meta-analysis suggests a similar effectiveness between a monovalent and a pentavalent rotavirus vaccine: a preliminary report of re-analyses of data from a Cochrane Database Systematic Review |
| 2014 | confirms it is open | Incorporating alternative design clinical trials in network meta-analyses |
| 2015 | confirms it is open | Design, Analysis, and Reporting of Crossover Trials for Inclusion in a Meta-Analysis |
| 2015 | confirms it is open | From Randomized Controlled Trials of Antidepressant Drugs to the Meta-Analytic Synthesis of Evidence: Methodological Aspects Lead to Discrepant Findings |
| 2015 | confirms it is open | Network meta-analysis incorporating randomized controlled trials and non-randomized comparative cohort studies for assessing the safety and effectiveness of medical treatments: challenges and opportunities |
| 2016 | confirms it is open | Review of the Reporting of Survival Analyses within Randomised Controlled Trials and the Implications for Meta-Analysis |
| 2017 | confirms it is open | 5-Aminosalicylates to maintain remission in Crohn’s disease: Interpreting conflicting systematic review evidence |
| 2017 | confirms it is open | Overcoming the Straw Man Effect in Oncology: Visualization and Ranking of Chemotherapy Regimens Using an Information Theoretic Approach |
| 2017 | confirms it is open | Pharmacological interventions for non-alcohol related fatty liver disease (NAFLD): an attempted network meta-analysis |
| 2018 | partly addresses it | Different Methods for Modelling Severe Hypoglycaemic Events: Implications for Effectiveness, Costs and Health Utilities |
| 2018 | partly addresses it | An innovative and comprehensive technique to evaluate different measures of medication adherence: The network meta-analysis |
| 2018 | partly addresses it | The number needed to treat and relevant between-trial comparisons of competing interventions |
| 2018 | confirms it is open | Dismantling cognitive-behaviour therapy for panic disorder: a systematic review and component network meta-analysis |
| 2018 | confirms it is open | Network meta-analysis: users’ guide for pediatricians |
| 2019 | partly addresses it | Shared parameter model for competing risks and different data summaries in meta-analysis: Implications for common and rare outcomes |
| 2019 | partly addresses it | Treatment sequence network meta-analysis in Crohn’s disease: a methodological case study |
| 2019 | confirms it is open | Choice of Methodology Impacts Outcome in Indirect Comparisons of Drugs for Idiopathic Pulmonary Fibrosis |
| 2019 | confirms it is open | Considering the methodological limitations in the evidence base of antidepressants for depression: a reanalysis of a network meta-analysis |
| 2019 | confirms it is open | Modelling time-course relationships with multiple treatments: Model-based network meta-analysis for continuous summary outcomes |
| 2019 | confirms it is open | Selective reporting bias in randomised controlled trials from two network meta-analyses: comparison of clinical trial registrations and their respective publications |
| 2019 | confirms it is open | Using Matching-Adjusted Indirect Comparisons and Network Meta-analyses to Compare Efficacy of Brexanolone Injection with Selective Serotonin Reuptake Inhibitors for Treating Postpartum Depression |
| 2020 | partly addresses it | Comparative effectiveness of biological medicines in rheumatoid arthritis: systematic review and network meta-analysis including aggregate results from reanalysed individual patient data |
| 2020 | confirms it is open | Summarising salient information on historical controls: A structured assessment of validity and comparability across studies |
| 2020 | confirms it is open | A Review of Two Regulatory Approved Anti-CD19 CAR T-Cell Therapies in Diffuse Large B-Cell Lymphoma: Why Are Indirect Treatment Comparisons Not Feasible? |
| 2020 | confirms it is open | Evidence synthesis in pulmonary arterial hypertension: a systematic review and critical appraisal |
| 2021 | confirms it is open | Individual Participant Data Meta-Analysis: A Handbook for Healthcare Research (Riley, Tierney, Stewart, eds.; Wiley Series in Statistics in Practice, 2021) |
| 2021 | confirms it is open | Methodological approaches for analysing data from therapeutic efficacy studies |
| 2022 | confirms it is open | A Response to: Letter to the Editor Regarding “Network Meta-analysis of Food and Drug Administration-approved Treatment Options for Adults with Aquaporin-4 Immunoglobulin G-positive Neuromyelitis Optica Spectrum Disorder” |
| 2022 | confirms it is open | Letter to the Editor Regarding “Network Meta-analysis of Food and Drug Administration-approved Treatment Options for Adults with Aquaporin-4 Immunoglobulin G-positive Neuromyelitis Optica Spectrum Disorder” |
| 2022 | confirms it is open | Duplicated network meta-analysis in advanced prostate cancer: a case study and recommendations for change |
| 2022 | confirms it is open | Feasibility Study of a Network Meta-Analysis and Unanchored Population-Adjusted Indirect Treatment Comparison of Niraparib, Olaparib, and Bevacizumab as Maintenance Therapies in Patients with Newly Diagnosed Advanced Ovarian Cancer |
| 2022 | confirms it is open | Has network meta-analysis resolved the controversies related to bowel preparation in elective colorectal surgery? |
| 2022 | confirms it is open | Integrating Systematic Reviews into Supportive Care Trial Design: The Rethinking Clinical Trials (REaCT) Program |
| 2022 | confirms it is open | Prevention of venous thromboembolic events in patients with lower leg immobilization after trauma: Systematic review and network meta-analysis with meta-epidemiological approach |
| 2022 | confirms it is open | The Value of Indirect Comparisons of Systemic Biologics for Psoriasis: Interpretation of Efficacy Findings |
| 2023 | confirms it is open | Physical Prehabilitation in Patients who Underwent Major Abdominal Surgery: A Comprehensive Systematic Review and Component Network Meta-Analysis Using GRADE and CINeMA Approach |
| 2023 | confirms it is open | Immunonutrition in patients who underwent major abdominal surgery: A comprehensive systematic review and component network metanalysis using GRADE and CINeMA approaches |
| 2023 | confirms it is open | Are indirect comparisons for treatments in migraine necessitas? Many inevitable challenges to overcome |
| 2023 | confirms it is open | A Critical Appraisal of Matching-Adjusted Indirect Comparisons in Spinal Muscular Atrophy |
| 2023 | confirms it is open | Methods to appraise available evidence and adequacy of data from a systematic literature review to conduct a robust network meta-analysis of treatment options for patients with hospital-acquired or ventilator-associated bacterial pneumonia |
| 2024 | confirms it is open | Interventions to prevent the onset of frailty in adults aged 60 and older (PRAE-Frail): a systematic review and network meta-analysis |
| 2024 | confirms it is open | Letter to the Editor Regarding ‘Matching-Adjusted Indirect Comparison of the Long-Term Efficacy Maintenance and Adverse Event Rates of Lebrikizumab Versus Dupilumab in Moderate-to-Severe Atopic Dermatitis’ |
| 2024 | confirms it is open | Broad versus narrow research questions in evidence synthesis: A parallel to (and plea for) estimands |
| 2024 | confirms it is open | Methodology for the selection and evaluation of outcomes for Chinese herbal injection in acute exacerbation of chronic obstructive pulmonary disease (AECOPD): a comprehensive study |
| 2025 | confirms it is open | Evaluating the feasibility of a network meta-analysis comparing treatment options in polycythemia vera |
| 2025 | confirms it is open | Comparison of Diagnostic Bronchoscopy Single-Arm Trials Using an Advanced Optimization Approach for Indirect Comparisons with Partial Individual Patient Data |
| 2025 | confirms it is open | The hazards of using hazard ratios from proportional hazard models in indirect treatment comparisons |
| 2025 | confirms it is open | Serious Infection Risk with Systemic Treatments for Psoriasis: A Systematic Review and Network Meta-analysis Combining Randomised and Non-randomised Evidence |
| 2026 | partly addresses it | Model-based network meta-analysis: Joint estimation of dose-response and time-course relationships. |
| 2026 | confirms it is open | Considerations for assessing the feasibility of network meta-analysis of seasonal vaccines |
Source
Derived from C:L214-215, C:L218 of the reviewed corpus.