Estimand-first guidance exists but is not enforced in PAIC practice
Source note. This entry derives in part from a source document whose inline citations were ChatGPT interface tokens rather than references, and so resolve to nothing. Claims below are attributed to that document and were checked independently where possible; anything that could not be independently sourced is marked in the verification trail.
Statement
Published PAIC practice is dominated by MAIC and by unanchored designs, with poor reporting of covariate handling and diagnostics: 89% (16/18) of NICE appraisals using population adjustment used MAIC and 89% were unanchored (10.1017/S0266462319000333), 72% of 117 oncology MAICs were unanchored with only 3 meeting all NICE recommendations (10.1016/j.vhri.2025.101088), and quality or bias of individual studies was assessed in 9.3% of PAIC analyses with outcome-model fitting adequately reported in 3 of 18 (10.1002/jrsm.1653). That this pattern is caused by choosing the estimator before the estimand is an inference from those figures rather than a measured finding, and the eight-step sequence is the source’s own construction, not an externally validated standard. Guidance already implements part of the ordering: NICE DSU TSD 18 makes connectedness of the randomized evidence determine whether anchored or unanchored adjustment is admissible and requires the decision target population to be stated. The deficit is therefore adoption and enforcement of rules that exist, not an absent workflow.
Why it is open
Software, guidance documents and submission templates are organized around methods rather than estimands, so an estimator can be fixed before the decision question is stated and the target population inferred afterwards from whatever data are available. Nothing in review or reimbursement requires the earlier steps to be documented, so they can be skipped without consequence, and because covariate handling and diagnostics are poorly reported the drift is invisible in the published record and not self-correcting. The existing rule that connected evidence permits only anchored adjustment constrains the admissible estimator set; what is absent is any mechanism that checks it was applied.
What has been tried
| Work | What it contributes |
|---|---|
| Truong et al. 2023 | Methodological systematic review of PAICs; sources MAIC dominance and quantifies poor covariate, diagnostic and bias reporting (quality or bias assessed in 9.3%; model fitting adequately reported in 3 of 18). |
| Farinasso et al. 2025 | Sources the frequency of unanchored designs (72% of 117 oncology MAICs) and the poverty of covariate-handling and diagnostic reporting, with only 3 studies meeting all NICE recommendations. |
| Phillippo et al. 2019 (NICE technology appraisals review) | Independent NICE-submission figures: 89% (16/18) used MAIC and 89% were unanchored. |
| NICE DSU Technical Support Document 18 (2016) | Ties admissible population-adjustment methods to connectedness of the randomized evidence and requires an explicit decision target population, implementing part of the proposed ordering. Read by the code auditor from a document corpus at the given commit. |
Probable solution or research direction
Enforce the ordering that guidance already implies: require the decision target, estimand, evidence graph, identification conditions and support assessment to be documented before the estimator is named, with the admissible estimator set tied to the evidence structure as TSD 18 already does for anchored versus unanchored adjustment. Make that documentation a mandatory reporting appendix so that skipping a step becomes visible rather than silent. Measure compliance empirically once the requirement exists, since current compliance is known only from retrospective reviews.
Verification
Verdict. Unverifiable. Auditors disagreed and the disagreement was evidence-bearing on both sides.
The source titled this. “Estimator is chosen before the estimand and the evidence structure” The title asserts a causal claim about how estimators are chosen, which the audit found to be an inference from prevalence figures rather than a measured finding.
The source said. “Applied practice selects an estimator first: most published PAICs still use MAIC, unanchored designs are common, and reporting of covariate handling, diagnostics and bias assessment is poor. The eight-step sequence that would begin with the target decision and estimand and end with evaluation of decision consequences remains a recommendation rather than the operational default.”
What was wrong with it. The source stated the practice pattern without attribution; the literature auditor sourced every figure (Phillippo et al. 2019, Farinasso et al. 2025, Truong et al. 2023) while GPT-5.6 Sol held the prevalence claims were not established by the material supplied to it, so the figures now appear with their locators. GPT-5.6 Sol also found the sequence is not wholly missing, since TSD 18 already ties admissible methods to connectedness, and rejected the claim that only a small subset of estimators stays credible when assumptions fail, since robustness is assumption-specific.
Dissent. literature dissented (open): Every empirical claim in this entry checks out against retrieved reviews, and each has an exact figure. Jiang et al. 2025 (doi 10.1017/rsm.2025.10, full text read) reports ‘In a recent methodological systematic review, 88.9% (144 out of 162) of PAIC studies used MAICs’, citing Truong et al. 2023; Phillippo et al. 2019 found MAIC in 89% (16/18) of NICE TAs using population adjustment. On unanchored designs: Farinasso et al. 2025 found 72% of 117 oncology MAICs unanchored; Phillippo et al. 2019 found 89% (16/18). On poor reporting: Truong et al. 2023 found extensive heterogeneity assessment in only 37.0% of analyses and quality/bias assessment of individual studies in only 9.3%; Farinasso et al. found only 3/117 MAICs met all NICE recommendations, with the least-reported items being adjustment for all effect modifiers and prognostic variables in unanchored analyses, evidence of effect-modifier status, and the distribution of weights. The eight-step workflow and its ordering are the report’s own construction and cannot be validated against an external source; treat them as a proposal, not a finding. The gap is a practice and enforcement gap and is unresolved. codex dissented (overstated): The proposed order is methodologically sound, but it is not a wholly missing technical workflow. Existing PAIC guidance already makes evidence connectedness determine whether anchored or unanchored adjustment is admissible and requires an explicit target population. C2-C4 are current-practice prevalence claims for which no current source is supplied. C5 is undefined and technically misleading: robustness is assumption-specific, and no estimator remains credible after its required identification assumptions fail. C6 and C7 correctly describe the proposed eight-step sequence. C8 is appropriate for consequential untestable assumptions in a decision-grade analysis, although sensitivity analysis cannot validate those assumptions. C9 is appropriate when the stated aim is reimbursement decision support, not as a universal requirement for every statistical methods paper. The remaining issue is adoption and enforcement rather than absence of a statistical method.
- Literature and prior-art check
- open / supported. Every empirical claim in this entry checks out against retrieved reviews, and each has an exact figure. Jiang et al. 2025 (doi 10.1017/rsm.2025.10, full text read) reports ‘In a recent methodological systematic review, 88.9% (144 out of 162) of PAIC studies used MAICs’, citing Truong et al. 2023; Phillippo et al. 2019 found MAIC in 89% (16/18) of NICE TAs using population adjustment. On unanchored designs: Farinasso et al. 2025 found 72% of 117 oncology MAICs unanchored; Phillippo et al. 2019 found 89% (16/18). On poor reporting: Truong et al. 2023 found extensive heterogeneity assessment in only 37.0% of analyses and quality/bias assessment of individual studies in only 9.3%; Farinasso et al. found only 3/117 MAICs met all NICE recommendations, with the least-reported items being adjustment for all effect modifiers and prognostic variables in unanchored analyses, evidence of effect-modifier status, and the distribution of weights. The eight-step workflow and its ordering are the report’s own construction and cannot be validated against an external source; treat them as a proposal, not a finding. The gap is a practice and enforcement gap and is unresolved. Cites 10.1002/jrsm.1653: Sources MAIC dominance (88.9%, 144/162 per Jiang et al.’s reading of it) and quantifies poor covariate/diagnostic/bias reporting.. Cites 10.1016/j.vhri.2025.101088: Sources the frequency of unanchored designs and the poverty of covariate-handling and diagnostic reporting.. Cites 10.1017/S0266462319000333: Independent NICE-submission figures for MAIC dominance and unanchored prevalence..
- GPT-5.6 Sol, technical and source-code lens
- partially-solved / overstated. The proposed order is methodologically sound, but it is not a wholly missing technical workflow. Existing PAIC guidance already makes evidence connectedness determine whether anchored or unanchored adjustment is admissible and requires an explicit target population. C2-C4 are current-practice prevalence claims for which no current source is supplied. C5 is undefined and technically misleading: robustness is assumption-specific, and no estimator remains credible after its required identification assumptions fail. C6 and C7 correctly describe the proposed eight-step sequence. C8 is appropriate for consequential untestable assumptions in a decision-grade analysis, although sensitivity analysis cannot validate those assumptions. C9 is appropriate when the stated aim is reimbursement decision support, not as a universal requirement for every statistical methods paper. The remaining issue is adoption and enforcement rather than absence of a statistical method. Cites choxos/NICE-DSU-TSD-RAG@75abcb9c226a9f8fa7a0a4ffeba42874f1712ecb: Already ties admissible population-adjustment methods to connectedness of the randomized evidence and requires an explicit decision target population..
- Decision path
-
R3:partially-solved(n=1) → R7:counter-and-support-both-present
How the evidence falls in time
6 papers in the reviewed corpus bear on this problem, published 2017 to 2024. Every paper found here asserts the problem is open; none reports progress.
| Year | Says | Paper |
|---|---|---|
| 2017 | confirms it is open | Methods for Population-Adjusted Indirect Comparisons in Health Technology Appraisal |
| 2019 | confirms it is open | Population Adjustment Methods for Indirect Comparisons: A Review of National Institute for Health and Care Excellence Technology Appraisals |
| 2021 | confirms it is open | Do reporting guidelines have an impact? Empirical assessment of changes in reporting before and after the PRISMA extension statement for network meta-analysis |
| 2022 | confirms it is open | Letter to the Editor Regarding “Network Meta-analysis of Food and Drug Administration-approved Treatment Options for Adults with Aquaporin-4 Immunoglobulin G-positive Neuromyelitis Optica Spectrum Disorder” |
| 2023 | confirms it is open | Comparison of indirect treatment methods in migraine prevention to address differences in mode of administration |
| 2024 | confirms it is open | Which clinical trial designs and statistical approaches have been used in assessments of orphan maintenance by the European Medicines Agency between 2012 and 2022? A cross-sectional study |
Source
Derived from A:L5, A:L19, A:L54, C:L274-283 of the reviewed corpus.