No PAIC-specific reporting standard or credibility appraisal tool
Statement
There is no risk-of-bias or credibility scoring instrument built for population-adjusted indirect comparisons. Farinasso et al. 2025 (10.1016/j.vhri.2025.101088) state this and find only 3 of 117 oncology MAICs met all NICE recommendations, and Truong et al. 2023 (10.1002/jrsm.1653) find conduct and reporting remarkably heterogeneous and suboptimal. The domains themselves are not unaddressed: NICE DSU TSD 18 already requires an explicit target population, assessment of covariate distributions, evidence of effect-modifier status and the distribution of weights, TSD 7 supplies a pro forma reviewer checklist covering the decision problem, disconnected networks, assumptions, sensitivity and uncertainty, and CINeMA (10.1371/journal.pmed.1003082) gives per-comparison credibility judgments for network meta-analysis, though with no items for target population, overlap, weighting moments or bridge assumptions. What is missing is a scored PAIC-specific instrument and a consolidated standard adding estimand classification, reproducibility, assumption stress tests and documented provenance for any quantitative bias analysis.
Why it is open
Existing guidance states requirements but does not score them, so an assessor has no instrument that turns a domain judgment into a comparable credibility rating and compliance is visible only through retrospective review. Each HTA agency issues its own template, so there is no cross-jurisdiction standard and no shared vocabulary for these items. The items that would expose an analysis as assumption-driven impose a cost on the submitting party, so voluntary adoption is unlikely without a requirement.
What has been tried
| Work | What it contributes |
|---|---|
| Farinasso et al. 2025 | Scoping review of 117 oncology MAICs; states that no risk-of-bias tool specific to MAIC exists and reports that only 3 fulfilled all NICE recommendations, 72% were unanchored and 78% did not report the IPD source. |
| Truong et al. 2023 | Methodological systematic review of PAICs in HTA; finds conduct and reporting heterogeneous and suboptimal, with quality or bias of individual studies assessed in 9.3% of analyses. |
| Cassidy et al. 2023 | Supplies the reporting-bias evidence: MAICs in connected networks reported more favorable estimates for the IPD holder’s treatment than the corresponding anchored ITCs. |
| CINeMA framework (2020) | Per-comparison credibility framework for network meta-analysis across six domains, built on the contribution matrix; has no items for target population, overlap, weighting moments or bridge assumptions. |
| NICE DSU Technical Support Document 18 (2016) | PAIC-specific recommendations covering target population, covariate overlap, effect-modifier evidence, distribution of weights, uncertainty and systematic error. Read by the code auditor from a document corpus at the given commit. |
Probable solution or research direction
Build on TSD 18 and TSD 7 rather than start from scratch: add the domains they do not cover, namely explicit estimand classification, reproducibility, assumption stress tests and documented rationale and provenance for any quantitative bias analysis, and turn the result into an instrument an assessor can score. Follow CINeMA in keeping domain-level judgments separate rather than collapsing identification judgments into one aggregate score, and require quantitative bias analysis only where it is relevant and supportable. Validate the instrument on a sample of submitted analyses.
Verification
Verdict. Partially addressed. Part of this problem is addressed by existing work; the entry names which part.
The source said. “Published population-adjusted indirect comparisons are reported inconsistently and there is no risk-of-bias or credibility instrument built for them. Nor is there a reporting standard covering the nine areas that decide whether a submission is interpretable: decision question and estimand, evidence graph, population adjustment, overlap and extrapolation, model specification, uncertainty, assumption stress tests, quantitative bias analysis and reproducibility.”
What was wrong with it. The source said existing tools contain no items for target population, overlap, weighting moments or bridge assumptions, so those features are never assessed. GPT-5.6 Sol found NICE DSU TSD 18 already requires exactly those disclosures and TSD 7 supplies a reviewer checklist, and the literature auditor added CINeMA as an existing per-comparison credibility framework for network meta-analysis, so the surviving gap is a scored PAIC-specific instrument rather than an absence of appraisal criteria.
Dissent. literature dissented (open): The central claim is asserted by a source, which is the strongest form of evidence available for an absence claim. Farinasso et al. 2025 (Value Health Reg Issues, doi 10.1016/j.vhri.2025.101088) state: ‘There is no risk of bias tool specific for MAICs, which can hinder their applicability in the health technology assessment (HTA) process. Without the risk of bias assessment, it is also not possible to apply the Grading of Recommendation, Assessment, Development, and Evaluation criteria.’ Their scoping review of 117 oncology MAICs found only 3 fulfilled all NICE recommendations, 72% were unanchored, 78% did not report the IPD source and 66% ran no systematic review. Truong et al. 2023 (doi 10.1002/jrsm.1653) independently concludes reporting is ‘remarkably heterogeneous and suboptimal’. Cassidy et al. 2023 supplies the reporting-bias evidence. Caveat: the why_open text says existing RoB tools were built for RCTs and conventional meta-analysis; that omits CINeMA (doi 10.1371/journal.pmed.1003082), a per-comparison credibility framework for NMA built on the contribution matrix. CINeMA still has no items for target population, overlap, weighting moments or bridge assumptions, so the PAIC gap stands. codex dissented (overstated): The inspected guidance directly defeats the claim that target population, overlap, weights, uncertainty, and bridge assumptions are never assessed. TSD 18 gives PAIC-specific reporting recommendations for those domains, while TSD 7 supplies a reviewer checklist covering the decision problem, disconnected networks, assumptions, sensitivity analysis, and uncertainty. Neither document is a formal PAIC-specific risk-of-bias scoring instrument or a complete modern nine-domain standard, and both leave room for an updated framework. C1-C3 still require a current literature review. C4-C6 are therefore universal negatives stated too strongly. C7 is broadly sound, but QBA should be required when relevant and supportable, and identification judgments should not be collapsed into a misleading aggregate score. C8 accurately describes a proposal.
- Literature and prior-art check
- open / supported-with-caveat. The central claim is asserted by a source, which is the strongest form of evidence available for an absence claim. Farinasso et al. 2025 (Value Health Reg Issues, doi 10.1016/j.vhri.2025.101088) state: ‘There is no risk of bias tool specific for MAICs, which can hinder their applicability in the health technology assessment (HTA) process. Without the risk of bias assessment, it is also not possible to apply the Grading of Recommendation, Assessment, Development, and Evaluation criteria.’ Their scoping review of 117 oncology MAICs found only 3 fulfilled all NICE recommendations, 72% were unanchored, 78% did not report the IPD source and 66% ran no systematic review. Truong et al. 2023 (doi 10.1002/jrsm.1653) independently concludes reporting is ‘remarkably heterogeneous and suboptimal’. Cassidy et al. 2023 supplies the reporting-bias evidence. Caveat: the why_open text says existing RoB tools were built for RCTs and conventional meta-analysis; that omits CINeMA (doi 10.1371/journal.pmed.1003082), a per-comparison credibility framework for NMA built on the contribution matrix. CINeMA still has no items for target population, overlap, weighting moments or bridge assumptions, so the PAIC gap stands. Cites 10.1016/j.vhri.2025.101088: Supplies a source that itself asserts the absence of a PAIC-specific risk-of-bias tool, and supplies the empirical reporting figures.. Cites 10.1002/jrsm.1653: Supports ‘reviews show poor methodological reporting of PAICs’ with measured figures.. Cites 10.1016/j.jval.2023.07.001: Supports ‘reviews show possible reporting bias in published PAICs’..
- GPT-5.6 Sol, technical and source-code lens
- partially-solved / overstated. The inspected guidance directly defeats the claim that target population, overlap, weights, uncertainty, and bridge assumptions are never assessed. TSD 18 gives PAIC-specific reporting recommendations for those domains, while TSD 7 supplies a reviewer checklist covering the decision problem, disconnected networks, assumptions, sensitivity analysis, and uncertainty. Neither document is a formal PAIC-specific risk-of-bias scoring instrument or a complete modern nine-domain standard, and both leave room for an updated framework. C1-C3 still require a current literature review. C4-C6 are therefore universal negatives stated too strongly. C7 is broadly sound, but QBA should be required when relevant and supportable, and identification judgments should not be collapsed into a misleading aggregate score. C8 accurately describes a proposal. Cites choxos/NICE-DSU-TSD-RAG@75abcb9c226a9f8fa7a0a4ffeba42874f1712ecb: Provides PAIC-specific recommendations for target population, covariate overlap, effect-modifier evidence, weights, uncertainty, systematic error, and reporting.. Cites choxos/NICE-DSU-TSD-RAG@75abcb9c226a9f8fa7a0a4ffeba42874f1712ecb: Provides a pro forma reviewer checklist for decision questions, evidence structure, disconnected networks, assumptions, sensitivity analyses, and uncertainty..
- Decision path
-
R3:partially-solved(n=1) → R9:substantiated-single-overstatement:codex
How the evidence falls in time
26 papers in the reviewed corpus bear on this problem, published 2011 to 2025. Partial progress is on the record, and later work still calls this open. Both belong in view.
This problem is recurrent: three or more papers spanning at least eight years assert it independently. Sustained restatement by authors who mostly do not cite each other is the strongest evidence this reading can offer that a gap is real rather than one group’s framing.
It is also being closed in pieces: partial results come from more than one paper in more than one year, each covering a different part.
| Year | Says | Paper |
|---|---|---|
| 2011 | confirms it is open | Network meta-analysis-highly attractive but more methodological research is needed |
| 2013 | partly addresses it | Evidence synthesis for decision making 7: a reviewer’s checklist |
| 2013 | confirms it is open | Evaluation of inconsistency in networks of interventions |
| 2014 | confirms it is open | An overview of conducting systematic reviews with network meta-analysis |
| 2014 | confirms it is open | Evaluating the quality of evidence from a network meta-analysis |
| 2014 | confirms it is open | The quality of reporting methods and results in network meta-analyses: an overview of reviews and suggestions for improvement |
| 2015 | confirms it is open | An Assessment of the Methodological Quality of Published Network Meta-Analyses: A Systematic Review |
| 2016 | partly addresses it | A scoping review of indirect comparison methods and applications using individual patient data |
| 2016 | confirms it is open | A threshold analysis assessed the credibility of conclusions from network meta-analysis |
| 2016 | confirms it is open | Network meta-analyses performed by contracting companies and commissioned by industry |
| 2016 | confirms it is open | Use of network meta-analysis in systematic reviews: a survey of authors |
| 2020 | partly addresses it | Summarising salient information on historical controls: A structured assessment of validity and comparability across studies |
| 2020 | confirms it is open | CINeMA: An approach for assessing confidence in the results of a network meta-analysis |
| 2020 | confirms it is open | CINeMA: Software for semiautomated assessment of the confidence in the results of network meta-analysis |
| 2020 | confirms it is open | Evidence synthesis in pulmonary arterial hypertension: a systematic review and critical appraisal |
| 2021 | confirms it is open | Methodological review to develop a list of bias items used to assess reviews incorporating network meta-analysis: protocol and rationale |
| 2021 | confirms it is open | ROB-MEN: a tool to assess risk of bias due to missing evidence in network meta-analysis |
| 2022 | confirms it is open | The development of network meta-analysis |
| 2023 | partly addresses it | A Critical Appraisal of Matching-Adjusted Indirect Comparisons in Spinal Muscular Atrophy |
| 2023 | confirms it is open | Increasing transparency in indirect treatment comparisons: is selecting effect modifiers the missing part of the puzzle? A review of methodological approaches and critical considerations |
| 2023 | confirms it is open | Mapping the characteristics, concepts and methodologies of matching-adjusted indirect comparison studies assessing pharmacological therapies in oncology: a scoping review protocol |
| 2023 | confirms it is open | The Acceptance of Indirect Treatment Comparison Methods in Oncology by Health Technology Assessment Agencies in England, France, Germany, Italy, and Spain |
| 2024 | confirms it is open | Methods for Indirect Treatment Comparison: Results from a Systematic Literature Review |
| 2024 | confirms it is open | Methodological review of NMA bias concepts provides groundwork for the development of a list of concepts for potential inclusion in a new risk of bias tool for network meta-analysis (RoB NMA Tool) |
| 2024 | confirms it is open | Which clinical trial designs and statistical approaches have been used in assessments of orphan maintenance by the European Medicines Agency between 2012 and 2022? A cross-sectional study |
| 2025 | confirms it is open | Risk of Bias in Network Meta-Analysis (RoB NMA) tool |
Source
Derived from C:L214-215, C:L259, C:L2363-2364, C:L2365-2448 of the reviewed corpus.