No diagnostic can falsify unmeasured effect modification in PAIC
Source note. This entry derives in part from a source document whose inline citations were ChatGPT interface tokens rather than references, and so resolve to nothing. Claims below are attributed to that document and were checked independently where possible; anything that could not be independently sourced is marked in the verification trail.
Statement
Balance, support, fit and inconsistency diagnostics can falsify limited observable restrictions, but none of them can establish the absence of unmeasured effect modification or cross-study noncomparability, which is the part of the identifying assumption the analysis depends on. Residual heterogeneity and inconsistency checks, and covariate-by-covariate relaxation of the shared effect modifier assumption, are already available and are used inside ML-NMR target-population projections (Phillippo et al. 2022, doi:10.1177/0272989X221117162), and a proof-of-concept falsification test built on partitioning a single arm’s IPD exists for measured-covariate sufficiency (Yi and Jiang 2025, doi:10.57264/cer-2024-0235). What has not been developed for PAIC is falsification of transport itself through held-out IPD or leave-one-study-out prediction.
Why it is open
Balance diagnostics are functions of measured covariates and fit statistics are functions of observed outcomes, so both speak to the measured, within-network part of the model and are silent on unmeasured effect modification and cross-study comparability. Under the classical two-trial configuration only one trial supplies IPD, so there is no second source from which observable implications of the transport assumption could be built. Held-out-study and deleted-edge designs do exist in disconnected NMA (Beliveau et al. 2017, doi:10.1002/jrsm.1256), but a failed reconstruction is not uniquely attributable to transport failure, so such procedures test a joint prediction procedure rather than the identifying assumption on its own.
What has been tried
| Work | What it contributes |
|---|---|
| Phillippo et al. 2023, Validating the Assumptions of Population Adjustment: Application of Multilevel Network Meta-regression to a Network of Treatments for Plaque Psoriasis | Integrates residual heterogeneity and unrelated-mean-effects inconsistency checks with ML-NMR target-population projection and relaxes the shared effect modifier assumption one covariate at a time; contradicts the claim that residual inconsistency checks are not integrated with target-population projections. |
| Yi and Jiang 2025, A process to validate prognostic factors for unanchored matching-adjusted indirect comparison of single-arm trials in oncology: a proof-of-concept study | Splits a single arm into two risk-stratified groups with an imposed hazard ratio and tests whether reweighting on the candidate covariate set drives it back to one; a falsification-style test of measured-covariate sufficiency built on an IPD partition. |
| Jiang et al. 2025, A critical assessment of matching-adjusted indirect comparisons in relation to target populations | States that the shared effect modifier assumption is statistically untestable under the classical two-trial scenario; supplies the published attribution the source lacked. |
| Beliveau et al. 2017, Network meta-analysis of disconnected networks: How dangerous are random baseline treatment effects? | Disconnects real connected networks in multiple ways and scores the reconstructed cross-gap comparisons against the connected-network analysis, showing that synthetic bridge deletion is an established design rather than a new proposal. |
Probable solution or research direction
Extend the falsification designs that already exist rather than build new machinery: generalize the single-arm IPD-partition test to multi-covariate and anchored settings, and carry the leave-one-study-out and edge-deletion designs already used in disconnected NMA into PAIC where IPD sit on both sides of the deleted edge. State for each check what it can and cannot rule out, since rejection implicates the joint model rather than transport alone and acceptance cannot validate a genuinely missing gap. Where negative-control outcomes or falsification endpoints are available, add them as a second observable implication rather than as a substitute for the transport check.
Verification
Verdict. Overstated. 2 independent auditors judged the claim as stated too strong. No falsification test exists for the unmeasured-covariate component of PAIC identifying assumptions. Residual heterogeneity/inconsistency checks and shared-effect-modifier relaxation are available and are used within ML-NMR target-population projections, but they interrogate the measured, within-network part of the model; held-out-IPD and leave-one-study-out falsification of transport has not been developed for PAIC.
The source titled this. “Reported diagnostics cannot falsify the assumptions they rest on” The title says reported diagnostics cannot falsify the assumptions at all, but the audit found balance, fit and inconsistency checks do falsify limited observable restrictions.
The source said. “Covariate balance and model fit can look excellent in a population-adjusted comparison even under severe residual confounding, so the diagnostics that are routinely reported carry no evidential content for the assumption the analysis actually depends on. No formal falsification test of PAIC identifying assumptions, built on held-out IPD or aggregate-data partitions, currently exists.”
What was wrong with it. The source said routine diagnostics carry no evidential content and that residual inconsistency checks are not integrated with target-population projections; the literature auditor found them integrated in Phillippo et al. 2022 (doi:10.1177/0272989X221117162) and found a proof-of-concept IPD-partition falsification test in Yi and Jiang 2025 (doi:10.57264/cer-2024-0235). The core claim was unattributed in the source and is now sourced to Jiang et al. 2025 (doi:10.1017/rsm.2025.10), which states that under the classical two-trial scenario the shared effect modifier assumption is statistically untestable.
Revised by the full-text reading. Applied 2026-07-26 full-text reading of the consolidated library (687 papers).
- Citation corrected
- normalized a citation year: ‘Phillippo et al. 2022, Validating the Assumptions of Population Adjustment: Application of Multilevel Network Meta-regression to a Network of Treatments for Plaque Psoriasis’ -> ‘Phillippo et al. 2023, Validating the Assumptions of Population Adjustment: Application of Multilevel Network Meta-regression to a Network of Treatments for Plaque Psoriasis’ 10.1177/0272989X221117162 is cited with two different years across the registry. CrossRef gives 2022 online and 2023 in print; the issue year is used throughout.
- closure-upheld
- 10 findings from the reading assert this problem is open; the closure was re-examined and stands Put to an independent reviewer with the closing auditors’ reasoning, the work they relied on, and the new findings in view together. The new findings repeatedly establish the boundary already preserved in the closure: observed-data diagnostics can test limited observable implications but cannot establish the absence of unmeasured effect modification or noncomparability. Phillippo et al. 2022 is the closure’s central counterexample and explicitly acknowledges the external-target limitation, so it supports rather than contradicts the residual. Yi and Jiang 2025 is likewise already conceded as a within-source measured-covariate check. Practice examples of overinterpreted null diagnostics and evidence of low power do not create a held-out-IPD or leave-one-study-out falsification procedure for PAIC transport. (reviewer confidence: high)
- Flagged for reopening
-
10 findings from the reading assert this problem is open while the registry carries
overstatedFlagged, not applied: reversing an earlier audit’s closure needs the closure’s own evidence re-examined alongside this, which the reading did not do. See documentation/audit/reading/PROPOSED.md. - Literature and prior-art check
- open / overstated. Retrieved Phillippo et al. 2022 (Med Decis Making, psoriasis ML-NMR), which does exactly what the entry says is not done: it fits UME ML-NMR to test residual inconsistency and RE ML-NMR to test residual heterogeneity AFTER population adjustment, inside a framework that projects to arbitrary target populations, and relaxes the shared-effect-modifier assumption one covariate at a time. So ‘residual inconsistency checks are not currently integrated with target-population projections’ is false. I also retrieved Yi & Jiang 2025, a proof-of-concept falsification procedure for unanchored MAIC built on partitioning a single arm’s IPD, inducing a known hazard imbalance and asking whether reweighting on candidate covariates removes it. The core claim survives: nothing I found tests the unmeasured-confounding/conditional-constancy assumption itself, and Jiang et al. 2025 assert its untestability outright. Synthetic bridge deletion is already used in disconnected NMA (Beliveau 2017), so that sub-claim is not novel. Cites 10.1177/0272989X221117162: Integrates residual heterogeneity and residual inconsistency (UME) checks with ML-NMR target-population projection, and relaxes the shared effect modifier assumption covariate-by-covariate. Contradicts the claim that residual inconsistency checks are not integrated with target-population projections.. Cites 10.57264/cer-2024-0235: Proposes a formal falsification-style test built on an IPD partition: split a single arm into two risk-stratified groups with a known imposed hazard ratio, then test whether reweighting on the candidate covariate set drives it back to 1. Partially contradicts ‘no formal falsification test built on held-out IPD partitions currently exists’; it is a proof-of-concept covering measured-covariate sufficiency only.. Cites 10.1017/rsm.2025.10: Supplies a real published assertion of untestability for the shared effect modifier assumption, converting the unsourced core claim into a sourced one..
- GPT-5.6 Sol, technical and source-code lens
- open / overstated. C5 and a limited form of C6 are correct: excellent measured balance or within-source fit can coexist with severe bias and cannot establish absence of unmeasured effect modification or cross-study constancy. However, “no evidential content” is false because balance, support, fit, and inconsistency checks can falsify observable necessary conditions. Synthetic bridge deletion and leave-one-study-out emulation test a joint prediction procedure only when the held-out unit is independent, relevant, and prespecified; failure is not uniquely attributable to transport, and success cannot validate a genuinely missing gap. The field-wide absence claims in C2 and C4 are unsourced and cannot be established within this audit. The identification limitation remains open, but the proposed procedures are not universal falsification tests of PAIC assumptions.
- Decision path
-
R5:default-open → R8:overstated-majority(n=2)
How the evidence falls in time
10 papers in the reviewed corpus bear on this problem, published 2015 to 2025. Every paper found here asserts the problem is open; none reports progress.
This problem is recurrent: three or more papers spanning at least eight years assert it independently. Sustained restatement by authors who mostly do not cite each other is the strongest evidence this reading can offer that a gap is real rather than one group’s framing.
| Year | Says | Paper |
|---|---|---|
| 2015 | confirms it is open | Network Meta-analysis: Users’ Guide for Surgeons: Part II - Certainty |
| 2015 | confirms it is open | Response to: ‘Heterogeneity, consistency and model fit should be assessed in Bayesian network meta-analysis’ by Wei et al. |
| 2016 | confirms it is open | GetReal in network meta-analysis: a review of the methodology |
| 2016 | confirms it is open | Meta-Analysis of a Complex Network of Non-Pharmacological Interventions: The Example of Femoral Neck Fracture |
| 2019 | confirms it is open | Population Adjustment Methods for Indirect Comparisons: A Review of National Institute for Health and Care Excellence Technology Appraisals |
| 2021 | confirms it is open | Individual Participant Data Meta-Analysis: A Handbook for Healthcare Research (Riley, Tierney, Stewart, eds.; Wiley Series in Statistics in Practice, 2021) |
| 2022 | confirms it is open | Validating the Assumptions of Population Adjustment: Application of Multilevel Network Meta-regression to a Network of Treatments for Plaque Psoriasis |
| 2022 | confirms it is open | A Response to: Letter to the Editor Regarding “Network Meta-analysis of Food and Drug Administration-approved Treatment Options for Adults with Aquaporin-4 Immunoglobulin G-positive Neuromyelitis Optica Spectrum Disorder” |
| 2024 | confirms it is open | Low awareness of the transitivity assumption in complex networks of interventions: a systematic survey from 721 network meta-analyses |
| 2025 | confirms it is open | A process to validate prognostic factors for unanchored matching-adjusted indirect comparison of single-arm trials in oncology: a proof-of-concept study |
Source
Derived from A:L60, C:L214-215, C:L253 of the reviewed corpus.