Node splitting compares paths without standardizing to a target
Unadjusted node splitting tests inconsistency plus population mismatch. With no true inconsistency it flagged 14% to 98% of analyses when the direct and indirect study sets differed in an effect modifier, and it fell to its null rate when the mismatch cancelled a real conflict. A split adjusted for the modifier held size (4% to 6%) and recovered some masked conflicts, but its precision fell as the gap between the study sets grew. Read the study below.
Statement
Standard aggregate-data node splitting, as implemented in netmeta and gemtc, compares direct and indirect evidence without standardizing either path to a common target population, so a population difference between the paths and a genuine loop conflict arrive as a single number. Node splitting is local to a comparison rather than a global test of the network, and it can already be fitted inside a population-adjusted regression: multinma has supported consistency = ‘nodesplit’ since 0.4.0 and the same call accepts a regression formula and integration points, so both paths can be compared after adjustment (Phillippo et al. 2022, doi:10.1177/0272989X221117162 assess inconsistency inside a population-adjusted psoriasis model with estimates produced for three external target populations). Two things remain undone: that adjusted split reports a conditional inconsistency factor rather than one standardized to a named target population on the marginal scale, and no study has quantified how often the unadjusted diagnostic flags population mismatch as inconsistency or misses true inconsistency that a compensating mismatch has masked. Whether consistency is population-indexed at all needs care, because marginal effects on non-collapsible measures are population-dependent (Chandler and Ishak, arxiv:2602.17041) but a loop whose every effect is defined in the same population telescopes algebraically, so population-varying inconsistency requires different populations along the evidence paths, design-specific effect surfaces, or an inconsistency-by-covariate interaction.
Why it is open
Unadjusted node splitting takes the direct estimate from the studies that made that comparison and the indirect estimate from a different set of studies, so the two paths carry different covariate distributions; a population difference registers as inconsistency, a real conflict can be cancelled by a compensating population difference, and one number is returned for both explanations. Fitting the split inside a population-adjusted regression removes that confounding at the conditional level, but only under the usual transport conditions: consistent estimands, positivity, measured effect modifiers, and correctly specified outcome and transport models. Even then the output is a conditional inconsistency factor, so no diagnostic is reported at a named target population on the marginal scale a decision uses. And the false-positive and masking rates of the unadjusted test, which is the one applied work actually runs, have not been measured for the population-indexed comparison this entry describes, though Song et al. 2012 measured them for the ordinary unadjusted test by simulation, so nobody knows how much reported inconsistency in the literature is population mismatch.
What has been tried
| Work | What it contributes |
|---|---|
| multinma: Bayesian Network Meta-Analysis of Individual and Aggregate Data (version 0.9.1), nma(consistency = “nodesplit”) | Node splitting, added in 0.4.0 (2022-01-18), can be fitted within a covariate-adjusted ML-NMR with a regression formula and numerical integration, so the two evidence paths are compared after population adjustment rather than on raw aggregate contrasts. |
| Phillippo et al., 2023, Validating the Assumptions of Population Adjustment: Application of Multilevel Network Meta-regression to a Network of Treatments for Plaque Psoriasis | Examines residual heterogeneity and inconsistency inside a population-adjusted model and relaxes the shared effect modifier assumption for each covariate in turn, with estimates produced for three external target populations. |
| Chandler and Ishak, 2026 | Shows that even when effect modifiers are shared across treatments, marginal effects are generally population-dependent for commonly used non-collapsible measures including hazard ratios and odds ratios; the audit used it as the support for consistency being population-indexed. |
| cpaic, unreleased development package on GitHub (not on CRAN), commit 9d150e9 | relative_effects() evaluates a conditional contrast at one covariate profile and explicitly does not marginally standardize over a target distribution, and the NAMESPACE exposes no node-splitting or design-by-treatment diagnostic. |
Probable solution or research direction
The immediate work is measurement rather than method: run the simulation that quantifies how often unadjusted node splitting flags population mismatch as inconsistency and how often a compensating mismatch masks a real conflict, since that number decides whether the adjusted version is worth its cost. Beyond that, marginalize the ML-NMR node split so the inconsistency factor is reported at a named target population on the decision scale rather than at a covariate profile, and treat population-indexed inconsistency parameters as the harder remaining case. Any such diagnostic inherits the transport assumptions and cannot separate genuine inconsistency from unmeasured population or design differences, so it should be reported with that limit stated. In cpaic, an unreleased development package on GitHub that is not on CRAN, neither piece exists at commit 9d150e9: relative_effects() evaluates a conditional contrast at a single covariate profile without marginal standardization, and no node-splitting or design-by-treatment diagnostic is exported.
Our study
A population difference arriving as inconsistency
The question put to it. When the studies supplying the direct and indirect paths of a loop differ in an effect modifier, how often does unadjusted node splitting flag the population difference as inconsistency or miss a real conflict that the difference cancels, and what does an adjusted split achieve?
What it found. Unadjusted node splitting tests inconsistency plus population mismatch. With no true inconsistency it flagged 14% to 98% of analyses when the direct and indirect study sets differed in an effect modifier, and it fell to its null rate when the mismatch cancelled a real conflict. A split adjusted for the modifier held size (4% to 6%) and recovered some masked conflicts, but its precision fell as the gap between the study sets grew.
Results.
- Refuting sentence fails: unadjusted flag rate 0.141 to 0.982 with no inconsistency and differing populations; adjusted 0.041 to 0.058.
- Exact masking: unadjusted 0.058 to 0.105; adjusted 0.074 to 0.254.
- Adjusted inconsistency SE rose from 0.10 to 0.30 as the covariate gap went from 0 to 1 (four studies per comparison, modification 0.4); power against 0.2 fell from 0.54 to 0.12.
- Unadjusted split at 0.10 even with no gap under modification 0.4, from covariate-driven heterogeneity.
Design.
- Estimand
- Loop inconsistency factor for the B versus C contrast
- Methods compared
- unadjusted node split, meta-regression-adjusted node split
- Data-generating mechanism
- Triangle networks with a covariate gap between the direct and indirect study sets, modification strength, true loop inconsistency and two or four studies per comparison; unadjusted and meta-regression-adjusted node splits. 54 cells, 2000 replicates each.
- Replicates
- 2000 per scenario, 108,000 in total
What this does not answer. Continuous study estimates, one loop, no Bayesian node split or marginalization to a named target. Peer review has not been done.
Peer review. Not yet done. The result stands on the study alone until it is.
Read it: PDF · Markdown · OpenDocument · Code and data: studies/HET-04-population-mismatch-arriving-as-inconsistency · Protocol: registered before the run
Verification
Verdict. Partially addressed. Part of this problem is addressed by existing work; the entry names which part.
The source titled this. “Consistency is population-indexed but tested as a global property” The title calls consistency tested as a global property, but node splitting is local to a comparison and can already be fitted inside a population-adjusted regression.
The source said. “A network loop can be consistent in one population and inconsistent in another, so consistency is a population-indexed property; standard node splitting and design-by-treatment interaction models nonetheless test it as a global property of the network.”
What was wrong with it. The source said node splitting and design-by-treatment models test consistency as a global property with no population standardization; the literature auditor found node splitting can already be fitted inside a covariate-adjusted ML-NMR (multinma since 0.4.0; Phillippo et al. 2022, doi:10.1177/0272989X221117162), and the code auditor noted that node splitting is local to a comparison rather than global, and that non-collapsibility on its own does not make consistency population-dependent because a loop defined in one population telescopes algebraically. The auditors split on whether the entry is open or partly solved, and it is published at the weaker claim.
Revised by the full-text reading. Applied 2026-07-26 full-text reading of the consolidated library (687 papers).
- Corrected
- why_open: corrected a factual claim Song et al. 2012 (BMC Med Res Methodol 12:138) ran 5000 replicates per scenario across four heterogeneity levels and six network sizes, reporting type I error and power for the unadjusted test. The flat claim that these were never measured is false; the narrower population-indexed claim in the statement stands. Previously: “Unadjusted node splitting takes the direct estimate from the studies that made that comparison and the indirect estimate from a different set of studies, so the two paths carry different covariate distributions; a population difference registers as inconsistency, a real conflict can be cancelled by a compensating population difference, and one number is returned for both explanations. Fitting the split inside a population-adjusted regression removes that confounding at the conditional level, but only under the usual transport conditions: consistent estimands, positivity, measured effect modifi”
- Citation corrected
- normalized a citation year: ‘Phillippo et al., 2022, Validating the Assumptions of Population Adjustment: Application of Multilevel Network Meta-regression to a Network of Treatments for Plaque Psoriasis’ -> ‘Phillippo et al., 2023, Validating the Assumptions of Population Adjustment: Application of Multilevel Network Meta-regression to a Network of Treatments for Plaque Psoriasis’ 10.1177/0272989X221117162 is cited with two different years across the registry. CrossRef gives 2022 online and 2023 in print; the issue year is used throughout.
- Literature and prior-art check
- partially-solved / supported-with-caveat. Claim 1 is supported. Chandler & Ishak (arXiv 2602.17041, in this corpus) state: ‘even when effect modifiers are shared across treatments, marginal effects are generally population-dependent for commonly used non-collapsible measures, including hazard ratios and odds ratios.’ If the effects being compared are population-dependent, then so is any consistency relation among them. Claim 2 needs a qualifier. It is true of aggregate-data node splitting in netmeta and gemtc, but false as stated for multinma: node-splitting was added in multinma 0.4.0 (consistency=‘nodesplit’) and the same nma() call accepts a regression formula and integration points, so a node-split can be fitted inside a covariate-adjusted ML-NMR where direct and indirect paths are compared at a common covariate profile and estimates can be produced in any target population. Phillippo et al. (2022) do exactly this: ‘We examine residual heterogeneity and inconsistency and relax the shared effect modifier assumption for each covariate in turn.’ What is genuinely open is the evaluation the entry asks for. I found no study quantifying how often standard node splitting flags population mismatch as inconsistency, or how often a compensating mismatch masks true inconsistency, and ML-NMR node splitting reports a conditional inconsistency factor, not a target-population-standardized marginal one. Cites cran:multinma 0.9.1: Node splitting can already be fitted within a covariate-adjusted, population-adjusted ML-NMR, so the two evidence paths are compared after adjustment rather than on raw aggregate contrasts. Feature added in multinma 0.4.0 (2022-01-18).. Cites doi:10.1177/0272989X221117162: Demonstrates assessment of inconsistency inside a population-adjusted model with estimates produced for three external target populations..
- GPT-5.6 Sol, technical and source-code lens
- open / supported-with-caveat. C1 is possible, but non-collapsibility alone does not make consistency switch across populations. If each effect is coherently defined as delta_jk(P) = g(mu_k(P)) - g(mu_j(P)) in the same population P, loop consistency telescopes algebraically for every P. Population-varying inconsistency requires different populations along the evidence paths, design-specific effect surfaces, or an inconsistency-by-covariate interaction. C2 is correct for ordinary node splitting, which generally compares estimates from their contributing study populations rather than transporting both paths to an explicit common target. Node splitting is local to a comparison, however, so describing it as a global test is inaccurate. C3 is promising only under consistent estimands, positivity, measured effect modifiers, and correctly specified outcome and transport models; it cannot separate genuine inconsistency from unmeasured population or design differences. At cpaic 9d150e9, relative_effects() evaluates a conditional contrast at one covariate profile and explicitly does not marginally standardize over a target distribution, while NAMESPACE exposes no node-splitting or design-by-treatment diagnostic. The proposed diagnostic therefore remains open in the package.
- Decision path
-
R3:partially-solved(n=1) → R11:caveated(n=2)
How the evidence falls in time
17 papers in the reviewed corpus bear on this problem, published 2013 to 2023. The most recent paper to touch this reports progress on it, and every paper calling it open is older.
This problem is recurrent: three or more papers spanning at least eight years assert it independently. Sustained restatement by authors who mostly do not cite each other is the strongest evidence this reading can offer that a gap is real rather than one group’s framing.
It is also being closed in pieces: partial results come from more than one paper in more than one year, each covering a different part.
Does the later work answer the earlier work? Put to an independent reviewer with the older and newer claims side by side, it judged that the later work answers a component of it, leaving a named part (high confidence). The 2017 and 2018 regression-based node-splitting work separates discrepancies in treatment effects from discrepancies in treatment-by-covariate coefficients, which meaningfully addresses the original conflation. But these are conditional covariate-value analyses, not standardization of both paths to a target covariate distribution, and they do not quantify the practical frequency or consequences of misclassification. What remains: A direct-versus-indirect diagnostic standardized marginally to a named target population, and empirical evidence on how often conventional node splitting mistakes population mismatch for inconsistency or masks real inconsistency.
| Year | Says | Paper |
|---|---|---|
| 2013 | confirms it is open | Evaluation of inconsistency in networks of interventions |
| 2013 | confirms it is open | Evidence synthesis for decision making 4: inconsistency in networks of evidence based on randomized controlled trials |
| 2014 | confirms it is open | A design-by-treatment interaction model for network meta-analysis with random inconsistency effects |
| 2014 | confirms it is open | Allowing for uncertainty due to missing continuous outcome data in pairwise and network meta-analysis |
| 2014 | confirms it is open | Characteristics of a loop of evidence that affect detection and estimation of inconsistency: a simulation study |
| 2014 | confirms it is open | Characteristics of networks of interventions: a description of a database of 186 published networks |
| 2014 | confirms it is open | Visualizing inconsistency in network meta-analysis by independent path decomposition |
| 2015 | confirms it is open | Automated generation of node-splitting models for assessment of inconsistency in network meta-analysis |
| 2015 | confirms it is open | The design-by-treatment interaction model: a unifying framework for modelling loop inconsistency in network meta-analysis |
| 2017 | partly addresses it | Network meta-analysis including treatment by covariate interactions: Consistency can vary across covariate values |
| 2017 | confirms it is open | Bayesian one-step IPD network meta-analysis of time-to-event data using Royston-Parmar models |
| 2018 | partly addresses it | Assessing the consistency assumptions underlying network meta-regression using aggregate data |
| 2018 | confirms it is open | Incorporating external evidence on between-trial heterogeneity in network meta-analysis |
| 2021 | confirms it is open | Individual Participant Data Meta-Analysis: A Handbook for Healthcare Research (Riley, Tierney, Stewart, eds.; Wiley Series in Statistics in Practice, 2021) |
| 2022 | confirms it is open | Pharmacological and non-pharmacological treatments for restless legs syndrome in end-stage kidney disease: a systematic review and component network meta-analysis (supplementary file and main article) |
| 2022 | confirms it is open | Analysis of adaptive platform trials using a network approach |
| 2023 | partly addresses it | A new approach to evaluating loop inconsistency in network meta-analysis |
Source
Derived from C:L214-215, C:L235 of the reviewed corpus.