The shared effect modifier check has low and uncalibrated power

Identification, transitivity & conditional constancy
Very high priority
Overstated
A published, software-implemented check on the shared effect modifier assumption already exists: Phillippo et al. (doi:10.1177/0272989X221117162) split the common class interaction into treatment-specific interactions one covariate at a time and compare posteriors and model fit, inside a single nine-study network and with no second IPD trial, and multinma implements the hierarchical relaxation this entry proposed through nma(class_interactions = c(‘common’, ‘exchangeable’, ‘independent’)). The check has low and uncharacterized power, which its authors concede, no calibration threshold and no agreed stability metric, and it is least feasible in the small, IPD-poor networks where the assumption does the most identifying work. The restriction is also not the universal mechanism that makes anchored adjustment work; it becomes an identifying restriction in particular class, component and sparse ML-NMR parameterizations, and hierarchical pooling regularizes weak information rather than creating identification.

IDN-05 Overstated Very high priority Promising

ImportantA study has been run against this problem

The check does not have the resolution to test the assumption it is read as testing. Against a violation already twice the size that would change a decision, the DIC rule fires in 0.045 [0.029, 0.070] of analyses, against 0.013 [0.005, 0.029] when the assumption holds exactly: a violation that doubles the error that matters moves the alarm rate by about three percentage points. The result does not depend on where the cut is put. Scored threshold-free, the DIC difference ranks a violated network above an intact one with an AUROC of 0.557 [0.517, 0.596] at that violation, so no threshold recovers a ranking the statistic does not contain. The posterior standard deviation of the interaction contrast runs 0.30 to 1.64 against the 0.153 that would make a transported estimate materially wrong, which is why. The equivalence reading, the only one that could confirm rather than merely fail to contradict, fired zero times in 3,200 opportunities, including every replicate where the assumption held exactly; that is a statement about precision, and partly about the prior, since independent normal(0, 2.5) priors already place 0.965 of the contrast beyond the margin. Following the check is still better than not: relaxing exactly what it fires on beats relaxing everything at every target displacement and beats imposing the restriction at every displacement except zero, where the two are indistinguishable. Read the study below.

Statement

A published, software-implemented check on the shared effect modifier assumption already exists: Phillippo et al. (doi:10.1177/0272989X221117162) split the common class interaction into treatment-specific interactions one covariate at a time and compare posteriors and model fit, inside a single nine-study network and with no second IPD trial, and multinma implements the hierarchical relaxation this entry proposed through nma(class_interactions = c(‘common’, ‘exchangeable’, ‘independent’)). The check has low and uncharacterized power, which its authors concede, no calibration threshold and no agreed stability metric, and it is least feasible in the small, IPD-poor networks where the assumption does the most identifying work. The restriction is also not the universal mechanism that makes anchored adjustment work; it becomes an identifying restriction in particular class, component and sparse ML-NMR parameterizations, and hierarchical pooling regularizes weak information rather than creating identification.

Why it is open

Interaction terms are estimated far less precisely than main effects, so the existing check can pass on intervals wide enough to accommodate almost any drift, while apparent discrepancies can come from non-collapsibility, differing covariate ranges or different outcome definitions instead of genuine modifier drift. Nothing states how much interaction stability is enough, so comparing posteriors and model fit returns a judgment rather than a calibrated verdict. The networks where the restriction carries the most identifying weight are the ones with the least data to relax it, and treatment-specific interactions are rarely estimable where one arm contributes only aggregate summaries.

What has been tried

Work What it contributes
Phillippo et al. 2023, Validating the Assumptions of Population Adjustment: Application of Multilevel Network Meta-regression to a Network of Treatments for Plaque Psoriasis, Medical Decision Making assesses population-adjustment assumptions inside a nine-study network: splits the common class interaction into treatment-specific interactions one covariate at a time, checks conditional constancy through residual heterogeneity and inconsistency in sufficiently large networks, and reports that the shared-interaction restriction was required to identify the model where a treatment was informed only by aggregate data
multinma R package, nma() argument class_interactions documents common, exchangeable (hierarchically shrunk) and independent treatment-class effect-modifier interactions. Correction, from study IDN-05: "exchangeable" is accepted by the argument but not implemented. In version 0.9.1.9002 (commit 8489bd83) make_nma_formula() aborts with “Exchangeable treatment class interactions (class_interactions = "exchangeable") not yet supported.” Only "common" and "independent" can be fitted, so the relaxation the source proposed as future work remains future work. The separate class_effects argument does support "exchangeable", but that shrinks treatment class effects, not effect-modifier interactions

Probable solution or research direction

The estimator and the diagnostic exist, so the work is characterization: establish the power of the split-interaction check across network sizes and IPD availability, define a stability metric reporting how far interactions can drift before conclusions move rather than a binary verdict, and set a calibration threshold. Use estimated between-trial interaction drift to set priors or sensitivity envelopes on modifier drift, and report how conclusions move as the restriction is relaxed from common to exchangeable to independent, which multinma already supports. Where mechanistic pharmacology or biomarker evidence exists, use it to inform the shrinkage prior instead of defaulting to a common interaction.

Our study

A violation twice the size that would change a decision raises the shared effect modifier check’s alarm rate from 1.3% to 4.5%

The question put to it. Population adjustment across a network assumes that a covariate modifies the effect of every active treatment in a class in the same way. There is one published within-network check on that assumption: fit the model with a single interaction shared across the class, refit with treatment-specific interactions one covariate at a time, and compare the posteriors and the model fit. Its authors say it likely has low power. Nobody has measured it, there is no threshold that separates a pass from a failure, and there is no statement of what a pass licenses. Scored against a known truth under a declared distribution of networks, what is it worth?

What it found. The check does not have the resolution to test the assumption it is read as testing. Against a violation already twice the size that would change a decision, the DIC rule fires in 0.045 [0.029, 0.070] of analyses, against 0.013 [0.005, 0.029] when the assumption holds exactly: a violation that doubles the error that matters moves the alarm rate by about three percentage points. The result does not depend on where the cut is put. Scored threshold-free, the DIC difference ranks a violated network above an intact one with an AUROC of 0.557 [0.517, 0.596] at that violation, so no threshold recovers a ranking the statistic does not contain. The posterior standard deviation of the interaction contrast runs 0.30 to 1.64 against the 0.153 that would make a transported estimate materially wrong, which is why. The equivalence reading, the only one that could confirm rather than merely fail to contradict, fired zero times in 3,200 opportunities, including every replicate where the assumption held exactly; that is a statement about precision, and partly about the prior, since independent normal(0, 2.5) priors already place 0.965 of the contrast beyond the margin. Following the check is still better than not: relaxing exactly what it fires on beats relaxing everything at every target displacement and beats imposing the restriction at every displacement except zero, where the two are indistinguishable.

Results.

  • Detection barely moves with the violation. Pooled over all eight design strata, 400 replicates each, the DIC cut at 5 fires in 0.013 [0.005, 0.029] under the global null, 0.045 [0.029, 0.070] at a contrast of 0.30 that leaves the transported estimate wrong by twice the material threshold, 0.147 at 0.60 and 0.453 at 1.20, which is eight times the decision boundary and a stress test rather than a plausible violation.
  • The finding survives without any threshold. Scored as a classifier, the DIC difference separates a violated network from an intact one with an AUROC of 0.557 [0.517, 0.596] at a contrast of 0.30, 0.681 [0.644, 0.718] at 0.60 and 0.843 [0.815, 0.870] at 1.20; against material error itself it reaches 0.666 at a target one standard deviation away. This analysis was prespecified, computed from the first run onward, and omitted from the first two drafts until three reviewers asked for it.
  • The reason is resolution. The posterior standard deviation of the difference between the two treatments’ effect modifiers runs 0.30 to 1.64 on the log-odds scale across strata, against 0.153 for the difference that makes the transported estimate materially wrong. An earlier draft led with a prior-removed version of this quantity; its own sensitivity arm showed the quantity is unstable, so the measured posterior standard deviation is what is reported.
  • The procedure cannot give an all-clear. The equivalence reading, P(|difference| <= 0.153) > 0.95, fired in 0 of 3,200 covariate-checks, including every replicate where the two modifiers were identical by construction. This is a precision result: firing needs a posterior standard deviation below about 0.078 and the tightest stratum reaches 0.30. It is also prior-dependent in the other direction, since independent normal(0, 2.5) priors on the two coefficients put 0.965 of their contrast beyond the margin before any data are seen, above the 0.95 threshold at which the complementary margin rule fires.
  • The registered verdict fails, and a perfect check would have failed it too. The upper 95% bound on the probability that a passed analysis is materially wrong is 0.697, deployment- weighted, against a registered 0.10. But an oracle that knows the true interaction specification is materially wrong 0.639 of the time on the same replicates, so the check’s excess over that oracle is 0.036 and is negative at small target displacements. With a threshold of 0.03 absolute risk, the estimation error of a six-study network alone exceeds it in about half of analyses when the assumption holds exactly. A threshold on total error charges a diagnostic for noise it was never built to detect.
  • The check is blind to what determines the harm. Its output reads only the fitted network, so its pass rate is identical at every target displacement to the last digit. Over the same fits the probability that a passed analysis is materially wrong rises from 0.427 to 0.723, an excess over the noise floor of 0.296 against a registered 0.20. The first half of that is an algebraic identity that verifies the bookkeeping; the second is empirical.
  • The three readings are not three procedures. The DIC rule fires without the interval rule in 0.0006 of covariate-checks and the interval rule fires without DIC in 0.106, so the DIC cut is very nearly a strict subset of the interval rule rather than a second opinion. Under the global null they differ in type I error by 0.080 on the drifting covariate and 0.087 pooling both. The 95% interval rule runs at about twice its nominal rate, 0.075 [0.046, 0.120] with no treatment-effect heterogeneity and 0.110 [0.074, 0.161] with it, so avoiding covariate selection does not deliver nominal coverage.
  • A fire is not a diagnosis. When a violation is present the interval rule fires on the covariate that is shared exactly in up to 0.388 of replicates.
  • Following the check beats the alternatives, but not uniformly, and the ranking of the alternatives is the surprise. Deployment-weighted and paired on the same replicates, relaxing exactly what fired beats relaxing everything at every displacement on both absolute and squared error, and beats imposing the restriction at displacements 0.5, 1.0 and 1.5 but is indistinguishable from it at 0 (-0.00005, 95% CI [-0.00069, 0.00059]), where there is nothing to transport. Relaxing everything is worse than imposing the restriction, because the relaxed model spends precision the network does not have; it also fails its sampler diagnostics in 0.068 of replicates and 0.24 in the worst cell. The worst diagnostic reached by any model is R-hat 5.13 with effective sample size 1.03, in a singly-relaxed fit.
  • Knowing the target is worth more than knowing the check statistic. Held out across design factors, the check statistic alone discriminates material error at an AUROC of 0.48 to 0.63, close to chance; adding the target displacement raises it to 0.69 to 0.83. A one-term mapping from the statistic to the probability of material error transports across network size, covariate spread and heterogeneity with absolute errors of 0.013 to 0.058, and fails across drift with an error of 0.469.
  • On a decision-curve basis, distrusting every adjusted analysis has higher net benefit than every rule in the panel at every action threshold from 0.05 to 0.60. This restates the noise floor rather than measuring the check: with material error present in 0.675 of passing analyses, most of it sampling noise no covariate check can predict, treat-all dominates almost regardless of what the check does.
  • Two software findings, against multinma 0.9.1.9002 (commit 8489bd83). The package documents class_interactions = ‘exchangeable’ and raises ‘not yet supported’, so the hierarchical relaxation this catalog entry proposed as the way forward cannot be fitted; the entry’s claim that it ships in the package is wrong for this version. And under the default class_interactions = ‘common’, an x:.trt term in a regression formula is silently demoted to the class level, so an analyst who follows the published recipe by editing the formula alone fits the shared model twice and can never find evidence against sharing.
  • Two adversarial critiques before the run found nine issues between them that changed the design, and each found something the other missed. Peer review then found more, including two errors in this study’s own code: contraction was divided by the standard deviation of a prior the prior-sensitivity arm was not fitted under, which inverted the reported direction of that arm’s result, and the primary bound was a Wilson interval on the unweighted risk printed beside a deployment-weighted point estimate. Both were derived by reviewers from the published numbers alone, without access to the code.

Design.

Estimand
Marginal risk difference for the aggregate-only treatment versus the individual-data treatment in a target population, with the population-average conditional log-odds effect as the mechanism scale
Methods compared
shared class interaction, treatment-specific interaction for one covariate, treatment-specific interactions for all covariates, check first, then relax exactly what fired
Data-generating mechanism
A multilevel network meta-regression with a binomial outcome and a logit link, one individual-level study and the rest aggregate, two active treatments in one class of which only one has individual data. Drift is carried by one covariate and parameterized as the contrast between the two treatments’ interactions; a second covariate is shared exactly and supplies a type I error observation from the same fits. 32 cells crossing drift, network size, between-study covariate spread and treatment-effect heterogeneity, 50 replicates each, four fits per replicate forming the complete model lattice, plus eight amendment cells at the decision boundary, a prior-sensitivity arm and an integration-sensitivity check. The target population is not a design factor because the check reads only the fitted network and cannot depend on it; one set of fits is scored at four target displacements, and the reference intercept is solved separately for each so displacement moves the treatment contrast and not the baseline risk.
Replicates
50 per cell, 2,000 networks and 8,960 ML-NMR fits: 6,400 over the 32 registered cells, 1,600 in the amendment arm, 960 in the prior-sensitivity arm. That last arm ran at 30 replicates per cell against the 100 registered, a departure found in round-two review and disclosed in the protocol.

What this does not answer. The design is a best case in several specific ways that favour the check, and failures attributable to those idealizations are conservative; no general lower-bound claim is made, because three features cut the other way. The true marginal reference risk in the target is supplied to the estimator. Covariates are normal with known target moments and study covariate means are drawn independently of study baselines and of the random treatment effects, so the ecological route identifying the aggregate-only treatment’s interaction is unbiased by construction. Conditional constancy holds and only one covariate drifts, linearly. Against that: the fits are fixed-effect while half the cells generate random treatment effects, which is a second specification error that inflates firing; the likelihood is integrated over 64 quadrature points and refitting at 256 changes the DIC verdict in 8.3% of a 24-network subset, which is noise belonging to the implementation and not to the evidence, and 256 points is not demonstrated to be converged; and a single individual-level study is an assumption about the network rather than necessarily a pessimistic one. Beyond that: what a second individual-level study would buy is not measured, that factor having been dropped to pay for treatment-effect heterogeneity. With two active treatments and two covariates, multiplicity across covariates and the selection involved in reading the widest of several pairwise differences are not measured. Where treatment-effect heterogeneity is present the estimand is not cleanly defined, since truth is set at a random effect of zero while a fixed-effect model targets a function of the realized studies; primary results are repeated on the sixteen homogeneous cells and do not change. The hierarchical middle ground between shared and independent interactions could not be fitted because the package does not implement it, and we did not implement it ourselves: that is a reason for its absence, not a justification. The five decision rules scored here are our formalizations of an exploratory published practice, not published rules, and their operating characteristics are ours. Fifty replicates per cell gives a per-cell standard error of 0.065, so nothing is read off a single cell.

Peer review. 2 rounds, two independent reviewers. The reports, the point-by-point responses and the editorial decision are published in full: read the review.

Read it: PDF · Markdown · OpenDocument · Code and data: studies/IDN-05-shared-modifier-check-power · Protocol: registered before the run · Review: full history

What this project measured

Study studies/IDN-05-shared-modifier-check-power, run and decided against a rule registered before the run. Six-study networks, deployment-weighted over a displacement grid.

Registered rule. The check earns the reassurance it is read as giving if the upper 95% confidence bound on P(material error | the check did not fire) is below 0.10, where material error is a C-versus-A marginal risk difference wrong by more than 0.03 absolute risk.

Result: the rule fails. Upper bound 0.697 on realized error and 0.563 on systematic error, against a 0.10 threshold.

The qualification matters more than the verdict. An oracle that knows which specification is true, fitted to the same replicates, is materially wrong 0.639 of the time at the same displacement. The check’s excess over that oracle is 0.036, and at displacements 0 and 0.5 the excess is negative. A perfect check would have failed this gate too: with a 0.03 absolute-risk threshold, the estimation error of a six-study network alone exceeds it in 0.46 to 0.58 of replicates when the assumption holds exactly. The registered threshold was unreachable by any procedure, so this is evidence about what a six-study network can support, not only about the check.

Mechanism claims: decoupling confirmed (conditional risk rises 0.296 while the pass rate is flat in displacement); identification asymmetry partly confirmed (contraction gap widens monotonically, clearing 0.20 in 5 of 8 strata); the two rules differ on the null confirmed (type I gap 0.087); the two rules disagree not clearly met (0.107, 95% CI [0.096, 0.118], against a 0.10 threshold).

Verification

Verdict. Overstated. 2 independent auditors judged the claim as stated too strong. A published, software-implemented check on the shared effect modifier assumption exists (split the common class interaction into treatment-specific interactions and compare posteriors and model fit, as in Phillippo et al. 2023 with multinma’s class_interactions argument), but it has low and uncharacterized power, no calibration threshold, and no agreed stability metric, and it is infeasible in the small, IPD-poor networks where the assumption does the most identifying work.

The source titled this. “Shared effect modification is assumed rather than estimated or tested” The title says shared effect modification is never estimated or tested, but a published within-network check exists. (Its hierarchical relaxation does not ship in multinma: see the correction in the sources table.)

The source said. “Population adjustment commonly assumes that a covariate modifies the effects of different treatments in the same way, although the same modifier may act differently on different treatments. The only empirical check on offer, testing whether modifier interactions estimated in one IPD trial predict the treatment-effect heterogeneity observed in another, is not standardized and is rarely feasible.”

What was wrong with it. The source called cross-trial replication with two IPD trials the only empirical check on offer and proposed hierarchical shrinkage of treatment-specific interactions as future work. Both auditors found the check published in 2023 as a within-network procedure needing no second IPD trial (doi:10.1177/0272989X221117162) and the shrinkage already shipped in multinma’s class_interactions argument, so what remains open is power, calibration and a stability metric rather than the method itself.

Revised by the full-text reading. Applied 2026-07-26 full-text reading of the consolidated library (687 papers).

closure-upheld
15 findings from the reading assert this problem is open; the closure was re-examined and stands Put to an independent reviewer with the closing auditors’ reasoning, the work they relied on, and the new findings in view together. The new findings restate the closure’s residual: the Phillippo check exists but has low, uncalibrated power and is often infeasible when only one IPD study or sparse aggregate evidence is available. The 2022 Phillippo paper expressly concedes low power and multiplicity, which was already central to the closure rationale. Applied examples imposing a common interaction because flexible interactions are not estimable show non-adoption or non-feasibility in particular data structures, not that the cited within-network method fails to exist or falls outside its claimed scope. The 2026 transportability finding also supports the closure’s qualification that shared effect modification is not universally required for anchored pairwise adjustment. (reviewer confidence: high)
Flagged for reopening
16 findings from the reading assert this problem is open while the registry carries overstated Flagged, not applied: reversing an earlier audit’s closure needs the closure’s own evidence re-examined alongside this, which the reading did not do. See documentation/audit/reading/PROPOSED.md.
Literature and prior-art check
partially-solved / overstated. I read the full text of Phillippo et al. 2023 (Med Decis Making), which the entry does not cite, and it directly contradicts the statement’s ‘only empirical check on offer’. They assess the shared effect modifier assumption inside a single 9-study network with no second IPD trial, by splitting the common class interaction into treatment-specific interactions one covariate at a time and comparing posteriors and DIC, and they report a result (weak evidence against sharing for weight). The entry’s own proposed direction, hierarchical shrinkage of treatment-specific interactions, is already released software: multinma’s nma() takes class_interactions = c(“common”, “exchangeable”, “independent”) (verified in man/nma.Rd), so exchangeable within-class interactions are fittable today. What remains genuinely open is exactly what Phillippo et al. concede: ‘It is likely that this approach to assessing the shared effect modifier assumption has low power, particularly when data are lacking.’ Power, calibration, a stability metric rather than a binary verdict, and the data conditions under which the check is informative are unresolved. So the problem is real but the framing overstates the void by roughly one decade and one R package. Cites 10.1177/0272989X221117162: Provides and applies a standardized within-network empirical check on the shared effect modifier assumption, not requiring two IPD trials. Cites cran:multinma: Implements common, exchangeable (hierarchically shrunk) and independent treatment-class effect-modifier interactions, i.e. the entry’s proposed direction.
GPT-5.6 Sol, technical and source-code lens
partially-solved / overstated. C1 and C2 are correct: treatment-specific modification can differ, and shared conditional interactions do not make marginal odds or hazard ratios population-invariant. Hierarchical pooling is a coherent relaxation but regularizes weak information rather than creating identification. The shared-interaction restriction is not the universal mechanism that makes anchored PAIC work; an anchored analysis targeting the comparator-trial population can transport the IPD trial contrast and subtract the contrast observed in the target trial without assuming identical interactions for every treatment. Shared interactions become identifying restrictions in particular class, component, or sparse ML-NMR parameterizations. The proposed two-trial replication is also not the only empirical check and does not invariably require two complete IPD datasets: treatment-specific interactions can be compared within a mixed network, and sufficiently detailed subgroup aggregate data may support a limited check. Existing within-network relaxation and model comparison partially address the problem; power, calibration, and prior sensitivity remain open. Cites 10.1177/0272989X221117162: Demonstrates a within-network empirical assessment of shared effect modification by relaxing common interactions to treatment-specific interactions and comparing the resulting models, so cross-trial replication using two IPD trials is not the only available check..
Decision path
R3:partially-solved(n=2) → R8:overstated-majority(n=2)

How the evidence falls in time

16 papers in the reviewed corpus bear on this problem, published 2013 to 2026. Partial progress is on the record, and later work still calls this open. Both belong in view.

This problem is recurrent: three or more papers spanning at least eight years assert it independently. Sustained restatement by authors who mostly do not cite each other is the strongest evidence this reading can offer that a gap is real rather than one group’s framing.

Year Says Paper
2013 confirms it is open Evidence synthesis for decision making 3: heterogeneity - subgroups, meta-regression, bias, and bias-adjustment
2014 confirms it is open The importance of baseline viral load when assessing relative efficacy in treatment-naive HBeAg-positive chronic hepatitis B: a systematic review and network meta-analysis
2015 confirms it is open Response to: ‘Heterogeneity, consistency and model fit should be assessed in Bayesian network meta-analysis’ by Wei et al.
2016 confirms it is open Exploratory Network Meta Regression Analysis of Stroke Prevention in Atrial Fibrillation Fails to Identify Any Interactions with Treatment Effect
2017 confirms it is open Methods for Population-Adjusted Indirect Comparisons in Health Technology Appraisal
2018 partly addresses it Assessing the consistency assumptions underlying network meta-regression using aggregate data
2021 confirms it is open Methods for population adjustment with limited access to individual patient data: A review and simulation study
2021 confirms it is open Individual Participant Data Meta-Analysis: A Handbook for Healthcare Research (Riley, Tierney, Stewart, eds.; Wiley Series in Statistics in Practice, 2021)
2022 confirms it is open Validating the Assumptions of Population Adjustment: Application of Multilevel Network Meta-regression to a Network of Treatments for Plaque Psoriasis
2022 confirms it is open A Response to: Letter to the Editor Regarding “Network Meta-analysis of Food and Drug Administration-approved Treatment Options for Adults with Aquaporin-4 Immunoglobulin G-positive Neuromyelitis Optica Spectrum Disorder”
2024 confirms it is open Implementing Multilevel Network Meta-Regression for Time-To-Event Outcomes: A Case Study in Relapsed Refractory Multiple Myeloma
2024 confirms it is open An empirical study on 209 networks of treatments revealed intransitivity to be common and multiple statistical tests suboptimal to assess transitivity
2024 confirms it is open Twenty years of network meta-analysis: Continuing controversies and recent developments
2025 confirms it is open A critical assessment of matching-adjusted indirect comparisons in relation to target populations
2026 confirms it is open Population-adjusted network meta-analyses provide new insights into the efficacy of treatment alternatives for metastatic castration-sensitive prostate cancer
2026 confirms it is open Reframing Population-Adjusted Indirect Comparisons as a Transportability Problem: An Estimand-Based Perspective and Implications for Health Technology Assessment (Chandler & Ishak; arXiv:2602.17041v3). The batch file points at media/media/Table_1_v2.pdf; the full LaTeX source Transportability_PAIC.tex sits in the same arXiv package and was read.

Source

Derived from C:L214-215, C:L223, C:L403 of the reviewed corpus.