Shared Gamma conflates within- and between-study modification
Fitting one shared interaction matrix on a globally centered covariate is not pooling: for a binary modifier with study prevalence p_s it is algebraically identical to imposing that the causal within-trial coefficient equals the confounded across-trial one. In the model simulated, coverage of a nominal 95% interval for the causal interaction falls from 93.8% when that happens to be true to 17.2% when the two are equal and opposite, and further in networks whose contrasts move two components at once. Separating them costs about 5% in interval width and removes an order of magnitude of bias. The catalog’s stated cause is incomplete: the loss is present with all twelve trials supplying individual data, so aggregate data are not necessary for it, but removing individual data roughly doubles the bias, so they amplify it. Read the study below.
Statement
Component ML-NMR estimates a single shared component-by-modifier interaction matrix Gamma from IPD and aggregate data together and does not separate within-trial from across-trial interaction information, so aggregate studies can pull the estimate toward a non-randomized across-trial association. It does not, however, plug in covariate means: ML-NMR integrates an individual-level model over each study’s covariate distribution precisely to avoid aggregation bias, so the ecological mechanism is not the one the source described. Aggregate data are also not categorically uninformative about Gamma, because a nonlinear integrated likelihood contains curvature-based information; in the implemented log-link case the aggregate data identify beta + log cosh(gamma) rather than beta alone. The within-trial and across-trial separation is standard practice in IPD meta-analysis and simply has not been ported to component models.
Why it is open
Between-study variation in covariate distributions is not randomized, so any association between a study’s covariate profile and its treatment effect is confounded by everything else that differs between studies, and a single Gamma lets that association share a parameter with the within-trial causal modification. The published fix for ordinary IPD meta-analysis, centering patient-level covariates at their trial mean so within-trial and across-trial interactions are separately estimated, has not been implemented for component ML-NMR, where the interaction is additionally shared across components and across subnetworks. Current implementations mitigate only by labeling whether a target contrast is supported by IPD or by aggregate arms and by excluding aggregate-only screen results from default rankings.
What has been tried
| Work | What it contributes |
|---|---|
| Hua et al. 2017 | shows that centering patient-level covariates at their trial mean is required to separate within-trial from across-trial interaction information, with a worked reversal of an interaction estimate |
| Freeman et al. 2018 | extends the within-trial and across-trial separation to IPD network meta-analysis and gives a practical framework including consistency checks between the two sources |
| Phillippo et al. 2020 | defines ML-NMR as integrating an individual-level model over each study’s covariate distribution rather than regressing on covariate means, which is what avoids aggregation bias |
| Phillippo et al. 2023 | describes ML-NMR as coherently synthesizing IPD and AgD while avoiding aggregation bias and non-collapsibility bias |
| cpaic 0.1.0 (development version, choxos/cpaic@9d150e9) | unreleased development R package on GitHub, not on CRAN; fits one shared Gamma across IPD and AgD, labels each target contrast as identified by IPD, aggregate evidence, or neither, and excludes aggregate-only elements from default rankings |
Probable solution or research direction
Port the established within-trial and across-trial separation into component ML-NMR: estimate Gamma_W and Gamma_B as distinct parameters and restrict causal interaction learning to the directions actually informed by IPD, with the across-trial parameter reported separately rather than pooled in. Use the existing framework for identifying treatment-covariate interactions in IPD network meta-analysis, including its visual consistency checks between the two information sources, as the template. Keep the labeling of which contrasts are IPD-supported, since that is the partial mitigation already in place.
Our study
Separating within-trial from across-trial component effect modification
The question put to it. Component ML-NMR fits one shared interaction on a globally centered covariate, which algebraically forces the causal within-trial interaction to equal the confounded across-trial association. How much does that cost, when, and is the damage really caused by aggregate data as the catalog claims?
What it found. Fitting one shared interaction matrix on a globally centered covariate is not pooling: for a binary modifier with study prevalence p_s it is algebraically identical to imposing that the causal within-trial coefficient equals the confounded across-trial one. In the model simulated, coverage of a nominal 95% interval for the causal interaction falls from 93.8% when that happens to be true to 17.2% when the two are equal and opposite, and further in networks whose contrasts move two components at once. Separating them costs about 5% in interval width and removes an order of magnitude of bias. The catalog’s stated cause is incomplete: the loss is present with all twelve trials supplying individual data, so aggregate data are not necessary for it, but removing individual data roughly doubles the bias, so they amplify it.
Results.
- Coverage under the shared restriction degrades monotonically with the discordance between the within-trial and across-trial coefficients, from 93.8% at no discordance to 17.2% when they are equal and opposite, and it is worse in bundled component networks whose contrasts move two components at once than where every contrast isolates one.
- Both separated estimators are nominal at every discordance examined, 92.6% to 96.3%, including under a nonlinear across-trial pattern where the linear split model is itself misspecified.
- CMP-13 says aggregate studies cause the pull. With all twelve trials supplying individual patient data the shared model still has median absolute standardized bias 0.15 and coverage as low as 44.6%, so aggregate data are not necessary. But bias rises to 0.29 and coverage falls to 17.2% as individual data are removed, so they amplify it. The registered refutation threshold was narrowly missed (47% of cells against 50%) and we report that rather than moving it.
- Where a component appears in no individual-data arm, the three methods behave in three different ways: the IPD-anchored estimator declines to report it in 100% of replicates; the shared model returns a confident -0.34 against a truth of zero, sourced entirely from the across-trial gradient; and the joint split returns intervals so wide that coverage approaches one, when it returns one at all.
- A study-clustered sandwich, included so the comparison could not be won on the variance estimator, covered 75.4% to 92.4% where the shared model is correctly specified. With twelve clusters it is badly anticonservative, and it is reported as a finding rather than used as the comparator.
Design.
- Estimand
- Within-trial component interaction on the conditional log-rate scale
- Methods compared
- shared-info (status quo), shared-sandwich, joint-split, ipd-anchored
- Data-generating mechanism
- Twelve-trial two-component Poisson network with a binary effect modifier whose prevalence varies across studies, aggregate arms generated by exact integration. 320 scenarios crossing network structure, the discordance between within-trial and across-trial modification, IPD availability including a case where one component has no within-trial data at all, arm size and prevalence spread. 2000 replicates each.
- Replicates
- 2000 per scenario, 640,000 networks
What this does not answer. The registered global analysis is uninformative and the regime split that produced a usable answer was prompted by observing that failure, so it is not prespecified. The general headline about component ML-NMR is not established: the likelihood and nuisance structures are not mapped onto the fixed- and random-effect specifications the implemented method offers, and free study-by-treatment effects or priors can absorb or alter the term the identity isolates. A maximum-likelihood analogue of component ML-NMR. Implemented ML-NMR is Bayesian and priors matter most where interactions are weakly identified, so the coverage numbers do not transfer to multinma; the algebra does, and therefore so does the direction. One binary modifier, a log link, a Poisson outcome, additive components with no synergy, no random treatment-effect heterogeneity, no inconsistency, and exactly reported study prevalences. Under the discordance patterns the split model is correctly specified, so the study measures the cost of an algebraically known restriction rather than establishing that real networks contain within/across disagreement or how large it typically is. Continuous or multiple correlated modifiers, misspecified aggregate covariate distributions, and time-to-event outcomes are untested. It does not touch IDN-05.
Peer review. 2 rounds, two independent reviewers. The reports, the point-by-point responses and the editorial decision are published in full: read the review.
Read it: PDF · Markdown · OpenDocument · Code and data: studies/CMP-13-within-between-interaction · Protocol: registered before the run · Review: full history
What this project measured
Study studies/CMP-13-within-between-interaction, run and decided against a rule registered before the run. 320 scenarios, 2000 replicates each, split before the run into a primary regime of 256 scenarios where every component carries within-trial information and 64 where one component carries none. The split is on a factor level fixed in advance, not on an observed outcome.
Conclusion: material at mild discordance. Coverage of the shared model degrades with discordance while the alternatives hold nominal throughout:
| discordance | shared | joint split | IPD anchored |
|---|---|---|---|
| rho = 1 | 0.939 to 0.958 | 0.938 to 0.960 | 0.938 to 0.959 |
| rho = 0 | 0.655 to 0.963 | 0.936 to 0.960 | 0.939 to 0.961 |
| rho = -1 | 0.172 to 0.960 | 0.926 to 0.960 | 0.931 to 0.963 |
The worst cell reaches 0.172 coverage. Note also that the cluster-robust comparator fails its own negative controls in 128 of 128 control scenarios (coverage 0.754 to 0.924), so it is reported but cannot serve as the status quo.
The catalog’s mechanism claim is not supported, and this is the useful part. This entry attributes the pull to aggregate studies. The study tested that directly by giving all twelve trials individual data: across 80 discordant scenario-components with complete individual data, the shared model still shows median absolute standardized bias 0.15, maximum 2.03, with 0.47 of them above 0.20 and coverage as low as 0.447. The registered support threshold (bias below 0.10 everywhere) fails; the registered refutation threshold (more than half above 0.20) is not quite met. So the claim is recorded as not supported, and the direction of the evidence is that the conflation is a property of the parameterization, not of the data type — which relocates the problem this entry describes.
Verification
Verdict. Overstated. 2 independent auditors judged the claim as stated too strong. Component ML-NMR would estimate a single shared component-by-modifier interaction matrix and, like ML-NMR generally, would not separate within-trial from across-trial interaction information, so aggregate studies can still pull the estimate toward a non-randomised across-trial association; unlike standard network meta-regression it does not plug in covariate means, so it does not incur aggregation bias from nonlinearity, and the within/across separation is already standard practice in IPD meta-analysis and merely needs porting to component models.
The source said. “Component ML-NMR estimates one shared component-by-modifier interaction matrix Gamma using both IPD and aggregate data. Aggregate covariate means identify ecological, between-study gradients, whereas randomization within a trial identifies only within-study effect modification, so combining them in one parameter conflates the two.”
What was wrong with it. The source said aggregate covariate means identify ecological gradients in ML-NMR; that describes standard network meta-regression, not ML-NMR, which was built to avoid aggregation bias by integrating over the covariate distribution (doi:10.1111/rssa.12579; doi:10.1177/0272989X221117162). The literature auditor also found the fix already exists outside component models: trial-mean centering separates within-trial from across-trial interaction information (Hua et al. 2017; Freeman et al. 2018), and GPT-5.6 Sol added that nonlinear aggregate likelihoods carry curvature-based information about Gamma, so the aggregate contribution is not purely ecological.
Revised by the full-text reading. Applied 2026-07-26 full-text reading of the consolidated library (687 papers).
- Citation corrected
- normalized a citation year: ‘Phillippo et al. 2022’ -> ‘Phillippo et al. 2023’ 10.1177/0272989X221117162 is cited with two different years across the registry. CrossRef gives 2022 online and 2023 in print; the issue year is used throughout.
- closure-upheld
- 8 findings from the reading assert this problem is open; the closure was re-examined and stands Put to an independent reviewer with the closing auditors’ reasoning, the work they relied on, and the new findings in view together. The new findings directly restate the closure’s residual: component ML-NMR and related ML-NMR models use unsplit interactions, while within-trial versus across-trial separation remains a method available in ordinary IPD meta-analysis but not yet ported to these models. Phillippo et al. 2020 and 2022 explicitly acknowledge the unsplit interaction and describe a split ML-NMR as possible future work, which supports rather than contradicts the closure. The findings do not show that ML-NMR uses covariate means, that aggregate likelihoods are categorically uninformative about Gamma, or that trial-mean centering has already been implemented for component ML-NMR. The closure already identifies the genuine open work: implementing the within/across decomposition in component ML-NMR. (reviewer confidence: high)
- Flagged for reopening
-
10 findings from the reading assert this problem is open while the registry carries
overstatedFlagged, not applied: reversing an earlier audit’s closure needs the closure’s own evidence re-examined alongside this, which the reading did not do. See documentation/audit/reading/PROPOSED.md. - Literature and prior-art check
- partially-solved / overstated. Two problems. First, the mechanism attributed to ML-NMR is wrong: ML-NMR does not regress on aggregate covariate means. Phillippo et al. 2020 built it precisely because ‘Existing meta-regression approaches incur aggregation bias’, by integrating an individual-level model over the covariate distribution; Phillippo et al. 2022 describe ML-NMR as ‘avoiding aggregation bias and noncollapsibility bias’. So ‘Aggregate-data covariate means identify ecological gradients’ describes standard network meta-regression, not ML-NMR. Second, the residual worry - that a single shared interaction does not separate within-trial from across-trial information - is real but already solved methodologically outside component models. Hua et al. 2017 show the fix (‘it is crucial to centre patient-level covariates by their mean value in each trial, in order to separate out within-trial and across-trial information’) with a worked reversal (interaction -0.011, p = 0.004 amalgamated, versus -0.007, p = 0.22 separated), and Freeman et al. 2018 give a full framework for IPD network meta-analysis. What is open is porting that separation into component ML-NMR. Cites doi:10.1002/sim.7171: Gives the established fix (trial-mean centring) that separates within-trial from across-trial interaction information; not yet applied to component or ML-NMR models.. Cites doi:10.1002/jrsm.1300: Extends the within/across separation to network meta-analysis and gives a practical framework including visual consistency checks between the two sources..
- GPT-5.6 Sol, technical and source-code lens
- partially-solved / overstated. C3 and C4 are correct: randomization does not randomize study-level covariate distributions, and cpaic fits only one Gamma. C1 describes a real ecological-bias risk when aggregate information about Gamma comes from across-study distribution shifts. C2 is too categorical across the supported families. For an identity-link, first-moment aggregate contrast, the centered within-study interaction averages out. In a nonlinear integrated likelihood, however, an aggregate arm mean depends on the interaction through the entire covariate distribution; cpaic’s own log-link example identifies beta + log cosh(gamma). Such information is model-dependent and generally insufficient for nonparametric causal effect-modification identification, but it is not literally zero or reducible solely to a gradient in covariate means. The model-level separation remains open. cpaic partially mitigates it by labeling whether target contrasts are supported by IPD or only aggregate arms and excluding aggregate-only screen results from rankings by default. Cites choxos/cpaic@9d150e9: Labels each target contrast as identified by IPD, aggregate evidence, or neither, exposing the information source.. Cites choxos/cpaic@9d150e9: Prevents aggregate-only first-order screens from entering default treatment or component hierarchies..
- Decision path
-
R3:partially-solved(n=2) → R8:overstated-majority(n=2)
How the evidence falls in time
10 papers in the reviewed corpus bear on this problem, published 2018 to 2022. Partial progress is on the record, and later work still calls this open. Both belong in view.
It is also being closed in pieces: partial results come from more than one paper in more than one year, each covering a different part.
| Year | Says | Paper |
|---|---|---|
| 2018 | partly addresses it | A framework for identifying treatment-covariate interactions in individual participant data network meta-analysis |
| 2019 | confirms it is open | Dismantling, personalising and optimising internet cognitive-behavioural therapy for depression: a study protocol for individual participant data component network meta-analysis |
| 2020 | confirms it is open | Multilevel network meta-regression for population-adjusted treatment comparisons |
| 2021 | partly addresses it | Individual Participant Data Meta-Analysis: A Handbook for Healthcare Research (Riley, Tierney, Stewart, eds.; Wiley Series in Statistics in Practice, 2021) |
| 2021 | confirms it is open | Behavioural interventions for smoking cessation: an overview and network meta-analysis |
| 2021 | confirms it is open | Supplementary appendix to Furukawa et al., Dismantling, optimising, and personalising internet cognitive behavioural therapy for depression: a systematic review and component network meta-analysis using individual participant data |
| 2022 | confirms it is open | Impact of program characteristics on weight loss in adult behavioral weight management interventions: systematic review and component network meta-analysis |
| 2022 | confirms it is open | Meta-Research: Methods and Protocols (Evangelou E, Veroniki AA, eds), Methods in Molecular Biology vol. 2345 |
| 2022 | confirms it is open | Validating the Assumptions of Population Adjustment: Application of Multilevel Network Meta-regression to a Network of Treatments for Plaque Psoriasis |
| 2022 | confirms it is open | Bayesian models for aggregate and individual patient data component network meta-analysis |
Source
Derived from C:L1367, C:L1237, scope:field-level (ML-NMR generally; implemented in cpaic) of the reviewed corpus.