ML-NMR interactions can rest solely on aggregate-data variation
By none of them reliably. With one aggregate trial the data carry no information about the treatment’s interaction, yet the shared-interaction model’s credible interval was as narrow as the individual-data trial’s and covered 0.68 and 0.15 at discordance 0.15 and 0.3; a separate model returned its prior. With four aggregate trials the separate model covered nominally until an ecological term of 0.1 entered the between-trial gradient, then 0.71. The shared model’s apparent recovery there came from the ecological bias offsetting the discordance. Read the study below.
Statement
In mixed IPD and aggregate-data ML-NMR, interaction directions without adequate within-study IPD support may be weakly identified or nonidentified and are then sensitive to aggregate-study variation, shared-interaction restrictions and priors, and one coefficient forced to carry both within-study and between-study variation conflates individual-level effect modification with an ecological gradient. How far this bites is design- and parameter-specific rather than a categorical property of the method. The auditors disagreed on scope and the disagreement stands: the literature auditor found each claim confirmed almost verbatim by the ML-NMR authors, who report that the secukinumab interaction estimates rest solely on aggregate data and that the shared-interaction restriction was required to identify the model (doi:10.1177/0272989X221117162), while the source-code auditor judged the diagnostic void overstated, because cpaic, an unreleased development package on GitHub that cannot be installed from CRAN, already labels whether a row is identified by IPD, by between-study gradients or by neither, and flags posteriors that simply reproduce their priors.
Why it is open
Studies contributing only aggregate data inform covariate effects mainly through between-study contrasts, so when the model imposes one coefficient for both sources an ecological gradient is read as individual-level modification and the fit still looks good. Flags help but do not identify: labeling a parameter as aggregate-informed or as prior-driven tells a reader where to distrust the estimate without supplying the missing within-study information. The remedy the source proposes, splitting interactions by treatment, is described by the ML-NMR authors as possible given sufficient data and an interesting avenue for further research, so it has not been evaluated, and the general-likelihood extension (doi:10.1093/jrsssa/qnaf169) leaves the integration-distribution requirement intact.
What has been tried
| Work | What it contributes |
|---|---|
| Phillippo et al. 2023, Validating the Assumptions of Population Adjustment: Application of Multilevel Network Meta-regression to a Network of Treatments for Plaque Psoriasis, Medical Decision Making | assesses population-adjustment assumptions inside a nine-study network: splits the common class interaction into treatment-specific interactions one covariate at a time, checks conditional constancy through residual heterogeneity and inconsistency in sufficiently large networks, and reports that the shared-interaction restriction was required to identify the model where a treatment was informed only by aggregate data |
| Phillippo et al. 2025, Multilevel network meta-regression for general likelihoods (arXiv:2401.12640), Journal of the Royal Statistical Society Series A | extends ML-NMR to general likelihoods; the specified integration distribution for aggregate-data studies remains a requirement of the method |
| cpaic, unreleased development package on GitHub (not on CRAN), commit 9d150e9, R/estimability.R and R/plot-mlnmr.R | estimable_effects_at() labels whether a component by effect-modifier row is identified by IPD, by between-study covariate gradients or by neither, and plot_prior_posterior() exposes parameters whose posteriors reproduce their priors; these flag the problem rather than solving it |
Probable solution or research direction
Do not rebuild the flags; make them standard and test them. Evaluate in simulation whether identified-by labeling, prior-to-posterior contraction and prior-data conflict checks actually detect the within-versus-between conflation when the truth is known, and report them next to estimates rather than on request. Fit within-study centered specifications that separate within-study from between-study coefficients, and treatment-specific interactions with hierarchical shrinkage, the avenue the ML-NMR authors flag as unexplored, then characterize how much IPD each interaction needs before its posterior stops tracking its prior.
Our study
An aggregate-only treatment’s interaction is identified by assumption, by an ecological gradient or by the prior, and each fails differently
The question put to it. When a treatment has only aggregate data, is its effect-modifier interaction identified by the data, by the shared-interaction assumption, or by the prior, and do credible intervals cover when the assumption fails?
What it found. By none of them reliably. With one aggregate trial the data carry no information about the treatment’s interaction, yet the shared-interaction model’s credible interval was as narrow as the individual-data trial’s and covered 0.68 and 0.15 at discordance 0.15 and 0.3; a separate model returned its prior. With four aggregate trials the separate model covered nominally until an ecological term of 0.1 entered the between-trial gradient, then 0.71. The shared model’s apparent recovery there came from the ecological bias offsetting the discordance.
Results.
- Direct share of posterior precision (information about the interaction itself): zero for every model with one aggregate trial.
- Shared-model coverage at nonzero discordance without ecological bias: 0.149 to 0.867; separate model with a gradient: 0.942 to 0.955.
- With the ecological term, separate-model bias about 0.09 and coverage 0.71 to 0.74.
Design.
- Estimand
- Effect-modifier interaction of the aggregate-only treatment
- Methods compared
- shared-interaction model, separate interaction with a vague prior, hierarchical interaction
- Data-generating mechanism
- Exact conjugate-normal calculation: an individual-data trial for one treatment, one to four aggregate trials for another at spread covariate means, discordance between the two interactions and an ecological term; shared, separate and hierarchical models.
- Replicates
- Exact calculation over 36 cells
What this does not answer. Conjugate normal models on contrast-level data rather than multinma fits; one covariate; fixed prior scales. Peer review has not been done.
Peer review. Not yet done. The result stands on the study alone until it is.
Read it: PDF · Markdown · OpenDocument · Code and data: studies/IDN-06-within-between-in-bayesian-mlnmr · Protocol: registered before the run
Verification
Verdict. Partially addressed. Auditors disagreed and the disagreement was evidence-bearing on both sides. Reopened by the full-text reading: 5 findings from 5 papers survived both the reader and an independent second reviewer, so the verdict moves from unverifiable to partially-addressed.
The source titled this. “ML-NMR interaction parameters are only partially identified” The title states partial identification as a categorical property of the method, which the audit found to be design- and parameter-specific.
The source said. “ML-NMR relies on conditional constancy of relative effects, a correct individual-level model, a specified integration distribution, and shared-interaction restrictions where needed. Partial identifiability of its interaction parameters and conflation of within-study with between-study effect modification remain unresolved weaknesses.”
What was wrong with it. The source stated partial identifiability as a categorical weakness and said nothing in standard output separates a parameter learned from data from one supplied by the prior. The source-code auditor found identifiability parameter- and design-specific and found both flags already implemented in cpaic at commit 9d150e9, while the literature auditor found the claims confirmed by the ML-NMR authors themselves; the entry is published as unverifiable because both readings carry evidence, and it is now sourced to doi:10.1177/0272989X221117162, for which it previously cited nothing.
Dissent. codex dissented (overstated): C1 and C2 are technically sound: ML-NMR transports conditional relative-effect models and must integrate aggregate-study likelihood contributions over specified covariate distributions. C3 is too categorical. Interaction identifiability is parameter- and design-specific: interactions may be well identified by adequate within-trial IPD variation, weakly identified, or nonidentified. Aggregate arms can identify model-dependent integrated functionals and sometimes interaction combinations, rather than supplying only a generic ecological gradient. C4 is correct as a possibility when one coefficient is forced to represent both within-study and between-study variation. The report also overstates the diagnostic void for cpaic at 9d150e9: estimable_effects_at() labels whether information is from IPD, aggregate gradients, or neither, while plot_prior_posterior() exposes parameters whose posteriors reproduce their priors. These diagnostics flag rather than solve the underlying identification problem.
Revised by the full-text reading. Applied 2026-07-26 full-text reading of the consolidated library (687 papers).
- Verdict changed
- unverifiable -> partially-addressed 5 surviving findings from: A framework for identifying treatment-covariate interactions; An overview of methods for network meta-analysis using indiv; Comparing the use of aggregate data and various methods of i; Individual Participant Data Meta-Analysis: A Handbook for He; Network meta-analysis combining individual patient and aggre. New citations: 10.1002/jrsm.1300, 10.1177/0962280216660741, 10.1186/s12874-015-0007-0, 10.1186/s12874-021-01254-5
- Literature and prior-art check
- open / supported. This entry has no prior_work at all, yet every one of its four claims is confirmed almost verbatim by a paper by the ML-NMR authors themselves, which I read in full. On partial identifiability: in the psoriasis network the secukinumab interaction estimates ‘are much more uncertain than for the ixekizumab regimens, as the secukinumab parameters are based solely on AgD, and the ixekizumab data largely drive the interaction estimates in the common interaction model’, and the shared-interaction restriction was ‘required to identify the model’. On within-versus-between conflation, Phillippo et al. state the ecological-bias mechanism explicitly and flag the entry’s own proposed remedy as unexplored: ‘it is possible to fit ML-NMR models where the interaction terms are split in this manner (given sufficient data), and this is an interesting avenue for further research.’ That is a source asserting the gap, which is the strongest evidence a ‘this is open’ claim can have. The integration-distribution requirement is intrinsic to ML-NMR and restated in the general-likelihood extension (arXiv 2401.12640 / JRSS-A 10.1093/jrsssa/qnaf169). I recommend the catalog attach Phillippo et al. 2023 as prior_work rather than publishing this entry unsourced.
- GPT-5.6 Sol, technical and source-code lens
- open / overstated. C1 and C2 are technically sound: ML-NMR transports conditional relative-effect models and must integrate aggregate-study likelihood contributions over specified covariate distributions. C3 is too categorical. Interaction identifiability is parameter- and design-specific: interactions may be well identified by adequate within-trial IPD variation, weakly identified, or nonidentified. Aggregate arms can identify model-dependent integrated functionals and sometimes interaction combinations, rather than supplying only a generic ecological gradient. C4 is correct as a possibility when one coefficient is forced to represent both within-study and between-study variation. The report also overstates the diagnostic void for cpaic at 9d150e9: estimable_effects_at() labels whether information is from IPD, aggregate gradients, or neither, while plot_prior_posterior() exposes parameters whose posteriors reproduce their priors. These diagnostics flag rather than solve the underlying identification problem.
- Decision path
-
R5:default-open → R7:counter-and-support-both-present
How the evidence falls in time
15 papers in the reviewed corpus bear on this problem, published 2013 to 2024. Partial progress is on the record, and later work still calls this open. Both belong in view.
This problem is recurrent: three or more papers spanning at least eight years assert it independently. Sustained restatement by authors who mostly do not cite each other is the strongest evidence this reading can offer that a gap is real rather than one group’s framing.
It is also being closed in pieces: partial results come from more than one paper in more than one year, each covering a different part.
| Year | Says | Paper |
|---|---|---|
| 2013 | confirms it is open | Evidence synthesis for decision making 3: heterogeneity - subgroups, meta-regression, bias, and bias-adjustment |
| 2014 | confirms it is open | Network meta-analysis of (individual patient) time to event data alongside (aggregate) count data |
| 2015 | partly addresses it | Network meta-analysis combining individual patient and aggregate data from a mixture of study designs with an application to pulmonary arterial hypertension |
| 2016 | partly addresses it | An overview of methods for network meta-analysis using individual participant data: when do benefits arise? |
| 2017 | confirms it is open | Network meta-analysis including treatment by covariate interactions: Consistency can vary across covariate values |
| 2018 | partly addresses it | A framework for identifying treatment-covariate interactions in individual participant data network meta-analysis |
| 2020 | confirms it is open | Multilevel network meta-regression for population-adjusted treatment comparisons |
| 2021 | partly addresses it | Individual Participant Data Meta-Analysis: A Handbook for Healthcare Research (Riley, Tierney, Stewart, eds.; Wiley Series in Statistics in Practice, 2021) |
| 2021 | partly addresses it | Comparing the use of aggregate data and various methods of integrating individual patient data to network meta-analysis and its application to first-line ART |
| 2021 | confirms it is open | A frequentist one-step model for a simple network meta-analysis of time-to-event data in presence of an effect modifier |
| 2022 | confirms it is open | Validating the Assumptions of Population Adjustment: Application of Multilevel Network Meta-regression to a Network of Treatments for Plaque Psoriasis |
| 2022 | confirms it is open | Calibrating a network meta-analysis of diabetes trials of sodium glucose cotransporter 2 inhibitors, glucagon-like peptide-1 receptor analogues and dipeptidyl peptidase-4 inhibitors to a representative routine population: a systematic review protocol |
| 2023 | confirms it is open | Individual participant data from digital sources informed and improved precision in the evaluation of predictive biomarkers in Bayesian network meta-analysis |
| 2024 | confirms it is open | Methods for Indirect Treatment Comparison: Results from a Systematic Literature Review |
| 2024 | confirms it is open | Comparing the efficacy of cipaglucosidase alfa plus miglustat with other enzyme replacement therapies for late-onset Pompe disease: a network meta-analysis utilizing patient-level and aggregate data |
Source
Derived from C:L187-188, C:L195 of the reviewed corpus.