Ecological versus within-study effect modification
Statement
Aggregate study means inform between-study associations, not individual-level treatment-by-covariate interactions, so meta-regression on study means can misattribute an ecological association to individual effect modification and then transport it to a target population. This is established rather than new: it is Berlin et al. 2002 (doi:10.1002/sim.1023), and the remedy the entry proposes, separating within-trial from across-trial information by trial-mean centering, was formalized, simulated and demonstrated by Hua et al. 2017 (doi:10.1002/sim.7171). The auditors split on ML-NMR (doi:10.1111/rssa.12579): the literature auditor read it as avoiding the ecological bias of aggregate-data network meta-regression, while GPT-5.6 Sol held that it avoids mean plug-in aggregation but that an interaction informed only by aggregate arms stays ecological unless within-study information or an identifying restriction separates the coefficients, pointing to a development implementation that fits one interaction matrix for IPD covariates and aggregate means and thereby imposes equality of the two (choxos/cpaic@9d150e9). What remains open is aggregate-only interaction identification and a formal ecological-bias sensitivity analysis over the gap between the coefficients, since an IPD-informed prior transfers information under exchangeability but does not create likelihood identification.
Why it is open
In an aggregate-data study the covariate takes a single value per arm, so no within-study contrast exists to identify the individual-level interaction and the fitted coefficient is a between-study slope confounded with everything else that varies across studies. The two coefficients coincide only under an assumption the aggregate data cannot check, and because the regression is well behaved either way the substitution leaves no signature in fit statistics. Where IPD supply within-study variation the remedy is known and validated, so the residual problem is specifically the aggregate-only case, in which the identifying restriction is imposed by the model rather than tested by the data.
What has been tried
| Work | What it contributes |
|---|---|
| Berlin et al. 2002, Individual patient- versus group-level data meta-regressions for the investigation of treatment effect modifiers: ecological bias rears its ugly head, Statistics in Medicine (PMID 11813224) | Original source for the entry’s central claim that group-level meta-regression can substitute an ecological association for an individual-level interaction. |
| Hua, Burke, Crowther, Ensor, Tudur Smith & Riley 2017, One-stage individual participant data meta-analysis models: estimation of treatment-covariate interactions must avoid ecological bias by separating out within-trial and across-trial information, Statistics in Medicine (PMID 27910122) | Formalizes and validates the entry’s proposed remedy, showing by simulation and an applied epilepsy example that trial-mean centering separates within-trial from across-trial information and that failing to do so distorts interaction estimates. |
| Phillippo et al. 2020, Multilevel network meta-regression for population-adjusted treatment comparisons, JRSS-A (PMID 32684669) | Integrates an individual-level model over the aggregate covariate distribution rather than plugging in means, removing the aggregation bias of standard aggregate-data meta-regression. Whether that also removes ecological bias in an aggregate-only interaction is what the two auditors disputed. |
| cpaic development source at commit 9d150e9 (choxos/cpaic) | Source-code evidence cited by GPT-5.6 Sol that fitting one interaction matrix across IPD covariates and aggregate means imposes equality of the within-study and between-study interaction. cpaic is an unreleased development package on GitHub; an auditor confirmed it is not on CRAN and cannot be installed from there. |
Probable solution or research direction
Use the established remedy wherever IPD allow it: separate within-study and between-study coefficients by trial-mean centering, and integrate an individual-level model over the aggregate covariate distribution rather than plugging in means. The work genuinely left is the aggregate-only case: report which coefficient a transported effect relies on, and run an explicit ecological-bias sensitivity analysis over the gap between the within-study and between-study coefficients rather than assuming it is zero. IPD-informed priors on the individual-level interaction can regularize a posterior and should be reported as an assumption, not as identification.
Verification
Verdict. Partially addressed. Auditors disagreed and the disagreement was evidence-bearing on both sides. Reopened by the full-text reading: 7 findings from 7 papers survived both the reader and an independent second reviewer, so the verdict moves from unverifiable to partially-addressed.
The source said. “Aggregate-data study means identify between-study gradients rather than individual-level treatment-by-covariate interactions, so meta-regression on study means can misattribute an ecological association to individual effect modification and then transport it to a target population.”
What was wrong with it. The source stated a result established in 2002 with an empty prior_work list; the literature auditor sourced it to Berlin et al. 2002 (doi:10.1002/sim.1023) and its proposed remedy to Hua et al. 2017 (doi:10.1002/sim.7171), so the claim is now sourced. The two auditors disagreed about how much ML-NMR (doi:10.1111/rssa.12579) resolves, and both readings are kept above.
Dissent. literature dissented (supported): prior_work is empty, but every substantive claim here is established literature I retrieved. The problem statement is Berlin et al. 2002, Stat Med 21:371-387 (PMID 11813224), whose title is literally ‘ecological bias rears its ugly head’. The entry’s own proposed remedy (separating within-study and between-study coefficients by mean-centering) was formalized, simulated and empirically demonstrated by Hua, Burke, Crowther, Ensor, Tudur Smith & Riley, Stat Med 2017 (PMID 27910122), which reports the epilepsy example where the age interaction moves from -0.011 (95% CI -0.019 to -0.003, p=0.004) when within- and across-trial information are amalgamated to -0.007 (95% CI -0.019 to 0.005, p=0.22) when they are separated. The ML-NMR claim is directly supported by Phillippo et al. 2020 JRSS-A. So claims 1-3 are solved and established, not ‘Promising’. What remains open is claim 4 (IPD-informed priors on the individual-level interaction) and formal ecological-bias sensitivity analysis over the within/between gap; I found no PAIC-specific treatment of either. Maturity rating ‘Promising’ understates a 2002-2017 established literature. codex dissented (overstated): C1 is correct if ‘identify’ means estimate a study-level association: aggregate means contain no within-study covariate contrast and cannot data-identify an individual treatment-by-covariate interaction. C2 is the standard remedy when IPD provide within-study variation. C3 is only conditionally true. ML-NMR avoids mean plug-in aggregation by integrating an individual-level model, but an interaction informed only by aggregate arms remains ecological unless within-study information or an identifying restriction separates the coefficients. cpaic ground truth is explicit: cmlnmr() uses one Gamma for IPD covariates and aggregate means and therefore imposes equality of within-study and between-study interactions. C4 is also too strong: an IPD-informed prior can transfer information under exchangeability assumptions and regularize a posterior, but a prior does not create likelihood identification. The broad ecological-bias problem and its main remedy are established; aggregate-only interaction identification remains structurally unresolved and decision-critical when a transported contrast relies on it.
Revised by the full-text reading. Applied 2026-07-26 full-text reading of the consolidated library (687 papers).
- Verdict changed
- unverifiable -> partially-addressed 7 surviving findings from: A framework for identifying treatment-covariate interactions; A frequentist one-step model for a simple network meta-analy; Deriving percentage study weights in multi-parameter meta-an; Individual Participant Data Meta-Analysis: A Handbook for He; Meta-Research: Methods and Protocols (Evangelou E, Veroniki ; Network meta-interpolation: Effect modification adjustment i; The importance of baseline viral load when assessing relativ. New citations: 10.1002/jrsm.1300, 10.1002/jrsm.1608, 10.1007/978-1-0716-1566-9, 10.1177/0962280216688033, 10.1186/2046-4053-3-21, 10.1371/journal.pone.0259121
- Literature and prior-art check
- partially-solved / supported. prior_work is empty, but every substantive claim here is established literature I retrieved. The problem statement is Berlin et al. 2002, Stat Med 21:371-387 (PMID 11813224), whose title is literally ‘ecological bias rears its ugly head’. The entry’s own proposed remedy (separating within-study and between-study coefficients by mean-centering) was formalized, simulated and empirically demonstrated by Hua, Burke, Crowther, Ensor, Tudur Smith & Riley, Stat Med 2017 (PMID 27910122), which reports the epilepsy example where the age interaction moves from -0.011 (95% CI -0.019 to -0.003, p=0.004) when within- and across-trial information are amalgamated to -0.007 (95% CI -0.019 to 0.005, p=0.22) when they are separated. The ML-NMR claim is directly supported by Phillippo et al. 2020 JRSS-A. So claims 1-3 are solved and established, not ‘Promising’. What remains open is claim 4 (IPD-informed priors on the individual-level interaction) and formal ecological-bias sensitivity analysis over the within/between gap; I found no PAIC-specific treatment of either. Maturity rating ‘Promising’ understates a 2002-2017 established literature. Cites doi:10.1002/sim.7171 (PMID 27910122): Establishes and validates the entry’s proposed remedy (separate within-study and between-study coefficients via trial-mean centering) with simulation and an applied demonstration.. Cites doi:10.1111/rssa.12579 (PMID 32684669): Confirms claim 3: ML-NMR avoids the aggregation/ecological bias of aggregate-data network meta-regression by integrating an individual-level model over the aggregate covariate distribution.. Cites doi:10.1002/sim.1023 (PMID 11813224): Original source for claim 1; the entry states a 24-year-old established result with no citation..
- GPT-5.6 Sol, technical and source-code lens
- partially-solved / overstated. C1 is correct if ‘identify’ means estimate a study-level association: aggregate means contain no within-study covariate contrast and cannot data-identify an individual treatment-by-covariate interaction. C2 is the standard remedy when IPD provide within-study variation. C3 is only conditionally true. ML-NMR avoids mean plug-in aggregation by integrating an individual-level model, but an interaction informed only by aggregate arms remains ecological unless within-study information or an identifying restriction separates the coefficients. cpaic ground truth is explicit: cmlnmr() uses one Gamma for IPD covariates and aggregate means and therefore imposes equality of within-study and between-study interactions. C4 is also too strong: an IPD-informed prior can transfer information under exchangeability assumptions and regularize a posterior, but a prior does not create likelihood identification. The broad ecological-bias problem and its main remedy are established; aggregate-only interaction identification remains structurally unresolved and decision-critical when a transported contrast relies on it. Cites doi:10.1002/sim.7171: Develops the within-study centering and separate within-trial versus across-trial coefficients proposed as the main remedy.. Cites doi:10.1111/rssa.12579: Provides ML-NMR integration of an individual-level outcome model over aggregate-study covariate distributions, addressing mean plug-in aggregation while retaining the need for interaction identification..
- Decision path
-
R3:partially-solved(n=2) → R7:counter-and-support-both-present
How the evidence falls in time
50 papers in the reviewed corpus bear on this problem, published 2011 to 2026. Partial progress is on the record, and later work still calls this open. Both belong in view.
This problem is recurrent: three or more papers spanning at least eight years assert it independently. Sustained restatement by authors who mostly do not cite each other is the strongest evidence this reading can offer that a gap is real rather than one group’s framing.
It is also being closed in pieces: partial results come from more than one paper in more than one year, each covering a different part.
| Year | Says | Paper |
|---|---|---|
| 2011 | confirms it is open | Conducting indirect-treatment-comparison and network-meta-analysis studies: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 2 |
| 2011 | confirms it is open | Interpreting indirect treatment comparisons and network meta-analysis for health-care decision making: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 1 |
| 2012 | confirms it is open | Efficacy of once-daily indacaterol 75 microgram relative to alternative bronchodilators in COPD: a study level and a patient level network meta-analysis |
| 2012 | confirms it is open | Meta-regression models to address heterogeneity and inconsistency in network meta-analysis of survival outcomes |
| 2013 | confirms it is open | Evidence synthesis for decision making 3: heterogeneity - subgroups, meta-regression, bias, and bias-adjustment |
| 2013 | confirms it is open | Is network meta-analysis as valid as standard pairwise meta-analysis? It all depends on the distribution of effect modifiers |
| 2014 | partly addresses it | The importance of baseline viral load when assessing relative efficacy in treatment-naive HBeAg-positive chronic hepatitis B: a systematic review and network meta-analysis |
| 2014 | confirms it is open | A process for assessing the feasibility of a network meta-analysis: a case study of everolimus in combination with hormonal therapy versus chemotherapy for advanced breast cancer |
| 2015 | confirms it is open | Accounting for Heterogeneity in Relative Treatment Effects for Use in Cost-Effectiveness Models and Value-of-Information Analyses |
| 2015 | confirms it is open | From Randomized Controlled Trials of Antidepressant Drugs to the Meta-Analytic Synthesis of Evidence: Methodological Aspects Lead to Discrepant Findings |
| 2015 | confirms it is open | Methods and characteristics of published network meta-analyses using individual patient data: protocol for a scoping review |
| 2016 | confirms it is open | Exploratory Network Meta Regression Analysis of Stroke Prevention in Atrial Fibrillation Fails to Identify Any Interactions with Treatment Effect |
| 2016 | confirms it is open | GetReal in network meta-analysis: a review of the methodology |
| 2016 | confirms it is open | Statistical Models and Methods for Network Meta-Analysis |
| 2017 | partly addresses it | Deriving percentage study weights in multi-parameter meta-analysis models: with application to meta-regression, network meta-analysis and one-stage individual participant data models |
| 2017 | confirms it is open | Bayesian one-step IPD network meta-analysis of time-to-event data using Royston-Parmar models |
| 2017 | confirms it is open | Network meta-analysis including treatment by covariate interactions: Consistency can vary across covariate values |
| 2018 | partly addresses it | A framework for identifying treatment-covariate interactions in individual participant data network meta-analysis |
| 2018 | confirms it is open | Component network meta-analysis identifies the most effective components of psychological preparation for adults undergoing surgery under general anesthesia |
| 2018 | confirms it is open | Assessing the consistency assumptions underlying network meta-regression using aggregate data |
| 2018 | confirms it is open | Graphs of study contributions and covariate distributions for network meta-regression |
| 2020 | confirms it is open | Optimal doses of antidepressants in dependence on age: Combined covariate actions in Bayesian network meta-analysis |
| 2020 | confirms it is open | Patient characteristics as effect modifiers for psoriasis biologic treatment response: an assessment using network meta-analysis subgroups |
| 2021 | partly addresses it | Individual Participant Data Meta-Analysis: A Handbook for Healthcare Research (Riley, Tierney, Stewart, eds.; Wiley Series in Statistics in Practice, 2021) |
| 2021 | partly addresses it | A frequentist one-step model for a simple network meta-analysis of time-to-event data in presence of an effect modifier |
| 2021 | confirms it is open | Cognitive effects and acceptability of non-invasive brain stimulation on Alzheimer’s disease and mild cognitive impairment: a component network meta-analysis (online supplementary data and main article) |
| 2021 | confirms it is open | Behavioural interventions for smoking cessation: an overview and network meta-analysis |
| 2021 | confirms it is open | Comparing the use of aggregate data and various methods of integrating individual patient data to network meta-analysis and its application to first-line ART |
| 2021 | confirms it is open | When does the use of individual patient data in network meta-analysis make a difference? A simulation study |
| 2022 | partly addresses it | Meta-Research: Methods and Protocols (Evangelou E, Veroniki AA, eds), Methods in Molecular Biology vol. 2345 |
| 2022 | confirms it is open | Impact of program characteristics on weight loss in adult behavioral weight management interventions: systematic review and component network meta-analysis |
| 2022 | confirms it is open | Identifying relative efficacy of components of prehabilitation in adult surgical patients: protocol for a systematic review and component network meta-analysis |
| 2022 | confirms it is open | Preventing persistent postsurgical pain: A systematic review and component network meta-analysis |
| 2022 | confirms it is open | Validating the Assumptions of Population Adjustment: Application of Multilevel Network Meta-regression to a Network of Treatments for Plaque Psoriasis |
| 2022 | confirms it is open | Bayesian network meta-analysis methods for combining individual participant data and aggregate data from single arm trials and randomised controlled trials |
| 2023 | partly addresses it | Network meta-interpolation: Effect modification adjustment in network meta-analysis using subgroup analyses (Harari, Soltanifar, Cappelleri, Verhoek, Ouwens, Daly, Heeg; Research Synthesis Methods 2023;14(2):211-233) |
| 2023 | confirms it is open | Dismantling cognitive-behavioural therapy components for attention-deficit hyperactivity disorder in adolescents and adults: protocol for a network meta-analysis |
| 2023 | confirms it is open | Comparison of indirect treatment methods in migraine prevention to address differences in mode of administration |
| 2023 | confirms it is open | A Bayesian (meta-)regression model for treatment effects on the risk difference scale |
| 2023 | confirms it is open | Individual participant data from digital sources informed and improved precision in the evaluation of predictive biomarkers in Bayesian network meta-analysis |
| 2023 | confirms it is open | Synthesizing cross-design evidence and cross-format data using network meta-regression |
| 2024 | confirms it is open | Assessing the relative efficacy of components of opioid-free anaesthesia in adult surgical patients: protocol for a systematic review and component network meta-analysis |
| 2024 | confirms it is open | A Complex Meta-Regression Model to Identify Effective Features of Interventions From Multi-Arm, Multi-Follow-Up Trials |
| 2024 | confirms it is open | A bayesian network meta-analysis to explore modifying factors in randomized controlled trials: what works for whom to reduce depression in nursing home residents? |
| 2024 | confirms it is open | Combining randomized and non-randomized data to predict heterogeneous effects of competing treatments |
| 2025 | confirms it is open | Impact of pulmonary rehabilitation programme design on effectiveness in COPD: a systematic review and component network meta-analysis |
| 2025 | confirms it is open | Combining treatment effects from mixed populations in meta-analysis: a review of methods |
| 2026 | confirms it is open | Effectiveness of interventions to increase vaccine uptake: component network meta-analysis |
| 2026 | confirms it is open | Between Rigor and Relevance: Why the EU HTA Guidelines on Indirect Comparisons Miss the Mark |
| 2026 | confirms it is open | Personalized treatment hierarchies in Bayesian network meta-analysis |
Source
Derived from C:L214-215, C:L232 of the reviewed corpus.