Single shared heterogeneity variance across classes and populations
No. Even with a fivefold difference in between-study variance, AIC chose class-specific variances in only 18% of networks with five studies per class, 40% with twelve and 79% with forty. Balanced selection accuracy reached 0.8 only at 40 studies per class and only for AIC; BIC and a likelihood-ratio test did not within 40. At the network sizes population-adjusted analyses use, the heterogeneity structure cannot be chosen from the data, so the shared default is untested rather than supported. Read the study below.
Statement
Network models in current software default to a single between-study heterogeneity variance shared across the whole network, adopted for estimability rather than for plausibility: multinma’s tau is one network-wide scalar, as is the common tau recorded by cMLNMR in cpaic, an unreleased development package on GitHub that is not on CRAN. The structured alternatives are not open research. Thorlund et al. 2013 (pmid:23311298) state the problem and build and compare four replacements for the homogeneous-variance assumption, empirically derived heterogeneity priors exist for 80 settings (Turner et al. 2015, doi:10.1002/sim.6381) and weakly informative ones for the very-few-studies case (Rover et al. 2021, doi:10.1002/jrsm.1475), and class-specific variance components with sharing across sparse classes ship in multinma via class_sd since 0.8.0 (Perren et al. 2025, doi:10.1177/0272989X251389887), although that is a within-class treatment standard deviation rather than tau. What has not been done is establishing how large a network must be before the competing variance structures are distinguishable, and no structure indexes heterogeneity by target population; nor is the direction of the error fixed, since a shared variance can overstate uncertainty in a sparse stratum as readily as understate it, depending on which strata are more heterogeneous.
Why it is open
Networks rarely contain enough studies per class or component to estimate separate variances, so a separate-variance model is often only nominally identified and its posterior is driven by the prior rather than by the data. Model comparison statistics have little power to distinguish competing variance structures at realistic network sizes, and no one has established the size at which they become distinguishable, so the default is neither tested nor visibly violated. Component-specific variances need further assumptions about the covariance among component deviations, and any structured alternative must keep the random-effects covariance positive semidefinite, which is why the existing proposals have not displaced the default in software. Population-indexed heterogeneity has no implementation at all: the software parameter is a single network-wide scalar, so the question of whether tau should differ by target population cannot currently be asked.
What has been tried
| Work | What it contributes |
|---|---|
| Thorlund, Thabane and Mills, 2013, Modelling heterogeneity variances in multiple treatment comparison meta-analysis, BMC Medical Research Methodology 13:2 | States the entry’s problem (the homogeneous variance assumption is unrealistic and can severely bias variance estimates, so credible intervals may lose nominal coverage and rank probabilities may be distorted) and builds and compares four alternatives: exchangeable variances, consistency-constrained variances, and two informative variance-prior schemes. |
| Turner et al., 2015, Predictive distributions for between-study heterogeneity and simple methods for their application in Bayesian meta-analysis | Derives empirically based predictive distributions for heterogeneity in 80 settings defined by outcome and comparison type, usable directly as heterogeneity priors; this is the robust-prior component the source lists as needed. |
| Rover et al., 2021, On weakly informative prior distributions for the heterogeneity parameter in Bayesian random-effects meta-analysis | Guidance on heterogeneity priors when very few studies contribute, which is the sparse-stratum case the entry names. |
| Perren, Pedder, Welton and Phillippo, 2026, Network Meta-Analysis with Class Effects: A Practical Guide and Model Selection Algorithm | Class-specific variance components with sharing of the standard deviation across groups of classes when a class has too few treatments to estimate its own, implemented in multinma as class_sd since 0.8.0, plus a model selection algorithm; note this is a within-class treatment standard deviation, not the between-study heterogeneity variance. |
| cpaic, unreleased development package on GitHub (not on CRAN), commit 9d150e9 | cMLNMR records a single common tau and offers half-normal and half-Student-t priors for it, with no class-specific or component-specific heterogeneity structure; the source-code auditor used this to confirm that the software default persists. |
Probable solution or research direction
The modeling work is largely done, so the useful contributions are evaluation and porting rather than invention. Run the distinguishability simulation the field lacks: how many studies per class or component are needed before exchangeable, consistency-constrained and separate-variance structures can be told apart, so a default can be defended or abandoned on the basis of network size rather than convention. Carry the existing structures and the empirically derived heterogeneity priors into population-adjusted network models, where the heterogeneity parameter is still a single scalar, and report prior sensitivity rather than claiming any prior is universally robust. The genuinely new case is heterogeneity indexed by target population, which none of the existing work addresses.
Our study
Shared and class-specific heterogeneity cannot be told apart below about 40 studies per class
The question put to it. How many studies per class are needed before model comparison reliably distinguishes a shared between-study variance from class-specific ones?
What it found. No. Even with a fivefold difference in between-study variance, AIC chose class-specific variances in only 18% of networks with five studies per class, 40% with twelve and 79% with forty. Balanced selection accuracy reached 0.8 only at 40 studies per class and only for AIC; BIC and a likelihood-ratio test did not within 40. At the network sizes population-adjusted analyses use, the heterogeneity structure cannot be chosen from the data, so the shared default is untested rather than supported.
Results.
- Twofold variance ratio: separate structure chosen in at most 27% of networks by any criterion up to 40 studies per class.
- AIC false-positive rate rose from 0.6% (2 studies per class) to 15% (40).
- BIC with two studies per class chose separate variances under the null in 12% because its penalty log(2m-2) is below 2.
Design.
- Estimand
- Probability of selecting the true heterogeneity structure
- Methods compared
- AIC, BIC, likelihood-ratio test
- Data-generating mechanism
- Two classes of 2 to 40 studies each with between-study variance ratio 1, 2 or 5; shared and separate REML fits compared by AIC, BIC and a likelihood-ratio test. 21 cells, 2000 replicates each.
- Replicates
- 2000 per cell, 21 cells
What this does not answer. Two classes, contrast-level data, frequentist criteria; no Bayesian model comparison, informative priors, ML-NMR or class-specific treatment SD. Peer review has not been done.
Peer review. Not yet done. The result stands on the study alone until it is.
Read it: PDF · Markdown · OpenDocument · Code and data: studies/HET-03-how-many-studies-before-the-structures-differ · Protocol: registered before the run
Verification
Verdict. Partially addressed. Part of this problem is addressed by existing work; the entry names which part.
The source said. “Network models default to one common between-study heterogeneity variance even when the network spans treatment classes, components, and populations for which a shared value is implausible. The default is adopted for estimability rather than for plausibility.”
What was wrong with it. The source presented structured heterogeneity models and robust heterogeneity priors as work still to be developed; the literature auditor found them published and evaluated years ago (Thorlund et al. 2013, pmid:23311298; Turner et al. 2015, doi:10.1002/sim.6381; Rover et al. 2021, doi:10.1002/jrsm.1475; class-specific variances in multinma since 0.8.0, doi:10.1177/0272989X251389887), so the residual gap is adoption plus the population-indexed case. The two auditors split on whether the entry is open or partly solved, and the code auditor added that ‘often unrealistic’ is an unsupported prevalence claim and that a shared variance does not automatically understate uncertainty where a new treatment sits; the entry is published at the weaker claim.
Dissent. literature dissented (overstated): Retrieved Thorlund, Thabane & Mills (2013), BMC Med Res Methodol 13:2, which is the entry’s problem and much of its proposed direction, already executed. Verbatim: ‘Random-effects MTCs have commonly modeled heterogeneity under the assumption that the between-trial variance for all involved treatment comparisons are equal… The homogeneous variance assumption, however, is unrealistic and can severely bias variance estimates. Consequently 95% credible intervals may not retain nominal coverage, and treatment rank probabilities may become distorted.’ They build and compare four alternatives (exchangeable variances, consistency-constrained variances, two informative variance-prior schemes) and conclude ‘All four approaches should be viable candidates for replacing or supplementing the conventional homogeneous variance MTC model.’ The ‘robust priors for sparse strata’ claim is likewise delivered rather than open: Turner et al. (2015) empirical predictive distributions for tau, Rover et al. (2021) weakly informative heterogeneity priors, Lilienthal et al. (2024) empirical HTA priors. Class-specific variance components with sharing across sparse classes are implemented in multinma 0.8.0+ via class_sd, though that is the within-class treatment SD, not between-study tau; multinma’s tau remains a single network-wide scalar, so the software-default half stands. Genuinely open: population-indexed heterogeneity, and the distinguishability simulation (Thorlund used two illustrative datasets).
Revised by the full-text reading. Applied 2026-07-26 full-text reading of the consolidated library (687 papers).
- Citation corrected
- normalized a citation year: ‘Perren, Pedder, Welton and Phillippo, 2025, Network Meta-Analysis with Class Effects: A Practical Guide and Model Selection Algorithm’ -> ‘Perren, Pedder, Welton and Phillippo, 2026, Network Meta-Analysis with Class Effects: A Practical Guide and Model Selection Algorithm’ 10.1177/0272989X251389887 is cited with two different years across the registry. CrossRef gives 2025 online and 2026 in print; the issue year is used throughout.
- Literature and prior-art check
- partially-solved / overstated. Retrieved Thorlund, Thabane & Mills (2013), BMC Med Res Methodol 13:2, which is the entry’s problem and much of its proposed direction, already executed. Verbatim: ‘Random-effects MTCs have commonly modeled heterogeneity under the assumption that the between-trial variance for all involved treatment comparisons are equal… The homogeneous variance assumption, however, is unrealistic and can severely bias variance estimates. Consequently 95% credible intervals may not retain nominal coverage, and treatment rank probabilities may become distorted.’ They build and compare four alternatives (exchangeable variances, consistency-constrained variances, two informative variance-prior schemes) and conclude ‘All four approaches should be viable candidates for replacing or supplementing the conventional homogeneous variance MTC model.’ The ‘robust priors for sparse strata’ claim is likewise delivered rather than open: Turner et al. (2015) empirical predictive distributions for tau, Rover et al. (2021) weakly informative heterogeneity priors, Lilienthal et al. (2024) empirical HTA priors. Class-specific variance components with sharing across sparse classes are implemented in multinma 0.8.0+ via class_sd, though that is the within-class treatment SD, not between-study tau; multinma’s tau remains a single network-wide scalar, so the software-default half stands. Genuinely open: population-indexed heterogeneity, and the distinguishability simulation (Thorlund used two illustrative datasets). Cites pmid:23311298: Already states the problem and supplies four evaluated alternatives to a single shared heterogeneity variance.. Cites doi:10.1002/sim.6381: Supplies empirically derived heterogeneity priors for 80 settings, the ‘robust priors for sparse strata’ the entry lists as needed.. Cites doi:10.1002/jrsm.1475: Guidance on robust heterogeneity priors when very few studies contribute, the sparse-stratum case the entry names.. Cites doi:10.1177/0272989X251389887: Class-specific variance components with sharing across sparse classes, implemented in multinma (class_sd argument, since 0.8.0), plus a model-selection algorithm..
- GPT-5.6 Sol, technical and source-code lens
- open / supported-with-caveat. C1 is technically plausible but context-dependent; the word often is an unsupported prevalence claim. A common heterogeneity variance can be implausible across distinct classes or populations, but it can either understate or overstate uncertainty depending on which strata are more heterogeneous. Understatement near a new treatment is not automatic. C2 is a valid modeling direction only if the resulting random-effects covariance is positive semidefinite and the network supplies enough replication. Component-specific variances also require assumptions about covariance among component deviations. In sparse strata, separate variances may be practically unidentified and dominated by priors. C3 is too categorical: proper regularization, hierarchical pooling, and prior-sensitivity analysis are advisable, but there is no universally robust prior that is always required. At cpaic 9d150e9, cMLNMR explicitly records a single common tau. It offers half-normal and half-Student-t priors but no class-specific or component-specific heterogeneity structure, so the core package limitation remains open. This is a potentially material sensitivity issue, not automatically a high-severity defect in every network.
- Decision path
-
R3:partially-solved(n=1) → R9:substantiated-single-overstatement:literature
How the evidence falls in time
91 papers in the reviewed corpus bear on this problem, published 2010 to 2026. Partial progress is on the record, and later work still calls this open. Both belong in view.
This problem is recurrent: three or more papers spanning at least eight years assert it independently. Sustained restatement by authors who mostly do not cite each other is the strongest evidence this reading can offer that a gap is real rather than one group’s framing.
It is also being closed in pieces: partial results come from more than one paper in more than one year, each covering a different part.
| Year | Says | Paper |
|---|---|---|
| 2010 | confirms it is open | Network meta-analysis on the log-hazard scale, combining count and hazard ratio statistics accounting for multi-arm trials: a tutorial |
| 2011 | confirms it is open | Conducting indirect-treatment-comparison and network-meta-analysis studies: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 2 |
| 2011 | confirms it is open | Interpreting indirect treatment comparisons and network meta-analysis for health-care decision making: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 1 |
| 2011 | confirms it is open | Network meta-analysis of survival data with fractional polynomials |
| 2012 | confirms it is open | Consistency and inconsistency in network meta-analysis: concepts and models for multi-arm studies |
| 2012 | confirms it is open | Consistency and inconsistency in network meta-analysis: model estimation using multivariate meta-regression |
| 2012 | confirms it is open | Evidence Synthesis for Decision Making 2: A Generalized Linear Modeling Framework for Pairwise and Network Meta-analysis of Randomized Controlled Trials |
| 2012 | confirms it is open | Sample size and power considerations in network meta-analysis |
| 2012 | confirms it is open | Simulation evaluation of statistical properties of methods for indirect and mixed treatment comparisons |
| 2013 | partly addresses it | Evaluation of inconsistency in networks of interventions |
| 2013 | partly addresses it | Graphical tools for network meta-analysis in STATA |
| 2013 | confirms it is open | Evidence synthesis for decision making 3: heterogeneity - subgroups, meta-regression, bias, and bias-adjustment |
| 2014 | confirms it is open | A design-by-treatment interaction model for network meta-analysis with random inconsistency effects |
| 2014 | confirms it is open | Characteristics of a loop of evidence that affect detection and estimation of inconsistency: a simulation study |
| 2014 | confirms it is open | Evaluating the quality of evidence from a network meta-analysis |
| 2014 | confirms it is open | Joint synthesis of multiple correlated outcomes in networks of interventions |
| 2014 | confirms it is open | Network meta-analysis of multiple outcome measures accounting for borrowing of information across outcomes |
| 2015 | partly addresses it | ‘Arm-based’ parameterization for network meta-analysis |
| 2015 | confirms it is open | Accounting for Heterogeneity in Relative Treatment Effects for Use in Cost-Effectiveness Models and Value-of-Information Analyses |
| 2015 | confirms it is open | Borrowing of strength and study weights in multivariate and network meta-analysis |
| 2015 | confirms it is open | Extending DerSimonian and Laird’s methodology to perform network meta-analyses with random inconsistency effects |
| 2015 | confirms it is open | Extending Treatment Networks in Health Technology Assessment: How Far Should We Go? |
| 2016 | partly addresses it | Statistical Models and Methods for Network Meta-Analysis |
| 2016 | confirms it is open | An overview of methods for network meta-analysis using individual participant data: when do benefits arise? |
| 2016 | confirms it is open | Continuously updated network meta-analysis and statistical monitoring for timely decision-making |
| 2016 | confirms it is open | GetReal in network meta-analysis: a review of the methodology |
| 2016 | confirms it is open | Implementing informative priors for heterogeneity in meta-analysis using meta-regression and pseudo data |
| 2016 | confirms it is open | Meta-Analysis of a Complex Network of Non-Pharmacological Interventions: The Example of Femoral Neck Fracture |
| 2016 | confirms it is open | Model-Based Network Meta-Analysis: A Framework for Evidence Synthesis of Clinical Trial Data |
| 2016 | confirms it is open | Two new methods to fit models for network meta-analysis with random inconsistency effects |
| 2017 | partly addresses it | Bayesian one-step IPD network meta-analysis of time-to-event data using Royston-Parmar models |
| 2017 | partly addresses it | Performing Arm-Based Network Meta-Analysis in R with the pcnetmeta Package |
| 2017 | partly addresses it | Using structural equation modeling for network meta-analysis |
| 2017 | confirms it is open | A matrix-based method of moments for fitting multivariate network meta-analysis models with multiple outcomes and random inconsistency effects |
| 2017 | confirms it is open | An investigation of the impact of using different methods for network meta-analysis: a protocol for an empirical evaluation |
| 2017 | confirms it is open | Characteristics and knowledge synthesis approach for 456 network meta-analyses: a scoping review |
| 2017 | confirms it is open | Paule-Mandel estimators for network meta-analysis with random inconsistency effects |
| 2017 | confirms it is open | Pharmacological interventions for acute hepatitis B infection: an attempted network meta-analysis |
| 2017 | confirms it is open | Pharmacological interventions for alcoholic liver disease (alcohol-related liver disease): an attempted network meta-analysis |
| 2018 | partly addresses it | A Bayesian hierarchical model for network meta-analysis of multiple diagnostic tests |
| 2018 | partly addresses it | Incorporating external evidence on between-trial heterogeneity in network meta-analysis |
| 2018 | confirms it is open | Borrowing of strength from indirect evidence in 40 network meta-analyses |
| 2018 | confirms it is open | An innovative and comprehensive technique to evaluate different measures of medication adherence: The network meta-analysis |
| 2018 | confirms it is open | Hypothesis testing in Bayesian network meta-analysis |
| 2018 | confirms it is open | Network meta-analysis explained |
| 2018 | confirms it is open | Planning a future randomized clinical trial based on a network of relevant past trials |
| 2018 | confirms it is open | The inclusion of real world evidence in clinical development planning |
| 2018 | confirms it is open | The number needed to treat and relevant between-trial comparisons of competing interventions |
| 2019 | confirms it is open | Dismantling, personalising and optimising internet cognitive-behavioural therapy for depression: a study protocol for individual participant data component network meta-analysis |
| 2019 | confirms it is open | Bivariate network meta-analysis for surrogate endpoint evaluation |
| 2019 | confirms it is open | Treatment sequence network meta-analysis in Crohn’s disease: a methodological case study |
| 2020 | confirms it is open | Supporting information (Bayesian analysis) to Ruecker, Petropoulou and Schwarzer, Network meta-analysis of multicomponent interventions |
| 2020 | confirms it is open | Supplementary online content to Miklowitz et al., Adjunctive psychotherapy for bipolar disorder: a systematic review and component network meta-analysis |
| 2020 | confirms it is open | Multilevel network meta-regression for population-adjusted treatment comparisons |
| 2020 | confirms it is open | A simulation study to compare different estimation approaches for network meta-analysis and corresponding methods to evaluate the consistency assumption |
| 2020 | confirms it is open | Bayesian splines versus fractional polynomials in network meta-analysis |
| 2020 | confirms it is open | CINeMA: Software for semiautomated assessment of the confidence in the results of network meta-analysis |
| 2021 | partly addresses it | A likelihood ratio test for the homogeneity of between-study variance in network meta-analysis |
| 2021 | confirms it is open | Behavioural interventions for smoking cessation: an overview and network meta-analysis |
| 2021 | confirms it is open | Non-pharmacological interventions for preventing delirium in hospitalised non-ICU patients |
| 2021 | confirms it is open | Component network meta-analysis compared to a matching method in a disconnected network: A case study |
| 2021 | confirms it is open | Supplementary appendix to Furukawa et al., Dismantling, optimising, and personalising internet cognitive behavioural therapy for depression: a systematic review and component network meta-analysis using individual participant data |
| 2021 | confirms it is open | Relative effectiveness of non-surgical interventions for pain management in knee osteoarthritis: a protocol for a component network meta-analysis of randomised controlled trials |
| 2021 | confirms it is open | Individual Participant Data Meta-Analysis: A Handbook for Healthcare Research (Riley, Tierney, Stewart, eds.; Wiley Series in Statistics in Practice, 2021) |
| 2021 | confirms it is open | Prevalence of evidence of inconsistency and its association with network structural characteristics in 201 published networks of interventions |
| 2021 | confirms it is open | When does the use of individual patient data in network meta-analysis make a difference? A simulation study |
| 2022 | partly addresses it | Network Meta-analysis on Disconnected Evidence Networks When Only Aggregate Data Are Available: Modified Methods to Include Disconnected Trials and Single-Arm Studies while Minimizing Bias |
| 2022 | partly addresses it | metapack: An R Package for Bayesian Meta-Analysis and Network Meta-Analysis with a Unified Formula Interface |
| 2022 | confirms it is open | Impact of program characteristics on weight loss in adult behavioral weight management interventions: systematic review and component network meta-analysis |
| 2022 | confirms it is open | Identifying relative efficacy of components of prehabilitation in adult surgical patients: protocol for a systematic review and component network meta-analysis |
| 2022 | confirms it is open | Bayesian models for aggregate and individual patient data component network meta-analysis |
| 2022 | confirms it is open | Bayesian unanchored additive models for component network meta-analysis |
| 2022 | confirms it is open | A dose-effect network meta-analysis model with application in antidepressants using restricted cubic splines |
| 2022 | confirms it is open | Bayesian network meta-analysis methods for combining individual participant data and aggregate data from single arm trials and randomised controlled trials |
| 2022 | confirms it is open | Incorporating dose effects in network meta-analysis |
| 2022 | confirms it is open | Network meta-analysis in psychology and educational sciences: A systematic review of their characteristics |
| 2022 | confirms it is open | Theory and practice of Bayesian and frequentist frameworks for network meta-analysis |
| 2023 | confirms it is open | Dismantling and personalising task-sharing psychosocial interventions for common mental disorders: a study protocol for an individual participant data component network meta-analysis |
| 2023 | confirms it is open | A new approach to evaluating loop inconsistency in network meta-analysis |
| 2024 | confirms it is open | Interventions to prevent the onset of frailty in adults aged 60 and older (PRAE-Frail): a systematic review and network meta-analysis |
| 2024 | confirms it is open | Implementing Multilevel Network Meta-Regression for Time-To-Event Outcomes: A Case Study in Relapsed Refractory Multiple Myeloma |
| 2024 | confirms it is open | A Complex Meta-Regression Model to Identify Effective Features of Interventions From Multi-Arm, Multi-Follow-Up Trials |
| 2024 | confirms it is open | Sharing information across patient subgroups to draw conclusions from sparse treatment networks |
| 2025 | confirms it is open | Community-based interventions for the prevention or management of dental caries in children (Protocol) |
| 2025 | confirms it is open | Understanding the effectiveness and underlying mechanisms of lifestyle modification interventions in adults with learning disabilities: a mixed-methods systematic review |
| 2025 | confirms it is open | Dismantling behavioural weight management interventions: component network meta-analysis of randomised controlled trials and real-world services |
| 2025 | confirms it is open | An investigation of the impact of using contrast- and arm-synthesis models for network meta-analysis |
| 2025 | confirms it is open | Network Meta-Analysis With Individual Participant-Level Data of Time-to-Event Outcomes Using Cox Regression |
| 2025 | confirms it is open | Network Meta-Analysis With Class Effects: A Practical Guide and Model Selection Algorithm |
| 2025 | confirms it is open | Risk of Bias in Network Meta-Analysis (RoB NMA) tool |
| 2026 | confirms it is open | Effectiveness of interventions to increase vaccine uptake: component network meta-analysis |
Source
Derived from C:L214-215, C:L234 of the reviewed corpus.