No corpus-derived prior distributions exist for the NMA inconsistency variance
Statement
Models for network meta-analysis with random inconsistency effects, and the design-by-treatment interaction model behind them, estimate an inconsistency variance that the likelihood barely identifies in typical networks, because it is informed only by the small number of independent loops. Empirically derived predictive distributions exist for the between-study heterogeneity variance, built from 14,886 Cochrane meta-analyses and published for 80 settings indexed by outcome and comparison type, but no analogous corpus-derived distribution exists for the inconsistency variance. Analysts therefore use either vague priors that leave the parameter unidentified or a heterogeneity prior rescaled by a factor chosen by hand, a derivation its own authors call somewhat speculative because it assumes the amount of prior information about heterogeneity matches that about inconsistency. The sensitivity is not hypothetical: on a topical antibiotics network estimating six parameters from sixteen comparisons, setting the inconsistency prior mean equal to the heterogeneity prior mean gives posterior means of 0.22 for heterogeneity and 0.35 for inconsistency, while setting it to one tenth gives 0.37 and 0.06, and the prior’s 2.5th and 97.5th percentiles of 0.03 and 1.20 essentially reproduce the posterior interval. Because the inconsistency variance is what a Bayesian analysis uses to judge whether a network is coherent enough to pool, a prior-driven estimate makes the coherence judgment prior-driven. The missing object is concrete: a database-derived predictive distribution for the inconsistency variance, indexed the same way as the heterogeneity distributions, together with an empirical estimate of the ratio of the two variances.
Why it is open
The heterogeneity corpus could be assembled from pairwise meta-analyses, of which the Cochrane database holds tens of thousands; an inconsistency variance exists only in networks containing loops, and the standard sampling frames of published networks are two orders of magnitude smaller, with the reference database describing 186 networks. Within each network the parameter is informed by a handful of independent loops, so every corpus entry is a noisy draw and a predictive distribution built by pooling point estimates would inherit that noise as if it were signal; a hierarchical extraction model is required and none has been specified. No indexing scheme has been agreed either, since the analogues of outcome type and comparison type for inconsistency would have to involve loop count and design structure, which vary far more across networks than outcome type does. Beyond the technical obstacles, nobody has attempted it: the proposal was left as future work by the same group that identified the need, and the random-inconsistency models it would serve remain rare in applied practice, so no user community is pressing for the prior.
What has been tried
| Work | What it contributes |
|---|---|
| Law, Jackson, Turner, Rhodes and Viechtbauer 2016, BMC Medical Research Methodology | Fits the random-inconsistency-effects model with empirically based lognormal priors on both variances, derives the inconsistency prior by rescaling the heterogeneity prior mean, calls that derivation speculative, demonstrates the prior sensitivity across scaling factors of one, one half and one tenth, and names corpus-derived predictive distributions for the inconsistency variance as possible future work. |
| Jackson, Barrett, Rice, White and Higgins 2014, Statistics in Medicine | Defines the design-by-treatment interaction model with random inconsistency effects, which introduces the inconsistency variance as a single parameter describing the extent of inconsistency across the network. |
| Higgins, Jackson, Barrett, Lu, Ades and White 2012, Research Synthesis Methods | Develops the design-by-treatment interaction framework for multi-arm studies, the modeling setting in which a global inconsistency parameter is defined and needs a prior. |
| Turner, Davey, Clarke, Thompson and Higgins 2012, International Journal of Epidemiology | The corpus method that would be imitated: predictive distributions for between-study heterogeneity derived from 14,886 Cochrane meta-analyses and indexed by outcome and comparison type. |
| Turner, Jackson, Wei, Thompson and Higgins 2015, Statistics in Medicine | Supplies the 80 published heterogeneity predictive distributions that current inconsistency priors are rescaled from, and sets the format any inconsistency-variance corpus would follow. |
| Nikolakopoulou, Chaimani, Veroniki, Vasiliadis, Schmid and Salanti 2014, PLoS ONE | Describes a database of 186 published networks with their size and connectivity characteristics; the kind of sampling frame a corpus-building exercise would use, and evidence of how small that frame is compared with the pairwise corpus. |
| Veroniki, Vasiliadis, Higgins and Salanti 2013, International Journal of Epidemiology | Evaluates inconsistency empirically across published networks using several methods and shows that detection depends on the method used; the closest existing empirical anchor for how much inconsistency is present in practice. |
| Song, Xiong, Parekh-Bhurke et al. 2011, BMJ | Meta-epidemiological study of disagreement between direct and indirect comparisons across many independent triangles, giving a second empirical source on the typical magnitude of inconsistency that a predictive distribution would have to reproduce. |
| Veroniki, Mavridis, Higgins and Salanti 2014, BMC Medical Research Methodology | Simulation study of which loop characteristics affect detection and estimation of inconsistency, which is what any corpus-building design would have to condition on when deciding how to index the distributions. |
| Jackson, Veroniki, Law, Tricco and Baker 2017, Research Synthesis Methods | Supplies classical Paule-Mandel, DerSimonian-Laird and REML estimators for the same two variances, which is the fast frequentist route a large-scale corpus refit would use instead of MCMC. |
Probable solution or research direction
Refit an existing database of published networks with a random-inconsistency model under matched weakly informative priors on both variances, using the classical estimators where MCMC is too slow, and publish the resulting predictive distribution for the inconsistency variance together with the distribution of its ratio to the heterogeneity variance, indexed in the same outcome-type and comparison-type scheme as the published heterogeneity distributions. Make the extraction hierarchical so that each network’s estimation uncertainty enters the predictive distribution rather than being discarded, since a naive histogram of point estimates from loops-poor networks would overstate the spread. Validate by leave-one-network-out prediction: does a prior derived this way beat both a vague prior and the current hand-scaled prior at predicting the held-out network’s design-level effects. Report explicitly whether the empirical ratio supports the one-half default that current practice assumes, since that single number is doing most of the work in every published application of these models.
Verification
Verdict. Confirmed open. One reader proposed this and the reviewer accepted it because the corpus-building task is concrete and distinct from generic prior-sensitivity concerns. The direct evidence is a single paper, but that paper both demonstrates the prior dominance numerically and names the missing corpus as future work, and no such corpus has appeared since.
Source
Derived from reading:nma-38 of the reviewed corpus.