Inconsistency conclusions depend on an uncalibrated heterogeneity error specification
Statement
The standard checks for network inconsistency, the design-by-treatment interaction test, generalized Cochran’s Q and node splitting, are conditional on how heterogeneity is specified, because the trial-by-treatment interaction is nested within the design-by-treatment interaction and therefore supplies the error term against which inconsistency is judged. On a single ten-treatment diabetes network of fifteen designs the two defensible specifications give opposite answers: with the trial-by-treatment interaction taken as fixed the inconsistency statistic is chi-squared 22.53 on 7 degrees of freedom, p = 0.0021, and with the same interaction taken as random and Kenward-Roger denominator degrees of freedom the test is F = 0.32 on 7 and 11 degrees of freedom, p = 0.9268. The residual diagnostics reverse with the test: two designs carry conspicuous studentized and PRESS residuals under the fixed specification and unremarkable ones under the random specification. A third specification in common use, dropping the heterogeneity term altogether, changes the effective weighting again, since the fixed specification weights all trials equally while dropping the term weights each trial by the variances of its means. No published criterion, calibration study or simulation says which specification a decision-grade analysis should use. Because consistency status decides whether a pooled network estimate is reported at all, and feeds credibility instruments in which incoherence is a scored domain, the specification is an analyst degree of freedom sitting at the gate and it is not reported as a choice.
Why it is open
The specifications encode genuinely different positions about what heterogeneity is, a nuisance to be conditioned on or a random error term against which design differences should be judged, and both are defensible in the abstract, so the question cannot be settled by argument and has to be settled by operating characteristics that nobody has computed. Calibrating them means simulating networks with known inconsistency and known heterogeneity and reporting type I error and power for each specification, which is straightforward and unglamorous; it also requires deciding what error rate is acceptable for a test whose consequence is refusing to pool, and the field has never stated one. The random specification carries an estimated variance component into the reference distribution, so its small-sample behavior depends on the denominator degrees-of-freedom approximation as well as on the specification, and those two sources have not been separated. Heterogeneity and inconsistency are also not cleanly separable in a sparse network, so at realistic sizes the specifications compete for the same variation and any calibration will be topology-dependent rather than universal.
What has been tried
| Work | What it contributes |
|---|---|
| Piepho 2014, BMC Medical Research Methodology | Formulates network meta-analysis as a factorial analysis of variance and demonstrates the reversal on the Senn diabetes network: inconsistency chi-squared 22.53 on 7 degrees of freedom with heterogeneity fixed against F = 0.32 on 7 and 11 degrees of freedom with the same effects random, with the studentized and PRESS residuals for two designs conspicuous under the first and unremarkable under the second, and states a preference rather than a calibration. |
| Piepho, Williams and Madden 2012, Biometrics | The two-way linear mixed model formulation of multitreatment meta-analysis in which the nesting of the heterogeneity interaction inside the inconsistency interaction is explicit, and in which fixed or random is an ordinary model option. |
| Higgins, Jackson, Barrett, Lu, Ades and White 2012, Research Synthesis Methods | Defines the design-by-treatment interaction model, the global inconsistency framework whose test statistic the heterogeneity specification determines. |
| White, Barrett, Jackson and Higgins 2012, Research Synthesis Methods | Estimates the design-by-treatment model by multivariate meta-regression, where the same fixed-versus-random heterogeneity choice appears as a fitting option with no guidance on which to select. |
| Jackson, Barrett, Rice, White and Higgins 2014, Statistics in Medicine | Design-by-treatment interaction model with random inconsistency effects, adding a second variance component and therefore a second specification choice on top of the one this entry concerns. |
| Dias, Welton, Caldwell and Ades 2010, Statistics in Medicine | Node splitting, the local consistency check that inherits the same dependence, since the direct and indirect estimates it compares are both conditioned on the assumed heterogeneity structure. |
| Krahn, Binder and Konig 2013, BMC Medical Research Methodology | Net-heat plot for locating inconsistency in a network under a baseline-contrast parameterization; the analysis of the same diabetes network whose fixed-specification statistic the reversal reproduces, which is what establishes that the disagreement is about specification rather than about parameterization. |
| Veroniki, Vasiliadis, Higgins and Salanti 2013, International Journal of Epidemiology | Applies competing inconsistency evaluation methods across many published networks and reports how often each flags inconsistency; the closest thing to an empirical calibration, and it varies the method rather than the heterogeneity specification. |
| Kenward and Roger 1997, Biometrics | Small-sample approximation to the denominator degrees of freedom for Wald-type F statistics from restricted maximum likelihood fits; the correction the random specification depends on, whose contribution to the reversal has not been separated from the specification’s own. |
| Nikolakopoulou et al. 2020, PLOS Medicine (CINeMA) | Confidence framework in which incoherence is a scored domain taking the inconsistency assessment as an input; establishes that the undisclosed specification choice propagates into a published confidence rating. |
| Dias, Welton, Sutton, Caldwell, Lu and Ades 2013, Medical Decision Making | Decision-making guidance on inconsistency in networks of randomized trials; sets out the checks a submission is expected to run without specifying the heterogeneity structure under which they are to be run. |
Probable solution or research direction
Run the calibration: simulate networks over a grid of topology, studies per design, true heterogeneity and true inconsistency, and report type I error and power for the design-by-treatment test with the heterogeneity term taken as fixed, as random, and omitted, with and without the Kenward-Roger correction, so the choice can rest on measured error rates. Report the same curves for node splitting, which inherits the specification through the same error term. Convert the result into a default with a stated condition, for instance the random specification whenever the heterogeneity variance is estimable and the fixed specification only below a stated number of studies per design, and require the analysis plan to name the specification before the data are seen. Where specifications disagree on real data the honest output is both results together with the statement that consistency status is not determined by the evidence, rather than the one the analyst prefers; a sensitivity display of the inconsistency p-value across specifications is a small addition to existing software and would make the degree of freedom visible immediately.
Verification
Verdict. Confirmed open. One reader proposed this and the reviewer accepted it as a well-posed operating-characteristics problem, since the heterogeneity specification changes the reference distribution and the practical conclusion of a global inconsistency assessment, which is distinct from the untestability of consistency itself. The reversal is demonstrated on one network in one paper, a narrow empirical base for how often the specifications disagree, though sufficient for the claim that no calibration of their error rates exists.
Source
Derived from reading:nma-42 of the reviewed corpus.