No error control for repeated looks at a living network
Statement
A living or cumulative network meta-analysis re-fits an accumulating network on a fixed cadence, and every release is read as the current answer, yet no convention states what error rate is being controlled across the series of looks. Sequential monitoring machinery exists for the single-contrast case: Nikolakopoulou et al. 2018 (doi:10.1177/0962280216659896) build efficacy and futility boundaries from each network estimate’s z-score and its accrued information fraction, and applied sequential network meta-analyses use them. Four things that machinery does not supply are missing at once: a required information size for a contrast when treatments and loops keep entering, an error-rate target for the K(K-1)/2 comparisons that are re-tested at every update and are correlated through the shared basic parameters and the consistency equations, a rule stating which assumptions must be re-checked at each update and which may be inherited, and a rule for retiring a settled comparison from the update cycle. The network case is not the pairwise case with more contrasts, because a comparison can cross its boundary purely on indirect evidence contributed by trials of other comparisons; the same methods paper shows a stent-versus-surgery contrast becoming significant when only stent-versus-stent studies are published. Living protocols decline adjustment explicitly and record it as a design choice: one cumulative antidepressant program states it will not adjust for multiple comparisons in successive network meta-analyses, and a live cumulative lung-cancer network planning updates every four months with about two new treatments per cycle declines adjustment on the stated grounds that Bayesian inference is not affected by repeated updates. Guideline panels consume these releases between updates, so a reader of update k has no stated basis for interpreting the reported interval, and no report says how many looks a conclusion has survived.
Why it is open
Alpha spending requires a fixed information target and a fixed hypothesis set, and a living network supplies neither: adding a treatment changes the required information size for contrasts nobody looked at, and adding an indirect path changes an estimate with no new direct trial, so the information fraction is not well defined per contrast. The monitored quantity also moves, because the best of k treatments and any rank summary are defined relative to the treatment set, so a node added, merged or split between looks redefines the estimand rather than adding information about it. The Bayesian defense is correct as far as it goes and does not reach the problem: a posterior for a fixed parameter is unaffected by optional stopping, but selecting a maximum over a growing set, and the decision rules a guideline consumer applies at each release, are not fixed-parameter inference. The dependence among comparisons is analytically available from the network covariance matrix, so an efficient simultaneous procedure is derivable in principle; what is absent is a decision about which family is being controlled, and that choice sits outside any model. Living-review methodology has developed the production side in detail while the statistical updating series stops at pairwise synthesis, so the network case has no owner.
What has been tried
| Work | What it contributes |
|---|---|
| Nikolakopoulou et al. 2018, Statistical Methods in Medical Research | Extends sequential monitoring to network meta-analysis: efficacy and futility boundaries per network estimate from the z-score and the information accrued after each study enters, with the demonstration that a contrast can become conclusive through indirect evidence alone. Adjusts for repeated looks at one comparison, defines no family-level error target, and declines multiplicity adjustment for the accompanying inconsistency tests. |
| Wetterslev et al. 2008, Journal of Clinical Epidemiology | Trial sequential analysis for cumulative pairwise meta-analysis: heterogeneity-adjusted required information size plus monitoring boundaries, finding early potentially spurious P below 0.05 in five of six neonatal meta-analyses. Assumes one fixed comparison and a monotone information target, which is what a living network breaks. |
| Simmonds et al. 2017, Journal of Clinical Epidemiology | Living systematic reviews part 3: compares the law of the iterated logarithm, the Shuster method, trial sequential analysis and sequential meta-analysis as ways of controlling type I and type II error under repeated updating, and states that the type I error rate rises rapidly as more updates are performed. Pairwise only, with a fixed comparison set. |
| Elliott et al. 2017, Journal of Clinical Epidemiology | Defines the living systematic review, its update cadence and the conditions under which continuous updating is worth doing, setting the production framework within which the inferential question arises. |
| Tsai et al. 2020, Journal of Clinical Periodontology | Applies sequential network meta-analysis to an updated 60-trial periodontal network specifically to control the type I error rate from multiple testing, and reports that the stricter sequential criteria give different conclusions from the ordinary analysis. A single application rather than a framework, with a fixed treatment set. |
| Vandvik, Brignardello-Petersen and Guyatt 2016, BMC Medicine | Sets out living cumulative network meta-analysis as a program and enumerates the challenges that must be solved before it can become routine. Error control across repeated releases is not among them, which is itself evidence that the question is unformulated. |
| Xin et al. 2022, BMC Medical Research Methodology | Interactive re-analysis and sensitivity analysis of a living COVID-19 network through the MetaInsight app, and surveys what exists for the repeated-looks problem: trial sequential analysis has been proposed, its network extension and R packages have started to be developed, and issues such as futility boundaries still need further work. |
| Crequit, Trinquart and Ravaud 2016, BMJ Open | Protocol for a live cumulative network meta-analysis in second-line non-small-cell lung cancer with updates every four months, which declines any adjustment for multiple testing on the stated grounds that Bayesian inference is not affected by repeated updates. |
| Luo et al. 2018, BMJ Open | Protocol for cumulative network meta-analyses of new-generation antidepressants feeding practice guidelines, which states that no adjustment for multiple comparisons will be made across successive network meta-analyses because particular hypotheses for specific comparisons are not being tested. |
Probable solution or research direction
Fix the monitored family before deriving anything: state whether the target is a per-contrast rate, a family-wise rate over the decision-relevant contrasts at one release, or a false-discovery rate across the whole update series, and take the correlation among contrasts from the network covariance matrix, which already supplies it analytically. Redefine the information fraction so it survives node entry and exit, for example by anchoring it to a declared decision contrast and treating a newly opened indirect path as information accrual rather than as a change of target. Write an assumption-inheritance rule saying which transitivity, heterogeneity and node-partition checks must be re-run when new studies or new nodes arrive and which results may carry over. Add a retirement rule that drops a settled comparison from further monitoring, plus a reporting convention stating how many looks a conclusion has survived and what fraction of the required information has accrued. Evaluate the result by simulating an accruing network and reporting the realized probability that at least one comparison crosses an efficacy boundary spuriously, as a function of the number of treatments, the update cadence and the rate at which new nodes enter.
Verification
Verdict. Confirmed open. Four independent readers proposed this from four different papers and the reviewer accepted each, holding that repeated publication and decision use of an evolving network raise questions that neither pairwise trial sequential analysis nor the fixed-parameter Bayesian argument answers. The basis is not only applied protocols: the methods paper that defines sequential boundaries for networks adjusts solely for repeated looks at one contrast, and a living-review feasibility study records that the network extensions of trial sequential analysis still leave issues such as futility boundaries unresolved.
Source
Derived from reading:nma-17, reading:nma-31, reading:nma-36, reading:nma-37 of the reviewed corpus.