No joint error rate for the many claims a network asserts
Statement
Network meta-analysis reports per-treatment marginal summaries such as SUCRA, P-scores and rankograms, plus a league table of pairwise intervals each at nominal 95 percent, while a recommendation uses several of those statements together. Nothing reports the joint posterior or family-wise confidence of the conjunction that is acted on, and nothing defines a tie by reference to a minimal clinically important difference rather than to exact equality of parameters. Both halves are computable and the omission is material: in the 12-drug antidepressants network the full set of 66 pairwise ordering statements holds jointly with posterior probability below 0.01, and only a heavily trimmed set survives at 0.97 (doi:10.1093/jrsssc/qlae038). The GRADE minimally contextualised framework interrogates each contrast against a decision threshold at nominal 95 percent and then builds effectiveness categories by iterative promotion; its worked 27-treatment example produces 351 comparisons, and the guidance states that how seriously multiplicity might compromise inferences remains uncertain and that no satisfactory or widely accepted strategy exists for dealing with it. Part of the clinical half is built: Mavridis et al. 2020 (doi:10.1002/bimj.201900026) extend the P-score to measure the mean certainty that a treatment beats its competitors by at least a stated minimum important difference, across several outcomes at once. What remains absent is an error-rate target for the family of claims a network report makes, a procedure that exploits the correlation among contrasts that the network covariance matrix already supplies, and any convention fixing the joint-probability target or choosing among the several distinct trimmed sets that reach it. This concerns the many claims made from one network at one time; error control across the repeated looks of a living network is a separate gap.
Why it is open
The family is not defined, and until it is no procedure can be evaluated: a network report asserts pairwise superiorities, category memberships and an ordering at the same time, and which of these a committee acts on jointly varies by decision, so the correction depends on a choice that sits outside the model. Standard corrections are also a poor fit, because the contrasts are strongly dependent through the shared basic parameters and the consistency equations, so a Bonferroni-style adjustment over T(T-1)/2 comparisons is badly conservative while the efficient alternative requires carrying the network covariance into a simultaneous procedure nobody has written. Making equality expressible at all requires a prior with an atom at zero, so a joint-probability report is a change of model rather than post-processing of an existing analysis. Even once computed the report is not unique, since several distinct trimmed sets of ordering statements can reach the same joint probability with no rule for choosing among them and no default for the target. The clinically meaningful half is blocked upstream by elicitation: an equivalence region needs a minimum important difference per outcome, network meta-analyses rarely declare one, and without it a statistically detected difference far below any plausible margin still produces an arrow in a ranking.
What has been tried
| Work | What it contributes |
|---|---|
| Barrientos, Page and Lin 2024, Journal of the Royal Statistical Society Series C | Nonparametric Bayesian model whose prior has an atom, so exact equality of treatment effects is expressible, and which reports the joint posterior probability of a whole set of ordering statements. On the 12-drug antidepressants network the untrimmed set of relationships holds jointly with probability below 0.01. Leaves the trimming threshold undefined and concedes that several distinct trimmed graphs can reach the same joint probability. |
| Brignardello-Petersen et al. 2020, BMJ (minimally contextualised framework) | Official GRADE guidance for drawing conclusions from a network: each contrast is interrogated against a decision threshold and treatments are sorted into effectiveness categories by iterative promotion. Names multiplicity as unresolved, states that no satisfactory or widely accepted strategy exists, and declines to address it. |
| Brignardello-Petersen et al. 2020, BMJ (partially contextualised framework) | The companion framework that additionally assigns magnitude labels from the same nominal intervals, so the unaddressed multiplicity carries through into the wording of the published conclusion. |
| Mavridis et al. 2020, Biometrical Journal | Extends the P-score so that it measures the mean certainty that a treatment is better than the competing treatments by at least a minimum clinically important difference, and so that several outcomes can be combined with an explicit benefit-harm tradeoff. Addresses the clinical-margin half for a per-treatment metric, not the joint credibility of a set of ordering statements. |
| Trinquart et al. 2016, Annals of Internal Medicine | Reanalyzes 58 networks and reports that the median width of the 95 percent credible interval for SUCRA is 65 percentage points and that in 28 percent of networks there is a 50 percent or greater probability that the best-ranked treatment is not the best. Quantifies the imprecision of one rank; says nothing about the credibility of a conjunction of claims. |
| Salanti, Ades and Ioannidis 2011, Journal of Clinical Epidemiology | Introduces SUCRA and the rank-based presentation that decisions are read off, including probabilities of lying within an acceptable range from a reference treatment. Each summary is marginal in the treatment, which is the form the joint statement is missing from. |
| Rucker and Schwarzer 2015, BMC Medical Research Methodology | Defines the frequentist P-score as the mean of one-sided p-values, the metric that Mavridis et al. later extended to a clinically important margin. Also marginal per treatment, and the paper itself notes it offers no major advantage over inspecting intervals. |
Probable solution or research direction
Fix the family before the procedure: write down the exact set of claims a network report asserts, separating pairwise superiorities, category memberships and the full ordering, and state which of them a committee acts on jointly. For that family, derive a simultaneous procedure that uses the network covariance matrix rather than treating contrasts as independent, and report both the joint credibility of the asserted set and the largest subset reaching a stated target. Standardize the equivalence side by requiring an outcome-specific minimum important difference in the protocol, then define a tie as a contrast whose interval lies inside that region, which puts ranking metrics on the same scale as the decision and makes the Mavridis-style P-score extension usable by default. Settle the reporting conventions that the joint-probability approach leaves open: a default target probability and a deterministic rule for choosing among trimmed sets that reach it. Score every candidate on realized decision error by simulating networks with known orderings and reporting how often a category separation or a superiority claim is spurious as a function of the number of treatments.
Verification
Verdict. Partially addressed. Two independent readers proposed this and the reviewer accepted both, but real work covers parts of it: Barrientos et al. 2024 compute the joint posterior of a set of ordering statements and Mavridis et al. 2020 build a minimum important difference into a ranking metric. What no work supplies is an error-rate target for the family of claims a network report makes, a procedure that uses the known correlation among contrasts, or a convention for the joint-probability threshold, and the GRADE working group’s own guidance states that no satisfactory or widely accepted strategy exists.
Source
Derived from reading:nma-10, reading:nma-25 of the reviewed corpus.