The treatment hierarchy is published without a certainty rating
Statement
GRADE and CINeMA rate the certainty of each pairwise relative effect a network meta-analysis produces; the treatment hierarchy is a separate output with its own decision use, and no rating framework covers it. Several partial devices exist and do not agree with one another: normalized entropy per treatment, credible intervals for SUCRA and predictive P-scores, rank-clustering models that admit ties, ranking metrics adjusted for a minimally important difference, and the precision of treatment hierarchy (POTH), a network-level scalar with a leave-one-treatment-out residual and a simulation reference distribution. Nothing states which of these a report must present, what value is too uncertain to rank on, how the answer changes when the ranking metric changes, or what a decision maker should do when the pairwise estimates are rated high certainty and the hierarchy is not. The quantity is neither degenerate nor self-interpreting: across 267 published networks POTH runs from 0.096 to over 0.999 with a median of 0.671 and an interquartile range of 0.528 to 0.833. Ranking output is the part of a network meta-analysis that reaches guideline panels, press releases and clinical summaries, and it is published as an ordered list with no certainty attached to the ordering itself.
Why it is open
A hierarchy is a nonlinear functional of the whole joint posterior, and the GRADE and CINeMA domains are defined comparison by comparison, so per-comparison ratings cannot be aggregated into a rating for an ordering. The candidate metrics measure different objects: entropy and rank intervals are per treatment, POTH is per network, minimally-important-difference adjustment changes the estimand rather than the uncertainty, and rank clustering changes the rated object from an ordering into a partition. None is on a scale comparable across networks without a reference distribution, so thresholds cannot be fixed by convention the way GRADE’s four levels were. The hierarchy question itself is not settled either: several distinct formalizations of which treatment is best have been described, and a certainty rating cannot precede a decision about which one is being rated. Ranking is also the output most attractive to report and least attractive to qualify, so nothing pushes an uncertainty statement into the abstract where the ordering appears.
What has been tried
| Work | What it contributes |
|---|---|
| Wigle et al. 2025, Precision of Treatment Hierarchy: A Metric for Quantifying Certainty in Treatment Hierarchies From Network Meta-Analysis, Statistics in Medicine | Defines POTH with a leave-one-treatment-out residual and a simulation-based null, notes that hierarchies from SUCRA and P-score are typically reported without any measure of certainty, and reports the empirical distribution across 267 published networks (minimum 0.096, maximum above 0.999, median 0.671, interquartile range 0.528 to 0.833); implemented in the CRAN package poth. |
| Salanti et al. 2011, Graphical methods and numerical summaries for presenting results from multiple-treatment meta-analysis: an overview and tutorial, Journal of Clinical Epidemiology | Introduces SUCRA, the ranking summary whose hierarchies are the object a certainty framework would have to rate. |
| Rucker and Schwarzer 2015, Ranking treatments in frequentist network meta-analysis works without resampling methods, BMC Medical Research Methodology | Gives the P-score, the frequentist analogue of SUCRA, so that the same reporting habit holds in both frameworks. |
| Trinquart et al. 2016, Uncertainty in Treatment Rankings: Reanalysis of Network Meta-analyses of Randomized Trials, Annals of Internal Medicine | Reanalyzes published networks to show that rank uncertainty is large and routinely omitted, establishing the empirical size of the problem a decade before a hierarchy-level metric existed. |
| Wu, Shih and Tu 2021, Using Normalized Entropy to Measure Uncertainty of Rankings for Network Meta-analyses, Medical Decision Making | Proposes normalized entropy as a per-treatment uncertainty measure for a treatment’s place in the hierarchy, one of the competing devices with no rule saying when to use it. |
| Veroniki et al. 2018, Is providing uncertainty intervals in treatment ranking helpful in a network meta-analysis?, Journal of Clinical Epidemiology | Examines whether ranking uncertainty intervals help interpretation, which is the reporting question a framework would have to answer rather than leave to preference. |
| Rosenberger et al. 2021, Predictive P-score for treatment ranking in Bayesian network meta-analysis, BMC Medical Research Methodology | Gives a predictive ranking metric whose intervals are continuous rather than discrete, a further alternative on a scale not comparable with the others. |
| Pearce and Zhou 2025, RaCE: A rank-clustering estimation method for network meta-analysis, Research Synthesis Methods | Estimates clusters of treatments that cannot be distinguished, changing the reported object from a strict ordering to a partition and so changing what a certainty rating would apply to. |
| Curteis et al. 2025, Ranking of treatments in network meta-analysis: incorporating minimally important differences, BMC Medical Research Methodology | Adjusts ranking metrics so that treatments differing by less than a clinically important amount are treated as equivalent, altering the hierarchy itself rather than its uncertainty. |
| Salanti et al. 2022, Introducing the Treatment Hierarchy Question in Network Meta-Analysis, American Journal of Epidemiology | Shows that which treatment is best has several distinct formalizations, so a certainty framework must first fix which hierarchy question it rates. |
| Chiocchia et al. 2020, Agreement between ranking metrics in network meta-analysis: an empirical study, BMJ Open | Measures how far published hierarchies depend on the choice of ranking metric, which is one of the sensitivities a certainty framework would need a domain for. |
| Davies and Galla 2021, Degree irregularity and rank probability bias in network meta-analysis, Research Synthesis Methods | Shows that network topology alone biases rank probabilities, a structural threat to a hierarchy that no per-comparison certainty domain captures. |
Probable solution or research direction
Fix the rated object first, using the existing taxonomy of hierarchy questions, since the certainty of a full ordering, of a best-treatment claim and of a top-k set are different quantities with different failure modes. Define domains by what actually degrades a hierarchy rather than by analogy to the pairwise domains: overlap of the effect posteriors, which POTH already measures; the contribution of low-certainty comparisons to the ordering; sensitivity of the order to the choice of ranking metric; and sensitivity to a minimally important difference. Calibrate thresholds against the empirical distribution across the 267 published networks rather than by convention, and require any published ordering to carry a hierarchy-level precision statement alongside it. Test the resulting rating by whether it separates orderings that survive a leave-one-study-out refit or a change of ranking metric from those that do not, which is the operating characteristic none of the current metrics has been shown to have.
Verification
Verdict. Partially addressed. One reader proposed this and the reviewer accepted that POTH and related quantities are useful partial measures of hierarchy precision but do not constitute a certainty framework with domains, interpretation thresholds and reporting rules. It is partially addressed rather than open because five distinct devices for hierarchy uncertainty were published between 2016 and 2025; what none supplies is the rule that says which to report and what value is too uncertain to act on.
Revised by the full-text reading. Applied 2026-07-26 full-text reading of the consolidated library (687 papers).
- Citation corrected
- normalized a citation year: ‘Davies and Galla 2020, Degree irregularity and rank probability bias in network meta-analysis, Research Synthesis Methods’ -> ‘Davies and Galla 2021, Degree irregularity and rank probability bias in network meta-analysis, Research Synthesis Methods’ 10.1002/jrsm.1454 is cited with two different years across the registry. CrossRef gives 2020 online and 2021 in print; the issue year is used throughout.
- Citation corrected
- normalized a citation year: ‘Salanti et al. 2021, Introducing the Treatment Hierarchy Question in Network Meta-Analysis, American Journal of Epidemiology’ -> ‘Salanti et al. 2022, Introducing the Treatment Hierarchy Question in Network Meta-Analysis, American Journal of Epidemiology’ 10.1093/aje/kwab278 is cited with two different years across the registry. CrossRef gives 2021 online and 2022 in print; the issue year is used throughout.
Source
Derived from reading:nma-06 of the reviewed corpus.