NMA rank output lacks a decision-theoretic reconciliation with net benefit
Statement
Evidence synthesis reports treatment orderings as probability-best, rankograms, SUCRA or P-scores computed on a clinical outcome, while the economic model reports a separate ordering on net benefit, and no convention reconciles the two. SUCRA is an unweighted average of the cumulative rank probabilities, and Epstein 2019 states that this leaves it without empirical or theoretical grounding, such as utility theory, for preference-based choice. The two orderings can conflict inside a single model: in a seven-option varicose-vein appraisal a dominated option had a 33 percent probability of greatest net benefit at a threshold of 20,000 pounds per quality-adjusted life year while ranking fifth or worse in 27 percent of simulations, and SUCRA placed it above an option that was more likely to be in the top four, 86 percent against 72 percent on cumulative rank probability. Rank probabilities had not previously been used for economic evaluation at all, so nothing links the synthesis-stage ordering that submissions publish to the decision-stage ordering that determines reimbursement. The synthesis side has moved: the treatment hierarchy question has been formalized as something the analyst must state before selecting a ranking metric, and complex hierarchy statements can now be evaluated as posterior probabilities in released software. What is still missing is the mapping from a rank functional to a loss function, a rule for what to do when the clinical-outcome ordering and the net-benefit ordering diverge, and an agreed partial order that is reportable without assuming a risk attitude.
Why it is open
Rank statistics discard exactly what a reimbursement decision needs. A rank is invariant to the magnitude of the differences between options, so two networks with the same ordering and wildly different effect sizes yield identical SUCRA values, while net benefit is a function of magnitudes, costs and utilities; no transformation of the rank distribution can recover information the ranking threw away. Rank metrics are also defined relative to the set of options in the network, so adding or removing a comparator changes every treatment’s rank without any new evidence about it, whereas net benefit at a given threshold is a per-option quantity that does not move. The work that has been done sits on one side of a boundary: hierarchy questions have been formalized inside the synthesis and against clinical outcomes, and it stops at the economic model, so nobody owns the join, because the synthesis team does not hold the cost and utility inputs and the economic team does not publish a rank distribution. And the divergence has never been measured, so there is no evidence base saying how often a SUCRA ordering and a net-benefit ordering actually disagree in appraisals that contain both.
What has been tried
| Work | What it contributes |
|---|---|
| Epstein 2019, Beyond the cost-effectiveness acceptability curve: the appropriateness of rank probabilities for presenting the results of economic evaluation in multiple technology appraisal, Health Economics 28:801-807 | Argues SUCRA is inappropriate for economic evaluation because it is an unweighted average of cumulative rank probabilities with no utility-theoretic basis, computes rank probabilities of net benefit in a seven-option case study, and shows a dominated option with a 33 percent probability of greatest net benefit that nonetheless ranks fifth or worse in 27 percent of simulations. States that rank probabilities had not previously been used in economic evaluation. |
| Salanti, Ades and Ioannidis 2011, Graphical methods and numerical summaries for presenting results from multiple-treatment meta-analysis, Journal of Clinical Epidemiology 64:163-171 | Introduces the rankogram, the cumulative ranking curve and SUCRA as presentational summaries of a network’s posterior ordering. They are defined as descriptive quantities, not as solutions to a decision problem. |
| Rucker and Schwarzer 2015, Ranking treatments in frequentist network meta-analysis works without resampling methods, BMC Medical Research Methodology 15 | Defines the P-score, the frequentist analogue of SUCRA computed from point estimates and standard errors. It inherits the same absence of a loss function. |
| Salanti et al. 2022, Introducing the Treatment Hierarchy Question in Network Meta-Analysis, American Journal of Epidemiology 191:930-938 | The closest existing thing to the mapping this entry asks for: it requires the analyst to state the clinical question a hierarchy is meant to answer before choosing a metric, and matches metrics to questions. It works with clinical outcomes and stops short of a loss function or of the economic model’s net benefit. |
| Papakonstantinou et al. 2022, Answering complex hierarchy questions in network meta-analysis, BMC Medical Research Methodology 22, with the nmarank package on CRAN | Evaluates the posterior probability of arbitrary specified hierarchy statements, which supplies the machinery for reporting a decision-relevant ordering once someone specifies which statement the decision needs. |
| Rucker and Schwarzer 2017, Resolve conflicting rankings of outcomes in network meta-analysis: partial ordering of treatments, Research Synthesis Methods 8:526-536 | Reports a partial order rather than forcing a total ranking when outcomes conflict, which is the form of preference-free output this entry asks for on the multi-outcome side. It does not extend to conflict between a clinical ordering and a net-benefit ordering. |
| Trinquart, Attiche, Bafeta, Porcher and Ravaud 2016, Uncertainty in treatment rankings: reanalysis of network meta-analyses of randomized trials, Annals of Internal Medicine 164:666-673 | Reanalyzes published network meta-analyses and shows how wide the uncertainty around a reported ranking is, which is the reason a point ordering cannot substitute for a decision rule. |
| Chiocchia et al. 2020, Agreement between ranking metrics in network meta-analysis: an empirical study, BMJ Open 10:e037744 | Measures agreement among ranking metrics computed on the same clinical outcome. It establishes the template for the divergence study this entry asks for while stopping inside the synthesis. |
| Claxton 1999, The irrelevance of inference: a decision-making approach to the stochastic evaluation of health care technologies, Journal of Health Economics 18:341-364 | The decision-theoretic argument that adoption should follow expected net benefit rather than inferential statements about which option is superior. It is the standard against which a rank statistic used as a decision quantity has to be justified. |
| Hopkin et al. 2019, Combining Multiple Treatment Comparisons with Personalized Patient Preferences, Medical Decision Making 39:264-277 | Builds a patient-facing statin selection platform whose treatment ordering is a preference-weighted average of SUCRA values drawn from two separate network meta-analyses, which is the practice this entry says has no decision-theoretic justification. |
| Ades, Welton, Dias, Phillippo and Caldwell 2024, Twenty years of network meta-analysis: continuing controversies and recent developments, Research Synthesis Methods 15:702-727 | Surveys the state of the field including the continuing dispute over ranking metrics, confirming that the interpretation of ranks remains contested within evidence synthesis independently of the economic question. |
Probable solution or research direction
Write down the loss functions and match each reported rank statistic to the one that makes it optimal, so a reader can tell which decision a SUCRA or a probability-best is the answer to: probability-best is the Bayes rule under a zero-one loss on the identity of the top option, which is defensible only when exactly one option will be recommended and all errors are equally costly, and it is not the rule for a reimbursement decision on net benefit. Require the rank distribution of net benefit at the relevant threshold to be reported alongside the clinical-outcome ranking, since the probabilistic economic model already generates the simulations that produce it, and require both orderings to be shown together with an explicit note when they diverge. Measure how often they diverge by re-running a set of completed appraisals that contain both a network meta-analysis and a probabilistic economic model, reporting the proportion in which the SUCRA ordering and the net-benefit ordering disagree in the top three. Where no threshold or utility set can be agreed, report a partial order instead of a ranking: the set of options not dominated by first-order stochastic dominance of net benefit is preference-free within the class of increasing utility functions and is computable from the same simulations.
Verification
Verdict. Partially addressed. One reader proposed this and the reviewer accepted it as a distinct decision-theoretic reporting and interpretation gap rather than the simulation-performance issue in DEC-01, on the ground that rank probabilities and SUCRA encode no effect magnitudes, costs, utilities or loss function. It is recorded as partially addressed rather than confirmed open because the synthesis half has moved: the treatment hierarchy question has been formalized and arbitrary hierarchy statements can be evaluated as posterior probabilities in released software, so what remains absent is the loss function and the reconciliation with the economic model’s net-benefit ordering.
Source
Derived from reading:nma-27 of the reviewed corpus.