Treatment hierarchies lack a declared target-population referent
Yes: where effect modification changed the true ranking across plausible targets, the estimated hierarchy moved 1.6 to 2.4 places per treatment across targets against 0.1 to 1.4 places across sampling replicates, so a hierarchy must name its population. But estimated hierarchies also moved 0.2 to 2.1 places across targets where the true ranking did not change, through the uncertainty of meta-regression slopes away from the data’s center, so a changed hierarchy is not by itself evidence of population dependence. Read the study below.
Statement
Ranking metrics in network meta-analysis, SUCRA, P-scores, probability of being best and rankograms, are functionals of the whole set of relative effects, so they inherit the population dependence of every contrast in the network at once. No ranking metric is defined with respect to a declared target covariate distribution, and no journal or health technology assessment template requires a published hierarchy to name the population it refers to. The dependence is demonstrated rather than theoretical: reweighting the trial samples in a four-medication cocaine dependence network to resemble successive target populations reordered the medications in both directions, with the medication ranked worst for abstinence becoming second best, while every pairwise interval still overlapped, so a reader of the unweighted hierarchy had no signal that the ordering was population-specific. Wigle and Moodie 2026 supply the conditional version, a hierarchy evaluated at one covariate profile from an individual participant data network meta-analysis with treatment-covariate interactions, and report a combination therapy ranked first for one hypothetical patient and fourth of six for another; they explicitly advise against the aggregate-data meta-regression route for this purpose because of ecological confounding, which closes it to the aggregate-only networks most submissions contain. What does not exist is the target-standardized version: a hierarchy computed from relative effects averaged over a named target population’s joint covariate distribution, a requirement that the population referent be stated, and a diagnostic that separates rank movement caused by sampling noise from rank movement caused by the choice of target.
Why it is open
The ingredients sit on both sides of a join nobody has made. Population adjustment already returns population-average effects for a declared target from a mixed individual and aggregate network, and ranking metrics are computable from any posterior or covariance of relative effects, but the ranking machinery lives in packages that carry no target-population representation and the adjustment machinery ships no ranking output. The estimand is also awkward in a way a pairwise effect is not: a hierarchy is a discontinuous ordering, so uncertainty in the reported target moments propagates non-smoothly and a small change in the target distribution can flip ranks without moving any effect materially. There is no agreed answer to which population a hierarchy should refer to when one review serves several decision-makers, so the honest output is a family of hierarchies indexed by target rather than the single ordering that made rankings popular in the first place. Nothing forces the issue either: no reporting standard asks for the population referent, so an unlabeled hierarchy is never rejected.
What has been tried
| Work | What it contributes |
|---|---|
| Wigle and Moodie 2026, Research Synthesis Methods | states that a standard hierarchy is not personalized to any patient and depends on the covariate distribution of the population, and adapts SUCRA to a single covariate profile from an individual participant data network meta-analysis with treatment-covariate interactions; conditional rather than target-standardized, and advised against for aggregate-data networks |
| Susukida et al. 2018, International Journal of Methods in Psychiatric Research | holds the evidence base fixed and reweights the trial samples to resemble target populations, showing that the ranking of four medications for cocaine dependence changes with the target alone |
| Salanti, Ades and Ioannidis 2011, Journal of Clinical Epidemiology | defines SUCRA and rankograms as summaries of the posterior of relative effects, with no target covariate distribution anywhere in the definition |
| Rucker and Schwarzer 2015, BMC Medical Research Methodology | gives the frequentist P-score as a resampling-free analogue of SUCRA, computed from the point estimates and standard errors of relative effects and inheriting the same unstated population referent |
| Trinquart et al. 2016, Annals of Internal Medicine | reanalyzes published network meta-analyses to show how much sampling uncertainty rankings carry; quantifies the noise component of rank instability, which a target-sensitivity analysis would have to be separated from |
| Phillippo et al. 2020, Journal of the Royal Statistical Society Series A | multilevel network meta-regression produces population-average effects for an arbitrary declared target from a mixed individual and aggregate network, supplying the effect draws a target-standardized hierarchy would be computed from, but returns no ranking output |
| Wigle et al. 2026, arXiv:2605.15142 | extends hierarchy construction to component network meta-analysis, including which hierarchy questions are estimable from a given design; extends the object without attaching a population referent to it |
| Phillippo et al. 2018, Medical Decision Making (NICE DSU TSD18) | requires the target population to be stated explicitly for population-adjusted indirect comparisons, the precedent this problem asks to be extended from effect estimates to hierarchies |
Probable solution or research direction
Define SUCRA, P-score and probability of being best as functionals of target-standardized relative effects and compute them from a fit that already returns population-average effects for an arbitrary covariate distribution, so the hierarchy inherits a named target instead of an implicit one. Add a target-sensitivity display: recompute the hierarchy over a prespecified set of plausible targets, each contributing trial population, the pooled network population and the jurisdictional decision population, and report each treatment’s rank range across that set alongside its rank interval from sampling variability. Implement it where the target representation already exists, as a ranking method applied to the marginal effect draws produced by multilevel network meta-regression, so no new inferential machinery is needed. Require any reported hierarchy to state the population it refers to, in the way the target population is already required for population-adjusted effects in health technology appraisal.
Our study
Treatment hierarchies across target populations
The question put to it. Does a treatment hierarchy move more across plausible target populations than across sampling variability, so that a hierarchy must name its population?
What it found. Yes: where effect modification changed the true ranking across plausible targets, the estimated hierarchy moved 1.6 to 2.4 places per treatment across targets against 0.1 to 1.4 places across sampling replicates, so a hierarchy must name its population. But estimated hierarchies also moved 0.2 to 2.1 places across targets where the true ranking did not change, through the uncertainty of meta-regression slopes away from the data’s center, so a changed hierarchy is not by itself evidence of population dependence.
Results.
- Refuting sentence fails: in all 10 scenarios with a true rank change, movement across targets exceeded movement across replicates beyond Monte Carlo error.
- Spurious movement across targets with no true change: 0.23 to 2.07 places per treatment.
- Best treatment misidentified at an outer target in up to 70% of analyses (two trials per comparison, closely spaced effects).
Design.
- Estimand
- Treatment ranks (P-scores) at declared target covariate means
- Methods compared
- fixed-effect meta-regression with P-scores at each target
- Data-generating mechanism
- Star networks of five treatments against a reference with treatment-specific linear modification; fixed-effect meta-regression and P-scores at three targets; rank movement across targets against movement across independent replicates. 24 cells, 1000 replicates each.
- Replicates
- 1000 per scenario, 24,000 in total
What this does not answer. Study-level meta-regression rather than ML-NMR; P-scores only; one covariate; fixed-effect. Peer review has not been done.
Peer review. Not yet done. The result stands on the study alone until it is.
Read it: PDF · Markdown · OpenDocument · Code and data: studies/EST-12-a-hierarchy-with-no-population · Protocol: registered before the run
Verification
Verdict. Confirmed open. Two independent readers proposed it from different literatures, a methods paper on personalized hierarchies and an applied reweighting study, and the reviewer accepted both, noting that a hierarchy can technically be calculated from target-standardized draws without the target-population ranking estimand having been defined, reported or implemented as a standard method.
Source
Derived from reading:nma-03, reading:nma-31 of the reviewed corpus.