Method evaluations score estimator error, not decision error
Source note. This entry derives in part from a source document whose inline citations were ChatGPT interface tokens rather than references, and so resolve to nothing. Claims below are attributed to that document and were checked independently where possible; anything that could not be independently sourced is marked in the verification trail.
For a threshold on a known side of the truth, yes: near the threshold in poor-overlap scenarios the RMSE and expected-loss rankings correlated at only 0.45, and the disagreement was reproducible, not noise. It came from the direction of bias: a biased method decided wrongly far less often when its bias pointed away from the threshold (0.11 against 0.64). Averaged over the two sides of the threshold, as an analyst who does not know the truth must, the rankings agreed (0.93, the same best method in every poor-overlap scenario). RMSE is therefore an adequate proxy for decision error unless the side of the threshold is known. Read the study below.
Statement
ADEMP does not require, and its performance-measure inventory does not contain, any decision or net-benefit measure: Table 6 of Morris et al. 2019 lists bias, EmpSE, relative precision, MSE, model standard error, coverage and rejection percentage only. Whether ITC and PAIC method comparisons routinely omit decision metrics such as wrong-decision probability, net-benefit error or rank reversal is contested: the literature auditor searched and found no such evaluation, while GPT-5.6 Sol holds that this prevalence claim is not established by the cited paper. Decision-level assessment already exists outside method-comparison simulation, in threshold analysis for applied network meta-analysis (Phillippo et al. 2018) and in NICE DSU TSD 7’s probability that a strategy is optimal at a threshold willingness to pay. Expected value of perfect or sample information does not belong on the list of estimator-quality metrics: it measures the value of collecting information and can rise as uncertainty rises.
Why it is open
The link from estimator error to decision error runs through a decision threshold and a loss function, neither of which ADEMP requires, so a method that wins on RMSE can lose on the recommendation; Figure 1 of Morris et al. already shows RMSE and coverage ranking methods differently. Computing a decision metric means simulating a decision rule and usually an economic model, machinery that method authors treat as outside statistical scope and that no ITC benchmark provides. The blocker is therefore adoption inside method-comparison simulation rather than missing technical machinery, since threshold-based decision assessment already exists for applied syntheses.
What has been tried
| Work | What it contributes |
|---|---|
| Morris, White and Crowther 2019 | Defines ADEMP; Table 6 enumerates the performance measures, none of which is a decision or net-benefit measure, and its methods element explicitly allows a decision rule. |
| Phillippo et al. 2018 | Derives bias-adjustment thresholds, the smallest changes to the data that change the treatment decision, giving decision-level assessment for applied network meta-analysis. |
| NICE DSU Technical Support Document 7 (2012) | Reviewer’s checklist for evidence synthesis in decision making; uses an explicit decision model, willingness-to-pay threshold and probability that a strategy is optimal. Read by the code auditor from a document corpus at the given commit. |
Probable solution or research direction
Carry the existing threshold-and-loss machinery into method-comparison simulation instead of inventing decision metrics: report wrong-decision probability, net-benefit error and rank reversal alongside bias, RMSE and coverage under a prespecified decision model, threshold and loss, which the ADEMP performance-measure element is broad enough to hold. Re-score published benchmark scenarios on both axes so the estimation ranking and the decision ranking can be compared. Keep expected value of information out of that set and use it where it belongs, as a research-design criterion.
Our study
Do estimation and decision rankings of population-adjustment methods agree?
The question put to it. When population-adjustment methods are ranked by the decisions their estimates lead to rather than by RMSE, does the ranking change, and how does it depend on where the truth sits relative to the decision threshold?
What it found. For a threshold on a known side of the truth, yes: near the threshold in poor-overlap scenarios the RMSE and expected-loss rankings correlated at only 0.45, and the disagreement was reproducible, not noise. It came from the direction of bias: a biased method decided wrongly far less often when its bias pointed away from the threshold (0.11 against 0.64). Averaged over the two sides of the threshold, as an analyst who does not know the truth must, the rankings agreed (0.93, the same best method in every poor-overlap scenario). RMSE is therefore an adequate proxy for decision error unless the side of the threshold is known.
Results.
- Registered primary (near threshold, poor overlap, symmetric loss): mean Spearman correlation 0.45 (95% bootstrap interval 0.29 to 0.60); RMSE-best also loss-best in 38% of strata; verdict CONFIRMED.
- Exploratory: split-half reliability of the rankings 1.00 (RMSE) and 0.96 (expected loss) in poor-overlap scenarios, against a cross-measure correlation of 0.44.
- Biased methods (|bias| > 0.05), threshold 0.1 from the truth: wrong-decision probability 0.114 with bias pointing away from the threshold and 0.635 toward it.
- Exploratory: loss averaged over both sides of the threshold correlated with RMSE at 0.93 (distance 0.1) and 0.96 (0.3) in poor-overlap scenarios.
Design.
- Estimand
- Wrong-decision probability and expected loss of adopting B over A, against the true recommendation
- Methods compared
- unadjusted, MAIC on means, MAIC on means and variances, MAIC on the prognostic index variance, G-computation
- Data-generating mechanism
- Re-scoring of COV-03’s 108,000 stored replicates (five methods) against thresholds 0.1 or 0.3 above or below the truth, with loss ratios 1, 2 and 5; rank correlation between RMSE and expected-loss rankings by stratum, plus exploratory split-half reliability and two-sided averaging.
- Replicates
- 108,000 replicates re-scored (108 scenarios x 1000), 5 methods, 12 threshold-loss settings
What this does not answer. Decision on the relative effect only, without a net-benefit model; one source of replicates (COV-03) and five methods; no ML-NMR; priors on the side of the threshold not modeled. Peer review has not been done.
Peer review. Not yet done. The result stands on the study alone until it is.
Read it: PDF · Markdown · OpenDocument · Code and data: studies/DEC-01-when-the-two-rankings-disagree · Protocol: registered before the run
Verification
Verdict. Partially addressed. Auditors disagreed and the disagreement was evidence-bearing on both sides. Reopened by the full-text reading: 5 findings from 5 papers survived both the reader and an independent second reviewer, so the verdict moves from unverifiable to partially-addressed.
The source said. “ITC and PAIC method comparisons are scored on bias, RMSE and coverage, and ADEMP contains no decision element, so no evaluation reports how often a method produces the wrong reimbursement decision. Decision measures such as wrong-decision probability, incremental net-benefit error, expected regret, rank-reversal frequency, probability of selecting a dominated treatment and expected value of sample information are absent from the standard reporting set.”
What was wrong with it. The source listed EVPI and EVSI among decision-quality metrics for scoring estimators; GPT-5.6 Sol found this a category error, since value-of-information measures the value of collecting data rather than estimator quality. The premise that decision-level assessment is possible was unattributed in the source and is now sourced to Phillippo et al. 2018 (10.1111/rssa.12341) and NICE DSU TSD 7, and the ADEMP claim was checked line by line against Table 6 of Morris et al. 2019.
Dissent. literature dissented (open): I read the full text of Morris, White & Crowther 2019 (PMC6492164). ADEMP is Aims, Data-generating mechanisms, Estimands, Methods, Performance measures; Table 6 enumerates exactly bias, EmpSE, relative % increase in precision, MSE, average ModSE, relative % error in ModSE, coverage, bias-eliminated coverage, and rejection % (power/type I error). No loss function, net benefit, regret or decision metric appears anywhere. The citation is used accurately. Two narrowings worth noting: the paper’s Estimands element says ‘Define estimands and/or other targets’, and it states a method ‘might refer to a design or some procedure (such as a decision rule)’, so ADEMP accommodates decision rules without requiring a decision-consequence element. Figure 1 also demonstrates that RMSE and coverage can rank methods differently, which is the mechanism behind the entry’s ‘lower RMSE, more wrong decisions’ claim. Prior art exists for decision-based assessment of applied NMAs (Phillippo threshold analysis, nmathresh), but I found no ITC/PAIC method-comparison simulation scoring wrong-decision probability, net-benefit error or rank reversal. The problem is open. ‘Most important metric addition’ is opinion, not verifiable. codex dissented (overstated): C5-C7 are mathematically correct. For example, with true incremental net benefit 1 and threshold 0, an estimator fixed at -0.1 has RMSE 1.1 and always gives the wrong decision, while one fixed at 3 has RMSE 2 and never does. C2 is also correct: ADEMP does not require a decision element. However, the report conflates estimator losses with value-of-information measures. Wrong-decision probability and regret are valid only after specifying actions, a threshold, and a loss or net-benefit model. EVPI and EVSI measure the value of resolving or collecting information, not estimator quality, and can increase when uncertainty increases. C3, C4, and C11 are literature-prevalence claims not established by the cited ADEMP paper; C10 is an unsupported priority judgment. Existing decision-analysis guidance already connects evidence synthesis to threshold-based decisions, so the remaining gap is routine PAIC application, not missing technical machinery.
Revised by the full-text reading. Applied 2026-07-26 full-text reading of the consolidated library (687 papers).
- Verdict changed
- unverifiable -> partially-addressed 5 surviving findings from: Bias in identification of the best treatment in a Bayesian n; Hypothesis testing in Bayesian network meta-analysis; Living network meta-analysis compared with pairwise meta-ana; Rapid methods including network meta-analysis to produce evi; The Personalised Randomized Controlled Trial: Evaluation of . New citations: 10.1002/sim.9663, 10.1136/bmj.k585, 10.1186/s12874-018-0574-y, 10.1186/s13643-018-0829-z, 10.2147/clep.s69660
- Literature and prior-art check
- open / supported. I read the full text of Morris, White & Crowther 2019 (PMC6492164). ADEMP is Aims, Data-generating mechanisms, Estimands, Methods, Performance measures; Table 6 enumerates exactly bias, EmpSE, relative % increase in precision, MSE, average ModSE, relative % error in ModSE, coverage, bias-eliminated coverage, and rejection % (power/type I error). No loss function, net benefit, regret or decision metric appears anywhere. The citation is used accurately. Two narrowings worth noting: the paper’s Estimands element says ‘Define estimands and/or other targets’, and it states a method ‘might refer to a design or some procedure (such as a decision rule)’, so ADEMP accommodates decision rules without requiring a decision-consequence element. Figure 1 also demonstrates that RMSE and coverage can rank methods differently, which is the mechanism behind the entry’s ‘lower RMSE, more wrong decisions’ claim. Prior art exists for decision-based assessment of applied NMAs (Phillippo threshold analysis, nmathresh), but I found no ITC/PAIC method-comparison simulation scoring wrong-decision probability, net-benefit error or rank reversal. The problem is open. ‘Most important metric addition’ is opinion, not verifiable. Cites 10.1002/sim.8086: Confirms the cited claim: ADEMP has five elements and its performance-measure inventory contains no decision or net-benefit measure.. Cites 10.1111/rssa.12341: Real reference for the entry’s unsourced premise that decision-level assessment is possible and already formalized for applied NMA (not for method-comparison simulations), so the proposed direction is building on existing machinery rather than starting fresh..
- GPT-5.6 Sol, technical and source-code lens
- partially-solved / overstated. C5-C7 are mathematically correct. For example, with true incremental net benefit 1 and threshold 0, an estimator fixed at -0.1 has RMSE 1.1 and always gives the wrong decision, while one fixed at 3 has RMSE 2 and never does. C2 is also correct: ADEMP does not require a decision element. However, the report conflates estimator losses with value-of-information measures. Wrong-decision probability and regret are valid only after specifying actions, a threshold, and a loss or net-benefit model. EVPI and EVSI measure the value of resolving or collecting information, not estimator quality, and can increase when uncertainty increases. C3, C4, and C11 are literature-prevalence claims not established by the cited ADEMP paper; C10 is an unsupported priority judgment. Existing decision-analysis guidance already connects evidence synthesis to threshold-based decisions, so the remaining gap is routine PAIC application, not missing technical machinery. Cites 10.1002/sim.8086: Provides the general ADEMP framework, including a performance-measures element broad enough to contain a prespecified decision loss, although it does not require one.. Cites choxos/NICE-DSU-TSD-RAG@75abcb9c226a9f8fa7a0a4ffeba42874f1712ecb: Shows that evidence-synthesis appraisal can already use an explicit decision model, willingness-to-pay threshold, and probability that a strategy is optimal..
- Decision path
-
R3:partially-solved(n=1) → R7:counter-and-support-both-present
How the evidence falls in time
8 papers in the reviewed corpus bear on this problem, published 2013 to 2024. Partial progress is on the record, and later work still calls this open. Both belong in view.
This problem is recurrent: three or more papers spanning at least eight years assert it independently. Sustained restatement by authors who mostly do not cite each other is the strongest evidence this reading can offer that a gap is real rather than one group’s framing.
It is also being closed in pieces: partial results come from more than one paper in more than one year, each covering a different part.
| Year | Says | Paper |
|---|---|---|
| 2013 | confirms it is open | Evidence synthesis for decision making 7: a reviewer’s checklist |
| 2014 | partly addresses it | Bias in identification of the best treatment in a Bayesian network meta-analysis for binary outcome: a simulation study |
| 2018 | partly addresses it | Hypothesis testing in Bayesian network meta-analysis |
| 2018 | partly addresses it | Living network meta-analysis compared with pairwise meta-analysis in comparative effectiveness research: empirical study |
| 2018 | partly addresses it | Rapid methods including network meta-analysis to produce evidence in clinical decision support: a decision analysis |
| 2020 | confirms it is open | Comparison of Methods for Estimating Therapy Effects by Indirect Comparisons: A Simulation Study |
| 2023 | partly addresses it | The Personalised Randomized Controlled Trial: Evaluation of a new trial design |
| 2024 | confirms it is open | Response to discussant comments on “NMA, the first 20 years” |
Source
Derived from A:L193, A:L198, C:L214-215, C:L260, C:L1769, C:L2046-2053 of the reviewed corpus.