No rule says what proves similarity from an indirect comparison
Statement
Reimbursement pathways that turn on whether a new treatment is clinically similar to an existing one, such as the NICE cost-comparison route, are routinely decided from indirect comparisons, and no guidance states what the estimate must show before similarity may be concluded. A systematic review of all 41 NICE cost-comparison evaluations from the pathway’s introduction in 2017 to April 2025 found companies relying heavily on statistically non-significant indirect comparison results, external assessment groups objecting, and only about 32% of appraisals applying any formal method to explore the uncertainty in those results. Absence of a detected difference and demonstrated similarity are different claims, and nothing requires a non-inferiority margin to be prespecified, fixes the posterior probability or interval position that suffices, or says what to do when no clinically validated margin exists. The disagreement is public: one external assessment group has concluded that non-significant differences or overlapping 95% intervals may be sufficient to demonstrate clinical similarity, and another has published the opposite position, with a comment and reply exchange in the record. The estimate carrying the claim is an indirect one, so it already rests on transitivity and, in population-adjusted analyses, on transportability, and none of that enters the threshold. The single published framework reports the posterior probability that a contrast lies within a margin and derives a hypothetical threshold by empirical distribution function where no validated margin exists, but concedes its own 95% probability cutoff is arbitrary and does not extend to population-adjusted comparisons.
Why it is open
The rule has to be set by the agency that uses it, and the NICE methods manual is silent, so no submission is ever rejected for lacking one and no submitter has an incentive to propose a stricter bar than the one accepted last time. Where a clinically validated margin does not exist, any rule needs a convention for manufacturing one, and manufacturing a margin from the evidence being tested is circular unless the convention is fixed in advance. A defensible threshold would also have to price the extra assumptions of the indirect route, a quantity nobody has calibrated and which cannot be estimated from the data that produced the comparison. The pathway’s attraction to all parties is speed, so requirements that add analysis are resisted from both sides. Because the two assessment groups disagree in print and neither position has been evaluated against realized decision error, there is no empirical basis on which the agency could choose between them.
What has been tried
| Work | What it contributes |
|---|---|
| Edwards et al. 2026, Clinical similarity in cost-comparison evaluations: a systematic review of current methods in NICE appraisals and the development of a framework for the formal assessment of clinical similarity, BMJ Open | Reviews all 41 NICE cost-comparison evaluations from 2017 to April 2025, states that NICE gives no guidance on interpreting non-significant indirect comparison results to prove clinical similarity, finds only about 32% applied any formal method, and proposes point-and-density plots reporting the probability of non-inferiority against a margin; applied to appraisal TA1019 it returns 95.9% and 86.3% probabilities of non-inferiority where the frequentist comparison was non-significant. |
| Lee et al. 2025, How Similar Is Similar Enough? Assessment of Indirect Treatment Comparisons to Support Similarity for National Institute for Health and Care Excellence’s Cost Comparison Route, Value in Health | An assessment group’s review of indirect comparisons used to support similarity in the cost-comparison route, raising concerns about reliance on non-significant results and recommending methods that would improve transparency. |
| Poley et al. 2025, Cost Comparisons in NICE Technology Appraisals: An External Assessment Group Perspective, PharmacoEconomics | States the opposing assessment-group position that a lack of statistically significant differences or overlapping 95% intervals may suffice to demonstrate clinical similarity; the disagreement is recorded in a published comment (doi 10.1007/s40273-025-01551-7) and reply (doi 10.1007/s40273-025-01553-5). |
| Hashim et al. 2021, A systematic review of noninferiority margins in oncology clinical trials, Journal of Comparative Effectiveness Research | Documents how non-inferiority margins are chosen when direct evidence exists, which is the comparison case against which an indirect-comparison rule would have to be set. |
| Bucher et al. 1997, The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials, Journal of Clinical Epidemiology | Gives the anchored indirect estimator whose interval position is what similarity claims are read off in practice. |
| NICE health technology evaluations: the manual (2023) | The process document governing the cost-comparison route; it contains no rule for interpreting a statistically non-significant indirect comparison as evidence of similarity. |
Probable solution or research direction
State the rule as a decision problem rather than a significance test: fix a margin on the scale the committee acts on, require the posterior probability that the contrast lies within it, and set the threshold from the error rate the pathway is willing to accept rather than from convention. Require the margin to be prespecified in the final scope, so that it cannot be chosen after the estimate is seen, which is the one procedural change the review recommends and no guidance imposes. Establish operating characteristics by simulation: for realistic cost-comparison evidence structures, compute how often a stated probability threshold declares similarity when a clinically meaningful difference exists, as a function of network sparsity, heterogeneity and the size of a transitivity violation. Extend the rule to population-adjusted comparisons, where the effective sample size after weighting rather than the nominal sample size determines what probability is attainable, so that a threshold set for an unadjusted comparison is not silently reused.
Verification
Verdict. Confirmed open. One reader proposed this and the reviewer accepted it as a decision-methods gap distinct from reporting requirements, noting that a single proposed Bayesian framework does not establish an agreed or validated rule. The evidence is a single systematic review, but it covers the complete population of 41 relevant appraisals and records a published disagreement between two assessment groups.
Source
Derived from reading:nma-01 of the reviewed corpus.