No rule says which of three conflicting estimates to use
Statement
When the direct, indirect and network estimates of the same contrast disagree and carry different certainty ratings, no convention states which one a decision should use. GRADE directs the reader to the estimate from the highest-certainty evidence, which can be the indirect estimate and can therefore require acting on a quantity no network model outputs as its answer; other authorities hold that having committed to a network analysis one should always use the network estimate. The two rules can recommend differently from the same fitted model: in a worked pediatric antidepressant example the direct estimate for fluoxetine versus placebo was a standardized mean difference of -0.26 rated very low certainty, the indirect estimate -1.41 rated moderate, and the network estimate -0.51 rated very low, so following GRADE the clinician acts on an effect more than twice the size of the network estimate and more than five times the size of the direct one. Nothing establishes which selection rule has the better decision properties, because no simulation or empirical study scores them against realized error or decision error. The GRADE rule is also applied with no account of how selecting on estimated certainty distorts the reported interval, which is a selection problem rather than a reporting one.
Why it is open
Detecting the divergence is solved and deciding what to do about it is not: node splitting, inconsistency models and the meta-epidemiological literature all measure disagreement between direct and indirect evidence, and none of them prescribes an action. The three estimates are not independent, since the network estimate is a weighted combination of the other two, so selecting whichever carries the best rating is selecting on a statistic correlated with the quantity being chosen, and the interval reported afterwards is not the interval of the selection procedure. Certainty ratings are ordinal categories rather than probabilities, so no arithmetic converts moderate versus very low into weights, and any rule that uses them is a convention rather than a derivation. The alternative to selecting is modeling, through an inconsistency-allowing or bias-adjusted synthesis that returns one estimate with the disagreement priced into its width, but that changes what is reported rather than answering the question the panel asked, and no guidance names it as the resolution.
What has been tried
| Work | What it contributes |
|---|---|
| Al Khalifah et al. 2018, Network meta-analysis: users’ guide for pediatricians, BMC Pediatrics | Poses the question directly, records the conflict between the GRADE highest-certainty rule and the position that committing to a network commits one to the network estimate, and resolves the worked fluoxetine example on the indirect estimate (-1.41, moderate certainty) rather than the network estimate (-0.51, very low). |
| Puhan et al. 2014, A GRADE Working Group approach for rating the quality of treatment effect estimates from network meta-analysis, BMJ | Sets out the rating approach and the rule of taking the higher-certainty of the direct and indirect estimates, which is the selection rule in dispute. |
| Izcovich et al. 2023, A guide and pragmatic considerations for applying GRADE to network meta-analysis, BMJ | Current practical guidance for applying the rating framework, which keeps the certainty-based selection without evaluating its decision properties. |
| Dias et al. 2010, Checking consistency in mixed treatment comparison meta-analysis, Statistics in Medicine | Gives node splitting, the standard device for detecting the disagreement that triggers the choice; it says nothing about what to report once disagreement is found. |
| van Valkenhoef et al. 2015, Automated generation of node-splitting models for assessment of inconsistency in network meta-analysis, Research Synthesis Methods | Automates node splitting so that the direct, indirect and network estimates of each contrast are routinely produced, which is what makes the selection problem a routine one. |
| Veroniki et al. 2013, Evaluation of inconsistency in networks of interventions, International Journal of Epidemiology | Compares methods for evaluating inconsistency, establishing that detection is mature while the downstream decision is unaddressed. |
| Song et al. 2011, Inconsistency between direct and indirect comparisons of competing interventions: meta-epidemiological study, BMJ | Measures empirically how often direct and indirect estimates disagree, which fixes how frequently the selection question arises in practice. |
| Dias et al. 2013, Evidence Synthesis for Decision Making 4: Inconsistency in Networks of Evidence Based on Randomized Controlled Trials, Medical Decision Making | Sets out the modeling alternative in which inconsistency is represented within the synthesis rather than resolved by choosing among estimates. |
| Chaudhry et al. 2015, Network Meta-analysis: Users’ Guide for Surgeons, Part II: Certainty, Clinical Orthopaedics and Related Research | Presents the same certainty-based reading to a second clinical audience, showing the rule is taught rather than debated at the point of use. |
| Phillippo et al. 2019, Threshold Analysis as an Alternative to GRADE for Assessing Confidence in Guideline Recommendations Based on Network Meta-analyses, Annals of Internal Medicine | Offers the nearest decision-level alternative, reporting how far the evidence could change before the recommendation changes rather than selecting among competing estimates. |
Probable solution or research direction
Score the rules rather than argue them: simulate networks with known truth and known inconsistency, apply the GRADE highest-certainty rule, the always-network rule and a bias-adjusted synthesis, and report realized error and decision error for each; nothing in that comparison requires machinery that does not already exist. Quantify the selection effect separately, since where a rule picks the estimate with the best rating the reported interval is conditional on that selection, so its coverage has to be derived or simulated rather than assumed. Where the disagreement is large, test whether an inconsistency-allowing or bias-adjusted model that returns a single estimate with the disagreement priced into its width outperforms any selection rule, which would replace the choice with a model. Where a selection rule is retained, require it to be prespecified in the protocol, since choosing after seeing which of the three estimates favors the intervention is otherwise unconstrained.
Verification
Verdict. Confirmed open. One reader proposed this and the reviewer accepted it, noting that the quoted guide records a real conflict between an evidence-certainty selection rule and the convention of using the network estimate once a network has been fitted. The evidence is a single users’ guide, which is a weaker basis than a survey, though the two rules it contrasts are both stated in widely used guidance and the numerical example shows the divergence is not hypothetical.
Source
Derived from reading:nma-33 of the reviewed corpus.