Network meta-analysis displays are tested for acceptability, not for accuracy
Statement
A network meta-analysis reaches guideline panels, reimbursement committees and clinicians as a display rather than as a posterior, and no display carries a measured reading-error rate. A systematic survey of 200 reviews published in 2020 and 2021 counted 1133 presentation formats in use, with forest plots in 108 reviews (54.0 percent), league tables in 93 (46.5 percent) and the surface under the cumulative ranking curve in 69 (34.5 percent), while a structured evidence summary appeared in 10 (5.0 percent) and covered benefits and harms together in 3 (1.5 percent). New formats keep arriving, among them the rank-heat plot, the Kilim plot, spie charts, color-coded multi-outcome tables and interactive summary tools, and each is introduced with a worked example rather than with evidence that a reader extracts the right answer from it. Development studies stop at acceptability and preference: the GRADE network summary-of-findings table came out of four rounds of semistructured interviews with 32 stakeholders, two color-coded multi-outcome formats were refined by qualitative content analysis with 26 participants against a single exemplar network in acute pain, and the MATCH-IT interactive tool was revised over five cycles of think-aloud testing with 26 physicians, 24 of whom (94 percent) reported no or sparse prior experience of interpreting a network meta-analysis. One randomized comparison exists: 149 third-year medical students were assigned to two MATCH-IT versions differing in color coding and in which outcomes appear on entry, 82.5 percent answered at least 4 of 5 interpretation items correctly and 75.8 percent reached the recommendation an expert panel had reached, and the design contrast made no difference to interpretation or to understandability. Nothing compares display families against each other on one network, nothing scores whether a reader recognizes that a top-ranked treatment is imprecisely estimated or that a contrast is carried entirely by indirect evidence, and nothing has been run on the panels and committees the displays exist to serve.
Why it is open
A comprehension study needs an answer key, and the field has not agreed what a correct reading of a network result is. Scoring whether a reader took the right message from a ranking display presupposes a position on what a ranking licenses, and that position is contested: the same fit yields orderings that differ by ranking metric and by whether uncertainty is penalized, so the answer key would encode a judgment the field is still arguing about. The audience that matters is also the audience that cannot be recruited at scale, since guideline panels and reimbursement committees have members in the tens, read a display once inside a deliberation that is itself the outcome of interest, and cannot be randomized without disturbing the process being measured; the substitute available in hundreds is students, who are not the audience. Display work is done by the groups that build displays, and a head-to-head trial can only demote one of the entrants, so no one whose format is in use has an incentive to run it and no funder treats presentation evaluation as a methods output. The comparison also has no natural unit, because a league table and a rank-heat plot answer different questions from one fit, so a trial must first fix the question, which is the step on which agreement is missing.
What has been tried
| Work | What it contributes |
|---|---|
| Lovsletten, Wang, Pitre, Odegaard, Veroniki, Lunny, Tricco, Agoritsas & Vandvik 2024, Journal of Clinical Epidemiology 173:111445 | Systematic survey of 200 systematic reviews with network meta-analysis published 2020 to 2021 across 158 journals; counts 1133 presentation formats, forest plots in 108 reviews (54.0 percent), league tables in 93 (46.5 percent), SUCRA in 69 (34.5 percent), structured evidence summaries in 10 (5.0 percent) and in 3 (1.5 percent) across benefits and harms, absolute effects in 16 of 133 reviews reporting dichotomous outcomes (11.7 percent) and certainty ratings inside a presentation format in 26 (13.0 percent). Establishes what is in use; measures no comprehension. |
| Stokke Hunskaar, Lovsletten, Muller & Vandvik 2024, BMJ Evidence-Based Medicine 29(1):29-36 | The only randomized comparison of network meta-analysis display design. 149 third-year medical students were randomized between two MATCH-IT versions, one using color coding to categorize interventions by magnitude and direction of effect with all outcomes shown on entry and one with neither; 82.5 percent (n=123) answered at least 4 of 5 interpretation items correctly, 75.8 percent (n=114) recommended what an expert panel recommended, and the design alternatives were preferred but had no bearing on interpretation or understandability. Compares two variants of one tool, not display families, and in students. |
| Lovsletten, Stokke Hunskaar, Heen, Bekkering, Vanden Poel, Zeraatkar, Vermandere, Aertgeerts, Delvaux, Achille, Busse, Agoritsas & Vandvik 2024, Journal of Clinical Epidemiology 172:111399 | Qualitative user testing of MATCH-IT over five development cycles with 26 practicing physicians in Norway, Belgium and Canada, 24 of whom (94 percent) had no or sparse prior experience of interpreting network meta-analysis; think-aloud plus semistructured interview with directed content analysis. Reports proof of concept and names comparison of MATCH-IT with similar tools and formats as the next step, which has not been done. |
| Phillips, Sadeghirad, Busse, Brignardello-Petersen, Cuello-Garcia, Kenji Nampo, Guo, Bzovsky, Bannuru, Thabane, Bhandari & Guyatt 2022, BMJ Open 12(6):e056400 | Design validation of two color-coded formats for presenting minimally and partially contextualized network results across multiple outcomes, using qualitative content analysis of interviews with 26 participants including 20 practicing clinicians. Ends with two formats each preferred by a substantial group and no criterion for choosing between them; validated against a single exemplar network. |
| Yepes-Nunez, Li, Guyatt, Jack, Brozek, Beyene, Murad, Rochwerg, Mbuagbaw, Zhang, Florez, Siemieniuk, Sadeghirad, Mustafa, Santesso & Schunemann 2019, Journal of Clinical Epidemiology 115:1-13 | Develops the GRADE summary-of-findings table for network meta-analysis through four rounds of semistructured interviews with 32 stakeholders who conduct or use meta-analysis, yielding six candidate tables and one judged highly acceptable. The endpoint is acceptability, not accuracy of reading. |
| Veroniki, Straus, Fyraridis & Tricco 2016, Journal of Clinical Epidemiology 76:193-199 | Introduces the rank-heat plot for many treatments across many outcomes in one figure and states that research on the understanding of the available approaches for presenting treatment ranking by clinicians, guideline developers and policy makers is needed and critical, because different approaches may complicate inference and decision-making. That research has not been carried out. |
| Daly, Mbuagbaw, Thabane, Straus & Hamid 2020, BMC Medical Research Methodology 20(1):266 | Proof-of-concept framework plotting each treatment’s performance across outcomes as a spie chart whose coverage area represents the probability of ranking best overall, illustrated on dietary-fat and psoriasis networks. Concludes that formal validation in the decision-making context and statistical comparisons with other recent approaches are required, and neither has followed. |
| Seo, Furukawa, Veroniki, Pillinger, Tomlinson, Salanti, Cipriani & Efthimiou 2021, Research Synthesis Methods 12(1):86-95 | The Kilim plot for multi-outcome network results, conveying uncertainty through a truncated color scale rather than intervals; states that it gives no relative effects and shows no confidence intervals, so what a reader can and cannot infer from it is a design decision that was never tested. |
| Carrasco-Labra, Brignardello-Petersen, Santesso, Neumann, Mustafa, Mbuagbaw, Etxeandia Ikobaltzeta, De Stio, McCullagh, Alonso-Coello, Meerpohl, Vandvik, Brozek, Akl, Bossuyt, Churchill, Glenton, Rosenbaum, Tugwell, Welch, Garner, Guyatt & Schunemann 2016, Journal of Clinical Epidemiology 74:7-18 | Randomized trial among 290 systematic review users comparing a new and the current summary-of-findings table format on item-by-item understanding, accessibility, satisfaction and preference, with 63 percent and 62 percent more correct answers for the new format on understanding risk difference and understanding certainty. This is the design template the network display literature has not executed; it was run on pairwise tables. |
| Kossmeier, Tran & Voracek 2020, BMC Medical Research Methodology 20(1):26 | Comprehensive review, taxonomy and feature analysis of graphical displays for meta-analysis and systematic reviews, correlating functionality features across the inventory; identifies displays that remain usable at medium to large numbers of studies while still identifying each study and its effect as rare. Maps the design space without evaluating any display against readers. |
Probable solution or research direction
Fix the readings before the formats: name the small set of judgments a display must support, which treatment the evidence best supports, how uncertain that ordering is, what the absolute difference is at a stated baseline risk and horizon, and which contrasts rest on indirect evidence alone, then derive the answer key for each from the fitted model rather than from opinion. Run on displays the design already executed for tables: the randomized trial that compared summary-of-findings formats among 290 review users and scored understanding item by item transfers directly to a comparison of league table, forest plot against a common comparator, rankogram or cumulative ranking curve, rank-heat plot and an interactive summary, all built from one network at fixed certainty ratings. Report accuracy per reading with the direction of the error and the time taken, so that a display can be described the way a diagnostic is described and a review can state which readings its chosen format supports and which it does not. Recruit guideline panelists and health technology assessment reviewers as the primary stratum even at sample sizes in the low hundreds, and report student samples separately rather than as evidence about committees. The cheap first step is to start with the two formats that dominate the surveyed corpus of 200 reviews, the forest plot and the league table, since those are what readers actually meet.
Verification
Verdict. Partially addressed. This rests on labeller-reported absence rather than on a source arguing for it: 6 of 12 gap labellers independently named it after failing to find it while attaching future-work sentences to the registry, and the supporting sentences span 15 distinct papers from 2014 to 2026, most of them proposals of a display conceding that their own format was never evaluated with readers. Checked against the registry, where DEC-05 asks what a committee page should contain, OUT-21 asks what a set of absolute effects must have in common, DEC-23 and DEC-29 concern the certainty and the distortion of the hierarchy, and DEC-30 concerns the reliability of credibility labels; none asks whether any display is read correctly. The verdict is partially-addressed rather than confirmed-open because quantitative evaluation is not wholly absent: one randomized trial with 149 medical students compared two variants of one interactive tool, and three qualitative user-testing and design-validation studies exist with 26, 26 and 32 participants. What remains unevaluated is the choice between display families, the accuracy of any of them against a defined answer key, and behavior in the panels and committees that use them. Provenance here is weaker than for neighboring entries, since no source argues the gap as a methodological problem; the evidence is a repeated admission inside display proposals that validation was not done, plus one explicit call that the research is critical because different presentations may lead to different decisions.
Source
Derived from reading:uncovered of the reviewed corpus.