ROB-MEN reliability is unmeasured and CINeMA agreement collapses on large networks

Decision analysis, regulation & reporting
Medium-high priority
Partially addressed
CINeMA and ROB-MEN attach graded labels to individual network estimates, confidence in the result and risk of bias due to missing evidence respectively, and those labels are carried into guidelines and appraisals as if they were properties of the evidence. The ROB-MEN guidance states that the reproducibility of CINeMA and ROB-MEN assessments has not been studied and that interrater agreement work would be informative. For CINeMA that has since been partly answered: 13 assessors applying both CINeMA and GRADE to four networks of differing size and complexity produced interrater agreement, measured by Gwet’s AC, between 0.02 and 0.73 for CINeMA against 0.49 to 0.70 for GRADE, with the 0.02 falling on the largest and most complex network, while concordance between the two approaches fell from 1 to 0.42 as the number of comparisons and the share of indirect evidence rose. Nothing comparable exists for ROB-MEN, whose judgment-based inputs are not incidental: the assessor sets the suspected direction of bias for each comparison, decides what percentage contribution from suspect evidence counts as substantial (set at 15% by user choice in the guidance paper’s own case study), and reads a difference between network meta-analysis and network meta-regression estimates as evidence of small-study effects, while the quantitative across-study assessment needs 10 studies per comparison and was therefore unavailable for all 15 comparisons of that example. Because both instruments compute the final label algorithmically from those inputs, the output looks objective while inheriting whatever variability sits upstream, and neither reports how far its label would move if an input were set differently. What is missing is a reliability estimate for ROB-MEN, a replication of the CINeMA result over more networks and assessors, and any evidence that either label is stable enough to be treated as a property of the evidence rather than of the assessor. The instrument landscape has since widened without closing the question: a 2026 map counts 22 appraisal instruments for network meta-analysis, 11 of them NMA-specific, and finds measurement-property data reported for only 9, while the RoB NMA tool published in 2025 reports no reliability estimate at all beyond a pilot completion time.

DEC-30 Partially addressed Medium-high priority Established Tractability 4/5

Statement

CINeMA and ROB-MEN attach graded labels to individual network estimates, confidence in the result and risk of bias due to missing evidence respectively, and those labels are carried into guidelines and appraisals as if they were properties of the evidence. The ROB-MEN guidance states that the reproducibility of CINeMA and ROB-MEN assessments has not been studied and that interrater agreement work would be informative. For CINeMA that has since been partly answered: 13 assessors applying both CINeMA and GRADE to four networks of differing size and complexity produced interrater agreement, measured by Gwet’s AC, between 0.02 and 0.73 for CINeMA against 0.49 to 0.70 for GRADE, with the 0.02 falling on the largest and most complex network, while concordance between the two approaches fell from 1 to 0.42 as the number of comparisons and the share of indirect evidence rose. Nothing comparable exists for ROB-MEN, whose judgment-based inputs are not incidental: the assessor sets the suspected direction of bias for each comparison, decides what percentage contribution from suspect evidence counts as substantial (set at 15% by user choice in the guidance paper’s own case study), and reads a difference between network meta-analysis and network meta-regression estimates as evidence of small-study effects, while the quantitative across-study assessment needs 10 studies per comparison and was therefore unavailable for all 15 comparisons of that example. Because both instruments compute the final label algorithmically from those inputs, the output looks objective while inheriting whatever variability sits upstream, and neither reports how far its label would move if an input were set differently. What is missing is a reliability estimate for ROB-MEN, a replication of the CINeMA result over more networks and assessors, and any evidence that either label is stable enough to be treated as a property of the evidence rather than of the assessor. The instrument landscape has since widened without closing the question: a 2026 map counts 22 appraisal instruments for network meta-analysis, 11 of them NMA-specific, and finds measurement-property data reported for only 9, while the RoB NMA tool published in 2025 reports no reliability estimate at all beyond a pilot completion time.

Why it is open

Measuring reliability requires a designed duplicate-assessment study across multiple assessors and multiple networks, it produces no new method, and the one study that exists needed 16 authors and 150 to 481 minutes of assessor time per network per tool. That study also shows a single summary agreement coefficient will not settle the question, since agreement was acceptable on small networks and collapsed on the largest, so any replacement design has to vary network size, connectivity and share of indirect evidence deliberately. ROB-MEN is harder to study than CINeMA because its most consequential input is unavailable in most real networks: funnel plots and small-study-effect tests need 10 studies per comparison, so in typical networks the across-study assessment reduces to unanchored qualitative judgment. ROB-MEN also consumes a percentage contribution matrix whose construction is itself contested, so its reliability cannot be separated from a definitional choice the field has not settled. Tool developers have little incentive to run a study that can only reduce confidence in their own instrument, and reliability measurement of an appraisal tool is not treated as a fundable research output.

What has been tried

Work What it contributes
Chiocchia, Holloway and Salanti 2023, BMC Medical Research Methodology Guidance paper for the ROB-MEN web application; documents the judgment-based inputs, sets the substantial-contribution threshold at 15% by user choice in its worked example, reports no funnel plot or small-study-effect test was applicable to any comparison there, and states that reproducibility of CINeMA and ROB-MEN has not been studied.
Chiocchia et al. 2021, BMC Medicine The ROB-MEN framework itself: per-comparison within-study and across-study judgments combined with the contribution matrix into a risk-of-bias label for every network estimate.
Nikolakopoulou et al. 2020, PLoS Medicine CINeMA: the domain framework producing per-comparison confidence ratings, weighting study-level judgments by contribution.
Minozzi et al. 2025, Journal of Clinical Epidemiology Thirteen assessors applied GRADE and CINeMA to four networks chosen for differing size and complexity; reports interrater agreement of 0.02 to 0.73 for CINeMA and 0.49 to 0.70 for GRADE by Gwet’s AC, overall concordance between approaches from 1 to 0.42, and median assessment times of 150 to 330 minutes for CINeMA and 160 to 481 for GRADE. Covers CINeMA, not ROB-MEN.
Battain et al. 2026, Journal of Clinical Epidemiology Companion qualitative survey of the same assessors: CINeMA’s automation made application fast but reduced understanding of how the conclusion was reached, GRADE was judged transparent but slow and subjective, and both were said to need further training before correct use.
Noori et al. 2024, Journal of Clinical Epidemiology Applies GRADE and CINeMA to the same opioid network: ratings differed for 36 of 105 pain-relief comparisons and 11 of 66 function comparisons, all by one level, with CINeMA typically lower because of transitivity and heterogeneity assessment. Measures agreement between approaches, not between assessors.
Minozzi et al. 2020, Journal of Clinical Epidemiology Interrater reliability study of RoB 2, reporting low agreement and application difficulties; the design template that a ROB-MEN reliability study would follow.
Minozzi et al. 2019, Journal of Clinical Epidemiology Interrater reliability of ROBINS-I across five raters and 31 studies, with only slight agreement on the overall judgment and a recommendation for calibration exercises and intensive training; shows how far an instrument’s labels can depend on who applies them.
Lunny et al. 2025, BMJ publishes the RoB NMA tool, a 17-item instrument covering network geometry and node definition, effect modifiers and transitivity, and statistical synthesis including multi-arm handling, heterogeneity assumptions, direct-versus-indirect conflict and selective reporting; its only reported operating characteristic is a pilot completion time of 79 minutes (range 30 to 150), so it inherits the reliability question this entry describes rather than answering it
Mondragon et al. 2026, Journal of Clinical Epidemiology maps 22 appraisal instruments applicable to network meta-analysis, 11 of them NMA-specific, and finds that only 9 of the 22 report any measurement-property data

Probable solution or research direction

Run for ROB-MEN what has been run for CINeMA: a duplicate-assessment study in which independent assessors label the same networks, powered to estimate agreement separately for the within-study component, the across-study component and the overall label, and deliberately sampling networks that do and do not reach 10 studies per comparison. Report agreement as a function of network size, connectivity and share of indirect evidence rather than as a single coefficient, since the CINeMA result shows the answer is not constant across networks. Instrument both applications to emit their own sensitivity: given an assessor’s inputs, each can recompute the label over a grid of substantial-contribution thresholds and suspected bias directions and report which labels are stable, at essentially no extra computational cost. Where agreement is poor because assessors differ, the response is calibration exercises and worked training examples before application, as recommended after the ROBINS-I study; where it is poor because the input is unavailable, the honest output is an explicit no-information label rather than a judgment.

Verification

Verdict. Partially addressed. One reader proposed this from the ROB-MEN guidance paper’s own statement that reproducibility is unstudied, and the reviewer accepted it as distinct from developing a population-adjustment credibility instrument or validating numeric diagnostics. It is not confirmed-open because a 13-assessor study has since measured CINeMA’s interrater reliability; the unmeasured half is ROB-MEN, and the CINeMA result itself rests on four networks.

Revised by the full-text reading. Applied full-text reading.

prior-work-added
added 2 citations and widened the statement to the current instrument landscape Surfaced while assessing a labeller-reported absence, ‘no validated instrument appraises NMA conduct’, which was rejected as resolved by the RoB NMA tool. The residue of that candidate, that reliability is reported for a minority of instruments, is this entry’s problem, so the tool and the instrument map belong here as prior work. Previously: “CINeMA and ROB-MEN attach graded labels to individual network estimates, confidence in the result and risk of bias due to missing evidence respectively, and those labels are carried into guidelines and appraisals as if they were properties of the evidence. The ROB-MEN guidance states that the reproducibility of CINeMA and ROB-MEN assessments has not been studied and that interrater agreement work would be informative. For CINeMA that has since been partly answered: 13 assessors applying both CINeMA and GRADE to four networks of differing size and complexity produced interrater agreement, measu”

Source

Derived from reading:nma-14 of the reviewed corpus.