No criterion selects among design-aware models for a mixed-design network
Statement
Reviews and protocols that admit non-randomized studies of interventions into a randomized network enumerate the same alternatives to naive pooling: design-adjusted synthesis, non-randomized evidence as informative or power priors, three-level or design-specific hierarchical models, bias-adjusted models, and stratified or sensitivity analysis. All of them are implemented, and crossnma provides four cross-design approaches behind one interface, so availability is not the obstacle. What does not exist is a rule that says which to fit, or evidence that ranks them on anything a decision reads. Where they have been run side by side the point estimates barely separate while the reported uncertainty does: in a type 2 diabetes network of sodium-glucose co-transporter 2 inhibitors and glucagon-like peptide-1 receptor agonists the mean glycated hemoglobin differences at 24 and 52 weeks were similar under every method, naive pooling narrowed the interval relative to randomized evidence alone, a hierarchical model separating designs widened it, and bias-adjustment models moved it in different directions for different treatments. Fit does not arbitrate either: in that network hierarchical and bias-adjustment models all fitted better than naive pooling and nothing in the fit ordered them against each other. Each family also needs a bias magnitude it cannot estimate, since the power-prior exponent runs from 0 to 1 with no data-based value and the mixture-model probability of bias requires informative priors elicited from trained risk-of-bias assessors, with the identifiability of the bias parameters depending on which data happen to be present. The analyst therefore picks the model and, with it, the width of the interval a committee reads.
Why it is open
Design-level bias is a study-level offset that enters the same observed dispersion as between-study heterogeneity and the treatment effect, so a mixed network does not separate the three; every family assumes rather than estimates the quantity on which they differ, and data cannot rank assumptions they never see. Fit is the natural tiebreaker and fails structurally, because the families carry similar numbers of free parameters and differ in a shape assumption about bias that a handful of non-randomized studies per contrast cannot resolve, which is why fit separates naive pooling from the rest and then stops. The empirical route that would calibrate the magnitude has been run and returned a near-null average: across 39 systematic reviews and 8 overviews covering 2869 randomized trials and 3924 observational studies the pooled ratio of ratios was 1.08 with a 95 percent confidence interval of 1.01 to 1.15, rated low certainty, so there is no corpus-level design bias worth plugging in even though individual discrepancies are large. Scoring a criterion needs a randomized answer for the same contrast as ground truth, and that exists only where the non-randomized evidence was not needed, so the validation set systematically excludes the cases the criterion is for. Nothing forces a declaration: the choice is made once, reported as the analysis, and naming it as a choice could only make the result look more fragile than it currently does.
What has been tried
| Work | What it contributes |
|---|---|
| Efthimiou, Mavridis, Debray, Samara, Belger, Siontis, Leucht and Salanti 2017, Combining randomized and non-randomized evidence in network meta-analysis, Statistics in Medicine 36(8):1210-1226 | Sets out and compares naive data synthesis, design-adjusted synthesis, non-randomized evidence as prior information and three-level hierarchical models, discussing the advantages and limitations of each in two clinical examples; it supplies the enumeration that later work inherits but no selection rule and no operating characteristics. |
| Schmitz, Adams and Walsh 2013, Incorporating data from various trial designs into a mixed treatment comparison model, Statistics in Medicine 32(17):2935-2949 | Proposes naive pooling, observational data as prior information, and a three-level hierarchical model as the three routes for mixing designs in one network, and notes that the hierarchical form allows bias adjustment while also accounting for between-design heterogeneity. |
| Hussein, Abrams, Gray, Anwer, Dias and Bujkiewicz 2023, Hierarchical network meta-analysis models for synthesis of evidence from randomised and non-randomised studies, BMC Medical Research Methodology 23:97 | Fits naive pooling, hierarchical, class-effect and bias-adjustment models to one type 2 diabetes network and reports that the effect estimates agree while the uncertainty moves differently under each, and that hierarchical and bias-adjustment models all fit better than naive pooling without the fit ordering them among themselves. |
| Hamza, Schwarzer and Salanti 2024, crossnma: An R package to synthesize cross-design evidence and cross-format data using network meta-analysis and network meta-regression, BMC Medical Research Methodology 24:169 | Ships four cross-design approaches behind one interface with default beta priors on the probability of bias keyed to risk-of-bias category and study type, and records that the identifiability of all model parameters, in particular those relating to bias, depends on the available data. |
| Hamza, Chalkou, Pellegrini, Kuhle, Benkert, Lorscheider, Zecca, Iglesias-Urrutia, Manca, Furukawa, Cipriani and Salanti 2023, Synthesizing cross-design evidence and cross-format data using network meta-regression, Research Synthesis Methods 14(2):283-300 | Develops the four approaches that crossnma implements and reports that the relative treatment effects did not change much once design and risk of bias were accounted for, in a multiple sclerosis network and in a network of 431 trials of 21 antidepressants; this is why point estimates cannot separate the model families. |
| Verde 2021, A bias-corrected meta-analysis model for combining studies of different types and quality, Biometrical Journal 63(2):406-422 | Mixture model in which the true effects are a mixture of a component of interest and a hidden bias structure; the source of the probability-of-bias parameter that the network versions require informative priors for. |
| Jenkins, Hussein, Martina, Dequen-O’Byrne, Abrams and Bujkiewicz 2021, Methods for the inclusion of real-world evidence in network meta-analysis, BMC Medical Research Methodology 21:207 | Runs the power-prior down-weighting exponent across its whole range in a relapsing remitting multiple sclerosis network, finds stable point estimates and credible intervals that widen as weight is given to real-world evidence, and concludes that empirical evidence on the biases of individual real-world studies and elicitation methods for their extent are both warranted. |
| Cameron, Fireman, Hutton, Clifford, Coyle, Wells, Dormuth, Platt and Toh 2015, Network meta-analysis incorporating randomized controlled trials and non-randomized comparative cohort studies for assessing the safety and effectiveness of medical treatments: challenges and opportunities, Systematic Reviews 4:147 | States that published research on the prior-based and design-level hierarchical routes is limited and that there is a lack of consensus on what degree of bias adjustment to apply to non-randomized studies. |
| Toews, Anglemyer, Nyirenda, Alsaid, Balduzzi, Grummich, Schwingshackl and Bero 2024, Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials: a meta-epidemiological study, Cochrane Database of Systematic Reviews 1:MR000034 | Pools 39 systematic reviews and 8 overviews covering 2869 randomized trials and 3924 observational studies and reports a ratio of ratios of 1.08 (95 percent confidence interval 1.01 to 1.15) at low certainty; the corpus-level anchor a design-bias prior would be calibrated against, and it is close to null. |
| Zhao, Song, Wang, Li and Kang 2026, Naive pooling of randomized and non-randomized evidence in network meta-analyses: a meta-epidemiological study protocol, Frontiers in Medicine 13:1868332 | Protocol to measure how often published networks published from 2020 to 2025 pool randomized and non-randomized evidence without any design-specific adjustment; it lists the design-aware alternatives as a menu and offers no criterion for choosing among them. |
Probable solution or research direction
Score the families rather than enumerate them again: simulate networks in which the design-level bias is known, crossing its size, its sign, whether it varies by treatment class, the number of non-randomized studies per contrast and their relative precision, and report coverage of the decision-relevant contrast, interval width, and how often the recommendation changes under each family, with naive pooling and randomized-only synthesis as the two reference behaviors. Score each family when it is wrong as well as when it is right, since the operational question is which degrades least under a bias structure it did not assume. Add an empirical arm over networks carrying both randomized and non-randomized evidence on the same contrast, using the randomized subnetwork as the reference and recording the discrepancy each family leaves; that is the only calibration data the problem admits, and it also yields a distribution of realized design bias by indication that could replace the near-null corpus average. Require the bias magnitude to carry a declared provenance in the output, stating whether it was elicited, taken from a corpus or set by convention, and report the value of the down-weighting exponent or bias prior at which the recommendation flips, which is a tipping point rather than a choice. Where nothing selects a family, report the envelope over all of them, since the spread across models is the quantity a committee actually needs.
Verification
Verdict. Partially addressed. This came from labeller-reported absence rather than from a paper arguing for it: 3 of 12 gap labellers named it independently while working on other gaps, and the eight future-research sentences behind it span eight distinct papers from 2013 to 2026. The absence as reported, that nothing models the bias when randomized and non-randomized studies enter one network, is false, and the entry has been narrowed to the claim the sources actually support. Two of the eight sentences, Dias et al. 2013 and Wei et al. 2015 on separating true between-trial variation from bias-induced variation, are already registered as QBA-30 and were dropped; a third, the call in Hussein et al. 2023 for a network extension of the mixture model with class-varying bias, has since been supplied by crossnma. Registry search found DEC-22, which rates confidence in a mixed-design network and explicitly sets the synthesis side aside as solved, DIS-07, which states the same calibration problem for external-control bridges across a disconnected gap, and DEC-24, which asks for a common scale for borrowing intensity rather than a rule for choosing a model; none states the selection problem. The verdict is partially-addressed because the model families, their implementations and risk-of-bias-keyed default priors exist and a corpus-level empirical anchor now exists in Toews et al. 2024; what is missing is a criterion with operating characteristics and any per-study calibration of the bias magnitude.
Source
Derived from reading:uncovered of the reviewed corpus.