No criterion or error-based evidence selects arm-synthesis over contrast-synthesis
Statement
Two model families synthesize a network of trials. Contrast-synthesis holds the study intercepts fixed, so relative effects are estimated from within-study differences alone; arm-synthesis makes them random, which recovers between-study information and lets the model return absolute as well as relative effects. The conditions under which the extra between-study information biases a treatment effect are established theoretically, including a constructed dataset in which a true odds ratio of 1 is estimated as 1.35, but nothing establishes whether those conditions arise in real networks or how large the error is when they do. The one large empirical comparison re-analyzed 118 published networks with binary outcomes under three contrast-synthesis and two arm-synthesis models: against the Bayesian contrast-synthesis reference, arm-synthesis with a common variance returned odds ratios a factor 0.69 (95% CI 0.51 to 0.95) apart and standard errors a factor 0.69 (0.68 to 0.71) apart, while arm-synthesis with treatment-specific variances returned 0.47 (0.37 to 0.61) and standard errors 1.64 times those of the common-variance arm model, with regression-based limits of agreement at the midpoint of the log odds ratios running from minus 0.7 to 0.4 and from minus 1.3 to 0.8; treatment ranks moved with them, and 28 of the 118 arm-synthesis fits either produced no estimate or failed convergence diagnostics. An agreement study of real networks can measure disagreement but never error, and no simulation has scored bias, coverage, interval width and rank recovery for the two families under realistic network conditions, so nothing ranks them by accuracy. There is consequently no rule an analyst can apply before fitting: the standing advice is to prespecify a model and justify it, or to run both as complementary sensitivity analyses. Two subsidiary questions are equally unsettled: whether adding a fixed study main effect makes arm-synthesis equivalent to contrast-synthesis by removing the recovery of between-study information, which has been proposed but never compared; and whether the random-effect variance should differ by treatment or by contrast, which was introduced as a stated belief and left undecided by a deviance information criterion of 298.4 against 298.6 in the worked example that proposed it.
Why it is open
An empirical comparison of published networks is the only design that keeps realism, and it can never say which model was right, because the truth is unobserved; only simulation can rank the families by error, and a simulation has to fix the exact quantity in dispute, namely whether study intercepts are associated with treatment effects, so its verdict is written into its data-generating mechanism before a single replicate runs. The comparison is also not purely about bias, because the two families target different estimands: contrast-synthesis returns a study-specific conditional effect while arm-synthesis returns a population-averaged marginal one, so a scoring exercise must first choose which target it is scoring against, and choosing determines much of the answer. Within arm-synthesis the data cannot separate the competing variance structures either, which is why the choice is defended by argument rather than by fit. Worse, the regimes where the choice matters most are the regimes the evidence excludes: empirical comparisons screen out networks failing an inconsistency test because the models being fitted do not represent inconsistency, and sparse networks drop out through convergence failure of the arm-synthesis fits, so the available agreement statistics are computed on the subset where the two families were always going to agree.
What has been tried
| Work | What it contributes |
|---|---|
| Karahalios, White, Turner, Salanti, Herbison, Veroniki, Nikolakopoulou and McKenzie 2025, Research Synthesis Methods | The largest empirical comparison: 118 published binary-outcome networks re-analyzed under three contrast-synthesis and two arm-synthesis models, reporting ratios of odds ratios and of standard errors, Bland-Altman limits of agreement, rank and SUCRA differences, and 28 arm-synthesis fits that failed to estimate or converge; states in its own future-research section that simulation evidence, a test of the fixed-study-effect equivalence and an assessment of the effect of model choice on inconsistency are all still needed. |
| White, Turner, Karahalios and Salanti 2019, Statistics in Medicine | Derives the four differences between the families, shows that random study intercepts admit between-study information that can cause important bias, gives the constructed example in which a true odds ratio of 1 is estimated as 1.35, and concludes that both families are suitable but that random study intercepts require a strong rationale; supplies the theory a criterion would have to operationalize. |
| Karahalios, McKenzie and White 2022, Methods in Molecular Biology volume 2345 | Sets out both models, their estimands, symmetry and use of between-study information, reviews the simulation and empirical evidence available, and records that this evidence is limited, that it is unknown whether the extra between-study information is a problem in practice, and that simulation studies under realistic scenarios crossing network geometry, sparsity and outcome type are needed. |
| Dias and Ades 2016, Research Synthesis Methods | The case against arm-synthesis: pooling arm-level summaries breaks the within-trial randomized comparison, and an underlying risk needed for absolute effects is better taken from an external source than from the network; one of the two positions the field has polarized into. |
| Hong, Chu, Zhang and Carlin 2016, Research Synthesis Methods | Casts arm-synthesis as a Bayesian missing-data problem in which absent arms are imputed under exchangeability of absolute effects; the formulation whose weaker missingness assumption is the main argument in its favor. |
| Zhang, Carlin, Neaton, Soon, Nie, Kane, Virnig and Chu 2014, Clinical Trials | The arm-synthesis formulation with a correlated multivariate random-effects vector across treatments, from which absolute risks and marginal contrasts are reported; the model implemented in the software applied studies use. |
| Piepho, Madden and Williams 2024, Research Synthesis Methods | Argues that recovery of between-study information in arm-synthesis is prevented by fitting a fixed study main effect, so the principle of concurrent control can be honored inside an arm-based analysis; the proposed reconciliation that no study has yet compared against either family. |
| Piepho and Madden 2022, Research Synthesis Methods | Implements the fixed-study-main-effect device in standard generalized linear mixed model procedures, establishing that the reconciliation is routinely fittable and therefore that the missing item is evidence rather than software. |
| Chu, Lin, Wang, Wang, Chen and Cappelleri 2024, WIREs Computational Statistics | Review of the assumptions, advantages and limitations of the two families for binary outcomes; declines to supply a selection rule and instead recommends fitting both as complementary sensitivity analyses, which is the current state of guidance. |
| Hawkins, Scott and Woods 2016, Research Synthesis Methods | Introduces the arm-based parameterization with treatment-specific rather than contrast-specific random-effect variances, states as a belief that variation by treatment is the more realistic assumption for many applied studies, and reports deviance information criteria of 298.4 and 298.6 for the two structures, which cannot separate them. |
| Lin, Zhang, Hodges and Chu 2017, Journal of Statistical Software | The pcnetmeta implementation, the only R package used to fit arm-synthesis in the empirical comparison; settles the dispute over exchangeability of absolute rather than relative effects by citing that both assumptions are accepted in the literature, which is the position a criterion would have to replace. |
| Lin, Chu and Hodges 2016, Epidemiology | Earlier empirical comparison on 14 networks using treatment exclusion as the perturbation, finding arm-synthesis estimates more stable on average but with large changes in some networks, and recommending that analysts consider both models; the precedent for measuring disagreement rather than error. |
| Li, Shih, Song and Tu 2023, Research Synthesis Methods | Simulates how bias in one direct comparison propagates through the network under contrast-based, baseline and arm-based models, showing the propagation depends on geometry; the closest existing error-based comparison, scored on how an injected bias spreads rather than on which family recovers the truth. |
| Luney et al. 2026, BMJ | A 158-trial, 52-intervention network meta-analysis of drugs to prevent postoperative delirium fitted with a Bayesian arm-based model; establishes that the family whose operating characteristics are unmeasured is carrying headline results in general medical journals rather than sitting in the methods literature. |
Probable solution or research direction
Settle the analytic question first, because it is cheap and it partitions the rest: if fitting a fixed study main effect does remove the recovery of between-study information, then the disagreement measured across the 118 networks is attributable to the random intercepts alone, and what remains is an estimand choice rather than a bias problem. Then run the simulation the field has been asking for since 2022, crossing the presence and strength of association between study intercepts and treatment effects, the degree of inconsistency, the number of studies per treatment, event rarity and network geometry, and scoring bias, coverage, interval width and rank recovery separately against a conditional and a marginal target so that the two families are each judged on the estimand they claim. Report non-convergence as a performance measure rather than an exclusion, since a family that fails on roughly one network in four is telling an analyst something the surviving fits cannot. Convert the output into a rule that can be applied before fitting, stated over network features an analyst can read off the data, and ship it as a default check in the packages that fit these models, because a criterion published as a table will not be consulted.
Verification
Verdict. Confirmed open. This came from an absence reported by 2 of 12 gap labelers at the end of their labeling rather than from a paper arguing for it, so it rests on weaker provenance than its neighbors. The evidence spans five papers, of which two carry it, the 2025 empirical comparison and the 2022 chapter, both of which state the missing item in their own words; the other three are secondary, being a software paper that resolves the exchangeability dispute by citation, an applied study whose complaint is about inconsistency detection under both frameworks and belongs to the inconsistency entries rather than here, and a parameterization paper that states a belief about the variance structure. The registry was searched for arm-based and contrast-based content and the near neighbors were read in full: HET-18 covers the between-treatment correlation matrix inside arm-based models, HET-16 the comparison-specific heterogeneity variances under the second-order consistency constraint, EVB-07 the missing-data assumption on trial design that separates the two likelihoods, and DIA-22 the exclusion of inconsistent networks from the empirical comparison; none of them states that no criterion and no error-based evidence selects between the families. PubMed was searched to July 2026 for a simulation comparison of arm-synthesis and contrast-synthesis and none has appeared, and the most recent review declines to give a rule and recommends fitting both.
Source
Derived from reading:uncovered of the reviewed corpus.