No two-stage adjusted estimator carries multi-arm covariance
Ignoring the within-trial covariance errs in a direction set by the contrast, not by how the arms’ effect modification aligns. The trial’s own B-versus-C interval became conservative in every scenario, while the network A-versus-B interval, which borrows through the trial, fell below nominal in 15 of 30 scenarios. The shared-weighting part of the covariance took both signs but never reversed a contrast’s error. Dropping the trial widened the network interval by 29% to 91%. Read the study below.
Statement
When a multi-arm trial’s contrasts are population-adjusted one at a time, the within-study covariance induced by a shared arm and a shared weighting step is not recovered by naive variance formulas, so standard errors and network precision would be wrong. Current implementations avoid the error by refusing the capability rather than by solving it: both the component MAIC and component STC routes reject IPD studies with more than two arms, so multi-arm IPD trials must be split or dropped. The covariance is handled natively on the synthesis side of frequentist CNMA and in the one-stage ML-NMR likelihood, which keeps multi-arm trials inside a single arm-level model; what is missing is a two-stage adjusted estimator that carries it, and any multi-arm handling in component PAIC.
Why it is open
Population adjustment reweights or re-fits per contrast, and a two-stage route that processes one contrast at a time has no natural place to carry the induced correlation, so multi-arm studies must be split, which double counts the shared arm, or dropped, which discards evidence. A stratified bootstrap could recover the covariance only if replicates are carried jointly across a trial’s contrasts rather than per contrast, which requires multivariate estimating equations or an arm-level likelihood and therefore changes the two-stage architecture. Until then the practical cost is lost evidence rather than wrong intervals.
What has been tried
| Work | What it contributes |
|---|---|
| Phillippo et al. 2020 | handles multi-arm trials natively within a one-stage arm-level NMA likelihood, so the within-study covariance is never discarded |
| Petropoulou et al. 2023 | records that multi-arm study weights are adjusted in the netmeta CNMA machinery, showing the covariance loss is introduced by the adjustment stage rather than by component modeling |
| cpaic 0.1.0 (development version, choxos/cpaic@9d150e9) | unreleased development R package on GitHub, not on CRAN; re-estimates weights and outcome models in arm-stratified bootstrap replicates, fails closed when too few replicates succeed, and rejects IPD studies with more than two arms in both cmaic() and cstc() |
Probable solution or research direction
Use a multivariate bootstrap or stacked estimating equations that carry all of a trial’s contrasts jointly, so the two-stage route can accept multi-arm IPD studies with the correct correlation structure. Alternatively move to a one-stage arm-level component likelihood, where the full contrast covariance is produced by the model rather than reconstructed afterwards, which is how the non-component version already handles multi-arm trials. Either way the fix should lift the current two-arm restriction rather than adding a variance correction on top of it.
Our study
Splitting a multi-arm trial’s adjusted contrasts is conservative for one contrast and anticonservative for another
The question put to it. When a three-arm trial’s population-adjusted contrasts share an arm and a weight vector, what is lost by treating them as independent, is the error ever anticonservative, and is dropping the trial cheaper?
What it found. Ignoring the within-trial covariance errs in a direction set by the contrast, not by how the arms’ effect modification aligns. The trial’s own B-versus-C interval became conservative in every scenario, while the network A-versus-B interval, which borrows through the trial, fell below nominal in 15 of 30 scenarios. The shared-weighting part of the covariance took both signs but never reversed a contrast’s error. Dropping the trial widened the network interval by 29% to 91%.
Results.
- Split, trial-alone d_BC: coverage 0.957 to 1.000, above nominal beyond 3 MCSE in 29 of 30 scenarios and below in none; DESIGN.md’s alignment-sign prediction is withdrawn by its registered test.
- Split, network d_AB: coverage 0.4 to 3.7 points below the stacked sandwich, below nominal beyond 3 MCSE in 15 of 30 scenarios against 0 for the stacked sandwich.
- The shared-weighting covariance term was -22% to +15% of the Monte Carlo covariance, dominated by the shared-arm term.
- Joint sandwiches recovered 0.76 to 1.15 (stacked) and 0.73 to 1.12 (weights fixed) of the Monte Carlo covariance; stacked trial-alone intervals covered 0.919 to 0.960.
- Dropping the trial was unbiased and widened the network d_AB interval by a median 53%.
Design.
- Estimand
- d_AB, d_AC and d_BC in the target population
- Methods compared
- split (covariance zero), joint sandwich, weights fixed, stacked sandwich, drop the trial
- Data-generating mechanism
- Three-arm individual-data trial weighted by MAIC to a target, plus aggregate A-versus-C and B-versus-C trials in the target; 30 scenarios crossing effect-modification strength, its alignment across arms, target shift and shared-arm size; split, weights-fixed, stacked and drop analyses, network by generalized least squares.
- Replicates
- 1000 per scenario, 30,000 in total
What this does not answer. Continuous outcome, linear modification, one three-arm trial in a fixed two-trial aggregate network; four-arm trials, the one-stage ML-NMR comparator and a bootstrap are not run, and target moments are fixed. Peer review has not been done.
Peer review. Not yet done. The result stands on the study alone until it is.
Read it: PDF · Markdown · OpenDocument · Code and data: studies/CMP-12-multi-arm-covariance-two-stage · Protocol: registered before the run
Verification
Verdict. Partially addressed. Part of this problem is addressed by existing work; the entry names which part.
The source titled this. “Two-stage adjustment loses multi-arm within-study covariance” The title asserts the covariance is lost in practice, but the audit found both component routes reject multi-arm IPD studies outright, so it is a refused capability rather than a wrong-standard-error defect.
The source said. “When a multi-arm trial’s contrasts are population-adjusted one at a time in a two-stage workflow, the within-study covariance among those contrasts can be lost or misstated, so standard errors and network precision can be wrong. The same limitation appears as a hard restriction in the cSTC route, which accepts only two-arm IPD studies.”
What was wrong with it. The source implied that adjusted multi-arm contrasts are emitted with lost covariance. GPT-5.6 Sol read choxos/cpaic@9d150e9 and found both cmaic() and cstc() reject IPD studies with more than two arms, so this is a refused capability rather than a current wrong-standard-error defect; the literature auditor added that the one-stage fix is published and implemented for non-component networks (doi:10.1111/rssa.12579) and that netmeta’s CNMA machinery already adjusts multi-arm weights.
- Literature and prior-art check
- partially-solved / supported-with-caveat. I found no paper that recovers within-study covariance among multi-arm contrasts after two-stage population adjustment, so that specific gap is real. Two qualifications. First, the multi-arm covariance problem is already solved on the synthesis side of CNMA: Petropoulou et al. 2023 record that in the netmeta CNMA machinery ‘For multi-arm studies, the weights are assumed to be adjusted as described in [Rucker and Schwarzer]’, so the loss is introduced by the adjustment stage, not by component modelling. Second, the one-stage alternative avoids the problem entirely: ML-NMR keeps multi-arm trials inside a single arm-level likelihood, and Phillippo et al. 2020 give the limitation of the two-stage route as a motivation for it. So the practical need is met for non-component networks; what is open is a two-stage estimator that carries the covariance, and any multi-arm handling in a component PAIC. The claim that cSTC accepts only two-arm IPD studies is a code fact I cannot check. Cites doi:10.1111/rssa.12579: Handles multi-arm trials natively within a one-stage NMA likelihood, so the covariance is never discarded; does not solve the two-stage or component case..
- GPT-5.6 Sol, technical and source-code lens
- partially-solved / supported-with-caveat. C1, C2, and C6 are correct for a workflow that estimates several shared-arm contrasts separately and passes only their marginal variances onward. C3 and C4 are source-confirmed: cMAIC jointly re-estimates weights and its arm-only outcome model within arm-stratified bootstrap replicates, fails closed below the success threshold, and both cMAIC and cSTC reject IPD studies with more than two arms. Consequently, current cpaic does not silently emit multi-arm adjusted contrasts with incorrect covariance; it avoids the error by refusing that capability. Joint multi-arm adjustment remains an open feature, but this is a roadmap limitation rather than a current wrong-standard-error defect. C5 is an empirical prevalence claim outside the assigned lens and was not assessed. Cites choxos/cpaic@9d150e9: Propagates weight and outcome-model uncertainty for supported two-arm studies and refuses unsupported multi-arm IPD rather than discarding covariance silently.. Cites choxos/cpaic@9d150e9: Prevents cSTC from producing a falsely independent set of adjusted multi-arm contrasts..
- Decision path
-
R3:partially-solved(n=2) → R11:caveated(n=2)
Source
Derived from C:L1374, C:L1291-1293, C:L1373, scope:field-level (all two-stage PAIC); manifests in cpaic’s cMAIC and cSTC, scope:package-specific of the reviewed corpus.