No calibrated small-study interval method for population-adjusted NMA
Coverage fails in two ways. Meta-analyzing per-trial adjusted contrasts that reuse one individual-data trial as if independent lost coverage as trials were added (down to 0.55 at twelve trials with DerSimonian-Laird), because the shared error does not average out. Pooling the aggregate trials first and subtracting the transported contrast once fixed that; with DerSimonian-Laird it stayed within 0.93 to 0.97 from four trials, and Hartung-Knapp was conservative (up to 1.00) at two or three. Read the study below.
Statement
Random-effects inference in multivariate and contrast-based network meta-analysis rests on a large-sample approximation in the number of studies, and it fails when that number is small: across 24 scenarios with 8, 12 and 16 studies, coverage of extended DerSimonian-Laird, maximum likelihood and restricted maximum likelihood confidence regions fell materially below the nominal 95 percent, worst for joint inference and, among marginal inferences, at small study counts with large heterogeneity. Exact permutation regions repair this for aggregate-data multivariate and contrast-based models. The repair does not carry to population-adjusted synthesis: a network fed by matching-adjusted or simulated treatment comparison estimates, or fitted as multilevel network meta-regression, combines estimated weights or a fitted outcome surface with a handful of studies, so the exchangeability under permutation of study labels that the exact method needs does not hold, and weights estimated from one individual-patient dataset supply several contrasts at once. Variance estimation for a single matching-adjusted comparison has been benchmarked across 108 scenarios, with sample size, population overlap and outcome type identified as what decides whether a given estimator is adequate, but that is a within-comparison question and says nothing about the interval for a network estimate assembled from few such comparisons under heterogeneity. No published work reports the realized coverage of population-adjusted network intervals as a function of the number of studies, and no exact, permutation-based or small-sample-calibrated alternative has been derived for that setting. Population-adjusted networks are where study counts are smallest, commonly two to a dozen trials, so this is the regime the methods are used in rather than an edge case.
Why it is open
The permutation argument needs a null under which study-level contributions are exchangeable, and a two-stage weighting workflow breaks it structurally, because the weights are estimated from one individual-patient dataset that supplies several contrasts, so the studies are not independent units and permuting their labels does not leave the null distribution unchanged. A one-stage mixed-granularity likelihood does not obviously possess the symmetry either, since individual-level and aggregate-level studies enter through different terms and are not interchangeable. Bayesian population-adjusted intervals do not escape by construction, because with a handful of studies the heterogeneity variance is determined by its prior, so the interval inherits a choice rather than the data. The natural fallbacks from pairwise meta-analysis, the Hartung-Knapp adjustment and its relatives, have no derivation for a network whose inputs carry estimated weights, and importing them would produce an interval of unknown coverage in exactly the situation where coverage is the question. Measuring the problem is cheap and nobody has done it, which is the more awkward half: the realized coverage of population-adjusted network intervals as a function of the number of studies is a simulation, not a theorem.
What has been tried
| Work | What it contributes |
|---|---|
| Noma, Nagashima and Furukawa 2020, Biometrics | Demonstrates serious undercoverage of extended DerSimonian-Laird, maximum likelihood and restricted maximum likelihood confidence regions across 24 scenarios at 8, 12 and 16 studies, and supplies exact permutation-based joint and marginal inference for multivariate meta-analysis and contrast-based network meta-analysis; the remedy that does not transfer to estimated-weight settings. |
| Chandler and Proskorovsky 2024, Research Synthesis Methods | Monte Carlo comparison of variance estimators across 108 anchored and unanchored matching-adjusted indirect comparison scenarios, covering conventional estimators with raw and with effective-sample-size weights, sandwich estimators with and without finite-sample adjustment, and bootstrap variants, and finding that raw-weight conventional estimators understate variability under poor overlap while bootstrapping is unstable at small sample sizes; settles the single-comparison part of the problem and leaves the few-studies network part untouched. |
| IntHout, Ioannidis and Borm 2014, BMC Medical Research Methodology | Shows the Hartung-Knapp-Sidik-Jonkman method substantially outperforms DerSimonian-Laird for random-effects meta-analysis with few studies; the small-study interval fix that has no population-adjusted network analogue. |
| Noma, Nagashima, Maruo, Gosho and Furukawa 2018, Statistics in Medicine | Bartlett-type corrections and bootstrap adjustments to likelihood-based inference for network meta-analysis; the other small-sample route for aggregate-data networks, likewise derived without estimated weights or a mixed-granularity likelihood. |
| Higgins and Thompson 2004, Statistics in Medicine | Permutation approach to meta-regression to control spurious findings from few studies; the origin of the exchangeability argument that population adjustment breaks. |
| Phillippo et al. 2020, Journal of the Royal Statistical Society Series A | Multilevel network meta-regression, the one-stage mixed-granularity likelihood in which individual and aggregate studies enter through different terms; its small-sample interval behavior as a function of the number of studies has not been studied. |
| Remiro-Azocar, Heath and Baio 2021, Research Synthesis Methods | Review and simulation study of population adjustment with limited individual patient data, reporting bias and precision for the competing estimators; does not report coverage as a function of the number of studies in a network, which is the quantity at issue. |
| Remiro-Azocar 2022, BMC Medical Research Methodology | Two-stage matching-adjusted indirect comparison, which separates weight estimation from outcome modeling; the workflow whose estimated weights and reused individual-patient data destroy the permutation symmetry the exact method needs. |
| Phillippo, Ades, Dias, Palmer, Abrams and Welton 2018, Medical Decision Making | Methods guidance for population-adjusted indirect comparisons in health technology appraisal; establishes that these comparisons are routinely built from very few studies, which is the regime in which the asymptotic interval is never justified. |
Probable solution or research direction
Measure first: simulate population-adjusted networks over the number of studies from two upward, the number of individual-patient studies, population overlap and heterogeneity, and report realized coverage and interval width for the intervals actually used, namely sandwich and bootstrap matching-adjusted contrasts combined in a network and credible intervals from multilevel network meta-regression under standard heterogeneity priors. Then derive the calibration: the promising route is a bootstrap that resamples at the level of the study and re-estimates the weights inside each replicate, giving a null that respects the dependence a naive permutation destroys, together with a Hartung-Knapp-style variance inflation whose degrees of freedom account for the estimated weights. Establish whether any permutation scheme is exact for the one-stage mixed-granularity likelihood, because if none is, the field should stop waiting for one and calibrate approximations instead. Report everything by number of studies rather than averaged over it, since the small-count regime is the point of the exercise, and state a minimum study count below which no population-adjusted network interval should be presented as calibrated.
Our study
Interval coverage for population-adjusted synthesis with few studies
The question put to it. How does interval coverage of population-adjusted random-effects synthesis depend on the number of studies, when one individual-data trial supplies the adjusted contrast for several aggregate trials?
What it found. Coverage fails in two ways. Meta-analyzing per-trial adjusted contrasts that reuse one individual-data trial as if independent lost coverage as trials were added (down to 0.55 at twelve trials with DerSimonian-Laird), because the shared error does not average out. Pooling the aggregate trials first and subtracting the transported contrast once fixed that; with DerSimonian-Laird it stayed within 0.93 to 0.97 from four trials, and Hartung-Knapp was conservative (up to 1.00) at two or three.
Results.
- Refuting sentence fails: coverage at two to four trials was 0.774 to 0.888 for the per-trial analysis and 0.909 to 0.962 for the two-step DerSimonian-Laird analysis.
- Minimum trial count with coverage 0.93 to 0.97 at every heterogeneity level: 4 for two-step DerSimonian-Laird; none for the per-trial analyses or two-step Hartung-Knapp.
- Per-trial analysis coverage fell with the number of trials even without heterogeneity.
Design.
- Estimand
- B versus A mean difference in the target population
- Methods compared
- per-trial adjusted contrasts, DerSimonian-Laird, per-trial adjusted contrasts, Hartung-Knapp, two-step, DerSimonian-Laird, two-step, Hartung-Knapp
- Data-generating mechanism
- One individual-data trial transported by MAIC and 2 to 12 aggregate trials with heterogeneity; naive and two-step random-effects analyses with DerSimonian-Laird or Hartung-Knapp intervals. 18 cells, 2000 replicates each.
- Replicates
- 2000 per scenario, 36,000 in total
What this does not answer. Pairwise synthesis with one individual-data trial; continuous outcome; no one-stage or Bayesian analysis; one overlap level. Peer review has not been done.
Peer review. Not yet done. The result stands on the study alone until it is.
Read it: PDF · Markdown · OpenDocument · Code and data: studies/HET-10-a-minimum-study-count-nobody-has-measured · Protocol: registered before the run
Verification
Verdict. Partially addressed. One reader proposed this and the reviewer accepted it as a distinct interval-calibration problem rather than a heterogeneity-parameterization or omitted-uncertainty-source problem, while noting that the cited evidence establishes undercoverage for conventional networks rather than the absence of all relevant population-adjusted methods. It is partially addressed because the aggregate-data half has both a demonstration and an exact remedy and single-comparison matching-adjusted variance estimation has been benchmarked across 108 scenarios; the intersection, a calibrated interval for a population-adjusted network with few studies, has neither a derivation nor a coverage measurement.
Source
Derived from reading:nma-29 of the reviewed corpus.