Multivariate synthesis needs within-study correlations that publications do not report
Statement
Multivariate meta-analysis and multivariate network meta-analysis take each study’s within-study covariance matrix as a known input. The diagonal is recoverable from published standard errors; the off-diagonal entries, which describe the covariance between effects on different outcomes or at different assessment times, are never printed, and the standard treatment is to import them from an external source or to impose a structure. Two devices remove the requirement in two dimensions: an alternative random-effects model replaces the within-study and between-study correlations with a single overall synthesis correlation and needs only the inputs of two separate univariate analyses, and a robust variance estimator repairs its coverage where the number of studies is large, around 50 or more. Neither extends to three or more outcomes, where the analogous reduction is a whole positive-definite matrix rather than a scalar, and applications beyond two or three outcomes remain rare partly for that reason. The repeated-measures case is worse: within-arm correlations across time points are almost never reported, so time-course model-based network meta-analysis is run either with the correlations set to zero or under an assumed first-order autoregressive structure estimated from the aggregate summaries, and importing an individual-level correlation from a trial that has individual data risks ecological bias with nothing available to check it; the variance-rescaling device that stands in for temporal correlation is derived for continuous outcomes only, and binomial and count outcomes have no counterpart. What does not exist is an inference procedure valid for three or more correlated outcomes or time points when nothing is known about the within-study correlations, a bound over the correlation matrices compatible with what was reported, or a diagnostic saying how far an estimate rests on the imported value. The error lands where it is least visible, since under a misspecified within-study correlation the damage falls on the between-study variance estimate rather than on the pooled effect.
Why it is open
The off-diagonal entries are functionals of the within-patient joint distribution of the outcomes, which a marginal publication does not contain, so the quantity is not an estimation target at all; no estimator recovers it, and the only routes are to impose structure, to import it, or to bound it. The bivariate workaround succeeds because one scalar can absorb both the within-study and the between-study correlation and still leave a model identified from univariate inputs; in higher dimensions the same argument requires a whole correlation matrix whose entries are coupled by the positive-definiteness constraint, so they cannot be profiled or varied one at a time the way a single correlation can, and the property that made the device attractive, that it needs only what a univariate analysis needs, is lost. Individual participant data supply the correlations but at the individual level, and correlation between summary measures need not equal correlation between patient-level outcomes, so borrowing them across trials substitutes an unverifiable ecological assumption for the missing input. The consequence surfaces in the between-study variance and in interval width rather than in the point estimate, so an analysis fitted under the wrong correlation looks like one fitted under the right one and nothing in the output flags the imported value. Multivariate synthesis is also promoted for the precision it gains, so a method that widened intervals to cover correlation uncertainty would remove part of the reason to fit it, and the incentive runs against the fix.
What has been tried
| Work | What it contributes |
|---|---|
| Riley, Thompson and Abrams 2008, An alternative model for bivariate random-effects meta-analysis when the within-study correlations are unknown, Biostatistics 9(1):172-186 | Replaces the within-study and between-study correlations with one overall synthesis correlation so that the model needs only the data required for two separate univariate analyses; the standard workaround, and it is defined for two endpoints. |
| Hong, Riley and Chen 2018, An improved method for bivariate meta-analysis when within-study correlations are unknown, Research Synthesis Methods 9(1):73-88 | Shows that the standard variance estimator for that alternative model is not correct under many important settings, so coverage of a function of the pooled estimates can miss its nominal level even with many studies, and supplies a robust replacement recommended at roughly 50 studies or more. |
| Wei and Higgins 2013, Estimating within-study covariances in multivariate meta-analysis with multiple outcomes, Statistics in Medicine 32(7):1191-1205 | Approximates the within-study covariances from information about likely correlations between the underlying outcomes, for continuous, dichotomous and mixed data; the imputation route, whose own inputs are assumed rather than observed. |
| Riley, Price, Jackson, Wardle, Gueyffier, Wang, Staessen and White 2015, Multivariate meta-analysis using individual participant data, Research Synthesis Methods 6(2):157-174 | Derives the within-study correlations directly by joint regression and bootstrapping when individual participant data are available; the only route that estimates rather than assumes them, and it needs exactly the data publications do not supply. |
| Jackson, Bujkiewicz, Law, Riley and White 2018, A matrix-based method of moments for fitting multivariate network meta-analysis models with multiple outcomes and random inconsistency effects, Biometrics 74(2):548-556 | Extends network meta-analysis to multiple outcomes while continuing to treat the within-study covariance matrices as fixed and known, and states that the entries describing covariance between treatment effects for different outcomes are harder to obtain in practice and are handled by imported strategies. |
| Noma, Nagashima and Furukawa 2020, Permutation inference methods for multivariate meta-analysis, Biometrics 76(1):337-347 | Gives permutation-based joint and marginal confidence regions that hold without large-sample approximations, under the standard model in which the within-study correlations are all known, and names accurate methods for the case where there is no information about them as future work. |
| Riley, Jackson, Salanti, Burke, Price, Kirkham and White 2017, Multivariate and network meta-analysis of multiple outcomes and multiple treatments: rationale, concepts, and examples, BMJ 358:j3932 | Records that convergence and estimation problems grow with the number of outcomes, that applications beyond two or three outcomes are rare, and that without individual participant data there are problems obtaining and estimating the correlations among outcomes. |
| Pedder, Dias, Bennetts, Boucher and Welton 2019, Modelling time-course relationships with multiple treatments: model-based network meta-analysis for continuous summary outcomes, Research Synthesis Methods 10(2):267-286 | Sets out the options for unknown within-arm correlations across time points, either importing an individual-level correlation and assuming it holds at the aggregate level and in other trials, which it notes may lead to ecological bias, or estimating a constrained structure such as compound symmetry from the aggregate summaries. |
| Strathe, Bøg and Gorst-Rasmussen 2026, Model-based network meta-analysis: joint estimation of dose-response and time-course relationships, Research Synthesis Methods | States that within-arm standard errors can be extracted or digitized from publications whereas within-arm correlations seldom are reported, runs the multivariate likelihood with the correlations set to zero or under an assumed first-order autoregressive structure, and checks the consequence only in the one illustrative dataset where individual data happened to be available. |
| Heinecke, Tallarita and De Iorio 2020, Bayesian splines versus fractional polynomials in network meta-analysis, BMC Medical Research Methodology 20:261 | Rescales the outcome variances to stand in for correlation between time points in continuous outcomes and records that other response types need other rescaling strategies, naming multivariate binomial and multivariate Poisson likelihoods as the devices that would be required and not building them. |
| Du 2026, A comparison of multivariate and univariate meta-analysis, Behavior Research Methods 58(7) | Two simulation studies varying the number of outcomes, the number of studies, heterogeneity, the true within-study and between-study correlations and the specified within-study correlation, reporting that misspecification hurts the between-study variance estimates more than the overall effect sizes and that univariate analysis outperformed multivariate in more conditions than the reverse. |
| Hattle, Burke, Trikalinos, Schmid, Chen, Jackson and Riley 2022, Multivariate meta-analysis of multiple outcomes: characteristics and predictors of borrowing of strength from Cochrane reviews, Systematic Reviews 11:149 | Across 43 Cochrane intervention reviews, identifies the largest absolute within-study correlation as one of four characteristics predicting how much strength a bivariate model borrows, which is direct evidence that the assumed value governs how much the model does. |
Probable solution or research direction
Take the two-outcome result as the template and ask what survives in higher dimensions: characterize the class of models over three or more outcomes that stay identified from marginal univariate inputs alone, and establish whether a structured synthesis correlation, exchangeable across outcome pairs or first-order autoregressive across time points, keeps the pooled effects and their intervals valid when the true matrix is something else. Where identification fails, bound rather than assume: the set of positive-definite matrices compatible with the reported margins is a convex constraint, so report the interval obtained by optimizing the estimate over that set, which converts a silent assumption into a stated worst case. Supply the diagnostic that is currently missing, namely the derivative of the pooled estimate and of the between-study variance with respect to each imported correlation, so that a reader can see which entries the answer rests on and which are inert. Test the ecological step directly in the datasets that carry both individual and aggregate data by computing the summary-level and the individual-level correlations in the same trials and reporting how far apart they are, then use that distribution as the prior for the imported value rather than a single number. Build the missing likelihoods for non-continuous repeated measures, since the continuous variance-rescaling device has no binomial or count counterpart and the multivariate binomial and Poisson forms named as the alternative have not been implemented for network synthesis.
Verification
Verdict. Partially addressed. This came from labeller-reported absence rather than from a paper arguing for it: 3 of 12 gap labellers named it independently while working on other gaps, and the eight future-research sentences behind it span eight distinct papers from 2017 to 2026. Registry search confirmed the labellers’ own note that the existing covariance entries concern other sources of dependence: HET-02 and CMP-12 are the shared-arm and shared-weighting covariance of a multi-arm trial, HET-11 is the covariance induced by overlapping reported subgroups, and HET-09 is the between-study and inconsistency covariance rather than the within-study one. The cross-outcome and cross-timepoint within-study correlation appears in DIS-17, OUT-19 and OUT-23 only as an input to something else and is the subject of no entry. Two of the eight sentences, Yoon et al. 2021 and Spineli and Kalyvas 2020, concern a different dependence, namely between an effect estimate and its own estimated variance, and did not support this entry. The verdict is partially-addressed because the bivariate case has been solved twice over, by the alternative synthesis-correlation model and by its robust variance estimator, the covariances can be imputed from assumed outcome correlations or computed exactly from individual data, and the consequences of misspecification have now been simulated; what is missing is any of this for three or more outcomes or time points, a bound over the compatible matrices, and a diagnostic.
Revised by the full-text reading. Applied 2026-07-26 full-text reading of the consolidated library (687 papers).
- Citation corrected
- normalized a citation year: ‘Riley, Thompson and Abrams 2007, An alternative model for bivariate random-effects meta-analysis when the within-study correlations are unknown, Biostatistics 9(1):172-186’ -> ‘Riley, Thompson and Abrams 2008, An alternative model for bivariate random-effects meta-analysis when the within-study correlations are unknown, Biostatistics 9(1):172-186’ 10.1093/biostatistics/kxm023 is cited with two different years across the registry. CrossRef gives 2007 online and 2008 in print; the issue year is used throughout.
- Citation corrected
- normalized a citation year: ‘Noma, Nagashima and Furukawa 2019, Permutation inference methods for multivariate meta-analysis, Biometrics 76(1):337-347’ -> ‘Noma, Nagashima and Furukawa 2020, Permutation inference methods for multivariate meta-analysis, Biometrics 76(1):337-347’ 10.1111/biom.13134 is cited with two different years across the registry. CrossRef gives 2019 online and 2020 in print; the issue year is used throughout.
Source
Derived from reading:uncovered of the reviewed corpus.