Nothing propagates sampling error in reported target moments
Two things, established by different means. Analytically: the variance a standard interval omits is (1-2k)Var_T(tau)/n_T to first order, where k is how far the target trial’s own treatment effect is modified by the same covariates as the source trial’s, so the sign is set by a quantity the usual inputs do not identify; and a correction adding only the positive term is closer to the right variance than doing nothing only when k < 1/4. Empirically, and only in an exploratory subset: the omission runs from 9.5% too much variance to 14.9% too little and moves coverage between 92.1% and 96.2%. The registered analysis is uninformative and the confirmatory coverage question remains open. Read the study below.
Statement
Published aggregate-data means and standard deviations are estimates when they are sample summaries of a superpopulation, yet population-adjustment estimators condition on them as fixed and no study has quantified the coverage that costs. The weight-estimation half of the problem is settled: sandwich, ESS-rescaled, and bootstrap variance estimators exist, are implemented in maicplus, and were benchmarked across 108 scenarios by Chandler and Proskorovsky (2024, doi:10.1002/jrsm.1759), who found the fixed-weight conventional estimator materially anticonservative only under poor or moderate overlap and all methods valid under strong overlap. G-MAIC (doi:10.1017/rsm.2025.10021) propagates population-adjustment uncertainty by Bayesian bootstrap, and cmaic() in the unreleased cpaic development package on GitHub re-estimates weights inside a stratified bootstrap. None of these resamples the target moments.
Why it is open
Weights are functions of estimated parameters and of published target moments, and only the first part has been addressed. Sandwich, ESS-rescaled, bootstrap, and Bayesian-bootstrap estimators now cover weight estimation, and the correction matters mainly when overlap is poor or moderate. The published-moment part is untouched because a table of means and standard deviations arrives without the covariance among those moments or a sampling model for them, so there is nothing to resample and no nuisance equation to stack. The gap bites only when the estimand is a superpopulation quantity; where the supplied summaries define the target by construction, or are exact administrative counts, conditioning on them is the correct thing to do.
What has been tried
| Work | What it contributes |
|---|---|
| Chandler and Proskorovsky 2024, Uncertain about uncertainty in matching-adjusted indirect comparisons? A simulation study to compare methods for variance estimation, Res Synth Methods 15(6):1094-1110 | compares conventional estimators on raw weights, conventional estimators on ESS-rescaled weights, robust sandwich estimators, and bootstrapping across 108 scenarios, anchored and unanchored, for binary and time-to-event outcomes; finds variability underestimated by conventional estimators with raw weights when population overlap is poor or moderate, and all methods producing valid coverage probabilities and standard errors under strong overlap |
| Gao, Heath and Baio 2025, Regression augmented weighting adjustment for indirect comparisons in health decision modelling | G-MAIC employs a Bayesian bootstrap to propagate the uncertainty of population-adjusted estimates rather than conditioning on estimated weights |
| cpaic, unreleased development package on GitHub (not on CRAN), commit 9d150e9 | cmaic() runs a stratified-by-arm bootstrap that resamples the IPD, re-estimates the MAIC weights, and refits the outcome model, so weight-estimation uncertainty enters the interval; target means and standard deviations are held fixed throughout that bootstrap, and cstc() and cmlnmr() likewise condition on the supplied target summaries |
Probable solution or research direction
Do not rebuild weight-estimation variance; it exists, is implemented, and is benchmarked. The open work is a sampling model for the published target moments, either deriving their covariance from reported sample sizes and design or a Bayesian formulation that places a distribution over the target covariate law and integrates the treatment effect over posterior draws. With such a model, resampling the moments alongside the individual data, or stacking a moment-estimation equation into the estimating system, becomes well defined; without one, a bootstrap over target summaries is not identified. Then run the simulation the source proposed, restricted to the part nobody has done: how much coverage is lost by conditioning on target moments at realistic aggregate-data sample sizes.
Our study
What conditioning on sampled target moments costs
The question put to it. Population adjustment treats a target trial’s published covariate means and standard deviations as exact, though they are sample estimates. By how much does that cost interval coverage at realistic sample sizes, and what would a publication have to report for a correction to work?
What it found. Two things, established by different means. Analytically: the variance a standard interval omits is (1-2k)Var_T(tau)/n_T to first order, where k is how far the target trial’s own treatment effect is modified by the same covariates as the source trial’s, so the sign is set by a quantity the usual inputs do not identify; and a correction adding only the positive term is closer to the right variance than doing nothing only when k < 1/4. Empirically, and only in an exploratory subset: the omission runs from 9.5% too much variance to 14.9% too little and moves coverage between 92.1% and 96.2%. The registered analysis is uninformative and the confirmatory coverage question remains open.
Results.
- The prespecified analysis returned uninformative. Its gates require the reference interval and the negative controls to be nominal, and at poor overlap the Wald sandwich undercovers by up to 7 points for every method including scenarios with no effect modification at all, where the population omission is exactly zero. That is a source-variance failure, not a target-moment one.
- The sign of the omission is an analytic identity, not a simulation finding: (1-2k)Var_T(tau)/n_T. Regressing the measured omission on it across all 216 scenarios with effect modification gives a slope of 1.003 (SE 0.042). An earlier draft claimed the simulation located the cancellation point; peer review corrected that and the claim was retracted.
- The two corrections an analyst can deploy add only the positive term and omit the cross-covariance. They beat doing nothing only when k < 1/4, and are worse than doing nothing between 1/4 and 1/2. The design has no interior k with which to test this, so no recommendation is made for intermediate alignment. A draft claim that they help whenever k < 1/2 was wrong and was corrected in review.
- In the exploratory subset the status quo was the only method to leave 93% to 97% coverage, and it left in both directions. Magnitude tracks the ratio of source to target information, so a large source trial matched to a small target publication is the worst case, not a small target trial.
- Peer review left a standing recommendation of major revision. The confirmatory question needs a fresh registered run with an ex ante overlap restriction or a validated source variance estimator.
Design.
- Estimand
- Target-superpopulation marginal A versus B mean difference, exact in closed form
- Methods compared
- status quo (target moments held fixed), normal reconstruction (deployable today), reported moment covariance (enhanced reporting), full joint score (all target information)
- Data-generating mechanism
- Anchored MAIC on a continuous outcome, with the target trial’s participants generated and then discarded so the estimator sees only a published baseline table. 252 scenarios crossing source size, target size, overlap, effect-modification strength calibrated to a fixed target-population standard deviation, the alignment between the two trials’ effect modification, and whether the analyst’s borrowed correlation is correct.
- Replicates
- 5000 per scenario, 1,260,000 in total
What this does not answer. The registered coverage question is not confirmatorily answered: every usable coverage result comes from a post-run restriction conditioning on an observed outcome. One estimator, one outcome type, one information structure. Covariates are multivariate normal, which is also the assumption the deployable correction is derived under, so it is measured under conditions favorable to it; skewed, bounded and categorical covariates are untested. The outcome is continuous and the effect additive, so marginal and conditional effects coincide, and nothing transfers directly to binary or time-to-event outcomes, where MAIC is most used. Effect modification lies exactly in the span of the matched moments, so this is a variance result under correct identification and says nothing about bias. The target moments and the target treatment effect come from the same trial, which is the common case but not the only one. STC, ML-NMR, ML-UMR and NMI have different sensitivities and are not covered. EST-07’s other components, model uncertainty in reconstructed correlations, inclusion-criteria ambiguity and secular drift, are not sampling error and are untouched. The alignment parameter imposes exact non-negative proportionality between source and target modifier coefficients, so modifier sets differing in direction or overlapping only partially are untested.
Peer review. 2 rounds, two independent reviewers. The reports, the point-by-point responses and the editorial decision are published in full: read the review.
Read it: PDF · Markdown · OpenDocument · Code and data: studies/MIS-03-target-moment-uncertainty · Protocol: registered before the run · Review: full history
What this project measured
Study studies/MIS-03-target-moment-uncertainty, run and decided against a rule registered before the run. 252 scenarios, 5000 replicates each.
Conclusion: uninformative, and the study says so rather than reporting its estimates. The convergence and bias gates passed, but the reference validity and negative-control gates failed, so the comparison the study was built to make cannot be read off it. The estimates exist; the prespecified rule refuses to interpret them, which is the outcome a registered decision rule exists to make available.
A successor study on non-collapsible scales, studies/EST-07-moment-variance-noncollapsible, is designed and under review but has not been run.
Verification
Verdict. Overstated. 2 independent auditors judged the claim as stated too strong. Published aggregate-data means and standard deviations are estimates, yet every population-adjustment estimator I could find conditions on them as fixed, and no study has quantified the resulting coverage loss. Uncertainty from weight estimation itself is a different matter: sandwich, ESS-rescaled and bootstrap estimators exist, are implemented in maicplus, and were benchmarked across 108 scenarios by Chandler & Proskorovsky (2024), who found the fixed-weight conventional estimator is only materially anticonservative under poor or moderate overlap.
The source titled this. “Weights and reported target moments are treated as known constants” The title says estimated weights are treated as known constants, but weight-estimation variance is solved, benchmarked across 108 scenarios and implemented in maicplus.
The source said. “Many population-adjusted analyses condition on estimated weights and on published aggregate-data moments as if both were fixed inputs, so the reported uncertainty omits weight-estimation and target-summary variability.”
What was wrong with it. The source said reported uncertainty omits both weight-estimation and target-summary variability, and that the omission is directional so intervals come out systematically too narrow. Both auditors judged that overstated: the weight-estimation half was benchmarked across 108 scenarios by Chandler and Proskorovsky (2024, doi:10.1002/jrsm.1759) and is shipped in maicplus, in G-MAIC (doi:10.1017/rsm.2025.10021), and in the cmaic() bootstrap of cpaic at commit 9d150e9, and the direction is not guaranteed, since estimating calibration weights can raise or lower variance and fixed-weight sandwich calculations can be conservative; only the target-moment half survives.
- Literature and prior-art check
- partially-solved / overstated. The weight-estimation half of this entry has already been done, including the simulation the proposed direction asks for. Chandler & Proskorovsky (2024), Res Synth Methods 15(6):1094-1110: ‘One of the primary challenges in MAIC is the need to properly account for the additional uncertainty introduced by the matching process… we conducted a comprehensive Monte Carlo simulation study to evaluate the performance of different statistical methods across 108 scenarios. Four general approaches for variance estimation were compared in both anchored and unanchored MAICs of binary and time-to-event outcomes: (1) conventional estimators (CE) using raw weights; (2) CE using weights rescaled to the effective sample size (ESS); (3) robust sandwich estimators; and (4) bootstrapping.’ Their finding narrows the entry materially: undercoverage from treating weights as fixed appears only under poor or moderate overlap, and ‘All methods produced valid coverage probabilities and standard errors in cases of strong population overlap.’ The tooling is shipped: maicplus 0.1.2 imports both sandwich and boot, and G-MAIC (Gao, Heath & Baio 2025) ‘employs Bayesian bootstrap to propagate the uncertainty of population-adjusted estimates.’ The target-moment half is where I found nothing: no method propagating sampling error in published aggregate means and SDs. Cites doi:10.1002/jrsm.1759: Executes the entry’s proposed simulation for the weight-estimation component: 108 scenarios, four variance approaches, anchored and unanchored, binary and time-to-event, reporting empirical coverage.. Cites doi:10.1017/rsm.2025.10021: G-MAIC uses Bayesian bootstrap to propagate population-adjustment uncertainty rather than conditioning on estimated weights..
- GPT-5.6 Sol, technical and source-code lens
- partially-solved / overstated. C1 and C2 are unsourced prevalence claims, although the corresponding conditional analyses certainly occur. C3 is true when published means and SDs are sample estimates of superpopulation moments, but not when they are exact administrative quantities, user-defined targets, or the supplied summaries define a conditional finite-population estimand. C4 is correct only conditionally: stacked equations must include every relevant nuisance equation and cross-covariance, while a bootstrap must resample both data sources at their actual sampling units. Published moments alone usually do not permit such a bootstrap without their covariance or an additional sampling model. Most importantly, omission is not directionally guaranteed to make intervals narrower; estimating calibration weights can increase or decrease variance, and fixed-weight sandwich calculations may be conservative in some settings. cpaic partially resolves C1: cmaic() stratified-bootstraps the IPD, re-estimates the weights, and refits the outcome model. It nevertheless holds target means and SDs fixed throughout that bootstrap; cstc() and cmlnmr() likewise condition on supplied target summaries. The remaining target-summary component can matter with small target samples or unstable overlap but is not universally a very-high-severity defect. Cites choxos/cpaic@9d150e9cad88f2e6fbcd94dbb92c3dee3b7c2dca: The cmaic() bootstrap resamples IPD within treatment arms, re-estimates MAIC weights, and refits the outcome model, resolving the weight-estimation component for that estimator. Target moments remain fixed..
- Decision path
-
R3:partially-solved(n=2) → R8:overstated-majority(n=2)
How the evidence falls in time
2 papers in the reviewed corpus bear on this problem, published 2018 to 2024. Partial progress is on the record, and later work still calls this open. Both belong in view.
| Year | Says | Paper |
|---|---|---|
| 2018 | partly addresses it | Sensitivity analyses for effect modifiers not observed in the target population when generalizing treatment effects from a randomized controlled trial: Assumptions, models, effect scales, data scenarios, and implementation details (Nguyen, Ackerman, Schmid, Cole, Stuart, PLoS ONE 2018;13(12):e0208795) |
| 2024 | confirms it is open | Advancing unanchored simulated treatment comparisons: A novel implementation and simulation study |
Source
Derived from C:L214-215, C:L251 of the reviewed corpus.