Reported target moments are treated as known constants
Two things, established by different means. Analytically: the variance a standard interval omits is (1-2k)Var_T(tau)/n_T to first order, where k is how far the target trial’s own treatment effect is modified by the same covariates as the source trial’s, so the sign is set by a quantity the usual inputs do not identify; and a correction adding only the positive term is closer to the right variance than doing nothing only when k < 1/4. Empirically, and only in an exploratory subset: the omission runs from 9.5% too much variance to 14.9% too little and moves coverage between 92.1% and 96.2%. The registered analysis is uninformative and the confirmatory coverage question remains open. Read the study below.
Statement
Reported target-population means and standard deviations are estimates, and standard PAIC estimators condition on them as fixed inputs, so the reported interval omits that variability whenever the inference is to a superpopulation rather than to the realized finite target sample. The first of the four uncertainty sources the source bundles together has been solved, though outside the ITC literature: Sheng, Sun and Huang (arxiv:2603.02474) give asymptotic normality and a variance estimator for an entropy-balancing transport estimator, which is MAIC-equivalent, that accounts for uncertainty in covariate summaries, and Chen, Chen and Yu (doi:10.1002/sim.70358) construct resampling-based perturbation intervals for a target effect estimated from summary-level target information. The other three sources are not sampling error and do not belong in the same interval: uncertainty in reconstructed correlations is model uncertainty, inclusion-criteria ambiguity is estimand ambiguity, and secular change is a transport bias over time. The claim that no software propagates target-moment uncertainty was verified only for cpaic, an unreleased development package on GitHub that is not installable from CRAN, where at commit 9d150e9 cmaic() bootstraps IPD patients while holding target means and standard deviations fixed.
Why it is open
Conditioning on published summaries as if exact omits variability from the reported interval in a way no convergence diagnostic reveals, and the fix is not a bootstrap of the source IPD, which carries no information about how the target summaries were estimated. Propagating that uncertainty needs target microdata, a sampling design, or at least the covariance or influence information behind the published summaries, which aggregate publications rarely provide. Two of the four listed sources have no conventional variance representation at all, so they need declared alternative estimands or explicit bias models rather than a wider interval. The solutions that do exist sit in the transportability literature and have not been carried into PAIC estimators or packages, which still take reported moments as given.
What has been tried
| Work | What it contributes |
|---|---|
| Sheng, Sun and Huang 2026, Transportable inference using target population summary statistics under covariate shift | Establishes asymptotic normality for the entropy balancing estimator, which is MAIC-equivalent, and proposes a variance estimator that accounts for uncertainty in covariate summaries, plus a second method handling covariate shift and summary uncertainty together. It is framed as transportability rather than PAIC and has not been carried into ITC software. |
| Chen, Chen and Yu 2026, Confidence Interval Construction for Causally Generalized Estimates With Target Sample Summary Information, Statistics in Medicine 45(1-2):e70358 | Proposes a resampling-based perturbation method for constructing confidence intervals for an estimated target average treatment effect using summary-level target information with entropy balancing weights. |
| Gao, Heath and Baio 2025, Regression augmented weighting adjustment for indirect comparisons in health decision modelling (G-MAIC), Research Synthesis Methods | Uses Bayesian bootstrap to propagate the uncertainty of population-adjusted estimates, but treats the reported comparator summaries as fixed balancing targets, so it addresses uncertainty from the matching process rather than sampling error in the target moments. |
| cpaic, unreleased development package on GitHub (not on CRAN), commit 9d150e9 | At commit 9d150e9, cmaic() holds target means and standard deviations fixed while bootstrapping only IPD patients, cstc() centers on fixed target means, and cmlnmr() builds fixed integration points from aggregate means, standard deviations and a supplied or borrowed correlation. This confirms the conditioning behavior in one unreleased package rather than across all PAIC software. |
Probable solution or research direction
Port the existing target-summary variance results into PAIC rather than deriving them again: the entropy-balancing variance estimator and the perturbation interval both apply to MAIC-equivalent weighting, so the work is validation under PAIC conditions and implementation in packages that currently hold target moments fixed. A Bayesian formulation placing a distribution over the target covariate law is a reasonable complement, but its posterior is identified only through the target data and the chosen law or prior, so that prior has to be stated rather than absorbed. Handle inclusion-criteria ambiguity and secular drift as declared alternative estimands or explicit bias models with sensitivity ranges, not as extra variance in one interval. Uncertainty in reconstructed correlations remains the genuinely unaddressed component and is the natural next target.
Our study
What conditioning on sampled target moments costs
This study was designed against MIS-03 and bears on this entry in part. What it does not answer is stated below.
The question put to it. Population adjustment treats a target trial’s published covariate means and standard deviations as exact, though they are sample estimates. By how much does that cost interval coverage at realistic sample sizes, and what would a publication have to report for a correction to work?
What it found. Two things, established by different means. Analytically: the variance a standard interval omits is (1-2k)Var_T(tau)/n_T to first order, where k is how far the target trial’s own treatment effect is modified by the same covariates as the source trial’s, so the sign is set by a quantity the usual inputs do not identify; and a correction adding only the positive term is closer to the right variance than doing nothing only when k < 1/4. Empirically, and only in an exploratory subset: the omission runs from 9.5% too much variance to 14.9% too little and moves coverage between 92.1% and 96.2%. The registered analysis is uninformative and the confirmatory coverage question remains open.
Results.
- The prespecified analysis returned uninformative. Its gates require the reference interval and the negative controls to be nominal, and at poor overlap the Wald sandwich undercovers by up to 7 points for every method including scenarios with no effect modification at all, where the population omission is exactly zero. That is a source-variance failure, not a target-moment one.
- The sign of the omission is an analytic identity, not a simulation finding: (1-2k)Var_T(tau)/n_T. Regressing the measured omission on it across all 216 scenarios with effect modification gives a slope of 1.003 (SE 0.042). An earlier draft claimed the simulation located the cancellation point; peer review corrected that and the claim was retracted.
- The two corrections an analyst can deploy add only the positive term and omit the cross-covariance. They beat doing nothing only when k < 1/4, and are worse than doing nothing between 1/4 and 1/2. The design has no interior k with which to test this, so no recommendation is made for intermediate alignment. A draft claim that they help whenever k < 1/2 was wrong and was corrected in review.
- In the exploratory subset the status quo was the only method to leave 93% to 97% coverage, and it left in both directions. Magnitude tracks the ratio of source to target information, so a large source trial matched to a small target publication is the worst case, not a small target trial.
- Peer review left a standing recommendation of major revision. The confirmatory question needs a fresh registered run with an ex ante overlap restriction or a validated source variance estimator.
Design.
- Estimand
- Target-superpopulation marginal A versus B mean difference, exact in closed form
- Methods compared
- status quo (target moments held fixed), normal reconstruction (deployable today), reported moment covariance (enhanced reporting), full joint score (all target information)
- Data-generating mechanism
- Anchored MAIC on a continuous outcome, with the target trial’s participants generated and then discarded so the estimator sees only a published baseline table. 252 scenarios crossing source size, target size, overlap, effect-modification strength calibrated to a fixed target-population standard deviation, the alignment between the two trials’ effect modification, and whether the analyst’s borrowed correlation is correct.
- Replicates
- 5000 per scenario, 1,260,000 in total
What this does not answer. The registered coverage question is not confirmatorily answered: every usable coverage result comes from a post-run restriction conditioning on an observed outcome. One estimator, one outcome type, one information structure. Covariates are multivariate normal, which is also the assumption the deployable correction is derived under, so it is measured under conditions favorable to it; skewed, bounded and categorical covariates are untested. The outcome is continuous and the effect additive, so marginal and conditional effects coincide, and nothing transfers directly to binary or time-to-event outcomes, where MAIC is most used. Effect modification lies exactly in the span of the matched moments, so this is a variance result under correct identification and says nothing about bias. The target moments and the target treatment effect come from the same trial, which is the common case but not the only one. STC, ML-NMR, ML-UMR and NMI have different sensitivities and are not covered. EST-07’s other components, model uncertainty in reconstructed correlations, inclusion-criteria ambiguity and secular drift, are not sampling error and are untouched. The alignment parameter imposes exact non-negative proportionality between source and target modifier coefficients, so modifier sets differing in direction or overlapping only partially are untested.
Peer review. 2 rounds, two independent reviewers. The reports, the point-by-point responses and the editorial decision are published in full: read the review.
Read it: PDF · Markdown · OpenDocument · Code and data: studies/MIS-03-target-moment-uncertainty · Protocol: registered before the run · Review: full history
Question. When MAIC or STC matches reported target moments that are themselves estimates from 100 to 1000 people, what does treating them as known constants cost in bias and coverage on a non-collapsible scale, where the moment-matched contrast is not the target estimand?
Design. Successor to MIS-03 on a non-collapsible scale; every number in the protocol is generated from the registered design file and checked by script. Not yet run: the registered design needs about 668 core-hours.
The protocol is registered before any result is seen: read it.
Verification
Verdict. Partially addressed. Part of this problem is addressed by existing work; the entry names which part.
The source said. “Reported target-population means and standard deviations are estimates, not known constants, yet standard PAIC estimators condition on them as fixed inputs. Direction B holds that a joint model should propagate sampling uncertainty in target moments, uncertainty in reconstructed correlations, ambiguity in target inclusion criteria, and secular change between the trial and current practice.”
What was wrong with it. The source treated all four uncertainty sources as unaddressed; the literature auditor found the first already solved for MAIC-equivalent estimators (arxiv:2603.02474; doi:10.1002/sim.70358) and noted that G-MAIC (doi:10.1017/rsm.2025.10021) propagates uncertainty by Bayesian bootstrap of the IPD only, leaving the target summaries fixed. GPT-5.6 Sol judged the entry overstated for treating the moments as necessarily random when the estimand may condition on the realized finite target, for bundling estimand ambiguity and secular drift with sampling variance, and for a universal software negative that source code confirms only for cpaic.
Dissent. codex dissented (overstated): C1 is correct for superpopulation inference from a finite target sample, but not when the estimand deliberately conditions on the realized finite target population and its exact summaries. C2 is an empirical ecosystem claim; source code confirms it for cpaic. At 9d150e9, cmaic() holds target means and SDs fixed while bootstrapping only IPD patients, cstc() centers on fixed target means, and cmlnmr() constructs fixed integration points from aggregate means, SDs, and a supplied or borrowed correlation. C3 improperly bundles sampling uncertainty, reconstructed-law model uncertainty, estimand ambiguity, and temporal transport bias. Inclusion criteria and secular change need explicit alternative estimands, scenarios, or bias models, not automatic addition to one interval. C4 is possible but posterior uncertainty is identified only through the target data and chosen law or prior. C5 requires target microdata, a sampling design, or at least the covariance or influence information for the reported summaries; bootstrapping source IPD alone cannot do it. C6 is an unsupported universal software negative, although it is true of cpaic. Frequentist methods already address target-summary sampling uncertainty, so only the broader structural components remain open. The priority is very high only when the target sample is small, the effect is sensitive to its law, or the decision changes.
- Literature and prior-art check
- partially-solved / supported-with-caveat. The descriptive half checks out. G-MAIC (Gao, Heath & Baio, Res Synth Methods 2025) uses the reported comparator summaries as fixed balancing targets and propagates uncertainty only by Bayesian bootstrap of the IPD; Chandler & Proskorovsky’s 108-scenario MAIC variance study (Res Synth Methods 2024) compares conventional, ESS-rescaled, sandwich and bootstrap estimators, all of which address uncertainty from the matching process, none the sampling error in the target moments; maicplus matches weighted moments to reported moments taken as given. But the first of the four listed uncertainty sources has been solved outside the ITC literature, using estimators that are equivalent to MAIC. Sheng, Sun & Huang (arXiv:2603.02474, March 2026) apply entropy balancing to transportability and ‘propose a variance estimator to account for uncertainty in covariate summaries’, then give a second method handling covariate shift and summary uncertainty simultaneously; Chen, Chen & Yu (Stat Med 45(1-2):e70358, 2026) give a resampling-perturbation confidence interval for a target ATE estimated from target summary-level information. The other three sources (reconstructed correlations, inclusion-criteria ambiguity, secular drift) remain genuinely unaddressed. Cites arxiv:2603.02474: Delivers valid inference for an entropy-balancing (MAIC-equivalent) transport estimator that treats the reported target covariate moments as estimated rather than known.. Cites doi:10.1002/sim.70358: Constructs confidence intervals for a target-population effect when only summary-level target information is available, using entropy balancing weights..
- GPT-5.6 Sol, technical and source-code lens
- partially-solved / overstated. C1 is correct for superpopulation inference from a finite target sample, but not when the estimand deliberately conditions on the realized finite target population and its exact summaries. C2 is an empirical ecosystem claim; source code confirms it for cpaic. At 9d150e9, cmaic() holds target means and SDs fixed while bootstrapping only IPD patients, cstc() centers on fixed target means, and cmlnmr() constructs fixed integration points from aggregate means, SDs, and a supplied or borrowed correlation. C3 improperly bundles sampling uncertainty, reconstructed-law model uncertainty, estimand ambiguity, and temporal transport bias. Inclusion criteria and secular change need explicit alternative estimands, scenarios, or bias models, not automatic addition to one interval. C4 is possible but posterior uncertainty is identified only through the target data and chosen law or prior. C5 requires target microdata, a sampling design, or at least the covariance or influence information for the reported summaries; bootstrapping source IPD alone cannot do it. C6 is an unsupported universal software negative, although it is true of cpaic. Frequentist methods already address target-summary sampling uncertainty, so only the broader structural components remain open. The priority is very high only when the target sample is small, the effect is sensitive to its law, or the decision changes. Cites arxiv:2603.02474: Provides variance estimation that accounts for uncertainty in target covariate summaries and also addresses covariate shift.. Cites doi:10.1002/sim.70358: Constructs intervals for target effects when only target summary information is available, resolving the first uncertainty source in a defined sampling setting..
- Decision path
-
R3:partially-solved(n=2) → R9:substantiated-single-overstatement:codex
How the evidence falls in time
4 papers in the reviewed corpus bear on this problem, published 2018 to 2022. Partial progress is on the record, and later work still calls this open. Both belong in view.
| Year | Says | Paper |
|---|---|---|
| 2018 | partly addresses it | Sensitivity analyses for effect modifiers not observed in the target population when generalizing treatment effects from a randomized controlled trial: Assumptions, models, effect scales, data scenarios, and implementation details (Nguyen, Ackerman, Schmid, Cole, Stuart, PLoS ONE 2018;13(12):e0208795) |
| 2018 | confirms it is open | Comparing pharmacological treatments for cocaine dependence: Incorporation of methods for enhancing generalizability in meta-analytic studies |
| 2020 | confirms it is open | Multilevel network meta-regression for population-adjusted treatment comparisons |
| 2022 | confirms it is open | Parametric G-computation for compatible indirect treatment comparisons with limited individual patient data |
Source
Derived from C:L326-335, C:L251 of the reviewed corpus.