Intervals omit selection, reconstruction and target-summary error
Selecting effect modifiers on the analyzed data (interactions with p < 0.2) left the naive STC interval covering 0.887 to 0.927, because its SE ignored which model was selected; a bootstrap that repeats the selection restored 0.938 to 0.968. Lasso at the one-standard-error penalty dropped real modifiers and coverage fell to 0.33 with three modifiers at 1000 per arm. Fitting all candidate interactions kept coverage at 0.93 to 0.96. Read the study below.
Statement
The estimated-weights component of this problem is solved and evaluated: Chandler and Proskorovsky 2024 (10.1002/jrsm.1759) compared four families of MAIC variance estimator across 108 scenarios and give guidance by overlap, effective sample size and outcome type, following Remiro-Azocar et al. 2021 (10.1002/jrsm.1511), which had shown the robust sandwich estimator understates variability at small ESS. What remains unpropagated is model-selection uncertainty where selection used the analyzed data, pseudo-IPD reconstruction uncertainty, target-summary uncertainty, and any decomposition separating sampling from prior and structural contributions. Bridge uncertainty belongs on that list only with a caveat: it is structural rather than sampling-based and should be reported through separate sensitivity or tipping analysis unless a defensible probability model supports joint propagation. The claim that standard software returns one interval with no components is also not true of cpaic, an unreleased GitHub development package not on CRAN, which re-estimates MAIC weights inside an arm-stratified bootstrap, preserves multi-arm covariance, and carries an integration-error diagnostic for component ML-NMR.
Why it is open
Interval estimates are conditional on the selected model, so a selection step carried out on the analyzed data leaves the reported interval too narrow, and no ITC reporting convention requires selection uncertainty to be accounted for. Each remaining component enters at a different stage of a multi-step pipeline, and the final stage that produces the interval has no access to what earlier stages introduced. Numerical integration error is a deterministic approximation error, so the fix is bounding and convergence assessment rather than treatment as another random variance component; the existing diagnostic traces nested approximations against the final grid rather than against the exact integral and is unavailable for survival models.
What has been tried
| Work | What it contributes |
|---|---|
| Chandler and Proskorovsky 2024 | 108-scenario simulation comparing four families of MAIC variance estimator; finds raw-weight conventional estimators understate variability under poor or moderate overlap, ESS-weighted conventional estimators accurate across most scenarios, and sandwich estimators downward-biased at small ESS without finite-sample adjustment. |
| Remiro-Azocar, Heath and Baio 2021 | Earlier evidence that the robust sandwich variance estimator underestimates variability in MAIC where effective sample sizes are small. |
| cpaic (development package, audited at commit 9d150e9) | Unreleased development package on GitHub, confirmed by an auditor not to be on CRAN. Re-estimates MAIC weights and refits the outcome model inside an arm-stratified bootstrap, preserves multi-arm covariance through netmeta for the frequentist bridge and the standard 0.5 baseline-shift correlation for random effects, and provides a nested-Sobol integration-error diagnostic; selection, target-summary, pseudo-IPD and bridge-uncertainty decomposition remain unimplemented. |
Probable solution or research direction
Adopt the published variance-estimation guidance for weights instead of treating it as open, and extend propagation to the stages still conditioned on: pseudo-IPD reconstruction, target summaries including their covariance, and model selection where the selection used the analyzed data. Require full model specification with priors and random-effects structure, explicit separation of within-study from between-study effect modification, justification of shared versus treatment- or component-specific interactions, and presentation of substantively plausible alternative models. Report structural bridge uncertainty and numerical error separately, as sensitivity regions and convergence bounds, rather than merging them into a single interval.
Our study
Coverage of an STC interval after selecting effect modifiers on the same data
The question put to it. When effect modifiers are selected on the analyzed data before an STC, does an interval that treats the selected model as fixed keep its coverage, and does resampling the whole procedure restore it?
What it found. Selecting effect modifiers on the analyzed data (interactions with p < 0.2) left the naive STC interval covering 0.887 to 0.927, because its SE ignored which model was selected; a bootstrap that repeats the selection restored 0.938 to 0.968. Lasso at the one-standard-error penalty dropped real modifiers and coverage fell to 0.33 with three modifiers at 1000 per arm. Fitting all candidate interactions kept coverage at 0.93 to 0.96.
Results.
- Refuting sentence fails: naive coverage after significance selection below 0.93 beyond Monte Carlo error in some scenarios (0.887 to 0.927); bias at most 0.046.
- Whole-procedure bootstrap: 0.938 to 0.968.
- Lasso (lambda 1se) kept on average 0.01 to 0.15 interactions; coverage as low as 0.333 and bias about -0.2 with three real modifiers.
- The shortfall was not monotone in sample size with three modifiers (largest at 300 per arm), as registered.
Design.
- Estimand
- Target-population marginal log odds ratio, A versus C
- Methods compared
- all interactions, oracle interactions, no interactions, p < 0.2 selection with naive interval, p < 0.2 selection with whole-procedure bootstrap, lasso selection
- Data-generating mechanism
- G-computation STC with six candidate modifiers, five selection rules (all, oracle, none, p < 0.2, lasso), naive and whole-procedure bootstrap intervals, three sample sizes and one or three true modifiers. 400 replicates per cell.
- Replicates
- 400 per scenario, 2,400 in total, each with a 60-resample bootstrap
What this does not answer. STC only; MAIC-based selection, overlap, larger candidate sets and target-summary uncertainty not run; 400 replicates. Peer review has not been done.
Peer review. Not yet done. The result stands on the study alone until it is.
Read it: PDF · Markdown · OpenDocument · Code and data: studies/DEC-11-selection-uncertainty-and-what-belongs-in-an-interval · Protocol: registered before the run
Verification
Verdict. Partially addressed. Part of this problem is addressed by existing work; the entry names which part.
The source titled this. “Reported intervals omit selection, weighting and bridge uncertainty” The title lists weighting uncertainty as unpropagated, but the audit found the estimated-weights variance question settled across a 108-scenario evaluation with actionable guidance.
The source said. “Reported ITC and PAIC intervals are computed conditional on one selected model and on upstream inputs treated as fixed. Model specification and the selection process are not fully reported, and uncertainty from estimated weights, multi-arm covariance, pseudo-IPD reconstruction, target summaries, priors, numerical integration and the bridge itself is neither propagated nor decomposed.”
What was wrong with it. The source treated estimated-weight uncertainty in MAIC as unaddressed and said standard software offers no decomposition. The literature auditor found the variance-estimation question settled by a 108-scenario evaluation with actionable guidance (10.1002/jrsm.1759), and GPT-5.6 Sol found weight bootstrapping, multi-arm covariance and an integration-error diagnostic already implemented in cpaic and objected that structural bridge uncertainty should not be folded into a confidence interval.
Dissent. codex dissented (overstated): C1 applies when model selection used the analyzed data, not to a genuinely prespecified model; its claimed frequency is not established here. C2-C3 are correct because between-study interaction gradients are ecologically confounded and shared interactions are structural assumptions. C4 is warranted for substantively plausible alternatives, not every imaginable model. C5 should say random-effects structures and priors can materially affect conclusions and must be reported. C6-C8 are sound when those stages are estimated, with target-summary covariance also needed where applicable. C7’s necessity is exact, but its frequency claim is unverified. C9 is important, but bridge uncertainty is structural and cannot be inserted into a confidence interval without a defensible probability model; sensitivity regions or tipping analyses should ordinarily remain separate. C10 should require convergence assessment and control of numerical error, not automatic treatment as a stochastic variance component. The blanket software claim is false for cpaic: cMAIC re-estimates weights inside an arm-stratified bootstrap, network engines preserve multi-arm dependence, and cML-NMR has an integration-error diagnostic. Selection, target-summary, pseudo-IPD, and full bridge-uncertainty decomposition remain unresolved.
- Literature and prior-art check
- partially-solved / supported-with-caveat. One component of this bundle has been formally settled. Chandler & Proskorovsky 2024 (Res Synth Methods, doi 10.1002/jrsm.1759, PMID 39323097) ran a 108-scenario Monte Carlo study comparing four families of variance estimator for MAIC (conventional with raw weights, conventional with ESS-rescaled weights, robust sandwich variants, bootstrap) across anchored and unanchored designs and binary and time-to-event outcomes, and gives concrete guidance: raw-weight conventional estimators understate variability under poor or moderate overlap, sandwich estimators are downward-biased at small ESS unless finite-sample-adjusted, bootstrapping is unstable at small ESS. Remiro-Azocar et al. 2021 (doi 10.1002/jrsm.1511) had already shown ‘the robust sandwich variance estimator underestimates variability where effective sample sizes are small’. So ‘uncertainty from estimated weights should be accounted for in MAIC standard errors’ is no longer an open problem; it is a solved-and-evaluated question with published recommendations. The remaining components, model-selection uncertainty, pseudo-IPD reconstruction uncertainty, target-summary uncertainty, multi-arm covariance preservation, numerical integration error, and a decomposition separating sampling from bridge and prior uncertainty, I found no work addressing, and they remain open. Cites 10.1002/jrsm.1759: Resolves the estimated-weights component: a 108-scenario evaluation of four variance-estimation approaches with actionable guidance by sample size, overlap and outcome type.. Cites 10.1002/jrsm.1511: Earlier evidence that the standard MAIC variance estimator misstates uncertainty at small effective sample size..
- GPT-5.6 Sol, technical and source-code lens
- partially-solved / overstated. C1 applies when model selection used the analyzed data, not to a genuinely prespecified model; its claimed frequency is not established here. C2-C3 are correct because between-study interaction gradients are ecologically confounded and shared interactions are structural assumptions. C4 is warranted for substantively plausible alternatives, not every imaginable model. C5 should say random-effects structures and priors can materially affect conclusions and must be reported. C6-C8 are sound when those stages are estimated, with target-summary covariance also needed where applicable. C7’s necessity is exact, but its frequency claim is unverified. C9 is important, but bridge uncertainty is structural and cannot be inserted into a confidence interval without a defensible probability model; sensitivity regions or tipping analyses should ordinarily remain separate. C10 should require convergence assessment and control of numerical error, not automatic treatment as a stochastic variance component. The blanket software claim is false for cpaic: cMAIC re-estimates weights inside an arm-stratified bootstrap, network engines preserve multi-arm dependence, and cML-NMR has an integration-error diagnostic. Selection, target-summary, pseudo-IPD, and full bridge-uncertainty decomposition remain unresolved. Cites choxos/cpaic@9d150e9cad88f2e6fbcd94dbb92c3dee3b7c2dca: Resolves C6 for source-IPD sampling uncertainty by recalculating MAIC weights and refitting the outcome model in every bootstrap resample. Target moments remain fixed.. Cites choxos/cpaic@9d150e9cad88f2e6fbcd94dbb92c3dee3b7c2dca: Addresses C7 through netmeta’s multi-arm covariance for the frequentist bridge and the standard 0.5 baseline-shift correlation for cML-NMR random effects.. Cites choxos/cpaic@9d150e9cad88f2e6fbcd94dbb92c3dee3b7c2dca: Partially resolves C10 by tracing nested Sobol-point approximations against the final fitted grid. It does not estimate error of the final grid against the exact integral and is unavailable for survival models..
- Decision path
-
R3:partially-solved(n=2) → R9:substantiated-single-overstatement:codex
How the evidence falls in time
10 papers in the reviewed corpus bear on this problem, published 2013 to 2026. The most recent paper to touch this reports progress on it, and every paper calling it open is older.
It is also being closed in pieces: partial results come from more than one paper in more than one year, each covering a different part.
Does the later work answer the earlier work? Put to an independent reviewer with the older and newer claims side by side, it judged that the later work answers a component of it, leaving a named part (high confidence). The later work substantially addresses uncertainty from estimated MAIC weights, including comparative evaluation and practical limits under poor overlap. It does not propagate the other named sources of uncertainty into the reported interval. What remains: Propagation of data-driven model-selection uncertainty, pseudo-IPD reconstruction uncertainty, target-summary uncertainty, and a defensible decomposition of sampling, structural, and prior contributions; structural bridge uncertainty still needs sensitivity or tipping analyses unless probabilistically modeled.
| Year | Says | Paper |
|---|---|---|
| 2013 | partly addresses it | Variance reduction in randomised trials by inverse probability weighting using the propensity score |
| 2018 | confirms it is open | Comparing pharmacological treatments for cocaine dependence: Incorporation of methods for enhancing generalizability in meta-analytic studies |
| 2019 | confirms it is open | Treatment sequence network meta-analysis in Crohn’s disease: a methodological case study |
| 2020 | confirms it is open | Performance of unanchored matching-adjusted indirect comparison (MAIC) for the evidence synthesis of single-arm trials with time-to-event outcomes |
| 2021 | confirms it is open | Methods for population adjustment with limited access to individual patient data: A review and simulation study |
| 2021 | confirms it is open | Network meta-analysis of biologic treatments for psoriasis using absolute Psoriasis Area and Severity Index values ≤1, 2, 3 or 5 derived from a statistical conversion method |
| 2022 | confirms it is open | Two-stage matching-adjusted indirect comparison |
| 2023 | confirms it is open | A Critical Appraisal of Matching-Adjusted Indirect Comparisons in Spinal Muscular Atrophy |
| 2024 | confirms it is open | A comprehensive review and shiny application on the matching-adjusted indirect comparison |
| 2026 | partly addresses it | Can We Trust PAICs in Rare Diseases? Methodological Challenges and Limitations |
Source
Derived from C:L2399-2406, C:L2408-2415 of the reviewed corpus.