No post-selection inference for model choice in population adjustment
For choices among MAIC and STC specifications the candidates are close: choosing the most precise method covered 0.938 to 0.956, like a prespecified one. Choosing the most favorable estimate shifted it by up to 0.26 on the log odds ratio scale while its interval still covered 0.927 or more, because the favored method often had a wide interval. Averaging with between-model variance over-covered (up to 0.991) because the candidates share data; that difference makes the refuting sentence fail as registered. Read the study below.
Statement
When an ITC or PAIC model is chosen or tuned adaptively and the reported interval conditions on that choice, unconditional coverage can be understated, and no post-selection or shrinkage-propagation machinery has been adapted to population adjustment; both auditors searched and found nothing PAIC-specific. The stronger form does not hold, because regularization is not the same as omitted uncertainty: a Bayesian fit that carries posterior draws through to the transported effect does propagate shrinkage, as the development source choxos/cpaic@9d150e9 does, so what is actually omitted is discrete model-selection uncertainty and the choice of prior scale. The word ‘commonly’ is an empirical claim the source did not support; the nearest evidence is a methodological review of PAIC publications finding that key methodological aspects were reported inconsistently, with appropriate uncertainty measures among the less-followed recommendations (doi:10.1016/j.jclinepi.2023.09.004).
Why it is open
Post-selection inference and honest sample splitting exist in general statistics but have not been adapted to population adjustment, where selection happens in the IPD trial while the estimand lives in a different population and further uncertainty enters through estimated weights and reported target moments. Splitting a single HTA-scale trial to buy honest inference costs precision these analyses cannot spare, so the same data keep being used for discovery, tuning, estimation and uncertainty. Bayesian implementations are not automatically safe either: carrying posterior draws through covers shrinkage under a fixed prior, but not the discrete model choice or the prior scale that was itself selected.
What has been tried
| Work | What it contributes |
|---|---|
| Serret-Larmande et al. 2023, Journal of Clinical Epidemiology (PMID 37717707) | Methodological review of published PAIC analyses reporting that key methodological aspects were reported inconsistently, with only three articles adequately reporting all aspects and appropriate uncertainty measures among the less-followed recommendations. This is the closest real evidence for the source’s unsourced ‘commonly’. |
| cpaic development source at commit 9d150e9 (choxos/cpaic) | Source-code counter-evidence cited by GPT-5.6 Sol: regularizing interaction priors are recorded in the fitted object and posterior draws are carried through to target-population effects and quantiles, so shrinkage uncertainty does reach the transported effect. cpaic is an unreleased development package on GitHub, not installable from CRAN. |
Probable solution or research direction
Port existing machinery instead of inventing it: stacking or model averaging over scientifically plausible modifier sets, projection-predictive selection from a rich reference model, shrinkage priors with the full posterior carried into the transported effect, and honest splitting or cross-fitting where sample size permits. Do not present cross-fitting as the only valid route, since valid selective inference, resampling that repeats the selection step, debiasing and full Bayesian model averaging are alternatives with their own conditions. The PAIC-specific gap is that no coverage assessment of an effect-modifier selection procedure has been run with the selection step included, which is where the work should start.
Our study
Choosing among population-adjustment models on the same data
This study was designed against MIS-04 and bears on this entry in part. What it does not answer is stated below.
The question put to it. When the adjustment model is chosen from the analyzed data, does the chosen model’s interval keep its coverage, and what does averaging over candidates change?
What it found. For choices among MAIC and STC specifications the candidates are close: choosing the most precise method covered 0.938 to 0.956, like a prespecified one. Choosing the most favorable estimate shifted it by up to 0.26 on the log odds ratio scale while its interval still covered 0.927 or more, because the favored method often had a wide interval. Averaging with between-model variance over-covered (up to 0.991) because the candidates share data; that difference makes the refuting sentence fail as registered.
Results.
- Registered verdict FAILS, through the averaged interval’s coverage differing from the prespecified one’s by more than 0.03.
- Selection by smallest SE: coverage 0.938 to 0.956, bias as the prespecified linear STC.
- Selection by most favorable estimate: bias -0.046 to -0.259, coverage 0.927 to 0.960.
- Equal-weight average with added between-model variance: coverage 0.949 to 0.991, the widest intervals.
Design.
- Estimand
- Marginal log odds ratio, B versus A, in the target (DIA-08)
- Methods compared
- prespecified linear STC, prespecified MAIC on means, selection by smallest SE, selection by most favorable estimate, equal-weight model averaging
- Data-generating mechanism
- Re-scoring of DIA-08’s stored replicates (16 scenarios x 2000) under prespecified, precision-selected, result-selected and averaged use of four adjustment methods.
- Replicates
- 32,000 stored replicates from DIA-08 re-scored
What this does not answer. Outcome and weighting models only; treatment partitions and bridges not studied; equal weights. Peer review has not been done.
Peer review. Not yet done. The result stands on the study alone until it is.
Read it: PDF · Markdown · OpenDocument · Code and data: studies/MIS-04-selection-over-models-bridges-and-partitions · Protocol: registered before the run
Verification
Verdict. Confirmed open. Auditors agreed the problem is open and no counterevidence surfaced.
The source titled this. “Selection and shrinkage uncertainty absent from transported effects” The title says shrinkage uncertainty is absent from transported effects, which the audit rejected since a Bayesian fit carrying posterior draws through does propagate shrinkage.
The source said. “When a population-adjustment outcome model is chosen by variable selection or fitted with regularization, the reported standard errors and intervals condition on the selected model, so shrinkage and selection uncertainty never reach the transported effect. The report lists regularization or selection not reflected in uncertainty as a distinct STC bias source.”
What was wrong with it. The source said shrinkage and selection uncertainty ‘never reach the transported effect’. GPT-5.6 Sol found that false for a full Bayesian fit that carries posterior draws through to the target effect (choxos/cpaic@9d150e9, an unreleased development package on GitHub that an auditor confirmed is not on CRAN), and the literature auditor could not verify the word ‘commonly’, pointing instead to doi:10.1016/j.jclinepi.2023.09.004 on inconsistent uncertainty reporting in PAIC.
Dissent. codex dissented (overstated): C2-C3 are correct for unconditional coverage of an adaptive procedure. C1 and C5 are correct only when model, modifier, or tuning selection is performed and the reported interval conditions on that choice; the word ‘commonly’ is an empirical claim not established here. Regularization is not synonymous with omitted uncertainty. At cpaic@9d150e9, cmlnmr() uses explicitly regularizing interaction priors and relative_effects.cpaic_mlnmr() carries posterior draws through target effects and quantiles, directly refuting the claim that shrinkage uncertainty never reaches a transported effect. Prior-scale choice or discrete model-selection uncertainty can still be omitted. C4 is also too absolute: reusing data without cross-fitting is not inherently invalid under parametric or Donsker conditions, selective-inference or debiasing theory, full Bayesian model averaging, or a bootstrap that repeats selection. The PAIC-specific practice and coverage problem remains open, but it is high priority only for analyses that actually select or tune adaptively and then ignore that process.
- Literature and prior-art check
- open / supported-with-caveat. Searched PubMed and the web for post-selection or shrinkage-uncertainty propagation adapted to population adjustment and found nothing PAIC-specific, so the core ‘not adapted to population adjustment’ claim survives. But two caveats. First, the empirical claim that conventional STC analyses ‘commonly’ fail to reflect selection uncertainty is not sourced anywhere in the entry and I could not verify the word ‘commonly’; the nearest real evidence is Serret-Larmande et al. 2023 (PMID 37717707), which found that across 133 PAIC publications ‘Key methodological aspects were reported inconsistently, with only three articles adequately reporting all aspects’ and that appropriate uncertainty measures were among the less-followed recommendations. Second, prior_work is empty although the entry’s whole proposed_direction (stacking, projection-predictive selection, horseshoe and spike-and-slab priors, honest splitting, cross-fitting) is off-the-shelf general Bayesian methodology with mature software; nothing in the entry acknowledges that the machinery exists and only the adaptation is missing. Claims 2 and 3 are uncontested general statistical facts.
- GPT-5.6 Sol, technical and source-code lens
- open / overstated. C2-C3 are correct for unconditional coverage of an adaptive procedure. C1 and C5 are correct only when model, modifier, or tuning selection is performed and the reported interval conditions on that choice; the word ‘commonly’ is an empirical claim not established here. Regularization is not synonymous with omitted uncertainty. At cpaic@9d150e9, cmlnmr() uses explicitly regularizing interaction priors and relative_effects.cpaic_mlnmr() carries posterior draws through target effects and quantiles, directly refuting the claim that shrinkage uncertainty never reaches a transported effect. Prior-scale choice or discrete model-selection uncertainty can still be omitted. C4 is also too absolute: reusing data without cross-fitting is not inherently invalid under parametric or Donsker conditions, selective-inference or debiasing theory, full Bayesian model averaging, or a bootstrap that repeats selection. The PAIC-specific practice and coverage problem remains open, but it is high priority only for analyses that actually select or tune adaptively and then ignore that process.
- Decision path
-
R5:default-open → R9:substantiated-single-overstatement:codex
How the evidence falls in time
11 papers in the reviewed corpus bear on this problem, published 2011 to 2025. Every paper found here asserts the problem is open; none reports progress.
This problem is recurrent: three or more papers spanning at least eight years assert it independently. Sustained restatement by authors who mostly do not cite each other is the strongest evidence this reading can offer that a gap is real rather than one group’s framing.
| Year | Says | Paper |
|---|---|---|
| 2011 | confirms it is open | Network meta-analysis of survival data with fractional polynomials |
| 2016 | confirms it is open | GetReal in network meta-analysis: a review of the methodology |
| 2016 | confirms it is open | Identifying back pain subgroups: developing and applying approaches using individual patient data collected within clinical trials |
| 2017 | confirms it is open | Rituximab biosimilar in rheumatoid arthritis: an enhanced-evidence assessment to evaluate equivalence with the originator based on network meta-analysis |
| 2018 | confirms it is open | A framework for identifying treatment-covariate interactions in individual participant data network meta-analysis |
| 2019 | confirms it is open | Dismantling, personalising and optimising internet cognitive-behavioural therapy for depression: a study protocol for individual participant data component network meta-analysis |
| 2021 | confirms it is open | A two-stage prediction model for heterogeneous effects of treatments |
| 2022 | confirms it is open | Validating the Assumptions of Population Adjustment: Application of Multilevel Network Meta-regression to a Network of Treatments for Plaque Psoriasis |
| 2024 | confirms it is open | Response to discussant comments on “NMA, the first 20 years” |
| 2025 | confirms it is open | Unanchored simulated treatment comparison on survival outcomes using parametric and Royston-Parmar models with application to lenvatinib plus pembrolizumab in renal cell carcinoma |
| 2025 | confirms it is open | Addressing challenges with Matching-Adjusted Indirect Comparisons to demonstrate the comparative effectiveness of entrectinib in metastatic ROS-1 positive Non-Small Cell Lung Cancer |
Source
Derived from C:L252, C:L943, C:L959, C:L1018 of the reviewed corpus.