Deciding which covariates are true effect modifiers
Source note. This entry derives in part from a source document whose inline citations were ChatGPT interface tokens rather than references, and so resolve to nothing. Claims below are attributed to that document and were checked independently where possible; anything that could not be independently sourced is marked in the verification trail.
Selecting effect modifiers on the analyzed data (interactions with p < 0.2) left the naive STC interval covering 0.887 to 0.927, because its SE ignored which model was selected; a bootstrap that repeats the selection restored 0.938 to 0.968. Lasso at the one-standard-error penalty dropped real modifiers and coverage fell to 0.33 with three modifiers at 1000 per arm. Fitting all candidate interactions kept coverage at 0.93 to 0.96. Read the study below.
Statement
There is no agreed default for choosing the effect-modifier set: the right set depends on the treatment contrast, estimand, effect scale, design and available support, and in the small samples typical of these analyses significance screening is unstable while balancing on every available covariate can destroy overlap and exhaust the information available for interaction terms. Reviews document what applied work does: NICE technology appraisals selected covariates by availability, expert opinion, effective sample size, statistical significance or cross-validation (Phillippo et al. 2019, doi:10.1017/S0266462319000333); of 117 oncology MAICs only 3 met all NICE recommendations, with evidence of effect-modifier status among the least reported items (Farinasso 2025, doi:10.1016/j.vhri.2025.101088); and no study among 27 CAR-T indirect comparisons described its identification methodology (Neumann 2026, doi:10.57264/cer-2026-0033). Adjustment sets are not uniformly small: those reviews report anywhere from no adjustment to 19 covariates, and a median of 6 (range 1 to 13) for unanchored analyses. A formal causal covariate-selection procedure does exist for transporting experimental results (Egami and Hartman, arxiv:1909.02669), but it assumes richer target data than an aggregate-data comparator supplies, and it has not been benchmarked for this setting.
Why it is open
Interaction terms carry far less information than main effects, so a data-driven modifier set fluctuates across resamples and the downstream interval is conditional on a selection event that is never reported. Balancing on everything trades that instability for a support problem: extra balancing constraints usually make the weights harder to satisfy and cut effective sample size, though not monotonically, and extra interaction terms consume information the network may not hold. Clinical elicitation, the remaining route, is incomplete and not free of the analyst’s interest in the result. Because the rationale is seldom documented the same evidence base yields different adjustment sets in different submissions, and getting the set wrong has measured consequences: all population adjustment methods incur bias when an effect modifier is missing (Phillippo et al. 2020, doi:10.1002/sim.8759).
What has been tried
| Work | What it contributes |
|---|---|
| Phillippo et al., 2019 | Reviews population adjustment in NICE technology appraisals and records that covariates were included ‘based on: availability, expert opinion, effective sample size, statistical significance, or cross-validation’; the audit uses it to source the ad hoc and significance-screening claims, and it reports a median of 6 covariates (range 1 to 13) for unanchored analyses. |
| Farinasso, 2025 | Scoping review of 117 oncology MAICs focused on reporting quality; only 3 fulfilled all NICE recommendations, and evidence of effect-modifier status and the distribution of weights were among the least reported items. |
| Neumann, 2026 | Systematic review of 27 CAR-T indirect comparisons; covariates were chosen from clinical expertise or literature review but ‘no study provided a detailed description of the identification methodology’, and adjustment ranged from none to 19 covariates. |
| Egami and Hartman, 2021 | Formal causal covariate-selection procedure based on estimated separating sets for generalizing experimental results; adjacent prior art that assumes richer target data than an aggregate-data comparator provides. |
| Phillippo et al., 2020 | Simulation study of anchored population adjustment methods; all methods incur bias when an effect modifier is missing, which is the consequence of getting the modifier set wrong. |
Probable solution or research direction
Prespecify the selection process and record it: causal and clinical knowledge, cross-trial interaction evidence, prognostic strength where the design is unanchored or the strategy is precision-oriented, measurement comparability, expected imbalance between source and target, support and estimability consequences, and predeclared sensitivity sets over alternative modifier lists. The gap to fill is not a way to encode nonlinear covariate effects, which splines and basis expansions already do, but a way to choose and estimate them when interaction information is sparse. Adapt and benchmark the existing separating-set procedure (arxiv:1909.02669) for the case where the target is known only through published summaries, and report uncertainty for the whole procedure including selection, whether by sample splitting, cross-fitting, resampling the entire selection procedure, selective inference, shrinkage or model averaging. Minimum covariate reporting sets for aggregate-data publications would make any of this auditable.
Our study
Coverage of an STC interval after selecting effect modifiers on the same data
This study was designed against DEC-11 and bears on this entry in part. What it does not answer is stated below.
The question put to it. When effect modifiers are selected on the analyzed data before an STC, does an interval that treats the selected model as fixed keep its coverage, and does resampling the whole procedure restore it?
What it found. Selecting effect modifiers on the analyzed data (interactions with p < 0.2) left the naive STC interval covering 0.887 to 0.927, because its SE ignored which model was selected; a bootstrap that repeats the selection restored 0.938 to 0.968. Lasso at the one-standard-error penalty dropped real modifiers and coverage fell to 0.33 with three modifiers at 1000 per arm. Fitting all candidate interactions kept coverage at 0.93 to 0.96.
Results.
- Refuting sentence fails: naive coverage after significance selection below 0.93 beyond Monte Carlo error in some scenarios (0.887 to 0.927); bias at most 0.046.
- Whole-procedure bootstrap: 0.938 to 0.968.
- Lasso (lambda 1se) kept on average 0.01 to 0.15 interactions; coverage as low as 0.333 and bias about -0.2 with three real modifiers.
- The shortfall was not monotone in sample size with three modifiers (largest at 300 per arm), as registered.
Design.
- Estimand
- Target-population marginal log odds ratio, A versus C
- Methods compared
- all interactions, oracle interactions, no interactions, p < 0.2 selection with naive interval, p < 0.2 selection with whole-procedure bootstrap, lasso selection
- Data-generating mechanism
- G-computation STC with six candidate modifiers, five selection rules (all, oracle, none, p < 0.2, lasso), naive and whole-procedure bootstrap intervals, three sample sizes and one or three true modifiers. 400 replicates per cell.
- Replicates
- 400 per scenario, 2,400 in total, each with a 60-resample bootstrap
What this does not answer. STC only; MAIC-based selection, overlap, larger candidate sets and target-summary uncertainty not run; 400 replicates. Peer review has not been done.
Peer review. Not yet done. The result stands on the study alone until it is.
Read it: PDF · Markdown · OpenDocument · Code and data: studies/DEC-11-selection-uncertainty-and-what-belongs-in-an-interval · Protocol: registered before the run
Question. At source-trial sizes typical of appraisals, does interaction screening detect an effect modifier reliably wherever omitting it would bias the target-population effect, and is balancing or adjusting for every candidate a workable alternative when there are 6 or 13 candidates?
Design. Continuous-outcome source trial with 6 or 13 candidate covariates and one true modifier at five strengths, targets shifted by 0.2 or 0.5 SD on every candidate; interactions screened at p < 0.05, MAIC balancing the true, screened or full set, and STC with all interactions. 20 cells, 1000 replicates each.
The protocol is registered before any result is seen: read it.
What this project measured
COV-04 answers this entry’s shrinkage and model-averaging route to uncertainty for the whole selection procedure. With eight candidate interactions in a continuous-outcome STC, 95% posterior intervals for the transported effect covered 0.794 to 0.948 under an empirical-Bayes ridge prior on the interactions and 0.868 to 0.959 under a hierarchical normal prior, against 0.939 to 0.955 with all interactions unpenalized. Shrinkage does not select, but when the modifiers the target weights push the effect the same way, their joint shrinkage biases the transported effect and the narrower interval does not carry that bias. Spike-and-slab averaging undercovered less (lowest 0.908); conditioning on the median-probability model behaved like frequentist selection (lowest 0.785).
Not answered: the regularized horseshoe, heredity constraints, binary and survival outcomes, and interactions of mixed sign relative to the target shift.
Verification
Verdict. Partially addressed. Auditors agreed the problem is open and no counterevidence surfaced. Reopened by the full-text reading: 3 findings from 3 papers survived both the reader and an independent second reviewer, so the verdict moves from confirmed-open to partially-addressed.
The source said. “Effect-modifier transport is the entire reason population adjustment exists, yet there is no defensible default for choosing the modifier set: significance-based screening rests on underpowered interaction tests, adjusting for every available covariate destroys overlap and can leave the interaction model unidentified, and clinical elicitation is incomplete. Applied work uses small ad hoc covariate sets with weak overlap diagnostics and poorly reported selection rationale.”
What was wrong with it. The claim was unattributed in the source, which also said applied work uses ‘small ad hoc covariate sets’; the literature auditor supplied real references, so the ad hoc-selection, weak-diagnostic and poor-rationale claims are now sourced to three reviews (doi:10.1017/S0266462319000333, doi:10.1016/j.vhri.2025.101088, doi:10.57264/cer-2026-0033), the last of which contradicts ‘small’ by reporting adjustment sets that range from none to 19 covariates. The code auditor struck two further claims: splines and basis expansions already encode nonlinear covariate effects, so only their selection and estimation under sparse data is unresolved, and honest sample splitting is one valid inferential strategy among several rather than a necessity.
Revised by the full-text reading. Applied 2026-07-26 full-text reading of the consolidated library (687 papers).
- Verdict changed
- confirmed-open -> partially-addressed 3 surviving findings from: A process for assessing the feasibility of a network meta-an; A two-stage prediction model for heterogeneous effects of tr; Extending inferences from a randomized trial to a new target. New citations: 10.1002/sim.9034, 10.1186/1741-7015-12-93
- Literature and prior-art check
- open / supported-with-caveat. Retrieved three empirical reviews that source this entry’s unsourced claims. Phillippo 2019 (NICE TAs) reports covariates ‘included based on: availability, expert opinion, effective sample size, statistical significance, or cross-validation’ — sourcing both the significance-screening and ad hoc claims. Farinasso 2025 (117 oncology MAICs) reports only 3 met all NICE recommendations and that ‘the least reported aspects were the adjustment for all effect modifiers and prognostic variables…, evidence of effect modifier status, and distribution of weights’ — sourcing the poor-rationale and weak-diagnostics claims. Neumann 2026 (27 CAR-T ITCs) states ‘no study provided a detailed description of the identification methodology’ and that standardized approaches are ‘urgently needed’. Caveat: the same review contradicts ‘small’ covariate sets (range: none to 19 covariates); Phillippo 2019 gives median 6, range 1-13 for unanchored. The ‘no benchmarked pipeline’ claim is an absence claim I cannot verify; a formal separating-set selection algorithm exists for generalization (Egami & Hartman), just not benchmarked with an aggregate-data target. Cites doi:10.1017/S0266462319000333: Sources the claims that applied PAIC selects covariates ad hoc and partly by statistical significance. Cites doi:10.1016/j.vhri.2025.101088: Sources the weak-diagnostics and poor-reporting-of-rationale claims with numbers (117 studies, 3 compliant). Cites doi:10.57264/cer-2026-0033: A source that itself asserts the absence of a standardized covariate-identification approach. Cites arxiv:1909.02669: A formal causal covariate-selection procedure (estimated separating sets) for transporting effects; adjacent prior art the entry does not cite, though it assumes richer target data than a PAIC aggregate-data comparator provides. Cites doi:10.1002/sim.8759: Sources the consequence of getting the modifier set wrong.
- GPT-5.6 Sol, technical and source-code lens
- open / overstated. The underlying selection problem is genuinely open because the needed adjustment set is specific to the treatment contrast, estimand, effect scale, design, and available support. The compound claim is nevertheless too broad. C3-C6 and C8 are prevalence or consensus claims with no supplied evidence. C7 is false as written: splines, basis expansions, and prespecified interactions already encode nonlinear effects; choosing and estimating them under sparse data is the unresolved part. C9 and C11 are common finite-sample risks, not universal facts. C12 holds when selection is ignored, and C15 is false as a necessity because cross-fitting, resampling the full selection procedure, selective inference, shrinkage or model averaging, and external prespecification are alternatives. Added balance constraints often worsen feasibility and ESS but need not reduce ESS monotonically. At cpaic@9d150e9, no modifier-selection pipeline is exported; cstc() and cmaic() accept analyst-specified sets and default to all IPD covariates. Very-high priority is warranted only when plausible modifier omission and weak support can materially move the target contrast.
- Decision path
-
R5:default-open → R9:substantiated-single-overstatement:codex
How the evidence falls in time
29 papers in the reviewed corpus bear on this problem, published 2012 to 2025. Partial progress is on the record, and later work still calls this open. Both belong in view.
This problem is recurrent: three or more papers spanning at least eight years assert it independently. Sustained restatement by authors who mostly do not cite each other is the strongest evidence this reading can offer that a gap is real rather than one group’s framing.
It is also being closed in pieces: partial results come from more than one paper in more than one year, each covering a different part.
| Year | Says | Paper |
|---|---|---|
| 2012 | confirms it is open | Efficacy of once-daily indacaterol 75 microgram relative to alternative bronchodilators in COPD: a study level and a patient level network meta-analysis |
| 2013 | confirms it is open | Evidence synthesis for decision making 3: heterogeneity - subgroups, meta-regression, bias, and bias-adjustment |
| 2013 | confirms it is open | Is network meta-analysis as valid as standard pairwise meta-analysis? It all depends on the distribution of effect modifiers |
| 2013 | confirms it is open | The use of individual patient-level data (IPD) to quantify the impact of pretreatment predictors of response to treatment in chronic hepatitis B patients |
| 2014 | partly addresses it | A process for assessing the feasibility of a network meta-analysis: a case study of everolimus in combination with hormonal therapy versus chemotherapy for advanced breast cancer |
| 2014 | confirms it is open | An overview of conducting systematic reviews with network meta-analysis |
| 2016 | confirms it is open | Exploratory Network Meta Regression Analysis of Stroke Prevention in Atrial Fibrillation Fails to Identify Any Interactions with Treatment Effect |
| 2016 | confirms it is open | Identifying back pain subgroups: developing and applying approaches using individual patient data collected within clinical trials |
| 2017 | confirms it is open | Management of people with early- or very early-stage hepatocellular carcinoma: an attempted network meta-analysis |
| 2018 | confirms it is open | Comparing pharmacological treatments for cocaine dependence: Incorporation of methods for enhancing generalizability in meta-analytic studies |
| 2018 | confirms it is open | The use of single armed observational data to closing the gap in otherwise disconnected evidence networks: a network meta-analysis in multiple myeloma |
| 2019 | confirms it is open | Population Adjustment Methods for Indirect Comparisons: A Review of National Institute for Health and Care Excellence Technology Appraisals |
| 2020 | partly addresses it | Extending inferences from a randomized trial to a new target population (Dahabreh, Robertson, Steingrimsson, Stuart, Hernan; tutorial, arXiv:1805.00550v3) |
| 2020 | confirms it is open | Assessing the performance of population adjustment methods for anchored indirect comparisons: A simulation study |
| 2020 | confirms it is open | A Review of Two Regulatory Approved Anti-CD19 CAR T-Cell Therapies in Diffuse Large B-Cell Lymphoma: Why Are Indirect Treatment Comparisons Not Feasible? |
| 2020 | confirms it is open | Patient characteristics as effect modifiers for psoriasis biologic treatment response: an assessment using network meta-analysis subgroups |
| 2021 | partly addresses it | A two-stage prediction model for heterogeneous effects of treatments |
| 2021 | confirms it is open | Supplementary appendix to Furukawa et al., Dismantling, optimising, and personalising internet cognitive behavioural therapy for depression: a systematic review and component network meta-analysis using individual participant data |
| 2021 | confirms it is open | Methods for population adjustment with limited access to individual patient data: A review and simulation study |
| 2021 | confirms it is open | Reporting of Cochrane systematic review protocols with network meta-analyses-A scoping review |
| 2023 | confirms it is open | Increasing transparency in indirect treatment comparisons: is selecting effect modifiers the missing part of the puzzle? A review of methodological approaches and critical considerations |
| 2023 | confirms it is open | A Critical Appraisal of Matching-Adjusted Indirect Comparisons in Spinal Muscular Atrophy |
| 2023 | confirms it is open | Network meta-interpolation: Effect modification adjustment in network meta-analysis using subgroup analyses (Harari, Soltanifar, Cappelleri, Verhoek, Ouwens, Daly, Heeg; Research Synthesis Methods 2023;14(2):211-233) |
| 2024 | confirms it is open | PD39 First Approach For Assessing Statistical Significance In Industry Funded Matching-Adjusted Indirect Comparison Studies: A Scoping Review |
| 2025 | confirms it is open | Unanchored simulated treatment comparison on survival outcomes using parametric and Royston-Parmar models with application to lenvatinib plus pembrolizumab in renal cell carcinoma |
| 2025 | confirms it is open | A process to validate prognostic factors for unanchored matching-adjusted indirect comparison of single-arm trials in oncology: a proof-of-concept study |
| 2025 | confirms it is open | Network connectivity, between-study heterogeneity and timepoint challenges in generalized myasthenia gravis: a feasibility assessment of indirect treatment comparisons |
| 2025 | confirms it is open | A database to initiate methodological advances in the evaluation of transitivity assumption in network meta-analysis: qualitative features and limitations of the tracenma R package |
| 2025 | confirms it is open | Selection of indirect treatment comparisons for health technology assessments: a practical guide for health economics and outcomes research scientists and clinicians |
Source
Derived from A:L63, C:L214-215, C:L228, C:L916-928 of the reviewed corpus.