Data-driven modifier and causal-structure discovery with few trials
No. Lasso discovery at the minimum-error penalty found the true modifiers but added 1.7 to 2.8 false ones among seven null covariates, and sets from independent datasets agreed with Jaccard similarity 0.37 to 0.57; the one-standard-error rule was stable only by selecting nothing (a single modifier found in 3% to 5% of analyses). With an unmeasured study-level cause of effect heterogeneity, pooled discovery selected the covariate whose trial means tracked it in 69% of analyses with 3 trials and 93% with 12 at the same total size, and raised false discoveries among the other covariates from 1.7 to 2.8 per analysis to 3.6 to 3.9; trial-specific treatment effects removed both. Read the study below.
Statement
The study-aware causal forest already exists: Multi-Study Causal Forest (arXiv:2502.02110) is built to borrow individual patient data across studies when the sources of heterogeneity differ, and causal forests are routinely fitted to pooled multi-trial individual data. What is not done is aggregating the fitted conditional average treatment effect to an external HTA target population inside an indirect comparison, and no work applies causal discovery to a treatment network. Neither method selects a causally sufficient transport adjustment set, resolves unmeasured transport bias, or stays reliable with sparse events and few informative environments, so both remain hypothesis-generation aids.
Why it is open
Aggregating a fitted conditional effect to a declared target distribution needs the target covariate distribution, overlap and a transport argument, none of which the forest supplies. Discovered heterogeneity can reflect between-study differences rather than modification, and neither method distinguishes a causal interaction from one induced by an unadjusted study-level factor. Causal discovery returns observationally equivalent graphs with different transport implications; randomized treatments can orient some edges, but nothing in the output selects an adjustment set or certifies transportability. The identifying signal for cross-study structure is the number of distinct environments, and evidence networks supply few.
What has been tried
| Work | What it contributes |
|---|---|
| Multi-Study Causal Forest (MCF): A flexible framework for data borrowing in the presence of varying treatment effect heterogeneity (2025) | Covers the study-aware causal forest for effect-modifier discovery across studies. Does not cover aggregation of the fitted conditional effect to an external target population within an indirect comparison. |
Probable solution or research direction
Take the existing multi-study causal forest and add what an indirect comparison needs: standardization of the fitted conditional effect to a declared external target distribution, with overlap and transport assumptions stated. Quantify how many trials and events are required before discovered structure is stable, and validate against known-truth simulations. Treat all output as candidate hypotheses for expert review and sensitivity analysis, requiring external confirmation before a discovered modifier enters an adjustment set.
Our study
Modifier discovery across trials: no penalty is both stable and powerful, and study-level heterogeneity creates a false modifier that more trials entrench
The question put to it. Are effect modifiers discovered from pooled trial data stable and correct enough to use, and do study-level confounders create false modifiers that more data do not remove?
What it found. No. Lasso discovery at the minimum-error penalty found the true modifiers but added 1.7 to 2.8 false ones among seven null covariates, and sets from independent datasets agreed with Jaccard similarity 0.37 to 0.57; the one-standard-error rule was stable only by selecting nothing (a single modifier found in 3% to 5% of analyses). With an unmeasured study-level cause of effect heterogeneity, pooled discovery selected the covariate whose trial means tracked it in 69% of analyses with 3 trials and 93% with 12 at the same total size, and raised false discoveries among the other covariates from 1.7 to 2.8 per analysis to 3.6 to 3.9; trial-specific treatment effects removed both.
Results.
- Usability condition (stability at least 0.8 with at most 0.5 false modifiers in every cell) failed for every method and penalty.
- Pooled minimum-error discovery selected the confounded covariate in 0.69, 0.81, 0.93 of analyses with 3, 6, 12 trials (one true modifier); within-trial discovery 0.14 to 0.46, the rate of any null covariate.
- Pooled minimum-error target bias under confounding 0.023 to 0.057; one-standard-error rule biased toward no modification by 0.09 to 0.21.
Design.
- Estimand
- Selection of true and false effect modifiers, stability of the selected set, and the target mean difference at covariate means 0.5
- Methods compared
- pooled lasso, minimum-CV penalty, pooled lasso, one-SE rule, within-trial lasso, minimum-CV penalty, within-trial lasso, one-SE rule
- Data-generating mechanism
- Three to twelve trials at a fixed total of 2400 patients, one or three true modifiers among ten candidates, study-level confounding absent or present; pooled and within-trial lasso at two penalty rules; power, false discoveries, stability and target error. 12 cells, 300 replicates each.
- Replicates
- 300 per cell, 12 cells, two datasets per replicate
What this does not answer. Lasso only (no multi-study causal forests); continuous outcome; linear modification; one overlap level; post-selection inference for the target not studied here. Peer review has not been done.
Peer review. Not yet done. The result stands on the study alone until it is.
Read it: PDF · Markdown · OpenDocument · Code and data: studies/ADJ-10-discovery-that-cannot-repair-transport · Protocol: registered before the run
Verification
Verdict. Partially addressed. Part of this problem is addressed by existing work; the entry names which part.
The source said. “Study-aware causal forests are proposed to discover effect modifiers and aggregate conditional average treatment effects to a target population, and causal discovery algorithms to generate candidate DAGs and interaction structures across studies. Both are offered as replacements for pre-specified modifier lists but are labeled hypothesis-generation aids rather than selection procedures.”
What was wrong with it. The source proposed study-aware causal forests as something still to develop and labeled them ‘Promising-emerging’; the literature auditor found a published multi-study causal forest (arXiv:2502.02110), so the forest half is not awaiting adaptation and the open part is aggregation to an external target. GPT-5.6 Sol also corrected the claim that few trials and rare events compromise validity: they damage power, stability and calibration, while a large randomized trial can still identify within-trial conditional effects.
- Literature and prior-art check
- partially-solved / supported-with-caveat. The ‘study-aware causal forest’ is already published: ‘Multi-Study Causal Forest (MCF): A flexible framework for data borrowing in the presence of varying treatment effect heterogeneity’ (arXiv:2502.02110), which is explicitly built to borrow individual patient data across studies when the sources of heterogeneity differ. Causal forests are also routinely fitted to pooled multi-trial IPD (e.g. the type 2 diabetes application, BMC Med Inform Decis Mak 2023; the psychiatry tutorial, Int J Methods Psychiatr Res 2025). So the forest half is not a proposal awaiting adaptation; what remains open is aggregating the fitted CATE to an external HTA target population inside an indirect comparison. I found nothing applying causal discovery to a treatment network, and the entry’s limitations (few trials, rare events, observational equivalence of returned graphs, inability to distinguish modification from between-study confounding) are all standard and correct. The caveat is the ‘Promising-emerging’ label for causal forests, which understates the multi-study work already done. Cites arxiv:2502.02110: Covers the study-aware causal forest for effect-modifier discovery across studies. Does not cover aggregation of the CATE to an external target population within an indirect comparison..
- GPT-5.6 Sol, technical and source-code lens
- open / overstated. C1 is feasible with sufficiently large randomized IPD, study-aware fitting or splitting, overlap, and a declared target covariate distribution. C2 is not a guaranteed improvement, and C3 confuses low information with validity: few trials and rare events primarily damage power, stability, calibration, and transport assessment, while a large randomized trial can still identify within-trial CATEs. C4 is correct for unmeasured transport factors. C5 and C6 are defensible only as candidate generation. C7 holds for purely observational discovery without added interventional, temporal, functional-form, or background-knowledge restrictions; randomized treatments and multiple interventions can orient some edges, so observational equivalence does not categorically prevent all orientation. C8 applies when cross-study shifts are the identifying signal, not simply because the individual-level sample came from few trials. C9 and C10 reproduce the labels. Calling either approach a replacement for pre-specified, causally justified modifiers is unsound, and Medium-high is inflated for tools restricted to hypothesis generation.
- Grok 4.5, recency lens
- open / supported. Causal forests are actively used on single RCTs and in multi-study synthesis contexts (e.g. Van Vogt et al. 2025 BMC Medical Research Methodology application; broader multi-study causal ML surveys 2024), and causal discovery remains a general tool for candidate DAGs. Neither closes the stated ITC problem. Forests still need substantial within-study signal and do not remove hidden confounding or certify that discovered heterogeneity is modification rather than study-level confounding; discovery still returns observationally equivalent graphs with different transport implications. No 2024–2026 work makes study-aware causal forests or multi-study discovery a reliable adjustment-set selector under typical NMA study counts and rare events. Treating output as hypothesis generation (as the report does) is still the honest status. Queries: causal forest network meta-analysis; study-aware causal forest IPD meta-analysis; causal discovery multi-trial transportability.
- Decision path
-
R3:partially-solved(n=1) → R9:substantiated-single-overstatement:codex
Source
Derived from C:L1646-1647, C:L1651, C:L1671 of the reviewed corpus.