IPD availability may be informative rather than random
Selection of which trials supply individual data on an observed trial variable caused no bias, but selection on the trials’ own effect modification biased the within-trial interaction evidence by up to 0.14 (true 0.3), with coverage down to 0.59. Combining with the across-trial association in all trials halved the bias. A leave-IPD-out influence diagnostic did not respond to the selection, so the problem is invisible without a stated selection model. Read the study below.
Statement
IPD availability may be informative rather than random, and the premise has empirical support: a survey of 31 IPD meta-analyses found that 16 (52 percent) did not obtain all the individual participant data requested, five of those did not mention it as a limitation, and only six examined how the trials without IPD might affect conclusions. Component PAIC conditions on the supplied IPD and provides neither a selection model for availability nor grouped leave-IPD-out validation. Nonrandomness has to be demonstrated or posited as a sensitivity assumption rather than assumed categorically: availability can be modeled where it depends on observed trial-level variables, and only selection on unobserved IPD quantities is unidentified without additional data or assumptions.
Why it is open
The IPD edges are the ones that get population-adjusted and the ones that inform interaction parameters, so a selection mechanism on availability propagates into which parts of covariate space are informed by causal evidence rather than by aggregate association. Selection depending on unobserved IPD characteristics cannot be estimated from within the network, because the trials with unobserved IPD are precisely the ones that would identify it. Federated approaches or a full selection model require access or assumptions that are currently unavailable, and no leave-IPD-out facility exists in component PAIC software to at least measure the dependence.
What has been tried
| Work | What it contributes |
|---|---|
| Ahmed, Sutton and Riley 2012 | surveys 31 IPD meta-analyses and documents that most did not obtain all requested individual participant data and few assessed the impact, establishing that IPD availability is not random |
| cpaic 0.1.0 (development version, choxos/cpaic@9d150e9) | unreleased development R package on GitHub, not on CRAN; conditions on the supplied IPD studies without modeling why IPD are available, and provides observation-level leave-one-out only, with no grouped leave-IPD-out facility |
Probable solution or research direction
Implement grouped leave-IPD-out validation first, since it measures how conclusions depend on which trials contributed individual data even though it does not identify or correct the mechanism. Model availability where it depends on observed trial-level characteristics, and treat selection on unobserved characteristics as an explicit sensitivity assumption with a stated selection model rather than as an unmodeled default. Pursue federated methods that let non-sponsor trials contribute individual-level information without releasing data, which is the only route that addresses the cause rather than the symptom.
Our study
Which trials supply individual data, and what selection on it does
The question put to it. When the trials that supply individual data are selected on observed trial variables or on their own effect modification, how biased is the interaction evidence, does combining with aggregate trials help, and can a leave-IPD-out diagnostic reveal it?
What it found. Selection of which trials supply individual data on an observed trial variable caused no bias, but selection on the trials’ own effect modification biased the within-trial interaction evidence by up to 0.14 (true 0.3), with coverage down to 0.59. Combining with the across-trial association in all trials halved the bias. A leave-IPD-out influence diagnostic did not respond to the selection, so the problem is invisible without a stated selection model.
Results.
- Selection on the observed trial variable only: within-trial bias -0.003 to 0.003.
- Selection on the latent interaction: within-trial bias 0.027 to 0.142; combined 0.014 to 0.065.
- Leave-IPD-out influence 0.026 to 0.046, varying with the IPD share but not with selection strength (correlation with bias -0.85 at half the trials supplying data).
Design.
- Estimand
- Mean treatment-by-covariate interaction and the effect at covariate 1
- Methods compared
- pooled within-trial interaction from IPD trials, combined within-trial and across-trial estimator, leave-IPD-out influence diagnostic
- Data-generating mechanism
- Twelve trials with trial-specific interactions and IPD availability depending on an observed trial variable and on the latent interaction; within-trial and combined estimators; leave-IPD-out influence. 18 cells, 1000 replicates each.
- Replicates
- 1000 per scenario, 18,000 in total
What this does not answer. Pairwise continuous-outcome meta-analysis rather than component ML-NMR; no selection-model sensitivity analysis; one network size. Peer review has not been done.
Peer review. Not yet done. The result stands on the study alone until it is.
Read it: PDF · Markdown · OpenDocument · Code and data: studies/CMP-22-measuring-the-dependence-before-modeling-it · Protocol: registered before the run
Verification
Verdict. Partially addressed. Auditors agreed the problem is open and no counterevidence surfaced. Reopened by the full-text reading: 3 findings from 3 papers survived both the reader and an independent second reviewer, so the verdict moves from confirmed-open to partially-addressed.
The source said. “The trials for which IPD is available are typically sponsor-owned and are not a random subset of the evidence base, yet component PAIC treats IPD availability as an exogenous feature of the network.”
What was wrong with it. The source asserted non-random IPD availability categorically and cited nothing. GPT-5.6 Sol flagged the categorical form as unsupported and distinguished selection on observed trial-level variables, which is modelable, from selection on unobserved IPD quantities, which is not identified; the literature auditor supplied the missing empirical source, doi:10.1136/bmj.d7762.
Dissent. literature dissented (supported): The premise is sourceable and the entry sources nothing. Ahmed, Sutton and Riley 2012 surveyed 31 recent IPD meta-analyses of randomised trials and found that ‘Sixteen (52%) of the 31 meta-analyses did not obtain all the individual participant data requested, yet five of these (31%) did not mention this as a potential limitation, and only six (38%) examined how trials without individual participant data might affect the conclusions’, concluding that ‘Publication, availability, and selection biases are a potential concern for meta-analyses of individual participant data’. That converts ‘IPD availability is not random’ from assertion to evidence, and it also shows the field’s partial response - their recommendation is to include aggregate data from the IPD-less studies to gauge impact, which is what a network with mixed IPD and AgD already does. What remains open is exactly what the entry says: a selection model for IPD availability, and leave-IPD-out validation, neither of which I found in the PAIC literature.
Revised by the full-text reading. Applied 2026-07-26 full-text reading of the consolidated library (687 papers).
- Verdict changed
- confirmed-open -> partially-addressed 3 surviving findings from: A Bayesian hierarchical model for network meta-analysis of m; Dismantling, personalising and optimising internet cognitive; Meta-Research: Methods and Protocols (Evangelou E, Veroniki . New citations: 10.1007/978-1-0716-1566-9, 10.1093/biostatistics/kxx025, 10.1136/bmjopen-2018-026137
- Literature and prior-art check
- open / supported. The premise is sourceable and the entry sources nothing. Ahmed, Sutton and Riley 2012 surveyed 31 recent IPD meta-analyses of randomised trials and found that ‘Sixteen (52%) of the 31 meta-analyses did not obtain all the individual participant data requested, yet five of these (31%) did not mention this as a potential limitation, and only six (38%) examined how trials without individual participant data might affect the conclusions’, concluding that ‘Publication, availability, and selection biases are a potential concern for meta-analyses of individual participant data’. That converts ‘IPD availability is not random’ from assertion to evidence, and it also shows the field’s partial response - their recommendation is to include aggregate data from the IPD-less studies to gauge impact, which is what a network with mixed IPD and AgD already does. What remains open is exactly what the entry says: a selection model for IPD availability, and leave-IPD-out validation, neither of which I found in the PAIC literature. Cites doi:10.1136/bmj.d7762: Empirically documents non-random IPD availability and its neglect; supplies the missing source for the entry’s premise without solving the selection-modelling problem..
- GPT-5.6 Sol, technical and source-code lens
- open / overstated. C2 is confirmed: cpaic records the supplied IPD studies and adjusts those studies without modeling why IPD are available. C3 is true for cpaic; its LOO method is observation-level and explicitly lacks grouped leave-one-study-out, and no leave-IPD-out facility is exported. C1, however, is an unsupported categorical empirical assertion. IPD availability may be informative, but nonrandomness must be demonstrated or posited as a sensitivity assumption. Availability and its associations with observed trial-level variables can be modeled; only selection depending on unobserved IPD quantities is not identified without additional data or assumptions. Leave-IPD-out analysis assesses stability but does not identify or correct the selection mechanism.
- Decision path
-
R5:default-open → R9:substantiated-single-overstatement:codex
How the evidence falls in time
12 papers in the reviewed corpus bear on this problem, published 2012 to 2025. Partial progress is on the record, and later work still calls this open. Both belong in view.
This problem is recurrent: three or more papers spanning at least eight years assert it independently. Sustained restatement by authors who mostly do not cite each other is the strongest evidence this reading can offer that a gap is real rather than one group’s framing.
It is also being closed in pieces: partial results come from more than one paper in more than one year, each covering a different part.
| Year | Says | Paper |
|---|---|---|
| 2012 | confirms it is open | Efficacy of once-daily indacaterol 75 microgram relative to alternative bronchodilators in COPD: a study level and a patient level network meta-analysis |
| 2016 | confirms it is open | An overview of methods for network meta-analysis using individual participant data: when do benefits arise? |
| 2016 | confirms it is open | A scoping review of indirect comparison methods and applications using individual patient data |
| 2016 | confirms it is open | Identifying back pain subgroups: developing and applying approaches using individual patient data collected within clinical trials |
| 2018 | partly addresses it | A Bayesian hierarchical model for network meta-analysis of multiple diagnostic tests |
| 2019 | partly addresses it | Dismantling, personalising and optimising internet cognitive-behavioural therapy for depression: a study protocol for individual participant data component network meta-analysis |
| 2020 | confirms it is open | Statistical analyses and quality of individual participant data network meta-analyses were suboptimal: a cross-sectional study |
| 2021 | confirms it is open | Individual Participant Data Meta-Analysis: A Handbook for Healthcare Research (Riley, Tierney, Stewart, eds.; Wiley Series in Statistics in Practice, 2021) |
| 2022 | partly addresses it | Meta-Research: Methods and Protocols (Evangelou E, Veroniki AA, eds), Methods in Molecular Biology vol. 2345 |
| 2022 | confirms it is open | Complex speech-language therapy interventions for stroke-related aphasia: the RELEASE study incorporating a systematic review and individual participant data network meta-analysis |
| 2022 | confirms it is open | Reporting and data sharing level for COVID-19 vaccine trials: A cross-sectional study |
| 2025 | confirms it is open | Combining treatment effects from mixed populations in meta-analysis: a review of methods |
Source
Derived from C:L1381, scope:field-level (evidence access across all PAIC); unmodeled in cpaic of the reviewed corpus.