No standardized sample-size planning guidance or software for PAIC
Source note. This entry derives in part from a source document whose inline citations were ChatGPT interface tokens rather than references, and so resolve to nothing. Claims below are attributed to that document and were checked independently where possible; anything that could not be independently sourced is marked in the verification trail.
The influence-function variance. It predicted the achieved SD within 10% in 85% of 72 scenarios, missing only at 150 patients with 10% prevalence. Kish ESS was within 10% in 58% and understated the achieved SD in 99% of scenarios, so it is optimistic, not the conservative quantity it is usually taken to be; nominal n was within 10% in 21%. Read the study below.
Statement
Nominal sample size alone does not determine the precision of a population-adjusted indirect comparison, although it remains an important determinant: information also depends on anticipated overlap, the balancing constraints, the outcome distribution, treatment allocation and the estimator’s own influence function, and for survival outcomes on weighted risk-set and event information rather than on patient counts. No universal effective-sample-size threshold is defensible, and the Kish ESS is not an exact information measure in any case, since Phillippo et al. 2018 note it ‘is likely to be an underestimation of the true ESS’ when the weights are estimated and correlated with the outcome (doi:10.1177/0272989X17725740). What is missing is standardized PAIC-specific planning guidance and software, not the possibility of information-based planning: scenario-based planning that posits source and target distributions, derives the anticipated weights and uses influence-function or simulation calculations is already feasible. Simulation design is also not starting from nothing, since the 162-scenario anchored study of Remiro-Azocar, Heath and Baio 2021 (doi:10.1002/jrsm.1511) and a 2025 study indexed on positivity-violation severity (arxiv:2507.12241) are the de facto templates the field reuses.
Why it is open
The information remaining after weighting depends on the mismatch with a target that is not fixed until the comparator population is chosen, so no pre-study calculation expressed in nominal n alone can be binding, and any calculation stated in Kish ESS inherits a quantity that understates the true effective sample size and ignores arm allocation, outcome variance and nuisance-parameter estimation. For survival outcomes the relevant unit is events rather than patients, and even total events and censoring fractions are insufficient without timing, allocation, risk-set composition and the censoring mechanism, so a single grid cannot serve both outcome types. Unanchored designs carry a strictly stronger identification burden, conditional constancy of absolute rather than relative effects, and omitted prognostic structure creates structural bias that a larger sample does not cure, so the harsher grid needed there is about misspecification and confounding scenarios rather than simply larger n. The gap is therefore in standardization and tooling: nothing prevents a design-specific information calculation, but there is no accepted PAIC procedure, no software, and no agreed reporting of what was assumed.
What has been tried
| Work | What it contributes |
|---|---|
| Remiro-Azocar, Heath and Baio 2021, Methods for population adjustment with limited access to individual patient data: a review and simulation study | sources the claim that effective sample size rather than nominal size governs precision in weighting-based PAIC, reporting that the robust sandwich variance estimator underestimates variability where effective sample sizes are small; its 162-scenario survival simulation grid is close to the design grid this entry proposes building |
| Phillippo et al. 2018, Medical Decision Making | sources the harsher requirements for unanchored comparisons, which assume conditional constancy of absolute effects and require all prognostic variables as well as all effect modifiers; also records that the standard Kish ESS is likely an underestimate of the true ESS, so planning guidance stated in ESS inherits a biased quantity |
| Serret-Larmande et al. 2025, Assessing the Impact of Covariate Distribution and Positivity Violation on Weighting-Based Indirect Comparisons: a Simulation Study | simulation study explicitly designed around positivity-violation severity and non-normal covariates, supporting the proposal to index design grids on overlap severity rather than on nominal sample size |
Probable solution or research direction
Build scenario-based planning rather than a threshold rule: posit source and target covariate distributions, derive the anticipated weights and their dispersion, and compute precision from the estimator’s influence function or by simulation, reporting weighted event and risk-set information for survival outcomes and stating the assumed overlap explicitly. Extend the existing simulation templates instead of duplicating them, since the 162-scenario anchored study (doi:10.1002/jrsm.1511) already varies overlap, imbalance and sample size and the 2025 positivity study (arxiv:2507.12241) already indexes on violation severity; what is added is unanchored scenarios with omitted prognostic variables and misspecification, not merely larger n. Where planning is reported in effective sample size, state that the Kish approximation understates the true ESS and that no threshold on it certifies validity (doi:10.1177/0272989X17725740). Package the result as software so that a planning calculation can be reproduced and audited rather than asserted.
Our study
Planning a MAIC from the estimator’s influence function, not from n or Kish ESS
The question put to it. Before any data exist, which planning quantity predicts the precision a MAIC will achieve: nominal sample size, Kish effective sample size, or the estimator’s influence-function variance evaluated at posited source and target laws?
What it found. The influence-function variance. It predicted the achieved SD within 10% in 85% of 72 scenarios, missing only at 150 patients with 10% prevalence. Kish ESS was within 10% in 58% and understated the achieved SD in 99% of scenarios, so it is optimistic, not the conservative quantity it is usually taken to be; nominal n was within 10% in 21%.
Results.
- Planned SE over achieved SD, median (range): influence function 0.963 (0.827 to 1.023); Kish ESS 0.912 (0.745 to 1.004); nominal n 0.797 (0.541 to 0.985).
- Kish ESS understated the achieved SD in 99% of scenarios; its error grew with the target shift.
- All 11 influence-function misses were at 150 patients and 10% prevalence; its median ratio was 0.908 at 150 patients and 0.981 at 1000.
- The refuting sentence, that nominal n with an overlap adjustment is good enough, fails its registered criterion (58% of scenarios within 10% against the 90% required).
Design.
- Estimand
- SD of the MAIC estimate of the target marginal log odds ratio
- Methods compared
- nominal n, Kish ESS, influence-function variance
- Data-generating mechanism
- Binary-outcome MAIC with 2 or 5 covariates, 72 scenarios crossing total size (150, 400, 1000), target shift, dimension, allocation and prevalence; three planned SEs computed from the posited laws compared with the SD achieved over replicates.
- Replicates
- 1000 per scenario, 72,000 in total
What this does not answer. The planner is assumed to know the source and target laws. Binary outcome, anchored-side precision and MAIC on means only; the unanchored bias floor and survival outcomes are not run. Peer review has not been done.
Peer review. Not yet done. The result stands on the study alone until it is.
Read it: PDF · Markdown · OpenDocument · Code and data: studies/OVL-06-planning-in-information-rather-than-in-n · Protocol: registered before the run
Verification
Verdict. Confirmed open. Auditors agreed the problem is open and no counterevidence surfaced.
The source titled this. “No sample-size or information-planning framework for PAIC” The title says no planning framework exists, which the audit called a non sequitur since scenario-based influence-function and simulation planning is already feasible.
The source said. “Nominal sample size is not the operative information metric for population-adjusted indirect comparisons: information content is driven by effective sample size after reweighting and by event information in the weighted pseudo-population. The report offers proposed design grids while explicitly declining to claim any universal threshold, so there is no framework for planning a PAIC or a PAIC simulation in information terms.”
What was wrong with it. The source moved from ‘no universal threshold’ to ‘no planning framework’, which GPT-5.6 Sol judged a non sequitur, since scenario-based planning conditional on anticipated overlap and estimator-specific influence is already possible and what is missing is standardized guidance and software; it also narrowed ‘nominal n is not the operative metric’ to the weaker claim that nominal n alone is insufficient. The claim was unattributed in the source and is now sourced: the literature auditor supplied Remiro-Azocar, Heath and Baio 2021 (doi:10.1002/jrsm.1511), whose 162-scenario simulation substantially overlaps the grid the source proposed building, Phillippo et al. 2018 (doi:10.1177/0272989X17725740) for the stronger unanchored assumptions and for the fact that the ESS such guidance would be stated in is itself a biased approximation, and Serret-Larmande et al. 2025 (arxiv:2507.12241) for a simulation already indexed on positivity-violation severity.
Revised by the full-text reading. Applied 2026-07-26 full-text reading of the consolidated library (687 papers).
- Verdict left unchanged
- verdict left unchanged Later work reports progress, but on the chronology review the second reviewer judged it ‘no’ at high confidence: The handbook supplies useful IPD meta-analysis simulation planning methods and software, including for interactions, but it does not plan for the defining information losses and assumptions of PAIC. It therefore does not provide the requested standardized PAIC planning framework or tool.
- Literature and prior-art check
- open / supported-with-caveat. PubMed searches for sample-size, power or information planning tied to MAIC, PAIC and external-control weighting returned nothing; I found no planning framework, so the entry’s central claim stands and the problem is open. Because the entry is source_unsourced I supply sources below. Two caveats. First, the entry’s own instrument undermines it: Phillippo et al. 2018 state the standard ESS ‘is likely to be an underestimation of the true ESS’ because the weights are estimated and correlated with the outcome, so planning guidance ‘stated in effective sample size’ inherits a known-biased quantity, which the entry does not acknowledge. Second, ‘no framework for planning a PAIC simulation’ is narrower than stated: Remiro-Azocar et al. 2021 already provide a 162-scenario survival-outcome simulation design varying covariate overlap, imbalance and sample size, which is the de facto template the field uses. It is not a planning framework in information terms, but it is not nothing, and the entry’s proposed grid overlaps with it substantially. The harsher-grid-for-unanchored claim is well supported by the assumption hierarchy in Phillippo et al. Cites doi:10.1002/jrsm.1511: Sources the claim that effective sample size, not nominal size, governs precision in weighting-based PAIC, and supplies the existing 162-scenario survival simulation grid the entry proposes to build.. Cites doi:10.1177/0272989X17725740: Sources the harsher-requirements-for-unanchored claim: unanchored comparisons assume conditional constancy of absolute effects and require all prognostic variables as well as all effect modifiers.. Cites arXiv:2507.12241: A recent simulation study explicitly designed around positivity violation severity, supporting the entry’s proposal to index design grids on overlap severity rather than nominal n..
- GPT-5.6 Sol, technical and source-code lens
- open / overstated. C2 is too absolute: nominal n remains an important information determinant, but it is insufficient by itself. C3 is directionally correct, yet Kish ESS is not an exact information measure for estimated weights, heterogeneous outcomes, treatment contrasts, or survival estimators. Precision is governed by the estimator’s influence function or information matrix, including arm allocation, outcome variance, nuisance-parameter estimation, weighted events, and risk-set composition. C4 faithfully describes the report’s refusal to impose a universal threshold, which is appropriate. C5 is sound because patient counts alone are inadequate for survival designs, although total events and censoring fractions are also insufficient without timing, allocation, risk sets, and censoring mechanisms. C6 correctly recognizes the stronger identification burden of unanchored comparisons, but omitted prognostic structure creates structural bias that a larger sample does not cure; simulations need broader misspecification and confounding scenarios, not necessarily merely larger n values. The logical step from no universal threshold to no planning framework is false. Design-specific planning can posit source and target distributions, derive anticipated weights, and use influence-function calculations or simulation. What remains open is standardized PAIC-specific guidance and software, not the possibility of information-based planning itself. This is primarily a design-roadmap gap rather than a universal validity defect in completed analyses.
- Decision path
-
R5:default-open → R9:substantiated-single-overstatement:codex
How the evidence falls in time
4 papers in the reviewed corpus bear on this problem, published 2017 to 2021. The most recent paper to touch this reports progress on it, and every paper calling it open is older.
Does the later work answer the earlier work? Put to an independent reviewer with the older and newer claims side by side, it judged that the later work does not answer it (high confidence). The handbook supplies useful IPD meta-analysis simulation planning methods and software, including for interactions, but it does not plan for the defining information losses and assumptions of PAIC. It therefore does not provide the requested standardized PAIC planning framework or tool. What remains: PAIC-specific planning guidance and software that incorporate overlap, balancing constraints, weighting, outcome type, and estimator-specific information remain absent.
| Year | Says | Paper |
|---|---|---|
| 2017 | confirms it is open | Pharmacological interventions for primary sclerosing cholangitis: an attempted network meta-analysis |
| 2018 | confirms it is open | Mapping the characteristics of network meta-analyses on drug therapy: A systematic review |
| 2020 | confirms it is open | Comparison of Methods for Estimating Therapy Effects by Indirect Comparisons: A Simulation Study |
| 2021 | partly addresses it | Individual Participant Data Meta-Analysis: A Handbook for Healthcare Research (Riley, Tierney, Stewart, eds.; Wiley Series in Statistics in Practice, 2021) |
Source
Derived from A:L200-204 of the reviewed corpus.