Failure signatures across PAIC estimator families are uncharacterized
At the level at which a diagnostic is actually read, one analysis at a time, effective sample size tracks the error a study got by chance far better than the error it got from adjusting on the wrong covariates: AUROC 0.847 against the outcome-noise component and 0.813 against chance covariate imbalance between the source arms, both of which are functions of the covariates, the assignment and the weights, against 0.653 for the transport component that adjustment exists to remove. At the quoted cutoff of 35 it catches between a quarter and a third of material errors in every misspecification stratum, 0.309 where an effect modifier is omitted and the omission is diagnosable in principle, at specificity 0.926. No evaluated fixed cutoff met the registered requirement of sensitivity 0.80 at specificity 0.50 in every stratum. Three of the panel’s members are provably one statistic, since ESS/n = 1/(1+CV^2(w)) exactly and area under the ROC curve is invariant to monotone transformation; a fourth, the post-weighting balance on matched moments, is zero at the solution of the calibration equations and among converged analyses cannot discriminate anything. On a decision-curve basis every rule in the panel is beaten by scrutinizing every analysis, up to an action threshold of 0.3. Read the study below.
Statement
Factorial simulations have already located the region where MAIC stops improving on an unadjusted indirect comparison: Phillippo et al. 2020 (doi:10.1002/sim.8759) sweeps overlap, effect-modification strength, sample size and missing modifiers and reports that MAIC may even increase bias compared with standard indirect comparisons; Remiro-Azocar et al. 2021 (doi:10.1002/jrsm.1511) locates the bias-variance crossover over 162 scenarios; and Liu et al. 2025 (doi:10.1007/s40258-025-00952-1) runs a 729-scenario full factorial and finds MAIC more biased than naive indirect comparison at low covariate overlap with high covariate strength. What no study does is characterize and contrast the failure signatures of weighting-based, outcome-model-based and machine-learning estimators as overlap deteriorates, or report whether those failures are visible in any reported diagnostic. Flexible learners are absent from this simulation literature entirely.
Why it is open
Adjustment trades a bias term that depends on unobserved cross-study modifier imbalance against an observable variance penalty, so the exact data-specific gain cannot be learned from the data at hand without causal and modeling assumptions, although observed imbalance, prespecified effect modifiers, trial design and external knowledge still inform the decision. Weighting estimators fail by variance explosion and weight concentration while outcome-model estimators fail silently by extrapolating a fitted surface into regions with no data, so the two signatures are not comparable on any single performance measure, and flexible learners make this worse because their extrapolation behavior is set by regularization rather than by an explicit functional form. The existing factorials fix the estimator set narrowly, so the differential behavior near the support boundary is never observed even in the studies that have mapped the crossover region.
What has been tried
| Work | What it contributes |
|---|---|
| Phillippo et al. 2020, Assessing the performance of population adjustment methods for anchored indirect comparisons: A simulation study | Factorial simulation crossing overlap, effect-modification strength, sample size, missing modifiers and covariate distributions across MAIC, STC and ML-NMR, reporting where adjustment fails to pay. |
| Remiro-Azocar et al. 2021, Methods for population adjustment with limited access to individual patient data: A review and simulation study | 162-scenario simulation locating the bias-variance crossover as covariate overlap degrades. |
| Liu et al. 2025, Is the Use of Unanchored Matching-Adjusted Indirect Comparison Always Superior to Naive Indirect Comparison on Survival Outcomes? A Simulation Study | 729-scenario three-level full factorial over six factors including covariate overlap and covariate strength, identifying the region where MAIC is more biased than naive indirect comparison. |
| cpaic (development version), commit 9d150e9 | Unreleased development package on GitHub, not available from CRAN. Records a factorial comparison across imbalance and overlap for naive bridging, cSTC and cMAIC; narrow in estimator coverage, and the auditor notes its validation pipeline was not publicly tracked and the result was not independently reproducible from the public commit. |
Probable solution or research direction
Extend the existing overlap-by-effect-modification factorials rather than repeat them: hold their design fixed and add outcome-model, ML-NMR, doubly robust and flexible-learner arms, sweep overlap from complete to near-disjoint, and record for each estimator family the onset of bias, the onset of variance inflation, and whether either is visible in any diagnostic the analyst would actually report. Report failure signatures rather than only average performance, so the output is a map of how each family degrades rather than a single winner. Publish the code so the comparison can be re-run as new estimators appear.
Our study
Effective sample size tracks the error an analysis got by chance better than the error it got from adjusting on the wrong covariates
This study was designed against DIA-03 and bears on this entry in part. What it does not answer is stated below.
The question put to it. Every population adjustment arrives with a panel of numbers beside it: effective sample size, the same figure as a percentage, the largest weight, standardized differences before and after adjustment, a distance between populations. A committee reads them as evidence about whether to believe the estimate, and a widely quoted rule says to worry below an effective sample size of about 30 to 35. Scored as classifiers of realized error, with discrimination and calibration, against known truth and under a declared distribution of analyses, what are they worth?
What it found. At the level at which a diagnostic is actually read, one analysis at a time, effective sample size tracks the error a study got by chance far better than the error it got from adjusting on the wrong covariates: AUROC 0.847 against the outcome-noise component and 0.813 against chance covariate imbalance between the source arms, both of which are functions of the covariates, the assignment and the weights, against 0.653 for the transport component that adjustment exists to remove. At the quoted cutoff of 35 it catches between a quarter and a third of material errors in every misspecification stratum, 0.309 where an effect modifier is omitted and the omission is diagnosable in principle, at specificity 0.926. No evaluated fixed cutoff met the registered requirement of sensitivity 0.80 at specificity 0.50 in every stratum. Three of the panel’s members are provably one statistic, since ESS/n = 1/(1+CV^2(w)) exactly and area under the ROC curve is invariant to monotone transformation; a fourth, the post-weighting balance on matched moments, is zero at the solution of the calibration equations and among converged analyses cannot discriminate anything. On a decision-curve basis every rule in the panel is beaten by scrutinizing every analysis, up to an action threshold of 0.3.
Results.
- Prespecified verdict: the panel does not classify realized error at the thresholds in use. No rule reached sensitivity 0.80 with specificity 0.50 in every misspecification stratum. ESS below 35 reaches sensitivity 0.251 to 0.326 across strata at specificity 0.909 to 0.926.
- The mechanism, registered before the run and holding at the level of a single analysis: effective sample size discriminates the outcome-noise component at AUROC 0.847 and the transport component at 0.653, a gap of 0.194, and the arm-imbalance component at 0.813. The two it reads well are the two that are functions of the covariates, the assignment and the weights, which is what the panel is built from.
- Peer review changed this claim and the paper’s title. Scored against SYSTEMATIC transport bias with the design cell as the unit, rather than against the realized transport component of one replicate, effective sample size reaches 0.808 rather than 0.653. That analysis does not rescue the panel either: the statistic that is identically zero reaches 0.708 on it, because 128 cells is a small sample in which the bias and every diagnostic move along the same overlap axis. Both levels are reported and the title now claims only the replicate-level result.
- The panel is smaller than it looks, by algebra rather than by simulation. ESS, ESS as a percentage and the coefficient of variation of the weights had identical areas under the ROC curve to 0.0e+00 within a fixed source size. The matched-moment balance statistic never exceeded 1.4e-14 and its apparent AUROC of 0.550 is discrimination of floating-point residual, not of balance.
- Silence is where the damage is. Across the 88 cells the ESS rule never fires in, the material-error rate runs from 0.045 to 0.835. The registered version of this claim, a spread across the cells it does fire in, was not met (0.150 against a prespecified 0.30) because those cells are all bad. A first draft called the silent-cell span a stronger version of the registered claim; peer review established that both spans are variation in predictive value, which moves with prevalence and case mix, so neither is a test of whether a threshold transports. That promotion is withdrawn and the leave-one-factor-level-out calibration is the evidence on transportability.
- Decision-curve analysis: at action thresholds up to 0.3, flagging every analysis beats every rule in the panel. From 0.4 upwards three distinct statistics beat both alternatives, relative effective sample size, pre-weighting imbalance and Mahalanobis distance; the absolute cutoff that gets quoted is not among them at any threshold examined.
- The same statistics work when they measure the actual gap. Pre-weighting imbalance predicts the unadjusted comparison’s transport error at AUROC 0.891, against 0.655 for the same statistic applied to MAIC. These diagnostics are informative about the problem and uninformative about the solution: weighting sets to zero the very quantity they measure.
- Our proposal, converting residual imbalance on a measured but unadjusted covariate into the units of the estimand with the source-estimated interaction, reaches 0.937 against the transport component over the diagnosable set where effective sample size as a percentage reaches 0.851. That gain of 0.086 misses the 0.10 we registered and is reported as a miss. Inside the omitted-modifier stratum alone the plain balance check on unmatched covariates beats it, 0.946 against 0.938, because there the covariate is always a modifier and converting imbalance into effect units only adds estimation noise. The conversion earns its keep exactly when the analyst does not know whether the unmatched covariate matters.
- An adversarial critique of the design before the run found two arithmetic errors in the protocol: the dispersion factor was not the pure variance instrument it was described as wherever the cross-moment channel was on, and the target trial’s variance was understated by a factor of two to three. Both were confirmed numerically and corrected before any result was seen.
- Round one of review found the load-bearing claim of the paper wrong, and all three reviewers found it independently. The arm-imbalance component uses no outcome at all, and the noise component’s variance is exactly sigma^2 (1/ESS_A + 1/ESS_C), so a weight statistic is its natural summary rather than merely correlated with it. Round one also found that the calibration split was perfectly confounded with the dispersion factor, because cells are enumerated with that factor varying fastest; leave-one-factor-level-out replaced it and the previous claim that overlap statistics calibrate worse than weight statistics was withdrawn as an artifact.
Design.
- Estimand
- Transported A versus C mean difference in the target superpopulation; the anchored A versus B contrast as a secondary reference
- Methods compared
- MAIC matching means and standard deviations, MAIC matching means only, STC with a heteroskedasticity-consistent variance, no adjustment
- Data-generating mechanism
- An anchored indirect comparison with a continuous outcome and an identity link, chosen so that the realized error of a linear estimator splits exactly into arm imbalance, transport error and outcome noise, so each diagnostic can be scored against each component separately: two of the three are functions of the covariates, the assignment and the weights, which is what the panel is built from, and the third is what adjustment exists to remove. 128 cells crossing an omitted effect modifier, a cross-moment modifier that no baseline table could reveal, overlap, source size, how correlated the omitted modifier is with the adjustment set, and a target dispersion ratio that destroys effective sample size while adding no bias where the modifier structure is correct. 4000 replicates per cell, four estimators. Primary results are reported within misspecification strata rather than over a mixture, so no investigator-chosen frequency enters the headline.
- Replicates
- 4000 per cell, 512,000 datasets and 2,048,000 estimates
What this does not answer. The second half of DIA-03 is untouched: no PSIS-LOO is computed at any hierarchical level and grouped study-level leave-one-out is no more implemented here than in the packages, so the mismatch between the level at which cross-validation is reported and the level at which the prediction is made remains open. DIA-06 is answered only in part; two estimator families and no adjustment are compared on identical replicates, but ML-NMR, NMI, doubly robust estimators and flexible learners are absent. The outcome is continuous with an identity link, which is what buys the exact error decomposition and is also the largest restriction: nothing here transfers to non-collapsible effect measures or time-to-event outcomes, where part of the error is a collapsibility artifact rather than an adjustment failure. Covariates are normal, comparisons are anchored, conditional constancy holds, and target moments are exact. The deployment weights are a declared judgment and nothing in this program measured them, which is why the primary results are within strata.
Peer review. 2 rounds, two independent reviewers. The reports, the point-by-point responses and the editorial decision are published in full: read the review.
Read it: PDF · Markdown · OpenDocument · Code and data: studies/DIA-03-diagnostics-as-classifiers · Protocol: registered before the run · Review: full history
Question. As source-target divergence grows, does MAIC fail by variance and a linear outcome model by extrapolation bias at onsets that no single overlap statistic orders, and does a doubly robust estimator or a random forest fail later than both, on a continuous and a log odds ratio scale?
Design. Gaussian covariate shift spaced evenly in log(1 + chi-squared), with modifier curvature varied at fixed divergence under common random numbers; unadjusted, MAIC, STC, augmented MAIC and a per-arm random forest; bias and variance onsets on a continuous and a binary scale. 36 cells, 1000 replicates each.
The protocol is registered before any result is seen: read it.
Verification
Verdict. Overstated. 2 independent auditors judged the claim as stated too strong. Factorial simulations have located the region where MAIC stops improving on an unadjusted indirect comparison, but no study characterizes and contrasts the failure signatures of weighting-based versus outcome-model-based versus machine-learning PAIC estimators as overlap deteriorates, nor reports whether those failures are visible in any diagnostic.
The source titled this. “When adjustment pays, and how each estimator fails, is unknown” The title says when adjustment pays is unknown, but three published factorial simulations already locate the region where MAIC stops improving on an unadjusted comparison.
The source said. “The field cannot say under which conditions population adjustment materially improves an ordinary ITC or NMA and under which it merely adds variance without reducing bias. Nor has it characterized how MAIC, standardized outcome regression, ML-NMR, doubly robust methods and flexible learners each fail as source-target overlap deteriorates.”
What was wrong with it. The source said there is no evidence base specifying when population adjustment materially improves an ordinary ITC; the literature auditor found three factorial simulations that map exactly that region (doi:10.1002/sim.8759, doi:10.1002/jrsm.1511, doi:10.1007/s40258-025-00952-1), and GPT-5.6 Sol found a factorial imbalance-and-overlap comparison of naive bridging, cSTC and cMAIC in the cpaic development package. What survives is the comparative failure-signature question across estimator families.
- Literature and prior-art check
- partially-solved / overstated. The claim that there is no evidence base specifying when population adjustment materially improves an ordinary ITC is contradicted by at least three factorial simulations I retrieved. Phillippo et al. 2020 crossed sample size, missing effect modifiers, strength of effect modification, shared-effect-modifier validity, extrapolation validity, between-study overlap (a proxy parameter kappa swept from no overlap to full overlap) and covariate distributions, and reported that MAIC ‘may even increase bias compared with standard indirect comparisons’ - i.e. the region where adjustment does not pay. Remiro-Azocar et al. 2021 ran 162 scenarios and located the crossover explicitly. Liu et al. 2025 ran 729 scenarios in a three-level full factorial over six factors including covariate overlap and covariate strength and identified the failure region for MAIC versus naive comparison. What remains genuinely open is the comparative failure-signature question: no study I found sweeps overlap to near-disjoint and records, per estimator family, onset of bias versus onset of variance inflation and whether any diagnostic sees it, and flexible learners are absent entirely. Cites 10.1007/s40258-025-00952-1: 729-scenario full factorial over six factors including covariate overlap and strength; identifies the region where adjustment is worse than not adjusting.. Cites 10.1002/sim.8759: Factorial simulation crossing overlap, effect-modification strength, sample size, missing modifiers and covariate distributions across MAIC, STC and ML-NMR, and reporting where adjustment fails to pay.. Cites 10.1002/jrsm.1511: 162-scenario simulation locating the bias-variance crossover as overlap degrades..
- GPT-5.6 Sol, technical and source-code lens
- partially-solved / overstated. C2, C5, and C6 are correct qualitative statements. Basic theory already identifies the crossover: without relevant effect modification or source-target imbalance, adjustment cannot remove that bias and may add variance; under positivity failure, weighting becomes concentrated while outcome regression relies on extrapolation and model correctness. Exact data-specific MSE improvement cannot be learned from observed data without causal and modeling assumptions, but adjustment decisions can still use observed imbalance, prespecified effect modifiers, trial design, and external knowledge. At cpaic commit 9d150e9, the README reports a factorial overlap-and-imbalance comparison of naive bridging, cSTC, and cMAIC, so “no evidence base” is false. That evidence is narrow, and the commit itself says its validation pipeline was not publicly tracked. A broad comparison including ML-NMR, doubly robust estimators, and flexible learners remains open. Cites choxos/cpaic@9d150e9cad88f2e6fbcd94dbb92c3dee3b7c2dca: The package records a factorial comparison across imbalance and overlap for naive bridging, cSTC, and cMAIC, partially characterizing when adjustment changes bias and uncertainty. It does not cover the full estimator set and was not independently reproducible from the public commit..
- Decision path
-
R3:partially-solved(n=2) → R8:overstated-majority(n=2)
How the evidence falls in time
4 papers in the reviewed corpus bear on this problem, published 2020 to 2024. The most recent paper to touch this reports progress on it, and every paper calling it open is older.
It is also being closed in pieces: partial results come from more than one paper in more than one year, each covering a different part.
Does the later work answer the earlier work? Put to an independent reviewer with the older and newer claims side by side, it judged that the later work answers a component of it, leaving a named part (high confidence). Later simulations characterize important failure regions for MAIC and STC, including poor overlap, sparse data, asymmetric effect modification, bias, and type-I-error behavior. They do not deliver the requested across-family comparison or assess flexible machine-learning estimators and diagnostic detectability. What remains: A comparative failure map spanning weighting, outcome-model, and machine-learning estimators, linked to observable diagnostics and broader endpoint/settings, remains absent.
| Year | Says | Paper |
|---|---|---|
| 2020 | partly addresses it | Assessing the performance of population adjustment methods for anchored indirect comparisons: A simulation study |
| 2020 | partly addresses it | Comparison of Methods for Estimating Therapy Effects by Indirect Comparisons: A Simulation Study |
| 2022 | confirms it is open | Parametric G-computation for compatible indirect treatment comparisons with limited individual patient data |
| 2024 | partly addresses it | Advancing unanchored simulated treatment comparisons: A novel implementation and simulation study |
Source
Derived from C:L1775-1778, C:L1779 of the reviewed corpus.