PAIC diagnostics are not evaluated as classifiers of realized error
At the level at which a diagnostic is actually read, one analysis at a time, effective sample size tracks the error a study got by chance far better than the error it got from adjusting on the wrong covariates: AUROC 0.847 against the outcome-noise component and 0.813 against chance covariate imbalance between the source arms, both of which are functions of the covariates, the assignment and the weights, against 0.653 for the transport component that adjustment exists to remove. At the quoted cutoff of 35 it catches between a quarter and a third of material errors in every misspecification stratum, 0.309 where an effect modifier is omitted and the omission is diagnosable in principle, at specificity 0.926. No evaluated fixed cutoff met the registered requirement of sensitivity 0.80 at specificity 0.50 in every stratum. Three of the panel’s members are provably one statistic, since ESS/n = 1/(1+CV^2(w)) exactly and area under the ROC curve is invariant to monotone transformation; a fourth, the post-weighting balance on matched moments, is zero at the solution of the calibration equations and among converged analyses cannot discriminate anything. On a decision-curve basis every rule in the panel is beaten by scrutinizing every analysis, up to an action threshold of 0.3. Read the study below.
Statement
Apart from isolated work relating low effective sample size to error, PAIC diagnostics have not been evaluated as classifiers of realized error with reported discrimination and calibration. Two exceptions exist: a simulation reported at ISPOR Europe 2024 (abstract MSR65) found bias in unanchored MAIC once absolute ESS fell below roughly 30 to 35, and Remiro-Azocar et al. 2021 (doi:10.1002/jrsm.1511) reports that the robust sandwich variance estimator underestimates variability where effective sample sizes are small. Cross-validation is also still reported at a hierarchical level that does not match the prediction task; in cpaic at commit 9d150e9, PSIS-LOO is observation-level and documented as incapable of validating cross-gap prediction, and grouped study-level leave-one-out is not implemented.
Why it is open
Diagnostics are computed from the same data and model that produced the estimate, which does not logically prevent them from predicting error but does mean the predictive value has to be demonstrated rather than assumed. Sensitivity, specificity and calibration are not intrinsic properties of a diagnostic; they are defined only relative to a specified distribution of scenarios and a prespecified threshold for material error, so establishing them requires a simulation program with an agreed scenario set, which the field has not built. Patient-level cross-validation measures within-study accuracy dominated by information the target task does not have, while the decision quantity is a prediction for a new population or study, and importance-sampling diagnostics behave differently at each hierarchical level and are rarely reported per level.
What has been tried
| Work | What it contributes |
|---|---|
| MSR65 Can Low Effective Sample Size in Matching-Adjusted Indirect Comparisons (MAICs) Lead to Bias? Findings From a Simulation Study, ISPOR Europe 2024 / Value in Health abstract | Simulates unanchored MAICs across sample size, event rate and overlap and reports bias once absolute ESS falls below roughly 30 to 35; quantifies one standard diagnostic against realized error. Conference abstract, retrieved at abstract level only. |
| Remiro-Azocar et al. 2021, Methods for population adjustment with limited access to individual patient data: A review and simulation study | Links a diagnostic quantity (small effective sample size) to a documented inferential failure, namely underestimation of variability by the robust sandwich variance estimator. |
| cpaic (development version), commit 9d150e9 | Unreleased development package on GitHub, not available from CRAN. Its PSIS-LOO is observation-level and documented as incapable of validating cross-gap prediction; grouped study-level leave-one-out is not implemented, which is the mismatch the entry describes. |
Probable solution or research direction
Specify a scenario distribution and a material-error threshold first, then score each diagnostic as a predictor of realized error against known truth and report its discrimination and calibration, taking the published ESS-to-bias findings as the starting calibration point rather than the endpoint. Separate descriptive from error-predictive diagnostics in reporting guidance and retain only the latter as evidence. Aggregate log-likelihood contributions at the deployment-relevant unit by adding grouped study-level, comparison-level and subnetwork-level leave-one-out to the observation-level PSIS-LOO that implementations currently provide, and report the diagnostics separately by level.
Our study
Effective sample size tracks the error an analysis got by chance better than the error it got from adjusting on the wrong covariates
The question put to it. Every population adjustment arrives with a panel of numbers beside it: effective sample size, the same figure as a percentage, the largest weight, standardized differences before and after adjustment, a distance between populations. A committee reads them as evidence about whether to believe the estimate, and a widely quoted rule says to worry below an effective sample size of about 30 to 35. Scored as classifiers of realized error, with discrimination and calibration, against known truth and under a declared distribution of analyses, what are they worth?
What it found. At the level at which a diagnostic is actually read, one analysis at a time, effective sample size tracks the error a study got by chance far better than the error it got from adjusting on the wrong covariates: AUROC 0.847 against the outcome-noise component and 0.813 against chance covariate imbalance between the source arms, both of which are functions of the covariates, the assignment and the weights, against 0.653 for the transport component that adjustment exists to remove. At the quoted cutoff of 35 it catches between a quarter and a third of material errors in every misspecification stratum, 0.309 where an effect modifier is omitted and the omission is diagnosable in principle, at specificity 0.926. No evaluated fixed cutoff met the registered requirement of sensitivity 0.80 at specificity 0.50 in every stratum. Three of the panel’s members are provably one statistic, since ESS/n = 1/(1+CV^2(w)) exactly and area under the ROC curve is invariant to monotone transformation; a fourth, the post-weighting balance on matched moments, is zero at the solution of the calibration equations and among converged analyses cannot discriminate anything. On a decision-curve basis every rule in the panel is beaten by scrutinizing every analysis, up to an action threshold of 0.3.
Results.
- Prespecified verdict: the panel does not classify realized error at the thresholds in use. No rule reached sensitivity 0.80 with specificity 0.50 in every misspecification stratum. ESS below 35 reaches sensitivity 0.251 to 0.326 across strata at specificity 0.909 to 0.926.
- The mechanism, registered before the run and holding at the level of a single analysis: effective sample size discriminates the outcome-noise component at AUROC 0.847 and the transport component at 0.653, a gap of 0.194, and the arm-imbalance component at 0.813. The two it reads well are the two that are functions of the covariates, the assignment and the weights, which is what the panel is built from.
- Peer review changed this claim and the paper’s title. Scored against SYSTEMATIC transport bias with the design cell as the unit, rather than against the realized transport component of one replicate, effective sample size reaches 0.808 rather than 0.653. That analysis does not rescue the panel either: the statistic that is identically zero reaches 0.708 on it, because 128 cells is a small sample in which the bias and every diagnostic move along the same overlap axis. Both levels are reported and the title now claims only the replicate-level result.
- The panel is smaller than it looks, by algebra rather than by simulation. ESS, ESS as a percentage and the coefficient of variation of the weights had identical areas under the ROC curve to 0.0e+00 within a fixed source size. The matched-moment balance statistic never exceeded 1.4e-14 and its apparent AUROC of 0.550 is discrimination of floating-point residual, not of balance.
- Silence is where the damage is. Across the 88 cells the ESS rule never fires in, the material-error rate runs from 0.045 to 0.835. The registered version of this claim, a spread across the cells it does fire in, was not met (0.150 against a prespecified 0.30) because those cells are all bad. A first draft called the silent-cell span a stronger version of the registered claim; peer review established that both spans are variation in predictive value, which moves with prevalence and case mix, so neither is a test of whether a threshold transports. That promotion is withdrawn and the leave-one-factor-level-out calibration is the evidence on transportability.
- Decision-curve analysis: at action thresholds up to 0.3, flagging every analysis beats every rule in the panel. From 0.4 upwards three distinct statistics beat both alternatives, relative effective sample size, pre-weighting imbalance and Mahalanobis distance; the absolute cutoff that gets quoted is not among them at any threshold examined.
- The same statistics work when they measure the actual gap. Pre-weighting imbalance predicts the unadjusted comparison’s transport error at AUROC 0.891, against 0.655 for the same statistic applied to MAIC. These diagnostics are informative about the problem and uninformative about the solution: weighting sets to zero the very quantity they measure.
- Our proposal, converting residual imbalance on a measured but unadjusted covariate into the units of the estimand with the source-estimated interaction, reaches 0.937 against the transport component over the diagnosable set where effective sample size as a percentage reaches 0.851. That gain of 0.086 misses the 0.10 we registered and is reported as a miss. Inside the omitted-modifier stratum alone the plain balance check on unmatched covariates beats it, 0.946 against 0.938, because there the covariate is always a modifier and converting imbalance into effect units only adds estimation noise. The conversion earns its keep exactly when the analyst does not know whether the unmatched covariate matters.
- An adversarial critique of the design before the run found two arithmetic errors in the protocol: the dispersion factor was not the pure variance instrument it was described as wherever the cross-moment channel was on, and the target trial’s variance was understated by a factor of two to three. Both were confirmed numerically and corrected before any result was seen.
- Round one of review found the load-bearing claim of the paper wrong, and all three reviewers found it independently. The arm-imbalance component uses no outcome at all, and the noise component’s variance is exactly sigma^2 (1/ESS_A + 1/ESS_C), so a weight statistic is its natural summary rather than merely correlated with it. Round one also found that the calibration split was perfectly confounded with the dispersion factor, because cells are enumerated with that factor varying fastest; leave-one-factor-level-out replaced it and the previous claim that overlap statistics calibrate worse than weight statistics was withdrawn as an artifact.
Design.
- Estimand
- Transported A versus C mean difference in the target superpopulation; the anchored A versus B contrast as a secondary reference
- Methods compared
- MAIC matching means and standard deviations, MAIC matching means only, STC with a heteroskedasticity-consistent variance, no adjustment
- Data-generating mechanism
- An anchored indirect comparison with a continuous outcome and an identity link, chosen so that the realized error of a linear estimator splits exactly into arm imbalance, transport error and outcome noise, so each diagnostic can be scored against each component separately: two of the three are functions of the covariates, the assignment and the weights, which is what the panel is built from, and the third is what adjustment exists to remove. 128 cells crossing an omitted effect modifier, a cross-moment modifier that no baseline table could reveal, overlap, source size, how correlated the omitted modifier is with the adjustment set, and a target dispersion ratio that destroys effective sample size while adding no bias where the modifier structure is correct. 4000 replicates per cell, four estimators. Primary results are reported within misspecification strata rather than over a mixture, so no investigator-chosen frequency enters the headline.
- Replicates
- 4000 per cell, 512,000 datasets and 2,048,000 estimates
What this does not answer. The second half of DIA-03 is untouched: no PSIS-LOO is computed at any hierarchical level and grouped study-level leave-one-out is no more implemented here than in the packages, so the mismatch between the level at which cross-validation is reported and the level at which the prediction is made remains open. DIA-06 is answered only in part; two estimator families and no adjustment are compared on identical replicates, but ML-NMR, NMI, doubly robust estimators and flexible learners are absent. The outcome is continuous with an identity link, which is what buys the exact error decomposition and is also the largest restriction: nothing here transfers to non-collapsible effect measures or time-to-event outcomes, where part of the error is a collapsibility artifact rather than an adjustment failure. Covariates are normal, comparisons are anchored, conditional constancy holds, and target moments are exact. The deployment weights are a declared judgment and nothing in this program measured them, which is why the primary results are within strata.
Peer review. 2 rounds, two independent reviewers. The reports, the point-by-point responses and the editorial decision are published in full: read the review.
Read it: PDF · Markdown · OpenDocument · Code and data: studies/DIA-03-diagnostics-as-classifiers · Protocol: registered before the run · Review: full history
What this project measured
Study studies/DIA-03-diagnostics-as-classifiers, run and decided against a rule registered before the run. 128 cells, 4000 replicates each, four estimators, ten diagnostic rules.
Conclusion: the panel does not classify realized error at the thresholds in use. Material error, meaning the transported effect wrong by more than 0.20, occurs on 0.479 of fitted MAIC replicates under the deployment weights, so the target is close to a coin flip and a useful diagnostic has room to be informative.
Operating points are reported within misspecification strata, so no assumed mixture of misspecification frequencies enters the headline. In the well-specified stratum the rules divide into two useless families: high specificity with little sensitivity (ess 0.326/0.923, max_w 0.350/0.919, bias_hat 0.099/0.962), or high sensitivity bought with false alarms (smd_pre 0.822/0.408, ess_pct 0.742/0.623). The matched standardized difference, smd_matched, fires on no replicate at all (sensitivity 0.000, specificity 1.000): after matching it is zero by construction and carries no information about error. The pattern is essentially unchanged when a modifier is omitted.
Verification
Verdict. Overstated. 2 independent auditors judged the claim as stated too strong. Apart from isolated work relating low effective sample size to bias, PAIC diagnostics have not been evaluated as classifiers of realized error with reported discrimination and calibration, and cross-validation for PAIC is not routinely performed at the hierarchical level matching the prediction task.
The source titled this. “PAIC diagnostics are never scored on whether they predict error” The title says diagnostics are never scored on predicting error, but the audit found two exceptions relating low effective sample size to bias and to variance underestimation.
The source said. “Whether the diagnostics used in PAIC (ESS, balance, overlap, influence, fit statistics) actually predict the error of the resulting estimate, or only describe the fitted model, has never been quantified. Relatedly, cross-validation is reported at a hierarchical level that often does not match the prediction the analysis is making, so a good statistic can accompany a failed transport.”
What was wrong with it. The source said predictive validity has never been quantified; the literature auditor found an ISPOR Europe 2024 simulation abstract identifying an ESS threshold below which bias appears, and Remiro-Azocar et al. 2021 (doi:10.1002/jrsm.1511) tying small ESS to sandwich-variance underestimation. Neither treats a diagnostic as a classifier with discrimination and calibration against known truth, so the narrowed claim stands.
- Literature and prior-art check
- open / overstated. The blanket ‘has never been quantified’ fails for at least one diagnostic. An ISPOR Europe 2024 / Value in Health simulation abstract (MSR65, ‘Can Low Effective Sample Size in Matching-Adjusted Indirect Comparisons (MAICs) Lead to Bias?’) simulated unanchored MAICs of time-to-event and binary outcomes across sample size, event rate and overlap and reported bias when absolute ESS fell below roughly 30-35, irrespective of whether the low ESS came from small n or poor overlap. That is a diagnostic evaluated against realized error with a threshold. Remiro-Azocar et al. 2021 similarly ties a diagnostic to a performance failure, reporting that the robust sandwich variance estimator underestimates variability where ESS is small. Neither treats a diagnostic as a classifier with discrimination and calibration against known truth, and I found nothing on hierarchical-level-matched cross-validation or per-level PSIS reporting for PAIC, so the substantive gap stands. I read only abstract-level material for the ISPOR item, hence the low confidence. Cites https://www.valueinhealthjournal.com/article/S1098-3015(24)05162-3/fulltext: Quantifies, in simulation with known truth, whether a standard PAIC diagnostic (ESS) predicts bias, identifying an approximate ESS threshold below which bias appears. Conference abstract; I retrieved title and venue plus a summary of findings, not the full text.. Cites 10.1002/jrsm.1511: Links a diagnostic quantity (small ESS) to a documented inferential failure (sandwich variance underestimation), i.e. a diagnostic with demonstrated predictive content for one error mode..
- GPT-5.6 Sol, technical and source-code lens
- open / overstated. C2, C4, and C5 are technically correct: descriptive fit is not automatically error-predictive, and patient-level versus study-level validation target different deployments. But computing a diagnostic from fitted data does not logically prevent it from predicting error. Sensitivity, specificity, ROC curves, and calibration are defined only relative to a specified distribution of scenarios and a prespecified material-error threshold; they are not intrinsic properties of a diagnostic. At cpaic commit 9d150e9, PSIS-LOO is explicitly observation-level and documented as incapable of validating cross-gap prediction, while grouped study-level LOO is not implemented. The field-wide claims that predictive validity has “never” been quantified are unsourced. A systematic evaluation remains an open research task, but the report overstates what follows from same-data computation.
- Decision path
-
R5:default-open → R8:overstated-majority(n=2)
How the evidence falls in time
3 papers in the reviewed corpus bear on this problem, published 2013 to 2025. Every paper found here asserts the problem is open; none reports progress.
This problem is recurrent: three or more papers spanning at least eight years assert it independently. Sustained restatement by authors who mostly do not cite each other is the strongest evidence this reading can offer that a gap is real rather than one group’s framing.
| Year | Says | Paper |
|---|---|---|
| 2013 | confirms it is open | Demystifying trial networks and network meta-analysis |
| 2016 | confirms it is open | A threshold analysis assessed the credibility of conclusions from network meta-analysis |
| 2025 | confirms it is open | A process to validate prognostic factors for unanchored matching-adjusted indirect comparison of single-arm trials in oncology: a proof-of-concept study |
Source
Derived from C:L214-215, C:L255, C:L1783 of the reviewed corpus.