No support diagnostic aggregates over a PAIC target distribution
Unsupported target mass caused bias only where the effect bent in the unsupported direction (up to 0.29 for G-computation and 0.30 for MAIC with 27% of the target unsupported), and none with linear modification or orthogonal alignment. Weight and covariate diagnostics, identical across those scenarios by construction, discriminated failed analyses at chance within an overlap level (AUROC 0.48 to 0.52). An outcome-informed score reached 0.59 to 0.66, missing the registered margin by 0.01; neither was usable. Trimming to the supported region was unbiased for the restricted estimand. Read the study below.
Statement
Outcome-regression PAIC such as STC, ML-NMR and parametric g-computation can rely on predictions at target covariate profiles that are sparsely represented or unsupported in the source data, and weight-concentration summaries such as ESS do not diagnose that reliance. General support diagnostics do exist and can be applied to a regression: covariate distance, convex-hull membership, density, prediction variance and classical leverage all flag risk, though none certifies that a transported causal effect is identified. What is missing is the PAIC-specific layer: aggregating those measures over a target distribution that is often known only as published moments, reporting them for a target-standardized marginal contrast rather than for a single prediction, and calibrating rules that map a flag to abstention, an explicit sensitivity model, or target redefinition. Two auditors searched and found no application of conformal prediction or representation-space out-of-distribution scoring to STC, ML-NMR or g-computation-based population adjustment; that is a negative search result rather than a verified absence.
Why it is open
A regression has no weights to inspect, so a positivity violation shows up as a numerically well-behaved prediction rather than as an extreme weight, and while prediction variance does rise with leverage in linear and generalized linear models, no model-based interval represents misspecification or nonidentification. The general geometric and leverage diagnostics that could flag the risk do not aggregate cleanly to the quantity a PAIC reports, which is a target-standardized marginal contrast integrated over a target distribution known only through published marginals, in a hierarchical network where the relevant covariate joint distribution must be assumed rather than observed. Conformal and out-of-distribution scores inherit exchangeability-type assumptions that fail precisely under the unsupported covariate shift they are meant to detect, and in any case deliver predictive coverage or novelty rather than identification of a causal contrast. There is no established mapping from any of these scores to a decision, and interval widening is not one, because widening without an explicit sensitivity model, bound or prior does not repair nonidentification.
What has been tried
| Work | What it contributes |
|---|---|
| Remiro-Azocar, Heath and Baio 2022, Parametric G-computation for compatible indirect treatment comparisons with limited individual patient data | counterweight to the entry’s framing of extrapolation as pure hazard: reports that parametric G-computation achieves more precise and more accurate estimates than MAIC particularly when covariate overlap is poor, and yields unbiased marginal treatment effect estimates under no failures of assumptions |
| cpaic, unreleased development package on GitHub (not on CRAN), commit 9d150e9, plot_leverage() | provides plot_leverage(), which is DIC point leverage based on posterior deviance variance; it is a model-fit diagnostic and not the covariate-support or extrapolation-leverage diagnostic this entry proposes |
Probable solution or research direction
Start from the support diagnostics that already exist for regression, namely covariate distance, convex-hull membership, density ratios, prediction variance and classical leverage, and solve the PAIC-specific problem, which is aggregating them over a target distribution reconstructed from published marginals and reporting them for a target-standardized marginal contrast rather than for a single prediction. Note that plot_leverage() in cpaic (https://github.com/choxos/cpaic, commit 9d150e9, an unreleased development package that cannot be installed from CRAN) is DIC point leverage computed from posterior deviance variance, so it is a model-fit diagnostic and not the covariate-support diagnostic proposed here; building the latter is genuinely new work. Treat conformal and out-of-distribution tools as candidates for the flagging step only in their weighted or anomaly-score forms, since ordinary conformal prediction carries no guarantee under the shift being diagnosed. Tie every flag to a stated response, whether abstention, target redefinition, or an explicit sensitivity model with declared bounds, rather than to ad hoc interval widening, and frame the whole battery as diagnostic so that a passing score is never read as certifying extrapolation.
Our study
Unsupported target mass matters only where the effect is modified
This study was designed against OVL-01 and bears on this entry in part. What it does not answer is stated below.
The question put to it. When part of the target population lies outside the source trial’s support, do weight and covariate diagnostics identify the analyses that fail, or is the modification in the unsupported region, which they cannot see, what decides it?
What it found. Unsupported target mass caused bias only where the effect bent in the unsupported direction (up to 0.29 for G-computation and 0.30 for MAIC with 27% of the target unsupported), and none with linear modification or orthogonal alignment. Weight and covariate diagnostics, identical across those scenarios by construction, discriminated failed analyses at chance within an overlap level (AUROC 0.48 to 0.52). An outcome-informed score reached 0.59 to 0.66, missing the registered margin by 0.01; neither was usable. Trimming to the supported region was unbiased for the restricted estimand.
Results.
- Registered verdict NEITHER USABLE: at the largest unsupported mass, alignment-aware score minus Kish ESS AUROC 0.090 (SE 0.013) against a 0.10 margin, both below 0.7.
- Bias with full alignment and bent modification: -0.29 (G-computation) and -0.30 (MAIC); orthogonal alignment at most 0.023; linear modification at most 0.017 at every truncation.
- Kish ESS, unsupported mass and largest weight share: AUROC 0.48 to 0.52 within each overlap level.
- Trimmed G-computation against the restricted estimand: bias at most 0.026, coverage 0.926 to 0.962.
Design.
- Estimand
- Target mean difference over the whole target law; restricted estimand over the supported region
- Methods compared
- MAIC on target means, spline G-computation, G-computation trimmed to the observed range, Kish ESS, unsupported mass, largest weight share and an outcome-informed alignment score as diagnostics
- Data-generating mechanism
- Anchored transport with one covariate truncated in the source; 36 cells crossing unsupported mass, whether the unsupported direction modifies the effect, linear or bent modification and its strength; MAIC, spline G-computation and trimmed G-computation; weight diagnostics and an outcome-informed score scored as classifiers of failure. 1000 replicates per cell.
- Replicates
- 1000 per scenario, 36,000 in total
What this does not answer. Continuous outcome, two covariates; ML-NMR, ML-UMR and NMI not run; failure defined by non-coverage. Peer review has not been done.
Peer review. Not yet done. The result stands on the study alone until it is.
Read it: PDF · Markdown · OpenDocument · Code and data: studies/OVL-01-support-alignment-and-abstention · Protocol: registered before the run
Question. When the target population is known only through published means and SDs, does a support diagnostic for G-computation integrated over the reconstructed law discriminate failed analyses as well as the same diagnostic integrated over the true law, and should it be aggregated as a mean or as a tail functional?
Design. Source trial truncated on the sum of two covariates, so unsupported target mass depends on a correlation the publication omits; linear or bent modification; linear G-computation; hull, Mahalanobis, extrapolated-modification and prediction-variance diagnostics integrated over the true law and over the independence reconstruction, scored against interval failure. 12 cells, 1000 replicates each.
The protocol is registered before any result is seen: read it.
What this project measured
OVL-01 answers the refuting sentence of this entry. Its G-computation arm fitted natural-spline modification, whose prediction variance grows where the source has few patients, and integrated it over the target law including 27% of target mass outside the source’s support. When the modification bent in the unsupported region its interval covered 0.70 with a bias of -0.29 on a 0.04 truth, although it had widened there: prediction variance rises with leverage, but no model-based interval represents misspecification in a region the data do not reach. A diagnostic that multiplied unsupported distance by the estimated interaction discriminated failed analyses better than weight-based ones (AUROC 0.59 to 0.66 against 0.48 to 0.52) and was still not usable alone.
Not answered: how a support diagnostic should be aggregated over a target distribution that is itself reconstructed from published marginals (mean against tail functionals), and how the reconstruction error propagates into that diagnostic. OVL-01 used a known target law.
Verification
Verdict. Partially addressed. Auditors agreed the problem is open and no counterevidence surfaced. Reopened by the full-text reading: 1 finding from 1 paper survived both the reader and an independent second reviewer, so the verdict moves from confirmed-open to partially-addressed, on a single finding.
The source titled this. “No support diagnostics for outcome-regression population adjustment” The title claims no support diagnostics exist for outcome-regression PAIC, which the audit judged overstated since distance, convex-hull, density, prediction-variance and leverage measures all apply.
The source said. “Weight-based overlap diagnostics do not transfer to outcome-regression PAIC such as STC, ML-NMR and g-computation, where an outcome model silently produces predictions at target covariate profiles with no empirical support. The report calls for extrapolation-leverage diagnostics and treats conformal prediction and representation-space out-of-distribution scores as speculative but useful flags whose role is abstention, never certification of extrapolation.”
What was wrong with it. The source’s title and framing claimed that no support diagnostics exist for outcome-regression PAIC; GPT-5.6 Sol judged that overstated, since covariate-distance, convex-hull, density, prediction-variance and classical leverage diagnostics can all be applied, and since model-based standard errors are not necessarily narrow outside support because prediction variance rises with leverage, and it also struck the claim that conformal prediction flags unsupported profiles, because under unsupported covariate shift ordinary conformal prediction loses its exchangeability guarantee and need not flag the profile at all. The literature auditor recorded the load-bearing absence claim as unverifiable rather than verified, and noted that Remiro-Azocar, Heath and Baio 2022 (doi:10.1002/jrsm.1565) treat extrapolation as the capability that lets regression outperform MAIC exactly where overlap is worst.
Revised by the full-text reading. Applied 2026-07-26 full-text reading of the consolidated library (687 papers).
- Verdict changed
- confirmed-open -> partially-addressed 1 surviving finding from: Graphs of study contributions and covariate distributions fo. New citations: 10.1002/jrsm.1292
- Literature and prior-art check
- open / supported-with-caveat. I searched arXiv (full-text and abstract queries pairing conformal/out-of-distribution with transportability, generalizability and external validity) and PubMed, and found no application of conformal prediction or representation-space OOD scoring to STC, ML-NMR or g-computation-based population adjustment, and no extrapolation-leverage diagnostic for outcome-regression PAIC. So the entry’s own hedge that these tools are ‘speculative rather than established’ matches what I could retrieve, and the problem is open. Two caveats. First, the load-bearing negative claim (no support diagnostics exist for outcome-regression PAIC) is an absence claim and no source I retrieved asserts it, so it is unverifiable rather than verified. Second, the entry’s framing of extrapolation as pure hazard is one-sided: Remiro-Azocar et al. 2022 present extrapolation as the capability that lets regression-based adjustment beat MAIC exactly where overlap is worst, reporting that parametric G-computation is ‘more precise and more accurate than MAIC, particularly when covariate overlap is poor.’ The diagnostic gap is real; the implied verdict on outcome regression is not established.
- GPT-5.6 Sol, technical and source-code lens
- open / overstated. C1, the qualified form of C2, C6, and C8 are correct: weight-concentration diagnostics do not diagnose an outcome model, predictions can be computed outside empirical support, and neither conformal nor OOD scoring identifies a causal effect or validates extrapolation. The title and rationale nevertheless overstate the absence of diagnostics. Covariate-distance, convex-hull, density, prediction-variance, and classical leverage diagnostics can be applied to outcome regression, although none certifies causal transport. Model-based standard errors are not necessarily narrow outside support; linear and generalized regression prediction uncertainty can increase with leverage, while still failing to represent model misspecification or nonidentification. C4 is not true of ordinary conformal prediction: under unsupported covariate shift its exchangeability guarantee fails, and a conventional conformal interval need not flag the profile. Specialized conformal anomaly scores or weighted conformal procedures can be heuristic flags under additional assumptions. Representation-space OOD scores are likewise model-dependent heuristics. C7 is incomplete because ad hoc interval widening does not repair nonidentification; widening requires an explicit sensitivity model, bound, or prior. At cpaic@9d150e9, plot_leverage() is DIC point leverage based on posterior deviance variance, not the proposed covariate-support diagnostic. PAIC-specific aggregation of support diagnostics and calibrated abstention rules remains open, but general support diagnostics do exist.
- Decision path
-
R5:default-open → R9:substantiated-single-overstatement:codex
How the evidence falls in time
5 papers in the reviewed corpus bear on this problem, published 2017 to 2023. Partial progress is on the record, and later work still calls this open. Both belong in view.
| Year | Says | Paper |
|---|---|---|
| 2017 | confirms it is open | Network meta-analysis including treatment by covariate interactions: Consistency can vary across covariate values |
| 2018 | partly addresses it | Graphs of study contributions and covariate distributions for network meta-regression |
| 2020 | confirms it is open | Extending inferences from a randomized trial to a new target population (Dahabreh, Robertson, Steingrimsson, Stuart, Hernan; tutorial, arXiv:1805.00550v3) |
| 2022 | confirms it is open | Parametric G-computation for compatible indirect treatment comparisons with limited individual patient data |
| 2023 | confirms it is open | Comparison of indirect treatment methods in migraine prevention to address differences in mode of administration |
Source
Derived from C:L884, C:L908-910, C:L956-958 of the reviewed corpus.