No agreed ESS definition and no support-geometry diagnostics
Source note. This entry derives in part from a source document whose inline citations were ChatGPT interface tokens rather than references, and so resolve to nothing. Claims below are attributed to that document and were checked independently where possible; anything that could not be independently sourced is marked in the verification trail.
It matters. Two source populations that remove the same mass at the same threshold, one in the effect-modifying coordinate and one in a prognostic coordinate, give panels that agree to within 1.6% while one estimate is biased by 0.016 on the marginal risk difference and the other by less than 0.0001, and material error is 24 times as frequent in the first. Across the two placements every panel member sits within 0.005 of chance; region-specific ESS separates them almost perfectly and sliced optimal transport cost at 0.73, but a convex hull distance maximized over coordinates is as blind as the panel. The registered within-arm primary outcome reads the other way (panel 0.574, best geometric 0.592, rule returns refuted) because within one arm the hole placement is fixed and the outcome measures sampling noise, not geometry. Both readings are reported. Balance on the omitted moment, the comparator most likely to overturn the headline, was never fairly tested. Read the study below.
Statement
A global effective sample size is a scalar function of the weight distribution, so it does not encode where in covariate space the support lies or whether the supported region is relevant to the decision contrast. Reporting is demonstrably incomplete in practice: of 117 oncology MAICs reviewed by Farinasso et al. 2025 (doi:10.1016/j.vhri.2025.101088), only three fulfilled all NICE recommendations and the distribution of weights was among the least reported items. The claim that software reports ESS alone and never computes contrast-specific influence is too strong: cpaic, an unreleased development package on GitHub that is not on CRAN, already exports a weight panel with effective sample size, entropy efficiency, weight variation, concentration and residual balance, plus an edge influence function for a named treatment contrast. What remains missing is a definition set agreed across packages and the support-geometry diagnostics that no package computes: region-specific ESS, convex-hull distance, density-ratio and optimal-transport cost, out-of-distribution maps, and balance on omitted moments.
Why it is open
Effective sample size is the squared sum of the weights divided by the sum of their squares, so it is invariant to where the weights sit in covariate space; the same value can come from a broad shifted distribution or from a decision-relevant subgroup with no source support at all. Balance is conventionally reported on the moments that were matched, which is tautological, while balance on the omitted moments is the informative check and is usually skipped. Because no body owns the definitions and at least three competing ESS calculations for population adjustment have been proposed, two analyses of the same evidence can report incomparable numbers, and no empirical knowledge accumulates about which diagnostic values predict bias.
What has been tried
| Work | What it contributes |
|---|---|
| Farinasso et al. 2025, Matching-Adjusted Indirect Comparison Studies in Oncology: A Scoping Review Focused on Reporting Quality | Scoping review of 117 oncology MAICs finding that only three fulfilled all NICE recommendations and that the distribution of weights was among the least reported items; sources the previously unattributed claim that diagnostic practice is heterogeneous and incomplete. |
| Three new methodologies for calculating the effective sample size when performing population adjustment, 2024 | Proposes three competing ESS calculations for population adjustment, evidence that no single ESS definition is fixed across packages and publications. The literature auditor retrieved this at title and venue level only. |
| cpaic (development version), commit 9d150e9 | Unreleased development package on GitHub, not available from CRAN. Exports a weight-quality panel (ESS, entropy efficiency, weight variation, concentration, residual balance) and an edge influence function for a named treatment contrast, documented as approximate for multi-arm networks. |
Probable solution or research direction
Standardize and extend what exists rather than specify a panel from scratch: agree one definition set across packages, starting from the multi-metric weight panel and contrast-specific influence already implemented in cpaic, and add the support-geometry diagnostics no package computes, namely region-specific ESS, convex-hull distance between source and target support, density-ratio and optimal-transport cost, out-of-distribution maps, and balance on omitted as well as fitted moments. Require the agreed panel in publications and submissions. Evaluate each member of the panel in simulation for its ability to predict realized error, and drop the ones that only describe the fit.
Our study
The reported weight panel cannot see where support is missing
The question put to it. Every member of the overlap panel MAIC reports (Kish ESS, ESS percentage, entropy efficiency, maximum weight, top-share) is a symmetric function of the weights, so none can depend on where in covariate space the source fails to cover the target. Does that blindness matter in an ordinary covariate law, or do weight concentration and support geometry move together tightly enough that the panel is an adequate proxy?
What it found. It matters. Two source populations that remove the same mass at the same threshold, one in the effect-modifying coordinate and one in a prognostic coordinate, give panels that agree to within 1.6% while one estimate is biased by 0.016 on the marginal risk difference and the other by less than 0.0001, and material error is 24 times as frequent in the first. Across the two placements every panel member sits within 0.005 of chance; region-specific ESS separates them almost perfectly and sliced optimal transport cost at 0.73, but a convex hull distance maximized over coordinates is as blind as the panel. The registered within-arm primary outcome reads the other way (panel 0.574, best geometric 0.592, rule returns refuted) because within one arm the hole placement is fixed and the outcome measures sampling noise, not geometry. Both readings are reported. Balance on the omitted moment, the comparator most likely to overturn the headline, was never fairly tested.
Results.
- The manipulation works in a plain multivariate normal law. With strong effect modification, a hole in the modifying coordinate biases MAIC’s marginal risk difference by 0.0164 and a same-sized hole in a prognostic coordinate by -0.0001, while every panel member agrees between the two arms to within 1.6%. The refuting sentence, that such a counterexample needs configurations that do not occur, fails.
- Across the two hole placements, over 16,000 replicates, the best panel member separates them at 0.505 against 0.5 for chance, standard error 0.004. Region-specific ESS separates them at 1.000 and sliced optimal transport cost at 0.726. Convex hull distance, though it reads position, separates at 0.503 because it is a maximum over coordinates and both arms remove an identical wedge.
- The registered primary, AUROC against material error within the high-modification arm, returns refuted: best panel member 0.574, best geometric 0.592, a gap of 0.018 against a registered materiality of 0.10. Within one arm every replicate shares the hole placement, so the registered outcome asks which replicate drew a bad sample, and a concentration measure answers that as well as anything. The registered outcome was the wrong measurement for the proposition; it is reported as registered, not relabeled.
- Two of three controls failed on parts of the grid. At eight covariates and moderate overlap MAIC’s own error exceeds the 0.03 material threshold in 0.289 of replicates with no hole at all, the same threshold-below-noise defect IDN-05 documented. Leaving the modifier’s second moment unmatched breaks the symmetry the manipulation rests on and the panels diverge by up to 91%. The analysis runs on the 12 of 48 cells the controls leave, all at good overlap.
- Two of the three ESS definitions implemented are the same statistic when weights are normalized, so the measured comparability spread is Kish against entropy ESS: median 1.55 times, 90th percentile 3.50, maximum 15.5 on identical data.
- A post hoc probe prices the rescue of the failed strata by the upper Clopper-Pearson bound: 48,000 source records at dim3/moderate and 192,000 at dim8/moderate. It was not run.
Design.
- Estimand
- Target-population marginal risk difference, and material error above 0.03 as the classifier label
- Methods compared
- Kish ESS, ESS percentage, entropy efficiency, maximum weight, top 5% share, balance on an omitted moment, region-specific ESS, coordinatewise convex hull distance, sliced optimal transport cost
- Data-generating mechanism
- Target covariates independent standard normal in 3 or 8 dimensions; source shifted by 0.25 or 0.60 with a wedge above the 90th percentile removed from the effect-modifying coordinate, from a prognostic coordinate, or not at all. Binary outcome on the logit scale with one modifying coordinate. 48 cells crossing dimension, hole placement, overlap, balancing set and modification strength, source size 16,000. Nine diagnostics scored by AUROC within the high-modification arm and across the two hole placements.
- Replicates
- 2000 per cell, 96,000 replicates; 24,000 on the 12 cells the controls leave
What this does not answer. Balance on the omitted moment, the free check registered as most likely to overturn the headline, was not tested: every cell where it had anything to see was removed by the matched-multiset control. Region-specific ESS was handed the easiest instance, since the region it examines is the region the hole empties, so its near-perfect separation is an upper bound; a hole in a high-modification region the analyst did not think to examine is untested and there it would fail like the panel. No diagnostic is calibrated to a threshold. The analysis subgrid was chosen by a mechanical rule written after the run was read. The realized high-modification hole holds 22% of the modification mass against a registered 25%. MAIC weights only, a binary outcome, normal covariates, one rectangular hole, good overlap only after the null control, and two of the three published ESS definitions. Peer review has not been done.
Peer review. Not yet done. The result stands on the study alone until it is.
Read it: PDF · Markdown · OpenDocument · Code and data: studies/DIA-02-the-whole-weight-panel-is-a-multiset-functional · Protocol: registered before the run
What this project measured
Study studies/DIA-02-the-whole-weight-panel-is-a-multiset-functional. Its primary outcome and controls were fixed before the run; the rule restricting the analysis to the cells the controls leave standing was written after it, as its protocol records. 48 cells, 2000 replicates each, 96,000 replicates. Nine diagnostics: the five reported panel members, all symmetric functions of the weight multiset, against four candidates that read position as well as weight.
The manipulation. Two source populations remove the same mass at the same threshold on exchangeable standard normal coordinates and differ only in which coordinate carries the hole: the one that modifies the treatment effect, or a purely prognostic one. The first is biased by +0.01639 on the target marginal risk difference; the second by -0.00009, which is zero to five decimal places. Material error, meaning the transported effect wrong by more than 0.03 absolute risk, occurs on 0.106 of replicates in the first and 0.005 in the second, against 0.003 with no hole at all.
Result on the registered primary: refuted. Section 7 registered the AUROC of each diagnostic against material error within the modifying-hole arm. There the best panel member reaches 0.574 (max_weight) and the best geometric candidate 0.592 (ess_region), a gap of 0.018 against a registered materiality of 0.10. By the registered rule the panel discriminates as well as the geometric diagnostics, and the proposition is refuted.
Result on the study’s own control: confirmed, emphatically. Section 8 registered a separate check, that at matched multisets every panel member must be numerically identical across the two support arms. Read as a discrimination task over 16,000 replicates, every panel member sits within 0.005 of chance (max_weight 0.505, ess_kish 0.502, entropy_eff 0.501, top_share 0.500) with a standard error of 0.005, while ess_region separates the two arms at 1.000 and optimal-transport cost at 0.726. The blindness is measured, not merely unrejected.
The disagreement is the finding. Within one arm every replicate shares one hole placement, so the only thing varying is sampling noise and the outcome asks which replicate drew a bad sample. A concentration measure answers that about as well as anything. The proposition is about comparing analyses at different placements, and only the cross-arm comparison varies it. The registered primary outcome was the wrong measurement for the question the study was built to ask, and the study’s own control was the right one. That is recorded rather than repaired by relabeling the control as primary after the fact.
The refuting sentence in the design fails. Two analyses differing by 0.016 in bias were built at a panel matched to 2% in an ordinary covariate law, not a contrived one, so the invariance bites in practice and not only in principle.
Two of three controls failed, and the failures cost three quarters of the grid. The matched-multiset control failed wherever a modifier’s second moment was left unmatched, by up to a 0.908 relative difference, because dropping that moment breaks the coordinate symmetry the manipulation rests on. The null control failed at dim8/moderate, where MAIC’s own sampling error exceeds the 0.03 threshold on 0.288 of replicates with no support hole at all, and at dim3/moderate at 0.021. The source size had been registered from a noise floor measured at the middle of the grid, and the middle is not the worst corner. This is the same defect IDN-05 documented, of a threshold set below its own measurement noise, and it is worth naming twice because such a floor reads as a finding. The analysis drops failing strata on the controls’ own evidence rather than on the results, leaving 12 of 48 cells and 24,000 replicates.
The comparability half, quantified for the first time. Three published ESS definitions computed on identical data disagree by a median factor of 1.55, a 90th percentile of 3.50, and a maximum of 15.48. The complaint that two analyses of the same evidence can report incomparable numbers had never been given a size.
The exclusion is not a boundary artifact. dim3/moderate is dropped on a null rate of 0.0213 against a 0.02 cut, a few Monte Carlo standard errors above it. Adding it back moves the panel from 0.5046 to 0.5024 and ess_region from 0.9999 to 0.9997, so the cut is not doing the work.
What this does not answer. Four limits, three of them live. hull_gap is a geometric diagnostic and is blind anyway, at 0.503, because it maximizes over coordinates and both arms remove an identical wedge; reading position is not sufficient if the reading is then aggregated away. balance_omitted, registered as the cheap comparator most likely to overturn the headline, was never given anything to see: the cells where it has content are exactly the ones the matched-multiset control eliminated, so the comparison the design wanted did not run. ess_region’s 1.000 is an upper bound, not an estimate of field performance: the region it examines and the region the hole empties are the same construction, so perfect separation is what the arithmetic requires. Two things keep that from being circular, namely that the region comes from the effect-modifier structure MAIC already requires naming rather than from knowledge of the hole, and that in the prognostic-hole arm the hole lies outside the examined region and the statistic correctly reads near-normal instead of firing on any hole at all. What is untested is a hole in a high-modification region the analyst did not think to examine, which is where this diagnostic would fail exactly as the panel does. And no diagnostic here is calibrated: separating two placements is not a threshold.
Verification
Verdict. Partially addressed. Auditors disagreed and the disagreement was evidence-bearing on both sides. Reopened by the full-text reading: 1 finding from 1 paper survived both the reader and an independent second reviewer, so the verdict moves from unverifiable to partially-addressed, on a single finding.
The source titled this. “ESS reported alone and overlap diagnostics are not standardized” The title says software reports effective sample size alone, which the audit judged too strong given an implemented multi-metric weight panel and contrast-specific edge influence.
The source said. “Overlap is summarized by a single effective sample size, which cannot show the geometry of covariate support, where the analysis is extrapolating, or whether the supported region is relevant to the target contrast. The definitions and displays of overlap, ESS, extrapolation and modifier-balance diagnostics also differ between packages and publications, so two analyses of the same evidence report incomparable numbers.”
What was wrong with it. The claim was unattributed in the source and is now sourced: the literature auditor supplied Farinasso et al. 2025 (doi:10.1016/j.vhri.2025.101088) for heterogeneous and incomplete diagnostic reporting, and a 2024 paper proposing three competing ESS calculations (doi:10.1186/s12874-024-02412-1) for the non-standardization of ESS itself. GPT-5.6 Sol read cpaic at commit 9d150e9 and found a multi-metric weight panel and contrast-specific edge influence already implemented, so the ESS-only claim fails there; the two auditors disagreed on whether the entry as written was supported or overstated, and the weaker reading is published here.
Dissent. literature dissented (open): This entry is source_unsourced; I found real references that support it. Farinasso et al. 2025 scoping-reviewed 117 oncology MAICs and reports that only 3 fulfilled all NICE DSU recommendations and that among the least reported items was the distribution of weights, with mean sample-size reduction 44.9%. That is direct published evidence that diagnostic reporting practice is heterogeneous and suboptimal. Independently, a 2024 BMC Med Res Methodol paper proposes three new ESS calculations for population adjustment, which is itself evidence that no single ESS definition is fixed across packages and publications. I found no PAIC package or publication reporting local (region-specific) ESS, optimal-transport cost, convex-hull distance, out-of-distribution maps, balance on omitted moments, or influence alignment between IPD study edges and the decision contrast; those remain absent, though absence found by search is weaker evidence than the reporting review. codex dissented (overstated): C4 through C7 follow directly from ESS = (sum w)^2 / sum(w^2): the same weights can be reassigned to different covariate locations without changing ESS, despite different local support and contrast relevance. The claim that software reports only ESS and never computes contrast-specific influence is already false for cpaic at the requested commit. It exports weight_diagnostics() with ESS, entropy efficiency, weight variation, concentration, and residual balance, plus edge_influence() for a named treatment contrast. The latter is documented as approximate for multi-arm networks. Local ESS, convex-hull distance, density ratios, and transport costs are complementary diagnostics requiring careful definitions and validation, not established replacements for ESS. Cross-package standardization and fuller support-geometry diagnostics remain open.
Revised by the full-text reading. Applied 2026-07-26 full-text reading of the consolidated library (687 papers).
- Verdict changed
- unverifiable -> partially-addressed 1 surviving finding from: Geometric approaches to assessing the numerical feasibility . New citations: 10.1002/pst.2210
- Literature and prior-art check
- open / supported. This entry is source_unsourced; I found real references that support it. Farinasso et al. 2025 scoping-reviewed 117 oncology MAICs and reports that only 3 fulfilled all NICE DSU recommendations and that among the least reported items was the distribution of weights, with mean sample-size reduction 44.9%. That is direct published evidence that diagnostic reporting practice is heterogeneous and suboptimal. Independently, a 2024 BMC Med Res Methodol paper proposes three new ESS calculations for population adjustment, which is itself evidence that no single ESS definition is fixed across packages and publications. I found no PAIC package or publication reporting local (region-specific) ESS, optimal-transport cost, convex-hull distance, out-of-distribution maps, balance on omitted moments, or influence alignment between IPD study edges and the decision contrast; those remain absent, though absence found by search is weaker evidence than the reporting review. Cites 10.1016/j.vhri.2025.101088: Sources the previously unsourced claim that PAIC diagnostic practice is heterogeneous and suboptimal, with hard numbers on non-reporting of weight distributions.. Cites 10.1186/s12874-024-02412-1: Existence of three competing new ESS calculations for population adjustment supports the claim that ESS definitions are not standardized. Retrieved at title/venue level only; I did not read the full text..
- GPT-5.6 Sol, technical and source-code lens
-
partially-solved / overstated. C4 through C7 follow directly from ESS = (sum w)^2 / sum(w^2): the same weights can be reassigned to different covariate locations without changing ESS, despite different local support and contrast relevance. The claim that software reports only ESS and never computes contrast-specific influence is already false for cpaic at the requested commit. It exports
weight_diagnostics()with ESS, entropy efficiency, weight variation, concentration, and residual balance, plusedge_influence()for a named treatment contrast. The latter is documented as approximate for multi-arm networks. Local ESS, convex-hull distance, density ratios, and transport costs are complementary diagnostics requiring careful definitions and validation, not established replacements for ESS. Cross-package standardization and fuller support-geometry diagnostics remain open. Cites choxos/cpaic@9d150e9cad88f2e6fbcd94dbb92c3dee3b7c2dca: The package implements a multi-metric weight panel and contrast-specific edge influence, refuting the universal ESS-only and no-influence-software claims. It does not standardize definitions across packages or provide all proposed geometric diagnostics.. - Decision path
-
R3:partially-solved(n=1) → R7:counter-and-support-both-present
How the evidence falls in time
8 papers in the reviewed corpus bear on this problem, published 2012 to 2024. Partial progress is on the record, and later work still calls this open. Both belong in view.
This problem is recurrent: three or more papers spanning at least eight years assert it independently. Sustained restatement by authors who mostly do not cite each other is the strongest evidence this reading can offer that a gap is real rather than one group’s framing.
| Year | Says | Paper |
|---|---|---|
| 2012 | confirms it is open | Sample size and power considerations in network meta-analysis |
| 2018 | confirms it is open | Quantifying and presenting overall evidence in network meta-analysis |
| 2019 | confirms it is open | Alternative Weighting Approaches for Anchored Matching-Adjusted Indirect Comparisons via a Common Comparator |
| 2019 | confirms it is open | Population Adjustment Methods for Indirect Comparisons: A Review of National Institute for Health and Care Excellence Technology Appraisals |
| 2019 | confirms it is open | Using Matching-Adjusted Indirect Comparisons and Network Meta-analyses to Compare Efficacy of Brexanolone Injection with Selective Serotonin Reuptake Inhibitors for Treating Postpartum Depression |
| 2022 | partly addresses it | Geometric approaches to assessing the numerical feasibility for conducting matching-adjusted indirect comparisons |
| 2023 | confirms it is open | A Critical Appraisal of Matching-Adjusted Indirect Comparisons in Spinal Muscular Atrophy |
| 2024 | confirms it is open | Response to “Letter to the Editor Regarding ‘Matching-Adjusted Indirect Comparison of the Long-Term Efficacy Maintenance and Adverse Event Rates of Lebrikizumab vs. Dupilumab in Moderate to Severe Atopic Dermatitis’” |
Source
Derived from A:L217, C:L214-215, C:L254, C:L2002-2010 of the reviewed corpus.