Rare events, separation, and zero-event studies in weighted synthesis
Source note. This entry derives in part from a source document whose inline citations were ChatGPT interface tokens rather than references, and so resolve to nothing. Claims below are attributed to that document and were checked independently where possible; anything that could not be independently sourced is marked in the verification trail.
EESS is the reciprocal of the sandwich variance of a weighted log odds, and it differs from ESS times the event rate by E_w[w]E_w[p]/E_w[wp]: below 1 when weights concentrate on high-risk patients, above 1 on low-risk ones (the design had the direction reversed). As a predictor of failure the two were equivalent (AUROC 0.839 and 0.858) and ESS alone was poor (0.662). Their predictive value comes from zero-event analyses; once an estimate exists, non-coverage is near nominal and neither predicts it. Read the study below.
Statement
Rare events and concentrated weights leave the binary likelihood weakly informative, so estimates can separate and become strongly sensitive to the prior or penalty that is then doing the identification work. The failure is narrower than the source said: a contrast-level log-odds estimate is infinite or undefined with a zero cell, but zero counts can enter an arm-based binomial, Bayesian or penalized likelihood without a continuity correction, and a double-zero study carries no conditional odds-ratio information that any likelihood choice can recover. Rare-event machinery does exist in the unweighted network meta-analysis literature, specifically penalized likelihood regression (Evrenoglou et al., arxiv:2109.10221, PMC9805041), which the source asserted without citing. Weight diagnostics still report a patient effective sample size that says nothing about how many events support the contrast, and no effective event sample size convention has been implemented.
Why it is open
Reweighting concentrates the effective sample on few individuals without reducing the number of covariates, so the weighted binary likelihood can flatten exactly where population adjustment is most needed. Where the prior or penalty supplies the identification, the resulting interval is still finite and reads as evidence, and there is no accepted way to report how much of the answer came from the data rather than the penalty. Effective sample size is computed from the weights alone and ignores the outcome, so it cannot detect that the weights landed on patients who contribute no events, and neither an effective event sample size nor a rare-event-plus-overlap workflow has been implemented anywhere the auditors looked. The literature auditor searched for any composition of rare-event likelihoods with MAIC weighting, STC or ML-NMR integration and found none, an absence they could not confirm from a source, so that part is recorded as unverified rather than established.
What has been tried
| Work | What it contributes |
|---|---|
| Evrenoglou, White, Afach, Mavridis and Chaimani, Network meta-analysis of rare events using penalized likelihood regression, Statistics in Medicine (PMC9805041) | penalized-likelihood machinery for rare-event network meta-analysis that avoids continuity corrections and handles zero-event arms; it does not touch population adjustment, so it is the unweighted half the entry said already exists |
| cpaic, unreleased development package on GitHub (not on CRAN), commit 9d150e9, inst/stan/cpaic_binomial.stan, R/fit_checks.R, R/bayesian_diagnostics.R | accepts zero-count aggregate arms in the Bayesian population-adjusted binomial likelihood, rejects two-stage cSTC and cMAIC fits showing separation or non-identification rather than returning a misleading finite contrast, reports entropy, weight concentration and residual balance alongside ESS, and refits under tighter and looser priors as an identification diagnostic; it has no effective event sample size |
Probable solution or research direction
Compose the existing penalized and Bayesian rare-event likelihoods with the weighting and integration steps that population adjustment adds, and report the prior or penalty contribution explicitly instead of only a finite interval. Define and report an effective event sample size alongside the weighted patient effective sample size, paired with local ESS, leverage, influence and density-ratio diagnostics, since the diagnostics that already exist stop at the weights and do not reach the outcome. Drop the expectation that a beta-binomial or noncentral hypergeometric likelihood rescues individual-level separation or double-zero studies, and benchmark the remaining candidates in a simulation that varies event risk independently of nominal sample size, includes single-zero and double-zero studies, and reports type I error, coverage and prior dominance.
Our study
Effective sample size counted in events: right direction, no gain over ESS times the event rate
The question put to it. With rare binary outcomes, does an effective sample size counted in events (EESS) predict when a MAIC interval fails better than the reported ESS multiplied by the event rate, and in which direction do the two diverge?
What it found. EESS is the reciprocal of the sandwich variance of a weighted log odds, and it differs from ESS times the event rate by E_w[w]E_w[p]/E_w[wp]: below 1 when weights concentrate on high-risk patients, above 1 on low-risk ones (the design had the direction reversed). As a predictor of failure the two were equivalent (AUROC 0.839 and 0.858) and ESS alone was poor (0.662). Their predictive value comes from zero-event analyses; once an estimate exists, non-coverage is near nominal and neither predicts it.
Results.
- Median EESS / (ESS p): 0.47 to 0.79 when weights concentrate on high-risk patients, 1.28 to 2.40 on low-risk patients, 1.00 to 1.03 when the shift is in a non-prognostic covariate.
- AUROC for failure (no estimate or non-coverage) over 108,000 analyses: EESS 0.839, ESS x event rate 0.858, ESS 0.662; the registered rule, requiring EESS to be better by 0.05, is refuted.
- No estimate (an arm with no events) in up to 86% of analyses at risk 0.005 and 300 per arm.
- Among analyses with an estimate, non-coverage was 5.7% and the diagnostics’ AUROC 0.54 to 0.59 (exploratory).
Design.
- Estimand
- Target-population marginal log odds ratio, A versus C; failure of its interval
- Methods compared
- event-based ESS (EESS), ESS times the event rate, Kish ESS
- Data-generating mechanism
- Binary-outcome MAIC with one prognostic covariate among three, 108 scenarios crossing event risk (0.005 to 0.1), size and direction of the target shift relative to risk, sample size and effect; EESS, ESS times the event rate and ESS as per-analysis predictors of failure.
- Replicates
- 1000 per scenario, 108,000 in total
What this does not answer. One prognostic covariate, MAIC on means; the penalized, Firth, Bayesian and fail-closed estimators were not run, and unanchored comparisons and multivariable separation are outside the design. Peer review has not been done.
Peer review. Not yet done. The result stands on the study alone until it is.
Read it: PDF · Markdown · OpenDocument · Code and data: studies/OUT-02-effective-event-sample-size · Protocol: registered before the run
Verification
Verdict. Partially addressed. Part of this problem is addressed by existing work; the entry names which part.
The source said. “With rare events, weighting estimators and generalized linear models become unstable and can separate, zero-event and double-zero studies cannot enter a standard log-odds likelihood, and priors or penalties rather than data determine the answer. Diagnostics report a weighted patient effective sample size that can look adequate while the effective number of events is negligible, and no simulation program benchmarks the competing rare-event remedies in this regime.”
What was wrong with it. The claim that rare-event NMA methods already exist was unattributed in the source and is now sourced to Evrenoglou, White, Afach, Mavridis and Chaimani, penalized likelihood regression for rare-event NMA (arxiv:2109.10221, PMC9805041), supplied by the literature auditor. The code auditor found several mechanisms overstated at commit 9d150e9 of cpaic, an unreleased development package on GitHub that is not installable from CRAN: zero-event aggregate arms already enter its Bayesian binomial likelihood, two-stage fits fail closed on separation, weight diagnostics already report entropy, concentration and residual balance beyond ESS, and prior-sensitivity refits already expose weak identification; the same auditor rejected the beta-binomial remedy for individual-level separation and the noncentral hypergeometric remedy for double-zero studies.
Dissent. literature dissented (open): This is a source_unsourced entry. I can source the claim ‘rare-event NMA methods already exist’: Evrenoglou, White, Afach, Mavridis and Chaimani, ‘Network meta-analysis of rare events using penalized likelihood regression’ (Stat Med; PMC9805041, arXiv 2109.10221), which is exactly the penalised-likelihood machinery the entry says exists in the unweighted NMA literature. I searched for any composition of rare-event likelihoods with MAIC weighting, STC or ML-NMR integration and found none; per the audit rules an absence claim cannot be confirmed by not finding anything, so ‘no settled rare-event likelihood has been integrated with population-adjustment machinery’, ‘no PAIC method combines rare-event likelihoods with overlap-aware diagnostics’, ‘no standard effective-event-sample-size convention’, and ‘no existing simulation study varies event risk independently of nominal sample size’ are all unverifiable rather than verified. The mechanistic claims (weighting concentrates mass, ESS ignores the outcome) are sound but I retrieved no paper demonstrating MAIC-induced separation empirically. The internal priority claim is not a literature claim. codex dissented (overstated): C2, C4, C5, C11 through C13, and the broad simulation recommendation in C16 are technically sound. C3 overstates prior or penalty influence as exclusive determination by the prior rather than joint influence from a weak or monotone likelihood. A zero cell makes a standalone saturated contrast-level log-odds estimate infinite or undefined, but zero counts can enter an arm-based binomial likelihood, and one zero cell need not make a shared multi-study effect’s MLE infinite. A double-zero study has no conditional odds-ratio information, so a noncentral hypergeometric likelihood cannot make it informative merely by avoiding a continuity correction. C9 conflates remedies for separation with models for overdispersion or sparse counts; a beta-binomial model does not itself cure individual-level separation. C6 through C8, C14, C15, and C18 are literature-absence or practice claims outside this audit lens. C17 is not an exhaustive operating-characteristic set, and C19 merely reproduces the agenda’s priority label. At the audited cpaic commit, Bayesian cML-NMR accepts zero-event aggregate arms, two-stage methods fail closed on separation, weight diagnostics go beyond ESS, and prior-sensitivity refits expose weak identification. No effective-event ESS or specialized rare-event-plus-overlap workflow is implemented, so the problem is only partially addressed.
- Literature and prior-art check
- open / supported-with-caveat. This is a source_unsourced entry. I can source the claim ‘rare-event NMA methods already exist’: Evrenoglou, White, Afach, Mavridis and Chaimani, ‘Network meta-analysis of rare events using penalized likelihood regression’ (Stat Med; PMC9805041, arXiv 2109.10221), which is exactly the penalised-likelihood machinery the entry says exists in the unweighted NMA literature. I searched for any composition of rare-event likelihoods with MAIC weighting, STC or ML-NMR integration and found none; per the audit rules an absence claim cannot be confirmed by not finding anything, so ‘no settled rare-event likelihood has been integrated with population-adjustment machinery’, ‘no PAIC method combines rare-event likelihoods with overlap-aware diagnostics’, ‘no standard effective-event-sample-size convention’, and ‘no existing simulation study varies event risk independently of nominal sample size’ are all unverifiable rather than verified. The mechanistic claims (weighting concentrates mass, ESS ignores the outcome) are sound but I retrieved no paper demonstrating MAIC-induced separation empirically. The internal priority claim is not a literature claim. Cites arxiv:2109.10221: Sources the unsourced claim that rare-event NMA methods already exist (penalised likelihood avoiding continuity corrections and handling zero-event arms); it does not touch population adjustment..
- GPT-5.6 Sol, technical and source-code lens
- partially-solved / overstated. C2, C4, C5, C11 through C13, and the broad simulation recommendation in C16 are technically sound. C3 overstates prior or penalty influence as exclusive determination by the prior rather than joint influence from a weak or monotone likelihood. A zero cell makes a standalone saturated contrast-level log-odds estimate infinite or undefined, but zero counts can enter an arm-based binomial likelihood, and one zero cell need not make a shared multi-study effect’s MLE infinite. A double-zero study has no conditional odds-ratio information, so a noncentral hypergeometric likelihood cannot make it informative merely by avoiding a continuity correction. C9 conflates remedies for separation with models for overdispersion or sparse counts; a beta-binomial model does not itself cure individual-level separation. C6 through C8, C14, C15, and C18 are literature-absence or practice claims outside this audit lens. C17 is not an exhaustive operating-characteristic set, and C19 merely reproduces the agenda’s priority label. At the audited cpaic commit, Bayesian cML-NMR accepts zero-event aggregate arms, two-stage methods fail closed on separation, weight diagnostics go beyond ESS, and prior-sensitivity refits expose weak identification. No effective-event ESS or specialized rare-event-plus-overlap workflow is implemented, so the problem is only partially addressed. Cites choxos/cpaic@9d150e9cad88f2e6fbcd94dbb92c3dee3b7c2dca:inst/stan/cpaic_binomial.stan#L19-L20: The Bayesian population-adjusted arm likelihood permits an aggregate arm with zero observed events without a continuity correction.. Cites choxos/cpaic@9d150e9cad88f2e6fbcd94dbb92c3dee3b7c2dca:R/fit_checks.R#L1-L14: Two-stage cSTC and cMAIC fits with separation or non-identification are rejected instead of yielding a misleading finite contrast.. Cites choxos/cpaic@9d150e9cad88f2e6fbcd94dbb92c3dee3b7c2dca:R/fit_checks.R#L128-L152: The package reports entropy, weight concentration, and residual balance in addition to ESS, although it still lacks an event-information ESS.. Cites choxos/cpaic@9d150e9cad88f2e6fbcd94dbb92c3dee3b7c2dca:R/bayesian_diagnostics.R#L141-L157: The package provides an empirical check for whether weakly identified contrasts move materially under tighter or looser priors..
- Decision path
-
R3:partially-solved(n=1) → R9:substantiated-single-overstatement:codex
How the evidence falls in time
10 papers in the reviewed corpus bear on this problem, published 2016 to 2023. Partial progress is on the record, and later work still calls this open. Both belong in view.
It is also being closed in pieces: partial results come from more than one paper in more than one year, each covering a different part.
| Year | Says | Paper |
|---|---|---|
| 2016 | confirms it is open | Implementing informative priors for heterogeneity in meta-analysis using meta-regression and pseudo data |
| 2017 | confirms it is open | Network meta-analysis: application and practice using Stata |
| 2018 | confirms it is open | A design-by-treatment interaction model for network meta-analysis and meta-regression with integrated nested Laplace approximations |
| 2019 | partly addresses it | Network meta-analysis of rare events using the Mantel-Haenszel method |
| 2020 | confirms it is open | Comparison of exclusion, imputation and modelling of missing binary outcome data in frequentist network meta-analysis |
| 2021 | confirms it is open | Pattern-mixture model in network meta-analysis of binary missing outcome data: one-stage or two-stage approach? |
| 2022 | partly addresses it | Network meta-analysis of rare events using penalized likelihood regression |
| 2022 | confirms it is open | Letter to the Editor Regarding “Network Meta-analysis of Food and Drug Administration-approved Treatment Options for Adults with Aquaporin-4 Immunoglobulin G-positive Neuromyelitis Optica Spectrum Disorder” |
| 2023 | confirms it is open | A Critical Appraisal of Matching-Adjusted Indirect Comparisons in Spinal Muscular Atrophy |
| 2023 | confirms it is open | A Bayesian (meta-)regression model for treatment effects on the risk difference scale |
Source
Derived from A:L72, A:L180, C:L214-215, C:L248, C:L254, C:L1517-1518, C:L1517-1519, C:L1526-1529, C:L1526-1530, C:L1526-1533, C:L1535 of the reviewed corpus.