Reference QBA software and a minimum reporting standard
Statement
QBA for unanchored PAIC is done with bespoke per-submission scripts and no validated reference package with a benchmark suite and model manifest exists; the nearest artefact the literature auditor found is the analysis code accompanying Ren et al. 2025 (doi:10.1017/rsm.2025.13), a repository with a simulated dataset. Parts of the proposed object are already built: cpaic (choxos/cpaic@9d150e9, an unreleased development package on GitHub, not installable from CRAN) exports estimability labels, effective sample size and weight diagnostics, edge influence, integration-error diagnostics, prior sensitivity and bridge fragility, but no qba(), net-benefit, tipping-surface, partial-identification, benchmark-suite or model-manifest interface. The reporting side is also partly occupied, since Ishak et al. 2025 (doi:10.1007/s40273-025-01503-1) publish a PAIC framework covering estimand definition, method and adjustment-variable selection, application, risk-of-bias assessment and comprehensive reporting, and Leahy et al. 2023 (doi:10.1017/S0266462323002702) already require transparent parameter sourcing and code and data sharing, so what no guideline covers is sensitivity-parameter provenance, dependence among bias parameters, feasibility surfaces and failed fits.
Why it is open
Bespoke scripting means results cannot be reproduced, compared across submissions or checked by a reviewer, and without a benchmark suite there is no way to establish that an implementation is correct, so software correctness is currently taken on trust. A reference API must expose objects that existing PAIC packages have no representation for, including nonidentified sensitivity parameters, feasibility maps, decision-reversal surfaces and partial-identification bounds, so several checklist items require output that does not yet exist. No reporting guideline covers sensitivity-parameter provenance, dependence assumptions among bias parameters, feasibility surfaces or failed fits, so a reviewer has no standard against which to judge a submitted sensitivity analysis, and the proposed checklist has been neither validated nor adopted by any appraisal body. Economic models consume absolute arm-level survival and event probabilities, so a bias scenario can leave the relative effect plausible while implying arm outcomes no clinician would accept, and reporting only the contrast hides that.
What has been tried
| Work | What it contributes |
|---|---|
| Leahy et al. 2023, International Journal of Technology Assessment in Health Care | Covers part of the proposed reporting standard: transparent input sourcing, a-priori specification, sharing of all code and data used to inform and conduct QBA analyses with the respective HTA bodies, elicitation and early engagement. |
| Ishak, Chandler, Liu and Klijn 2025, A Framework for Reliable, Transparent, and Reproducible Population-Adjusted Indirect Comparisons, PharmacoEconomics 43:691-710 | Published PAIC framework covering six elements: estimand definition, PAIC method selection, adjustment-variable selection, application of the adjustment method, risk-of-bias assessment and comprehensive reporting. Occupies part of the reporting-standard space the entry said was empty; does not cover sensitivity-parameter provenance, dependence among bias parameters, feasibility surfaces or failed fits. |
| cpaic, unreleased development package on GitHub (not on CRAN), commit 9d150e9 | Supplies several proposed ingredients: estimability labels, a per-study weight panel (effective sample size, weight-entropy efficiency, coefficient of variation, largest normalized weight, mass in the top 5% of weights), edge influence, integration-error diagnostics, prior sensitivity and bridge fragility. It exports no qba(), net-benefit, tipping-surface, partial-identification, benchmark-suite or model-manifest interface, and its cMAIC returns within-study contrasts rather than bias-adjusted absolute arm outcomes. |
| Ren et al. 2025, Research Synthesis Methods | The nearest artefact to reference QBA software the literature auditor located: analysis code with a simulated dataset accompanying the paper, rather than a validated package with a benchmark suite or model manifest. |
Probable solution or research direction
Assemble the reference API from the diagnostics that already exist rather than specifying it from scratch, and release it so a reader can install and check it: make the sensitivity specification, the mode of analysis and the decision threshold explicit arguments, and have the returned object report the estimand and effect scale, identified versus sensitivity-specified quantities, provenance, feasible versus unsupported scenarios and decision-reversal surfaces, invoking effective sample size, partial-identification bounds, dependence, dominance and cancellation only where they are defined for the analysis at hand. Pair it with a benchmark suite and a model manifest, and embed the checklist items in the output so compliance is a by-product of running the analysis. Extend the published PAIC reporting framework with the QBA-specific items it does not cover rather than proposing a rival checklist, and report bias-adjusted absolute arm outcomes wherever the estimator produces them, using clinical implausibility of implied arm outcomes as grounds for excluding scenarios.
Verification
Verdict. Partially addressed. Part of this problem is addressed by existing work; the entry names which part.
The source said. “QBA for unanchored PAIC is done with bespoke per-submission scripts and there is no validated reference package, benchmark suite or model manifest. The report proposes a fourteen-item minimum reporting checklist covering target population and exact estimand, causal diagram and bias mechanism, classification of each sensitivity variable, every parameter with scale, range or distribution and evidence source, dependence assumptions, interpretation mode, models used, support and feasibility diagnostics throughout the sensitivity region, all uncertainty layers and failed fits, decision thresholds, the proportion of the region leading to each decision, code, seeds, source data and elicitation materials, bias-adjusted absolute arm outcomes, and a statement of what QBA does not prove.”
What was wrong with it. The literature auditor found the claim that no ITC or PAIC reporting guideline exists overstated, because Ishak et al. 2025 (doi:10.1007/s40273-025-01503-1) already publish a PAIC risk-of-bias and reporting framework, so the claim narrows to the QBA-specific items it does not cover, and the requirement to share code and data is already in Leahy et al. 2023 (doi:10.1017/S0266462323002702). GPT-5.6 Sol read cpaic at commit 9d150e9, found several proposed diagnostics already implemented, held that the object’s contents should be required only where they are defined for the analysis at hand, and rejected attributing bias-adjusted absolute arm outcomes to cpaic’s cMAIC, whose README defines cmaic() and cstc() as anchored in the common-comparator sense and whose weighted fit returns within-study contrasts.
Dissent. literature dissented (open): Claim 1 stands: the closest artefact I found is the analysis code accompanying Ren et al. 2025 (a GitHub repository with a simulated dataset), not a validated package with a benchmark suite or model manifest. Claim 7 is already published guidance, not a gap: Leahy et al. 2023 recommend that ‘All code and data used to inform and conduct QBA analyses should be shared with the respective HTA bodies’. Claim 9 is overstated as written, because a PAIC-specific reporting framework the entry does not cite now exists: Ishak, Chandler, Liu and Klijn 2025 (PharmacoEconomics 43:691-710) propose a systematic framework covering six elements of PAIC analyses explicitly including ‘(5) risk-of-bias assessment’ and ‘(6) comprehensive reporting’, motivated by exactly the variability and lack of transparency the entry describes. It does not, as far as I can establish from the abstract, cover sensitivity-parameter provenance, dependence among bias parameters, feasibility surfaces or failed fits, so the narrow version of claim 9 survives, but the sweeping version does not. Claims 2 to 6, 8, 10 and 11 are specifications for a future object and are internally coherent. codex dissented (overstated): C1’s field-wide absence cannot be established from source code or under the instructed non-literature lens. What is source-confirmed is narrower: cpaic 9d150e9 exports bridge_fragility(), prior_sensitivity(), estimability, ESS, edge-influence, weight, and integration diagnostics, but no qba(), net-benefit, tipping-surface, partial-identification, benchmark-suite, or model-manifest API. C2 is a useful design menu, not a universal object contract: ESS is relevant only when weights change, partial-identification bounds only in a bounds mode, and layer dominance or cancellation requires a scale- and model-dependent decomposition. C3 is factually correct only under the report’s grouping of the list into fourteen items. C4’s four categories are neither exhaustive nor mutually exclusive. C5 applies to a probabilistic joint specification; a deterministic grid has constraints or allowed combinations but no correlations to report. C6 should require diagnostics where they vary and apply, not mechanically at every point. C7 is a sound transparency aim, but seeds matter only for stochastic computation and source data may require controlled access rather than public release. C8 is correct. C9 is overstated because the cited Leahy guidance already covers transparent parameter sourcing and code/data disclosure, even if it does not supply this PAIC-specific checklist. C10 is usually valuable for unanchored comparisons. C11 is method-dependent and must not be attributed to current cpaic cMAIC: its README defines cmaic() and cstc() as anchored in the common-comparator sense, and its weighted fit returns within-study contrasts rather than an exported bias-adjusted absolute-arm object. The integrated reference API remains open, but several diagnostics and reporting principles are already solved; this is primarily tooling and standards work, not necessarily a very-high-priority methodological defect.
- Literature and prior-art check
- open / supported-with-caveat. Claim 1 stands: the closest artefact I found is the analysis code accompanying Ren et al. 2025 (a GitHub repository with a simulated dataset), not a validated package with a benchmark suite or model manifest. Claim 7 is already published guidance, not a gap: Leahy et al. 2023 recommend that ‘All code and data used to inform and conduct QBA analyses should be shared with the respective HTA bodies’. Claim 9 is overstated as written, because a PAIC-specific reporting framework the entry does not cite now exists: Ishak, Chandler, Liu and Klijn 2025 (PharmacoEconomics 43:691-710) propose a systematic framework covering six elements of PAIC analyses explicitly including ‘(5) risk-of-bias assessment’ and ‘(6) comprehensive reporting’, motivated by exactly the variability and lack of transparency the entry describes. It does not, as far as I can establish from the abstract, cover sensitivity-parameter provenance, dependence among bias parameters, feasibility surfaces or failed fits, so the narrow version of claim 9 survives, but the sweeping version does not. Claims 2 to 6, 8, 10 and 11 are specifications for a future object and are internally coherent. Cites 10.1007/s40273-025-01503-1: A published PAIC-specific framework covering estimand definition, method and adjustment-variable selection, risk-of-bias assessment and comprehensive reporting. Occupies part of the reporting-standard space claim 9 says is empty; does not cover the QBA-specific items..
- GPT-5.6 Sol, technical and source-code lens
- partially-solved / overstated. C1’s field-wide absence cannot be established from source code or under the instructed non-literature lens. What is source-confirmed is narrower: cpaic 9d150e9 exports bridge_fragility(), prior_sensitivity(), estimability, ESS, edge-influence, weight, and integration diagnostics, but no qba(), net-benefit, tipping-surface, partial-identification, benchmark-suite, or model-manifest API. C2 is a useful design menu, not a universal object contract: ESS is relevant only when weights change, partial-identification bounds only in a bounds mode, and layer dominance or cancellation requires a scale- and model-dependent decomposition. C3 is factually correct only under the report’s grouping of the list into fourteen items. C4’s four categories are neither exhaustive nor mutually exclusive. C5 applies to a probabilistic joint specification; a deterministic grid has constraints or allowed combinations but no correlations to report. C6 should require diagnostics where they vary and apply, not mechanically at every point. C7 is a sound transparency aim, but seeds matter only for stochastic computation and source data may require controlled access rather than public release. C8 is correct. C9 is overstated because the cited Leahy guidance already covers transparent parameter sourcing and code/data disclosure, even if it does not supply this PAIC-specific checklist. C10 is usually valuable for unanchored comparisons. C11 is method-dependent and must not be attributed to current cpaic cMAIC: its README defines cmaic() and cstc() as anchored in the common-comparator sense, and its weighted fit returns within-study contrasts rather than an exported bias-adjusted absolute-arm object. The integrated reference API remains open, but several diagnostics and reporting principles are already solved; this is primarily tooling and standards work, not necessarily a very-high-priority methodological defect. Cites choxos/cpaic@9d150e9cad88f2e6fbcd94dbb92c3dee3b7c2dca: Provides several proposed ingredients, including estimability, ESS and weight diagnostics, edge influence, integration-error diagnostics, prior sensitivity, and bridge fragility, but no unified multi-bias QBA or decision object.. Cites doi:10.1017/S0266462323002702: Covers part of the proposed reporting standard, including transparent input sourcing, a-priori specification, code and data sharing with HTA bodies, elicitation, and early engagement..
- Decision path
-
R3:partially-solved(n=1) → R9:substantiated-single-overstatement:codex
How the evidence falls in time
1 paper in the reviewed corpus bears on this problem, published 2021. Every paper found here asserts the problem is open; none reports progress.
| Year | Says | Paper |
|---|---|---|
| 2021 | confirms it is open | Applying Quantitative Bias Analysis to Epidemiologic Data, Second Edition (Fox, MacLehose, Lash; Springer, Statistics for Biology and Health, 2021) |
Source
Derived from C:L645-681, C:L800-813, C:L815-832, C:L828 of the reviewed corpus.