Prespecification and outcome-driven choice of sensitivity ranges
Statement
Sensitivity parameters, their scales, ranges, distributions and dependence assumptions should be prespecified where feasible, and this is already published HTA guidance rather than a new proposal: Leahy et al. 2023 (doi:10.1017/S0266462323002702) recommend that input parameters be identified through a systematic and transparent approach and specified a priori in a protocol or analysis plan, with code and data shared with HTA bodies and early engagement on the nature of the QBA. Ranges tuned after seeing which values preserve statistical significance or cost-effectiveness must not be presented as prespecified, although a disclosed post hoc tipping search is a legitimate exploratory analysis. What is missing is enforcement rather than the principle: no dedicated registration or auditable tooling makes a post hoc range change visible in the record, and no QBA-specific correction repairs undisclosed data-dependent selection over models and ranges, even though a specification can already be time-stamped in an ordinary protocol or registry and general selective-inference, multiverse and worst-case methods exist.
Why it is open
QBA multiplies analyst degrees of freedom because every bias parameter has a scale, a range, a distribution and a dependence structure that are chosen rather than estimated, and the analysis is typically performed by the sponsor after the primary result is known. A reviewer reading the submission cannot distinguish a prespecified range from one tuned until the conclusion survived, because the tuning leaves no trace in the output and nothing in the record time-stamps the specification unless the analyst volunteers it. No statistical correction can repair an undisclosed choice of bias model and range, so the guarantee has to come from governance and auditable tooling rather than from inference, and transparency currently depends entirely on voluntary disclosure.
What has been tried
| Work | What it contributes |
|---|---|
| Leahy et al. 2023, International Journal of Technology Assessment in Health Care | HTA workshop recommendations requiring that input parameters be identified through a systematic and transparent approach and specified a priori in a protocol or analysis plan, that evidence be collected transparently, that all code and data be shared with HTA bodies, and that decision makers be engaged early. Already covers the prespecification principle; supplies no registration mechanism or correction for selection over ranges. |
Probable solution or research direction
Operationalize the guidance that already exists rather than restate it: time-stamp the sensitivity specification, including scales, ranges, distributions and dependence assumptions, in the protocol or analysis plan lodged with the decision maker before results are known, and engage the appraisal body early as the workshop recommends. Build auditable tooling that records the specification and any later change so a post hoc range change is visible in the record alongside reproducible code, seeds and elicitation materials. Label a post hoc tipping search as exploratory and disclose the full search rather than presenting it as confirmatory, using existing selective-inference or multiverse reporting where a disclosed search has to be summarized.
Verification
Verdict. Partially addressed. Part of this problem is addressed by existing work; the entry names which part.
The source said. “Sensitivity parameters should be prespecified where feasible, and the analyst must not choose the range after seeing which values preserve statistical significance or cost-effectiveness. No registration infrastructure exists for sensitivity parameters and no statistical adjustment exists for selection over ranges.”
What was wrong with it. Both auditors found the entry proposing what Leahy et al. 2023 (doi:10.1017/S0266462323002702) already recommend, so prespecification is settled guidance and only the absence of registration infrastructure and of a correction for selection over ranges is genuinely open. GPT-5.6 Sol added that a sensitivity specification can already be time-stamped in an ordinary protocol, analysis plan or general registry without a QBA-specific field, that general selective-inference, multiverse and worst-case methods exist even though none repairs undisclosed choices, and that a disclosed post hoc tipping search is legitimate exploratory work.
Dissent. codex dissented (overstated): C1 is correct and is already explicit HTA guidance in the cited Leahy recommendations. C2 is correct for covert outcome-driven tuning intended to preserve a conclusion, but post hoc tipping searches or alternative ranges are legitimate exploratory analyses if labeled as such and the full search is disclosed. C3 is supported by the cited workshop recommendations. C4 is too absolute: a sensitivity specification can be time-stamped in an ordinary protocol, analysis plan, or general registry even without a QBA-specific registry field. General simultaneous-inference, selective-inference, multiverse, and worst-case methods also exist, although no universal statistical correction can repair undisclosed choice of model and range. The remaining gap is dedicated governance and auditable tooling, not the prespecification principle itself and not a very-high-priority unresolved statistical method.
- Literature and prior-art check
- open / supported-with-caveat. Retrieved the Leahy et al. 2023 recommendations. Claim 3 is verified almost item by item: ‘Input parameters should be identified through a systematic and transparent approach and specified a priori in a protocol or analysis plan’; ‘The evidence used to inform the analysis must be collected in a transparent manner and have sufficiently high accuracy’; ‘All code and data used to inform and conduct QBA analyses should be shared with the respective HTA bodies’; ‘Early engagement with HTA bodies to align on the nature of the QBA is recommended’; and expert elicitation appears as recommendation 3 and 5. The caveat is that this makes claims 1 and 2 restatements of existing published HTA guidance rather than open problems: prespecification of bias parameters in a protocol is already the recommendation of a 2023 multi-stakeholder expert workshop. What is genuinely open is only claim 4, the absence of registration infrastructure and of any statistical correction for selection over ranges, which I could not falsify and which is the part worth publishing. The entry should be reframed so it does not read as proposing what Leahy et al. already recommended.
- GPT-5.6 Sol, technical and source-code lens
- partially-solved / overstated. C1 is correct and is already explicit HTA guidance in the cited Leahy recommendations. C2 is correct for covert outcome-driven tuning intended to preserve a conclusion, but post hoc tipping searches or alternative ranges are legitimate exploratory analyses if labeled as such and the full search is disclosed. C3 is supported by the cited workshop recommendations. C4 is too absolute: a sensitivity specification can be time-stamped in an ordinary protocol, analysis plan, or general registry even without a QBA-specific registry field. General simultaneous-inference, selective-inference, multiverse, and worst-case methods also exist, although no universal statistical correction can repair undisclosed choice of model and range. The remaining gap is dedicated governance and auditable tooling, not the prespecification principle itself and not a very-high-priority unresolved statistical method. Cites doi:10.1017/S0266462323002702: Already recommends systematic, transparent, a-priori specification of QBA inputs in a protocol or analysis plan, along with transparent evidence, reproducible code, elicitation, and early HTA engagement..
- Decision path
-
R3:partially-solved(n=1) → R9:substantiated-single-overstatement:codex
How the evidence falls in time
7 papers in the reviewed corpus bear on this problem, published 2012 to 2026. Partial progress is on the record, and later work still calls this open. Both belong in view.
This problem is recurrent: three or more papers spanning at least eight years assert it independently. Sustained restatement by authors who mostly do not cite each other is the strongest evidence this reading can offer that a gap is real rather than one group’s framing.
| Year | Says | Paper |
|---|---|---|
| 2012 | confirms it is open | A fully Bayesian application of the Copas selection model for publication bias extended to network meta-analysis |
| 2014 | confirms it is open | Incorporating alternative design clinical trials in network meta-analyses |
| 2018 | confirms it is open | Allowing for informative missingness in aggregate data meta-analysis with continuous or binary outcomes: Extensions to metamiss |
| 2021 | partly addresses it | Applying Quantitative Bias Analysis to Epidemiologic Data, Second Edition (Fox, MacLehose, Lash; Springer, Statistics for Biology and Health, 2021) |
| 2021 | confirms it is open | How robust are findings of pairwise and network meta-analysis in the presence of missing participant outcome data? |
| 2022 | confirms it is open | Residual Disease After Primary Surgical Treatment for Advanced Epithelial Ovarian Cancer, Part 2: Network Meta-analysis Incorporating Expert Elicitation to Adjust for Publication Bias |
| 2026 | confirms it is open | Between Rigor and Relevance: Why the EU HTA Guidelines on Indirect Comparisons Miss the Mark |
Source
Derived from C:L694-695 of the reviewed corpus.