Selective non-publication of commissioned indirect comparisons is unmeasured
Statement
Publication bias in evidence synthesis is handled at the level of the primary trial, through funnel plots, comparison-adjusted funnel plots, selection models fitted inside a network and structured appraisal of missing evidence. Nothing operates at the level of the synthesis itself. A survey of the contracting firms that build indirect comparisons for industry found that 19 responding companies had conducted 476 network meta-analyses of which 102 were published, that sponsor refusal was the most common specified reason for having no plan to publish among the 129 unpublished analyses whose reasons were given, that 13 of those had been used in health technology assessment submissions rather than published, and that a protocol was publicly available for 11 of 162 published company-affiliated network meta-analyses. The denominator is therefore unobservable: nobody knows how many indirect comparisons were run on a given treatment, on which outcomes, or in which direction the withheld ones pointed. No registration requirement, agency disclosure rule or bounding method exists at this level, so an evidence map, a meta-research study or a decision that treats published network meta-analyses as an unbiased sample of the network meta-analyses performed cannot check that assumption or quantify its failure. The two conventional remedies both act on trials: funnel-plot asymmetry reads the studies inside a synthesis, and trial registries record trials rather than syntheses.
Why it is open
The suppressed object is an analysis rather than a trial, so it leaves no trace in any register that exists: review registration is voluntary and, on the measured evidence, a publicly available protocol exists for only a small minority of published company-affiliated analyses, and no regulator or reimbursement agency publishes an inventory of the indirect comparisons submitted to it. Every trial-level detection method reads a pattern among the studies within one synthesis, so none can see a synthesis that was never written up; there is no analogue of small-study asymmetry at the level of a corpus of analyses. Whether the non-publication decision followed the results is unknown even in the one survey that measured the rate, because the firms responsible for two-thirds of the conducted analyses gave no reasons, so the selection mechanism cannot be characterized let alone modeled. A bound would need an assumption about how the withheld analyses differ from the published ones, and there is no dataset in which both are observed against which to calibrate that assumption. The parties who could make the denominator observable are the sponsors and the agencies, so the fix is a governance action rather than a statistical one, and no methods group has an incentive to own it.
What has been tried
| Work | What it contributes |
|---|---|
| Schuit and Ioannidis 2016, Systematic Reviews | The only measurement of the phenomenon: 162 of 822 network meta-analysis papers had contracting-company authors, 93 firms were contacted and 37 responded, 19 disclosed 476 conducted analyses of which 102 were published, sponsor refusal was the leading specified reason for having no plan to publish, and a public protocol existed for 11 of 162 published company-affiliated analyses. |
| Lundh, Lexchin, Mintzes, Schroll and Bero 2017, Cochrane Database of Systematic Reviews | Establishes at trial level that industry sponsorship is associated with results and conclusions favoring the sponsor; supplies the direction of the mechanism that is assumed, but not shown, to operate one level up at the synthesis. |
| Page, Sterne, Higgins and Egger 2021, Research Synthesis Methods | Review of methods for investigating and dealing with publication and reporting biases in meta-analysis; maps the state of the art and shows that every method in it takes the studies within a synthesis as its unit. |
| Page, Sterne, Boutron et al. 2023, BMJ (ROB-ME) | Structured tool for judging risk of bias due to missing evidence in a meta-analysis; the closest existing instrument, again aimed at missing studies rather than missing syntheses. |
| Chaimani and Salanti 2012, Research Synthesis Methods | Comparison-adjusted funnel plot, the network-level detection tool; requires the studies inside the network and therefore cannot register a network that was never published. |
| Mavridis, Welton, Sutton and Salanti 2014, Statistics in Medicine | A selection model fitted inside a network; shows what the correction machinery looks like when the selection unit is the trial, and therefore what would have to be re-derived if the selection unit were the analysis. |
| Hutton et al. 2015, Annals of Internal Medicine (PRISMA-NMA) | Requires a published network meta-analysis to report its registration and protocol; the natural hook for a registration requirement, but it binds only analyses that reach publication and places no obligation on commissioned analyses that do not. |
Probable solution or research direction
Make the denominator observable before trying to model it. The cheapest route is agency-side, since reimbursement bodies already receive indirect comparisons in submissions: publishing the inventory of analyses submitted, including those withdrawn or superseded, would convert an unknown denominator into a measured one for exactly the decisions that matter most. The second route is a registration requirement for commissioned syntheses with the status trial registration has, recorded before the analysis is run and naming the treatments, outcomes and comparators. Failing both, derive a sensitivity bound: treat publication of the synthesis as a selection event, parameterize it by the probability that an analysis of a given direction and magnitude is published, and report how far the published corpus in a therapy area can be displaced across a plausible range of that parameter, calibrating the range against the one measured publication fraction rather than assuming it. A companion empirical study should compare estimates in published company-affiliated network meta-analyses against those in the corresponding health technology assessment submissions, which is the only setting where the same analysis is sometimes visible in both states.
Verification
Verdict. Confirmed open. One reader proposed this and the reviewer accepted it as a selection process acting on commissioned syntheses rather than on included trials, which trial-level diagnostics cannot see. The base is a single survey with a 40 percent response rate and self-reported counts, and the entry does not hide that: it establishes that most commissioned analyses go unpublished but not that the decision followed the results.
Source
Derived from reading:nma-38 of the reviewed corpus.