Open Problems in Indirect Treatment Comparisons
An audited catalog of what is actually unsolved in ITC and population-adjusted indirect comparison methodology
The field is not short of estimators. It is short of identification discipline, estimand discipline, target-population discipline, and diagnostics that expose how much of an answer is supplied by data rather than by assumption.
This site catalogs the open problems, one entry at a time. Each says what the problem is, why it stays open, what has already been tried, and the most probable route to a solution.
336 problems across 17 categories, carrying 1904 individual claims that were checked.
| Verdict | Entries | |
|---|---|---|
| Confirmed open | 123 | 37% |
| Partially addressed | 145 | 43% |
| Overstated | 54 | 16% |
| Unverifiable | 14 | 4% |
| Not supported | 0 | 0% |
| Resolved since report | 0 | 0% |
Roughly a third of the problems presented as open turn out to be partly addressed already, usually by work the source material does not cite. Those entries name what covers which part.
Start here
Browse the catalog
Every problem in one filterable table. Sort by priority, filter by category or verdict, or search across all fields.
The prioritized agenda
What should change in practice now, what should be built next, and what is high-risk and high-reward.
The method landscape
What each approach identifies, where it breaks, and a decision path from a comparison problem to a defensible method.
How this was verified
The provenance of this material, the audit protocol, and an honest account of what the verdicts do and do not establish.
The organizing distinction
The decisive division in this field is anchored versus unanchored evidence, not matching versus regression versus multilevel modeling.
In an anchored comparison, randomization still identifies the within-trial relative effect, and population adjustment is mainly about transporting effect modification. In an unanchored or disconnected comparison, the analyst has to reconstruct a causal contrast across studies: every prognostic factor and effect modifier that creates cross-study outcome differences becomes relevant, and residual confounding is generally not testable. Most of the hardest entries in this catalog live on the unanchored side of that line.
Three consequences run through the whole collection:
Adjustment to one trial population is not transport to the decision population. Pairwise matching and regression approaches usually target the population of the trial that supplied aggregate data. Applying that effect to a reimbursement population is a second transport step, and it is commonly hidden.
Weak overlap is not a nuisance diagnostic; it changes the scientific question. When weights are extreme or a model extrapolates beyond its support, the nominal target estimand may cease to be credibly identified. Reporting an effective sample size does not repair that.
In a disconnected network, some assumption carries the information across the gap. Baseline risk, a dose-response curve, a treatment class, a shared component — each answers a different question and imports different bias. That software produced a connected posterior is not evidence that the effect is identified.
Reading a verdict
Every entry carries a verdict recording what happened when its claims were checked against the literature and against two independent external auditors. A verdict of Overstated does not mean the problem is not real; it usually means the underlying issue is real and a claim about its severity was pitched too strongly. Entries marked Resolved since report are kept rather than deleted, because knowing that something was recently closed is as useful as knowing it is open.
The source material for this catalog was produced by a large language model and then audited. How this was verified explains what that means in practice, including where the audit’s own limits are.