Diagnostics & empirical validation

26 open problems

Balance and fit can look excellent under severe residual confounding. What would actually falsify the analysis?

ID Problem Priority Verdict Maturity prank
DIA-01 No diagnostic can falsify unmeasured effect modification in PAIC Very high Overstated Emerging 1
DIA-02 No agreed ESS definition and no support-geometry diagnostics Very high Partially addressed Promising 1
DIA-03 PAIC diagnostics are not evaluated as classifiers of realized error Very high Overstated Promising 1
DIA-04 ADEMP has no identification map for simulation estimands Very high Overstated Emerging 1
DIA-05 No shared benchmark platform with common scenarios, seeds and truth Very high Confirmed open Promising 1
DIA-06 Failure signatures across PAIC estimator families are uncharacterized Very high Overstated Emerging 1
DIA-08 PAIC simulations generate only shared, linear effect modification Very high Overstated Promising 1
DIA-09 Narrow outcome families and hazard-ratio-only survival scoring Very high Confirmed open Promising 1
DIA-11 No scenario library or performance measures for estimability failure Very high Partially addressed Emerging 1
DIA-12 No study ranks competing bridges by worst-case misspecification Very high Overstated Emerging 1
DIA-13 QBA bias mechanisms are not crossed with the primary factors Very high Confirmed open Emerging 1
DIA-14 QBA is judged on oracle recovery, not honest classification Very high Partially addressed Emerging 1
DIA-16 No curated public suite for bridge-deletion validation with IPD Very high Overstated Promising 1
DIA-17 Held-out IPD is not used to benchmark transport against truth Very high Confirmed open Promising 1
DIA-07 Scenario grids omit network topology and information placement High Partially addressed Promising 2
DIA-10 Data access is simulated as benign rather than informative High Partially addressed Promising 2
DIA-15 Coding-error bias, rank reversal and drift calibration go unscored High Overstated Emerging 2
DIA-18 Constancy assumptions are never tested across calendar time High Partially addressed Promising 2
DIA-24 No reference distribution says when inconsistency is large High Partially addressed Promising 2
DIA-25 Rival study-contribution definitions feed the same certainty rating High Partially addressed Established 2
DIA-26 No validated diagnostic localizes inconsistency to a design High Partially addressed Promising 2
DIA-19 NMA outlier and influence screens have no calibrated null Medium-high Confirmed open Promising 3
DIA-20 Transitivity dissimilarity screens have no calibrated threshold Medium-high Confirmed open Emerging 3
DIA-21 Competing contribution measures disagree and none is agreed correct Medium-high Confirmed open Established 3
DIA-22 Empirical method comparisons exclude the networks where methods diverge Medium-high Confirmed open Established 3
DIA-23 No convention requires network simulations to score basic parameters Medium-high Confirmed open Established 3
No matching items

Back to the full catalog