Hack-Nation Challenge 06 · E. coli · 5 drugs
Genome Firewall
Per-antibiotic verdicts from a reconstructed genome —
likely to fail ·
likely to work ·
no-call —
with calibrated confidence as held-out frequencies, spectrum-confirmed
evidence, and a target-locus callability gate. The genome doesn't
replace the lab; it tells you which cases to rush and which antibiotic
not to start with.
loading data…
How this model can mislead — and what we did about it
| Failure mode | Failure story | Countermeasure (built, not promised) | Anchor |
|---|---|---|---|
| Homolog leakage | Random splits put near-identical strains in train and test; the model memorizes clones, not resistance biology, and collapses on unseen lineages. | All splits are by skani cluster (99.5% ANI de-dup); every reported number comes from genetically grouped splits, with the held-out genetic group as the headline — never the in-distribution one. | Walsh et al. 2021 DOME, Nat Methods; Hicks et al. 2019, PLoS Comput Biol |
| False confidence | A confidently wrong “likely to work” is the most dangerous output; raw ML scores are miscalibrated and users over-trust automated suggestions. | Platt calibration on a held-out split only; Brier score and reliability curve published per drug; asymmetric conformal no-call band (α susceptible-side 0.02) plus an ANI-distance hard override; confidence is always shown as a bin-level frequency, never a bare number. | Van Calster et al. 2019, BMC Med; FDA CDS guidance (automation bias) |
| Spurious correlation | The model latches onto lineage markers that correlate with resistance in this sample but encode no mechanism (cf. Caruana's pneumonia–asthma model). | Evidence is decoupled from the model: category (i) curated determinant (AMRFinderPlus + allele-aware mapping), category (ii) statistical association only, category (iii) no signal — shown separately, so a mechanism-free prediction is visibly weaker. | Caruana et al. 2015, KDD; Hicks et al. 2019 (r > 0.98 confounding) |
| Absent-target false-susceptible | “No resistance gene found” reported as “susceptible”, including when the drug's target locus was never actually sequenced. | Locus callability gate: the quinolone target loci (gyrA / parC / parE) must be verified present in the assembly before a wild-type reading is trusted; not-called loci force suspicion, never default-susceptible. | BioFire K212727 labeling; Ellington et al. 2017 EUCAST WGS-AST report |