When a multi-label classifier makes a prediction — say, flagging a patient record for Diabetes, Hypertension, and COVID-19 — how confident should you be? This question is harder than it looks. In a single-label setting, the probability score attached to a prediction is a straightforward measure of confidence. In the multi-label world, it gets complicated fast.

A new paper by Laurence A. F. Park (Western Sydney University) and Jesse Read (École Polytechnique) takes a rigorous look at this problem, testing seven candidate functions for estimating expected accuracy from a multi-label probability distribution — and finding clear winners depending on how accuracy is measured.