Introduction by Sébastien:
For this first reading group, I propose to go to the basics. The objective is to be able to read ROC and PR plots in order to interpret them correctly. There a thousands of papers about the evaluation of classifiers. It would be impossible to have all the knowledge in a single reading group, so I decided to focus on classifiers with two classes, and putting the emphasis mostly on the signification of the points in these spaces, and also on what can arrive to the classifiers they represent if they are combined or tuned. Curves are also a little bit addressed. The distribution of the readings I propose is the following. I only selected papers published after the year 2000. Moreover, most of the selected papers have been cited nearly one thousand times.
- Unachievable region in precision-recall space and its effect on empirical evaluation (Boyd et al., 2012)
- The relationship between precision-recall and ROC curves (Davis et al., 2006)
- The foundations of cost-sensitive learning (Elkan, 2001)
- The geometry of ROC space: Understanding machine learning metrics through ROC isometrics (Flach, 2003)
- Boolean combination of classifiers in the ROC space (Khreich et al., 2010)
- Robust classification for imprecise environments (Provost et al., 2001)
ROC Space seems to be easier to construct and understand, however both ROC and PR plots give similar information, so providing one or the other is pretty much equivalent. Follow the standard that is used in the literature of your application.