Computers in Cardiology Challenge 2000 Top Scores (Final)
Can sleep apnea be diagnosed based on evidence from the ECG? Based on results obtained by entrants in the Computers in Cardiology Challenge 2000, yes. Each entrant has analyzed a set of 35 recordings, either for screening (in which each recording is classified as containing apnea or not), or for quantitative assessment (in which each minute of each recording is classified as containing apnea or not). Summarized below are the scores of the top finishers in each event. These are outstanding results that demonstrate conclusively that apnea can be detected reliably from the ECG using a variety of methods. (Brief descriptions of the methods used can be viewed by following the links in the tables below to abstracts submitted by each entrant for presentation at Computers in Cardiology 2000; please note that these abstracts were written prior to May 2000, and do not mention results achieved since then.) For details on the CinC Challenge 2000, follow the links at the bottom of this page.
Event 1 (Apnea Screening)
In this event, the five recordings in class B are not counted. The score is the total number of correct classifications of the 20 class A (apnea) and 10 class C (control/no apnea) recordings, so that the maximum possible score is 30. Since four entrants achieved a perfect score, the date of the top-scoring entrant's submission is the tiebreaker.
The top scores in event 1 are:
| Score | Entrant | Date | Entries |
|---|---|---|---|
| 30/30 |
MR Jarvis and PP Mitra Caltech, Pasadena, CA, USA |
3 May | 3 |
| 30/30 |
B Raymond, R Cayton, R Bates, and M Chappell Birmingham Heartlands Hospital, Birmingham, UK |
10 May | 3 |
| 30/30 |
P de Chazal, C Henehan, E Sheridan, R Reilly, P Nolan, and M O'Malley University College - Dublin, Ireland |
17 July | 1 |
| 30/30 |
J McNames, A Fraser, and A Rechtsteiner Portland State University, Portland, OR, USA |
12 September | 3 |
| 29/30 |
PK Stein and PP Domitrovich Washington University School of Medicine, St. Louis, MO, USA |
12 September | 2 |
| 28/30 |
JE Mietus, C-K Peng, PCh Ivanov, and AL Goldberger Beth Israel Deaconess Medical Center, Boston, MA, USA (unofficial entry) |
27 April | 2 |
| 28/30 |
Z Shinar, A Baharav, and S Akselrod Tel-Aviv University, Ramat-Aviv, Israel |
29 April | 1 |
| 28/30 |
MJ Drinnan, J Allen, P Langley, and A Murray Freeman Hospital, Newcastle upon Tyne, UK |
3 May | 1 |
| 28/30 |
C Maier, M Bauch, and H Dickhaus University of Heidelberg, Heilbronn, Germany |
3 May | 2 |
| 28/30 |
M Schrader, C Zywietz, V von Einem, B Widiger, G Joseph Medical School Hannover, Hannover, Germany |
7 August | 8 |
| 27/30 |
C Marchesi, M Paoletti, S Di Gaetano University of Firenze, Firenze, Italy |
28 April | 1 |
| 27/30 |
M Ballora, L Glass, B Pennycook, PCh Ivanov, and AL Goldberger McGill University, Montreal, Quebec, Canada (unofficial entry) |
3 May | 1 |
Each entrant's best score is shown, along with the date when they achieved that score. Many entrants submitted multiple entries, and the 'Entries' shown indicate how many entries were submitted by each entrant up to and including the one that scored highest (later entries, and entries that did not receive scores because of formatting errors were not counted); this gives some sense of how much 'tuning' may have taken place. Entries noted as 'unofficial' came from two of the PhysioNet core research groups, and were therefore not eligible for awards, although they followed all of the rules of the competition.
Only one entry, from PK Stein, was submitted for an unofficial 'three-way score' (the total of correct classifications of all 35 records, including the 5 class B (borderline) records). This is a significantly more difficult task, since the amount of apnea in the class B records must be accurately determined in order to classify them correctly, and recognizing apnea in recordings that have only small amounts of apnea is more difficult than in recordings with frequent apneas. The single entry received a score of 33/35, an excellent result; the only errors occurred when a class A record was put in class B, and a class B record was put in class A.
Event 2 (Apnea Quantification)
In this event, each minute of each of the 35 recordings in the test set must be classified as containing apnea (A) or not (N). The maximum possible score is 17268 (the total number of minutes in the 35 recordings for which reference classifications are available).
The top scores in event 2 are:
| Score | Entrant | Date | Entries |
|---|---|---|---|
| 15994/17268 92.62% |
J McNames, A Fraser, and A Rechtsteiner Portland State University, Portland, OR, USA |
21 September | 4 |
| 15939/17268 92.30% |
B Raymond, R Cayton, R Bates, and M Chappell Birmingham Heartlands Hospital, Birmingham, UK |
22 September | 8 |
| 15432/17268 89.36% |
P de Chazal, C Henehan, E Sheridan, R Reilly, P Nolan, and M O'Malley University College - Dublin, Ireland |
22 September | 15 |
| 15120/17268 87.56% |
M Schrader, C Zywietz, V von Einem, B Widiger, G Joseph Medical School Hannover, Hannover, Germany |
12 September | 9 |
| 15075/17268 87.30% |
MR Jarvis and PP Mitra Caltech, Pasadena, CA, USA |
21 September | 3 |
| 14788/17268 85.63% |
Z Shinar, A Baharav, and S Akselrod Tel-Aviv University, Ramat-Aviv, Israel |
11 May | 1 |
| 14772/17268 85.54% |
C Maier, M Bauch, and H Dickhaus University of Heidelberg, Heilbronn, Germany |
20 September | 5 |
| 14591/17268 84.49% |
JE Mietus, C-K Peng, and AL Goldberger Beth Israel Deaconess Medical Center, Boston, MA, USA (unofficial entry) |
19 May | 3 |
As in event 1, each entrant's best score is shown above, along with the date it was achieved and the number of entries submitted (excluding any entries submitted after the one that received the best score, and any that were not scored because of formatting errors). Notably, four of the top five finishers in this event also achieved perfect scores in event 1. The classification accuracy achieved by the top finishers is comparable to the roughly 90% concurrence of human experts in classification of the original polysomnograms with reference to the full set of signals, including nasal airflow, respiratory effort, and oxygen saturation.
Links:
- Read about the Computers in Cardiology Challenge 2000.
- View or download the training and test data sets.
- Although the competition has ended, you may still submit your results for (unofficial) scoring.
- The correct classifications for event 1 are available here, and those for event 2 are here. Additional information about the records (which was not available to the participants) is here . (But wait! Are you sure you don't want to try the challenge first?)
- Read about Computers in Cardiology 2000, which was held in Cambridge, Massachusetts, 24-27 September 2000.
- Read about the Computers in Cardiology Challenge 2001.
- What should be the topic of next year's challenge? Do you have a data set that can help in creating a challenge? Please send us your ideas!