next up previous
Next: Studying Discrepancies Up: Evaluating ECG Analyzers Previous: Obtaining Reference Heart Rate Data

Comparing Annotation Files

Once the test annotation files and the reference heart rate annotation files have been obtained, the remainder of the evaluation procedure is straightforward. All of the information needed to characterize the analysis performed by the device under test is encoded in the test annotation files; similarly, all of the information needed to characterize the actual contents of the test signals is encoded in the reference annotation and reference heart rate annotation files. The evaluation procedure thus entails comparison of the test and reference annotation files for each record.

Four programs are provided in the DB Software Package for this purpose:

The DB Software Package also includes three related programs:

To obtain a concise summary of how to use any of these programs, including a list of any command-line options, simply run the program without any command-line arguments. Refer to the ECG Database Applications Guide, which accompanies the DB Software Package, for details.

In most cases, it will be easiest to collect all of the annotation files before beginning the comparison, and then to perform the comparison by typing:

The program asks for the test annotator name, the names of the databases used for testing, and what optional detector outputs should be evaluated.

Only the statistics required by ECAR and EC38 are reported by ecgeval. If more detailed evaluation data are needed, it will be necessary to run bxb, rxr, etc., separately. If file space is extremely limited, it may be necessary to delete each test annotation file after it has been compared against the reference file, before the next test annotation file can be created; in such cases, it may also be necessary to prompt the user to change media containing signal or reference annotation files, or to reset the device under test before beginning each record. Optionally, ecgeval can generate a script (batch) file of commands, which can be edited to accommodate special requirements such as these.

For example, suppose we have obtained a set of test annotation files with the annotator name ``yow'', which we wish to compare against the reference annotation files (annotator name ``atr'')gif and reference heart rate annotation files (annotator name ``htr''). The portion of the evaluation script generated by ecgeval for MIT DB record 100 is:

bxb -r 100 -a atr yow -L bxb.out sd.out
rxr -r 100 -a atr yow -L vruns.out sruns.out
mxm -r 100 -a htr yow -L hr0.out -m 0
epic -r 100 -a atr yow -L -A af.out
    -V vf.out -S st.out stm.out
(The last two lines shown above form a single command. The mxm command gathers statistics on measurement number 0; if other heart rate measurements are defined, mxm should be run once for each such measurement, substituting the appropriate measurement numbers for 0 in the output file name, hr0.out, and the final argument.) Statistics for the remainder of the MIT DB are obtained by repeating these commands, substituting in each the appropriate record names for 100. Once these commands have been run for all of the records, the record-by-record statistics will be found in nine files (bxb.out, sd.out, vruns.out, sruns.out, hr0.out, af.out, vf.out, st.out, and stm.out). The first eight of these files contain one line for each record.gif sumstats can read any of these files, and calculates aggregate performance statistics; to use it, type ``sumstats file'', where file is the name of one of these files. The output of sumstats contains a copy of its input, with aggregate statistics appended to the end. Typically this output might be saved in a file to be printed later, e.g.,
sumstats bxb.out >>report.out

A scatter plot of the ST measurement comparisons performed by epic can be produced using plotstm, the output of which can be printed directly on any PostScript printer. For example, to make a plot file for stm.out, type:

plotstm stm.out >

next up previous
Next: Studying Discrepancies Up: Evaluating ECG Analyzers Previous: Obtaining Reference Heart Rate Data

George B. Moody (
Sat May 24 04:20:05 EDT 1997