Evaluation Protocols

Between 1984 and 1987, the Association for the Advancement of Medical Instrumentation (AAMI) sponsored the development of a protocol for the use of the first two of these databases, which was published as an AAMI Recommended Practice.² Between 1990 and 1998, the ambulatory ECG subcommittee of the AAMI ECG committee developed and revised a standard for ambulatory ECG monitors, significant portions of which address the issue of the accuracy of automated analysis performed by some of these devices.³ The ambulatory ECG standard EC38:1998, and the ``testing and reporting performance results'' standard EC57:1998, build on the evaluation protocol adopted for the earlier Recommended Practice (ECAR), incorporating provisions for the use of all five of the databases listed above, with extensions for assessing detection of supraventricular arrhythmias and transient ischemic ST changes. The standard breaks new ground in establishing specific reporting requirements for the performance of automated ECG analyzers on standard tests using the databases listed above.

A significant constraint imposed on evaluators by the EC38 standard is that they must obtain annotation files containing the analysis results of the device under test. Although the device itself need not produce these files, EC38 specifically requires that they be produced by an automated procedure, which must be fully disclosed. The intent of this requirement is to permit reproducible independent evaluations in which neither the proprietary data of the developers (the analysis algorithms) nor that of the evaluators (the test signals and reference annotations) need necessarily to be disclosed. By defining the interface between the developer and the evaluator to be the annotation file, the responsibilities of each party are clearly defined: the developer must make certain that the device's outputs are recorded in the annotation file in the manner intended by the developer, but in the language of the standard; the evaluator must make certain that the algorithms used to compare the device's annotation files with the reference annotation files conform to the specification of the standard. The format and content of these annotation files is specified in detail below. For many existing devices, it may be difficult or impossible to obtain such annotation files without the cooperation of the developers. Newly-designed devices should incorporate the necessary ``hooks'' for producing annotation files.

Please e-mail your comments and suggestions to webmaster@physionet.org, or post them to:

Updated 8 March 2019