Comparison of the GRACE, HEART and TIMI score to predict major adverse cardiac events in chest pain patients at the emergency department

  • Author Footnotes
    1 This author takes responsibility for all aspects of the reliability and freedom from bias of the data presented and their discussed interpretation.
    J.M. Poldervaart
    Correspondence
    Corresponding author at: Julius Center for Health Sciences and Primary care, University Medical Center, Str. 6.101, PO box 85500, 3508AB Utrecht, The Netherlands.
    Footnotes
    1 This author takes responsibility for all aspects of the reliability and freedom from bias of the data presented and their discussed interpretation.
    Affiliations
    Julius Center for Health Sciences and Primary care, University Medical Center Utrecht, Utrecht, The Netherlands
    Search for articles by this author
  • Author Footnotes
    1 This author takes responsibility for all aspects of the reliability and freedom from bias of the data presented and their discussed interpretation.
    M. Langedijk
    Footnotes
    1 This author takes responsibility for all aspects of the reliability and freedom from bias of the data presented and their discussed interpretation.
    Affiliations
    Department of Internal Medicine, Gelderse vallei hospital, Ede, The Netherlands
    Search for articles by this author
  • Author Footnotes
    1 This author takes responsibility for all aspects of the reliability and freedom from bias of the data presented and their discussed interpretation.
    B.E. Backus
    Footnotes
    1 This author takes responsibility for all aspects of the reliability and freedom from bias of the data presented and their discussed interpretation.
    Affiliations
    Department of Emergency Medicine, Medical Center Haaglanden, Den Haag, The Netherlands
    Search for articles by this author
  • Author Footnotes
    1 This author takes responsibility for all aspects of the reliability and freedom from bias of the data presented and their discussed interpretation.
    I.M.C. Dekker
    Footnotes
    1 This author takes responsibility for all aspects of the reliability and freedom from bias of the data presented and their discussed interpretation.
    Affiliations
    Department of Emergency Medicine, Atrium Medical Center, Heerlen, The Netherlands
    Search for articles by this author
  • Author Footnotes
    1 This author takes responsibility for all aspects of the reliability and freedom from bias of the data presented and their discussed interpretation.
    A.J. Six
    Footnotes
    1 This author takes responsibility for all aspects of the reliability and freedom from bias of the data presented and their discussed interpretation.
    Affiliations
    Department of Cardiology, Zuwe Hofpoort Hospital, Woerden, The Netherlands
    Search for articles by this author
  • Author Footnotes
    1 This author takes responsibility for all aspects of the reliability and freedom from bias of the data presented and their discussed interpretation.
    P.A. Doevendans
    Footnotes
    1 This author takes responsibility for all aspects of the reliability and freedom from bias of the data presented and their discussed interpretation.
    Affiliations
    Department of Cardiology, University Medical Center Utrecht, Utrecht, The Netherlands
    Search for articles by this author
  • Author Footnotes
    1 This author takes responsibility for all aspects of the reliability and freedom from bias of the data presented and their discussed interpretation.
    A.W. Hoes
    Footnotes
    1 This author takes responsibility for all aspects of the reliability and freedom from bias of the data presented and their discussed interpretation.
    Affiliations
    Julius Center for Health Sciences and Primary care, University Medical Center Utrecht, Utrecht, The Netherlands
    Search for articles by this author
  • Author Footnotes
    1 This author takes responsibility for all aspects of the reliability and freedom from bias of the data presented and their discussed interpretation.
    J.B. Reitsma
    Footnotes
    1 This author takes responsibility for all aspects of the reliability and freedom from bias of the data presented and their discussed interpretation.
    Affiliations
    Julius Center for Health Sciences and Primary care, University Medical Center Utrecht, Utrecht, The Netherlands
    Search for articles by this author
  • Author Footnotes
    1 This author takes responsibility for all aspects of the reliability and freedom from bias of the data presented and their discussed interpretation.
Open AccessPublished:October 30, 2016DOI:https://doi.org/10.1016/j.ijcard.2016.10.080

      Abstract

      Background

      The performance of the GRACE, HEART and TIMI scores were compared in predicting the probability of major adverse cardiac events (MACE) in chest pain patients presenting at the emergency department (ED), in particular their ability to identify patients at low risk.

      Methods

      Chest pain patients presenting at the ED in nine Dutch hospitals were included. The primary outcome was MACE within 6 weeks. The HEART score was determined by the treating physician at the ED. The GRACE and TIMI score were calculated based on prospectively collected data. Performance of the scores was compared by calculating AUC curves. Additionally, the number of low-risk patients identified by each score were compared at a fixed level of safety of at least 95% or 98% sensitivity.

      Results

      In total, 1748 patients were included. The AUC of GRACE, HEART, and TIMI were 0.73 (95% CI: 0.70–0.76%), 0.86 (95% CI: 0.84–0.88%) and 0.80 (95% CI: 0.78–0.83%), respectively (all differences in AUC highly statistically significant). At an absolute level of safety of at least 98% sensitivity, the GRACE score identified 231 patients as “low risk” in which 2.2% a MACE was missed; the HEART score identified 381 patients as “low risk” with 0.8% missed MACE. The TIMI score identified no “low risk” patients at this safety level.

      Conclusions

      The HEART score outperformed the GRACE and TIMI scores in discriminating between those with and without MACE in chest pain patients, and identified the largest group of low-risk patients at the same level of safety.

      Keywords

      1. Background

      Up to 6.3% of emergency department (ED) visits are related to chest pain [
      • Niska R.
      • Bhuiya F.
      • Xu J.
      National hospital ambulatory medical care survey: 2007 emergency department summary.
      ]. An urgent question in these patients is whether they have an acute coronary syndrome (ACS), as any delay in diagnosis and treatment can have a negative impact on their prognosis [
      • Thygesen K.
      • Alpert J.S.
      • Jaffe A.S.
      • Simoons M.L.
      • Chaitman B.R.
      • White H.D.
      • Joint ESC/ACCF/AHA/WHF Task Force for Universal Definition of Myocardial Infarction
      Third universal definition of myocardial infarction.
      ,
      • Maroko P.R.
      • Kjekshus J.K.
      • Sobel B.E.
      • et al.
      Factors influencing infarct size following experimental coronary artery occlusions.
      ,
      • Windecker S.
      • Kolh P.
      • Alfonso F.
      • et al.
      ESC/EACTS guidelines on myocardial revascularization.
      ]. Normal values of troponin and a normal electrocardiogram (ECG) still do not exclude ACS completely. As a result, many patients presenting with chest pain are currently hospitalized and extensively evaluated with non-invasive stress testing or imaging, or with an invasive coronary angiography [
      • Amsterdam E.A.
      • Kirk J.D.
      • Bluemke D.A.
      • et al.
      Testing of low-risk patients presenting to the emergency department with chest pain: a scientific statement from the American Heart Association.
      ]. However, of all chest pain patients <25% will have an ACS [
      • Amsterdam E.A.
      • Kirk J.D.
      • Bluemke D.A.
      • et al.
      Testing of low-risk patients presenting to the emergency department with chest pain: a scientific statement from the American Heart Association.
      ]. If patients at low risk for ACS could be recognized early in the diagnostic process, it has the potential to reduce patient burden, length of stay at the ED, frequency of hospitalization and costs [
      • Six A.J.
      • Backus B.E.
      • Kingma A.
      • Kaandorp S.I.
      Consumption of diagnostic procedures and other cardiology care in chest pain patients after presentation at the emergency department.
      ,
      • Hoffmann U.
      • Truong Q.A.
      • Schoenfeld D.A.
      • for the ROMICAT-II Investigators
      • et al.
      Coronary CT angiography versus standard evaluation in acute chest pain.
      ,
      • National Institute for Health and Care Excellence
      Chest pain of recent onset Assessment and diagnosis of recent onset chest pain or discomfort of suspected cardiac origin.
      ].
      To diagnose ACS, physicians use patient history, ECG abnormalities, cardiac markers (notably troponin) and several other potential variables [
      • Thygesen K.
      • Alpert J.S.
      • Jaffe A.S.
      • Simoons M.L.
      • Chaitman B.R.
      • White H.D.
      • Joint ESC/ACCF/AHA/WHF Task Force for Universal Definition of Myocardial Infarction
      Third universal definition of myocardial infarction.
      ,
      • Cannon C.P.
      • Brindis R.G.
      • Chaitman B.R.
      • et al.
      2013 ACCF/AHA key data elements and definitions for measuring the clinical management and outcomes of patients with acute coronary syndromes and coronary artery disease.
      ]. International cardiac guidelines state that chest pain patients presenting to the ED should be assessed with a risk stratification tool or risk score [
      • Thygesen K.
      • Alpert J.S.
      • Jaffe A.S.
      • Simoons M.L.
      • Chaitman B.R.
      • White H.D.
      • Joint ESC/ACCF/AHA/WHF Task Force for Universal Definition of Myocardial Infarction
      Third universal definition of myocardial infarction.
      ,
      • VMS veiligheidsprogramma
      Praktijkgids ACS [Document on the Internet].
      ,
      • Roffi M.
      • Patrono C.
      • Collet J.P.
      • et al.
      ESC guidelines for the management of acute coronary syndromes in patients presenting without persistent ST-segment elevation.
      ] and over the years, a number of tools have been developed [
      • Hess E.P.
      • Knoedler M.A.
      • Shah N.D.
      • et al.
      The chest pain choice decision aid; a randomized trial.
      ,
      • Than M.
      • Flaws D.
      • Sanders S.
      • et al.
      Development and validation of the emergency department assessment of chest pain score and 2 h accelerated diagnostic protocol.
      ,
      • Lagerqvist B.
      • Diderholm E.
      • Lindahl B.
      • et al.
      5-year outcomes in the FRISC-II randomised trial of an invasive versus a non-invasive strategy in non-ST-elevation acute coronary syndrome: a follow-up study.
      ,
      • Six A.J.
      • Backus B.E.
      • Kelder J.C.
      Chest pain in the emergency room: value of the HEART score.
      ,
      • Granger C.B.
      • Goldberg R.J.
      • Dabbous O.
      • et al.
      Global registry of acute coronary events investigators. Predictors of hospital mortality in the global registry of acute coronary events.
      ,
      • Body R.
      • Carley S.
      • McDowell G.
      • et al.
      The Manchester acute coronary syndromes (MACS) decision rule for suspected cardiac chest pain: derivation and external validation.
      ,
      • Boersma E.
      • Pieper K.S.
      • Steyerberg E.W.
      • et al.
      Predictors of outcome in patients with acute coronary syndromes without persistent ST-segment elevation; results from an international trial of 9461 patients.
      ,
      • Antman E.M.
      • Cohen M.
      • Bernink P.J.L.M.
      • et al.
      The TIMI risk score for unstable angina/non-ST elevation MI; a method for prognostication and therapeutic decision making.
      ,
      • Christenson J.
      • Innes G.
      • McKnight D.
      • et al.
      A clinical prediction rule for early discharge of patients with chest pain.
      ]. Three well-known risk scores are the GRACE score, the HEART score and the TIMI score, see Table 1 and Supplementary material A.1 [
      • Six A.J.
      • Backus B.E.
      • Kelder J.C.
      Chest pain in the emergency room: value of the HEART score.
      ,
      • Granger C.B.
      • Goldberg R.J.
      • Dabbous O.
      • et al.
      Global registry of acute coronary events investigators. Predictors of hospital mortality in the global registry of acute coronary events.
      ,
      • Antman E.M.
      • Cohen M.
      • Bernink P.J.L.M.
      • et al.
      The TIMI risk score for unstable angina/non-ST elevation MI; a method for prognostication and therapeutic decision making.
      ]. Risk scores combine and weigh various predictors to calculate the risk of ACS for an individual patient. They are based on readily available information collected during the initial work-up of chest pain patients.
      Table 1Variables present in GRACE score, HEART score and TIMI score.
      VariablesGRACE scoreHEART scoreTIMI score
      AgeXXX
      Gender
      HistorySuspicious (physicians' opinion)X
      Severe angina (≥2 events in last 24 h)X
      Use of aspirin last 7 daysX
      Killip classX
      Physical examinationHeart rateX
      Systolic blood pressureX
      ECGST deviationXXX
      Repolarization disorder, LBBB or pacemakerX
      Cardiac arrest at admissionX
      Laboratory resultsCreatinin levelX
      Positive cardiac enzyme
      Troponin or creatin kinase–MB.
      XXX
      Risk factorsPrevious atherosclerotic disease
      Previous atherosclerotic disease was defined as myocardial infarction, coronary arterial bypass grafting, percutaneous coronary intervention, stroke or transient ischemic attack, peripheral artery disease.
      X
      Previous coronary artery disease ≥50%X
      Current smoking
      Smoking in the HEART –impact trial was defined as smoking currently or stopped <3months.
      XX
      Diabetes mellitusXX
      Family history of cardiovascular diseaseXX
      HypercholesterolemiaXX
      HypertensionXX
      Obesity (body mass index >30)X
      ECG: electrocardiogram, LBBB: left bundle branch block.
      low asterisk Troponin or creatin kinase–MB.
      Previous atherosclerotic disease was defined as myocardial infarction, coronary arterial bypass grafting, percutaneous coronary intervention, stroke or transient ischemic attack, peripheral artery disease.
      Smoking in the HEART –impact trial was defined as smoking currently or stopped <3 months.
      Studies directly comparing the performance of risk scores in the same population of chest pain patients report conflicting results in terms of which score is preferred to use at the ED [
      • Mahler S.A.
      • Miller C.D.
      • Hollander J.E.
      • et al.
      Identifying patients for early discharge: performance of decision rules among patients with acute chest pain.
      ,
      • Six A.J.
      • Cullen L.
      • Backus B.E.
      • et al.
      The HEART score for the assessment of patients with chest pain in the emergency department.
      ,
      • Cullen L.
      • Greenslade J.
      • Hammett C.J.
      • et al.
      Comparison of three risk stratification rules for predicting patients with acute coronary syndrome presenting to an Australian emergency department.
      ,
      • Ramsay G.
      • Podogrodzka M.
      • McClure C.
      • Fox K.
      Risk prediction in patients with suspected cardiac pain: the GRACE and TIMI risk scores versus clinical evaluation.
      ]. Furthermore, it is unclear which risk score performs best in identifying patients at “low risk” of ACS, as these patients are candidates for early discharge from the ED (triage role). Therefore, we compared the performance of the GRACE, HEART and TIMI risk scores in identifying best patients at “low risk” of ACS in patients presenting with chest pain at the ED.

      2. Methods

      2.1 Design and study population

      Our study population consisted of patients participating in the HEART-impact trial. In short, this trial investigated the impact of the use of the HEART score in daily practice on safety, quality of life and use of health care resources. The trial was designed as a pragmatic, stepped wedge, cluster randomized trial and compared usual care with HEART score care (i.e. calculation of the HEART score and adherence to recommended patient management depending on the score; see Fig. A.1 and [
      • Poldervaart J.M.
      • Reitsma J.B.
      • Koffijberg H.
      • et al.
      The impact of the HEART risk score in the early assessment of patients with acute chest pain: design of a stepped wedge, cluster randomised trial.
      ]). In this stepped wedge design all hospitals (clusters) started with an initial period of usual care. Subsequently, at regular intervals (“steps”), each hospital switched to using the HEART score. At the end of the trial all hospitals had crossed over to using the HEART score. The order in which hospitals switched was randomized. A total of 9 hospitals in the Netherlands participated. Any patient with chest pain presenting to the ED was eligible for inclusion. Patients directly recognized as having ST-elevation myocardial infarction (STEMI) were excluded, because of the lack of diagnostic uncertainty. All included patients provided written informed consent. The study was conducted according to the principles of the current version of the declaration of Helsinki and in accordance with the Dutch law on Medical Research Involving Human Subjects Act (WMO). The trial was approved by the Institutional Review Board of the University Medical Center Utrecht, the Netherlands, and subsequently by the Boards of the participating hospitals. Characteristics of the used troponin assays are shown in Table A.1. Further details can be read in the published study protocol [
      • Poldervaart J.M.
      • Reitsma J.B.
      • Koffijberg H.
      • et al.
      The impact of the HEART risk score in the early assessment of patients with acute chest pain: design of a stepped wedge, cluster randomised trial.
      ]. For our current study, we only analysed patients who were included during the HEART care period (half of the total HEART-Impact trial population), since specific measures were taken during the usual care period of the HEART-Impact trial to ensure the HEART score was not calculated.

      2.2 Endpoints

      The main endpoint in our study was major adverse cardiac events (MACE) within 6 weeks after the initial ED presentation (including the index event). MACE consisted of unstable angina (UA), non-ST elevation myocardial infarction (NSTEMI), STEMI, percutaneous coronary intervention (PCI), coronary arterial bypass grafting (CABG), stenosis managed conservatively, cardiovascular death, non-cardiovascular death and death with unknown cause. The potential occurrence of MACE was identified by means of a phone call with each patient at 3 months after presentation [
      • Poldervaart J.M.
      • Reitsma J.B.
      • Koffijberg H.
      • et al.
      The impact of the HEART risk score in the early assessment of patients with acute chest pain: design of a stepped wedge, cluster randomised trial.
      ]. In all cases the patient could not be contacted, the patient's general practitioner was contacted. In all patients with a possible MACE or unknown status, the electronic hospital records were investigated. All information possibly indicating MACE was further investigated by examining medical records from the hospital and/or the general practitioner. All potential events were then adjudicated by two independent cardiologists and it was decided whether a MACE occurred or not. The adjudication was done blinded for the GRACE, HEART and TIMI scores [
      • Poldervaart J.M.
      • Reitsma J.B.
      • Koffijberg H.
      • et al.
      The impact of the HEART risk score in the early assessment of patients with acute chest pain: design of a stepped wedge, cluster randomised trial.
      ].

      2.3 Calculation of the risk scores and their performance

      All variables used in the risk scores were collected at time of presentation at the ED and are depicted in Table 1. The GRACE score and TIMI score were calculated automatically from the prospectively collected data, without interpretation by the investigators. The HEART score was calculated by physicians at the moment of admission at the ED during the HEART care period [
      • Poldervaart J.M.
      • Reitsma J.B.
      • Koffijberg H.
      • et al.
      The impact of the HEART risk score in the early assessment of patients with acute chest pain: design of a stepped wedge, cluster randomised trial.
      ]. One of the key roles of these risk scores is to identify patients at low risk for MACE. Therefore, we compared the number of patients identified as “low risk” at a fixed level of safety. In scenario 1, we calculated the cut-off for each risk score with an absolute safety level of no >5% of all patients with MACE being missed, i.e. at least 95% sensitivity. The risk score with the highest number of patients identified as low risk considering this safety level can then be considered the most efficient score. We also considered scenario 2, with an absolute safety level of missing not >2% of all patients with MACE, i.e. at least 98% sensitivity. Both scenarios of safety levels were based on the first measurement of troponin at the ED. To reflect current clinical practice most closely, we also calculated all three scores based on the first and (when available) second troponin measurement and again assessed the scores' efficiency and safety. Furthermore, to facilitate comparison with other studies, we also assessed the efficiency and safety, when the primary endpoint of MACE consisted of only acute myocardial infarction (AMI) and/or death.

      2.4 Statistical analysis

      Continuous variables were presented as means with standard deviations, categorical variables were presented as absolute number of patients with corresponding percentages. Cut-off values of troponin were provided by all participating hospitals to assess whether the level of this cardiac marker was elevated. We compared the discrimination of the three scores by examining their ROC curves and calculating the areas under the ROC curve (AUCs), also known as the c-statistic, and the corresponding 95% confidence intervals (CI). To compare the c-statistics we used the method of DeLong [
      • Carlton E.W.
      • Khattab A.
      • Greaves K.
      Identifying patients suitable for discharge after a single-presentation high-sensitivity troponin result: a comparison of five established risk scores and two high-sensitivity assays.
      ], which takes into account the paired nature of our data as all three scores were determined in each patient. Because of the within-patient comparison of scores, the effect of clustering within hospital was not taken into account. We used Statistical Package for the Social Sciences (IBM SPSS statistics, version 21) for all statistical analyses, except for the comparison of the paired ROC curves and AUC for which we used SAS version 9.1.

      3. Results

      3.1 Study population

      Patients were enrolled between July 1, 2013 and August 31, 2014 in nine hospitals in the Netherlands. For patient flow see Fig. 1. In total 3666 patients were included in the HEART-Impact trial, with 1833 (50%) patients included during HEART care period. Of these 1833 HEART care patients, 10 patients (0.5%) were lost to follow-up, and 2 patients (0.1%) withdrew their informed consent. In a total of 73 (4.0%) patients, one or more risk score could not be calculated and therefore 1748 patients were used in the analysis. The mean age of these patients was 62 years and 54% were male. Patient characteristics are shown in Table 2.
      Fig. 1
      Fig. 1Patient flow chart for patients included in current comparison of performance of the GRACE score, HEART score and TIMI score.
      Table 2Baseline characteristics.
      All patients (n = 1748)Patients with MACE (n = 326)Patients without MACE (n = 1422)
      Demographics
       Male937 (54%)227 (70%)710 (50%)
       Mean age (SD)62 (14)67 (11)60 (15)
      Vital signs at presentation
       Mean systolic blood pressure in mm Hg (SD)144 (23)147 (23)143 (23)
       Mean diastolic blood pressure in mm Hg (SD)81 (13)82 (13)81 (13)
       Mean heart frequency per minute (SD)73 (15)75 (17)73 (15)
       Killip class I1723 (99%)317 (97%)1406 (99%)
      Cardiac risk factors
       Diabetes Mellitus271 (16%)68 (21%)203 (14%)
       Obesity (BMI >30 kg/m2)319 (18%)58 (18%)261 (18%)
       Hypercholesterolemia559 (32%)117 (36%)442 (31%)
       Hypertension846 (48%)209 (64%)637 (48%)
       Positive family history629 (36%)117 (36%)512 (36%)
       Current smoking441 (25%)81 (25%)360 (25%)
       History of cardiovascular disease576 (33%)154 (47%)422 (30%)
       History of AMI277 (16%)65 (20%)212 (15%)
       History of PCI331 (19%)91 (28%)240 (17%)
       History of CABG128 (7%)36 (11%)92 (6%)
       History of CVA/TIA98 (6%)27 (8%)71 (5%)
       History of peripheral artery disease69 (4%)25 (8%)44 (3%)
      Laboratory results at presentation
       Mean creatinin in μmol/l (SD)80 (33)85 (22)78 (35)
      Medication at presentation
       Aspirin597 (34%)153 (47%)444 (31%)
       P2Y12-inhibitor (clopidogrel)107 (6%)40 (12%)67 (5%)
       Vitamin K antagonists (coumarin)162 (9%)33 (10%)129 (9%)
       Other (Dipyridamol, Ticagrelor, DOAC)62 (4%)14 (4%)48 (3%)
      SD: standard deviation, mm Hg: millimetres of mercury, BMI: Body Mass Index, AMI: acute myocardial infarction, PCI: percutaneous coronary intervention, CABG: coronary arterial bypass grafting, CVA: cerebrovascular attack, TIA: transient ischemic attack, DOAC: direct oral anticoagulant.

      3.2 Endpoints

      A total of 542 MACE occurred in 325 (19%) patients, consisting of 99 UA, 201 NSTEMI, 11 STEMI, 41 stenosis managed conservatively, 140 PCI, 45 CABG, one cardiovascular death, one non-cardiovascular death and three deaths from an unknown cause.

      3.3 Performance of the risk scores

      In Fig. 2, the ROC curves of the GRACE score, HEART score and TIMI score to predict major adverse cardiac events within 6 weeks are shown. The AUC of the HEART score was highest with 0.86 (95% CI: 0.84–0.88), followed by the AUC of the TIMI score with 0.80 (95% CI: 0.78–0.83) and the GRACE score with an AUC of 0.73 (95% CI: 0.70–0.76). All differences in AUC were highly statistically significant: all p-values <0.001.
      Fig. 2
      Fig. 2Receiver-operating-characteristic (ROC) curves and corresponding Areas under the curve (AUCs) of the GRACE, HEART and TIMI score to predict major adverse cardiac events within 6 weeks.
      Table 3 shows the comparison of performance of GRACE, HEART and TIMI score in terms of safety and efficiency. Scenario 1 used an absolute level of safety of missing no >5% of all patients with MACE to define a “low-risk” group. At this absolute safety level, the GRACE score classified 334 patients as “low risk” of whom 12/334 (3.6%) patients developed MACE. Using the same absolute safety level, the HEART score classified 708 patients as “low risk” with 14/708 (2.0%) patients developing MACE. Lastly, the TIMI score identified 439 patients as “low risk” with 14/439 (3.2%) having a MACE.
      Table 3Comparison of performance of GRACE score, HEART score and TIMI score in terms of safety and efficiency.
      Scenario 1: at least 95% sensitivityGRACE scoreHEART scoreTIMI score
      Corresponding cut-off for “low risk”72 points3 points0 points
      Number of patients classified “low risk” / total number of patients334/1748 (19.1%)708/1748 (40.5%)439/1748 (25.1%)
      Percentage of MACE in “low risk” group3.6% (12/334)2.0% (14/708)3.2% (14/439)
       MACE, of which AMI530
       MACE, of which death010
      Negative predictive value (NPV)96% (94%–98%)98% (97–99%)97% (95–98%)
      Scenario 2: at least 98% sensitivityGRACE scoreHEART scoreTIMI score
      Corresponding cut-off for “low risk”66 points2 points
      At the lowest TIMI score, this absolute safety level is not reached unless all patients are classified as high risk.
      Number of patients classified “low risk” / total number of patients231/1748 (13.2%)381/1748 (21.8%)
      Percentage of MACE in “low risk” group2.2% (5/231)0.8% (3/381)
       MACE, of which AMI11
       MACE, of which death00
      Negative predictive value (NPV)98% (95–99%)99% (98–100%)
      MACE: major adverse cardiac events, AMI: acute myocardial infarction.
      low asterisk At the lowest TIMI score, this absolute safety level is not reached unless all patients are classified as high risk.
      We repeated the analyses at a different absolute safety level of missing no >2% of MACE in all patients with MACE (scenario 2). This sensitivity analysis showed that the HEART score again was the more efficient score with a low-risk group of 381 patients versus 231 and no patients for the GRACE and TIMI scores, respectively. The proportion of MACE in these low-risk groups were 0.8% and 2.2% for respectively the HEART and GRACE scores. Furthermore, to facilitate comparison with other studies, we also identified the number of patients with an AMI or death, which are shown in Table 3.
      In current practice, serial measurements of troponin are being used. To reflect this we performed an additional analysis in which the scores were based on the first and second troponin measurement. A second troponin measurement was performed in 955 of the 1748 patients (55%). Addition of the second troponin (when performed) in the calculation of the risk scores did not show different results, as depicted in Table 4.
      Table 4Comparison of performance of GRACE score, HEART score and TIMI score in terms of safety and efficiency, based on the first and second troponin measurement (performed in N = 955).
      Scenario 1: at least 95% sensitivityGRACE scoreHEART scoreTIMI score
      Corresponding cut-off for “low risk”73 points3 points0 points
      Number of patients classified as “low risk” / total number of patients340/1748 (19.5%)707/1748 (40.5%)430/1748 (24.6%)
      Percentage of MACE in “low risk” group4.1% (14/340)1.8% (13/707)1.9% (8/430)
       Of which AMI530
       Of which death010
      Negative predictive value (NPV)96% (93–98%)98% (97–99%)98% (96–99%)
      Scenario 2: at least 98% sensitivityGRACE scoreHEART scoreTIMI score
      Corresponding cut-off for “low risk”67 points2 points
      At the lowest TIMI score, this absolute safety level is not reached unless all patients are classified as high risk.
      Number of patients classified as “low risk” / total number of patients243/1748 (13.9%)381/1748 (21.8%)
      Percentage of MACE in “low risk” group2.5% (6/243)0.8% (3/381)
       Of which AMI11
       Of which death00
      Negative predictive value (NPV)97% (95–99%)99% (98–100%)
      MACE: major adverse cardiac events, AMI: acute myocardial infarction.
      low asterisk At the lowest TIMI score, this absolute safety level is not reached unless all patients are classified as high risk.

      4. Discussion

      Our head-to-head comparison of three well-known and extensively validated risk scores in 1748 patients presenting with chest pain at the ED, showed that at a same level of safety not missing >5% of all patients with ACS, the number of low-risk patients identified by the HEART score was higher than by the GRACE or TIMI scores. At a maximum of 2% missed cases, results were similar. Furthermore, the HEART score had the highest overall discrimination to predict MACE with an area under the ROC curve of 0.86 (95% CI: 0.84–0.88), followed by the TIMI score with an AUC of 0.80 (95% CI: 0.78–0.83) and the GRACE score (0.73, 95% CI: 0.70–0.76).
      In the literature, mostly comparable results were found when comparing the HEART and TIMI scores. In one study, the AUC of the HEART score was 0.83 (95% CI: 0.81–0.85) and the AUC of the TIMI score (0.75, 95% CI: 0.72–0.77) was slightly lower than the AUC of 0.80 we found [
      • Six A.J.
      • Cullen L.
      • Backus B.E.
      • et al.
      The HEART score for the assessment of patients with chest pain in the emergency department.
      ]. In one other study comparing performance of GRACE and TIMI risk scores, the TIMI score AUC was 0.79 (95% CI: 0.74–0.83), a similar result we found in our analysis. The AUC for the GRACE score was considerably higher, namely 0.83 (95% CI: 0.79–0.87), which may possibly be explained by the smaller definition of MACE and shorter duration of follow-up [
      • Cullen L.
      • Greenslade J.
      • Hammett C.J.
      • et al.
      Comparison of three risk stratification rules for predicting patients with acute coronary syndrome presenting to an Australian emergency department.
      ]. On the other hand, Carlton et al. showed that TIMI would be the more effective risk score, but neither HEART nor TIMI reached a 1% miss-rate for AMI with addition of either high-sensitive troponin [
      • Christenson J.
      • Innes G.
      • McKnight D.
      • et al.
      A clinical prediction rule for early discharge of patients with chest pain.
      ]. Additionally, in this study only AMI was included as an outcome measure, instead of the broad endpoint definition we used [
      • DeLong E.R.
      • DeLong D.M.
      • Clarke-Pearson D.L.
      Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach.
      ].
      One valuable role for cardiac risk scores is to identify patients as low-risk in order to avoid further testing and hospital admission in these patients (triage role). An ideal triage instrument would identify the largest number of patients at low risk (i.e. efficiency) without compromising safety, meaning that the number of patients classified as low risk but developing MACE (i.e. false negatives) should be low. When setting an absolute safety level for missed MACE of 5% of total patients, the HEART score identifies the most patients as “low risk”, namely 708 patients, with 14 patients missed of the total 325 patients with MACE. This corresponds to a proportion of MACE in the low-risk group of 2.0%. Although the definition of an acceptable false-negative rate is susceptible to personal opinions, and may vary between countries, Than et al. and Kline et al. estimate that the most clinicians would accept a false-negative rate of 1 to 2% [
      • Than M.
      • Herbert M.
      • Flaws D.
      • et al.
      What is an acceptible risk for major adverse cardiac event in chest pain patients soon after discharge from the emergency department?.
      ,
      • Kline J.A.
      • Johnson C.L.
      • Pollack Jr., C.V.
      • et al.
      Pretest probability assessment derived from attribute matching.
      ]. When repeating the analyses at a different absolute safety level of missing no >2% of all patients with MACE, the HEART score was again the most efficient score with 381 patients identified as low risk, resulting in a cumulative incidence of MACE in this low-risk group of 0.8%, which is below the mentioned more conservative 1% false-negative rate.
      The better performance of the HEART score compared to the TIMI and GRACE scores may be explained by the differences in the patient populations in which the three risk scores were developed. The HEART score was specifically developed for unselected patients with chest pain presenting at the ED, thus, a clinical domain characterized by diagnostic uncertainty [
      • Six A.J.
      • Backus B.E.
      • Kelder J.C.
      Chest pain in the emergency room: value of the HEART score.
      ]. The GRACE score was developed in patients already diagnosed with ACS [
      • Kaa F.
      • Eagle K.A.
      • Gore J.M.
      • Steg P.G.
      • Anderson F.A.
      • for the GRACe investigators
      The global registry of acute coronary events, 1999 to 2009-GRACE.
      ,
      • The GRACE investigators
      Rationale and design of the GRACE (global registry of acute coronary events) project: a multinational registry of patients hospitalized with acute coronary syndromes.
      ], thus with a higher risk of AMI and/or death than an unselected population with chest pain at the ED. Similarly, the TIMI score was developed in a group of patients already diagnosed with UA or NSTEMI [
      • Antman E.M.
      • Cohen M.
      • Bernink P.J.L.M.
      • et al.
      The TIMI risk score for unstable angina/non-ST elevation MI; a method for prognostication and therapeutic decision making.
      ]. Importantly, our HEART-impact trial cohort consisted only of patients in whom a diagnostic dilemma persisted and patients with STEMI were excluded. The GRACE and TIMI scores are well-known scores and are supported by current clinical guidelines [
      • Thygesen K.
      • Alpert J.S.
      • Jaffe A.S.
      • Simoons M.L.
      • Chaitman B.R.
      • White H.D.
      • Joint ESC/ACCF/AHA/WHF Task Force for Universal Definition of Myocardial Infarction
      Third universal definition of myocardial infarction.
      ,
      • Windecker S.
      • Kolh P.
      • Alfonso F.
      • et al.
      ESC/EACTS guidelines on myocardial revascularization.
      ,
      • VMS veiligheidsprogramma
      Praktijkgids ACS [Document on the Internet].
      ,
      • Roffi M.
      • Patrono C.
      • Collet J.P.
      • et al.
      ESC guidelines for the management of acute coronary syndromes in patients presenting without persistent ST-segment elevation.
      ], but seem more suitable as a (short-term) prognostic score in patients already diagnosed with ACS. A strength of the GRACE score is that it was derived in a large dataset of 11,389 patients [
      • Kaa F.
      • Eagle K.A.
      • Gore J.M.
      • Steg P.G.
      • Anderson F.A.
      • for the GRACe investigators
      The global registry of acute coronary events, 1999 to 2009-GRACE.
      ,
      • The GRACE investigators
      Rationale and design of the GRACE (global registry of acute coronary events) project: a multinational registry of patients hospitalized with acute coronary syndromes.
      ]. The range of the outcome of the GRACE score is very wide (1 to 372), therefore small differences in patient characteristics will result in a specific score for every patient. However, the large range of total score outcomes with the GRACE score demands the use of a computer, making it more difficult to apply at the bedside, although smartphone apps might diminish this issue. The HEART and TIMI score have a smaller range of total scores from 0 to 10 and 0–7 respectively. The HEART scores' strength is that all five variables included in the score are derived from clinical practice which makes it simple to calculate the score at the bedside, improving applicability for physicians. Interestingly, the HEART score was not developed using mathematical modelling from real-life data, but developed by a cardiologist based on clinical experience and later on validated in clinical databases [
      • Six A.J.
      • Backus B.E.
      • Kelder J.C.
      Chest pain in the emergency room: value of the HEART score.
      ]. A limitation of the HEART score is the subjectivity of the first element, (i.e. whether history taking indicates ACS), although it is widely accepted that this is a clinically relevant element. Furthermore, the score uses a cut-off of 1.7% as being “low risk”, which can arguably be too high in some countries [
      • Than M.
      • Herbert M.
      • Flaws D.
      • et al.
      What is an acceptible risk for major adverse cardiac event in chest pain patients soon after discharge from the emergency department?.
      ,
      • Kline J.A.
      • Johnson C.L.
      • Pollack Jr., C.V.
      • et al.
      Pretest probability assessment derived from attribute matching.
      ]. However, the aim of this study was not to determine an optimal cut-off for the risk scores, as this is subject to debate. The TIMI score has as strength that it is comprised of statistically significant predictors, is derived in a large dataset of 1957 patients and consists of only seven clinical elements that can be calculated at the bedside. However, as shown, the TIMI score only identifies a small proportion of patients as “low risk” who are eligible for early discharge, making it not the most efficient score for triage.

      4.1 Study limitations

      A number of limitations of our study should be mentioned. Firstly, we chose to validate the GRACE, HEART and TIMI scores, while currently several other risk scores are available [
      • Hess E.P.
      • Knoedler M.A.
      • Shah N.D.
      • et al.
      The chest pain choice decision aid; a randomized trial.
      ,
      • Lagerqvist B.
      • Diderholm E.
      • Lindahl B.
      • et al.
      5-year outcomes in the FRISC-II randomised trial of an invasive versus a non-invasive strategy in non-ST-elevation acute coronary syndrome: a follow-up study.
      ,
      • Body R.
      • Carley S.
      • McDowell G.
      • et al.
      The Manchester acute coronary syndromes (MACS) decision rule for suspected cardiac chest pain: derivation and external validation.
      ,
      • Boersma E.
      • Pieper K.S.
      • Steyerberg E.W.
      • et al.
      Predictors of outcome in patients with acute coronary syndromes without persistent ST-segment elevation; results from an international trial of 9461 patients.
      ]. We consulted several experienced cardiologists, who found that most currently available risk scores were not used in daily practice, or that the scores included variables not routinely assessed by clinicians. Secondly, the GRACE score and TIMI score were calculated from prospectively collected variables, blinded for the primary endpoints. These variables were defined before the start of the trial and included in our data collection form at the ED. Clinicians might take other variables into account when calculating a risk score in daily practice; although the GRACE and TIMI score consist of more objective variables than the HEART score, we cannot rule out that in our study the performance of the GRACE and TIMI scores could have been underestimated to some extent. Lastly, we did not include serial troponin measurements in our study, while this is current policy in most hospitals. However, we did perform additional analyses based on available second troponin measurements into the calculation of all three risk scores, with the aim to more closely reflect current clinical practice. It should be noted physicians did not perform second troponin measurements in all patients, but only in the patients of whom they deemed this was necessary. Also in these additional analyses, the HEART score had the highest discriminative power.

      5. Conclusions

      From our head-to-head comparison of the GRACE, HEART and TIMI score in a large prospective cohort of chest pain patients presenting to the ED, we conclude that the HEART score performed best in discriminating between those with and without MACE. The HEART score identified the largest number of patients (40.5%) as low risk without compromising safety. We recommend the use of the HEART score in the work-up of patients with chest pain at the ED.

      Conflict of interest

      Two authors (BEB and AJS) were involved in the development of the HEART score. The authors report no relationships that could be construed as a conflict of interest.

      Acknowledgement of grant support

      Research grant was obtained from the “ Netherlands Organization for Health Research and Development ” (ZonMw grant Project number 171202015 ), as part of the Effectiveness Program (Doelmatigheidsprogramma ZonMw). The funding source had no role in the study's design, conduct, and reporting.

      Acknowledgements

      We thank all participating hospitals, especially all (cardiac) residents, nurses, secretaries, R&D departments and local laboratories, namely Diagnostisch Laboratrium Saltro Utrecht, ATAL medial, MDC Amstelland, Trombosedienst Gelderse vallei Ede, Diagnostiek voor U and the laboratory of Atrium MC. Furthermore we like to thank our monitor Gerda Kuiper, the Data Monitoring Safety Board (DSMB) consisting of Arend Mosterd, Kit Roes, Carla Bruijnzeel-Koomen and Caroline van Baal, the datamanagers Jildou Zwerver, Susan van Hemert and Joost Schotsman, and the trial nurses Lydeke Zwart, Carla Tims-Polderman, Els Kooiman, Octavie Brand-Rijssenbeek and Mieke de Haas for their extensive help during the data collection. We also would like to thank the members of our outcome committee for their precise adjudication, especially the extra members of the adjudication consensus committee Adriaan van Kraaijeveld and Jeroen Smits.

      Appendix A. Supplementary data

      References

        • Niska R.
        • Bhuiya F.
        • Xu J.
        National hospital ambulatory medical care survey: 2007 emergency department summary.
        Natl. Health Stat. Rep. 2010; 26: 1-31
        • Thygesen K.
        • Alpert J.S.
        • Jaffe A.S.
        • Simoons M.L.
        • Chaitman B.R.
        • White H.D.
        • Joint ESC/ACCF/AHA/WHF Task Force for Universal Definition of Myocardial Infarction
        Third universal definition of myocardial infarction.
        Eur. Heart J. 2012; 33: 2551-2567
        • Maroko P.R.
        • Kjekshus J.K.
        • Sobel B.E.
        • et al.
        Factors influencing infarct size following experimental coronary artery occlusions.
        Circulation. 1971; 43: 67-82
        • Windecker S.
        • Kolh P.
        • Alfonso F.
        • et al.
        ESC/EACTS guidelines on myocardial revascularization.
        Eur. Heart J. 2014; 35: 2541-2619
        • Amsterdam E.A.
        • Kirk J.D.
        • Bluemke D.A.
        • et al.
        Testing of low-risk patients presenting to the emergency department with chest pain: a scientific statement from the American Heart Association.
        Circulation. 2010; 122: 1756-1776
        • Six A.J.
        • Backus B.E.
        • Kingma A.
        • Kaandorp S.I.
        Consumption of diagnostic procedures and other cardiology care in chest pain patients after presentation at the emergency department.
        Neth. Heart J. 2012; 20: 499-504
        • Hoffmann U.
        • Truong Q.A.
        • Schoenfeld D.A.
        • for the ROMICAT-II Investigators
        • et al.
        Coronary CT angiography versus standard evaluation in acute chest pain.
        N. Engl. J. Med. 2012; 367: 299-308
        • National Institute for Health and Care Excellence
        Chest pain of recent onset Assessment and diagnosis of recent onset chest pain or discomfort of suspected cardiac origin.
        in: [Document on the Internet] NICE Clinical Guideline 95. March 2010 (Published, Available from: http://guidance.nice.org.uk/CG95)
        • Cannon C.P.
        • Brindis R.G.
        • Chaitman B.R.
        • et al.
        2013 ACCF/AHA key data elements and definitions for measuring the clinical management and outcomes of patients with acute coronary syndromes and coronary artery disease.
        J. Am. Coll. Cardiol. 2013; 61: 992-1025
        • VMS veiligheidsprogramma
        Praktijkgids ACS [Document on the Internet].
        Webpage VMS zorg, March 2010 (Published, Available from: http://www.vmszorg.nl/_page/vms_inline?nodeid=4571&subjectid=6705&configid=6490)
        • Roffi M.
        • Patrono C.
        • Collet J.P.
        • et al.
        ESC guidelines for the management of acute coronary syndromes in patients presenting without persistent ST-segment elevation.
        Eur. Heart J. 2016; 37: 267-315
        • Hess E.P.
        • Knoedler M.A.
        • Shah N.D.
        • et al.
        The chest pain choice decision aid; a randomized trial.
        Circ. Cardiovasc. Qual. Outcomes. 2012; 5: 1-9
        • Than M.
        • Flaws D.
        • Sanders S.
        • et al.
        Development and validation of the emergency department assessment of chest pain score and 2 h accelerated diagnostic protocol.
        Emerg. Med. Australas. 2014; 26: 34-44
        • Lagerqvist B.
        • Diderholm E.
        • Lindahl B.
        • et al.
        5-year outcomes in the FRISC-II randomised trial of an invasive versus a non-invasive strategy in non-ST-elevation acute coronary syndrome: a follow-up study.
        Heart. 2005; 91: 1047-1052
        • Six A.J.
        • Backus B.E.
        • Kelder J.C.
        Chest pain in the emergency room: value of the HEART score.
        Neth. Heart J. 2008; 16: 191-196
        • Granger C.B.
        • Goldberg R.J.
        • Dabbous O.
        • et al.
        Global registry of acute coronary events investigators. Predictors of hospital mortality in the global registry of acute coronary events.
        Arch. Intern. Med. 2003; 163: 2345-2353
        • Body R.
        • Carley S.
        • McDowell G.
        • et al.
        The Manchester acute coronary syndromes (MACS) decision rule for suspected cardiac chest pain: derivation and external validation.
        Heart. 2014; 100: 1462-1468
        • Boersma E.
        • Pieper K.S.
        • Steyerberg E.W.
        • et al.
        Predictors of outcome in patients with acute coronary syndromes without persistent ST-segment elevation; results from an international trial of 9461 patients.
        Circulation. 2000; 101: 2557-2567
        • Antman E.M.
        • Cohen M.
        • Bernink P.J.L.M.
        • et al.
        The TIMI risk score for unstable angina/non-ST elevation MI; a method for prognostication and therapeutic decision making.
        JAMA. 2000; 284: 835-842
        • Christenson J.
        • Innes G.
        • McKnight D.
        • et al.
        A clinical prediction rule for early discharge of patients with chest pain.
        Ann. Emerg. Med. 2006; 47: 1-10
        • Mahler S.A.
        • Miller C.D.
        • Hollander J.E.
        • et al.
        Identifying patients for early discharge: performance of decision rules among patients with acute chest pain.
        Int. J. Cardiol. 2013; 168: 795-802
        • Six A.J.
        • Cullen L.
        • Backus B.E.
        • et al.
        The HEART score for the assessment of patients with chest pain in the emergency department.
        Crit. Pathw. Cardiol. 2013; 12: 121-126
        • Cullen L.
        • Greenslade J.
        • Hammett C.J.
        • et al.
        Comparison of three risk stratification rules for predicting patients with acute coronary syndrome presenting to an Australian emergency department.
        Heart Lung Circ. 2013; 22: 844-851
        • Ramsay G.
        • Podogrodzka M.
        • McClure C.
        • Fox K.
        Risk prediction in patients with suspected cardiac pain: the GRACE and TIMI risk scores versus clinical evaluation.
        QJM. 2007; 100: 11-18
        • Poldervaart J.M.
        • Reitsma J.B.
        • Koffijberg H.
        • et al.
        The impact of the HEART risk score in the early assessment of patients with acute chest pain: design of a stepped wedge, cluster randomised trial.
        BMC Cardiovasc. Disord. 2013; 13: 77
        • Carlton E.W.
        • Khattab A.
        • Greaves K.
        Identifying patients suitable for discharge after a single-presentation high-sensitivity troponin result: a comparison of five established risk scores and two high-sensitivity assays.
        Ann. Emerg. Med. Dec. 2015; 66 (e1): 635-645
        • DeLong E.R.
        • DeLong D.M.
        • Clarke-Pearson D.L.
        Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach.
        Biometrics. 1988; 44: 837-845
        • Than M.
        • Herbert M.
        • Flaws D.
        • et al.
        What is an acceptible risk for major adverse cardiac event in chest pain patients soon after discharge from the emergency department?.
        Int. J. Cardiol. 2013; 166: 752-754
        • Kline J.A.
        • Johnson C.L.
        • Pollack Jr., C.V.
        • et al.
        Pretest probability assessment derived from attribute matching.
        BMC Med. Inform. Decis. Mak. 2005; 5: 26
        • Kaa F.
        • Eagle K.A.
        • Gore J.M.
        • Steg P.G.
        • Anderson F.A.
        • for the GRACe investigators
        The global registry of acute coronary events, 1999 to 2009-GRACE.
        Heart. 2010; 96: 1095-1101
        • The GRACE investigators
        Rationale and design of the GRACE (global registry of acute coronary events) project: a multinational registry of patients hospitalized with acute coronary syndromes.
        Am. Heart J. 2001; 141: 190-199