If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
1 This author takes responsibility for all aspects of the reliability and freedom from bias of the data presented and their discussed interpretation.
J.M. Poldervaart
Correspondence
Corresponding author at: Julius Center for Health Sciences and Primary care, University Medical Center, Str. 6.101, PO box 85500, 3508AB Utrecht, The Netherlands.
The performance of the GRACE, HEART and TIMI scores were compared in predicting the probability of major adverse cardiac events (MACE) in chest pain patients presenting at the emergency department (ED), in particular their ability to identify patients at low risk.
Methods
Chest pain patients presenting at the ED in nine Dutch hospitals were included. The primary outcome was MACE within 6 weeks. The HEART score was determined by the treating physician at the ED. The GRACE and TIMI score were calculated based on prospectively collected data. Performance of the scores was compared by calculating AUC curves. Additionally, the number of low-risk patients identified by each score were compared at a fixed level of safety of at least 95% or 98% sensitivity.
Results
In total, 1748 patients were included. The AUC of GRACE, HEART, and TIMI were 0.73 (95% CI: 0.70–0.76%), 0.86 (95% CI: 0.84–0.88%) and 0.80 (95% CI: 0.78–0.83%), respectively (all differences in AUC highly statistically significant). At an absolute level of safety of at least 98% sensitivity, the GRACE score identified 231 patients as “low risk” in which 2.2% a MACE was missed; the HEART score identified 381 patients as “low risk” with 0.8% missed MACE. The TIMI score identified no “low risk” patients at this safety level.
Conclusions
The HEART score outperformed the GRACE and TIMI scores in discriminating between those with and without MACE in chest pain patients, and identified the largest group of low-risk patients at the same level of safety.
]. An urgent question in these patients is whether they have an acute coronary syndrome (ACS), as any delay in diagnosis and treatment can have a negative impact on their prognosis [
]. Normal values of troponin and a normal electrocardiogram (ECG) still do not exclude ACS completely. As a result, many patients presenting with chest pain are currently hospitalized and extensively evaluated with non-invasive stress testing or imaging, or with an invasive coronary angiography [
]. If patients at low risk for ACS could be recognized early in the diagnostic process, it has the potential to reduce patient burden, length of stay at the ED, frequency of hospitalization and costs [
2013 ACCF/AHA key data elements and definitions for measuring the clinical management and outcomes of patients with acute coronary syndromes and coronary artery disease.
]. International cardiac guidelines state that chest pain patients presenting to the ED should be assessed with a risk stratification tool or risk score [
5-year outcomes in the FRISC-II randomised trial of an invasive versus a non-invasive strategy in non-ST-elevation acute coronary syndrome: a follow-up study.
Predictors of outcome in patients with acute coronary syndromes without persistent ST-segment elevation; results from an international trial of 9461 patients.
]. Risk scores combine and weigh various predictors to calculate the risk of ACS for an individual patient. They are based on readily available information collected during the initial work-up of chest pain patients.
Table 1Variables present in GRACE score, HEART score and TIMI score.
Studies directly comparing the performance of risk scores in the same population of chest pain patients report conflicting results in terms of which score is preferred to use at the ED [
]. Furthermore, it is unclear which risk score performs best in identifying patients at “low risk” of ACS, as these patients are candidates for early discharge from the ED (triage role). Therefore, we compared the performance of the GRACE, HEART and TIMI risk scores in identifying best patients at “low risk” of ACS in patients presenting with chest pain at the ED.
2. Methods
2.1 Design and study population
Our study population consisted of patients participating in the HEART-impact trial. In short, this trial investigated the impact of the use of the HEART score in daily practice on safety, quality of life and use of health care resources. The trial was designed as a pragmatic, stepped wedge, cluster randomized trial and compared usual care with HEART score care (i.e. calculation of the HEART score and adherence to recommended patient management depending on the score; see Fig. A.1 and [
]). In this stepped wedge design all hospitals (clusters) started with an initial period of usual care. Subsequently, at regular intervals (“steps”), each hospital switched to using the HEART score. At the end of the trial all hospitals had crossed over to using the HEART score. The order in which hospitals switched was randomized. A total of 9 hospitals in the Netherlands participated. Any patient with chest pain presenting to the ED was eligible for inclusion. Patients directly recognized as having ST-elevation myocardial infarction (STEMI) were excluded, because of the lack of diagnostic uncertainty. All included patients provided written informed consent. The study was conducted according to the principles of the current version of the declaration of Helsinki and in accordance with the Dutch law on Medical Research Involving Human Subjects Act (WMO). The trial was approved by the Institutional Review Board of the University Medical Center Utrecht, the Netherlands, and subsequently by the Boards of the participating hospitals. Characteristics of the used troponin assays are shown in Table A.1. Further details can be read in the published study protocol [
]. For our current study, we only analysed patients who were included during the HEART care period (half of the total HEART-Impact trial population), since specific measures were taken during the usual care period of the HEART-Impact trial to ensure the HEART score was not calculated.
2.2 Endpoints
The main endpoint in our study was major adverse cardiac events (MACE) within 6 weeks after the initial ED presentation (including the index event). MACE consisted of unstable angina (UA), non-ST elevation myocardial infarction (NSTEMI), STEMI, percutaneous coronary intervention (PCI), coronary arterial bypass grafting (CABG), stenosis managed conservatively, cardiovascular death, non-cardiovascular death and death with unknown cause. The potential occurrence of MACE was identified by means of a phone call with each patient at 3 months after presentation [
]. In all cases the patient could not be contacted, the patient's general practitioner was contacted. In all patients with a possible MACE or unknown status, the electronic hospital records were investigated. All information possibly indicating MACE was further investigated by examining medical records from the hospital and/or the general practitioner. All potential events were then adjudicated by two independent cardiologists and it was decided whether a MACE occurred or not. The adjudication was done blinded for the GRACE, HEART and TIMI scores [
2.3 Calculation of the risk scores and their performance
All variables used in the risk scores were collected at time of presentation at the ED and are depicted in Table 1. The GRACE score and TIMI score were calculated automatically from the prospectively collected data, without interpretation by the investigators. The HEART score was calculated by physicians at the moment of admission at the ED during the HEART care period [
]. One of the key roles of these risk scores is to identify patients at low risk for MACE. Therefore, we compared the number of patients identified as “low risk” at a fixed level of safety. In scenario 1, we calculated the cut-off for each risk score with an absolute safety level of no >5% of all patients with MACE being missed, i.e. at least 95% sensitivity. The risk score with the highest number of patients identified as low risk considering this safety level can then be considered the most efficient score. We also considered scenario 2, with an absolute safety level of missing not >2% of all patients with MACE, i.e. at least 98% sensitivity. Both scenarios of safety levels were based on the first measurement of troponin at the ED. To reflect current clinical practice most closely, we also calculated all three scores based on the first and (when available) second troponin measurement and again assessed the scores' efficiency and safety. Furthermore, to facilitate comparison with other studies, we also assessed the efficiency and safety, when the primary endpoint of MACE consisted of only acute myocardial infarction (AMI) and/or death.
2.4 Statistical analysis
Continuous variables were presented as means with standard deviations, categorical variables were presented as absolute number of patients with corresponding percentages. Cut-off values of troponin were provided by all participating hospitals to assess whether the level of this cardiac marker was elevated. We compared the discrimination of the three scores by examining their ROC curves and calculating the areas under the ROC curve (AUCs), also known as the c-statistic, and the corresponding 95% confidence intervals (CI). To compare the c-statistics we used the method of DeLong [
Identifying patients suitable for discharge after a single-presentation high-sensitivity troponin result: a comparison of five established risk scores and two high-sensitivity assays.
], which takes into account the paired nature of our data as all three scores were determined in each patient. Because of the within-patient comparison of scores, the effect of clustering within hospital was not taken into account. We used Statistical Package for the Social Sciences (IBM SPSS statistics, version 21) for all statistical analyses, except for the comparison of the paired ROC curves and AUC for which we used SAS version 9.1.
3. Results
3.1 Study population
Patients were enrolled between July 1, 2013 and August 31, 2014 in nine hospitals in the Netherlands. For patient flow see Fig. 1. In total 3666 patients were included in the HEART-Impact trial, with 1833 (50%) patients included during HEART care period. Of these 1833 HEART care patients, 10 patients (0.5%) were lost to follow-up, and 2 patients (0.1%) withdrew their informed consent. In a total of 73 (4.0%) patients, one or more risk score could not be calculated and therefore 1748 patients were used in the analysis. The mean age of these patients was 62 years and 54% were male. Patient characteristics are shown in Table 2.
Fig. 1Patient flow chart for patients included in current comparison of performance of the GRACE score, HEART score and TIMI score.
A total of 542 MACE occurred in 325 (19%) patients, consisting of 99 UA, 201 NSTEMI, 11 STEMI, 41 stenosis managed conservatively, 140 PCI, 45 CABG, one cardiovascular death, one non-cardiovascular death and three deaths from an unknown cause.
3.3 Performance of the risk scores
In Fig. 2, the ROC curves of the GRACE score, HEART score and TIMI score to predict major adverse cardiac events within 6 weeks are shown. The AUC of the HEART score was highest with 0.86 (95% CI: 0.84–0.88), followed by the AUC of the TIMI score with 0.80 (95% CI: 0.78–0.83) and the GRACE score with an AUC of 0.73 (95% CI: 0.70–0.76). All differences in AUC were highly statistically significant: all p-values <0.001.
Fig. 2Receiver-operating-characteristic (ROC) curves and corresponding Areas under the curve (AUCs) of the GRACE, HEART and TIMI score to predict major adverse cardiac events within 6 weeks.
Table 3 shows the comparison of performance of GRACE, HEART and TIMI score in terms of safety and efficiency. Scenario 1 used an absolute level of safety of missing no >5% of all patients with MACE to define a “low-risk” group. At this absolute safety level, the GRACE score classified 334 patients as “low risk” of whom 12/334 (3.6%) patients developed MACE. Using the same absolute safety level, the HEART score classified 708 patients as “low risk” with 14/708 (2.0%) patients developing MACE. Lastly, the TIMI score identified 439 patients as “low risk” with 14/439 (3.2%) having a MACE.
Table 3Comparison of performance of GRACE score, HEART score and TIMI score in terms of safety and efficiency.
Scenario 1: at least 95% sensitivity
GRACE score
HEART score
TIMI score
Corresponding cut-off for “low risk”
≤72 points
≤3 points
0 points
Number of patients classified “low risk” / total number of patients
We repeated the analyses at a different absolute safety level of missing no >2% of MACE in all patients with MACE (scenario 2). This sensitivity analysis showed that the HEART score again was the more efficient score with a low-risk group of 381 patients versus 231 and no patients for the GRACE and TIMI scores, respectively. The proportion of MACE in these low-risk groups were 0.8% and 2.2% for respectively the HEART and GRACE scores. Furthermore, to facilitate comparison with other studies, we also identified the number of patients with an AMI or death, which are shown in Table 3.
In current practice, serial measurements of troponin are being used. To reflect this we performed an additional analysis in which the scores were based on the first and second troponin measurement. A second troponin measurement was performed in 955 of the 1748 patients (55%). Addition of the second troponin (when performed) in the calculation of the risk scores did not show different results, as depicted in Table 4.
Table 4Comparison of performance of GRACE score, HEART score and TIMI score in terms of safety and efficiency, based on the first and second troponin measurement (performed in N = 955).
Scenario 1: at least 95% sensitivity
GRACE score
HEART score
TIMI score
Corresponding cut-off for “low risk”
≤73 points
≤3 points
0 points
Number of patients classified as “low risk” / total number of patients
Our head-to-head comparison of three well-known and extensively validated risk scores in 1748 patients presenting with chest pain at the ED, showed that at a same level of safety not missing >5% of all patients with ACS, the number of low-risk patients identified by the HEART score was higher than by the GRACE or TIMI scores. At a maximum of 2% missed cases, results were similar. Furthermore, the HEART score had the highest overall discrimination to predict MACE with an area under the ROC curve of 0.86 (95% CI: 0.84–0.88), followed by the TIMI score with an AUC of 0.80 (95% CI: 0.78–0.83) and the GRACE score (0.73, 95% CI: 0.70–0.76).
In the literature, mostly comparable results were found when comparing the HEART and TIMI scores. In one study, the AUC of the HEART score was 0.83 (95% CI: 0.81–0.85) and the AUC of the TIMI score (0.75, 95% CI: 0.72–0.77) was slightly lower than the AUC of 0.80 we found [
]. In one other study comparing performance of GRACE and TIMI risk scores, the TIMI score AUC was 0.79 (95% CI: 0.74–0.83), a similar result we found in our analysis. The AUC for the GRACE score was considerably higher, namely 0.83 (95% CI: 0.79–0.87), which may possibly be explained by the smaller definition of MACE and shorter duration of follow-up [
]. On the other hand, Carlton et al. showed that TIMI would be the more effective risk score, but neither HEART nor TIMI reached a 1% miss-rate for AMI with addition of either high-sensitive troponin [
One valuable role for cardiac risk scores is to identify patients as low-risk in order to avoid further testing and hospital admission in these patients (triage role). An ideal triage instrument would identify the largest number of patients at low risk (i.e. efficiency) without compromising safety, meaning that the number of patients classified as low risk but developing MACE (i.e. false negatives) should be low. When setting an absolute safety level for missed MACE of 5% of total patients, the HEART score identifies the most patients as “low risk”, namely 708 patients, with 14 patients missed of the total 325 patients with MACE. This corresponds to a proportion of MACE in the low-risk group of 2.0%. Although the definition of an acceptable false-negative rate is susceptible to personal opinions, and may vary between countries, Than et al. and Kline et al. estimate that the most clinicians would accept a false-negative rate of 1 to 2% [
]. When repeating the analyses at a different absolute safety level of missing no >2% of all patients with MACE, the HEART score was again the most efficient score with 381 patients identified as low risk, resulting in a cumulative incidence of MACE in this low-risk group of 0.8%, which is below the mentioned more conservative 1% false-negative rate.
The better performance of the HEART score compared to the TIMI and GRACE scores may be explained by the differences in the patient populations in which the three risk scores were developed. The HEART score was specifically developed for unselected patients with chest pain presenting at the ED, thus, a clinical domain characterized by diagnostic uncertainty [
Rationale and design of the GRACE (global registry of acute coronary events) project: a multinational registry of patients hospitalized with acute coronary syndromes.
], thus with a higher risk of AMI and/or death than an unselected population with chest pain at the ED. Similarly, the TIMI score was developed in a group of patients already diagnosed with UA or NSTEMI [
]. Importantly, our HEART-impact trial cohort consisted only of patients in whom a diagnostic dilemma persisted and patients with STEMI were excluded. The GRACE and TIMI scores are well-known scores and are supported by current clinical guidelines [
], but seem more suitable as a (short-term) prognostic score in patients already diagnosed with ACS. A strength of the GRACE score is that it was derived in a large dataset of 11,389 patients [
Rationale and design of the GRACE (global registry of acute coronary events) project: a multinational registry of patients hospitalized with acute coronary syndromes.
]. The range of the outcome of the GRACE score is very wide (1 to 372), therefore small differences in patient characteristics will result in a specific score for every patient. However, the large range of total score outcomes with the GRACE score demands the use of a computer, making it more difficult to apply at the bedside, although smartphone apps might diminish this issue. The HEART and TIMI score have a smaller range of total scores from 0 to 10 and 0–7 respectively. The HEART scores' strength is that all five variables included in the score are derived from clinical practice which makes it simple to calculate the score at the bedside, improving applicability for physicians. Interestingly, the HEART score was not developed using mathematical modelling from real-life data, but developed by a cardiologist based on clinical experience and later on validated in clinical databases [
]. A limitation of the HEART score is the subjectivity of the first element, (i.e. whether history taking indicates ACS), although it is widely accepted that this is a clinically relevant element. Furthermore, the score uses a cut-off of 1.7% as being “low risk”, which can arguably be too high in some countries [
]. However, the aim of this study was not to determine an optimal cut-off for the risk scores, as this is subject to debate. The TIMI score has as strength that it is comprised of statistically significant predictors, is derived in a large dataset of 1957 patients and consists of only seven clinical elements that can be calculated at the bedside. However, as shown, the TIMI score only identifies a small proportion of patients as “low risk” who are eligible for early discharge, making it not the most efficient score for triage.
4.1 Study limitations
A number of limitations of our study should be mentioned. Firstly, we chose to validate the GRACE, HEART and TIMI scores, while currently several other risk scores are available [
5-year outcomes in the FRISC-II randomised trial of an invasive versus a non-invasive strategy in non-ST-elevation acute coronary syndrome: a follow-up study.
Predictors of outcome in patients with acute coronary syndromes without persistent ST-segment elevation; results from an international trial of 9461 patients.
]. We consulted several experienced cardiologists, who found that most currently available risk scores were not used in daily practice, or that the scores included variables not routinely assessed by clinicians. Secondly, the GRACE score and TIMI score were calculated from prospectively collected variables, blinded for the primary endpoints. These variables were defined before the start of the trial and included in our data collection form at the ED. Clinicians might take other variables into account when calculating a risk score in daily practice; although the GRACE and TIMI score consist of more objective variables than the HEART score, we cannot rule out that in our study the performance of the GRACE and TIMI scores could have been underestimated to some extent. Lastly, we did not include serial troponin measurements in our study, while this is current policy in most hospitals. However, we did perform additional analyses based on available second troponin measurements into the calculation of all three risk scores, with the aim to more closely reflect current clinical practice. It should be noted physicians did not perform second troponin measurements in all patients, but only in the patients of whom they deemed this was necessary. Also in these additional analyses, the HEART score had the highest discriminative power.
5. Conclusions
From our head-to-head comparison of the GRACE, HEART and TIMI score in a large prospective cohort of chest pain patients presenting to the ED, we conclude that the HEART score performed best in discriminating between those with and without MACE. The HEART score identified the largest number of patients (40.5%) as low risk without compromising safety. We recommend the use of the HEART score in the work-up of patients with chest pain at the ED.
Conflict of interest
Two authors (BEB and AJS) were involved in the development of the HEART score. The authors report no relationships that could be construed as a conflict of interest.
Acknowledgement of grant support
Research grant was obtained from the “Netherlands Organization for Health Research and Development” (ZonMw grant Project number 171202015), as part of the Effectiveness Program (Doelmatigheidsprogramma ZonMw). The funding source had no role in the study's design, conduct, and reporting.
Acknowledgements
We thank all participating hospitals, especially all (cardiac) residents, nurses, secretaries, R&D departments and local laboratories, namely Diagnostisch Laboratrium Saltro Utrecht, ATAL medial, MDC Amstelland, Trombosedienst Gelderse vallei Ede, Diagnostiek voor U and the laboratory of Atrium MC. Furthermore we like to thank our monitor Gerda Kuiper, the Data Monitoring Safety Board (DSMB) consisting of Arend Mosterd, Kit Roes, Carla Bruijnzeel-Koomen and Caroline van Baal, the datamanagers Jildou Zwerver, Susan van Hemert and Joost Schotsman, and the trial nurses Lydeke Zwart, Carla Tims-Polderman, Els Kooiman, Octavie Brand-Rijssenbeek and Mieke de Haas for their extensive help during the data collection. We also would like to thank the members of our outcome committee for their precise adjudication, especially the extra members of the adjudication consensus committee Adriaan van Kraaijeveld and Jeroen Smits.
2013 ACCF/AHA key data elements and definitions for measuring the clinical management and outcomes of patients with acute coronary syndromes and coronary artery disease.
5-year outcomes in the FRISC-II randomised trial of an invasive versus a non-invasive strategy in non-ST-elevation acute coronary syndrome: a follow-up study.
Predictors of outcome in patients with acute coronary syndromes without persistent ST-segment elevation; results from an international trial of 9461 patients.
Identifying patients suitable for discharge after a single-presentation high-sensitivity troponin result: a comparison of five established risk scores and two high-sensitivity assays.
Rationale and design of the GRACE (global registry of acute coronary events) project: a multinational registry of patients hospitalized with acute coronary syndromes.
Troponin or creatin kinase–MB.
33282-X&id=gr1.jpg){kind=link}
33282-X&id=gr2.jpg){kind=link}