Development and Validation of a Clinical Risk Score to Predict the Occurrence of Critical Illness in Hospitalized Patients With COVID-19 | Critical Care Medicine | JAMA Internal Medicine | JAMA Network
[Skip to Content]
[Skip to Content Landing]
Figure.  The Online Web-Based Calculatora for Predicting Critical Illness Among Patients With COVID-19
The Online Web-Based Calculatora for Predicting Critical Illness Among Patients With COVID-19

aGuangzhou Institute of Respiratory Health is responsible for the web-based calculator (http://118.126.104.170/).

Table 1.  Demographics and Clinical Characteristics Among Patients In the Development Cohort Who Did or Did Not Develop Critical Illnessa
Demographics and Clinical Characteristics Among Patients In the Development Cohort Who Did or Did Not Develop Critical Illnessa
Table 2.  Laboratory Findings Among Patients Who Did or Did Not Develop Critical Illness
Laboratory Findings Among Patients Who Did or Did Not Develop Critical Illness
Table 3.  Multivariable Logistic Regression Model for Predicting Development of Critical Illness in 1590 Patients Hospitalized With COVID-19 in Wuhan
Multivariable Logistic Regression Model for Predicting Development of Critical Illness in 1590 Patients Hospitalized With COVID-19 in Wuhan
Table 4.  Demographics and Clinical Characteristics of Patients in Validation Cohorts
Demographics and Clinical Characteristics of Patients in Validation Cohorts
1.
Zhou  F, Yu  T, Du  R,  et al.  Clinical course and risk factors for mortality of adult inpatients with COVID-19 in Wuhan, China: a retrospective cohort study.   Lancet. 2020;395(10229):1054-1062. doi:10.1016/S0140-6736(20)30566-3PubMedGoogle ScholarCrossref
2.
Wu  Z, McGoogan  JM.  Characteristics of and important lessons from the coronavirus disease 2019 (COVID-19) outbreak in China: summary of a report of 72 314 cases from the Chinese Center for Disease Control and Prevention.   JAMA. 2020;(Feb):24. doi:10.1001/jama.2020.2648PubMedGoogle Scholar
3.
Wu  C, Chen  X, Cai  Y,  et al.  Risk factors associated with acute respiratory distress syndrome and death in patients with coronavirus disease 2019 pneumonia in Wuhan, China.   JAMA Intern Med. 2020;(Mar):13. doi:10.1001/jamainternmed.2020.0994PubMedGoogle Scholar
4.
Guan  WJ, Ni  ZY, Hu  Y,  et al; China Medical Treatment Expert Group for Covid-19.  Clinical characteristics of coronavirus disease 2019 in China.   N Engl J Med. 2020;(Feb):28. doi:10.1056/NEJMoa2002032PubMedGoogle Scholar
5.
Metlay  JP, Waterer  GW, Long  AC,  et al.  Diagnosis and treatment of adults with community-acquired pneumonia. an official clinical practice guideline of the American Thoracic Society and Infectious Diseases Society of America.   Am J Respir Crit Care Med. 2019;200(7):e45-e67. doi:10.1164/rccm.201908-1581STPubMedGoogle ScholarCrossref
6.
Gao  HN, Lu  HZ, Cao  B,  et al.  Clinical findings in 111 cases of influenza A (H7N9) virus infection.   N Engl J Med. 2013;368(24):2277-2285. doi:10.1056/NEJMoa1305584PubMedGoogle ScholarCrossref
7.
Iasonos  A, Schrag  D, Raj  GV, Panageas  KS.  How to build and interpret a nomogram for cancer prognosis.   J Clin Oncol. 2008;26(8):1364-1370. doi:10.1200/JCO.2007.12.9791PubMedGoogle ScholarCrossref
8.
Neill  AM, Martin  IR, Weir  R,  et al.  Community acquired pneumonia: aetiology and usefulness of severity criteria on admission.   Thorax. 1996;51(10):1010-1016. doi:10.1136/thx.51.10.1010PubMedGoogle ScholarCrossref
9.
Xu  J, Long  JS.  Confidence intervals for predicted outcomes in regression models for categorical outcomes.   Stata Journal. 2005;5(4):537-559. doi:10.1177/1536867X0500500405Google ScholarCrossref
10.
Liang  W, Guan  W, Chen  R,  et al.  Cancer patients in SARS-CoV-2 infection: a nationwide analysis in China.   Lancet Oncol. 2020;21(3):335-337. doi:10.1016/S1470-2045(20)30096-6PubMedGoogle ScholarCrossref
Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    2 Comments for this article
    EXPAND ALL
    Severity scoring of CT and CXR
    Arjun Nair, MD FRCR | UCLH
    A brilliant paper. I would like to know the definition of the severity scoring grades for CT and CXR. No definition is provided in the manuscript or supplementary material.
    CONFLICT OF INTEREST: None Reported
    Predicting Critical COVID-19 Illness Using a Clinical Risk Score
    Michael McAleer, PhD(Econometrics),Queen's | Asia University, Taiwan
    The novel Original Investigation is concerned with the important issue of using epidemiological and clinical characteristics for early identification and detection, and subsequent prediction of critical illness, namely a composite measure of whether COVID-19 hospital patients will be admitted to an ICU, will need invasive mechanical ventilation, or will die.

    The sample of patients was based on laboratory-confirmed hospitalized cases, an initial nationwide cohort of 1590 patients and a validation treatment of 710 patients from 575 hospitals in 31 provincial administrative regions, with data from 21 November 2019 to 31 January 2020. 

    The mean ages for the development
    and validation cohorts were 48.9 and 48.2 years, respectively.

    An experienced team of medical researchers at a number of leading hospitals and universities in China identified 10 independent factors that are available upon admission to hospital to develop a freely available online risk score to identify and predict the development of critical illness in order to render optimal treatment.

    The screening was based on Least Absolute Shrinkage and Selection Operator (LASSO) regression in pretesting patients.

    The estimated coefficients from logistic regression were used to construct a predictive risk score to estimate the risk that a hospitalized patient with COVID-19 will develop critical illness, with accuracy measured by the area under the receiver operating characteristic curve.

    In order to enhance the accuracy and usefulness of the innovative clinical risk score, it would be helpful to examine additional samples based on:

    (1) a larger sample of patients from a wide range of countries to establish an unbalanced dynamic panel data set;

    (2) extending the data set of patients beyond 31 January 2020;

    (3) using samples from different countries comprising younger patients;

    (4) using samples from different countries comprising older patients with comorbidities;

    (5) using samples from different countries according to low, moderate and high risk categories of patients;

    (6) testing the statistical validity of the LASSO and logistic regression models using functional form tests;

    (7) testing the significance of the 62 omitted variables from the original 72;  

    (8) validating the predicted critical illnesses using the mortality rates of critically ill patients;

    (9) checking the sensitivity of the estimates from the logistic regressions used in the clinical risk score;

    (10) varying the threshold of missing values from the arbitrary 20%;    

    (11) using a larger data set to determine how the estimates of the LASSO and logistic regressions change dynamically over time;

    (12) using a larger data set to check the accuracy of the predicted critical illnesses according to different rates of confirmed COVID-19 cases.

    Further extensive empirical analysis based on clinically controlled trials across different countries and patient cohorts will enable an accurate evaluation of the innovative Clinical Risk Score as a powerful predictor of critical COVID-19 illness.
    CONFLICT OF INTEREST: None Reported
    READ MORE
    Original Investigation
    May 12, 2020

    Development and Validation of a Clinical Risk Score to Predict the Occurrence of Critical Illness in Hospitalized Patients With COVID-19

    Author Affiliations
    • 1National Clinical Research Center for Respiratory Disease, The First Affiliated Hospital of Guangzhou Medical University, Guangzhou, China
    • 2Department of Thoracic Surgery, The First Affiliated Hospital of Guangzhou Medical University, Guangzhou, China
    • 3Department of Rheumatology, The First Affiliated Hospital, Sun Yat-sen University, Guangzhou, China
    • 4Hankou Hospital, Wuhan, China
    • 5Department of Intensive Care Unit, The First Affiliated Hospital of Guangzhou Medical University, Guangzhou, China
    • 6Jinyintan Hospital, Wuhan, China
    • 7Tongji Medical College, Union Hospital, Huazhong University of Science and Technology, Wuhan, Hubei, China
    • 8The First People Hospital of Foshan, Foshan, China
    • 9Nanhai Hospital, Foshan, China
    • 10Daye Hospital, Hubei, China
    • 11The Sixth Affiliated Hospital of Guangzhou Medical University, Qingyuan, China
    JAMA Intern Med. 2020;180(8):1081-1089. doi:10.1001/jamainternmed.2020.2033
    Key Points

    Question  What epidemiological and clinical characteristics are associated with the development of critical illness among patients with novel coronavirus disease 2019 (COVID-19)? Can these characteristics be used to predict which patients admitted to the hospital with COVID-19 will need admission to an intensive care unit, mechanical ventilation, or will die?

    Findings  In this study with a development cohort of 1590 patients and a validation cohort of 710 patients, a risk score was developed and validated to predict development of critical illness. We identified 10 independent predictors and developed a risk score (COVID-GRAM) that predicts development of critical illness. The risk score predictors included: chest radiography abnormality, age, hemoptysis, dyspnea, unconsciousness, number of comorbidities, cancer history, neutrophil-to-lymphocyte ratio, lactate dehydrogenase, and direct bilirubin.

    Meaning  The COVID risk score may help identify patients with COVID-19 who may subsequently develop critical illness.

    Abstract

    Importance  Early identification of patients with novel coronavirus disease 2019 (COVID-19) who may develop critical illness is of great importance and may aid in delivering proper treatment and optimizing use of resources.

    Objective  To develop and validate a clinical score at hospital admission for predicting which patients with COVID-19 will develop critical illness based on a nationwide cohort in China.

    Design, Setting, and Participants  Collaborating with the National Health Commission of China, we established a retrospective cohort of patients with COVID-19 from 575 hospitals in 31 provincial administrative regions as of January 31, 2020. Epidemiological, clinical, laboratory, and imaging variables ascertained at hospital admission were screened using Least Absolute Shrinkage and Selection Operator (LASSO) and logistic regression to construct a predictive risk score (COVID-GRAM). The score provides an estimate of the risk that a hospitalized patient with COVID-19 will develop critical illness. Accuracy of the score was measured by the area under the receiver operating characteristic curve (AUC). Data from 4 additional cohorts in China hospitalized with COVID-19 were used to validate the score. Data were analyzed between February 20, 2020 and March 17, 2020.

    Main Outcomes and Measures  Among patients with COVID-19 admitted to the hospital, critical illness was defined as the composite measure of admission to the intensive care unit, invasive ventilation, or death.

    Results  The development cohort included 1590 patients. the mean (SD) age of patients in the cohort was 48.9 (15.7) years; 904 (57.3%) were men. The validation cohort included 710 patients with a mean (SD) age of 48.2 (15.2) years, and 382 (53.8%) were men and 172 (24.2%). From 72 potential predictors, 10 variables were independent predictive factors and were included in the risk score: chest radiographic abnormality (OR, 3.39; 95% CI, 2.14-5.38), age (OR, 1.03; 95% CI, 1.01-1.05), hemoptysis (OR, 4.53; 95% CI, 1.36-15.15), dyspnea (OR, 1.88; 95% CI, 1.18-3.01), unconsciousness (OR, 4.71; 95% CI, 1.39-15.98), number of comorbidities (OR, 1.60; 95% CI, 1.27-2.00), cancer history (OR, 4.07; 95% CI, 1.23-13.43), neutrophil-to-lymphocyte ratio (OR, 1.06; 95% CI, 1.02-1.10), lactate dehydrogenase (OR, 1.002; 95% CI, 1.001-1.004) and direct bilirubin (OR, 1.15; 95% CI, 1.06-1.24). The mean AUC in the development cohort was 0.88 (95% CI, 0.85-0.91) and the AUC in the validation cohort was 0.88 (95% CI, 0.84-0.93). The score has been translated into an online risk calculator that is freely available to the public (http://118.126.104.170/)

    Conclusions and Relevance  In this study, a risk score based on characteristics of COVID-19 patients at the time of admission to the hospital was developed that may help predict a patient’s risk of developing critical illness.

    Introduction

    The outbreak of the novel coronavirus disease 2019 (COVID-19) began in Wuhan, China in December 2019. Since then, it has rapidly spread around the world. As of April 16, 2020, the WHO reported a total of 1 995 983 COVID-19 cases globally, with average mortality of 6.57%.

    The clinical spectrum of COVID-19 pneumonia ranges from mild to critically ill cases. Patients with mild disease present with symptoms of fever and cough, followed by sputum production and fatigue. Sepsis, respiratory failure, acute respiratory distress syndrome, heart failure, and septic shock are commonly observed in critically ill patients.1

    According to the largest current report from the Chinese Center for Disease Control and Prevention with 72 314 cases, 58 574 patients (81%) were classified as mild, 10 124 (14%) were classified as severe, and 3616 (5%) were considered critical illness. The average case-fatality rate was 2.3%, but mortality was as high as 49% in patients with critical illness.2 Among 201 patients in Wuhan, Wu et al3 reported that risk factors associated with development of acute respiratory distress syndrome and death included older age, neutrophilia, organ dysfunction, coagulopathy, and elevated D-dimer levels.

    Early detection of patients who are likely to develop critical illness is of great importance and may aid in delivering proper care and optimizing use of limited resources. We aimed to construct a risk prediction score based on a nationwide cohort of Chinese patients with COVID-19 to help identify patients at the time of hospital admission who are likely to develop critical illness.

    Methods
    Data Sources and Processing

    This study was approved by the ethics committee of the First Affiliated Hospital of Guangzhou Medical University; written informed consent was waived owing to the use of deidentified retrospective data. On behalf of the National Clinical Research Center for Respiratory Disease and collaborating with the National Health Commission of the People’s Republic of China, we established a retrospective cohort to study COVID-19 cases throughout China. We obtained medical records from laboratory-confirmed hospitalized cases with COVID-19 reported to the China National Health Commission between November 21, 2019 and January 31, 2020, as previously described.4 The National Health Commission of China requested that all 1855 hospitals in China that were designated to care for COVID-19 patients submit the clinical records of all hospitalized COVID-19 cases without selection to the database by January 31, 2020. For the development cohort, we used data from the 575 hospitals that contributed clinical data by the deadline.

    COVID-19 diagnoses were confirmed by positive high-throughput sequencing or real-time reverse-transcription polymerase-chain-reaction (RT-PCR) assay for nasal and pharyngeal swab specimens. A team of experienced respiratory clinicians reviewed, abstracted and cross-checked the data. Each record was checked independently by 2 clinicians. We included all patients with data on clinical status at hospitalization (laboratory findings, clinical symptoms and signs, severity, and discharge status).

    Potential Predictive Variables

    Potential predictive variables included the following patient characteristics at hospital admission: clinical signs and symptoms, imaging results, laboratory findings, demographic variables, and medical history. Demographic variables collected for the study included age, sex, smoking status, exposure to Wuhan (including Wuhan residency, travel history to Wuhan, or contact with people from Wuhan), residency in Hubei province, and time between onset of symptoms to admission. Medical history included number of comorbidities, chronic obstructive pulmonary disease, diabetes, hypertension, coronary artery disease, cerebrovascular disease, hepatitis B, cancer, chronic renal disease, immunodeficiency disease, and pregnancy. Clinical signs and symptoms included categorical and continuous variables: first body temperature, respiratory rate, heart rate, cardiac arrhythmia, systolic blood pressure, diastolic blood pressure, symptoms rating, fever, conjunctival congestion, nasal congestion, headache, cough, expectoration, sore throat, fatigue, hemoptysis, dyspnea, nausea and vomiting, diarrhea, arthralgia and myalgia, rigor, throat blockage, tonsillar enlargement, enlarged lymph nodes, skin rash, and unconsciousness. Imaging results included chest radiography (CXR) abnormality, the severity of CXR abnormality, chest computed tomographic (CT) imaging abnormality, and the severity of CT abnormality. Laboratory findings included partial arterial oxygen pressure, oxygen saturation, white blood cell, lymphocyte, and platelet counts, neutrophil to lymphocyte ratio, and levels of hemoglobin, C-reactive protein, procalcitonin, lactate dehydrogenase, aspartate transaminase, direct bilirubin, indirect bilirubin, total bilirubin, creatine kinase, creatinine, hypersensitive troponin I, albumin, serum sodium, serum potassium, serum chlorine, D-dimer levels, prothrombin time, and activated partial thromboplastin time.

    Outcomes

    We defined the severity of COVID-19 (severe vs nonsevere) based on the American Thoracic Society guidelines for community-acquired pneumonia given the extensive acceptance of this guideline.5 We defined critical COVID-19 illness as a composite of admission to the intensive care unit (ICU), invasive ventilation, or death. We adopted this composite end point because admission to ICU, invasive ventilation, and death are serious outcomes of COVID-19 that have been adopted in previous studies to assess the severity of other serious infectious diseases.5,6

    Variable Selection and Score Construction

    All 1590 patients hospitalized with COVID-19 in the development cohort were included for variable selection and risk score development. As described herein, 72 variables were entered into the selection process. Least Absolute Shrinkage and Selection Operator (LASSO) regression was applied to minimize the potential collinearity of variables measured from the same patient and over-fitting of variables. Imputation for missing variables was considered if missing values were less than 20%. We used predictive mean matching to impute numeric features, logistic regression to impute binary variables, and Bayesian polytomous regression to impute factor features. We used L1-penalized least absolute shrinkage and selection regression for multivariable analyses, augmented with 10-fold cross validation for internal validation. This is a logistic regression model that penalizes the absolute size of the coefficients of a regression model based on the value of λ. With larger penalties, the estimates of weaker factors shrink toward zero, so that only the strongest predictors remain in the model. The most predictive covariates were selected by the minimum (λ min). The R package “glmnet” statistical software (R Foundation) was used to perform the LASSO regression. Subsequently, variables identified by LASSO regression analysis were entered into logistic regression models and those that were consistently statistically significant were used to construct the risk score (COVID-GRAM),7 which was then used to construct a web-based risk calculator (http://118.126.104.170/). Data were analyzed between February 20, 2020 and March 17, 2020.

    Assessment of Accuracy

    The accuracy of COVID risk score was assessed using the area under the receiver-operator characteristic curve (AUC). We also used the AUC to compare the accuracy of the COVID-GRAM with CURB-6 models,8 which have been used in classification of the severity of community-acquired pneumonia. For internal validation of the accuracy estimates and to reduce overfit bias, we used 200 bootstrap resamples. Statistical analysis was performed with R software (version 3.6.2, R Foundation), and P < .05 was considered statistically significant.

    Score Validation

    To validate the generalizability of COVID risk score, we used data from hospitals that were not included in the development cohort including 710 patients. Data for the validation cohort were pooled from 4 sources: (1) a multicenter cohort of hospitals from 10 cities in Hubei province that missed the deadline for data submission, but subsequently submitted data on cases admitted before the January 31, 2020; (2) Daye Hospital (near Wuhan); (3) The First People’s Hospital of Foshan (Guangdong province), and Nanhai People’s Hospital of Foshan (Guangdong province). The later 3 hospitals reported up-to-date data as of February 28, 2020.

    The variables required for calculating the COVID risk score from the validation cohort were collected and cross-checked by 2 experienced physicians (C.Z.S. and C.A.L.) and the risk score was calculated as described herein for the development cohort.

    Results
    Characteristics of the Development Cohort

    In the development cohort, we collected data from 1590 patients from 575 hospitals in 31 provincial administrative regions between November 21, 2019 and January 31, 2020. At hospital admission, 24 of 1590 patients (1.5%) were considered to be severe and the rest (1566 [98.5%]) were considered to be mild according to the American Thoracic Society guideline.5 A total of 131 patients eventually developed critical illness (8.2%). The overall mortality was 3.2% and 1334 patients (83.9%) had a history of exposure to Wuhan.

    Overall, the mean (SD) age of patients in the cohort was 48.9 (15.7) years; 904 patients (57.3%) were men and 399 (25.1%) had at least 1 coexisting condition, including hypertension (269 [16.9%]), diabetes (130 [8.2%]), and cardiovascular disease (59 [3.7%]) as the top 3 comorbidities (Table 1). Fever (1351 [88.0%]), dry cough (1052 [70.2%]), fatigue (584 [42.8%]), productive cough (513 [36.0%]), and shortness of breath (331 [23.7%]) were the most common symptoms. Most patients (1130 [71.1%]) had abnormal chest CT findings. Laboratory findings of the development cohort are presented in Table 2.

    Predictor Selection

    Seventy-two variables measured at hospital admission (Table 1 and Table 2) were included in the LASSO regression. After LASSO regression selection (eFigure 1 in the Supplement), 19 variables remained significant predictors of critical illness, including clinical features and blood test results, CXR abnormality, age, exposure to Wuhan, first and highest body temperature, respiratory rate, systolic blood pressure, hemoptysis, dyspnea, skin rash, unconsciousness, number of comorbidities, chronic obstructive pulmonary disease (COPD), cancer, oxygen saturation levels, neutrophils, neutrophil to lymphocyte ratio, lactate dehydrogenase, direct bilirubin, and creatinine levels.

    Inclusion of these 19 variables in a logistic regression model resulted in 10 variables that were independently statistically significant predictors of critical illness and were included in risk score. These variables included CXR abnormality (OR, 3.39; 95% CI, 2.14-5.38; P < .001), age (OR, 1.03; 95% CI, 1.01-1.05; P = .002), hemoptysis (OR, 4.53; 95% CI, 1.36-15.15; P = .01), dyspnea (OR, 1.88; 95% CI, 1.18-3.01; P = .01), unconsciousness (OR, 4.71; 95% CI, 1.39-15.98; P = .01), number of comorbidities (OR, 1.60; 95% CI, 1.27-2.00; P < .001), cancer history (OR, 4.07; 95% CI, 1.23-13.43; P = .02), neutrophil-to-lymphocyte ratio (OR, 1.06; 95% CI, 1.02-1.10; P = .003), lactate dehydrogenase (OR, 1.002; 95% CI, 1.001-1.004, P < .001), and direct bilirubin (OR, 1.15; 95% CI, 1.06-1.24; P = .001) (Table 3).

    Construction of the Risk Score and Web-Based Calculator

    The COVID risk score was constructed based on the coefficients from the logistic model. We used the following formulas for the logistic model to calculate the probability and 95% confidence intervals9: probability = exp( ∑ β × X)/[1+ exp( ∑ β × X)], lower limit of 95% CI = exp[ ∑ Xn × βn− ∑ z × SE(β)]/{1+exp[ ∑ Xn × βn- ∑ z × SE(β)]}, upper limit of 95% CI = exp[ ∑ Xn × βn+ ∑ z × SE(β)]/{1+exp[ ∑ Xn × βn+ ∑ z × SE(β)]}.

    An online calculator based on COVID-GRAM was developed to allow clinicians to enter the values of the 10 variables required for the risk score with automatic calculation of the likelihood (with 95% CIs) that a hospitalized patient with COVID-19 will develop critical illness (http://118.126.104.170/) (Figure)

    The Performance of COVID Risk Score

    By internal bootstrap validation, the mean AUC based on data from the development cohort was 0.88 (95% CI, 0.85-0.91) (eFigure 2 in the Supplement). The AUC of COVID risk score for patients in the epicenter at Hubei was 0.87 (95% CI, 0.83-0.91) and outside Hubei was 0.82 (95% CI, 0.73-0.90). The predictive value of COVID-GRAM was higher than the CURB-6 model, which had an AUC of 0.75 (95% CI, 0.70-0.80) for correct prediction of development of critical illness (P < .001).

    Validation of COVID-GRAM

    The validation cohort included 710 patients with a mean (SD) age of 48.2 (15.2) years, 382 (53.8%) were men and 172 (24.2%) had at least 1 coexisting condition. Critical illness eventually developed in 87 (12.3%) of these patients and 8 (1.1%) died. Variables used in COVID risk score for the validation cohort are shown in Table 4; eTable 1 in the Supplement. The accuracy of COVID risk score in the validation cohort was similar to that of the development cohort with an AUC in the validation cohort of 0.88 (95% CI, 0.84-0.93) (eFigures 3 and 4 and eTable 2 in the Supplement).

    Discussion

    In this study, we developed and validated a clinical risk score and a web-based risk calculator to predict the development of critical illness among hospitalized COVID-19 infected patients. The performance of this risk score was satisfactory with accuracy based on AUCs in both the development and validation cohorts of 0.88. The web-based calculator can be used by clinicians to estimate an individual hospitalized patient’s risk of developing critical illness. The 10 variables required for calculation of the risk of developing critical illness are generally readily available at hospital admission, and the web-based calculator is easy to use. If the patient’s estimated risk for critical illness is low, the clinician may choose to monitor, whereas high-risk estimates might support aggressive treatment or admission to the ICU. We deliberately did not categorize risk into low-, moderate-, and high-risk groups, as we believe that clinicians are better informed by calculating the risk estimate for each individual patient and making decisions based on local or regional conditions. For example, in areas with good access to clinical and supportive care, patient outcomes might be optimized by deciding to provide more aggressive care to moderate risk patients. In contrast, in areas with high case volume and/or limited resources, the decision might be to provide less aggressive care to moderate-risk patients to maximize availability of ICU beds and ventilators.

    Chest radiography abnormality, age, hemoptysis, dyspnea, unconsciousness, number of comorbidities, cancer history, neutrophil-to-lymphocyte ratio, lactate dehydrogenase, and direct bilirubin were included in the COVID risk score. Previous studies have found several of these variables to be risk factors for severe illness related to COVID-19. Wu et al3 found that older age and more comorbidities were associated with a higher risk of developing ARDS in patients infected with COVID-19. A previous study10 from our group found that patients with COVID-19 with cancer had higher risk of severe events compared with patients without cancer (39% vs 18%). Zhou and colleagues1 found lower lymphocyte count, higher lactate dehydrogenase, and more imaging abnormalities in patients who died from COVID-19 disease.

    Limitations

    Potential limitations of this study include a modest sample size for constructing the risk score and a relatively small sample for validation. The data for score development and validation are entirely from China, which could potentially limit the generalizability of the risk score in other areas of the world. Additional validation studies of the COVID risk score from areas outside China should be completed.

    Conclusions

    In this study, we developed a risk score and web-based calculator to estimate the risk of developing critical illness among patients with COVID-19 based on 10 variables commonly measured on admission to the hospital. Estimating the risk of critical illness could help identify patients who are and are not likely to develop critical illness, thus supporting appropriate treatment and optimizing the use of medical resources.

    Back to top
    Article Information

    Corresponding Author: Jianxing He, MD, PhD (drjianxing.he@gmail.com), and Nan-Shan Zhong, MD, Guangzhou Institute of Respiratory Health (nanshan@vip.163.com), China State Key Laboratory of Respiratory Disease, National Clinical Research Center for Respiratory Disease, The First Affiliated Hospital of Guangzhou Medical University, 151 Yanjiang Rd, Guangzhou, Guangdong 510120, China.

    Accepted for Publication: April 20, 2020.

    Published Online: May 12, 2020. doi:10.1001/jamainternmed.2020.2033

    Author Contributions: Drs W. Liang and He had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Drs W. Liang, H. Liang, Ou, B. Chen, A. Chen, and C. Li are joint first authors.

    Concept and design: W. Liang, H. Liang, A. Chen, Xu, G. Chen, H. Guo, Zhang, Zhong, He.

    Acquisition, analysis, or interpretation of data: W. Liang, Ou, B. Chen, C. Li, Y. Li, Guan, Sang, Lu, Xu, J. Guo, Z. Chen, Zhao, S. Li, Zhang, Zhong.

    Drafting of the manuscript: W. Liang, H. Liang, B. Chen, C. Li, G. Chen, Zhao, Zhong, He.

    Critical revision of the manuscript for important intellectual content: W. Liang, Ou, A. Chen, Y. Li, Guan, Sang, Lu, Xu, H. Guo, J. Guo, Z. Chen, S. Li, Zhang, Zhong.

    Statistical analysis: W. Liang, H. Liang, Ou, B. Chen, H. Guo, Z. Chen, He.

    Obtained funding: W. Liang, A. Chen, Xu.

    Administrative, technical, or material support: Y. Li, G. Chen, J. Guo, Z. Chen, S. Li, Zhong.

    Supervision: Sang, Lu, Zhang, Zhong.

    Other - Construction of the online calculator: B. Chen.

    Conflict of Interest Disclosures: None reported.

    Funding/Support: This study is supported by China National Science Foundation (Grant No. 81871893), Key Project of Guangzhou Scientific Research Project (Grant No. 201804020030), High-level university construction project of Guangzhou medical university (Grant No. 20182737, 201721007, 201715907, 2017160107); National key R & D Program (Grant No. 2017YFC0907903 & 2017YFC0112704) and the Guangdong high level hospital construction “reaching peak” plan.

    Role of the Funder/Sponsor: The funding organizations had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

    The China Medical Treatment Expert Group for COVID-19: Zong-jiu Zhang, MD, Ya-hui Jiao, MD, Bin Du, MD, Xin-qiang Gao, MD and Tao Wei, MD (National Health Commission), Yu-fei Duan, MD and Zhi-ling Zhao, MD (Health Commission of Guangdong Province), Yi-min Li, MD, Zi-jing Liang, MD, Nuo-fu Zhang, MD, Shi-yue Li, MD, Qing-hui Huang, MD, Wen-xi Huang, MD, and Ming Li, MD (Guangzhou Institute of Respiratory Health), Zheng Chen, MD, Dong Han, MD, Li Li, MD, Zheng Chen, MD, Zhi-ying Zhan, MD, Jin-jian Chen, MD, Li-jun Xu, MD, Xiao-han Xu, MD (State Key Laboratory of Organ Failure Research, Department of Biostatistics, Guangdong Provincial Key Laboratory of Tropical Disease Research, School of Public Health, Southern Medical University); Li-qiang Wang, MD, Wei-peng Cai, MD, Zi-sheng Chen, MD (the sixth affiliated hospital of Guangzhou medical university), Chang-xing Ou, MD, Xiao-min Peng, MD, Si-ni Cui, MD, Yuan Wang, MD, Mou Zeng, MD, Xin Hao, MD, Qi-hua He, MD, Jing-pei Li, MD, Xu-kai Li, MD, Wei Wang, MD, Li-min Ou, MD, Ya-lei Zhang, MD, Jing-wei Liu, MD, Xin-guo Xiong, MD, Wei-juna Shi, MD, San-mei Yu, MD, Run-dong Qin, MD, Si-yang Yao, MD, Bo-meng Zhang, MD, Xiao-hong Xie, MD, Zhan-hong Xie, MD, Wan-di Wang, MD, Xiao-xian Zhang, MD, Hui-yin Xu, MD, Zi-qing Zhou, MD, Ying Jiang, MD, Ni Liu, MD, Jing-jing Yuan, MD, Zheng Zhu, MD, Jie-xia Zhang, MD, Hong-hao Li, MD, Wei-hua Huang, MD, Lu-lin Wang, MD, Jie-ying Li, MD, Li-fen Gao, MD, Jia-bo Gao, MD, Cai-chen Li, MD, Xue-wei Chen, MD, Jia-bo Gao, MD, Ming-shan Xue, MD, Shou-xie Huang, MD, Jia-man Tang, MD, Wei-li Gu, MD, Jin-lin Wang, MD (Guangzhou Institute of Respiratory Health).

    Acknowledgement: Special thanks to Ling Sang, MD, supporting Wuhan Jinyintan Hospital; Yuan-da Xu, MD, supporting Wuhan Union Hospital; Ai-lan Chen MD, supporting Wuhan Hankou Hospital; Guo-qiang Chen, MD, and Hai-yan Guo, MD, Foshan Hospital; and Jun Guo, MD, Daye Hospital; and the China Medical Treatment Expert Group for COVID-19 for providing validation data. See further acknowledgements in the eAppendix in the Supplement.

    References
    1.
    Zhou  F, Yu  T, Du  R,  et al.  Clinical course and risk factors for mortality of adult inpatients with COVID-19 in Wuhan, China: a retrospective cohort study.   Lancet. 2020;395(10229):1054-1062. doi:10.1016/S0140-6736(20)30566-3PubMedGoogle ScholarCrossref
    2.
    Wu  Z, McGoogan  JM.  Characteristics of and important lessons from the coronavirus disease 2019 (COVID-19) outbreak in China: summary of a report of 72 314 cases from the Chinese Center for Disease Control and Prevention.   JAMA. 2020;(Feb):24. doi:10.1001/jama.2020.2648PubMedGoogle Scholar
    3.
    Wu  C, Chen  X, Cai  Y,  et al.  Risk factors associated with acute respiratory distress syndrome and death in patients with coronavirus disease 2019 pneumonia in Wuhan, China.   JAMA Intern Med. 2020;(Mar):13. doi:10.1001/jamainternmed.2020.0994PubMedGoogle Scholar
    4.
    Guan  WJ, Ni  ZY, Hu  Y,  et al; China Medical Treatment Expert Group for Covid-19.  Clinical characteristics of coronavirus disease 2019 in China.   N Engl J Med. 2020;(Feb):28. doi:10.1056/NEJMoa2002032PubMedGoogle Scholar
    5.
    Metlay  JP, Waterer  GW, Long  AC,  et al.  Diagnosis and treatment of adults with community-acquired pneumonia. an official clinical practice guideline of the American Thoracic Society and Infectious Diseases Society of America.   Am J Respir Crit Care Med. 2019;200(7):e45-e67. doi:10.1164/rccm.201908-1581STPubMedGoogle ScholarCrossref
    6.
    Gao  HN, Lu  HZ, Cao  B,  et al.  Clinical findings in 111 cases of influenza A (H7N9) virus infection.   N Engl J Med. 2013;368(24):2277-2285. doi:10.1056/NEJMoa1305584PubMedGoogle ScholarCrossref
    7.
    Iasonos  A, Schrag  D, Raj  GV, Panageas  KS.  How to build and interpret a nomogram for cancer prognosis.   J Clin Oncol. 2008;26(8):1364-1370. doi:10.1200/JCO.2007.12.9791PubMedGoogle ScholarCrossref
    8.
    Neill  AM, Martin  IR, Weir  R,  et al.  Community acquired pneumonia: aetiology and usefulness of severity criteria on admission.   Thorax. 1996;51(10):1010-1016. doi:10.1136/thx.51.10.1010PubMedGoogle ScholarCrossref
    9.
    Xu  J, Long  JS.  Confidence intervals for predicted outcomes in regression models for categorical outcomes.   Stata Journal. 2005;5(4):537-559. doi:10.1177/1536867X0500500405Google ScholarCrossref
    10.
    Liang  W, Guan  W, Chen  R,  et al.  Cancer patients in SARS-CoV-2 infection: a nationwide analysis in China.   Lancet Oncol. 2020;21(3):335-337. doi:10.1016/S1470-2045(20)30096-6PubMedGoogle ScholarCrossref
    ×