National Academies Offers Guidance to Decision-Makers on Evaluating COVID-19 Data | Infectious Diseases | JAMA Health Forum | JAMA Network
[Skip to Content]
Curated health policy research and original commentary from across the JAMA Network
health-forum_ChannelsBanner
[Skip to Content Landing]
In the News
COVID-19

National Academies Offers Guidance to Decision-Makers on Evaluating COVID-19 Data

  • 1Consulting Editor, JAMA Health Forum and JAMA

A rapid expert consultation from the National Academies of Sciences, Engineering, and Medicine says that decision-makers leading the response to COVID-19 should consider specific criteria when evaluating numbers of confirmed cases, hospitalizations, excess deaths, and other data to make COVID-19–related policy decisions, such as restricting or allowing public gatherings or mandating the wearing of face masks in public places.

Such decisions are often colored by competing interests and political concerns. For example, the governors of Arizona and Texas, who insisted on reopening businesses despite rising numbers of COVID-19 cases, clashed with mayors and other local leaders in their respective states who requested authority in their own cities or counties over such actions as reimposing lockdowns or requiring people to wear masks in public.

The National Academies’ guide was developed by its recently formed Societal Experts Action Network (SEAN), a group of experts in the social, behavioral, and economic sciences created to help decision-makers, such as governors, mayors, city council members, and others who are leading the response to COVID-19 in their jurisdictions. The new guide is intended to help these leaders better understand both the strengths and weaknesses of 7 types of data used as indicators for evaluating the course of COVID-19 in a community or population and use the types of data best suited to inform their decisions.

The 7 data types comprise the numbers of confirmed COVID-19 cases, emergency department visits, hospitalizations, reported confirmed COVID-19 deaths, excess deaths (from all causes, compared with comparable time periods in the past), the proportion of viral tests that are positive, and representative prevalence surveys (based on viral and antibody testing of a representative sample of individuals).

“Our intent is not to discourage decision-makers from using any of these data, as they represent the best of what is available,” said Mary Bassett, MD, MPH, cochair of SEAN’s executive committee and director of the François-Xavier Bagnoud Center for Health and Human Rights at Harvard University, in a statement. “Rather, the goal of our rapid expert consultation is to clarify the limitations of these data points and help leaders as they make decisions, such as when to allow public gatherings or reopen businesses.”

The document notes that decision-makers should consider 5 criteria when evaluating COVID-19 data: representativeness (how representative the data are for the population of interest, such as the population of a state, city, or county), bias (whether there are systematic factors that could cause the reported values to underestimate or overestimate the true value), uncertainty (such as uncertainty stemming from small sample sizes or tests with limited accuracy), time range (issues such as a time lag in reporting the data and whether time lags differ across sources), and space (whether the measures cover all geographic areas and particular areas of interest, but not beyond those areas).

For example, although the number of confirmed cases is a readily available measure, it’s likely to substantially underestimate COVID-19 prevalence in the population. However, the usefulness of this measure will increase as testing widens to include asymptomatic people and populations with milder symptoms. Another measure, the fraction of viral tests that are positive, is widely used, but it may tend to overestimate the disease’s prevalence because the people who are tested are often not representative of the population as a whole.

Hospitalization data reflect only the most severe cases of infection, but as the proportion of hospitalizations to confirmed cases decreases, it probably reflects a decline in a community’s total number of infections.

Similarly, 2 measures of deaths provide different kinds of information. The number of reported COVID-19 deaths reflects the state of the outbreak several weeks earlier (because of the long course of COVID-19 infection). The best indicator of the effects of the pandemic on mortality is excess deaths, but this measure represents a mix of confirmed COVID-19 deaths and deaths from other causes because of the potential for misclassifying cause of death.

“Each data type represents a piece of the puzzle, and when used in combination, the various types form a clearer picture of how the disease is spreading and its severity,” the authors wrote. “Because any single data type is likely to yield an under- or over-estimate of the extent and spread of the disease, it is important to consider multiple data types and be cautious in relying on estimates without considering sources of bias.”

The network also released a new interactive web tool designed to help decision-makers further explore the strengths and weaknesses of the 7 data types.

“With greater understanding of the strengths and limitations of these data, decision makers can make better decisions,” the authors noted.

Limit 200 characters
Limit 25 characters
Conflicts of Interest Disclosure

Identify all potential conflicts of interest that might be relevant to your comment.

Conflicts of interest comprise financial interests, activities, and relationships within the past 3 years including but not limited to employment, affiliation, grants or funding, consultancies, honoraria or payment, speaker's bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued.

Err on the side of full disclosure.

If you have no conflicts of interest, check "No potential conflicts of interest" in the box below. The information will be posted with your response.

Not all submitted comments are published. Please see our commenting policy for details.

Limit 140 characters
Limit 3600 characters or approximately 600 words
    ×