Get it fast or get it right? Covid-19 testing and the FDA*

Five months after it first issued nonbinding policy guidance on Covid-19 test authorization, the FDA still has not set specific performance parameters (limit of detection) for these tests, according to spokesperson for the agency.

Why does this matter?

When I was developing and validating new, rapid methods for finding Salmonella in a food sample, I always had to keep one question in mind.

The question had nothing to do with the speed of the test. It had nothing to do with the cost of the test.

It had everything to do with the sensitivity of the test and how its performance compared to the FDA’s “gold standard” test.

How does this relate to Covid-19 testing?

In its rush to make Covid-19 diagnostic tests available in large quantities and across the country, the FDA has resorted to an Emergency Use Authorization (EUA) policy. This means that the FDA has relied upon the developers of these tests to validate their own tests.

On May 11, 2020, the FDA issued an updated nonbinding guidance, outlining the agency’s EUA policy for diagnostic tests. In this guidance document, the agency recommended that developers of Covid-19 diagnostic tests evaluate and submit the analytical limit of detection and assess traceability of their product with any FDA-recommended reference material as a condition of the authorization.

The FDA’s Molecular Diagnostic Template for Commercial Manufacturers sets out these requirements and recommendations in greater detail, including maximum acceptable criteria for false positive (5%) and false negative (5%) results.

However, the FDA has not – nor does it plan to – set specific limit of detection criteria for Emergency Use Authorization of a Covid-19 diagnostic test.

The limit of detection is the lowest concentration of Covid-19 virus particles or virus RNA required to produce a positive test result. The lower the limit of detection, the less likely it is that a test will produce a false negative result.

What has FDA done so far?

The FDA has authorized 270 diagnostic tests for Covid-19, including 213 molecular (i.e., nucleic acid tests), 52 antibody tests and 5 antigen tests so far.

As of September 23rd, the FDA contacted developers of 165 of the authorized molecular assays, to request shipping information for the purpose of sending a Covid-19 diagnostic test reference panel. The purpose of the panel was to obtain directly comparable limit of detection data from the various tests developers.

Eighteen of the developers did not provide shipping information to the FDA and were not sent the test panel.

Forty-five of the developers either have not returned any data to the FDA or the data they returned were “not interpretable.”

Eighteen sets of data are currently “under interactive review.”

Among the tests for which data have been returned and reviewed, the limit of detection ranges from 180 units per millileter (units/ml) to 600,000 units/ml. (One milliliter is roughly equivalent to ¼ teaspoon of liquid.)

The lower the number, the more sensitive the test.

The Abbott ID NOW Covid-19 test, which purports to produce a result in 15 minutes, reported a limit of detection of 300,000 units/ml.

What does the FDA consider to be an acceptable limit of detection?

“The FDA has not yet set optimal ranges,” according to an FDA spokesperson.

When asked for additional clarification, the spokesperson replied that the limit of detection data are a relative, rather than an absolute, indication of test performance. While acknowledging that a lower limit of detection value indicates a more sensitive test, FDA has declined to set a baseline acceptable performance.

What is viral load and how does this relate to limit of detection?

Viral load is the number of virus particles in a sample from a patient, and will vary depending on the individual, the severity of symptoms, how carefully the sample was taken, and the stage of progression of the disease.

In a May 2020 issue of The Lancet, a group of researchers published viral load data from 23 Covid-19 patients in two Hong Kong hospitals. The initial viral load in respiratory tract samples for these patients was as low as ~8,100/ml and as low as ~15,000/ml in patients with severe disease.

The highest recorded viral loads were in the tens of millions/ml in both sets of patients.

No data were available for asymptomatic individuals.

Developers of thirteen of the Covid-19 diagnostic tests authorized by FDA reported a limit of detection of 180,000 units/ml, or more than ten times higher than the low end of the viral loads reported in the Hong Kong study. An additional seventeen tests reported a limit of detection of 18,000 units/ml, teetering on the brink of being unable to detect a low viral load.

If all patients and asymptomatic infected individuals carried viral loads in the tens of millions, this wouldn’t matter.

Unfortunately, that is not the case. A test that needs a high number of virus particles present in order to find the infection is more likely to produce false negative results in patients with a low viral load.

Why do false negative results matter?

  • They matter because false negative results produce a false sense of security.
  • They matter because the medical community relies on test results to trigger contact tracing and quarantine actions.
  • They matter because they affect the overall statistics on Covid-19 infections.
  • They matter because a person can test negative, have no symptoms, and still be able to infect others.

We learned of an example of this just today, when White House Press Secretary Kayleigh McEnany revealed that she had tested positive for coronavirus after having tested negative several days in a row and having interacted during that time, without a mask, with multiple individuals.

Where does Canada stand?

As of today, Canada has authorized 37 Covid-19 diagnostic tests for use, including 27 tests based on nucleic acid technology (so-called molecular tests). 

Canada has established minimum requirements for sensitivity and specificity of Covid-19 serological tests (i.e., for finding antibodies to the virus in the blood), but not for nucleic acid tests.

A nucleic acid tests developer applying for authorization in Canada must provide, in addition to other information, “the known information in relation to the quality, safety and effectiveness of the device.” 

A test is not approved unless Health Canada concludes that there is, “sufficient evidence to support the conclusion that the benefits associated with the COVID-19 medical device outweigh the risks, having regard to the uncertainties relating to the benefits and risks and the urgent public health need,” and that “the health or safety of patients, users or other persons will not be unduly affected.”

Although Canada has decided to use the FDA’s EUA as a starting point for authorizing a diagnostic test for use, Health Canada will be assessing the tests authorized in Canada in light of the data published by the FDA “to see whether the published data would indicate a need for any revisions to the labelling information,” according to a Health Canada spokesperson.

Ontario, Canada’s largest province, is using two of the approved tests on a routine basis: one from Roche and the other from Abbott, according to a spokesperson for the Ontario Ministry of Health. 

The Roche test has a lower limit of detection of 1800 units/ml and the detection limit of the Abbott test is 5400 units/ml, according to the data posted by FDA. Both of these tests should be sensitive enough to find Covid-19 carriers with relatively low virus loads, at least in theory.

What are the FDA’s plans going forward?

At the moment, the FDA has no specific plans to delist developers who have not responded to the limit of detection reference panel invitation or who responded but never returned data.

When asked whether the FDA would review its EUA decisions for the worst performing tests in the list, an agency spokesperson responded, “In the published data, a lower LoD represents a test’s ability to detect a smaller amount of viral material in a given sample, signaling a more sensitive test. However, the data does not indicate how sensitive a particular test is, and, therefore, cannot be used by itself to determine whether to authorize a test or other regulatory action. The FDA will continue to update the table as it receives additional results. The data does not indicate the clinical sensitivity of a particular test, and, therefore, cannot be used by itself to determine whether or not to authorize a test or take other regulatory action. Instead, the data gives laboratories, healthcare providers, and patients a new resource on the relative performance of tests they can use to better inform which tests they select to use.”

Is there a place for a rapid point-of-care test with a high limit of detection?

Some scientists think there is.

Writing in the New England Journal of Medicine, Dr. Michael Mina (Harvard T.H. Chan School of Public Health) and Drs. Roy Parker and Daniel Larremore (Univeristy of Colorado) have proposed shifting emphasis away from the slower, more sensitive tests to frequent use of inexpensive, less sensitive, rapid testing.

They believe that, “…the FDA, the CDC, the National Institutes of Health, and others must encourage structured evaluations of tests in the context of planned testing regimens to identify those that will provide the best Covid filters. Frequent use of cheap, simple, rapid tests will accomplish that aim, even if their analytic sensitivities are vastly inferior to those of benchmark tests. Such a regimen can help us stop Covid in its tracks.”

What’s the bottom line?

Decisions about patient care, contact tracing, quarantine and epidemiology must be based on knowledge to be effective. This includes solid information on the performance of the various tools used to make these decisions.

At the moment, the medical profession is feeling its way down a dark passage, trying to interpret the results of diagnostic tests without having in hand basic information as to the reliability of these tests.

The FDA’s survey of limit of detection performance is a first step. The next step will be for the agency, and for Health Canada, to decide which of the surveyed diagnostic tests deserve to be authorized for use.

At the very least, test developers who decline to participate in a limit of detection study, who agree to participate but does not return data, or whose data are unintelligible should not be granted continued Emergency Use Authorization for their tests.

*Content revised and updated to reflect additional information received after the article was posted.

Thomson Onion Salmonella Outbreak: Is CDC Missing in Action?

On July 21, 2020, the US Centers for Disease Control and Prevention (CDC) informed the public of an outbreak of Salmonella Newport infections.

On July 24th, the Public Health Agency of Canada (PHAC) informed the public of an outbreak of Salmonella Newport infections apparently caused by the same outbreak strain as CDC was finding in the United States.

At the time of the initial reports, neither agency had determined the source of the outbreak.

On July 30th, PHAC updated its outbreak advisory, informing Canadians that the outbreak was linked to consumtion of red onions imported from the United States. That same day, the Canadian Food Inspection Agency (CFIA) posted a recall notice for red onions imported by Sysco in Western Canada.

Using the Canadian data as its starting point, on July 31st, the US Food and Drug Administration (FDA) and CDC announced that the US outbreak was linked to consumption of red onions produced by Thomson International, Inc. of Bakersfield, California.

Thomson International is a family-owned business, incorporated in California.

On August 1st, Thomson recalled its entire harvest of red, yellow, white, and sweet yellow onions from the 2020 growing season – approximately 18,750 tons of onions. The onions were distributed across the United States and exported to Canada.

CDC issued status updates of the size and scope of the US outbreak on August 3rd, August 7th, August 18th and September 1st, and has not been heard from since.

PHAC issued status updates of the size and scope of the Canadian outbreak on August 2nd, August 7th, August 14th, August 21st, August 31st and September 14th.

By August 7th, FDA had initiated its on-site investigation of Thomson’s Bakersfield facility, looking for the source of the Salmonella Newport contamination. By August 11th, FDA personnel had submitted 370 samples to the agency’s lab for Salmonella testing, including 278 swab samples, 82 onion samples, and 10 miscellaneous environmental samples, according to information obtained by eFoodAlert in response to a Freedom of Information Act request.

Not a single sample contained Salmonella.

The FDA investigation is still in progress. However, with the growing season complete and the packing plant idle, the chances of finding the source of the Salmonella Newport diminish day by day.

As of the last report from CDC, 1012 individuals in 47 states have been infected with Salmonella Newport as a result of having consumed contaminated onions. Only Louisiana, Oklahoma and Vermont have not reported any outbreak cases. Although there have been no deaths associated with this outbreak, 136 (more than 13%) of the victims have required a hospital stay.

In Canada, there have been 506 confirmed cases of Salmonella Newport in seven provinces, and 71 people (14%) have been hospitalized.

Canada v. USA – A Performance Comparison

Why was CDC unable to determine the link between red onions and the Salmonella Newport outbreak until after PHAC had made the connection?

Why has CDC not provided an update to its outbreak status report in three weeks?

Why does Canada appear to have been much harder hit by this outbreak than the United States – 13.7 cases per million Canadians versus only 3.1 per million Americans? Is this due to some quirk of distribution, or have PHAC and its provincial partners done a better job of reporting than CDC and the various state health agencies?

Has the Covid-19 pandemic hit CDC so hard that it no longer has the resources to follow-up on illness outbreaks elsewhere?





Recalls and Alerts: September 13 – 16, 2020

Here is today’s list of food safety recalls, product withdrawals, allergy alerts and miscellaneous compliance issues. The live links will take you directly to the official recall notices and company news releases that contain detailed information for each recall and alert.

If you would like to receive automatic email alerts for all new articles posted on eFoodAlert, please submit your request using the sidebar link.

United States

REGULATORY ACTION UNDER FDA PRODUCE SAFETY RULE: FDA announces a consent decree of permanent injunction against Fortune Food Product, Inc., an Illinois-based processor of sprouts and soy products. The consent decree prohibits Fortune Food Product, Inc. from growing, harvesting, packing and holding sprouts and soy products at or from their facility, or any other facility, until certain requirements are met. The consent decree requires the defendants to, among other things, take corrective actions and notify the FDA before such operations may resume.

Allergy Alert: Willow Tree Poultry Farm, Inc. recalls Willow Tree Premium White Meat CHICKEN SALAD Classic (15 oz; Sell by 9/30/20 and a time stamp of 13:00:00 through 17:00:00) due to undeclared walnuts.


OUTBREAK ALERT UPDATE: PHAC reports 506 confirmed cases of Salmonella Newport illness linked to onions in British Columbia (116), Alberta (292), Saskatchewan (34), Manitoba (25), Ontario (14), Quebec (24) and Prince Edward Island (1). The onions were produced by Thomson International, Inc. (Bakersfield, California) and imported into Canada.

Food Safety Recall: Poissonnerie Québec Océan recalls three products due to lack of required labeling regarding mandatory refrigeration. Please refer to the recall notice for a list of affected products.


Allergy Alert (Iceland): Matvælastofnun alerts the public that Bónus Kjarnabrauði bread (Best before 15.09.2020) contains undeclared lupines.

Allergy Alert (Italy): Shankara Bulgaria ltd recalls GIUSTO Senza Glutine brand COOKIE AVENA MIRTILLI ROSSI E CIOCCOLATO / Oat cookies with cranberries and chocolate (50g; Lot No./Best before 03-05-2021) due to undeclared soy.

Allergy Alert (Italy): Shankara Bulgaria ltd recalls GIUSTO Senza Glutine brand COOKIE AVENA COCCO ARANCIA / Oat cookies with orange and coconut (50g; Lot No./Best before 23-04-2021) due to undeclared soy.

Allergy Alert (Luxembourg): Industry recalls TARTICHOC brand Pâte à tartiner au chocolat fondant (Best before 05/2021 to 08/2021) and TARTICHOC brand Pâte à tartiner au chocolat lait (Best before 05/2021 to 07/2021) due to undeclared hazelnuts.

Allergy Alert (UK): SPAR recalls SPAR Pesto Chicken Breast Sizzlers (330g; Batch codes 110920016 & 140920026; Use by 22 September 2020 & 23 September 2020, respectively) due to undeclared egg.

Food Safety Recall (Belgium): WESTVLEES recalls Carrefour bio brand BIO AMERICAIN NATURE / raw ground meat (250g; Lot 637; Expiry date 20/09/2020) due to Listeria monocytogenes contamination.

Food Safety Recall (Belgium): Delikip Bv recalls pilons de poulet marinés / marinated chicken legs (4 pieces / 4.2-4.8 kg; Lot #11092020; Best before 18/09/2020) due to possible Salmonella contamination.

Food Safety Recall (France): La Fromagerie du Pays Welsche recalls La Fromagerie du Pays Welsche brand Munster fermier AOP traditionnel / Traditional Munster cheese (Lot #A189; Best before 15/09/2020) due to Listeria monocytogenes contamination.

Food Safety Recall (Ireland): James O’Brien Farmhouse Cheese recalls All cheeses produced or cut by James O’Brien Farmhouse Cheese including brie, cheddar, feta, gouda and halloumi (all batches and ‘best before’ dates) due to Listeria monocytogenes contamination.

Food Safety Recall (Italy): Sicily Food SRL recalls UNES brand SALMONE NORVEGESE AFFUMICATO / Norwegian smoked salmon (50g; Lot #SA75-22620; Best before 22-09-2020) due to Listeria monocytogenes contamination.