Five months after it first issued nonbinding policy guidance on Covid-19 test authorization, the FDA still has not set specific performance parameters (limit of detection) for these tests, according to spokesperson for the agency.

Why does this matter?

When I was developing and validating new, rapid methods for finding Salmonella in a food sample, I always had to keep one question in mind.

The question had nothing to do with the speed of the test. It had nothing to do with the cost of the test.

It had everything to do with the sensitivity of the test and how its performance compared to the FDA’s “gold standard” test.

How does this relate to Covid-19 testing?

In its rush to make Covid-19 diagnostic tests available in large quantities and across the country, the FDA has resorted to an Emergency Use Authorization (EUA) policy. This means that the FDA has relied upon the developers of these tests to validate their own tests.

On May 11, 2020, the FDA issued an updated nonbinding guidance, outlining the agency’s EUA policy for diagnostic tests. In this guidance document, the agency recommended that developers of Covid-19 diagnostic tests evaluate and submit the analytical limit of detection and assess traceability of their product with any FDA-recommended reference material as a condition of the authorization.

The FDA’s Molecular Diagnostic Template for Commercial Manufacturers sets out these requirements and recommendations in greater detail, including maximum acceptable criteria for false positive (5%) and false negative (5%) results.

However, the FDA has not – nor does it plan to – set specific limit of detection criteria for Emergency Use Authorization of a Covid-19 diagnostic test.

The limit of detection is the lowest concentration of Covid-19 virus particles or virus RNA required to produce a positive test result. The lower the limit of detection, the less likely it is that a test will produce a false negative result.

What has FDA done so far?

The FDA has authorized 270 diagnostic tests for Covid-19, including 213 molecular (i.e., nucleic acid tests), 52 antibody tests and 5 antigen tests so far.

As of September 23rd, the FDA contacted developers of 165 of the authorized molecular assays, to request shipping information for the purpose of sending a Covid-19 diagnostic test reference panel. The purpose of the panel was to obtain directly comparable limit of detection data from the various tests developers.

Eighteen of the developers did not provide shipping information to the FDA and were not sent the test panel.

Forty-five of the developers either have not returned any data to the FDA or the data they returned were “not interpretable.”

Eighteen sets of data are currently “under interactive review.”

Among the tests for which data have been returned and reviewed, the limit of detection ranges from 180 units per millileter (units/ml) to 600,000 units/ml. (One milliliter is roughly equivalent to ¼ teaspoon of liquid.)

The lower the number, the more sensitive the test.

The Abbott ID NOW Covid-19 test, which purports to produce a result in 15 minutes, reported a limit of detection of 300,000 units/ml.

What does the FDA consider to be an acceptable limit of detection?

“The FDA has not yet set optimal ranges,” according to an FDA spokesperson.

When asked for additional clarification, the spokesperson replied that the limit of detection data are a relative, rather than an absolute, indication of test performance. While acknowledging that a lower limit of detection value indicates a more sensitive test, FDA has declined to set a baseline acceptable performance.

What is viral load and how does this relate to limit of detection?

Viral load is the number of virus particles in a sample from a patient, and will vary depending on the individual, the severity of symptoms, how carefully the sample was taken, and the stage of progression of the disease.

In a May 2020 issue of The Lancet, a group of researchers published viral load data from 23 Covid-19 patients in two Hong Kong hospitals. The initial viral load in respiratory tract samples for these patients was as low as ~8,100/ml and as low as ~15,000/ml in patients with severe disease.

The highest recorded viral loads were in the tens of millions/ml in both sets of patients.

No data were available for asymptomatic individuals.

Developers of thirteen of the Covid-19 diagnostic tests authorized by FDA reported a limit of detection of 180,000 units/ml, or more than ten times higher than the low end of the viral loads reported in the Hong Kong study. An additional seventeen tests reported a limit of detection of 18,000 units/ml, teetering on the brink of being unable to detect a low viral load.

If all patients and asymptomatic infected individuals carried viral loads in the tens of millions, this wouldn’t matter.

Unfortunately, that is not the case. A test that needs a high number of virus particles present in order to find the infection is more likely to produce false negative results in patients with a low viral load.

Why do false negative results matter?

• They matter because false negative results produce a false sense of security.
• They matter because the medical community relies on test results to trigger contact tracing and quarantine actions.
• They matter because they affect the overall statistics on Covid-19 infections.
• They matter because a person can test negative, have no symptoms, and still be able to infect others.

We learned of an example of this just today, when White House Press Secretary Kayleigh McEnany revealed that she had tested positive for coronavirus after having tested negative several days in a row and having interacted during that time, without a mask, with multiple individuals.

Where does Canada stand?

As of today, Canada has authorized 37 Covid-19 diagnostic tests for use, including 27 tests based on nucleic acid technology (so-called molecular tests). 

Canada has established minimum requirements for sensitivity and specificity of Covid-19 serological tests (i.e., for finding antibodies to the virus in the blood), but not for nucleic acid tests.

A nucleic acid tests developer applying for authorization in Canada must provide, in addition to other information, “the known information in relation to the quality, safety and effectiveness of the device.” 

A test is not approved unless Health Canada concludes that there is, “sufficient evidence to support the conclusion that the benefits associated with the COVID-19 medical device outweigh the risks, having regard to the uncertainties relating to the benefits and risks and the urgent public health need,” and that “the health or safety of patients, users or other persons will not be unduly affected.”

Although Canada has decided to use the FDA’s EUA as a starting point for authorizing a diagnostic test for use, Health Canada will be assessing the tests authorized in Canada in light of the data published by the FDA “to see whether the published data would indicate a need for any revisions to the labelling information,” according to a Health Canada spokesperson.

Ontario, Canada’s largest province, is using two of the approved tests on a routine basis: one from Roche and the other from Abbott, according to a spokesperson for the Ontario Ministry of Health. 

The Roche test has a lower limit of detection of 1800 units/ml and the detection limit of the Abbott test is 5400 units/ml, according to the data posted by FDA. Both of these tests should be sensitive enough to find Covid-19 carriers with relatively low virus loads, at least in theory.

What are the FDA’s plans going forward?

At the moment, the FDA has no specific plans to delist developers who have not responded to the limit of detection reference panel invitation or who responded but never returned data.

When asked whether the FDA would review its EUA decisions for the worst performing tests in the list, an agency spokesperson responded, “In the published data, a lower LoD represents a test’s ability to detect a smaller amount of viral material in a given sample, signaling a more sensitive test. However, the data does not indicate how sensitive a particular test is, and, therefore, cannot be used by itself to determine whether to authorize a test or other regulatory action. The FDA will continue to update the table as it receives additional results. The data does not indicate the clinical sensitivity of a particular test, and, therefore, cannot be used by itself to determine whether or not to authorize a test or take other regulatory action. Instead, the data gives laboratories, healthcare providers, and patients a new resource on the relative performance of tests they can use to better inform which tests they select to use.”

Is there a place for a rapid point-of-care test with a high limit of detection?

Some scientists think there is.

Writing in the New England Journal of Medicine, Dr. Michael Mina (Harvard T.H. Chan School of Public Health) and Drs. Roy Parker and Daniel Larremore (Univeristy of Colorado) have proposed shifting emphasis away from the slower, more sensitive tests to frequent use of inexpensive, less sensitive, rapid testing.

They believe that, “…the FDA, the CDC, the National Institutes of Health, and others must encourage structured evaluations of tests in the context of planned testing regimens to identify those that will provide the best Covid filters. Frequent use of cheap, simple, rapid tests will accomplish that aim, even if their analytic sensitivities are vastly inferior to those of benchmark tests. Such a regimen can help us stop Covid in its tracks.”

What’s the bottom line?

Decisions about patient care, contact tracing, quarantine and epidemiology must be based on knowledge to be effective. This includes solid information on the performance of the various tools used to make these decisions.

At the moment, the medical profession is feeling its way down a dark passage, trying to interpret the results of diagnostic tests without having in hand basic information as to the reliability of these tests.

The FDA’s survey of limit of detection performance is a first step. The next step will be for the agency, and for Health Canada, to decide which of the surveyed diagnostic tests deserve to be authorized for use.

At the very least, test developers who decline to participate in a limit of detection study, who agree to participate but does not return data, or whose data are unintelligible should not be granted continued Emergency Use Authorization for their tests.

*Content revised and updated to reflect additional information received after the article was posted.