Comparative sensitivity evaluation for 122 CE-marked rapid diagnostic tests for SARS-CoV-2 antigen, Germany, September 2020 to April 2021

Abstract

Introduction

A large number of antigen-detecting rapid diagnostic tests (Ag RDT) for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) are available on the European market, both for professional use and as self-tests. Rapid tests are based on lateral flow immunochromatography using antibodies against SARS-CoV-2 proteins (antigens), present in respiratory tract specimens. By far most Ag RDT target the viral nucleoprotein, only very few assays work with spike protein detection. Viral variants of concern (VOC) contain mainly mutations in the gene encoding the spike protein, leaving the vast majority of SARS-CoV‑2 Ag RDT unaffected; however, the few SARS-CoV-2 Ag RDT based on spike protein detection should be checked at regular intervals for potential deficiencies. While PCR is still the gold standard for virus detection, there is increasing evidence that infectivity of respiratory secretions correlates with high viral loads present in the early phase of infection, e.g. before and 0–10 days after onset of symptoms. In addition to more complex and time-consuming PCR systems, Ag RDT allow rapid identification of acutely infected and potentially infectious individuals facilitating fast decisions on containment of virus spread, patient care, isolation and contact tracing [1,2]. Furthermore, Ag RDT may save limited reagents of more sensitive molecular diagnostics to serve other diagnostic needs, e.g. disease management or confirmation of Ag RDT reactive results.

In the European Union (EU), regulatory requirements for SARS-CoV-2 in vitro diagnostic medical devices (IVD) are defined by the IVD Directive 98/79/EC (IVDD) and have to be addressed by the manufacturer before access to the EU Common Market [3]. However, certification (CE marking) of SARS-CoV-2 diagnostics is currently done solely by the manufacturer (self-certification), without third party intervention. The exception are SARS-CoV-2 self-tests, where a notified body has to assess studies with lay persons performing the tests. However, owing to the urgency in the coronavirus disease (COVID-19) situation, a national derogation for CE-certification of self-tests can be agreed by the national competent authority, e. g. by relying on the performance of the same RDT cassette offered for professional use. Starting from May 2022, the IVDD will be replaced by the IVD Regulation (EU) 2017/746 (IVDR) where a risk-based classification of IVD is the basis for the scrutiny of their assessment [4,5]. The SARS-CoV-2 IVD will belong to the high-risk devices (class D) under the IVDR, requiring a notified body both for certification of the manufacturer`s quality management system and for assessment of the technical documentation of the device. Furthermore, EU reference laboratories (EURL) will be responsible for independent laboratory testing of class D devices to verify performance features and to assure batch-to-batch consistency [4]. However, for the time being, independent evaluations of SARS-CoV-2 Ag RDT that allow conclusions on their performance are largely missing.

In the current situation with absence of strict regulatory requirements for most SARS-CoV-2 IVD, the German Ministry of Health decided to link the reimbursement of SARS-CoV-2 Ag RDT to provision of evidence of essential quality features of these assays. This evidence consisted of two parts: (i) compliance with minimum criteria for RDT sensitivity (detection of > 80% of PCR-positive symptomatic patients during the first 7 days after symptom onset) and specificity (> 97% for asymptomatic persons) in studies performed by or on behalf of the manufacturer with clinical specimens, and (ii) successful outcome in the independent laboratory evaluation. Minimum criteria were jointly defined by the Paul-Ehrlich Institute (PEI) and the Robert Koch Institute (RKI), two governmental authorities in Germany [6]. Manufacturers or distributors of SARS-CoV-2 Ag RDT document for the respective SARS-CoV-2 Ag RDT compliance with these criteria before the device can be listed as eligible for reimbursement on a dedicated webpage of Bundesinstitut für Arzneimittel und Medizinprodukte (BfArM), another governmental authority [7].

We selected devices from the BfArM list for the comparative evaluation performed by PEI/RKI. The aim of this comparative evaluation was to both determine the state of the art sensitivity of proficient devices and identify devices not reaching the minimum sensitivity level. The concept of 'state of the art' is also mentioned in the IVD Regulation (EU) 2017/746 (IVDR) [5], describing a defined level of quality features achieved by the majority of assays at a certain time point after their comparative evaluation using an uniform sample set (head-to-head comparison); with continuous improvement of devices, the state of the art level increases over time and would therefore need to be re-assessed at certain intervals. Subsequently, devices with sensitivity below state of the art are removed from the BfArM list while all devices with successful evaluation outcome are published on the PEI webpage [8]. We evaluated 122 SARS-CoV-2 Ag-RDT in direct comparison using a common panel of SARS-CoV-2 specimens.

Methods

Results

Discussion

There is convincing evidence that infectivity of SARS-CoV-2 correlates directly with high viral loads in respiratory specimens of acutely infected persons [11-13]. It has therefore been suggested in many countries to use antigen tests to detect potential infectivity and help control the spread of infection rather than for the purpose of clinical diagnosis. Thus Ag RDT have become a key part of testing strategies since the autumn of 2020. Hundreds of different Ag RDT, most often from East-Asian manufacturers, are available in Europe. Nearly all tests state in their IFU sensitivity values of > 90% for PCR-confirmed specimens. Such statements, being in strong contrast to the results of our study and to other independent evaluations, may be explained by preselection of specimens with strong PCR positivity and/or studies including only few specimens.

Lack of independent evaluation combined with unjustified statements of quality features led the German Ministry of Health to request in autumn 2020 a comparative evaluation of the sensitivity of test kits offered in Germany. At the time we performed our study (autumn 2020 to spring 2021), there were no EU-wide requirements for quality features of COVID-19 IVD such as a defined minimum sensitivity or minimum specificity, and manufacturers may themselves certify their devices as compliant with basic requirements of the IVDD. Therefore, it was mainly left to individual countries or international organisations to define minimum requirements for the acceptance of tests. However, in autumn 2021, a Guidance on performance evaluation of SARS-CoV-2 in vitro diagnostic medical devices was endorsed by the EU Medical Device Coordination Group which will be the basis for future Common Specifications of the IVD Regulation (EU) 2017/746 [14].

In Germany, the Ministry of Health decided to link the reimbursement of SARS-CoV-2 Ag RDT to quality requirements that needed to be fulfilled by acceptable devices. Minimum requirements were jointly formulated by the PEI and RKI and state for SARS-CoV-2 Ag RDT a minimum sensitivity of 80% for PCR-positive specimens obtained within the first 7 days after symptom onset; the minimum specificity was defined as > 97%, and for both requirements, a study population of at least 100 persons is required [6]. Analogous requirements for SARS-CoV-2 Ag RDT have been proposed by the World Health Organization (WHO) for the emergency use listing [15], the United States Food and Drug Administration [16], the European Centre for Disease Prevention and Control [17], the Swiss Authority Bundesamt für Gesundheit [18] and the non-governmental Foundation for Innovative New Diagnostics [19]. Furthermore, RDT reimbursable in Germany had to pass our comparative evaluation, the first part of which is summarised in this manuscript; the evaluation has been continued for further RDT with an equivalent panel version 3 (data not included in this manuscript).

The definition of 75% minimal detection rate (analytical sensitivity) for panel members with Cq ≤ 25 in our comparative evaluation was based on different reasons. Firstly, infectious virus determined by cell culture propagation was reported for specimens with virus concentrations corresponding to an RNA level of around 10⁶ copies/mL and higher [11-13] (Supplement). Secondly, early in the evaluation, this limit proved to differentiate between RDT with different levels of analytical sensitivity, widely in accordance with diagnostic sensitivity determined in independent SARS-CoV-2 RDT evaluations using clinical specimens [19,20]. Thirdly, there is intrinsic variation between different nucleic acid amplification tests with regard to reported Cq values because of assay-specific nucleic acid extraction/elution volumes combined with assay-specific amplification input volume and amplification efficacy. This fact explains the urgent need for standardisation in this field using a common reference preparation, e.g. the WHO International Standard (IS), in combination with common unitage reporting, e.g. international units associated with the WHO IS [21]. Finally, Cq values of our panel members are not directly comparable to those of clinical specimens in other studies: we quantified the panel members by pipetting an aliquot into the amplification reaction while viral RNA in clinical specimens is measured after its elution from swabs, with probable swab-dependant retention of viral compounds, as described in Puyskens et al [9].

We recognise as potential limitation that clinical specimens are defined according time point of symptom onset and may not necessarily reflect the same viral load pattern as in our panel. We followed routine use of the tests as far as possible, including pre-analytical steps such as antigen absorption using the test-specific swabs, and subsequent elution into the test-specific buffer. Although this procedure does not follow the IFU exactly, we estimate that it is very close to the routine steps prescribed in the IFU of each test for processing clinical specimens. The vast majority (79%) of Ag RDT included in our study showed sufficient sensitivity according to our criteria. Nevertheless, the results showed a wide range of varying sensitivity. There were few tests with high and many tests with sufficient sensitivity, but also quite a few tests (21%) that did not meet the minimum criterion. Our study shows that the majority of SARS-CoV-2 Ag RDT correctly identify high viral loads of Cq ≤ 25 (> 10⁶ virus RNA copies/mL) in samples from the respiratory tract with a sensitivity of > 75%, supporting their use in the early symptomatic phase. However, although sensitivity declined with Cq > 25, there were few SARS-CoV-2 Ag RDT (4/122; 3.3%) with highest sensitivity: 97.5% for Cq < 30 or up to 86% for the complete Cq range (Cq 17–36).

There are scientific publications of further independent head-to-head evaluations for SARS-CoV-2 Ag RDT which, at the time of writing this manuscript, were limited to the comparison of only few tests [22-28]. Respective conclusions based on clinical specimens are widely consistent with our results, and the sensitivity ranking of different tests was often in line with our evaluation panel. For a valid comparison between different RDT, it is essential to follow the instructions for use, including the use of the swabs provided with the specific RDT, potentially impacting the release of virus compounds into the elution buffer (see also [9]). This precondition is not always fulfilled by studies comparing different RDT.

Since most of the SARS-CoV-2 Ag RDT offered in Europe are provided without a read-out device, visual interpretation of test results is indispensable. We would like to emphasise that few discrepant tests results obtained by two experienced laboratory technicians were reported. These equivocal results were ultimately interpreted as reactive, in favour of the tests under investigation. However, visual read-out and subjective interpretation of faint test lines, potentially caused by borderline concentration of the analyte, presents a challenge for less experienced users, e.g. lay persons using Ag RDT as self-tests.

A limitation of this study is its spot check nature since it cannot address variations between different batches of the same product, or variations between different test locations (see also [9]).

Conclusion

By using the same panel for a large number of different SARS-CoV-2 Ag RDT, we were able to evaluate the comparative performance of the different tests under the same conditions. The evaluation panel proved to be of appropriate design for sensitivity differentiation of SARS-CoV-2 Ag RDT, distinguishing better performing from less suitable tests. The continuation of the comparative evaluation is needed to cope with the rapidly growing market of SARS-CoV-2 Ag RDT. Since the panel is now close to exhaustion, we will continue the evaluation with a new set of samples with similar features, calibrated against the current panel. Although the study has not been performed with individual clinical samples, the respective limitation may be small because of the concept to use pooled specimens from clinical samples; we are confident that the results reflect well pre-analytical and analytical features of the RDT.

Supplementary Data

Notes

Conflict of interest: None declared.

References