Appraising and evaluating the search results
In our previous blog, we focused on performing a successful database search and doing the first screening of search results through titles and abstracts.
In this final, third part of our three-part series, we would like to show you how to do the second level of screening, i.e., full-text analysis, and capture relevant data for the clinical evaluation report.
If you are interested to join our newsletter email list and receive our template for the Clinical Evaluation Report (CER), please write us directly.
Step One: Find and gather the full text of selected documents
In our previous post, we talked about the first level of screening the results. On that stage we excluded publications that are obviously not relevant on the basis of the contents of the abstract or title. We left the ones that are, at first glance, important for our Literature Review.
In this step, your aim will be to find the full texts of all the references you’ve selected.
According to MEDDEV 2.7/1 Rev. 4 and MDCG 2020 13, the full-text papers and documents should be obtained for the appraisal stage.
Step Two: Evaluating the Relevance
Read and critically appraise the full text of each article you selected. It is crucial to determine if the identified data is relevant for the medical device under evaluation. According to MEDDEV 2.7/1 revision 4, when evaluating the relevance of the full-text articles, it is essential to consider whether the data are intended to:
- directly demonstrate adequate clinical performance and clinical safety of the device (pivotal data), or
- serve as an indirect supportive role.
The pivotal data must have the quality to demonstrate the clinical performance and the safety of the device under evaluation. Therefore, it must be generated with the device in case or with an equivalent device used for its intended purpose.
For more information on this topic, see the MEDDEV 2.7/1 Appendix A6, Appraisal of clinical data – examples of studies that lack scientific validity for demonstration of adequate clinical performance and/or clinical safety and MEDDEV 2.7/1 Appendix A1, Demonstration of equivalence.
Data that are not pivotal are generally appraised for their contribution for identifying and defining:
– the current knowledge/ state of the art in the corresponding medical field, to define acceptability criteria for the evaluation of the benefit/risk profile and of specific side-effects of the device under evaluation;
– hazards; individual case reports may be used for identification of new and previously unknown hazards that are associated with the device;
– the validity of criteria used for the demonstration of equivalence (if equivalence is claimed) etc.
Questions that help determine whether data are relevant are listed in Section 9.3.2.c of MEDDEV 2.7.1 rev. 4, we provide here in the summarized form:
- To what extent are the generated data representative of the device under evaluation?
- What aspects are covered?
- Are the data relevant to the intended purpose or claims of the device?
- Suppose the data are relevant to specific aspects of the intended purpose or claims. Are they applicable to particular device model, user group, medical indication, age, gender, the severity of condition, or range of time?
If the article is relevant
If you decide that an article is relevant, the next step is to weigh its clinical contribution. Unfortunately, there is no single method for weighting clinical data. You should choose a rating system you believe it is the most adequate for the device under evaluation (for example, OCEBM, SORT, GRADE, or other rating scales.)
Scientific validity of the clinical data set: Level of Evidence of the publication
|Level of Evidence||Type of publications|
|MA / SR||Evidence obtained from meta-analysis or systematic review|
|I||Ia||Evidence obtained from properly designed randomized and controlled clinical trial|
|Ib||Evidence obtained from pseudo randomized controlled trials, randomized controlled trials with limitation|
|II||IIa||Evidence obtained from not randomized prospective studies comparative with current control|
|IIb||Evidence obtained from not randomized prospective studies comparative with historical control|
|IIc||Evidence obtained from not randomized prospective studies not comparative|
|III||IIIa||Retrospective trial : consecutive patients extracted from prospective database or registry|
|IIIb||Retrospective trial, not consecutive patients|
|IV||Evidence obtained from case report|
|V||Data presented from poster, abstract or conference|
If the article is not relevant
You are expected to provide rationale for each article excluded from the analysis. The reasons might vary:
- Different device, Different indication, Different population
- Inappropriate number of patients, duration, no statistical analysis,
- Non-comparative study, inappropriate clinical outcome
- Preclinical study not on the device under evaluation
- Socio-economic assessment, cost-effectiveness study
- Comments on an article, poster, abstract, video not on the device under evaluation
- Non-peer-reviewed opinion/journal not on the device under evaluation
- Single Case study report without any relevant safety information
- Out of the scope
- Foreign language not generally understood
- Duplicate of article (same author, same study…)
You can add the table with the codes for the exclusion rationale:
|NE||Non-Equivalence||Different device, Different indication, Different population|
|M||Methodology||Inappropriate number of patients, duration, no statistical analysis,
Non-comparative study, inappropriate clinical outcome
|T||Type of publication||Preclinical study not on the device under evaluation|
|Socio-economic assessment, cost-effectiveness study|
|Comments on an article, poster, abstract, video not on the device under evaluation|
|Non-peer-reviewed opinion/journal not on the device under evaluation|
|Single Case study report without any relevant safety information|
|Review, guidance or guideline published from more than 2 year|
|OS||Out of scope||Out of the scope|
|L||Language||Foreign language not generally understood|
|D||Duplicate||Duplicate of article (same author, same study…)|
Important! The sum of included and excluded articles should sum up to the number of articles you’ve got by the literature search.
If the available data is limited
According to MEDDEV 2.7/1 revision 4, in exceptional situations, when an evaluation is based on limited data, you can describe and justify that in the clinical evaluation report. Please, see additional information and specific considerations in Appendix A8 (Devices for unmet medical needs – aspects to consider).
Step Three: Write the Literature Search Review Report
Finally, you can write the Literature Search Review Report which contains the results of your search that address safety, performance, benefits to the patient, side-effects, adverse reactions, state of the art of the device in question.
A CER literature review report should include Introduction (Purpose), Objectives, Method, Search results, Analysis, and Conclusion.
For Introduction, Objectives and Methods you should use the info pre-defined in your literature review protocol. Provide a summary of the searches that were performed, the number of results, and the number of articles excluded. The analysis of the reviewed literature should contain a comprehensive summary and analysis of the information identified during the literature review process, both favorable and unfavorable, and it should address each of the CER literature review objectives. In Conclusion, you should provide a summary to indicate how the objectives of the literature review have been met.
At the end, you should add Appendices to your Report: Bibliography, Detailed data analysis tables and spreadsheets, Literature search results list and detailed criteria for the article exclusions.
BioReg Services specializes in systematic literature searches and reviews in compliance with the MDR Regulation (EU) 2017/745. Our team can design a comprehensive literature search protocol with complete documentation of the literature screening and data extraction process. We can obtain clinical evidence for the literature review for your new medical device, or we can update the literature review of a CE-marked or legacy device to create an MDR compliant document.
Don’t hesitate to contact us.