Evaluating Clinical Studies and Evidence-Based Care

Fiona Collins BDS, MBA, MA, FPFA

TO GET C.E. CREDITS FOR THIS ARTICLE
PLEASE LOGIN OR REGISTER

Evaluating Clinical Studies and Evidence-Based Care

Well-designed randomized controlled trials (RCT) with a sufficiently large sample size and a low risk of bias provide the most robust data for systematic reviews that contribute to evidence-based dentistry focuses on the best available evidence and patient-centered care.1American Dental Association. Policy on Evidence-Based Dentistry. Available from: http://www.ada.org/en/about-the-ada/ada-positions-policies-and-statements/policy-on-evidence-based-dentistry. ,2Brignardello-Petersen R, Carrasco-Labra A, Glick M, Guyatt GH, Azarpazhooh A. A practical approach to evidence-based dentistry. Understanding and applying the principles of EBD. J Am Dent Assoc 2014;145(11):1105-7. http://jada.ada.org However, a systematic review may contain less robust RCT, a combination of RCT and non-randomized clinical trials, and other types of studies such as prospective or retrospective cohort studies, which in descending order provide lower levels of evidence. Case-controlled studies and case series may also contribute and absenting studies an expert opinion may be noted in developing recommendations. Laboratory (in vitro) data may also be considered and in some situations may be the only viable option. Similar to computer-generated data, the validity of the output ultimately relies on the input, including the individual studies and methodology. There are a number of questions and considerations to address when reading a paper on a clinical study or a systematic review and considering the outcomes.

Evaluating Individual Clinical Studies

The CONSORT 2010 checklist contains 25 items to address when reporting a parallel group RCT, as well as a flow diagram.3CONSORT transparent reporting of trials. CONSORT 2010. Available at: http://www.consort-statement.org/consort-2010. The checklist addresses the trial’s design, analysis and interpretation of the results. It can also be used to avoid potential pitfalls in study design and execution. A completed CONSORT 2010 checklist can be submitted with the article, and studies that adequately address CONSORT criteria and provide a checklist give clarity. However, many studies do not have a CONSORT checklist available. Some of the considerations for these are: Was the study randomized or non-randomized? Was the sample size adequate? Were the results statistically significant? Did the study compare ‘apples to apples’? Were there sources of bias and were confounders considered? Bias can consist of one or more types of bias, weakens clinical data, and may influence the results and conclusions.4Pannucci CJ, Wilkins EG. Identifying and avoiding bias in research. Plast Reconstr Surg 2010;126(2):619-25. How was the data compiled and compared and what is its validity? Are the results clinically relevant? (Table 1) These are important questions in the context of the evidence base, everyday practice and patient care.

Table 1. Questions to consider for a clinical study
Was the study randomized or non-randomized?
Was the sample size adequate?
Did the study compare ‘apples to apples’?
Were there sources of bias?
Were confounders considered?
How was the data compiled and compared?
Are the results statistically significant?
Are the results clinically relevant?
Bias can consist of one or more types of bias, weakens clinical data, and may influence the results and conclusions

Sample Size
Small sample sizes preclude statistical analysis.5Faber J, Fonseca LM. How sample size influences research outcomes. Dental Press J Orthod 2014;19(4):27-9. That means that it would not be possible to conclusively demonstrate an effect in epidemiological or clinical studies, e.g., statistical significance for efficacy of an anti-gingivitis chemotherapeutic.6Röhrig B, du Prel JB, Wachtlin D, Kwiecien R, Blettner M. Sample size calculation in clinical trials: part 13 of a series on evaluation of scientific publications. Dtsch Arztebl Int 2010;107(31-32):552-6. Nonetheless, small samples can be helpful in providing preliminary data. Conversely, if an oversized sample is used, this can magnify differences beyond what is realistic, and may place more individuals at risk of potential adverse events in drug trials.5Faber J, Fonseca LM. How sample size influences research outcomes. Dental Press J Orthod 2014;19(4):27-9. ,6Röhrig B, du Prel JB, Wachtlin D, Kwiecien R, Blettner M. Sample size calculation in clinical trials: part 13 of a series on evaluation of scientific publications. Dtsch Arztebl Int 2010;107(31-32):552-6. In addition, differing sample sizes across groups is a source of bias.

Blinding
Clinical trials that are double-blinded mean that neither patients nor the researchers evaluating outcomes know which intervention or treatment a given patient had received. One example of double-blinding is concealed allocation of a product, such as toothpaste delivered in a plain tube/wrap and coded for notation in the records, with neither the patient nor the evaluating researcher knowing which product the subject received. In contrast, single-blinding or non-blinding can result in bias. If an intervention is known and anticipated to be the ‘latest and greatest’ then results, such as subjective relief from hypersensitivity, may be perceived by a patient to be superior to the alternative. Similarly, a researcher could inadvertently evaluate results differently.

One example of double-blinding is concealed allocation of a product, such as toothpaste delivered in a plain tube/wrap and coded for notation in the records.
Are you comparing apples to apples?

Does the study compare apples to apples?
A study may compare a test drug/device to a placebo, a positive control, and/or a negative control. One question to look at is whether a study with positive control(s), in particular, is comparing apples to apples. In a study with a test group and a positive control group, criteria such as sample sizes, demographics, and disease levels should be similar for both groups, and subjects randomly allocated. For topical fluoride agents, for example, caries reductions in studies are greater in a group with lower access to fluoridated water and a higher level of dental caries.7Centers for Disease Control and Prevention. Recommendations for Using Fluoride to Prevent and Control Dental Caries in the United States. MMWR 2001/50(RR14);1-42. Therefore, one question to ask is whether disease levels were dissimilar, and if so were individuals with more severe disease or risk allocated disproportionately to one of the groups? That would influence the reduction in dental caries and could make one product appear to be more efficacious than the other whether or not that was true. This is ‘transfer bias.’ To account for varying levels of disease among subjects, and avoid transfer bias, did the study stratify subjects in groups, based on disease level, for each intervention?

Table 2. Some potential sources of bias
Lack of blinding/single blinding
Disproportionate allocation to groups, e.g., by demographics, disease level
Differing drop-out rates
Incomplete data
Selective reporting
Outcomes assessment

Other sources of bias and confounders
Other sources of bias include different drop-out rates, incomplete data, inaccurate recall in retrospective studies, selective reporting and assessment of results. (Table 2) Confounders are factors, other than the intended intervention, that could influence outcomes and conclusions. When confounders are known, an adjusted outcome can be obtained by performing statistical analyses, such as regression analysis, to take confounders into account. Consideration of unknown confounders is one of the reasons for using large sample sizes. Lack of consideration of confounders weakens studies.8Romo-Romo A, Aguilar-Salinas CA, Brito-Córdova GX, Gómez Díaz RA, Vilchis Valentín D, Almeda-Valdes P. Effects of the non-nutritive sweeteners on glucose metabolism and appetite regulating hormones: Systematic review of observational prospective studies and clinical trials. PLoS ONE 2016;11(8):e0161264.

Confounders are factors other than the intervention itself that could influence outcomes and conclusions.

Clinical applicability and statistical significance
Can the same outcome be expected in your patient population as in the study, i.e., are they similar and is it applicable? Are there different factors such as disease level or dietary habits that could influence the results? Another consideration is whether or not the outcome for a given study is clinically relevant/impactful even if the RCT shows statistically significance.9Prato G, Pagliaro U, Buti J, Rotundo R, Newman MG. Evaluation of the literature: evidence assessment tools for clinicians. J Evid Based Dent Pract 2013;13(4):130-41. doi: 10.1016/j.jebdp.2013.08.001. For example, a minimal reduction in a measurement, e.g., pocket probing depth, may not be clinically relevant or impactful. In addition, a measurement difference may not be reliably measurable.

Evaluating Systematic Reviews

The quality and usefulness of systematic reviews relies upon the design and quality of the included studies. In one systematic review on the efficacy of splints for temporomandibular joint dysfunction, the overall risk of bias was mostly high.10Zhang C, Wu J-Y, Deng D-L, He B-Y, Tao Y, Niu Y-M, Deng M-H. Efficacy of splint therapy for the management of temporomandibular disorders: a meta-analysis. Oncotarget 2016;7(51):84043-53. For all 13 studies, the risk of bias was high for blinding of participants and outcome assessment, and low for selective reporting and incomplete outcome data. For other categories, the risk of bias was ‘unclear, low or high.’ In addition, different types of splints were evaluated across studies and there were issues with follow-up. In other systematic reviews, study limitations impacted evaluation of the risks and benefits of artificial sweeteners.8Romo-Romo A, Aguilar-Salinas CA, Brito-Córdova GX, Gómez Díaz RA, Vilchis Valentín D, Almeda-Valdes P. Effects of the non-nutritive sweeteners on glucose metabolism and appetite regulating hormones: Systematic review of observational prospective studies and clinical trials. PLoS ONE 2016;11(8):e0161264.,11Johnson RK, Lichtenstein AH, Anderson CAM, Carson JA, Després JP, Hu FB, et al. Low-calorie sweetened beverages and cardiometabolic health: a science advisory from the American Heart Association. Circulation 2018;138:e126-40.,12Pereira MA. Sugar-sweetened and artificially-sweetened beverages in relation to obesity risk 1–3. Adv Nutr 2014;5:797-808.,13Lohner SI, Küllenberg de Gaudry D, Sommer H, Meerpohl JJ. Association between intake of non-sugar sweeteners and health outcomes: systematic review and meta-analyses of randomised and non-randomised controlled trials and observational studies. Br Med J 2019;364:k4718. doi: https://doi.org/10.1136/bmj. Outcomes on disease risk were conflicting, and studies were also heterogeneous for methodology, sample size, duration, scope, and identification or consideration of bias and confounding factors. In reviews of studies on or involving diagnostic testing, the use of different indices across studies with varying levels of sensitivity (false negatives) and specificity (false positives) is an example of a confounder, again limiting the usefulness of systematic review outcomes.14Lechien JR, Chiesa-Estomba CM, Calvo Henriquez C, Mouawad F, Ristagno C, Barillari MR, et al. Laryngopharyngeal reflux, gastroesophageal reflux and dental disorders: A systematic review. PLoS ONE 2020;15(8):e0237581. ,15Senneby A, Mejàre I, Sahlin NE, Svensäter G, Rohlin M. Diagnostic accuracy of different caries risk assessment methods. A systematic review. J Dent 2015;43(12):1385-93.

In diagnostic testing, the use of different indices across studies with varying levels of sensitivity and specificity limits the usefulness of a systematic review.

Tools for designing and evaluating systematic reviews
Available tools can help with the design and evaluation of systematic reviews. The ‘Preferred reporting items for systematic reviews and meta-analyses’ (PRISMA statement) can be used in their design.16Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Int J Surg 2010;8:336-41. Another online tool, ‘A MeaSurement Tool to Assess systematic Reviews’ (AMSTAR) contains criteria for use while designing or evaluating systematic reviews containing RCT.17Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, et al. Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol 2007;7:10. The AMSTAR 2 criteria were developed for systematic reviews that contain both randomized and non-randomized clinical trials.18AMSTAR 2. Available at: https://amstar.ca/docs/AMSTAR-2.pdf. The assessment tool contains 11 criteria. (Table 3) In addition, for systematic reviews of diagnostic studies, the QUADAS -2 tool enables determination of bias related to patient selection, the index tests used, the references standard, flow and timing of the study and evaluation of the outcomes.19Whiting PF, Rutjes AW, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, et al. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med 2011;155:529-36. doi: 10.7326/0003-4819-155-8-201110180-00009. The Grading of Recommendations Assessment, Development, and Evaluation (GRADE) criteria are used to assess the results and quality of the evidence for systematic reviews, which could range from low to high depending on the included studies.20Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al. GRADE: An emerging consensus on rating quality of evidence and strength of recommendations. Br Med J 2008;336:924-6.

Table 3. AMSTAR Criteria
1. Was an a priori design provided?
2. Were study selection and data extraction duplicated?
3. Was a comprehensive literature search performed?
4. Was the status of publication (i.e., gray literature) used as an inclusion criterion?
5. Was a list of studies (included & excluded) provided?
6. Were the characteristics of the included studies provided?
7. Was the scientific quality of the included studies assessed and documented?
8. Was the scientific quality of the included studies used appropriately in formulating conclusions?
9. Were the methods used to combine the findings of studies appropriate?
10. Was the likelihood of publication bias assessed?
11. Was the conflict of interest included?

Critical appraisals use these tools. In one such critical appraisal, the AMSTAR criteria were considered, 97% of studies were found to be at high risk of bias and there was considerable inter-study variation.21Barrow SL. Critical Appraisal of the Cochrane Systematic Review on Water Fluoridation for the Prevention of Dental Caries. Int J Evid Based Pract Dent Hygienist 2016;2:3-36. doi: 10.11607/ebh.47. This limited the ability to draw definitive conclusions.

Clinical relevance

Guidelines and recommendations for clinical practice are derived from the evidence and note its strength or in the case of an expert opinion not that.1American Dental Association. Policy on Evidence-Based Dentistry. Available from: http://www.ada.org/en/about-the-ada/ada-positions-policies-and-statements/policy-on-evidence-based-dentistry. ,22ADA Evidence-Based Clinical Recommendations. www.ada.org/prof/resources/ebd/clinical.asp. They are extremely useful, and save clinicians having to undertake time-consuming literature review to find outcomes for decision-making. However, the clinic setting differs from a research setting where more factors can be controlled for. Decision-making is more complicated, and findings of research studies may or may not be translatable to the clinic.23Kay E. How do we decide? Knowledge? Experience? Research? Evid Based Dent 2020;21:4. https://doi.org/10.1038/s41432-020-0087-5. ,24Song M, Liu K, Abromitis R, Schleyer TL. Reusing electronic patient data for dental clinical research: a review of current status. J Dent 2013;41:1148-63. It has been recommended that the results of clinical research be combined with experiential knowledge from the clinical setting for decision-making purposes and to benefit patients.23Kay E. How do we decide? Knowledge? Experience? Research? Evid Based Dent 2020;21:4. https://doi.org/10.1038/s41432-020-0087-5.

It has been recommended that the results of clinical research be combined with experiential knowledge from the clinical setting.

Practice-based networks such as the National Dental Practice-Based Research Network (NDPBRN) conduct research on outcomes and best practices in real-life clinical settings (dental offices).25The National Dental Practice-Based Research Network. Available at: https://www.nationaldentalpbrn.org/. One example is research on influencers for the clinical acceptability of single-unit crowns.26Lawson NC, Litaker MS, Sowell E, Gordan VV, Mungia R, Ronzo KR, et al. Clinical acceptance of single-unit crowns and its association with impression and tissue displacement techniques: Findings from the National Dental Practice-Based Research Network. J Prosthet Dent 2020;123(5):701-9. doi: 10.1016/j.prosdent.2019.05.016. The evaluation included 3,730 single-unit crowns provided by dentists in the NDPBRN. Among the findings, digital scans resulted in 99.5% of crowns being clinically acceptable compared to 95.8% for dual-arch trays, and a dual-cord technique for tissue retraction reduced impression retakes. In another study, 107 dentists in the NDPBRN assessed suspicious occlusal lesions based on color, luster, lesion roughness, and caries risk level for individual patents using case presentations.27Makhija SK, Robinson ME, Bader JD, Shugars DA, Litaker MS, Im HR, et al. Dentists' decision strategies for suspicious occlusal caries lesions in a National Dental PBRN study. J Dent 2018;69:83-7. doi: 10.1016/j.jdent.2017.11.004. Large variations were found in how these criteria were applied by dentists in deciding whether or not caries lesions had reached into dentin. It was concluded that this information could be applied in interventions to optimize decision-making on treatment. In addition, electronic health records (electronic medical and/or dental records) offer an opportunity to mine the results of interventions based on the patient data entered into the records and can be used to evaluate practice-based treatment outcomes, quality of care and evidence-based treatment. Large sample sizes are readily available by using EHR, and the findings are based on the real-life clinical setting.24Song M, Liu K, Abromitis R, Schleyer TL. Reusing electronic patient data for dental clinical research: a review of current status. J Dent 2013;41:1148-63.

Conclusions

Several evaluation tools are available to determine the strength and validity of studies and systematic reviews from which recommendations can be developed together. When reviewing individual studies and systematic reviews, there are questions to consider such as their overall design, potential biases and, importantly, clinical relevance and applicability. More standardized methods across studies would increase the ability to determine outcomes and conclusions. Given that findings of research studies may or may not be translatable to the clinic setting, it has also been recommended that the results could be combined with experiential knowledge. Resources for outcomes in clinic settings, such as the NDPBRN and mining of electronic health records, offer opportunities to obtain such data for consideration. Ultimately, outcomes must be clinically applicable and beneficial.

References

  • 1.American Dental Association. Policy on Evidence-Based Dentistry. Available from: http://www.ada.org/en/about-the-ada/ada-positions-policies-and-statements/policy-on-evidence-based-dentistry.
  • 2.Brignardello-Petersen R, Carrasco-Labra A, Glick M, Guyatt GH, Azarpazhooh A. A practical approach to evidence-based dentistry. Understanding and applying the principles of EBD. J Am Dent Assoc 2014;145(11):1105-7. http://jada.ada.org
  • 3.CONSORT transparent reporting of trials. CONSORT 2010. Available at: http://www.consort-statement.org/consort-2010.
  • 4.Pannucci CJ, Wilkins EG. Identifying and avoiding bias in research. Plast Reconstr Surg 2010;126(2):619-25.
  • 5.Faber J, Fonseca LM. How sample size influences research outcomes. Dental Press J Orthod 2014;19(4):27-9.
  • 6.Röhrig B, du Prel JB, Wachtlin D, Kwiecien R, Blettner M. Sample size calculation in clinical trials: part 13 of a series on evaluation of scientific publications. Dtsch Arztebl Int 2010;107(31-32):552-6.
  • 7.Centers for Disease Control and Prevention. Recommendations for Using Fluoride to Prevent and Control Dental Caries in the United States. MMWR 2001/50(RR14);1-42.
  • 8.Romo-Romo A, Aguilar-Salinas CA, Brito-Córdova GX, Gómez Díaz RA, Vilchis Valentín D, Almeda-Valdes P. Effects of the non-nutritive sweeteners on glucose metabolism and appetite regulating hormones: Systematic review of observational prospective studies and clinical trials. PLoS ONE 2016;11(8):e0161264.
  • 9.Prato G, Pagliaro U, Buti J, Rotundo R, Newman MG. Evaluation of the literature: evidence assessment tools for clinicians. J Evid Based Dent Pract 2013;13(4):130-41. doi: 10.1016/j.jebdp.2013.08.001.
  • 10.Zhang C, Wu J-Y, Deng D-L, He B-Y, Tao Y, Niu Y-M, Deng M-H. Efficacy of splint therapy for the management of temporomandibular disorders: a meta-analysis. Oncotarget 2016;7(51):84043-53.
  • 11.Johnson RK, Lichtenstein AH, Anderson CAM, Carson JA, Després JP, Hu FB, et al. Low-calorie sweetened beverages and cardiometabolic health: a science advisory from the American Heart Association. Circulation 2018;138:e126-40.
  • 12.Pereira MA. Sugar-sweetened and artificially-sweetened beverages in relation to obesity risk 1–3. Adv Nutr 2014;5:797-808.
  • 13.Lohner SI, Küllenberg de Gaudry D, Sommer H, Meerpohl JJ. Association between intake of non-sugar sweeteners and health outcomes: systematic review and meta-analyses of randomised and non-randomised controlled trials and observational studies. Br Med J 2019;364:k4718. doi: https://doi.org/10.1136/bmj.
  • 14.Lechien JR, Chiesa-Estomba CM, Calvo Henriquez C, Mouawad F, Ristagno C, Barillari MR, et al. Laryngopharyngeal reflux, gastroesophageal reflux and dental disorders: A systematic review. PLoS ONE 2020;15(8):e0237581.
  • 15.Senneby A, Mejàre I, Sahlin NE, Svensäter G, Rohlin M. Diagnostic accuracy of different caries risk assessment methods. A systematic review. J Dent 2015;43(12):1385-93.
  • 16.Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Int J Surg 2010;8:336-41.
  • 17.Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, et al. Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol 2007;7:10.
  • 18.AMSTAR 2. Available at: https://amstar.ca/docs/AMSTAR-2.pdf.
  • 19.Whiting PF, Rutjes AW, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, et al. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med 2011;155:529-36. doi: 10.7326/0003-4819-155-8-201110180-00009.
  • 20.Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al. GRADE: An emerging consensus on rating quality of evidence and strength of recommendations. Br Med J 2008;336:924-6.
  • 21.Barrow SL. Critical Appraisal of the Cochrane Systematic Review on Water Fluoridation for the Prevention of Dental Caries. Int J Evid Based Pract Dent Hygienist 2016;2:3-36. doi: 10.11607/ebh.47.
  • 22.ADA Evidence-Based Clinical Recommendations. www.ada.org/prof/resources/ebd/clinical.asp.
  • 23.Kay E. How do we decide? Knowledge? Experience? Research? Evid Based Dent 2020;21:4. https://doi.org/10.1038/s41432-020-0087-5.
  • 24.Song M, Liu K, Abromitis R, Schleyer TL. Reusing electronic patient data for dental clinical research: a review of current status. J Dent 2013;41:1148-63.
  • 25.The National Dental Practice-Based Research Network. Available at: https://www.nationaldentalpbrn.org/.
  • 26.Lawson NC, Litaker MS, Sowell E, Gordan VV, Mungia R, Ronzo KR, et al. Clinical acceptance of single-unit crowns and its association with impression and tissue displacement techniques: Findings from the National Dental Practice-Based Research Network. J Prosthet Dent 2020;123(5):701-9. doi: 10.1016/j.prosdent.2019.05.016.
  • 27.Makhija SK, Robinson ME, Bader JD, Shugars DA, Litaker MS, Im HR, et al. Dentists' decision strategies for suspicious occlusal caries lesions in a National Dental PBRN study. J Dent 2018;69:83-7. doi: 10.1016/j.jdent.2017.11.004.