More than 10 years ago, Bess (1995) highlighted the need for audiologists to engage in Evidence-Based Practice (EBP), pointing out that the time had passed when few questioned the audiologist's clinical procedures and decisions. Rather, continued provision of quality hearing healthcare within an arena that emphasized reducing overall healthcare costs necessitated that audiologists become evidence-based practitioners (Bess, 1995). The evidence-based audiologist places emphasis on the integration of knowledge of the best available evidence (research), with his or her clinical judgment and the patient's preferences, for making clinical decisions (Cox, 2005).
While few audiologists are likely to argue about the importance of maintaining up-to-date knowledge of scientific evidence, doing so within busy clinical practices presents many challenges. Not only must the audiologist be able to engage in critical evaluation of the evidence, often without much formal training in research methodology, but the clinician also must be able to set time aside to access and read the literature. This can be a daunting task for any healthcare practitioner as the medical literature, including that in Audiology, is vast and growing rapidly. Indeed, the results of a recent study indicated that keeping current with the peer-reviewed literature in primary care medicine would take a physician 627.5 hours per month, or 21 hours per day seven days per week (Alper et al., 2004). Within Audiology, the number of papers published in respected refereed journals has grown since 1960 from about 200 per year to over 1700 per year, resulting in the audiologist needing to read 5 papers per day, 365 days per year to keep up with current peer-reviewed research (Thorne, 2003). Clearly, if EBP is to become the standard way by which audiologists provide clinical services, there is a critical need for efficient and effective methods of accessing relevant clinical evidence.
The systematic review has emerged as a powerful method for gathering, analyzing, and summarizing the scientific evidence on a specific clinical topic. In a systematic review, critical evaluation of individual studies is completed, various study results synthesized, and a single document written, summarizing the best current evidence for making clinical decisions. Systematic reviews may or may not include a meta-analysis, which is a quantitative synthesis of the results of individual studies.
As demonstrated in a special issue of the Journal of the American Academy of Audiology (JAAA) published in 2005 (Vol. 16, No. 7), the availability of systematic reviews is growing in the field. In the JAAA special issue, leaders in the field of amplification published seven systematic reviews on topics ranging from hearing aid fitting using clinical pre-fitting speech measures to the effectiveness of signal processing strategies in the pediatric population. In the same issue, Dr. Robyn Cox (2005) published an article on the topic of EBP, in which she pointed out that there was "a desperate need for published systematic reviews and meta-analyses of related investigations that currently exist in the literature" (p. 436). Indeed, systematic reviews and meta-analyses provide the highest level of evidence on which to make a clinical decision (Cox, 2005). As another example, a task force organized by the American Academy of Audiology utilized an EBP review approach in the establishment of the most recent guideline for the audiologic management of adults with hearing impairment (AAA, 2006; Valente, 2006). In addition to being extremely useful resources for practitioners searching for evidence, systematic reviews are becoming more common and important for guiding the development of clinically-relevant research agendas that can rapidly move the field forward.
Although systematic reviews provide a high level of evidence, it is important to realize that just as the quality of the primary studies appraised as part of the systematic review process can vary greatly, there can also be a great deal of variability in the quality of systematic reviews. The quality of a systematic review, and thus its worth, depends on the extent to which the scientific review methods utilized in its development were able to minimize the risks of error and bias. Moreover, critical appraisal is important for determining the usefulness of the review for guiding clinical decisions. Numerous guidelines are available to assist in the critical appraisal of systematic reviews* Consistent across the various review guidelines are at least six questions, focusing on:
- the review question;
- inclusion/exclusion criteria;
- the literature search;
- critical appraisal;
- data synthesis; and,
- the summary of the findings.
- The review question
A good systematic review will provide the reader with concise background information and an explanation of current practices leading to the clinical question posed. In appraising the review, the reader should ask, "Was the review question clearly and explicitly stated?" (JBI, 2000). A clear, explicit clinical question will then follow the guidelines of what is referred to as a PICO format. The question will specify the Patient or population; the Intervention (or test, or agent) of interest; the Comparison intervention (or control) if relevant; and the Outcome(s) of interest. In providing background information leading to the AAA-TF's PICO question, Chisolm et al (in press) cited data demonstrating that untreated hearing loss in adults impacted negatively the self-perception of health related quality-of-life (HRQoL). Although hearing aids provide many benefits to adults with hearing loss, there remains a lack of a clear understanding of whether or not hearing aids resulted in improvements in HRQoL. Thus, the AAA-TF developed the PICO question: In individuals 18 years old and older with adult-onset sensorineural hearing loss (P) does the use of hearing aids (I) improve HRQoL (O)? - Inclusion/Exclusion Criteria
Authors who develop a systematic review should set, clearly and explicitly, the criteria used for the selection of studies for review. Inclusion and exclusion criteria typically include specification of study designs and levels of evidence, participant characteristics, intervention characteristics and delivery, and specific outcomes. While the proposed PICO question specifies the area of general interest, the inclusion and exclusion criteria determine the focus, nature, and limits of the review. In appraising a systematic review, the reader needs to ask, "How were the studies selected?" (JBI, 2000).
Study Design and Levels of Evidence
Perhaps the most important criterion that systematic reviewers set relates to the study design and levels of evidence. In EBP, it is commonplace to refer to design hierarchies, based on the ability of particular study designs to minimize or eliminate biases in the effect studied. There are many hierarchies available for ranking clinical research, and the various schemes are not always consistent with one another. This is, in part, because different design types are better suited to answer different types of clinical questions. Furthermore, the reviewers must select the evidence hierarchy that best fits the type of clinical question under review (Robey, 2004). For example, the highest level of evidence for a question regarding a diagnosis is a prospective study, wherein patients are administered both the diagnostic test under investigation and a gold standard test, with both patients and test administrators blinded as to which test is being administered. For a treatment question, the highest level of evidence is a double-blinded, randomized controlled trial.
The AAA-TF adapted the Scottish Intercollegiate Guideline Network (SIGN) (www.sign.ac.uk) hierarchy for use in completion of the systematic review. The SIGN hierarchy consists of four basic levels, with finer gradations contained within the two highest levels. Level 1 studies include randomized controlled trials (RCTs) and systematic reviews of RCTs, with or without meta-analyses. Level 2 includes quasi-experimental, controlled studies without randomization, such as parallel group and crossover designs, and systematic reviews of these types of studies. Level 3 includes non-analytic studies, such as case reports and case studies. Finally, expert opinion comprises level 4, the lowest level of evidence. In utilizing the SIGN hierarchy, the AAA-TF elected to include studies of the highest three levels, excluding studies based on expert opinion.
Participants
The studies selected for review should include participants similar to those specified in the PICO question. For the AAA-TF systematic review, participant criteria for article selection were: adult, 18 years or older, post-lingual sensorineural hearing loss of any degree, new or experienced hearing aid wearer with normal cognitive function, and no restriction based on living condition.
Intervention
The type of intervention is the means for improving patient function. The characteristics of the intervention may encompass a broad or narrow range, dependent on the review focus, and studies are only included when the intervention of interest is part of the study methodology. The AAA-TF review was not designed to compare various technical aspects of hearing aid fittings. Thus, intervention inclusion criteria were hearing aids regardless of: style (e.g., behind-the-ear, in-the-ear, completely-in-the ear, etc.), signal processing circuitry (e.g., analog or digital), microphone technology (e.g., omnidirectional or directional), or fitting strategy (e.g., monaural vs. binaural).
Outcomes
Outcomes, or the effects on patient status following intervention, must be specified, and guidelines must be defined as to how the outcomes of interest must be quantified in the studies selected for review. The AAA-TF systematic review included studies that measured HRQoL utilizing either generic or disease-specific instruments (National Institutes for Health, 1993). Generic instruments are broad in scope and can be applied across conditions or diseases, while disease-specific instruments focus on one condition, attempting to define its effects on daily functioning and well-being. - The Literature Search
Critical to any systematic review is a carefully planned method for searching the literature. All steps in the search process should be carefully detailed in the systematic review, including databases searched, other sources investigated, search terms used, and how the terms were combined to form search strings, timeframe of the search, and the results obtained at each stage of the search process. In evaluating the quality of the literature search, readers of systematic reviews should ask two questions:
(1) Were comprehensive search methods used to locate studies?, and,
(2) Were relevant databases searched and other potentially important sources of information thoroughly explored? (JBI, 2000).
To be as comprehensive as possible the search would not be limited to English-only publications, although the reading of non-English literature may be problematic. In addition, to identify all the published literature, more than one database should be searched. A thorough search also would include non-peer reviewed literature and hand searching of conference proceedings. While these latter sources of evidence may not provide research reports that adhere to the rigorous standards required by peer-reviewed journals, there are high quality non-peer-reviewed publications. Further, non-peer-reviewed journals tend to publish articles faster and they, as well as conference proceedings, can be a good source for the most recent data in the area of clinical interest (Law, 2002). Deciding to limit the systematic review to studies published in peer-reviewed journals also could result in a bias, since there is a strong tendency for peer-reviewed journals to publish only statistically significant findings (Scargle, 2000).
To complete its search, the AAA-TF was divided into three teams: One team searched non-peer-reviewed journals (e.g., The Hearing Journal) and recent conference proceedings, while the other two teams conducted database searches. The databases included the Communication Sciences and Disorders DOME (ComDisDome), and the Cumulative Index to Nursing and Allied-Health Literature (CINAHL), amongst others. Key words used in the database searches included hearing aids, amplification, quality-of-life, and the names of commonly used generic and disease-specific HRQoL instruments. The search was limited to articles published in English and the database search occurred during July 2004.
The comprehensive nature of the search is demonstrated by careful recording of the results obtained at each stage. First, the reviewers should record the number of hits obtained with each search term and combinations of terms. Then reviewers examine article titles and discard those that do not appear relevant to the topic, recording the number eliminated. Next, reviewers examine article abstracts, making a preliminary judgment on which ones meet inclusion/exclusion criteria and dismissing studies that do not meet criteria; the number of excluded studies should again be noted. The full-texts of the remaining articles are then retrieved for further review. After reading the full-text, if an article does not meet criteria, is the reviewer should discard the article, recording the reason(s) for dismissal. The remaining studies form the basis for the systematic review. The AAA-TF search revealed 171 articles whose key words and titles suggested they would meet inclusion/exclusion criteria. Abstract review reduced the number of studies to 74 for full-text review. Of these 74, only 16 qualified for further evaluation. - Critical Appraisal
Each study included in the systematic review is evaluated to determine the extent to which the methods used were able to minimize the potential for biases influencing the results. The reader of the systematic review must ask "was the quality and validity of studies assessed appropriately?" (JBI, 2000). Factors commonly considered important to the quality and validity of research findings include:
(1) use of a power analysis to ensure appropriate sample size;
(2) use of experimental and control groups;
(3) adequate specification of subject inclusion and exclusion criteria;
(4) use of appropriate statistical analyses;
(5) accountability for mortality threats, namely drop outs;
(6) allocation concealment; and
(7) effect sizes. In addition to these factors, the AAA-TF considered the presence of a well-described hearing aid fitting protocol and the verification of hearing aid fit using real-ear measures, as important quality indicators. - Data synthesis
There are two ways to synthesize individual study data. The first, which is included in all systematic reviews, is a descriptive analysis of the reported results for individual studies. All of the systematic reviews in the aforementioned special JAAA issue provided descriptive analyses. The second way to synthesize the data from individual studies is through meta-analysis, the statistical analysis of the results from two or more individual studies. Researchers should only conduct meta-analyses for studies with similar characteristics, whose methodological quality is adequate, and which utilize similar outcome measures. Synthesis from individual studies occurs by converting individual results to a common metric and then applying standard statistical procedures to yield a single point estimate, with confidence intervals, which specifies the magnitude of the effect size. In evaluating a systematic review, the reader needs to determine "how the results of the studies were combined"; and, if the "findings were combined appropriately" (JBI, 2000).
The AAA-TF systematic review included four meta-analyses. Separate analyses were conducted for studies comparing results between groups of subjects vs. those comparing results within groups of subjects; and, studies using generic instruments vs. studies using disease-specific instruments. Thus, there were four meta-analyses:
(1) between-group, generic;
(2) between-group, disease-specific;
(3) within-group, generic; and,
(4) within-group, disease-specific.
Effect size (ES) estimates, reported as Cohen's d, were: 0.28 (CI.05,.95 = 0.09, 0.48) for the between-group, generic data; 2.07 (CI.05,.95 = 0.51, 3.63) for between-subject, disease-specific data; 0.02 (CI.05,.95 = -0.04, 0.07) for the within-group, generic data; and, 1.01(CI.05,.95 = 0.76, 1.26) for the within-group, disease-specific data. As discussed by Cox (2005), we can interpret ES estimates in relation to benchmarks provided by Cohen (1988). Application to the above ES estimates indicated hearing aids had no effect (0.02) to a small (0.28) effect on HRQoL as measured with generic instruments; and, a medium (0.51) to large (1.01) effect as measured with disease-specific instruments. - Summary of the Findings
Systematic reviews should conclude with a discussion of the overall results, their application to the clinical question, recommendations for clinical practice, and the proposal of additional directions for research. Often, a "grading system" is used to indicate the strength of the recommendation for clinical practice (Cox, 2005). A grade of "A" indicates a clinician can be very confident in making the recommendation because a body of high-quality relevant research supports it. A grade of "D" indicates that the recommendation can be made only with a great deal of caution. As a consumer of a systematic review, the clinician should consider "whether the recommendations made were supported by the reported data" (JBI, 2000). Based on the levels of evidence, the quality of the evidence, and the effect size estimates, the AAA-TF concluded that a grade of "B" was appropriate for the recommendation of the use of hearing aids to improve self-perception of HRQoL. In other words, the research data support a conclusion that improvements in HRQoL are most likely the outcome from hearing aid use by individuals with adult-onset hearing loss, particularly when assessment focuses on the effects of hearing loss on quality of life.
Systematic reviews promise to be very useful for busy clinicians seeking to engage in EBP. Systematic reviews, however, can vary in quality. The more transparent and thorough the systematic review report, the easier it will be for consumers to judge its usefulness. The six questions posed in this article should help audiologists in critically appraising systematic reviews that address questions relevant to their clinical practice.
Footnotes
* see The Joanna Briggs Institute (JBI; 2000) for review.
** a preliminary draft of this publication can be found on the American Academy of Audiology website at www.audiology.org/NR/rdonlyres/B5ADF071-3B37-43BC-99BA-6838C885ABEC/0/qualityoflife.pdf
References
Alper, B. S., Hand, J. A., Elliott, S. G., Kinkade, S., Hauan, M. J., Onion, D. K., & Sklar, B. M. (2004). How much effort is needed to keep up with the literature relevant for primary care? Journal of the Medical Library Association, 4, 430-437.
American Academy of Audiology (2006). Guidelines for the Audiologic Management of Adult Hearing Impairment. Retrieved February 16, 2007 from www.audiology.org/NR/rdonlyres/5DE475B4-58F3-40A7-934E-584AC11EABE9/0/haguidelines.pdf.
American Speech-Language-Hearing Association. (2004). Introduction to Evidence-based Practice. Retrieved May 15, 2006, from www.asha.org/members/ebp.
Bess, F. (1995). Evidence-based audiology. American Journal of Audiology, 4, 5.
Chisolm, T. H., Johnson, C. E., Danhauer, J. L., Portz, L. J. P., Abrams, H. B., Lesner, S. A., McCarthy, P., & Newman, C. W. (in press). A systematic review of health-related quality of life and hearing aids: Final report of the American Academy of Audiology task force on the non-acoustic benefits of amplification. Journal of the American Academy of Audiology.
Cox, R. (2005). Evidence-Based Practice in Provision of Amplification. Journal of the American Academy of Audiology, 16(7), 419-438.
The Joanna Briggs Institute (JBI). (2000). Changing Practice: Appraising Systematic Reviews. Best Practice Information Sheets, Supp 1. Retrieved November 28, 2005, from www.joannabriggs.edu.au/pubs/best_practice.php?pageNum_rsBestPractice=3&totalRows_rsBestPractice=113
Law, M. (Ed.). (2002). Evidence-based rehabilitation. A guide to practice. Thorofare, NJ: Slack.
National Institutes of Health. (1993) Quality of Life Assessment: Practice, Problems, and Promise - Proceedings of a workshop, October 15-17, 1990. Washington, DC: US Dept of Health and Human Services.
Robey, R. (2004, April 13) Levels of evidence. The ASHA Leader, 5.
Scargle, J. D. (2000). Publication Bias: The "File-Drawer" Problem in Scientific Inference [Electronic version]. Journal of Scientific Exploration, 14(1), 91-106.
Scottish Intercollegiate Guidelines Network. (2001). SIGN 50: A guideline developer's handbook. Retrieved September 8, 2004, from www.sign.ac.uk/guidelines/fulltext/50/index.html
Thorne, P.R. (2003). Evidence-based audiology and clinical experience. The Australian and New Zealand Journal of Audiology, 25, 10-15.
Valente, M. (2006, October 30). Guideline for audiologic management of the adult patient. Audiology Online, Article 1716. Retrieved on February 16, 2007 from www.audiologyonline.com/articles/article_detail.asp?article_id=1716
