The American Journal of Sports

Medicine
http://ajs.sagepub.com/

How to Write a Systematic Review

Joshua D. Harris, Carmen E. Quatman, M.M. Manring, Robert A. Siston and David C. Flanigan
Am J Sports Med published online August 7, 2013
DOI: 10.1177/0363546513497567
The online version of this article can be found at:
http://ajs.sagepub.com/content/early/2013/08/05/0363546513497567

Published by:
http://www.sagepublications.com

On behalf of:
American Orthopaedic Society for Sports Medicine

Additional services and information for The American Journal of Sports Medicine can be found at:

P<P

Published online August 7, 2013 in advance of the print journal.

Email Alerts: http://ajs.sagepub.com/cgi/alerts

Subscriptions: http://ajs.sagepub.com/subscriptions
Reprints: http://www.sagepub.com/journalsReprints.nav
Permissions: http://www.sagepub.com/journalsPermissions.nav

>> OnlineFirst Version of Record - Aug 7, 2013

What is This?

Downloaded from ajs.sagepub.com at UNIVERSITAETSSPITAL on July 5, 2014

AJSM PreView, published on August 7, 2013 as doi:10.1177/0363546513497567

Basic Science Update

How to Write a Systematic Review

Joshua D. Harris,*y MD, Carmen E. Quatman,z MD, PhD, M.M. Manring, PhD,
Robert A. Siston,z|| PhD, and David C. Flanigan,z MD
Investigation performed at Sports Medicine Center, The Ohio State University, Columbus, Ohio
Background: The role of evidence-based medicine in sports medicine and orthopaedic surgery is rapidly growing. Systematic
reviews and meta-analyses are also proliferating in the medical literature.
Purpose: To provide the outline necessary for a practitioner to properly understand and/or conduct a systematic review for publication in a sports medicine journal.
Study Design: Review.
Methods: The steps of a successful systematic review include the following: identification of an unanswered answerable question; explicit definitions of the investigations participant(s), intervention(s), comparison(s), and outcome(s); utilization of PRISMA
(Preferred Reporting Items for Systematic Reviews and Meta-analyses) guidelines and PROSPERO registration; thorough systematic data extraction; and appropriate grading of the evidence and strength of the recommendations.
Results: An outline to understand and conduct a systematic review is provided, and the difference between meta-analyses and
systematic reviews is described. The steps necessary to perform a systematic review are fully explained, including the study purpose, search methodology, data extraction, reporting of results, identification of bias, and reporting of the studys main findings.
Conclusion: Systematic reviews or meta-analyses critically appraise and formally synthesize the best existing evidence to provide a statement of conclusion that answers specific clinical questions. Readers and reviewers, however, must recognize that the
quality and strength of recommendations in a review are only as strong as the quality of studies that it analyzes. Thus, great care
must be used in the interpretation of bias and extrapolation of the reviews findings to translation to clinical practice. Without
advanced education on the topic, the reader may follow the steps discussed herein to perform a systematic review.
Keywords: evidence-based medicine; meta-analysis; systematic review; PRISMA; PROSPERO

In addition, these reviews account for and attempt to

limit individual studies biases. Thus, systematic reviews
and meta-analyses are powerful in their ability to combine
patient outcomes from distinct, yet similar, trials. Therefore, they have the potential to provide sufficient patient
numbers and generalizable population information to
make more powerful evidence-based conclusions. The quality and strength of recommendations in a review are only
as strong as the quality of studies that are analyzed.1,5,22,24
Even randomized trials with high-level evidence are not
without limitations. Thus, great care must be used in the
interpretation of bias and extrapolation of the reviews
findings to translation to clinical practice. Thus, a systematic review of high-quality randomized controlled trials is
a high-quality review, as these investigations are at the
top of the evidence-based medicine hierarchy (Figure 1).
Similarly, a systematic review of retrospective case series
with level IV evidence is limited by the same biases that
qualify the individual studies as level IV evidence. Given
the recent rapid expansion of electronic and written medical publishing, systematic reviews and meta-analyses are
very useful in that they synthesize and present large bodies of evidence to the busy clinician. The purpose of this
review is to provide an outline for a practitioner to properly
understand and/or conduct a systematic review for publication in a sports medicine journal.

Evidence-based medicine utilizes the available medical literature to guide clinical decision making and assess the
strength of clinical recommendations. When diagnosing
and treating patients, practitioners employ evidence-based
guidelines to advocate for or against an intervention. Metaanalyses and systematic reviews critically appraise and
formally synthesize the best existing evidence to provide
a statement of conclusion that answers specific clinical
questions. Conduct of performance of this type of investigation is transparent, with explicit selection, evaluation, and
reporting of the analyzed evidence.
*Address correspondence to Joshua D. Harris, MD, The Methodist
Orthopedics and Sports Medicine Center, 6560 Fannin Street, Scurlock
Tower, Suite 400, Houston, TX 77030 (e-mail: joshuaharrismd@
gmail.com).
y
The Methodist Orthopedics and Sports Medicine Center, Houston,
Texas.
z
Sports Medicine Center, The Ohio State University, Columbus, Ohio.

Department of Orthopaedic Surgery, The Ohio State University,

Columbus, Ohio.
||
Department of Mechanical and Aerospace Engineering, The Ohio
State University, Columbus, Ohio.
The authors declared that they have no conflicts of interest in the
authorship and publication of this contribution.
The American Journal of Sports Medicine, Vol. XX, No. X
DOI: 10.1177/0363546513497567
2013 The Author(s)

1
Downloaded from ajs.sagepub.com at UNIVERSITAETSSPITAL on July 5, 2014

Harris et al

The American Journal of Sports Medicine

checklist in 2009.17 The 27-item checklist was intended

to improve review quality, not generate a review quality
score. Incorporation of PRISMA guidelines in sports medicine and orthopaedic surgery literature is being increasingly recognized through support and promotion by highquality publications.36 Further, PRISMA has been
included as an integral component in the international
EQUATOR (Enhancing the Quality and Transparency of
Health Care Research) initiative to improve the reporting
of published research.11

GETTING STARTED

Figure 1. Hierarchy triangle of evidence-based medicine.

META-ANALYSIS AND SYSTEMATIC REVIEW

Although commonly used interchangeably, systematic
reviews and meta-analyses are not the same. The
Cochrane Collaboration defines a systematic review as
a comprehensive high-level summary of primary research
on a specific research question that attempts to identify,
select, synthesize, and appraise all high-quality evidence
relevant to that question to answer it. Further, systematic
reviews collate all evidence pertinent to a priori selected
criteria for eligibility to address the specific research question. Additionally, they identify and minimize bias via
transparent, explicit, and systematic methodology.
A meta-analysis utilizes statistical methods (as differentiated from systematic reviews) to quantitatively evaluate
pooled data from single studies. Individual studies are
assigned a weight based on the sample size. Conclusions
are reported based on the accuracy and precision (mean
and confidence interval [CI] relative to a zero effect line
on a forest plot) of individual studies results. The width of
individual studies 95% CIs graphically depicts the degree
of clinical, statistical, and methodological heterogeneity
that is inevitably found in any clinical study. The forest
plot conveys the take-home point conclusion of a metaanalysis in one simple figure. Familiarity in reading and
interpreting a forest plot is essential in understanding the
effect of the analyzed treatment(s) and their magnitudes.
A meta-analysis does not necessarily mandate comprehensive inclusion of all studies relevant to a specific topic (eg,
as in a systematic review), only the mathematical assimilation of studies. Thus, not all meta-analyses are systematic
reviews. Likewise, not all systematic reviews are metaanalyses unless all studies are identified, included, and analyzed (systematic review) quantitatively (meta-analysis).
The QUORUM (Quality of Reporting of Meta-analyses)
statement was established in 1996 by a group of epidemiologists, biostatisticians, medical editors, and researchers to
improve the quality of conduct and reporting of systematic
reviews and meta-analyses.19 Developments in the performance and reporting of reviews led to the revision of
QUORUM to the PRISMA (Preferred Reporting Items for
Systematic Reviews and Meta-analyses) guidelines and

The PRISMA guidelines recommend open registration of all

systematic reviews.20 Given the recent proliferation of systematic reviews and meta-analyses (Figure 2), this reduces
the potential for duplication of resources devoted to a specific
clinical topic. The Journal of Bone and Joint Surgery, Clinical Orthopaedics and Related Research, and Journal of
Pediatric Orthopaedics have recently proposed an expectation of all authors of systematic reviews or meta-analyses
to identify any similar reviews and justify why a new investigation is unique and illustrates different findings than
prior reviews.36 More importantly, registration improves
the quality of conduct of the review and its eventual reporting. Thus, on February 22, 2011, PROSPERO, the online,
free, prospective international systematic review register,
was launched for any and all health carerelated
research.2,33 Nonpublication of finished systematic reviews,
just as with clinical trials (publication bias), is a problem
that may lead to overestimates of the effect of an intervention reported in the medical literature.29 Review registration is expected to reduce that bias, which preferentially
selects the publication of only studies that have positive
findings by ensuring public awareness before study commencement. Further, it has been demonstrated that studies
with statistically significant positive findings are more
likely to be reported in the English-language literature.7
Thus, the journal publications language for included studies is a necessary source of bias that must be accounted
for either before or during the review.

PURPOSE OF THE REVIEW: IDENTIFYING

AN ANSWERABLE QUESTION(S)
The purpose of a systematic review should be to answer an
important answerable clinical question or identify areas of
high clinical importance that are underreported in the medical literature.8,21 The question must be specific; however, it
must not be too specific. If the posed question is too broad
(eg, Is exercise beneficial?), then the reviewer is unable
to properly focus the literature to a manageable number of
studies to review and analyze. If the posed question is too
narrow (eg, Is exercising 43 minutes per day 3 times per
week better than exercising 38 minutes per day 4 times
per week?), then there are not enough published reports
available to answer the question. An appropriate question
should be original and focused on the precise identification

Downloaded from ajs.sagepub.com at UNIVERSITAETSSPITAL on July 5, 2014

Vol. XX, No. X, XXXX

Orthopaedic Systematic Review

Figure 2. According to the PubMed database, over the past

20 years, the number of systematic reviews and metaanalyses has increased each year.
of the participant(s), intervention(s), comparison(s), outcome(s), and study design (PICOS criteria). A meticulous
approach to the PICOS criteria is what makes a review systematic, contrary to a simple narrative review. It is the
information within PICOS that establishes the inclusion
and exclusion criteria for the studies analyzed.

ELIGIBLE STUDIES: INCLUSION

AND EXCLUSION CRITERIA
Once the study purpose is identified and an answerable
question posed, the reviewer must determine the study
inclusion and exclusion criteria. Generation of a PRISMA
flowchart (Figure 3) that demonstrates the identification
and screening of potentially eligible studies determines
the final number of studies included for analysis. Greater
specificity of the inclusion criteria (improved review internal validity) limits the heterogeneity (review external validity) of the studies in the final analysis. However, depending
on the study topic, greater study homogeneity is achieved at
the expense of the number of studies (and patients) analyzed. A priori, these criteria should be established. However, it is not uncommon to need to alter criteria as the
study search strategy ensues. For example, consider the following scenario: A novice reviewer asks the question What
is the rate of dislocation following total hip replacement?
The original inclusion criteria presented all clinical outcome
studies reporting a minimum 2-year follow-up after total hip
replacement. The search commences, and the reviewer
encounters studies on different surgical approaches (eg, posterior, lateral, anterior). The researcher must now make
a decision: (1) include all surgical approaches and combine
their rates, (2) include all surgical approaches and report
and compare the rates for each separate approach, or (3)
only include one approach and report its individual rate.
Preliminary research for the study background should
develop enough of a knowledge base to set up an appropriate and thorough list of inclusion and exclusion criteria so
that significant alterations are unnecessary during the
course of the search, as shown in the total hip example.

Figure 3. PRISMA flowchart that shows the step-by-step

process of the application of inclusion and exclusion criteria
to generate a final number of studies for analysis in the systematic review.
Critical evaluation of eligible studies also involves the
assessment of the studys level of evidence.35 A minimum
level of evidence may be an inclusion criterion in a systematic review. The highest level of evidence should always be
sought. However, important studies that evaluate the clinical question should not necessarily be excluded to meet the
highest level of evidence. Sometimes, the better performed
study with more accurate information may be provided in
a lower level investigation. Leaving this information out
to meet only level I studies may actually reduce the effectiveness of a well-performed systematic review. Further,
depending on the relevant topic, it may be discovered that
prospective randomized trials do not exist. In certain situations, level III and IV evidence may be the best possible evidence. For example, consider the following scenario: A
reviewer asks the question What are the clinical outcome
differences between compartment syndrome of the leg treated with 4-compartment fasciotomy within 4 hours and
within 24 hours? Clearly, no researcher would ever randomize a patient to the latter group if the patient presented
within 4 hours of the time of diagnosis. Only retrospective
case series would be available to study this patient. Therefore, a systematic review on this topic is the same level of
evidence as the minimum level of evidence of the studies
that it analyzes. Nevertheless, the best available evidence
does not always need to be a randomized controlled trial.

PRIMARY AND EXPLORATORY OUTCOMES

Just as part of the inclusion and exclusion criteria are
established a priori, the primary study outcomes should
be ascertained before the studies are selected and analyzed. Primary end points of a systematic review should

Downloaded from ajs.sagepub.com at UNIVERSITAETSSPITAL on July 5, 2014

Harris et al

The American Journal of Sports Medicine

be analogous to those of a clinical trial. It is the measure

upon which success or failure of an intervention is based
(ie, acceptance or rejection of the null hypothesis). It is
the main effect tested statistically for a difference and
whether that difference is clinically meaningful. Beyond
primary and secondary outcome measure(s) evaluated,
exploratory end points are generally assessed post hoc.
As with clinical trials, in a review, a priori planned outcome comparisons of an intervention on 2 groups of participants have a greater degree of statistical power compared
with post hoc comparisons.

SEARCH METHODOLOGY
The search strategy should be strictly focused on the
PICOS criteria. Before beginning the search, the authors
need to be aware that a PRISMA flowchart (Figure 3 and
Appendices 1-3 [available in the online version of this article at http://ajsm.sagepub.com/supplemental]) should be
created to illustrate study identification, screening, eligibility, inclusion, and analysis. There are several nonmutually exclusive electronic databases publicly available (free
of charge) for the extraction of studies to be included in systematic review analyses. Pay-per-use databases are also
accessible. Use of only a single database is insufficient.
PubMed is a free database that utilizes the MEDLINE
database maintained by the United States National
Library of Medicine of the National Institutes of Health.
As of December 31, 2012, there are 22.4 million citations
in the PubMed database. Embase is a subscription-based
database maintained by Elsevier BV, with more than
25.2 million citations (as of May 13, 2012). Embase contains all MEDLINE records and 5 million citations not
included in MEDLINE (including more than 2000 exclusively Embase-indexed journals). The Embase database is
growing by more than 1 million citations annually. PEDro
(Physiotherapy Evidence Database) is a free database
maintained by the Centre for Evidence-Based Physiotherapy. It contains more than 23,000 randomized controlled
trials, systematic reviews, and clinical practice management guidelines.
The Cochrane Library is a collection of 6 unique databases maintained by John Wiley & Sons Ltd. Included in
the Cochrane Library is the Cochrane Database of Systematic Reviews (CDSR), which contains 7626 systematic
reviews as of December 1, 2012. The reviews in the latter
are excellent examples upon which to guide a novice
author of systematic reviews. The Cochrane Central Register of Controlled Trials (CENTRAL) contains more than
680,000 controlled trials, most of which are concurrently
found in MEDLINE, Embase, Cochrane Review Groups
specialized registers, and hand-search results register.
SciVerse Scopus is a subscription-based database that contains 41 million records of peer-reviewed journals, books,
conferences, and scientific web pages.
Other commonly used databases for the identification of
studies relevant to the investigation for systematic reviews
include CINAHL (Cumulative Index to Nursing and Allied
Health Literature), SPORTDiscus, and Google Scholar.

Several publications have documented the necessity of utilizing at least 2 databases and, in certain circumstances,
hand-searching important selected journals.16,27,34 Nevertheless, in reference to sports medicine/orthopaedic surgery
meta-analyses, recall rates (defined as the proportion of primary studies analyzed within orthopaedic meta-analyses
that are indexed in either MEDLINE or Embase) of 90%
and 81% for MEDLINE and Embase were obtained individually for all primary studies included in the meta-analyses,
respectively.25 Combining MEDLINE and Embase
increased recall rates to 91%, and the addition of Cochrane
databases increased recall rates to 97%.25
Regardless of the database(s) used, the initial searchs
specificity must not compromise the sensitivity. In other
words, it is better to manually review more journal title(s),
abstract(s), and full-text article(s) in the database than to
be too specific and omit potentially inclusive studies. The
entire take-home point of the review, its results and conclusions, is based on the studies that it analyzes. Thus, it is
imperative to ensure that all of the relevant studies are
included. It may take a longer amount of time to perform
manual searching; however, this is absolutely necessary.
In addition to studies identified from the database(s)
searched, all reference lists from these studies should be
analyzed for the potential inclusion of studies omitted
from the initial search. A minimum of 2 reviewers should
perform the initial study identification, secondary study
screening, and final determination of eligibility and study
inclusion. Although the inclusion of conference abstracts
reduces publication bias, the bias introduced by the lack
of a formal rigorous peer-review process (and the oftenpresent differences between conference abstracts and final
published articles) often warrants the exclusion of
abstracts from high-quality systematic reviews.
Once the database review of study titles, abstracts, and
full-text articles has commenced, the investigator must be
sure to eliminate duplicate patient populations found in
different studies. This is applicable when authors publish
more than one article on the same group of patients, usually with different lengths of follow-up or with analysis
and reporting of a different primary or secondary outcome.
The easiest way to identify duplicate patient populations is
in the articles Methods section, which reports the dates of
patient enrollment. If 2 separate studies with the same
authors and the same intervention have overlapping dates
of patient enrollment, then only one study may be inclusive. In this situation, the reviewer should select the study
with the higher level of evidence, greater number of
patients, longer follow-up, or more thorough reporting of
the primary outcome of interest.

EXTRACTION AND ANALYSIS OF STUDY DATA:

REPORTING THE RESULTS
Once all exclusion criteria are applied and the final list of
studies is identified for analysis, there are several effective
methods of extraction of study data into a coherent group
of PICOS parameters. These data collection forms can be
either written paper checklists or electronic spreadsheets.

Downloaded from ajs.sagepub.com at UNIVERSITAETSSPITAL on July 5, 2014

Vol. XX, No. X, XXXX

Orthopaedic Systematic Review

Although a generic template form may be used to begin

data extraction from included studies for a systematic
review, each written review requires the individualization
of data extracted to the topic of interest. Data can be input
into a custom spreadsheet such as Microsoft Excel (Microsoft Corp, Redmond, Washington) (see Table 1 for an example of parameters sought and extracted). Multiple different
checklists have been published by various academic institutions and private groups. Review authors should find
one that suits their research style, research question, and
outcomes of interest most appropriately.
Two commonly used checklists to assist with the extraction of trial details are publicly available from the Centre
for Evidence-Based Medicine (CEBM)31 and the Cochrane
Collaboration.4 In 2005, Spindler et al26 were the first to
publish a checklist for use in the conduct and reporting
of systematic reviews in orthopaedic surgery (4 written
pages). It is not uncommon to continue the application of
review exclusion criteria during data extraction. Study
details not revealed in the title or abstract are revealed
upon review of the full text and may warrant exclusion.

STATISTICAL ASSIMILATION
OF DATA (META-ANALYSIS)
Once all studies data have been extracted, a brief tabular
narrative of each investigation may be presented for publication within the article.12 Columnar data usually include
the (1) year of publication, (2) lead author, (3) number of
study participants, (4) participant group(s), (5) intervention(s), (6) follow-up period, and (7) outcomes. Unique
study details may warrant the addition of further study
details in a tabular/graphical form.14 Additional tables
may be added to illustrate complications and reoperations14 and study limitations/biases.13 After each study
has been critically evaluated, the decision must be made
to quantitatively group the data with like outcome tools
together (meta-analysis).15 Homogeneous studies allow
for the assimilation of separate studies results and the creation of a forest plot, which is an illustration that displays
the relative strength (effect size) of an individual studys
results that evaluate the same intervention with the
same outcome measure. Effect size analysis allows for
a direct numerical comparison between different studies.
Performance of a meta-analysis is not always possible
because of heterogeneity among studies. Heterogeneity
may be assessed in 1 of 2 ways. One is visually with the
eyeball test, published by the CEBM in 2005.31 This
test simply seeks to find the overlap of CIs of the trials
with the summary estimate on the forest plot. A second
method involves a quantitative assessment of statistical
heterogeneity via tests such as the Cochrane Q (x2) test,
index test (I2), or t2 test (or T2). It is recommended to
seek the assistance of a biostatistician who is experienced
in meta-analyses early in the timeline of the investigation.37 Additionally, it is important to understand, identify,
and directly report the difference between statistical significance and clinical relevance within the individual

TABLE 1
Relevant Information Sought to Be Extracted
From Individual Trialsa
Study details
Study author(s)
Journal
Year of publication
Years of patient enrollment
Presence, absence, or lack of reporting of financial COI
Level of evidence
Study design (RCT, prospective, retrospective, etc)
Single vs multicenter
Country of study performance
Key study statements
Primary and secondary purposes
Hypothesis(es)
Study inclusion and exclusion criteria
Intended primary and secondary outcome measures
Evaluation of study methodological quality
Cochrane Bone, Joint and Muscle Trauma Groups methodological
quality score
CLEAR-NPT
Delphi list
Detsky Quality Assessment Scale
Coleman (and modified) methodology score
Quality appraisal tool
Jadad score
CONSORT
STROBE statement
Newcastle-Ottawa Quality Assessment Scale
Participants
Number of patients enrolled
Blinded? Yes/no
Number of male/female patients
Number of body parts analyzed (eg, knees)
Number of right vs left sides
Number of dominant vs nondominant
Number of surgical interventions (eg, total knee arthroplasty)
Patient age, mean 6 SD
Clinical follow-up, mean 6 SD
% follow-up (eg, .80%)
Radiographic follow-up (if applicable), mean 6 SD
Duration of symptoms warranting an intervention, mean 6 SD
Length of treatment before intervention, mean 6 SD
Number of prior surgical procedures, mean 6 SD
Other treatments performed
Interventions
Number of interventions performed in each group, mean 6 SD
Number of providers performing the intervention
Blinded? Yes/no
Comparators
Number of patients in control or comparator groups
Description of treatment in the control or comparator groups
Outcomes
Explicit description of all outcome measures and scores utilized
Preintervention score
1-year score
2-year score
5-year score
Etc
Number of complications
Number of reinterventions
Independent blinded evaluator/assessor?
Were study conclusions based on study results?
a
CLEAR-NPT, Checklist to Evaluate and Report a Non-Pharmacologic
Trial; COI, conflict of interest; CONSORT, Consolidated Standards of
Reporting Trials; RCT, randomized controlled trial; SD, standard deviation;
STROBE, Strengthening the Reporting of Observational Studies in
Epidemiology.

Downloaded from ajs.sagepub.com at UNIVERSITAETSSPITAL on July 5, 2014

Harris et al

The American Journal of Sports Medicine

TABLE 2
Pearls and Pitfalls for the Conduct of a High-Quality Systematic Reviewa
Pearls

Pitfalls

Answerable question
Does the review improve significantly upon existing reviews?
PICOS strategy protocol
PROSPERO registration
PRISMA guidelines, checklist, and flowchart
Thorough data extraction
Quantitative synthesis of studies data (if applicable, meta-analysis)
Grade the evidence and strength of recommendations (eg, SORT,
GRADE)
Explicit statement of the reviews take-home point

Underestimating the length of time to complete the review

Failing to identify if the review has already been conducted recently
Question nonspecific or too broad (unanswerable)
Failure to identify explicit study inclusion and exclusion criteria
Failure of review transparency
Not excluding duplicate study populations within different studies
Failure to recognize and report heterogeneity between studies
Failure to recognize and report studies bias
Making claims in conclusions that are beyond the facts/results of the
review

a
GRADE, Grading of Recommendations Assessment, Development, and Evaluation; PICOS, Participants, Interventions, Comparisons/
Controls, Outcomes, and Study Design; PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-analyses; SORT, Strength
of Recommendation Taxonomy.

studys findings. For example, the postoperative outcome

difference between 2 new surgical techniques for cartilage
repair is statistically significant with regard to the International Knee Documentation Committee (IKDC) subjective score. However, the effect size of this difference is
small, not meeting the threshold of minimally detectable
change or minimal clinically important difference.10 Therefore, although the difference is statistically significant, the
difference is not clinically large enough to be perceived by
the patient as different.18 These differences are key in the
ability to report whether patients have improved (treatment response/responders), whether that improvement is
back to normal,18 and whether the outcome is acceptable
to the patient (patient acceptable symptom state [PASS]).30

EVALUATION OF STUDY
METHODOLOGICAL QUALITY
The quality of a systematic review (Table 2) is only as good
as the studies that it analyzes. Thus, a review of only randomized controlled trials with level I evidence is a level I
review. Further, a review of multiple level I randomized
trials and multiple level III retrospective case comparison
studies is a level III review. In addition to level of evidence
ratings (updated by the CEBM in 2011),32 there are several
different study methodological quality scores available to
numerically grade the quality of a trial (Table 1). Some
questionnaires are designed as guides to help improve
the conduct and reporting of trials and are not intended
to generate a quantitative result (eg, CONSORT). When
intended to numerically grade the quality of an investigation, these scores simply grade the quality of study reporting and not necessarily the quality of study performance
and conduct. It is important to recognize that, just as the
review search and study selection be performed by at least
2 reviewers, it is also important to assess study quality
with at least 2 reviewers as well. These quality evaluation
tools describe the potential sources of bias within studies
(eg, selection, performance, transfer, detection, publication, study design). The reviewers must be aware that

these steps are critical to grading the strength of evidence.

Therefore, they must be accurate, and reviewers may note
that this step in the review takes a significant length of
time, depending on how many studies are being analyzed.
Other similar assessment tools for grading the evidence
include SORT (Strength of Recommendation Taxonomy)6
and GRADE (Grading of Recommendations Assessment,
Development, and Evaluation).9
Beyond the analysis of individual study quality, questionnaires exist to grade the quality of systematic reviews
and meta-analyses as well. Therefore, these assessment
tools may be used by authors of reviews to guide them in
the relevant steps of performance and reporting of the
review. Evaluation of systematic review methodological
quality via AMSTAR (assessment of multiple systematic
reviews) has recently been introduced to improve the conduct and reporting of systematic reviews.23 The Cochrane
Collaboration has recently introduced the MECIR (Methodological Expectations of Cochrane Intervention Reviews)
guidelines, which list 803 and 10828 items for conducting3
and reporting28 systematic reviews, respectively. The
MECIR guidelines complement and supplement PRISMA
guidelines. They explicitly ask whether an item relevant
to an investigation is either mandatory or highly desirable.

SUMMARIZE THE FINDINGS

The final, and most important, step in the systematic
review is the reporting of the take-home point. Often,
busy clinicians only have time to read an abstract or the
study conclusions. This limited amount of text is the
authors single opportunity to convey the key findings of
the review. Therefore, authors must address their answerable question(s) and whether their hypothesis or hypotheses were confirmed. Authors must also acknowledge the
limitations identified in the analyzed studies, as this bias
parallels the bias of the review. The authors must make
conclusions based on the reviews results. There is no
room for speculation or discussion in this section, only
the facts. The reviews conclusions should, in essence, be

Downloaded from ajs.sagepub.com at UNIVERSITAETSSPITAL on July 5, 2014

Vol. XX, No. X, XXXX

Orthopaedic Systematic Review

the answer to the following question: If a reader were to

remember one thing about my review, what would it be?

REFERENCES
1. Bhandari M, Morrow F, Kulkarni AV, Tornetta P 3rd. Meta-analyses in
orthopaedic surgery: a systematic review of their methodologies.
J Bone Joint Surg Am. 2001;83(1):15-24.
2. Booth A, Clarke M, Dooley G, et al. The nuts and bolts of PROSPERO: an international prospective register of systematic reviews.
Syst Rev. 2012;1(1):2.
3. Chandler J, Churchill R, Higgins J, Lasserson T, Tovey D. Methodological standards for the conduct of new Cochrane Intervention
Reviews. Version 2.2. December 17, 2012. Available at: http://
www.editorial-unit.cochrane.org/sites/editorial-unit.cochrane.org/
files/uploads/MECIR_conduct_standards%202.2%2017122012.pdf.
Accessed December 22, 2012.
4. Cochrane Collaboration. Cochrane Handbook for Systematic
Reviews of Interventions. Available at: http://handbook.cochrane
.org/. Accessed January 17, 2013.
5. Dijkman BG, Abouali JA, Kooistra BW, et al. Twenty years of metaanalyses in orthopaedic surgery: has quality kept up with quantity?
J Bone Joint Surg Am. 2010;92(1):48-57.
6. Ebell MH, Siwek J, Weiss BD, et al. Strength of recommendation
taxonomy (SORT): a patient-centered approach to grading evidence
in the medical literature. Am Fam Physician. 2004;69(3):548556.
7. Egger M, Zellweger-Zahner T, Schneider M, Junker C, Lengeler C,
Antes G. Language bias in randomised controlled trials published in
English and German. Lancet. 1997;350(9074):326-329.
8. Fazalare JA, Griesser MJ, Siston RA, Flanigan DC. The use of continuous passive motion following knee cartilage defect surgery: a systematic review. Orthopedics. 2010;33(12):878.
9. Grade Working Group. Grading of Recommendations Assessment,
Development and Evaluation. Available at: http://www.gradeworkinggroup.org/ Accessed January 20, 2013.
10. Greco NJ, Anderson AF, Mann BJ, et al. Responsiveness of the International Knee Documentation Committee Subjective Knee Form in
comparison to the Western Ontario and McMaster Universities Osteoarthritis Index, modified Cincinnati Knee Rating System, and Short
Form 36 in patients with focal articular cartilage defects. Am J Sports
Med. 2010;38(5):891-902.
11. Harms M. The EQUATOR network and the PRISMA statement for the
reporting of systematic reviews and meta-analyses. Physiotherapy.
2009;95(4):237-240.
12. Harris J, Brophy R, Siston R, Flanigan D. Treatment of chondral
defects in the athletes knee. Arthroscopy. 2010;26(6):841-852.
13. Harris JD, Cavo M, Brophy R, Siston R, Flanigan D. Biological knee
reconstruction: a systematic review of combined meniscal allograft
transplantation and cartilage repair or restoration. Arthroscopy.
2011;27(3):409-418.
14. Harris JD, Siston RA, Brophy RH, Lattermann C, Carey JL, Flanigan
DC. Failures, re-operations, and complications after autologous
chondrocyte implantation: a systematic review. Osteoarthritis Cartilage. 2011;19(7):779-791.
15. Harris JD, Siston RA, Pan X, Flanigan DC. Autologous chondrocyte
implantation: a systematic review. J Bone Joint Surg Am. 2010;
92(12):2220-2233.
16. Hopewell S, Clarke M, Lefebvre C, Scherer R. Handsearching versus
electronic searching to identify reports of randomized trials.
Cochrane Database Syst Rev. 2007;(2):MR000001.
17. Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement for
reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J Clin
Epidemiol. 2009;62(10):e1-e34.

18. Mann BJ, Gosens T, Lyman S. Quantifying clinically significant

change: a brief review of methods and presentation of a hybrid
approach. Am J Sports Med. 2012;40(10):2385-2393.
19. Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF.
Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Metaanalyses. Lancet. 1999;354(9193):1896-1900.
20. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items
for systematic reviews and meta-analyses: the PRISMA statement.
J Clin Epidemiol. 2009;62(10):1006-1012.
21. Quatman CE, Quatman-Yates CC, Schmitt LC, Paterno MV. The clinical utility and diagnostic performance of MRI for identification and
classification of knee osteochondritis dissecans. J Bone Joint Surg
Am. 2012;94(11):1036-1044.
22. Sharma R, Vannabouathong C, Bains S, et al. Meta-analyses in joint
arthroplasty: a review of quantity, quality, and impact. J Bone Joint
Surg Am. 2011;93(24):2304-2309.
23. Shea BJ, Hamel C, Wells GA, et al. AMSTAR is a reliable and valid
measurement tool to assess the methodological quality of systematic
reviews. J Clin Epidemiol. 2009;62(10):1013-1020.
24. Simunovic N, Sprague S, Bhandari M. Methodological issues in systematic reviews and meta-analyses of observational studies in orthopaedic
research. J Bone Joint Surg Am. 2009;91 Suppl 3:87-94.
25. Slobogean GP, Verma A, Giustini D, Slobogean BL, Mulpuri K. MEDLINE, EMBASE, and Cochrane index most primary studies but not
abstracts included in orthopedic meta-analyses. J Clin Epidemiol.
2009;62(12):1261-1267.
26. Spindler KP, Kuhn JE, Dunn W, Matthews CE, Harrell FE Jr, Dittus
RS. Reading and reviewing the orthopaedic literature: a systematic,
evidence-based medicine approach. J Am Acad Orthop Surg.
2005;13(4):220-229.
27. Suarez-Almazor ME, Belseck E, Homik J, Dorgan M, Ramos-Remus
C. Identifying clinical trials in the medical literature with electronic
databases: MEDLINE alone is not enough. Control Clin Trials.
2000;21(5):476-487.
28. Tovey D. Standards for the reporting of new Cochrane Intervention
Reviews. Version 1.1. December 17, 2012. Available at: http://
www.editorial-unit.cochrane.org/sites/editorial-unit.cochrane.org/
files/uploads/MECIR%20Reporting%20standards%201.1_1712201
2_1.pdf. Accessed December 22, 2012.
29. Tricco AC, Pham B, Brehaut J, et al. An international survey indicated
that unpublished systematic reviews exist. J Clin Epidemiol. 2009;
62(6):617-623.e615.
30. Tubach F, Giraudeau B, Ravaud P. The variability in minimal clinically
important difference and patient acceptable symptomatic state values did not have an impact on treatment effect estimates. J Clin Epidemiol. 2009;62(7):725-728.
31. University of Oxford, CEBM. Systematic review critical appraisal
sheet. Available at: http://www.cebm.net/index.aspx?o=1913.
Accessed January 20, 2013.
32. University of Oxford, CEBM. OCEBM levels of evidence system.
Available at: http://www.cebm.net/index.aspx?o=5653. Accessed
December 20, 2012.
33. University of York, Centre for Reviews and Dissemination. PROSPERO: international prospective register of systematic reviews.
Available at: http://www.crd.york.ac.uk/PROSPERO/. Accessed
December 25, 2102.
34. Whiting P, Westwood M, Burke M, Sterne J, Glanville J. Systematic
reviews of test accuracy should search a range of databases to identify primary studies. J Clin Epidemiol. 2008;61(4):357-364.
35. Wright JG, Swiontkowski MF, Heckman JD. Introducing levels of evidence to the journal. J Bone Joint Surg Am. 2003;85:1-3.
36. Wright JG, Swiontkowski MF, Tolo VT. Meta-analyses and systematic reviews: new guidelines for JBJS. J Bone Joint Surg Am.
2012;94(17):1537.
37. Wright RW, Brand RA, Dunn W, Spindler KP. How to write a systematic review. Clin Orthop Relat Res. 2007;455:23-29.

For reprints and permission queries, please visit SAGEs Web site at http://www.sagepub.com/journalsPermissions.nav

Downloaded from ajs.sagepub.com at UNIVERSITAETSSPITAL on July 5, 2014