Editing Evidence-based medicine (section)

== Methods ==

=== Steps ===
The steps for designing explicit, evidence-based guidelines were described in the late 1980s: formulate the question (population, intervention, comparison intervention, outcomes, time horizon, setting); search the literature to identify studies that inform the question; interpret each study to determine precisely what it says about the question; if several studies address the question, synthesize their results ([[meta-analysis]]); summarize the evidence in evidence tables; compare the benefits, harms and costs in a balance sheet; draw a conclusion about the preferred practice; write the guideline; write the rationale for the guideline; have others review each of the previous steps; implement the guideline.<ref name=eddy4 />

For the purposes of medical education and individual-level decision making, five steps of EBM in practice were described in 1992<ref>{{cite journal | vauthors = Cook DJ, Jaeschke R, Guyatt GH | title = Critical appraisal of therapeutic interventions in the intensive care unit: human monoclonal antibody treatment in sepsis. Journal Club of the Hamilton Regional Critical Care Group | journal = Journal of Intensive Care Medicine | volume = 7 | issue = 6 | pages = 275–282 | year = 1992 | pmid = 10147956 | doi = 10.1177/088506669200700601 | s2cid = 7194293 }}</ref> and the experience of delegates attending the 2003 Conference of Evidence-Based Health Care Teachers and Developers was summarized into five steps and published in 2005.<ref name="Sicily statement on evidence-based">{{cite journal | vauthors = Dawes M, Summerskill W, Glasziou P, Cartabellotta A, Martin J, Hopayian K, Porzsolt F, Burls A, Osborne J | display-authors = 6 | title = Sicily statement on evidence-based practice | journal = BMC Medical Education | volume = 5 | issue = 1 | pages = 1 | date = January 2005 | pmid = 15634359 | pmc = 544887 | doi = 10.1186/1472-6920-5-1 | doi-access = free }}</ref> This five-step process can broadly be categorized as follows:

# Translation of uncertainty to an answerable question; includes critical questioning, study design and levels of evidence<ref>{{cite journal | vauthors = Richardson WS, Wilson MC, Nishikawa J, Hayward RS | title = The well-built clinical question: a key to evidence-based decisions | journal = ACP Journal Club | volume = 123 | issue = 3 | pages = A12–A13 | year = 1995 | pmid = 7582737 | doi = 10.7326/ACPJC-1995-123-3-A12 }}</ref>
# Systematic retrieval of the best evidence available<ref>{{cite journal | vauthors = Rosenberg WM, Deeks J, Lusher A, Snowball R, Dooley G, Sackett D | title = Improving searching skills and evidence retrieval | journal = Journal of the Royal College of Physicians of London | volume = 32 | issue = 6 | pages = 557–563 | year = 1998 | doi = 10.1016/S0035-8819(25)01804-5 | pmid = 9881313 | pmc = 9662986 }}</ref>
# Critical appraisal of evidence for [[internal validity]] that can be broken down into aspects regarding:<ref name="doi"/>
#* Systematic errors as a result of selection bias, information bias and confounding
#* Quantitative aspects of diagnosis and treatment
#* The effect size and aspects regarding its precision
#* Clinical importance of results
#* External validity or generalizability
# Application of results in practice<ref>{{cite journal | vauthors = Epling J, Smucny J, Patil A, Tudiver F | title = Teaching evidence-based medicine skills through a residency-developed guideline | journal = Family Medicine | volume = 34 | issue = 9 | pages = 646–648 | date = October 2002 | pmid = 12455246 }}</ref>
# Evaluation of performance<ref>{{cite journal | vauthors = Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, O'Brien MA, Johansen M, Grimshaw J, Oxman AD | display-authors = 6 | title = Audit and feedback: effects on professional practice and healthcare outcomes | journal = The Cochrane Database of Systematic Reviews | volume = 6 | issue = 6 | pages = CD000259 | date = June 2012 | pmid = 22696318 | doi = 10.1002/14651858.CD000259.pub3 | pmc = 11338587 }}</ref>

=== Evidence reviews ===
[[Systematic review]]s of published research studies are a major part of the evaluation of particular treatments. The [[Cochrane Collaboration]] is one of the best-known organisations that conducts systematic reviews. Like other producers of systematic reviews, it requires authors to provide a detailed study protocol as well as a reproducible plan of their literature search and evaluations of the evidence.<ref name="Tanjong2009">{{cite journal | vauthors = Tanjong-Ghogomu E, Tugwell P, Welch V | title = Evidence-based medicine and the Cochrane Collaboration | journal = Bulletin of the NYU Hospital for Joint Diseases | volume = 67 | issue = 2 | pages = 198–205 | year = 2009 | pmid = 19583554 | url = http://www.nyuhjdbulletin.org/Permalink.aspx?permalinkId=07a150ec-2275-4bbf-9003-6ea200b62a4f | url-status = dead | archive-url = https://web.archive.org/web/20130601140339/http://www.nyuhjdbulletin.org/Permalink.aspx?permalinkId=07a150ec-2275-4bbf-9003-6ea200b62a4f | archive-date = 1 June 2013 }}</ref> After the best evidence is assessed, treatment is categorized as (1) likely to be beneficial, (2) likely to be harmful, or (3) without evidence to support either benefit or harm.<ref>{{Citation |title=Grading of Recommendations and Levels of Evidence |date=2015-04-27 |work=Treatment for Post-Traumatic Stress Disorder, Operational Stress Injury, or Critical Incident Stress: A Review of Guidelines [Internet] |url=https://www.ncbi.nlm.nih.gov/books/NBK304764/ |access-date=2025-03-31 |publisher=Canadian Agency for Drugs and Technologies in Health |language=en}}</ref>

A 2007 analysis of 1,016 systematic reviews from all 50 Cochrane Collaboration Review Groups found that 44% of the reviews concluded that the intervention was likely to be beneficial, 7% concluded that the intervention was likely to be harmful, and 49% concluded that evidence did not support either benefit or harm. 96% recommended further research.<ref name="Mapping2007">{{cite journal | vauthors = El Dib RP, Atallah AN, Andriolo RB | title = Mapping the Cochrane evidence for decision making in health care | journal = Journal of Evaluation in Clinical Practice | volume = 13 | issue = 4 | pages = 689–692 | date = August 2007 | pmid = 17683315 | doi = 10.1111/j.1365-2753.2007.00886.x }}</ref> In 2017, a study assessed the role of systematic reviews produced by Cochrane Collaboration to inform US private payers' policymaking; it showed that although the medical policy documents of major US private payers were informed by Cochrane systematic reviews, there was still scope to encourage the further use.<ref>{{cite journal | vauthors = Singh A, Hussain S, Najmi AK | title = Role of Cochrane Reviews in informing US private payers' policies | journal = Journal of Evidence-Based Medicine | volume = 10 | issue = 4 | pages = 293–331 | date = November 2017 | pmid = 29193899 | doi = 10.1111/jebm.12278 | s2cid = 22796658 }}</ref>

=== Assessing the quality of evidence ===
{{Main|Levels of evidence}}
Evidence-based medicine categorizes different types of clinical evidence and rates or grades them<ref>{{cite web|url=http://www.essentialevidenceplus.com/product/ebm_loe.cfm?show=grade |title=EBM: Levels of Evidence |publisher=Essential Evidence Plus |access-date=23 February 2012}}</ref> according to the strength of their freedom from the various biases that beset medical research. For example, the strongest evidence for therapeutic interventions is provided by systematic review of [[randomized]], well-blinded, [[placebo-controlled trials]] with allocation concealment and complete follow-up involving a homogeneous patient population and medical condition. In contrast, patient testimonials, [[case report]]s, and even expert opinion have little value as proof because of the placebo effect, the biases inherent in observation and reporting of cases, and difficulties in ascertaining who is an expert (however, some critics have argued that expert opinion "does not belong in the rankings of the quality of [[empirical evidence]] because it does not represent a form of empirical evidence" and continue that "expert opinion would seem to be a separate, complex type of knowledge that would not fit into hierarchies otherwise limited to empirical evidence alone.").<ref name="Tonelli99">{{cite journal | vauthors = Tonelli MR | title = In defense of expert opinion | journal = Academic Medicine | volume = 74 | issue = 11 | pages = 1187–1192 | date = November 1999 | pmid = 10587679 | doi = 10.1097/00001888-199911000-00010 | doi-access = free }}</ref>

Several organizations have developed grading systems for assessing the quality of evidence. For example, in 1989 the U.S. Preventive Services Task Force (USPSTF) put forth the following system:<ref name="USPrevServTaskForce">{{cite book|author=U.S. Preventive Services Task Force|title=Guide to clinical preventive services: report of the U.S. Preventive Services Task Force|url=https://books.google.com/books?id=eQGJHgI_dR8C&pg=PR24 |date=August 1989|publisher=DIANE Publishing|isbn=978-1-56806-297-6|pages=24–}}</ref>
* Level I: Evidence obtained from at least one properly designed [[randomized controlled trial]].
* Level II-1: Evidence obtained from well-designed controlled trials without [[randomization]].
* Level II-2: Evidence obtained from well-designed [[Cohort study|cohort studies]] or [[case-control]] studies, preferably from more than one center or research group.
* Level II-3: Evidence obtained from multiple [[time series]] designs with or without the intervention. Dramatic results in uncontrolled trials might also be regarded as this type of evidence.
* Level III: Opinions of respected authorities, based on clinical experience, descriptive studies, or reports of expert committees.

Another example are the Oxford CEBM Levels of Evidence published by the [[Centre for Evidence-Based Medicine]]. First released in September 2000, the Levels of Evidence provide a way to rank evidence for claims about prognosis, diagnosis, treatment benefits, treatment harms, and screening, which most grading schemes do not address. The original CEBM Levels were Evidence-Based On Call to make the process of finding evidence feasible and its results explicit. In 2011, an international team redesigned the Oxford CEBM Levels to make them more understandable and to take into account recent developments in evidence ranking schemes. The Oxford CEBM Levels of Evidence have been used by patients and clinicians, as well as by experts to develop clinical guidelines, such as recommendations for the optimal use of phototherapy and topical therapy in [[psoriasis]]<ref>{{cite web | author = OCEBM Levels of Evidence Working Group |title=The Oxford Levels of Evidence 2' |url=http://www.cebm.net/index.aspx?o=5653 |date=May 2016 |access-date=9 December 2013 |archive-date=5 December 2013 |archive-url=https://web.archive.org/web/20131205135701/http://www.cebm.net/index.aspx?o=5653 |url-status=dead }}</ref> and guidelines for the use of the BCLC staging system for diagnosing and monitoring [[hepatocellular carcinoma]] in Canada.<ref>{{cite journal | vauthors = Paul C, Gallini A, Archier E, Castela E, Devaux S, Aractingi S, Aubin F, Bachelez H, Cribier B, Joly P, Jullien D, Le Maître M, Misery L, Richard MA, Ortonne JP | display-authors = 6 | title = Evidence-based recommendations on topical treatment and phototherapy of psoriasis: systematic review and expert opinion of a panel of dermatologists | journal = Journal of the European Academy of Dermatology and Venereology | volume = 26 | issue = Suppl 3 | pages = 1–10 | date = May 2012 | pmid = 22512675 | doi = 10.1111/j.1468-3083.2012.04518.x | s2cid = 36103291 }}</ref>

In 2000, a system was developed by the Grading of Recommendations Assessment, Development and Evaluation ([[GRADE]]) working group. The GRADE system takes into account more dimensions than just the quality of medical research.<ref name="GRADE">{{cite web|url=http://www.gradeworkinggroup.org/|title=Welcome to the GRADE working group|access-date=24 September 2007|archive-url=https://web.archive.org/web/20060207191324/http://www.gradeworkinggroup.org/|archive-date=7 February 2006|url-status=live|website=www.gradeworkinggroup.org}}</ref> It requires users who are performing an assessment of the quality of evidence, usually as part of a systematic review, to consider the impact of different factors on their confidence in the results. Authors of GRADE tables assign one of four levels to evaluate the quality of evidence, on the basis of their confidence that the observed effect (a numeric value) is close to the true effect. The confidence value is based on judgments assigned in five different domains in a structured manner.<ref name="balshem2011">{{cite journal | vauthors = Balshem H, Helfand M, Schünemann HJ, Oxman AD, Kunz R, Brozek J, Vist GE, Falck-Ytter Y, Meerpohl J, Norris S, Guyatt GH | display-authors = 6 | title = GRADE guidelines: 3. Rating the quality of evidence | journal = Journal of Clinical Epidemiology | volume = 64 | issue = 4 | pages = 401–406 | date = April 2011 | pmid = 21208779 | doi = 10.1016/j.jclinepi.2010.07.015 | doi-access = free | author3-link = Holger Jens Schünemann }}</ref> The GRADE working group defines 'quality of evidence' and 'strength of recommendations' based on the quality as two different concepts that are commonly confused with each other.<ref name=balshem2011 />

Systematic reviews may include randomized controlled trials that have low risk of bias, or observational studies that have high risk of bias. In the case of randomized controlled trials, the quality of evidence is high but can be downgraded in five different domains.<ref name="GRADE Handbook">{{cite book|title=GRADE handbook for grading quality of evidence and strength of recommendation|year=2009|edition=Version 3.2|editor3-link=Holger Jens Schünemann|veditors=Schünemann H, Brożek J, Oxman A}}<br />{{cite web|title=GRADEPro|website=Cochrane Informatics and Knowledge Management Department|url=http://tech.cochrane.org/gradepro|access-date=1 March 2016|archive-url=https://web.archive.org/web/20160305115426/http://tech.cochrane.org/gradepro|archive-date=5 March 2016|url-status=dead}}<br />{{cite book|title=GRADE handbook for grading quality of evidence and strength of recommendations. Updated October 2013|date=2013|publisher=The GRADE Working Group|veditors=Schünemann H, Brożek J, Guyatt G, Oxman A|url=https://gdt.gradepro.org/app/handbook/handbook.html|access-date=3 September 2019}}</ref>
* Risk of bias: A judgment made on the basis of the chance that bias in included studies has influenced the estimate of effect.
* Imprecision: A judgment made on the basis of the chance that the observed estimate of effect could change completely.
* Indirectness: A judgment made on the basis of the differences in characteristics of how the study was conducted and how the results are actually going to be applied.
* Inconsistency: A judgment made on the basis of the variability of results across the included studies.
* Publication bias: A judgment made on the basis of the question whether all the research evidence has been taken to account.<ref>{{Cite journal |last1=DeVito |first1=Nicholas J. |last2=Goldacre |first2=Ben |date=April 2019 |title=Catalogue of bias: publication bias |journal=BMJ Evidence-based Medicine |volume=24 |issue=2 |pages=53–54 |doi=10.1136/bmjebm-2018-111107 |issn=2515-4478 |pmid=30523135|doi-access=free }}</ref>

In the case of observational studies per GRADE, the quality of evidence starts off lower and may be upgraded in three domains in addition to being subject to downgrading.<ref name="GRADE Handbook" />
* Large effect: Methodologically strong studies show that the observed effect is so large that the probability of it changing completely is less likely.
* Plausible confounding would change the effect: Despite the presence of a possible confounding factor that is expected to reduce the observed effect, the effect estimate still shows significant effect.
* Dose response gradient: The intervention used becomes more effective with increasing dose. This suggests that a further increase will likely bring about more effect.

Meaning of the levels of quality of evidence as per GRADE:<ref name="balshem2011" />
* High Quality Evidence: The authors are very confident that the presented estimate lies very close to the true value. In other words, the probability is very low that further research will completely change the presented conclusions.
* Moderate Quality Evidence: The authors are confident that the presented estimate lies close to the true value, but it is also possible that it may be substantially different. In other words, further research may completely change the conclusions.
* Low Quality Evidence: The authors are not confident in the effect estimate, and the true value may be substantially different. In other words, further research is likely to change the presented conclusions completely.
* Very Low Quality Evidence: The authors do not have any confidence in the estimate and it is likely that the true value is substantially different from it. In other words, new research will probably change the presented conclusions completely.

=== Categories of recommendations ===
In guidelines and other publications, recommendation for a clinical service is classified by the balance of risk versus benefit and the level of evidence on which this information is based. The U.S. Preventive Services Task Force uses the following system:<ref>{{cite journal | vauthors = Sherman M, Burak K, Maroun J, Metrakos P, Knox JJ, Myers RP, Guindi M, Porter G, Kachura JR, Rasuli P, Gill S, Ghali P, Chaudhury P, Siddiqui J, Valenti D, Weiss A, Wong R | display-authors = 6 | title = Multidisciplinary Canadian consensus recommendations for the management and treatment of hepatocellular carcinoma | journal = Current Oncology | volume = 18 | issue = 5 | pages = 228–240 | date = October 2011 | pmid = 21980250 | pmc = 3185900 | doi = 10.3747/co.v18i5.952 }}</ref>
* Level A: Good [[scientific evidence]] suggests that the benefits of the clinical service substantially outweigh the potential risks. Clinicians should discuss the service with eligible patients.
* Level B: At least fair scientific evidence suggests that the benefits of the clinical service outweighs the potential risks. Clinicians should discuss the service with eligible patients.
* Level C: At least fair scientific evidence suggests that the clinical service provides benefits, but the balance between benefits and risks is too close for general recommendations. Clinicians need not offer it unless individual considerations apply.
* Level D: At least fair scientific evidence suggests that the risks of the clinical service outweigh potential benefits. Clinicians should not routinely offer the service to asymptomatic patients.
* Level I: Scientific evidence is lacking, of poor quality, or conflicting, such that the risk versus benefit balance cannot be assessed. Clinicians should help patients understand the uncertainty surrounding the clinical service.

GRADE guideline panelists may make strong or weak recommendations on the basis of further criteria. Some of the important criteria are the balance between desirable and undesirable effects (not considering cost), the quality of the evidence, values and preferences and costs (resource utilization).<ref name="GRADE Handbook" />

Despite the differences between systems, the purposes are the same: to guide users of clinical research information on which studies are likely to be most valid. However, the individual studies still require careful critical appraisal<ref>{{Cite web |title=Appraise the evidence {{!}} BMJ Best Practice |url=https://bestpractice.bmj.com/info/us/toolkit/learn-ebm/appraise-the-evidence/ |access-date=2025-02-10 |language=en-US}}</ref>

=== Statistical measures ===
Evidence-based medicine attempts to express clinical benefits of tests and treatments using mathematical methods. Tools used by practitioners of evidence-based medicine include:
* Likelihood ratio {{Main|Likelihood ratios in diagnostic testing}} The [[pre-test odds]] of a particular diagnosis, multiplied by the likelihood ratio, determines the [[pre- and post-test probability|post-test odds]]. (Odds can be calculated from, and then converted to, the [more familiar] probability.) This reflects [[Bayes' theorem]]. The differences in likelihood ratio between clinical tests can be used to prioritize clinical tests according to their usefulness in a given clinical situation.
* AUC-ROC The area under the [[receiver operating characteristic]] curve (AUC-ROC) reflects the relationship between [[sensitivity and specificity]] for a given test. High-quality tests will have an AUC-ROC approaching 1, and high-quality publications about clinical tests will provide information about the AUC-ROC. Cutoff values for positive and negative tests can influence specificity and sensitivity, but they do not affect AUC-ROC.
* [[Number needed to treat]] (NNT)/[[Number needed to harm]] (NNH). NNT and NNH are ways of expressing the effectiveness and safety, respectively, of interventions in a way that is clinically meaningful. NNT is the number of people who need to be treated in order to achieve the desired outcome (e.g. survival from cancer) in one patient. For example, if a treatment increases the chance of survival by 5%, then 20 people need to be treated in order for 1 additional patient to survive because of the treatment. The concept can also be applied to diagnostic tests. For example, if 1,339 women age 50–59 need to be invited for breast cancer screening over a ten-year period in order to prevent one woman from dying of breast cancer,<ref name="BandolierStatins2004">"Patient Compliance with statins" ''[[Bandolier (journal)|Bandolier]]'' [http://www.medicine.ox.ac.uk/bandolier/booth/cardiac/patcomp.html Review] 2004 {{Webarchive|url=https://archive.today/20150512021122/http://www.medicine.ox.ac.uk/bandolier/booth/cardiac/patcomp.html |date=12 May 2015 }}</ref> then the NNT for being invited to breast cancer screening is 1339.

=== Quality of clinical trials ===
Evidence-based medicine attempts to objectively evaluate the quality of clinical research by critically assessing techniques reported by researchers in their publications.
* Trial design considerations: High-quality studies have clearly defined eligibility criteria and have minimal missing data.<ref name=":03">{{Cite journal |last1=Bellomo |first1=Rinaldo |last2=Bagshaw |first2=Sean M. |date=2006 |title=Evidence-based medicine: classifying the evidence from clinical trials—the need to consider other dimensions |journal=Critical Care |volume=10 |issue=5 |pages=232 |doi=10.1186/cc5045 |issn=1466-609X |pmc=1751050 |pmid=17029653 |doi-access=free }}</ref><ref>{{Cite journal |last=Greeley |first=Christopher |date=December 2016 |title=Demystifying the Medical Literature |journal=Academic Forensic Pathology |volume=6 |issue=4 |pages=556–567 |doi=10.23907/2016.055 |issn=1925-3621 |pmc=6474497 |pmid=31239931}}</ref>
* Generalizability considerations: Studies may only be applicable to narrowly defined patient populations and may not be generalizable to other clinical contexts.<ref name=":03"/>
* Follow-up: Sufficient time for defined outcomes to occur can influence the prospective study outcomes and the [[statistical power]] of a study to detect differences between a treatment and control arm.<ref name=":2">{{Cite journal |last=Akobeng |first=A. K. |date=August 2005 |title=Understanding randomised controlled trials |journal=Archives of Disease in Childhood |volume=90 |issue=8 |pages=840–844 |doi=10.1136/adc.2004.058222 |issn=1468-2044 |pmc=1720509 |pmid=16040885}}</ref>
* Power: A mathematical calculation can determine whether the number of patients is sufficient to detect a difference between treatment arms. A negative study may reflect a lack of benefit, or simply a lack of sufficient quantities of patients to detect a difference.<ref name=":2"/><ref name=":03"/><ref name=":1">{{Cite web |date=2007 |title=Statistical Power |url=http://www.bandolier.org.uk/booth/glossary/statpow.html |website=Bandolier}}</ref>