Assessment of Literature Adequacy and the GRADE Framework

After identifying the relevant literature, its adequacy and certainty should be assessed using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) framework, which classifies evidence as high, moderate, low, or very low certainty.^5,6 This framework is widely used by organizations such as the World Health Organization (WHO), the Cochrane Collaboration, and the American Society of Plastic Surgeons (ASPS).

Recommendations are characterized by both their direction—either in favor of or against an intervention—and their strength. Recommendations are considered strong when the benefits clearly outweigh the risks and conditional (weak) when greater uncertainty exists.⁷

GRADE incorporates Summary of Findings (SoF) tables, which provide a standardized summary of intervention effects and the certainty of the evidence, as well as Evidence to Decision (EtD) frameworks, which support the translation of evidence into recommendations.⁸ While SoF tables summarize what the studies show, EtD frameworks guide how decisions should be made based on that evidence. These frameworks can be operationalized through tools such as the Interactive Evidence to Decision (iEtD) platform (available at https://ietd.epistemonikos.org/).⁹

Criteria for Relevance, Credibility, and Currency

After identifying the relevant literature—guidelines, systematic reviews, or, in their absence, primary studies—its adequacy should be assessed according to three criteria: relevance, credibility, and currency. Relevance refers to the direct applicability of the evidence to the clinical question; discrepancies between the evidence and the question increase indirectness. Credibility relates to methodological quality and may be assessed using instruments such as AGREE II¹⁰ (guidelines), AMSTAR-2,¹¹ and ROBIS¹² (systematic reviews). Currency depends on how recent the literature searches were and whether new evidence has emerged since publication, which can be assessed through a rapid update.⁴ This assessment does not need to encompass the entire body of literature, but only the evidence used to support the recommendation. Furthermore, it is not a dichotomous process: different sources may present limitations to varying degrees. In general, relevance, credibility, and currency should be prioritized in that order, such that more relevant evidence may be preferred even when it is less recent.⁴

Depending on the availability and adequacy of the literature, recommendations may be developed using one of four strategies: adoption of existing guideline recommendations, adaptation of existing recommendations, development of new recommendations based on available systematic reviews, or, in the absence of such evidence, reliance on expert opinion.⁴

Adoption or Adaptation of Existing Recommendations

The literature search may identify an existing guideline that is considered appropriate for the urgent question, meaning that it is relevant, credible, and current. In such cases, the expert panel should determine whether the recommendation will be adopted as originally presented or adapted to the new context. In practice, a recommendation is considered adopted when neither its direction nor its strength is modified. Conversely, when either of these elements is changed, the recommendation is considered adapted.⁴

To decide whether a recommendation should be adopted or adapted, the panel should assess whether contextual differences may affect its direction or strength, drawing particularly on the SoF and EtD tables provided in the original guideline. Factors such as the importance of outcomes, potential indirectness of the evidence, and reassessment of evidence certainty—including considerations related to risk of bias, imprecision, inconsistency, or the availability of new evidence—may justify modifications.

Accordingly, it is essential to critically review the original guideline’s GRADE assessment in light of the new context, which requires a basic understanding of the GRADE framework.⁴

Classification of Evidence Certainty in GRADE

The GRADE framework classifies the certainty of evidence into four levels (high, moderate, low, and very low), considering the body of evidence for each outcome rather than individual studies.^7,13 Randomized controlled trials begin as high-certainty evidence, whereas non-randomized studies begin as low-certainty evidence. Evidence may be downgraded due to limitations such as risk of bias, imprecision, inconsistency, indirectness, and publication bias, or upgraded in specific circumstances, such as a large magnitude of effect or the presence of a dose–response gradient. For both randomized and non-randomized studies, downgrading is performed according to the severity of the identified limitations and may occur by one level (e.g., from high to moderate certainty) or by two levels (e.g., from high to low certainty).⁷

Importance of Outcomes and the Minimally Important Difference (MID)

When moving from evidence to recommendations, the panel should consider the importance of benefits, harms, and burdens from the patient perspective. The overall certainty of evidence is determined by the lowest certainty rating among the critical outcomes.⁷ In addition to determining whether an effect exists, it is essential to assess its clinical relevance.

For this purpose, the concept of the minimally important difference (MID) is used. The MID represents the smallest change perceived as important by patients. Accordingly, the assessment may focus either on the presence of an effect (relative to the null effect) or on its clinical importance (whether it exceeds the MID threshold). The certainty of evidence is then judged considering limitations such as imprecision, inconsistency, risk of bias, indirectness, and publication bias.¹⁴

Imprecision

Intervention studies aim to estimate the true effect of a treatment. In this context, the pooled estimate from a metaanalysis represents the best point estimate of that effect, whereas the confidence interval (CI), typically set at 95%, reflects the range within which the true effect is likely to lie. Studies with small sample sizes tend to produce more imprecise estimates,¹⁵ which is reflected in wider confidence intervals, particularly when they encompass both benefit and no effect.¹⁴

Within the GRADE framework, imprecision is assessed according to the position of the confidence interval relative to either the null effect or the minimally important difference (MID), and may result in downgrading the certainty of evidence by one or two levels. If the confidence interval crosses the selected threshold of interest—whether the null effect or the MID—the evidence should be downgraded for imprecision, regardless of sample size. When the confidence interval does not cross the threshold but the observed effect is very large (e.g., a relative risk reduction greater than 40%), caution is warranted because such large effects are uncommon. In these situations, if the number of participants is small, downgrading for imprecision is also recommended. This assessment is supported by the concept of the Optimal Information Size (OIS), which represents the ideal number of participants required in a meta-analysis. If the meta-analysis reaches the OIS, downgrading for imprecision is generally not warranted; otherwise, downgrading should be considered.

Inconsistency

In the GRADE framework, inconsistency refers to differences in the results of studies evaluating the same intervention. It is suggested when point estimates of treatment effects differ substantially, when confidence intervals show little overlap, or when they point in opposite directions, thereby reducing the certainty of the evidence.¹⁶

Inconsistency may arise from random variation or from true heterogeneity among studies, with the I² statistic serving as the primary indicator of such variability. Low I² values (<30%) rarely indicate important inconsistency, whereas higher values increase the likelihood that the certainty of evidence will be downgraded. However, interpretation of the I² statistic should be undertaken cautiously, particularly when confidence intervals are very narrow.^15,16

Risk of Bias

Risk of bias is a central domain within the GRADE framework and should be assessed using appropriate appraisal tools.¹⁷ In randomized controlled trials, instruments such as ROBUST-RCT¹⁸ evaluate domains including randomization, allocation concealment, blinding, and missing outcome data, classifying studies as having either low or high risk of bias. The certainty of evidence may be downgraded when important limitations are identified in one or more of these domains, and at the body-of-evidence level when studies at high risk of bias predominate (>55%) within the available evidence.

The impact of bias, however, depends on its likely direction. If a bias would be expected to overestimate an effect that was not observed, downgrading may not be warranted. Furthermore, when discrepancies exist between studies at high and low risk of bias, the analysis may focus on the lowrisk studies without necessarily downgrading the certainty of evidence.

Non-randomized controlled studies begin as low-certainty evidence and may be further downgraded according to assessments performed using instruments such as the Newcastle–Ottawa Scale.¹⁹ In the absence of factors warranting downgrading, their certainty may, under specific circumstances, be upgraded—for example, when a large magnitude of effect is observed (such as doubling a beneficial effect or halving the risk of an adverse outcome) or when a dose– response gradient is present—while remaining cautious regarding the potential influence of residual confounding. Case series and single-arm studies, by contrast, are generally considered very low-certainty evidence.

Publication Bias

Publication bias, another domain within the GRADE framework, occurs when studies—particularly those with negative or unfavorable results—remain unpublished, thereby distorting estimates of treatment effects. Its assessment is indirect and is often based on the evaluation of funnel plots. Although asymmetry in a funnel plot may suggest the presence of publication bias, it is not definitive evidence of its existence. The risk of publication bias is considered higher when the available evidence is dominated by small studies, particularly those funded by industry. In such situations, downgrading the certainty of evidence may be warranted.¹⁷

Indirectness

Indirectness, the final GRADE domain, refers to discrepancies between the clinical question (population, intervention, comparator, outcome [PICO]) addressed by the guideline under development and the available evidence. It occurs when there are differences in the population, intervention, comparator, or outcomes. Not every difference requires downgrading; however, downgrading may be warranted when such discrepancies are likely to influence the expected effect.

This may occur, for example, when evidence is extrapolated across populations with substantially different clinical characteristics or when it is based on outdated technologies. Indirectness may also arise when the outcomes evaluated do not directly reflect what is important to patients, such as when surrogate outcomes are used. Surrogate outcomes are indirect measures—typically laboratory or intermediate outcomes—that do not directly represent clinically important benefits but instead serve as markers of the intervention’s effects. In such situations, when the available evidence does not adequately correspond to the target context or to clinically relevant outcomes, the certainty of evidence should be downgraded.²⁰

Additional Considerations

When meta-analysis is not feasible because of incompatible or insufficient data, narrative synthesis may be used. In such cases, evidence should continue to be assessed by outcome, and the same GRADE domains should be applied to classify the certainty of evidence.⁸ The Synthesis Without Meta-analysis (SWiM) reporting guideline provides additional guidance on this process.²¹

When both randomized and non-randomized studies are available, higher-certainty evidence should generally be prioritized. When the certainty of evidence is similar, findings from both study designs may be presented. In addition to the GRADE domains, contextual factors may also be considered when formulating recommendations, including the perspective of the guideline (individual or population-based), resource use, equity, acceptability, and feasibility.²² The decision to adapt recommendations depends on the existence of meaningful differences between the context of the original guideline and the current setting. When recommendations are adapted, the rationale for these modifications should be explicitly described, and the wording of the recommendations may be adjusted to improve their applicability to the target audience.⁴

Formulation of Recommendations in GRADE

Within the GRADE framework, recommendations are characterized by both their direction (for or against an intervention) and their strength (strong or conditional), resulting in four principal categories, in addition to the rare option of restricting an intervention to research settings. Strong recommendations are issued when there is a clear balance between benefits and harms favoring one course of action and when well-informed patients would be expected to make similar choices. Conditional recommendations, by contrast, reflect greater uncertainty or variability in patient values and preferences and generally require shared decision-making²³ (►Supplementary Figure S3).

Supplementary Figure S3 - Framework for formulating recommendations using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach.

Supplementary Figure S3 - Framework for formulating recommendations using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach.

The formulation of recommendations is based on three pillars: the magnitude of benefits, harms, and burdens; the certainty of the evidence; and patient values and preferences, often organized using Evidence to Decision (EtD) frameworks. The magnitude of benefits is interpreted in light of the minimally important difference (MID), assessing whether the intervention exceeds this threshold. The MID may be derived from the literature, determined through patient interviews, or, in the absence of such information, established by consensus among panel members.

The determination of the direction and strength of recommendations depends initially on the certainty of the evidence and subsequently on the balance between the benefits and harms of the intervention.

When evidence is of high or moderate certainty and the balance of benefits and harms clearly favors an intervention, a strong recommendation should be issued. Conversely, when the balance between benefits and harms is less clear, the recommendation will generally be conditional.

When evidence is of low or very low certainty, recommendations will generally be conditional, regardless of whether the balance of benefits and harms favors the intervention. However, in exceptional circumstances, a strong recommendation may still be issued despite low-certainty evidence, such as when an intervention is associated with substantial risk or important harm. GRADE also standardizes recommendation wording: “we recommend” is used for strong recommendations, whereas “we suggest” is used for conditional recommendations, avoiding ambiguous expressions such as “consider.” The present text provides a simplified overview of the GRADE framework; more detailed information is available in the methodological literature.^{7,8,14,16,17,20,23}

Recommendations in Settings with Limited Evidence or Extreme Urgency

In addition to adopting or adapting recommendations from existing guidelines, new recommendations may be developed in urgent situations (1–2 weeks) when new evidence emerges after the publication of previous guidelines or, in the absence of such guidelines, on the basis of appropriate systematic reviews. In these circumstances, recommendations should follow the GRADE framework, with methodological simplifications introduced when necessary to meet time constraints.²⁴

When neither guidelines nor systematic reviews are available, recommendations developed in urgent situations (1–2 weeks) may be based on expert evidence, as there is insufficient time to conduct rapid systematic reviews. In such cases, the panel reviews the available literature in a non-systematic manner within the limited timeframe and records its judgments using EtD frameworks. These assessments are subsequently consolidated and discussed until consensus is reached through deliberation or voting.⁴

In ultra-rapid emergency situations (1–2 hours), it may be necessary to rely on indirect evidence, with corresponding downgrading for indirectness. Under these extreme circumstances, recommendations may be based exclusively on expert opinion, without systematic reviews or the participation of methodologists. Such recommendations should be explicitly considered provisional and subject to revision as additional time becomes available for panel deliberation or as new evidence emerges^25–27 (►Supplementary Figure S2).

Rapid Guidelines and the Use of Methodological Shortcuts

When neither ultra-rapid emergency situations (1–2 hours) nor urgent situations (1–2 weeks) apply, rapid guidelines (1–3 months) may be developed. These guidelines employ abbreviated versions of the traditional methodological process, with the central challenge being the reduction of procedural steps without compromising validity or credibility. To achieve this, it is essential to preserve the core methodological elements, thereby ensuring that recommendations remain trustworthy²² (►Supplementary Figure S2). For traditional guidelines (>3 months), the Guideline International Network (GIN)–McMaster Guideline Development Checklist (GDC) is one of the principal methodological references. This comprehensive checklist provides guidance for all stages of the guideline development process, from planning to implementation and updating, and has become an international standard for the development of traditional guidelines.^3,28

To specifically support the development of rapid guidelines, an extension of the GDC was published in 2018, defining the essential principles that should be maintained even when abbreviated processes are used.²²

The initial principles of rapid guideline development emphasize structured planning and pragmatism. The available timeframe should first be defined, along with the components of the GDC that will be retained. The methodology should then be adapted according to resource limitations without compromising quality. Whenever possible, a target completion date should be established in advance.²²

Early planning is essential and includes the development of standard operating procedures, preparation of templates, early identification of reviewers, and organization of panel meetings. The rationale for developing a rapid guideline should also be clearly justified, typically based on the emergence of important new evidence regarding safety, effectiveness, or cost-effectiveness, or on the need for timely guidance.

The need for emergency recommendations prior to the rapid guideline should likewise be assessed and, when necessary, incorporated into the planning process. The development group should include key members, although its size may be reduced to facilitate efficiency. When feasible, additional resources may be allocated to optimize the process, including the recruitment of additional personnel and dedicated full-time staff. Conflict-of-interest management should be conducted efficiently and may require restrictions on panel composition. Nevertheless, in certain circumstances, the participation of experts with highly specialized knowledge may still be necessary despite potential conflicts of interest.

Subsequent principles focus on organizational structure and operational efficiency. The creation of a database of subject-matter experts facilitates rapid panel assembly and reviewer selection. When timelines are limited, virtual meetings should be prioritized to accelerate decision-making. The target audience and topic of the guideline should also be defined in advance and, whenever possible, communicated early in the process.

The formulation of PICO questions should reflect available resources, emphasizing precision and, when necessary, reducing both the number of questions and the overall scope of the guideline. Appropriate prioritization of questions is essential and should be clearly documented to ensure transparency.

The assessment should be restricted to the most important (critical) outcomes for decision-making while ensuring that both benefits and harms are considered. These outcomes may be defined iteratively by panel members, with their relevance progressively refined throughout the process.

Patient values and preferences remain central to recommendation development and may be assessed indirectly through the literature or, when such information is unavailable, through the collective experience of the panel, provided that this approach adequately reflects the patient perspective.

The search and evidence-synthesis strategy should reflect the resources available, prioritizing the use of existing guidelines or systematic reviews, which may be adopted, adapted, or updated as previously described in urgent situations. When adaptation is required, specific methodologies such as ADOLOPMENT²⁴ and ADAPTE may be used.^29,30

In the context of rapid guidelines (1–3 months), when neither guidelines nor systematic reviews are available, rapid systematic reviews are recommended, and their methodology has been well described in the medical literature.^13,31–38

In addition, primary studies and expert contributions may complement the Summary of Findings (SoF) tables of the new rapid guideline. Recommendation development may be accelerated through preliminary voting, with SoF tables distributed to panel members in advance. Areas of disagreement may subsequently be discussed during virtual meetings with the goal of reaching consensus, supported by tools such as GRADEpro (available at https://www.gradepro.org/). For many clinical questions, published evidence may be limited or unavailable. In such circumstances, the panel may decide not to issue recommendations, rely on expert opinion, or seek to generate primary data. When sufficient time is available, one possible strategy is to obtain information through structured online surveys involving experts with relevant subject-matter expertise, thereby collecting unpublished observations and case series. The resulting recommendations will generally be based on very low-certainty evidence; nevertheless, such a synthesis, despite its limitations, is often more robust than decisions based exclusively on expert opinion.³⁹

The wording of recommendations should be determined during panel meetings, ideally in parallel with the evaluation of the evidence, using standardized language that explicitly communicates the direction of the recommendation first and its strength second. In general, recommendations should be formulated in favor of a particular course of action rather than against a specific approach. However, when an ineffective or potentially harmful intervention is widely used, a recommendation against that practice may be appropriate.²³ Transparency is essential throughout the process, particularly when evidence is limited, and the methods used for evidence review and appraisal should be clearly documented. Peer review remains a critical stage of guideline development and may be expedited through the prior identification of reviewers and the establishment of predefined timelines.

Consideration should also be given to the implementation of the rapid guideline, including the assessment of feasibility and potential barriers, which should be explicitly described. Finally, it is advisable to establish in advance a timeline for the development of a traditional guideline following the recommendations of the Guideline International Network (GIN)–McMaster Guideline Development Checklist (GDC),²⁸ thereby ensuring a future update and a more comprehensive evaluation of the recommendations.²²