Quality of randomized clinical trials published by plastic surgeons: a long-term follow-up study
Qualidade dos ensaios clínicos aleatórios publicados por cirurgiões plásticos: seguimento de longo prazo

INTRODUCTION

A randomized clinical trial (RCT) is a prospective study design that compares the effect of interventions on humans in one or more groups against a control group. RCTs are the best source of evidence for health interventions. Evidence-based medicine (EBM) is defined as the use of the best existing scientific evidence, provided by appropriately designed and conducted RCTs results, combined with individual clinical expertise and patient preferences and values, for deciding on individual patient care^1-4.

EBM became popular in the 1980s and impacted all fields of medicine, including Plastic Surgery⁵. The application of EBM principles can not only determine the best treatment for the patient but can also reduce the costs of healthcare systems². However, particularly in the surgical areas, there are challenges to overcome, including the belief that the application of EBM could reduce the autonomy of surgeons, and that the best scientific evidence does not always exceed the best practices. This concern is even greater in Plastic Surgery, in which the results are measured not only by the occurrence of complications and need for reintervention, but mainly by patient satisfaction with aesthetic results⁶.

Nevertheless, the EBM practice in Plastic Surgery is no longer a trend, it is a reality⁶. Information obtained from research with methodological rigor has become the key point of EBM and translation of knowledge⁷. Therefore, the results of well-conducted RCTs can have a significant impact on medical care by contributing to the establishment of solid scientific evidence that will serve as basis for clinical care protocols and interventions⁸.

Research in Plastic Surgery will have a much greater influence on clinical practice if studies with greater impact are published^9,10. Thus, the identification and systematic evaluation of the RCTs conducted by plastic surgeons, and their impact on the specialty, allow the implementation of evidence-based practice, with direct benefits for patients^11,12.

Several studies have indicated that plastic surgeons recognize the need to improve the level of evidence of research in Plastic Surgery, and this recognition reflects in the continuous increase in publications of clinical trials by this specialty^2,8,12-17.

A previous study identified RCTs with properly described allocation concealment, published by plastic surgeons between 1966 and 2003, and evaluated their quality¹⁸. Subsequently, in another study, the evolution was assessed over a subsequent five-year period (2004 to 2008), and a quantitative and qualitative increase in the RCTs published by plastic surgeons were observed⁸. This study aimed to test whether there was a quantitative and qualitative improvement in the RCTs in Plastic Surgery in another five-year period (2009 to 2013) compared to previously studied periods.

OBJECTIVE

To evaluate the evolution of randomized clinical trials in Plastic Surgery with adequately described allocation concealment, published between 2009 and 2013, compared to previously studied periods (1966-2003 and 2004-2008).

METHODS

The project was approved by the Research Ethics Committee of the Federal University of São Paulo – Paulista School of Medicine (UNIFESP - EPM), under the number 842.388, CAAE 37661814.8.0000.5505. The cases were selected by convenience, consisting of all the recovered RCTs that met the study eligibility criteria, published over a five-year period (January 2009 to December 2013)

Electronic searches were performed to identify the largest number of RCTs that were published by plastic surgeons in English. Specific search strategies were developed for each studied database, CCTR (Cochrane Central Register of Controlled Trials), LILACS (Latin American and Caribbean Health Science Literature), EMBASE (Excerpta Medica Database), and MEDLINE (MEDLARS- Medical Literature Retrieval System - online).

The abstracts of all retrieved articles were read by an evaluator and those that met the eligibility criteria (possible RCTs in which at least one plastic surgeon participated and published in English between 2009 and 2013) were selected for the reading of their full texts. Studies that were not conducted by plastic surgeons or without the participation of at least one plastic surgeon, and those published in a language other than English, were excluded. At this stage, whenever there was any doubt, the study was selected for reading in its entirety.

The full texts of the articles which had their abstracts selected were read by an evaluator to confirm the eligibility criteria. Subsequently, two independent evaluators classified the selected articles according to their allocation concealment¹⁹, and the disagreements were resolved in a consensus meeting. The RCTs published by plastic surgeons, with adequately described allocation concealment, were selected and constituted the sample of this study.

The selected RCTs were then evaluated for their quality. The evaluation was conducted independently by two evaluators, followed by a consensus meeting. Two validated instruments were used for quality assessment: the Delphi List²⁰ and the Jadad Scale²¹.

The Delphi List is a list of generic criteria for quality evaluation of clinical trials that should be used in combination with other instruments. It does not use scores and all items have two answer choices: “yes” or “no” (Chart 1)²⁰.

Chart 1 - Delphi List Items²⁰

The Jadad Scale is based on scores: one point is given for each “yes” answer, and a zero point for each “no” answer. Points counted for the first two items (randomization and double blinding) depend not only on how they are described, but also on the use of appropriate methods for this purpose. If the methods are described and appropriate, an additional point is given for each item. If the methods used to generate the randomization sequence or create the blinding conditions are described but inappropriate, the item will receive a zero point. Therefore, the scale encompasses scores from 0 to 5. The study will be considered of poor quality if it receives two or fewer points (Chart 2)²¹.

Chart 2 - Jadad Scale Items²¹

The results were compared to those obtained in the two previous studies that used the same method to evaluate the quality of the RCTs published by plastic surgeons in the 1966-2003 and 2004-2008 periods^8,18.

RESULTS

The electronic search identified 6,997 articles in the analyzed databases. One evaluator selected 616 articles, excluding 6,381 that were clearly not RCTs, were repeated in different databases, or did not include the participation of at least one plastic surgeon.

After reading the full texts of the 616 publications, the evaluator excluded 336 articles for the following reasons: one article was published in a language other than English (Russian); 219 did not include the participation of at least one plastic surgeon; 116 were not RCTs. Of the 280 remaining articles, 19 were repeated within the same database. Thus, the final selection comprised 261 studies.

Two evaluators independently classified the 261 selected studies according to their allocation concealment¹⁹. The kappa coefficient of agreement (kw) between the two evaluators was 0.94 (p = 0.000). After a consensus meeting, 43 RCTs published in English, with the participation of at least one plastic surgeon, and with adequately described allocation concealment were selected.

The comparison of the Delphi list items between the 2009-2013 period and previous study periods (1966- 2003 and 2004-2008) is shown in Table 1. A significant improvement in scores was observed when comparing the items “Groups were comparable in terms of the most important characteristics of the prognosis” (p < 0.001), “Inclusion and exclusion criteria were specified” (p = 0.0029), and “Measures of variability and estimation of points were presented for the primary variable” (p = 0.0057) in the 1966-2003 and 2004-2008 periods.

Table 1 - Quality assessment using the Delphi List after consensus meeting and comparison with the 1966-2003 and 2004-2008 periods.

The kappa coefficient of agreement (kw) between the two evaluators in the evaluation of the 43 RCTs for Jadad scores²¹ was 0.67 (p = 0.0000). Table 2 shows the distribution of the RCTs according to Jadad scores.

Table 2 - Quality Scale scores after consensus meeting.

DISCUSSION

The present study evaluated the evolution of the quality of RCTs published by plastic surgeons since 1966. The long follow-up period of 47 years, using the same method, facilitated a clear analysis of the evolution of studies over this period.

Considering the growing demand for specialized treatments and the limitation of health resources, there has been a growing interest in the practice of so-called “evidence-based medicine,”^12,14,23,24 and the qualitative and quantitative increase of publications of RCTs by plastic surgeons evaluated in the 47 years confirms this interest.

Conducting RCTs in surgical areas is accompanied by many difficulties and challenges, which includes ethical issues that make it impossible to use placebo procedures or even all procedures, compared to a surgical intervention; the impossibility of blinding the surgeon; learning curves; technical differences between surgeons, among others^25,26. Thus, the production of RCTs in this field has been slower compared to other medical specialties^24,27-31.

Despite the difficulties in conducting a randomized clinical trial in plastic surgery, evidence-based medicine is the key to the progress of this specialty¹⁵. Thus, the clinical trials that are effectively conducted should follow strict quality standards and methodological rigor, so that they may have more impact on clinical practice^11,12,32.

In order to improve the quality of RCTs, a group of researchers and editors elaborated the CONSORT (Consolidated Standards of Reporting Trials) Statement, initially published in 1996 and updated in 2001^31,32. From the review on the use of the CONSORT Statement, published in 2010, which comprises a checklist and a flow diagram, it became popular and was adopted by most medical journals as the standard for describing RCTs³³.

The use of the CONSORT Statement has contributed to an increase in the quality level of the published RCTs. An important limitation of the present study was the lack of use of the CONSORT checklist to assess the quality of published RCTs. However, it should be noted that the checklist, in its current form, was not available when the two previous studies were designed and conducted, and this study aimed to strictly use the same method as the two previous studies, thus, allowing a long-term evolution assessment.

A progressive increase in the number of RCTs published by plastic surgeons was observed over time. No RCT appropriately describing allocation concealment was published by plastic surgeons from 1966 to 1983. The first RCT with these characteristics was published in 1984, comparing the occurrence of capsular contracture after the use of saline implants or silicone gel for breast reconstruction³⁴. From then on, a progressive increase was observed, following the popularization of EBM from the 1980 decade⁶, but with a substantial increase only from the 2000s onwards.

In the present study, a higher concentration of publications was observed in Europe and North America, following a trend described in the other two periods for comparison (1966-2003 and 2004-2008). Momeni et al.¹¹, when evaluating RCTs in three major plastic surgery journals from 1990 to 2005, also observed a higher number of publications in Europe and North America.

The Delphi List²⁰, used in this study, evaluates three dimensions of the quality of an RCT: internal validity (degree of validity of the study for the assessed sample), external validity (degree of validity of the study in extrapolating its results to the population), and statistical analysis. The comparison of the Delphi List items between the present study (2009-2013) and the first studied period (1966-2003) ²⁰ indicates a greater number of responses, with statistical significance, for the items: “The groups were comparable in terms of the most important characteristics of the prognosis,” “The inclusion and exclusion criteria were specified,” and “Measures of variability and estimation of points were presented for the primary variable.” This shows an increase in the quality of RCTs published in the current period regarding these items. However, there was no change in the quality of the studies when the 2009-2013 period was compared to the previous five-year period (2004 to 2008).

The Jadad Scale is a short, simple, reliable, valid, and widely used instrument^21,35. Originally designed to assess pain in RCTs, it can be applied in other fields of medicine, since the items are not specific²¹. Olivo et al.³⁵, in a systematic review, analyzed the scales used to assess the methodological quality of RCTs in the health domain. They found that most of the scales did not have strict control over their development, nor were they tested for validity and applicability. They also observed that the Jadad Scale²¹ has been one of the most cited and used in the academic community of the health domain, besides having the best evidence for validity and applicability.

The evaluation of the Jadad scores²¹ indicated that there was a statistically significant increase in quality when comparing the 1966-2003 period to the 2004-2008 and 2009-2013 periods. However, there was no improvement in quality when comparing the 2009-2013 period to the previous five-year period (2004 to 2008), indicating a stabilization in the quality of RCTs.

Yu et al.³⁶ conducted a cross-sectional study aimed at assessing the quality of RCTs publications on surgery that were published in the 2003-2013 period. They used the conformity of the items to the CONSORT 2010^32,36 checklist as quality criteria. They observed that the studies published in 2013 obtained higher scores than those published in 2003, and this was statistically significant, suggesting an improvement in the quality of publications. They concluded that there has been an increase in the quality of RCTs publications on surgery in the last decade. However, this quality remains at suboptimal levels, especially regarding surgical interventions³⁶.

In this study, it was observed that the studies indicated a stabilization in quality compared to the last analyzed period (2009 to 2013) and the period before that (2004 to 2008) regarding the methodological criteria assessed using the Delphi List²⁰ and the Jadad Scale²¹. This could indicate that plastic surgeons, after a significant improvement, may have reached a basic level of quality in terms of criteria used for the publication of RCTs. However, several other items would need to be incorporated in order to increase their quality. This need reflects in the constant improvement of the CONSORT 2010³⁷checklist, and in the requirement, by an increasing number of journals, of compliance with thechecklist items for an RCT to be accepted for publication.

Plastic Surgery is characterized by a long history of innovation, which continues to this day, and has many contributions to share with other medical, clinical, or surgical specialties³¹. Existing barriers should not be considered as obstacles to scientific growth of the specialty, but as challenges to be overcome.

CONCLUSION

There was no difference in the quality of randomized clinical trials (RCTs) with appropriately described concealment allocation, published by plastic surgeons from 2009 to 2013 compared to the previous five-year period (2004 to 2008). However, both periods had better quality RCTs than the 1966-2003 period.