AGREE II tool
We used the AGREE II tool to assess the quality of the CPGs development.
(6) AGREE II is a validated instrument
composed of 23 items grouped under 6 domains and one final item to
evaluate the overall quality of the CPG. The domains are: Scope and
Purpose (3 items), Stakeholder Involvement (3 items), Rigor of
Development (8 items), Clarity of Presentation (3 items), Applicability
(4 items), and Editorial Independence (4 items). For each item, each
appraiser is asked to score based on the statement, using a Likert scale
from 1 ((Strongly disagree) to 7 (Strongly agree). A score between 2 and
6 is assigned when the reporting of the AGREE II item does not meet the
full criteria or considerations depending on the completeness and
quality of reporting. Scores increase as more criteria are met and
considerations addressed.(5,
14, 15)
Following the recommendations by the AGREE collaboration, each CPG was
evaluated independently by three assessors. With the aim of reducing the
variability among the assessors we performed an intensive training in
the tool led by one of the investigators (IDF), expert in the use of
AGREE II. After initial training, the assessors were calibrated by
independently assessing two CPGs (one low quality and one high quality),
followed by feedback from the trainer (IDF), and by a group discussion
to solve any issues that arose.
We performed two rounds of assessment and feedback. We considered
disagreement between the assessors when the differences among the scores
were of 3 or more points, and they were resolved by discussion. If
agreement was not achieved after the discussion, two reviewers (IDF,
LEC) made the final assessment and decision. Final scores per domain
were calculated by summing up scores from the 3 assessors per individual
item in a domain and by scaling the total as a percentage of the maximum
possible score for that domain. Therefore, each domain was scored in a
range from 0 to 100%.
The AGREE collaboration does not define minimum domain scores or
patterns of scores across domains to differentiate between high quality
and poor-quality guidelines. (20) Instead, the
recommendation for users is to use the scores to compare CPG among them
and determine specific thresholds based on the context in which the
guidelines are to be used. All the assessment were performed using the
My AGREE-PLUS online tool provided by the AGREE collaboration
(http://www.agreetrust.org/resource-centre/agree-plus/).