You are here

  • Home
  • Archive
  • Volume 15, Issue 2
  • Clinicians’ experience with infographic summaries from the BMJ Rapid Recommendations: a qualitative user-testing study among residents and interns at a large teaching hospital in Switzerland

Article Text

Article menu

Download PDFPDF

Medical education and training
Original research
Clinicians’ experience with infographic summaries from the BMJ Rapid Recommendations: a qualitative user-testing study among residents and interns at a large teaching hospital in Switzerland
  1. Tiffany Hirschel1,
  2. Per Vandvik2,3,4,
  3. Thomas Agoritsas2,5,6,7
  1. 1Anesthesiology Division, Department of Acute Medicine, Geneva University Hospitals, Geneva, Switzerland
  2. 2MAGIC Evidence Ecosystem Foundation, Oslo, Norway
  3. 3Institute of Health and Society, Faculty of Medicine, University of Oslo, Oslo, Norway
  4. 4Department of Medicine, Lovisenberg Diakonale Hospital, Oslo, Norway
  5. 5Faculty of Medicine, UNIGE, Geneva, Switzerland
  6. 6Division General Internal Medicine, Department of Medicine, Geneva University Hospitals, Geneva, Switzerland
  7. 7Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada
  1. Correspondence to Professor Thomas Agoritsas; Thomas.Agoritsas{at}unige.ch

Abstract

Objective Clinicians need trustworthy clinical practice guidelines to succeed with evidence-based diagnosis and treatment at the bedside. The BMJ Rapid Recommendations explore innovative ways to enhance dissemination and uptake, including multilayered interactive infographics linked to a digitally structured authoring and publication platform (the MAGICapp). We aimed to assess user experiences of physicians in training in various specialties when they interact with these infographics.

Design We conducted a qualitative user-testing study to assess user experience of a convenience sample of physicians in training. User testing was carried out through guided think-aloud sessions. We assessed six facets of user experience using a revised version of Morville’s framework: usefulness, understandability, usability, credibility, desirability and identification.

Setting Setting include Geneva’s University Hospital, a large teaching hospital in Switzerland.

Participants Participants include a convenience sample of residents and interns without restriction regarding medical field or division of care.

Results Most users reported a positive experience. The infographics were understandable and useful to rapidly grasp the key elements of the recommendation, its rationale and supporting evidence, in a credible way. Some users felt intimidated by numbers or the amount of information, although they perceived there could be a learning curve while using generic formats. Plain language summaries helped complement the visuals but could be further highlighted. Despite their generally positive experience, several users had limited understanding of key GRADE (Grading of Recommendations Assessment, Development and Evaluation) domains of the quality of evidence and remained uncertain by the implication of weak or conditional recommendations.

Conclusion Our study allowed to identify several aspects of guideline formats that improve their understandability and usefulness. Guideline organisations can use our findings to adapt their presentation format to enhance their dissemination and uptake in clinical practice. Avenues for research include the interplay between infographics and the digital authoring platform, multiple comparisons and living guidelines.

  • MEDICAL EDUCATION & TRAINING
  • Protocols & guidelines
  • Information technology

Data availability statement

Data used during the current study are available from the corresponding author upon request.

http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

https://doi-org.ezproxy.u-pec.fr/10.1136/bmjopen-2023-083032

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    STRENGTHS AND LIMITATIONS OF THIS STUDY

    • We conducted a rigorous user testing and analysis that translated the users’ experience into practical and transferable findings.

    • Interviews, in the form of guided think-aloud sessions, took place in French-speaking teaching hospital, and may not be fully generalisable to other settings.

    • We did not investigate knowledge retention, or impact on guideline adherence and related patient outcomes.

    • Our study did not compare different formats, although the sample of infographics tested included various iterations in design and across the RapidRecs.

    Introduction

    Incorporating current best evidence into decisions at the point of care is an ongoing challenge for physicians and care providers. An important gap remains between results from clinical research and their implementation in clinical practice, both because of challenges related to the production and dissemination of research findings and to the increasing demands and time constraints of physicians’ daily practice.1–3 The volume of evidence grows exponentially with more than 4000 articles published daily, including more than 100 randomised trial and 20 systematic reviews.4 Most published findings—even from randomised trials—suffer from substantial risk of bias,5 resulting in a majority of clinical decisions being informed by low certainty evidence.6 In this context of information overload, clinicians need trustworthy and readily available evidence to succeed with evidence-based diagnosis and treatment at the bedside.7–9 Haynes Evidence-based healthcare pyramid 5.010 models the information hierarchy when it comes to clinical decision-making. This pyramid highlights the importance of summarised, synthetised and filtered information of preappraised literature in clinical decision-making. It consists of five major layers: studies, systematic reviews, systematically derived recommendations (guidelines), synthesised summaries for clinical reference, and systems (that is the integration of evidence and guidance within computerised decision support systems and electronic health records). Each layer builds on the lower levels and provides progressively more useful information for guiding clinical practice. The need for synthetised information has been highlighted by previous research and guideline adherence may have a positive impact on patient outcomes and healthcare costs.11–14

    However, clinical practice guidelines can also suffer from major limitations. They are vulnerable to financial and intellectual interests.15 16 They can still be more consensus based than evidence based, with limited to no involvement from patient partners.17–19 Their development process is cumbersome and time-consuming, and they end up rapidly outdated along the rapid production of primary evidence.20 21 Correa et al22 conducted a meta-review on barriers and facilitators to the implementation of clinical practice guidelines and identified the absence of leadership within organisations, lack of time among healthcare professionals, and doubts regarding the credibility and applicability of clinical practice guidelines as major factors hindering the implementation of guidelines. Major advances in guideline methodology have occurred in the last decade, including standards for their development and assessment of transparency and rigour.7 23 The Institute of Medicine (IOM) has clarified explicit trustworthiness criteria.24 The Grading of Recommendations Assessment, Development and Evaluation (GRADE) working group has developed explicit approaches on how to summarise evidence, rate its quality and move in a transparent manner from evidence to recommendations.25–29 Application of these standards and methods, as well as management of conflict of interest, heavily impacts the strength of recommendations,6 30 31 yet they will not suffice to overcome barriers of their implementation at the point of care.8 32 33

    Another challenge relates to presentation formats of guidelines. Sharp et al34 conducted a systematic review examining various formats used for summarising evidence synthesis and their effectiveness and acceptability. The findings suggest that alternative formats to standard systematic review tables improved knowledge and understanding, but many trials lacked adequate reporting on study quality and no ‘gold standard’ of format emerged.

    To address these issues, the MAGIC Evidence Ecosystem Foundation (www.magicevidence.org), in collaboration with The British Medical Journal (BMJ), launched the BMJ Rapid Recommendations (also known as RapidRecs), whose aim is to produce and disseminate a new generation of trustworthy, timely and actionable recommendations on the basis of new practice-changing evidence, or complex ignored evidence.35 36 Using an online authoring platform—the MAGICapp (https://app.magicapp.org)—all data are digitally structured for dissemination in various formats and channels: multilayered formats for recommendations, evidence summaries, patient decision aids and widgets for other online platforms.8 37–40 Each RapidRec begins with an interactive infographic at the top, codesigned by MAGIC and the BMJ. The infographic operates as a doorway to each multilayered elements of the online version of the guideline published in the MAGICapp (www.bmj.com/rapid-recommendations).

    Infographics—that is, the visual representations of data using icons, illustrations or charts with a minimum of text—have been widely used to disseminate research findings or in medical education.41 Their potential merits among clinicians include the increase of dissemination and readership,42 43 who find them user-friendly and enjoyable.44–46 They answer the need for rapid information retrieval.47 48 They may help overcome statistical illiteracy by facilitating the understanding of complex data with less cognitive load than text.49 50 However, some have raised concern about actual knowledge retention or understanding, suggesting infographics may oversimplify information or prevent in-depth reading.41 42 46 51 52

    Less is known about the value of infographics as synopsis of clinical practice guideline.53 This study aims to assess the user experiences of clinicians when they interact with the RapidRecs’ infographics. Our main objectives were to identify guideline formats that help to facilitate usability and understanding or, on the contrary, formats that hinder their use. Since their creation in 2016, RapidRecs have been widely disseminated and reached high level of popularity, as assessed by their high Altmetric scores.54 55 They even have been adopted by the WHO for its COVID-19 therapeutic guidelines.56 They thus provide a unique opportunity to study the usefulness of infographics for the dissemination of trustworthy recommendations for clinical practice.

    Material and methods

    Study design

    We applied a user-testing design to assess user experiences of residents and interns in a large teaching hospital in Switzerland, when using BMJ RapidRecs infographics relevant to their practice and learning. User testing is a widely used method in web design research.57 It tries to capture and study the users experience through interview methods. Seven different facets of user’s experience have been described in Morville’s ‘honey-comb’ model.58 59 These include findability, accessibility, usability, usefulness, credibility, desirability and value. In our study, we decided to conduct non-directed interviews and encouraged participant to ‘think aloud’. These interviews occurred at their workplace. Interviews were recorded and analysed using Morville’s framework for user experience. The research is part of an MD thesis (TH).

    Intervention tested: the RapidRec Infographics

    The BMJ RapidRecs infographics have been codesigned in a unique collaboration between the BMJ and the MAGIC Evidence Ecosystem Foundation, combining skills from data graphic designers, journal editors, practising clinicians and experts in evidence-based medicine. Each RapidRec has its own infographic, presented in a synoptic view at the top of the publication.

    The content of the RapidRec infographics is specific to the clinical question being addressed, while its design is meant to be generic across each RapidRec. For each guideline, an unconflicted and multimultidisciplinary international panel, coordinated by MAGIC and the BMJ, appraised the evidence on a clinical question and issued their recommendations. Each panel included front-line clinicians (generalists and specialised in the question), methodologist, as well as patient partners. Each guideline panel had about 20–25 members, including 2–4 patient partners.35 36 60 The design of the infographic is multilayered, interactive and is organised in a similar generic way across RapidRecs. The content and data of the guideline are authored in the MAGICapp, a digitally structured authoring and publication platform, whose structure applies key elements of the IOM trustworthiness standards, GRADE rigorous methodology and the Evidence to Decision Framework (EtD).24–29 The data in the MAGICapp are then transformed into an infographic in collaboration with data graphic designers. The infographic then operates as a doorway to each multilayered element of the online version of the guideline in the MAGICapp, using widgets technology.61

    All existing RapidRecs can be found at the BMJ online portal (www.bmj.com/rapid-recommendations). Figures 1–4 illustrate the static view of one RapidRec on ‘Corticosteroids for sore throat’—the actual infographic being interactive and linking to MAGICapp content (direct link: https://www-bmj-com.ezproxy.u-pec.fr/content/358/bmj.j4090).62 The first layer of the guideline has three parts (figure 1): (1) the population to whom the recommendation applies (or may not apply); (2) the intervention and comparison (which can become several interventions in multiple comparison guidelines); and (3) the actual recommendation(s). The direction (for vs against) and strength (strong vs weak) of the recommendation are both depicted with arrows and text. The arrows are meant to visually convey to the user the four main options according to GRADE. The written statement of the recommendations applies formal GRADE guidance, using the word ‘recommend’ for strong recommendations, and the word ‘suggest’ for weak or conditional recommendations.63 64

    Figure 1
    Figure 1

    Example of an infographic: initial view (reproduced with permission from BMJ).

    Figure 2
    Figure 2

    Example of an infographic: strength of recommendation and selected GRADE (Grading of Recommendations Assessment, Development and Evaluation) summary of findings (reproduced with permission from BMJ).

    Figure 3
    Figure 3

    Example of infographic: key practical issues, values and preferences and additional considerations (reproduced with permission from BMJ).

    Figure 4
    Figure 4

    Modified Morville’s model for testing the experience of users, adapted from Semantic studio design63 and Rosenbaum et al.64

    When the user wishes to see more details, they can ‘click for details’ and access the next interactive layer: the main elements of GRADE summary of findings (figure 2). The magnitude of effects (in absolute terms) and the certainty of evidence are displayed for each patient-important outcome (figure 2). Continuous outcomes can either display absolute values (eg, kilograms, points on a quality of life scale, and/or be categorised as a proportion of responders (eg, according to the proportion of patient who may reach a minimally important difference).65 The quality of the evidence following GRADE guidance—also known as certainty of estimate of effect—can be high, moderate, low and very low quality or certainty.66 67 From each outcome, users can click on the ‘More button’ to access the next layer (figure 3) displaying GRADE plain language summary68 along with the detailed assessment of GRADE domains for quality of the evidence.

    Finally, at the bottom of each infographic, are listed key additional considerations identified by the panel, such as statements about patients’ values and preferences, resources or accessibility considerations, or other elements of EtD (figure 3).27–29 Our previous research has shown the critical importance of practical issues in decision-making. These are also generated by the RapidRec process and digitally structured in the MAGICapp. Key practical issues are then also displayed at the end of the infographic.39 40

    Participants and setting

    The setting of our study was at the University Hospitals of Geneva, in Switzerland, serving from primary and tertiary inpatient and outpatient care. We performed user tests among a convenience sample of physicians in training, focusing recruitment on residents and interns without restriction to the medical field or division of care. We sent to every resident and intern in training an email invitation to participate to our user testing (approximately 1000 people). Responses were voluntary with no additional incentive. A meeting was arranged with the convenience sample of those who manifested their interest spontaneously. Working in the same hospital, some participants were known by the interviewer, yet with no notable conflict of interest (TH). Participants provided written informed consent (see consent form as online supplemental file).

    Data collection

    Volunteer residents and interns met individually by the principal interviewer (TH), who had been trained for user-testing and think-aloud sessions by an experienced researcher (TA). The interviewer did not participate in the design of the infographic, nor in any of the RapidRecs’ panels, and thus was independent of known bias, as pertains to our research question. Interviews lasted 30–45 min, within the workplace at the hospital, so that residents and interns were in the usual clinical environment. The interviewer sat next to the participant, looking together at a computer showing the interactive infographics on BMJ’s website. They were no other participants during the interviews nor were there any patient present as it was not part of a clinical consultation. The sessions were audio recorded and transcribed anonymously. The interview started with a few general questions about the participant’s specialty they were training into, their age and years of experience, and previous awareness of the RapidRecs. Participants were then invited to freely choose which rapid recommendations they wished to explore, to ensure the topic was of sufficient relevance to the user. The investigator briefly explained the principle of the think-aloud session, in which the focus was to capture their direct experience while interacting with the RapidRec infographic. The interviews did not use a prespecified set of questions but rather a non-directive think-aloud approach, focusing on following the user experience of the formats displayed, asking open-ended questions and clarification of vague terms, listening carefully and only intervening when strictly necessary as per established guidance on user testing methodology.57 The interview ended with open questions about any general comments or suggestions to improve the tool. After each interview, quick notes were also written down to summarise the overall impression of the interview and help with the analysis.

    Data analysis

    The think-aloud sessions were transcribed by the principal interviewer. All identifiable features were removed. The interviews were then stored solely in an encrypted folder on the principal investigator’s computer. We analysed content using both deductive and inductive approaches, searching for units of meaning for user-experience, but within predetermined themes according to Morville’s framework.69 In this directed approach, two investigators (TH and TA) coded each element of meaning using a revised version of Morville’s framework, categorising them into the six following facets of user experience: usefulness, understandability, usability, credibility, desirability and identification(figure 4).39 58 59 Within each category, we then coded the quality of the reported experience—that is, showstoppers (preventing further use), major frustration (hindering further use, with participants eventually figuring out the issue), minor frustration, positive feedback and suggestions for improvement of the infographic.39 The coded elements were further discussed iteratively in the team to enhance reliability of the findings. Once we transcribed, coded and classified all comments into Morville’s categories, we searched for themes that came up at least more than twice. All comments regarding the same theme were then analysed together. We selected representative quotes for the issues most often experienced or that felt critical to users.

    Patient and public involvement

    Although each RapidRec tested included patients, no patient partner was involved in the design and conduct of the present study.40

    Results

    Participants

    We performed 32 think-aloud sessions, 9 among residents and 23 among interns, with a median age of 29 years (IQR 26–32 years). Men and women were represented equally (15 men and 17 women). Although all reported a good understanding of English, only one was a native speaker. Most participants reported between 2 and 4 years of clinical experience, and nine reported some previous experience in clinical or non-clinical research. At the time of their interview, 13 (40%) were in the Division of General Internal Medicine department, five (15%) in Paediatrics, two in the Emergency Room, two in Gastroenterology, two in Neurology, one in Nephrology (3%), one in Haematology, one in Rheumatology, one in Anaesthesiology (3%), one in Maxillofacial surgery, one in Urology (3%), one in Gynaecology-obstetrics (3%) and one in Psychiatry. More than half (17 participants) reported that they had heard of the BMJ Rapid Recommendations before, as several had been mentioned in educational rounds, but few had directly accessed the publications and their infographics.

    Overview of user experiences

    Participants chose freely among available RapidRecs on the BMJ (www.bmj.com/rapid-recommendations) at the time the user testing was conducted, between March 2019 and April 2020. Topics covered in the tested infographics included: transcatheter versus surgical aortic valve replacement for aortic stenosis,70 antiretroviral therapy in pregnant women living with HIV,71 corticosteroids for sore throat,62 antibiotics after drainage for uncomplicated skin abscesses,72 type of needles for lumbar puncture,73 patent foramen ovale closure, antiplatelets or anticoagulation for cryptogenic stroke,74 corticosteroid therapy for sepsis,75 prostate cancer screening with prostate-specific antigen,76 oxygen for acutely ill medical patients,77 dual antiplatelet therapy for transient ischaemic attack and minor stroke,78 low-intensity pulsed ultrasound for bone healing,79 treatment for subclinical hypothyroidism,55 gastrointestinal bleeding prophylaxis for critically ill patients,80 subacromial decompression surgery for shoulder pain81 and arthroscopic surgery for arthritis.54 We discuss findings in each of the facet of Morville’s user experience (See figure 4), with illustrative quotes from the interviews. Table 1 lists a summary of the key findings.

    View this table:
    Table 1

    Summary of findings per domain of Morville’s user experience with infographics

    Usefulness

    Most users found the infographics useful, allowing them to rapidly grasp the essence of the recommendation, its rationale and the overall quality of the underlying evidence. Key elements supporting rapid insights were the visual organisation of the infographic and its interactive multilayered feature.

    It’s quite telling visually, it is clear. It’s great. Sometimes it’s hard to find a good article, with good references, where the essential come out quickly. Frankly, yes, I’ll use it in practice, I think it’s pretty cool. When you have to present a meta-analysis, why bother presenting tables, when this speaks more to people, it’s more interactive, and, in the end, it gives exactly the same messages but more quickly. I prefer that to tables.—Intern in General Internal Medicine

    It allows you to answer a question that’s very pertinent by having arguments to fall back on. We can say yes because of this, or no because of that. All the while keeping in mind that there may be some evidence that is of insufficient quality, and it allows us to understand the state of the literature fairly quickly—Intern in General Internal Medicine

    Users had contrasting views about the optimal amount of information they needed in the infographic. Some felt that information was missing, such as number of randomised clinical trial included, and how many patients they included. This information is part of a separate infographic or table in each RapidRec manuscript, but participants tended not to scroll down the webpage to find it, as they most often remained within the infographic. Several participants were looking for a p value or CIs. These were also available in the MAGICapp, clickable at the bottom of each summary of finding or at the end of the infographic, but only few opted to click on it. Those looking for p values and CIs were typically not familiar with the GRADE wording, such as risk of bias, inconsistency, imprecision, indirectness and publication bias present at the deeper layer of the infographic, and thus did not recognise its relevance to the quality of the evidence. In contrast, most participants noticed the stars rating the evidence quality into very low, low, moderate and high and felt they understood their meaning. Other information that participants felt was missing was additional information about drug type, dosage and duration of treatment.

    Fewer participants found the infographic displayed too much information and were initially put-off by the summary of findings at the deeper layer, although some perceived it required getting familiar to the format.

    Every time I opened the thing [i.e., more details] I thought: wooh there’s so much info, there’s lots of little colours, numbers, for each thing you have to think about what it means… probably more time than if you just read the abstract of the metanalysis. […] To understand the information, you have to make a little more effort, but once you’ve made the effort and you understand how it works, you're faster… I think it’s a question of getting used to it—Intern in General Internal Medicine

    Several participants were puzzled by some core GRADE concepts—such as weak recommendations and their relationship with patients’ values and preferences—or even challenged their usefulness in a clinical practice guideline.

    I think they are pushing the definition of a guideline. Because a guideline tells you what you should do—Resident in Paediatrics

    Others appraised them with scepticism expressing that they wouldn’t change their practice on their basis, or would need to discuss them with more experienced clinicians.

    The recommendation is weak, this is why I wouldn’t apply it to my clinical practice—Intern in Haematology

    […] the weak recommendations I’d kind of ask myself if it’s right to follow them. I’d probably look at it with someone who has more experience than me.—Intern in Haematology

    Similarly, about half of the users felt somewhat puzzled by the box providing the panel’s statement on ‘values and preference’. Some participants read the statement as some sort of final conclusion or a written summary of the recommendation. Other correctly interpreted the statement but expressed that they were not interested in that type of information, while some did not seem to notice it.

    We get the impression that the recommendation is strong, based on patients' values and preferences, but not on “medical” grounds. So, you told me that they included patients, but I wonder if we shouldn’t just have the medical opinion, and maybe the patients’ opinion at a second stage, and maybe it shouldn’t be part of the recommendation.—Resident in General internal medicine

    Understandability

    All users easily understood the population and to whom the recommendation did or did not apply to (see figure 1). A large majority understood the recommendation, its direction and strength and main summary of findings, many found the information intuitive, although some users were quicker than others in understanding the key messages.

    Participants reported that a key feature supporting their understanding was that infographics were similar in format throughout the different recommendations, suggesting that there may be a learning curve in understanding the generic part of the infographic.

    I think when you’ve seen two or three, it’s easier to use them and you find the information faster […] it’s always the same presentation with exactly the same outcomes, so that’s pretty good.—Intern in General Internal Medicine

    Once you’ve seen it, it’s clear how it works and how it’s structured, and I think it’s really quite good. The winner is clear between that and reading a meta-analysis, it’s much clearer.—Intern in General Internal Medicine

    The use of colour and the organisation of the visual field enhanced understandability. All information concerning the same intervention is presented on the same side and is of the same colour (see figures 1 and 2).

    What I like about it is that it’s very simple and minimalistic to start with. And in fact, it allows us to get straight to the point, with recommendations and results, in a very strategic way, using colour codes, with a real differentiation of the visual space, with everything on the left concerning surgery and everything on the right concerning conservative treatment, highlighting the arrows in favour of a beneficial outcome. The star code is also a nice touch.—Intern in Gastroenterology

    Understandability was mediated by both symbols and written text, one complementing each other. For example, some participants expressed minor confusion understanding the symbols supporting the direction and strength of the recommendation (see also previous ‘usefulness’ section) but understood it better or felt reassured by the written summary underneath the arrows.

    So, there’s a strong recommendation to give hormones… Ah no, it’s NOT to give hormones. So, I corrected myself by reading the summary sentence, see? So it’s always good to make sentences.—Intern in Psychiatry

    Overall understandability of the summary of finding was also good, although a few participants were at first intimidated by numerical information, especially when associated with visual overload (see usefulness).

    The numbers themselves… I think you could have a box to find out more with the numbers, and then just have the sentences that tell you… for someone like me who isn't really into research.—Intern in Paediatrics intern

    So, in fact… There are numbers… so it’s true that since I do surgery and not really internal medicine, the numbers… I don't really have the values in my head, and there, I don’t see any units—Intern in Surgery

    A plain language summary for each outcome is included in the infographic, when clicking on ‘More’ button. One user mentioned a better understanding with the plain language summary, when clicking on the ‘more’ button for an outcome.

    Ahhh but this is good, because it’s much clearer when you open and see the little bubble, it’s much simpler. Frankly, I think it’s much clearer with the little bubble than with the arrows. Arrows aren’t very intuitive for me. If I’d opened it like that, I’d have understood straight away.—Intern in Paediatrics

    Understanding of the GRADE domains of quality of evidence was limited for about half the users (see also previous ‘usefulness’ section), and they looked for definitions. Infobuttons (information appearing when hovering over an item) were often received with enthusiasm.

    It’s quite nice that there’s a little commentary to help understand the GRADE score.—Intern in Paediatrics

    Usability

    Usability was very good in general. Only minor frustrations were identified such as too small a font size, or too much material on the same page, this was particularly the case when presented with the summary of findings containing a lot of information and numbers.

    Users appreciated the colours, but some felt they conveyed some hidden colour coding, or even overlooked information that was not highlighted with colour and felt that the information was generally less important.

    It feels like there is a hidden color code, the PPI and H2 are in green are better than the sucralfates in orange. I get the impression that this means there are not as good, when it’s probably just the color code.—Resident in General internal medicine

    I didn’t stop on preferences and values because they’re small and gray. Intern in Paediatrics

    Dropdown menus helped navigate the multilayered content, but the number of layers was overlooked by several users. While the ‘click for more details’ button going from the recommendation to the summary of findings was clicked by many, fewer spontaneously clicked on the more discreet ‘more’ button to get the plain language summary and GRADE full quality of evidence assessment, which stresses the importance of making dropdown menus visible enough.

    And it’s nice to have drop-down menus that open only on demand, if you really want to back up what’s been pre-summarized and visually pleasing to read. Intern in Gastroenterology.

    In addition to improve understandability (see above), the use of infobuttons (information that appeared when hovering over a text or an icon) enhanced navigational experience.

    it’s very nice when you hover the mouse over the abbreviations and the whole name is immediately visible Intern in Paediatrics

    Credibility

    Most users found the RapidRec infographic, highly credible, particularly given its affiliation to a major clinical journal like the BMJ, and the transparency in the date of publication.

    Yes, I trust the BMJ, I think we can easily rely on it. If you have any doubts, you can look up what you’re looking for in the studies. Resident in the Emergency Room

    Several participants spontaneously checked the guideline’s authors, looking up for familiar names.

    As discussed above in the section on the infographic’s usefulness, some users felt that some information lacked (ie, the number of trials and patients included), which affected their perception of the credibility of the guideline. Most of them did not scroll down the manuscript to find the information, which could have been found in another infographic or table, as they seemed to expect it at the top level. The same issue applied to those who expected to find p values and CIs to appraise impression at the top level, in place of the full GRADE assessment of the quality of evidence, although more comprehensive. For the same users, use of the GRADE approach did not enhance their experience of trustworthiness, as they lacked familiarity about guideline standards and methods.

    Identification

    Even though most users easily identified with the content of the RapidRec that they chose to explore, and found it relevant to their practice, they judged it on how aligned it was their current knowledge and practice. When the recommendation was in line with it, they were more inclined to consider the content as trustworthy. Conversely, users expressed more doubts when the recommendation differed from their current practice, in particular when the recommendation was issued as weak.

    I think it’s aimed more at general practitioners than specialists […] Sometimes we see 80-year-olds with high PSA levels, and we think it’s useless. So, I understand why they say no screening, but I'd be inclined to nuance this a little more, by saying that the shared decision should be made with someone who knows more than what’s being said here.—Intern in Urology.

    Although they were free to choose among the list of existing RapidRecs on the BMJ, these did not cover all fields of medicine, and some physicians, such as residents in psychiatry or paediatrics commented on how they would be keener to explore a content closer to their area of interest.

    Desirability

    One aspect that diminished desirability were the ads and the text surrounding the infographic. Some users looked for a way to show the infographic full screen, which is not possible in its interactive form.

    It starts with these huge ads on top of the page that take up half of the screen. Then there are links everywhere, you can click on a thousand things. You can like and tweet the article before having read it. It is confusing at the start, and you don’t really know where to begin—Intern in General internal medicine

    However, participants were generally positive about the look of this presentation format. Recurrent adjectives to describe the RR were ‘beautiful’ and ‘clear’, particularly the use of icons, images, colour, the visual organisation of space and text.

    Some mentioned how they could use this presentation format in the future, for example, during their next journal club or to ‘brag about their newly acquired knowledge’ at work:

    All this is real data? There’s no catch? There’s no “lorem ipsum” just filling in text like that? So tomorrow I can sound clever when I go to work?’ Intern in Neurology

    Discussion

    Principal findings

    Clinicians need trustworthy clinical practice guidelines to succeed in evidence-based diagnosis and treatment at the bedside, but their dissemination remains limited and their implementation lagging. Our study assessed the user experiences of physicians training in various specialties when they interact with the RapidRecs’ multilayered interactive infographics.

    We found that users had a mostly positive experience using these infographics. They were understandable and useful to rapidly grasp the essence of the recommendation, rationale and supporting evidence in a credible way. The usability of multilayered features through drop-down menus, the hover over infobuttons, the colours and the visual organisation were enjoyable and supported understanding. However, some features could be overlooked if too discreet or diluted in content, and colours could unwillingly convey more (or less) importance or suggest some hidden colour code. Visuals, numbers and text complement each other to enhance understandability. Some users felt intimidated by numbers or the amount of information, although they perceived there could be a learning curve using generic formats across RapidRecs. Plain language summaries helped but could be highlighted instead of being available only at deeper layers. Finally, several users had limited understanding of key GRADE domains of the quality of evidence and were puzzled by the implication of weak recommendations, or the panel’s statements about patients’ values and preferences.

    Comparison with current literature

    Infographics have been widely used to disseminate research findings or in medical education.41 But their use to disseminate clinical practice guideline has been limited, particularly the new generation of clinical practice guideline following trustworthiness standards and methods, including GRADE and the EtD. Our results mostly concur with those of Van Bostraeten et al53 who also conducted user testing among five selected BMJ RapidRecs suited from primary care. Although these were conducted among a difference population—namely GPs in outpatients practice, with a mean age of 48 years old—both positive and challenging experiences were remarkably similar. GPs reported that the selected infographics were time-efficient, easy to understand, trustworthy and supportive for decision-making. They reported similar challenges around the use of complex scales and terminology related statistics and guideline methodology.53

    Indeed, the most challenging finding of our study is also that, despite the user’s overall positive experience of the infographics, the format still fails to intuitively convey the meaning and implication of key GRADE concepts of evidence appraisal and strength of recommendations. From a substantial proportion of our clinicians in training, only a few were familiar with the terms ‘risk of bias’, ‘imprecision’, ‘indirectness’, ‘inconsistency’ and ‘publication bias’. This observation was surprising considering their exposure to GRADE in their medical curriculum, and its widespread use across many guidelines, including commonly used online summaries such DynaMed or UpToDate.6 These challenges had been observed in previous studies as has been highlighted by the review of Sharp et al.34 In their systematic review, they reported seven articles that highlighted the need to be explicit about how the scale is used and recommended to provide distinct explanations of the GRADE rating scale. The use of hover buttons to include definition of these terms might have a positive impact on understanding. The negative attitude, even paternalistic for some participants, towards the incorporation of patients values and preferences in the panel’s rationale, was previously observed by Brandt et al82 and conveys a well-known discomfort of clinicians with the very notion of weak recommendations. The GRADE working group has produced a whole body of methodological research to understand its numerous causes6 64 83 84 and attempts to innovate visual or narrative presentation formats.85 86 This is particularly important as at least two-thirds of recommendations across all medical fields are likely to be weak according to GRADE guidance.38 The solution may well be beyond the scope of infographics, or any specific formats that hope to intuitively convey these notions so that they feel acceptable and useful to clinicians, but rather through combined didactic efforts on guideline methodology and shared decision-making.38 87 88

    Strengths and weakness of the study

    Our study has limitations. Given its qualitative nature, our sample size was small. However, by processing iterative interviews, we felt that no substantial new theme was arising, suggesting that we had reached saturation and further inclusion of users would likely result in repetition. Another limitation is that interviews took place in a French-speaking teaching hospital, and in a controlled and think-aloud setting that may not be fully generalisable, although the intention of interviewing resident and interns at the workplace was to be close to their clinical practice and learning. User experience may differ when accessing infographics around a specific patient’s care. Moreover, we did not investigate knowledge retention, or impact on guideline adherence and related patient outcomes. Finally, our study did not compare different formats, although there were differences of design across the RapidRecs tested.

    Strengths of our study include a thorough process of user testing conduct and analysis, that has been validated to translate the user’s experience into practical and transferable findings.57 We conducted it among a population of clinicians in training with different backgrounds and specialties of interest. Our study adds to a rising body of evidence on how infographics can be used for the dissemination and uptake of clinical practice guidelines.44 46 50 53

    Meaning of the study

    The BMJ RapidRecs, and these accompanying infographics, are the result of a unique collaboration and cocreation between independent researchers and a wide-audience publisher. This model is offering new opportunities of interdisciplinary innovation between graphics designers, journal editors, practising clinicians, experts in evidence-based medicine and patient partners. Ongoing and future avenues of research include the interplay between infographics, digital authoring and publication platforms—including the enhanced support from A.I.—and the adaptation of infographics to specific methodological challenges such as complex risk stratification and personalised guidance,89 multiple comparisons of interventions90 ,91 and living evidence and guidelines.56

    Data availability statement

    Data used during the current study are available from the corresponding author upon request.

    Ethics approval

    The Geneva’s cantonal ethical commission for research (Commission Cantonale d’Ethique de la Recherche, CCER, Req-2019-00048) reviewed the protocol and exempted this study. Participants gave informed consent to participate in the study before taking part.

    Acknowledgments

    We wish to thank the BMJ, and in particular Dr Helen Mac Donald (BMJ Editor) and Dr Will Stahl-Timmins (BMJ Data Graphics Designer) for their invaluable contribution in the codevelopment and codesign of the infographics and RapidRecs formats

    References

    1. 1.↵
      1. Morris ZS,
      2. Wooding S,
      3. Grant J
      . The answer is 17 years, what is the question: understanding time lags in translational research. J R Soc Med 2011;104:510–20. doi:10.1258/jrsm.2011.110180
      OpenUrlCrossRefPubMed
    2. 2.↵
      1. Kristensen N,
      2. Nymann C,
      3. Konradsen H
      . Implementing research results in clinical practice- the experiences of healthcare professionals. BMC Health Serv Res 2016;16:48. doi:10.1186/s12913-016-1292-y
    3. 3.↵
      1. Fonarow GC,
      2. Yancy CW,
      3. Hernandez AF, et al
      . Potential impact of optimal implementation of evidence-based heart failure therapies on mortality. Am Heart J 2011;161:1024–30. doi:10.1016/j.ahj.2011.01.027
      OpenUrlCrossRefPubMedWeb of Science
    4. 4.↵
      1. Bastian H,
      2. Glasziou P,
      3. Chalmers I
      . Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med 2010;7:e1000326. doi:10.1371/journal.pmed.1000326
    5. 5.↵
      1. Dechartres A,
      2. Trinquart L,
      3. Atal I, et al
      . Evolution of poor reporting and inadequate methods over time in 20 920 randomised controlled trials included in Cochrane reviews: research on research study. BMJ 2017;357. doi:10.1136/bmj.j2490
    6. 6.↵
      1. Agoritsas T,
      2. Merglen A,
      3. Heen AF, et al
      . UpToDate adherence to GRADE criteria for strong recommendations: an analytical survey. BMJ Open 2017;7:e018593. doi:10.1136/bmjopen-2017-018593
    7. 7.↵
      1. Laine C,
      2. Taichman DB,
      3. Mulrow C
      . Trustworthy clinical guidelines. Ann Intern Med 2011;154:774–5. doi:10.7326/0003-4819-154-11-201106070-00011
      OpenUrlCrossRefPubMedWeb of Science
    8. 8.↵
      1. Vandvik PO,
      2. Brandt L,
      3. Alonso-Coello P, et al
      . Creating clinical practice guidelines we can trust, use, and share: a new era is imminent. Chest 2013;144:381–9. doi:10.1378/chest.13-0746
      OpenUrlCrossRefPubMedWeb of Science
    9. 9.↵
      1. Neumann I,
      2. Alonso-Coello P,
      3. Vandvik PO, et al
      . Do clinicians want recommendations? A multicenter study comparing evidence summaries with and without GRADE recommendations. J Clin Epidemiol 2018;99:33–40. doi:10.1016/j.jclinepi.2018.02.026
      OpenUrlCrossRefPubMed
    10. 10.↵
      1. Alper BS,
      2. Haynes RB
      . EBHC pyramid 5.0 for accessing preappraised evidence and guidance. Evid Based Med 2016;21:123–5. doi:10.1136/ebmed-2016-110447
      OpenUrl
    11. 11.↵
      1. Wöckel A,
      2. Kurzeder C,
      3. Geyer V, et al
      . Effects of guideline adherence in primary breast cancer--a 5-year multi-center cohort study of 3976 patients. Breast 2010;19:120–7. doi:10.1016/j.breast.2009.12.006
      OpenUrlPubMed
    12. 12.↵
      1. Proietti M,
      2. Nobili A,
      3. Raparelli V, et al
      . Adherence to antithrombotic therapy guidelines improves mortality among elderly patients with atrial fibrillation: insights from the REPOSI study. Clin Res Cardiol 2016;105:912–20. doi:10.1007/s00392-016-0999-4
      OpenUrl
    13. 13.↵
      1. Komajda M,
      2. Lapuerta P,
      3. Hermans N, et al
      . Adherence to guidelines is a predictor of outcome in chronic heart failure: the MAHLER survey. Eur Heart J 2005;26:1653–9. doi:10.1093/eurheartj/ehi251
      OpenUrlCrossRefPubMedWeb of Science
    14. 14.↵
      1. Wilke M,
      2. Grube RF,
      3. Bodmann KF
      . Guideline-adherent initial intravenous antibiotic therapy for hospital-acquired/ventilator-associated pneumonia is clinically superior, saves lives and is cheaper than non guideline adherent therapy. Eur J Med Res 2011;16:315. doi:10.1186/2047-783X-16-7-315
      OpenUrlCrossRefPubMed
    15. 15.↵
      1. Guyatt G,
      2. Akl EA,
      3. Hirsh J, et al
      . The vexing problem of guidelines and conflict of interest: a potential solution. Ann Intern Med 2010;152:738–41. doi:10.7326/0003-4819-152-11-201006010-00254
      OpenUrlCrossRefPubMedWeb of Science
    16. 16.↵
      1. Akl EA,
      2. Karl R,
      3. Guyatt GH
      . Methodologists and context experts disagreed regarding managing conflicts of interest of clinical practice guidelines panels. J Clin Epidemiol 2012;65:734–9. doi:10.1016/j.jclinepi.2011.12.013
      OpenUrlCrossRefPubMed
    17. 17.↵
      1. Djulbegovic B,
      2. Guyatt G
      . Evidence vs Consensus in Clinical Practice Guidelines. JAMA 2019;322:725–6. doi:10.1001/jama.2019.9751
      OpenUrlCrossRefPubMed
    18. 18.↵
      1. Kung J,
      2. Miller RR,
      3. Mackowiak PA
      . Failure of clinical practice guidelines to meet institute of medicine standards: Two more decades of little, if any, progress. Arch Intern Med 2012;172:1628–33. doi:10.1001/2013.jamainternmed.56
      OpenUrlCrossRefPubMedWeb of Science
    19. 19.↵
      1. Guyatt G,
      2. Vandvik PO
      . Creating clinical practice guidelines: problems and solutions. Chest 2013;144:365–7. doi:10.1378/chest.13-0463
      OpenUrlCrossRefPubMed
    20. 20.↵
      1. Alonso-Coello P,
      2. Martínez García L,
      3. Carrasco JM, et al
      . The updating of clinical practice guidelines: insights from an international survey. Implement Sci 2011;6:107. doi:10.1186/1748-5908-6-107
      OpenUrlCrossRefPubMed
    21. 21.↵
      1. Martínez García L,
      2. Arévalo-Rodríguez I,
      3. Solà I, et al
      . Strategies for monitoring and updating clinical practice guidelines: a systematic review. Implement Sci 2012;7:109. doi:10.1186/1748-5908-7-109
      OpenUrlCrossRefPubMed
    22. 22.↵
      1. Correa VC,
      2. Lugo-Agudelo LH,
      3. Aguirre-Acevedo DC, et al
      . Individual, health system, and contextual barriers and facilitators for the implementation of clinical practice guidelines: a systematic metareview. Health Res Policy Syst 2020;18:74. doi:10.1186/s12961-020-00588-8
    23. 23.↵
      1. Qaseem A,
      2. Forland F,
      3. Macbeth F, et al
      . Guidelines International Network: toward international standards for clinical practice guidelines. Ann Intern Med 2012;156:525–31. doi:10.7326/0003-4819-156-7-201204030-00009
      OpenUrlCrossRefPubMedWeb of Science
    24. 24.↵
      1. Institute of Medicine (US) Committee on Standards for Developing Trustworthy Clinical Practice Guidelines
      . Clinical Practice Guidelines We Can Trust. Washington (DC): National Academies Press (US), 2011.
    25. 25.↵
      1. Guyatt GH,
      2. Oxman AD,
      3. Vist GE, et al
      . GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ 2008;336:924–6. doi:10.1136/bmj.39489.470347.AD
      OpenUrlFREE Full Text
    26. 26.↵
      1. Guyatt GH,
      2. Oxman AD,
      3. Schünemann HJ, et al
      . GRADE guidelines: a new series of articles in the Journal of Clinical Epidemiology. J Clin Epidemiol 2011;64:380–2. doi:10.1016/j.jclinepi.2010.09.011
      OpenUrlCrossRefPubMed
    27. 27.↵
      1. Alonso-Coello P,
      2. Schünemann HJ,
      3. Moberg J, et al
      . GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction. BMJ 2016;353. doi:10.1136/bmj.i2016
    28. 28.↵
      1. Alonso-Coello P,
      2. Oxman AD,
      3. Moberg J, et al
      . GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 2: Clinical practice guidelines. BMJ 2016;353. doi:10.1136/bmj.i2089
    29. 29.↵
      1. Neumann I,
      2. Brignardello-Petersen R,
      3. Wiercioch W, et al
      . The GRADE evidence-to-decision framework: a report of its testing and application in 15 international guideline panels. Implement Sci 2016;11:93. doi:10.1186/s13012-016-0462-y
    30. 30.↵
      1. Agoritsas T,
      2. Neumann I,
      3. Mendoza C, et al
      . Guideline conflict of interest management and methodology heavily impacts on the strength of recommendations: comparison between two iterations of the American College of Chest Physicians Antithrombotic Guidelines. J Clin Epidemiol 2017;81:141–3. doi:10.1016/j.jclinepi.2016.07.007
      OpenUrl
    31. 31.↵
      1. Guyatt GH,
      2. Akl EA,
      3. Crowther M, et al
      . Introduction to the ninth edition: Antithrombotic Therapy and Prevention of Thrombosis, 9th ed: American College of Chest Physicians Evidence-Based Clinical Practice Guidelines. Chest 2012;141:48S–52S. doi:10.1378/chest.11-2286
      OpenUrlCrossRefPubMedWeb of Science
    32. 32.↵
      1. Hobbs FDR,
      2. Erhardt L
      . Acceptance of guideline recommendations and perceived implementation of coronary heart disease prevention among primary care physicians in five European countries: the Reassessing European Attitudes about Cardiovascular Treatment (REACT) survey. Fam Pract 2002;19:596–604. doi:10.1093/fampra/19.6.596
      OpenUrlCrossRefPubMedWeb of Science
    33. 33.↵
      1. Lugtenberg M,
      2. Burgers JS,
      3. Besters CF, et al
      . Perceived barriers to guideline adherence: a survey among general practitioners. BMC Fam Pract 2011;12:98. doi:10.1186/1471-2296-12-98
    34. 34.↵
      1. Sharp MK,
      2. Baki DABA,
      3. Quigley J, et al
      . The effectiveness and acceptability of evidence synthesis summary formats for clinical guideline development groups: a mixed-methods systematic review. Implement Sci 2022;17:74. doi:10.1186/s13012-022-01243-2
    35. 35.↵
      1. Siemieniuk RA,
      2. Agoritsas T,
      3. Macdonald H, et al
      . Introduction to BMJ Rapid Recommendations. BMJ 2016;354:i5191. doi:10.1136/bmj.i5191
    36. 36.↵
      1. Agoritsas T,
      2. Darbellay Farhoumand P,
      3. Siemieniuk R, et al
      . The BMJ Rapid Recommendations: towards a new model for the production of clinical practice guidelines. Rev Med Suisse 2019;15:149–55.
      OpenUrl
    37. 37.↵
      1. Kristiansen A,
      2. Brandt L,
      3. Alonso-Coello P, et al
      . Development of a novel, multilayered presentation format for clinical practice guidelines. Chest 2015;147:754–63. doi:10.1378/chest.14-1366
      OpenUrlCrossRefPubMed
    38. 38.↵
      1. Agoritsas T,
      2. Heen AF,
      3. Brandt L, et al
      . Decision aids that really promote shared decision making: the pace quickens. BMJ 2015;350:g7624. doi:10.1136/bmj.g7624
    39. 39.↵
      1. Heen AF,
      2. Vandvik PO,
      3. Brandt L, et al
      . Decision aids linked to evidence summaries and clinical practice guidelines: results from user-testing in clinical encounters. BMC Med Inform Decis Mak 2021;21:202. doi:10.1186/s12911-021-01541-7
      OpenUrlCrossRef
    40. 40.↵
      1. Heen AF,
      2. Vandvik PO,
      3. Brandt L, et al
      . A framework for practical issues was developed to inform shared decision-making tools and clinical guidelines. J Clin Epidemiol 2021;129:104–13. doi:10.1016/j.jclinepi.2020.10.002
      OpenUrlCrossRefPubMed
    41. 41.↵
      1. Spicer JO,
      2. Coleman CG
      . Creating Effective Infographics and Visual Abstracts to Disseminate Research and Facilitate Medical Education on Social Media. Clin Infect Dis 2022;74:e14–22. doi:10.1093/cid/ciac058
      OpenUrl
    42. 42.↵
      1. Thoma B,
      2. Murray H,
      3. Huang SYM, et al
      . The impact of social media promotion with infographics and podcasts on research dissemination and readership. CJEM 2018;20:300–6. doi:10.1017/cem.2017.394
      OpenUrl
    43. 43.↵
      1. Huang S,
      2. Martin LJ,
      3. Yeh CH, et al
      . The effect of an infographic promotion on research dissemination and readership: A randomized controlled trial. CJEM 2018;20:826–33. doi:10.1017/cem.2018.436
      OpenUrl
    44. 44.↵
      1. Crick K,
      2. Hartling L
      . Preferences of Knowledge Users for Two Formats of Summarizing Results from Systematic Reviews: Infographics and Critical Appraisals. PLoS ONE 2015;10:e0140029. doi:10.1371/journal.pone.0140029
      OpenUrlPubMed
    45. 45.↵
      1. Mc Sween-Cadieux E,
      2. Fillol A,
      3. Ridde V, et al
      . Letter to the editor in response to “No difference in knowledge obtained from infographic or plain language summary of a Cochrane systematic review: three randomized controlled trials” by Buljan et al. (2017). J Clin Epidemiol 2018;100:132–3. doi:10.1016/j.jclinepi.2018.02.020
      OpenUrl
    46. 46.↵
      1. Turck CJ,
      2. Silva MA,
      3. Tremblay SR, et al
      . A Preliminary Study of Health Care Professionals’ Preferences for Infographics Versus Conventional Abstracts for Communicating the Results of Clinical Research. J Contin Educ Health Prof 2014;34:S36–8. doi:10.1002/chp.21232
      OpenUrl
    47. 47.↵
      1. Murray IR,
      2. Murray AD,
      3. Wordie SJ, et al
      . Maximising the impact of your work using infographics. Bone Joint Res 2017;6:619–20. doi:10.1302/2046-3758.611.BJR-2017-0313
      OpenUrlFREE Full Text
    48. 48.↵
      1. Gupta A
      . Speaking Concisely and Comprehensively: Importance of an Efficient Communication Model Between the Physician and the Patient. Cureus 2018;10:e3156. doi:10.7759/cureus.3156
    49. 49.↵
      1. Gigerenzer G,
      2. Gaissmaier W,
      3. Kurz-Milcke E, et al
      . Helping Doctors and Patients Make Sense of Health Statistics. Psychol Sci Public Interest 2007;8:53–96. doi:10.1111/j.1539-6053.2008.00033.x
      OpenUrlCrossRefPubMed
    50. 50.↵
      1. Martin LJ,
      2. Turnquist A,
      3. Groot B, et al
      . Exploring the Role of Infographics for Summarizing Medical Literature. Health Professions Education 2019;5:48–57. doi:10.1016/j.hpe.2018.03.005
      OpenUrl
    51. 51.↵
      1. Mayer RE
      . Applying the science of learning to medical education. Med Educ 2010;44:543–9. doi:10.1111/j.1365-2923.2010.03624.x
      OpenUrlCrossRefPubMed
    52. 52.↵
      1. Issa N,
      2. Schuller M,
      3. Santacaterina S, et al
      . Applying multimedia design principles enhances learning in medical education. Med Educ 2011;45:818–26. doi:10.1111/j.1365-2923.2011.03988.x
      OpenUrlCrossRefPubMed
    53. 53.↵
      1. Van Bostraeten P,
      2. Aertgeerts B,
      3. Bekkering GE, et al
      . Infographic summaries for clinical practice guidelines: results from user testing of the BMJ Rapid Recommendations in primary care. BMJ Open 2023;13:e071847. doi:10.1136/bmjopen-2023-071847
    54. 54.↵
      1. Siemieniuk RAC,
      2. Harris IA,
      3. Agoritsas T, et al
      . Arthroscopic surgery for degenerative knee arthritis and meniscal tears: a clinical practice guideline. BMJ 2017;357:j1982. doi:10.1136/bmj.j1982
    55. 55.↵
      1. Bekkering GE,
      2. Agoritsas T,
      3. Lytvyn L, et al
      . Thyroid hormones treatment for subclinical hypothyroidism: a clinical practice guideline. BMJ 2019;365:l2006. doi:10.1136/bmj.l2006
    56. 56.↵
      1. Agarwal A,
      2. Hunt B,
      3. Stegemann M, et al
      . A living WHO guideline on drugs for covid-19. BMJ 2020;370:m3379. doi:10.1136/bmj.m3379
    57. 57.↵
      1. Kuniavsky M
      . Observing the User Experience: A Practitioner’s Guide to User Research. Morgan Kaufmann, 2003.
    58. 58.↵
      User experience design. Semantic Studio; 2004. Available: https://semanticstudios.com/user_experience_design/ [Accessed 30 Jul 2020].
    59. 59.↵
      1. Rosenbaum S
      . Improving the user experience of evidence: a design approach to evidence-informed health care. 1 bflere pag ill. 2011.
    60. 60.↵
      MAGICevidence website. Available: https://www.magicevidence.org/ [Accessed 23 Nov 2023].
    61. 61.↵
      MAGICapp top 10 features – get help, and give us suggestions and feedback. Available: https://help.magicapp.org/knowledgebase/articles/2012114-magicapp-top-10-features [Accessed 23 Jun 2024].
    62. 62.↵
      1. Aertgeerts B,
      2. Agoritsas T,
      3. Siemieniuk RAC, et al
      . Corticosteroids for sore throat: a clinical practice guideline. BMJ 2017;358:j4090. doi:10.1136/bmj.j4090
    63. 63.↵
      1. Andrews JC,
      2. Schünemann HJ,
      3. Oxman AD, et al
      . GRADE guidelines: 15. Going from evidence to recommendation-determinants of a recommendation’s direction and strength. J Clin Epidemiol 2013;66:726–35. doi:10.1016/j.jclinepi.2013.02.003
      OpenUrlCrossRefPubMed
    64. 64.↵
      1. Neumann I,
      2. Santesso N,
      3. Akl EA, et al
      . A guide for health professionals to interpret and use recommendations in guidelines developed with the GRADE approach. J Clin Epidemiol 2016;72:45–55. doi:10.1016/j.jclinepi.2015.11.017
      OpenUrlCrossRefPubMed
    65. 65.↵
      1. Busse JW,
      2. Vankrunkelsven P,
      3. Zeng L, et al
      . Medical cannabis or cannabinoids for chronic pain: a clinical practice guideline. BMJ 2021;374:n2040. doi:10.1136/bmj.n2040
    66. 66.↵
      1. Guyatt G,
      2. Oxman AD,
      3. Akl EA, et al
      . GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol 2011;64:383–94. doi:10.1016/j.jclinepi.2010.04.026
      OpenUrlCrossRefPubMed
    67. 67.↵
      1. Balshem H,
      2. Helfand M,
      3. Schünemann HJ, et al
      . GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol 2011;64:401–6. doi:10.1016/j.jclinepi.2010.07.015
      OpenUrlCrossRefPubMedWeb of Science
    68. 68.↵
      1. Santesso N,
      2. Glenton C,
      3. Dahm P, et al
      . GRADE guidelines 26: informative statements to communicate the findings of systematic reviews of interventions. J Clin Epidemiol 2020;119:126–35. doi:10.1016/j.jclinepi.2019.10.014
      OpenUrlCrossRefPubMed
    69. 69.↵
      1. Hsieh H-F,
      2. Shannon SE
      . Three approaches to qualitative content analysis. Qual Health Res 2005;15:1277–88. doi:10.1177/1049732305276687
      OpenUrlCrossRefPubMedWeb of Science
    70. 70.↵
      1. Vandvik PO,
      2. Otto CM,
      3. Siemieniuk RA, et al
      . Transcatheter or surgical aortic valve replacement for patients with severe, symptomatic, aortic stenosis at low to intermediate surgical risk: a clinical practice guideline. BMJ 2016;354:i5085. doi:10.1136/bmj.i5085
    71. 71.↵
      1. Siemieniuk RAC,
      2. Lytvyn L,
      3. Mah Ming J, et al
      . Antiretroviral therapy in pregnant women living with HIV: a clinical practice guideline. BMJ 2017;358:j3961. doi:10.1136/bmj.j3961
    72. 72.↵
      1. Vermandere M,
      2. Aertgeerts B,
      3. Agoritsas T, et al
      . Antibiotics after incision and drainage for uncomplicated skin abscesses: a clinical practice guideline. BMJ 2018;360:k243. doi:10.1136/bmj.k243
    73. 73.↵
      1. Rochwerg B,
      2. Almenawer SA,
      3. Siemieniuk RAC, et al
      . Atraumatic (pencil-point) versus conventional needles for lumbar puncture: a clinical practice guideline. BMJ 2018;361:k1920. doi:10.1136/bmj.k1920
    74. 74.↵
      1. Kuijpers T,
      2. Spencer FA,
      3. Siemieniuk RAC, et al
      . Patent foramen ovale closure, antiplatelet therapy or anticoagulation therapy alone for management of cryptogenic stroke? A clinical practice guideline. BMJ 2018;362:k2515. doi:10.1136/bmj.k2515
    75. 73.↵
      1. Lamontagne F,
      2. Rochwerg B,
      3. Lytvyn L, et al
      . Corticosteroid therapy for sepsis: a clinical practice guideline. BMJ 2018;362:k3284. doi:10.1136/bmj.k3284
    76. 76.↵
      1. Tikkinen KAO,
      2. Dahm P,
      3. Lytvyn L, et al
      . Prostate cancer screening with prostate-specific antigen (PSA) test: a clinical practice guideline. BMJ 2018;362:k3581. doi:10.1136/bmj.k3581
    77. 77.↵
      1. Siemieniuk RAC,
      2. Chu DK,
      3. Kim LH-Y, et al
      . Oxygen therapy for acutely ill medical patients: a clinical practice guideline. BMJ 2018;363:k4169. doi:10.1136/bmj.k4169
    78. 78.↵
      1. Prasad K,
      2. Siemieniuk R,
      3. Hao Q, et al
      . Dual antiplatelet therapy with aspirin and clopidogrel for acute high risk transient ischaemic attack and minor ischaemic stroke: a clinical practice guideline. BMJ 2018;363:k5130. doi:10.1136/bmj.k5130
    79. 79.↵
      1. Poolman RW,
      2. Agoritsas T,
      3. Siemieniuk RAC, et al
      . Low intensity pulsed ultrasound (LIPUS) for bone healing: a clinical practice guideline. BMJ 2017;356:j576. doi:10.1136/bmj.j576
    80. 80.↵
      1. Ye Z,
      2. Reintam Blaser A,
      3. Lytvyn L, et al
      . Gastrointestinal bleeding prophylaxis for critically ill patients: a clinical practice guideline. BMJ 2020;368:l6722. doi:10.1136/bmj.l6722
    81. 81.↵
      1. Vandvik PO,
      2. Lähdeoja T,
      3. Ardern C, et al
      . Subacromial decompression surgery for adults with shoulder pain: a clinical practice guideline. BMJ 2019;364:l294. doi:10.1136/bmj.l294
    82. 82.↵
      1. Brandt L,
      2. Vandvik PO,
      3. Alonso-Coello P, et al
      . Multilayered and digitally structured presentation formats of trustworthy recommendations: a combined survey and randomised trial. BMJ Open 2017;7:e011569. doi:10.1136/bmjopen-2016-011569
    83. 83.↵
      1. Alexander PE,
      2. Gionfriddo MR,
      3. Li S-A, et al
      . A number of factors explain why WHO guideline developers make strong recommendations inconsistent with GRADE guidance. J Clin Epidemiol 2016;70:111–22. doi:10.1016/j.jclinepi.2015.09.006
      OpenUrl
    84. 84.↵
      1. Alexander PE,
      2. Li S-A,
      3. Gionfriddo MR, et al
      . Senior GRADE methodologists encounter challenges as part of WHO guideline development panels: an inductive content analysis. J Clin Epidemiol 2016;70:123–8. doi:10.1016/j.jclinepi.2015.09.003
      OpenUrl
    85. 85.↵
      1. Akl EA,
      2. Maroun N,
      3. Guyatt G, et al
      . Symbols were superior to numbers for presenting strength of recommendations to health care consumers: a randomized trial. J Clin Epidemiol 2007;60:1298–305. doi:10.1016/j.jclinepi.2007.03.011
      OpenUrlCrossRefPubMed
    86. 86.↵
      1. Santesso N,
      2. Wiercioch W,
      3. Barbara AM, et al
      . Focus groups and interviews with the public led to the development of a template for a GRADE Plain Language Recommendation (PLR). J Clin Epidemiol 2022;141:18–25. doi:10.1016/j.jclinepi.2021.09.018
      OpenUrl
    87. 87.↵
      1. Stiggelbout AM,
      2. Van der Weijden T,
      3. De Wit MPT, et al
      . Shared decision making: really putting patients at the centre of healthcare. BMJ 2012;344. doi:10.1136/bmj.e256
    88. 88.↵
      1. Elwyn G,
      2. Quinlan C,
      3. Mulley A, et al
      . Trustworthy guidelines - excellent; customized care tools - even better. BMC Med 2015;13:199. doi:10.1186/s12916-015-0436-y
    89. 89.↵
      1. Khan SU,
      2. Yedlapati SH,
      3. Lone AN, et al
      . PCSK9 inhibitors and ezetimibe with or without statin therapy for cardiovascular risk reduction: a systematic review and network meta-analysis. BMJ 2022;377:e069116. doi:10.1136/bmj-2021-069116
    90. 90.↵
      1. Siemieniuk RA,
      2. Bartoszko JJ,
      3. Zeraatkar D, et al
      . Drug treatments for covid-19: living systematic review and network meta-analysis. BMJ 2020;370. doi:10.1136/bmj.m2980
    91. 91.↵
      1. Busse JW,
      2. Casassus R,
      3. Carrasco-Labra A, et al
      . Management of chronic pain associated with temporomandibular disorders: a clinical practice guideline. BMJ 2023;383:e076227. doi:10.1136/bmj-2023-076227

    Footnotes

    • Contributors TA was involved in the design of the study and is the guarantor. TH collected the data. TH and TA analysed the data. TH wrote the initial manuscript. PV and TA revised the manuscript. All authors read and approved the final manuscript. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted. The lead author affirms that the manuscript is an honest, accurate and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.

    • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

    • Competing interests TA is the Chair and PV the Chief Research and Innovation Officer of the MAGIC Evidence. Ecoystem Foundation, who coinitiated and is coordinating the BMJ Rapid Recommendations. TA has participated to the design of all infographics. The other authors declare no other competing interest. All authors have completed the ICMJE uniform disclosure form at (59) and declare no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.

    • Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.