Team-based research appraisal activities among allied health in rural and regional health services

Dr Olivia A. King1,2,3
Dr Rosalie A. Boyce1,2,4
  1. University Hospital Geelong
  2. South West Healthcare
  3. Monash Centre for Scholarship in Health Education
  4. The University of Queensland Office of Sponsored Research

Abstract

Rational and aim

Health services are expected to provide evidence-informed care and services. Journal clubs have been the mainstay of evidence appraisal activities for many clinical teams however the translation of findings to changes in clinical practice are less certain. The current study examines the operationalization of evidence appraisal activities by allied health teams in a regional and rural area and their connection to practice change.

Method

A cross-sectional online survey of allied health managers and team leaders across three health services in a regional and rural area in Victoria, Australia. Participants were asked to describe the evidence appraisal activities undertaken within their teams with respect to operational factors such as the approach, forum, frequency and platform. Participants were also asked about their perceptions of the capacity within teams to undertake evidence appraisal, impact of the activities and the importance of different stakeholder groups in the clinical practice change process.

Results

Sixteen allied health managers or team leaders responded to the survey. Almost all teams engaged in some form of regular evidence appraisal activity, either within a unidisciplinary or multidisciplinary format. Features of the activities varied however participants commonly reported the perceived impact of such activities on clinical practice was moderate or low. Participants considered themselves, as managers and their clinicians key to identifying the need for, and leading changes to clinical practice.

Conclusion

Allied health teams regularly engage in evidence-appraisal activities in regional and rural health services. While impact of these activities on clinical practice remains unclear, the findings of this survey study suggest the impact is moderate at best. A region-wide approach to team-based evidence appraisal activities underpinned by research translation framework(s) may improve the impact of these activities on clinical practice.

Keywords

Allied health, evidence-based practice, research translation, journal club

Introduction

Appropriate and timely translation of research evidence into clinical practice is central to health service capacity development strategies in both developed and developing countries, worldwide.1-4 Research translation is a non-linear process by which the best available evidence is adopted, adapted to the local context and implemented into practice with a view to improve service efficiency and effectiveness and ultimately, healthcare outcomes.5, 6 Successful and sustained research translation relies on appropriate adaptation of research evidence to the local context, leadership (and followership), adequate resources, organisational support and engagement with stakeholders beyond the team driving translation.5, 7-9 Research translation is a complex and social process.5, 7, 10
Health services with a vibrant research culture and high levels of clinician research capacity are associated with evidence-informed care, perform more relevant clinical and health service research and increased clinician job satisfaction and retention.4, 11-14 Research capacity encompasses an array of individual skills and competencies which contribute to proficiency in key research activities. For this paper, three key domains to define individual healthcare professionals’ research capacity are identified: 1) research consumption (i.e. the reading and appraising of research evidence); 2) research activity (i.e. actively undertaking research and thereby generating relevant research evidence) and 3) implementing research evidence to change clinical practice in accordance with the best available evidence).4, 14-16 Developing individuals’ skills in these three domains enhances organisational research capacity leading to increased service and system-wide adoption and implementation of evidence-informed healthcare practices.17
Clinicians’ skills in research evidence consumption rely on their ability to identify a clinically-relevant knowledge gap, search for and identify relevant research evidence, read, comprehend and critically appraise the evidence to inform their decision to adopt and apply the findings in clinical practice. A critical finding limiting the appetite of allied health clinicians for research engagement and consumption is a decline in confidence in their evidence-based practice skills, including critical appraisal within five years of commencing clinical practice.18
Evidence consumption activities can be undertaken individually, within a uniprofessional or a multiprofessional setting. Journal clubs have been an institution among health professionals since the late 1800s and have provided a forum for team-based evidence consumption. Touted as a means to support and promote evidence-based practice, little is known about the features of journal clubs that influence practice change.19 Despite the enduring nature of the journal club format, there is no assurance or systematic methodology for ensuring that it translates into clinical practice review and change. Evidence is notably lacking that high-level research consumption skills influence successful and sustained research translation.9, 19
While efforts to enhance research capacity have historically focused on the skills required to competently consume and conduct healthcare research,16 attention has recently turned to the mechanisms by which research knowledge is mobilised or translated into practice and the capabilities required of health services and clinicians to facilitate this.9, 14, 20 In the Australian context, investment in allied health research translation capacity development has been increasing over the last decade and includes the recent establishment of allied health service situated researchers to support research translation in Victoria.14, 21 Allied health represents more than a quarter of the healthcare workforce in Australia22. Professions included in allied health may number more than twenty, depending on the jurisdiction, but typically include disciplines such as speech pathology, dietetics, podiatry, physiotherapy, occupational therapy, social work.23
Recent research has focused on new formats for journal clubs to improve practice change outcomes for allied health. For example, Wenke and colleagues measured the effectiveness of an enhanced journal club format compared with the traditional format, with teams of allied health clinicians. However, the results indicated that further research needed.24 The purpose of the current study is to elucidate how team-based evidence appraisal activities are operationalised by allied health teams in regional and rural health services, to influence and inform research translation. The study serves as a precursor to establishing processes to promote research translation capacity development in health services across the region.
A key contextual feature to note is that infrastructure supporting allied health research in the region is comparatively underdeveloped with respect to major hospitals in metropolitan areas and other regions. Anecdotal evidence collected from allied health clinicians and managers within the region identified two areas of perceived weakness:
  1. Structures and processes governing team-based evidence review and appraisal were inconsistent
  2. review and appraisal activities did not consistently inform systems-level clinical practice change.
Managers and team leaders expressed a desire to ensure clinical practice was aligned with contemporary, evidence-based guidelines and make efficient use of the time allocated for clinical appraisal activities.

Methods

To gain an understanding of the processes and systems in place, as well as perceptions of the role of such processes in informing practice change, a survey of allied health managers and leaders of teams with allied health clinicians was undertaken. The use of survey methodology provided anonymity for participants to disclose perceived inadequacies in team-based evidence appraisal activities and processes.
The survey study was conducted across three health services and between May and August 2019. The decision to restrict the survey to manager and team leader levels was to capture formal and organisationally approved evidence appraisal activities. Furthermore, the study was seeking to assess the manager and team leader satisfaction with the translational output from appraisal activities rather than those of frontline staff. The survey (available on request) was sent to 27 allied health managers or team leaders. Ethics approval was obtained from the region’s largest health service research office. Responses to close-ended questions the and free text responses were analysed and summarized descriptively.25

Results

Sixteen allied health team or program managers responded to the survey. All participants are allied health discipline managers or managers of programs or teams of which allied health clinicians are members. The survey asked questions which relate to team characteristics and activities. One question asked participants about their perceptions of the impact of clinical evidence appraisal activities undertaken within and by their team. Due to concerns about the identifiability of participants and teams, very few demographic questions were included.
Participant and team characteristics
Sixteen allied health team or program managers responded to the survey. The size of the teams represented varied, with five participants representing teams of between 10 and 20 clinicians and another five representing teams of more than 30 clinicians. Four participants represented a team with less than 10 clinicians and two had between 20 and 30 clinicians. Therefore, the respondents represent at least 280 allied health clinicians.
Nine of the participants described their team as being unidisciplinary (which may include allied health assistants) and seven participants’ teams were multidisciplinary. More than half (n = 9) of the participants had completed their most recent qualification over 10 years ago, four had completed a qualification within the last five years, two within the last six to 10 years and one participant was currently undertaking a further qualification.
Types of clinical evidence appraisal activities
Participants were asked to select the clinical evidence appraisal activities that their teams engage in, from a pre-defined list. They were given the opportunity to select “other” and describe the activity. Participants were able to select all applicable activities and among the eight types of activities, there were 52 selections. Two participants indicated that they were not aware of any organized or regular evidence appraisal activities in their teams. Table 1 shows the most commonly employed approaches to team-based clinical evidence appraisal.

Table 1 Clinical evidence appraisal activities according to team type

One participant indicated that they were not aware of any organized or regular activities, one indicated that individual team members engaged in activities with teams outside of theirs (only) and one indicated that they (manager) engaged in evidence appraisal privately or as part of formal study. Of the participants whose teams do engage in team-based evidence appraisal activities (n = 13), all engage in multiple types of activities.
Forums and media for evidence appraisal activities
Participants were asked to select from a list, the forums within which clinical evidence appraisal occurs. Participants were also asked to select the medium utilized. For both questions, participants could select all that apply and had the option to select and describe “other”. Tables 2 and 3 present the types of forums and media according to team sizes. Most teams reviewed and appraised clinical evidence within regular team meetings or during meetings which are scheduled in response to the identification of clinical or service issues. See Table 2.

Table 2 Team size and forums for conducting clinical evidence appraisal activities

The most commonly utilized medium is face-to-face meetings (n = 14), followed by email (n = 6) and video- or teleconference (n = 4). See Table 3.

Table 3 Team size and media utilised to conduct team-based clinical evidence appraisal activities

Frequency of evidence appraisal activities and team member attendance
Participants were asked to indicate how frequently team-based clinical evidence appraisal activities are undertaken (single answer option) and typical team-member attendance. More than one third of the teams represented meet quarterly (n=6), a quarter met monthly (n=4) and the remaining teams represented meet every two months (n=2) or every six months (n=1). Two respondents selected not-applicable and one did not answer.
Half of the participants (n=8) indicated that typically most team members attend clinical evidence appraisal activities, a quarter (n=4) indicated that some of the team members attend regularly and the remaining responded that all attend regularly (n=2) or not applicable (n=2).
Perceived levels of expertise within the teams and impact of evidence appraisal activities
Participants were asked to indicate their perception of the level of capacity or expertise within their teams, with regard to clinical evidence appraisal. The majority of participants indicated that they had adequate (n = 7, 44%) or very good (n = 6, 38%) capacity or expertise within their teams to undertake clinical evidence appraisal activities. There were no participants that indicated they had excellent capacity and few indicated they had inadequate capacity (n = 2, 13%) or very poor capacity (n = 1, 6%).
Participants were also asked to indicate their perceived impact of the clinical evidence appraisal, where the following definitions, for the purpose of the current study, were applied:
The majority of participants indicated that their activities were perceived to lead to moderate impact (n = 8, 50%), fewer perceived high impact (n = 4, 25%), low impact (n = 3, 19%) and for one participant, this question was not applicable.
On interrogating the data, there did not appear to be any patterns in the type or size of the team and the perceived impact of evidence appraisal activities. This is likely due to the small number of responses. Interestingly, there did not appear to be any link between teams with higher levels of evidence appraisal expertise and a higher impact of their activities.
Descriptions of evidence appraisal activities and how these inform changes to clinical practice
Fifteen participants responded to two open-ended questions which asked them to (1) describe the clinical evidence appraisal activities which take place within their teams and (2) how these activities inform decisions and actions to change clinical practice.
In response to the first question, several participants described having a structured process in place, which is led by a designated person (either the manager / team leader or a senior clinician with an interest in research or a clinical area). Others described more ad hoc approaches to reviewing evidence, based on current and emerging trends and issues.
Responses to the second question were also varied. Several participants did however, speak about establishing small working groups to effect clinical practice changes in response to evidence and a perceived need to address clinical or service issues.
Who is best placed to identify the need for clinical practice change and who is best placed to lead the endeavour?
Participants were asked to rank the importance of the list of stakeholders / factors below from a pre-determined list, with respect to their role in (1) identifying the need for clinical practice change and (2) in leading the implementation of changes. Consumers were not listed as a stakeholder group, as it is implied that through consumer stories, experiences and adverse events involving consumers, the need for changes to clinical practice will be highlighted by one of the above-listed stakeholders. For example, receiving and acting on consumer feedback is incorporated in the work of the Safety and Quality team. Further, the purpose of the study was to examine the appraisal activities and their impact of the frequency of change to practice, rather than to understand how consumers are involved in identifying the need for appraisal activities.
Participants were asked to use a ranking of 1 to indicate which stakeholder group was considered to be most important in identifying or leading the need for changes to clinical practice. Those considered least important were ranked 10. Therefore, the stakeholder group or factor with the lowest sum were considered most important.
Table 4 Stakeholder group or factor most important in identifying the need for changes to clinical practice
Table 4 shows, the participants ranked managers or team leaders, and clinicians as the most important stakeholder group in identifying the need for changes to clinical practice. Policy directives and important Safety and quality teams were also considered very important.
Similarly, Table 5 shows that participants ranked managers or team leaders and clinicians as the most important groups in leading changes to clinical practice. Senior managers, project workers and safety and quality teams were also considered important.

Table 5 Stakeholder group most important in leading changes to clinical practice

Discussion
The respondents represented 16 teams of least 280 clinicians from three health services across the Region. This is approximately 40% of the allied health workforce in the employing institutions Allied health teams, both unidisciplinary and multidisciplinary, take different approaches to appraising clinical evidence within their teams and there are no clear patterns according to team composition. There is variety with respect to the types of activities, forums and media used to appraise the clinical evidence, the frequency at which these activities are undertaken, typical attendance, perceived expertise within the teams and impact of such activities in terms of changes to clinical practice.
Most of the participants who reported having team-based processes to review and appraise clinical evidence in place, frequently described themselves (i.e. as the manager or team leader) to be the leader of such processes. Of the participants that reported having a team-based approach to discussing and planning changes to clinical practice in place, the establishment of small working groups was commonly described. The size or disciplinary profile of the teams does not appear to influence the types of activities undertaken or any of the other aforementioned factors.
The majority of participants (n = 12, 75%) perceive the impact of their various evidence appraisal activities to be either low or moderate; that is, these activities seldom lead to positive changes to clinical practice. That is, regardless of perceived capacity, engagement and expertise, team-based research consumption rarely leads to changes in clinical practice. This resonates with Hitch and colleagues’ (2019) observations that knowledge and research consumption skills are rarely the key predictors of clinicians’ or teams’ ability to translate research into practice. This highlights the need for a different approach to promoting evidence-informed clinical practice.
On ranking the importance of the different groups and factors in terms of identifying the need for change to clinical practice and then again in leading changes to clinical practice, participants pointed to managers / team leaders (i.e. themselves) and the clinicians within their teams. This adds weight to the argument for ongoing investment in research translation support and resources in health services.4, 26-28 Moreover, that managers themselves were identified as central to the clinical practice change process reflects the findings of previous research on the factors influencing middle managers’ commitment to driving clinical practice innovation.29 The authors found that the two most important factors were managers’ perceptions of the ease of implementation and of the likely benefits on patient outcomes.
Clinicians and managers are aptly placed to identify the contextually relevant enabling and constraining factors influencing research translation endeavors.8, 9 Indeed, the results of the current study indicate that allied health managers and team leaders consider themselves and the clinicians in their teams aptly placed to identify the need for practice change and to lead the practice change process. However, support to identify relevant, high quality guidelines and evidence appropriate for clinical teams and areas is enhanced by the presence of, or ease of access to, resources including experienced researchers as knowledge brokers or facilitators.24, 30
While the evidence supporting the impact of traditional team-based evidence appraisal activities on clinical practice remains weak, it is clear that for allied health, journal clubs as a concept represent the mainstay of research and quality improvement activities. They are conducted regularly: quarterly in more than one third of cases, generally well-attended: most team members attending regularly in half of the teams represented and using a number of different platforms. Accordingly, efforts to develop research translation capacity within allied health clinicians, teams and health services more broadly, can leverage off the enduring commitment to and engagement with, these sort of activities. The focus however, can be shifted from research consumption (review, appraisal and judgment of the worthiness of published research), to a process guided by one of the numerous research translation frameworks that have proven effective in identifying and address the various factors impeding the successful and sustained translation of research into clinical practice, on a systems level.31, 32
There are several implications for practice emanating from the current study. The traditional journal club can be revitalized, refocused and rebranded as a research translation group or contemporary clinical practice club. Membership, frequency of meetings and platforms should remain as they are so as to ensure the current level of engagement, however the focus should be placed on translation discussions with the support and guidance of a suitably qualified clinician-researcher or a skilled translation lead.

Limitations and future direction

This study has several limitations. The findings are based on a small sample of 16 allied health managers and team leaders. Further, the survey tool utilized is not validated. However, it is important to note that it was designed to enable the researchers to attain an accurate illustration of the various team-based research consumption activities across the region to form the basis of research capacity development activities. The definitions related to impact (i.e. high, moderate, low), were employed for the purpose of the survey and it was beyond the scope of the study to measure or make claims about impact, or to generate a definition of impact with respect to research appraisal activities.
The findings of the survey study nonetheless, provide a baseline against which subsequent research translation capacity building strategies and efforts in Rural and Regional areas, can be measured. The study also sets the scene for investing in resources to support the identification of key areas of focus for research and the opportunity to take a concerted, region-wide approach to research translation.

Conclusion

Team-based evidence appraisal activities continue to occur regularly in most allied health teams. The size, composition of the teams, forums, platforms and frequency of activities varies across teams. Attendance is reportedly strong and perceived capacity to undertake evidence appraisal is adequate. Notwithstanding, perceived impact on changes to clinical practice is consistently low. Managers are aptly placed to identify and lead innovation in healthcare practices. With supportive infrastructure including evidence-based frameworks, there are clearly opportunities for research translation activities and capacity development strategies in allied health to leverage off existing and well-subscribed to forums, and to introduce a systematic and regional approach to identifying the need for, and leading innovative, evidence-informed clinical practice change.

Acknowledgements

We would like to thank the Directors of Allied Health from the participating health services for supporting the study. We particularly thank Kait Brown for her feedback on an earlier version of the paper. We also acknowledge and thank the participants.

References

1. Pickstone C, Nancarrow S, Cooke J, et al. Building research capacity in the allied health professions.Evidence & Policy . 2008;4(1):53-68. doi:10.1332/174426408783477864
2. Ekeroma AJ, Kenealy T, Shulruf B, McCowan LM, Hill A. Building reproductive health research and audit capacity and activity in the pacific islands (BRRACAP) study: methods, rationale and baseline results. BMC Med Educ . 2014;14(1):121. doi:10.1186/1472-6920-14-121
3. Gill SD, Gwini SM, Otmar R, Lane SE, Quirk F, Fuscaldo G. Assessing research capacity in Victoria’s south‐west health service providers. Aust J Rural Health . 2019;27(6):505-513. doi:10.1111/ajr.12558
4. Hitch D, Lhuede K, Vernon L, Pepin G, Stagnitti K. Longitudinal evaluation of a knowledge translation role in occupational therapy. BMC Health Serv Res . 2019;19(1):154. doi:10.1186/s12913-019-3971-y
5. Braithwaite J, Churruca K, Long JC, Ellis LA, Herkes J. When complexity science meets implementation science: a theoretical and empirical analysis of systems change.BMC Med . 2018;16(1):63. doi:10.1186/s12916-018-1057-z
6. May CR, Johnson M, Finch T. Implementation, context and complexity. Implementation Science . 2016;11(1):141. doi:10.1186/s13012-016-0506-3
7. Greenhalgh T, Wieringa S. Is it time to drop the ‘knowledge translation’metaphor? A critical literature review. J R Soc Med . 2011;104(12):501-509. doi:10.1258/jrsm.2011.110285
8. Greenhalgh T, Papoutsi C. Spreading and scaling up innovation and improvement. BMJ . 2019;365:l2068. doi:10.1136/bmj.l2068
9. Hitch D, Pepin G, Lhuede K, Rowan S, Giles S. Development of the translating allied health knowledge (TAHK) framework. Int J Health Policy Manag . 2019;8(7):412. doi:10.15171/ijhpm.2019.23
10. May CR, Johnson M, Finch T. Implementation, context and complexity. Implement Sci . 2016;11(1):141. doi:10.1186/s13012-016-0506-3
11. Matus J, Walker A, Mickan S. Research capacity building frameworks for allied health professionals–a systematic review. BMC Health Serv Res . 2018;18(1):716. doi:10.1186/s12913-018-3518-7
12. Williams C, Miyazaki K, Borkowski D, McKinstry C, Cotchet M, Haines T. Research capacity and culture of the Victorian public health allied health workforce is influenced by key research support staff and location. Aust Health Rev . 2015;39(3):303-311. doi:10.1071/AH14209
13. Jonker L, Fisher SJ, Dagnan D. Patients admitted to more research‐active hospitals have more confidence in staff and are better informed about their condition and medication: Results from a retrospective cross‐sectional study. J Eval Clin Pract . 2020;26(1):203-208. doi:10.1111/jep.13118
14. Department of Health and Human Services. Victorian allied health research framework. Accessed 06/09/2020, https://www2.health.vic.gov.au/health-workforce/allied-health-workforce/allied-health-research
15. Slade SC, Philip K, Morris ME. Frameworks for embedding a research culture in allied health practice: a rapid review. Health Res Policy Syst . 2018;16(1):29. doi:10.1186/s12961-018-0304-2
16. Kislov R, Waterman H, Harvey G, Boaden R. Rethinking capacity building for knowledge mobilisation: developing multilevel capabilities in healthcare organisations.Implement Sci . 2014;9(1):166. doi:10.1186/s13012-014-0166-0
17. Harding K, Lynch L, Porter J, Taylor NF. Organisational benefits of a strong research culture in a health service: a systematic review. Aust Health Rev . 2017;41(1):45-53. doi:10.1071/AH15180
18. Klaic M, McDermott F, Haines T. How soon do allied health professionals lose confidence to perform EBP activities? A cross‐sectional study. J Eval Clin Pract . 2019;25(4):603-612. doi:10.1111/jep.13001
19. Wilson M, Ice S, Nakashima CY, et al. Striving for evidence-based practice innovations through a hybrid model journal club: A pilot study. Nurse Educ Today . May 2015;35(5):657-62. doi:10.1016/j.nedt.2015.01.026
20. Wenke RJ, Tynan A, Scott A, Mickan S. Effects and mechanisms of an allied health research position in a Queensland regional and rural health service: a descriptive case study. Aust Health Rev . 2018;42(6):667-675. doi:10.1071/AH17086
21. Hulcombe J, Sturgess J, Souvlis T, Fitzgerald C. An approach to building research capacity for health practitioners in a public health environment: an organisational perspective. Aust Health Rev . 2014;38(3):252-258. doi:10.1071/AH13066
22. Allied Health Professions Australia. What is allied health? Allied Health Professions Australia. Accessed 06/09/2020, https://ahpa.com.au/what-is-allied-health/
23. Nancarrow SA, Young G, O’Callaghan K, Jenkins M, Philip K, Barlow K. Shape of allied health: an environmental scan of 27 allied health professions in Victoria.Aust Health Rev . 2017;41(3):327-335. doi:10.1071/AH16026
24. Wenke RJ, Thomas R, Hughes I, Mickan S. The effectiveness and feasibility of TREAT (tailoring research evidence and theory) journal clubs in allied health: a randomised controlled trial. BMC Med Educ . 2018;18(1):104. doi:10.1186/s12909-018-1198-y
25. Cooksey RW. Descriptive Statistics for Summarising Data. Illustrating Statistical Procedures: Finding Meaning in Quantitative Data . Springer; 2020:61-139.
26. Wenke R, Mickan S. The role and impact of research positions within health care settings in allied health: a systematic review. BMC Health Serv Res . 2016;16(1):355. doi:10.1186/s12913-016-1606-0
27. Moran A, Haines H, Raschke N, et al. Mind the gap: is it time to invest in embedded researchers in regional, rural and remote health services to address health outcome discrepancies for those living in rural, remote and regional areas?Aust J Prim Health . 2019;25(2):104-107. doi:10.1071/PY18201
28. Cooke J, Gardois P, Booth A. Uncovering the mechanisms of research capacity development in health and social care: a realist synthesis. Health Res Policy Syst . 2018;16(1):93. doi:10.1186/s12961-018-0363-4
29. Urquhart R, Kendell C, Folkes A, Reiman T, Grunfeld E, Porter G. Factors influencing middle managers’ commitment to the implementation of innovations in cancer care. J Health Serv Res Policy . 2019;24(2):91-99. doi:10.1177/1355819618804842
30. Coates D, Mickan S. The embedded researcher model in Australian healthcare settings: comparison by degree of “embeddedness”. Transl Res . 2020;218:29-42. doi:10.1016/j.trsl.2019.10.005
31. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci . 2009;4(1):1-15. doi:10.1186/1748-5908-4-50
32. Keith RE, Crosson JC, O’Malley AS, Cromp D, Taylor EF. Using the Consolidated Framework for Implementation Research (CFIR) to produce actionable findings: a rapid-cycle evaluation approach to improving implementation.Implement Sci . 2017;12(1):15. doi:10.1186/s13012-017-0550-7