Submitted: 30 June 2022
Accepted: 31 October 2022
Published online: 4 July, TAPS 2023, 8(3), 26-34
https://doi.org/10.29060/TAPS.2023-8-3/OA2834

Noorjahan Haneem Md Hashim¹, Shairil Rahayu Ruslan¹, Ina Ismiarti Shariffuddin¹, Woon Lai Lim¹, Christina Phoay Lay Tan² & Vinod Pallath³

¹Department of Anaesthesiology, Faculty of Medicine, Universiti Malaya, Malaysia; ²Department of Primary Care Medicine, Faculty of Medicine, Universiti Malaya, Malaysia; ³Medical Education Research & Development Unit, Dean’s Office, Faculty of Medicine, Universiti Malaya, Malaysia

Abstract

Introduction: Examiner training is essential to ensure the trustworthiness of the examination process and results. The Anaesthesiology examiners’ training programme to standardise examination techniques and standards across seniority, subspecialty, and institutions was developed using McLean’s adaptation of Kern’s framework.

Methods: The programme was delivered through an online platform due to pandemic constraints. Key focus areas were Performance Dimension Training (PDT), Form-of-Reference Training (FORT) and factors affecting validity. Training methods included interactive lectures, facilitated discussions and experiential learning sessions using the rubrics created for the viva examination. The programme effectiveness was measured using the Kirkpatrick model for programme evaluation.

Results: Seven out of eleven participants rated the programme content as useful and relevant. Four participants showed improvement in the post-test, when compared to the pre-test. Five participants reported behavioural changes during the examination, either during the preparation or conduct of the examination.  Factors that contributed to this intervention’s effectiveness were identified through the MOAC (motivation, opportunities, abilities, and communality) model.

Conclusion: Though not all examiners attended the training session, all were committed to a fairer and transparent examination and motivated to ensure ease of the process. The success of any faculty development programme must be defined and the factors affecting it must be identified to ensure engagement and sustainability of the programme.

Keywords:           Medical Education, Health Profession Education, Examiner Training, Faculty Development, Assessment, MOAC Model, Programme Evaluation

Practice Highlights

• A faculty development initiative must be tailored to faculty’s learning needs and context.
• A simple framework of planning, implementing, and evaluating can be used to design a programme.
• Target outcome measures and evaluation plans must be included in the planning process.
• The Kirkpatrick model is a useful tool to use in programme evaluation: to answer if the programme has met its objectives.
• The MOAC model is a useful tool to explain why a programme has met its objective.

I. INTRODUCTION

Anaesthesiology specialist training in Malaysia comprises a 4-year clinical master’s programme. At the time of our workshop, five local public universities offer the programme. The course content is similar in all universities, but the course delivery may differ to align with each university’s rules and regulations. The summative examinations are held as a Conjoint Examination. Examiners include lecturers from all five universities, specialists from the Ministry of Health and external examiners from international Anaesthesiology training programmes. The examination consists of a written and a viva voce examination. The areas examined are the knowledge and cognitive skills in patient management.

A speciality training programme’s exit level assessment is an essential milestone for licensing. In our programme, the exit examination occurs at the end of the training before trainees practise independently in the healthcare system and are eligible for national specialist registration. Therefore, aligning the curriculum and assessment to licensing requirements is necessary.

Examiners play an important role during this high-stakes summative examination, making decisions regarding allowing graduating trainees to work as specialists in the community. Therefore, examiners must understand their role. In recent years, the anaesthesiology training programme providers in Malaysia have been taking measures to improve the validity of the examination. These include a stringent vetting process to ensure examination content reflects the syllabus, questions are unambiguous, and the examiners agree on the criteria for passing. However, previous examinations revealed that although examiners were clear on the aim of the examination, some utilised different assessment approaches, which were possibly coloured by personal and professional experiences, and thus needed constant calibration on the passing criteria. In addition, during examiner discussions, different examiners were found to have different skill levels in constructing focused higher-order questions and were not fully aware of potential cognitive biases that may affect the examination results.

These insights from previous examinations warranted a specific skill training session to ensure the trustworthiness of the examination process and results (Blew et al., 2010; Iqbal et al., 2010; Juul et al., 2019, Chapter 8, pp. 127-140; McLean et al., 2008). The examiners and the Specialty committee were keen to ensure that these issues were addressed with a training programme that complements the current on-the-job examiner training.

II. METHODS

An examiner training module was developed using McLean’s adaptation of Kern’s framework for curriculum development: Planning, Implementation and Evaluation (McLean et al., 2008; Thomas et al., 2015). A conceptual framework for the examiner training programme was drawn up from the programme’s conception stage to the evaluation of its outcome, as illustrated in Figure 1 (Steinert et al., 2016).

Figure 1: The conceptual framework for the examiner training programme and evaluation of its effectiveness

A. Planning

Three key focus areas were identified for the training programme: (1) Performance Dimension Training (PDT); (2) examiner calibration with Frame-Of-Reference Training (FORT); as well as (3) identifying factors affecting the validity of results and measures that can be taken to prevent them.

1) Performance dimension training (Feldman et al., 2012): The aim was to improve examination validity by reducing examiner errors or biases unrelated to the examinees’ targeted performance behaviours. Finalised marking schemes outlining competencies to be assessed required agreement by all the examiners ahead of time. These needed to be clearly defined and easily understood by all the examiners, and consistency was key to reducing examiner bias.

2) Examiner calibration with Frame-of-Reference Training (FORT) (Newman et al., 2016): Differing levels of experience among all the participants meant that there were differing expectations and performances among them. The examiner training programme needed to assist examiners in resetting expectations and criteria for assessing the candidates’ competencies. This examiner calibration was achieved using pre-recorded simulated viva sessions in which the participants rated candidates’ performances in each simulated viva session and received immediate feedback on their ability and criteria for scoring the candidates.

3) Identifying factors affecting the validity of results (Lineberry, 2019): Factors that may affect the validity of examination results may be related to construct underrepresentation (CU), where the results only reflect one part of an attribute being examined; or construct-irrelevant variance (CIV), where the results are being affected by areas or issues other than the attribute being examined.

An example of CU is sampling issues where only a limited area of the syllabus is examined, or an answer key is limited by the availability of evidence or content expertise.

Examples of CIV include the different ways a concept can be interpreted in different cultures or training centres, ambiguous questions, examiner cognitive biases, examiner fatigue, examinee language abilities, and examinees guessing or cheating. The examiner training programme was designed with the objectives listed in Table 1.

B. Implementation

The faculty intervention programme was designed as a one-day online programme to be attended by potential examiners for the Anaesthesiology Exit Examination. The programme objectives were prioritised from the needs assessment and designed based on Tekian & Norcini’s recommendations (Tekian & Norcini, 2016). Due to time constraints, training was performed using an online platform closer to the examination dates after obtaining university clearance on confidentiality regarding assessment issues.

The structure and contents of the examiner training programme are outlined in Table 2 and is further elaborated in Appendix A.

Based on the objectives, the organisers invited a multidisciplinary group of facilitators. The group consisted of anaesthesiologists, medical education experts in assessment and faculty development, and a technical and logistics support team to ensure efficient delivery of the online programme.

A multimodal approach to delivery was adopted to accommodate the diversity of the examiner group (gender, seniority, subspeciality, and examination experience). Explicit ground rules were agreed upon to underpin the safe and respectful learning environment. The educational strategy included interactive lectures, hands-on practice using rubrics created and calibration using video-assisted scenarios. The programme objectives were embedded and reinforced with each strategy. Pre- and post-tests were performed to help participants gauge their learning and assist the programme organisers in evaluating the participants’ learning.

This would be the first time such a programme was held within the local setting. Participants were all anaesthesiologists by profession, were actively involved in clinical duties within a tertiary hospital setting and consented to participate in this programme. As potential examiners, they all had prior experience as observers of the examination process, with the majority having previous experience as examiners as well.

The programme was organised during the peak of the COVID-19 pandemic and was managed on a fully online platform to ensure safety and minimise the time taken away from clinical duties. In addition, participants received protected time for this programme, a necessary luxury as anaesthesiologists were at the forefront of managing the pandemic.

C. Evaluation

The Kirkpatrick model (McLean et al., 2008; Newstrom, 1995;) was used to evaluate the programme’s effectiveness described and elaborated in Figure 2.

Figure 2: The Kirkpatrick model, elaborated for this programme

The MOAC model (Vollenbroek, 2019), expanded from the MOA (Marin-Garcia & Martinez Tomas, 2016) model by Blumberg & Pringle (Blumberg & Pringle, 1982) was used to examine factors that contributed to the effectiveness of the programme. Motivation, opportunity, ability, and communality are factors that drives action and performance.

III. RESULTS

Eleven participants attended the programme. These participants were examiners for the 2021 examinations from the university training centres and the Ministry of Health, Malaysia. Only one of the participants would be a first-time examiner in the Exit Examination. Four of the would-be examiners could not attend due to service priorities.

A. Level 1: Reaction

Seven of the eleven participants completed the programme evaluation form, which is openly available in Figshare at https://doi.org/10.6084/m9.figshare.20189309.v1 (Tan & Pallath, 2022). All of them rated the programme content as useful and relevant to their examination duties and stated that the content and presentations were pitched at the correct level, with appropriate visual aids and reference materials. The online learning opportunity was also rated as good.

All seven also aimed to make behavioural changes after attending the programme, as indicated below. Some of the excerpts include:

“I am more cognizant of the candidates’ understanding to questions and marking schemes”

“Yes. We definitely need the rubric/marking scheme for standardisation. Will also try to reduce all the possible biases as mentioned in the programme.”

“Yes, as I will be more agreeable to question standardisation in viva examination because it makes it fairer for the candidates.”

The participants also shared their understanding of the importance of standardisation and examiner training and would recommend this programme to be conducted annually. They agreed that the examiner training programme should be made mandatory for all new examiners, with the option of refresher courses for veteran examiners if appropriate.

B. Level 2: Learning

All 11 participants completed the pre-and post-tests. The data supporting these findings of this is openly available in Figshare at https://doi.org/10.6084/m9.figshare.20186582.v1 (Md Hashim, 2021). The participants’ marks in both tests are shown in Appendix B. The areas that showed improvement in scores were identifying why under-sampling is a problem and methods to prevent validity threats. Understanding the source of validity threat from cognitive biases showed a decline in scores (question 2 with scores of 11 to 8 and question 3 with scores of 10 to 8), respectively.

Comparing the post-test scores to pre-test scores, four participants showed improvement, four showed no change (one of the participants answered all questions correctly in both tests) and three participants showed a decline in test scores.

C. Level 3: Behavioural Change

Six participants responded to the follow-up questionnaire, which is openly available in Figshare at https://doi.org/10.6084/m9.figshare.20186591.v2 (Md Hashim, 2022). This questionnaire was administered about a year after the examiner training programme and after the completion of two examinations. Only one respondent did not make any self-perceived behavioural change while preparing the examination questions and conducting the viva examinations. Two respondents did not make any changes while marking or rating candidates.

The specific changes in the three areas of behavioural change that were consciously noted by the respondents were explored. Respondents reported increased awareness and being more systematic in question preparation, making questions more aligned to the curriculum, preparing better quality questions, and being more cognizant of candidates’ understanding of the questions.

They also reported being more objective and guided during marking and rating as the passing criteria were better defined and structured.

Regarding the conduct of the viva examination, respondents shared that they were better prepared during vetting and felt it was easier to rate candidates as the marking schemes and questions were standardised and could ensure candidates could answer all the required questions to pass.

D. Level 4: Results

The examiners who attended the training programme were able to prepare questions as blueprinted and were able to identify areas to be examined and provided recommended criteria for passing each question. This has led to a smooth vetting process and examination.

E. Factors Affecting Effectiveness

Even though the programme was not attended by all the potential examiners, those who did were committed to the idea of a fairer and more transparent examination process. This formed the motivation aspect of the model.

In terms of opportunity, protected training time is important, followed by prioritising the content of the training material according to the most pressing needs.

The ability aspect encompassed the abilities of the facilitators and participants. To emphasise the learning process, credible trainers were invited to this programme to facilitate the lectures and experiential learning sessions. In this aspect, the Faculty Development team comprised an experienced clinician, a basic medical scientist, and an anaesthesiologist, all with medical education qualifications and were vital in ensuring the success of this programme. The whole team was led by the Chief Examiner who focused on the dimensions to be tested and calibrated, while simultaneously managing the expectations of the examiners and their abilities to give and accept feedback. Communication and the skill to be receptive to the proposed changes were also crucial to make the intervention work.

In terms of communality, all the participants were of similar professional backgrounds and shared the common realisation that this training programme was essential and would only yield positive results. Hence this ensured the programme’s overall success.

IV. DISCUSSION

The progressive change seen in this attempt to improve the examination system is aligned with the general progress in medical education. Training of examiners is important (Holmboe et al., 2011), as it is not the tool used for assessment, but rather the person using the tool, that makes the difference. As it is difficult to design the ‘perfect tool’ for performance tests and redesigning a tool only changes 10% of the variance in rating (Holmboe et al., 2011; Williams et al., 2003), educators must now train the faculty in observation and assessment. It is not irrational to extrapolate this effect on written and oral examinations. Holmboe et al. (2011) also share the reasons for a training programme for assessors, which are changing curriculum structure, content and delivery and emerging evidence regarding assessment, building a system reserve, utilising training programmes as opportunities to identify and engage change agents and allow the faculty to form a mental picture of how changes will affect them and improve practice. Enlisting the help and support of a respected faculty member during training will promote the depth and breadth of change.

Khera et al. (2005) described their paediatric examination experiences, in which the Royal College of Paediatrics and Child Health defined examiners’ competencies, selection process and training programme components. The training programme included principles of assessment, examination design, writing questions, interpersonal skills, professional attributes, managing diversity, and assessing the examiners’ skills. They believe these contents will ensure the assessment is valid, reliable, and fair. As Anaesthesiology examiners have different knowledge levels and experiences, it had been crucial to assess their learning needs and provide them with appropriate learning opportunities.

In the emergency brought on by the COVID-19 pandemic, online training was the safest and most feasible platform for conducting this programme. Online faculty development activities have the perceived advantages of being convenient, flexible, and allowing interdisciplinary interaction and providing an experience of being an online student(Cook & Steinert, 2013). Forming the facilitation team together with the dedicated technical and logistics team and creating a chat group prior to conducting the programme were key in anticipating and handling communication and technical issues (Cook & Steinert, 2013).

Though participants were engaged and the results of the workshop were encouraging, the programme delivery and the content will be reviewed based on the feedback received. The convenience of an online activity must be balanced with the participant engagement and facilitator presence of a face-to-face-activity. Since the results of both methods of delivery differs (Arias et al., 2018; Daniel, 2014; Kemp & Grieve, 2014), the best solution may to ask the participants what would best work for them, as they are adult learners and experienced examiners. The programme must be designed with participants involvement, with opportunities to participate and engaging facilitators and support teams that would be able to support the participants’ learning need (Singh et al., 2022).

At the end of the programme, the effectiveness of the programme was measured by referencing the Kirkpatrick model. The Kirkpatrick model (Newstrom, 1995; Steinert et al., 2006) was the most helpful in helping us identify the success of the intervention, which included behavioural change. Measuring behavioural change and impact on the examination results, organisational changes and changes in student learning may be difficult and may not be directly caused by a single intervention (McLean et al., 2008). The key, is perhaps to involve examiners, students and other stakeholders in the evaluation process, using various validated tools, and to ensure that the effort is ongoing, with sustained support, guidance and feedback (McLean et al., 2008).

To explain the overall effectiveness of the programme (with regards to reaction, learning and behavioural change), the MOAC model (Vollenbroek, 2019) expanded from the original MOA model was used. The MOAC model not only describes factors that affect an individual’s performance in a group, but also the group behaviour.

Motivation is an important driving force of action, and members are more motivated when a subject becomes relevant on a personal level, leading to action. The motivation to be informed and to improve has led to active participation in the knowledge sharing session, processing new information presented in the programme and adopting changes learnt during the programme. Presence of a group of motivated individuals with the same goals supported each other’s learning.

Opportunity, especially time, space and resources, must be allocated to reflect the value and relevance of any activity. Work autonomy, allows professionals to engage in what they consider relevant or important, and be accountable for their work outcomes. Facilitating conditions, for example, technology, facilitators, and a platform to practise what is being learnt are also important aspects of opportunity. Allowing protected time with the appropriate facilitating conditions, indicates institutional support and has enabled participants to fully optimise the learning experience.

Ability positively affects knowledge exchange and willingness to participate. Having prior knowledge improves a participant’s ability to absorb and utilise new knowledge. The programme participants, being experienced clinical teachers and examiners are fully aware of their capabilities and are able to process and share important information. Experienced faculty development facilitators who are also clinical teachers and examiners were able to identify areas to focus and provide relevant examples for application.

Communality is the added dimension to the original MOA model. Participants of this programme are members in a complex system, who already know each other. Having shared identity, language and challenges have allowed them to develop trust while pursuing the common goal of improving the system they were working in. This facilitated knowledge sharing and behavioural change.

The limitation in our programme is the small sample size. However, we believe that is important to review the effectiveness of a programme, especially with regards to behavioural change, and to share how other programmes can benefit from using the frameworks we shared. The findings from this programme will also inform how we conduct future faculty development programmes. With pandemic restrictions lifted, we hope to conduct this programme face-to-face, to facilitate engagement and communication.

V. CONCLUSION

For this faculty development programme to succeed, targets for success must first be defined and factors that contribute to its success need to be identified. This will ensure active engagement from the participants and promote the sustainability of the programme.

Notes on Contributors

Noorjahan Haneem Md Hashim designed the programme, assisted in content creation, curation and matching learning activities, moderated the programme, and conceptualised and wrote this manuscript.

Shairil Rahayu Ruslan participated as a committee of the programme, assisted as a simulated candidate during the training sessions, as well as contributed to the conceptualisation, writing, and formatting of this manuscript. She also compiled the bibliography and cross-checked the references for this manuscript.

Ina Ismiarti Shariffuddin created the opportunity for the programme (Specialty board and interdisciplinary buy-in, department funding), prioritised the programme learning outcomes, chaired the programme, and contributed to the writing and review of this manuscript.

Woon Lai Lim participated as a committee member of the programme and contributed to the writing of this manuscript.

Christina Phoay Lay Tan designed and conducted the faculty development training programme, and reviewed and contributed to the writing of this manuscript. She also cross-checked the references for this manuscript.

Vinod Pallath designed and conducted the faculty development training programme, and reviewed and contributed to the writing of this manuscript.

All authors verified and approved the final version of the manuscript.

Ethical Approval

Ethical approval was applied for the follow-up questionnaire that was distributed to the participants, which was approved on the 6th of May 2022 (Reference number: UM.TNC2/UMREC_1879). The programme evaluation and pre- and post-tests are accepted as part of the programme evaluation procedures.

Data Availability

De-identified individual participant data collected are available in the Figshare repository immediately after publication without an end date, as below :

https://doi.org/10.6084/m9.figshare.20189309.v1

https://doi.org/10.6084/m9.figshare.20186582.v1

https://doi.org/10.6084/m9.figshare.20186591.v2

The authors confirm that all data underlying the findings are freely available for view from the Figshare data repository. However, the reuse and resharing of the programme evaluation form, pre- and posttest questions, as well as followup questionnaire, despite being easily accessible from the data repository, should warrant a reasonable request from the corresponding author out of courtesy.

Acknowledgement

The authors would like to acknowledge Dr Selvan Segaran and Dr Siti Nur Jawahir Rosli from the Medical Education, Research and Development Unit (MERDU) for their logistics and technical support in all stages of this programme; Professor Dr Jamuna Vadivelu, Head, MERDU for her insight and support; Dr Nur Azreen Hussain and Dr Wan Aizat Wan Zakaria from the Department of Anaesthesiology, UMMC and UM, for their acting skills in the training videos; and the Visibility and Communication Unit, Faculty of Medicine, Universiti Malaya for their video editing services.

Funding

There is no funding source for this manuscript.

Declaration of Interest

There are no conflicts of interest among the authors of this manuscript.

References

Arias, J. J., Swinton, J., & Anderson, K. (2018). Online vs. face-to-face: A comparison of student outcomes with random assignment. E-Journal of Business Education & Scholarship of Teaching, 12(2), 1–23. https://eric.ed.gov/?id=EJ1193426

Blew, P., Muir, J. G., & Naik, V. N. (2010). The evolving Royal College examination in anesthesiology. Canadian Journal of Anesthesia/Journal canadien d’anesthésie, 57(9), 804-810. https://doi.org/10.1007/s12630-010-9341-1

Blumberg, M., & Pringle, C. D. (1982). The missing opportunity in organizational research: Some implications for a theory of work performance. The Academy of Management Review, 7(4), 560–569. https://doi.org/10.2307/257222

Cook, D. A., & Steinert, Y. (2013). Online learning for faculty development: A review of the literature. Medical Teacher, 35(11), 930–937. https://doi.org/10.3109/0142159X.2013.827328

Daniel, C. M. (2014). Comparing online and face-to-face professional development [Doctoral dissertation, Nova Southeastern University]. https://doi.org/10.13140/2.1.3157.5042

Feldman, M., Lazzara, E. H., Vanderbilt, A. A., & DiazGranados, D. (2012). Rater training to support high-stakes simulation-based assessments. Journal of Continuing Education in the Health Professions, 32(4), 279–286. https://doi.org/10.1002/chp.21156

Holmboe, E. S., Ward, D. S., Reznick, R. K., Katsufrakis, P. J., Leslie, K. M., Patel, V. L., Ray, D. D., & Nelson, E. A. (2011). Faculty development in assessment: The missing link in competency-based medical education. Academic Medicine, 86(4), 460–467. https://doi.org/10.1097/ACM.0b013e31820cb2a7

Iqbal, I., Naqvi, S., Abeysundara, L., & Narula, A. (2010). The value of oral assessments: A review. The Bulletin of the Royal College of Surgeons of England, 92(7), 1–6. https://doi.org/10.1308/147363510×511030

Juul, D., Yudkowsky, R., & Tekian, A. (2019). Oral Examinations. In R. Yudkowsky, Y. S. Park, & S. M. Downing (Eds.), Assessment in Health Professions Education. Routledge. https://doi.org/10.4324/9781315166902-8

Kemp, N., & Grieve, R. (2014). Face-to-face or face-to-screen? Undergraduates’ opinions and test performance in classroom vs. online learning. Frontiers in Psychology, 5. https://doi.org/10.3389/fpsyg.2014.01278

Khera, N., Davies, H., Davies, H., Lissauer, T., Skuse, D., Wakeford, R., & Stroobant, J. (2005). How should paediatric examiners be trained? Archives of Disease in Childhood, 90(1), 43–47. https://doi.org/10.1136/adc.2004.055103

Lineberry, M. (2019). Validity and quality. Assessment in Health Professions Education, 17-32. https://doi.org/10.4324/9781315166902-2

Marin-Garcia, J. A., & Martinez Tomas, J. (2016). Deconstructing AMO framework: A systematic review. Intangible Capital, 12(4), 1040. https://doi.org/10.3926/ic.838

McLean, M., Cilliers, F., & Van Wyk, J. M. (2008). Faculty development: Yesterday, today and tomorrow. Medical Teacher, 30(6), 555–584. https://doi.org/10.1080/01421590802109834

Md Hashim, N. H. (2021). Pre- and Post-test [Dataset]. Figshare. https://doi.org/10.6084/m9.figshare.20186582.v1

Md Hashim, N. H. (2022). Followup Questionnaire [Dataset]. Figshare. https://doi.org/10.6084/m9.figshare.20186591.v2

Newman, L. R., Brodsky, D., Jones, R. N., Schwartzstein, R. M., Atkins, K. M., & Roberts, D. H. (2016). Frame-of-reference training: Establishing reliable assessment of teaching effectiveness. Journal of Continuing Education in the Health Professions, 36(3), 206–210. https://doi.org/10.1097/CEH.0000000000000086

Newstrom, J. W. (1995). Evaluating training programs: The four levels, by Donald L. Kirkpatrick. (1994). San Francisco: Berrett-Koehler. 229 pp., $32.95 cloth. Human Resource Development Quarterly, 6(3), 317-320. https://doi.org/10.1002/hrdq.3920060310

Singh, J., Evans, E., Reed, A., Karch, L., Qualey, K., Singh, L., & Wiersma, H. (2022). Online, hybrid, and face-to-face learning through the eyes of faculty, students, administrators, and instructional designers: Lessons learned and directions for the post-vaccine and post-pandemic/COVID-19 World. Journal of Educational Technology Systems, 50(3), 301–326. https://doi.org/10.1177/00472395211063754

Steinert, Y., Mann, K., Anderson, B., Barnett, B. M., Centeno, A., Naismith, L., Prideaux, D., Spencer, J., Tullo, E., Viggiano, T., Ward, H., & Dolmans, D. (2016). A systematic review of faculty development initiatives designed to enhance teaching effectiveness: A 10-year update: BEME Guide No. 40. Medical Teacher, 38(8), 769-786. https://doi.org/10.1080/0142159x.2016.1181851

Steinert, Y., Mann, K., Centeno, A., Dolmans, D., Spencer, J., Gelula, M., & Prideaux, D. (2006). A systematic review of faculty development initiatives designed to improve teaching effectiveness in medical education: BEME Guide No. 8. Medical Teacher, 28(6), 497–526. https://doi.org/10.1080/01421590600902976

Tan, C. P. L., & Pallath, V. (2022). Workshop Evaluation Form [Dataset]. Figshare. https://doi.org/10.6084/m9.figshare.20189309.v1

Tekian, A., & Norcini, J. J. (2016). Faculty development in assessment : What the faculty need to know and do. In M. Mentkowski, P.F. Wimmers (Eds.), Assessing Competence in Professional Performance across Disciplines and Professions (1st ed., pp. 355–374). Springer Cham. https://doi.org/10.1007/978-3-319-30064-1

Thomas, P. A., Kern, D. E., Hughes, M. T., & Chen, B. Y. (2015). Curriculum development for medical education : A six-step approach. John Hopkins University Press. https://jhu.pure.elsevier.com/en/publications/curriculum-development-for-medical-education-a-six-step-approach

Vollenbroek, W. B. (2019). Communities of Practice: Beyond the Hype – Analysing the Developments in Communities of Practice at Work [Doctoral dissertation, University of Twente]. https://doi.org/10.3990/1.9789036548205

Williams, R. G., Klamen, D. A., & McGaghie, W. C. (2003). SPECIAL ARTICLE: Cognitive, social and environmental sources of bias in clinical performance ratings. Teaching and Learning in Medicine, 15(4), 270–292. https://doi.org/10.1207/S15328015TLM1504_11

*Shairil Rahayu Ruslan
50604, Kuala Lumpur,
Malaysia
03-79492052 / 012-3291074
Email: shairilrahayu@gmail.com, shairil@ummc.edu.my

Submitted: 23 August 2022
Accepted: 3 January 2023
Published online: 4 July, TAPS 2023, 8(3), 15-25
https://doi.org/10.29060/TAPS.2023-8-3/OA2871

Iroro Enameguolo Yarhere¹, Tudor Chinnah² & Uche Chineze³

¹Department of Paediatrics, College of Health Sciences, University of Port Harcourt, Nigeria; ²Department of Anatomy, University of Exeter, United Kingdom; ³Department of Education and Curriculum studies, University of Port Harcourt, Nigeria

Abstract

Introduction: This study aimed to compare the paediatric endocrinology curriculum across Southern Nigeria medical schools, using reports from learners. It also checked the learners’ perceptions about different learning patterns and competency in some expected core skills.

Methods: This mixed (quantitative and qualitative) study was conducted with 7 medical schools in Southern Nigeria. A multi-staged randomized selection of schools and respondents, was adopted for a focus group discussion (FGD), and the information derived was used to develop a semi-structured questionnaire, which 314 doctors submitted. The FGD discussed rotation patterns, completion rates of topics and perceptions for some skills. These themes were included in the forms for general survey, and Likert scale was used to assess competency in skills. Data generated was analysed using statistical package for social sciences, SPSS 24, and p values < 0.05 were considered significant

Results: Lectures and topics had various completion rates, 42.6% – 98%, highest being “diabetes mellitus”. Endocrinology rotation was completed by 58.6% of respondents, and 58 – 78 % perceived competency in growth measurement and charting. Significantly more learners, 46.6% who had staggered posting got correct matching of Tanner staging, versus learners who had block posting, 33.3%, p = 0.018.

Conclusion: Respondents reported high variability in the implementation of the recommended guidelines for paediatric endocrinology curriculum between schools in Southern Nigeria. Variabilities were in the courses’ completion, learners’ skills exposure and how much hands-on were allowed in various skills acquisitions. This variability will hamper the core objectives of human capital development should the trend continue.

Keywords:          Paediatric Endocrinology Curriculum, Perception, Compliance, Completion Rate, Learners

Practice Highlights

• Medical and dental council of Nigeria has a recommended benchmark for minimum academic standards in all medical schools to which total compliance is expected.
• Evaluation of paediatric endocrinology curriculum content and training methods was conducted using reports from learners.
• Variability in the content, and training methods of the intended competency were reported across medical schools.
• Compliance rate of the recommended curriculum was less than 50% in some contents and some learners reported low skill performance training.
• The lack of uniformity can prevent achievement of the overarching objective of the curriculum in Nigeria with wide variations in competence among graduating doctors.

I. INTRODUCTION

The primary aim of the Medical and Dental council of Nigeria (MDCN) undergraduate curriculum is “to train doctors and dentists who can work effectively in a health team to provide comprehensive health care to individuals in any community in the nation, and keep up to date on issues of global health” (Federal Ministry of Health of Nigeria, 2012). In Nigeria today, there are 49 federal, 59 states and 111 private universities, and 44 of these have full or partially accredited medical schools and while these schools have a prescribed curriculum, some are not following explicitly (Federal Ministry of Health of Nigeria, 2012). This curriculum advocates for universities to develop syllabus to meet the benchmark for minimum academic standards (BMAS) across schools, however there is no uniform template developed for assessing graduates to know how their competence converge as is applicable in United States of America (USA), Canada and United Kingdom (UK) (Santen et al., 2019; Shah et al., 2020; Sosna et al., 2021).  Diabetes mellitus, thyroid disorders, puberty, rickets and growth abnormalities are topics included in the MDCN paediatric curriculum under endocrinology which learners are expected to acquire competence in cognitive and psychomotor skills to diagnose and treat or refer appropriately children presenting with these diseases.

A. Problem

Most deaths from diseases in Nigeria and other resource-limited countries are consequent upon general public ignorance of disease, late presentation to the health care systems, poverty and lack of funds to access healthcare facilities and reduced knowledge of some disease patterns by the healthcare providers (Yarhere & Nte, 2018). Addressing the gaps in reduced knowledge can be done by developing competency-based curriculum for all graduating doctors to have as near-similar competence as possible but achieving this may not be feasible. Training activities are not uniform throughout medical schools in Nigeria and elsewhere, and depend on schools’ vision, mission and objectives, and the structures and processes put in place. There are barriers to positive implementation across schools including but not limited to individual school’s determination of what is relevant in the curriculum, access to the materials needed to teach the curriculum content and getting trainers to use these curriculums (Polikoff, 2018). The lack of uniformity of curriculum across universities may not be contending issues, but when the graduating doctors have varying degree of competencies in skills and cognition, then a template for imparting uniform and up to date knowledge and to evaluate this is needed to find ways of reducing the variability (McManus, 2003; McManus et al., 2020; Rimmer, 2014).

The curriculum uniformity across schools is one way of improving competency and thus, healthcare standards, and there is need to explore this uniformity or diversity within the paediatric undergraduate training. In some countries, there is a uniform board certification examination before doctors can practice and this is also done for doctors immigrating into these countries (Hohmann & Tetsworth, 2018; Puri et al., 2021; Tiffin et al., 2017; van Zanten et al., 2022) but Nigeria is exempt from this uniform exit examination. This uniform exit board examination makes these schools align course contents, and therefore reduces the variabilities between medical schools and undergraduate training.

B. Curriculum Evaluation for Change or Improvement

Curriculum evaluation is a means by which educators understand whether the curriculum used to train learners is working as intended, and whether there is need to change the entire programme or redesign aspects (Burton & McDonald, 2001; Ornstein & Hunkins, 2009). It is also a way of identifying deficiencies in training syllabus across universities, (Rufai et al., 2016) or whether compliance to a curriculum is being achieved (Grant, 2014; Olson et al., 2000). Kirkpatrick’s curriculum evaluation method is widely acceptable in medical education using the 4 steps; learners’ reaction or satisfaction, knowledge, behavioural changes and results or impact, and in Nigeria, for paediatric endocrinology, this has not been done (Alsalamah, 2021; Bates, 2004).

Universities have variabilities in organisation, students’ numbers in classes, duration of specific posting, posting types and whether the courses are elective or core. In medical schools in Nigeria, paediatric postings are undertaken in the 5^th or 6^th year of a 6-year programme. While some stagger the posting to be done within the last 2 years, others do theirs in the 5^th or the 6^th year exclusively, and the extent of these variabilities and how they affect the training processes and products has not been evaluated in Nigeria and this can be done using learners’ or graduates’ perceptions.

The aim of this research was to evaluate learners’ report and perception of some aspects of the paediatric endocrinology curriculum contents and learning methods across Southern Nigeria medical schools. Endocrinology was taken from the paediatric course to reduce the volume of information to be analysed.

II. METHODS

This was a cross sectional study design with qualitative and quantitative data analyses, evaluating learners’ report and their perception of the curriculum being used by various medical schools in Southern Nigeria to deliver the MDCN paediatric endocrinology curriculum. Survey was conducted across 10 medical schools in Southern Nigeria that have learners who have either completed their final year, or are doing their internship. Two steps were used to retrieve the information needed; a focus group discussion of sampled learners, and a questionnaire survey sent out to randomly selected respondents and these 2 methods complemented each other. The focus group discussion was used to explore in depth, the minds of the respondents and what they perceived was being done well and what needed to be changed in the syllabus in their respective schools. The questionnaire survey was then used to collect reports and perceptions from a wider set of learners who had completed their paediatric posting within the past 6 – 12 months. Some of these were already doing their internship and others were in their final year in preparation for their final examinations.

Sample size for respondents will be calculated using the formula:

N = (Z score)² x SD x (1 – SD) 

                       (CI)²

Z score = 1.96, SD (standard deviation of the mean) = estimated at ± 0.5, Confidence interval = 0.05

= 384 respondents, with an attrition rate of 10% will be added 10% of 384 = 38

384 + 38 = 422 respondents.

A. Sampling Technique

Multi-staged sampling technique was used to determine the schools, and respondents that participated in the study. There are 29 Southern Universities with medical / health colleges and 16 of these had more than 50 learners in their final year or had graduated. Ten schools were randomly selected using the excel formula [= rand ()], and a proportionate stratified sampling was done using the matriculation numbers of the students in each school to arrive at 422 respondents. Total number of learners that studied paediatrics in various institutions was 800; Ibadan 150, Port Harcourt, 128, Lagos 128, Niger delta University 69, UNN Enugu 128, University of Benin 128, Others 69. From the total number of learners in each school, selected learners and interns were sent the questionnaire using their email addresses. Selection for the FGD was done using simple random sampling from each school and these were sent separate emails with details for the meeting.

B. Focus Group Discussions Process

Focus group discussion was conducted with the respondents using zoom video platform, and the process lasted for 2 hours, 30 minutes. Ten learners’ representatives from the selected schools were contacted for this FGD, however, 7 (70%) agreed to participate after several email reminders.  The interview was semi-structured with a flexible topic guide, which covered issues relating to the respondents’ views and opinions on the curriculum in paediatric endocrinology; description of posting type in each school, whether block, or staggered, topics received and/or completed, perception of their competence in a key psychomotor skill. The focus group interview discussions were recorded in the zoom meeting platform and transcribed verbatim. The data were analysed using the thematic framework content analyses method. The themes generated were categorised into; 1. Lecture contents and completion rate, 2. Types of paediatric rotation and posting, 3. Skill competence acquisition and clinical postings. Their perceptions about these themes were also sought and discussed. The transcription of the groups’ discussions was reviewed by IY and TC to help categorise the data and pull-out important quotes used.

C. Questionnaire Survey

Following thematic analyses of the FGD, the themes generated were converted to questions in a survey for a larger sample population. Themes generated were the type of paediatric posting, rotations through units in the departments and paediatric endocrinology topics, training methods and competency acquired. Demographic characteristics of responders such as level/year of study, age, gender and university of study were collected. The respondents were also asked to select topics from a poll, included in their paediatric endocrinology syllabus, with result in Figure 1, and to state the various methods used to learn growth and growth disorders in their schools. A means of assessing cognitive (recall) skills of the learners was conducted using animated pictures of Tanner staging and matching-type multiple choice, and the responses were crossmatched with the type of posting learners were exposed to, i.e. block posting or staggered posting. Tanner staging was chosen as it cuts across general paediatrics and endocrinology as part of growth and puberty (endocrinology).

Data retrieved were analysed statistically by using chi-square test, and Pearson correlation for categorical variables. The level of competence perceived by learners in height measurement and charting on growth chart was retrieved using 5-point Likert scale (where 1 = not competent; 2 = low competence; 3 = neutral; 4 = competent; and 5 = proficient). The association between level of competence and whether learners rotated through paediatric endocrinology was checked using Pearson’s correlation test. For all statistics, p value < 0.05 was considered significant.

D. Ethics

The research commenced after the Research Ethics committee of the University of Port Harcourt granted approval (UPH/CEREMAD/REC/MM80/056). Verbal informed consent was obtained from the participants during the focus group discussion, who also gave consent for video and recording of the process. Informed consent was also obtained from all participants who filled and submitted the online survey. The focus group discussants received N3,000 ($10) for internet data only as monetary compensation.

III. RESULTS

There were 314 learners from the 422 calculated sample size, responded to the questionnaire survey, giving a response rate of 74.4%. There were more final year respondents than early career doctors and more of the respondents were in the age bracket 20 – 24 years, with a mean of 25.02 ±2.71 years. The male: female ratio was 1:1.01, and the data that support the findings of this study are available in Figshare at https://doi.org/10.6084 /m9.figshare.20730937.v1 (Yarhere et al., 2022).

RESPONDENTS	Frequency	Percentage
Year of study
Early career doctor (graduate/intern)	130	41.4	p = 0.002
Final year	184	58.6
University attended (calculated cohort)
University of Port Harcourt (63)	62	19.7
Niger Delta University (54)	54	17.2
University of Ibadan (76)	50	15.9
University of Benin (65)	44	14.0
University of Lagos (65)	40	12.7
University of Nigeria (65)	42	13.4
Other western Universities (34)	22	7.0
Age
20-24	140	44.6
25-29	162	51.6
>=30	12	3.8
Mean	25.02 ± 2.71
Gender
Male	152	48.4	p = 0.612
Female	162	51.6

Table 1. Demographic characteristics of all respondents and the universities attended

There were basically 2 modes of paediatric posting in the institutions sampled; 4-months block posting where respondents have a month of didactic lectures and 3 months of clinical rotations through various units in the Paediatric departments, and 4 months of staggered rotations with junior and senior postings in the clinical classes. While some learners rotated through all the units (core and electives) in the departments, some went through core units, emergency and neonatal units, and 2 other units randomly selected for the respondents by the departments.

C. Learners’ Responses to Rotation through Paediatrics and Posting Types

Participant 2 shared:

The way it works in University of xxxx, we rotate through 2 elective postings with core (CHEW and SCBU) postings in the junior and senior postings. These elective postings are randomly selected by the department (meaning heads or coordinators). I did neurology and gastroenterology in my junior posting and haemato-oncology and I really can’t remember the other one in my senior posting.

 “Will I be wrong to say you did not see a patient with Diabetic keto acidosis?”

Facilitator

“I saw a child with diabetic keto acidosis in the ward but it wasn’t my unit managing the patient. I only went to the ward to do some other thing.”

Participant 2 

“If you were given the opportunity to design a curriculum or programme for your university, will you prefer what is being practiced now, or will you rather have every student go through every unit and get titbits from each unit?”

Facilitator

Participant 2 responded:

Yes, I will prefer that situation where you get to be exposed to every unit in the department but …. emmm, that creates a problem because you may be in a unit for a week, and no patient comes in but the next group rotating to the unit gets to see many patients. I would want to suggest that perhaps, instead of focusing on more of clinical posting, that a unified tutorial class which will expose everyone to the core diseases in the various disciplines.

Table 2 corroborates the information given by the focus group discussants. Testing the competency outcome in either method can give some estimated guess as to which is better, however, there are several confounding factors that will not allow fair comparison (See Table 3).

Though there were differences in the mode of paediatric postings where staggered or block, c² = 4.59, p = 0.032. the difference in proportion of respondents who had core and selected elective posting as against all units posting was not significant, c² = 0.318, p = 0.573.

In the 2×2 table above where recall was tested in the learners based on their paediatric posting type, higher percentage of those who had staggered posting got the correct matching of Tanner stage, and the difference was significant, c² = 5.630, p = 0.018. However, the total number of respondents with the correct response was low.

D. Perception of Core Competency Skill in Growth Measurement and Charting by Learners

One of the most important courses in paediatrics is growth and development and training future medical doctor to acquire skills and competence in growth and management is a key component of the BMAS. While growth measurement may seem easy to the uninformed, the whole task is daunting especially in children with complex growth abnormalities and malformation, and for more complex skills like arm span. Which of the more complex skills should the learner be expected to be competent in, will be debated in an expert forum of trainers.

“So, did you do anthropometric measures?”

Facilitator

Participant 1 shared:

Yes, anytime we clerk a patient, we must check the weight and height and interpret using age-appropriate charts, but we did not plot them in the charts. We carry the age-appropriate chart and interpreted our patients, as this is a requirement.

Using the chart may not be emphasised by all paediatric lecturers, so learners can be smart to know those lecturers who will request this skill from them during the clerkship period or the unit rotations.

“We did not quite get the concept of mid parental height, height percentile, it was just mentioned in passing. I never saw a severely short child that needed growth hormone. I was only told by a classmate of mine.”

Participant 3

The charting and interpretation of weight and height measurements of children was not done in all schools as shown in Table 4 below, which tells that only 65.8% of total respondents were taught interpretation of measured and charted growth parameters. The level of competence in these tasks will also be varied as seen in Appendix 1. Two hundred and thirty-eight (75.8%) learners perceived they had competency/ proficiency in height measures using stadiometer, and 44.6 % of the learners with these perceptions actually had paediatric endocrinology clinical rotation (Appendix 1).

Bone age and orchidometers are used to assess skeletal maturation and puberty, which are advanced for the undergraduate learners and certainly not compulsory, but some respondents were taught with the tools showing the variabilities in contents and skills delivery between these schools. From Table 4 above, framers of the syllabus for endocrinology aspect of paediatrics curriculum are unlikely to include use of orchidometer and bone age during the undergraduate paediatric endocrinology rotation as the skill is complex, and not necessary for their level of development.

IV. DISCUSSION

This study has highlighted differences in course contents and training methods across medical schools in Southern Nigeria. While many schools have used the BMAS prescribed by the MDCN, the syllabus used are different and the intended learning outcomes are diverse based on the respondents’ reports. Some learners reported not having diabetes lectures in their school through no fault of theirs, as lecturer rescheduled the lectures and never gave them. While learners have the responsibility to attend lectures, trainers are also obligated to be present at their scheduled lectures or transfer this to their teacher-assistants, or use technologies (Grant, 2014; Ruiz et al., 2006). Some learners had little participation in the Emergency Room, others participated fully in DKA management, learning empathy, specialised skills and communication. The intended competencies to be acquired can be achieved through shadowing and participation, bed-side teaching, and tutorial to improve the cognitive and psychomotor skill, and these opportunities must be created for them in experiential settings (Ryan et al., 2020; Shah et al., 2020).

More learners had staggered postings, going through junior and senior paediatric postings in what may be considered as integrated learning departing from the traditional method (Patel et al., 2005; Watmough et al., 2006, 2009). In the staggered posting type of rotation, we noticed that not all learners went through paediatric endocrinology unit posting, and like one of the discussants said, they would rather everyone went through each unit getting bits of everything and having opportunity to study specific and prevalent diseases in paediatric units rather than leaving them with the possibility of not learning important disorders. As it is not always possible to encounter specific diseases like DKA during entire posting in the schools that use staggered posting types, the likelihood of exposure was higher in schools that had block posting from the FGD conducted, but this did not translate to better retention of skills or cognitions as depicted in the Tanner staging matching question.

Having learners train in all special postings may not be the best approach in undergraduate medicine because the specialised skills may not be utilised in general practice and even in general paediatrics should the learners plan paediatric specialisation (Bindal et al., 2011). While some trainers may argue that all information and skill should be taught to the learners, the time to acquire and achieve mastery may be short for the learners (Jensen et al., 2018; Offiah et al., 2019). This study can be referenced in curriculum designing and implementation so the framers understand what society needs should be filled at any time. The concept of cognitive overload has actually reduced the duration of core specialty in clinical medicine while increasing the duration for others with emphasis on psychomotor, affective skills and professionalism. Some medical schools have core paediatric posting of 7 – 8 weeks, but Nigeria is still fixed with the traditional 3 – 4 months. In some schools in South Africa, the clinical posting is run as modular block for 3 years, with paediatric curriculum running from year 4 through year 6 (Dudley & Rohwer, 2015). With the long duration in the Nigeria curriculum, skills competencies are still deficient, so there is need to revamp the curriculum to make it more competency driven. It is excusable that more sophisticated competence like use of orchidometer were not known by more than half the learners, but if some were taught, the level of confidence in these skills at this stage of their learning should also be assessed as was done for diabetes by George et al. (2008).

Medical schools in Nigeria and other countries will have to continually evolve and produce curricula that are competency based, using problem-based learning, simulations, mannikin training for skills as is done in other countries (Watmough et al., 2006). Diabetes, thyroid, ambiguous genitalia with congenital adrenal hyperplasia, short stature and calcium disorders are common in Nigeria and should be taught in structured and integrated formats. Integrated curriculum where skills are graded from simple to complex can also be tested e g, skills of height measurements and charting using the stadiometer and growth charts can be taught in the 1^st clinical year, and then the mid parental heights, target height calculation and bone age may be taught in the 2^nd and 3^rd clinical years. (Brauer & Ferguson, 2015; Grant, 2014).

A. Strength of the Research

Articulating the perceptions of learners is not always easy as they are varied and subjective, but getting them to come together, discuss and give suggestions on how curriculum can be designed and achieved increases the strength of this research. There was no sense of victimisation of the learners as many had already graduated from their schools, and the discussants admitted to not missing classes, or clinical learning. They spoke freely, with courtesy to others and there was little or no argument among them.

B. Limitations of the Research

As this research is based on past experiences of the cognitive and psychomotor skills achieved during the learners’ training period, the possibility of recall bias is high, and respondents may underestimate or exaggerate their skills. Using respondents who had just concluded their paediatric postings was an attempt at reducing this limitation. The best time to evaluate a programme is usually soon after the programme has been concluded however, as there has been no report of this type of evaluation, there was need to embark on it and make recommendations.

V. CONCLUSION

Respondents reported high variability in the implementation of the recommended guidelines for paediatric endocrinology curriculum between schools in Southern Nigeria. Variabilities were in the courses’ completion, learners’ skills exposure and how much hands-on were allowed in various skills acquisitions. This variability will hamper the core objectives of human capital development should the trend continue.

A. Area of Future Research

Noting the differences exist between schools, curriculum strategists and implementation teams in universities should commission a DELPHI study by experts, where core competencies and objectives for paediatric endocrinology will be agreed on and sent to the regulatory bodies for endorsement and implementation.

Notes on Contributors

IY conceived, designed, planned, executed and conducted interviews and the research. He also collected the data, analysed it and wrote the manuscript.

TC helped in designing the methodology for the data colllection and analyses, and reviewed the manuscript.

CU gave critical appraisal of the manuscript and all authors have approved the final manuscript.

Ethical Approval

The research ethics committee of the Univeristy of Port Harcourt gave ethical approval before the start of the study with the number: UPH/CEREMAD/REC/MM80/056.

Data Availability

The data supporting this research is available for publication purposes, without editing. Data can be shared only with express permission from the corresponding author as deposited in Figshare repository, using the private url:

https://figshare.com/articles/dataset/Copy_of_CURRICULUM_STUDENTS_xls/21154396

Acknowledgement

We acknowledge the early career doctors and final year students who participated in the online survey especially the selected ones who took part in the focus group discussion.

Declaration of Interest

Authors declare that there are no conflicts of interest, including financial, consultant, institutional and other relationships that might lead to bias or a conflict of interest.

Funding

There was no funding for this survey.

References

Alsalamah, A., & Callinan, C. (2021). Adaptation of Kirkpatrick’s four level model of training criteria to evaluate training programmes for head teachers. Education Science, 11(116), 1-25. https://doi.org/10.3390/educsci11030116

Bates, R. (2004). A critical analysis of evaluation practice: The Kirkpatrick model and the principle of beneficence. Evaluation and Program Planning, 27, 341-347. https://doi.org/10.1016/j.evalprogplan.2004.04.011

Bindal, T., Wall, D., & Goodyear, H. M. (2011). Medical students’ views on selecting paediatrics as a career choice. European Journal of Pediatrics, 170(9), 1193-1199. https://doi.org/10.1007/s00431-011-1467-9

Brauer, D. G., & Ferguson, K. J. (2015). The integrated curriculum in medical education: AMEE Guide No. 96. Medical Teacher, 37(4), 312-322. https://doi.org/10.3109/0142159X.2014.970998

Burton, J. L., & McDonald, S. (2001). Curriculum or syllabus: Which are we reforming? Medical Teacher, 23(2), 187-191. https://doi.org/10.1080/01421590020031110

Dudley, L. D., Young, T. N., Rohwer, A. C., Willems, B., Dramowski, A., Goliath, C., Mukinda, F. K., Marais, F., Mehtar, S., & Cameron, N. A. (2015). Fit for purpose? A review of a medical curriculum and its contribution to strengthening health systems in South Africa. African Journal Health Profession Education, 7(1), 81-84. https://doi.org/10.7196/AJHPE.512

Federal Ministry of Health of Nigeria, Health Systems 20/20Project. (2012). Nigeria undergraduate medical and dental curriculum template. Health systems 20/20 Project, Abt Associates Inc.

George, J. T., Warriner, D. A., Anthony, J., Rozario, K. S., Xavier, S., Jude, E. B., & Mckay, G. A. (2008). Training tomorrow’s doctors in diabetes: Self-reported confidence levels, practice and perceived training needs of post-graduate trainee doctors in the UK. A multi-centre survey. BMC Medical Education, 8, Article 22. https://doi.org/10.1186/1472-6920-8-22

Grant, J. (2014). Principles of curriculum design. In T. Swanwick, K. Forrest, B. C. O’Brien (Eds.), Understanding medical education evidence, theory and practice Sussex, UK: Wiley Blackwell, 31-46.

Hohmann, E., & Tetsworth, K. (2018). Fellowship exit examination in orthopaedic surgery in the commonwealth countries of Australia, UK, South Africa and Canada. Are they comparable and equivalent? A perspective on the requirements for medical migration. Medical Education Online, 23(1), Article 1537429. https://doi.org/10.1080/10872981.2018.1537429

Jensen, J. K., Dyre, L., Jørgensen, M. E., Andreasen, L. A., & Tolsgaard, M. G. (2018). Simulation-based point-of-care ultrasound training: a matter of competency rather than volume. Acta Anaesthesiology Scandanavia, 62(6), 811-819. https://doi.org/10.1111/aas.13083

McManus, I. C. (2003). Medical school differences: beneficial diversity or harmful deviations. BMJ Quality and Safety in Health Care, 12(5), 324-325. https://doi.org/10.1136/qhc.12.5.324

McManus, I. C., Harborne, A. C., Horsfall, H. L., Joseph, T., Smith, D. T., Marshall-Andon, T., Samuels, R., Kearsley, J. W., Abbas, N., Baig, H., Beecham, J., Benons, N., Caird, C., Clark, R., Cope, T., Coultas, J., Debenham, L., Douglas, S., Eldridge, J., . . . Devine, O. P. (2020). Exploring UK medical school differences: the MedDifs study of selection, teaching, student and F1 perceptions, postgraduate outcomes and fitness to practise. BMC Medicine, 18(1), Article 136. https://doi.org/10.1186/s12916-020-01572-3

Offiah, G., Ekpotu, L. P., Murphy, S., Kane, D., Gordon, A., O’Sullivan, M., Sharifuddin, S. F., Hill, A. D. K., & Condron, C. M. (2019). Evaluation of medical student retention of clinical skills following simulation training. BMC Medical Education, 19(1), Article 263. https://doi.org/10.1186/s12909-019-1663-2

Olson, A. L., Woodhead, J., Bekow, R., Kaufman, N., & Marshal, S. (2000). A national general pediatric clerkship curriculum: The process of development and implementation. Pediatrics, 160(S1), 216 -222. https://doi.org/10.1542/peds.106.S1.216

Ornstein A, H. F., & Hunkins, F. P. (2009). Curriculum: Foundations, principles and issues (5th Ed.). Pearson.

Patel, V. L., Arocha, J. F., Chaudhari, S., Karlin, D. R., & Briedis, D. J. (2005). Knowledge integration and reasoning as a function of instruction in a hybrid medical curriculum. Journal of Dental Education, 69(11), 1186-1211. https://www.ncbi.nlm.nih.gov/pubmed/16275683

Polikoff, M. S. (2018). The challenges of curriculum materials as a reform lever Evidence Speaks Reports, 2, 58

Puri, N., McCarthy, M., & Miller, B. (2021). Validity and reliability of pre-matriculation and institutional assessments in predicting USMLE STEP 1 success: Lessons from a traditional 2 x 2 curricular model. Frontiers in Medicine (Lausanne), 8, Article 798876. https://doi.org/10.3389/fmed.2021.798876

Rimmer, A. (2014). GMC will develop single exam for all medical graduates wishing to practise in UK. BMJ, 349, g5896. https://doi.org/10.1136/bmj.g5896

Rufai, S. R., Holland, L. C., Dimovska, E. O., Bing Chuo, C., Tilley, S., & Ellis, H. (2016). A national survey of undergraduate suture and local anesthetic training in the United Kingdom. Journal of Surgical Education, 73(2), 181-184. https://doi.org/10.1016/j.jsurg.2015.09.017

Ruiz, J. G., Mintzer, M. J., & Leipzig, R. M. (2006). The impact of E-learning in medical education. Academic Medicine, 81(3), 207-212. https://doi.org/10.1097/00001888-200603000-00002

Ryan, A., Hatala, R., Brydges, R., & Molloy, E. (2020). Learning with patients, students, and peers: Continuing professional development in the solo practitioner workplace. Journal of Continuing Education in the Health Profession, 40(4), 283-288. https://doi.org/10.1097/CEH.0000000000000307

Santen, S. A., Feldman, M., Weir, S., Blondino, C., Rawls, M., & DiGiovanni, S. (2019). Developing comprehensive strategies to evaluate medical school curricula. Medical Science Educator, 29(1), 291-298. https://doi.org/10.1007/s40670-018-00640-x

Shah, S., McCann, M., & Yu, C. (2020). Developing a national competency-based diabetes curriculum in undergraduate medical education: A Delphi study. Canadian Journal of Diabetes, 44(1), 30-36. https://doi.org/10.1016/j.jcjd.2019.04.019

Sosna, J., Pyatigorskaya, N., Krestin, G., Denton, E., Stanislav, K., Morozov, S., Kumamaru, K. K., Jankharia, B., Mildenberger, P., Forster, B., Schouman-Clayes, E., Bradey, A., Akata, D., Brkljacic, B., Grassi, R., Plako, A., Papanagiotou, H., Maksimović, R., & Lexa, F. (2021). International survey on residency programs in radiology: similarities and differences among 17 countries. Clinical Imaging, 79, 230-234. https://doi.org/10.1016/j.clinimag.2021.05.011

Tiffin, P. A., Paton, L. W., Mwandigha, L. M., McLachlan, J. C., & Illing, J. (2017). Predicting fitness to practise events in international medical graduates who registered as UK doctors via the Professional and Linguistic Assessments Board (PLAB) system: a national cohort study. BMC Medicine, 15(1), Article 66. https://doi.org/10.1186/s12916-017-0829-1

van Zanten, M., Boulet, J. R., & Shiffer, C. D. (2022). Making the grade: licensing examination performance by medical school accreditation status. BMC Medical Education, 22(1), Article 36. https://doi.org/10.1186/s12909-022-03101-7

Watmough, S., Garden, A., & Taylor, D. (2006). Does a new integrated PBL curriculum with specific communication skills classes produce Pre Registration House Officers (PRHOs) with improved communication skills. Medical Teachers, 28(3), 264-269. https://doi.org/10.1080/01421590600605173

Watmough, S., O’Sullivan, H., & Taylor, D. (2009). Graduates from a traditional medical curriculum evaluate the effectiveness of their medical curriculum through interviews. BMC Medical Education, 9, Article 64. https://doi.org/10.1186/1472-6920-9-64

Yarhere, I., Chinnah, T., & Uche, C. (2022). Learners’ report and perception of differences in undergraduate paediatric endocrinology curriculum content and delivery across Southern Nigeria. [Data set]. Figshare. https://doi.org/10.6084/m9.figshare.20730937.v1

Yarhere, I. E., & Nte, A. R. (2018) A ten-year review of all cause paediatric mortality in University of Port Harcourt Teaching Hospital, Nigeria (2006–2015). Nigerian Journal of Paediatrics, 45(4), 185-91.

*Iroro Enameguolo Yarhere
East/West Road,
PMB 5323 Choba,
Rivers State, Nigeria
+2347067987148
Email: iroro.yarhere@uniport.edu.ng

Submitted: 16 May 2022
Accepted: 3 January 2023
Published online: 4 July, TAPS 2023, 8(3), 5-14
https://doi.org/10.29060/TAPS.2023-8-3/OA2813

Bikramjit Pal¹, Aung Win Thein², Sook Vui Chong³, Ava Gwak Mui Tay⁴, Htoo Htoo Kyaw Soe⁵ & Sudipta Pal⁶

¹Department of Surgery, Manipal University College Malaysia, Melaka, Malaysia; ²Department of Surgery, Manipal University College Malaysia, Melaka, Malaysia; ³Department of Medicine, Manipal University College Malaysia, Melaka, Malaysia; ⁴Department of Surgery, Manipal University College Malaysia, Melaka, Malaysia; ⁵Department of Community Medicine, Manipal University College Malaysia, Melaka, Malaysia; ⁶Department of Community Medicine, Manipal University College Malaysia, Melaka, Malaysia

Abstract

Introduction: The practice of high-fidelity simulation-based medical education has become a popular small-group teaching modality across all spheres of clinical medicine. High-fidelity simulation (HFS) is now being increasingly used in the context of undergraduate medical education, but its superiority over traditional teaching methods is still not established. The main objective of this study was to analyse the effectiveness of HFS-based teaching over video-assisted lecture (VAL)-based teaching in the enhancement of knowledge for the management of tension pneumothorax among undergraduate medical students.

Methods: A cohort of 111 final-year undergraduate medical students were randomised for this study. The efficacy of HFS-based teaching (intervention group) and VAL-based teaching (control group), on the acquisition of knowledge, was assessed by single-best answer multiple choice questions (MCQ) tests in the first and eighth week of their surgery posting. Mean and standard deviation (SD) for the total score of MCQ assessments were used as outcome measures. ANCOVA was used to determine the difference in post-test MCQ marks between groups. The intragroup comparison of the pre-test and post-test MCQ scores was done by using paired t-test. The P-value was set at 0.05.

Results: The mean of post-test MCQ scores were significantly higher than the mean of pre-test MCQ scores in both groups. The mean pre-test and post-test MCQ scores in the intervention group were slightly more than those of the control group but not statistically significant.

Conclusion: There was a statistically significant enhancement of knowledge in both groups but the difference in knowledge enhancement between the groups was insignificant.

Keywords:           High-Fidelity Simulation, Video-Assisted Lecture, Simulation-Based Medical Education (SBME), Randomized Controlled Trial (RCT), Medical Education, Pre-test and Post-test Knowledge Assessments

Practice Highlights

• An RCT study to evaluate the effectiveness of HFS over video-assisted lecture teaching method.
• HFS seems to be not superior than VAL-based teaching for knowledge acquisition and retention.
• HFS may be used judiciously when the objectives are mainly knowledge based.
• Further research may determine curricular areas where HFS is superior and worth adopting.

I. INTRODUCTION

High-Fidelity Simulation (HFS) is an innovative healthcare education methodology that involves the use of sophisticated life-like mannequins to create a realistic patient environment. HFS can be considered an innovative teaching method that aids students in translating knowledge and psychomotor skills from the classroom to the actual clinical setting. Kolb’s Experiential Learning Cycle (Kolb, 1984) provides a basis for the integration of active learning of simulation with conventional teaching methods for a comprehensive learning experience in undergraduate medical education. HFS-based education is potentially an efficacious pedagogy that is now available for teaching. The usefulness of HFS has been recognized by the Accreditation Council of Graduate Medical Education (Accreditation Council of Graduate Medical Education [ACGME], 2020). HFS has the added benefit of increasing students’ confidence and their ability to care for the patients at the bedside (Kiernan, 2018). HFS-based education and video-assisted lecture-based teaching are both effective in achieving factual learning. Despite the increasing acceptance of HFS, there are limited studies to compare the usefulness of HFS with conventional teaching methods for factual learning among undergraduate medical students. At present, the different research studies have not provided enough evidence to establish HFS-based teaching’s superiority over traditional educational methods in the acquisition and retention of knowledge. There is inconsistent and variable outcome regarding the effectiveness of HFS on student learning (Yang & Liu, 2016). HFS-based education is both time-consuming and resource intensive. Its long-term merits in retaining knowledge and translating it into enhanced patient care need further research. As educators, we need to systematically evaluate the expensive newer teaching-learning modules like HFPS for their effectiveness by using rigorous research methodology and protocols. This is to ensure that we are providing the best learning opportunities conceivable for the students. Previous studies were mostly done in North America and, therefore, the generalisability of these results is guarded and might not be applicable in the context of Europe and Asia due to many differences in academic and curriculum aspects (Davies, 2008). The purpose of this study was to establish the feasibility of the use of HFS to deliver critical care education to final-year medical students and to find its efficacy in the enhancement of knowledge when compared to video-assisted lectures. The study compared the effectiveness of two methods of teaching pedagogy in the enhancement of knowledge acquisition using pre-test and post-test MCQ. This study was designed to provide insights that may be applied to the future development and improvement of HFS-based education among undergraduate medical students and its possibility of integrating it into course curricula.

II. METHODS

A. Study Design

Randomized Controlled Trial (RCT) with parallel groups and 1:1 allocation. Please see Appendix 1 for the Flow Chart.

B. Sample Size

G*Power software was used to calculate the sample size (Faul et al., 2007). Based on the preliminary RCT study of our institute done with the same protocol in 2018, the calculated sample size was 114 with a power of 0.95 for this study.

C. Inclusion and Exclusion Criteria

All male and female final-year undergraduate medical (MBBS) students in our institute were recruited after obtaining their written informed consent. All final-year students in the institute consented to the study. The participants were between the ages of 22-26 years.

The total number of participants recruited was 123.

The number of participants dropped out was 12 (9.77%).

Out of 111 participants who completed the study, 61 (54.95%) were female and 50 (45.05%) were male.

The study was conducted in the Clinical Skills Simulation Lab of Melaka Manipal Medical College (presently known as Manipal University College Malaysia).

The study period was from March 2019 to February 2020 (12 months).

D. Interventions

1) Description of HFPS-based teaching: It was an interactive session using a high-fidelity patient simulator demonstrating the management of tension pneumothorax by performing Needle Decompression on METIman (Pre-Hospital) following the Advanced Trauma Life Support Manual developed by the American College of Surgeons (ATLS Subcommittee et al., 2013).

2) Description of Hi-fidelity simulator: METIman Pre-Hospital HI-Fidelity Simulator (MMP-0418) was used for the simulation sessions. It was a fully wireless, adult High-Fidelity Patient Simulator (HFPS) with modelled physiology. It comes with extensive clinical features and capabilities designed specifically for learners to practice, gain experience, and develop clinical mastery in a wide range of patient care scenarios.

3) Description of video-assisted lecture-based teaching: It was a small group interactive session delivered face-to-face to the participants using a recorded video clip demonstrating the management of tension pneumothorax by performing Needle Decompression on METIman (Pre-Hospital) following the Advanced Trauma Life Support Manual developed by the American College of Surgeons (ATLS Subcommittee et al., 2013).

E. Outcome

The tool for measurement of knowledge was an identical set of single-best answer A-type MCQs. These MCQs were used for both Pre-test and Post-test knowledge assessments. MCQs were constructed based on the teaching sessions to assess their learning outcome.

The efficacy of HFPS-based teaching when compared to video-assisted lecture-based teaching is enhancement of knowledge for management of tension pneumothorax.

F. Recruitment

The students were recruited in the study during their final year surgical posting.

G. Randomisation

A cohort of 12 to 14 students from each rotation was randomised into intervention (HFPS-based teaching) and control (video-assisted lecture-based teaching) groups following random sequence generation method.

A computer-generated random sequence number was developed from randomizer.org. The independent randomiser was a biostatistician who did not participate in the delivery of interventions. The allocated interventions were then sealed in a sequentially numbered, opaque envelope.

Block randomisation with a block size of two was used to assign the students into intervention and control groups.

H. Implementation

A biostatistician generated the allocation sequence. One independent investigator enrolled the participants, and another independent investigator assigned the participants to interventions. The outcome assessor and the biostatistician were kept blinded to the randomisation.

I. Procedure for Data Collection

The participants who gave consent were enrolled in the study. Each session was conducted with a group of 12 to 14 participants. On the first day, the participants were briefed about the sessions and expected learning outcomes. As part of the briefing process, they were explained the confidentiality of the HFPS, the video-assisted lecture sessions and the ethical issues involved. All the participants were introduced to the high-fidelity patient simulator (METIman) in the clinical lab set-up to make them aware of its functions and familiarise them with the handling of the mannequin. An assurance was given to the students that the training course was not part of the evaluation process for the surgical curriculum. The briefing was followed by the first knowledge assessment (Pre-test MCQ) of all the participants. Pre-test MCQ was designed to collect the score of initial background knowledge about tension pneumothorax and its management following the ATLS protocol. The module for the aetiology, pathophysiology and clinical presentation of tension pneumothorax and its steps of management following the ATLS protocol was part of their final year course curriculum. It was taught before they participated in the study. After the Pre-test MCQ session, they were randomized into intervention and control groups consisting of 6 to 7 participants each. For the intervention group, an independent investigator used the high-fidelity simulator (METIman Pre-hospital) to demonstrate the diagnosis and management of tension pneumothorax (Needle Decompression) in an emergency setting. The demonstration time was 20 minutes followed by hands-on training for another 20 minutes. For the control group, a recorded video clip of the identical facilitated simulation session on the diagnosis and management of tension pneumothorax (Needle Decompression) was shown by another investigator. The video demonstration lasted for 20 minutes. This session was followed by a 20-minute interactive discussion session with the faculty. All the participants in both groups were apprised of the importance of aetiology, pathophysiology and clinical presentation in arriving at the diagnosis and management of tension pneumothorax during these interactive teaching sessions. The participants were encouraged to explore how they would manage the stated clinical situation through discussion. The faculty were instructed to emphasize the teaching points related to the outcome of the study. The total duration for both types of teaching was 40 minutes. There were no more additional hands-on practice or video-assisted lecture sessions for the participants during the course of the research study. In the seventh/eighth week, both the intervention and the control groups again participated in the second knowledge assessment (Delayed Post-test MCQ) to assess their gain and retention of knowledge. Delayed Post-test MCQ assessment may minimise the recall bias and test their retained memory better.

Both Pre-test and Post-test knowledge assessments comprised 20 MCQs which were to be completed in 20 minutes. The single-best answer A-type MCQs with five options of answers were prepared following the guidelines framed by the National Board of Medical Examiners (Case & Swanson, 2001). For each correct response, a score of one point was awarded. No negative marking was awarded for incorrect response. Based on the learning objectives, the MCQs were constructed by 6 experts in the field of Surgery, Medicine and Medical Education who were not part of this research study. The MCQs covered the items on pathophysiology, diagnosis, and management of tension pneumothorax, and assessed for knowledge comprehension and knowledge application. The order of the questions was changed between the Pre-test and the Post-test. The MCQ answer sheets were scanned by Konica Minolta FM (172.17.5.12) scanner and graded by using Optical Mark Recognition (OMR) software (Remark Office OMR, version 9.5, 2014; Gravic Inc., USA). Before the main study, a preliminary study involving 56 students was conducted to explore the time management, feasibility, acceptability, and validation of the MCQs (Pal et al., 2021). In the preliminary study, the Pre-test and the Post-test were administered in the first week and the fourth week respectively to note the short-term retention of knowledge. This study is an extension of the preliminary study with a different cohort of students where the Pre-test and the Delayed Post-test were administered in the first week and the seventh/eighth week respectively to determine the medium-term retention of knowledge. The MCQs were reviewed based on the feedback from the preliminary study on the appropriateness of the content, clarity in wording, and difficulty level. The difficulty index and the bi-serial correlation for item discrimination of all MCQs were checked. The value between 30 and 95 in the difficulty index and the bi-serial correlation value > 0.2 were chosen as the accepted standard for this study. 

At the end of the study, the participants in the intervention group were provided with access to the identical video-assisted lecture sessions as designed for the control group. Similarly, the participants in the control group were provided with access to the same HFS sessions. This is to ensure parity between the groups for their professional development of knowledge.

J. Statistical Analysis

SPSS software (version 25) was used for data analysis. The descriptive statistics such as frequency and percentage for categorical data and the mean and standard deviation for the total score of the assessments were calculated. ANCOVA was used to determine the difference in post-test MCQ marks between intervention and control groups with pre-test MCQ marks as a covariate. Intragroup comparison of pre-test and post-test MCQ marks was also done by calculating paired t-test. For intergroup comparison, the effect size – Partial Eta Squared was calculated in ANCOVA. Cohen’s dz was calculated for the comparison of dependent means. The level of significance was set at 0.05 and the null hypothesis was rejected when P < 0.05. We measured the scale-level content validity index (SCVI) and item-level content validity index (ICVI) for the validity and Cronbach alpha for the internal consistency (reliability) of the MCQs. The average values of SCVI and ICVI were 0.94 & 0.89 respectively. The value of Cronbach’s alpha was 0.78.

III. RESULTS

The data that support the findings this RCT study are openly available at https://doi.org/10.6084/m9.figshare.19932053 (Pal et al., 2022).

A. General Data Analysis

There was no difference in the highest Pre-test scores achieved by the participants in both intervention and control groups. The lowest scores recorded in the intervention group were better than the control group in both Pre-test and Post-test. There was a negligible difference between the highest Post-test scores among control and intervention groups (See Table 1).

Test score	Intervention	Control
PRE-TEST
Mean (SE)	12.31 (0.34)	12.23 (0.36)
95% CI for Mean	11.64 – 12.98	11.50 – 12.96
Min – Max	6.0 – 18.0	6.0 – 18.0
POST-TEST
Mean (SE)	13.65 (0.27)	13.60 (0.30)
95% CI for Mean	13.12 – 14.19	12.98 – 14.20
Min – Max	8.0 – 18.0	7.0 – 17.0
Table 1. Highest, lowest and unadjusted mean MCQ scores among intervention and control groups
SE – Standard Error                CI – Confidence Interval Min – Minimum                      Max – Maximum

There was a statistically significant difference between pre-test and post-test MCQ scores among the intervention and control groups. The mean of post-test MCQ scores was significantly higher than the mean of pre-test MCQ scores in both intervention and control groups. The effect size was moderate in both groups (See Table 3).

IV. DISCUSSION

Multiple studies have revealed slight to the modest enhancement of knowledge in simulation-based medical education (SBME) when compared to other instructional teaching methods (Cook et al., 2012; Gordon et al., 2006; Lo et al., 2011; Ray et al., 2012; Ten Eyck et al., 2009). Notwithstanding the increasing popularity of SBME, there is little evidence to conclude that it is superior to other small-group teaching modalities for the acquisition of knowledge (Alluri et al., 2016). The common perception is that knowledge lies at the lowest level of competence in Miller’s model of clinical acumen (Miller, 1990), but it is also important to note that knowledge is the basic foundation of competence and proficiency (Norman, 2009). Theoretically, SBME is advantageous for assessment of both knowledge and skills but there are few studies which directly evaluated the effectiveness of HFS in the assessment of knowledge (McGaghie et al., 2009; Rogers, 2008).

The mean scores of both Pre-test and the Post-test were higher in the intervention group in this study. In comparison, our preliminary study demonstrated that the control group had higher mean MCQ marks than the intervention group in Pre-test whereas at Post-test, the intervention group had higher mean MCQ marks than the control group (Pal et al., 2021).

In our study, there is significant enhancement of knowledge (P < 0.001) in both modes of teaching which corroborates the findings of Alluri et al. (2016). Their RCT study demonstrated that the participants in both the simulation and lecture groups had improved post-test scores (p < 0.05). The comparison of Pre-test and Post-test MCQ scores in our preliminary study also revealed significant higher mean MCQ scores at Post-test than Pre-test in both intervention and control groups (Pal et al., 2021). A study by Couto et al. (2015) showed improved post-test scores in both methods. Similar results were noted in the studies by Chen et al. (2017) and Vijayaraghavan et al. (2019). The finding of a study by Hall (2013) showed a slight increase in post-test scores in both the HFPS and control groups.

A systematic review by La Cerra et al. (2019) revealed that HFS was superior to other teaching methods in improving knowledge and performance. Significant higher scores for participants in the HFS group in the studies by Larsen et al. (2020) and Solymos et al. (2015) demonstrated that HFS may be superior to conventional teaching methods for factual learning. In another study by Bartlett et al. (2021), HFS showed a significant long-term gain in knowledge over traditional teaching methods, but short-term knowledge gain was insignificant. Our study revealed that the Post-test MCQ score was higher in the HFS group but after adjustment of pre-test scores, there was no significant difference in knowledge gain between the control and intervention groups. The findings were similar in our preliminary study where  the intervention group had higher mean change score of MCQ scores than the control group but it was not statistically significant (Pal et al., 2021).

On the other hand, there was no significant knowledge improvement in both simulation and traditional teaching methods as observed in the studies (Corbridge et al., 2010; Kerr et al., 2013; Moadel et al., 2017). The findings of Alluri et al. (2016) also showed no difference in knowledge gain between simulation and lecture-based teaching. The studies by Morgan et al. (2002) and Tan et al. (2008), demonstrated equal efficacy between simulation and conventional lectures. The findings of a study by Kerr et al. (2013) demonstrated that SBME was not beneficial in acquisition and retention of knowledge. There was no significant improvement in knowledge after simulation-based education as revealed by the findings of three RCTs (Cavaleiro et al., 2009; Cherry et al., 2007; Kim et al., 2002).

Despite simulation being effective in acquisition of knowledge, it may not be the most efficient modality when compared to other traditional educational methods (Bordage et al., 2009). There is ample evidence that SBME usually leads to enhancement of knowledge and skills among undergraduate students but its superiority over other conventional teaching methods is yet to be defined (Nestel et al., 2015).

A. Limitations

There is a possibility of potential biases in the form of design, recruitment, sample populations and data analysis that could have influenced the findings. Due to randomization in blocks of two, the allocation of participants may be predictable which may result in selection bias. The confounding factors such as communication between the different groups of students prior to the second MCQ assessment, participants’ recall memory and preparation for the post-test after 7 – 8 weeks need to be considered. As it was a single-centre study which included final-year medical students only, the validity of the findings may not be applicable to other settings.

V. CONCLUSION

Conventional teaching modalities and HFS, when used in conjunction with bedside teaching, may complement clinical practice, leading to higher retention of knowledge. Therefore, more studies are required to measure the efficacy of simulation for a better understanding of the differences that it can make in the acquisition of knowledge. Our study revealed that the efficacy of high-fidelity simulation-based teaching was not superior to video-assisted lecture-based teaching in terms of knowledge acquisition and retention. The substantially higher cost and maintenance associated with HFS need to be considered before planning a teaching-learning activity. It may be used judiciously with conventional teaching when the objectives are mainly knowledge-based. More studies are required to determine its effectiveness and further evaluation as a teaching-learning tool in medical education.

Notes on Contributors

Bikramjit Pal was involved in Conceptualization, Formal Analysis, Literature Review, Methodology, Project administration & Supervision, Data Analysis and Writing (original draft & editing).

Aung Win Thein was involved in Formal Analysis, Literature Review, Methodology, Supervision and Writing (review & editing).

Sook Vui Chong was involved in Literature Review, Methodology, Supervision and Writing (review & editing).

Ava Gwak Mui Tay was involved in Formal Analysis, Literature Review, Supervision and Writing (review & editing).

Htoo Htoo Kyaw Soe was involved in Formal Analysis, Methodology, Data curation, Statistical Analysis and Validation.

Sudipta Pal was involved in Literature Review, Methodology, Formal Analysis, Data curation and Writing (review & editing).

Ethical Approval

Ethical approval was duly obtained from the Ethical Committee / IRB of Manipal University College Malaysia. Informed consent was taken from all the participants. All information about the participants was kept confidential.

Approval number: MMMC/FOM/Research Ethics Committee – 11/2018.

Data Availability

The data that supports the findings of this RCT study are openly available at Figshare repository, https://doi.org/10.6084/m9.figshare.19932053.v2 (Pal et al., 2022).

Acknowledgement

The authors would like to acknowledge the final year MBBS students of Manipal University College Malaysia who had participated in this research project, the faculty of the Department of Surgery, the lab assistants and technicians of Clinical Skills Lab and the Management of Manipal University College Malaysia.

Funding

The researchers had not received any funding or benefits from industry or elsewhere to conduct this study.

Declaration of Interest

The researchers had no conflicts of interest.

References

Accreditation Council for Graduate Medical Education. (2020, July 1). Program requirements for graduate medical education in general surgery. https://www.acgme.org/globalassets/pfassets/programrequirements/440_generalsurgery_2020.pdf.

Alluri, R. K., Tsing, P., Lee, E., & Napolitano, J. (2016). A randomized controlled trial of high-fidelity simulation versus lecture-based education in preclinical medical students. Medical Teacher, 38(4), 404–409. https://doi.org/10.3109/0142159X.2015.1031734

ATLS Subcommittee, American College of Surgeons’ Committee on Trauma, & International ATLS working group. (2013). Advanced trauma life support (ATLS®): The ninth edition. Journal of Trauma and Acute Care Surgery, 74(5), 1363–1366. https://doi.org/10.1097/TA.0b013e31828b82f5

Bartlett, R. S., Bruecker, S., & Eccleston, B. (2021). High-fidelity simulation improves long-term knowledge of clinical swallow evaluation. American Journal of Speech-Language Pathology, 30(2), 673–686. 

Bordage, G., Carlin, B., Mazmanian, P. E., & American College of Chest Physicians Health and Science Policy Committee (2009). Continuing medical education effect on physician knowledge: Effectiveness of continuing medical education: American College of Chest Physicians evidence-based educational guidelines. Chest, 135(3 Suppl), 29S–36S. https://doi.org/10.1378/chest.08-2515

Case, S. M., & Swanson, D. B. (2001). Constructing written test questions for the basic and clinical sciences. 3rd ed. Philadelphia: National Board of Medical Examiners.

Cavaleiro, A. P., Guimarães, H., & Calheiros, F. (2009). Training neonatal skills with simulators? Acta Paediatrica, 98(4), 636–639.    

Chen, T., Stapleton, S., Ledford, M., & Frallicciardi, A. (2017). Comparison of high-fidelity simulation versus case-based discussion on fourth- year medical student performance. Western Journal of Emergency Medicine: Integrating Emergency Care with Population Health, 18(5.1). https://escholarship.org/uc/item/5k73f4qc

Cherry, R. A., Williams, J., George, J., & Ali, J. (2007). The effectiveness of a human patient simulator in the ATLS shock skills station.  Journal of Surgical Research, 139(2), 229–235. https://doi.org/10.1016/j.jss.2006.08.010

Cook, D. A., Brydges, R., Hamstra, S. J., Zendejas, B., Szostek, J. H., Wang, A. T., Erwin, P. J., & Hatala, R. (2012). Comparative effectiveness of technology-enhanced simulation versus other instructional methods: A systematic review and meta-analysis. Simulation in Healthcare: The Journal of the Society for Simulation in Healthcare, 7(5), 308–320. https://doi.org/10.1097/SIH.0b013e3182614f95

Corbridge, S. J., Robinson, F. P., Tiffen, J., & Corbridge, T. C. (2010). Online learning versus simulation for teaching principles of mechanical ventilation to nurse practitioner students. International Journal of Nursing Education Scholarship, 7(1), Article 12. https://doi.org/10.2202/1548-923X.1976

Couto, T. B., Farhat, S. C., Geis, G. L., Olsen, O., & Schvartsman, C. (2015). High-fidelity simulation versus case-based discussion for teaching medical students in Brazil about pediatric emergencies. Clinics (Sao Paulo, Brazil), 70(6), 393–399. https://doi.org/10.6061/clinics/2015(06)02

Davies, R. (2008). The Bologna process: the quiet revolution in nursing higher education. Nurse Education Today, 28(8), 935–942. https://doi.org/10.1016/j.nedt.2008.05.008

Ellis, P. (2010). The essential guide to effect sizes: Statistical power, meta-analysis, and the interpretation of research results. Cambridge University Press. https://doi.org/10.1017/CBO9780511761676

Faul, F., Erdfelder, E., Lang, A. G., & Buchner, A. (2007). G*Power 3: a flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175–191. https://doi.org/10.3758/bf03193146

Gordon, J. A., Shaffer, D. W., Raemer, D. B., Pawlowski, J., Hurford, W. E., & Cooper, J. B. (2006). A randomized controlled trial of simulation-based teaching versus traditional instruction in medicine: A pilot study among clinical medical students. Advances in Health Sciences Education , 11(1), 33–39. https://doi.org/10.1007/s10459-004-7346-7

Hall, R. M. (2013). Effects of high fidelity simulation on knowledge acquisition, self-confidence, and satisfaction with baccalaureate nursing students using the solomon-four research design [Doctoral dissertation, East Tennessee State University]. East Tennessee State University Higher Education Commons. https://dc.etsu.edu/etd/2281

Kerr, B., Hawkins, T. L., Herman, R., Barnes, S., Kaufmann, S., Fraser, K., & Ma, I. W. (2013). Feasibility of scenario-based simulation training versus traditional workshops in continuing medical education: a randomized controlled trial. Medical Education Online, 18(1), Article 21312. https://doi.org/10.3402/meo.v18i0.21312

Kiernan, L. C. (2018). Evaluating competence and confidence using simulation technology. Nursing, 48(10), 45–52. https://doi.org/10.1097/01.NURSE.0000545022.36908.f3

Kim, J. H., Kim, W. O., Min, K. T., Yang, J. Y., & Nam, Y. T. (2002). Learning by computer simulation does not lead to better test performance on advanced cardiac life support than textbook study. The Journal of Education in Perioperative Medicine, 4(1), Article E019. 

Kolb, D. A. (1984). Experiential learning: Experience as the source of learning and development. Prentice Hall. 

La Cerra, C., Dante, A., Caponnetto, V., Franconi, I., Gaxhja, E., Petrucci, C., Alfes, C. M., & Lancia, L. (2019). Effects of high-fidelity simulation based on life-threatening clinical condition scenarios on learning outcomes of undergraduate and postgraduate nursing students: A systematic review and meta-analysis. BMJ Open, 9(2), Article e025306. https://doi.org/10.1136/bmjopen-2018-025306

Larsen, T., Jackson, N., & Napolitano, J. (2020). A comparison of simulation-based education and problem-based learning in pre-clinical medical undergraduates. MedEdPublish, 9(1), Article 172.

Lo, B. M., Devine, A. S., Evans, D. P., Byars, D. V., Lamm, O. Y., Lee, R. J., Lowe, S. M., & Walker, L. L. (2011). Comparison of traditional versus high-fidelity simulation in the retention of ACLS knowledge. Resuscitation, 82(11), 1440–1443. https://doi.org/10.1016/j.resuscitation.2011.06.017

McGaghie, W. C., Siddall, V. J., Mazmanian, P. E., Myers, J., & American College of Chest Physicians Health and Science Policy Committee (2009). Lessons for continuing medical education from simulation research in undergraduate and graduate medical education: Effectiveness of continuing medical education: American College of Chest Physicians Evidence-Based Educational Guidelines. Chest, 135(3 Suppl), 62S–68S. https://doi.org/10.1378/chest.08-2521

Miller, G. E. (1990). The assessment of clinical skills/competence/performance. Academic Medicine, 65(9), S63–S67. https://doi.org/10.1097/00001888-199009000-00045

Moadel, T., Varga, S., & Hile, D. (2017). A prospective randomized controlled trial comparing simulation, lecture and discussion-based education of sepsis to emergency medicine residents. Western Journal of Emergency Medicine: Integrating Emergency Care with Population Health, 18(5.1). https://escholarship.org/uc/item/0132981t

Morgan, P. J., Cleave-Hogg, D., McIlroy, J., & Devitt, J. H. (2002). Simulation technology: A comparison of experiential and visual learning for undergraduate medical students. Anesthesiology, 96, 10–16. https://doi.org/10.1097/00000542-200201000-00008

Nestel, D., Harlim, J., Smith, C., Krogh, K., & Bearman, M. (2015). Simulated learning technologies in undergraduate curricula: An evidence check review for HETI.

Norman, G. (2009). The American College of Chest Physicians evidence-based educational guidelines for continuing medical education interventions: a critical review of evidence-based educational guidelines. Chest, 135(3), 834–837. https://doi.org/10.1378/chest.09-0036

Pal, B., Chong, S. V., Thein, A. W., Tay, A. G., Soe, H. H., & Pal, S. (2021). Is high-fidelity patient simulation-based teaching superior to video-assisted lecture-based teaching in enhancing knowledge and skills among undergraduate medical students? Journal of Health and Translational Medicine, 24(1), 83-90. https://doi.org/10.22452/jummec.vol24no1.14

Pal, B., Thein, A. W., Chong, S. V., Tay, A., Htoo, H., & Pal, S. (2022). A randomized controlled trial study to compare the effectiveness of high-fidelity based teaching with video-assisted based lecture teaching in enhancing knowledge [Dataset]. Figshare. https://doi.org/10.6084/m9.figshare.19932053

Ray, S. M., Wylie, D. R., Shaun Rowe, A., Heidel, E., & Franks, A. S. (2012). Pharmacy student knowledge retention after completing either a simulated or written patient case. American Journal of Pharmaceutical Education, 76(5), 86. https://doi.org/10.5688/ajpe76586

Rogers, D. A. (2008). The role of simulation in surgical continuing medical education. Seminars in Colon and Rectal Surgery, 19(2), 108-114. https://doi.org/10.1053/j.scrs.2008.02.007

Solymos, O., O’Kelly, P., & Walshe, C. M. (2015). Pilot study comparing simulation-based and didactic lecture-based critical care teaching for final-year medical students. BMC Anesthesiology, 15, Article 153. https://doi.org/10.1186/s12871-015-0109-6

Tan, G. M., Ti, L. K., Tan, K., & Lee, T. (2008). A comparison of screen-based simulation and conventional lectures for undergraduate teaching of crisis management. Anaesthesia and Intensive Care, 36(4), 565–569.

Ten Eyck, R. P., Tews, M., & Ballester, J. M. (2009). Improved medical student satisfaction and test performance with a simulation-based emergency medicine curriculum: a randomized controlled trial. Annals of Emergency Medicine, 54(5), 684–691. https://doi.org/10.1016/j.annemergmed.2009.03.025

Vijayaraghavan, S., Rishipathak, P., & Hinduja, A. (2019). High-fidelity simulation versus case-based discussion for teaching bradyarrhythmia to emergency medical services students. Journal of Emergencies, Trauma, and Shock, 12(3), 176–178. https://doi.org/10.4103/JETS.JETS_115_18

Yang, Y., & Liu, H. P. (2016). Systematic evaluation influence of high-fidelity simulation teaching on clinical competence of nursing students. Chinese Nursing Research, 30(7), 809–814. https://caod.oriprobe.com/articles/47628779/Systematic_evaluation_influence_of_high_fidelity_s.htm

*Bikramjit Pal
RCSI & UCD Malaysia Campus (RUMC),
4 Jalan Sepoy Lines,
Georgetown, Penang, 10450, Malaysia
+6042171908-1908 (Ext)
Email: bikramjit.pal@rcsiucd.edu.my

Submitted: 16 March 2022
Accepted: 26 May 2022
Published online: 4 April, TAPS 2023, 8(2), 57-65
https://doi.org/10.29060/TAPS.2023-8-2/OA2778

Vijay Kautilya Dayanidhi¹, Arijit Datta², Shruti P Hegde³ & Preeti Tiwari⁴

¹Department of Forensic Medicine, Medicine, Manipal Tata Medical College, MAHE, India; ²Department of Forensic Medicine, Medicine, Pramukhswamy Medical college, India; ³Department of Ophthalmology, Medicine, Manipal Tata Medical College, MAHE, India; ⁴Department of Community Medicine, Medicine, Pramukhswamy Medical college, India

Abstract

Introduction: Summative assessments play a major role in shaping the student’s learning. There is little literature available on validity of summative assessment question papers in Forensic Medicine & Toxicology. This study analyses 30 question papers from 6 reputed universities for content validity.

Methods: A retrospective cross-sectional record-based observational study was conducted where 30 university summative question papers in Forensic Medicine & Toxicology from 6 universities across India were evaluated for content validity. The learning domain assessed, the type of questions asked, and sampling of the content was compared and presented in the results.

Results: From the results of the study, it was noted that 80% weightage was allotted to recall in most papers and only one paper tested for application. 70 to 80% of the marks were allotted to Forensic Pathology leading to disproportionate sampling. Core areas in Toxicology and Medical Jurisprudence were sparsely assessed. 

Conclusion: The content validity of the summative question papers in Forensic Medicine and Toxicology was unsatisfactory, emphasising the need for evaluation of the clarity and efficacy of the blueprints being used by the universities. Faculty training to motivate and influence a change in the mindset is necessary to bring about a course correction.

Keywords:           Forensic Medicine & Toxicology, Summative Assessments, University Assessments, Blueprint, Content Validity, Learning Domains

Practice Highlights

• Content validity of the Forensic Medicine & Toxicology university exam question papers form six universities was studied.
• It was observed that certain subtopics like Forensic Pathology over time have been over value (80% Weightage).
• Core areas in Medical Jurisprudence and toxicology like substance abuse, environmental toxicology, and pharmaceutical toxicity have been undervalued.
• None of the QPs analysed tested for application. Most items in the assessment tested only recall.
• The blueprints for the paper setters, considering the competencies to be assessed must be designed and validated.

I. INTRODUCTION

Reflecting on our learning experience during MBBS, we realised that we have always had issues with the examination system. The questions are vague and clustered around few important topics. Undergraduate students look up to previous examination question papers as references to decide the stake to be allotted to the topics while preparing for examination. Invariable all students attempt to predict the examination pattern and allot appropriate time and efforts to different subjects, skills, and topics. This reiterates George E Miller’s quote “Assessment drives Learning”. Summative assessments need to be planned appropriately as medicine has high stakes (Amin et al., 2006). Properly designed and executed assessments are known to have a “positive steering effect on the student’s learning. They are also needed to evaluate the programs. Improper assessments can drive a hidden curriculum leading to a completely unintended outcomes (Amin & Khoo, 2003, pp. 260).

Competency Based Medical Education (CBME) model being adopted in India as per the new Graduate Medical Education Regulations 2019, has attempted to bring about a radical change in the educational process. Undergraduate examinations in India are shifting towards a criteria-based process (Aggarwal & Agarwal., 2017; Bhattacharya et al., 2017; Mehta & Kikani, 2019). Outcome based education demands that the examinations be designed to sample and evaluate specific competencies prescribed. The success of these models strongly depended on the validity of the examination process. Summative assessments require that the assessment tool be validated. Key outcomes need to be tested (Amin & Khoo, 2003, pp. 260; McAleer, 2001). Content validity and construct validity are two very important aspect that support the effectiveness of an assessment. Content validity tests the representativeness of the learning objectives in the assessment tool and construct validity represents the congruence of the assessment tool with the intended purpose (Amin & Khoo, 2003, pp. 260).

Forensic Medicine and Toxicology in India, trains the undergraduate to apply their knowledge gained in Medicine for the benefit of law. It is a culmination of Forensic Pathology, Medical Jurisprudence and Toxicology put together. Its key objective is to empower Indian Medical Graduates in handling Medical Legal issues and critically apply their medical skills in delivering justice. Emphasis is also placed on training in etiology, identification, and management of Poisoning ( Sharma et al., 2005). Studies on student perception suggest that teaching is significantly teacher centric and theory oriented. Skill training in Medical Jurisprudence and Toxicology is significantly neglected. Students allege though they value the subject, they spend less time as only select concepts are emphasised (Gupta et al., 2017; Parmar, 2018; Sharma et al., 2005; Sudhan & Raj, 2019). As the new CBME UG curriculum 2019 is being rolled out it is necessary that deficiencies in the traditional curriculum be identified in order to deliver an efficient and effective Forensic Medicine & Toxicology curriculum (National Medical Commission, 2018).

Summative theory exams inherently have a challenge with distribution of the items being tested (Aggarwal & Agarwal, 2017;  Amin et al., 2006; Amin & Khoo, 2003, pp. 260; Bhattacharya et al., 2017). Validity of the content being tested in examination is always in question. Selecting appropriate questions, question types and domain can make all the difference in the validity of the examination (Amin et al., 2006; Amin & Khoo, 2003, pp. 260; McAleer, 2001). Particularly in Forensic Medicine which is purely application-based course, testing critical thinking and synthesis is necessary. This is found wanting in the traditional curriculum (Parmar, 2018; Sharma et al., 2005; Sudhan & Raj, 2019). Published literature on systematic analysis of summative assessment question papers in Forensic Medicine & Toxicology are sparsely available.  In this study, we have analysed and compared undergraduate summative examination question papers of Forensic Medicine & Toxicology from six reputed universities all over India for the distribution of content tested, Domain of learning and Construct of the question.

II. METHODS

A retrospective cross-sectional record-based observational study was conduct at Government Medical College, Bharatpur after obtaining ethical approval from the Institutional Ethics Committee between October to December 2020. For the study, 30 summative exam question papers from six reputed medical universities were selected based on the availability of the University question papers in public domain. The last five-year (2016-20) undergraduate question papers in Forensic Medicine & Toxicology were collected from the university websites and the college records from constituent colleges after thorough web search. The names of the universities have been kept anonymous during the analysis of results. All the data was collected form sources in public domain hence explicit consent was not taken.  Two of the selected universities were based in North India and four universities were based in South India. The identity of the Medical Universities was kept confidential during the analysis of the question papers. 

The Summative theory examination in Forensic Medicine & Toxicology as per the Medical Council of India (MCI) regulations consists of one theory paper of minimum 40 marks. The question paper consists of essay type questions and objective questions like very short answer questions or Multiple-choice questions depending on the universities (National Medical Commission, 2018).

For analysis, the questions were categorised based on the question type as LEQ (Long Essay Question), SAQ (Short Answer Question) & VSAQ (Very Short Answer Question including MCQs). The Questions were also categorised based on the domain of learning as Recall Based, Comprehension Based and Application Based Questions.

The Topics in Forensic Medicine & Toxicology can be broadly subdivided into Medical Jurisprudence, Forensic Pathology and Toxicology. These were further subdivided as Six Categories as Legal Procedure, Medical jurisprudence, Forensic Pathology, Forensic Psychiatry, Lab Technique, emerging trends, and Toxicology (Medical Council of India, 1997). Percentage of marks allotted to each of these topics was analysed in each of the papers.

Further, Forensic Pathology was Sub divided into Subtopics like Identification, Postmortem Changes, Mechanical Injuries, Mechanical Asphyxia, Thermal Deaths, Sexual Offences and Medico Legal issues related to Pregnancy, Delivery, Abortion. Toxicology was Sub divided into General Toxicology, Chemical Toxicology, Drug, Pharmacy & Substance abuse Toxicology, Bio toxicology (Medical Council of India, 1997).  Percentage allotment of Marks in each of the question papers was analysed for each of the subtopics.

The data thus collected was tabulated in an Excel Sheet and the percentage distribution of marks in various subtopics noted. The SPSS Statistical Software (IBM SPSS Statistics for Windows, Version 23.0) was used to analyse the data. Radar Graphs and line graphs were plotted to represent and compare pattern of distribution of marks in various topics in each question paper. The type of questions asked, the weightage allotted to the subtopics were compared keeping in mind the expected outcomes in the Forensic Medicine & Toxicology curriculum proposed by National Medical Commission and Medical Council of India for content validity (Medical Council of India, 1997; National Medical Commission, 2018). The learning domain targeted in the questions was compared for construct validity of the question papers.

III. RESULTS

In this study, five question papers(n=30) from each university(n=6) were analysed and compared. The data that supports the findings of this study are openly available in Figshare at https://doi.org/10.6084/m9.Fig share.19367864 (Kautilya et al., 2022).

As regulated the university Summative examination in Forensic Medicine & Toxicology consists of one theory assessment and one practical assessment (Medical Council of  India, 1997; National Medical Commission, 2018). The theory paper is allotted a minimum of 40 marks. Five universities conducted exam for 40 marks and one university paper was of 100 marks. All question papers had three types of questions, namely Essay questions (Long Answer Questions-LAQs) of 8 to 10 marks each, short essays (Short Answer questions-SAQs) of 3-5 marks each and Objective questions (like Multiple choice questions-MCQs or Very short answer questions- VSAQs) of 1-2 marks each. Table 1 presents the percentage distribution of the marks allotted to each question type.

Submitted: 19 August 2022
Accepted: 5 December 2022
Published online: 4 April, TAPS 2023, 8(2), 47-56
https://doi.org/10.29060/TAPS.2023-8-2/OA2869

Edyta Truskowska¹, Yvonne Emmett² & Allys Guerandel¹

¹Department of Psychiatry, Faculty of Medicine, University College Dublin, Ireland; ²National College of Ireland, Ireland

Abstract

Introduction: Digital Badges have emerged as an alternative credentialing mechanism in higher education. They have data embedded in them and can be displayed online. Research in education suggests that they can facilitate student motivation and engagement. The authors introduced digital badges in a Psychiatry module in an Irish University. Completion of clinical tasks during the student’s clinical placements, which were previously recorded on a paper logbook, now triggers digital badges. The hope was to increase students’ engagement with the learning and assessment requirements of the module.

Methods: The badges – gold, silver and bronze level – were acquired on completion of specific clinical tasks and an MCQ. This was done online and student progress was monitored remotely. Data was collected from the students at the end of the module using a questionnaire adapted from validated questionnaires used in educational research.

Results: The response rate was 68%. 64% of students reported that badges helped them achieve learning outcomes. 68% agreed that digital badges helped them to meet the assessment requirements. 61% thought badges helped them to understand their performance. 61% were in favour of the continuing use of badges. Qualitative comments suggested that badges should contribute to a higher proportion of the summative mark, and identified that badges helped students to structure their work.

Conclusions: The findings are in keeping with the literature in that engagement and motivation have been facilitated. Further evaluation is required but the use of badges as an educational tool is promising.

Keywords:           Medical Education, Digital Badges, Students’ Engagement, Continuous Assessment Gamification, Health Profession Education

Practice Highlights

• Digital badges may enhance student engagement.
• Digital badges may promote motivation for learning.
• Evaluation of digital badges using a questionnaire with ordinal analysis of data and coding of free comments.
• Majority of students reported working harder than in a non-gamified module.
• Digital badges provided structure and direction to the student’s learning.

I. INTRODUCTION

Educational research recognises student engagement as valuable and as having significant impact on their learning (Mandernach, 2015). While searching for tools impacting on engagement, educators observed that games have been good at engaging players for decades, through their ability to sustain players’ attention and keep them motivated throughout the games (Przybylski et al., 2010). This level of engagement is desirable to both students and educators. This achievable level of engagement in gaming strategies has led to the exploration of its use in education. Elements from game design applied in non-game contexts to influence, engage and motivate individuals and groups have resulted in the development of a new field known as gamification (Deterding et al., 2011).

Digital badges are common tools of gamification (Barata et al., 2013). They are frequently used by game designers and in recent years also by educators. A digital badge used in education can be a validated symbol of academic achievement, accomplishment, skill, quality or interest (HASTAC, n.d.). Digital badges are digital images obtained through the completion of some pre-specified goals that are annotated with metadata and that can be displayed online (Hensiek et al., 2017). In higher education badges have been used to recognise a student’s participation in a learning activity, to help students explicitly and visually capture and monitor progress made on learning tasks, to recognise the achievement of skills and competencies and to serve as a means of certifying these achievements. They are reported to have a positive effect on the learners’ motivation if they are considered as awards or if they trigger competition among peers (Yildirim et al., 2016).  

It appears that the value of digital badges depends on their design (when awarded, for what, and what they mean). For example, the use of badges as credentials only, has been criticised for focusing exclusively on extrinsic motivating factors, which have less impact on engagement than intrinsic ones (Seaborn & Fels, 2015). This is why combining the use of badges, as credentials as well as using them within the assessment process appears to be a better idea. Considering, that assessment has proven to have the most impact on effective learning, the use of badges during structured assessment has been favoured by educators (Abramovich, 2016; Rolfe & McPherson, 1995). The assessments that have potential to generate formative and summative feedback are presented as particularly useful (Armour-Thomas & Gordon, 2013). Digital Badges represent a viable alternative to existing methods of assessment in educational institutions and in the work environment (Dowling-Hetherington & Glowatz, 2017). It was also noted that access to regular feedback (broadly available in games) is helpful to learners. Students that are given opportunities to complete a task and learn from their mistakes do better in overall assessment. Games are a great example of the design where a player learns through feedback, gets better and eventually becomes successful (McGonigal, 2011). Similarly, the literature states that the use of badges has potential to offer a sort of “covert assessment”, meaning that students can approach a task as if it was a game. This helps to maintain the benefits of assessment while minimizing the potential for unhelpful levels of test anxiety (McGonigal, 2011) (Abramovich et al., 2013).

Another advantage given for the use of digital badges is their potential for remote monitoring of students’ progress and their difficulties by instructors and tutors (Huang & Soman, 2013). There is a growing momentum for the use of digital badges as an innovative instruction and credentialling strategy in higher education (Noyes et al., 2020).

In our University, Psychiatry is taught as a 10-credit module to both undergraduate and graduate entry students in the final stage of their degree in medicine. Typically, approximately 240 students are taught the module in four different groups: two groups in the spring and two groups in autumn, for six weeks at a time. Face to face teaching is centralized on Mondays and Fridays. Clinical teaching is delivered during the rest of the week and takes place in multiple different clinical centres. The overall assessment of this module comprised a continuous assessment with specific formative and summative tasks recorded in a paper logbook. The summative tasks were worth 20% of their overall assessment.

Standardising the student clinical experience, engaging them in their clinical placements and monitoring their attendance and progress can be challenging. The paper logbook/portfolio we were using was inadequate in that it did not allow for central monitoring of progress and often the difficulties students were encountering came to the attention of the teaching staff only when the logbook was handed over at the end of the module. Provision of feedback on progress was also limited. We felt that, in particular, students that were slow to progress were missing potential remediation before the summative assessments. We also encountered practical difficulties such as lost logbooks that affected the continuous assessment process.

We felt that digital badges offered a way of monitoring attendance and participation in tasks remotely, providing feedback, facilitating remediation and allowing students’ gauge how they are doing in relation to their peers while optimizing engagement in the clinical placements and structuring the learning to sustain progress through the module. We introduced and piloted the use of digital badges in the Psychiatry module as part of the continuous assessment. We carried out a descriptive study to appraise the potential usefulness of digital badges as part of our teaching strategy.

II. METHODS

A. Course Design

Students taking the 6-week Psychiatry module start their clinical placement on day 2 of the Module. Each week, students participate in their continuous assessment in order to collect their weekly badge. To acquire a badge, they need to complete and upload specific clinical tasks including formative clinical cases scheduled for them, to upload a Clinical Placement Form signed by the consultant on the team they are attached to and do an online multiple-choice question test at the end of the week. As all of this is done online their progress can be monitored remotely by the teaching team independent of the location of their clinical placement. Collecting their weekly badges provides them with 5% of their continuous assessment mark. The other marks for continuous assessment come from a summative clinical case (90%) and a reflective assignment (5%). Continuous assessment contributes to 20% of overall assessment mark.

B. Badges Design

Tutors in Psychiatry in conjunction with the University’s Teaching and Learning Department created Badges. It was part of an institution-wide digital badging pilot project (UCD Teaching and learning, 2017). It was agreed that there would be three types of badges -bronze, silver and gold – obtained and displayed on the university’s virtual learning environment (currently, Brightspace). As noted above students receive a digital badge on completion of assigned tasks, which are part of their continuous assessment. The type of badge awarded depends on the MCQ score and it is displayed on the student’s Blackboard profile. Figure 1 depicts the process. It shows that all badges are contributing to 5% of the module continuous assessment. Every week students receive information as to what percentage of the group has acquired a bronze, silver or gold badge so they have an idea of their performance in relation to that of the rest of the group.

Figure 1. Step-by-step the process of getting/awarding a badge

C. Questionnaire

The ‘Digital Badges Experience Survey’ questionnaire was designed based on previously described surveys: the ARCS Badge Motivation Survey (Foli et al., 2016) and the Badge Opinion Survey (Abramovich et al., 2013) with some additional questions suggested by the literature on digital badges. The authors and faculty members identified and agreed on the following constructs as being relevant to our teaching delivery and our study: previous knowledge of digital badges (items 1 & 2), their meaning and relevance to students (items 22, 13, 14, 12, 8 & 5) motivation and engagement (items 3, 4 & 23), relevance to assessment and feedback (items 11, 9, 10, 24, 18 & 19), their use in structuring learning (items 15, 6, 7 & 16), self-efficacy (items 17, 20, 21 & 25), social context implication (items 26, 27 & 28). Under each construct, items from the above questionnaires were discussed and agreement reached on the ones to be used, altered or added assessing relevance and acceptability for our aims and teaching context.

Our survey consisted of 30 items with answers displayed on a seven-point Likert Scale. The 31^st (final) question required dichotomous (yes/no) answer with space for respondents to explain their reasons for it. We also provided for free commenting from students (Questionnaire is available in Table 1).

Please rate your agreement with each of the statements using the following scale:

Strongly Agree	Agree	Somewhat Agree	Neutral	Somewhat Disagree		Disagree	Strongly Disagree
+3	+2	+1	0	– 1		-2	-3
Please circle one number for each statement					Strongly Agree			Strongly Disagree

1. I knew what digital badges were before I began this module.	+3	+2	+1	0		-1	-2	-3
2. I have earned digital badges before beginning this module.	+3	+2	+1	0		-1	-2	-3
3. I felt motivated to complete the module because I was earning digital badges.	+3	+2	+1	0		-1	-2	-3
4. Compared to other modules on my programme, the digital badges motivated me to work harder.	+3	+2	+1	0		-1	-2	-3
5. The digital badges helped me to understand the learning outcomes for this module.	+3	+2	+1	0		-1	-2	-3
6. The digital badges helped me to achieve the learning outcomes for this module.	+3	+2	+1	0		-1	-2	-3
7. The badge helped draw my attention to the clinical seminars.	+3	+2	+1	0		-1	-2	-3
8. The digital badges helped me to understand the content of this module.	+3	+2	+1	0		-1	-2	-3
9. The digital badges helped me to understand the assessment requirements for this module.	+3	+2	+1	0		-1	-2	-3
10. I was more aware of the module continuous assessment requirements because I would be earning digital badges.	+3	+2	+1	0		-1	-2	-3
11. Because I was earning digital badges, I knew the continuous assessment requirements were important.	+3	+2	+1	0		-1	-2	-3
12. Earning digital badges made a difference in how I viewed completing the continuous assessment requirements.	+3	+2	+1	0		-1	-2	-3
13. Earning badges made the assignments more significant to me.	+3	+2	+1	0		-1	-2	-3
14. The badges increased how relevant the assignments were.	+3	+2	+1	0		-1	-2	-3
15. The digital badges helped me to structure my work in this module.	+3	+2	+1	0		-1	-2	-3
16. The digital badges helped me to meet the assessment requirements of this module.	+3	+2	+1	0		-1	-2	-3
17. The badges increased my confidence that I could demonstrate the content of my knowledge.	+3	+2	+1	0		-1	-2	-3
18. The digital badges helped me to understand my performance in this module.	+3	+2	+1	0		-1	-2	-3
19. The digital badges helped me to understand my progress through the module.	+3	+2	+1	0		-1	-2	-3
20. The badges were symbols that I had mastered content.	+3	+2	+1	0		-1	-2	-3
21. The badges increased my overall level of satisfaction with completing the continuous assessment requirements.	+3	+2	+1	0		-1	-2	-3
22. By earning the badges I was more fulfilled as a student by completing the assessment requirements.	+3	+2	+1	0		-1	-2	-3
23. The digital badges made me want to keep on working.	+3	+2	+1	0		-1	-2	-3
24. I understand why I earned all of my badges.	+3	+2	+1	0		-1	-2	-3
25. The badges I earned represent what I learned on this module.	+3	+2	+1	0		-1	-2	-3
26. I talked to others about the badges I earned.	+3	+2	+1	0		-1	-2	-3
27. I compared the badges I earned with others’ on the module.	+3	+2	+1	0		-1	-2	-3
28. The potential to earn digital badges at gold, silver and bronze levels made me feel competitive.	+3	+2	+1	0		-1	-2	-3
29. I think digital badges are a good addition to the programme.	+3	+2	+1	0		-1	-2	-3
30. I would like to earn digital badges in other modules on my programme.	+3	+2	+1	0		-1	-2	-3
31. I think the badges are helpful and should be used in the coming years: tick as appropriate and give 3 reasons why	Yes ☐				No ☐
32. Any other comments

Thank you for your participation

Table 1. Digital Badges Experience Survey

D. Participants

The questionnaires were distributed to all the students of final year of Medicine in our university at the beginning of the final or sixth week of the module and collected by their tutors. Informed verbal consent was obtained from study participants. As described above, the course was run four times in one academic year, and we collected data from all four groups of students: two in the spring and two in the autumn.

E. Data Collection and Analysis

As noted above, questionnaires were distributed and collected by tutors at the beginning of the sixth (last) week of the Psychiatry Course. The level of student’s agreement with various statements was marked on the 7-point Likert-type scale. Those data were uploaded to Excel. Data from Likert scales can be analysed as ordinal as well as interval data (Sullivan & Artino 2013), (Norman, G. 2010) and we have considered both options. We concluded that using descriptive statistics such as the mean in relation to students’ opinions had limited value (Sullivan & Artino 2013), (Knapp 1990). This is why we decided to analyse our data as ordinal. To simplify the answers, we organised them into three groups: “agreed”, “neutral”, “disagreed”.

Students’ comments were entered into an excel sheet and analysed by two independent researchers. The comments relating to the use of badges in teaching of Psychiatry were coded according to topics, which were identified and agreed upon by the two independent researchers (Johnson & LaMontagne, 1993), (Sundler et al., 2019). Topics were further codified as positive or negative. This way of coding is described and performed in more details in other studies (Quesenberry et al., 2011).

III. RESULTS

A. Demographics

161 out of 237 students completed questionnaires giving a 68% response rate. The response rate was 75% in the first half (from springtime) of the students and 61% in the second (autumn rotation).

B. Analysis of Answers

65% of students had no previous knowledge of digital badges and 93% had never earned a badge before the module as per items 1 & 2 of questionnaire.

1) Meaning and relevance: Item 22: 48% of respondents agreed that by earning the badges they felt more fulfilled as a student when completing the assessment requirements. 31% disagreed. 45% agreed and 39% disagreed that earning badges made the assignments more significant to them (item 13) and similarly only 42% felt the badges increased the sense of how relevant the assignments were, while 40 % disagreed with this view as per item 14.

Earning digital badges made a difference in how 59% of students viewed completing the continuous assessment requirements, (item 12). 29% disagreed with this. 68% students felt that the digital badges helped them to understand the content of this module and 18% disagreed, (item 8). 66% students agreed and 24% disagreed about the fact that digital badges helped them to understand the learning outcomes for this module (item 5).

2) Motivation and engagement: 51% of students that responded felt motivated to complete the module because they earned a digital badge (item 3). 33% did not agree with this. The possibility of earning a digital badge motivated 43% of respondents to work harder (item 4). 39% disagreed with this. The digital badges made 45% of students want to keep on working, while 29% were not impacted (item 23).

3) Assessment and feedback: Item 11: 50% felt that because they were earning digital badges, they knew that the continuous assessment requirements were important. 36% disagreed. The digital badges also helped 69% respondents to understand the assessment requirements for this module but not so for the 15% respondents (item 9). Out of all respondents, 78% were more aware of the module continuous assessment requirements because of the digital badges, and only 15% disagreed with that (item 10). Item 24: As many as 74% of all respondents did and 14% did not understand why they earned their badges. The digital badges helped 61% of students to understand their performance in this module (item18). 27% did not find that badges helped in that way. Similarly, the digital badges helped 61% of students to understand their progress through the module. 22% disagreed with this (item 19).

4) Structure: Out of all respondents 50% agreed and 33% disagreed that the digital badges helped them to structure their work in the module (item 15). 64% of respondents agreed and 21% disagreed that the digital badges helped them to achieve the learning outcomes for this module (item 6). Similarly, the badge helped draw attention to the clinical seminars for 57% of respondents, but not so for 25%. 68% of students (vs 19% who have disagreed) felt that the digital badges helped them to meet the assessment requirements of this module (item 16).

5) Self-efficacy: Item 17: 48% of students agreed (vs 33% who disagreed) that earning the digital badges made them more confident that they could demonstrate the content of their knowledge. Item 20: 41% of all respondents agreed that the badges were symbols of mastering the content of the module. A similar number (40%) of students disagreed and 17% stayed neutral. 59% found the badges increased their overall level of satisfaction with completing the continuous assessment requirements (item 21). 27% disagreed with this. 44% of students agreed and 41% disagreed with the statement that the badges they earned represented their learning in the module (item 25).

6) Social context and competitiveness: 42% of students did and 44% did not talk to others about the badges they earned (item 26). The potential to earn digital badges at gold, silver and bronze levels made 49% of students more competitive (item 28). 39% disagreed with this. 29% of students did and 56% did not compare the badges they earned with others on the module (item 27).

7) Overall: 56% agreed that digital badges were a good addition to the program. 61% of students found digital badges helpful and felt that they should be used in the future. 31% did not agree with the statement and 8% did not answer.

Students’ opinions: 136 students did and 25 students did not write any comment. Students’ comments related either to one or to several topics. Students were positive about the use of badges (in various topics) 134 times (See Figure 1) and negative/critical 106 times (See Figure 2). It is important to note however that as many as 67 out of the 106 negative comments related to the low value of the badges.

Figure 2 depicts information about positive topics included in students’ comments. 50 students liked the structure and focus that the badge system provided. Twenty-eight students found badges motivating and 24 valued feedback they received in the process. A number of students found the whole process enjoyable, rewarding and fun (See Figure 2). Figure 3 indicates the negative opinions. The most frequent topic of all was repeated 67 times as was related to the value of badges (See Figure 3). Figure 4 shows some comments made by students in the free comment box provided on the survey (See Figure 4).

Figure 2. Frequency of positive comment grouped by topic

Figure 3. Frequency of negative comment grouped by topic

Figure 4. Comments written by students

In summary, the majority of the students liked the way Digital Badges were used in the teaching of Psychiatry, however both groups (those that liked and those that dislike badges) criticized them for their low value of the overall assessment.

IV. DISCUSSION

Students met digital Badges piloted in the teaching of Psychiatry to Medical Students of our university originally with apprehension. The majority of the students had never heard of digital badges and have never earned a digital badge before this module. However, data from the study looking at students’ perception of the use of digital badges in medical education provided encouraging results.

A. Sense of Reward

Students found badges rewarding yet complained about the small value of the badges. This reflected our design intention in which we wanted to support and engage students rather than focus on extrinsic motivating factors such as sense of reward.

As mentioned above the badge was awarded for completion of weekly tasks, and the acquisition of a badge was functioning more as a method of feedback to students rather than for grading. However, a number of students complained about this, and reported a sense of frustration and a lack of motivation to try harder when the assessment value of the badge was so low.

Nevertheless, our design was supported by other studies. One such study concluded that achievement of badges could influence students’ behaviour even if they do not interfere with grading (Hakulinen & Auvinen, 2014). It seems that competitiveness was triggered in those who wanted to do better anyway.

We were pleased to see learners’ comments about reduced stress during the module. We wanted our award system to potentiate a sense of safety around assessment, giving participants freedom to learn from their mistakes without influencing their final grade. This is a well-recognized principle in gamification as a facilitator of students’ engagement (De Byl & Hooper, 2013).

In addition, our design was guided by the fact that the best use of badges was linked with the recognition of already occurred learning, therefore more viewed as an assessment tool, providing feedback and possibly self-reflection (Reid et al., 2015).

B. Impact on Structure, Assessment and Feedback

We were pleased to note that students in our study felt that digital badges provided direction and structure to their learning. This was also reflected in their comments: students mentioned how badges impacted on their study structure, helping them to focus attention on important aspects of the seminars. These findings are consistent with a study that reported that students who enjoyed badges, found them helpful in giving them the direction they needed to work in (Abramovich et al., 2013). These students also praised the alignment between badge topics and course content (Abramovich et al., 2013).

We were also hoping that badges designed, as part of an assessment that generates formative feedback would help students know if they are progressing enough to meet the requirements of their class (HASTAC n.d.). Based on responses to items 24, 18 and 19 it appeared that students benefited somewhat from the potential guidance and feedback provided by the digital badges system.

It is important to remember that students were asked about the badges at their review seminar and few days prior to their exams. This timing could have influenced their answers. For example, students’ opinion was divided on the statement that earning a badge gave them a sense that they have mastered the course content. Similarly, opinion was divided on whether badges increased students’ confidence that they could demonstrate their knowledge, nevertheless more students felt that they had an impact. It would be interesting to see if students’ responses had been different after their exams. We know gamification has already been described as a powerful strategy that can help achieve learning objectives by affecting the way students behave (Huang & Soman, 2013).

C. Impact on Motivation and Engagement

In our study, a majority of students responded that they worked harder in this module compared to non-gamified modules. Similarly, about 30% more students stated that they were more motivated to work harder through the module because they were earning digital badges. Interestingly when they were given space to provide free comment, many have noted that they did feel more motivated, and a few felt more engaged. Previous studies also reported that students were more likely to engage in the game-like tasks providing rapid feedback (Thamvichai & Supanakorn-Davila, 2012). In other publications students also considered gamified courses to be more motivating, interesting and easier to learn as compared to other courses (Barata et al., 2013), (Dicheva et al., 2015), (Hakulinen & Auvinen, 2014). It is suggested that badges are most valued by learners who are extrinsically motivated and value external validation (Foli et al., 2016).

D. Impact on Outcomes

We did not compare outcomes in overall performance in assessment between students before and after implementation of badges. Having considered this, we decided against it. We felt there were too many variables influencing students’ performance and it would be difficult to definitely attribute potential change to the implementation of digital badges.

E. View of Badges and Learner Type

The impact of an educational tool could also depend on characteristics of the student as a learner. It is reported that students with high expectation for learning and those that value their learning tasks may view the badge as validating if designed as a performance assessment (having impact on intrinsic motivation), but it may devalue their learning if it was viewed as an external reward (Reid et al., 2015). On the other hand, badges used as an assessment model can have a negative impact on students with low expectancy values (Reid et al., 2015). Another study concluded that engagement in the gamified classroom was dependent on students’ playfulness (De Byl & Hooper, 2013). In this study, we have not addressed the learner types and other such specifications of the individuals in our group of students.

In a systematic review of digital badges in health care education, it is mentioned that digital badges represent an innovative approach to learning and assessment and evidence in further education literature demonstrates that their use increases knowledge, retention and motivation to learn. However, they also report a lack of empirical research investigating digital badges within the health care education context (Noyes et al., 2020).

F. Limitations

Our study is limited by the lack of demographic data from all participants. This reduces the potential for comparison between genders and undergraduate vs postgraduate students. As mentioned above, we have not addressed learner characteristics and types (intrinsic versus extrinsic motivation, playfulness). The timing of the data collection (students completed questionnaires before their exams, rather than after) may also be limiting factors. It is also important to remember that this study allowed only for assessment of subjective impact (via students’ opinions and experience) of the badges on students learning and did not perform objective measures of students’ overall performance.

V. CONCLUSION

This study was performed at the start of the implementation of the digital badges in the module and at the time, it was the only module with elements of gamification throughout the whole undergraduate medical curriculum in our university. Like most changes in the assessment process, the students greeted this with a level of apprehension. It would be interesting to see if students’ opinion has evolved after a few years of digital badges being integrated in the module and when other modules are using them. Nevertheless, our data shows that our group of students felt, that they benefited from the learning structure provided by the digital badges. The online process of obtaining the badges enabled tutors to provide timely feedback and monitor students’ progress. In addition, our findings are in keeping with the literature in that engagement and motivation have been facilitated by introducing the digital badges and as such, they indicate that the use of digital badges is a promising tool in education.

The use of digital badges in Medical Education is only starting and would benefit from more research in its judicious integration in higher education curriculum as appropriate.

Notes on Contributors

Dr Edyta Truszkowska did the literature search, collected and analysed the data and gave feedback on methodology and questionaire developed and wrote paper.

Dr Yvonne Emmett designed the methodology and developed questionaire. She gave feedback on the data analysis and edited the writing of the paper.

Prof Allys Guerandel suggested the implementation of digital badges and research project. She gave feedback on data analysis and wrting of paper making revisions to same.

All three authors have read and approved the final manuscript.

Ethical Approval

Our project has been exempted by ethics committee of our institution Human Research Ethics Committee – Sciences (Exemption number LS-E-17-56).

Data Availability

Data is available on reasonable request and data is shared in the institution.

Acknowledgement

We would like to acknowledge our institution psychiatry teaching team for their support in the implementation of the digital badges.

Funding

There are no sources of funding.

Declaration of Interest

 There is no conflict of interest for any of the authors.

References

Abramovich, S. (2016). Understanding digital badges in higher education through assessment. On the Horizon, 24(1), 126-131. https://doi.org/10.1108/OTH-08-2015-0044

Abramovich, S., Schunn, C., & Higashi, R. M. (2013). Are badges useful in education? It depends upon the type of badge and expertise of learner. Educational Technology Research and Development, 61(2), 217-232. https://doi.org/10.1007/s11423-013-9289-2

Armour-Thomas, E., & Gordon, E. (2013). Toward an Understanding of Assessment as a Dynamic Component of Pedagogy. https://www.ets.org/Media/Research/pdf/armour_thomas_gordon_understanding_assessment.pdf

Barata, G., Gama, S., Jorge, J., & Gonçalves, D. (2013, September). Engaging engineering students with gamification. In 2013 5th International Conference on Games and Virtual Worlds for Serious Applications IEEE, UK (VS-GAMES) (pp. 1-8). https://doi.org/10.1109/VS-GAMES.2013.6624228

De Byl, P., & Hooper, J. (2013). Key attributes of engagement in a gamified learning environment. In ASCILITE-Australian Society for Computers in Learning in Tertiary Education Annual Conference, Australia (221-230). https://www.learntechlib.org/p/171232/

Deterding, S., Dixon, D., Khaled, R., & Nacke, L. (2011, September). From game design elements to gamefulness: defining” gamification”. In Proceedings of the 15th international academic MindTrek conference: Envisioning future media environments (pp. 9-15). https://doi.org/10.1145/2181037.2181040

Dicheva, D., Dichev, C., Agre, G., & Angelova, G. (2015). Gamification in education: A systematic mapping study. Journal of Educational Technology & Society, 18(3), 75-88. https://www.jstor.org/stable/jeductechsoci.18.3.75

Dowling-Hetherington, L., & Glowatz, M. (2017). The usefulness of digital badges in higher education: Exploring the student perspectives. Irish Journal of Academic Practice, 6(1), 1-28. https://researchrepository.ucd.ie/handle/10197/9691

Foli, K. J., Karagory, P., & Kirby, K. (2016). An exploratory study of undergraduate nursing students’ perceptions of digital badges. Journal of Nursing Education, 55(11), 640-644. https://doi.org/10.3928/01484834-20161011-06

Hakulinen, L., & Auvinen, T. (2014, April). The effect of gamification on students with different achievement goal orientations. In 2014 international conference on teaching and learning in computing and engineering, Malaysia (pp. 9-16). IEEE. https://doi.org/10.1109/LaTiCE.2014.10

HASTAC. (n.d.). Digital Badges. http://www.hastac.org/digital-badges

Hensiek, S., DeKorver, B. K., Harwood, C. J., Fish, J., O’Shea, K., & Towns, M. (2017). Digital badges in science: A novel approach to the assessment of student learning. Journal of College Science Teaching, 46(3), 28. https://www.proquest.com/scholarly-journals/digital-badges-science-novel-approach-assessment/docview/1854234735/se-2

Huang, W. H. Y., & Soman, D. (2013). Gamification of education. Report Series: Behavioural Economics in Action, 29, 11 -12.

Johnson, L. J., & LaMontagne, M. J. (1993). Research methods using content analysis to examine the verbal or written communication of stakeholders within early intervention. Journal of Early Intervention, 17(1), 73-79.

Knapp, T. R. (1990). Treating ordinal scales as interval scales: An attempt to resolve the controversy. Nursing Research, 39(2), 121-123.

Mandernach, B. J. (2015). Assessment of student engagement in higher education: A synthesis of literature and assessment tools. International Journal of Learning, Teaching and Educational Research, 12(2), 1-14.

McGonigal, J. (2011). Reality is broken: Why games make us better and how they can change the world. Penguin.

Norman, G. (2010). Likert scales, levels of measurement and the “laws” of statistics. Advances in health sciences education, 15(5), 625-632.

Noyes, J. A., Welch, P. A., Johnson, J. W., & Carbonneau, K. J. (2020). A systematic review of digital badges in health care education Medical Education, 54(7), 600-615. https://doi.org/10.1111/medu.14060

Przybylski, A. K., Rigby, C. S., & Ryan, R. M. (2010). A motivational model of video game engagement. Review of General Psychology, 14(2), 154. https://doi.org/10.1037/a0019440

Quesenberry, A. C., Hemmeter, M. L., & Ostrosky, M. M. (2011). Addressing challenging behaviors in Head Start: A closer look at program policies and procedures. Topics in Early Childhood Special Education, 30(4), 209-220. https://doi.org/10.1177/0271121410371985

Reid, A. J., Paster, D., & Abramovich, S. (2015). Digital Badges in undergraduate composition courses: effects on intrinsic motivation. Journal of Computers in Education, 2(4), 377-98. https://doi.org/10.1007/s40692-015-0042-1

Rolfe, I., & McPherson, J. (1995). Formative assessment: How am I doing? The Lancet, 345(8953), 837-839. https://doi.org/10.1016/S0140-6736(95)92968-1

Seaborn, K., & Fels, D. I. (2015). Gamification in theory and action: A survey. International Journal of Human Computer Studies, 1(74), 14-31. https://doi.org/10.1016/j.ijhcs.2014.09.006

Sullivan, G. M., & Artino, A. R. (2013). Analyzing and interpreting data from likert-type scales. Journal of Graduate Medical Education, 5(4), 541-542 https://doi.org/10.4300/JGME-5-4-18

Sundler, A. J., Lindberg, E., Nilssonn, C., & Palmer, L. (2019). Qualitative thematic analysis based on descriptive phenomenology. Nursing Open, 6(3), 733-739. https://doi.org/10.1002/nop2.275

Thamvichai, R., & Supanakorn-Davila, S. (2012). A pilot study: Motivating students to engage in programming using game-like instruction. Proceedings of Active Learning in Engineering Education. St Cloud University. https://nms.asee.org/wp-content/uploads/sites/47/2020/02/St_Cloud_2012_Conference_Proceedings.pdf#page=18

UCD Teaching and learning. (2017). UCD digital/open badges pilot 2016/2017. Implementation and evaluation report. UCD Teaching and Learning, 1-23. https://www.ucd.ie/t4cms/UCD%20Digital%20Badges%20Pilot%20Report.pdf

Yildirim, S., Kaban, A., Yildirim, G., & Celik, E. (2016). The effect of digital badges specialization level of the subject on the achievement, satisfaction and motivation levels of the students. Turkish Online Journal of Educational Technology- TOJET, 15(3), 169-182. https://eric.ed.gov/?id=EJ1106420

*Allys Guerandel
University College Dublin,
School of Medicine and Medical Sciences,
Belfield, Dublin 4, Ireland D04V1W8.
00353868590063
Email: allys.guerandel@ucd.ie

Submitted: 1 August 2022
Accepted: 1 November 2022
Published online: 4 April, TAPS 2023, 8(2), 36-46
https://doi.org/10.29060/TAPS.2023-8-2/OA2855

Marina C. Jenkins¹, Caroline R. Paul², Shobhina Chheda¹ & Janice L. Hanson³

¹School of Medicine and Public Health, University of Wisconsin-Madison, United States; ²Langone Health, Grossman School of Medicine, New York University, United States; ³School of Medicine, Washington University in St. Louis, United States

Abstract

Introduction: Increases in vaccine hesitancy continue to threaten the landscape of public health. Literature provides recommendations for vaccine communication and highlights the importance of patient trust, yet few studies have examined medical student perspectives on vaccine hesitancy in clinical settings. Therefore, we aimed to explore medical student experiences encountering vaccine hesitancy, mistrust, and personal biases, with the goal of informing medical student education.

Methods: A health disparities course including simulated clinical scenarios required students to complete a written reflection. We sorted reflections written in 2014-2016 to identify common topics and used inductive thematic analysis to identify themes relevant to vaccine hesitancy by group consensus.

Results: Our sample included 84 de-identified essays sorted into three non-exclusive topics: vaccine hesitancy (n=42), mistrust (n=34), and personal bias (n=39). We identified four themes within medical students’ reflections: 1) Building a Relationship, including emphasis on patient-centred approaches; 2) Preparedness and Need to Prepare for Future Encounters, including highlighting gaps in medical education; 3) Reactions to Encountering Hesitant Patients, including frustration; 4) Insights for Providing Information and Developing a Plan with Hesitant Patients, including approaches to presenting knowledge. 

Conclusion: Reflections in the context of simulated encounters and discussion are useful in students identifying their preparedness for vaccine discussion with patients. Student reflections can assist educators in identifying missing educational frameworks for particular scenarios such as vaccine hesitancy. Without a structured framework regarding addressing vaccine hesitancy, students draw upon other skills that may contradict recommended practices.

Keywords:           Medical Education, Vaccine Hesitancy, Reflective Writing, Bias, Mistrust

Practice Highlights

• Reflective writing can be a useful tool in medical education toward addressing vaccine hesitancy.
• Medical student reflective writing can be used to demonstrate curricular gaps.
• Medical students expressed feeling unprepared to care for vaccine hesitant patients.
• Without a framework for vaccine communication, students may draw on other inappropriate skills.

I. INTRODUCTION

Increases in vaccine hesitancy and refusal threaten public health (He et al., 2022; Hough-Telford et al., 2016; Kempe et al., 2020; Santibanez et al., 2020), especially with the COVID-19 pandemic introducing a need for quick and widespread uptake of a new vaccine (Hamel et al., 2022; Ognyanova et al., 2022). Patients, especially parents, are increasingly seeking alternative forms of health information, such as online sources that can include misinformation (Broniatowski et al., 2018; Hara & Sanfilippo, 2016; Jenkins & Moreno, 2020; Meleo-Erwin et al., 2017). Patient trust in their clinician and the health care system delivering the vaccine strongly influence vaccination decisions (Goldenberg, 2016; Kennedy et al., 2011; Larson, 2016). Trust remains the most important barrier to acceptance and uptake of the COVID-19 vaccine, with mistrust of government, medicine, and science presenting major barriers to vaccine uptake (Ognyanova et al., 2022). Vaccine hesitant patients may bring preconceptions and concerns from their own research to in-clinic vaccine communication. Thus, it is important for clinicians to be well-prepared to work with vaccine-hesitant patients and parents.

Existing recommendations for clinicians encountering vaccine hesitancy emphasise centring patient views and voice instead of a medical, academic perspective (Holt et al., 2016; Koski et al., 2019). Approaches including motivational interviewing, presumptive language around vaccine recommendations, and persistent vaccine reminders without pressuring or dismissing patients have been shown to be effective in addressing vaccine hesitancy in medical practice (Dempsey et al., 2018; Gagneur et al., 2018; Hofstetter et al., 2017), while correcting misinformation and offering evidence to patients have been found to be counterproductive (Holt et al., 2016; Koski et al., 2019). These pre-COVID recommendations remain the same for addressing COVID-19 vaccine hesitancy, and lack of physician preparedness for encountering these patients is still an important issue (Centres for Disease Control and Prevention, 2021). Physicians may have misconceptions about patients’ reasons for vaccine hesitancy, often assuming lack of understanding or information on the safety, effectiveness, and necessity of vaccines (Hough-Telford et al., 2016), rather than recognising the more central roles of trust and validation of concerns. If physicians do not learn approaches for centring patient voices in vaccine communication, these pre-conceived biases may present a barrier to vaccine uptake and patient-physician trust.

While valuable recommendations for addressing vaccine hesitancy in the clinical setting exist, current efforts center around informing practicing clinicians on these approaches and providing more educational resources to patients (Centres for Disease Control and Prevention, 2021). These may not represent a sufficient, long-term solution. Furthermore, resources available for healthcare workers may be inaccessible or overwhelming for physicians independently seeking tools (Karras et al., 2019). Incorporating vaccine hesitancy-centred curriculum into medical education may be the optimal, long-term solution to the lack of physician preparedness for these encounters, especially in the face of future pandemics and introduction of new vaccines. With curriculum renewal efforts incorporating early clinical experiences, students could encounter patients for whom vaccines are recommended, including vaccine hesitant patients, early in medical school. It would provide a better educational experience for students and a better health care experience for patients if students receive education to prepare them for these conversations. However, few studies have examined medical student perspectives on vaccine hesitancy in the clinical setting. Existing studies have found mixed findings around medical students’ reflections on their preparedness for encountering vaccine hesitant patients and highlight the need for expansion of related curriculum in medical education (Brown et al., 2017; Kernéis et al., 2017). While COVID vaccine hesitancy literature lacks exploration of medical student perspectives and preparedness, recent studies have highlighted an additional barrier of vaccine hesitancy among medical students in some settings (Lucia et al., 2021). These findings provide additional motivation for including vaccine hesitancy-specific curriculum in medical education.

Understanding medical students’ reactions to vaccine hesitancy is critical in preparing students to address vaccine hesitancy while maintaining patient trust. In the present study, which used a scholarship-of-teaching approach, we aimed to expand on existing research on medical student preparedness for encountering vaccine hesitancy to examine written reflections on mistrust and personal bias in clinical encounters more broadly and use a larger sample of student narratives. We analysed students’ structured reflections regarding assigned reading, simulated patient encounters, peer discussions, and faculty-facilitated discussions to evaluate medical students’ learning during a health disparities curriculum. Structured reflection on simulated encounters has been shown to be a useful tool for understanding student perspectives (Koski et al., 2018); this approach can inform development of medical curriculum for addressing vaccine hesitancy and may be a useful teaching tool as well for students to practice, discuss, and reflect on their own biases in an educational setting. Therefore, the purpose of this study was to explore medical student reflections on encountering vaccine hesitancy, patient mistrust, and personal biases, with the goal of informing medical student education.

II. METHODS

In this qualitative study, we analysed written reflections from a third-year medical student Skills to Impact Health Disparities course, to evaluate their learning about interacting with vaccine-hesitant patients and parents. This study was determined to be exempt by the relevant institutional review boards, including a waiver of informed consent.

From 2006-2018, a medical school at a U.S., Midwestern university required a one-day core session with the goal of developing learner skills to impact health disparities. Small groups of approximately six students went through five to six standardized patient scenarios, each designed to generate discussion and reflection about clinician bias that can unintentionally influence patient care. During the learning activity, each student spent 3-5 minutes interacting with a standardized patient who presented a challenge designed to provoke a level of discomfort in the learners to allow for discussion and reflection. One of these six scenarios included a parent with a history of vaccine refusal for their child expressing concerns about a recommended vaccine.

Following each case, students engaged in a 15-minute, non-facilitated discussion based on a list of focused questions. After all cases, students joined another group of six students for a 75-minute faculty-facilitated debrief. In addition, students were required to complete a brief critical reflection based on a theme of the core day activity using the LeAP framework (Aronson et al., 2012). This framework is modelled on a clinical framework, the SOAP note (Chief complaint, Subjective, Objective, Assessment, and Plan). Students were asked to consider a specific experience that led to concern or questions; describe the experience as fully as possible; reconsider the experience by getting other perspectives; synthesize learning; and make a plan to address future similar challenges. Students could choose to reflect on simulated or real clinical experiences.

Written reflective essays were available for analysis from years 2014-2016, providing qualitative data about students’ observations and experiences with health disparities and health equity. All available essays (n=292) from 2014, 2015, and 2016 that were submitted as a course requirement for the Skills to Impact Health Disparities Core Day required course were de-identified and organized by year.

To ascertain the topics that the students addressed, three investigators (two involved in this study and one from another study using the larger set of all essays) read all essays. Each investigator then designated each essay to a topic from a jointly-developed list of non-exclusive topics derived from the data. After individually assigning topics for a sample of essays, the investigators met to compare their sorting and reconcile any differences before they went on to sort through another set of essays. This process continued until all essays were assigned to one or more topics. Most topic labels matched topics of the simulated scenarios that the students encountered in the course, while others related to broader issues highlighted across scenarios. With the goal of selecting reflections relevant to the issue of vaccine hesitancy, all reflections designated under the topics of vaccine hesitancy, mistrust, and personal bias were gathered for qualitative data analysis. Literature review and initial reading of essays suggested essays on encountering mistrust and bias relate to students’ experiences when encountering vaccine hesitant patients, despite not all essays relating directly to vaccine hesitancy. Each essay was assigned an identifier with cohort year and an essay number. Individual essays were excluded based on group consensus on lack of relevance to vaccine hesitancy.

Inductive thematic analysis was used to identify codes and themes in the reflection data using a semantic, realist approach to identify explicit reactions from students grounded in clinical experiences to identify themes that could be directly applied to clinical practice (Braun & Clarke, 2006). Four investigators, including two involved in topic assignment (CRP, SC) and two additional investigators (MCJ, JLH), read and discussed six essays to develop a preliminary codebook, applied these codes to the same six essays, then met to discuss and revise the codebook. Subsequently, investigators coded the remaining essays in pairs using the revised codebook through four rounds of coding, making further iterative changes to the codebook and reconciling differences within pairs. The full team then met to discuss the coding, revise code descriptions, refine the grouping of the codes, and agree on descriptions of the groups. Any changes made to the codebook during the analysis process were retrospectively updated in all previous coding, so that all coding data reflected the final version of the codebook. Data were organized with qualitative analysis software (HyperResearch version 4.5.4). After all data were coded, investigators discussed and reached consensus on the themes.

III. RESULTS

A total of 90 reflections were collected from the Skills to Impact Health Disparities course across three cohorts of third-year medical students from 2014-2016 at one U.S., Midwestern university. Based on investigator consensus on lack of content relevance, six reflections were excluded from our study sample. Our final study sample included 84 de-identified reflections across three, non-exclusive topics: 42 categorized as relating to vaccine hesitancy, 34 as mistrust, and 39 as personal bias. We identified four major themes in medical students’ reflections on encountering vaccine hesitancy, mistrust and personal bias: 1) Building a Relationship, 2) Preparedness and Need to Prepare for Future Encounters, 3) Reactions to Encountering Hesitant Patients, and 4) Insights for Providing Information and Developing a Plan with Hesitant Patients. Representative quotes for each theme can be found in Table 1. Supplemental Table 1 lists each theme with the codes that informed the theme.

A. Building a Relationship

In our first theme, medical students recognized the importance of Building a Relationship with hesitant parents or patients as the foundation for discussions about vaccines or other care about which patients expressed hesitance. They focused on approaches such as building rapport, centring the parent/patient’s views during the discussion, acknowledging their efforts to gather information about their health decisions, expressing empathy, and avoiding direct confrontation of the patient’s viewpoint during the discussion. Many of these observations occurred during the core day experience. For example, one student wrote:

“I learned the importance of letting the patient try to teach the doctor what they know rather than the doctor jumping in and lecturing to the patient. In the future I will try to talk less and let the patient explain more about why they oppose vaccinations to better gauge what they understand about the literature before I try to explain why vaccinations are important and the facts about vaccinations.”

[Year3_61]

The students saw the importance of finding points of commonality between their perspectives and those of the patient and moving the conversation toward establishing goals that they could work together with the patient to accomplish.

One student described, “I learned that a big part of approaching this difficult conversation is establishing the correct approach: common goal, shared decision making.”

[Year3_65]

B. Preparedness and Need to Prepare for Future Encounters

Another major theme identified in medical student reflections on encountering hesitant patients was Preparedness and Need to Prepare for Future Encounters. This theme included discussion of whether the student expressed feeling ready for the encounter or whether they thought it was successful, as well as specific plans for preparing for similar encounters in the future. One way that students discussed their own feelings of preparedness was by recognizing their own biases upon reflection of the encounter. For example, one student wrote:

“I realized my own prejudices influenced my care of my patients more than I would have liked. … It was an eye opener that I am not as impartial as I would like to be and that it takes a lot more self-reflection and awareness to be the best care provider I can be.

[Year3_16] 

When discussing a need to prepare for future encounters, many students referenced plans to independently seek additional resources, especially those referenced by patients in encounters.

Other students mentioned plans to practice patient interactions related to the reflection encounter; including, “For me, practicing acknowledging a patient’s views and concerns without endorsing or validating false information is paramount.”

[Year1_07]

Some students also referenced plans to request feedback or advice from more senior clinicians. Additionally, several students identified gaps in their medical school curriculum that contributed to their lack of preparedness or that needed to be filled to support future preparedness. Students specifically referred to needing more resources, support, and training for encountering hesitant patients. They sometimes called for system-wide changes to address this gap in knowledge.

C. Reactions to Encountering Hesitant Patients

One of the themes identified in the students’ self-reflection was related to their own and others’ Reactions to Encountering Hesitant Patients. While some students expressed frustration with patients/parents who expressed hesitance about vaccines, they acknowledged that they can be passionate about the topic of vaccines in their patient care, but ultimately, patients and parents make their own decisions.

One student shared, “I have always found it quite distressing when an otherwise healthy child goes unvaccinated, given the enormous amount of evidence in favour of vaccination efficacy and its effect on public health.”

[Year2_86]

Another student shared, “I knew I could not force the patient, and I knew that she ultimately was in control of what she would do.”

[Year3_78]

In some reflections patient and parents were labelled, for example, as “anti-vaxxers.” Some reflections described parents’ and patients’ bias towards the physician or clear messaging of a desire for a different doctor.  In encountering standardized patients in our scenarios or in reflecting on patients seen in clinical settings, students acknowledged that these conversations were difficult, and they were able to self-assess their level of comfort with conversations.

This was well-summarized in one reflection: “It was remarkable to me how such a strong reaction from this patient’s mother elicited an equally strong reaction in me.”

[Year2_34]

 

At times students recognized a point where these difficult conversations could reach a dead end. One student stated, “No matter how hard I would try, nothing seemed to work.”

[Year2_03]

Especially in this context, students reflected ambivalence towards the patient’s decision. For example:

“I personally feel that providers allowing for healthy children on their patient panels to remain unvaccinated indirectly reinforces non-vaccination as being acceptable by the medical establishment. That said, I also see and appreciate that turning a child away from one’s practice because their parents refuse to vaccinate them not only does not solve the problem at hand, but it also leaves a child at a very critical developmental age with no health care at all until an alternative provider can be found. Ultimately, I found attempting to reconcile these seemingly incompatible sides of the issue of dealing with anti-vaccination quite confusing and uncomfortable.”

[Year2_86]

D. Insights for Providing Information and Developing a Plan with Hesitant Patients

A fourth theme centred on students’ insights regarding how to provide information appropriately to patients and how to create a plan with patients who were hesitant regarding the medical recommendations given to them. Medical students suggested a variety of ways to provide information to patients who were hesitant. They noted the importance of contributing relevant facts and evidence, stressing that such information and knowledge in general needed to be presented in an understandable manner.

As one student described, “Finding the appropriate words to use in such conversations with a patient is essential.”

[Year1_44]

Students often wrote that they needed to provide reputable information to inform the patient’s decision-making. Some suggested strategies for how to present information to patients, including the sharing of stories and the use of scary information to convey the level of seriousness of the medical recommendation and advice.

One student referenced storytelling in the literature, “…the use of storytelling, the same method used by the anti-vaccination movement, [can be] a way to counteract the barrage of misinformation regarding vaccines.”

[Year1_90]

Sharing these insights about how to present information, students also moved towards how to develop a plan with their patients with some deliberate suggestions. Some students felt they needed to be persistent in their recommendations for vaccines. Some students explained how intentional discussions on the risks and benefits of their recommendations can help in their negotiation about a care plan with their patients.

One student noted, “This draws along the line of patient autonomy, and as long as we are clear about the risks and benefits with the patient, then ultimately, it’s up to the patient to make the decision about which medications she will take.”

[Year1_52]

Submitted: 11 March 2022
Accepted: 28 June 2022
Published online: 4 April, TAPS 2023, 8(2), 14-35
https://doi.org/10.29060/TAPS.2023-8-2/OA2762

Sayaka Oikawa¹, Ruri Ashida² & Satoshi Takeda³

¹Center for Medical Education and Career Development, Fukushima Medical University, Fukushima, Japan; ²Center for International Education and Research, Tokyo Medical University, Tokyo, Japan; ³Department of Emergency Medicine, The Jikei University School of Medicine, Tokyo, Japan

Abstract

Introduction: There are various difficulties in treating foreign patients; however, the existing educational programs are still insufficient for addressing this issue. The purpose of this study is to investigate what difficulties are encountered in the treatment of foreigners in emergency departments, and to create scenarios for simulation-based education using real-life cases.

Methods: A cross-sectional anonymous survey to 457 emergency departments was conducted in 2018. Additionally, we conducted a survey of 46 foreign residents who had visited hospitals for treatment in Japan. The data was analysed quantitatively, and the narrative responses were thematically analysed.

Results: Of the 141 hospitals that responded (response rate: 30.9%), 136 (96.5%) answered that they had treated foreign patients. There were 51 and 66 cases with cultural and linguistic difficulties, respectively. In the qualitative analysis, different ideas/beliefs towards treatments or examinations (51.0%) and communication with non-English speaking patients (65.2%) were most common categories in the cases with cultural and linguistic difficulties, respectively. The survey of 46 foreign residents on the surprising aspects of Japanese healthcare showed, 14% mentioned difference in treatment plans between own country and Japan, 12% each mentioned a lack of explanation by medical staff, and a lack of privacy in the examination room. Based on the survey results, we created 2 scenarios of simulation.

Conclusions: Scenarios of simulation-based education using real-life cases may be effective materials for cultivating cultural awareness of medical staff.

Keywords:           Cultural Awareness, Cultural Humility, Emergency Department, Foreign Patients, Simulation-based Education

Based on the survey results, we decided the main topic of the scenarios based on the contents overlapped in multiple cases. “Gender restriction of doctors who treated patients” and “communication difficulty in languages other than Japanese or English” were the most frequent topics in cultural and linguistic difficulties respectively. Following the selection of topics, we synthesised the similar responses to create a scenario that could occur in any size of ED setting. We developed the settings including patient age, sex, language, and backgrounds, regarding that the patient characteristics can be demonstrated by simulated patients. As a result, we developed two scenarios: a scenario of abdominal pain in a Muslim female patient and a scenario of forearm fracture in a Chinese male patient (Appendices 5 and 6).  In the abdominal pain scenario, no female doctor was available, and a learner, a male doctor, had to examine and treat a simulated patient who refused to be seen by a male doctor. In the forearm fracture scenario, no interpreter was available, and a learner had to communicate with a simulated patient who spoke Chinese only. The learning objective for the learners was to communicate appropriately with patients with different cultural and linguistic backgrounds. Based on the results of the survey for foreign residents, we indicated the importance of listening to the patient’s concerns carefully as a teaching point. Also, we reflected the survey results of how each hospital handled the cases on the information for instructors and teaching points.

IV. DISCUSSION

At the time of writing this paper, 96.5% of the EDs had accepted foreign patients; and 82.3% had English-speaking staff. However, only 32.6% of the EDs had multilingual signs for patients, which is listed as actions to be taken in the manual for treating foreign patients (Japan Ministry of Health, Labour and Welfare, 2021).

In the present study, most of the EDs used translation tools when treating foreign patients. Various types of translation methods were found to be used in the EDs, the use of which is consistent with the manual for treating foreign patients (Japan Ministry of Health, Labour and Welfare, 2021). However, we found that the EDs still encountered a significant number of cases with linguistic difficulties. This suggests that even though the EDs own the translation tools, medical staff are not able to utilise them in communicating with foreign patients. According to our survey result, it was revealed that more than half of the cases with linguistic difficulties were of non-English speaking patients. To overcome the linguistic difficulties, medical staff need to be capable of using them enough to communicate with patients of various native languages. In addition to the use of translation tools, multilingual medical explanation/consent forms or signs in hospitals may be effectively used in the aim of communication with foreign patients.

Regarding culturally difficult cases, our survey showed the various issues caused by differences of religious background, lifestyles, and ideas and beliefs on treatment and testing between medical staff and patients. This result is consistent with the reports which elaborated difficulties in treating foreign patients in Japan (Tatsumi et al., 2016). Our study showed that different ideas/beliefs towards treatments or examinations was most common theme in the cases with cultural difficulties in EDs. Knowing beliefs of other culture is one of individual’s capabilities to manage effectively in culturally diverse settings (Ang et al., 2007), and a report on psychiatric hospitals showed that medical staff adapted to hospitalised foreign patients’ culture and religion as they built the relationships with the patients over a long period of time (Kobayashi et al., 2014). Whereas, it is difficult to build relationships with foreign patients in the acute ED setting. Thus, we realised that practical training of communication with foreign patients provide knowledge about their cultures and religions in limited time and is critically important for medical staff in EDs.

SBE is an effective educational format which makes learners’ unconscious incompetence to conscious incompetence (Morell et al., 2002), in other words, medical staff may be able to recognise their unconscious biases towards foreign patients by participating in SBE. As consistent with the previous survey by the MHLW (2021), the culturally difficult cases included complicated issues that require the cooperation of administrative staff and full-time English interpreters in the hospital. In the present study, we created the two scenarios targeting medical staff as learners based on the real-life cases with the many responses in the survey. However, we need to create more varieties of scenarios that can involve other professions than health care professionals. Furthermore, the acquisition and retention of learners’ skills in a single training session of SBE is limited (Legoux et al., 2021). SBE aimed at cultivating cultural awareness cannot be completed in a single session but in continuous sessions with multiple scenarios.

The results of our survey of foreign residents showed that they had been surprised at the differences in treatment plans between their country and Japan, a lack of explanation by medical staff, and a lack of privacy in the examination room. We found that it is important to investigate the opinions of those who receive medical care in a country different from their home because their perspectives allow us to recognise the things taken for granted among medical staff.  Medical staff’s unconscious biases about patients of different cultural backgrounds or national origins influence their decision-making (Tervalon & Murray-Garcia, 1998), and implicit bias can contribute to miscommunication (Bartlett et al., 2019). Therefore, listening to the concerns of foreign patients is important in order to avoid providing treatment based solely on medical staff’s biases. Furthermore, in creating scenarios, referring to the survey results of multiple stakeholders made the contents more multi-dimensional and relevant. This study was conducted in the contexts of EDs in Japan, however, scenarios created with perspectives of both medical staff and patients who have various cultural backgrounds may effectively address to the real-life problems triggered by unconscious biases, even in other contexts. 

In Emergency situations, we often focus on the patients’ cultural backgrounds, national origins, languages, and religious background in order to provide effective treatments. However, recognising our own bias is not achieved by only focusing on the patients’ culture. Self-reflection is necessary to recognise one’s own cultural biases. The process of self-reflection of our own culture is important for cultivating cultural awareness. Furthermore, the importance of cultural humility – discovering one’s own values toward other cultures through continuous self-reflection and becoming aware of one’s own relationship to the world – has been recently noted in medical education (Chang et al., 2012). As a further research, the development of scenarios that include the study guide which ensure the learners’ self-reflection is required for SBE in emergency settings.

There are several limitations in this study. The response rate of a survey for EDs was 30.9%, which is unable to deny sampling bias. We conducted a survey for EDs with a focus on English, however, it is necessary to conduct surveys on languages other than English. In addition, the survey was only for the EDs of training hospitals in the top 10 prefectures with the most foreign tourists. We may consider expanding the number of hospitals to collect more information about difficulties they encounter in treating foreign patients. For the sub study, the snowball sampling had a methodological limitation in calculating total number the survey sent. As a further research, impacts of SBE using these scenarios on the treatment of foreign patients is less clear. To assess whether foreign patients’ satisfaction of medical care will change, and whether unconscious bias towards foreign patients among medical staff will decrease by conducting these scenarios are necessary.

V. CONCLUSION

In the current study, we were able to clarify linguistic and cultural difficulties in treating foreign patients in the EDs. We developed the scenarios for SBE using the real-life difficult cases of foreign patients with linguistic or cultural differences in medical care in Japan. The simulation training using these scenarios may be useful for promoting cultural awareness of medical staff in EDs. In future, more varieties of scenarios of SBE need to be created and shared in order to treat foreign patients safely and adequately.

Notes on Contributors

SO contributed to the design of the study and conducted data collection and analysis. RA devised the project, the main conceptual ideas, and conducted data collection and analysis. ST contributed to the design of the study and the interpretation of the data.

Ethical Approval

This study was approved by the Institutional Review Board of The Jikei University School of Medicine Japan (Approval No. 28-211(8454), 28-276(8519)). An informed consent was obtained from all the participants responded to the survey.

Data Availability

The data that support the findings of this study are not openly available due to privacy. The materials are available from the corresponding author on reasonable request.

Acknowledgement

The authors would like to acknowledge the respondents at the EDs of training hospitals, the foreigners living in Japan, and the young clinicians at The Jikei University School of Medicine for their cooperation in the study. 

Funding

This work has been supported by JSPS KAKENHI, grant number 16K08883.

Declaration of Interest

The authors report no conflicts of interest. The authors alone are responsible for the content of the article.

References

Ang, S., Van Dyne, L., Koh, C., Ng, K. Y., Templer, K. J., Tay, C., & Chandrasekar, N. A. (2007). Cultural intelligence: Its measurement and effects on cultural judgment and decision making, cultural adaptation and task performance. Management and Organization Review, 3(3), 335-371. https://doi.org/10.1111/j.1740-8784.2007.00082.x

Ashida, R., & Otaki, J. (2022). Survey of Japanese medical schools on involvement of English-speaking simulated patients to improve students’ patient communication skills. Teaching and Learning in Medicine, 34(1), 13-20. https://doi.org/10.1080/10401334.2021.1915789

Bartlett, K., Strelitz, P., Hawley, J., Sloane, R., & Staples, B. (2019). Explicitly addressing implicit bias in a cultural competence curriculum for pediatric trainees. MedEdPublish, 8. https://doi.org/10.15694/mep.2019.000102.1

Chang, E. S., Simon, M., & Dong, X. (2012). Integrating cultural humility into health care professional education and training. Advances in Health Sciences Education, 17(2), 269–278. https://doi.org/10.1007/s10459-010-9264-1

Hobgood, C., Sawning, S., Bowen, J., & Savage, K. (2006). Teaching culturally appropriate care: a review of educational models and methods. Academic Emergency Medicine, 13(12), 1288-1295. https://doi.org/10.1197/j.aem.2006.07.031

Japan Ministry of Economy, Trade and Industry. (2019, January). Kokunai iryokikan ni okeru gaikokujin kanja no ukeire jittai chosa [Survey on the actual conditions of foreign patients accepted at domestic medical institutions in Japan]. https://www.meti.go.jp/policy/mono_info_service/healthcare/iryou/inbound/activity/survey_report.html

Japan Ministry of Health, Labour and Welfare. (2021, March). Gaikokujin kanja no ukeire no tameno iryokikan muke manyuaru [A manual for medical institutions to accept foreign patients]. https://www.mhlw.go.jp/content/10800000/000795505.pdf

Japan Tourism Agency. (2018, February). Shukuhakuryoko tokeichosa hokokusho [Report on the survey of accommodations and travel statistics]. Ministry of Land, Infrastructure, Transport and Tourism. https://www.mlit.go.jp/common/001220398.pdf

Japan Tourism Agency. (2019, March). Honichi gaikokujin ryokosha no iryo ni kansuru jittaichosa ukeire kankyo no seibikyoka wo okonaimashita [Conducted a survey on the actual conditions of medical care for foreign visitors to Japan and strengthened the development of the receiving environment]. Ministry of Land, Infrastructure, Transport and Tourism. https://www.mlit.go.jp/kankocho/news08_000272.html

Japan Tourism Agency. (2021, June). Shukuhakuryoko tokeichosa [Survey of accommodations and travel statistics]. Ministry of Land, Infrastructure, Transport and Tourism. https://www.mlit.go.jp/kankocho/siryou/toukei/content/001413644.pdf

Kelly, M. A., Balakrishnan, A., & Naren, K. (2018). Cultural considerations in simulation-based education. The Asia Pacific Scholar, 3(3), 1-4. https://doi.org/10.29060/TAPS.2018-3-3/GP10 70

Kobayashi, Y., Yoshimitsu, Y., & Kato, S. (2014). Super kyukyu ni okeru kangoshi no gaikokujinkanja nitaishite ninshiki suru mondai to taio no jissai [Nurses’ perceptions of and responses to foreign patients in a super emergency hospitals]. Nihon Seishinka Kango Gakujutsu Shukaishi [The Japanese Psychiatric Nursing Society], 57(3), 379 383.

Kubo, Y., Takaki, S., Nomoto, Y., Maeno, Y., & Kawaguchi, Y. (2014). Nihon no byoin ni okeru kyukyugairai deno gaikokujinkanja heno kango no genjo ni kansuru chosa. [A survey on the current status of nursing care for foreign patients in emergency departments in Japanese jospitals]. Kosei no shihyo [Journal of Health and Welfare Statistics], 61(1), 17-25.

Leake, R., Holt, K., Potter, C., & Ortega, D. M. (2010). Using simulation training to improve culturally responsive child welfare practice. Journal of Public Child Welfare, 4(3), 325-346. https://doi.org/10.1080/15548732.2010.496080

Legoux, C., Gerein, R., Boutis, K., Barrowman, N., & Plint, A. (2021). Retention of critical procedural skills after simulation training: a systematic review. AEM Education and Training, 5(3), e10536. https://doi.org/10.1002/aet2.10536

Mechanic, O. J., Dubosh, N. M., Rosen, C. L., & Landry, A. M. (2017). Cultural competency training in emergency medicine. The Journal of Emergency Medicine, 53(3), 391-396. https://doi.org/10.1016/j.jemermed.2017.04.019

Morell, V. W., Sharp, P. C., & Crandall, S. J. (2002). Creating student awareness to improve cultural competence: creating the critical incident. Medical Teacher, 24(5), 532-534. https://doi.org/10.1080/0142159021000012577

Motola, I., Devine, L. A., Chung, H. S., Sullivan, J. E., & Issenberg, S. B. (2013). Simulation in healthcare education: a best evidence practical guide. AMEE Guide No. 82. Medical Teacher, 35(10), e1511-e1530. https://doi.org/10.3109/0142159X.2013.818632

NHS England. (2016). NHS England response to the specific duties of the Equality Act. Equality information relating to public facing functions. https://www.england.nhs.uk/wp-content/uploads/2016/02/nhse-specific-duties-equality-act.pdf

Office of Disease Prevention and Health Promotion. (2021, August). Disparities. U.S. Department of Health and Human Services, Office of Disease Prevention and Health Promotion. https://www.healthypeople.gov/2020/about/foundation-health-measures/Disparities

Osegawa, M., Morio, H., Nomoto, K., Nishizawa, M., & Sadahiro, T. (2002). Present medical practice and problems in emergency disease in foreign travelers requiring hospital admission. Nihon Kyukyu Igakukai Zasshi [Journal of Japanese Association for Acute Medicine], 13(11), 703-710. https://doi.org/10.3893/jjaam.13.703

Paroz, S., Daele, A., Viret, F., Vadot, S., Bonvin, R., & Bodenmann, P. (2016). Cultural competence and simulated patients. The Clinical Teacher, 13(5), 369-373. https://doi.org/10.1111/tct.12466

Serizawa, A. (2007). Developing a culturally competent health care workforce in Japan: implications for education. Nursing education perspectives, 28(3), 140-144.

Seropian, M. A. (2003). General concepts in full scale simulation: getting started. Anesthesia & Analgesia, 97(6), 1695-1705. https://doi.org/10.1213/01.ane.0000090152.91261.d9

Tatsumi, Y., Sasaki-Otomaru, A., & Kanoya, Y. (2016). The actual situation and issues of emergency medical services for foreigners staying in Japan extracted by systematic review. Nihon Kenko Igakukai Zasshi [Journal of Japan Health Medicine Association], 25(2), 91-97.

Tervalon, M., & Murray-Garcia, J. (1998). Cultural humility versus cultural competence: A critical distinction in defining physician training outcomes in multicultural education. Journal of Health Care for the Poor and Underserved, 9(2), 117-125. https://doi.org/10.1353/hpu.2010.0233

*Sayaka Oikawa
Center for Medical Education and Career Development,
Fukushima Medical University,
1 Hikarigaoka, Fukushima, 960-1295, Japan
Email: sayaka9@fmu.ac.jp

Submitted: 11 February 2022
Accepted: 21 July 2022
Published online: 4 April, TAPS 2023, 8(2), 4-13
https://doi.org/10.29060/TAPS.2023-8-2/OA2755

Tri Nur Kristina¹, Fatikhu Yatuni Asmara², Sudaryanto Sudaryanto¹, Nuryanto Nuryanto³ & Saekhol Bakri¹

¹Department of Medicine, Faculty of Medicine, Universitas Diponegoro, Indonesia; ²Department of Nursing, Faculty of Medicine, Universitas Diponegoro, Indonesia; ³Department of Nutrition, Faculty of Medicine, Universitas Diponegoro, Indonesia

Abstract

Introduction: This study aimed to examine the usefulness of Community Based-Education (CBE) and Interprofessional Education (IPE) to community health outcomes.

Methods: The design was a mixed-method study. Each small group worked together to identify family health problems, implement interventions, and evaluate the results. The Readiness for Interprofessional Learning Scale (RIPLS) questionnaire was applied to compare students’ perceptions before and after the implementation. In addition, qualitative data were acquired from students’ comments on the questionnaire, interviews with community representatives, and FGDs with instructors at the end of the program.

Results: Three hundred and sixty-seven out of 465 students returned the completed paired questionnaire (78.9 % response rate). Paired t-test showed that student perception of responsibility slightly increased, whereas teamwork and collaboration, negative and positive professional identity decreased somewhat. However, most pre-post students’ responses already trended to the positive side (scales 4 and 5). Moreover, the Chi-square test showed that pre-post students’ confidence levels significantly increased. Most small groups of students successfully empowered the community to solve health problems. Students, instructors, and representatives of community members appreciated the program. However, several students felt their instructors were not motivated to mentor their tasks, and sometimes they had difficulty conducting home visits together. 

Conclusion: This study showed that the combination of CBE-IPE contributes to more favourable community health outcomes. However, it was challenging in several aspects of preparation, including highly motivated instructors. Nevertheless, pre and post-implementation students’ attitudes are still positive.

Keywords:           Interprofessional Education (IPE), Community Based Education (CBE), Health Profession Students, Readiness

Submitted: 28 July 2022
Accepted: 12 October 2022
Published online: 3 January, TAPS 2023, 8(1), 25-32
https://doi.org/10.29060/TAPS.2023-8-1/OA2850

Choon Peng Jeremy Wee¹, Mingwei Ng¹ & Pim W. Teunissen²

¹Department of Emergency Medicine, Singapore General Hospital, Singapore; ²School of Health Professions Education (SHE), Faculty of Health Medicine and Life Sciences, Maastricht University, Maastricht, The Netherlands

Abstract

Introduction: This study was performed to understand how fixed clinical teams affected informal learning in Emergency Medicine Residents. Better understanding the effects of team dynamics on informal learning may help to optimise learning and improve performance.

Methods: From 8^th February 2020 till 27^th September 2020, the Singapore General Hospital Emergency Department adopted a fixed team system. Zoom interviews were carried out amongst Emergency Medicine Residents who worked in the fixed team system using a semi-structured iterative interview guide. A qualitative content analysis was used for this exploratory study. The interviews were transcribed verbatim, anonymised and coding via template analysis performed. Data collection and analysis were performed until data sufficiency.

Results: The themes identified centred around relationship dynamics, team composition and motivation for learning. The first was how improved relationships led to improved trust, communications and camaraderie among team members. This improved peer learning and clinical supervision and provided a more personalised learning experience. A balanced team composition allowed learners to be exposed to experts in various subspecialties. Finally, there was an initial increase in motivation, followed by a decrease with time.

Conclusion: In postgraduate medical education, working in a fixed team system with balanced members had positive effects on informal learning by strengthening relationships and communications.

Keywords:           Informal Learning, Workplace Learning, Fixed Teams, Medical Education, Postgraduate

Practice Highlights

• Fixed teams can strengthen relationships between members through better trust, familiarity and communication.
• A balanced fixed team with members having different areas of expertise allows a variety of perspectives.
• Rotation of team members may achieve a balance between the stronger relationship and familiarity of the members of fixed teams and the greater variance in perspectives from a non-fixed team system.

I. INTRODUCTION

Fixed teams (FT) and non-fixed teams (NFT) exist in medicine because of differing service requirements and manpower resources. Examples of FTs can exist in ward round teams and operating surgical teams (Eddy et al., 2016; Stepaniak et al., 2012) where personnel stay within the same work team for long periods. In other areas of healthcare like the Emergency Departments (ED), a NFT system is usually employed where teams are formed according to the personnel rostered to work on that shift and team members change every shift. This allows a more flexible system for the team members as they can request off days and leave according to their personal schedule and yet allows for 24-hour coverage in the ED. 

High levels of performance are required of medical teams, both fixed and non-fixed, to achieve good patient outcomes. Therefore, team members should learn how to work effectively together to deliver the best patient care. There are studies, both within healthcare and other industries, which showed mixed results when FT were compared with NFT with regards to team performance. A systematic review on minimal invasive surgical teams found that the FTs had better teamwork and had reduced rates of technical mistakes compared to NFTs (Gjeraa et al., 2016). However, an aviation study showed that FTs made more minor errors compared to NFTs due to FT members being too familiar with each other and overlooking errors (Barker et al., 1996). Although it is unclear how transferable specific research findings from non-healthcare domains are, what is clear is that FTs and NFTs are different in the way teams were formed and the amount of time team members spend working together. There is a lack of understanding if and how these differences affect the way learning takes place in FT and NFTs; which could translate to the performance of the team and its members. 

Workplace learning occurs through informal learning by experiencing work challenges and via interpersonal relationships. Informal learning can be supported through learner engagement by encouraging active participation in work activities and aligning learners’ interests with that of the organisation’s objectives towards improving the individuals’ and organisation’s capabilities (Billett, 2007). Informal learning is now widely accepted as a form of workplace learning that occurs out of a formal planned teaching program. It usually occurs during work activities which are not primarily aimed towards education, with learning objectives not planned beforehand (Callanan et al., 2011; Rogoff et al., 2016; Watkins et al., 2018). Although informal learning had been studied, there are very few studies looking at how being in FTs affects informal learning. A review on the involvement of employees in workplace learning (Kyndt & Baert, 2013) revealed that there was a paucity of literature on whether any team system improves the involvement of employees in informal learning. Thus, it remains to be studied what effects working in a FT system has on informal learning especially of the more junior team members. 

An integrative literature review on informal learning found that some of the important components of informal learning within members of a team and between teams included interpersonal relationships, feedback, networking and leadership (Jeong et al., 2018). Therefore, there may be differences in interpersonal relationships and feedback between the different team systems. It is known that good interpersonal relationships include good supervisor and peer support and both affect whether what is learnt is applied at the workplace (Burke & Hutchins, 2016), there was little published data on whether supervisor and peer support or even the supervisory relationship were affected by the amount of time spent together. Within some healthcare systems both FTs and NFTs consists of members with varying levels of experience, differing expertise and roles. In a FT, the learners are only supervised by supervisors within that team; therefore, each learner’s supervised time is divided within a small group of supervisors as compared to a NFT system where each learner’s supervised time is spread amongst a larger number of supervisors. Thus, despite this increased time spent together in a FT, it is not clear if working in a FT impacts supervisory or interpersonal relationship. 

Having a good trusted learner-supervisor relationship can result in the establishment of an “Educational Alliance” (Telio et al., 2015). This is because feedback from a credible trusted source was more readily accepted and feedback is another important component of effective informal learning within a team (Jeong et al., 2018).

Furthermore, a study among social work students and their supervisors in a rotational placement model, found that the longer the amount of time they spent with each other the greater the trust between them (Vassos et al., 2017). On the other hand, being in a FT could restrict networking and socialisation to a smaller group of people as contact with other teams’ members could be reduced however it is not known how this could affect informal learning. 

Understanding how informal learning takes place within FT and NFT may allow optimisation of learning within each and perhaps even configure teams to enhance learning and thus ultimately improve performance. Our study aimed to fill this gap in the literature by exploring how fixed clinical teams affected the experience of informal learning for Emergency Medicine (EM) Residents. By doing so we hope to understand how informal learning can be supported via the appropriate implementation of team systems especially where high performance is expected from the teams.

II. METHODS

To study how being in fixed clinical teams affected the experience of informal learning for EM Residents we conducted an exploratory qualitative study based on a constructivism research paradigm using content analysis of individual interviews. This is because informal learning could not be quantified with specific learning outcomes.

A. Setting

EDs teams manage a large number of critically ill patients who may need time sensitive interventions. These teams would comprise of experienced Emergency Physicians (EPs) and more junior Medical Officers (MOs) and Residents. The Residents are postgraduate doctors who are training to graduate as EPs; therefore, informal workplace learning is a crucial part of their training. Hence the residents would be good study subjects to investigate the effect of team systems on informal learning. 

The Singapore General Hospital (SGH) ED functioned via NFTs where the composition and number of members in the team differed with each 8 hour shifts accordingly to the anticipated patient load. The COVID-19 pandemic provided a naturalistic setting where the effect of a FT system can be compared to a NFT system which had existed before hand. After COVID-19 was declared a pandemic by the World Health Organisation (WHO) on the 11^th March 2020 (World Health Organisation, 2020). There was an emphasis on infection control to contain the pandemic. Many countries had instituted social distancing measures which included curfew-like measures and travel restrictions (Lake, 2020). Similarly, the Singapore government had instituted legislative measures to limit face-to-face interactions. In the ED of SGH, measures were put in place to limit the spread of COVID-19.

Thus, from 8^th February till 27^th September 2020, as part of infection control measures, doctors were organised into 5 FTs, each having between 5-7 Consultants, 3-5 EM Senior Residents, 2-3 EM Junior Residents and 7-9 non-EM Medical Officers (MO) (Liu et al., 2020; Quah et al., 2020). The Senior Emergency Physicians (SEPs) consisted of certified specialists in EM (Associate Consultants, Consultants and Senior Consultants); they played supervisory and educational roles to the junior doctors (JD) which included MOs, Junior Residents and Senior Residents. Each FT worked 12-hour shifts. Interactions between teams were kept to a minimum and members from different teams were not allowed to mingle. Thus, the residents were only supervised by their team’s SEPs. Informal learning would now occur within these FTs. 

Formal learning was converted to a remote online platform because of infection control measures. Lectures and tutorials were held and recorded using software which enabled online asynchronous access e.g. Zoom (Zoom, 2016) as not all residents could be given protected learning time together. Sessions which could not be transferred onto an online platform (e.g. hands on simulation and procedure skills training) were cancelled. Formal summative examinations were also cancelled.

B. Interviews

Interviews were conducted and recorded via Zoom (Zoom, 2016) to maintain social distancing. The primary investigator performed 11 interviews and a coinvestigator performed the remaining 4 out of a total of 15 interviews. A semi-structured iterative interview guide was developed based on Eraut’s typology on informal learning which included team activities, tasks and enabling/disabling factors (Eraut, 2010) was used, and the interviews were audio-recorded and transcribed verbatim and anonymised. The interviews allowed positive and negative aspects to be explored and being semi structured the questions asked varied according to the interviewees’ responses. This helped to focus the interviewees to what informal learning was with examples when it could occur within team activities. The guide was iteratively amended with each interview to enhance clarity which helped to obtain more in-depth data in later interviews. 

C. Participants

Twenty-four ED residents were working in the FT system in the ED of SGH during COVID-19. Fifteen were Junior Residents and 9 were Senior Residents. Purposive sampling was carried out with at least two Residents from each team being sampled. This is to ensure that there was good representation for all of the fixed clinical teams. All 24 residents were invited to participate via email and WhatsApp messaging platform with written consent being obtained. Fifteen individual interviews were conducted before data sufficiency was achieved where no new data would change the outcome of the study, thus no further interviews were conducted beyond data sufficiency (Varpio et al., 2017). Eleven interviewees were Junior Residents (4 females and 7 males) and 4 were Senior Residents (3 females and 1 male).

D. Data Analysis

Data analysis was performed via template analysis by the primary and coinvestigator independently (Brooks et al., 2015). Eraut’s typology developed from his research on informal learning was used as a priori themes in the initial coding template (Eraut, 2010). Coding template modifications were made as the analysis of the transcripts continued. Themes were categorised into hierarchical clusters and relationships between them were studied and defined. After final modifications, the coding template was applied to the entire data set. Coding themes were compared and discussed between the primary investigator and the coinvestigator until consensus was reached. 

E. Ethics

Waiver for approval was granted by the Singhealth Institutional Ethics Board. The primary investigator was a core faculty within the Singhealth Emergency Medicine Residency Program and although the interview was conducted among EM residents the primary investigator did not conduct the interviews when the interviewees were from the same team as the primary investigator. These were conducted by the coinvestigator. The coinvestigator was an EM Senior Resident who was not involved in the FT system. A reflexivity diary was kept, and peer debrief was done.

III. RESULTS

Three main themes emerged on how working in FTs affected informal learning amongst our participants (Figure 1). These included changes in relationship dynamics between members, effect of FT composition on informal learning of the participants and influence on motivation.

Figure 1. Themes and sub-themes

Theme 1: Changes in Relationship Dynamics between FT Members

From the interviews, the participants felt that the FT system resulted in more familiarity, trust, teamwork and improved communications between team members including SEPs, Residents and MOs. Interviewees felt that this strengthened the relationship dynamics between FT members as compared to a NFT. This meant that FT members were able to coordinate and exchange information better. It led to an increase in familiarity in knowing each member’s style of practice and way of thinking. The team members could understand each other better and how they reasoned.

The strengthened relationship between residents and SEPs changed with dynamics. Having a “closer rapport” and “deeper bond” allowed the residents to “tag along” with SEPs “more often” and gave the residents more insight as to why the SEPs behaved in a certain way as to “how they had practiced medicine” and the rationale behind each step was “more easily communicated to the residents who were tagging along” (Resident A), resulting in residents having a deeper understanding of why things were done in addition to how things were done. This strengthened relationship was also present between the residents and their peers. Therefore, peer learning improved within the FTs as junior residents reported feeling less reservation in asking each other questions. 

Resident D felt that peer learning was better within FTs because of the improved relationship, there were less reservations which had prevented him from asking his peers questions in a NFT setting. 

He elaborated:

“Fixed team [was] definitely better for peer learning. For the same reasons, because you know each other better, you’re more familiar. We don’t only talk to each other about work… after a while, when you go for meals together… or rather like resting together and no cases around you sit and talk. So there’s a lot more familiarity with the person you’re working with, and… you’re just more comfortable with asking questions… you don’t feel like this is somebody who’s going to judge you if [you] asked a stupid question”

This was not just amongst the residents but also with SEPs. Resident G felt that it was easier for the residents to speak to the SEPs because of familiarity and resulted in less workplace stress:

“Over the time as we knew each other better … the workplace stress was much lower… so I could… work with less stress at workplace… Because if you didn’t really know the consultant you tend to be afraid to talk to them; then of course your stress levels will be higher. But if you know that consultant and you know what kind of person, he or she is then you could be more comfortable to talk to them…”

“…It is more comfortable to… approach the senior because you know every day… we have a fixed team so naturally we feel that our relationship is closer…”

“… so, I won’t be too afraid to speak out or to talk to them to discuss with them.”

Contributing to additional ways of informal learning, communication within FTs even during work took on a more “friendly [and] social” form with greater congeniality and via more “communication platforms” (WhatsApp and Tiger Text) which continued even “outside of work” (Resident H). These platforms were also used as learning tools to facilitate case discussions, share learning points and experiences. This was not previously present in the NFTs. The residents felt that learning was more customised because of the change in relationship dynamics. In FTs, there was closer supervision of residents by SEPs. The residents “spent more time” with the same group of supervisors, thus the supervisors were able to better “assess both strengths, weaknesses and address any particular loopholes” of the residents (Resident A). 

However, there were some adverse effects of this change in relationship dynamics. Some residents felt that with a closer relationship between team members, supervisors sometimes were more tolerant of the learners’ shortcomings and be less likely to point it out because of not wanting to affect the relationship. This closer relationship could result in residents taking “shortcuts” and “flying under the radar” because they knew the SEPs could tolerate or would not scrutinise the learners closely once “trust” was established (Resident L). Resident H gave an example of how familiarity could lead to less critical thinking by the learner:

“If… the senior always does like… [Rapid Sequence Intubation] … even though I question the first time I saw him do that… subsequently … every time I work with him I will do [it] this way. I won’t really think does the patient really need this way [of management] or will the patient benefit in a different way… if you are working with different bosses then every case you need to restart your thinking…”

Theme 2: Effect of Composition of the FT

All interviewees felt that the composition of the FT had affected informal learning, and that having a balanced team in terms of a wide range of years of practice amongst both the SEPs as well as the JDs would help improve informal learning. Having a team where the JDs were of differing seniorities of practice could help with peer learning because the senior ones could help the junior ones more. This also applied to the SEPs as that provided a wider perspective to clinical issues due to having different clinical experience and expertise in different subspecialties. Furthermore, if the JDs in a FT were of a similar level of practice, Resident C expressed that they could be “competing with each other for cases and procedures” which adversely affected informal learning with fewer opportunities to perform procedures. In a NFT the members would be constantly changing and it would be unlikely its JD would be always of the same level of practice.

The interviewees expressed concern that within a FT system that, although residents had close contact with a fixed group of supervisors, they lacked contact with the other teams’ SEPs. Many residents felt that this had adversely impacted informal learning because the SEPs were experts in different subspecialties (e.g., Trauma, Toxicology, Ultrasound, etc). By not interacting with many SEPs, residents were unable to learn from them. Furthermore, different supervisors had different perspectives and approaches to patients which the residents may not be exposed to if they were not in the same team as these seniors. However, this was mitigated by having a team with a balanced variation in the areas of specialty of the seniors. Resident F summarised this:

“…a team with… people from different seniorities are essential… (even) juniors can teach seniors… the way that my team was composed… it was a good mix… there are people from different… specialties… from different seniorities even within the juniors … like first posting to… [senior post graduate years]… offers different perspectives, learning of different things… people from different [subspecialties] can offer insight into the systems-based learning or component from other parts of the institution…”

Theme 3: Influence of a FT System on Motivation

Many residents felt that having FTs increased their motivation to learn. This resulted from their supervisors being able to inspire them and follow up with their learning progress more closely. Resident M was motivated to learn because his “friends (FT members) were very motivating” and “enthusiastic”; this encouraged him to learn more. Furthermore, resident C felt more motivated to learn in a FT because he “always sees the same senior” and this more frequent contact results in him being “more likely to take their feedback and opinion more seriously and work harder”

However, “after some time everyone is comfortable with each other” and some participants feared that their motivation “might go down” (Resident N). This was because there was a feeling of complacency as time went on within a FT, hence the motivation to learn started to dwindle after an initial increase.

Other reasons for this decline were related to COVID-19, the focus was more on facing the threat rather than learning and the priority to learn was secondary. The motivation to learn “was a bit less” as “the mood was more to survive than to learn”; Resident L was “less driven to learn” because there was a “general bleakness in the whole situation of [COVID-19] which made his “inner desire to learn… wane a bit”

IV. DISCUSSION

This study explored how working in fixed clinical teams affected informal learning for EM Residents. There are many pros and cons to fixed team rostering however the focus of this study is on informal learning. The findings highlighted the importance of having a balanced team composition where team members were able to establish trust and a strong bidirectional relationship because of the longer time spent working together. Motivation to learn increased initially; however after some time, some felt a decrease. This was consistent with prior work which highlighted team dynamics and commitment and that feedback which was given often and in a socially interactive environment were factors which helped to enable effective work-based learning (Attenborough et al., 2019; Jeong et al., 2018; Kyndt et al., 2016). Unlike in Attenborough’s work where team leadership was one of the factors identified; our respondents did not mention the effect of team leadership on informal learning. Their focus was more on the relationships between the different team members. From this study the predominant factors which positively affected informal learning included teamwork, collaborative task performance, where good communication was needed between different people, and personal development especially in building interpersonal relationships and group decision making.

Limited studies were done on how FTs affect informal learning. Our study found that FTs resulted in more (informal) communication channels (e.g. WhatsApp) being formed which was not present in NFTs, resulting in more learning activities including sharing ideas, resources and experiences. These sharing activities were some of the major forms of informal learning activities identified in the literature (Lohman, 2006). FTs resulted in open communication and quality feedback which was well received, and were found to be beneficial towards informal learning (Jeong et al., 2018). Our study showed that working in a FTs led to more customised learning. Findings of improved communications and strengthened relationships in a bidirectional manner involving teacher and learner alike, supports a shift from a predominantly teacher to learner type of dynamics to a team learning dynamics where all team members can learn with and from one another. This is important because informal learning takes place effectively when learning from past mistakes and feedback exchange occurs, involving both cognitive and social interactions (Jeong et al., 2018).

FTs had negative effects on informal learning as well. Familiarity resulted in supervisors being more tolerant of shortcomings and FTs limited learners’ contact with other teams’ supervisors and adversely affected informal learning. This was because informal learning also takes place when there is sharing of ideas, expertise and experience (Lohman, 2006) and limiting the number of supervisors limits the variance of shared viewpoints.

Our study has limitations. Firstly, interviewee recall bias was possible because 6 months had passed after the FT system was stopped before the interviews. Therefore, some details may not have been accurately recalled given this period which could affect the trustworthiness of results. Secondly, the participants were likely to be comparing their experiences in the FT system during COVID-19 to a NFT system without a pandemic. Thus, some of the experienced changes may be because of the pandemic rather than purely due to the FT system. Thirdly, there are many pros and cons to FTs however the focus of this study is on informal learning thus other factors not investigated with this study may affect the feasibility of FT.  Lastly there could be power differential effects between the interviewers and the interviewees because the interviewers performed supervisory and roles to the residents. However, to mitigate this, a reflexivity diary was kept, and peer debrief between the two interviewers was performed. Furthermore, the interviewers did not interview members who had been in the same team as them.

V. CONCLUSION

In conclusion, FTs impact informal learning by building strong relationships with improved team communications and adding a social dimension for learning. A balance of team members as well as rotating the residents across different FTs may be beneficial for improving informal learning for EM Residents.

Notes on Contributors

Dr Wee Choon Peng Jeremy submitted the CIRB application, (with the help of the last author) conceptualised the study and its design. He performed the literature review, recruited and interviewed the participants, collected and transcribed the data, performed the thematic analysis of the data and wrote the manuscript. 

Dr Ng Mingwei helped to recruit and interview some of the participants, transcribed and collected the data. Dr Ng helped perform the thematic analysis of the data and helped edit the manuscript.  

Prof. Dr. Pim Teunissen was central to the conceptualisation of the study, advised on the design of the study and gave critical feedback to the writing of the manuscript and edited the manuscript extensively

All the authors have read and approved the final manuscript. 

Ethical Approval

We have included the letter for waiver of CIRB via email. Ethics approval for waiver of written informed consent was obtained from the Singhealth Institutional Review Board (CIRB Ref: 2020/3114).

Data Availability

As the data set is qualitative in nature, we are not able to upload that in any public repository.

Funding

There is no funding for this paper/study.

Declaration of Interest

The authors report there are no competing interests to declare. 

References

Attenborough, J., Abbott, S., Brook, J., & Knight, R. A. (2019). Everywhere and nowhere: Work-based learning in healthcare education. Nurse Education in Practice, 36, 132-138. https://doi.org/10.1016/j.nepr.2019.03.004

Barker, J. M., Clothier, C. C., Woody, J. R., McKinney, E. H., Jr., & Brown, J. L. (1996). Crew resource management: A simulator study comparing fixed versus formed aircrews. Aviation, Space, and Environmental Medicine, 67(1), 3-7.

Billett, S. (2007). Constituting the workplace curriculum. Journal of Curriculum Studies, 38(1), 31-48. https://doi.org/10.1080/00220270500153781

Brooks, J., McCluskey, S., Turley, E., & King, N. (2015). The utility of template analysis in qualitative psychology research. Qualitative Research in Psychology, 12(2), 202-222. https://doi.org/10.1080/14780887.2014.955224

Burke, L. A., & Hutchins, H. M. (2016). Training transfer: An integrative literature review. Human Resource Development Review, 6(3), 263-296. https://doi.org/10.1177/1534484307303035

Callanan, M., Cervantes, C., & Loomis, M. (2011). Informal learning. Wiley Interdisciplinary Reviews Cognitive Science, 2(6), 646-655. https://doi.org/10.1002/wcs.143

Eddy, K., Jordan, Z., & Stephenson, M. (2016). Health professionals’ experience of teamwork education in acute hospital settings: a systematic review of qualitative literature. JBI Database of Systematic Reviews Implementation Reports, 14(4), 96-137. https://doi.org/10.11124/JBISRIR-2016-1843

Eraut, M. (2010). Informal learning in the workplace. Studies in continuing education, 26(2), 247-273. https://doi.org/10.1080/158037042000225245

Gjeraa, K., Spanager, L., Konge, L., Petersen, R. H., & Ostergaard, D. (2016). Non-technical skills in minimally invasive surgery teams: A systematic review, Surgical Endoscopy, 30(12), 5185-5199. https://doi.org/10.1007/s00464-016-4890-1

Jeong, S., Han, S. J., Lee, J., Sunalai, S., & Yoon, S. W. (2018). Integrative literature review on informal learning: Antecedents, conceptualizations, and future directions. Human Resource Development Review, 17(2), 128-152. https://doi.org/10.1177/1534484318772242

Kyndt, E., & Baert, H. (2013). Antecedents of employees’ involvement in work-related learning. Review of Educational Research, 83(2), 273-313. https://doi.org/10.3102/0034654313478021

Kyndt, E., Vermeire, E., & Cabus, S. (2016). Informal workplace learning among nurses. Journal of Workplace Learning, 28(7), 435-450. https://doi.org/10.1108/JWL-06-2015-0052

Lake, M. A. (2020). What we know so far: COVID-19 current clinical knowledge and research. Clinical Medicine, 20(2), 124-127. https://doi.org/10.7861/clinmed.2019-coron

Liu, Z., Teo, T. L., Lim, M. J., Nadarajan, G. D., Segaram, S. S. C., Thangarajoo, S., Wee, L. E., Wee, J. C. P., & Tan, K. B. K. (2020). Dynamic emergency department response to the evolving COVID-19 pandemic: The experience of a tertiary hospital in Singapore. Journal of the American College of Emergency Physicians Open, 1(6),1395-1403. https://doi.org/10.1002/emp2.12264

Lohman, M. C. (2006). Factors influencing teachers’ engagement in informal learning activities. Journal of Workplace Learning, 18(3), 141-156. https://doi.org/10.1108/13665620610654577

Quah, L. J. J., Tan, B. K. K., Fua, T. P., Wee, C. P. J., Lim, C. S., Nadarajan, G., Zakaria, N. D., Chan, S. J., Wan, P. W., Teo, L. T., Chua, Y. Y., Wong, E., & Venkataraman, A. (2020). Reorganising the emergency department to manage the COVID-19 outbreak. International Journal of Emergency Medicine, 13(1), Article 32. https://doi.org/10.1186/s12245-020-00294-w

Rogoff, B., Callanan, M., Gutiérrez, K. D., & Erickson, F. (2016). The organization of informal learning. Review of Research in Education, 40(1), 356-401. https://doi.org/10.3102/0091732×16680994

Stepaniak, P. S., Heij, C., Buise, M. P., Mannaerts, G. H., Smulders, J. F., & Nienhuijs, S. W. (2012). Bariatric surgery with operating room teams that stayed fixed during the day: A multicenter study analyzing the effects on patient outcomes, teamwork and safety climate, and procedure duration. Anesthesia and Analgesia, 115(6), 1384-1392. https://doi.org/10.1213/ANE.0b013e31826c7fa6

Telio, S., Ajjawi, R., & Regehr, G. (2015). The “Educational Alliance” as a framework for reconceptualizing feedback in medical education. Academic Medicine, 90(5), 609-614. https://doi.org/10.1097/ACM.0000000000000560

Varpio, L., Ajjawi, R., Monrouxe, L. V., O’Brien, B. C., & Rees, C. E. (2017). Shedding the cobra effect: Problematising thematic emergence, triangulation, saturation and member checking. Medical Education, 51(1), 40-50. https://doi.org/10.1111/medu.13124

Vassos, S., Harms, L., & Rose, D. (2017). Supervision and social work students: Relationships in a team-based rotation placement model. Social Work Education, 37(3), 328-341. https://doi.org/10.1080/02615479.2017.1406466

Watkins, K. E., Marsick, V. J., Wofford, M. G., & Ellinger, A. D. (2018). The evolving Marsick and Watkins (1990) theory of informal and incidental learning. New Directions for Adult and Continuing Education, 2018(159), 21-36. https://doi.org/10.1002/ace.20285

World Health Organisation. (2020). World Health Organisation emergencies press conference on coronavirus disease outbreak – 11 March 2020. https://www.who.int/director-general/speeches/detail/who-director-general-s-opening-remarks-at-the-media-briefing-on-covid-19—11-march-2020

Zoom. (2016). Security guide. Zoom Video Communications Inc.

*Wee Choon Peng Jeremy
Outram Road,
Singapore 169608
Email: jeremy.wee.c.p@singhealth.com.sg

Submitted: 26 November 2021
Accepted: 21 July 2022
Published online: 3 January, TAPS 2023, 8(1), 33-42
https://doi.org/10.29060/TAPS.2023-8-1/OA2712

Jaime Maria Tan¹, Junaidah Binte Badron¹ & Sashikumar Ganapathy^1,2

¹Department of Emergency Medicine, KK Women’s & Children’s Hospital, Singapore; ²Duke-NUS Medical School, Singapore

Abstract

Introduction: Perceptions towards the working and learning environment as well as coping mechanisms have been studied across different healthcare sectors. They have shown to reduce stress and burnout. However, perceptions of the work environment in the Emergency Department (ED) setting have not been studied in depth. The literature surrounding coping mechanisms also mostly focuses on their impacts rather than the mechanisms utilised. In addition, these were often investigated using surveys. This study aimed to use a phenomenological approach to explore the perceptions and coping strategies of junior doctors working in a paediatric ED.

Methods: Sixteen junior doctors working in the Paediatric ED were recruited. Semi-structured interviews were conducted after conducting literature reviews. Data was collected until saturation point. All interviews were recorded and transcribed verbatim manually and subsequently analysed.

Results: The greatest fears of junior doctors starting their paediatric emergency posting were lack of knowledge due to inexperience in the subspecialty; fear of the work environment due to unfamiliarity as well as workload and the intrinsic high-stress environment. The main coping strategies were ensuring clinical safety, obtaining psychosocial support from loved ones and colleagues, and placing focus on spirituality and wellbeing.

Conclusion: In this study, the perceptions and coping strategies of the junior doctors in the Paediatric ED were explored. The findings from this study will help to structure and improve the support given to future junior doctors who rotate to the department as well as better orientate them to allay their pre-conceived notions.

 Keywords:          Coping Behaviours, Perceptions, Paediatric Emergency Department, Stressors, Interviews

Practice Highlights

• Participants worried about knowledge, workload and responsibilities prior to starting their posting.
• Perceptions were mostly of an anticipatory nature, influenced by seniors’ past experiences.
• In work, support from senior staff was helpful in allaying their fears and increasing patient safety.
• Participants felt psychosocial support, spirituality and wellness were useful coping strategies.
• Maladpative coping strategies did not come up as a main theme in our study.

I. INTRODUCTION

Perceptions toward the work and learning environment can strongly impact experiences and even lead to large amounts of stress (Chan et al., 2016). A poorer perception of the learning environment is also associated with greater levels of burnout (Chew et al., 2019; Sum et al., 2019). Conversely, a positive perception of the work environment helps to alleviate stress (Abraham et al., 2018). Workers’ perception of their work environment contributes significantly to their overall experiences.

Main factors contributing to stress in the ED include heavy workload and critically ill patients. Workplace violence, trauma, abuse and morbidity also add to the stress and burnout experienced (Burbeck et al., 2002; Copeland & Henry, 2018; Healy & Tyrrell, 2011; Xu et al., 2019). In the paediatric setting, added stressors include dealing with sexual abuse and non-accidental injury as well as death and the inability to provide optimal care for children (Alomari et al., 2021; Basu et al., 2016; Durand et al., 2019; Greenslade et al., 2019; Shanafelt et al., 2012; Watson et al., 2019).

Given these significant stressors, individuals utilise different coping mechanisms to mediate these experiences (Howlett et al., 2015).

Some coping strategies discussed in previous studies include socialising with friends and family (Gribben et al., 2019). Focusing on physical wellbeing, clinical variety, reflectivity, and organizational activities were also helpful in alleviating burnout in other areas of healthcare (Barham et al., 2019; Koh et al., 2015).

Several studies also found that the use of maladaptive coping mechanisms, such as alcohol use and self-blame increased with the frequency of burnout (Jackson et al., 2016; Oreskovich et al., 2015; Ryali et al., 2018; Talih et al., 2018).

While many studies studied stressors and the effectiveness of the coping mechanisms used, the actual components of coping mechanisms were not well studied. In studies that did look at coping mechanisms and their effectiveness, these studies were also often done via the survey method and were only evaluated on the surface.

Most studies looked at healthcare workers in general. Few studies looked solely at the doctor population. This makes conclusively evaluating the doctor component of coping mechanisms and their effects difficult.

While other studies looked at an adult emergency perspective, there were also few studies looking at the paediatric ED. It has been reported that dealing with paediatric emergencies causes more stress compared to their adult equivalents. Some of the contributing factors are related to the nature of working with children. These, in itself, are unmodifiable (Guise et al., 2017). Therefore, it is important to study how the paediatric context can affect the experiences of the doctors who care for them.

In our study, we studied the perceptions of junior doctors at the beginning of their posting. We subsequently explored their coping mechanisms in the Singaporean context.

The element of stress in the ED among junior doctors is significant as the ED is often part of many specialist training pathways (Mason et al., 2015). During the time of training, the doctors are still learning and developing. Hence, many doctors experience sharp learning curves during their postings. This brings about more stress (McPherson et al., 2003). In some cases, the stress can even lead to doctors thinking about leaving clinical practice altogether (Degen et al., 2014).

In the Singaporean context, paediatric emergency postings are part and parcel of speciality training for junior doctors (especially for those in emergency medicine and family medicine training). Because of this, junior doctors spend the majority of their paediatric postings in the paediatric emergency. As such, a Singapore-specific context would give light on the challenges of this sizable group.

The nature of the healthcare system in Singapore is unique. Up to 60% of the consultations in the paediatric ED were for nonurgent conditions due to the overall perception of the severity of symptoms and parental preference towards paediatric specialist facilities (Ganapathy et al., 2015). This would lead to an increased workload for the paediatric ED. The distribution in workload may also differ compared to a global perspective, with the load of severe paediatric trauma in Singapore being low (Pek et al., 2019).

These subtleties in the paediatric ED in Singapore can influence the experiences of junior doctors differently. With these key differences in mind, we aimed to investigate the perceptions of junior doctors towards their paediatric ED posting in Singapore and how they subsequently coped with the challenges faced.

II. METHODS

A. Design

In this study, we examined the experiences of doctors in their paediatric ED rotation and how their thoughts and actions influenced their stress during their rotation. We deemed the phenomenological approach to be the most appropriate for this study. Phenomenology is defined as the study of how individuals see and experience a phenomenon and what this means to the individuals in their own experience (Neubauer et al., 2019; Smith, 2021).

The approach we chose was that of an interpretive phenomenological analysis in which we aimed to investigate the experience through the participants’ own experiences and perceptions. With the help of the various participants’ accounts, themes and ideas bound by their experiences were explored (Tuffour, 2017).

B. Methods

The members of the study performed a preliminary literature review on the topic and explored plausible methods of data collection. The study team decided on semi-structured interviews as it promotes sharing and would allow for sufficient privacy.

The team members included a senior consultant, a staff physician and a medical officer. Together, after discussions about concepts that the team was keen to explore, an interview guide was drawn up.

Subsequently, a proposal was submitted to the Hospital Centralised Institutional Review Board for approval.

One-on-one interviews were conducted with the participants by investigator A, a medical officer who was rotating within the department at the time of the study. This was done to reduce the power differential. Interviews were conducted at a location and time convenient to the participant.

Prior to the interviews, consent was sought and all interviews were recorded and subsequently transcribed verbatim. The interviews were conducted over a 1-month period in December 2019.

Questions were open-ended and allowed participants to share ideas that they were keen to raise with no restrictions to the topics brought up. Interview questions were tweaked alongside subsequent interviews so that they were easier to understand and would encourage sharing. Additional questioning in subsequent interviews was adopted to improve clarity. For example, one of the questions that featured early in the interviews was ‘What are some of the coping mechanisms you use?’ During subsequent interviews we noticed some participants utilised coping mechanisms before work to prepare themselves, some used other strategies during work to cope with the stress, while others dealt with their stressors after getting off work. We tweaked the question to include ‘during the shift or outside of the shift’ to help participants widen their perspective about certain coping methods they may have used but were not immediately conscious of when answering the questions. No new questions referring to particular themes were inserted although interviewers were aware of the themes that had been highlighted in previous interviews. This was done in addition to the initial interview guide and ensured the broad nature of questioning was not compromised and the breadth of interviews was maintained.

Themes were identified from the interviews until data saturation was reached. Data saturation was noted at the 12th interview. The team continued to learn from subsequent interviews, with interviews contributing additional depth to the issues explored. Further interviews were conducted to confirm that no new theme was being identified.

The interviews were then transcribed and de-identified. They were subsequently reviewed by 2 reviewers (Investigator A and Investigator B). Data was analysed using a step-by-step thematic analysis method (Braun & Clarke, 2006). Investigators A and B independently analysed the transcripts, identified themes and later reported the  common themes. These themes were discussed for concurrence. When any differences in opinion arose, these would be reviewed by investigator C to resolve any disagreement.

C. Setting

The research was conducted within the Paediatric ED in KK Women’s and Children’s Hospital, a tertiary paediatric hospital.

The Children’s Emergency of KK Women’s and Children’s Hospital is the largest paediatric emergency unit in Singapore. During the time of the interviews, the department treated over 400 patients daily. The Children’s Emergency sees all children under the age of 18 years for all medical complaints.

The department is staffed by over 60 junior doctors at a single time. These junior doctors come from various backgrounds and pass through the department for varying amounts of time. Thus, their experience can be very heterogeneous.

The job scope and responsibilities of all the junior doctors are primarily the same despite the different levels of experience. They are expected to treat the patients that present to the ED. These doctors can seek advice from the senior doctors who are on the ground. However, for the majority of the time, they would be tasked to treat patients on their own.

D. Participants

Participants were recruited through an email that was circulated to all junior doctors in the department. Participation in the study was voluntary and participants were not remunerated.

A total of 16 junior doctors were recruited and interviewed over a 1-month period. Due to the busyness of the ED and the limited time frame in which the interviews were conducted, only 16 interviews were conducted. Convenience sampling was chosen for the sampling method. The first 16 volunteers who had volunteered were interviewed. However, it was noted that saturation point was reached prior to the conclusion of the interview process.

The variety within the ED was well represented. The details of participant breakdown are elaborated on in Table 1.