Submitted: 30 August 2020
Accepted: 9 December 2020
Published online: 13 July, TAPS 2021, 6(3), 56-66
https://doi.org/10.29060/TAPS.2021-6-3/OA2440

Gayathri Devi Nadarajan¹, Kirsty J Freeman², Paul Weng Wan¹, Jia Hao Lim¹, Abegail Resus Fernandez² & Evelyn Wong¹

¹Department of Emergency Medicine, Singapore General Hospital, Singapore; ²Office of Education, Duke-NUS Medical School, Singapore

Abstract

Introduction: COVID-19 challenged a graduate medical student Emergency Medicine Clinical Clerkship to transform a 160-hour face-to-face clinical syllabus to a remotely delivered e-learning programme comprising of live streamed lectures, case-based discussions, and telesimulation experiences. This paper outlines the evaluation of the telesimulation component of a programme that was designed as a solution to COVID-19 restriction.

Methods: A mixed methods approach was used to evaluate the telesimulation educational activities. Via a post-course online survey student were asked to rate the pre-simulation preparation, level of engagement, confidence in recognising and responding to the four clinical presentations and to evaluate telesimulation as a tool to prepare for working in the clinical environment. Students responded to open-ended questions describing their experience in greater depth.

Results: Forty-two (72.4%) out of 58 students responded. 97.62% agreed that participating in the simulation was interesting and useful and 90.48% felt that this will provide a good grounding prior to clinical work. Four key themes were identified: Fidelity, Realism, Engagement and Knowledge, Skills and Attitudes Outcomes. Limitations of telesimulation included the inability to examine patients, perform procedures and experience non-verbal cues of team members and patients; but this emphasised importance of non-verbal cues and close looped communication. Additionally, designing the telesimulation according to defined objectives and scheduling it after the theory teaching contributed to successful execution.

Conclusion: Telesimulation is an effective alternative when in-person teaching is not possible and if used correctly, can sharpen non-tactile aspects of clinical care such as history taking, executing treatment algorithms and team communication.  

Keywords:           Telesimulation, COVID-19, Emergency Medicine, Programme Evaluation

Practice Highlights

• Telesimulation doesn’t replace but can be an effective alternative when in-person teaching is not possible.
• When implemented correctly, it can sharpen non-tactile aspects of clinical care.
• It is possible to achieve a level of fidelity and realism in a telesimulation environment.
• Simulation faculty needs to be skilled in debriefing techniques that enable the learner to reflect.
• Limitations of telesimulation can be reframed as learning opportunities.

I. INTRODUCTION

COVID-19 brought about unexpected challenges to medical education, especially to student clinical clerkships where medical students would spend time within a clinical discipline, interacting with clinicians and learning from patients. Healthcare institutions restricted student movement within clinical environments and barred students from entering the high-risk frontline areas to reduce exposure risk.

Prior to COVID-19, students undertaking an Emergency Medicine (EM) Clinical Clerkship, would have the opportunity to manage and deliver care to high acuity patients, with bedside teaching, small group tutorials, problem-based learning and simulation modalities. With COVID-19, students were not permitted into the Emergency Department (ED) and face-to-face teaching activities were halted. Hence this clerkship had to be conducted remotely. The EM clerkship was transformed from a 160-hour clinical programme to a remotely administered programme comprising 40 hours of e-learning, 40 hours of interactive live online session and 15 hours of telesimulation. As part of this programme, we decided to utilise telesimulation to help students achieve some of the objectives of a clinical clerkship.

Telesimulation is defined as the “Process by which telecommunication and simulation resources are utilised to provide education, training and/or assessment to learners at an off-site location” (McCoy et al., 2017). By allowing simulation to be conducted through devices such as the computer and phone, it mitigates the problem of physical proximity. Though telesimulation has existed for about a decade, its utilisation appears limited to the rural settings and studies mainly describe its usage for learning skills (Mikrogianakis et al., 2011; Naik et al., 2020; Okrainec et al., 2010) rather than for emergency management of patients. With the need to adapt teaching to remote experiences, telesimulation is gaining popularity (Sa-Couto & Nicolau, 2020).

A. Programme Overview

This remote learning programme was developed in a tertiary ED in Singapore which receives both undergraduate and postgraduate medical students for their EM clerkship. There were 58 postgraduate medical students undertaking their 4-week EM clerkship in June 2020. The EM core clinical training curriculum was taught by EM faculty via online modules and interactive classroom sessions delivered via the video conferencing platform, Zoom. The learners spent the mornings in interactive online sessions with faculty, and afternoons in self-study as part of a flipped classroom learning, using provided learning materials. The telesimulation session was scheduled in the last week, over five days. The students were split into ten groups each comprising of five to six students, where two groups participated in one telesimulation session each day.

Our objectives for this telesimulation programme was to ensure that the students could take a focused clinical history from the simulated patients, communicate with them, construct a list of differentials and manage them  accordingly in the emergency setting. The secondary objectives were to train them to prioritise the investigations and management of critically ill patients and to communicate and work effectively within a team. Using Kern’s six step approach, the team of simulation and clinical educators’ planned and implemented the telesimulation activity to achieve these outcomes (Harden et al., 1999; Smith & Dollase, 1999) during their EM clerkship.

As medical students, the learners are at a novice stage according to The Dreyfus Model of Skill Acquisition (Benner, 2004; Dreyfuss & Dreyfus, 1980). Hence, the deliberate attempt not to assess skills such as intubation or defibrillation through telesimulation as it may create unnecessary anxiety and feelings of incompetence (Papanagnou, 2017). Furthermore, it was deemed challenging to conduct procedural skill teaching through this modality. Instead, the focus was on clinical reasoning and patient management. The clinical students fall under the category of “show how” within the Miller’s pyramid, with regards to history taking, clinical reasoning and management. As adult learners, a problem centred (Knowles, 1990), experiential learning approach (Kolb, 1984) would be more valuable. Hence telesimulation was an appropriate modality. 

Each telesimulation session was conducted by two simulation and one physician faculty. There was a total of four scenarios for each session, where one group, consisting of five to six students, will participate in the scenario, while the other group observes, before switching. This allowed each group of students to participate in two clinical scenarios. The topics chosen for telesimulation were Anaphylaxis, Cardiac Arrest, Meningitis, and Syncope where the theory was covered in the core topics in the preceding weeks. Each of these scenarios began with the students taking a history from the simulated patient, before the patient progressively deteriorated and required resuscitation. Figure 1 shows a summary of the scenarios. The scenarios were selected as they did not require much procedure-based interventions (e.g. chest tube insertion in a poly trauma victim) which would be difficult to assess via Zoom.

Figure 1. Brief case description of simulation scenarios and visual presentation of the flow of the telesimulation experience

The sessions commenced with a briefing where the students were orientated to the online environment, including the use of video and microphones. As depicted in Figure 1, using the share screen feature, the simulation technician switched between different views. The briefing included a photo of the patient as a visual cue, along with the text of the presenting case. One of the simulation faculty played the circulating nurse, providing prompts to aid students’ engagement and asking participants to clarify their statements or orders as the scenario progressed. Using existing mannequin software and ensuring sharing of screen sounds, real-time patient monitoring was provided to the learners when requested. Upon request, additional visual cues of investigation results would be displayed, reverting back to either a picture of the patient or the patient monitoring. With their video off, the clinical faculty voiced the patient. At the conclusion of the scenario, all participants and faculty turned their video and microphones back on to participate in the large group debrief before proceeding on to the next scenario. 

The objective of this paper is to describe the students’ experience of telesimulation as part of an online clerkship programme and how such techniques can be used to meet learning outcomes (Harden et al., 1999) in various settings. At the time of writing, there is no literature describing evaluation of the use of a telesimulation programme within the ED for medical student education, with this paper aiming to address this gap.

 II. METHODS

A mixed methods approach was used to evaluate the introduction of telesimulation to the EM online clerkship, and to gain students’ perspective on learning through telesimulation. Programme evaluation research aligns with a mixed methods approach as the collection of both quantitative and qualitative data provide a deeper understanding of the student experience (Cohen et al., 2011).

A. Participants, Data Collection and Analysis

58 final year medical students who participated in the EM Online Clerkship programme were invited to participate in a post-telesimulation activity evaluation survey. Using a 5-point Likert scale, students were asked to indicate their agreement on 11 items addressing pre-simulation preparation, their level of engagement, confidence in recognising and responding to the four clinical presentations and telesimulation as a tool to prepare for working in the clinical environment. Seven open-ended questions were asked to enable the students to describe their experience in greater depth. 24hrs after completing the telesimulation session, students received an email with a link to the survey. Qualtrics online survey software was used to build, distribute and collect the survey responses. Voluntary consent was assumed by participation in the anonymised online evaluation. A statement outlining the purpose of the survey was included at the start of the survey and require an agreement before the survey could be commenced. Completion of the survey therefore implied consent.  The survey took between three and five minutes to complete, all responses were anonymous, with no identifiable data collected.

Descriptive statistics was used to analyse the responses to the Likert scale questions, with thematic analysis of the open-ended survey questions. Author one (GN) and author two (KF) reviewed the transcripts separately, making note of key phrases, outline possible categories or themes. Both authors then jointly rearranged and renamed the codes, developing higher order themes. NVivo 12™ was used to store, code and manage the qualitative data.

III. RESULTS

Of the 58 students who were invited to participate in the survey, 42 complete responses were received, a response rate of 72.4%. As seen in table 2, the results demonstrated that 97.62% of respondents agreed/strongly agreed that participating in the telesimulation session was interesting and useful to their learning. In relation to the use of visual and auditory cues, 93% of respondents felt that these helped them engage in the simulation. In relation to their level of preparedness to participate in the telesimulation experience, nearly 17% of respondents reported that the pre-session handout did not adequately preparing them for what to expect in the session.

Table 1. Results of the student responses to the Likert scale items

When asked to rate if they felt more confident recognising and responding to the four clinical presentations (anaphylaxis, cardiac arrest, meningitis, and syncope), between 90% and 93% agreed/strongly agreed that participating in the telesimulation sessions resulted in them feeling more confident in recognising and responding to the specific clinical presentations (Figure 2).

Figure 2. Student rating to the question “I feel more confident in recognising and responding to a patient with …”

Four key themes were identified following the data analysis of the open-ended survey questions, describing around the telesimulation experience of the respondents: 1) Fidelity; 2) Realism; 3) Engagement; and 4) Outcomes. As demonstrated in Figure 3, the themes do not exist in isolation, but intersect as they describe the telesimulation experience that the students had. The students feedback reflected the benefits and limitations which fall under these main themes.

Figure 3. Themes reflecting the students experience with telesimulation

A. Fidelity – Physical, Psychological and Conceptual

The theme Fidelity reflects how closely the telesimulation mimics or reproduces, reality.  Subthemes of conceptual, physical and psychological fidelity were also reflected. The students’ feedback reflected limitations in physical fidelity while conceptual and psychological fidelity was present mostly.

They reported that the auditory and visual stimulus from the dynamic display of investigations and real-time vital signs monitoring, provided a high level of physical fidelity.        

“Auditory and visual information on patients’ vitals and results were really helpful in generating the differential list.”

Student 37 

“The noise and sights is a good proxy for real life cases in a virtual environment.”

Student 4 

“Seeing the vitals of the patient in real-time allows us to experience the importance of time in managing critically ill patients”

Student 20 

However, aspects of physical fidelity, particularly related to the patient assessment, were reported as challenging via telesimulation. With the patient represented as a static picture, voiced by the clinical faculty, students shared how the lack of non-verbal and visual cues from the patient impacted their ability to perform a physical assessment of their patient.

“….we don’t get to observe the body language of the patient as much as we would like”  

Student 12 

“More difficult than in real life. Seeing and hearing a real patient gives much information”

 Student 15 

“I think what is lacking is being able to visually evaluate the patient”

Student 21

In terms of the level of psychological fidelity, the auditory cues from the ‘patient’ and the real-time vital signs monitoring simulated the ED resuscitation room, which appears to have instilled a similar sense the stress and the need to think under pressure, as reflected by the students’ feedback.

“Have to work around the distractions of beeping monitors, seizing patient, teammates asking questions/suggestions.”

Student 33

“It’s dynamic and gives us the opportunity to think under pressure.” 

Student 13 

“Stressful but probably close to reality?” 

Student 35

The students’ statements reflected the subtheme of conceptual fidelity, where they felt the context and sequence was similar to what they would encounter in the ED, where they are required to deliver timely and lifesaving treatment. This was possibly because the faculty made deliberate attempts to ensure that events during the simulation would unfold as it usually would in the ED room, based on the learners’ actions. 

“It simulates a clinical environment with real time updates of vitals and test results in addition to the history and communications.”

Student 5 

What the students did report struggling with however, was the limitations of the platform in terms of multiple actions occurring simultaneously. Unlike in real life, multiple tasks could not be performed at the same time over the online platform, and this impeded the conceptual fidelity.

“In reality, multiple interventions would be carried out in tandem.”

Student 35 

“More challenging to perform tasks concurrently over Zoom.”

Student 36 

“…many things cannot occur concurrently.”

Student 2

B. Realism

The theme Realism captures the degree to which the experience represented details or situations in a way that is accurate or true to life. Students reported that aspects of the telesimulation experience represented what they thought an actual ED encounter would be like.

“I think the process is similar to the actual clinical environment, it is difficult, especially when the patient is deteriorating in front of you, and your team are waiting for you to make the decisions.”

Student 41

“The pictures/videos and the beeping of the monitor, they make it more real”.

                                                                           Student 42 

“It was realistic as getting the differentials was time sensitive”.

Student 13

The students also acknowledged the limitations in achieving realism presented by telesimulation as the various team members could not perform tasks simultaneously and take in cues from the patient to assess the outcome of their actions. 

“Harder to communicate with my teammates compared to real life because only one person could speak at a time while in real life, multiple conversations could be occurring”

                                                                           Student 3 

“More difficult than taking a history in real life – more technical issues (can’t hear properly), Can’t see the patient”

Student 8 

C. Engagement

The theme Engagement relates to the level of interest, attention and involvement in the learning experience. The level of fidelity and realism impacts the level of engagement of the learners. Most students were able to immerse themselves and fully engage in the scenarios, possibly because aspects of fidelity and realism were deliberately given close attention during the preparation phase.

“I actually forgot that the patient was being voiced by the clinical tutor”

Student 41

“My heart was racing doing the simulation – what will I be like when I am there for real?”

                                                                           Student 41

However, on the downside, without being together in the same place, some felt that the scenario was too “messy” and “chaotic” and found it difficult to follow.

“It was a little hectic with the many other ongoing tasks in the background”

                                                                           Student 6 

“Easier to detach oneself from the patient (less affected by patient’s appearance, tone of voice, blood, gore, suffering etc.)” 

Student 28

 

At the same time, some students faced technical difficulties, such as small or flickering screen, poor internet connection or poor audio, which were barriers to their engagement.

“…there were some issues hearing the faculty clearly which may affect the quality of learning.”

                                                                           Student 6 

By addressing the concepts of realism and fidelity, the students reported increased levels of engagement, although it appears that technical barriers unique to telesimulation provide challenges for some students achieving a greater level of engagement.

D. Outcomes- Knowledge, Skills and Attitudes

The theme Outcomes encompasses what students feel they learnt from the experience that could be useful in future clinical encounters. Under outcomes, there were sub-themes of knowledge, skills and attitudes. From a knowledge perspective, students reported that the telesimulation reinforced their clinical reasoning to arrive at a differential list.

“It was very useful and helped with our clinical reasoning. It was also useful in learning how to generate differential diagnoses as a team and going down the path of a working differential diagnosis while keeping others in mind.”

Student 32 

Whilst the lack of hands-on practice was acknowledged, the telesimulation environment required them to practice the skills of prioritisation, leadership, teamwork and effective, close loop communication to manage the patient and this accounted for their skills gained.

“I will be able to apply the concept of teamwork, thinking on my feet, thinking broad, and constant reassessment of the unstable patient in my clinical training over the next few months”

                                                                           Student 6 

“Stay calm, go back to first principles, have the approach at your fingertips, make an effort to remember drug doses and administration route”

Student 2

IV. DISCUSSION

In relation to Kirkpatrick’s model for evaluating educational outcomes, the results of this study (table 1) demonstrate achievement of both level one (reactions) and level two (learning) outcomes (Kirkpatrick & Kirkpatrick, 2009). Whilst these findings may not determine the effectiveness of telesimulation, it does however provide insight into the learners’ experience that have highlighted the strengths and limitations of telesimulation, which the authors of this paper believe provides a foundation upon which others can build. 

It is well documented in the simulation literature that fidelity and realism are important concepts that need to be considered when planning simulation-based education (Oliver, 2002). And this was an initial concern by the educators. The lack of a physical ‘patient’ on which to carry out procedures and physical examination could limit the effectiveness of the telesimulation experience. To address this limitation, faculty briefed the learners about the limitations and the strategies, such as the use of a ‘nurse confederate’ to provide clinical information, as well as having visual cues such as pictures and videos to trigger their actions. Interestingly, the feedback suggests that the lack of a physical ‘patient’ to examine, resulted in more emphasis being placed on the audio and visual cues during the session. This allowed the learners to proactively compensate for the lack of tactile cues with audio and visual ones, reinforcing the importance of clinical alerts and alarms. The inability to perform a physical examination provided an opportunity for the debriefer to emphasise the importance of the skill in clinical care.

There was a deliberate attempt to create scenarios that were commonly seen in the ED in as much details as possible to achieve realism in the virtual space. This limited the scenarios that could be used as we had to use ensure procedures were not required for the patient management to progress (for example, trauma was deemed inappropriate). The faculty feel that the typical, non-complex ED scenarios compounded with the sequence of events as it would occur in real life possibly contributed to the student’s perception of realism during the telesimulation.

Instructional scaffolding was key to student engagement.  The faculty configured the telesimulation session to be held after three weeks of interactive and didactic sessions on Zoom. This allows the learners to acquire essential knowledge which they can then apply during the telesimulation session. With the background baseline knowledge, the telesimulation setup and audio and visual prompts of a real ED environment, the faculty felt that they were able to immerse the students within the scenario rather than conducting it as an online Problem-Based-Learning session. This may have contributed to their engagement.

Communication skills were a common thread reported by the students, both positively and negatively in many of their statements. They described the shortcomings of communication over Zoom and felt that the session highlighted how non-verbal cues and the physical presence influences the way one communicates. At the same time, the absence of visual and tactile stimuli forced them to practice good communication to get their points across when managing the patients.

Interestingly, though many students scored high on the Likert scale about feeling confident in managing emergencies, with the open-ended questions, they reflected feelings of nervousness, fear and a lack of confidence to working in the ED, showing that perhaps this cannot replace patient contact.

Cognisant of the limitations of telesimulation, most of the learners enjoyed the session. This may have been due to the novelty of it and ED room mimics such as the beeping of the monitors and the realistic scenarios. Faculty also realise that the limitations of telesimulation and used them as discussion points to highlight elements that one may take for granted during their patient encounters, such as the non-verbal cues and the tactile stimuli.

Key to this successful telesimulation session was establishing realistic and focused objectives (Harden et al., 1999). Failing to recognise that telesimulation differs from conventional simulation and therefore emphasising on tactile skills such as procedures and physical examination will minimise the effectiveness of the session. Knowing the inherent limitations helped faculty to prepare holistically for the session. Learning objectives focused on non-tactile aspects, such as history taking and executing treatment algorithms. In addition, as tactile cues are limited in the telesimulation setting, all other cues such as visual and audio were optimised.

Debriefing during the telesimulation sessions has an even more vital role in student learning compared to conventional simulation sessions (Fanning & Gaba, 2007). The debriefer not only has to highlight salient clinical points regarding the case, but also probe learners to think about limitations of the telesimulation modality. Therein, understanding the importance of highlighting tactile and visual feedback. For example, one learner recognised that he was “unable to visually observe and direct the teammate”; another came to the conclusion that “being able to see the patient and physical expression of fellow doctors/nurses is crucial”. This allows the educator to discuss the importance of situational awareness and non-verbal cues in enhancing team dynamics. However, if the debriefer fails to address this limitation, the learners may leave the session feeling dissatisfied or inadequate with their performance at the session. The uniqueness of telesimulation adds another facet to debriefing where the debriefer needs to be able to address the limitations of telesimulation and relate it back to clinical relevance. Therefore, there might be a need to provide additional training for educators in debriefing telesimulation sessions.

Simulation-based training is an effective modality to teach procedural skills, put into practice treatment algorithms and hone soft-skill relevant to team dynamics. (Lateef, 2010; Sirimanna & Aggarwal, 2014). As demonstrated through this innovative programme, it has an important role to play in medical education during such a pandemic where it might be used to mitigate the negative educational impact of no patient contact, team-based training and protocol development and testing  (Chaplin et al., 2020; Dieckmann et al., 2020). All this is done within a psychologically and physically safe environment.

Based on the feedback collected, a conceptual framework below (Figure 4) was drawn, showing the relationship between the concepts of fidelity and realism in the telesimulation experience to the level of engagement and therefore outcomes perceived by the learners. This is supported by the objectives, experiential learning and a safe learning environment.

Figure 4. Conceptual framework

V. CONCLUSION

The role of face-to-face interactions with patients and immersing oneself in the acute care environment in bridging the theory to practice gap experienced by all healthcare students is essential to their clinical training.  The restrictions encountered due to COVID-19 have required clinical educators to be agile and innovative in their approach the clinical clerkships. The EM clerkship telesimulation programme set out to provide an avenue for medical students to hone their clinical skills (history taking) and clinical reasoning (deriving differential diagnosis) in a safe environment. The evaluation of this programme has highlighted key areas of telesimulation which educators need to consider when planning to use it. The feedback from the students is promising and it highlights certain teaching points which may not be reflected upon during in-person simulation. Educators who wish to implement a telesimulation programme should pay particular attention to the learning objectives and debriefing methods. Whilst this paper has outlined how telesimulation can be implemented during a pandemic, it is envisaged that educators from other healthcare disciplines could use these findings to support the adoption of telesimulation in a variety of educational contexts. Telesimulation is a good alternative in settings such as this pandemic or during distance training programmes and may be a convenient way to hone history taking, clinical reasoning and communication skills without the use of an expensive simulation laboratory. The modality needs to meet the learning objectives and the debriefing should be adopted for telesimulation. However, the authors/faculty feel it cannot replace the full benefits of in-person simulation or learning from direct patient contact.

Notes on Contributors

Gayathri Devi Nadarajan and Kirsty J Freeman conceptualised the article, undertook the thematic analysis, contributed to article sections, and reviewed and revised manuscript based on suggestions from the other authors.

Lim Jia Hao, Wan Paul Weng and WONG Evelyn contributed to the conceptualisation of the paper, contributed to the article sections, reviewed and revised drafts.

Abegail Resus Fernandez undertook the quantitative analysis, and reviewed drafts.

All authors were involved in the development and delivery of the EM Clerkship Telesimulation Programme. All the authors have read and approved the final manuscript.

Ethical Approval

The SingHealth Centralised Institutional Review Board (CIRB) granted an exemption, CIRB Ref. No.: 2020/2719, as this study was assessed as a quality improvement project.

Data Availability

All relevant data are within the manuscript.

Acknowledgement

We would like to thanks the students who participated in this unit and their willingness to adapt to the online platform with grace and enthusiasm. 

Funding

This work has not received any external funding. 

Declaration of Interest

All authors declare that there are no conflicts of interest.

References

Benner, P. (2004). Using the dreyfus model of skill acquisition to describe and interpret skill acquisition and clinical judgment in nursing practice and education. Bulletin of Science, Technology and Society, 24(3), 188–199. https://doi.org/10.1177/0270467604265061

Chaplin, T., McColl, T., Petrosoniak, A., & Hall, A. K. (2020). Building the plane as you fly: Simulation during the COVID-19 pandemic. Canadian Journal of Emergency Medicine, 22(5), 576–578. https://doi.org/10.1017/cem.2020.398

Cohen, L., Manion, L., & Morrison, K. (2011). Research methods in education. Routledge. https://doi.org/10.4324/9780203720967

Dieckmann, P., Torgeirsen, K., Qvindesland, S. A., Thomas, L., Bushell, V., & Langli Ersdal, H. (2020). The use of simulation to prepare and improve responses to infectious disease outbreaks like COVID-19: Practical tips and resources from Norway, Denmark, and the UK. Advances in Simulation, 5(1), 1–10. https://doi.org/10.1186/s41077-020-00121-5

Dreyfuss, S. E., & Dreyfus, H. L. (1980). A five-stage model of the mental activities involved in directed skill acquisition. Berkeley. https://apps.dtic.mil/dtic/tr/fulltext/u2/a084551.pdf

Fanning, R. M., & Gaba, D. M. (2007). The role of debriefing in simulation-based learning. Simulation in Healthcare, 2(2), 115–125. https://doi.org/10.1097/SIH.0b013e3180315539

Harden, R. M., Crosby, J. R., & Davis, M. H. (1999). AMEE Guide No. 14: Outcome-based education: Part 1 – An introduction to outcome-based education. Medical Teacher, 21(1), 7–14. https://doi.org/10.1080/01421599979969

Kirkpatrick, D. L., & Kirkpatrick, J. D. (2009). Evaluating: Part of a ten-step process. In evaluating training programs. Berrett-Koehler Publishers.

Knowles, M. S. (1990). The adult learner: A neglected species. Gulf Publishing Co.

Kolb, D. A. (1984). The process of experiential learning. Experiential learning: Experience as the source of learning and development (pp. 20-38). Prentice Hall.

Lateef, F. (2010). Simulation-based learning: Just like the real thing. Journal of Emergencies, Trauma, and Shock, 3(4), 348. https://doi.org/10.4103/0974-2700.70743

McCoy, C. E., Sayegh, J., Alrabah, R., & Yarris, L. M. (2017). Telesimulation: An innovative tool for health professions education. Academic Emergency Medicine Education and Training, 1(2), 132–136. https://doi.org/10.1002/aet2.10015

Mikrogianakis, A., Kam, A., Silver, S., Bakanisi, B., Henao, O., Okrainec, A., & Azzie, G. (2011). Telesimulation: An innovative and effective tool for teaching novel intraosseous insertion techniques in developing countries. Academic Emergency Medicine, 18(4), 420–427. https://doi.org/10.1111/j.1553-2712.2011.01038.x

Naik, N., Finkelstein, R. A., Howell, J., Rajwani, K., & Ching, K. (2020). Telesimulation for COVID-19 Ventilator management training with social-distancing restrictions during the coronavirus pandemic. Simulation and Gaming, 51(4), 571–577. https://doi.org/10.1177/1046878120926561

Okrainec, A., Henao, O., & Azzie, G. (2010). Telesimulation: An effective method for teaching the fundamentals of laparoscopic surgery in resource-restricted countries. Surgical Endoscopy, 24(2), 417–422. https://doi.org/10.1007/s00464-009-0572-6

Oliver, R. G. (2002). Simulation-based medical education. In R. M. Harden & J. A. Dent  (Eds.), A practical guide for medical teachers (4th ed., Vol. 29, Issue 2, pp. 226–233). Churchill Livingstone.

Papanagnou, D. (2017). Telesimulation: A paradigm shift for simulation education. Academic Emergency Medicine Education and Training, 1(2), 137–139. https://doi.org/10.1002/aet2.10032

Sa-Couto, C., & Nicolau, A. (2020). How to use telesimulation to reduce COVID-19 training challenges: A recipe with free online tools and a bit of imagination. MedEdPublish, 9(1), 1–7. https://doi.org/10.15694/mep.2020.000129.1

Sirimanna, P. V., & Aggarwal, R. (2014). Patient safety. In Levine, A., DeMaria, S., Jr.,  Schwartz, A. D., & Sim. A. J. (Eds.).  The comprehensive textbook of healthcare simulation. Springer.

Smith, S. R., & Dollase, R. (1999). AMEE guide No. 14: Outcome-based education: Part 2 – Planning, implementing and evaluating a competency-based curriculum. Medical Teacher, 21(1), 15–22. https://doi.org/10.1080/01421599979978

*Gayathri Devi Nadarajan
Department of Emergency Medicine
Singapore General Hospital
1 Outram Road, Singapore 169608
Tel: +65 96804724
Email: gayathri.devi.nadarajan@singhealth.com.sg

Submitted: 14 August 2020
Accepted: 6 November 2020
Published online: 13 July, TAPS 2021, 6(3), 45-55
https://doi.org/10.29060/TAPS.2021-6-3/OA2377

Nathalie Khoueiry Zgheib¹, Ahmed Ali² & Ramzi Sabra¹

¹Department of Pharmacology and Toxicology, American University of Beirut Faculty of Medicine, Beirut, Lebanon; ²Medical Education Unit, American University of Beirut Faculty of Medicine, Beirut, Lebanon

Abstract

Introduction: The forced transition to online learning due to the COVID-19 pandemic has impacted medical education significantly.

Methods: In this paper, the authors compare the performance of Year 1 and 2 classes of medical students who took the same courses either online (2019-2020) or face-to-face (2018-2019), and compare their evaluation of these courses. The authors also present results of three survey questions delivered to current Year 1 medical students on the perceived advantages and disadvantages of online learning and suggestions for improvement.

Results: Performance and evaluation scores of Year 1 and 2 classes was similar irrespective of the mode of delivery of the course in question. 30 current (2019-2020) Year 1 students responded to the survey questions with a response rate of 25.4%. Some of the cited disadvantages had to do with technical, infrastructural and faculty know-how and support. But the more challenging limitations had to do with the process of learning and what facilitates it, the students’ ability to self-regulate and to motivate themselves, the negative impact of isolation, loss of socialisation and interaction with peers and faculty, and the almost total lack of hands-on experiences.

Conclusion: Rapid transition to online learning did not affect student knowledge acquisition negatively. As such, the sudden shift to online education might not be a totally negative development and can be harnessed to drive a more progressive medical education agenda. These results are particularly important considering the several disadvantages that the students cited in relation to the online delivery of the courses. 

Keywords:           Online Learning, COVID-19 Pandemic, Medical Students

Practice Highlights

• The authors report on the forced transition to online learning due to the COVID-19 pandemic.
• The performance and evaluation scores were similar in online delivery vs face-to face. 
• The sudden shift to online education might not be a totally negative development despite the several disadvantages that students cited.

Survey question	Theme	Quote
Advantages of online teaching and learning in comparison to face to face teaching and learning	Time flexibility with asynchronous learning	“Better scheduling that allows us to sleep and rest at night in order to wake up better prepared to ace those PowerPoints” (S9) “Easier to manage our time” (S18)
	Control of learning pace with VOP	“Being able to speed through slides/concepts we already understood and pausing and replaying concepts that we have trouble with makes the whole learning process a lot more efficient and focused” (S27)
	More wellbeing	“Less time to commute which allows more time to rest and take care of oneself” (S9) “Having a very healthy diet with my family in the village” (S12) “The [exam] performance is better and stress in minimal” (S23)
Disadvantages of online teaching and learning in comparison to face to face teaching and learning	Potential for procrastination and loss of motivation	“Less motivation, harder to follow the schedule, requires strong time management skills” (S5) “Face to face teaching helps me organize my day better” (S4) “Being at university with other students around studying during the day motivated me” (S26)
	Bad internet connection	“Internet connection in our country is not stable to hold a class or an exam, so we are resorting to 3g/4g. This leads to a lot of extra expense” (S3) “Time consuming” (S2) and “Sessions would run for more than their original allocated time” (S3) “Longer exams might coincide with the times of the electricity shut offs. This would automatically freeze Respondus and the student will have to restart their computer and so on. Although we are given extra time this adds a lot of stress to an already stressful situation” (S19)  “Asking questions are much more difficult and needs much more time” (S7) “WebEx needed a stronger Wi-Fi in some sessions which leads to a harder way to grasp the information” (S18) “The internet connection everywhere in Lebanon is not the best, sometimes we have trouble listening. Sometimes it also gets really crowded when everyone wants to talk at the at the same” (S21)
	Home environment less conducive to learning	“Not everybody has the privilege of adjusting their environments to their liking, whether that be because of their dog barking or their family members not respecting their study time” (S28) “This experience helped my appreciate how much I concentrate better in the library” (S9)
	Loss of interaction with teachers	“No direct interaction, harder to communicate directly with professors” (S2) “Face to face interaction was lost: no clues to non-verbal clues, no gestures seen” (S17)  “It is true that we can always email the doctors for any additional questions but that does not compare to in person interactions” (S19)
	Loss of interaction with students	“not being able to interact with my friends” (S12) “Students lose their social skills as they interact less with each other-more into introversion” (S17) “You feel there is a barrier between you and the students” (S17)
	Lack of immediate feedback	“One problem is during exams not being able to see my mistakes” (S15) “Not correcting our exam and not seeing our mistakes was a huge disadvantage for the online learning” (S18) “Restricting questions to only emails” (S11) and “some professors don’t respond to emails” (S16) and “the response may be delayed” (S29)
	Loss of hands on experiences	“No hands-on experience for courses like clinical skills” (S3) “Mainly missing out on clinical skills” (S22)
	Faculty’s lack of IT knowhow or experience	“Professors have different abilities and effectiveness in knowing how to do a VOP/online lecture” (S11) “Most Drs. don’t know how to use zoom or WebEx” (S6) “Many instructors are not technically inclined or are outright aversive to it” (S13) “So much time is wasted on technical issues” (S19) “Professors sometimes don’t see the raised hands and sometimes it doesn’t even work. In some lectures we had to wait for the professor to give us access, so we spent time waiting while they didn’t see that some people are trying to access the lecture” (S21) “One of the disadvantages is using the live WebEx sessions. Some professors are losing their recordings, others have a poor connection” (S23)
	Effect on faculty’s teaching skills	“Some professors …just read instead of teaching” (S7) “Many professors are not exactly cooperative in terms of explaining mainly because they read their PowerPoints” (S23) “Can’t explain a topic and be passionate about it if talking to a screen or microphone” (S23)
Suggestions to improve the current online teaching	Technical support and knowhow	“Train the staff on the proper way of utilising the platforms” (S2) “Make IT staff more readily available to help instructors” (S16) “Agree on one way to give the lecture via WebEx as some professors used WebEx team, where we had to ask permission for access, and it was kind of chaotic. It would also be better if the professor agreed on one way to have the questions asked to avoid interruptions and multiple people talking at the same time” (S21)
	More VOPs and less WebEx for lectures	“I think VOP is a much safer option and a less tiring one” (S23) “Revert from live WebEx sessions to VOP” (S3)
	More interaction and immediate feedback	“Open forums for discussion” (S3)  “Adding analytical questions in PowerPoints” (S9) “See exams and mistakes” (S15) “If the professors want to use WebEx … then they should allow questions at all times and not only at the end of the session” (S11) “Include small assessment questions (clicker like questions) at the end of each major concept so that the students can assess their understanding” (S19) “Recording voice over PowerPoint for lectures, with every group of lectures followed by a WebEx session where the professor answers questions” (S24) “Review/Q&A session once a week” (S25)

Table 1. Themes Generated from the Three Survey Questions with Selected Representative Quotes

VOP: Voice Over PowerPoint

As shown in Tables 2 and 3, there were no statistically significant differences in the MCAT scores between any two classes that were compared. The performance of the Year 1 students on the CMM course during the current academic year (online) was higher than that of students during the previous year (face-to-face); however, the passing grade for the two courses was slightly different.  When the passing grades were equalised, there was no longer a difference between the two classes. Similarly, there was no difference in the performance of the Year 2 students on either the CMM course they took in Year 1, or on The Kidney and Urinary System course between the current class and the previous year’s class (all face-to-face).

With regard to The Blood course, the grade on the final examination was significantly lower for current students (online) relative to their predecessors (face-to-face); however, the passing grades on these examinations were different, with the current year’s examination having a lower passing grade than last year’s. When the passing grades were equalised, there was no longer a difference in the performance on the final examination.

The performance of the students in the Human Development and Psychopathology course’s final examination was almost identical in the online group compared with their predecessors (all face-to-face). Interestingly, the passing grade on this year’s examination was lower than that on last year’s examination, such that when the passing scores were equalized, the current class had better performance on the final examination than last year’s class.

IV. DISCUSSION

Medical education scholars have been increasingly disseminating opinions about sudden transitioning to online education to COVID-19 and the adaptations that are being implemented. Few studies have documented the actual institutional experiences, the perspectives of students, and the lessons learned in different medical courses or curricula such as TBL (Gaber et al., 2020), anatomy (Srinivasan, 2020) and continuing medical education in obstetrics and gynaecology (Kanneganti et al., 2020). Only one report from Wuhan, China, evaluated nursing interns’ outcomes on emergency medicine theoretical and practical examination scores (Zhou et al., 2020). The current paper is the first to examine the impact of this abrupt transition to online learning, which occurred in numerous countries worldwide, on the performance of our medical students in knowledge-based examinations. It reveals that the sudden shift to full online learning that our medical school had to adopt did not have a negative influence on the students’ knowledge acquisition as judged by their performance on final examinations. It also did not affect their overall reception and evaluation of the courses. These results are particularly interesting and important considering the many disadvantages that the students cited in relation to the online delivery of the courses. 

Many of the limitations and disadvantages of online education cited by students had to do with technical and infrastructural matters and with faculty know-how and IT support. These are problems that can, theoretically, be easily remedied. The more challenging, however, limitations had to do with the process of learning, what facilitates or hampers it, the students’ ability to self-regulate and to motivate themselves, the negative impact of isolation, loss of socialisation and interaction with peers and faculty, and the almost total lack of hands-on experiences. 

These limitations did not affect the students’ ability to achieve learning, at least in the domain of knowledge acquisition and application. It is clear that students in the three classes that were examined had, at baseline, a similar level of achievement meaning that any differences in student performance in the courses that were given online this year cannot be ascribed to differences in the academic performance or ability of the students. Therefore, the lack of difference in performance between classes taking the course online versus those taking it face-to-face suggests a consistency in performance that was not affected adversely by the sudden transition to online learning.

One reason for this lack of difference in performance between online and face-to-face delivery of the courses may be that the outcomes that were being sought and assessed were essentially knowledge acquisition and knowledge application. This agrees with the overall results of multiple studies that compared online vs offline learning in medical school, and which, in fact, tended to favour online learning (Pei & Wu, 2019). Indeed, even before our sudden shift to total online education, many of our students had adopted their own approaches to achieve the knowledge learning outcomes. Even though lectures were not available online, attendance at face-to-face lectures (which was not mandatory) was never complete, and for the majority of students, the rate of attendance ranged between 25% and 75% (unpublished data). In fact, the students indicated that they depended instead on notes and voice recordings made during the lecture that were shared by their classmates or predecessors, and that they used several Web-based resources. In contrast, attendance at TBL exercises and other interactive and small group sessions is mandatory at our school, and students uniformly participated in them, as they did in the online Zoom-based sessions. Thus, our students were probably well prepared for this sudden shift. In line with this view, Ferrel and Ryan (2020), in a recent editorial on the impact of COVID-19 on medical education, predicted that many medical students in their didactic years may perceive little change in their study schedule, since many of them already use outside resources and watch school lectures after they have been presented.

The lack of significant differences in scores and attitudes may also attest to our – and indeed all – medical students’ resilience and adaptability to difficult situations, for they are high-achieving and resourceful students who have been selected from among an exceedingly competitive group of applicants, and likely have the cognitive powers and non-cognitive qualities to meet such challenges.  Ferrel and Ryan (2020) also emphasised the need for medical students to adapt and be innovative during the pandemic, and to devise ways by which they can exhibit their skills, work ethics and teamwork. In fact, one of the advantages of the online shift that our students cited was the flexibility this approach afforded them in managing their time, setting their schedules, controlling their pace of learning and achieving better self-care. Nevertheless, some of them found it challenging to do so, and to regulate their environment and motivate themselves; rather, they seemed to require external cues or assistance to get into a learning mode, and found difficulty in establishing boundaries between work and home, as suggested recently by Rose (2020). In this context, it is noteworthy that our students preferred asynchronous to synchronous learning, and this is consistent with Daniel’s recommendation to use this approach because it gives teachers “flexibility in preparing learning materials and enables students to juggle the demands of home and study” (Daniel, 2020).

Our findings also raise questions about certain assumptions regarding student learning and the optimal teaching approaches for knowledge-based objectives, such as the value and benefits of face-to-face interactions among students and with faculty in a didactic context. Our results suggest that students can achieve these knowledge objectives without the personal interaction and contact with faculty. This, of course, does not address the non-cognitive learning outcomes that might be negatively affected by pure online learning. As summarised by Fawn et al. (2020), while content may be covered well in such abrupt transitions to online learning, we cannot be sure that the valuable non-cognitive learning that happens as a result of the “social activity, the relationship-building, the problem-solving, the dialogue and generation of ideas and the students’ own discovery of other content that has not been pre-defined by the teacher” has been achieved.

We cannot make definite, long-term conclusions from this single account that is restricted to 2 courses in the preclinical years, a brief period of time, and one institution, and a low response rate for the survey questions, but the results are encouraging, and may have implications for educational practice. The lack of decline in cognitive performance may suggest that the sudden shift to online education might not be a totally negative development. If our findings are reproduced or generalised, one can use them to validate what progressive medical educators have been advocating for years, that: online educational technology must change the way we educate our students; didactic lecturing should give way to flipped classrooms; and valuable teacher time must be expended to help students apply knowledge rather than to simply transfer information in scheduled lectures. Quoting Ezekiel Emanuel (2020), who in a recent article stated that the reconfiguration of medical education, fuelled by online educational technology, seemed inevitable, Wolanskyj-Spinner (2020) suggested that the coronavirus epidemic appears to be an inflection point that is forcing a disruption in how we teach medicine. At AUBFM, we have long pressed the faculty who teach medical students to record their lectures and use the scheduled class time thus saved to implement flipped classrooms, employing small-group-based, problem-solving and interactive sessions. While many responded, many also hesitated, objected, and even resisted. The following two additional comments provided by two students illustrate their frustration with the resistance of faculty and their hopes to move in that direction: 

“I really hope we can make online learning standard coming out of this phase … There was an attempt a few years ago but many instructors refused to be recorded or to fiddle with computers; we must seize the opportunity now.”

“Please never stop recording lectures, regardless of the status of live classes!”

Ahmed et al. (2020) recently reported that during the 2003 SARS epidemic in China, novel online problem-based learning techniques had to be implemented in one medical school that proved to be so popular that they were applied as part of the regular curriculum in later years. We believe that medical educators can harness the current disruption in how we teach medical students, and make use of to implement novel and sound educational practices and adopt a wide variety of valid approaches and tools that, otherwise, might have been resisted by unwilling individuals with entrenched ideas.

V. CONCLUSION

In conclusion, rapid transition to online learning did not affect student knowledge acquisition negatively. As such, the sudden shift to online education might not be a totally negative development and can be harnessed to drive a more progressive medical education agenda. These results are particularly important considering the several disadvantages that the students cited in relation to the online delivery of the courses. 

Notes on Contributors

Nathalie Zgheib developed the concept, collected and analysed data, and wrote the first draft of the manuscript. Ahmed Ali also developed the concept, performed the literature review, and revised the manuscript write-up. Ramzi Sabra also developed the concept, collected and analysed data, and revised the manuscript write-up. The three authors read and approved the final version of the manuscript.

Data Availability

The data that support the findings of this manuscript are available from the corresponding author upon reasonable request.

Ethical Approval

This is a report of experience with educational practices. It was confirmed by our Institutional Review Board (IRB) that the activities described in this article do not constitute human subject research.

Acknowledgments

 The authors wish to thank AUBFM faculty and medical students for their support, diligence and flexibility during the COVID-19 pandemic.

Funding

This study did not receive any funding.

Declaration of interest

The authors do not have any conflict of interest to declare.

References

Ahmed, H., Allaf, M., & Elghazaly, H. (2020). COVID-19 and medical education. The Lancet Infectious Diseases, 20(7), 777-778. https://doi.org/10.1016/S1473-3099(20)30226-7

Ashcroft, J., Byrne, M. H. V., Brennan, P. A., & Davies, R. J. (2020). Preparing medical students for a pandemic: A systematic review of student disaster training programmes. Postgraduate Medical Journal, Advance online publication. https://doi.org/10.1136/postgradmedj-2020-137906

Daniel, S. J. (2020). Education and the COVID-19 pandemic. Prospects, 49, 91-96. https://doi.org/10.1007/s11125-020-09464-3  

Emanuel, E. J. (2020). The inevitable reimagining of medical education. Journal of the American Medical Association, 323(12), 1127-1128. https://doi.org/10.1001/jama.2020.1227

Fawn, T., Jones, D., & Aitken, G. (2020). Challenging assumptions about “moving online” in response to COVID-19, and some practical advice. MedEdPublish, 9(1), 83. https://doi.org/10.15694/mep.2020.000083.1

Ferrel, M. N., & Ryan, J. J. (2020). The impact of COVID-19 on medical education. Cureus, 12(3), e7492. https://doi.org/10.7759/cureus.7492

Frehywot, S., Vovides, Y., Talib, Z., Mikhail, N., Ross, H., Wohltjen, H., Koumare, A. K., & Scott, J. (2013). E-learning in medical education in resource constrained low- and middle-income countries. Human Resources for Health, 4(11), 4. https://doi.org/10.1186/1478-4491-11-4

Gaber, D. A., Shehata, M. H., & Amin, H. A. A. (2020). Online team-based learning sessions as interactive methodologies during the pandemic. Medical Education, 54(7), 666-667. https://doi.org/10.1111/medu.14198

Kanneganti, A., Lim, K. M. X., Chan, G. M. F., Choo, S. N., Choolani, M., Ismail-Pratt, I., & Logan, S. J. S. (2020). Pedagogy in a pandemic – COVID-19 and virtual continuing medical education (vCME) in obstetrics and gynecology. Acta Obstetetricia et Gynecologica Scandinavica, 99(6), 692-695. https://doi.org/10.1111/aogs.13885 

Liang, Z. C., Ooi, S. B. S., & Wang, W. (2020). Pandemics and their impact on medical training: Lessons from Singapore. Academic Medicine, 95(9), 1359-1361. https://doi.org/10.1097/ACM.0000000000003441

Lim, E. C., Oh, V. M., Koh, D. R., & Seet, R. C. (2009). The challenges of “continuing medical education” in a pandemic era. Annals of Academic Medicine Singapore, 38(8), 724-726.

McKimm, J., Mclean, M., Gibbs, T., & Pawlowicz, E. (2019). Sharing stories about medical education in difficult circumstances: Conceptualizing issues, strategies, and solutions. Medical Teacher, 41(1), 83-90. https://doi.org/10.1080/0142159X.2018.1442566

Patil, N. G., Chan, Y., & Yan, H. (2003). SARS and its effect on medical education in Hong Kong. Medical Education, 37(12), 1127-1128. https://doi.org/10.1046/j.1365-2923.2003.01723.x

Pei, L., & Wu, H. (2019). Does online learning work better than offline learning in undergraduate medical education? A systematic review and meta-analysis. Medical Education Online, 24(1), 1666538. https://doi.org/10.1080/10872981.2019.1666538

Rose, S. (2020). Medical student education in the time of COVID-19. Journal of the American Medical Association, 323(21), 2131-2132. https://doi.org/10.1001/jama.2020.5227

Ross, D. (2020). Creating a “quarantine curriculum” to enhance teaching and learning during the COVID-19 pandemic. Academic Medicine, 95(8), 1125-1126.

Sandars, J., Correia, R., Dankbaar, M., de Jong, P., Sun Goh, P., Hege, I., Oh, S., Patel, R., Premkumar, K., Webb, A., & Pusic, M. (2020). Twelve tips for rapidly migrating to online learning during the COVID-19 pandemic. MedEdPublish, 9(1), 82. https://doi.org/10.15694/mep.2020.000082.1

Srinivasan, D. K. (2020). Medical students’ perceptions and an Anatomy teacher’s personal experience using an e-learning platform for tutorials during the Covid-19 crisis. Anatomical Sciences Education, 13(3), 318-319. https://doi.org/10.1002/ase.1970

Wolanskyj-Spinner, A. (2020). COVID-19: The global disrupter of medical education. ASH Clinical News, https://www.ashclinicalnews.org/viewpoints/editors-corner/covid-19-global-disrupter-medical-education/

Woodward, A., & McLernon-Billows, D. (2018). Undergraduate medical education in Sierra Leone: A qualitative study of the student experience. BMC Medical Education, 18(1), 298. https://doi.org/10.1186/s12909-018-1397-6

Zhou, T., Huang, S., Cheng, J., & Xiao, Y. (2020). The distance teaching practice of combined mode of massive open online course micro-video for interns in emergency department during the COVID-19 epidemic period. Telemedicine Journal and E-Health, 26(5), 584-588. https://doi.org/10.1089/tmj.2020.0079

*Ramzi Sabra
Department of Pharmacology and Toxicology,
Faculty of Medicine,
American University of Beirut,
PO. Box 11-0236, Riad El Solh,
Beirut, Lebanon
Tel: +961-1-350000 Ext: 4848
Email: rsabra@aub.edu.lb

Submitted: 28 August 2020
Accepted: 3 March 2021
Published online: 13 July, TAPS 2021, 6(3), 32-44
https://doi.org/10.29060/TAPS.2021-6-3/OA2385

Judy McKimm¹, Subha Ramani² & Vishna Devi Nadarajah³

¹Swansea University Medical School, United Kingdom; ²Harvard Medical School, United States of America; ³International Medical University, Malaysia

Abstract

Introduction: The COVID-19 pandemic has caused huge change and uncertainty for universities, faculty, and students around the world. For many health professions’ education (HPE) leaders, the pandemic has caused unforeseen crises, such as closure of campuses, uncertainty over student numbers and finances and an almost overnight shift to online learning and assessment. 

Methods: In this article, we explore a range of leadership approaches, some of which are more applicable to times of crisis, and others which will be required to take forward a vision for an uncertain future. We focus on leadership and change, crisis and uncertainty, conceptualising ‘leadership’ as comprising the three interrelated elements of leadership, management and followership. These elements operate at various levels – intrapersonal, interpersonal, organisational and global systems levels. 

Results: Effective leaders are often seen as being able to thrive in times of crisis – the traditional ‘hero leader’ – however, leadership in rapidly changing, complex and uncertain situations needs to be much more nuanced, adaptive and flexible.

Conclusion: From the many leadership theories and approaches available, we suggest some specific approaches that leaders might choose in order to work with their teams and organisations through these rapidly changing and challenging times.

Keywords:            Leadership, Followership, Management, Health Professions Education, Change, Crisis, Uncertainty, Emotional Intelligence, COVID-19 Pandemic, Universities

Practice Highlights

• In rapid change and uncertainty, different leadership approaches are needed.
• Primal leadership and emotional intelligence are essential.
• Followers need to feel safe, physically and psychologically.
• Authentic and inclusive leadership draws from diverse views.
• Adaptive and regenerative leadership acknowledges interrelated systems.

I. INTRODUCTION

The COVID-19 pandemic has caused huge change and uncertainty for universities and their stakeholders around the world. For many health professions education (HPE) leaders, the pandemic has caused an unforeseen crisis, the ripples from which will probably be felt for years to come. Effective leaders are often seen as being able to thrive in times of crisis – the traditional ‘hero leader’ – however, leadership in rapidly changing, complex and uncertain situations need to be much more nuanced and flexible. In this article we explore leadership approaches, some of which are more applicable to times of crisis, and others which will be required to take forward a vision for the ‘new normal’ to ensure that we learn from our experiences during the pandemic. 

In this article we focus on leadership and change. We start with an overview of the leadership triad, a discussion of the educational challenges imposed by the COVID-19 pandemic, followed by detailed discussion of effective leadership styles and competencies during challenging situations, approaching these through three lenses: Intrapersonal, referring to characteristics that successful leaders possess; interpersonal, referring to leadership styles and approaches leaders can adopt when they interact with others; and system level, which refers to leadership attributes to effectively lead organisations during a crisis. We conceptualise ‘leadership’ as comprising three interrelated elements: leadership, management, and followership (see Figure 1), which we call the ‘leadership triad’ (McKimm & O’Sullivan, 2016).

Figure 1: The Leadership Triad

Note: From “When I say … leadership,” by J. McKimm, and H. O’Sullivan, 2016, Medical Education, 50(9), 896–897. https://doi.org/10.1111/medu.13119

Figure 2: Four levels of leadership for addressing challenges during a crisis

Note: Drawing from “ABC of clinical leadership,” by T. Swanwick and J. McKimm, 2017, John Wiley & Sons.

A. The Intrapersonal Level: Working with Emotional and Social Intelligence

Figure 3: Intrapersonal leadership attributes: Emotional and social intelligence competencies essential to lead and manage change during challenging circumstances.

Note: Adapted from “Competencies as a behavioral approach to emotional intelligence,” by R. E. Boyatzis, 2009, Journal of Management Development, 28(9), 749–770. https://doi.org/10.1108/02621710910987647

While the construct of EI and competencies can serve as a useful guide to leaders in leading and managing change, the actual behaviours that are most effective depend on the organisational culture and the societal culture within which an organisation is situated. Moreover, at institutions which feature diversity in the composition of its leaders, staff and learners, leaders should recognise that individuals on a team might have different emotional reactions even when working towards a common goal. All four domains of EI competencies are essential for leaders to manage the groups of people they lead (See Table 1).

Submitted: 28 July 2020
Accepted: 3 December 2020
Published online: 13 July, TAPS 2021, 6(3), 75-82
https://doi.org/10.29060/TAPS.2021-6-3/OA2366

Kirsty J Freeman¹ & Brid Phillips²

¹Office of Education, Duke-NUS Medical School, Singapore; ²Health Professions Education Unit, The University of Western Australia, Australia

Abstract

Introduction: Healthcare requires its practitioners, policymakers, stakeholders, and critics to have empathetic awareness and skills in critical thinking. Often these skills are neglected or lost in current educational programs aimed at those interested in the field of health. Health humanities and, in particular narrative medicine, aim to redress this omission.

Methods: We used a mixed methods approach to explore the experience of health humanities students in creative writing and staged readings to foster empathic awareness and critical thinking. Data was collected from 20 second-year students enrolled in an undergraduate health humanities unit via a post-assessment survey, and thematic analysis of a reflective paper.

Results: 92.9% of the students felt that writing a creative piece helped them to understand the health topic from a different perspective, with 85.7% reporting that the use of creative writing helped to create emotional connections. From the reflective paper, six themes were elicited through the thematic data analysis: (1) The value of language; (2) Insights into the patient experience; (3) Giving voice to the patient; (4) Creating empathic exchanges; (5) Illness and emotion; and (6) Reflective practice. 

Conclusion: By offering a mode of experiential learning involving both creative writing and staged readings, students develop empathic ways of thinking and being while deepening their critical engagement with a range of health topics. Students were able to understand the need to make humanistic sense of the health and well-being narrative, providing them with a range of transferable skills which will be an asset in any workplace.

Keywords:           Narrative Medicine, Empathy, Critical Thinking, Staged Reading, Health Humanities

Practice Highlights

• Creative writing and staged readings are effective in fostering empathetic awareness and critical thinking.
• Narrative medicine techniques result in greater understanding about the perspectives of others.
• Developing creative language leads to enhanced communication skills and nuanced ways of thinking.
• Staged readings delivered online provide effective teaching and learning opportunities.

I. INTRODUCTION

Health humanities, and the attendant field of medical humanities, refer to the application of the creative or fine arts (visual arts, performing arts etc.) and humanities disciplines (literary studies, law, history, philosophy, etc.) to discussions and explorations on the nature of human health and well-being (Crawford et al., 2010). Within this broad umbrella lies the discipline of narrative medicine. The application of humanities to health has had a long pedigree, but the distinct disciplines of both narrative medicine and health humanities only began to emerge over the first decade of the 21st Century. In part, they emerged from a growing concern about an increasing lack of empathy in health professionals (Dean & McAllister, 2018; Lai, 2020). Narrative medicine with its interest in creativity and ambiguity strives to address this concern. Through narrative medicine, skills in thinking reflectively, listening actively, observing more closely and writing creatively can be developed. It has been shown that there is a positive impact on empathy and communication following narrative medicine education (Barber & Moreno-Leguizamon, 2017). This is important as the empathy conveying physician is more successful in promoting better clinical outcomes for patients. However, ‘[d]espite the centrality of stories to many of the tasks that clinicians perform it remains that explicit and formal teaching of knowledge and methods in narrative is relatively novel’ (Boudreau et al., 2012, p. 152).

One of the educational techniques embraced in narrative medicine is staged readings.  A staged reading is an event which may have some rehearsal time, but the readers use scripts on stage. There is minimal staging, costuming, and props. This is pertinent as the use of theatre in academic teaching represents a new model of education that reminds students of the humanity of people (Baker et al., 2019). This form of engagement involves an emotional transaction through the spectacle of theatre which, as the Greeks understood it, was an occasion that provided recognition, catharsis, and release for both the individual and the wider community (Shapiro & Hunt, 2003). Health topics are also more easily understood through the medium of performance (Ünalan et al., 2009). Ünalan et al. (2009) also surmised that theatrical performance could increase empathy levels. There are similar benefits to be had from staged medical readings which foster introspection and reflection (Matharu et al., 2011). The purpose of this study was to determine whether the use of creative writing and staged readings could develop empathy and critical thinking in second year university students enrolled in an undergraduate health humanities unit of study. Student enrolment information confirms majority of students are on a pathway to studying medicine, pharmacy, dentistry, ophthalmology, or global health. 

A. Context

In semester one of 2020, 20 second-year students were enrolled in a narrative medicine unit, as part of their undergraduate bachelor’s degree. By delivering an undergraduate unit in narrative medicine, the goal is to give students the opportunity:

• To dip their toes into the world of literary fiction.
• To present their own creative pieces relating to health topics.
• To gain an understanding of health issues from the perspective of others as this increases empathetic awareness.

Through a series of scaffolded assessments, students have a unique opportunity to develop empathetic awareness and critical thinking skills through creative presentations mimicking staged readings.

Two of the three assessments related to the staged reading, the second assessment was a creative piece and the third was a reflection on process of creating the piece and the health topic to which it related. The purpose of these assessments was to demonstrate different modes of narrative writing. This was achieved through the construction of a creative piece that explored a health topic using narrative medicine techniques including but not limited to short story writing, poetry, and play writing.

The unit involved supporting students to devise a short creative writing piece. Within the piece, themes of empathy, communication, cultural difference, and societal biases and assumptions around the students’ chosen health topics were explored. The piece was then to be presented as a staged reading by the students for an invited audience. The audience would include the wider university community of undergraduate and postgraduate students, staff, and invited guests such as health professionals to the reading. Immediately following their reading, the students, supported by academic staff, were to hold a guided discussion on the significance of health topics in the piece. This discussion would bring biases and assumptions into focus and heighten the individual’s awareness of emotional dynamics at work in the healthcare context while also offering insights into the perspective of others. Similar programs have been used to educate bioethical students, help them to develop discussion questions, and enhance their critical self-reflection (Kerr et al., 2020; Robeson & King, 2017).

Students were required to read a literary fiction novel, Extinctions (Wilson, 2020). The act of reading itself has demonstrated benefits of improving processing of experiences and developing empathy. Reading literature also improves our social awareness and our ability to see the perspectives of others (Fennelly, 2020; Kaptein et al., 2018). In the novel Extinctions, there are many discernible health topics such as traumatic brain injury, drug use, ageing, mobility issues, death and dying, and loss of independence. There is also a range of characters involved in these issues from a young girl with a drug problem to the protagonist, Fred, a man in his declining years. It was important to foster student engagement with the project by offering a fully scaffolded experience to allay performance anxieties. Scaffolding has been shown to support students as they negotiate a challenging environment and allow them to make meaning for themselves rather than have it imposed on them from an autocratic perspective (Wilson, 2016). Each week from the first week of semester, the seminar included both a close reading exercise and a creative writing exercise. The scaffolding also included several resource folders addressing the main genres the students were encouraged to explore – short story, poetry, drama. The folders contained videos, book chapters, blog posts, and journal articles that introduced the students to ways of writing creatively. There was also a dedicated workshop which explored these techniques and answered any questions the students had on creative writing.

Due to COVID-19 restrictions on social gatherings and the cancellation of face-to-face interactions, the presentation aspect had to be cancelled at short notice and instead, presentations took place online without the wider audience participation. The creative pieces were read in an online forum limited to students and the unit coordinator. The students had the opportunity to read their work to other participating students and to lead a short discussion on their health topic. This sharing is important as ‘representation is always a dialogue, in which, the receiver of the work contributes a necessary response to the creator of the work’ (Charon et al., 2016, p. 347). As the students based their pieces on fully rounded characters from the novel, this process shares similarities with verbatim theatre. Verbatim theatre has been shown to allow positive exploration of emotional behaviours (Scott et al., 2017). After this process, students were encouraged to incorporate feedback from the presentations into their pieces before submission of the creative piece.

The third assessment component of the unit required the students to submit a reflective essay on the experience where they discuss the creative process and their representation of the health topic which they had chosen. They had to discuss the significance of the health topic and examine their personal responses to the topic and how it was influenced by their research, the creative process, and the discussions which took place following the presentation of the piece. To support the reflective process, we developed a reflective writing toolkit which illustrates both the process and its importance.

II. METHODS

A cross-sectional mixed methods design was used to evaluate the experience of health humanities students in creative writing and staged readings to foster empathic awareness and critical thinking.  20 second-year students who were enrolled in a narrative medicine unit between January and June 2020 were invited to participate.

A. Data Collection and Analysis

Data was collected at two points over the semester, an online survey in week 10, and a reflective paper at the end of semester in week 13.

1) Creative writing and staged reading assessment student experience survey: All 20 undergraduate students enrolled in the unit in semester one, 2020, were invited to participate in an online survey examining the student experience of participating in the creative writing and staged reading assessment. The survey tool curated by author one (KF) was designed to collect basic demographic data about the student, along with information about their current enrolment. The survey was designed to evaluate the first two level of Kirkpatrick’s model of program evaluation, level one being reaction and level two learning (Frye & Hemmer, 2012). Given that this is the first time the program had been offered, the authors felt that the data collected would provide a baseline upon which further detailed evaluations can build upon. Students were asked to rate their experience with the staged reading project using a five-point Likert scale, as well as responding to open-ended questions designed to further expand on the students’ experience. A statement of voluntary consent was included at the start of the survey and the participant had to agree to the consent before the survey could commence. Thus participation in the anonymized online evaluation indicates consent. Descriptive statistics were calculated for the demographic data. Categorical data are presented as number and percentage. The analyses were performed using IBM SPSS Statistics Version 25.0. Thematic analysis of the open-ended questions was then undertaken. Researcher bias was minimised by having author one (KF), who was not involved in delivering the course, undertake the analysis of the survey data.

2) Staged reading reflective essay: The second data collection point was a reflective essay (n=20).  Thematic analysis of the text was undertaken by both authors. Each author reviewed the transcripts separately, making note of key phrases, outline possible categories or themes. Discussion of our interpretations took place over teleconference, as we then jointly rearranged and renamed the codes, developing higher order themes.  NVivo 12™ was used to manage the qualitative data (QSR International., 2018). This mixed methods design combines quantitative and qualitative data to provide a richer source of information about the experience of staged readings.

III. RESULTS

A. Creative Writing and Staged Reading Assessment Student Experience Survey

Of the 20 students enrolled in the unit, 14 completed the student experience survey, a response rate of 70%. Table 1 summarises the demographic characteristics of the respondents. As can be seen in Table 1, the students were enrolled in one of three bachelor degrees. The degree enrolled by majority of students in is science based, with only 1 respondent studying an Arts based degree. 92.8% of respondents had not previously completed an undergraduate degree. Three respondents spoke languages other than English at home.

When asked to rate their experience in the staged reading project 92.9% of the respondents felt that writing a creative piece helped them to understand the health topic from a different perspective; with  85.7% reporting that the use of creative writing helped to create emotional connections (Table 2).  

Table 1. Summary of demographic characteristics of respondents

Table 2. Student rating of creative writing and staged reading assessment

When asked to describe their experience of the staged reading assessment in the free text survey questions students reported feeling daunted, nervous and apprehensive about the prospect of writing a creative piece, as many of them shared that they had little or no experience with creative writing. With several students describing to task as challenging, on reflection they expressed feeling fulfilled, enriched, sharing that they found the task rewarding.

A. Staged Reading Reflective Essay

The qualitative data analysis resulted in six themes being identified: (1) The value of language; (2) Insights into the patient experience; (3) Giving voice to the patient; (4) Creating empathic exchanges; (5) Illness and emotion; and (6) Reflective practice (Figure 2). These themes are described in this section, illustrated with representative quotes.

Figure 1. Overview of the staged reading themes

1) The value of language: as they worked through their creative piece, students discovered the value and power of language as a tool for expression and communication. Students commented on their appreciation of language:

Since I wanted to create a powerful and emotional piece, I experimented using literacy techniques to achieve a desperate and anxious tone.

Student 2

I learnt that the use of language is also vital in writing a creative piece, emphasising the importance of communication between the author and readers.

Student 14

2) Insights into the patient experience: Students developed an awareness of the value of research when trying to understand the issues surrounding health topics. Better equipped with quality knowledge and research, they were able to give more nuanced accounts of health experiences. Students learned the value of looking beyond the symptom to see the whole person and, thus the value and importance of person-centred care:

Acknowledging that my research and reflections had resulted in greater understandings of both the health topic and the importance of seeing a patient beyond their physical disease.

Student 2

It is very important to take into consideration all aspects of what a person is experiencing in order to make the best assessment and to come up with the best route of action to help the patient overcome whatever it is they are suffering from.                                                                                                                                                                                                                                               

Student 1

3) Giving voice to the patient: Students gained an insight into the need to give a voice to patients in order to gain better understanding of the perspectives of others. They reflected on the powerlessness and silence that often surrounds certain conditions and situations:

I felt that I was able to give Katie an authentic voice, through which readers were then able to empathise with, and better understand, her struggles.

Student 4

Furthermore, the narratives of victims can empower them by giving them a voice in times when they are often silenced.

Student 7

4) Creating empathetic exchanges: Students gained an understanding of empathetic exchanges and in some instances understood the need to create opportunities for the development of empathy. The students did not always articulate the word empathy but instead alluded to the concept by talking about experiencing and understanding the emotions and perspectives of others:

I felt increased levels of empathy towards individuals dealing with the disorder as I now understand the many other challenges and hurdles that accompany eating disorders that I didn’t know prior to this assignment.

Student 8

When using these descriptions, I felt that I was able to build an emotional connection with Katie’s character, thus encouraging readers to also empathise with her situation.

Student 4

5) Illness and emotion: Students discovered that emotional transactions and states are intertwined with health and illness. They understood, through their work, the interconnectiveness of emotional responses and illness. Some of their observations included:

I have become more sensitive to the idea that human beings are inherently emotional and can be affectively moved when provided with an impetus.

Student 3

The purpose of the piece was unearthing the complex thoughts and emotions individuals with eating disorders and substance abuse go through.

Student 8 

6) Reflective Practice: On topics where students previously had felt knowledgeable, deeper reflection and considerations revealed their own misconceptions and lack of knowledge. Within this paradigm, they also showed a maturation of habits and behaviours. Their comments were insightful:

I entered a phase of reflection where I realised that I would have previously contributed to these harmful views.

Student 2

I learnt and later researched further the differences in the perception of STEM women in both Western and Asian populations. This is something I would attempt to change. It was really interesting for me to discover that I had this unconscious bias forcing me to further expand my learning and knowledge in the area.

Student 18

IV. DISCUSSION

While the use of creative writing and staged readings is a developing area in health humanities, the findings of this study suggest that they are effective in fostering emotional awareness and critical thinking. McDonald et al. (2015) found that ‘[a]nalysis of the student’s writing showed that they demonstrated the ability to “stand in another’s shoes” and, interestingly, the students’ comments on their own writing showed that their ability to relate to characters they initially felt little affinity for deepened’ (McDonald et al., 2015, p. 9). 

By taking part in the creative writing project and the accompanying reflection piece, students were exposed to an innovative and experiential form of learning that provided a unique pedagogical experience. Whilst the students reported being daunted at the thought of constructing a creative piece, the self-reflective processes and actively engaging with the perspectives of others, ensured students were able to enhance their critical thinking skills. In the creative writing and staged reading assessment student experience survey, 100% of students agreed, “Listening to the creative pieces of others helped me to understand the perspectives of others”. This accords with the work of Deloney and Graham who note “[e]xperiential learning activities increase student engagement and are a helpful tool to connect abstract ideas with concrete knowledge” (Deloney & Graham, 2003, p. 249).

The findings of this study highlight the value of language as a tool for expression and communication. Educators need to be mindful of their student population when contemplating incorporating creative writing and staged readings into their programs. For students to be accepted for enrolment at The University of Western Australia they must demonstrate a minimum level of English language proficiency necessary for academic studies. Whilst there has been several studies examining what Hyland calls “linguistics disadvantage in terms of a Native/non-Native divide” (Hyland, 2016, p. 61) in academic writing (Badenhorst et al., 2015; Bocanegra-Valle, 2014; Zhao, 2017), the impact of this linguistic divide in creative writing and staged readings has not been fully explored.

The students gained insights into holistic care relating to both themselves and others. As these are undergraduate students, many on a pathway to a career in health, insights into holistic care and self-care are valuable life lessons. There was also an indication that the process contributed to the social and cultural well-being of themselves and others. Nagji et al. (2013) note that theatre based programs/novel humanities based curriculum items contribute to student well-being, an increasingly important area for universities to address particularly during a pandemic. Although the student population participating in this unit are studying science-based degrees where creative writing is not commonplace, the learning strategies were structured in a way that enabled development of narrative writing skills.

By integrating the narrative medicine techniques of creative writing and staged readings the students were able to give more nuanced accounts of health experiences. Students learned the value of looking beyond the symptom to see the whole person and, thus the value and importance of person-centred care. This gave them valuable insights into the patient experience. During this process, students gained an insight into the need to give a voice to patients in order to gain better understanding of the perspectives of others. They learnt about the powerlessness and silence that often surrounds certain conditions and situations. In their responses, the students did not always articulate the word empathy but instead alluded to the concept by talking about experiencing and understanding the emotions and perspectives of others. Students gained an understanding of the power of empathetic exchanges and in some instances understood the importance of creating empathetic exchanges opportunities. Another important theme which was uncovered, was a growing awareness amongst students that emotional transactions and states are intertwined with health and illness. They understood, through their work, the interconnectiveness of illness and emotion. On topics where students previously had felt knowledgeable, deeper reflection and considerations revealed their own misconceptions and lack of knowledge. Within this paradigm, they also showed a maturation of habits and behaviours leading to improved reflective practices.

There were limitations with the project as we had to move from face-to-face to online teaching due to constraints placed upon teaching during the COVID-19 pandemic. However, the online format did have some unexpected benefits, with a number of students feeling less intimidated when presenting online compared to a face-to-face workshop. Despite the move to online teaching, the cohort remained present and engaged in the project. Many expressed regrets at losing physical contact and the incidental discussions that happen before and after classes but overall with support they adapted well.

V. CONCLUSION

This study has described the experience of students engaging in creative writing and staged readings as part of a narrative medicine unit. Students completing the unit and its attendant assessments developed useful life skills including critical thinking, understanding the perspectives of others, and the positive use of narrative in appreciating the experiences of others. The work engaged them in new and innovative ways evidenced by some statements which noted that their experience in this unit was unique in their university journey. Having autonomy over the health topic they chose, the character they explored, and the creative medium they used to express their thinking, enhanced the learning experience and allowed them to meet the learning outcomes of the unit.

Notes on Contributors

Kirsty J Freeman crafted the paper with her co-author, performed the data collection and analysis of the survey, and undertook thematic analysis of the reflective essays. Dr Brid Phillips crafted the paper with her co-author, conducted the literature search, and undertook thematic analysis of the reflective essays. All the authors have read and approved the final manuscript.

Ethical Approval

Ethics approval was granted by The University of Western Australia Human Research Ethics Committee: HREC RA/4/20/5254.

Data Availability

All relevant quantitative data are within the manuscript. The qualitative data collected for this manuscript originates from assessment items submitted as part of the participants’ academic studies.  The authors do not have consent to upload this data into a data repository.

Acknowledgement

We would like to thanks the students who participated in this unit and their willingness to adapt to the online platform with grace and enthusiasm.

Funding

This work has not received any external funding. 

Declaration of Interest

All authors declare that there are no conflicts of interest.

References

Badenhorst, C., Moloney, C., Rosales, J., Dyer, J., & Ru, L. (2015). Beyond deficit: Graduate student research-writing pedagogies. Teaching in Higher Education, 20(1), 1-11. https://doi.org/10.1080/13562517.2014.945160

Baker, H. F., Moreland, P. J., Thompson, L. M., Clark-Youngblood, E. M., Solell-Knepler, P. R., Palmietto, N. L., & Gossett, N. A. (2019). Building empathy and professional skills in global health nursing through theatre monologues. The Journal of Nursing Education, 58(11), 653-656. https://doi.org/10.3928/01484834-20191021-07

Barber, S., & Moreno-Leguizamon, C. J. (2017). Can narrative medicine education contribute to the delivery of compassionate care? A review of the literature. Medical Humanities, 43(3), 199-203. https://doi.org/10.1136/medhum-2017-011242

Bocanegra-Valle, A. (2014). ‘English is my default academic language’: Voices from LSP scholars publishing in a multilingual journal. Journal of English for Academic Purposes, 13(1), 65-77. https://doi.org/10.1016/j.jeap.2013.10.010

Boudreau, J. D., Liben, S., & Fuks, A. (2012). A faculty development workshop in narrative-based reflective writing. Perspectives on Medical Education, 1(3), 143-154. https://doi.org/10.1007/s40037-012-0021-4

Charon, R., Hermann, N., & Devlin, M. J. (2016). Close reading and creative writing in clinical education: Teaching attention, representation, and affiliation. Academic Medicine, 91(3), 345-350. https://doi.org/10.1097/ACM.0000000000000827

Crawford, P., Brown, B., Tischler, V., & Baker, C. (2010). Health humanities: The future of medical humanities? Mental Health Review Journal, 15(3), 4-10. https://doi.org/10.5042/mhrj.2010.0654

Dean, S., & McAllister, M. (2018). How education must reawaken empathy. Journal of Advanced Nursing, 74(2), 233-234. https://doi.org/10.1111/jan.13239

Deloney, L. A., & Graham, C. J. (2003). Wit: Using drama to teach first-year medical students about empathy and compassion. Teaching and Learning in Medicine, 15(4), 247-251. https://doi.org/10.1207/S15328015TLM1504_06

Fennelly, B. A. (2020, May 15). What’s the use of reading? Literature and empathy [Video] . https://youtu.be/9nJv8sxpUKU

Frye, A. W., & Hemmer, P. A. (2012). Program evaluation models and related theories: AMEE guide no. 67. Medical Teacher, 34(5), e288-299. https://doi.org/10.3109/0142159x.2012.668637

Hyland, K. (2016). Academic publishing and the myth of linguistic injustice. Journal of Second Language Writing, 31, 58-69. https://doi.org/10.1016/j.jslw.2016.01.005

Kaptein, A., Hughes, B., Murray, M., & Smyth, J. (2018). Start making sense: Art informing health psychology. Health Psychology Open, 5(1), 205510291876004. https://doi.org/10.1177/2055102918760042

Kerr, A., Biechler, M., Kachmar, U., Palocko, B., & Shaub, T. (2020). Confessions of a reluctant caregiver palliative educational program: Using readers’ theater to teach end-of-life communication in undergraduate medical education. Health Communication, 35(2), 192-200. https://doi.org/10.1080/10410236.2018.1550471

Lai, C.-W. (2020). “Booster shots” of humanism at bedside teaching. The Asia Pacific Scholar, 5(2), 45-47. https://doi.org/10.29060/TAPS.2020-5-2/PV1085

Matharu, K. S., Howell, J., & Fitzgerald, F. (2011). Drama and empathy in medical education: Drama and empathy. Literature Compass, 8(7), 443-454. https://doi.org/10.1111/j.1741-4113.2011.00778.x

McDonald, P., Ashton, K., Barratt, R., Doyle, S., Imeson, D., Meir, A., & Risser, G. (2015). Clinical realism: A new literary genre and a potential tool for encouraging empathy in medical students. BMC Medical Education, 15(1), 112-112. https://doi.org/10.1186/s12909-015-0372-8

Nagji, A., Brett-MacLean, P., & Breault, L. (2013). Exploring the benefits of an optional theatre module on medical student well-being. Teaching and Learning in Medicine, 25(3), 201-206. https://doi.org/10.1080/10401334.2013.801774

QSR International. (2018). NVivo 12 [Software]. In https://qsrinternational.com/nvivo/nvivo-products/

Robeson, R., & King, N. (2017). Performable case studies in ethics education. Healthcare, 5(3), 57. https://doi.org/10.3390/healthcare5030057

Scott, K. M., Berlec, Š., Nash, L., Hooker, L., Dwyer, P., Macneill, P., River, J., & Ivory, K. (2017). Grace under pressure: A drama-based approach to tackling mistreatment of medical students. Medical Humanities, 43(1), 68. https://doi.org/10.1136/medhum-2016-011031

Shapiro, J., & Hunt, L. (2003). All the world’s a stage: The use of theatrical performance in medical education. Medical Education, 37(10), 922-927. https://doi.org/10.1046/j.1365-2923.2003.01634.x

Ünalan, P. C., Uzuner, A., Ifçili, S., Akman, M., Hancolu, S., & Thulesius, H. O. (2009). Using theatre in education in a traditional lecture oriented medical curriculum. BMC Medical Education, 9(1), 73-73. https://doi.org/10.1186/1472-6920-9-73

Wilson, J. (2020). Extinctions. UWA Publishing.

Wilson, K. (2016). Critical reading, critical thinking: Delicate scaffolding in english for academic purposes (EAP). Thinking Skills and Creativity, 22, 256-265. https://doi.org/10.1016/j.tsc.2016.10.002

Zhao, J. (2017). Native speaker advantage in academic writing? Conjunctive realizations in EAP writing by four groups of writers. Ampersand, 4, 47-57. https://doi.org/10.1016/j.amper.2017.07.001

*Kirsty J Freeman
Duke-NUS Medical School
8 College Road, Singapore 169857
Tel:+65 89219676
Email: kirsty.freeman@duke-nus.edu.sg

Submitted: 1 October 2020
Accepted: 3 December 2020
Published online: 4 May, TAPS 2021, 6(2), 78-87
https://doi.org/10.29060/TAPS.2021-6-2/OA2406

Yoshitaka Maeda¹, Yoshikazu Asada², Yoshihiko Suzuki¹, Akihiro Watanabe³, Satoshi Suzuki³ & Hiroshi Kawahira¹

¹Medical Simulation Center, Jichi Medical University, Japan; ²Center for Information, Jichi Medical University, Japan; ³Faculty of Health and Medical Sciences, Kanagawa Institute of Technology, Japan

Abstract

Introduction: Students in the early years of medical school should learn clinical site risk assessment skills. However, the effect of this training on clinically inexperienced students is not clear, and it is difficult for students to predict risks from a wide range of perspectives. This study aims to develop and implement three patterns of safety walk rounds (SWR) in a class of students with no clinical experience.

Methods: Three types of SWR were conducted: (A) 37 students observed a familiar classroom and predicted safety risks; (B) 39 students created a profile of a fictitious student in advance and then used Type A parameters; (C) 100 students participated. First, Type A was conducted as a practice. Next, students observed a hospital and predicted risks. All participants in Types A to C had no clinical experience. We classified all risks into perception, comprehension, and action.

Results: For each safety walk-round, there were two types of risk prediction. In Type A, risks such as perception and comprehension were more than 80%. In Types B and C, action risks were 60%. Students had little experience in observing facilities and none at finding safety risks.

Conclusion: Each method had a different risk prediction tendency. Combining the methods could enable students to acquire comprehensive skills in assessing hidden environmental patient safety risks.

Keywords:            Patient Safety Education, Undergraduate Education, Risk Assessment Skill, Safety Walk-Rounds

Practice Highlights

• Proposes a patient safety education method incorporating safety walk rounds (SWR).
• Clarifies the risk prediction tendency of clinically inexperienced students in each SWR pattern.
• Students conducting SWR in familiar classrooms tend to predict certain risks.
• Creating fictitious user profiles before SWR enables prediction of action risks.
• Combining different SWR types could enable comprehensive risk assessment skills.

I. INTRODUCTION

In Japan, first-year medical students are recent high school graduates. 60% of universities that train medical professionals provide patient safety education to fourth year medical students at the start of clinical training (Ishikawa et al., 2008). Further, lower grade educational methods do not include specific guidelines for patient safety education, and students in lower grades do not have sufficient medical knowledge to immediately apply their patient safety knowledge in clinical practice. This problem has been pointed out not only in Japan but also in the US and Canada (Alper et al., 2009). Conversely, the Telluride Interdisciplinary Roundtable (Mayer et al., 2009) and Lucian Leape Institute (2010) showed that patient safety education should be included in the curriculum of all grades. This would enable students to learn the necessity and importance of patient safety knowledge and consider patient safety as an implementation science while continuously practising patient safety skills (Nakajima, 2012).

However, many medical schools teach basic patient safety knowledge through lectures on accident analysis tools, legal responsibility knowledge, ethics, and infection (Mayer et al., 2009); however, students lack education on non-technical skills (Mayer et al., 2009; Nakajima, 2012; Walton et al., 2010). Students should be trained in awareness of safety weaknesses, threats (risks) in the workplace or operations, and how to avoid these risks (Doi et al., 2012). Topic 6 of the World Health Organization’s (WHO’s) Patient Safety Curriculum Guide indicates that students need to take appropriate corrective action when they see an unsafe situation or environment (Walton et al., 2010). However, the WHO guidelines do not explain how these risk assessment skills can be taught to students. Literature that examines the effectiveness of risk assessment skills training for early-year medical students is deficient.

To address these issues, we focused on Safety walk rounds (SWR), in which a safety manager goes to a workplace, listens to staff opinions on safety, and observes the workplace to identify safety issues before an accident (Hafey, 2017; Womack, 2013). Singer and Tucker (2014) pointed out that SWR enhances safety culture. The effects of SWR in the radiology department have reduced the number of unsafe events by half (Donnelly et al., 2008). Additionally, other studies reported that safety managers grow more sensitive to safety issues using SWR and that motivation regarding safety is increased (Frankel et al., 2003; Singer & Tucker, 2014). However, its educational effect and applicability to educating clinically inexperienced students are not clear since SWR has not been used for education.

This study aims to develop and implement three patterns of SWR in a class of students with no clinical experience. We clarify the risk prediction tendency of students in each SWR pattern and discuss the effects.

 II. METHODS

A. Development Process of Three Types of SWR

We developed three patterns of SWRs to help clinically inexperienced students predict risk and considered what motivates students to learn. We used the ARCS model proposed by J. M. Keller in 1983, which is a framework using four elements: Attention (stimulating the learner’s interest, intellectual curiosity, and inquisitiveness); Relevance (making the content familiar and meaningful); Confidence (encouraging learners to learn); and Satisfaction (giving the learner a sense of satisfaction and motivation to learn more) (Keller, 1987). ARCS is an acronym for these elements.

· SWR in daily situations (SWR-D): Experts are better at predicting risks than novices as the latter has limited knowledge of important aspects of each situation (Murata et al., 2009). Hence, clinically inexperienced students might find it difficult to predict risks in clinical situations. Using the ARCS model, students need to be given Attention, Confidence and Satisfaction. Therefore, we developed SWR in daily situations (SWR-D). Students observe daily situations in classrooms and school buildings for instances of safety risk and take pictures. We use daily situations as classrooms and school buildings are familiar environments for students, and there are many safety risks for educating students.

· SWR in daily situations using the Persona method (SWR-DP): It may be difficult for students who have never performed SWR to observe safety risks in daily situations, and students’ Confidence should be high. Therefore, we combined the persona method with SWR to create a virtual profile of a virtual user, including name, gender, age, and information about the system (e.g., technological literacy). The Persona method has been used to examine the safety of driving support systems (Lindgren et al., 2007). We hypothesize that the Persona method would help students to predict risk from the perspective of a specific user. The student considers the problems the virtual user will face and their behaviour (Cooper, 2004; Mulder & Yaar, 2006). Students created fictitious student profiles (personas) and conducted SWR assuming that the persona students would spend one day in school buildings and classrooms.

· SWR in clinical situations (SWR-C): It may be difficult for students to associate SWR with patient safety in clinical practice, as SWR in daily situations were not related to clinical practice conducted. Further, students might not be motivated to learn—using the ARCS model, students need to see Relevance and Satisfaction. Therefore, we developed the SWR in clinical situations (SWR-C). First, to practice SWR, students performed SWR-D. After SWR-D, they observed clinical situations in hospitals to predict risks (SWR-C).

B. Description of Participants and SWR Implementation Process

This study involved first-year medical students and third-year students in the medical engineer training courses who had no clinical experience. After participating in each SWR pattern, students were asked for their opinions.

1) SWR-D: The participants included 37 students in the third-year medical engineer training course and 100 students who had been in medical school for one month. SWR-D was administered to the third-year (2018) medical engineer training students in one session. SWR-D was implemented as one of the required general education courses for first-year medical students in 2019. An exercise using SWR-D was given to all participants who worked in groups of four to five. Students photographed incidences of safety risk (30 minutes) and collaborated to identify the risks in each photo (20 minutes).

2) SWR-DP: The participants included 39 students (different from SWR-D) in the third-year medical engineer training course. SWR-DP was administered to the third-year (2019) medical engineering students in one session and conducted in groups of four to five. Each group considered one persona (virtual student profile) for the first 20 minutes, and SWR-D was conducted as before.

3) SWR-C: The participants included 100 students (same as SWR-D) who had been in medical school for one month. Each student was assigned one clinical department in advance. Two weeks after the SWR-D, early exposure training was conducted. During training, students found safety risks in clinical situations and outlined the identified risks in reports as photography was not allowed for confidentiality.

C. Statistical Analysis

The risks predicted by students in each SWR pattern were counted and classified into the following: perception—difficulty perceiving something that exists in the outside world (e.g., signs that are difficult to read); comprehension—difficulty understanding the meaning of something that exists in the outside world and in planning what action to take (e.g., signs that are difficult to understand); and action—difficulty performing the intended action (e.g., places where it is difficult to walk).

These classifications are based on Norman’s seven stages of action (Norman & Draper, 1986) where human actions are classified into seven stages: forming the goal, forming the intention, specifying an action, executing the action, perceiving the state of the world, interpreting the perception, and evaluating the outcome (Norman, 1988). This is a representative model widely used for the design evaluation of man-machine systems such as computers (Fleming & Koman, 1998) to understand human cognitive behaviour that leads to human error in medical treatment (Zhang et al., 2002; Zhang et al., 2004).

D. Ethical Considerations in This Research

The ethical requirements in this study are in accord with the Declaration of Helsinki. We emphasized and explained to students that participation was voluntary and that declining to cooperate would have no influence on their grades. We also explained that consent to participate could be withdrawn at any time, that the results of this study may be published after processing, and that the students’ personal information would not be revealed. The students entered their consent in the e-learning system Moodle. This study was considered exempt by the Jichi Medical University Review Board (Number 18-014).

III. RESULTS

The total number of perception, comprehension, and action risks in each SWR is shown in Figure 1. Table 1 shows typical predicted risks and some of the images taken by students. The data that support the findings of this study are openly available in Figshare at <http://doi.org/10.6084/m9.figshare.13012664 (Maeda et al., 2021)>.

Figure 1. Classification results of risks predicted by students in each SWR pattern

Table 1. Typical risks and captured images for each SWR pattern

IV. DISCUSSION

A. Student Risk Predicting Tendency in Each SWR Pattern

The importance of institutional design in patient safety has been pointed out in many publications. For example, environmental design is being considered to avoid various risks, such as falls and patient suicides, mixing up patients, and improper handling of tubes and connectors (Joseph & Rashid, 2007; Michalska & Szewieczek, 2007; Reiling, 2006; Reiling et al., 2003; Reiling et al., 2008). In particular, it has been pointed out that the indoor environment (e.g., noise and lighting) and interior design (e.g., furniture and materials) are important (Joseph & Rashid, 2007). When considering the design, it is necessary to predict both the direct impact risks and the indirect impact risks of accidents as points of view when predicting onsite risks. Direct impact risks are the aspects of hospital design that can directly impact safety outcomes, such as patient falls and medical errors (Joseph & Rashid, 2007). This is considered to correspond to “action” in this study. For example, a tall counter design is directly linked to the undesirable consequence of a wheelchair user being unable to receive a meal. Indirect impact risks are the aspects of hospital design that can cause users to make incorrect decisions that lead to accidents and errors (Joseph & Rashid, 2007). This is considered to correspond to “perception” and “comprehension” in this study. For example, a paediatric refrigerator’s “be careful of allergies” poster does not directly lead to an unsafe outcome; however, if a child is unable to understand, then an incorrect decision to eat food to which the child is allergic may lead to an unsafe outcome.

In SWR-D, many indirect impact risks (perception and comprehension) were predicted. Conversely, few action risks (direct impact risks) were predicted. SWR-DP and SWR-C results were the opposite to SWR-D; many direct impact risks and few indirect impact risks were identified.

In SWR, it is necessary to identify risks through brainstorming and simulation using the operators’ experience and reasoning (Okubo et al., 2014). However, different techniques identify different risk types. Indirect impact risks such as perception and comprehension are related to human internal thinking, such as incorrect decisions. To identify these risks, students need to observe from the perspective of the person concerned and brainstorm the risks. Conversely, the action is observable—a risk that students can predict by observing the actions of the concerned person or simulating behaviour as the concerned person. Subsequently, what caused each SWR to favour predicting one type of risk over another?

In SWR-D, most students observed a familiar daily situation from their (able-bodied) perspective. In SWR-DP, students observed a familiar daily situation from a persona’s point of view that differed from that of an able-bodied person. In SWR-C, they observed unfamiliar clinical situations from the patients’ perspective (it is unknown whether this perspective was different from their own). This suggests that when students observe a familiar environment from their own perspective, they concentrate on brainstorming about risks but do not conduct much action simulation (acting on a simulated basis and identifying risks). Consequently, the risks related to action were few.

Conversely, if students observe risk from others’ perspective, they may not be able to brainstorm well, and they may tend to predict risks by performing action simulations. In SWR-DP, many images of simulations, such as trying to use a cafeteria as a persona (for example, a wheelchair user), were recorded. Notably, novice nurses tend to observe bedsides without being able to imagine the patient’s condition or behaviour (Daikoku & Saito, 2017); it seems difficult to predict risks associated with unfamiliar subjects (people/environment) only by brainstorming.

However, during SWR-C, we asked students to predict risks only by observation to avoid interference with patient care, and students were unable to perform action simulations. Nevertheless, the risks associated with actions were the most predicted. According to the risks predicted by the students (Table 1), students likely found patients who were confused and observed their behaviour. In the clinical situation, there were several observable patients that students could predict many risks based on observable actions.

In summary, for clinically inexperienced students to predict many indirect impact risks (perception, comprehension), it is better for them to make observations from their own perspectives in a daily situation where brainstorming can be easily conducted. It is better to ask students to observe an unfamiliar person and environment to identify more direct impact risks (actions). It would be better to create a fictitious user profile (persona) and conduct SWR (SWR-DP) or conduct SWR in a clinical situation unfamiliar to students (SWR-C). In any case, if each SWR is implemented independently, the predicted risks are biased. Therefore, by combining each SWR, it may be possible to develop skills that enable students to find direct impact and indirect impact risks in a well-balanced manner.

B. Limitations

This study evaluates the educational benefits of three SWR patterns and discusses their effectiveness. We could not compare the three patterns of SWR for the following aspects. First, we could not examine the relationship between participants’ background and SWR. This study targeted first-year medical students (SWR-C) and third-year students in the medical engineer training course (SWR-D, SWR-DP). Each student’s age and expertise were different. Therefore, the background of each participant could have affected the participant’s risk prediction tendency. However, all students shared the common background of having no clinical experience and had basic education on patient safety and conducted SWR with a basic knowledge of human factors, such as medical accident analysis methods. Second, we could not conduct a comparative study of each SWR by statistical analysis as there were differences in the way each SWR was conducted. SWR-D and SWR-DP were administered as a group, while SWR-C was administered individually because of restrictions in clinical practice. We could not calculate the average number of hazard predictions per student.

Further, each SWR was conducted in a compulsory class; therefore, there were a large number of students per faculty member. Consequently, we could not observe all students. Therefore, we have little record of how students predicted risks, especially in SWR-C. This is because the simulation actions in clinical situations and photography were restricted.

Additionally, when examining the safety of facility design, it is important to predict risks to healthcare professionals as well as patients (Reiling et al., 2003). However, in SWR-C, 90% of the risks were related to patients. Future studies should examine training methods that enable clinically inexperienced students to predict risks to healthcare professionals.

V. CONCLUSION

In this study, we proposed a patient safety education method incorporating SWR. We conducted SWR-D, SWR-DP, and SWR-C sessions and clarified the risk assessment tendency of students in each SWR pattern. For students to predict many indirect impact risks (perception comprehension), it is better to have students observe a daily situation in which it is easy to identify risks from their own points of view (SWR-D). To find many direct impact risks (action), it is better for students to create a persona and observe a daily situation (SWR-DP) or clinical situation (SWR-C). This suggests that a combination of these SWRs would provide students with the skills to comprehensively predict the patient safety risks in facilities and the environment. By continuously conducting all SWR session types starting at lower grade levels, it is expected that skills related to risk assessment will be effectively acquired. It is expected that SWR education from pre-graduates will increase the number of medical professionals who can conduct an appropriate risk assessment in the field, resulting in improved quality and safety of healthcare.

Notes on Contributors

Yoshitaka Maeda, PhD, a Research Associate at the Medical Simulation Center at Jichi Medical University, Japan, contributed to the conceptualization, methodology, validation, formal analysis, investigation, resources, data curation, writing (original draft, review, and editing), and visualization.

Yoshikazu Asada, PhD, an Assistant Professor at the Center for Information at Jichi Medical University, Japan, contributed to the methodology, validation, formal analysis, and visualization.

Yoshihiko Suzuki, MD, an Assistant Professor at the Medical Simulation Center at Jichi Medical University, Japan, contributed to the conceptualization and methodology.

Akihiro Watanabe, MS, a Research Associate at the Faculty of Health and Medical Sciences at Kanagawa Institute of Technology, Japan, contributed to the validation and investigation.

Satoshi Suzuki, PhD, a Professor at the Faculty of Health and Medical Sciences at Kanagawa Institute of Technology, Japan, contributed to the data curation, writing (review and editing), and visualization.

Hiroshi Kawahira, MD, PhD, FACS, a Professor at the Medical Simulation Center at Jichi Medical University, Japan, contributed to the writing (review and editing), supervision, and project administration.

All the authors have read and approved the final manuscript.

Ethical Approval

This study was approved by the Jichi Medical University Institutional Review Board (protocol number 18-014).

Data Availability

The data that support the findings of this study are openly available in Figshare repository, http://doi.org/10.6084/m9.figshare.13012664

Funding

There is no funder for this study.

Declaration of Interest

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of this article.

 References

Alper, E., Rosenberg, E. I., O’Brien, K. E., Fischer, M., & Durning, S. J. (2009). Patient safety education at US and Canadian medical schools: Results from the 2006 clerkship directors in internal medicine survey. Academic Medicine, 84(12), 1672-1676. https://doi.org/10.1097/acm.0b013e3181bf98a4

Cooper, A. (2004). The inmates are running the asylum: Why high-tech products drive us crazy and how to restore the sanity (Vol. 2). Sams Publishing.

Daikoku, R., & Saito, Y. (2017). Characteristics of gaze in expert nurses during observation at the bedside. Japanese Journal of Nursing Art and Science, 15(3), 218-226. https://doi.org/10.18892/jsnas.15.3_218

Doi, T., Kawamoto, K., & Yamaguchi, K. (2012). Difference by level of awareness and the years of experience to patient treatment safety. Japanese Journal of Radiological Technology, 68(5), 608-616. https://doi.org/10.6009/jjrt.2012_JSRT_68.5.608

Donnelly, L. F., Dickerson, J. M., Lehkamp, T. W., Gessner, K. E., Moskovitz, J., & Hutchinson, S. (2008). IRQN award paper: Operational rounds: A practical administrative process to improve safety and clinical services in radiology. Journal of the American College of Radiology, 5(11), 1142-1149. https://doi.org/10.1016/j.jacr.2008.05.017

Fleming, J., & Koman, R. (1998). Web navigation: Designing the user experience. O’Reilly Media.

Frankel, A., Gandhi, T. K., & Bates, D. W. (2003). Improving patient safety across a large integrated health care delivery system. International Journal for Quality in Health Care, 15(suppl_1), i31-i40. https://doi.org/10.1093/intqhc/mzg075

Hafey, R. B. (2017). Lean safety gemba walks: A methodology for workforce engagement and culture change. Productivity Press.

Ishikawa, M., Hirao, T., & Maezawa, M. (2008). Study of patient safety education for undergraduates. Medical Education, 39(2), 115-119. https://doi.org/10.11307/mededjapan1970.39.115

Joseph, A., & Rashid, M. (2007). The architecture of safety: Hospital design. Current Opinion in Critical Care, 13(6), 714-719. https://doi.org/10.1097/mcc.0b013e3282f1be6e

Keller, J. M. (1987). Development and use of the ARCS model of instructional design. Journal of Instructional Development, 10(3), 2-10. https://doi.org/10.1007/BF02905780

Lindgren, A., Chen, F., Amdahl, P., & Chaikiat, P. (2007). Using personas and scenarios as an interface design tool for advanced driver assistance systems. In: Stephanidis C. (eds) Universal Access in Human-Computer Interaction. Ambient Interaction. UAHCI 2007. Lecture Notes in Computer Science, vol 4555. Springer. https://doi.org/10.1007/978-3-540-73281-5_49　

Lucian Leape Institute. (Eds.). (2010). Unmet needs: Teaching physicians to provide safe patient care. National Patient Safety Foundation.

Maeda, Y., Asada, Y., Suzuki, Y., Watanabe, A., Suzuki, S., & Kawahira, H. (2021). Proposal for safety walk-round education to develop risk prediction skills of novice health professional students [Data set]. Figshare. http://doi.org/10.6084/m9.figshare.13012664

Mayer, D., Klamen, D. L., Gunderson, A., & Barach, P. (2009). Designing a patient safety undergraduate medical curriculum: The Telluride interdisciplinary roundtable experience. Teaching and Learning in Medicine, 21(1), 52-58. https://doi.org/10.1080/10401330802574090

Michalska, J., & Szewieczek, D. (2007). The 5S methodology as a tool for improving the organization. Journal of Achievements in Materials and Manufacturing Engineering, 24(2), 211-214.

Mulder, S., & Yaar, Z. (2006). The user is always right: A practical guide to creating and using personas for the web. New Riders.

Murata, A., Hayami, T., & Moriwaka, M. (2009). Visual information processing characteristics of drivers in prediction of dangerous situation-Comparison among novice, expert and non-licensed person. In IEEE SMC Hiroshima Chapter (Eds.), Proceedings of Fifth International Workshop on Computational Intelligence & Applications, (pp. 254-257). IEEE SMC Hiroshima Chapter.

Nakajima, K. (2012). Patient safety and quality of medical care. Topics: III. Management of patient safety and quality of medical care: Theory and practice; 4. Patient safety and quality improvement education for undergraduate medical students. Nihon Naika Gakkai zasshi. The Journal of the Japanese Society of Internal Medicine, 101(12), 3477. https://doi.org/10.2169/naika.101.3477

Norman, D. A. (1988). The psychology of everyday things. Basic Books.

Norman, D. A., & Draper, S. W. (1986). User centered system design: New perspectives on human-computer interaction. L. Erlbaum Associates.

Okubo, Y., Kuroda, R., & Yamamoto, K. (2014). Health and safety patrol in universities. Journal of Environment and Safety, 5(3), 169-175. https://doi.org/10.11162/daikankyo.14C0702

Reiling, J. (2006). Safe design of healthcare facilities. BMJ Quality & Safety, 15(suppl 1), i34-i40. https://dx.doi.org/10.1136%2Fqshc.2006.019422

Reiling, J., Breckbill, C., Murphy, M., McCullough, S., & Chernos, S. (2003). Facility designing around patient safety and its effect on nursing. Nursing Economics, 21(3), 143.

Reiling, J., Hughes, R. G., & Murphy, M. R. (2008). The impact of facility design on patient safety. In Hughes, R. G (Ed.), Patient safety and quality: An evidence-based handbook for nurses. Agency for Healthcare Research and Quality.

Singer, S. J., & Tucker, A. L. (2014). The evolving literature on Safety walkrounds: Emerging themes and practical messages. BMJ Quality & Safety, 23(10), 789–800. http://dx.doi.org/10.1136/bmjqs-2014-003416

Walton, M., Woodward, H., Van Staalduinen, S., Lemer, C., Greaves, F., Noble, D., Ellis, B., Donaldson, L., & Barraclough, B. (2010). The WHO patient safety curriculum guide for medical schools. BMJ Quality & Safety, 19(6), 542-546. http://dx.doi.org/10.1136/qshc.2009.036970

Womack, J. (2013). Gemba walks: Expanded 2nd edition. Lean Enterprise Institute.

Zhang, J., Patel, V. L., & Johnson, T. R. (2002). Medical error: Is the solution medical or cognitive? Journal of American Medical Informatics Association, 9(6 Suppl 1), s75-77. https://dx.doi.org/10.1197%2Fjamia.M1232

Zhang, J., Patel, V. L., Johnson, T. R., & Shortliffe, E. H. (2004). A cognitive taxonomy of medical errors. Journal of Biomedical Informatics, 37(3), 193-204. https://doi.org/10.1016/j.jbi.2004.04.004

*Yoshitaka Maeda, PhD
3311-1, Yakushiji, Shimotsuke-shi,
(Kinen-tou 7th floor),
Tochigi,Japan, 329-0498
Tel: +81-285-58-7455
Email: y-maeda@jichi.ac.jp

Submitted: 30 April 2020
Accepted: 8 September 2020
Published online: 4 May, TAPS 2021, 6(2), 66-77
https://doi.org/10.29060/TAPS.2021-6-2/OA2391

Pauline Luk & Julie Chen

The University of Hong Kong, Hong Kong

Abstract

Introduction: A novel initiative allowed third year medical students to pursue experiential learning during a year-long Enrichment Year programme as part of the core curriculum. ‘connect*ed’, an online virtual community of learning was developed to provide learning and social support to students and to help them link their diverse experiences with the common goal of being a doctor. This study examined the nature, pattern, and content of online interactions among medical students within this community of learning to identify features that support learning and personal growth.

Methods: This was a quantitative-qualitative study using platform data analytics, social network analysis, thematic content analysis to analyse the nature and pattern of online interactions. Focus group interviews with the faculty mentors and medical students were used to triangulate the results.

Results: Students favoured online interactions focused on sharing and learning from each other rather than structured tasks. Multimedia content, especially images, attracted more attention and stimulated more constructive discussion. We identified five patterns of interaction. The degree centrality and reciprocity did not affect the team interactivity but mutual encouragement by team members and mentors can promote a positive team dynamic.

Conclusion: Online interactions that are less structured, relate to personal interests, and use of multimedia appear to generate the most meaningful content and teams do not necessarily need to have a leader to be effective. A structured online network that adopts these features can better support learners who are geographically separated and engaged in different learning experiences.

Keywords:            Online Learning, Undergraduate, Interaction, Experiential Learning

Practice Highlights

• Image-based messages and less structured online activities focused on experience-sharing engage students and stimulate a more constructive discussion.
• The proactivity of students and mentors can foster a positive team dynamic and learning experience.
• A team or group leader is not always necessary to promote group interaction.

In general, all mentors were more active than students as teachers initiated new posts and were often keen to share information with students (Appendix 3). Even when students were encouraged to create new posts, they tended to focus on completing the tasks in the Inquiry Pods.

Diagram 1: Patterns of online interactions by teams

3) Pattern 3: Student degree centrality: Team 3 is such an example, showing that the thick arrows are pointing towards students, meaning that the interaction is initiated by students. The mentor took a less important role in the conversations. The degree centrality is towards students and the mentor was outside of the interaction centre. For instance, in team 3, the mentor posted 12 posts, 19 comments, and 20 reactions, while the seven students posted 1 to 12 post, 17 to 68 comments, and 0 to 64 reactions respectively. In this pattern, mentor was situated outside the conversation circle. The degree of centrality shifted to students.

4) Pattern 4: Student degree centrality: The dynamics of interaction leaned towards active students, which were represented by the thick edges towards certain students. In this pattern, there were usually multiple centre points that did not include the mentor. For instance, in team 4, the mentor posted 11 posts, 27 comments, and 90 reactions, while the seven students posted 0 to 3 post, 28 to 78 comments, and 0 to 39 reactions respectively. The degree of centrality shifted to multiple students.

5) Pattern 5: Diversified degree centrality: Mentors were active in posting, having similar frequency of comments as the students and the number of comments among all members are the same, and having low reactions. In this pattern, there are multiple conversation nodes and most are interactions between students. Those interactions are more student-driven and indicate multi-centred conversation. For instance, in team 5, the mentor posted 18 posts, 36 comments, and 3 reactions, while the six students posted 0 to 3 post, 31 to 62 comments, and 4 to 29 reactions respectively. The degree of centrality shifted to more than one student. In this pattern, there is more interaction between students as shown by the bi-directional arrows. The degree centrality is low with diversified centres.

These patterns show that teams could have single-centred interaction (Pattern 1 & 2) or multi-centred interaction (Pattern 3, 4, & 5) with each representing different team interactions. Team activity, and not the centeredness of the interaction, was associated with the effectiveness of collaboration and the completion of tasks. In addition, most teams demonstrated one-way communication when interacting. That means the reciprocity of a network is low. In Teams 1 and 2, the interaction dynamics favoured the mentors, while in Teams 3, 4, and 5, the dynamics leaned towards active students. In contrast, team 5 demonstrated strong reciprocity.

However, after comparing the patterns of all 33 teams, there is no indication that a certain pattern was better than the others. There was no significant difference in the on-time completion rate for the IP assignments for the five most active teams (86.9%) compared with all 33 teams (85.5%).

C. Content of Posts

In connect*ed, students shared their Enrichment Year experience using text, photo, video, or related links of other websites. The nature of interactions was predominantly text-based, as it is easier to post and interact using the text. However, image-based messages attracted more attention and stimulated a more constructive discussion.

There were three particular areas that generated greater levels of interactivity. Firstly, students were very willing to share and reflect on their personal experiences. Taking the ‘Communicator’ Inquiry Pod as an example, students shared their observations on communication in their respective settings by posting on the team wall:

“There is a huge contrast here, where students actively ask questions even if the setting involves 80+ students. I suppose the background behind the two nationalities have a huge role in it, as Asians tend to be a bit more shy compared to the extrovertiveness commonly shared by Westerners. While we should embrace who we were and are, I think it is also beneficial to observe others and learn from such observations.”

Student A (studied abroad)

This text-based conversation thread compared and contrasted effective classroom communication in different countries. It enabled students to reflect and to draw on their own experiences to benefit all team members.

The second area of interest for students was social support. One of the most popular activities was the posting of photos and videos about their Enrichment Year activities including when they are performing social service missions, cooking a gourmet meal or joining group gatherings during festive occasions. Those posts generated numerous responses and reactions indicating a keen interest in reaching out and maintaining social connectedness.

Thirdly, students were more active online when there was information being shared related to the medical practice and they are more willing to discuss their views as shown in this sequence from Team 3:

“Being a MBBS student, people around may ask us for medical advice. They think we are knowledgeable to make a diagnosis based on their description and believe we are able to help. However, as we are not yet qualified, it is inappropriate for us to give any professional opinion. Sometimes, I would like to share what I have learnt and suggest some possible solutions. Nevertheless, at the same time, I am afraid my opinion would affect their health seeking behaviour, for instance, they might just follow what I share instead of seeing a doctor.”

Student B 

“It’s true that we are not knowledgeable enough to give medical advice and it will be misleading to our relatives and friends if they take our opinions as professional advice and decide not to seek proper medical opinions. Thus, we should always remind ourselves of the role as medical students and think about the impacts of our words.”

Student C 

“I understand your feeling as my relatives and friends also ask me for medical advice. It will be safer to advise them to seek help from medical professionals for diagnosis or other serious health issues. However, as a medical student, I think it is possible for us to give them some lifestyle advice without causing harmful consequences, for instance, smoking cessation, diet with lower cholesterol content and moderate exercise. Although we are not qualified to make a diagnosis at the moment, we can still use our medical knowledge as a way to promote public health and arise their health conscious.”

Student D

Students more actively express their opinions when the topics under discussion are related to the profession they are aiming to join.

D. Feedback and Focus Group Interviews

Although there were only 9 student interviewees, we found repeated themes and content suggesting data saturation.  This may be because connect*ed comprised only 10% of the overall Enrichment Year and students did not pay particular attention to this component resulting in little variation in responses during the FGD.

The main theme that arose from the FGD with students and mentors was about the most rewarding aspect of the online interaction in connect*ed. Both groups indicated that this was the social connectivity attained through student sharing of day-to-day life during the Enrichment Year.

“The photo and video did not need much effort to share with others, but they are more interesting and can let me know more about how others were doing during their Enrichment Year.”

Student E

 “I am very interested in knowing the life of others in other universities. I hope they can share more and we could see others’ videos.”

Student F

 “Sharing things we learned with the team could help us to be more socialized.”

Student G

Mentors also enjoyed knowing more about students’ Enrichment Year life and believed that students should enjoy themselves while learning.

“The connect*ed is a good example helping students to bridge their knowledge and core value. The sharing of experience (related to the Enrichment Year) is important, it engages students in the discussion”

Mentor X

“The platform support each other very well. This is a platform for socializing and communicating. I know what students are doing if they posted on the team wall.”

Mentor Y

The value of social connectivity for support was further emphasized by students suggesting that the platform was more useful for social networking than learning.

 “connect*ed provided a platform for us sharing the struggles and support each other when I was having my Enrichment Year.”

Student I

This view was echoed by mentors who believed that connect*ed provided necessary support for students especially those who were overseas. Mentors used the chat function on Workplace to have personalized communication with their members and to offer advice.

 “I used the chat function on the Workplace, which is more personal and can support each other very well…. I can have immediate interaction with students.”

Mentor Z

Students found mentors were motivating and encouraged them to interact in teams which led to some mentor-centric team interactions.

 “Our mentor is very motivating and encourages all members to participate in the discussion. She guided us through to complete the inquiry pods.”

Student J

 “In my team, there are some inactive members who have demotivated me to interact. If there is an active member, I think it would help.”

Student K

The findings also indicated that proactivity by student members, participation by the mentor, responsiveness, and social/non-academic discussions fostered a positive team dynamic and a positive online learning experience, regardless of whether the team interaction was primarily single-centred or multi-centred.

IV. DISCUSSION

This study examined the nature, pattern, and content of online interactions among medical students within a virtual community of learning among the inaugural cohort of the Enrichment Year to identify features that support learning and personal growth.

Our results found that students favoured online interactions that were less structured, image-rich and focused on sharing of experience to learn from each other and to support one another. Multimedia content, especially images, attracted more attention and stimulated discussion that was more constructive. This is consistent with findings in the literature that show that images have a positive influence on learning and engagement (Chan & Unsworth, 2011; Stuijfzand et al., 2016). Sharing of personal experience helped students to reflect on their own experience and explore how others experienced their Enrichment Year. The results support previous studies that suggest self-reflection and community building enhances experiential learning (Arnold & Paulus, 2010; Pai, 2016). This builds a virtual community that allows students to share their struggles which students found to be a crucial aspect of giving and receiving social support. The use of the different modes of communication available on Workplace, including the text messaging and voice calls as well as social media posting provided flexible avenues of support to students. The finding is very similar to the outcomes of a project involving a mobile application for experiential learning activities (Schnepp & Rogers, 2017). We also observed that the number of positive reactions (like, love, haha or wow) far exceeded the number of negative reactions (sad and angry). This is a visual form of encouragement from mentors and peers that reflected their interest in engaging with each other.

From the patterns derived from our social network analysis, we found that the interaction could be uni-directional or bi-directional, but there was no correlation of the interaction with team effectiveness in completing tasks on time. As seen in the social network patterns identified, active mentors can drive team interaction. However, in contrast with other findings in the literature, the degree centrality and reciprocity do not affect the team interactional dynamic (Jan & Vlachopoulos, 2019). Regardless of the directionality of the predominant interaction, if there are active members in the team or the mentors are motivating, these individuals are the key to generating more interaction and enhancing online learning experience of students.

Ideally, both mentors and all students should be active, but having at least one or two more active students, can raise the team dynamic. Once some students are willing to share their experience and give timely responses, it can stimulate others to join. Continued encouragement of active members and mentors can promote a positive team dynamic. In terms of degree centrality, the observation that no pattern of interaction was superior to the others suggests that a single leader is not always necessary to for the team to be effective.

This study suggests that interactions will occur most naturally when students are doing what they feel is useful such as maintaining social support with each other and their mentor. In order to be accepted, learning initiatives such as linking learning with experiential activities will need to be less formal and integrate more smoothly with students’ demonstrated desire for social support and interest to share experience. In addition, attention to team formation and ensuring opportunity develop team cohesion would be essential as students in the FGD, commented that when there were members they do not know well, it will be a hindrance for interaction. As the connect*ed is one of the graded components of the Enrichment Year, we observed that the assessment could serve as an external motivator encouraging students to contribute to the work and support their team. However, it could also have a negative impact as it is perceived as an additional burden and may pressure some students to participate for the sake of participating and doing so in an inauthentic way.

 V. CONCLUSION

The online virtual community, connect*ed, to support experiential learning for medical students is still at an early stage. Features of connect*ed that facilitated learning and personal growth included a focus on student support and sharing especially with multimedia, less structured interactions, and teams with active members and/or mentor. It is important to note that interaction does not equate to learning (Jan & Vlachopoulos, 2019), and so the use of an online network that adopts these features may better support learners but the effectiveness of achieving formal learning outcome should be further studied. We will continue to modify and evaluate the functionality of the connect*ed community to ensure it is fit-for-purpose to support students’ needs and learning.

Notes on Contributors

Pauline Luk and Julie Chen contributed to the design and implementation of the research, analysis of the results and writing of the manuscript. PL drafted the manuscript, and JL edited and contributed to the intellectual content of the manuscript and provided overall supervision of the project. Both authors and approved the final manuscript.

Ethical Approval

This research received approval from the HKU Institutional Review Board (UW18-121). Consent was obtained from participants for the research study.

Acknowledgements

The authors sincerely appreciate the support from the mentors and students who participated in this study, the collaboration with the Education University of Hong Kong, and the administrative and technical support rendered by Mr. Francis Tsoi and Miss Joyce Tsang throughout the project.

Funding

This project was funded by the Hong Kong University Grants Committee (UGC) Funding Scheme for Teaching and Learning Related Proposals (2016-19 Triennium).

Declaration of Interest

The authors report no conflicts of interest.

References

Alamodi, A. A., Abu-Zaid, A., Eshaq, A. M., & Al-Kattan, K. (2018). The summer enrichment program: A multidimensional experiential enriching experience for junior medical students. The American Journal of the Medical Sciences, 356(2), 185-186. https://doi.org/10.1016/j.amjms.2018.05.005

Arnold, N., & Paulus, T. (2010). Using a social networking site for experiential learning: Appropriating, lurking, modeling and community building. The Internet and Higher Education, 13(4), 188-196.  https://doi.org/10.1016/j.iheduc.2010.04.002

Bastian, M., Heymann, S., & Jacomy, M. (2009). Gephi: an open source software for exploring and manipulating networks. In the International AAAI Conference on Weblogs and Social Media. 361-362. Retrieved April 1, 2020, from https://vbn.aau.dk/ws/files/328840013/154_3225_1_PB.pdf

Chan, E., & Unsworth, L. (2011). Image–language interaction in online reading environments: Challenges for students’ reading comprehension. Australian Educational Researcher, 38(2), 181-202.  https://doi.org/10.1007/s13384-011-0023-y

Greenberg, G. (2006). Can we talk?  Electronic portfolios as collaborative learning spaces. In A. Jafari & C. Kaufman (Eds.), Handbook of Research on ePortfolios. Idea Group Inc.

HKU. (2017). Educational aims and institutional learning outcomes. Retrieved April 1, 2020, from http://www.handbook.hku.hk/ug/full-time-2017-18/important-policies/educational-aims-and-institutional-learning-outcomes

Jan, S., & Vlachopoulos, P. (2019). Social network analysis: A framework for identifying communities in higher education online learning. Technology, Knowledge and Learning, 24(4), 621-639. https://doi.org/10.1007/s10758-018-9375-y   

Kallail, K. J., Shaw, P., Hughes, T., & Berardo, B. (2020). Enriching medical student learning experiences. Journal of Medical Education and Curricular Development, 7, 4. https://doi.org/10.1177/2382120520902160

Lave, J., & Wenger, E. (1991). Situated learning: legitimate peripheral participation. Cambridge University Press

Lyons, Z. (2017). Establishment and implementation of a psychiatry enrichment programme for medical students. Australasian Psychiatry, 25(1), 69-72. https://doi.org/10.1177/1039856216671663

Ma, L. (2012). Some philosophical considerations in using mixed methods in library and information science research. Journal of the American Society for Information Science and Technology, 63(9), 1859-1867.  https://doi.org/10.1002/asi.22711

Pai, H. C. (2016). An integrated model for the effects of self-reflection and clinical experiential learning on clinical nursing performance in nursing students: A longitudinal study. Nurse Education Today, 45, 156. https://doi.org/10.1016/j.nedt.2016.07.011

Saqr, M., Fors, U., Tedre, M., & Nouri, J. (2018). How social network analysis can be used to monitor online collaborative learning and guide an informed intervention. PLoS One, 13(3). http://dx.doi.org/10.1371/journal.pone.0194777

Schnepp, J., & Rogers, C. (2017). Evaluating the acceptability and usability of EASEL: A mobile application that supports guided reflection for experiential learning activities. Journal of Information Technology Education: Innovations in Practice, 16, 195.

Stuijfzand, B. G., van Der Schaaf, M. F., Kirschner, F. C., Ravesloot, C. J., van Der Gijp, A., & Vincken, K. L. (2016). Medical students’ cognitive load in volumetric image interpretation: Insights from human-computer interaction and eye movements. Computers in Human Behavior, 62, 394-5632.

Vygotsky, L. S. (1978). Mind in Society. Harvard University Press. https://doi.org/10.2307/j.ctvjf9vz4

*Pauline Luk
5/F William MW Mong Block
21 Sassoon Road,
Pokfulam, Hong Kong
Email: pluk@hku.hk

Submitted: 21 August 2020
Accepted: 12 November 2020
Published online: 4 May, TAPS 2021, 6(2), 57-65
https://doi.org/10.29060/TAPS.2021-6-2/OA2378

Nicholas Beng Hui Ng^1,2, Mae Yue Tan^1,2, Shuh Shing Lee³, Nasyitah binti Abdul Aziz³, Marion M Aw^1,2 & Jeremy Bingyuan Lin^1,2

¹Khoo Teck Puat-National University Children’s Medical Institute, National University Health System Singapore; ²Department of Paediatrics, Yong Loo Lin School of Medicine, National University of Singapore, Singapore; ³Centre for Medical Education (CenMED), Yong Loo Lin School of Medicine, National University of Singapore, Singapore

Abstract

Introduction: The coronavirus disease 2019 (COVID-19) pandemic has brought about additional challenges beyond the usual transitional stresses faced by a newly qualified doctor. We aimed to evaluate the impact of COVID-19 on interns’ stress, burnout, emotions, and implications on their training, while exploring their coping mechanisms and resilience levels.

Methods: Newly graduated doctors interning in a Paediatric department in Singapore, who experienced escalation of the pandemic from January to April 2020, were invited to participate. Participants completed the Perceived Stress Scale (PSS), Maslach’s Burnout Inventory (MBI), and Connor Davidson Resilience Scale 25-item (CD-RISC 25) pre-pandemic and 4 months into COVID-19. Group interviews were conducted to supplement the quantitative responses to achieve study aims.

Results: Response rate was 100% (n=10) for post-exposure questionnaires and group interviews. Despite working through the pandemic, interns’ stress levels were not increased, burnout remained low, while resilience remained high. Four themes emerged from the group interviews – the impacts of the pandemic on their psychology, duties, training, as well as protective mechanisms. Their responses, particularly the institutional mechanisms and individual coping strategies, enabled us to understand their unexpected low burnout and high resilience despite the pandemic.

Conclusion: This study demonstrated that it is possible to mitigate stress, burnout and preserve resilience of vulnerable healthcare workers such as interns amidst a pandemic. The study also validated a multifaceted approach that targets institutional, faculty as well as individual levels, can ensure the continued wellbeing of healthcare workers even in challenging times. 

Keywords:            COVID-19, Stress, Burnout, Resilience, Junior Doctor, Intern

Practice Highlights

• Intern doctors face additional and unique challenges in a pandemic, besides the usual stresses of their school-to-work transition.
• Our study shows that a multi-faceted approach that target institution, faculty and individual can lead to reduced burnout and preserved resilience in these doctors.

1) Theme 1 – Psychological Impact (Feelings): Most interns perceived that the pandemic had caused drastic changes in their personal and work lives, with various psychological impacts. They expressed increased emotional exhaustion such as stress and burnout, that is mainly related to changes in their clinical duties (Theme 2). The interns also shared about risks of COVID-19 infection to self and especially to family and loved ones, increasing their worries and stress. Interns followed physical distancing measures and team segregation at work, but several interns avoided their loved ones at home, especially the elderly and immunocompromised. For these interns, they further shared feelings of isolation and loneliness. Positive emotions such as feeling secure, valued and protected existed simultaneously and were mainly associated with the protective measures and support systems (Theme 4) in the workplace. Some also reported that the posting was still enjoyable and felt proud to be working in the pandemic.

2) Theme 2 – Impact on Duties: The interns highlighted there were many changes in institutional work processes and their duties due to the pandemic. Due to manpower changes, there were pervasive reports of physical fatigue. There were however those who felt the workload was still manageable. Interns also raised the issue of non-timely information and unclear protocols which often led to confusion and uncertainty in their work.

3) Theme 3 – Impact on Teaching and Learning: There were mixed comments on this. As a result of strict physical distancing and team segregation, initial planned teaching sessions on general paediatrics were cancelled and the interns felt they “missed out” on their clinical training.  Sessions were subsequently conducted using web-based platforms, which many found helpful. All interns felt that learning was restricted in the pandemic. Although it was beneficial to learn about pandemic response and management of suspected or affected COVID-19 patients, they felt their exposure to general paediatrics was reduced due to the limited variety of ward cases. However, there were some who felt there was better quality of teaching on the ward rounds as consultants had more time to teach with fewer elective and non-urgent cases in the rotations of non-COVID care.

4) Theme 4 – Preventive Measures and Support Systems: Despite the impacts on the interns’ psychology, duties and learning, they also shared on the various protective measures and support systems they perceived helped them cope. This was also the main reason for reported positive feelings of protection and support. Departmental and institutional work processes were implemented to take care of the interns’ physical and psychological welfare such as a rotational system of team segregation, which they reported provided a strict work-rest cycle as well as respite from COVID-care. In addition, seniors and faculty also ensured interns were competent and comfortable dealing with COVID-19 patients prior to taking on high risk duties such as swabbing patients. Support from multiple levels (seniors, department, institution) helped them through. In particular, the seniors and faculty provided support to the interns through regular “check-in” meetings where they could share concerns and provide feedback. The interns also shared that as a result of the strong support received, they were able to develop adaptability, perseverance and resilience, and they were even grateful to be in healthcare at this time.

IV. DISCUSSION

According to the demand-control-support model (Thomas, 2004), occupational stress causes burnout when job demands are high, individual autonomy is low and when job stress interferes with home life (Campbell et al., 2001; Linzer et al., 2001). On that note, we hypothesised that with the COVID-19 pandemic, interns would have increased stress and burnout, in addition to their routine difficulties in the transition from student to doctor. The pandemic-related concerns our interns had were similar to many healthcare workers globally – including the fear of contracting COVID-19 and more so transmitting it to vulnerable loved ones (Chen et al., 2020). Physical fatigue was also seen in our interns given the more intensive work schedule (Sasangohar et al., 2020). Although the total amount of admissions during the period was reduced to 40% of the usual load, the need for team segregation had led to a smaller pool of interns covering each clinical area. In addition, each intern had to do more in-house night calls while on active service. Segregation also meant that there would be less cross-coverage of duties where interns would receive less support from peers who would otherwise have been able to help with the workload on the ground. Another important aspect that had led to reported stress among many was the frequent changes in clinical workflows coupled with the lack of timely and reliable information (Wu et al., 2020). Many interns also highlighted concerns with regards to compromise and interference with their paediatric internship training (Liang et al., 2020). Despite all these, objectively the interns’ perceived stress was maintained without increase in burnout.

Burnout is known to be inversely related to resilience – this pattern is also reflected in our results. Resilience is the process of adapting well in the face of adversity, trauma, tragedy, threats or even significant sources of stress (Southwick et al., 2014). Our interns had high resilience scores, above what has previously been published among physicians (McKinley et al., 2020). One reason for this may be the development of resilience through a time of crisis, a phenomenon well encapsulated by the Crisis Theory: during a crisis or disequilibrium such as the current pandemic, people make attempts to adapt and seek solutions to restore stability. (Brooks et al., 2017; Caplan, 1964). The development of resilience is increasingly emphasised as an integral strategy to combat burnout. Potentially, the mitigating factors, coping mechanisms and support shared by our interns in the interviews, could explain their low burnout and high resilience.

Our interns perceived many systemic measures helped them cope with the pandemic – giving testament to the importance of institutional leadership in implementing safeguards for psychological health (Dewey et al., 2020; Wu et al., 2020). Protocols relating to staff protection, availability of personal protective equipment (Rasmussen et al., 2020) were some of the measures common to institutions worldwide. Furthermore, interns being the most junior member of the team, were spared from doing aerosolising procedures such as intubation, nebulisation administration and airway suctioning that were deferred to clinicians with prior experience and training. This allowed interns time to learn and improve in their competency and confidence prior to assuming these responsibilities. The interns were also thankful for the protected work-rest cycles (Wu et al., 2020), and that they were allowed to take paid leave – which is essential, more so in the pandemic to reduce fatigue and allowed time for rejuvenation.

Other than institutional support, direct support from seniors and faculty were significant in our interns’ responses in helping them, supporting the importance of mentorship (Ramanan et al., 2006). Despite feeling that they might not have reliable and timely access to important updates, they felt supported under the direct guidance of seniors who took the lead on the ground. Regular fortnightly ‘check-in’ sessions were conducted to elicit concerns, obtain feedback, and ensure continual wellbeing. This channel of communication was well received by interns: they appreciated the faculty’s concerns, had the autonomy of being able to input and contribute to the care of patients, the opportunity to air grievances confidentially and importantly, had closure on concerns they have raised regarding their rotations and training (Fischer et al., 2019). The enhanced collegiality between interns, support from seniors and improved cooperation among healthcare workers during this time of crisis naturally also contributed to reduced burnout levels, a finding well established in literature.(Li et al., 2013)

In terms of the impact of training, teaching sessions were initially discontinued to maintain physical distancing. Moreover, the interns had a higher proportion of time spent in the provision of COVID-19 care, which meant traditional general paediatric exposure was compromised. However, within 4 weeks of the pandemic, departmental teaching activities were restored via web-based sessions which interns found useful. The role of faculty in persisting with academic continuity, is again important in mitigating the impact of the pandemic on learning – some interns felt they had more teaching on the wards as consultants had more time to teach for each patient.

We believe that the perceived continual institutional and senior support for our interns allowed them to maintain high personal resilience, that could have mitigated their stress and burnout. In this pandemic, interns demonstrated adaptability and perseverance to the many changes, ability to persevere as well as finding gratitude amidst the challenges and focusing on their goal to help patients and fight the pandemic, which are all known features of resilience (Bird & Pincavage, 2016; Zwack & Schweitzer, 2013).

To our knowledge, this is the first research study in the pandemic that objectively evaluated the impact of the COVID-19 on interns’ psychological state, resilience and training. However, we recognise our study limitations. The small population would mean that it would be difficult to derive statistical comparisons in the pre- and post-exposure results. However, we believe the temporal exposure of the pandemic for this group of interns during their posting, made the pre- and post-pandemic results valid. The results were further supported by qualitative findings from a good group interview participation (100%) and in-depth discussion, that provided substantial explanations to the trend of results. We recognise that 2-4 months might be a short duration for negative psychological effects such as stress, and burnout to set in. Nonetheless, the amount of unprecedented changes and intensity of work for the interns involved within this period, were undoubtedly high. Another study limitation is the inclusion of Paediatric interns only and the possible lower exposure to COVID-19 as compared to their adult counterparts due to decreased disease morbidity and mortality in children. Although this factor could potentially result in less impact on the psychological factors studied, we believe other interns are likely to face similar concerns and challenges in the pandemic, due to their similar backgrounds and job scopes across most departments and disciplines.

This study elucidated the impact of the pandemic on interns in terms of their stress, burnout, as well as clinical duties and training. Despite increasing concerns on the psychological well-being of healthcare workers in the pandemic, our study has demonstrated that it is possible to mitigate their stress, burnout and preserve resilience, even in vulnerable new medical graduates. Our findings objectively validated the importance and effectiveness of the multi-faceted approach that target institution, faculty as well as the individual level, to build resilience and combat burnout in healthcare providers in this pandemic and beyond.

Notes on Contributors

Nicholas BH Ng contributed to conception and design of study, interpretation of data, drafting and critical revising of the article. Mae Yue Tan contributed to analysis and interpretation of data, drafting and critical revising of the article. Shuh Shing Lee contributed to analysis and interpretation of data, drafting and critical revising of the article. Nasyitah bte Abdul Aziz contributed to analysis and interpretation of data, drafting of the article. Marion M Aw contributed to interpretation of data, drafting and critical revising of the article. Jeremy BY Lin contributed to conception and design, interpretation of data, drafting and critical revising of the article. All authors gave final approval of the version to be published.

Data Availability

The data for this study can be found at https://doi.org/10.6084/m9.figshare.12924029.v1. The access to these datasets are available for use subject to approval of the authors of this article.

Ethical Approval

Ethics approval was obtained from the NHG Domain Specific Review Board (DSRB), with NHG DSRB reference number of 2020/00392.

Acknowledgement

The authors would like to thank the interns who participated in this study. 

Funding

Funding for this study was obtained from NUHS Fund Limited – Medical Affairs (Education) Fund.

Declaration of Interest

All authors have no conflicts of interest to declare.

References

Bird, A., & Pincavage, A. (2016). A curriculum to foster resident resilience. MedEdPORTAL, 12, 10439. https://doi.org/10.15766/mep_2374-8265.10439

Brooks, S. K., Dunn, R., Amlôt, R., Rubin, G. J., & Greenberg, N. (2017). Social and occupational factors associated with psychological wellbeing among occupational groups affected by disaster: A systematic review. Journal of Mental Health, 26(4), 373-384. https://doi.org/10.1080/09638237.2017.1294732

Campbell, D. A., Jr., Sonnad, S. S., Eckhauser, F. E., Campbell, K. K., & Greenfield, L. J. (2001). Burnout among American surgeons. Surgery, 130(4), 696-702; discussion 702-695. https://doi.org/10.1067/msy.2001.116676

Caplan, G. (1964). Principles of preventive psychiatry. Basic Books.

Chen, Q., Liang, M., Li, Y., Guo, J., Fei, D., Wang, L., He, L., Sheng, C., Cai, Y., Li, X., Wang, J., & Zhang, Z. (2020). Mental health care for medical staff in China during the COVID-19 outbreak. Lancet Psychiatry, 7(4), e15-e16. https://doi.org/10.1016/s2215-0366(20)30078-x

Chia, R., & Moynihan, Q. (2020, February 20). This alarming map shows where the coronavirus has spread in Singapore, one of the worst-hit areas outside of China Business Insider Singapore. Business Insider. https://www.businessinsider.com/coronavirus-singapore-map-shows-spread-worst-hit-outside-china-2020-2?IR=T.

Cohen, S., Kamarck, T., & Mermelstein, R. (1983). A global measure of perceived stress. Journal of Health and Social Behaviour, 24(4), 385-396.

Connor, K. M., & Davidson, J. R. (2003). Development of a new resilience scale: The Connor-Davidson resilience scale (CD-RISC). Depression and Anxiety, 18(2), 76-82. https://doi.org/10.1002/da.10113

Dewey, C., Hingle, S., Goelz, E., & Linzer, M. (2020). Supporting clinicians during the COVID-19 pandemic. Annals of Internal Medicine, 172(11), 752-753. https://doi.org/10.7326/M20-1033

Fischer, J., Alpert, A., & Rao, P. (2019). Promoting intern resilience: Individual chief wellness check-ins. MedEdPORTAL, 15, 10848. https://doi.org/10.15766/mep_2374-8265.10848

Li, B., Bruyneel, L., Sermeus, W., Van den Heede, K., Matawie, K., Aiken, L., & Lesaffre, E. (2013). Group-level impact of work environment dimensions on burnout experiences among nurses: A multivariate multilevel probit model. International Journal of Nursing Studies, 50(2), 281–291. https://doi.org/10.1016/j.ijnurstu.2012.07.001

Liang, Z. C., Ooi, S. B. S., & Wang, W. (2020). Pandemics and their impact on medical training: Lessons From Singapore. Academic Medicine. https://doi.org/10.1097/acm.0000000000003441

Linzer, M., Visser, M. R., Oort, F. J., Smets, E. M., McMurray, J. E., & de Haes, H. C. (2001). Predicting and preventing physician burnout: results from the United States and the Netherlands. The American Journal of Medicine, 111(2), 170-175. https://doi.org/10.1016/s0002-9343(01)00814-2

Low, Z. X., Yeo, K. A., Sharma, V. K., Leung, G. K., McIntyre, R. S., Guerrero, A., Lu, B., Lam, C. C. S. F., Tran, B. X., Nguyen, L. H., Ho, C. S., Tam, W. W., & Ho, R. C. (2019). Prevalence of burnout in medical and surgical residents: A meta-analysis. International Journal of Environmental Research and Public Health, 16(9). https://doi.org/10.3390/ijerph16091479

Maslach, C. J. S., & Leiter, M. P. (2016). Maslach burnout inventory manual. Mind Garden Inc.

McKinley, N., McCain, R. S., Convie, L., Clarke, M., Dempster, M., Campbell, W. J., & Kirk, S. J. (2020). Resilience, burnout and coping mechanisms in UK doctors: A cross-sectional study. British Medical Journal Open, 10(1), e031765. https://doi.org/10.1136/bmjopen-2019-031765

Ministry of Health (MOH), Singapore. (2020). Circuit breaker to minimise further spread of COVID-19. https://www.moh.gov.sg/news-highlights/details/circuit-breaker-to-minimise-further-spread-of-covid-19. (Retrieved April 3, 2020)

Ng, N. B. H (2020). The COVID-19 Pandemic: Impact on Paediatric Postgraduate Year One Doctors [Data set]. Figshare. https://figshare.com/s/74c81ca193638a553ea2

Ramanan, R. A., Taylor, W. C., Davis, R. B., & Phillips, R. S. (2006). Mentoring matters. Mentoring and career preparation in internal medicine residency training. Journal of General Internal Medicine, 21(4), 340-345. https://doi.org/10.1111/j.1525-1497.2006.00346.x

Rasmussen, S., Sperling, P., Poulsen, M. S., Emmersen, J., & Andersen, S. (2020). Medical students for health-care staff shortages during the COVID-19 pandemic. The Lancet, 395(10234), e79-e80. https://doi.org/10.1016/s0140-6736(20)30923-5

Rotenstein, L. S., Torre, M., Ramos, M. A., Rosales, R. C., Guille, C., Sen, S., & Mata, D. A. (2018). Prevalence of burnout among physicians: A systematic review. The Journal of the American Medical Association, 320(11), 1131-1150. https://doi.org/10.1001/jama.2018.12777

Sasangohar, F., Jones, S. L., Masud, F. N., Vahidy, F. S., & Kash, B. A. (2020). Provider burnout and fatigue during the COVID-19 pandemic: Lessons learned from a high-volume intensive care unit. Anesthesia and Analgesia, 131(1), 106–111. https://doi.org/10.1213/ane.0000000000004866

Southwick, S. M., Bonanno, G. A., Masten, A. S., Panter-Brick, C., & Yehuda, R. (2014). Resilience definitions, theory, and challenges: Interdisciplinary perspectives. European Journal of Psychotraumatology, 5,(1), 25338. https://doi.org/10.3402/ejpt.v5.25338

Sturman, N., Tan, Z., & Turner, J. (2017). “A steep learning curve”: Junior doctor perspectives on the transition from medical student to the health-care workplace. BMC Medical Education, 17(1), 92. https://doi.org/10.1186/s12909-017-0931-2

Thomas, N. K. (2004). Resident burnout. The Journal of the American Medical Association, 292(23), 2880-2889. https://doi.org/10.1001/jama.292.23.2880

Wu, P. E., Styra, R., & Gold, W. L. (2020). Mitigating the psychological effects of COVID-19 on health care workers. Canadian Medical Association Journal, 192(17), E459-e460. https://doi.org/10.1503/cmaj.200519

Zwack, J., & Schweitzer, J. (2013). If every fifth physician is affected by burnout, what about the other four? Resilience strategies of experienced physicians. Academic Medicine, 88(3), 382-389. https://doi.org/10.1097/ACM.0b013e318281696b

*Jeremy Bingyuan Lin
1E Kent Ridge Road,
NUHS Tower Block Level 12,
Singapore 119228
Tel: (65) 6772 4847
Email: jeremy_lin@nuhs.edu.sg

Submitted: 28 July 2020
Accepted: 18 November 2020
Published online: 4 May, TAPS 2021, 6(2), 48-56
https://doi.org/10.29060/TAPS.2021-6-2/OA2367

Oscar Gilang Purnajati¹, Rachmadya Nur Hidayah² & Gandes Retno Rahayu²

¹Faculty of Medicine, Universitas Kristen Duta Wacana, Yogyakarta, Indonesia; ²Department of Medical Education, Faculty of Medicine, Universitas Gadjah Mada, Yogyakarta, Indonesia

Abstract

Introduction: Objective Structured Clinical Examination (OSCE) examiners come from various backgrounds. This background variability may affect the way they score examinees. This study aimed to understand the effect of background variability influencing the examiners’ score agreement in OSCE’s procedural skill.

Methods: A mixed-methods study was conducted with explanatory sequential design. OSCE examiners (n=64) in the Faculty of Medicine Universitas Kristen Duta Wacana (FoM-UKDW) took part to assess two videos of Cardio-Pulmonary Resuscitation (CPR) competence to get their level of agreement by using Fleiss Kappa. One video portrayed CPR according to performance guideline, and the other portrayed CPR not according to performance guidelines. Primary survey, CPR procedure, and professional behaviour were assessed. To confirm the assessment results qualitatively, in-depth interviews were also conducted.

Results: Fifty-one examiners (79.7%) completed the assessment forms. From 18 background categories, there was a good agreement (>60%) in: Primary survey (4 groups), CPR procedure (15 groups), and professional behaviour (7 groups). In-depth interviews revealed several personal factors involved in scoring decisions: 1) Examiners use different references in assessing the skills; 2) Examiners use different ways in weighting competence; 3) The first impression might affect the examiners’ decision; and 4) Clinical practice experience drives examiners to establish a personal standard.

Conclusion: This study identifies several factors of examiner background that allow better agreement of procedural section (CPR procedure) with specific assessment guidelines. We should address personal factors affecting scoring decisions found in this study in preparing faculty members as OSCE examiners.

Keywords:            OSCE Score, Background Variability, Agreement, Personal Factor

Practice Highlights

• The examiners’ background variability influences the OSCE scoring agreement results.
• The reason for assessment inaccuracy remains unclear regarding the score agreement.
• The absence of assessment instruments that could provide a loophole for examiners to improvise.
• Personal factors affecting scoring decisions found in this study should be addressed in preparing OSCE examiners.

I. INTRODUCTION

To assess medical students’ competencies in a variety of skills, most medical schools in Indonesia implement the Objective Structured Clinical Examination (OSCE) both as a clinical skills examination at the undergraduate stage and as a national exit exam (Rahayu et al., 2016; Suhoyo et al., 2016). Most OSCE stations test both communication domains and specific clinical skills that will be assessed based on rubrics and scoring checklists which relies on examiners’ observations (Setyonugroho et al., 2015).  The OSCE has a challenge in its complexity to standardise  the scores, which are very depend on OSCE examiners’ perceptions (Pell et al., 2010). In a well-designed OSCE the examinees performance should only influence the examinees’ score, with minimal effects from other sources of variance (Khan et al., 2013). Research showed that there are influences of examiner’s background variability on OSCE results although they have been asked to standardise their behaviour (Pell et al., 2010) The decision and behaviour of OSCE examiners will affect the quality of assessment, including making a pass or fail decision, considering the complexity of knowledge, skill, and attitude in medical education (Colbert-Getz et al., 2017; Fuller et al., 2017).  

Examiners’ observations also rely on their clinical practice experience, OSCE examining experience, and gender conformity (Mortsiefer et al., 2017). Even in OSCE that is held in the most standard conditions, the examiner factor has the biggest role in scoring inaccurately (Mortsiefer et al., 2017). However, the reason for this inaccuracy remains unclear since there are concerns regarding the scoring agreement of examiners in OSCE and how the result might be affected by this issue. There is a need to consider the influence of examiners’ background variability (gender, educational level, clinical practice experiences, length of clinical practice experiences, OSCE experience, and OSCE training experience) when preparing teachers as OSCE examiners. This study aimed to understand background variability as a factor influencing examiners’ scoring agreement in assessing students’ performance in procedural skill, as the first step of faculty development program to ensure the standard quality for examiners.

II. METHODS

A. Study Design

This mixed-method study used a sequential explanatory design. This mixed-method approach is expected to provide more comprehensive results and better understanding than using a separated method (Creswell & Clark, 2018).

This study comprised of 2 sequential phases of data collection and analysis (QUANTITATIVE: qualitative) using sequential design. First, quantitative data were collected as a cross-sectional study of the examiners’ strength of agreement using Fleiss Kappa while assessing the clinical skill performance recorded in the 2 videos: one video portrayed CPR according to performance guideline and the other portrayed CPR not according to performance guideline. We used these 2 videos in order to portray more comprehensively how the consistency of OSCE examiner agreement both on good and poor clinical skill performance.

In the second phase, in-depth interviews were used to complement the quantitative results to gain more information and a detailed confirmation about how the scores were decided (Stalmeijer et al., 2014). In this stage of study, researchers explored and explained the examiners’ OSCE experiences and behaviour when they give a score on a clinical skill examination and the influences on their scoring regarding their backgrounds.

B. Materials and/or Subjects

The strength of agreement of the videos’ score came from 64 OSCE examiners FoM UKDW. Mortsiefer et al., (2017), explained that more subjects are better when investigate examiner characteristics associated with inter-examiner reliability (Mortsiefer et al., 2017).  In the second phase, in-depth interviews were conducted with 6 examiners of FoM UKDW, selected by purposive sampling regarding their scores and how they represented their own unique background (Table 1).

Researcher (OGP) provided all the participants with written information about this research and addressed ethical issues in an informed consent form. Researcher ensured participants understand the research protocol and clarified any questions regarding this study. Participants who agreed to take part, sign the informed consent form prior to the data collection. 

We held interviews in FoM UKDW with maximum 30 minutes of duration each interview. The inclusion criteria for examiners who were selected for this study were involved as full-time faculty members, had over 4 times OSCE examination experience, and had done OSCE examiner training, expecting that they had enough interaction with other faculty members and had influences from medical doctor education (Park et al., 2015). The exclusion criteria were participant did not answer the research invitation and did not fill the assessment form completely. Main researcher (OGP) conducted the interview. Main researcher was a male, student of Master of Health Profession Education Universitas Gadjah Mada, and the staff of FoM UKDW.

C. Statistics

1)Quantitative data analysis: We grouped examiners into 18 groups based on their background which were gender, educational level, clinical practice experiences, length of clinical practice experiences, OSCE experience, and OSCE training experience as shown in Table 1. We analysed all gathered data using IBM SPSS Statistics 25 and Microsoft Office Excel 365 (IBM Corp., Chicago). We presented quantitative data as a strength of agreement in percentage. The strength of agreement was calculated using Fleiss Kappa to determine the agreement between each group of each examiner background on whether CPR performances (primary survey, CPR Procedure, and professional behaviour), that portrayed in those 2 videos, were exhibiting score either “0”, “1”, “2”, or “3” based on the assessment guideline and rubric’s criteria (Purnajati, 2020). Based on recent research, agreement above 60% was considered as a substantial and adequate agreement (Stoyan et al., 2017; Vanbelle, 2019).

2) Qualitative data analysis: In-depth interviews were analysed using thematic analysis. We prepared a structured list of questions. It consisted of one key question: What was your experience in scoring the OSCE? The other additional questions evaluated the experiences of examiners in OSCE scoring including: the use of other references, differences in assessment weighting, use of own decision, clinical practice experience affecting the decision, and gender related decision making. Next, the collected data resulting from in-depth interviews were recorded using audio file recorder, read, and categorised into themes whenever they were related. The transcripts and identified themes were then given to an external coder in this study. This step was followed by our agreement for each theme. There was no repeated interview.

III. RESULTS

A. Quantitative Data Result

We deposited both quantitative and qualitative data in an online repository  (Purnajati, 2020).  The study participants in this quantitative phase were 64 OSCE examiners who are full-time faculty members. Twelve participants were excluded because did not fulfil the inclusion criteria. Fifty-one (79.7%) examiners who returned the completed assessment form are described below in Table 1.

The assessment rubric was divided into three main competencies: (1) primary survey, (2) CPR procedure, and (3) professional behaviour. The results showed overall agreement on each main competency based on each examiners’ background variability by using Fleiss Kappa. The percentage of agreement is shown in Figure 2, 3, and 4.

Figure 2. Primary Survey percentage of overall agreement (n = 51). Agreement above 60% (*) is considered as a substantial and adequate agreement

Figure 3. CPR Procedure percentage of overall agreement (n=51). Agreement above 60% (*) is considered as a substantial and adequate agreement

After completing the CPR competency assessment, all examiners’ background characteristics met a cutoff of approval above 60% in assessing CPR procedure except for examiners with clinical practice experience <3 years, OSCE testing experience <2 years, and OSCE examiner training> 5 years (Figure 3). This finding showed a good strength of agreement in assessing CPR procedure regardless of examiners’ background. However, there were many instances where the cut-off point of 60% was not achieved in the aspects of primary surveys and professional behaviour (Figure 2 and 4), which showed fair strength of agreement between examiners when they examined these competencies.

B. Qualitative Data Results

Two theme categories were determined: (1) OSCE experience and (2) specific behaviour in OSCE. The first theme contains of 3 sub-themes: (1) student performance, (2) examiner background effect, and (3) using assessment instrument. The second theme consists of 5 sub-themes: (1) use of assessment references, (2) score weighting, (3) personal inferences, (4) clinical experience, and (5) gender conformity.

Theme 1: Examiners argued that they understand the difference in student performance in performing clinical skills and can distinguish from the coherent skills performed by students according to checklist.

“Very easy in giving an assessment, because everything is in accordance with the assessment rubric”

                                                                                      (ID 35)

“The plot is clear, well organised”                      

      (ID 26)

“You can compare the inadequacies; it is enough to be compared”

                                                                                      (ID 11)

“The 2 different students are quite striking, so in my opinion it is not too difficult”

      (ID 28)

Nevertheless, some examiners had difficulty to distinguish student performance when only used a checklist. Examiner background did not affect their way in scoring clinical skills performance, but some background may have the potential to affect their scoring, such as clinical practice experience.

“I am trying to avoid personal interpretations, as much as possible, but of course that cannot be 100 percent. In my opinion, the assessment rubric still gives room for subjectivity”

                                                                                     (ID 28)

In this research, it seemed easy for examiners to understand the assessment instrument when giving score to those 2 videos and their understanding were good.

Theme 2: Interviews revealed that: 1) Examiners use other references such as their clinical experience in assessing the skills;

“If the assessment guideline is unclear, the students are also unclear, yes I will improvise. Or when the assessment guideline is clear and the students are unclear which criteria are included, yes I will improvise”

     (ID 35)

“Maybe yes, because once again the template at the beginning is not very clear”

 (ID 23)

2) Examiners use different ways in giving weight of competence, for example, procedural steps are considered more important than primary survey;

“For those that I feel have a small weight because the instructions are also short, so I don’t have to look carefully”

 (ID 24)

“When I feel that competence is not important, it does not get my emphasis, the more emergency that will get more attention.”

 (ID 28)

3) The first impression of examinees might affect their decision in scoring their performance;

“That first impression will affect me in giving value. I will be more critical. I see more, pay more attention to the small things they do”

(ID 24)

4) Clinical practice experience drives examiners to establish a personal standard on how a doctor should be;

“Clinical experience when practice is one of the judgments”

(ID 24)

“The reference is just my instinct because it has been running as a doctor after all these years. Yes, I use my previous knowledge”

(ID 26)

And 5) Gender of examinees does not affect their decision, while their professionalism (e.g. showing respect to patients) will surely affect their decision.

“I pay more attention especially to politeness and professional behaviour”

(ID 24)

“Students of any gender still have the same standard of evaluation, a score of professionalism which is more influential”

(ID 23)

 IV. DISCUSSION

Examiners’ agreement in this study was high in assessing the CPR procedure, which has a fixed and specific procedure in almost all groups of examiners. These results are consistent and can be explained by results from previous studies, which show that assessment with specific cases will provide high inter-examiner agreement (Erdogan et al., 2016). The differences in the examiner’s background will not have much influence on their agreement in giving an assessment in a specific case. This was supported by the opinions of examiners in the in-depth interviews who stated that in the CPR assessment procedure, assessment instruments are clear, easy to understand, with clear procedure flow, and performance that is easily distinguished, which made it easier for examiners to be able to distinguish student performance. A specific assessment instrument that could not provide a loophole for examiners to improvise assessment, made the opportunity for examiners to portray their subjectivity was minimised. This simplicity could lead to high agreement among examiners in specific competencies as shown in this study and based on clear evidence can increase the reliability of the assessment (Daniels et al., 2014) .

In this study, it was found in the primary survey assessment and professional behaviour which has an assessment guide that is not as specific as the CPR procedure, the percentage of agreement between examiner groups was lower, with only a few of them reaching 60% of agreement. This difference happened for reasons confirmed in the in-depth interviews which raised the issue that although the examiners tried to minimise their subjectivity in assessing, but it was said that there were still gaps in the assessment guide that still gives room for subjectivity. There are also examiners who were dissatisfied with the checklist, so they used their personal decisions in evaluating students.

According to a recent study, this could be due to the lack of specific instructions in the general assessment guidelines which will result in lower inter-examiner reliability compared to the use of more specific assessment guidelines (Mortsiefer et al., 2017). In the primary survey section and professional behaviour, there were also aspects of communication that were judged to be more susceptible to bias than physical examination skills because physical examination is more well-documented, clear instructions, and more widely accepted by examiners (Chong et al., 2018) The validity and reliability of a clinical skills assessment depend on factors including how the student’s performance on the exam, the character of the population, the environment, and even the assessment instrument itself can affect how examiners carry out the assessment (Brink & Louw, 2012). These phenomena were seen in the in-depth interviews which revealed that there were certain moments namely when the student being tested does not match the expectations written in the assessment guide and when the assessment guide is not clear so that it still gives room for subjectivity examiner. In addition, in the in-depth interviews the results also revealed that the examiners differentiated their attention on certain competencies with certain criteria such as the length of information in the assessment rubric, so that competencies that were considered not important did not get as much attention.

This finding may be in line with previous research which stated that constructs and conceptual definitions in this category that still provide a gap in the subjectivity of examiners cause shifting attention focus and weighting of their judgments to be different so that there are differences in important aspects between examiners (Schierenbeck & Murphy, 2018; Yeates et al., 2013). The difference in these important aspects can bring examiners to reorganise competency weights so that simpler and easier competencies (in this case those that have clearer and more detailed assessment guidelines) will be done first, and more complex ones (in this case, guides that have lower rigidity ratings) will be assessed later with the possibility of using more narratives (Chahine et al., 2015). This reorganisation can reflect how the examiners’ decision, allowing them to direct their attention to the more important aspects as the testers revealed in in-depth interviews with this research.

The personal factor, such as assessment references is a potential variability of the assessment conducted by the examiner. Examiners are trained and understand the use of assessment instruments, but produce varying assessments because they do not apply assessment criteria appropriately, but use personal best practice, use other test participants better as benchmarks, use patient outcomes (e.g. correct diagnosis, do patients understand, etc.), and use themselves as a comparison (Gingerich et al., 2014; Kogan et al., 2011; Yeates et al., 2013).

Another personal factors, including first impressions, can occur spontaneously unconsciously and can be a source of difference in judgment between examiners (Gingerich et al., 2011). First impressions based on observers’ observations have the same decisions and influences as social interactions, so it makes sense that first impressions are able to influence judgments, can be accurate and have a relationship with the final assessment results, but do not occur in examiners in general (Wood, 2014; Wood et al., 2017).

In providing assessments, there are gaps for examiners to give different competency weights to other examiners. Providing assessments based on targets that differ from competency standards and comparisons with the performance of other examinees will make the examiners recalibrate their own weighting and this is an explanation why there are variations in assessment and differences in the important points of the examinees’ performance among examiners (Gingerich et al., 2018; Yeates et al., 2015; Yeates et al., 2013).

The variability of personal factors between examiners can be conceptualise more as a different emphasis on building doctor-patient relationships and / or certain medical expertise rather than variations in the examiner’s background itself. The examiners’ own understanding can be conceptualized as a combination of whether what the examinees do is good enough and whether what they do is enough to build a doctor-patient relationship.

This research had some limitations such as it only used specific cases (i.e., CPR) to minimise the bias of the assessment instrument so that it would reveal more bias in the examiners themselves. In more complicated cases such as communication skills and clinical reasoning it is also necessary to provide a more complete picture of how the examiners’ scores agree in other cases. Generalization also became a limitation in this study because it only involved examiners from one medical education institution, however the study participants sufficiently described the variability of the examiner’s background.

V. CONCLUSION

This study identifies several factors of examiner background variability that influence examiners’ judgment in terms of inter-examiner agreement. Female examiners, bachelor education, less OSCE experience, and non-clinician examiners allow better agreement of procedural section (CPR procedure) with specific assessment guidelines. Cases that have unspecified assessment guidelines in this research, primary survey and professional behaviour, have lower agreement among examiners and must be examined deeper. We should note that personal factors of OSCE examiners can influence assessment discrepancies. However, the reasons for using these personal factors in scoring OSCE performance might be affected by unknown biases that require further research. Therefore, to improve clinical skills assessment such as OSCE for undergraduate medical programme, we must address personal factors affecting scoring decisions found in this study in preparing faculty members as OSCE examiners.

Notes on Contributors

Oscar Gilang Purnajati, MD was student of Master of Health Professions Education Study Program, Faculty of Medicine, Universitas Gadjah Mada, Indonesia. He concepted the research, reviewed the literature, designed the study, acquisited funding, conducted interviews, analysed quantitative data and transcripts, and wrote the manuscript.

Rachmadya Nur Hidayah, MD., M.Sc., Ph.D is lecturer of Department of Medical Education, Faculty of Medicine, Universitas Gadjah Mada, Yogyakarta, Indonesia. She supervised author Oscar Gilang Purnajati, developed the concepted framework for the study, critically analysed the data, cured the data, and reviewed the final manuscript.

Prof. Gandes Retno Rahayu, MD., M.Med.Ed, Ph.D is professor at the Department of Medical Education, Faculty of Medicine, Universitas Gadjah Mada, Yogyakarta, Indonesia. She supervised author Oscar Gilang Purnajati, advised the design of the study, critically analysed the data, gave critical feedback to the conducted interviews, reviewed the final manuscript.

All the authors have read and approved the final manuscript.

Ethical Approval

This study was approved by Health Research Ethics Committee Faculty of Medicine Universitas Kristen Duta Wacana (Reference No.1068/C.16/FK/2019).

Data Availability

All data were deposited in an online repository. The data is available at Open Science Framework with DOI: https://doi.org/10.17605/OSF.IO/RDP65

Acknowledgements

The author would like to thank Hikmawati Nurrokhmanti, MD, M.Sc for helping with the process of coding the in-depth interview transcripts. The author also would like to thank the staffs of Faculty of Medicine, Universitas Kristen Duta Wacana for supporting the research.

Funding Statement

This work was supported by the Universitas Kristen Duta Wacana (No. 075/B.03/UKDW/2018) as a part of study scholarship.

Declaration of Interest

No potential conflict of interest relevant to this article was reported.

Abbreviations and specific symbols

OSCE: Objective Structured Clinical Examination.

References

Brink, Y., & Louw, Q. A. (2012). Clinical instruments: Reliability and validity critical appraisal. Journal of Evaluation in Clinical Practice,18(6), 1126-1132. https://doi.org/10.1111/j.1365-2753.2011.01707.x

Chahine, S., Holmes, B., & Kowalewski, Z. (2015). In the minds of OSCE examiners: Uncovering hidden assumptions. Advances in Health Sciences Education : Theory and Practice, 21(3), 609–625. https://doi.org/10.1007/s10459-015-9655-4

Chong, L., Taylor, S., Haywood, M., Adelstein, B.-A., & Shulruf, B. (2018). Examiner seniority and experience are associated with bias when scoring communication, but not examination, skills in objective structured clinical examinations in Australia. Journal of Educational Evaluation for Health Professions, 15(17). https://doi.org/10.3352/jeehp.2018.15.17

Colbert-Getz, J. M., Ryan, M., Hennessey, E., Lindeman, B., Pitts, B., Rutherford, K. A., Schwengel, D., Sozio, S. M., George, J., & Jung, J. (2017). Measuring assessment quality with an assessment utility rubric for medical education. MedEdPORTAL : The Journal of Teaching and Learning Resources, 13, 1-5. https://doi.org/10.15766/mep_2374-8265.10588

Creswell, J. W., & Clark, V. L. P. (2018). Designing and conducting mixed method research. SAGE Publications, Inc.

Daniels, V. J., Bordage, G., Gierl, M. J., & Yudkowsky, R. (2014). Effect of clinically discriminating, evidence-based checklist items on the reliability of scores from an internal medicine residency OSCE. Advances in Health Sciences Education : Theory and Practice, 19(4), 497-506. https://doi.org/10.1007/s10459-013-9482-4

Erdogan, A., Dong, Y., Chen, X., Schmickl, C., Berrios, R. A. S., Arguello, L. Y. G., Kashyap, R., Kilickaya, O., Pickering, B., Gajic, O., & O’Horo, J. C. (2016). Development and validation of clinical performance assessment in simulated medical emergencies: An observational study. BMC Emergency Medicine, 16, 4. https://doi.org/10.1186/s12873-015-0066-x

Fuller, R., Homer, M., Pell, G., & Hallam, J. (2017). Managing extremes of assessor judgment within the OSCE. Medical Teacher, 39(1), 58-66. https://doi.org/ 10.1080/0142159X.2016.1230189

Gingerich, A., Kogan, J., Yeates, P., Govaerts, M., & Holmboe, E. (2014). Seeing the ‘black box’ differently: Assessor cognition from three research perspectives. Medical Education, 48(11), 1055–1068. https://doi.org /10.1111/medu.12546

Gingerich, A., Regehr, G., & Eva, K. W. (2011). Rater-based assessments as social judgments: Rethinking the etiology of rater errors. Academic Medicine : Journal of the Association of American Medical Colleges, 86(10), S1-S7. https://doi.org/10.1097/ACM.0b013e31822a6cf8

Gingerich, A., Schokking, E., & Yeates, P. (2018). Comparatively salient: Examining the influence of preceding performances on assessors’ focus and interpretations in written assessment comments. Advances in Health Sciences Education: Theory and Practice, 23(5), 937-959. https://doi.org /10.1007/s10459-018-9841-2

Khan, K. Z., Ramachandran, S., Gaunt, K., & Pushkar, P. (2013). The Objective Structured Clinical Examination (OSCE): AMEE Guide No. 81. Part I: An historical and theoretical perspective. Medical Teacher, 35(9), 1437-1446. https://doi.org/ 10.3109/0142159X.2013.818634

Kogan, J. R., Conforti, L., Bernabeo, E., Iobst, W., & Holmboe, E. (2011). Opening the black box of clinical skills assessment via observation: A conceptual model. Medical Education, 45(10), 1048-1060. https://doi.org/10.1111/j.1365-2923.2011.04025.x

Mortsiefer, A., Karger, A., Rotthoff, T., Raski, B., & Pentzek, M. (2017). Examiner characteristics and interrater reliability in a communication OSCE. Patient Education and Counseling, 100(6), 1230-1234. https://doi.org/ 10.1016/j.pec.2017.01.013

Park, S. E., Kim, A., Kristiansen, J., & Karimbux, N. Y. (2015). The Influence of Examiner Type on Dental Students’ OSCE Scores. Journal of Dental Education, 79(1), 89-94.

Pell, G., Fuller, R., Homer, M., & Roberts, T. (2010). How to measure the quality of the OSCE: A review of metrics – AMEE guide no. 49. Medical Teacher, 32(10), 802-811. https://doi.org/ 10.3109/0142159X.2010.507716

Purnajati, O. G. (2020). Does objective structured clinical examination examiners’ backgrounds influence the score agreement? [Data set]. Open Science Framework. https://doi.org/ 10.17605/OSF.IO/RDP65

Rahayu, G. R., Suhoyo, Y., Nurhidayah, R., Hasdianda, M. A., Dewi, S. P., Chaniago, Y., Wikaningrum, R., Hariyanto, T., Wonodirekso, S., & Achmad, T. (2016). Large-scale multi-site OSCEs for national competency examination of medical doctors in Indonesia. Medical Teacher, 38(8), 801-807. https://doi.org /10.3109/0142159X.2015.1078890

Schierenbeck, M. W., & Murphy, J. A. (2018). Interrater reliability and usability of a nurse anesthesia clinical evaluation instrument. Journal of Nursing Education, 57(7), 446-449. https://doi.org/10.3928/01484834-20180618-12

Setyonugroho, W., Kennedy, K. M., & Kropmans, T. J. B. (2015). Reliability and validity of OSCE checklists used to assess the communication skills of undergraduate medical students: A systematic review. Patient Education and Counseling, 98(12), 1482-1491. https://doi.org/ 10.1016/j.pec.2015.06.004

Stalmeijer, R. E., McNaughton, N., & Van Mook, W. N. (2014). Using focus groups in medical education research: AMEE Guide No. 91. Medical Teacher, 36(11), 923-939. https://doi.org/ 10.3109/0142159X.2014.917165

Stoyan, D., Pommerening, A., Hummel, M., & Kopp-Schneider, A. (2017). Multiple-rater kappas for binary data: Models and interpretation. Biometrical Journal, 60(5), 381-394. https://doi.org/ 10.1002/bimj.201600267

Suhoyo, Y., Rahayu, G. R., & Cahyani, N. (2016). A national collaboration to improve OSCE delivery. Medical Education, 50(11), 1150–1151. https://doi.org/ 10.1111/medu.13189

Vanbelle, S. (2019). Asymptotic variability of (multilevel) multirater kappa coefficients. Statistical Methods in Medical Research, 28(10-11), 3012-3026. https://doi.org /10.1177/0962280218794733

Wood, T. J. (2014). Exploring the role of first impressions in rater-based assessments. Advances in Health Sciences Education : Theory and Practice, 19(3), 409-427. https://doi.org/ 10.1007/s10459-013-9453-9

Wood, T. J., Chan, J., Humphrey-Murto, S., Pugh, D., & Touchie, C. (2017). The influence of first impressions on subsequent ratings within an OSCE station. Advances in Health Sciences Education : Theory and Practice, 22(4), 969-983. https://doi.org/10.1007/s10459-016-9736-z  

Yeates, P., Moreau, M., & Eva, K. (2015). Are examiners’ judgments in osce-style assessments influenced by contrast effects? Academic Medicine : Journal of the Association of American Medical Colleges, 90(7), 975-980. https://doi.org /10.1097/ACM.0000000000000650

Yeates, P., O’Neill, P., Mann, K., & Eva, K. (2013). Seeing the same thing differently: Mechanisms that contribute to assessor differences in directly-observed performance assessments. Advances in Health Sciences Education : Theory and Practice, 18(3), 325-341. https://doi.org/10.1007/s10459-012-9372-1

*Oscar Gilang Purnajati
Faculty of Medicine, 
Universitas Kristen Duta Wacana,
Jl. Dr. Wahidin Sudirohusodo No. 5-25.
Yogyakarta City,
Special Region of Yogyakarta
55224, Indonesia.55224, Indonesia.
Tel: +62-274-563929
Email: oscargilang@staff.ukdw.ac.id

Submitted: 8 July 2020
Accepted: 23 October 2020
Published online: 4 May, TAPS 2021, 6(2), 38-47
https://doi.org/10.29060/TAPS.2021-6-2/OA2338

Enjy Abouzeid¹, Rebecca O’Rourke², Yasser El-Wazir¹, Nahla Hassan¹, Rabab Abdel Ra’oof¹ & Trudie Roberts²

¹Faculty of Medicine, Ismailia, Egypt; ²LIME, University of Leeds, United Kingdom

Abstract

Introduction: Although, several factors have been identified as significant determinants in online learning, the human interactions with those factors and their effect on academic achievement are not fully elucidated. This study aims to determine the effect of self-regulated learning (SRL) on achievement in online learning through exploring the relations and interaction of the conception of learning, online discussion, and the e-learning experience.

Methods: A non-probability convenience sample of 128 learners in the Health Professions Education program through online learning filled-out three self-reported questionnaires to assess SRL strategies, the conception of learning, the quality of e-Learning experience and online discussion. A scoring rubric was used to assess the online discussion contributions. A path analysis model was developed to examine the effect of self-regulated learning on achievement in online learning through exploring the relations and interaction among the other factors.

Results: Path analysis showed that SRL has a statistically significant relationship with the quality of e-learning experience, and the conception of learning. On the other hand, there was no correlation with academic achievement and online discussion. However, academic achievement did show a correlation with online discussion.

Conclusion: The study showed a dynamic interaction between the students’ beliefs and the surrounding environment that can significantly and directly affect their behaviour in online learning. Moreover, online discussion is an essential activity in online learning.

Keywords:            Online Learning, Conception of Learning, E-learning Experience, Human-Computer Interface, Self-regulated Learning, Path Analysis

Practice Highlights

• The learner who views learning as a constructive process will show better use of self-regulated learning strategies.
• Learners’ beliefs and perceptions can shape the learning experience.
• Online discussion can directly and significantly affect academic achievement in online learning.
• Self-regulated learning is responsible for a small portion of the change in academic achievement.
• Online discussion may affect self-regulated learning negatively.

In just a few years, online e-learning has become part of the mainstream in medical education for postgraduates in both developed and developing countries. The use of online e-learning may provide solutions for many educational problems, especially for health professions graduates. It can help them achieving their developmental and educational goals despite the lack of time and overburdened schedules. This raised the need for better understanding of learning in online learning context.

The training that most schools offer to students and instructors on online leaning is mainly limited to using technologies that allow learners to interact with instructors and other learners effectively and flexibly. However, learners in online learning are facing several and complex challenges due to the nature of this context. Online learning is a form of distance learning that represent not only the access to learning experience via the use of technology and internet but also it relies on connectivity, flexibility and ability to promote varied interactions (Hiltz & Turoff, 2005). It characterised by autonomy and relative isolation due to the lack of face-to-face support. One of these important challenges is the need for self-regulated skills. It has been reported that these skills are more important in online learning as compared to traditional one (Azevedo et al., 2008).

Self-regulation is defined as the degree to which students are metacognitively, motivationally, and behaviourally active participants in their learning process (Zimmerman, 1986). This definition focused on students’ proactive use of specific behaviours to improve their academic achievement. In short, the ability to regulate one’s learning process is a critical skill to achieve personal learning objectives in online courses due to the absence of the support and guidance that is typically available in face-to-face learning environments (e.g., an instructor setting deadlines and structuring the learning process). Therefore, online learners need to determine when and how to engage with course content without any other support than the course content and structure, which can pose a challenge for many learners (Lajoie & Azevedo, 2006).

Hence, it seems reasonable to assume that SRL may be a reliable predictor of academic performance. It has been shown that self-regulated learners are more effective learners (Toering et al., 2012), who attain higher grades in medical education (Lucieer et al., 2016). However, the effect of SRL on academic achievement in online learning is still unclear.

Several factors may interact and affect learning in online learning. However, some had received only limited discussion in the medical education literature while others had relatively little empirical testing. Although several research studies have investigated the effect of conception of learning on learners’ approaches, efforts, and motivation, however the effect of conception of learning on self-regulation is still insufficiently explored. Moreover, it can be assumed that students in online learning context may show different conceptions of learning as studies have shown that conception of learning is a context-depended construct that may differ according to the domain of the study or the surrounding context (Chiu et al., 2016; Tsai & Tsai, 2014). Additionally, SRL processes depend on both the learner and the surrounding environment (Bembenutty, 2006). As a result, we assumed that the learners’ perception of the quality of the surrounding learning environment might directly affect their behaviour and outcomes.  In other words, the quality and interactivity of the learning environment may shape the learners’ attitude towards the learning experiences and influence the behavioural control of the learner (Zhao, 2016).

Figure 1: The study conceptual framework

Therefore, a model was hypothesized to explore the interaction between self-regulated learning, the conception of learning, online discussion, and the e-learning experience in an online environment, and how this interaction may affect academic achievement. This cross-sectional study provides an exciting opportunity to advance our knowledge about the learning process in online learning by raising the following questions: 

1. What is the relationship between SRL and academic achievement in online learning?
2. What are the interactions between personal characteristics, beliefs, behaviours, and environment in online learning?
3. Does these interactions affect academic achievement in online learning?

II. METHODS

A. Type of the Study and Setting

An observation cross-sectional study was performed at the Faculty of Medicine, Suez Canal University, Egypt. The Medical Education Department offers postgraduate online learning programs in Medical Education to the graduates of Health Professions Education specialties. The program is one of the first online programs in health professions education in the Arab region. It is a two-year program in which students submitted weekly assignments through WordPress / Eleum and receive online feedback on the same Learning Management system (LMS). Also, participate in an online discussion forum through the web-based application Listserv on Google group.

B. Participants and Sampling

‘Out of 231 learners in the online program, a non-probability convenience sample of 128 learners was recruited in the current study; of which, 88 participants had an input in the online discussion’. The subjects were selected from all the program fellows based on their approval to be included in the study sample. The participants were asked to participate in the study through a mass email composed of a detailed description of the nature of the study, the purpose of the study and its relevance to the field of medical education. In all cases, fellows were informed that any information they included in the questionnaires would be treated with confidentiality.

C. Data Collection Tools

Instruments were selected in the current study because it was constructed and used in relevant contexts and the design of the final version of the questionnaires were validated using factor; reliability and test- retest analysis.

1) Measuring learners’ self-regulated learning: The Online Self-Regulated Learning Questionnaire (OSLQ) was used to measure the self-regulated learning behaviours of the fellows (Barnard et al., 2008). The OSLQ consists of six subscale constructs including: environment structuring; goal setting; time management; help seeking; task strategies; and self-evaluation.

2) Measuring learners’ conception of learning: The mental model section of the Inventory of Learning Style (ILS) was used to explore the learners’ conception of learning. The questionnaire was kindly provided by J.D. Vermunt, who originally developed this inventory (Vermunt, 1998). The conception of learning section is composed of 25 items categorised under five scales: construction of knowledge, intake of knowledge, use of knowledge, stimulating education & cooperation of learning.

3) Measuring of the quality of e-learning experience: The e-Learning Experience Questionnaire was used to explore the role of the learning environment (Ginns & Ellis, 2007). The questionnaire consisted of subscales which would reflect students’ perceptions of Good Teaching, Good Resources Clear Goals and Standards, Appropriate Assessment, Generic skills, Appropriate Workload and student interaction.

4)Online discussion: The assessment of the fellows’ input in the online discussion was done by using a scoring rubric that was included in a framework proposed by Nandi et al. (2009). This framework defines several themes on which qualitative online interaction can be designed and assessed. The scoring rubric composed of three broad categories: content, interaction quality and participation.

5) Academic achievement: The fellows’ final grade is the sum of the educational units’ mean which, in turn, is the sum of the unit assignments’ mean was used as an indicator of academic achievement. The academic achievement was categorized into four categories according to the final mean of the units: excellent: means 9-10, very good: means 8, good: means 7 and pass: means 6 and fail means > 6.

III. RESULTS

Data analysis was conducted using Statistical Package for the Social Sciences (SPSS®) version 20 software and International Business Machines SPSS Amos™ version 20. Out of the 231 learners in the Health Professions Education program through distance learning, 128 postgraduate learners were included in the study. The sample composed of 40 males and 88 female learners. Furthermore, they were divided according to their previous academic rank into 2 groups (Dr: 69 & Prof: 59 students). Student t-test revealed that there is no significant difference between male and female in SRL, t (126) = 1.43, conception of learning, t (126) = 0.13, quality of E-learning experience, t (126) = 0.78, online discussion, t (126) = -1.46 and academic achievement, t (126) = -0.79, p<0.05.

*Enjy Abouzeid
6A Hassan El Bassry Street, 
Ismailia, Egypt
Email: Enjyabouzeid@yahoo.com

Submitted: 24 June 2020
Accepted: 8 September 2020
Published online: 4 May, TAPS 2021, 6(2), 31-37
https://doi.org/10.29060/TAPS.2021-6-2/OA2328

Julie Yun Chen^1,2, Weng-Yee Chin¹, Agnes Tiwari³, Janet Wong³, Ian C K Wong⁴, Alan Worsley⁴, Yibin Feng⁵, Mai Har Sham⁶, Joyce Pui Yan Tsang^1,2 & Chak Sing Lau⁷

¹Department of Family Medicine and Primary Care, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Hong Kong; ²Bau Institute of Medical and Health Sciences Education, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Hong Kong; ³School of Nursing, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Hong Kong; ⁴Department of Pharmacology and Pharmacy, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Hong Kong; ⁵School of Chinese Medicine, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Hong Kong; ⁶School of Biomedical Sciences, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Hong Kong; ⁷Department of Medicine, Li Ka Shing Faculty of Medicine, The University of Hong, Hong Kong

Abstract

Introduction: The demanding nature of medical and health sciences studies can cause stress among students in these disciplines affecting their wellbeing and academic performance. The Perceived Stress Scale (PSS-10) is a widely used measure of perceived stress among medical students and healthcare professionals that has not yet been validated among medical and health sciences students in Hong Kong. The aim of this study is to establish the construct validity and reliability of the PSS-10 in this context.

Methods: 267 final year medical and health sciences students were surveyed using the PSS-10. The data were analysed using exploratory factor analysis for construct validity and Cronbach’s alpha coefficient and corrected item-total correlations for reliability.

Results: Exploratory factor analysis revealed a two-factor structure for PSS-10, with Cronbach’s alpha of 0.865 and 0.796, indicating good internal consistency. Corrected item-total correlations showed satisfactory correlation ranged from 0.539 to 0.748 for all items and their respective subscale. Both tests supported PSS-10 as a two-factor scale.

Conclusion: The PSS-10 is a valid measure for assessing perceived stress in Hong Kong medical and health sciences students.

Keywords:            Undergraduate Students, Medicine, Nursing, Pharmacy, Health Sciences, Validation, Perceived Stress

Practice Highlights

• It is important to have a valid instrument for early detection of stress in health science students.
• Perceived Stress Scale (PSS-10) has a two-factor structure, a finding that is consistent with most other studies.
• PSS-10 has satisfactory internal consistency and reliability.
• PSS-10 can be used to assess the level of stress in medical and health sciences students.

I. INTRODUCTION

Undertaking studies in healthcare disciplines can be stressful as the programmes are demanding and students are often competing with higher achieving peers from admission to graduation. Significant stress can lead to psychological distress that has negative implications on current and future performance. Medical students have a higher prevalence of distress and poorer mental quality of life than their non-medical peers (Dyrbye et al., 2006; Shin et al., 2016), and also experience sleep deprivation, anxiety, and feelings of social isolation as revealed in focus group interviews conducted by Henning et al. (2010). There may also be a negative impact in quality of patient care (Firth-Cozens, 2001) and higher rate of medical errors (West et al., 2009). High perceived stress level correlated to impaired clinical performances in nursing students, including application of knowledge, clinical skills and communication (Ye et al., 2018). High level of stress and impaired quality of life were also found in third year pharmacy students in the United States (Marshall et al., 2008). In a study on pre-medical and health sciences students, higher perceived stress was a predictor of poor academic achievement (Henning et al., 2018).

As in many Asian cultures, Hong Kong students in general are under pressure to perform well in school as education is viewed as a crucial stepping-stone to success (S. Chan, 1999; Tan & Yates, 2011). This pressure may be particularly pronounced in medical students who manifest a greater degree of psychological distress, including perceived stress, depressive symptoms and anxiety, than other university students (Wong et al., 2005). A survey on medical students from the University of Hong Kong also revealed that majority of medical students were screened positive for minor psychiatric disorders and up to 95% of them were burned out (Chau et al., 2019). Many students may be “pushed” into a career path by extrinsic factors such as parental expectation (Sreeramareddy et al., 2007) or as a part of family tradition. Asian medical students may also tend to focus on academic achievement and seek to outperform their peers (Henning et al., 2011). Given the risk for developing high level of stress for these students, and the particularly intense environment in Hong Kong, it is important to have a valid instrument for early detection of stress so that appropriate strategies may be instituted at an early stage.

The Perceived Stress Scale (PSS-10) (Cohen, 1988) is widely used to measure perceived stress among healthcare students and doctors in different countries (Jones et al., 2015; Wongpakaran & Wongpakaran, 2010), and healthcare workers in Hong Kong (Chua et al., 2004). Healthcare students and healthcare workers may respond differently to a stressful event, as shown in the studies by Chua et al. (2004) and Wong et al. (2004), where the psychological effects of the SARS outbreak were different for healthcare students and workers. PSS-10 has been translated and validated in various languages, including Spanish, Turkish, Portuguese, Chinese, Thai and Japanese, among different populations such as patients, students, pregnant women, and adults in the general population (Lee, 2012). These validation studies are fundamentally robust, yet validating the PSS-10 is important in the specific undergraduate medical and health professions educational context in Hong Kong. Our study population is subject to different cultural, societal and educational influences that affect the perception of stress and the understanding of the items in the instrument so validation studies done elsewhere may not be applicable to our local context. The aim of this study is therefore, to establish the construct validity and reliability of the PSS-10 for use in this population.

 II. METHODS

A. Participants and Data Collection

All final year students undertaking studies in Li Ka Shing Faculty of Medicine in the University of Hong Kong (HKUMed) in the academic year of 2014-2015 were the target population of this study. A research assistant, who was not involved in teaching and assessment of the students, invited the students to participate in the study during a designated compulsory face-to-face teaching session for each programme. Those who provided written consent completed a written questionnaire in January – February 2015 or June 2015. The specific time for each cohort was chosen to avoid known stressful periods such as exams. The questionnaire included the PSS-10 and demographic information.

 B. Measure

The Perceived Stress Scale (PSS-10) (Cohen, 1988) was chosen as the instrument for measuring perceived stress. We considered other often-used instruments including the Depression Anxiety Stress Scale (DASS) (Lovibond & Lovibond, 1995) that measures depression and anxiety, in addition to stress and the General Health Questionnaire (GHQ) (Goldberg & Hillier, 1979) that measures medical complaints as a reflection of emotional stress, but these looked at broader conceptualisations of psychological distress beyond the scope of our study. PSS-10 was the most fit-for-purpose in measuring stress in terms of respondents’ views about their lives. In addition, we wished to be able to compare the stress in medical and health professions students to other key local comparator populations (e.g. university students, doctors, general population etc) and using the same instrument would facilitate this comparison.

PSS-10 is a 10-item instrument that assesses the extent of stress of respondents. PSS-10 is the abbreviated version of the original instrument with 14 items (PSS-14). A brief version with four items (PSS-4) is also available. Among the three versions of PSS, PSS-10 was found to be superior in psychometric properties, in terms of validity and reliability, than the other two versions (Lee, 2012). In the PSS-10, respondents rate statements about how unpredictable, uncontrollable, and overloaded they find their lives on a 5-point Likert scale from “never” to “very often”. Each response is converted to a score of 0 to 4 with the overall PSS score computed as the total score of the 10 items, with four reverse-coded items. The higher the score, the worse the perceived stress, with a maximum score of 40. There is no specific cut-off score that corresponds to high or low stress. We used the original English version of PSS-10 because as an English-medium university, students at HKUMed are taught in English (except during bedside teaching and clinical practicums) and students are proficient in English.

 C. Data Analysis

To establish the construct validity of the PSS-10, exploratory factor analysis (EFA) was performed on the responses to PSS-10 items by final year medical students, using principal component extraction with varimax rotation and the criterion of eigenvalue greater than 1.00. The Kaiser-Meyer-Olkin (KMO) measure equal to or greater than 0.5 was used to indicate sampling adequacy, while the Barlett’s Test of Sphericity with p<0.001 was used to ensure the appropriateness of the data set for EFA. Cumulative variance explained in the factor structure identified by EFA model was reported.

Cronbach’s alpha coefficient and corrected item-total correlations were used to examine reliability. Cronbach’s alpha coefficient was calculated to assess the internal consistency of each scale, which was considered acceptable if greater than 0.7 (Nunnally, 1994). Corrected item-total correlations were evaluated by Pearson’s correlation coefficient. A correlation of more than 0.4 was considered satisfactory (Wolfinbarger & Gilly, 2003).

 III. RESULTS

A total of 267 students completed the survey, with an overall response rate of 86.5%. 104 (39%) of the respondents were male (Table 1). Female students had significantly higher perceived stress than male students (20.84 vs 18.59; p<0.001). Table 2 shows the descriptive statistics for PSS-10 items and total score by programme of study.

*Julie Chen
Department of Family Medicine &
Bau Institute of Medical and Health Sciences Education,
Li Ka Shing Faculty of Medicine,
University of Hong Kong
21 Sassoon Rd, Pok Fu Lam
Hong Kong
Email: juliechen@hku.hk