Submitted: 7 December 2021
Accepted: 8 February 2022
Published online: 5 April, TAPS 2022, 7(2), 27-36
https://doi.org/10.29060/TAPS.2022-7-2/OA2718

Tan Charmaine¹ & Ivan Cherh Chiet Low^1,2

¹Department of Physiology, Yong Loo Lin School of Medicine, National University of Singapore, Singapore; ²Human Potential Translational Research Programme, Yong Loo Lin School of Medicine, National University of Singapore, Singapore

Abstract

Introduction: Practicals are core components of an undergraduate health sciences curriculum to promote experiential learning and motivation in students. With restrictions on traditional forms of face-to-face practicals during the COVID-19 pandemic, we designed and investigated the efficacy of remote practicals as a viable learning strategy in exercise physiology teaching.

Methods: Student volunteers were instructed to perform a graded exercise test in a remote setting and provide their collected data for subsequent discussion in an online lecture. The effectiveness of this remote practical in promoting students’ motivation and learning outcomes achievement was assessed via an anonymous questionnaire containing 29 closed-ended and 2 open-ended items. Continual Assessment (CA) results were also investigated. Unpaired student’s t-tests were performed for comparisons between interventions with significance level set at P<0.05.

Results: Sixty-one (out of 81; 75%) students responded to the questionnaires and 49 (60%) consented to the use of their CA results for this study. Results revealed that students were moderately motivated and attained strong achievement of learning outcomes. When compared to students who did not volunteer for the hands-on component of the remote practical, students who participated in the hands-on component reported significantly higher self-efficacy (P<0.05) in explaining the practical procedures to their peers. Qualitative analysis further revealed that experiential learning and real-life data analysis were the central reasons supporting the effectiveness of the remote practical. Students were generally satisfied and would recommend the remote practical to future students.

Conclusion: Our study highlights the potential of remote practicals as viable alternatives to traditional practicals.

Keywords:           Remote Practical, Experiential Learning, Student Motivation, Learning Outcomes

Practice Highlights

• Remote practical aids in promoting experiential learning in exercise physiology teaching.
• Remote practical can promote motivation by enabling students to see the relevance of their learning.
• Students perceived that they could achieve the necessary learning outcomes via remote practicals.

I. INTRODUCTION

Laboratory work or practical classes are considered as core components of health sciences curriculum in higher education (Colthorpe & Ainscough, 2021; Dohn et al., 2016; Hofstein & Lunetta, 2004). Past studies have revealed the strong educational value of practical classes in promoting student motivation (Bruce, 1988; Dohn et al., 2016), student learning outcomes achievement (Brinson, 2015) as well as the ability to draw theory to practical applications (Neves et al., 2017).

One of the key subjects in undergraduate health sciences education is human physiology, a discipline seeking to understand the underlying mechanisms and dynamics of the human body (Lellis-Santos & Abdulkader, 2020). The role of practical classes in the teaching and learning of physiology is highly valued by educators and students alike (Dohn et al., 2016; Neves et al., 2017). Experiential learning in physiology practicals commonly takes the form of interactive hands-on activities, real-time data collection and analysis of physiological responses. When such practicals are carried out in a traditional face-to-face manner, students are able to utilise laboratory equipment in an authentic experimental setting and generate real-time data from their peers and/or themselves (Colthorpe & Ainscough, 2021). Data analysis and discussion following the hands-on component of practicals can further promote contextualised learning and facilitate the understanding of the theoretical content (Lewis & Williams, 1994). It has been reported that such an interactive learning approach in physiology enhances the achievement of learning outcomes and increases the level of motivation for students (Dohn et al., 2016).

The emergence of the COVID-19 pandemic has led to increasing safety management restrictions being imposed on physical classes in higher education institutes around the world (Ali, 2020). As such, educators were faced with the sudden need to switch from face-to-face lessons to online and remote teaching (Ali, 2020; Lellis-Santos & Abdulkader, 2020). Amidst the uncertainty of this transition, traditional face-to-face practicals have seen a sharp decline (Ray & Srivastava, 2020; Vasiliadou, 2020). As we examine these educational trends during crises, it becomes apparent that harnessing creativity to adapt and invent novel solutions is vital to maintain and even advance current standards of teaching and learning. Lellis-Santos and Abdulkader (2020) rightfully exemplify this notion by proposing the use of smartphone applications as a creative teaching approach to enable scientific data collection and practical learning from home even during social isolation. Along similar lines, we have designed a remote practical for students to carry out hands-on experiments outside of a traditional laboratory environment as an innovative alternative to circumvent the restrictions on face-to-face practicals and to provide them with experiential learning opportunities on cardiovascular concepts in exercise physiology.

To the best of our knowledge, there are few studies conducted to date regarding the efficacy of such remote practicals, particularly in the field of life sciences and exercise physiology. Therefore, our study aims to address this research gap by evaluating the effectiveness of our remote exercise physiology practical on (1) student learning outcomes, (2) student motivation and (3) students’ perceptions on the effectiveness and relevance of the remote practical. In addition, we also compared the quantitative and qualitative responses between students who participated and did not participate in the hands-on component of the remote practical. The Continual Assessment (CA) results from these two groups of students were also compared to assess if differences in academic performance existed between the two groups.

II. METHODS

A. Description of the Module

LSM3212 Human Physiology: Cardiopulmonary System is a third-year module in Life Sciences conducted by the Department of Physiology, Yong Loo Lin School of Medicine at the National University of Singapore. A total of 81 undergraduate Life Sciences students were enrolled in the module in the Academic Year 2020/2021, Semester 2. Traditionally, both lectures and practicals are carried out in a face-to-face manner for this module. However, due to restrictions imposed by the COVID-19 pandemic, lessons were forced to go online. As a result, a remote practical was designed and conducted as an innovative alternative for this batch of students.

B. Description of the Remote Practical

The remote practical was designed to provide students with experiential learning on cardiovascular concepts in exercise physiology. Conventionally, the practical involved a treadmill-based graded exercise test performed by a student volunteer equipped with specialised electrocardiogram-based heart rate monitors to illustrate how the cardiovascular system changes with increasing exercise stress. For the remote practical, however, students were given a set of practical handouts comprising a novel graded exercise protocol developed by the authors and could choose to perform it in their own time asynchronously, or not carry it out at all. In the graded exercise protocol, students were instructed to carry out a series of graded knee raise exercises and record their heart rate measured via a smartphone application together with other subjective exercise prescription ratings (ratings of perceived exertion and talk test ratings) after each set of exercise. The graded exercise protocol was developed with the intent to encourage contextualised learning from the practical content to real-life exercise routines. The consolidated data was subsequently used for discussion in a virtual lecture to illustrate how heart rate responses and cardiovascular adaptations may differ across individuals, as well as how the consolidated data may serve to guide exercise prescription. Participation in the hands-on component (knee raise exercises) of the remote practical was optional. However, participation in the analysis and discussion of the collated data in the virtual lecture conducted after the graded exercise was made compulsory. Via this design, the remote practical (comprising both the graded exercise and post-exercise discussion) not only replicated the pedagogy of the original in-class practical, but also augmented the opportunity for students to volunteer and take part in the graded exercise component of the practical.

C. Instruments

After the virtual lecture, students completed an anonymous (no informed consent required) questionnaire containing 29 closed-ended and 2 open-ended items. The purpose of this self-report questionnaire was to evaluate students’ perceived effectiveness of the remote practical on their motivation and achievement of learning outcomes of the virtual lecture.

Student motivation was measured by the Lab Motivation Scale (Dohn et al., 2016) containing 21 closed-ended statements based on three aspects – student interest, effort and self-efficacy. Multiple instruments had previously been employed to assess dimensionality and reliability of the validated Lab Motivation Scale (Dohn et al., 2016). A set of six closed-ended items were employed to measure students’ perception on whether they had achieved the intended learning outcomes of the remote practical. Lastly, two closed-ended items were included to elicit a general satisfaction score from students regarding the remote practical and/or the virtual lecture. All the closed-ended statements in the questionnaire were scored on a 5-point Likert Scale, ranging from 5 (strongly agree) to 1 (strongly disagree).

Furthermore, there were two open-ended questions focusing on the effectiveness of the remote practical and the relevance of the remote practical to students’ daily lives. The first question was posed to uncover specific reasons supporting the analysis of the closed-ended items, while the second question aimed to encourage contextualisation of concepts learnt through the remote practical in students’ daily lives.

Finally, the CA results of students who participated in the hands-on component of the remote practical were compared with students who did not participate. To ensure a fair comparison, we took into consideration only the CA result from the questions corresponding to the content covered in the remote practical and virtual lecture. The questions taken into consideration made up 40% of the entire examination score.

D. Analysis

A mixed method approach was employed in analysing the questionnaire responses. An initial 66 questionnaire responses were collected but five incomplete responses were excluded, leaving a total of 61 responses that were included in the final analysis. Responses to the closed-ended items were coded accordingly to a 5-point Likert scale, ranging from strongly agree (5) to strongly disagree (1). Scores were reversed for statements phrased in a negative manner (items 9, 15 and 18). All closed-ended responses were presented in the form of mean ± standard deviation. As a measure of internal consistency, Cronbach’s α was used as an instrument and measured across all scales. Unpaired student’s t-tests were carried out to find out if differences between students who participated and did not participate in the hands-on exercise component were significant with significance level set at P < 0.05. All data analysis and statistical tests were performed using Microsoft Excel 2016.

Open-ended responses were analysed in a 3-part process: informal reviewing, open coding and thematic analysis. Firstly, all responses were informally reviewed to familiarise with general ideas and main themes were identified. Next, open coding was performed where each response was analysed in detail and coded to the most appropriate theme (Braun & Clarke, 2006). Finally, thematic analysis was carried out through ranking themes according to frequency and analysing the results (Braun & Clarke, 2006). The open-ended questions were made optional, and all responses collected were subsequently analysed.

III. RESULTS

Out of the 61 participants, 29 (48%) participated in the hands-on exercise component of the remote practical while 32 (52%) did not participate. Both groups attended the compulsory virtual lecture conducted after the remote practical, where the data collected from the remote practical was consolidated and discussed with the entire class.

Internal consistency was calculated using Cronbach’s α and the reliability coefficient was found to be 0.95 across all closed-ended items, indicating an excellent level of interrelatedness across the overall scale (Cronbach, 1951). Individual scales of learning outcomes and motivation were also subjected to the analyses of Cronbach’s α. The alpha coefficient value was calculated to be 0.86 for perceived achievement of learning outcomes and 0.94 for motivation (Tables 1 & 2). Motivation was further divided into three individual subscales assessing student interest, effort and self-efficacy, with the reliability coefficients returning 0.85, 0.88 and 0.88 respectively (Table 1). These reliability coefficients correlate strongly with those of Dohn et al. (2016), hence providing support for the internal consistency of the Lab Motivation Scale.  Data supporting these findings is openly available via Figshare at https://doi.org/10.6084/m9. figshare. 17170 964 (Low, 2021).

A. Remote Practical and Learning Outcomes

In general, students rated between “Agree” and “Strongly Agree” for perceived achievement of learning outcomes, with an overall mean score of 4.16 ± 0.68 on a 5-point Likert scale (Table 1). Students who participated in the hands-on component reported a mean score of 4.20 ± 0.54, which was similar to that rated by students who did not participate in the hands-on component (4.13 ± 0.68, P = 0.567; Table 1). For the first closed-ended statement: I have gained a stronger understanding of how heart rate responds to increasing exercise intensity, students who participated indicated a higher mean score of 4.52 ± 0.51 as compared to the lower mean score of 4.09 ± 0.59 (P = 0.004) for students who did not participate in the hands-on exercise component (Table 1).

Table 1. Students’ perceived achievement of learning outcomes in cardiovascular physiology

n = 61. Responses were coded from 1 (Strongly Disagree) to 5 (Strongly Agree). All means are shown with ±SD of the mean. **P < 0.01.

B. Remote Practical and Student Motivation

Students generally rated between “Agree” and “Somewhat Agree/Somewhat Disagree” for student motivation, with an overall mean score of 3.66 ± 0.71 (Table 2). Students who participated in the hands-on component reported a mean score of 3.70 ± 0.64, which was similar to that rated by students who did not participate in the hands-on component (3.62 ± 0.78, P = 0.187; Table 2). Students rated between “Agree” and “Somewhat Agree/Somewhat Disagree” regarding the three aspects of student motivation, with a score of 3.92 ± 0.69 for interest, 3.66 ± 0.75 for effort and 3.50 ± 0.68 for self-efficacy respectively (Table 2). For statement 23, students who participated indicated a higher mean score of 3.68 ± 0.61 as compared to the lower mean score of 3.29 ± 0.71 (P = 0.031) for students who did not participate in the hands-on component (Table 2).

Table 2. Students’ perceived motivation towards the remote practical

n = 61. Adapted from the Lab Motivation Scale (Dohn et al., 2016). Responses were coded from 1 (Strongly Disagree) to 5 (Strongly Agree). Scores were reversed for statements phrased in a negative manner. All means are shown with ± SD of the mean. **P < 0.05.

C. Qualitative Explanations on Perceived Effectiveness and Relevance of Remote Practical

The first open-ended item sought to investigate the reasons underlying the perceived effectiveness or ineffectiveness of the remote practical in enhancing students’ learning. Of the 37 responses, 34 (92%) felt that the remote practical was effective while 3 (8%) felt it was ineffective and of little to no added value to them (Table 3). Experiential learning and real-life data emerged as the most common themes cited across all responses (n = 12), followed by reinforc(ing) concepts taught in lecture (n = 10; Table 3). Experiential learning was reflected as the most common response among students who participated in the hands-on component (n = 10) in comparison to real-life data indicated by students who did not participate in the hands-on component (n = 8; Table 3).

The aim of the second open-ended item was to investigate the relevance and application of the remote practical to students’ daily lives. Of the 36 responses, 29 (81%) felt the remote practical was relevant while 7 (19%) felt that it was irrelevant to their daily lives (Table 3). Overall, the remote practical was found to be most relevant in improving current exercise routine (n = 12), followed by understanding one’s own fitness level (n = 10) and understanding the importance of exercise (n = 7; Table 3). This trend was similar for both students who participated and did not participate in the hands-on component of the remote practical (Table 3). 

Table 3. Themes identified from the open-ended responses, ranked by frequency

D. Remote Practical and Satisfaction Score

Students rated close to “Agree” for satisfaction, with an overall mean score of 3.95 ± 0.75 (Table 4). Those who participated in the hands-on component reported a mean score of 4.11 ± 0.70, which was similar to that rated by students who did not participate in the hands-on component (3.81 ± 0.77, P = 0.054; Table 4).

Table 4. Students’ satisfaction score

n = 61. Responses were coded from 1 (Strongly Disagree) to 5 (Strongly Agree). All means are shown with ± SD of the mean. **P < 0.05.

E. Remote Practical and Academic Performance

Out of the 49 students who consented to the use of their CA results for this research study, 30 (61%) participated in the hands-on component of the remote practical while 19 (39%) did not (Table 5). Only the questions corresponding to the content covered in the remote practical and relevant virtual lecture were taken into consideration for this study. The overall mean mark was 7.3 ± 1.64 out of 10 (Table 5). Students who participated in the hands-on component (7.3 ± 1.84) and did not participate in the hands-on component (7.3 ± 1.32) exhibited similar mean marks as well (P = 0.940; Table 5).

Table 5. Students’ CA results

n = 49. CA scores are shown as mean ± SD, with *P < 0.05 considered significant.

IV. DISCUSSION

This study sought to evaluate if remote physiology practicals could be viable alternatives to traditional face-to-face practicals, especially during emergency remote teaching in a pandemic. Our results demonstrated that the students who participated in the remote practical perceived that they could achieve the learning outcomes in cardiovascular and exercise physiology teaching with reasonable satisfaction, regardless of whether they participated in the hands-on component of the remote practical. However, students who had actively participated in the hands-on component (graded exercise) reported that the remote practical had particularly benefitted them in better achieving certain learning outcomes as compared to their classmates who did not participate in the graded exercise. Specifically, students indicated that their participation in the graded exercise allowed them to have a better grasp of the concepts concerning heart rate response to increasing exercise intensity. This finding was not unexpected as the remote graded exercise was specifically designed to provide experiential learning opportunities to better comprehend the concepts underlying this particular learning outcome.

Perceived scores for the achievement of learning outcomes were otherwise similar between the participated and non-participated group. This could be attributed to the fact that the remote practical was used as a complement to the virtual lecture, wherein the interpretation and analysis of data collected from the optional graded exercise was discussed with the whole class during the compulsory virtual lecture. The perception scores of learning outcome achievement were well supported by the students’ academic performance as all of them shared similar mean CA marks regardless of their participation in the remote practical. This similarity is especially prominent as the CA questions were set based on the principle of constructive alignment (Biggs, 1996; Bloom, 1956; Stamov Roßnagel et al., 2020).

Interestingly, open-ended responses revealed “experiential learning” as the key reason supporting the efficacy of the remote practical in students who participated in the graded exercise while “having real-life data which reinforces concepts taught in lecture” were key reasons indicated by students who did not participate in the hands-on component of the remote practical. These findings are in line with studies recommending experiential learning as one of the seven “principles of good practice” to achieve excellence in higher education (Chickering & Gamson, 2006). This is accomplished by generating real-life data to allow students to draw the link between theoretical content and practical applications, before applying it to analyse real-life situations in view of course material (Lewis & Williams, 1994). This suggests that the remote practical is able to foster environments which could encourage hands-on learning and real-time data generation to enhance student learning, even if not conducted in a traditional laboratory setting.

Overall, students were satisfied with the remote practical and/or virtual lecture, with those who participated in the hands-on component generally being more satisfied. Even though the remote practical was not compulsory, those who took part in the hands-on component generally did not view it as an additional academic burden and instead would recommend it to future batches of students. This reinforces the potential of such remote practicals in helping students to achieve learning outcomes without imposing unnecessary pressure on them. 

With motivation being a strong indicator of self-directed learning and academic achievement (Cortright et al., 2013), it is crucial for educators to assess and understand the importance of motivating students. In fact, educators play a critical role in determining the motivation levels of their students through the nature of their classes and assignments (Cortright et al., 2013). Specifically, Dohn et al. (2016) states that students’ motivation could be negatively impacted by limited equipment or restricted time for practicals. Majority of students face similar limitations for a graded exercise test carried out in a traditional face-to-face practical. Typically, only one volunteer would carry out the actual exercise experimental protocol due to equipment and time constraints, while other students would passively watch and learn from the data collected. The novel remote practical proposed in this study could potentially overcome these limitations as students are able to personally experience the hands-on exercise component within their own spaces at home and at their own convenience, thereby possibly enhancing their motivation levels.

Our results indicated that overall motivation towards the remote practical and/or the virtual lecture ranged from 3 to 5, corresponding to “somewhat agree” and “strongly agree”, with a mean value of 3.62 ± 0.78. This is comparable to the motivation scores previously reported by Dohn et al. (2016) for in-class biomedical laboratory classes. The positive motivation score could be explained by the fact that majority of students (81%) could see the relevance (Table 3) of the remote practical in their daily lives. Learning activities which guide students towards finding ‘personal meaning and value’ in the educational content is known to positively influence their motivation levels (Cortright et al., 2013). By providing opportunities for students to reflect on, find meaning and draw relevance to their personal lives, such remote practicals can potentially address common limitations of traditional practicals and boost student motivation and learning.

Delving further into the three aspects of student motivation – interest, effort and self-efficacy, students rated the highest scores for interest, followed by effort and lastly self-efficacy. The score for effort placed into the practical could have been understandably affected due to the non-compulsory nature of the graded exercise. The exercise component of the practical could not be made compulsory as not all students were medically/physically fit enough to undergo a graded exercise test. Nonetheless, the similarity in perceived learning outcomes and academic results between students who did and did not participate in the graded exercise suggests that the follow up analysis and peer-based discussion of the tabulated data involving the entire class was sufficient to bridge the learning gap between the two groups of students. Overall, the favourable perceived learning outcomes (ranging from “agree” to “strongly agree”) and academic scores (corresponding to a grade of “A-” to “A”) reinforces the value of the remote practical as a teaching strategy to promote learning in exercise physiology, regardless of the students’ ability or interest to participate in strenuous physical activity. However, whether the remote practical is more effective than a conventional face-to-face practical or no practical at all remains an interesting question which necessitates future research as this cannot be addressed given the limitations of our current study design.  

Notably, self-efficacy scores were rated the lowest amongst the three aspects of motivation. This could be due to the fact that students are not closely supervised during a remote practical, unlike face-to-face practicals. Without the physical presence and continuous guidance of an instructor, students could have faced uncertainty as to whether instructions were properly executed. Thus, strategies to enhance pre-practical instructions using asynchronous video instructions or the incorporation of remote supervision methods may aid to further enhance the effectiveness of the remote practical. Interestingly, participation in the hands-on component of the remote practical appeared to have nonetheless enhanced the confidence of students in explaining the procedures of the practical to their peers (Table 2). This finding is of particular importance, as the ability to teach and explain is an indication of higher order learning corresponding to the second and third levels of the Bloom’s taxonomy (Bloom, 1956).

A. Limitations

Our study sought to evaluate the effectiveness of a remote exercise physiology practical in promoting student motivation and learning in a cohort of Life Sciences undergraduates. However, the current study design does not permit immediate comparison with conventional face-to-face practicals as students could not be randomly allocated into different comparison groups (remote or face-to-face) owing to pandemic restrictions and ethical reasons. Also, we could only investigate the effects of practical participation on the effectiveness of the remote practical in enhancing student motivation and learning outcomes achievement using a quasi-experimental approach. This is so, for we were unable to randomly allocate students into two comparison groups given that not all students were medically/physically fit enough to undergo a graded exercise test for the hands-on component of the remote practical. Based on this study design, some degree of self-selection bias could have been present as physically active students who had volunteered to take part in the exercise component of the remote practical could have seen greater relevance to their daily lives and could have been more intrinsically motivated to partake and learn from the practical session. In spite of that, the overall student motivation score appeared comparable between the two groups of students who participated and did not participate in the exercise component of the practical, suggesting that the degree of self-selection bias may not be of significant concern in the present study.

V. CONCLUSION

Overall, students reported that experiential learning and real-life data were the main reasons supporting the effectiveness of the remote practical. With experiential learning and real-life data as key components of traditional practicals (Dohn et al., 2016; Randall & Burkholder, 1990), the present study demonstrates the potential of remote practicals as viable and innovative alternatives for face-to-face practicals in exercise physiology teaching. In cases of sudden shifts to emergency remote education, such alternatives offer the possibility of incorporating experiential learning even during social isolation. 

Notes on Contributors

Tan conducted the study, analysed and interpreted the data, and drafted the manuscript. I.C.C. Low was involved in experimental conception and design, as well as critically reviewed and revised the manuscript. All authors have read and approved the final version of the manuscript.

Ethical Approval

All students were provided with a participant information sheet containing the purpose and details of the research study. The questionnaire was made completely anonymous. Informed consent was obtained from students for use of their CA results only after the release and confirmation of their results. Circulation of research materials was done virtually and students were never approached directly for recruitment. The study was approved by the National University of Singapore – Institutional Review Board (NUS-IRB) with study code NUS-IRB-2020-631.

Data Availability

Data supporting these findings is openly available via Figshare at DOI: https://doi.org/10.6084/m9.figshare. 17170964.

Acknowledgements

We greatly appreciate the study participants who have spent their time and effort to complete the questionnaires and have provided consent in the use of their results for research purposes.

Funding

There was no funding support accorded for this study.

Declaration of Interest

No potential conflict of interest was reported by the authors.

References

Ali, W. (2020). Online and remote learning in higher education institutes: A necessity in light of COVID-19 pandemic. Higher Education Studies, 10(3), 16-25. https://doi.org/10.5539/hes.v10n3p16

Biggs, J. (1996). Enhancing teaching through constructive alignment. Higher Education, 32(3), 347-364. https://doi.org/10.1007/BF00138871

Bloom, B. S. (1956). Taxonomy of educational objectives. Vol. 1: Cognitive domain. David McKay.

Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77-101. https://doi.org/10.1191/1478088706qp063oa

Brinson, J. R. (2015). Learning outcome achievement in non-traditional (virtual and remote) versus traditional (hands-on) laboratories: A review of the empirical research. Computers & Education, 87, 218-237. https://doi.org/10.1016/j.compedu.2015.07.003

Bruce, R. H. (1988). Enhancing student motivation and learning: Experiences with a simple simulation. Higher Education Research & Development, 7(2), 103-110. https://doi.org/10.1080/0729436880070201

Chickering, A., & Gamson, Z. (2006). Seven principles of good practice in undergraduate education. New Directions for Teaching and Learning, 1991, 63-69. https://doi.org/10.1002/tl.37219914708

Colthorpe, K., & Ainscough, L. (2021). Do-it-yourself physiology labs: Can hands-on laboratory classes be effectively replicated online? Advances in Physiology Education, 45(1), 95-102. https://doi.org/10.1152/advan.00205.2020

Cortright, R. N., Lujan, H. L., Blumberg, A. J., Cox, J. H., & DiCarlo, S. E. (2013). Higher levels of intrinsic motivation are related to higher levels of class performance for male but not female students. Advances in Physiology Education, 37(3), 227-232. https://doi.org/10.1152/advan.00018.2013

Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297-334. https://doi.org/10.1007/BF02310555

Dohn, N. B., Fago, A., Overgaard, J., Madsen, P. T., & Malte, H. (2016). Students’ motivation toward laboratory work in physiology teaching. Advances in Physiology Education, 40(3), 313-318. https://doi.org/10.1152/advan.00029.2016

Hofstein, A., & Lunetta, V. N. (2004). The laboratory in science education: Foundations for the twenty‐first century. Science education, 88(1), 28-54. https://doi.org/10.1002/sce.10106

Lellis-Santos, C., & Abdulkader, F. (2020). Smartphone-assisted experimentation as a didactic strategy to maintain practical lessons in remote education: Alternatives for physiology education during the COVID-19 pandemic. Advances in Physiology Education, 44(4), 579-586. https://doi.org/10.1152/advan.00066.2020

Lewis, L. H., & Williams, C. J. (1994). Experiential learning: Past and present. New Directions for Adult and Continuing Education, 1994(62), 5-16. https://doi.org/10.1002/ace.36719946203

Low, I. C. C. (2021). Remote physiology practical: Viable alternative to in-person practical in health sciences education? [Data set]. Figshare. https://doi.org/10.6084/m9.figshare.17170964

Neves, B. S., Altermann, C., Goncalves, R., Lara, M. V., & Mello-Carpes, P. B. (2017). Home-based vs. laboratory-based practical activities in the learning of human physiology: The perception of students. Advances in Physiology Education, 41(1), 89-93. https://doi.org/10.1152/advan.00018.2016

Randall, W. C., & Burkholder, T. (1990). Hands-on laboratory experience in teaching-learning physiology. Advances in Physiology Education, 259(6), S4-S7. https://doi.org/10.1152/advances.1990.259.6.s4

Ray, S., & Srivastava, S. (2020). Virtualization of science education: A lesson from the COVID-19 pandemic. Journal of proteins and proteomics, 11, 77-80. https://doi.org/10.1007%2Fs42485-020-00038-7

Stamov Roßnagel, C., Fitzallen, N., & Lo Baido, K. (2020). Constructive alignment and the learning experience: Relationships with student motivation and perceived learning demands. Higher Education Research & Development, 40(4), 838-851. https://doi.org/10.1080/07294360.2020.1787956

Vasiliadou, R. (2020). Virtual laboratories during coronavirus (COVID‐19) pandemic. Biochemistry and Molecular Biology Education, 48(5), 482-483. https://doi.org/10.1002/bmb.21407

*Ivan Cherh Chiet LOW
Department of Physiology,
Yong Loo Lin School of Medicine
Block MD9, 2 Medical Drive Level 4
Singapore 117593
Email: phsilcc@nus.edu.sg

Submitted: 6 April 2021
Accepted: 2 December 2021
Published online: 5 April, TAPS 2022, 7(2), 17-26
https://doi.org/10.29060/TAPS.2022-7-2/OA2510

Nicola Ngiam^1,2, Kristy X Fu^1,3 & Jacqueline SM Ong^1,3

¹Khoo Teck Puat- National University Children’s Medical Institute, National University Health System, Singapore; ²Centre for Healthcare Simulation, Yong Loo Lin School of Medicine, National University of Singapore, Singapore; ³Department of Paediatrics, Yong Loo Lin School of Medicine, National University of Singapore, Singapore

Abstract

Introduction: Personal protection in aerosol-generating procedures is an important skill to safely deliver care to patients in the COVID-19 pandemic. This aim of this study was to evaluate the impact of “just-in-time” simulation training for airway management in a suspected COVID-19 patient.

Methods: This was a prospective mixed-method cohort study in a tertiary paediatric department. A mandatory “just-in-time” simulation training session for intubation of a suspected COVID-19 patient was conducted. Pre- and post-simulation questionnaires were administered. Participants were invited to attend focus group interviews to further delineate their experience. Quantitative and qualitative methods were employed to analyse the data.

Results: Thirty-three participants, including doctors, nurses and respiratory therapists attended the training. Self-confidence in intubation, managing and leading a resuscitation team and dealing with problems with intubation significantly improved. Simulation was valued for the experiential learning as well as for increasing confidence and awareness. Process improvement suggestions from both participants and trainers were raised.  There was a small signal of skill translation to real life scenarios.

Conclusion: Simulation-based training is a useful tool for infectious disease outbreak preparedness. Further research will need to be done to determine the impact on actual clinical practice in pandemics.

Keywords:           Simulation, COVID-19, Pandemic Preparedness, Training, Intubation

Practice Highlights

• The authors report their experience with “just-in-time” in situ simulation training for emergency preparedness in the face of an infectious disease outbreak.
• Simulation training was well received and improved the confidence as well as awareness of frontline staff in managing intubation and resuscitation in a suspected COVID-19 paediatric patient.
• Process improvement suggestions from participants and trainers was a useful by-product of the simulation training activity.

I. INTRODUCTION

Since the World Health Organization (WHO) declared the Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) outbreak a public health emergency of international concern on 30 January 2020 (World Health Organization, 2020), the COVID-19 pandemic has now affected millions of people worldwide, with a wide range of case fatality rates amongst the 210 countries and territories affected (The Centre for Evidence-Based Medicine Nuffield Department of Primary Care Health Sciences University of Oxford, 2020). In Wuhan, China, one of the first epicentres of this international pandemic, out of 44672 confirmed cases of COVID-19, 1716 were healthcare workers (HCWs) (Wu & McGoogan, 2020). Of the confirmed cases among HCWs, 14.6% were classified as severe or critical, and 5 deaths were observed (The Novel Coronavirus Pneumonia Emergency Response Epidemiology Team, 2020). Early reports suggest that modes of transmission of SARS-CoV-2 include droplet and contact (via the fecal-oral route and fomites); airborne transmission may occur during aerosol-generating procedures (World Health Organization, 2020).

During the 2003 SARS-CoV outbreak, despite the presence of existing safety protocols, up to half of the SARS-CoV cases in some centers were HCWs as a result of transmission within hospital units (Booth et al., 2003). Critical care and emergency care providers are often involved in high-risk aerosol-generating procedures such as oronasal suctioning, bag-valve-mask ventilation, non-invasive positive pressure ventilation, and endotracheal intubation in patients with COVID-19 with respiratory failure, and so must be prepared (Caputo et al., 2006; Wax & Christian, 2020; Zuo et al., 2020). The challenge for providers would be to provide high-quality and timely care to infected patients, without compromising their own safety. Apart from having adequate supplies of personal protective equipment (PPE), a safe environment for HCWs requires the provision of up-to-date information regarding the disease, smooth dissemination of protocols, and easy accessibility to portals reinforcing education and training in infection control procedures. Lau et al. (2004) reported that HCWs who underwent more than 2 hours of training were far less likely to be infected with SARS-CoV during the 2003 SARS-CoV outbreak in Hong Kong. In addition to PPE training, we felt that a simulated airway management scenario in a suspected COVID-19 patient was a vital component of training in order for staff to experience the profound challenges of performing high risk aerosol-generating procedures while in PPE and in isolation facilities. During the recent 2014 to 2016 West Africa Ebola outbreak, Grillet et al. (2015) with the use of simulation, found that commonly performed procedures in the intensive care unit becomes more complicated, more stressful and less comfortable in appropriate PPE. We were fortunate to find a window of opportunity close to the beginning of the COVID-19 outbreak in Singapore to prepare our healthcare teams using “just-in-time” in-situ simulation. The main objective of this study was to evaluate the impact of this training on our resuscitation teams when managing a respiratory emergency of a suspected COVID-19 pediatric patient.

II. METHODS

This was a prospective mixed-method cohort study in a paediatric department of a tertiary university hospital. Residents were put through a mandatory “just-in-time” simulation training session to prepare them for intubation of a suspected COVID-19 patient. The doctors involved were senior paediatric residents who regularly manage emergencies on call. Nurses and respiratory therapists were rostered to participate whenever available on shift. Participants worked in teams of 5 to 6 to manage a simulation scenario involving a patient with bronchiolitis who was suspected to have COVID-19. Each team comprised of participants from each of the healthcare professional groups listed above.  Actual personal protective equipment (PPE) including powered air-purifying respirators (PAPR) were used in the simulation. Participants received training in the use of PPE and PAPR prior to the simulation session. The simulation was conducted in-situ in the paediatric intensive care unit (PICU) using a SimBaby (Laerdal). The scenario used can be found in the appendix. Each group participated in one scenario. Each scenario lasted 30 to 40 minutes. The end point was successful intubation of the patient. There were 2 instructors (paediatric intensive care clinicians) present, one in the anteroom and one in the patient’s room. Debriefing was conducted as a group by both instructors immediately after every simulation. Each debriefing session lasted 30 to 45 minutes and focused on technical and non-technical skills.

Anonymous pre-simulation and post-simulation questionnaires were administered. Questions focused on confidence levels with managing intubation in a COVID-19 suspect patient, using the PAPR, and anxiety levels. Responses were rated on a Likert scale of 1 to 5. Quantitative data was analysed using Microsoft EXCEL, 2016. Pre and post responses on the Likert scale were analysed using the paired t-test.

After the simulation and debriefing, participants were invited via email to attend focus group interviews to gain better insight into their responses and attitudes towards the simulation sessions. The format and logistics of the interviews were included in a participant information sheet in the email. Participation was voluntary. We aimed for maximal representation from the various groups of healthcare professionals who participated in the simulation. A semi-structured interview was conducted by 2 researchers (NN, JO) in groups of 3 to 5 and interviews were audio-recorded. Two focus group interviews were conducted. Participants were asked questions regarding how they felt, what they learnt and what the benefits of the simulation experience were. Each interview lasted 30 to 40 minutes and were conducted in a quiet room in the PICU. Audio recordings were transcribed verbatim and anonymised at the point of transcription. Participants were only identified by role in the transcript.

Thematic analysis of the transcripts was performed by the 2 interviewers using an inductive approach. Each interviewer coded the data independently, after which both interviewers discussed the codes and generated common themes together. Data was reviewed for commonality in responses, degree of specificity (detailed explanation), and extensiveness (number of different people who had similar responses). Where there was disagreement, review of the data and existing literature was used for resolution. Descriptive summaries were developed for each theme, and participant quotes provided further evidence for interpretation and recommendations made.

This study was approved by the institutional review board (National Healthcare Group, Domain Specific Review Board, NHG DSRB Ref. 2020/00234) and waiver of consent was obtained.

III. RESULTS

A. Pre- and Post-Simulation Questionnaire Responses

A total of 33 participants took part in the training and completed the pre- and post-simulation questionnaires. There were 19 doctors, 12 nurses and 2 respiratory therapists. Confidence, as assessed by the questionnaire, increased significantly after the simulation in the areas of intubation, use of the PAPR, airway management in a COVID-suspect patient, leading a team and dealing with problems that arise during resuscitation (p < 0.05). Regarding anxiety over intubating a COVID-suspect patient, most participants were less anxious after the simulation (p < 0.05). Interestingly, on looking more closely at the responses, 9% (n=3) of participants were more anxious after the simulation than before they started. 42% (n=14) reported no change in their level of anxiety. Table 1 shows the mean scores of the questions asked in the pre- and post-simulation questionnaire.

Table 1: Pre- and post-simulation responses

After the simulation training, 96% of participants agreed or strongly agreed that they were more aware of who to call for help and how to communicate effectively when wearing PPE or PAPR while in an isolation room and 93% agreed or strongly agreed that they were more able to anticipate problems. All participants agreed that simulation training was useful in preparing to deal with a similar situation while on call and in fact, 23% disagreed or strongly disagreed that 1 simulation session was sufficient. Figure 1 illustrates the perceived effectiveness of simulation training by participants.

Figure 1: Post-simulation responses on effectiveness of simulation training

B. Focus Group Interviews

Eight participants agreed to focus group interviews, 4 doctors, 3 nurses and 1 respiratory therapist. Comments from the focus group interviews helped to further delineate the benefits and lessons learnt from this simulation exercise. Key benefits were grouped into the following themes:

1) Experiential learning: Participants commented that getting to put theory into practice during the simulation was key to understanding what to expect. Being able to practice before an actual patient encounter helped them to be more prepared. Even simple things like setting an intravenous cannula could not be accomplished with ease. Participants stated that they had to rethink the way that things needed to be done as how they were done previously would not work in this situation. This led to a paradigm shift in the clinical processes and also the application of knowledge. For example, a lesson learnt was that there is a lower threshold for intubation in COVID-19 patients. In the words of a participant about his experience:

“But the fact that you actually go through it, firstly you realize that it takes 3 times the length of the duration of what you would do, and the difficulties in communicating with the people around. So I thought that was the most…an awakening…. the most beneficial part.”

Participants appreciated that this was a complex scenario that was high stakes for the healthcare team as well as for the patient. Getting to practice this, even though it was likely to be an infrequent occurrence, helped with familiarity with protocols and equipment used. Participants also felt that more people should attend this training and that they needed more practice.

2) Increasing confidence: Participants felt that the simulation training was a timely intervention in light of the developing COVID-19 situation. This experience took away the “first-time” feeling and made the participants feel more comfortable with the protective equipment. This took away some of the anxiety and fear about their own safety and the safety of their COVID-19 patients. One participant stated that more practice in simulations may reduce the resistance to wearing the PAPR, which may in itself pose a safety issue if PAPRs are not used when required. They felt that this experience made them a more valuable member to the resuscitation team.

3) Increasing awareness: Participants appreciated the feedback and debriefing that came after the simulation. Experiencing the simulation made taught them to anticipate problems, plan ahead and to prioritize as resources were less accessible than usual. They developed insight into a possible “unconscious incompetence” as they expressed that if they had not gone through the simulation, they would not have known what they did not know and would be inappropriately confident in a real situation. Exploring resource management and considering aspects of waste reduction were thought to be beneficial for future encounters with similar situations. Some of our participants said:

“It really gives you…. the best kind of idea of what to expect in a real life scenario.”

“I would feel quite…. in a sense, bad for the patient that I was doing it for the first time on that patient. So I do feel that it is a responsible thing to do, as healthcare providers that we… that this was actually undertaken.”

“If we had not had this, I probably won’t even have that fear of how terrible it could have been……… But if I had not known, I would still be confident not knowing what I’m expecting.”

Some participants mentioned developing more empathy for colleagues caring for COVID-19 patients through this experience. However, one participant found herself more frustrated after going through the simulation as the experience of managing a resuscitation in this situation was worse than she had thought it would be.

4) Process improvement: In the focus group discussions, participants mentioned the need for process improvement that was discovered while encountering difficulties during the simulation scenario. Communication challenges were brought up multiple times and participants suggested making laminated signs for communication with staff outside the room when assistance was required, having pen, paper and drug labels in the room, as well as using intercoms or walkie-talkies for more efficient communication.

5) Translation: A couple of participants encountered a real subsequent clinical situation which required them to apply skills that they had learnt during the simulation training. The fact that they had gone through the training made them less anxious and more able to take control of the emergency situation. The respiratory therapist in the group expressed more confidence in the nurses that had gone through the simulation training when compared to those who had not when managing a COVID-19 patient. Participants also commented that the skills learnt could be translated to a different institution in the future and perhaps also to a different infective agent. Participants expressed the following:

“And so it would make you, I think… more valuable as a team member in providing care for a COVD patient or any other sort of respiratory pathogen X anywhere.”

“Because I observe that those nurses that attended the simulation, in the actual scenario or in the actual patient handling, they are more confident and competent in doing their PPE and flow of the sequence inside.”

“Because for me, for example, I actually had to initiate and help…coordinate initiation of ECMO for a COVID suspect patient and so I think if not for that…that mock… I would probably have been much more nervous and unsure.”

Key challenges encountered during the simulation were related to the simulation scenario and to the process of simulation itself. The most emphasised challenges encountered during the simulation were cognitive overload and communication barriers. Not only did the team have to deal with a deteriorating patient who was suspected to have COVID-19, they also had to deal with infection control protocols and the inherent challenges that it posed to the resuscitation process. As many tasks needed to be coordinated in a stressful situation, prioritisation was key. Communication barriers came in the form of restricted movement in and out of the isolation room, not being able to use their mobile devices, hearing impairment with the PAPR on and not being able to be heard clearly while wearing the N95 mask. Other challenges raised relating to the scenario were unfamiliarity with the safety equipment as it was not used frequently and having to wait a long time for orders to be carried out. An interesting challenge brought up was a tension between the patient’s safety or well-being and healthcare worker’s own safety. Healthcare professionals frequently put their patients first and in this unique situation, the need for donning personal protective equipment before entering the patient’s environment does not allow for as rapid a response to a deteriorating patient as they are used to:

“I feel like I need to go in as soon as possible but donning the PAPR takes some…. more time than I thought.”

Logistical challenges in planning for this simulation training included the availability and timely attendance of all frontline healthcare workers. With the developing crisis, healthcare professionals were kept busy with their clinical responsibilities, therefore this simulation training was seen as an unwelcome interruption. There was also a perceived resistance to participating by the nurses although the reasons are unclear.

Participants also commented on the design of this simulation training. Prior training in the process of intubation and use of PPE and PAPR were deemed necessary to optimize the benefits of this exercise. Participants appreciated that there were 2 trainers, 1 in the ante-room and 1 in the patient’s room, who were observing different processes and able to give valuable feedback during the debriefing. They also appreciated that the simulation exercise was done in-situ, and therefore was realistic in design.

Suggestions for improvement included providing a variety of clinical scenarios, training junior physicians so that the whole medical team is familiar with the processes, mimicking the typical manpower available on-call in the simulation exercise, and increasing the complexity of the scenarios to address resource allocation issues faced in a pandemic.

IV. DISCUSSION

With the development of the COVID-19 outbreak and patients requiring intensive care, our unit thought it would be imperative to train our frontline staff to be prepared to manage airway emergencies and to be able to resuscitate infected patients. Simulation provides a safe environment for mistakes to be made without compromising patient safety (Ziv et al., 2000). It also provides a platform for deliberate practice (Ericsson, 2004) and not allow for a chance clinical encounter to develop expertise in an area. Simulation has also been utilised in high consequence infectious outbreak training, resulting in improved self-efficacy, reduced anxiety and improved inter-professional teamwork (Marrs et al., 2019; O’Keeffe et al., 2016). As the consequences for patient and individual safety were high in the COVID-19 outbreak, we felt it was prudent to use simulation-based education as a platform for upskilling our staff.

Despite training in the use of personal protection equipment (PPE), including the powered air-purifying respirator (PAPR), Watson et al found that there was an inadequate adherence to the use of PPE and resuscitation guidelines in simulated cardiac arrest in paediatric influenza (H1N1) patients (Watson et al., 2011). Intensive care procedures have been found to be more difficult, stressful and uncomfortable when performed in PPE (Grillet et al., 2015). Simulation training has also been able to detect breaches in infection control procedures (Seet et al., 2009) and potentially improve compliance with infection prevention measures (Tan et al., 2021).

For the above reasons, healthcare professionals who worked in the paediatric intensive care unit were put through a mandatory simulation-based training session on airway management of a deteriorating patient who was suspected to have COVID-19 infection. An in-situ model was chosen as it could be used to evaluate system competence and identify latent conditions that predispose to medical error (Patterson et al., 2013). In this study, in situ simulation provided a means to identify areas for process improvement and knowledge gaps. It provided the ability to test the actual clinical care system, including equipment, processes, and staff response. This form of “just-in-time” training takes place in close proximity to the clinical encounter in a focused concise manner (Itoh et al., 2019). This provides the participants with contextually relevant hands-on experience in dealing with an impending low-frequency event before it actually happens. This has shown to improve confidence levels and clinical skills (Sam et al., 2012).

As expected, there was improvement in self-efficacy as shown in the improvement in pre- and post-simulation responses for all questions relating to management of the patient and clinical team. This has been shown in many previous studies using simulation as a methodology (McLaughlin et al., 2019; Secheresse et al., 2020). Evidence does suggest that clinicians have a limited ability for self-assessment of competence (Davis et al., 2006) and self-assessment. From the Kirkpatrick levels of evaluation, this would be a level 2 evaluation of knowledge, skills and attitudes (Kirkpatrick & Kirkpatrick, 2006). From the qualitative data, the benefits that were emphasised where related to experiential learning and increasing confidence as well as awareness. Kolb’s framework of experiential learning includes the phases of concrete experience and reflective observation (Kolb, 1984). These phases were evident in the simulation experience. Participants reflected that theoretical knowledge does not guarantee perfect execution in real life. The cognitive load of managing a high consequence, low incidence event along with the concomitant risk of exposure to a highly infectious agent makes clinical decision making harder. Although the participants underwent prior PPE training, they appreciated the opportunity to put it all into practice.

There was a definite signal for increased self-efficacy and confidence. This is seen in the literature on simulation-based healthcare education across disciplines (Bragard et al., 2018; Cohen et al., 2013; Fisher et al., 2011; Fouilloux et al., 2019). A study on influenza pandemic preparedness showed that simulation improved staff confidence and also demonstrated that many tasks and procedures took longer to complete with the implementation of guidelines (Phin et al., 2009). This was similarly evident in our participants as they became more aware of the challenges and the additional time needed for most resuscitative actions due to infection control requirements. Going through the simulated scenario and debriefing made them feel more prepared for an actual emergency. In addition, participants highlighted that the simulation experience alleviated anxiety as it successfully removed the “first-time” feeling for them, and felt that it was the responsible thing to do for healthcare workers in the midst a public health crisis. Lessons learnt by the participants included anticipatory planning, prioritisation and resource management. We had a small signal that the skills learnt translated to real life as one participant had a clinical encounter that required initiating extracorporeal life support in a patient with suspected COVID-19. She reported feeling relieved that she had the simulation experience before the real-life clinical encounter, and felt she was less nervous because of her prior stimulated experience. A respiratory therapist also observed that the nurses who he worked with in the PICU who had gone through the simulation seemed more confident and more aware of the necessary processes when intubating suspected COVID-19 patients.

An interesting phenomenon was the signal that there was an increase in anxiety levels after participants went through the simulation as seen in the pre- and post-simulation response. A possible explanation could be what participants brought up in the interviews about the simulation and debriefing revealing their “unconscious incompetence”. This made them more aware of the complexity and so increased their anxiety with dealing with intubation in a COVID-19 patient. Other studies in the literature generally report a reduction in anxiety after simulation (Bragard et al., 2019; Shrestha et al., 2019, 2020). This may not be a negative impact arising from the simulation experience itself as it may reflect increased awareness in an uncommon, yet stressful and complex clinical situation for our healthcare workers. Anxiety levels in frontline healthcare workers in a pandemic may also be due to other factors such as being at higher risk of exposure to COVID-19 at work and the possibility of bringing the infection home to their family (Holmes et al., 2020; Shanafelt et al., 2020).

Another benefit derived from running this simulation-based training is the process improvement suggestions made by the participants. This is a known benefit of simulation (Paige et al., 2018) and was utilised during the 2003 SARS-CoV outbreak (Abrahamson et al., 2005). The main challenge faced by participants during this scenario was the communication barrier which arose from not being able to communicate with personnel outside the room and the presence of the PPE and PAPR physically obstructing hearing and clarity of speech. Also, restriction of movement in and out of the patient’s room experienced during simulation highlighted the need to rethink resuscitation practices when dealing with COIVD patients. These systemic issues surfaced allowed us to brainstorm for practical solutions as a unit, and some have been implemented in our PICU. We have trialled the use of infant monitors as a 2-way communication device. Pre-packed resuscitation drug kits containing intravenous adrenaline and intravenous atropine as well as pre-packed intravenous cannulation disposables have been put in every isolation room so that these would be easily accessible in an emergency. As suggested by the participants, we have also extended the simulation training to include all junior doctors in the department, more nurses and all respiratory therapists in the PICU to facilitate better teamwork. We are also exploring the provision of a dedicated COVID-19 crash cart to minimise waste and prevent cross-contamination.

There are several limitations to this study. Firstly, the sample size is small and selection bias is possible due to the study design. Next, focus group interviews were conducted with a small subset of the participants who agreed to participate, and therefore our findings may not have been representative of the entire cohort. However, we are reassured by the fact that each healthcare professional group that took part in the simulation training was represented in the focus groups. As this simulation exercise was designed to be “just in time” training and we were limited by the urgency of the situation as well as the need to train as many staff in the shortest amount of time, we could only conduct a one-time simulation scenario specific to the highest risk procedure in resuscitating a suspected COVID-19 patient. We were therefore not able to assess the impact of this training on the subsequent performance of the participants either in another simulated scenario or a real life one.

V. CONCLUSION

Simulation-based training is a useful tool for infectious disease outbreak preparedness for the healthcare team. It improves confidence and awareness around managing emergencies while maintaining personal protection through deliberate practice in a safe environment. It also provides an opportunity for process improvement in a new and evolving pandemic situation. It was well-received by the participants and perhaps more sessions are needed for adequate practice. This is limited by the resource-intensive nature of in-situ simulation and the heavy clinical workload at this time of crisis. Further research will need to be done to determine if simulation-based training has a significant impact on actual clinical practice

Notes on Contributors

Nicola Ngiam conceptualised and designed the study, collected data, analysed and interpreted it, drafted the manuscript and approved the final version submitted.

Kristy X Fu did the background work and literature review, assisted in drafting the manuscript and critically revised the manuscript for important intellectual content and approved the final version submitted.

Jacqueline SM Ong designed the study, collected data, interpreted it, critically revised the manuscript for important intellectual content and approved the final version submitted.

Ethical Approval

Ethical approval has been granted by National Healthcare Group Domain Specific Review Board (Ref: NHG DSRB Ref. 2020/00234)

Data Availability

Transcripts from the interviews are confidential and the authors do not have consent to upload onto a repository. Data from questionnaires can be made available on request.

Acknowledgement

We would like to thank Dr Dimple Rajgor for helping with formatting, and submission of the manuscript for publication.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Declaration of Interest

The Authors declare that there is no conflict of interest.

References

Abrahamson, S. D., Canzian, S., & Brunet, F. (2005). Using simulation for training and to change protocol during the outbreak of severe acute respiratory syndrome. Critical Care, 10(1), R3. https://doi.org/10.1186/cc3916

Booth, C. M., Matukas, L. M., Tomlinson, G. A., Rachlis, A. R., Rose, D. B., Dwosh, H. A., Walmsley, S. L., Mazzulli, T., Avendano, M.,  Derkach, P., Ephtimios, I. E., Kitai, I., Mederski, B. D., Shadowitz, S. B., Gold, W. L., Hawryluck, L. A., Rea, E., Chenkin, J. S., Cescon, D. W., Poutenan, S.M. & Detsky, A. S. (2003). Clinical features and short-term outcomes of 144 patients with SARS in the greater Toronto area. JAMA, 289(21), 2801-2809. https://doi.org/10.1001/jama.289.21.JOC30885

Bragard, I., Farhat, N., Seghaye, M.-C., Karam, O., Neuschwander, A., Shayan, Y., & Schumacher, K. (2019). Effectiveness of a high-fidelity simulation-based training program in managing cardiac arrhythmias in children: A randomized pilot study. Pediatric Emergency Care, 35(6), 412-418. https://doi.org/10.1097/PEC.0000000000000931

Bragard, I., Seghaye, M.-C., Farhat, N., Solowianiuk, M., Saliba, M., Etienne, A.-M., & Schumacher, K. (2018). Implementation of a 2-day simulation-based course to prepare medical graduates on their first year of residency. Pediatric Emergency Care, 34(12), 857-861. https://doi.org/10.1097/PEC.0000000000000930

Caputo, K., Byrick, R., Chapman, M., Orser, B., & Orser, B. (2006). Intubation of SARS patients: Infection and perspectives of healthcare workers. Canadian Journal of Anaesthesia, 53(2), 122-129. https://doi.org/10.1007/BF03021815

Cohen, E. R., Barsuk, J. H., Moazed, F., Caprio, T., Didwania, A., McGaghie, W. C., & Wayne, D. B. (2013). Making July safer: Simulation-based mastery learning during intern boot camp. Academic Medicine, 88(2), 233-239. https://doi.org/10.1097/ACM.0b013e31827bfc0a

Davis, D. A., Mazmanian, P. E., Fordis, M., Van Harrison, R., Thorpe, K. E., & Perrier, L. (2006). Accuracy of physician self-assessment compared with observed measures of competence: A systematic review. JAMA, 296(9), 1094-1102. https://doi.org/10.1001/jama.296.9.1094

Ericsson, K. A. (2004). Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Academic Medicine, 79(10), S70-S81. https://doi.org/10.1097/00001888-200410001-00022

Fisher, N., Eisen, L. A., Bayya, J. V., Dulu, A., Bernstein, P. S., Merkatz, I. R., & Goffman, D. (2011). Improved performance of maternal-fetal medicine staff after maternal cardiac arrest simulation-based training. American Journal of Obstetrics and Gynecology, 205(3), e1-e5. https://doi.org/10.1016/j.ajog.2011.06.012

Fouilloux, V., Gran, C., Guervilly, C., Breaud, J., El Louali, F., & Rostini, P. (2019). Impact of education and training course for ECMO patients based on high-fidelity simulation: A pilot study dedicated to ICU nurses. Perfusion, 34(1), 29-34. https://doi.org/10.1177/0267659118789824

Grillet, G., Marjanovic, N., Diverrez, J., Tattevin, P., Tadié, J., & L’Her, E. (2015). Intensive care medical procedures are more complicated, more stressful, and less comfortable with Ebola personal protective equipment: A simulation study. The Journal of Infection, 71(6), 703-706. https://doi.org/10.1016/j.jinf.2015.09.003

Holmes, E. A., O’Connor, R. C., Perry, V. H., Tracey, I., Wessely, S., Arseneault, L., Ballard, C., Christensen, H., Silver, R. C., & Everall, I. (2020). Multidisciplinary research priorities for the COVID-19 pandemic: a call for action for mental health science. The Lancet Psychiatry, 7(6), 547-560. https://doi.org/10.1016/S2215-0366(20)30168-1

Itoh, T., Lee-Jayaram, J., Fang, R., Hong, T., & Berg, B. (2019). Just-in-Time Training for Intraosseous Needle Placement and Defibrillator Use in a Pediatric Emergency Department. Pediatric Emergency Care, 35(10), 712-715. https://doi.org/10.1097/PEC.0000000000001516

Kirkpatrick, D., & Kirkpatrick, J. (2006). Evaluating training programs: The four levels (3rd ed.). Berrett-Koehler Publishers.

Kolb, D. (1984). The experiential learning theory of development. Experiential Learning: Experience as the Source of Learning and Development (pp. 132-160). Prentice Hall, Englewood Cliffs, NJ, 132-160.

Lau, J., Fung, K., Wong, T., Kim, J., Wong, E., Chung, S., Ho, D., Chan, L., Lui, S., & Cheng, A. (2004). SARS transmission among hospital workers in Hong Kong. Emerging Infectious Diseases, 10(2), 280-286. https://doi.org/10.3201/eid1002.030534

Marrs, R., Horsley, T. L., Hackbarth, D., & Landon, E. (2019). High consequence infectious diseases training using interprofessional simulation and TeamSTEPPS. American Journal of Infection Control, 48(6), 615-620. https://doi.org/10.1016/j.ajic.2019.10.007

McLaughlin, C., Barry, W., Barin, E., Kysh, L., Auerbach, M. A., Upperman, J. S., Burd, R. S., & Jensen, A. R. (2019). Multidisciplinary simulation-based team training for trauma resuscitation: A scoping review. Journal of Surgical Education, 76(6), 1669-1680. https://doi.org/10.1016/j.jsurg.2019.05.002

O’Keeffe, D., Bradley, D., Evans, L., Bustamante, N., Timmel, M., Akkineni, R., Mulloy, D., Goralnick, E., & Pozner, C. (2016). Ebola Emergency Preparedness: Simulation Training for Frontline Health Care Professionals. MedEdPORTAL, 12, 10433-10433. https://doi.org/10.15766/mep_2374-8265.10433

Paige, J. T., Fairbanks, R. J. T., & Gaba, D. M. (2018). Priorities related to improving healthcare safety through simulation. Simulation in Healthcare, 13(3S), S41-S50. https://doi.org/10.1097/SIH.0000000000000295

Patterson, M., Geis, G., Falcone, R., LeMaster, T., & Wears, R. (2013). In situ simulation: detection of safety threats and teamwork training in a high risk emergency department. BMJ Quality & Safety, 22(6), 468-477. https://doi.org/10.1136/bmjqs-2012-000942

Phin, N., Rylands, A., Allan, J., Edwards, C., Enstone, J., & Nguyen-Van-Tam, J. (2009). Personal protective equipment in an influenza pandemic: a UK simulation exercise. Journal of Hospital Infection, 71(1), 15-21. https://doi.org/10.1016/j.jhin.2008.09.005

Sam, J., Pierse, M., Al-Qahtani, A., & Cheng, A. (2012). Implementation and evaluation of a simulation curriculum for paediatric residency programs including just-in-time in situ mock codes. Paediatrics & Child Health, 17(2), e16-20. https://doi.org/10.1093/pch/17.2.e16

Secheresse, T., Pansu, P., & Lima, L. (2020). The impact of full-scale simulation training based on Kolb’s learning cycle on medical prehospital emergency teams: A multilevel assessment study. Simulation in Healthcare, 15(5), 335-340. https://doi.org/10.1097/SIH.0000000000000461

Seet, R., Lim, E., Oh, V., Ong, B., Goh, K., Fisher, D., Ho, K., & Yeoh, K. (2009). Readiness exercise to combat avian influenza. QJM : Monthly Journal of the Association of Physicians, 102(2), 133-137. https://doi.org/10.1093/qjmed/hcn159

Shanafelt, T., Ripp, J., & Trockel, M. (2020). Understanding and addressing sources of anxiety among health care professionals during the COVID-19 pandemic. JAMA, 323(21), 2133-2134. https://doi.org/10.1001/jama.2020.5893

Shrestha, R., Badyal, D., Shrestha, A., & Shrestha, A. (2020). In-situ simulation-based module to train interns in resuscitation skills during cardiac arrest. Advances in Medical education and Practice, 11, 271-285. https://doi.org/10.2147/AMEP.S246920

Shrestha, R., Shrestha, A. P., Shrestha, S. K., Basnet, S., & Pradhan, A. (2019). Interdisciplinary in situ simulation-based medical education in the emergency department of a teaching hospital in Nepal. International Journal of Emergency Medicine, 12(1), 19. https://doi.org/10.1186/s12245-019-0235-x

Tan, K. W., Ong, H. K., & Mok, U. S. (2021). Using simulation and inter-professional education to teach infection prevention during resuscitation. The Asia Pacific Scholar, 6(1), 93-108. https://doi.org/10.29060/TAPS.2021-6-1/OA2229

The Centre for Evidence-Based Medicine Nuffield Department of Primary Care Health Sciences University of Oxford. (2020). Global Covid-19 case fatality rates. https://www.cebm.net/covid-19/global-covid-19-case-fatality-rates/. Accessed April 26, 2020.

The Novel Coronavirus Pnuemonia Emergency Response Epidemiology Team. (2020). The epidemiological characteristics of an outbreak of 2019 novel coronavirus diseases (COVID-19)—China, 2020. China CDC Weekly, 2(8), 113-122. https://doi.org/10.46234/ccdcw2020.032

Watson, C., Duval-Arnould, J., McCrory, M., Froz, S., Connors, C., Perl, T., & Hunt, E. (2011). Simulated pediatric resuscitation use for personal protective equipment adherence measurement and training during the 2009 influenza (H1N1) pandemic. The Joint Commission Journal on Quality and Patient Safety, 37(11), 515-523. https://doi.org/10.1016/S1553-7250(11)37066-3

Wax, R. S., & Christian, M. D. (2020). Practical recommendations for critical care and anesthesiology teams caring for novel coronavirus (2019-nCoV) patients. Canadian Journal of Anesthesia/Journal canadien d’anesthésie, 67(5), 568–576. https://doi.org/10.1007/s12630-020-01591-x

World Health Organization. (2020). Statement on the second meeting of the International Health Regulations (2005) Emergency Committee regarding the outbreak of novel coronavirus (2019- nCoV). Retrieved 6 April, 2021, from https://www.who.int/news-room/detail/30-01-2020-statement-on-the-second-meeting-of-the-international-health-regulations-(2005)-emergency-committee-regarding-the-outbreak-of-novel-coronavirus-(2019-ncov)

Wu, Z., & McGoogan, J. (2020). Characteristics of and important lessons from the coronavirus disease 2019 (COVID-19) outbreak in China: Summary of a report of 72 314 cases from the Chinese Center for Disease Control and Prevention. JAMA, 323(13), 1239-1242. https://doi.org/10.1001/jama.2020.2648

Ziv, A., Small, S. D., & Wolpe, P. R. (2000). Patient safety and simulation-based medical education. Medical Teacher, 22(5), 489-495. https://doi.org/10.1080/01421590050110777

Zuo, M., Huang, Y., Ma, W., Xue, Z., Zhang, J., Gong, Y., & Che, L. (2020). Expert recommendations for tracheal intubation in critically ill patients with Noval Coronavirus Disease 2019. Chinese Medical Sciences Journal, 35(2), 105–109. https://doi.org/10.24920/003724

*Nicola Ngiam
Centre for Healthcare Simulation,
National University of Singapore,
14 Medical Dr,
Singapore 117599
Email: nicola_ngiam@nuhs.edu.sg

Submitted: 5 July 2021
Accepted: 13 September 2021
Published online: 5 April, TAPS 2022, 7(2), 6-16
https://doi.org/10.29060/TAPS.2022-7-2/OA2654

Ming-Jung Ho^1,2, Yu-Che Chang^3,4,5 & Steven L. Kanter⁶

¹Center for Innovations and Leadership in Education, Georgetown University Medical Center (CENTILE), Washington, D.C., United States; ²Department of Family Medicine, Georgetown University School of Medicine, Washington D.C., United States; ³Chang Gung Medical Education Research Centre (CG-MERC), Chang Gung Medical Foundation, Taoyuan, Taiwan; ⁴Chang Gung University College of Medicine, Taoyuan, Taiwan; ⁵Department of Emergency Medicine, Chang Gung Memorial Hospital, Linkou, Taoyuan, Taiwan; ⁶Association of Academic Health Centers, Washington, D.C., United States.

Abstract

Introduction: The COVID-19 pandemic forced medical schools worldwide to transition online. While there are ample reports about medical education adaptations to this crisis, there are limited studies evaluating the impact.

Methods: This study includes a case study of how Taiwanese medical school deans maintained in-person education during the COVID-19 pandemic. Additionally, it demonstrates how SWOT analyses can help medical educators reflect on adaptations during the COVID-19 pandemic and future crises.  This study employed two online surveys and a semi-structured interview regarding curricular adaptations. Eligible participants were deans or associate deans of all medical schools in Taiwan.

Results: Through a SWOT analysis, this study identified Strengths as strong leadership, prior experience with SARS, and ability to promptly adapt curriculum; Weaknesses as lack of faculty confidence in online education, limited numbers of administrative staff, and inability to rapidly add new topics; Opportunities as centralised anti-epidemic policies, inter-institutional collaborations, and educational innovations; and Threats as concerns from parents, patients, and teaching hospitals. It is reported that the quality of education was maintained as students’ evaluations of courses, performances in written exams, clerkship supervisors’ assessments, national OSCE, and national board exams remained comparable to pre-COVID times. Strengths and Opportunities such as strong leadership and centralised anti-epidemic policies can overcome Weaknesses and Threats, such as lack of confidence in online education and familial pressure to suspend hospital-based education.

Conclusion: This study recommends that medical school leaders conduct SWOT analyses as early as possible to plan strategies to continue safe and quality medical education during COVID-19 and future crises.

Keywords:           SWOT Analysis, Adaptation, Medical School, COVID-19, Pandemic, National Survey, Medical Education, Taiwan

Practice Highlights

• Medical school deans in Taiwan documented critical observations made during a global pandemic.
• SWOT analysis can help us reflect on medical education adaptations during the pandemic and crises.
• Strong leadership and centralised anti-epidemic policies can aid in overcoming weaknesses/threats.
• Schools can benefit from using SWOT analyses to plan for continuing education during emergent crises.

Table 1. Characteristics of Survey Schools

^aOne medical school had both 6-year program for high school graduates and 4-year program for college graduates.

Table 2. Themes and quotes of curriculum adaptations

Table 3. SWOT Analysis^b

^b This table is created using the blank SWOT form from Topor et al.²¹

A. Curricular Adaptations

1) Pre-clinical curriculum adaptations:

Taiwanese medical schools rapidly adapted their curricula to comply with government policy on social distancing prohibiting indoor gatherings of more than 100 individuals. Whereas a school with a small student body and sufficient classroom capacity did not have to transition online, a school with a large student body and insufficient classroom capacity moved almost all pre-clinical classes online. Some schools went partially online, converting only lectures with more than 100 students to online learning and capping the number of students in the classroom, with live stream available for those not in the classroom. While schools adapted different modalities for lecture-based courses, all schools made efforts to keep laboratory courses face-to-face. Some schools shared how they adjusted laboratory sessions to maintain social distance, including simultaneously using more lab rooms or holding lab sessions in different time slots.

2) Clinical curriculum adaptations:

In contrast to approaches in the United States suspending clinical rotations in spring semester recommended by AAMC (Association of American Medical Colleges, 2021), all medical schools in Taiwan continued clinical rotations while protecting the safety of students. Several strategies were implemented rapidly: Orientation to clinical rotations articulated pandemic prevention and included online modules required by the hospital. Clinical students were provided with surgical masks. In the emergency room, students were given N95 respirators each day. Exposure to suspected and confirmed COVID-19 cases was avoided for students replacing rotations in specialties with high risk of COVID-19 exposure. To prevent cross infection, inter-institutional and overseas exchange programs were cancelled. Students rotated in only one of the affiliated hospitals. Social distancing is also mentioned in clinical rotation adjustments. Several respondents mentioned that the number of students in outpatient clinics, in classrooms, and in conferences was reduced. One school reported an increase of clinical teachers to meet social distancing demands.

3) Assessment adaptations:

Eight schools replied that they did not change the assessment of students by written examinations in classrooms but arranged empty seats between students to keep social distance and protected students from airborne droplet transmissions by requiring everyone to wear a mask. Some schools decreased the use of written exams in classrooms by substituting reports, reflections, online exams, online presentations, and other approaches. Some evaluations involving hands-on practices were cancelled or replaced by simulation. One school reported that their admission exams included tests in laboratories. To prevent face-to-face transmission of airborne droplets, transparent acrylic partitions were installed on the benches in front of each student.

B. Impact of curricular adaptations

Medical school deans tried to evaluate the impact of curricular adaptions by comparing student performance in the 2020 spring semester with previous years. Most deans reviewed the passing rate of courses and reported no statistical differences. They also reported monitoring students’ performances in national board exams in July and national OSCE clinical skills exam held before graduation. The deans were relieved to see that the national exam scores were comparable to previous years and felt that curricular adaptations didn’t impact student competencies negatively.

C. Strengths

The strengths of Taiwanese medical schools’ responses to COVID-19 can be summarised into three themes: strong leadership, prior experience with SARS, and ability to rapidly adapt the curriculum.

1) Strong leadership: The key strength most Taiwanese deans cited in their medical schools’ response to COVID-19 is the strong leadership in the early deployment of epidemic prevention measures. The guidelines from the Central Epidemic Command Center were implemented on each campus, including requiring students to wear masks, to maintain social distance, to monitor temperature and symptoms of COVID-19, and to quarantine international students in separate dormitory areas. Communication also was highlighted by respondents as manifesting strong leadership. Furthermore, two deans made videos in collaboration with students to promote anti-epidemic measures (Chang Gung University, 2020; Fu Jen Catholic University, 2020).

2) Prior experience with SARS: Another strength most respondents noted was the experience of SARS. When COVID-19 spread to Taiwan, most clinical teachers had experienced SARS and were able to share critical lessons to face new emerging infectious diseases with the younger generation. The experience of SARS also prepared medical educators to deal with professionalism and ethics issues such as distribution of limited medical resources. Many participants mentioned that SARS had a great impact on the medical education system in Taiwan because it exposed the lack of primary care workforce with medical school graduates entering specialty training directly after medical school. After SARS, a postgraduate year program was established with one year of required general medicine training prior to specialty training programs with the goal of producing more physicians competent in general medicine to respond to emergent infectious diseases.

3) Ability to rapidly adapt the curriculum: All schools reported rapid curricular adaptations when spring semester started one or two weeks later than the scheduled time. Different strategies were employed in pre-clinical and clinical curricula to prevent and control the spread of COVID-19 as described in 3.1 curricular adaptation.

D. Weaknesses

The weaknesses of Taiwanese medical schools’ responses to COVID-19 can be summarised into three themes: lack of faculty confidence in online education, limited number of administrative staff, and inability to rapidly add new topics.

1) Lack of faculty confidence in online education: All respondents mentioned online teaching as challenging to teachers. Many voiced concerns about the effectiveness of online learning and found assessment of online learning challenging. Online learning is also considered challenging to students. Some respondents mentioned that online learning requires self-directed learning, which is not a strength of Taiwanese students since didactic teaching and rote memorisation is the norm of the K-12 education system.

2) Limited number of administrative staff: In response to the question about weaknesses of their medical schools’ responses to the COVID-19 pandemic, administrative challenges were mentioned frequently. Although Taiwan did not experience high burdens of COVID-19, all schools were preparing for large-scale outbreaks with frequent scenario planning, describing these planning activities using the military metaphor of “military sand table planning” and “early deployment.”

3) Inability to rapidly add new topics: The pandemic also revealed the need for medical educators to engage in some topics traditionally not articulated in medical curricula. It was challenging to add new topics in the ongoing 2020 spring semester curricula but respondents suggested to add the following topics in the future: infectious disease control, emergent infectious disease, crisis management and communication, public policy and ethical issues related to pandemics such as distribution of medical resources, and government control and restrictions on human rights vs quarantine policy.

E. Opportunities

Taiwanese medical school deans highlighted three opportunities which enabled them to continue to open medical schools during the COVID-19 pandemic: centralised anti-epidemic policies, collaborations among medical schools, and educational innovations.

1) Centralised anti-epidemic policies: Taiwanese deans all reported that the policies from the central government are helpful in their implementation of anti-epidemic measures. Several deans shared the policy guidelines issued by the Ministry of Education with clear guidelines to follow the principles of infection containment. Although each university has the flexibility to decide how to comply with the guidelines, there was external oversight to ensure compliance. For example, when a teaching hospital had an intra-hospital transmission of COVID-19, the medical school had to work with the Central Epidemic Command Center and Ministry of Education in its response.

2) Inter-institutional collaborations: Collaborations among medical schools through the Taiwan Association of Medical Education (TAME) also helped Taiwanese medical school deans to adapt to COVID-19. For instance, the deans worked together at TAME to ensure that final-year students graduated on time. Every year, TAME organises a national OSCE examination required for graduation for final year Taiwanese medical students. Medical schools and teaching hospitals collaborated to administer these exams at OSCE centers in selected teaching hospitals with infection mitigating measures.

3) Educational innovations: All respondents mentioned that the COVID-19 pandemic provided opportunities to engage in innovations in medical education, especially online education.  Online education was quickly implemented in some schools because access to internet is prevalent in Taiwan and they utilised the technology developed before the pandemic. One school reported an innovation using the Interactive Response System (IRS) developed three years ago and is now applied to enhance online teaching amid COVID-19. Another school adapted communication software instead of face-to-face counseling meetings. One school reported using “LINE” (a messaging app widely used in Taiwan) for a group chat for students to reduce student anxiety related to COVID-19. 

F. Threats

Three themes were identified during the COVID-19 pandemic as threats to medical education, but external to medical schools: parental pressure to remove students from clinical settings, decrease in patient volume, and hospital infection control measures limited educational opportunities.

1) Parental pressure to remove students from clinical settings:  Several deans mentioned that students’ parents reached out to them, pressuring them to discontinue clinical rotations. One respondent stated, “Parents are more worried than teachers and students. They are concerned that clinical rotations in the hospitals are risky for students. However, parents accepted our explanation that clinical rotations will not proceed unless there is adequate PPE and infection control measures.”

2) Decrease in patient volume: Another threat commonly mentioned was the decrease in patient volume. A participant explained, “Our hospital experienced a 30% decrease in patient volume. It is inconvenient for patients to come to the hospital due to tightened entrance control. The restriction on patients’ family visits is another factor. We are not letting students rotate on wards with suspected COVID-19 patients.” The limited patient encounters are perceived as potential threat to students’ clinical competence.

3) Hospital infection control measures limited educational opportunities: Finally, another threat reported was associated with infection control measures by the teaching hospitals. Deans expressed concerns that students’ clinical learning opportunities were compromised during the pandemic due to cancelled bedside teaching in high-risk specialties, cancelled presentations in conferences, cancelled interdepartmental grand rounds, and suspension of exchange programs both domestically and internationally. Some mentioned that reduced clinical exposures might result in less role modeling.

IV. DISCUSSION

During this historical event, medical schools around the world are challenged to navigate a path forward to meet missions of education, research, and patient care during an ongoing pandemic. As many countries are loosening or tightening restrictions after their curves of infection have flattened or resurged, medical schools globally are at various stages of deliberating when and how to resume campus-based and hospital-based medical education (Blaaza et al., 2020; Cleland et al., 2020; Tolsgaard et al., 2020). A systematic rapid review of published reports of medical educational developments in response to the COVID-19 pandemic concluded that most papers lacked evaluation data and focused on transitioning online and removing students from the clinical contexts (Gordon et al., 2020). The review suggested that medical schools learn from a few postgraduate papers reporting how face-to-face patient contact could be maintained while mitigating risk using PPE and social distancing (G. Choi et al., 2020; Hanel et al., 2020; Kang et al., 2020). Our study of medical school deans in Taiwan on how they continued in-person medical education throughout the COVID-19 pandemic offers lessons for the medical education community to engage medical students in campus-based and workplace-based learning with evaluation of impacts.

This study offers a conceptual advance by demonstrating how SWOT analysis can help us reflect on medical education adaptations to COVID-19 pandemic and future crises. Our analysis of the strengths, weaknesses, opportunities and threats in Taiwanese medical school adaptations to COVID-19 highlights certain strategies to leverage strengths and opportunities to overcome weaknesses and threats. A critical factor in the decision to continue face-to-face medical education, clearly identified by the opportunity-strength strategy in Table 3, was the early deployment of epidemic prevention measures under centralised government policies and strong medical school leadership. Epidemic prevention measures were deployed at all levels of schools in Taiwan as students returned to campus for spring semester in February 2020. To enact anti-epidemic policies in different contexts, this study presented various preventive measures to keep the recommended social distance in classrooms, laboratories, and teaching hospitals. To implement these measures, transparent communication and administrative support from medical school leadership are essential.

SWOT analysis on opportunity-weakness and threat-weakness strategies offers lessons on how to overcome the weakness of lack of faculty confidence in online education. Although online education is an important curricular adaptation to maintain social distance in campus-based medical education, our survey shows that respondents found online teaching challenging to faculty and are concerned about the effectiveness of online education. Strategies identified to overcome this weakness is to take advantage of the educational technology readily available and to transform the threats from hospital infection control to suspend hospital-based education into opportunities for integrating innovative online education. It is important to leverage innovative online education in workplace learning in the context of reduced patient case exposures before the pandemic subsides (B. Choi et al., 2020; Lo et al., 2020). There is sufficient evidence in the education literature that well-designed online education is as effective as traditional classroom teaching and shows equivalent or better learning outcomes (George et al., 2014; Jung & Rha, 2000; Khalil et al., 2020). We hope that the evaluations reported in the fall semester survey showing equivalent outcomes of online adaptations during COVID-19 would encourage medical educators to take advantage of innovations reported by this study, existing literature on online education in general, and the growing literature about innovative online medical education (Ahmed et al., 2020; Medical Education, 2020).

This study highlighted that another enabling factor for Taiwanese medical educators to continue face-to-face medical education was the experience of SARS. The Taiwanese population became accustomed to anti-epidemic measures, including wearing masks after SARS, which facilitated the implementation of anti-epidemic measures in schools during COVID-19. Taiwanese health providers were severely challenged during the 2003 SARS epidemic. 103 of 318 confirmed cases  (Centers for Disease Control and Prevention, 2003) and 11 of 60 deaths (including two first-year residents) (Taiwan Centers for Disease Control, 2013) were healthcare workers. Medical education leaders had a difficult decision to make regarding the final year students who were working as interns in hospitals (Lai, 2020). Following closures of clinics and resignation of healthcare providers, there were heated debates about physician duties to treat patients while being at risk of exposure to a deadly infectious disease and of being stigmatised by society (Lee, 2003). After SARS, not only were curricula on ethics and infection control strengthened, but the medical education system was reformed to convert the final-year of medical school to a formal post-graduate year program (Ho et al., 2017). In addition, as articulated by the respondents of the survey, SARS prepared Taiwanese medical educators and clinical teachers to be confident and competent to perform their educational and clinical duties in the time of COVID-19.

Although this study has many lessons to offer both conceptually and practically, there are some limitations to be considered. We conducted the survey with open-ended questions to explore diverse perspectives. The reported theme frequencies should be considered with caution since some respondents record more comprehensive answers than others. While the most frequently mentioned themes might be most important to the respondents, the least frequently given answer might be valuable due to its innovative nature. Another limitation of an open-ended questionnaire is that respondents might not have the time to enter answers in detail. To overcome this limitation, we invited the respondents to participate in a follow-up interview. However, only about one-third of deans were available to be interviewed during the study period. The four interviews conducted nonetheless enriched the study with more context and details. Finally, this study only surveyed medical school deans. It would be interesting to conduct further studies comparing these results to the perspectives of students and faculty members.

V. CONCLUSION

This nationwide study of all medical schools in Taiwan documents critical observations of top medical school leaders during an active pandemic. This case study demonstrated that strengths and opportunities such as strong leadership and centralised anti-epidemic policies can overcome weaknesses and threats such as faculty lack of confidence in online education and parental pressure to stop hospital-based education. We recommend that medical school leaders conduct SWOT analyses as early as possible to plan strategies to continue safe and quality medical education during COVID-19 and forthcoming emergencies. Similar to the impact of SARS, which led to reform of the Taiwanese postgraduate training system, there is hope that thoughtful reflection on medical education adaptations to the COVID-19 pandemic will improve medical education in general and lead to better preparation for future crises.

Notes on Contributors

MH contributed to the conception, acquisition and analysis of data, drafted the article, and approved the final version; YC contributed to the design, acquisition and analysis of data, drafted the article, and approved the final version; SK contributed to analysis and interpretation of data, revised the article critically, and approved the final version.

Ethical Approval

This study was approved as exempt by the Georgetown University Institutional Review Board (STUDY00002812). Since the study poses no greater than minimal risk to subjects, a waiver of written consent was granted. . We emailed invitations with information about study purpose, data confidentiality, and a weblink to the online survey. A verbal consent was obtained before data collection began. Participation was voluntary without incentives offered. All methods were carried out in accordance with relevant guidelines and regulations.

Data Availability

It is not possible to share data since we did not get consent from participants to share all data collected. They agreed to have results reported in a summary format with selected quotes.

Acknowledgement

The authors wish to thank survey respondents and interviewees for providing insights on the research topic. Fred Hafferty and Carrie Chen provided useful feedback to improve the manuscript.

Funding

This study did not receive any funding.

Declaration of Interest

The authors declare that they have no competing interests.

References

Abbas, M., Dhane, M., Beniey, M., Meloche-Dumas, L., Eissa, M., Guerard-Poirier, N., El-Raheb, M., Lebel-Guay, F., Dubrowski, A., & Patocskai, E. (2020). Repercussions of the COVID-19 pandemic on the well-being and training of medical clerks: A pan-Canadian survey. BMC Medical Education, 20(1), 385. https://doi.org/10.1186/s12909-020-02293-0

Ahmed, S. A., Hegazy, N. N., Abdel Malak, H. W., Cliff Kayser, W., 3rd, Elrafie, N. M., Hassanien, M., Al-Hayani, A. A., El Saadany, S. A., Ai-Youbi, A. O., & Shehata, M. H. (2020). Model for utilizing distance learning post COVID-19 using (PACT)^TM a cross sectional qualitative study. BMC Medical Education, 20(1), 400. https://doi.org/10.1186/s12909-020-02311-1

Al-Balas, M., Al-Balas, H. I., Jaber, H. M., Obeidat, K., Al-Balas, H., Aborajooh, E. A., Al-Taher, R., & Al-Balas, B. (2020). Distance learning in clinical medical education amid COVID-19 pandemic in Jordan: Current situation, challenges, and perspectives. BMC Medical Education, 20(1), 341. https://doi.org/10.1186/s12909-020-02257-4

Association for Medical Education in Europe. (2020). COVID-19. Retrieved April 1, 2021, from https://amee.org/covid-19

Association of American Medical Colleges. (2021). Coronavirus (COVID-19) resource hub. Retrieved April 1, 2021, from https://www.aamc.org/coronavirus-covid-19-resource-hub

Blaaza, M., Akhtar, H. N., & Tanzim, U. (2020). Returning to clinics amidst COVID-19: A student response to Compton et al. Medical Education, 54(12), 1193. https://doi.org/10.1111/medu.14282

Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative research in psychology, 3(2), 77-101.

Centers for Disease Control and Prevention. (2003). Severe acute respiratory syndrome–Taiwan, 2003. Morbidity and mortality weekly report, 52(20), 461‐466.

Chang Gung University. (2020, March 16). Chang Gung University COVID-19 anti-epidemic advanced adaptation measures [Video]. YouTube. https://www.youtube.com/watch?v=WyoOoShXVa8

Choi, B., Jegatheeswaran, L., Minocha, A., Alhilani, M., Nakhoul, M., & Mutengesa, E. (2020). The impact of the COVID-19 pandemic on final year medical students in the United Kingdom: a national survey. BMC medical education, 20(1), 206. https://doi.org/10.1186/s12909-020-02117-1

Choi, G., Wan, W., Chan, A., Tong, S., Poon, S., & Joynt, G. (2020). Preparedness for COVID-19: In situ simulation to enhance infection control systems in the intensive care unit. British Journal of Anaesthesia, 125(2), e236-e239. https://doi.org/10.1016/j.bja.2020.04.001

Cleland, J., McKimm, J., Fuller, R., Taylor, D., Janczukowicz, J., & Gibbs, T. (2020). Adapting to the impact of COVID-19: Sharing stories, sharing practice. Medical Teacher, 42(7), 772-775. https://doi.org/10.1080/0142159X.2020.1757635

Compton, S., Sarraf-Yazdi, S., Rustandy, F., & Radha Krishna, L. K. (2020). Medical students’ preference for returning to the clinical setting during the COVID-19 pandemic. Medical Education, 54(10), 943-950. https://doi.org/10.1111/medu.14268

Frieden, J. (2020, May 9). Medical schools grapple with challenges for the coming year— Three-quarters hope to have students back by the end of July. MedPage Today. Retrieved April 1 fromhttps://www.medpagetoday.com/publichealthpolicy/medicaleducation/86416

Fu Jen Catholic University. (2020, February 16). COVID-19: Fu Jen Catholic University anti-epidemic promotion video. [Video]. YouTube. https://youtu.be/-n3KvlVHpzY

Gardner, L. (2020, January 31). Update January 31: Modeling the spreading risk of 2019-nCoV. Johns Hopkins University Center for Systems Science and Engineering. Retrieved April 1 from https://systems.jhu.edu/research/public-health/ncov-model-2/

George, P. P., Papachristou, N., Belisario, J. M., Wang, W., Wark, P. A., Cotic, Z., Rasmussen, K., Sluiter, R., Riboli-Sasco, E., Tudor Car, L., Musulanov, E. M., Molina, J. A., Heng, B. H., Zhang, Y., Wheeler, E. L., Al Shorbaji, N., Majeed, A., & Car, J. (2014). Online eLearning for undergraduates in health professions: A systematic review of the impact on knowledge, skills, attitudes and satisfaction. Journal of Global Health, 4(1), 010406. https://doi.org/10.7189/jogh.04.010406

Ghazinoorya, S., Abdia, M., & Azadegan-Mehr, M. (2011). SWOT methodology: A state-of-the-art review for the past, a framework for the future. Journal of Business Economics and Management, 12(1), 24-48. https://doi.org/10.3846/16111699.2011.555358

Gordon, L., & Cleland, J. A. (2021). Change is never easy: How management theories can help operationalise change in medical education. Medical Education, 55(1), 55-64. https://doi.org/10.1111/medu.14297

Gordon, M., Patricio, M., Horne, L., Muston, A., Alston, S. R., Pammi, M., Thammasitboon, S., Park, S., Pawlikowska, T., Rees, E. L., Doyle, A. J., & Daniel, M. (2020). Developments in medical education in response to the COVID-19 pandemic: A rapid BEME systematic review: BEME Guide No. 63. Medical Teacher, 42(11), 1202-1215. https://doi.org/10.1080/0142159X.2020.1807484

Hanel, E., Bilic, M., Hassall, K., Hastings, M., Jazuli, F., Ha, M., Trotter, B., Fraser, C., & Rutledge, G. (2020). Virtual application of in situ simulation during a pandemic. Canadian Journal of Eemergency Medicine, 22(5), 563-566. https://doi.org/10.1017/cem.2020.375

Harries, A. J., Lee, C., Jones, L., Rodriguez, R. M., Davis, J. A., Boysen-Osborn, M., Kashima, K. J., Krane, N. K., Rae, G., Kman, N., Langsfeld, J. M., & Juarez, M. (2021). Effects of the COVID-19 pandemic on medical students: A multicenter quantitative study. BMC Medical Education, 21(1), 14. https://doi.org/10.1186/s12909-020-02462-1

Ho, M.-J., Shaw, K., Shih, J., & Chiu, Y.-T. (2017). Mission and modernity: The history and development of medical education in Taiwan. In L. C. Chen, M. R. Reich, & J. Ryan (Eds.), Medical Education in East Asia: Past and Future (pp. 84-111). Indiana University press.

International Association of Medical Science Educators. (2020, April 22). #IAMSECafe Archives & COVID-19 resources for medical science educators. Retrieved April 1 from http://www.iamse.org/covid-19-resources-for-medical-science-educators/

Journal of the Association of American Medical Colleges. (2021). COVID-19 and medical education. Retrieved April 1 from https://journals.lww.com/academicmedicine/pages/collectiondetails.aspx?TopicalCollectionId=68

Jung, I., & Rha, I. (2000). Effectiveness and cost-effectiveness of online education: A review of the literature. Educational Technology, 40(4), 57-60.

Kang, Y., Deng, L., Zhang, D., Wang, Y., Wang, G., Mei, L., Zhou, G., & Shu, H. (2020). A practice of anesthesia scenario design for emergency cesarean section in patients with COVID-19 infection based on the role of standard patient. BioScience Trends, 14(3), 222-226. https://doi.org/10.5582/bst.2020.03066

Keegan, D. A., & Bannister, S. L. (2021). More than moving online: Implications of the COVID-19 pandemic on curriculum development. Medical Education, 55(1), 101-103. https://doi.org/10.1111/medu.14389

Khalil, R., Mansour, A. E., Fadda, W. A., Almisnid, K., Aldamegh, M., Al-Nafeesah, A., Alkhalifah, A., & Al-Wutayd, O. (2020). The sudden transition to synchronized online learning during the COVID-19 pandemic in Saudi Arabia: A qualitative study exploring medical students’ perspectives. BMC Medical Education, 20(1), 285. https://doi.org/10.1186/s12909-020-02208-z

Lai, C. (2020, March 20). 2003 SARS: Reflections on Taiwanese medical education. UDN. https://health.udn.com/health/story/6631/4421448

Lazarus, G., Findyartini, A., Putera, A. M., Gamalliel, N., Nugraha, D., Adli, I., Phowira, J., Azzahra, L., Ariffandi, B., & Widyahening, I. S. (2021). Willingness to volunteer and readiness to practice of undergraduate medical students during the COVID-19 pandemic: A cross-sectional survey in Indonesia. BMC Medical Education, 21(1), 138. https://doi.org/10.1186/s12909-021-02576-0

Lee, M. (2003). SARS 風暴的啟示[Learning from the SARS event] [Editorial]. Journal of Medical Education, 7(1), 1-2.

Lo, H.-Y., Lin, S.-C., Chaou, C.-H., Chang, Y.-C., Ng, C.-J., & Chen, S.-Y. (2020). What is the impact of the COVID-19 pandemic on emergency medicine residency training: An observational study. BMC Medical Education, 20(1), 348. https://doi.org/10.1186/s12909-020-02267-2

Medical Education. (2020, May 7). Medical education adaptations. Retrieved April 1, 2020, from https://onlinelibrary.wiley.com/doi/toc/10.1002/(ISSN)1365-2923.med.adaptations.vi

Reid, M. D., & Sam, A. H. (2021). Reflections on assessment in the wake of change from the COVID-19 pandemic. Medical Education, 55(1), 128-130. https://doi.org/10.1111/medu.14368

Soled, D., Goel, S., Barry, D., Erfani, P., Joseph, N., Kochis, M., Uppal, N., Velasquez, D., Vora, K., & Scott, K. W. (2020). Medical student mobilization during a crisis: Lessons from a COVID-19 medical student response team. Academic Medicine, 95(9), 1384-1387. https://doi.org/10.1097/ACM.0000000000003401

Taiwan Center for Disease Control. (2020). Prevention and control of COVID-19 in Taiwan. Retrieved April 1, 2021, from https://www.cdc.gov.tw/Category/Page/0vq8rsAob_9HCi5GQ5jH1Q

Taiwan Centers for Disease Control. (2013). Severe acute respiratory syndrome (SARS). Retrieved April 1, 2021 from https://www.cdc.gov.tw/File/Get/InG8jagjxffXBDW1UexnrA

Tolsgaard, M. G., Cleland, J., Wilkinson, T., & Ellaway, R. H. (2020). How we make choices and sacrifices in medical education during the COVID-19 pandemic. Medical Teacher, 42(7), 741-743. https://doi.org/10.1080/0142159X.2020.1767769

Topor, D. R., Dickey, C., Stonestreet, L., Wendt, J., Woolley, A., & Budson, A. (2018). Interprofessional health care education at academic medical centers: Using a SWOT analysis to develop and implement programming. MedEdPORTAL, 14, 10766. https://doi.org/10.15766/mep_2374-8265.10766

UNICEF. (2021). COVID-19 Education: From disruption to recovery. Retrieved April 1, 2021, from https://en.unesco.org/covid19/educationresponse

Wang, C. J., Ng, C. Y., & Brook, R. H. (2020). Response to COVID-19 in Taiwan: Big data analytics, new technology, and proactive testing. The Journal of the American Medical Association, 323(14), 1341-1342. https://doi.org/10.1001/jama.2020.3151

Zhu, N., Zhang, D., Wang, W., Li, X., Yang, B., Song, J., Zhao, X., Huang, B., Shi, W., Lu, R., Niu, P., Zhan, F., Ma, X., Wang, D., Xu, W., Wu, G., Gao, G. F., & Tan, W. (2020). A novel coronavirus from patients with pneumonia in China, 2019. The New England Journal of Medicine, 382(8), 727-733. https://doi.org/10.1056/NEJMoa2001017

*Yu-Che Chang
No. 5, Fuxing St.,
Guishan Dist.,
Taoyuan City 333012,
Taiwan (R.O.C.)
Tel: +886-33281200 ext 3807
Email: changyuche@gmail.com

Submitted: 21 April 2021
Accepted: 7 October 2021
Published online: 4 January, TAPS 2022, 7(1), 87-97
https://doi.org/10.29060/TAPS.2022-7-1/OA2519

Nisakorn Deesaen¹, Kongpop Sutantikorn¹, Punyanuch Phonngoenchai¹, Sakchai Chaiyamahapruk² & Patcharada Amatyakul³

¹Faculty of Medicine, Naresuan University, Thailand; ²Department of Community Medicine, Faculty of Medicine, Naresuan University, Thailand; ³Department of Obstetrics and Gynaecology, Faculty of Medicine, Naresuan University, Thailand

Abstract

Introduction: Pelvic examination of patients in the department of obstetrics and gynaecology (Ob–Gyn) is an important skill for medical students. Because it involves a physical assessment of the patients’ genitalia, patients may refuse medical students to participate in the examination, affecting the medical students’ clinical skills.

Methods: This cross–sectional study was conducted at Naresuan University Hospital to determine the factors that influence the acceptance of medical student participation in the pelvic examinations. A total of 198 out–patients from the Ob–Gyn department were included. A Likert scale questionnaire was designed which featured topics on patients’ attitudes and circumstances related to medical student involvement in gynaecological procedures.

Results: The majority of outpatients (71.7%) accepted the participation of medical students in pelvic examinations. Patients with prior experiences in physical and pelvic examination by medical students had a significant impact on the patients’ acceptance (P–value<0.001). The patients’ impressions had an influence on the decision to accept students in pelvic exam participation. Approximately 40% of patients were concerned about the breach of confidentiality. However, most patients strongly agreed that allowing medical students to perform pelvic examination would benefit their medical education.

Conclusion: Most of the participants permitted medical students to participate in pelvic examinations and preferred that the medical instructor be the one to request permission. The patients’ impressions of medical students were crucial factors that significantly influence their decision whether to allow or deny them to participate in the procedure. Disclosure of confidentiality was found to be matters of concern to most patients.

Keywords:           Pelvic Examination, Medical Students, Acceptance, Performance, Clinical Teaching

Practice Highlights

• Most of the patients consent to a medical student participating in a pelvic examination.
• Medical instructors should ask for patients’ permission.
• Confidentiality and privacy of patients are critical issues that must be addressed.
• Patients’ impressions and prior examination experiences by medical students influence patients’ decisions.

I. INTRODUCTION

Medical students should learn how to perform pelvic examinations on patients during clinical years. History taking, physical examination, and pelvic examination are taught during the fourth to sixth year of medical training in our curriculum. Pelvic examination is defined as the assessment of external genitalia, speculum examination of the vagina and cervix, bimanual palpation of the uterus, adnexa and sometimes rectovaginal examination. This procedure is used to screen for gynaecological diseases and cancers in asymptomatic women and to diagnose gynaecological diseases in symptomatic women. Pelvic examination differs from other physical examinations because it involves an inspection of the genitalia, which, according to studies, commonly causes anxiety, fear, embarrassment (10–80% of women), pain, and discomfort (11–60% of women) (Bloomfield et al., 2014). Some factors why patients feel uncomfortable during a physical examination by medical students are concerns about privacy, confidentiality, and embarrassment (Rizk et al., 2002). As a result, patients may refuse to allow medical students to participate in the pelvic examination, affecting the learning experience and clinical skills of medical students. Furthermore, factors related to age, race, religion, cross–cultural differences, marital status, and previous delivery may also influence the patients’ decision to allow medical students in conducting pelvic examination (Anfinan et al., 2014; McLean et al., 2010). Compared to other ethnic and religious groups, Muslim women had a higher rate of refusing medical students, particularly male students (Nicum & Karoo, 1998). During intrapartum care, approximately 50% of Hindu and Muslim patients refused to accept medical students. Sikh and Muslim patients accepted only female medical students, 41% and 40%, respectively, whereas 59% of Christian patients accepted both female and male medical students (Nicum & Karoo, 1998). However, patients’ reasons for accepting and refusing medical student participation in pelvic examination have not been established in Southeast Asia because of limited investigation.

According to the patient’s bill of rights, patients have a right to accept or refuse the medical students’ participation. They should have the opportunity to decide if they want to contribute to the medical education or decline care from medical students (Teunissen, 2018). Approximately 26% of the patients refused to have a medical student perform a pelvic examination (Fortier et al., 2006). Previous study reported that the refusal rate increased as the level of student involvement increased, from observation to history taking to examinations and procedure, particularly with digital vaginal and rectal examination (Salah et al., 2015). In contrast, some patients accepted the medical students’ participation because they wanted to contribute to the clinical training in medical education. Because of the training process, the patients felt more confident that they would receive appropriate treatments. From the literature review, most studies were conducted in developed and some Arab countries. There are limited data from women in Southeast Asia countries on the factors and attitudes that influence their decision whether they allow or refuse medical students to participate in a pelvic examination. This research aimed at studying the patients’ characteristics, influencing factors and correlation with the acceptance of medical students, which may increase the Ob–Gyn patients to participate in clinical education.

II. METHODS

This cross–sectional descriptive study was conducted at the out–patient unit of the Department (OPD) of Obstetrics and Gynaecology, Naresuan University Hospital. Patients who visited Ob–Gyn department between November 2018 and May 2019 were included in the study. Patients under 20 years old, mentally or critically ill and unable to understand the questionnaire in Thai language were excluded in this study. All the participants were able to complete the consent forms and questionnaires independently, and were guaranteed anonymity.

The questionnaire was developed to evaluate 4 domains, as follows: (1) demographic and socio–economic data (age, gender, educational level, occupation and parity), (2) patient preference, (3) factors influencing patient receptivity of medical student involvement in pelvic examination (gender, hygiene, manner and demeanour, patients’ impressions of the medical students, prior experience in physical and pelvic examination), and (4) patients’ attitudes toward accepting medical students to conduct pelvic examination under supervision of medical instructors. Influencing factors and attitudes included in the questionnaire were modified based on a literature review. A 5–point Likert scale was used to determine the level of agreement (1= strongly agree; 2= agree; 3= neutral; 4= disagree; and 5= strongly disagree). The questionnaire was initially tested for content validity using the item-objective congruence (IOC) index, and then used in a pilot study on 30 patients who were not included in the study to ensure clarity and reliability.

A. Statistical Analysis

A Microsoft Excel spreadsheet was created for data entry and statistical analysis. Descriptive data was presented in the form of frequency, mode, and percentage. Chi-square test was calculated for proportions. The p–value of less than 0.05 was considered statistically significant.

B. Ethical Considerations

All collected data was kept confidential, and the information was used for research only. This study was approved by Naresuan University Institutional Review Board in compliance with the principles of the Helsinki Declaration. 

III. RESULTS

During the study period, 236 participants were recruited from the out–patient department of Ob–Gyn, but only 198 completed the questionnaire. Most of them (99.0%) were aware that Naresuan University Hospital is a primary teaching hospital of the Faculty of Medicine, Naresuan University which provides clinical training for medical students and residents. The socio–demographic data of the study population are shown in Table 1. The data that supports the findings of this study are openly available at http://doi.org/10.17605/OSF.IO/HBV68 (Amatyakul, 2021).

Table 1. Demographics and characteristics of the study population

From 198 participants, 71.7% accepted the medical students to participate in pelvic examination. Fifty–seven percent of the participants in the acceptor group allowed both male and female medical students, while 42.9% in the same group allowed only female medical students. The correlation between patients’ acceptance and refusal for the medical students to participate in pelvic examination showed that age, occupation and parity of the patients were not statistically different (p> 0.05). Thirty–one percent of bachelor’s degree holders and 37% of postgraduate degree holders refused the participation of medical students. The higher the patient’s educational attainment, the more likely they are to refuse a pelvic examination performed by medical students, as observed in this study. In the conduct of pelvic exam procedure, 69.3% of the patients in the acceptor group and 53.6% of the patients in the non-acceptor group felt comfortable with students present as observers. Before the students participate in a pelvic examination, most of the participants (81.4%) preferred that the medical instructors (56.6%) ask permission rather than the medical students (21.7%) or nurses (21.7%).

This research recorded 52.7% of the patients with prior experience of physical examination by medical students from other departments in Naresuan University Hospital. Thirty–two percent of the patients previously received pelvic examination by medical students (from our department or other medical training hospitals). Patients who had their physical examinations performed by medical students from other departments or had pelvic examinations performed by medical students had a statistically significant increase in allowing medical students to participate in pelvic examinations under the supervision of a medical instructor (Table 2).

Table 2. The correlation between the acceptance of medical student participation in pelvic examination and prior experience of pelvic examination performed by medical students.

Table 3. The positive and negative attitudes of the participants about medical students’ participation in pelvic examination under the supervision of medical instructors.

IV. DISCUSSION

Our study demonstrated that 71.7% of the participants agreed to have their pelvic examinations performed by medical students. This result was comparable to the study of Western women that reported an acceptance rate ranging from 58 to 77% (Nicum & Karoo, 1998). Conversely, our acceptance rate was lower when compared to a study conducted in United Arab Emirates by Rizk et al. (2002), in which 87.1% of the out-patients in Ob-Gyn accepted the involvement of medical students. According to the results of our study, there were no statistically significant differences between acceptors and non-acceptors in terms of age, educational level, occupation, parity, or prior pelvic examination. Hartz and Beal (2000) also reported similar findings, stating that the age and education of the patients were not statistically different between the two groups.  However, Rizk et al. (2002) stated that the acceptance of the patients with older age, higher parity, and higher education was statistically significant. Interestingly, there was a trend discovered in our research that patients who are highly educated were more reluctant to allow medical students in performing pelvic examination, even when supervised by medical instructors. This reluctance could be because of a strong concern for their privacy, which should be investigated further through an in-depth interview.

Prior experience of the patients receiving physical examination by medical students from other departments, and prior experience of the patients receiving pelvic examination performed by medical students significantly increased the rate of acceptance. These findings are consistent with those of Ghobain et al. (2016) who reported that a positive prior experience with a medical student was significantly related to giving medical students permission to perform a physical examination. This can be explained by the fact that these patients were already aware of the medical student involvement in performing physical examinations. Therefore, they are more likely to accept medical student participation in subsequent Ob-Gyn consultations.

One interesting finding from our study was the positive patient receptivity of medical students acting as observers during pelvic examination. Patients in approximately 70% of the acceptor group allowed other medical students to observe the examination process. Remarkably, 53.6% in the non-acceptor group was comfortable with medical students observing a pelvic examination performed by medical instructor. This would imply that even if students cannot perform pelvic examinations, they can still gain clinical knowledge through observation, and clinical instructors can take advantage of this valuable opportunity to educate their students.

Other major concern of patients is the requirement for students to obtain permission before participating in any procedures. The majority of the participants prefer that medical instructors be in charge of obtaining patient permission to allow students to perform physical examinations on their behalf. This tendency may lead to a higher rate of acceptance of student participation.

The characteristics and performance of the medical students in our study had a significant impact on patients’ decision. The external part of the female reproductive organ is a sensitive and the most private area for every woman, and patients would only allow medical students who practice good hygiene and cleanliness to participate in the examination. Furthermore, the appropriate manner and demeanour, including respectfulness and politeness toward patients, and trustworthiness of the medical students during the clinical procedure may increase the patients’ receptivity of medical students to conduct pelvic examination. Professional appearance reflecting cultural backgrounds also impacts patient preference and acceptance; therefore, medical students should dress properly (Aljoudi et al., 2016).

Several studies, including ours, found that participants felt more at ease with female students than with male students (Salah et al., 2015; Subki et al., 2018). Chang et al. (2010) reported that male students were refused by patients to participate in clinical interviews and physical examinations, including pelvic examinations. In a study conducted at Taibah University in India (Shetty et al., 2021), women significantly preferred female students during abdominal and genital examinations. As a result, it is difficult for obstetrics and gynaecology educators to consider methods of encouraging patients to accept participation of medical students, regardless of their genders.

Patients’ positive attitudes toward medical students’ participation contributed to clinical teaching, which is an important process for professional development. The patients agreed that they would receive more accurate and comprehensive treatment if they had two pelvic examinations. These details are consistent with the findings of a study conducted by Rizk et al. (2002), which revealed that 69.7% of the participants were willing to contribute to the education of students. Most participants were satisfied with the overall service because they were well informed about their care by the health care team and were actively involved in the treatment decision-making process. Like other literature (Nicum & Karoo, 1998), medical students also provided patients with useful medical information and support. Furthermore, patients strongly supported the idea that real patient encounters and practices under clinical supervision are more effective methods for improving student’s clinical skills than just mere observation or skill laboratory practice (Subki et al., 2018).

Patients’ main reasons for refusing medical students’ participation in pelvic examination were concerns about breaching their confidentiality and privacy, which were similar to a study conducted in London. According to the findings, the common reasons for patients’ uneasiness with participation of the medical students were related to privacy, receiving lower standard of care, examinations, lack of control over the student’s level of involvement, and a longer consultation time (Ryder et al., 2005). The participants in our study were also concerned about taking a longer than usual routine visit. Not only applicable in Ob-Gyn department, patients’ perceptions of students’ professionalism and respect for privacy were significantly related to the acceptance of medical students’ participation in surgical ward (Ghobain et al., 2016). Thus, all medical students must be informed about the doctor-patient relationship and the importance of maintaining patient confidentiality. Medical instructors must inform and reassure patients about their confidentiality. Before participating in any clinical teachings, medical instructors should explain to students their roles and responsibilities. To minimise patients’ feeling of discomfort, these roles and responsibilities should be conveyed and explained to them before asking for their approval.

This paper has highlighted the significance of patients’ attitudes toward medical students’ manner and demeanor, which greatly influences patients’ decision-making process.  However, some limitations should be considered, such as the fact that all participants were Thai, and that the study was carried out in one of the Southeast Asia countries where data on the attitudes of female patients had not been thoroughly investigated. Since socio-demographic factors and cultural issues vary across Southeast Asia, the results of this research only represent the characteristics of the Thai population and not the entire region. Additionally, this cross-sectional study could not establish the reasons for patients’ negative attitudes toward student involvement in pelvic examination. Therefore, it is suggested that future research use in-depth interview methodology to gather more information from both the acceptor and non-acceptor groups.

V. CONCLUSION

Most of patients agreed to medical students participating in pelvic examinations and preferred medical instructors to be the persons to ask patients for permission. Essentially, patients’ confidentiality and privacy must always be safeguarded. The performance of medical students, and their observance of patient privacy and confidentiality are crucial factors in gaining the patient’s approval. Furthermore, the gender of the medical student influences the patient’s acceptance and comfort level in student’s involvement. Clinical instructors must effectively convince patients in gynaecology department to allow male medical students to perform gynaecologic procedures.

Notes on Contributors

Nisakorn Deesaen, Punyanuch Phonngoenchai, and Kongpop Sutantikorn contributed to the literature review, concept development, questionaire design, data collection, data analysis, and manuscript finalisation. Sakchai Chaiyamahapruk was involved in the study design, data analysis, and manuscript finalisation. Patcharada Amatyakul contributed to the literature review, concept development, study design, data analysis, and manuscript writing and finalisation.

Ethical Approval

Ethical approval for this study was obtained from the Naresuan University Institutional Review Board, Naresuan University, Thailand (Ethics approval number IRB 0653/60).

Data Availability

The data that support the findings of this study are publicly available in Open Science Framework repository, http://doi.org/10.17605/OSF.IO/HBV68.

Acknowledgement

The authors would like to express their gratitude to the nurses at the out-patient unit of the Department of Obstetrics and Gynaecology, Naresuan University Hospital, for their help in distributing and collecting questionnaires from the patients.

Funding

This research received no specific grant from any funding agency.

Declaration of Interest

The authors declare no conflict of interest.

References

Aljoudi, S. B., Alsolami, S. S., Farahat, F. M., Alsaywid, B., & Abuznadah, W. (2016). Patients’ attitudes towards the participation of medical students in clinical examination and care in Western Saudi Arabia. Journal of Family and Community Medicine, 23(3), 172‑178. https://doi.org/10.4103/2230-8229.189133

Amatyakul, P. (2021). Proposal for patients’ attitude and factors influencing the acceptance of medical students’ participation in pelvic examination. [Data set]. Open Science Framework. http://doi.org/10.17605/OSF.IO/HBV68

Anfinan, N., Alghunaim, N., Boker, A., Hussain, A., Almarstani, A., Basalamah, H., Sait, H., Arif, R., & Sait, K. (2014). Obstetric and gynecologic patients’ attitudes and perceptions toward medical students in Saudi Arabia. Oman Medical Journal, 29(2), 106-109. https://doi.org/10.5001/omj.2014.26

Bloomfield, H. E., Olson, A., Greer, N., Cantor, A., MacDonald, R., Rutks, I., & Wilt, T. J. (2014). Screening pelvic examinations in asymptomatic, average-risk adult women: An evidence report for a clinical practice guideline from the American College of Physicians. Annals of Internal Medicine, 161(1), 46–53. https://doi.org/10.7326/M13-2881

Chang, J. C., Odrobina, M. R., & McIntyre-Seltman, K. (2010). The effect of student gender on the obstetrics and gynecology clerkship experience. Journal of Women’s Health, 19(1), 87-92. https://doi.org/10.1089/jwh.2009.1357

Fortier, A. M., Hahn, P. M., Trueman, J., & Reid, R. L. (2006). The acceptance of medical students by women with gynaecology appointments. Journal of Obstetrics and Gynaecology Canada, 28(6), 526-530. https://doi.org/10.1016/S1701-2163(16)32179-X

Ghobain, M. A., Alghamdi, A., Arab, A., Alaem, N., Aldress, T., & Ruhyiem, M. (2016). Patients’ perceptions towards the participation of medical students in their care. Sultan Qaboos University Medical Journal, 16(2), 224–229. https://doi.org/10.18295/squmj.2016.16.02.014

Hartz, M. B., & Beal, J. R. (2000). Patients’ attitudes and comfort levels regarding medical students’ involvement in Obstetrics–Gynecology outpatient clinics. Academic Medicine, 75(10), 1010-1014. https://doi.org/10.1097/00001888-200010000-00018 

McLean, M., Al Ahbabi, S., Al Ameri, M., Al Mansoori, M., Al Yahyaei, F., & Bernsen, R. (2010). Muslim women and medical students in the clinical encounter. Medical Education, 44(3), 306-315. https://doi.org/10.1111/j.1365-2923.2009.03599.x.

Nicum, R., & Karoo, R. (1998). Expectations and opinions of pregnant women about medical students being involved in care at the time of delivery. Medical Education, 32(3), 320-324. https://doi.org/10.1046/j.1365-2923.1998.00205.x.

Rizk, D. E. E., Al-Shebah, A., El-Zubeir, M. A., Thomas, L. B., Hassan, M. Y., & Ezimokhai, M. (2002). Women’s perceptions of and experiences with medical student involvement in outpatient obstetric and gynecologic care in the United Arab Emirates. American Journal of Obstetrics and Gynecololgy, 187(4), 1091-1100. https://doi.org/10.1067/mob.2002.126284.

Ryder, N., Ivens, D., & Sabin, C. (2005). The attitude of patients towards medical students in a sexual health clinic. Sexually Transmitted Infection, 81(5), 437–439. https://doi.org/10.1136/sti.2004.014332.

Salah, A. B., Mhamdi, S. E., Bouanene, I., Sriha, A., & Soltani, M. (2015). Patients’ attitude towards bedside teaching in Tunisia. International Journal of Medical Education, 6, 201-207. https://doi.org/10.5116/ijme.5669.ea24.

Shetty, P. A., Magazine, R., & Chogtu, B. (2021). Patient outlook on bedside teaching in a medical school. Journal of Taibah University Medical Sciences, 16(1), 50-56. https://doi.org/10.1016/j.jtumed.2020.10.002

Subki, A. H., Algethami, M. R., Addas, F. A., Alnefaie, M. N., Hindi, M. M., & Abduljabbar, H. S. (2018). Women’s perception and attitude to medical students’ participation in obstetrics and gynecology care. Saudi Medical Journal, 39(9), 902-909.  https://doi.org/10.15537/smj.2018.9.22668

Teunissen, P. W. (2018). An inconvenient discussion. Medical education, 52(11), 1104-1110. https://doi.org/10.1111/medu.13689

*Patcharada Amatyakul
Department of Obstetrics and Gynaecology,
Faculty of Medicine, Naresuan University,
99 Thaphoe District, Muang,
Phitsanulok, 65000 Thailand
Tel: 66-86-397-3455
Email: pamatyakul@hotmail.com

Submitted: 3 June 2021
Accepted: 4 October 2021
Published online: 4 January, TAPS 2022, 7(1), 76-86
https://doi.org/10.29060/TAPS.2022-7-1/OA2547

Nadia Greviana^1,2, Dewi Anggraeni Kusumoningrum², Ardi Findyartini^1,2, Chaina Hanum¹ & Garry Soloan^1,3

¹Medical Education Center, Indonesian Medical Education & Research Institute (IMERI) Faculty of Medicine Universitas Indonesia, Jakarta, Indonesia; ²Department of Medical Education, Faculty of Medicine Universitas Indonesia, Jakarta, Indonesia; ³Undergraduate Program in Medicine, Faculty of Medicine Universitas Indonesia, Jakarta, Indonesia

Abstract

Introduction: As significant autonomy is given in a Massive Open Online Course (MOOC), online self-regulated learning (SRL) ability is crucial in such courses. We aim to measure the online SRL abilities of early-career medical doctors enrolled in a MOOC.

Methods: We performed a cross-sectional study using the Self-Regulated Online Learning Questionnaire-revised version (SOL-Qr). We conducted a three-stage cross-cultural validation of the SOL-Qr, followed by Confirmatory Factor Analysis (CFA). The online SRL ability of 5,432 medical doctors enrolled in a MOOC was measured using the validated SOL-Qr.

Results: The CFA of the cross-translated SOL-Qr confirmed its comparability to the original version, with excellent validity & reliability. Participants showed high levels of online SRL during their early careers. Despite high online SRL scores, MOOC completion rate was low. Male participants showed slightly better time management ability than female participants. Participants working in the primary epicentrum for COVID-19 in the country showed lower online SRL scores, while participants who graduated from higher accreditation levels showed better time management ability.

Conclusion: The SOL-Qr and its subscales are suitable and valid for measuring the online SRL abilities of medical doctors in a MOOC during their early-career period. Time management ability was associated with previous experience during the medical education period, while other online SRL subscales were mostly associated with workload. However, as the scores did not correlate with the time spent for learning in MOOC, the corresponding learning effort or time spent may be beyond just the commitment to the described MOOC.

Keywords:           Self-Regulated Learning, MOOC, Online Learning

Practice Highlights

• It is important to take into account learners’ online SRL ability in MOOCs as it is dynamic across online learning contexts.
• The use of the SOL-Qr is beneficial for providing learners’ online SRL profiles in MOOC during medical doctors’ early career period.
• Understanding online SRL abilities helps MOOC developers to evaluate learning activities in MOOC and support learners’ online SRL ability.

I. INTRODUCTION

Massive Open Online Courses (MOOCs) are open academic platforms in which students can access learning resources interactively. The self-paced nature of MOOCs provides time-flexibility, facilitating deeper learning (Bullock & De Jong, 2013). MOOCs are useful for replacing direct interaction for knowledge transfer and learning processes during the ongoing pandemic because they utilise various formats, such as video lectures, reading resources, assignments, tests, and asynchronous discussion within the platform (Sandars & Patel, 2020). Because MOOC’s aim to give learners useful resources and empower teachers to provide vital knowledge, curation of the platform, with its copious information, it is beneficial for achieving high-quality content that fits the learning objectives and learners’ characteristics (Asarbakhsh & Sandars, 2013). As demand for technological solutions in education rise during the COVID-19 pandemic, MOOCs have been promoted as forms of disruption that accelerate adaptation to balance safety with the achievement of competencies by medical students and graduates (Hall et al., 2020; Liang et al., 2020).

MOOCs have generally been designed as open in access, location, pace, and time of completion; therefore, learners must control their learning process. Learning goals are usually set less strictly in MOOCs than in other courses. Unlike traditional, face-to-face teaching, MOOCs require highly engaged & strategic students. Learners must plan their study, set goals, evaluate their knowledge related to the course material, adapt their learning strategies, and assess their performance. They are solely responsible for managing their time and study environment (Jansen et al., 2017).

With high enrollment rates, the majority of learners fail to complete MOOCs, for various reasons: lack of time, insufficient prior knowledge, inadequate supervision, and difficulties in understanding the course materials (Hew & Cheung, 2014). Time management, effort regulation, metacognition, and critical thinking as part of Self-Regulated Learning (SRL) relate to success in online academic activities (Broadbent & Poon, 2015). Because MOOCs give students significant autonomy in completing the course, SRL is crucial for successful completion of MOOCs (Chung, 2015; Wong et al., 2018).

SRL encompasses a student’s ability to actively and constructively control their thoughts, acts, and emotions towards learning objectives (Jouhari et al., 2015), using the cycle of forethought, performance, and self-reflection phases, which should be implemented in an online learning environment (Panadero, 2017). Various external factors may interplay and affect self-regulated learning among students. This underscores the importance of a supportive family, helpful peers, and motivational, feedback-centered instruction methods; together, these factors support SRL (Jouhari et al., 2015).

Virtanen and Nevgi (2010) recognised gender as a factor affecting how SRL is perceived by students, especially during the forethought stage; they found that male students scored slightly higher on the sub-scale for self-efficacy, while female students demonstrated greater help-seeking strategies, performance anxiety, and beliefs in the value of studying. Bembenutty (2009) also found that female students perceive learning as a more valuable task in SRL than male students do.

Several instruments were developed to measure SRL, including structured interviews like the Self-Regulation Interview Schedule (SRLIS), questionnaires, teachers’ judgments, think-aloud techniques, and performance observations (Magno, 2011). The Motivated Strategies for Learning Questionnaire (MSLQ) is a common measure of SRL. It assesses two aspects: motivational orientation—encompassing values, expectations, and affective components—and the use of different learning strategies, e.g. cognitive and metacognitive strategies or resource management strategies (Pintrich et al., 1993). Another instrument for measuring students’ SRL in higher education is the Academic Self-Regulated Learning Scale (A-SRL-S), evaluating memory strategy, goal-setting, self-evaluation, seeking assistance, environmental structuring, learning responsibility, and planning and organising (Magno, 2011).

In the context of online learning, several instruments have been developed to measure students’ SRL, such as the Online Self-regulated Learning Questionnaire (OSLQ) and the Self-regulated Online Learning Questionnaire (SOL-Q). The OSLQ consists of six subscales: environment structuring, goal-setting, time management, help-seeking, task strategies, and self-evaluation (Barnard et al., 2009). However, this questionnaire cannot measure SRL activities in the appraisal or self-reflection phase. Meanwhile, the SOL-Q was developed by Jansen et al. (2017), using other existing SRL questionnaires to measure SRL activities, specifically in MOOCs, for all three SRL phases: preparatory, performance, and appraisal. The SOL-Q consists of five sub-scales: metacognitive skills, environmental structuring, help-seeking, time management, and persistence (Jansen et al., 2017). In 2018, a revision was made to split the metacognitive skills scale into three subscales: metacognitive activities before, during, and after a learning task. The revised questionnaire (SOL-Qr) demonstrated improved validity, usability, and reliability (Jansen et al., 2018).

Recognising the importance of learners’ during the use of MOOCs—and that SRL is affected by various factors—we attempted to measure the online self-regulated learning of early-career medical doctors enrolled in a MOOC that provides essential knowledge about COVID-19 to support medical doctors’ early careers during the COVID-19 pandemic. We seek to answer the following questions in this study:

1. Is the Self-Regulated Online Learning Question (SOL-Qr) valid for use in our setting?
2. What is the profile of students’ SRL scores, and are there any relationships between the SRL score, course completion, gender, respondents’ former medical school, and internship location

 

II. METHODS

A. Context

With approximately 270 million inhabitants across 34 provinces, Indonesia is one of the largest archipelagos in the world. According to a recent report, there is one medical doctor per 2500 people across the country (National Ministry of Health, 2020). Recently graduated medical doctors must undergo a one-year compulsory internship program upon graduation, where they serve as front-liners in primary health care settings across the country to serve in societies in very diverse sociocultural contexts and ethnicities. Those who graduated in 2020, mostly finished high school and entered medical schools in 2013–2014, completed their clinical stages and graduated from different medical schools in Indonesia at the beginning of the COVID-19 pandemic, during which some of the clinical duties in teaching hospitals were suspended and amended for safety reasons (Findyartini et al., 2020). A total of 93 public and private medical schools are distributed across Indonesia, all accredited by the National Accreditation Agency for Higher Education in Health with A-level as the highest accreditation indicating that the medical school has reached an established quality in teaching-learning processes and faculty members.

Considering the need for the newly graduated medical doctors serving as front liners to learn about the current pandemic situation and the importance of safety for both patients and healthcare workers, the Ministry of Health equipped these new graduates with essential COVID-19 knowledge before their involvement in patient management during the ongoing pandemic. Given the geographical reality of the country’s archipelago and the nature of the workplace setting during the internship program, the use of a Massive Open Online Course (MOOC) was preferred.

Little was known about COVID-19 at the beginning of the pandemic. Guidelines created by organisations (such as the CDC and WHO) are mostly amended living documents based on newly published articles, data, and clinical expertise. Studies on COVID-19 are being conducted on a massive scale worldwide, which may create information overload and overwhelm learners, especially those serving as front–liners (Poonia & Rajasekaran, 2020). A MOOC approach would facilitate a prompt response for equipping medical and health students, professionals, and the broader community to learn about the pandemic situation and prepare them to contribute to the pandemic response in community and hospital settings (Ortiz-Martìnez et al., 2021).

B. The COVID-19 MOOC Description

All internship doctors in Indonesia batch 3 and 4 in 2020, were compulsorily enrolled in an open course on COVID-19 at the beginning of their internship period. The open course was sequenced into two, sequentially accessed parts. The mandatory part A consists of fundamental knowledge on COVID-19 (such as COVID-19 screening, triage, infection control, management of patients, preventive strategy, etc); Part B consists of supplemental knowledge about COVID-19 (such as management of patients with comorbidity, ethics, and medicolegal aspects, perioperative management, etc); it is optional for participants to choose which topics to learn based on their interests and needs. Participants were encouraged to complete Part A during the first two weeks of their internship program to ensure sufficient knowledge before their service. However, participants were given full access to revisit the MOOC for up to 9–12 months of their internship programs. More details on the online course are reported elsewhere (Findyartini et al., 2021).

C. Study Design and Instruments

Our cross-sectional study uses the SOL-Qr (Jansen et al., 2018), which was adapted to Bahasa Indonesia, to assess online self-regulated learning ability in a Massive Open Online Course. Secondary data were obtained from the Moodle-based MOOC platform, including the total number of respondents, gender, internship location, former medical school, and course completion.

SOL-Qr measures seven aspects of SRL as mentioned in Table 1. Respondents answer each item on a 7-point Likert scale (1 for “not at all true for me” and 7 for “very true for me”). This questionnaire was translated to Bahasa Indonesia and back-translated to English to ensure similarity in meaning. Cognitive interviews with four respondents resembling the study participants were also conducted to obtain clarity of the items. The two respondents in this phase were final year medical school students from the authors’ institution (prospective participants of the national internship program) and the other two respondents were medical doctors who have just completed the national internship program.

Table 1. Descriptions of SOL-Qr (Jansen et al., 2018)

Course completion, as analysed in this study, refers to the completion rate (in percentage) of the optional topics in part B that were accessed and completed by study participants in the open course.

D. Data Collection

Data were collected from the Moodle-based MOOC platform of the COVID-19 module from January to February 2021, two months after each course had started. This study uses a total sampling approach, with a minimum sample size of 204 respondents, calculated from standard deviation of 3.43 (Yen et al., 2016) and alpha (type I error) of 0.05 and beta (type II error) of 0.20 and 10% estimated drop out level.

All study participants were National Internship Medical Doctors in the year 2020 who were enrolled in the COVID-19 Open Course on the MOOC platform.

The SOL-Qr questionnaire was embedded in the evaluation questionnaire placed at the end of Part A, before participants moved forward to Part B. The questionnaire was completed voluntarily by participants who agreed to participate in this study. They were given adequate written information about the study and assured that there were no consequences of participation in regards to the course or the internship program evaluation. All data included in this study were collected from participants who signed and agreed upon the written consent embedded in the questionnaire. This study obtained ethical clearance from Faculty of Medicine Universitas Indonesia/ dr. Cipto Mangunkusumo General Hospital Research Ethics Committee Board (KET-1395/UN2.F1/ETIK/PPM.00.02/2020) in 2020.

E. Data Analysis

We conducted a three-stage validation process for the SOL-Qr, including the process of translation to Bahasa Indonesia by the research investigator, the back-translation process, and a review process by four newly graduated medical doctors who represented the characteristics of the study participants. This process ensured that the Bahasa Indonesia version of SOL-Qr was comparable to the original version. Furthermore, a Confirmatory Factor Analysis (CFA) was performed to confirm the model proposed by Jansen et al. (2018) as a fit model in the Bahasa Indonesia version compared to the original version. The fit model of CFA analysis determined whether normality, multicollinearity, residual values, and multivariate outliers were met. Furthermore, the Two-Index Presentation Strategy, the fit index combination of at least two indicators among the root mean square error of approximation (RMSEA), the standardised root mean square residual (SRMR), and the comparative fit index (CFI) all indicated the fit model of CFA analysis (Hu & Bentler, 1999; Schreiber et al., 2006). Internal consistency analysis of the Bahasa Indonesia version of SOL-Qr was also completed. Items were concluded as valid if the corrected item-total item correlation value was higher than 0.3. The questionnaire was considered reliable if alpha Cronbach ≥ 0.700; an alpha value higher than 0.900 indicates excellent internal consistency (Blunch, 2008).

All survey data obtained from the questionnaire were statistically analysed using IBM SPSS Statistic version 21. Demographic data were processed using descriptive statistics (proportion, mean, and standard deviation). Study participants were classified into two groups according to their internship location:

• Medical doctors who serve in Java- Bali Region, which was the primary epicenter of COVID-19 in the country in 2020.
• Medical doctors who serve in Outside Java- Bali Region.

We also classified participants according to their former medical school accreditation.

Bivariate analysis using the t-independent test was used to find relationships between the online self-regulated learning scores and gender, internship location, the former medical school, and the course completion. The Pearson correlation test was also used to analyse correlations between online SRL subscales and the course completion rate.

III. RESULTS

A. Validation of the SOL-Qr

The three stages of validation were conducted in the SOL-Qr instrument to ensure the content validity of the Bahasa Indonesia version of SOL-Qr. CFA was performed on the Bahasa Indonesia version of SOL-Qr, with the results showing the goodness-of-fit according to Hu and Bentler’s Two Index Presentation; the SRMR value was 0.056 (<0.08) while RMSEA value was 0.078 (<0.08). Meanwhile, the χ2/df value was < 0.001; the CFI value was 0.874 (Hu & Bentler, 1999). The model (Figure 1) also confirms the comparability of the subscales to the original SOL-Qr.

The Bahasa Indonesia version of SOL-Qr also shows excellent validity and reliability, with a Cronbach’s alpha of 0.974. The reliability of each subscale ranges from 0.971 to 0.975.

Figure 1. CFA results of the National Language Version of SOL-Qr (MB: Metacognitive Before, MD: Metacognitive During, MA: Metacognitive After, TM: Time Management, ES: Environmental Structuring, P: Persistence, and HS: Help Seeking)

B. Profile of Participants’ Online SRL Scores

A total of 5,846 internship doctors from all 34 provinces in Indonesia were enrolled and accessed the MOOC; as many as 5,432 participants, graduated from 74 medical schools (of which 46% are A-accredited while 54% are B-accredited), agreed to participate in the study (response rate of 92.9%). Details on the study participants are presented in Table 2.

Table 2. Study participants (N = 5,432)

C. Relationship Between Online SRL Score, Course Completion, Gender, and Internship Location

Data on the online SRL scores and course completion were analysed according to gender, internship location, and medical school accreditation. The data are described in means and standard deviations, as they were normally distributed, as shown in Table 3.

According to Table 3, the average total scores of participants’ online self-regulated learning in all groups show high levels of online SRL (SRL score > 5). When comparing the male and female participants, the finding suggests that only the Time Management subscale shows a significant difference (p = 0.001). Male participants show higher scores in time management than female participants. Participants from the Outside Java-Bali Region had significantly higher online SRL scores in all subscales, except for the Persistence subscale (p = 0.181), than participants from the Java-Bali Region, which was the primary epicenter of COVID-19 in the country.

Meanwhile, the online SRL scores of participants who graduated from A-level versus B-level accredited medical schools mostly show similar online self-regulated learning scores, except in the time management scale (p = 0.009).

Although Table 3 shows no significant difference regarding course completion across gender, internship region, or former internship location, it does show a low course completion rate. Participants only completed less than approximately 40% of all optional topics in Part B.

The results of the Pearson correlation test show significant differences between the course completion rate and several SRL subscales, with very low correlation values (r < 0.1 in all subscales) for the SOL-Qr in the whole samples, as well as between the internship location and former medical education groups. Only the Time Management score was correlated to Course Completion in the Outside Java-Bali group, with a very low correlation value (r = 0.102). Therefore, the course completion rate does not correlate with the online SRL scores.

IV. DISCUSSION

Self-regulated learning is a dynamic process and may change while learners undergo various learning contexts (Barnard et al., 2009). The rapidly changing pandemic has accelerated the adaptation of new learning approaches and methods worldwide, including MOOCs that had gained popularity. Therefore, the use of the SOL-Qr is beneficial for providing MOOC learners’ SRL profiles (Jansen et al., 2018).

This study represents the first successful attempt to cross-culturally validate the SOL-Qr and determine the suitability of all its subscales for profiling online SRL of medical doctors in their early career. The three-stage validation process for the SOL-Qr was conducted as a form of cross-cultural adaptation of the instrument to facilitate its use in measuring self-regulated learning abilities in an online setting in the context of medical education—specifically in the MOOC used for continuing medical education and professional development programs for early-career, newly graduated medical doctors (Hambleton, 2005). The CFA conducted in this study also demonstrates a good fit, with excellent internal consistencies of the SOL-Qr and its subscales; this demonstrates the comparability of the instrument with the original SOL-Qr (Jansen et al., 2018).

The use of the SOL-Qr in this study demonstrates online SRL abilities during the transition phase in the early careers of medical doctors, from medical students to medical interns. This study demonstrates the high average level of online SRL ability among newly graduated medical doctors (SRL score > 5). Participants’ online SRL may increase due to the positive online learning experience obtained, considering that the knowledge on COVID-19 provided in this particular MOOC was vital and timely knowledge for these recent graduates. While participant perceptions of MOOCs have been reported elsewhere, it is known that positive perceptions of an online learning experience and environments are significantly correlated with the online SRL score (Abouzeid et al., 2021; Findyartini et al., 2020; Liaw & Huang, 2013).

The high scores for online SRL in our study were found in total and in most of the subscales, except for the metacognitive skills before learning (MB) scale in participants from the Java-Bali region. A similar finding on the low level of metacognitive ability was reported during the transition phase from preclinical to clinical learning, which was associated with previous clinical experience (Cho et al., 2017). Albeit being assessed using a different instrument, the decreased level of metacognitive skill in this study may also be affected by a similar factor. Since the Java-Bali Region had been the primary epicenter of COVID-19 in the country, with the greatest number of COVID-19 cases (69.41%) out of all other regions (National Ministry of Health, 2021), most interns in the Java-Bali Region may have experienced being removed from their duties at the beginning of the pandemic for safety reasons during their last clinical rotations as medical students (Findyartini et al., 2020). This may have caused discomfort because they felt useless and unable to contribute to patient care, further affecting their sense of competence and motivation, despite previous clinical experiences. Hence, this may have affected the process of goal-setting and reflecting on their prior knowledge during the transition to becoming medical interns (Dornan et al., 2014; Dubé et al., 2015).

Findings of this study demonstrate no significant correlation between online SRL and the number of optional topics achieved by participants. This result was supported by the fact that, despite the high level of online SRL ability depicted in this study, the duration of which participants accessed the compulsory Part A was lower than the expected minimum duration as estimated by course developers although we did not explore whether participants learned about COVID-19 from any other learning resources (Findyartini et al., 2021). The current study also supports the results of a previous study with similar findings regarding online SRL scores and academic achievement, although this study does report a significant relationship between online discussion and academic achievement (Abouzeid et al., 2021). The MOOC examined in this study does not provide two-way interactions among learners or between learners and instructors, which may affect the low number of optional courses completed.

Our study also shows that the online SRL scores of participants in the Outside Java-Bali Region were significantly higher (p < 0.001) across all scales than those of participants in the Java-Bali Region, except on the Persistence scale (p = 0.181). This suggests that participants in the Outside Java-Bali Region also have better time management, environmental structuring, and help-seeking abilities, which may relate to the workload at the internship locations. Participants in the Java-Bali Region face a higher workload and stress as front-liners in managing patients during the COVID-19 pandemic (as health educators, contact tracers, vaccinators, etc.). This aligns with other findings suggesting that an optimal workload determines the quality of students’ SRL in the early transition and adaptation phases (Barbosa et al., 2018).

Furthermore, the MOOC was given to the participants at the beginning of their internship programs, in which several orientations and patient management also took place. Hence, it is confirmed that the participants’ workload affected the time allocated to learn the MOOC (Eriksson et al., 2017). This study also suggests that the use of the MOOC for knowledge provision would preferably occur with sufficient time before immersion in other workplace-based learning experiences.

Hew and Cheung (2014) report few challenges in MOOC completion, including a lack of time and having other priorities to fulfill the course; therefore, time management ability remains crucial. The results of this study indicate that male participants had slightly better time management abilities than female participants. Although the absolute difference of the scores between groups is small, this study involved 5,432 respondents (much more than the minimum sample size), thus small differences in results can be statistically significant. As newly graduated doctors, study participants were interns who worked in a new environment hence new challenges regarding workplace relationships and workload may be faced. Previous studies show that female and male physicians may perceive these challenges differently (Babaria et al., 2009). Female physicians report that they feel uncertain and stressed when facing different clinical environments. Because they tend to need more time to adapt than male physicians (Malau-Aduli et al., 2020), completing a MOOC may not be their priority. Moreover, with masculinity/femininity level of cultural determinant, Indonesians have a clear cut between gender roles, thus affecting roles of female participants in most settings (Mangundjaya, 2013). Female participants may culturally face different roles in their personal lives, such as the expectation to perform housework and childcare, in addition to their internship obligation resulting in conflicting time and higher stress levels. The conflicting time and higher stress level in both work and personal life (Isaac et al., 2013) may influence female participants’ ability to commit time to learn and work and may explain the lower time management scores among female participants.

This study also highlights that, overall, the online SRL scores of participants graduating from the A-level and B-level of accredited medical schools show similar results, except on the time management scale (p = 0.009), where the participants from the higher level accredited schools show higher scores than the participants from the lower level accredited schools. Prior online learning experience has been reported as an important determinant of online learning success (Vilkova, 2019). For newly graduated doctors completing an internship program, prior online learning experience may be largely attributed to the use of online learning formally in their medical school’s curriculum. Our study suggests that the higher accredited medical schools may provide more online learning experiences, leading to better time management skills among the participants from this group.  Furthermore, MOOC completion and the fulfillment of learning outcomes were determined by the forethought phase in the students’ SRL; thus, the goal-setting, self-efficacy, and task values should be emphasised by the participants and facilitated by the MOOC (Vilkova, 2019). Our study further indicates that time management in the use of the MOOC should be considered by learners in the forethought, performance, and self-reflection phases of their SRL; the planning stage of the MOOC development must determine the estimated completion time for the whole course and its sections about the course learning outcomes and the participant’s characteristics (Stracke et al., 2018).

Similar to the SRL ability in offline learning, our findings further imply the importance of accounting for learners’ online SRL abilities, which are dynamic across online learning contexts, including MOOCs. Certain characteristics of MOOCs, such as their open access and self-paced nature, stress the importance of online SRL ability, especially for MOOCs used in the transition phase in the early career of medicine. Therefore, using validated and reliable instruments, such as the SOL-Qr, to measure the online SRL abilities of MOOC participants would help course developers to identify whether the online learning context supports or hinders learners’ SRL abilities, thus helping course administrators further improve MOOC to provide further support for learners’ SRL (Barnard et al., 2009; Sandars & Patel, 2020).

V. LIMITATIONS OF THE STUDY

We identify several limitations of this study. As this study examined online SRL ability of participants using a self-reported questionnaire, it was unable to examine the actual online SRL ability, given the poor correlation with the course completion rate. Furthermore, as the evaluation of online SRL was only conducted once, it was not possible to observe changes in SRL ability throughout the use of the MOOC or in a longer period. With the data that we obtained, we were also unable to analyse whether participants used other online resources to learn about COVID-19 before the internship program or parallel with accessing the provided module as the marker of online SRL nor to explain any causal relationships between online SRL scores and the factors under investigation. However, measurement of online SRL ability using the cross-culturally validated SOL-Qr reveals that this instrument can be used for MOOCs on continuing medical education and professional development in the early-career context. 

VI. CONCLUSION

Our study demonstrates the cross-cultural validity of the SOL-Qr and the suitability of its subscales for use in the medical and health fields. The results portray the online SRL ability of medical doctors as participants in a MOOC during the transition phase in their early career. We found that the time management ability was associated with previous experience during the medical education period, while other subscales were mostly associated with workload during the transition phase. However, as the scores did not correlate with the completion rate of MOOC, it can be concluded that the questionnaire is a possible valid tool to assess self-regulated learning in the MOOC environment. Yet, the corresponding learning effort or completion rate may be beyond just the commitment to the described MOOC.

Notes on Contributors

NG designed and led the study, led data collection and analysis, and led manuscript development. AF, DAK, and CH contributed in data collection, completed data analysis, and contributed to manuscript development.  GS contributed in the data analysis and manuscript development. All authors approved the final version of the manuscript. 

Ethical Approval

This study obtained ethical clearance from Research Ethics Committee of Faculty of Medicine Universitas Indonesia/ dr. Cipto Mangunkusumo General Hospital (KET-1395/UN2.F1/ETIK/PPM.00.02/2020) in 2020.

Data Availability

Data will be available upon request to corresponding author due to conditions of consent provided by respondents in this study and that it should abide data sharing policy from the authors’ institution and the Republic of Indonesia Ministry of Health.

Acknowledgement

The authors would like to acknowledge the Ministry of Health Republic of Indonesia for the trust that has been given to develop and organise the MOOC for the national internship program participants. We would also like to thank all national internship program participants for participation in this study and to Vernonia Yora Saki for assisting the authors with statistical analysis of the study. The preliminary results of this study were presented in Niigata Meeting 2020.

Funding

The development of MOOC and data analysis has been supported by the Ministry of Health Republic of Indonesia through a direct appointment decree to our institution. 

Declaration of Interest

All authors state no possible conflicts of interest, including financial, consultant, institutional, and other relationships that might lead to bias or a conflict of interest.

References

Abouzeid, E., O’Rourke, R., El-Wazir, Y., Hassan, N., Ra’oof, R. A., & Roberts, T. (2021). Interactions between learner’s beliefs, behaviour and environment in online learning: Path analysis. The Asia Pacific Scholar, 6(2), 38-47. https://doi.org/10.29060/TAPS.2021-6-2/OA2338

Asarbakhsh, M., & Sandars, J. (2013). E-learning: The essential usability perspective. The Clinical Teacher, 10(1), 47-50. https://doi.org/10.1111/j.1743-498X.2012.00627.x

Babaria, P., Abedin, S., & Nunez-Smith, M. (2009). The effect of gender on the clinical clerkship experiences of female medical students: Results from a qualitative study. Academic Medicine, 84(7), 859-866. https://doi.org/10.1097/ACM.0b013e3181a8130c

Barbosa, J., Silva, Á., Ferreira, M. A., & Severo, M. (2018). Do reciprocal relationships between academic workload and self-regulated learning predict medical freshmen’s achievement? A longitudinal study on the educational transition from secondary school to medical school. Advances in Health Sciences Education, 23, 733-748. https://doi.org/10.1007/s10459-018-9825-2

Barnard, L., Lan, W. Y., To, Y. M., Paton, V. O., & Lai, S.-L. (2009). Measuring self-regulation in online and blended learning environments. The Internet and Higher Education, 12(1), 1-6. https://doi.org/10.1016/j.iheduc.2008.10.005

Bembenutty, H. (2009). Academic delay of gratification, self-regulation of learning, gender differences, and expectancy-value. Personality and Individual Differences, 46(3), 347-352. https://doi.org/10.1016/j.paid.2008.10.028

Blunch, N. J. (2008). Introduction to structural equation modelling using SPSS and AMOS. SAGE.

Broadbent, J., & Poon, W. L. (2015). Self-regulated learning strategies & academic achievement in online higher education learning environments: A systematic review. The Internet and Higher Education, 27, 1–13. https://doi.org/10.1016/j.iheduc.2015.04.007

Bullock, A., & De Jong, P. G. M. (2013). Technology-enhanced learning. In T. Swanwick (Ed.), Understanding medical education: Evidence, theory, and practice (2nd ed., pp. 149-160). Wiley Blackwell. https://doi.org/10.1002/9781118472361.ch11

Cho, K. K., Marjadi, B., Langendyk, V., & Hu, W. (2017). Medical student changes in self-regulated learning during the transition to the clinical environment. BMC Medical Education, 17(59). https://doi.org/10.1186/s12909-017-0902-7

Chung, L.-Y. (2015). Exploring the effectiveness of self-regulated learning in massive open online courses on non-native English speakers. International Journal of Distance Education Technologies, 13(3), 61-73. https://doi.org/10.4018/ijdet.2015070105

Dornan, T., Tan, N., Boshuizen, H., Gick, R., Isba, R., Mann, K., Scherpbier, A., Spencer, J., & Timmins, E. (2014). How and what do medical students learn in clerkships? Experience based learning (ExBL). Advances in Health Sciences Education, 19(5), 721-749. https://doi.org/10.1007/s10459-014-9501-0

Dubé, T. V., Schinke, R. J., Strasser, R., Couper, I., & Lightfoot, N. E. (2015). Transition processes through a longitudinal integrated clerkship: A qualitative study of medical students’ experiences. Medical Education, 49(10), 1028-1037. https://doi.org/10.1111/medu.12797

Eriksson, T., Adawi, T., & Stöhr, C. (2017). “Time is the bottleneck”: A qualitative study exploring why learners drop out of MOOCs. Journal of Computing in Higher Education, 29, 133-146. https://doi.org/10.1007/s12528-016-9127-8

Findyartini, A., Anggraeni, D., Husin, J. M., & Greviana, N. (2020). Exploring medical students’ professional identity formation through written reflections during the COVID-19 pandemic. Journal of Public Health Research, 9(s1). https://doi.org/10.4081/jphr.2020.1918

Findyartini, A., Greviana, N., Hanum, C., Husin, J. M., Sudarsono, N. C., Krisnamurti, D. G. B., & Rahadiani, P. (2021). Supporting newly graduate medical doctors in managing COVID-19: An evaluation of Massive Open Online Course in a limited-resource setting. PLoS ONE, 16(9), e0257039. https://doi.org/10.1371/journal.pone.0257039

Hall, A. K., Nousiainen, M. T., Campisi, P., Dagnone, J. D., Frank, J. R., Kroeker, K. I., Brzezina, S., Purdy, E., & Oswald, A. (2020). Training disrupted: Practical tips for supporting competency-based medical education during the COVID-19 pandemic. Medical Teacher, 42(7), 756-761. https://doi.org/10.1080/0142159X.2020.1766669

Hambleton, R. K. (2005). Issues, designs, and technical guidelines for adapting tests into multiple languages and cultures. In R. K. Hambleton, P. F. Merenda, & C. D. Spielberger (Eds.), Adapting educational and psychological tests for cross-cultural assessment (pp. 3–38). Lawrence Erlbaum.

Hew, K. F., & Cheung, W. S. (2014). Students’ and instructors’ use of massive open online courses (MOOCs): Motivations and challenges. Educational Research Review, 12, 45–58. https://doi.org/10.1016/j.edurev.2014.05.001

Hu, L.-t., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. https://doi.org/10.1080/10705519909540118

Isaac, C., Petrashek, K., Steiner, M., Manwell, L. B., Carnes, M., & Byars-Winston, A. (2013). Male spouses of women physicians: Communication, compromise, and carving out time. The Qualitative Report, 18(52), 1-12. https://doi.org/10.46743/2160-3715/2013.1423

Jansen, R. S., van Leeuwen, A., Janssen, J., & Kester, L. (2018). Validation of the revised self-regulated online learning questionnaire. In V. Pammer-Schindler, M. Pérez-Sanagustín, H. Drachsler, R. Elferink, & M. Scheffel (Eds.), Lifelong Technology-Enhanced Learning (pp. 116-121). Springer. https://doi.org/10.1007/978-3-319-98572-5_9

Jansen, R. S., van Leeuwen, A., Janssen, J., Kester, L., & Kalz, M. (2017). Validation of the self-regulated online learning questionnaire. Journal of Computing in Higher Education, 29, 6-27. https://doi.org/10.1007/s12528-016-9125-x

Jouhari, Z., Haghani, F., & Changiz, T. (2015). Factors affecting self-regulated learning in medical students: A qualitative study. Medical Education Online, 20(1). https://doi.org/10.3402/meo.v20.28694

Liang, Z. C., Ooi, S. B. S., & Wang, W. (2020). Pandemics and their impact on medical training: Lessons from Singapore. Academic Medicine, 95(9), 1359-1361. https://doi.org/10.1097/ACM.0000000000003441

Liaw, S.–S., & Huang, H.–M. (2013). Perceived satisfaction, perceived usefulness and interactive learning environments as predictors to self-regulation in e-Learning environments. Computers & Education, 60(1), 14-24. https://doi.org/10.1016/j.compedu.2012.07.015

Magno, C. (2011). Validating the academic self-regulated learning scale with the Motivated Strategies for Learning Questionnaire (MSLQ) and Learning and Study Strategies Inventory (LASSI). The International Journal of Educational and Psychological Assessment, 7(2), 56-73.

Malau-Aduli, B. S., Roche, P., Adu, M., Jones, K., Alele, F., & Drovandi, A. (2020). Perceptions and processes influencing the transition of medical students from pre-clinical to clinical training. BMC Medical Education, 20, 279. https://doi.org/10.1186/s12909-020-02186-2

Mangundjaya, W. L. H. (2013). Is there cultural change in the national cultures of Indonesia? In Y. Kashima, E. S. Kashima, & R. Beatson (Eds.), Steering the cultural dynamics: Selected papers from the 2010 Congress of the International Association for Cross-Cultural Psychology. https://scholarworks.gvsu.edu/iaccp_papers/105/

National Ministry of Health. (2020, December 31). Health care workforce statistics database in Indonesia. Retrieved May 10, 2021, from http://bppsdmk.kemkes.go.id/info_sdmk/info/index?rumpun=101

National Ministry of Health. (2021, April 23). Current situations of the development of novel coronavirus (COVID-19). Retrieved May 10, 2021, from https://covid19.kemkes.go.id/situasi-infeksi-emerging/situasi-terkini-perkembangan-coronavirus-disease-covid-19-23-april-2021

Ortiz-Martìnez, Y., Castellanos-Mateus, S., Vergara-Retamoza, R., Gaines-Martìnez, B., & Vergel-Torrado, J. A. (2021). Online medical education in times of COVID-19 pandemic: A focus on Massive Open Online Courses (MOOCs), Educación Médica, 22, S40. https://doi.org/10.1016%2Fj.edumed.2020.12.001

Panadero, E. (2017). A review of self-regulated learning: Six models and four directions for research. Frontiers in Psychology, 8, Article 422. https://doi.org/10.3389/fpsyg.2017.00422

Pintrich, P. R., Smith, D. A. F., Garcia, T., & Mckeachie, W. J. (1993). Reliability and predictive validity of the Motivated Strategies for Learning Questionnaire (MLSQ). Educational and Psychological Measurement, 53(3), 801-813. https://doi.org/10.1177%2F0013164493053003024

Poonia, S. K., & Rajasekaran, K. (2020). Information overload: A method to share updates among frontline staff during the COVID-19 pandemic. Otolaryngology-Head and Neck Surgery, 163(1), 60-62. https://doi.org/10.1177%2F0194599820922988

Sandars, J., & Patel, R. (2020). The challenge of online learning for medical education during the COVID-19 pandemic. International Journal of Medical Education, 11, 169-170. https://doi.org/10.5116/ijme.5f20.55f2

Schreiber, J. B., Nora, A., Stage, F. K., Barlow, E. A., & King, J. (2006). Reporting structural equation modelling and confirmatory factor analysis results: A review. The Journal of Educational Research, 99(6), 323–338. https://doi.org/10.3200/JOER.99.6.323-338

Stracke, C. M., Tan, E., Moreira Texeira, A., do Carmo Pinto, M., Vassiliadis, B., Kameas, A., Sgouropoulou, C., & Vidal, G. (2018). Quality Reference Framework (QRF) for the Quality of Massive Open Online Courses (MOOCs): Developed by MOOQ in close collaboration with all interested parties worldwide. MOOQ. http://www.mooc-quality.eu/QRF

Vilkova, K. A. (2019, May 20-22). Self-regulated learning and successful MOOC completion [Europeaan MOOCs Stakeholders Summit 2019 Conference]. Proceedings of EMOOCs: Work in Progress Papers of the Research, Experience and Business Tracks.

Virtanen, P., & Nevgi, A. (2010). Disciplinary and gender differences among higher education students in self-regulated learning strategies. Educational Psychology, 30(3), 323-347. https://doi.org/10.1080/01443411003606391

Wong, J., Baars, M., Davis, D., Van Der Zee, T., Houben, G.-J., & Paas, F. (2018). Supporting self-regulated learning in online learning environments and MOOCs: A systematic review. International Journal of Human-Computer Interaction, 35(4-5), 356-373. https://doi.org/10.1080/10447318.2018.1543084

Yen, C.-J., Tu, C.-H., Sujo-Montes, L., & Sealander, K. (2016). A predictor for PLE management: Impacts of self-regulated online learning on students’ learning skills. Journal of Educational Technology Development and Exchange, 9(1), 29-48. https://doi.org/10.18785/jetde.0901.03

*Nadia Greviana
Faculty of Medicine Universitas Indonesia,
Jalan Salemba 6 Central Jakarta, Indonesia,
Email: nadiagreviana@ui.ac.id/ nadia.greviana@gmail.com

Submitted: 19 May 2021
Accepted: 26 August 2021
Published online: 4 January, TAPS 2022, 7(1), 66-75
https://doi.org/10.29060/TAPS.2022-7-1/OA2521

Lay Ling Tan¹, Pim W. Teunissen², Wee Shiong Lim³, Vanessa Wai Ling Mok¹ & Hwa Ling Yap¹

¹Department of Psychological Medicine, Changi General Hospital, Singapore; ²School of Health Professions Education (SHE), Maastricht University, Netherlands; ³Cognition and Memory Disorders Service, Tan Tock Seng Hospital, Singapore

Abstract

Introduction: Development of expertise and counselling skills in psychiatry can be mastered only with effective supervision and mentoring. The conceptualisations of educational supervision amongst supervisors and residents were explored in this study to understand how supervisory roles may have been affected by the adoption of competency-based psychiatry residency training.

Methods: A qualitative research approach with thematic analysis was adopted. Individual in-depth interviews using a semi-structured interview guide with a purposive sample of six supervisors and six newly graduated residents were conducted. Transcripts of the interview were analysed and coded using the Atlas Ti software.     

Results: Four major themes emerged from analysis of the transcripts: (1) Meaning and definition of supervision; (2) Expectations and responsibilities of the educational supervisor; (3) Elusiveness of mentoring elements in educational supervision and (4) Personal and professional development of residents in supervision. Supervisors and residents perceived educational supervision narrowly to be transactional with acquisition of knowledge and skills, but residents yearned for more relational interactions.

Conclusion: This study showed that the roles and functions of supervisors in educational supervision were unclear. It also highlighted the lack of a mentoring orientation in supervision in the psychiatry residency training program. An emphasis on assessment of competencies might have contributed to tension in the supervisory relationship and lack of a mentoring role, with concerns on residents’ personal and professional identity development in their psychiatry training. 

Keywords:           Psychiatry, Mentoring, Educational Supervision, Competency-Based Medical Education, Professional Identity Development

Practice Highlights

• Supervision in psychiatry has been described to encompass more than just a teaching and learning relationship but also a supportive and mentoring one.
• Educational supervision has been purported to offer the unique opportunity for effective mentoring within supervision.
• This qualitative study highlighted significant differences in definitions, roles and expectations of educational supervision.
• The conflict between mentoring and appraisal of competency needs to be addressed.
• Roles and expectations of the educational supervisor must be articulated clearly to both supervisors and residents.

Table 1: Faculty’s understanding of the roles and functions of the educational supervisor

2) Residents’ perspectives: Residents’ expected the educational supervisor to be “approachable and open”, “easy to talk to”, “relaxed”, “able to attend to personal growth”, “helping to reflect” and “build rapport”. Residents thus expected a more relational as opposed to transactional interaction with the supervisor.

“…apart from the bread and butter clinical aspects of work…talk to you a bit more about your personal stage in life, how things are coming along… It is this stuff that I find it hard to find in textbooks or anywhere along the clear training roadmap”

(R1)

Residents tended to see the educational supervisor’s role as conflictual in view of the attendant assessor function, and how boundaries between the two roles are often not clearly delineated.

“Ultimately is quite hierarchical in the system…. so if a supervisee has a lot of problems and come to the supervisor for advice…the supervisor might negatively evaluate the supervisee”

(R2)

C. Elusiveness of Mentoring Elements in Educational Supervision

1) Supervisor’s perspectives: Faculty viewed mentoring as “broader”, “longer term” and “beyond clinical and professional development”. “Trust” and “special” characterised a mentoring relationship but the focus was more on “professional development” rather than “personal development”. Faculty did not see themselves as fulfilling a mentoring role but generally agreed that there could be opportunities in offering elements of mentoring in educational supervision and to “contribute to the growth of residents beyond the short-term focus on clearing examinations”.

“…take about certain other aspects you know…mentorship if possible…beyond the pure clinical and professional development”

(S1)

Supervisors alluded to their roles in educational supervision as “facilitating the growth as a person”; “giving advice” and “role modelling” but they did not consider these as mentoring roles even though these were generally accepted as mentoring in nature.

“…never seen myself in a mentoring position…mentoring goes beyond just the supervisor-supervisee relationship…but I don’t think it has really extended beyond that”

(S3)

2) Residents’ perspectives: There was a similar reference to mentoring as “adopting a broader view of the development of the resident” over “a longer period of time”. Residents described mentoring as having a “deeper emotional connect” and “beyond the clinical development”. The evaluator role was viewed as potentially inhibiting the development of a mentoring relationship.

“So, it muddles the role…when they are so tied up to assess…they may not be able to mentor as much…”

(R1) 

D. Personal and Professional Development of Residents

1) Supervisor’s perspectives: Professional development was referred to as the “professional attitudes and motivation”, “duty of care”, “ethical boundaries” and the attributes which are more “formal” as contrasted to personal development. The latter being referred to as “one’s character”, “temperament”, “personality”; “the way we see things”; “development of self-awareness and self-actualisation” and the “need to know yourself and what problems you have in order to be able to help your patients struggling with personal problems”. Faculty also referred to personal development as “developing as a person and handling of stress”, “how to handle complaints”, and “how to juggle various roles to have a work-life balance”. Other views of personal development included “extracurricular activities outside of psychiatry” and “some hobbies” which some faculty opined as “more important in psychiatry because of the importance for self-care”.

However, some faculty held the opposing view that personal development should not be the focus of educational supervision. Lapses in personal development would only be brought up during supervision if there were “negative effects on professional roles and clinical practice and impeding progress” for risk of being accused of “prying into the residents’ personal lives and being nosy”.

“…but I don’t focus so much unless they hinder the work side of things. But maybe we should…”

(S3)

The reluctance of some residents to discuss issues of personal development were interpreted as an “Asian thing”, for instance being “uncomfortable” talking about personal struggles and “residents not expecting it”’.

2) Residents’ perspectives: Residents’ referred to personal developmental aspects as “be as a person”, “religious growth”, “personal well-being”, “how you are getting on with life as a whole”, “finding out about the person’s preferences”, “strengths and weaknesses” and “outside of the career”.

There was the fear of the lack of confidentiality and of being evaluated negatively if residents were to portray themselves as having personal struggles.

“though resident want to grow and develop but exposing these shortcomings could be very sensitive…”

(R1)

Residents considered personal aspects of their development to be “more private”; “should not be covered unless interferes with professional development”; “not so important”; “not fair for the supervisee and supervisor” and “something you should sort out on your own”. They perceived supervision to be formal and mainly moments of assessment for their professional development and so it would be inappropriate to discuss about personal struggles. Some residents also held the view that “personal and professional lives are separate” and the “supervisor may not be interested”.

“…to say that the supervisor should cover personal growth I don’t think that is very fair as well”

(S4)

IV. DISCUSSION

A. Conceptualisation of Educational Supervision

The concept of educational supervision was alien to both residents and supervisors. Supervision was seen mainly as teaching the residents to acquire knowledge and clinical skills with a focus on the transactional aspects. Our residents preferred a more relational supervisory interaction. The finding of psychiatry trainees valuing a supervisor’s emotional supportiveness more highly than clinical competence was also reported in another study (Chur-Hansen & McLean, 2007).

Our results showed that roles of teaching and assessing were more consistently endorsed by the supervisors. Relational roles like facilitating, role modelling, rapport building, and mentoring were considered important but cited less frequently by both groups. Participants tended to attribute this to the training’s emphasis on evaluation and assessment such that the focus of the supervision was more on the transactional rather than the relational components. This might have resulted in tension within the supervisory relationship as expectations for assessment of competencies take precedence (Julyan, 2009). This phenomenon was both ironic and worrying as the original intention for setting up the supervisory framework in the psychiatry residency program was to ensure that the training and learning would be supplemented by the presence of an educational supervisor with a mentoring role, entrusted with the fostering of the personal and professional components in clinical supervision (Kilminster et al., 2007).

B. Assessor Role of Supervisors

Although residency assessments were mainly formative, residents in our study still harbored concerns about the supervisor fulfilling the assessor role and determining their readiness to progress. If assessments were to be perceived as high stake in our examination-oriented training environment, the role of the teacher as helper might be compromised (van der Vleuten et al., 2012). As highlighted by the data in our study, combining the roles of helper and judge could confront the supervisor with a conflict of interest (Cavalcanti & Detsky, 2011). There might be situations either of inflation of judgement (Govaerts et al., 2007) or trivialisation of the assessment process (Dudek et al., 2005), which would potentially impair the professional development of the residents.

C. Personal and Professional Development

There was a common reference by both faculty and residents to the growth of the resident. Based on our data, there appeared to be some overlap of the concept of personal and professional development of residents in psychiatry training. Supervisors viewed the ability to handle stress and developing resilience not only as aspects of personal development but a reflection of the professional competence as well. But supervisors also opined that within the current supervisory framework, they would not be able to support the personal development of the residents. Although residents indicated their desires for supervisors to facilitate their holistic development and growth towards becoming a psychiatrist, they did not expect supervisors to be interested in their personal development but to focus more on the professional development of their clinical competencies.

There was avoidance by supervisors and residents to discuss personal struggles and frustrations in our study. This would be of concern if in the context of educational supervision in psychiatry, the personal aspects and development of better self-awareness could not be achieved, bearing in mind that this had been considered by some to be the fundamental goal of supervision in counselling (Kadushin, 1985). Concerns of boundary violations within the supervision relationship were raised by both groups of participants. In our study, the supervisors’ strict adherence to the boundaries as accustomed to in psychotherapeutic practice might deter self-disclosure. In contrast, our data suggested that sharing of clinical experiences by their supervisors were very much welcomed by the residents. Research has shown that there might be improvement in alliance when supervisors disclosed relevant past clinical experiences (Matazzoni, 2008). Self-disclosure by the supervisor usually normalised clinical struggles experienced by the residents and could enhance the supervision relationship (Knox et al., 2011). In our study, supervisors held fairly rigid boundaries within supervision, which had been shown to hinder the development of authentic emotional relationships or deeper mentoring relationships (Kozlowski et al., 2013).

Supervisors in our study reflected that it might have been cultural or an ‘Asian thing’ for residents to avoid discussion of personal struggles. Eastern cultures were noted to appreciate a larger power distance than Western cultures (Hofstede, 2011). Cultural value theory also opined that Eastern cultures tended to be more conservative and hierarchical and valued mastery to a lesser extent than the West (Schwartz, 1999). In our predominantly Asian context, the perception of a hierarchical training system might result in supervisors maintaining a stricter teacher-student boundary, or residents being more reluctant to share personal frustrations, particularly with the more senior supervisors. The evaluator role of the supervisor might also result in the resident erecting certain boundaries in the supervisory relationship. This would have implications for nurturing the personal and professional growth of the residents, which require guidance through mentorship and self-reflection within a trusting relationship (Holden et al., 2015).

D. Psychological Safety Within a Trusting Supervisory Relationship

Previous research has shown that any feedback which invoked the self potentially carried with it social judgements which might threaten the educational alliance (Pugh & Regehr, 2016; Telio et al., 2015) and there was a tendency for both supervisors and supervisees to interpret performance assessments as part of a judgement of personal worth (Hawe, 2003). The psychological safety within the supervisory relationship would be particularly important as awareness of the residents’ own inadequacies might be unpleasant and threatening as they faced their imperfect understanding and subjective theories (Vec et al., 2014). As such, the goals of supervision would be best attained with a trusting supervisor-supervisee relationship. However, our study showed that residents are unwilling to reveal too much of their inadequacies as this was too threatening for them, considering that their supervisors also evaluated their overall work performance and ability to deal with stress. This had resulted in tension in the supervisory relationship. The failure to pay heed to this, whether it was inherent to the training program or secondary to the supervisors’ lack of awareness, might further hinder and jeopardise the supervisory process.

The tension between assessment for support and assessment for high stakes decision-making will continue to challenge supervisors. The conflict between mentoring and appraisal of competency would need to be addressed. It would be important for residency training programs to create opportunities to allow the fostering of trusting supervisory relationships. Roles and expectations of the educational supervisor would need to be articulated clearly to both supervisors and residents. Supervisor training would need to focus not only on supervisor ability and competencies but more importantly, supervisor motivation. There should be the emphasis on instilling awareness of internal values and beliefs encompassing competency assessment, accountability, potential role conflicts, feedback delivery and “skills for establishing trusting, open and non-defensive yet problem–confronting relationships” (Govaerts et al., 2007).

E. Limitations of the Study

In our study, newly graduated residents who were agreeable to participate were recruited. The views of the ‘unwilling’ participants regarding supervision which might be more diverse and contentious might be inadvertently excluded. The content of the interview guide used was also not validated. Another limitation concerned the dual roles of researcher and APD. The view of the resident group might be subjected to biases and undue influence due to power differentials (Kotter, 2010). The researcher had minimised such potential biases by being reflexive and addressed concerns of imposing on the participants’ views openly during the interview (Creswell & Miller, 2000).

V. CONCLUSION

The mentoring role in supervision was found to be lacking in our current residency training. The residency program structure, with its focus on assessments of competencies and examinations, might have the unintended consequences of encouraging a transactional supervisory structure at the expense of a relational and mentoring relationship. This qualitative study highlighted significant differences in definitions, roles and expectations of educational supervision. It was our intention that this research endeavor contribute towards better appreciation of the dynamics within educational supervision in a competency-based residency training framework and further inform developments in the mentoring component of supervisory practices in the other training programs as well.

Notes on Contributors

Dr Lay Ling Tan formulated the research question and designed the research methodology. She conducted the semi-structured interviews and wrote the first draft of the manuscript.  

Prof Pim W. Teunissen provided guidance for the research methodology and data analysis. He was also involved with the revision of the manuscript drafts.

Dr Wee Shiong Lim provided guidance for the research methodology, assisted with the initial analysis of the first three interviews independently as well as the revision of the manuscript drafts.

Both Dr Vanessa Wai Ling Mok and Dr Hwa Ling Yap were involved with recruitment of participants and data analysis.

Ethical Approval

Ethics approval was sought from the Institutional Review Board (CIRB Ref: 2017/2319) and informed consent was received from all participants.

Data Availability

Due to the nature of this research, participants of this study did not agree for their data to be shared publicly, so supporting data is not available.

Acknowledgement

We would like to acknowledge the contributions of the National Psychiatry Residency Program Supervisors and Residents for their willingness to participate.

Funding

We did not receive any funding for this research study.

Declaration of Interest

Dr Lay Ling Tan is the Associate Program Director and the PI of this research. Dr Hwa Ling Yap and Dr Vanessa Mok are both teaching faculty of the teaching site. They have a vested interest in ensuring the quality of supervision of residents. The other authors have no other conflicts of interest, including financial, consultant, institutional and other relationships that might lead to bias or a conflict of interest.

References

Akhtar, S. (2009). Comprehensive dictionary of psychoanalysis. Routledge.

Carroll, M. (2006). Counselling supervision: Theory, skills and practice. SAGE.

Cavalcanti, R. B., & Detsky, A. S. (2011). The education and training of future physicians: Why coaches can’t be judges. JAMA, 306(9), 993-994. https://doi.org/10.1001/jama.2011.1232

Chay, O. M. (2019). Transformation of medical education over the years – A personal view. The Asia Pacific Scholar, 4(1), 59–61. https://doi.org/10.29060/TAPS.2019-4-1/PV1076

Chur-Hansen, A., & McLean, S. (2007). Trainee psychiatrists’ views about their supervisors and supervision. Australasian Psychiatry, 15(4), 269-272.

Creswell, J. W. (2014). Research design: Qualitative, quantitative, and mixed method approaches (4th ed.). SAGE.

Creswell, J. W., & Miller, D. L. (2000). Determining validity in qualitative inquiry. Theory Into practice, 39(3), 124-130. https://doi.org/10.1207/s15430421tip3903_2

Driessen, E. W., Overeem, K., & van der Vleuten, C. P. M. (2011). Get yourself a mentor. Medical Education, 45(5), 438-439. https://doi.org/10.1111/j.1365-2923.2011.03948.x

Dudek, N. L., Marks, M. B., & Regehr, G. (2005). Failure to fail: The perspectives of clinical supervisors. Academic Medicine, 80(10), S84-S87.

Falender, C. A., & Shafranske, E. P. (2007). Competence in competency-based supervision practice: Construct and application. Professional Psychology: Research and Practice, 38(3), 232–240. https://doi.org/10.1037/0735-7028.38.3.232

Frank, J. R., Snell, L., Englander, R., & Holmboe, E. S. (2017). Implementing competency-based medical education: Moving forward. Medical Teacher, 39(6), 568–573. https://doi.org/10.1080/0142159x.2017.1315069.

Govaerts, M. J., van der Vleuten, C. P., Schuwirth, L. W., & Muijtjens, A. M. (2007). Broadening perspectives on clinical performance assessment: rethinking the nature of in-training assessment. Advances in Health Sciences Education Theory Practical, 12(2), 239-260. https://doi.org/10.1007/s10459-006-9043-1

Hansman, C. A. (2001). Mentoring as continuing professional education. Adult learning, 12(1), 7–8. https://doi.org/10.1177/104515950101200104

Hawe, E. (2003). “It’s Pretty Difficult to Fail”: The reluctance of lecturers to award a failing grade. Assessment & Evaluation in Higher Education, 28(4), 371–382. https://doi.org/10.1080/0260293032000066209

Hofstede, G. (2011). Dimensionalizing Cultures: The Hofstede Model in Context. Online Readings in Psychology and Culture, 2(1), 1–26. https://doi.org/10.9707/2307-0919.1014

Holden, M. D., Buck, E., Luk, J., Ambriz, F., Boisaubin, E. V., Clark, M. A., Mihalic, A. P., Sadler, J. Z., Sapire, K. J., Spike, J. P., Vince, A., & Dalrymple, J. L. (2015). Professional identity formation: Creating a longitudinal framework through TIME (Transformation in Medical Education). Academic Medicine, 90(6), 761–767. https://doi.org/10.1097/acm.0000000000000719

Julyan, T. E. (2009). Educational supervision and the impact of workplace-based assessments: a survey of psychiatry trainees and their supervisors. BMC Medical Education, 9(1), 51. https://doi.org/10.1186/1472-6920-9-51

Kadushin, A. (1985). Supervision in social work. Columbia University Press.

Kilminster, S., Cottrell, D., Grant, J., & Jolly, B. (2007). AMEE Guide No. 27: Effective educational and clinical supervision. Medical Teacher, 29(1), 2–19. https://doi.org/10.1080/01421590701210907

Knox, S., Edwards, L. M., Hess, S. A., & Hill, C. E. (2011). Supervisor self-disclosure: Supervisees’ experiences and perspectives. Psychotherapy, 48(4), 336–341. https://doi.org/10.1037/a0022067

Kolb, D. A. (2014). Experiential learning: Experience as the Source of Learning and Development (2nd ed.). Pearson Education, Inc. (Original work published 1984).

Kotter, J. P. (2010). Power and influence. Simon and Schuster.

Kozlowski, J. M., Pruitt, N. T., DeWalt, T. A., & Knox, S. (2013). Can boundary crossings in clinical supervision be beneficial? Counselling Psychology Quarterly, 27(2), 109–126. https://doi.org/10.1080/09515070.2013.870123

Launer, J. (2013). Supervision, mentoring and coaching. In Swanwick. T (Ed.), Understanding medical education. Evidence, theory and practice (2nd ed., pp 111-122). Wiley Blackwell. https://doi.org/10.1002/9781118472361.ch8

MacDonald, J. (2002). Clinical supervision: A review of underlying concepts and developments. Australian & New Zealand Journal of Psychiatry, 36(1), 92-98. https://doi.org/10.1046/j.1440-1614.2002.00974.x

Matazzoni, T. A. (2008). The influence of supervisor self-disclosure on the supervisory working alliance in beginning and advanced therapists-in-training [Doctoral dissertation]. ProQuest.

Passi, V. (2016). The importance of mentoring during educational supervision. Perspectives on Medical Education, 5, 195-196.

Patel, P. (2016). An evaluation of the current patterns and practices of educational supervision in postgraduate medical education in the UK. Perspectives on Medical Education, 5(4), 205-214. https://doi.org/10.1007/s40037-016-0280-6

Pront, L., Gillham, D., & Schuwirth, L. W. (2016). Competencies to enable learning-focused clinical supervision: a thematic analysis of the literature. Medical Education, 50(4), 485–495. https://doi.org/10.1111/medu.12854

Pugh, D., & Regehr, G. (2016). Taking the sting out of assessment: is there a role for progress testing? Medical Education, 50(7), 721-729.

‌Schon, D. A. (1987). Educating the reflective practitioner: Toward a new design for teaching learning in the professions. John Wiley.

Schwartz, S. H. (1999). A theory of cultural values and some implications for work. Applied psychology, 48(1), 23-47. https://doi.org/10.1111/j.1464-0597.1999.tb00047.x

Tebes, J. K., Matlin, S. L., Migdole, S. J., Farkas, M. S., Money, R. W., Shulman, L., & Hoge, M. A. (2011). Providing competency training to clinical supervisors through an international supervision approach. Research on Social Work Practice, 21(2), 190-199. https://doi.org/10.1177/1049731510385827

Telio, S., Ajjawi, R., & Regehr, G. (2015). The “educational alliance” as a framework for reconceptualizing feedback in medical education. Academic Medicine, 90(5), 609-614. https://doi.org/10.1097/acm.0000000000000560

Torby, A., Turner, M. B., Kinzie, J. M., & Usher, C. (2015). What we talk about when we talk about supervision: The clear and the confusing in graduate psychiatric education. Journal of the American Psychoanalytic Association, 63(5), NP7-NP12. https://doi.org/10.1177/0003065115609724

van der Vleuten, C. P. M., Schuwirth, L. W. T., Driessen, E. W., Dijkstra, J., Tigelaar, D., Baartman, L. K. J., & van Tartwijk, J. (2012). A model for programmatic assessment fit for purpose. Medical Teacher, 34(3), 205-214. https://doi.org/10.3109/0142159X.2012.652239

Vec, T., Rupnik Vec, T., & Žorga, S. (2014). Understanding how supervision works and what it can achieve. In Watkins C. E. &. Milne D. I (Eds.) The Wiley International Handbook of Clinical Supervision (pp 103-127). Wiley. https://doi.org/10.1002/9781118846360.ch5

Wald, H. S. (2015). Professional identity (trans) formation in medical education: Reflection, relationship, resilience. Academic Medicine, 90(6), 701-706.

*Lay Ling Tan
2 Simei Street 3 S529889
Email: tan.lay.ling@singhealth.com.sg

Submitted: 23 January 2021
Accepted: 11 August 2021
Published online: 4 January, TAPS 2022, 7(1), 55-65
https://doi.org/10.29060/TAPS.2022-7-1/OA2475

Marcus A Henning¹, Vanamali Joseph¹, Roger J Booth², Christian U Krägeloh³ & Craig S Webster¹

¹School of Medicine, University of Auckland, New Zealand; ²Department of Medical Science, University of Auckland, New Zealand; ³Department of Psychology and Neuroscience, Auckland University of Technology, New Zealand

Abstract

Introduction: This study investigates psychological distress and quality of life (QoL) amongst first year premedical and health science students. The primary aim of this study was to investigate potential differences in QoL and psychological distress between students who sought entry into a medicine programme when compared to those opting for a non-medicine career. 

Methods: We examined participant responses to measures of QoL, psychological distress, and course preference (medicine or other). A structural equation model was conducted to consider the interrelationships among future course preference, gender, QoL, depression, anxiety and stress.

Results: Three hundred and sixty-five students completed the online survey. An a priori conceptual model was developed and then evaluated using a structural equation model. The values obtained for RMSEA (0.027), CFI (0.999), and SRMR (0.016) indicated an excellent model fit. The overall model fit statistic, chi-square (χ2 = 7.626, df=6, p= .267), confirmed a good model fit. Students aiming to enrol in medicine generated higher psychological health and environmental QoL scores compared to their non-medicine oriented peers. In addition, physical QoL and psychological health QoL scores significantly predicted psychological distress measures.

Conclusion: The study raises a potential debate regarding placing students with mixed career intentions into the same course and the potential implications this may have on teaching in interprofessional and large student groups in relation to wellbeing, pedagogy, equity, and expenditure. The findings clearly indicated that medical students are not as adversely impacted upon in terms of QoL and psychological distress compared with their non-medical peers.

Keywords:           Medical and Health Science Students, Quality of Life, Psychological Distress, Course Preference

Practice Highlights

• Students focusing on medicine report higher levels of psychological health and environmental QoL.
• Course preference is associated with psychological QoL and psychological distress.
• Issues with psychological QoL are more common among females.
• QoL and learning in large lecture theatres requires further investigation.

I. INTRODUCTION

At the University of Auckland, the Biomedical Common Year or Overlapping Year 1 (BCY) programme is a first-year syllabus and the gateway for many students aiming to enrol into either medicine or other healthcare disciplines (e.g., nursing, optometry, or pharmacy). The requirements for entry into medicine is highly competitive such that approximately 700 students who study in BCY aim to apply for medicine out of which only 250 students are chosen (Henning et al., 2017b). Entry into other healthcare disciplines (e.g., nursing, optometry, or pharmacy) is less competitive and selection is thus likely less distressing (Al Alwan et al., 2013). However, the large class sizes within the pre-medical and health science year (of approximately 1,300 students) containing students applying for either medicine or non-medicine programmes may also create a marginalising learning dynamic that may hinder the motivation and wellbeing of non-medicine students (Henning et al., 2017a). Hence, the aim of this research was to assess whether those students aiming for selection into medicine were more at risk of poor quality of life (QoL) and psychological distress relative to those students aiming for selection into other healthcare disciplines.

Numerous studies have researched the areas of QoL and psychological distress amongst medical students (Lovibond & Lovibond, 1995; Szabó, 2010; Wickramasinghe et al., 2019).  In their systematic review, Dyrbye et al. (2006) reported concerning rates of depression, anxiety and general psychological distress among medical students, citing for example in one report that 37% of medical students in their first year of study experienced mental health issues of some kind. The adverse impact of distress among students can contribute to problems associated with academic attainment, professional development, academic honesty, substance abuse, cynicism, humanitarian attitude, and development of empathy. Moreover, psychological distress appears to be more of a concern among female medical students. The implication underlying the research investigating QoL and psychological distress suggests that medical students are more compromised than their non-medical peers.

Our conceptual model for the present study was developed in accordance with self-determination theory (Ryan & Deci, 2000). According to this theory, the act of choosing can positively affect motivation, well-being and learning if the needs for autonomy, relatedness, and competence in the individual are high. In this study, students were free to choose their course of study and hence freedom of choice was perceived as equivalent to the notion of course preference. According to this theory, students can be intrinsically motivated (engaged in their studies because of inherent enjoyment) and extrinsically motivated with a focus on outcome (Vansteenkiste et al., 2006). Extrinsic motivation, however, can be differentiated into aspects of regulation dependent upon the degree of autonomy. Therefore, there are facets of autonomous motivation (which involves volition and choice) versus that of controlled motivation (feeling pressured or coerced) (Vansteenkiste et al., 2006). Hence, intrinsic motivation and well-intentioned aspects of extrinsic motivation are considered autonomous, whilst the poorly internalised form of extrinsic motivation are less well incorporated into an individual’s sense of self.  In our proposed model, we suggest features of choice with links to QoL.

We cannot assume that the medical or non-medical students differ in terms of their intrinsic or extrinsic motivation, but we do know that the majority of non-medical students at this university (e.g. nursing and pharmacy) have been given conditional entry into their courses on the proviso that they pass their courses in the BCY.  However, medical students do not have this luxury and are in a more competitive mode of action. They are, thus, likely intrinsically and extrinsically motivated, and their extrinsic motivation could be either autonomous or controlled.  Our assumption is that the non-medical students are also likely to be intrinsic and extrinsically motivated, but their extrinsic motivation is likely more autonomous as they have conditional entry. Therefore, non-medical students would likely experience more heightened levels of QoL than medical students who are experiencing higher levels of competition (Katz & Assor, 2007; Ryan & Deci, 2000). If this is not true another more pressing variable may be affecting the QoL outcome, such as mixing high numbers of students together from different disciplines into one course (Hornsby & Osman, 2014). In addition, given the established empirical evidence (Haist et al., 2000; Henning et al., 2013; Naylor et al., 2018; Ramsbottom-Lucier et al., 1995), we argued that gender would likely moderate choice and QoL. Therefore, the first stage of our conceptual model proposes that, once students have chosen a particular course of study (medicine or other) this will directly impact their QoL given the increased expectation associated with the decision process (Katz & Assor, 2007).  Therefore, those students aiming for a programme (i.e., medicine) requiring higher stakes will likely be more at risk of ill-health.

The next element of our argument proposes that QoL will likely impact psychological distress. For example, if a student experiences good health-related physical QoL they are then likely to have lowered risk of psychological distress (Burris et al., 2009). In their study of first year medical students, Verger et al. (2009) reported that psychological distress was moderated by gender and influenced by wellbeing protective factors, such as social support, coping with academic pressure, and socio-economic factors. Consequently, we suggest that there is likely a follow-on effect such that those students with compromised QoL will likely experience more psychological distress (Dyrbye et al., 2006). Nonetheless, the causes of poor QoL are likely complex and vary substantially between individuals given that some studies have shown that students aiming to study medicine may be more motivated and experience higher levels of QoL (Henning et al., 2017a).

The primary aim of this study was to empirically investigate potential differences in QoL and psychological distress between students who sought entry into a medicine programme when compared to those opting for a non-medicine career.  The issue of follow-on effects from QoL to psychological distress has not been studied within this learning context. To illustrate potential differences between the two groups, we scrutinised conceptual links between choice, gender, QoL and psychological distress in reference to students’ course preference (medicine or other) whilst studying in the BCY programme. 

II. METHODS

A. Participants

All students (N=1277) studying within the BCY programme were invited to participate in this survey. Students aiming to study in any health-related course at the University of Auckland first enrol in the BCY programme in preparation for formalised entry into degrees for Bachelor of Health Science, Bachelor of Pharmacy, Bachelor of Optometry or Bachelor of Medicine (The University of Auckland, 2019).

B. Procedure

Ethics approval for the collection and use of data was obtained from the University of Auckland Human Participants Ethics Committee (Ref. 021985).

Students were informed about the study by email in late July (2018), and two reminder emails were sent out. The anonymous survey was conducted prior to final course completion in Semester 2 (2018), but after grades for the first semester had been released. Informed consent was implied given that study participants received a participant information sheet prior to the start of the questionnaire.  Data were collected online using a Google Forms questionnaire (Google.com, 2015), which requires respondents to fully complete the survey before they are able to submit it.

C. Measures

Demographic and background details were obtained to assess the representativeness of the sample included: (1) Age in years; (2) Gender; (3) Future course preference (medicine or other); (4) Enrolment status (international or domestic); and (5) Ethnicity.

Quality of life was measured using the New Zealand version of the WHOQOL (NZ WHOQOL-BREF) questionnaire (Krägeloh et al., 2016). It has four domains (physical QoL, psychological health QoL, social relationships QoL and the environmental QoL) and contains 31 items. In this study, physical QoL had six items, psychological health QoL had 10 items, social relationships QoL had four items and environmental QoL had eight items (The NZ WHOQOL Group, 2017). Questions 1 and 2 are global QoL and health items and, thus, not included in the scoring of the four quality of life domains. Question 3 was deemed to have little relevance with respect to the research question and had been previously identified as having a ceiling effect in early career medical students (Krägeloh et al., 2011). All items utilised a 5-point Likert scale with varying anchors (the scores for two items were reversed). High scores indicated higher levels of QoL.

Psychological distress was measured using the Depression, Anxiety and Stress Scale (DASS-21), which captures respondents’ emotional states in reference to three domains: depression, anxiety and stress (within the past seven days). Each domain is comprised of seven items. The DASS-21 instrument has been developed from a previous 42-item version (Lovibond & Lovibond, 1995), and has undergone psychometric investigation among a variety of populations (Le et al., 2017; Medvedev et al., 2018; Szabó, 2010; Yıldırım et al., 2018). Each item utilises a 4-item Likert scale. Higher scores indicate more frequency and severity in the experience of negative emotions.

D. Data Analysis

First, the response rate (n/N) was calculated as a percentage measure, and the details of the participants were described. Second, descriptive statistics (means and standard deviations (SDs)) were computed for the two self-reported questionnaires measures (NZ WHOQOL-BREF and DASS-21). These measures were computed for each of the questionnaire domains with respect to course preference (medicine; other) and gender (male; female). A multivariate analysis of variance (ANOVA) approach was used to assess levels of significance for the independent variables, course preference and gender, with respect to the aforementioned dependent variables.

Lastly, one structural equation model (SEM) was developed to consider the interrelationships among course preference, gender, and QoL and DASS-21 domain measures. This SEM was developed using the software AMOS v25 (Arbuckle, 2017). The primary goodness-of-fit indices used to assess the model fit were inspection of the model chi-square value, root mean square error of approximation (RMSEA), comparative fit index (CFI), and standardised root mean square residual (SRMR). Model fits were considered acceptable if they met the following recommendations: (1) RMSEA < 0.06; (2) SRMR < 0.08; and (3) CFI close to 0.95 or higher (Hu & Bentler, 1998). Additionally, using maximum likelihood bootstrapping with 1,000 samples, regression weights were computed, and bias-corrected 95% confidence intervals were determined (Byrne, 2010).

III. RESULTS

A. Response Rate and Participant Data

Three hundred and sixty-five students (n = 365) fully completed the online survey resulting in a response rate of 29%. The average age of participants was 19 years (SD = 1.84) with 75% of respondents being female. Table 1 provides the demographic profile for the sample, and upon examination, the distributions of the sample data appeared representative of the population profile. In this study, more than 75% of participants stated a preference to study medicine as opposed to other non-medicine health options, which is a rate higher than that found elsewhere (Henning et al., 2017b). The data that support the findings of this study are openly available in Figshare at https://figshare.com/articles/dataset/_/1405 0463 (Henning et al., 2021).

Table 1. Demographic and academic achievement details of the sample versus the population.

The means and standard deviations for the variables of interest are shown in Table 2. The NZ WHOQOL-BREF questionnaire means for these participants are marginally lower than those found elsewhere (Henning et al., 2017b). No comparable New Zealand medical student reference values for the DASS-21 are available for this population. In reference to internal consistency coefficients computed for each of the measures (Table 2), all Cronbach’s alpha values were equal to or greater than 0.70 thus meeting ‘acceptable’ requirements (Field, 2005).

Table 2. Means, standard deviations (SD) and internal consistency scores (Cronbach alpha) of the measures used in the study.

Note: Using a multivariate ANOVA approach, significant differences (p<.05) for course preference (medicine vs. other) were noted for psychological health QoL, environmental QoL, and depression. In addition, significant differences (p<.05) for gender (male vs. female) were noted for psychological health QoL, depression, anxiety and stress.

B. Structured Equation Modelling

Based on previous literature (Dyrbye et al., 2006) and self-determination theory (Katz & Assor, 2007; Ryan & Deci, 2000), an a priori conceptual model was developed and tested using a SEM framework. The primary criterion variable was the students’ course preference (medicine, scored as 1 vs. other, scored as 0). The values obtained for RMSEA, SRMR, and CFI indicated that the model fit was excellent, that is, RMSEA = 0.027, CFI = 0.999, and SRMR = 0.016. As a secondary check, the overall model fit determined by the chi-square (χ2 = 7.626, df=6. p= .267) also indicated the model fitted very well. We noted that some of the error variances within measures of the NZ WHOQOL-BREF and DASS-21 were required to be correlated.

Figure 1 shows the path diagram for all the data collected (n = 365). The path coefficients are shown for significant relationships only (p < .05). The bias-corrected 95% confidence intervals for the regression weights (path coefficients) obtained using the maximum likelihood bootstrapping method were generated (Table 3) to ensure the accuracy and stability of the parameter estimates (Byrne, 2010).

Figure 1. Path diagram of the tested structural equation model for all study participants.

Note. The path standardised regression weight estimates were shown only when statistically significant (p<.05). For non-significant relationships, directive arrows are shown with dashed lines (see Table 3 for details). Errors for all pertinent measures were removed to improve clarity of the figure. Error variances among the WHOQOL- NZ version and DASS-21 domian measures were were allowed to be correlated. No other error variances were correlated. Key fit indices and chi-square calculations were standardised RMR = .016, CFI = .999, RMSEA = .027, χ2 = 7.626, df=6. p= .267

Table 3. Standardised regression weights: Lower and upper ends of bias-corrected 95% confidence intervals of the regression weights of the directional paths of the SEM.

Note. * p<.05

When inspecting Table 2 and Figure 1, we can make certain assertions related to the differences between the two groups (medical and other students) with respect to QoL and psychological distress. Figure 1 indicates two significant associations between course preference and psychological health QoL and environmental QoL. Therefore, students indicating a preference for enrolling in a medical course were likely to experience greater psychological health QoL and environmental QoL. The findings in Table 2, clarify that students aiming to study medicine have higher levels of QoL in these two domains, compared to students aiming to study in non-medicine courses. In addition, the multivariate analyse indicate that depression is more of a concern for non-medical students than intending medicine students.

IV. DISCUSSION

The known health concerns associated with first year students (Naylor et al., 2018; Tinto, 2009) appear consistent with the current findings. The actual values (Table 2) for three of the four QoL domains from the NZ WHOQOL-BREF -psychological health QoL, social relationships QoL, and environmental QoL – are lower than previously published values for year 4 and 5 students at the same university (Henning et al., 2012), although the physical QoL score was higher. This indicates that year 1 students at this university appear to have relatively lower QoL than their year 4 and 5 peers and are consequently lower than the general population norms, particularly in the areas of psychological health QoL, social relationships QoL, and environmental QoL (Henning et al., 2012). In addition, the values for all domains are lower than those acquired from a 2015 first-year cohort (overall) studying in the same programme (Henning et al., 2017b).

Similarly, the DASS-21 findings (Table 2) show that this group scored higher on the domains of depression and anxiety compared to early career doctors also graduating from the same university (Henning et al., 2014). In comparison to a Malaysian student group, the mean values for the present study show distinct similarity in terms of stress and anxiety scores, but a marked lower score for depression (Yusoff et al., 2013). Therefore, we have provided good evidence to suggest that this group of students, irrespective of choosing either medicine or other as their preferred course option, studying in the BCY programme are experiencing relatively lower levels of QoL and higher levels of psychological distress compared to previous cohorts studying at the same university. However, their experiences may not differ greatly when compared to those students’ psychological distress perceptions in other universities.

In reference to the conceptual model, the findings also confirm that preference for course of study is linked with aspects of QoL. This in turn influences psychological distress, suggesting a feedback loop. Both of these associations were confirmed by the path analysis resulting in an excellent model fit for the SEM (Figure 1). The first part of the model considered how the preferred course option (medicine versus other) could directly impact QoL measures and the second part of the model considered the flow-on (indirect) effects to psychological distress (depression, anxiety and stress).

A. Choosing to Study medicine and Its Impact on Quality of Life

Figure 1 clearly shows that choosing to study medicine has an influence on aspects of QoL, although gender was not found to be significantly correlated with course preference. The decision to choose to study medicine was directly and significantly associated with psychological health QoL and environmental QoL (with higher scores noted for intending medicine students), but did not influence either physical QoL or social relationships QoL. In Table 2, we noted that psychological health QoL and environmental QoL for those vying for a place in the medical programme was higher than students’ scores for other health options (e.g., nursing, pharmacy, or optometry).  The direction of the arrows in the model implies that the act of deciding to study medicine is likely influencing QoL on these two domains. These findings can be interpreted in several ways.

First, it is likely that students who aim for admission to medicine may be more psychologically robust to begin with. The psychological health QoL domain of the New Zealand version of the WHOQOL encompasses 10 facets (Krägeloh et al., 2016; WHOQoL Group, 1998). The facets that likely emphasise the difference between those applying for medicine and those applying for another degree are self-esteem, perception of negative or positive feelings, concentration, ability to meet expectation, and management of personal circumstances. In a previous study (Henning et al., 2017b), perceived stress was found to be highly correlated with psychological QoL and also noted to be a strong predictor of course preference (medicine versus other), with potential medical students scoring higher than their other heath professional peers.

Second, the aspects of cognitive capacity, personal management, and mental wellbeing need to be weighed in comparison to environmental QoL (measured in perceived access to resources, i.e., finance, health care, transport, security, technology, and housing) (WHOQoL Group, 1998). The current findings confirm previous research linking course preference to environmental QoL, suggesting that those students who come from affluent and supportive backgrounds will likely have more interest in pursuing a medical degree (Henning et al., 2017b).

B. Impact of Quality of Life on Psychological Distress

The SEM also revealed an interesting flow-on effect suggesting that those students not choosing medicine will have more issues with psychological wellbeing than those choosing to pursue medicine, which translates to greater risk of psychological distress in the form of depression, anxiety and stress for non-medical students. Furthermore, intending medicine students generated significantly lower depression scores than their non-medicine peers. Even though physical QoL is also directly connected with psychological distress factors, this association was not influenced by course preference. Hence, the key issues linked to course preference are related to psychological health QoL, which is incidentally linked to gender. Environmental QoL appears to have no flow-on psychological distress effects.

These findings indicate that some perceptions of QoL are likely linked to choice and ultimately psychological distress factors. The flow-on effect suggests that those students who struggle with psychological health QoL tend to further struggle with psychological distress. The point of difference between the two groups (medical and other) is that the intending medical students appear less affected by depression than their non-medical peers. The concept of choice has been established as a key component in motivation and wellbeing, for example, the perceived expectancies and values regarding a task are influenced by weighing up the benefits and costs associated with the forthcoming task (Katz & Assor, 2007). Hence, in the medical context the preferred choice of medicine likely has a powerful impact on motivation, learning and wellbeing (Katz & Assor, 2007). The current findings indicate that students focussing on medicine have higher psychological health QoL and lower psychological distress (particularly in the area of depression) than non-medicine students in the same year. This may imply that the current programme is more suited to those aiming for a medical career than those heading for a non-medicine health-related vocation, although this proposition requires further investigation.

C. Implications for Learning and Wellbeing

There are two clear implications associated with these results.

First, the findings indicate that intending medical students are more likely to have good access to environmental resources. The elevation of environmental QoL scores for students intending to study medicine as compared to students in other disciplines is consistent with findings reported elsewhere (Collins et al., 1993; Henning et al., 2012; Ng et al., 2009), suggesting that these students likely have better access to the needed resources. Equity arguments regarding selection of medical students is a global concern and impacts areas related to communication, empathetic listening, and shared understandings between clinician and patient (Rao & Flores, 2007). The resources differential suggests a clear argument that students who prefer to study medicine may not equitably mirror the less-resourced population at large and thus may have difficulty relating to large proportions of society.  It is important to also emphasise that equity initiatives at the university in which this study was conducted are in place, although the complexity of the issues are multifaceted and may require further development (Curtis et al., 2015; Poole et al., 2009).

Second, the findings in this study clearly indicate that intending medical students are more likely to experience higher levels of psychological health QoL. Therefore, they are more likely to have a different wellbeing outcome than those students who aim for an alternative health sciences career, which may warrant investigating the rationale for placing students aiming for different health professions in one course of study.  An argument for placing all health-career students together is likely based on the idea of interprofessional learning (Salamonson et al., 2009). Some of the benefits include improved knowledge gains especially for those students preparing for non-medical courses and positive attitudinal shifts regarding each profession (Carpenter, 1995). The findings in this study revealed that one lecturer was able lecture to 1277 students at one time (using asynchronous technology), suggesting this is a cost effective teaching option. Nonetheless, the findings in this study do indicate that the implication of large classes (or ‘massification’) in reference to wellbeing gains requires further investigation (Hornsby & Osman, 2014).

It has been established that students preferring medicine to other health science options will lead to greater academic attainment. For example, Henning et al. (2017b) reported that those students preferring to pursue medicine attained a bioscience grade average of 6.77 which was significantly higher than the non-medical comparison cohort which attained a grade average of 4.42. Therefore, creating smaller, discipline-specific groups could create more equitable learning, as well as engendering more opportunities for staff-student engagement, more engagement with relevant content, and greater likelihood of motivation for learning (Hornsby & Osman, 2014). However, in the context of pre-medical courses, such as the BCY programme, even though there are clear differences in resulting grade achievement, more research is required to tease out the subtle learning and wellbeing interconnections and to further investigate whether one group is more disadvantaged in their learning than the other (in this case the non-medicine group).

A limitation of this study is the relatively low response rate, although this is comparable with other studies using online questionnaires (Nulty, 2008) and the participants engaged in this study had similar characteristics to the total population (Table 1). Moreover, the response rate is within the range considered acceptable for surveys of representatives of organisations (36%±13%) – a category arguably appropriate for students entering their respective professions (Baruch, 1999). A further limitation of the study is that the research design is cross-sectional suggesting that the links between variables cannot infer temporal causality. Lastly, we acknowledge that it was outside the scope of the study to fully explore some of the speculations raised in this paper, such as whether choice of course of study was influenced by either intrinsic or extrinsic factors (i.e., degree of autonomy).

V. CONCLUSION

The main implication of this study is the direct relationships between course preference (medicine versus other) and psychological health QoL and environmental QoL, as well as the further indirect or flow-on effects to psychological distress via the psychological QoL pathway. Thus, indicating that medical students are not as adversely impacted upon in terms of QoL and psychological distress compared with their non-medicine peers. Lastly, the study findings provide useful debate and scope for further research in the areas of pedagogy, educational equity, career aspiration, and financial cost related to interprofessional education and teaching in large student groups.

Notes on Contributors

Marcus A. Henning is an associate professor in the Centre for Medical and Health Sciences Education at the University of Auckland, New Zealand. MAH was involved in the conceptualisation of the idea, data curation, formal analysis, developing the design of the study, writing the original draft and ongoing editing.

Vanamali Joseph was, at the time of this study, a third year medical student studying in the Faculty of Medical and Health Sciences at the University of Auckland, New Zealand. VJ was involved in the conceptualisation of the idea, data curation, developing the design of the study and ongoing editing.

Roger J. Booth is an associate professor in molecular medicine and pathology, in the Faculty of Medical and Health Sciences at the University of Auckland, New Zealand. RB was involved in the conceptualisation of the idea, data curation, developing the design of the study and ongoing editing.

Christian U. Krägeloh is an associate professor in the Department of Psychology at AUT University, New Zealand. CUK was involved in the conceptualisation of the idea, formal analysis and ongoing editing.

Craig S. Webster is an associate professor in the Centre for Medical and Health Sciences Education at the University of Auckland, New Zealand. CSW was involved in the conceptualisation of the idea, data curation, formal analysis, developing the design of the study, writing the original draft and ongoing editing.

Ethical Approval

Ethics approval for the collection and use of data was obtained from the University of Auckland Human Participants Ethics Committee (Ref. 021985).

Data Availability

The anonymised data that supports the findings of this study are openly available in the Figshare repository, https://figshare.com/articles/dataset/_/14050463

Funding

No funding source required for this study.

Declaration of Interest

The authors declare no conflicts of interest.

References

Al Alwan, I., Al Kushi, M., Tamim, H., Magzoub, M., & Elzubeir, M. (2013). Health sciences and medical college preadmission criteria and prediction of in-course academic performance: A longitudinal cohort study. Advances in Health Sciences Education, 18(3), 427-438. https://doi.org/10.1007/s10459-012-9380-1

Arbuckle, J. L. (2017). IBM® SPSS® Amos™ 25 User’s Guide. https://usermanual.wiki/Document/IBMSPSSAmosUserGuide.2983928102/view

Baruch, Y. (1999). Response rate in academic studies- A comparative analysis. Human Relations, 52(4), 421-438. https://doi.org/10.1177/001872679905200401

Burris, J. L., Brechting, E. H., Salsman, J., & Carlson, C. R. (2009). Factors associated with the psychological well-being and distress of university students. Journal of American College Health, 57(5), 536-544. https://doi.org/10.3200/JACH.57.5.536-544

Byrne, B. M. (2010). Structural Equation Modeling with AMOS: Basic concepts, applications and programming (2nd ed.). Routledge.

Carpenter, J. (1995). Interprofessional education for medical and nursing students: Evaluation of a programme. Medical Education, 29(4), 265-272. https://doi.org/10.1111/j.1365-2923.1995.tb02847.x

Collins, J., Jones, J., & White, G. (1993). Demographic variables in Auckland medical students. New Zealand Medical Journal, 106(960), 306-308.

Curtis, E., Wikaire, E., Jiang, Y., McMillan, L., Loto, R., Airini., & Reid, P. (2015). A tertiary approach to improving equity in health: Quantitative analysis of the Māori and Pacific admission scheme (MAPAS) process, 2008–2012. International Journal for Equity in Health, 14(1), 1-15. https://doi.org/10.1186/s12939-015-0133-7

Dyrbye, L. N., Thomas, M. R., & Shanafelt, T. D. (2006). Systematic review of depression, anxiety, and other indicators of psychological distress among US and Canadian medical students. Academic Medicine, 81(4), 354-373. https://doi.org/10.1097/00001888-200604000-00009

Field, A. P. (2005). Discovering statistics using SPSS : And sex, drugs and rock’n’roll (2nd ed.). SAGE.

Google.com. (2015). Homepage. https://www.google.co.nz/forms/about/

Haist, S. A., Wilson, J. F., Elam, C. L., Blue, A. V., & Fosson, S. E. (2000). The effect of gender and age on medical school performance: An important interaction. Advances in Health Sciences Education: Theory and Practice, 5(3), 197-205. https://doi.org/10.1023/A:1009829611335

Henning, M. A., Joseph, V., Booth, R. J., Krageloh, C. U., & Webster, C. S. (2021). Biomedical students’ course preference and links with quality of life and psychological distress [Data set]. Figshare. https://figshare.com/articles/dataset/_/14050463

Henning, M. A., Krägeloh, C. U., Hawken, S. J., Zhao, Y., & Doherty, I. (2012). The quality of life of medical students studying in New Zealand: A comparison with non-medical students and a general population reference group. Teaching and Learning in Medicine, 24(4), 334-340.https://doi.org/10.1080/10401334.2012.715261

Henning, M. A., Krägeloh, C. U., Booth, R., Hill, E. M., Chen, J., & Webster, C. S. (2017a). Biomedical students in their first year of study: Factors explaining performance in a high stakes examination. Medical Science Educator, 27(4), 633–643.  https://doi.org/10.1007/s40670-017-0444-y

Henning, M. A., Krägeloh, C. U., Booth, R., Hill, E. M., Chen, J., & Webster, C. S. (2017b). Profiling potential medical students and exploring determinants of career choice. The Asia Pacific Scholar, 2(1), 7-15. https://doi.org/10.29060/TAPS.2017-2-1/OA1019

Henning, M. A., Krägeloh, C. U., Manalo, E., Doherty, I., Lamdin, R., & Hawken, S. J. (2013). Medical students in early clinical training and achievement motivation: Variations according to gender, enrolment status, and age. Medical Science Educator, 23(1), 6-15. https://doi.org/10.1007/BF03341798

Henning, M. A., Sollers, J., Strom, J. M., Hill, A. G., Lyndon, M. P., Cumin, D., & Hawken, S. J. (2014). Junior doctors in their first year: Mental health, quality of life, burnout and heart rate variability. Perspectives in Medical Education, 3, 136-143. https://doi.org/10.1007/s40037-013-0075-y

Hornsby, D. J., & Osman, R. (2014). Massification in higher education: Large classes and student learning. Higher Education Research & Development, 67(6), 711-719. https://doi.org/10.1007/s10734-014-9733-1

Hu, L.-t., & Bentler, P. M. (1998). Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification. Psychological Methods, 3(4), 424-453. https://doi.org/10.1037/1082-989X.3.4.424

Katz, I., & Assor, A. (2007). When choice motivates and when it does not. Educational Psychology Review, 19(4), 429. https://doi.org/10.1007/s10648-006-9027-y

Krägeloh, C., Henning, M., Hawken, S., Zhao, Y., Shepherd, D., & Billington, R. (2011). Validation of the WHOQOL-BREF quality of life questionnaire for use with medical students. Eductaion for Health, 24(2), 545.

Krägeloh, C. U., Billington, D. R., Hsu, P. H.-C., Feng, X. J., Medvedev, O. N., Kersten, P., Landon, J., & Siegert, R. J. (2016). Ordinal-to-interval scale conversion tables and national items for the New Zealand version of the WHOQOL-BREF. PLOS ONE, 11(11), e0166065. https://doi.org/10.1371/journal.pone.0166065

Le, M. T. H., Tran, T. D., Holton, S., Nguyen, H. T., Wolfe, R., & Fisher, J. (2017). Reliability, convergent validity and factor structure of the DASS-21 in a sample of Vietnamese adolescents. PLOS ONE, 12(7), e0180557. https://doi.org/10.1371/journal.pone.0180557

Lovibond, P. F., & Lovibond, S. H. (1995). The structure of negative emotional states: Comparison of the Depression Anxiety Stress Scales (DASS) with the Beck Depression and Anxiety Inventories. Behaviour Research and Therapy, 33(3), 335-343.

Medvedev, O. N., Krägeloh, C. U., Titkova, E. A., & Siegert, R. J. (2018). Rasch analysis and ordinal-to-interval conversion tables for the Depression, Anxiety and Stress Scale. Journal of Health 25(10–11), 1374–1383. https://doi.org/10.1177/1359105318755261

Naylor, R., Baik, C., & Arkoudis, S. (2018). Identifying attrition risk based on the first year experience. Higher Education Research & Development, 37(2), 328-342. https://doi.org/10.1080/07294360.2017.1370438

Ng, C. L., Tambyah, P. A., & Wong, C. Y. (2009). Cost of medical education, financial assistance and medical school demographics in Singapore. Singapore Medical Journal, 50(5), 462-467.

Nulty, D. D. (2008). The adequacy of response rates to online and paper surveys: What can be done? Assessment and Evaluation in Higher Education, 33(3), 301-314. https://doi.org/10.1080/02602930701293231

Poole, P. J., Moriarty, H. J., Wearn, A. M., Wilkinson, T. J., & Weller, J. M. (2009). Medical student selection in New Zealand: looking to the future. New Zealand Medical Journal, 122(1306), 88-100.

Ramsbottom-Lucier, M., Johnson, M. M., & Elam, C. L. (1995). Age and gender differences in students’ preadmission qualifications and medical school performances. Academic Medicine, 70(3), 236-239. https://doi.org/10.1097/00001888-199503000-00016

Rao, V., & Flores, G. (2007). Why aren’t there more African-American physicians? A qualitative study and exploratory inquiry of African-American students’ perspectives on careers in medicine. Journal of the National Medical Association, 99(9), 986-993.

Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist, 55(1), 68-78. https://doi.org/10.1037/0003-066X.55.1.68

Salamonson, Y., Everett, B., Koch, J., Wilson, I., & Davidson, P. M. (2009). Learning strategies of first year nursing and medical students: A comparative study. International Journal of Nursing Studies, 46(12), 1541-1547. https://doi.org/10.1016/j.ijnurstu.2009.05.010

Szabó, M. (2010). The short version of the Depression Anxiety Stress Scales (DASS-21): Factor structure in a young adolescent sample. Journal of Adolescence, 33(1), 1-8. https://doi.org/10.1016/j.adolescence.2009.05.014

The NZ WHOQOL Group. (2017). WHOQOL tools. Auckland University of Technology. https://cpcr.aut.ac.nz/new-zealand-whoqol

The University of Auckland. (2019). Biomedical Science. https://www.auckland.ac.nz/en/study/study-options/find-a-study-option/biomedical-science.html

Tinto, V. (2009). Taking student retention seriously: Rethinking the first year of university. FYE curriculum design symposium. Queensland University of Technology, Australia.

Vansteenkiste, M., Lens, W., & Deci, E. L. (2006). Intrinsic versus extrinsic goal contents in self-determination theory: Another look at the quality of academic motivation. Educational Psychologist, 41(1), 19-31. https://doi.org/10.1207/s15326985ep4101_4

Verger, P., Combes, J.-B., Kovess-Masfety, V., Choquet, M., Guagliardo, V., Rouillon, F., & Peretti-Wattel, P. (2009). Psychological distress in first year university students: Socioeconomic and academic stressors, mastery and social support in young men and women. Social Psychiatry and Psychiatric Epidemiology, 44(8), 643-650. https://doi.org/10.1007/s00127-008-0486-y

WHOQoL Group. (1998). Development of the World Health Organization WHOQOL-BREF quality of life assessment. Psychological Medicine, 28(3), 551-558. https://doi.org/10.1017/S0033291798006667

Wickramasinghe, D. P., Almeida, I. S., & Samarasekera, D. N. (2019). Depression and stressful life events among medical students during undergraduate career: Findings from a medical school in South Asia. The Asia Pacific Scholar, 4(1), 42-47. https://doi.org/10.29060/TAPS.2019-4-1/OA2025

Yıldırım, A., Boysan, M., & Kefeli, M. C. (2018). Psychometric properties of the Turkish version of the Depression Anxiety Stress Scale-21 (DASS-21). British Journal of Guidance & Counselling, 46(5), 582-595. https://doi.org/10.1080/03069885.2018.1442558

Yusoff, M. S. B., Rahim, A. F. A., Baba, A. A., Ismail, S. B., Pa, M. N. M., & Esa, A. R. (2013). Prevalence and associated factors of stress, anxiety and depression among prospective medical students. Asian Journal of Psychiatry, 6(2), 128-133. https://doi.org/10.1016/j.ajp.2012.09.012

*Marcus A. Henning
The University of Auckland,
Private Bag 92019,
Auckland 1142, New Zealand
Tel: 0064 923 7392
Email: m.henning@auckland.ac.nz

Submitted: 26 January 2021
Accepted: 25 June 2021
Published online: 4 January, TAPS 2022, 7(1), 44-54
https://doi.org/10.29060/TAPS.2022-7-1/OA2471

Mitsumi Masuda¹, Machiko Saeki Yagi² & Fumino Sugiyama³

¹Nagoya City University, Nagoya, Aichi, Japan; ²Jichi Medical University, Shimotuke, Tochigi, Japan; ³National College of Nursing, Japan, Kiyose, Tokyo, Japan

Abstract

Introduction: Simulation-based learning (SBL) is a practical and efficient learning method that involves the replacement of a portion of clinical education with quality simulation experiences. It has been utilised in various countries, such as the United States, Canada, and South Korea. However, based on current regulations in Japan, clinical education cannot be replaced with simulation experience. For future curriculum integration, it is necessary to clarify the current use of SBL and tackle systematic educational strategies of SBL. Therefore, this national survey aimed to clarify the prevalence and practices of SBL in undergraduate nursing education programs in Japan.

Methods: This article presents the results of our national survey in Japan. It presents the questionnaire based on the International Nursing Association for Clinical Simulation and Learning Standards of Best Practice and demonstrates the use of simulation-based learning in Japanese undergraduate nursing programs.

Results: Overall, the schools using simulation-based education (SBE) comprised 346 schools (82.4%) of the sample. Those equipped with high-fidelity simulators were 146 schools (27.6%); the rest owned medium-fidelity simulators. Almost all undergraduate nursing education systems were equipped with simulators, however, the frequency of use was low. SBL was incorporated into the curriculum at many undergraduate nursing education institutions, and awareness of the INACSL Standard of Best Practice: Simulation^SM was extremely low.

Conclusion: This study shows that SBL is not properly utilised in undergraduate nursing programs, even though many schools are equipped with simulators. Thus, further study on barriers to simulator use is needed.

Keywords:           Simulation-based Learning, Curriculum, International Nursing Association for Clinical Simulation and Learning Standard of Best Practice: Simulation^SM, Japan, Undergraduate Nursing Education

Practice Highlights

• Almost all undergraduate nursing education programs use SBL.
• The frequency of use of SBL was low in almost all undergraduate nursing education systems.
• SBL was incorporated into the curriculum at many undergraduate nursing education institutions.
• Awareness of the INACSL Standard of Best Practice: Simulation^SM was extremely low.

Table 1. School type, entrant capacity, and number of faculty

B. SBL Use

1) SBL use and equipping a high-fidelity and a medium-fidelity simulator and task trainer use: Concerning the status of SBL use, 346 schools (82.4%) responded that they are using SBL, while 74 schools (17.6%) responded that they are not. Among all schools surveyed, 346 (82.4%) were equipped with a simulator for medical/nursing education, while 74 (17.6%) were not. Concerning simulator types, the most common high-fidelity simulator capable of modelling biological reactions and pathology and performing drug administration and other procedures was the SimMan^®, which was owned by 27 schools (6.4%) followed by a delivery simulator owned by 25 schools (6.0%). SCENARIO was also common (21 schools, 5.0%). SCENARIO is a high-fidelity simulator similar to SimMan^® which hit the market in Japan in 2017 but is cheaper than SimMan^®. Meanwhile, roughly 70% of schools (304 schools, 72.4%) were not equipped with a high-fidelity simulator. There were 361 schools (86.0%) equipped with at least one type of medium-fidelity simulator, allowing control of the creation of biological responses such as breath sounds, consciousness, and blood pressure. Regarding virtual reality (VR), 409 schools (97.4%) did not have VR facilities. VR refers to a computer-generated simulation in which a person can interact within an artificial three-dimensional environment using electronic devices, such as special goggles with a screen or gloves fitted with sensors. In this simulated artificial environment, the user is able to have a realistic-feeling experience. Concerning standardised patient (SP) simulation, 164 schools (39.0%) reported that they used SP while 254 schools (60.5%) did not, as shown in Table 2 (Masuda, 2021).

		n (%)	n (%)
			Foundation	Adult	Pediatric	Maternal	Geriatric	Psychiatric	Home
Using simulation-based education (schools)
	Yes	346 (82.4)
	No	74 (17.6)
Using simulation-based education (specialties)
	Yes	223 (53.1)	58 (16.8)	55 (15.9)	44 (12.7)	48 (13.9)	57 (16.5)	45 (13.0)	39 (11.3)
	No	197 (46.9)	22 (11.2)	27 (13.7)	20 (10.2)	19 (9.6)	40 (20.3)	43 (21.8)	26 (13.2)
Equipped with a simulator for medical/nursing education (schools)
	Yes	346 (82.4)
	No	74 (17.6)
Equipped with a high-fidelity simulator*1 (may select multiple answers)
	SimMan®	27 (6.4)	6 (22.2)	4 (14.8)	4 (14.8)	2 (7.4)	7 (25.9)	3 (11.1)	1 (3.7)
	iStan®	2 (0.5)	0 (0.0)	0 (0.0)	0 (0.0)	1 (50.0)	0 (0.0)	0 (0.0)	1 (50.0)
	Apollo™	1 (0.2)	0 (0.0)	0 (0.0)	0 (0.0)	0 (0.0)	0 (0.0)	0 (0.0)	1 (100.0)
	BabySim®	11 (2.6)	2 (18.2)	1 (9.1)	3 (27.3)	1 (9.1)	2 (18.2)	1 (9.1)	1 (9.1)
	Delivery simulator	25 (6.0)	3 (12.0)	3 (12.0)	1 (4.0)	7 (28.0)	4 (16.0)	3 (12.0)	4 (16.0)
	SCENARIO	21 (5.0)	7 (33.3)	5 (23.8)	2 (9.5)	2 (9.5)	2 (9.5)	3 (14.3)	0 (0.0)
	Equipped with other high-performance simulator	59 (14.0)	13 (22.0)	9 (15.3)	10 (16.9)	5 (8.5)	9 (15.3)	8 (13.6)	5 (8.5)
	Not equipped with a high-fidelity simulator	304 (72.4)	44 (14.5)	45 (14.8)	34 (11.2)	43 (14.1)	54 (17.8)	38 (12.5)	46 (15.1)
Equipped with a medium-fidelity simulator*2 (may select multiple answers)
	Physiko	265 (63.1)	61 (23.0)	50 (18.9)	21 (7.9)	25 (9.4)	44 (16.6)	26 (9.8)	38 (14.3)
	Nursing Anne®	46 (11.0)	7 (15.2)	11 (23.9)	3 (6.5)	6 (13.0)	5 (10.9)	7 (15.2)	7 (15.2)
	SimManALS®	13 (3.1)	4 (30.8)	3 (23.1)	0 (0.0)	0 (0.0)	1 (7.7)	4 (30.8)	1 (7.7)
	Other	37 (8.8)	10 (27.0)	5 (13.5)	9 (24.3)	9 (24.3)	2 (5.4)	1 (2.7)	1 (2.7)
	Not equipped with a medium-fidelity simulator	138 (32.9)	7 (5.1)	11 (8.0)	18 (13.0)	29 (21.0)	28 (20.3)	27 (19.6)	18 (13.0)
Equipped with a task trainer*3
	Yes	331 (78.8)	62 (18.7)	53 (16.0)	42 (12.7)	49 (14.8)	50 (15.1)	30 (9.1)	45 (13.6)
	No	89 (21.2)	7 (7.9)	11 (12.4)	7 (7.9)	9 (10.1)	22 (24.7)	22 (24.7)	11 (12.4)
Equipped with VR
	Yes	11 (2.6)	1 (9.1)	3 (27.3)	2 (18.2)	0 (0.0)	2 (18.2)	1 (9.1)	2 (18.2)
	No	409 (97.4)	68 (16.6)	61 (14.9)	47 (11.5)	58 (14.2)	70 (17.1)	51 (12.5)	54 (13.2)
Using standardised patients in simulation (specialties)
	Yes	164 (39.0)	38 (23.2)	30 (18.3)	16 (9.8)	22 (13.4)	27 (16.5)	10 (6.1)	21 (12.8)
	No	254 (60.5)	31 (12.2)	34 (13.4)	33 (13.0)	35 (13.8)	45 (17.7)	41 (16.1)	35 (13.8)

*1 High-fidelity simulator: The broad range of full body manikins that have the ability to mimic, at a very high level, human body functions (Lioce et al., 2020).
*2 Medium-fidelity simulator: The broad range of full body mannequins with installed human qualities such as breath sounds without chest rise (Smiley, 2019).
*3 Task trainer: A device designed to provide training in just the key elements of the procedure or skill being learned, such as lumbar puncture, or part of total system (Lioce et al., 2020).

Table 2. Simulation-based education use, equipping a high-fidelity and medium-fidelity simulator, and task trainer use

2) Frequency of simulator use and adoption in the curriculum: The application of SBL was infrequent with most schools (98 schools, 23.3%) using such programs once every year, followed by 97 schools (23.1%), which used SBL once every six months, and 72 schools (17.1%) using them once every two to three months, as shown in Table 3 (Masuda, 2021).

Table 3. Frequency of simulator use and adoption in the curriculum

C. The Status of INACSL Standards Application

Awareness of the INACSL Standards was low with over 90% of respondents stating, “I don’t know of them” (383 respondents, 91.2%) and only 8.8% (37 respondents) stating “I know of them.” Nonetheless, looking at specific items concerning simulation design, over half of the respondents reported using learning objectives, scenario design, methods for ensuring fidelity, learner levels and outcomes, and debriefing. However, rates of application were low for all other items, reaching only 20 to 30%, as shown in Supplementary Table S1 (Masuda, 2021).

IV. DISCUSSION

A. The Status of SBL Use

We surveyed the status of SBL use at nursing schools throughout Japan. The results revealed that 82.4% of those surveyed, use simulation-based education, validating that SBL is widespread in undergraduate nursing education. A nationwide survey in the United States in 2015 found the rate to be 99% (Breymier et al., 2015). Thus, although SBL is gaining popularity in Japan, the country continues to trail the United States in overall usage. Concerning the use of SBL by specialty, simulation is commonly used in foundation of nursing, followed by adult nursing and geriatric nursing. Meanwhile, simulation is used by less than 50% of schools for maternal nursing, pediatric nursing, psychiatric nursing, and home nursing. In countries such as the United States and South Korea, it is primarily used for specialties that require clinical practice in hospitals, such as adult nursing, and clinical nursing, but use for psychiatric and home nursing is limited (Kardong-Edgren et al., 2012; Shin et al., 2015). Thus, our findings were similar to those in these previous studies (Kardong-Edgren et al., 2012; Shin et al., 2015).  The less frequent use of psychiatric and home nursing may be due to the inappropriate use of simulators such as SimMan^® for medical field or the limited number of simulators in schools.

Concerning schools being equipped with simulators, roughly only 30% of schools had a high-performance simulator, but in comparison mid-level performance simulators were common, being present at approximately 80% of schools. Previous studies (Hayden, 2010; Smiley, 2019) and a meta-analysis (Kim et al., 2016) have shown that roughly 90% of schools implement programs, using either a high- or medium-fidelity simulator. It seems that Japan is catching up to countries with advanced SBL in terms of access to high- and medium-fidelity simulators. In addition, it is that the results were similar to 77.8% equipped with a task trainer of the schools. Nonetheless, the results of the present survey revealed that the frequency of use remains low. In other words, our findings suggest that although Japanese schools are sufficiently equipped with simulators, they are not optimally used.

Simulators are an expensive instructional tool, but it is not enough to simply purchase them. It is essential to further study the obstacles to their application and prepare an environment in which their use is possible. While high- and medium-fidelity simulators from Laerdal Medical were common, use of SCENARIO—a simulator put on the market by Kyoto Kagaku Co., Ltd. in 2017—is rapidly increasing. This rise may be influenced by the fact that SCENARIO is made in Japan. Moreover, not only is SCENARIO lightweight and relatively cheap, it also features native Japanese instructions reflecting the clinical situation, culture, and background of Japan rather than scenarios and an instruction manual written in English. Regarding simulator access by different specialties, it is only natural that the results matched those for the status of simulator-based education use, with foundation of nursing, adult nursing, and geriatric nursing having the most simulators. The rate of 39% of SP utilisation found in this survey was on par with the rate of 36% in bachelor’s programs in the United States (Kardong-Edgren et al., 2012), and a systematic review of 40 published studies from countries such as the United States, South Korea, UK, Australia, found that 25% used SPs (Kim et al., 2016). On the other hand, our results revealed that VR is virtually unused in undergraduate nursing education in Japan. This lags behind the US results of 25% for use in internet-based virtual hospitals and 34% for virtual intravenous injection programs (Kardong-Edgren et al., 2012).

Regarding the adoption of SBL into the curriculum, over half of the schools surveyed featured SBL in their present curriculum, and this inclusion can be expected to increase further as schools plan to transition to new curriculums shortly. Integration of simulation-based education into the curriculum is already in progress in the United States with studies finding that up to 50% of the time spent on clinical practice could be replaced with simulation-based learning with no change in learning outcomes (Hayden et al., 2014). Studies exploring how much of the curriculum has been replaced are also underway (Breymier et al., 2015; Gore et al., 2012; Hayden, 2010; Smiley, 2019). Based on current regulations of Ministry of Education, Culture, Sports, Science and Technology of Japan, the clinical experience cannot be replaced by simulation but is expected to become possible in the future as curriculum integration progresses. As such, integration of simulation into a curriculum requires a meticulous review of the program of study to identify gaps where simulation best fits to increase learning (Hodge et al., 2008). Additionally, past research on barriers to curriculum integration of simulation-based experience found that lack of time for faculty development was the primary barrier (Adamson, 2015; Sole et al., 2013). Therefore, it will be necessary to account for all obstacles to curriculum integration.

B. The Status of the INACSL Standards Application

Among the theoretical frameworks of SBLs, it was inferred that the Japanese version was the INACSL Standards to be developed and relatively known. Therefore, in this study, we investigated the awareness of the INACSL Standards. As a result, unfortunately, we found that awareness was extremely low and that these standards are not widespread in Japanese undergraduate nursing education. However, the results also suggested that over half of respondents implemented learning objectives, scenario design, methods for ensuring fidelity, learner levels and outcomes, and debriefing on simulation design. A survey of the status of simulation used by the INACSL found the use of conceptual frameworks and theories to be approximately 50% and described this result as extremely low (Beroz, 2017). The results of the present survey found an even lower rate with roughly 90% of respondents stating that they did not know of the INACSL Standards. However, even without having prior knowledge regarding the INACSL Standards, over half of the respondents were following the best practice standards in terms of simulation design. Therefore, it is imperative to correctly understand and apply theories and conceptual frameworks to teaching strategies rather than to know their official names. Nearly half of the educators surveyed were designing their simulation programs according to the INACSL Standards, which is considered a positive result. Despite these positive results for simulation design, other items were found to have low utilisation. It is thought that further popularisation of other items will lead to a comprehensive understanding of simulation program design.

C. Study Limitations

Online surveys are not yet widespread in undergraduate nursing education in Japan. It is also presumed that in Japanese undergraduate nursing educations, the coordinators of the nursing specialties are often held by professors or associate professors, and that the age groups of the subjects were relatively high. Therefore, it is considered that the participants may not be familiar with the online surveys. The inclusion criterion for subjects should have included someone familiar with on-line manipulations such as SurveyMonkey. In addition, it may be a deficiency in due to the lengthiness of the questionnaire, improper selection of respondents, response items or the data collection period has not been sufficiently taken. This aspect was the bias of this survey and may explain the low response rate.

In addition, the development of the questionnaire used in this study is limited to content validation, and no examination of construct validity has been conducted. Therefore, there is a limitation that the validity of the questionnaire is insufficient.

Regarding the classification of simulators, in our study, SimMan ALS^® was included in a medium-fidelity simulator, and SimMan^® was included in a high-fidelity simulator. However, SimManALS^® can also be captured as a high-fidelity simulator. This may have influenced the scoring of “others”. In this study, a completed survey was conducted, and randomisation of confounders was not possible.

Therefore, based on the results obtained in this study, we will plan a more focused survey in the future. We suggest that the results of this study can be used to promote SBL in a more specific method.

Lastly, in 2020-21, opportunities for nursing students to complete practical training in medical facilities are limited as a result of the global pandemic of COVID-19. Hence, it can be said that SBL will play a major role in maintaining learning opportunities and clinical competence in students while ensuring the safety of patients, students, and medical staff. We plan to continue and expand upon our survey research to enable more educational institutions to begin practicing SBL.

V. CONCLUSION

In conclusion, this study provided an initial view of the current status of SBL in Japan. Our results suggested that while many schools are equipped with simulators, they are not properly utilised in the foundation of nursing programs in Japan. Thus, further study of barriers to simulator use is needed. Also, although awareness of INACSL Standards was extremely low, over half of the respondents were designing simulations following the standards, implying that they were making use of some theories or conceptual frameworks in their designs.  Our results act as a foundational resource for studying strategies intended to systemise SBL in Japanese undergraduate nursing education. In the future, it will be necessary to specifically survey awareness and use of theories and conceptual frameworks to recommend methods for increasing their application and use in SBL.

Notes on Contributors

Dr. Mitsumi Masuda, PhD, RN, is an associate professor at the Graduate School of Nursing, Nagoya City University. She reviewed the literature, designed the study, performed data collection, data analysis and wrote the manuscript.

Dr. Machiko Saeki Yagi, MS, RN, is a lecturer at the School of Nursing, Jichi Medical University. She developed the methodological framework for the study, performed data collection, data analysis and gave critical feedback to the writing of the manuscript.

Dr. Fumino Sugiyama, PhD, RN, is an associate professor at the School of Nursing, National College of Nursing. She performed data collection and data analysis. All the authors have read and approved the final manuscript.

Ethical Approval

This research was approved by the research ethics review committee of the Nagoya City University, Graduate School of Nursing (Approval no.: 19019-2).

Data Availability

Datasets generated and/or analysed during the current study are available from the following DOI. https://doi.org/10.6084/m9.figshare.14010755.v1.

Acknowledgement

This study was completed with the support of The Murata Science Foundation (2019 No. 28). In addition, we would like to thank Editage (www.editage.com) for assisting with English language editing. We would like to offer our heartfelt thanks to the nursing schools across the country who participated in this study.

Zarifsanaiey, N., Amini, M., & Saadat, F. (2016). A comparison of educational strategies for the acquisition of nursing student’s performance and critical thinking: Simulation-based training vs. integrated training (simulation and critical thinking strategies). BMC Medical Education, 16(1), 294. https://doi.org/10.1186/s12909-016-0812-0

*Mitsumi Masuda
1, Kawasumi, Mizuho-cho,
Mizuho-ku, Nagoya,
467-8601, Japan.
Tel: +81-52-853-8063
Email: m.masuda@med.nagoya-cu.ac.jp

Submitted: 19 February 2021
Accepted: 17 June 2021
Published online: 4 January, TAPS 2022, 7(1), 33-43
https://doi.org/10.29060/TAPS.2022-7-1/OA2493

Chan Choong Foong*, An Jie Lye*, Che Rafidah Aziz, Wei-Han Hong, Vinod Pallath, Jessica Grace Cockburn, Siti Nurjawahir Rosli, Kuhan Krishnan, Prahaladhan Sivalingam, Noor Filzati Zulkepli & Jamuna Vadivelu

Medical Education & Research Development Unit (MERDU), Faculty of Medicine, University of Malaya, Malaysia

*Joint first authors

Abstract

Introduction: Medical schools universally responded by migrating teaching and learning to virtual learning environments (VLE) due to the impact of the COVID-19 pandemic. The use of virtual problem-based learning (PBL) in lieu of face-to-face sessions seems to be an appropriate response, but its effectiveness was understudied. The study compared the learning experiences of pre-clinical medical students at the beginning and completion of the virtual PBL.

Methods: The study was conducted at the University of Malaya, a public-funded university in Malaysia. A 12-item questionnaire was developed and validated to assess the learning experiences of students conducting virtual PBL sessions. Principal component analysis and test for internal consistency suggested that the questionnaire is valid and reliable. The questionnaire was administered to pre-clinical students (Year 1 and Year 2) twice: at the beginning and the end of the virtual PBL implementation. Their responses were compared for the domains “learning”, “confidence” and “concern”.

Results: Three hundred and forty-four pre-clinical students were recruited but only 275 students (80%) responded to both the initial and final questionnaires. Based on the responses, the learning experiences of students generally improved by the completion of the virtual PBL implementation. Students were most convinced that they obtained and understood the information given during the virtual PBL. However, they continued to be worried about passing the clinical examination and content acquisition.

Conclusion: The study supports the feasibility of virtual PBL as an acceptable alternative to replace face-to-face PBL during the COVID-19 pandemic.

Keywords:           Problem-based Learning, COVID-19, Undergraduate Medical Education, Virtual Learning Environment

Practice Highlights

• The study compared the learning experiences of pre-clinical medical students in virtual Problem Based Learning (PBL).
• The learning experiences of students in virtual PBL generally improved.
• Students obtained and understood the information given during the virtual PBL.
• Students continued to be worried about passing the clinical examination and content acquisition.
• Virtual PBL could be an acceptable alternative to replace face-to-face PBL during the pandemic.

Submitted: 31 August 2020
Accepted: 8 February 2021
Published online: 5 October, TAPS 2021, 6(4), 118-130
https://doi.org/10.29060/TAPS.2021-6-4/OA2481

Isabella E. Supnet, Jose Alvin P. Mojica, Sharon D. Ignacio & Carl Froilan D. Leochico

Department of Rehabilitation Medicine, College of Medicine and Philippine General Hospital, University of the Philippines Manila, Philippines

Abstract

Introduction: In the Philippines, telerehabilitation has been at the forefront of integrating telemedicine into the medical curriculum. However, the course evaluation tool used for traditional classroom-based courses is not appropriate in evaluating the unique teaching-learning tool that is telerehabilitation. This study aimed to develop a questionnaire that will aid in addressing this gap.

Method: A mixed methods study was devised to gather information from medical students exposed to telerehabilitation from the College of Medicine, University of the Philippines Manila as well as the residents from the Department of Rehabilitation Medicine, Philippine General Hospital from October to November 2019. The investigators obtained informed consent from all participants as well as their demographics before undergoing interviews. Themes were identified to create questions under the previously identified constructs, along with items derived from other course evaluation tools and opinions gathered from experts in telerehabilitation.

Results: In total, 26 individuals participated in the study. Most of the respondents had experience or ownership of various communication technologies and were well-versed in communication strategies through these technologies. There were 52 questions formulated from the interviews and review of previous tools.

Conclusion: This study is the first step in providing more research in the student evaluation of telerehabilitation and telemedicine. To match the needs of changing times evaluation of new standards and methods should follow. More research must be done to standardise teaching evaluation tools to validate the data gathered, and allow courses, such as telerehabilitation, an opportunity to adapt and promote further learning.

Keywords:           Assessment, Educational, Education, Medical, Telemedicine, Medical Informatics, Physical and Rehabilitation Medicine

Practice Highlights

• Telemedicine has been used as a tool to educate students on rehabilitation medicine.
• Almost all students at this time have had experience in the usage of information and communication technologies.
• Evaluation tools must be more specific to the method of instruction used.
• Clarity, congruence, and relevance are the most sought-after characteristics in telerehab courses.
• The recipient of the evaluation must account for the effect of the teacher or logistical concerns.

I. INTRODUCTION

A. Background

Telemedicine is defined as the use of advanced telecommunications technologies to exchange health information and provide health care services across geographic, temporal, social and cultural barriers (Myers, 2003). It has been widely used in the treatment and care of patients as we go through the COVID-19 pandemic and has also been vital in improving medical education in the age of online classes and social distancing. Through telemedicine, students are exposed to a variety of specialties, and gain experience in their observation of the management of diseases, such as in the case of telesurgery, or hone their skills in communication and counselling, such as through telerehabilitation (Jumreornvong et al., 2020).

Now more than ever, due to the capacity of students to interact and adapt with digital infrastructure (Pathipati et al., 2016), integration of telemedicine into the medical curriculum is increasingly important and evaluations of telemedicine curricula are necessary to ensure quality and to detect areas for growth and improvement. Previous studies have used general evaluation forms to evaluate their telemedicine programmes or created general surveys from their course objectives, which have yielded some valuable insights (Brockes et al., 2017; Bulik & Shokar, 2010). However, due to the nature of instruction of telemedicine, specialised and validated tools are necessary to provide a comprehensive assessment.

In the Philippines, telerehabilitation has been at the forefront of integrating telemedicine into the curriculum, particularly in the University of the Philippines. Because it has been four years since the start of the initiative, and one year since its implementation, it is important to evaluate the previous courses in order to improve them for the next generation.

Through this research, a preliminary evaluation questionnaire for telerehabilitation as a teaching-learning tool was developed. From this questionnaire, perceptions of students regarding telerehabilitation courses will be more efficiently gathered and evaluated, which will serve to further improve the telerehabilitation curriculum and possibly bring forth interventions to improve medical education in general.

B. Literature Review

1) Teaching programmes in telemedicine: In the process of conducting telemedical evaluations in order to aid patients, clinical training may also be received. Telemedicine has had good acceptance in training institutions abroad, with Neurology trainees agreeing that it should be part of their curriculum and supporting a formalised telemedicine rotation within their residency. Dermatology programmes in particular observed that telemedicine supported rather than detracted from the core competencies required from them (Lee & Nambudiri, 2019). No significant differences were seen between clinical outcomes of patients who underwent surgery through telementoring versus the traditional method, and case supervision using e-mails and voice-over applications afforded similar psychiatry education as compared to rotations in mental health clinics (O′Shea et al., 2015).

The methods through which telemedicine education is given also vary considerably per institution (Waseh & Dicker, 2019). Some schools have telemedicine included primarily as didactic sessions. Others allow their students to take part in patient encounters and interprofessional training. Some institutions also allow scholarly projects to be done in telemedicine.

Because telemedicine in medical education has yet to be explored formally, no evaluation tools have been developed to assess its application. Other institutions used generalised forms and made their own questionnaires (Brockes et al., 2017; Bulik & Shokar, 2010). Literature review for formally developed evaluation materials showed the presence of a Telehealth Usability Questionnaire, with items and domains encompassing many telehealth assessment tools (Parmanto et al., 2016). It was made for use with various types of telehealth systems, including computer-based systems, videoconferencing programmes, and adaptable for progressive innovations, particularly for mobile telemedicine applications. However, this was primarily utilised as an assessment tool between clinicians and patients and does not assess instruction on the usage of the programme or the organisation of the implementation.

2) Teaching programmes in telerehabilitation: In 2015, the Department of Rehabilitation Medicine, College of Medicine and Philippine General Hospital, University of the Philippines Manila (PGH DRM) has initiated education on this aspect of telemedicine, with telerehabilitation used as a teaching-learning tool for medical students in full implementation in the 2018 curriculum. The programme has expanded from its origins since then; from students engaging in telerehabilitation to actual telerehabilitation consultations and teletherapy services with the rural health unit of Alfonso, Cavite, as part of the University of the Philippines Community Health and Development Programme (UP CHDP).

Telerehabilitation as a teaching-learning tool was formally included as part of the curriculum for rehabilitation medicine in 2018 (Philippine General Hospital Department of Rehabilitation Medicine, 2018). The students start their engagement during their third year of medical school (known as Learning Unit 5), their first year of clinical exposure, with the concepts and theories behind telerehabilitation, and do observations of an actual telerehabilitation session. A year later, during Learning Unit 6, they then get to do a simulated telerehabilitation encounter, with guidance and techniques on how to present a rehabilitation case through telemedicine. During their final year of medical education (Learning Unit 7), the student is then evaluated on the presentation, evaluation and management given during an actual patient encounter. All students from the UP College of Medicine receive two sessions of telerehabilitation instruction per year.  On the other hand, Post-Graduate Interns (PGIs), who are students who have received their first four years of medical education in other medical schools and have chosen to spend their last year of medical school in the Philippine General Hospital, receive only one compressed session. Overall, the participants reported the experience to be excellent, and should be explored further (Leochico & Mojica, 2017).

Telerehabilitation was also included in January 2019 as a formal service for training residents in the Department of Rehabilitation Medicine, featuring the same concepts in a more compressed manner, and highlighting the hybridisation of standard rehabilitation practice with telemedicine.

The initiative is currently being evaluated by the students in part through the Course Evaluation by Students (CEBS) given by the University of the Philippines College of Medicine (UPCM) (University of the Philippines Manila College of Medicine, 2005), and through a Devised Telerehabilitation Feedback Form for Students (Philippine General Hospital Department of Rehabilitation Medicine, n.d.), which does not distinguish usability attributes of the telerehabilitation system from the telerehabilitation curriculum itself.

3) The usage of student evaluations: Student evaluation forms are a commonly used tool in determining teacher and course effectiveness in many areas of higher education. These evaluations commonly serve three purposes: to improve teaching quality, to support faculty decisions, and to provide evidence for institutional accountability. Many evaluation measures have been constructed and performed with students as the main and sometimes sole indicator of education quality; however, the implementation, reliability and validity of these methods and instruments have been a source of concern within academic circles (Soto-Estrada et al., 2018). Due to this, student evaluation remains a field of active study.

Although helpful, some caution must be undertaken in interpreting results of student evaluations. A recent meta-analysis by Uttl et al. (2017) argues that studies looking into student evaluations of teaching (SETs) were scant in terms of data to support the equivalence of high student evaluation ratings to student performance, and positive correlations between the two were primarily due to small study size effects. However, it is unfair to assume that SETs have no value whatsoever. Responses to the meta-analysis have been published as well, and aside from critiquing the method Uttl used to form his study, they support the view that, while it is poor practice to use student ratings of instruction alone in evaluating teaching, it remains to be necessary (Ryalls et al., n.d.).

C. Objectives

1) General objective:

• To develop a questionnaire that will evaluate telerehabilitation as a teaching-learning tool for medical students.

2) Specific objectives:

• To determine themes regarding the evaluation of telerehabilitation as a teaching-learning tool from residents of the PGH Department of Rehabilitation Medicine, as well from the medical students who have undergone the telemedicine courses.
• To formulate questionnaire items for the telerehabilitation course evaluation tool.

II. METHODS

A mixed methods study was devised to gather information from medical students exposed to telerehabilitation from the Philippine General Hospital and the College of Medicine, University of the Philippines Manila (with exposures corresponding to Learning Units 5, 6, and 7) as well as the residents from the PGH DRM from October to November 2019 (Figure 1). All participants were greater than 18 years old.

Figure 1. Flowchart of study methodology