Submitted: 26 January 2021
Accepted: 25 June 2021
Published online: 4 January, TAPS 2022, 7(1), 44-54
https://doi.org/10.29060/TAPS.2022-7-1/OA2471

Mitsumi Masuda¹, Machiko Saeki Yagi² & Fumino Sugiyama³

¹Nagoya City University, Nagoya, Aichi, Japan; ²Jichi Medical University, Shimotuke, Tochigi, Japan; ³National College of Nursing, Japan, Kiyose, Tokyo, Japan

Abstract

Introduction: Simulation-based learning (SBL) is a practical and efficient learning method that involves the replacement of a portion of clinical education with quality simulation experiences. It has been utilised in various countries, such as the United States, Canada, and South Korea. However, based on current regulations in Japan, clinical education cannot be replaced with simulation experience. For future curriculum integration, it is necessary to clarify the current use of SBL and tackle systematic educational strategies of SBL. Therefore, this national survey aimed to clarify the prevalence and practices of SBL in undergraduate nursing education programs in Japan.

Methods: This article presents the results of our national survey in Japan. It presents the questionnaire based on the International Nursing Association for Clinical Simulation and Learning Standards of Best Practice and demonstrates the use of simulation-based learning in Japanese undergraduate nursing programs.

Results: Overall, the schools using simulation-based education (SBE) comprised 346 schools (82.4%) of the sample. Those equipped with high-fidelity simulators were 146 schools (27.6%); the rest owned medium-fidelity simulators. Almost all undergraduate nursing education systems were equipped with simulators, however, the frequency of use was low. SBL was incorporated into the curriculum at many undergraduate nursing education institutions, and awareness of the INACSL Standard of Best Practice: Simulation^SM was extremely low.

Conclusion: This study shows that SBL is not properly utilised in undergraduate nursing programs, even though many schools are equipped with simulators. Thus, further study on barriers to simulator use is needed.

Keywords:           Simulation-based Learning, Curriculum, International Nursing Association for Clinical Simulation and Learning Standard of Best Practice: Simulation^SM, Japan, Undergraduate Nursing Education

Practice Highlights

• Almost all undergraduate nursing education programs use SBL.
• The frequency of use of SBL was low in almost all undergraduate nursing education systems.
• SBL was incorporated into the curriculum at many undergraduate nursing education institutions.
• Awareness of the INACSL Standard of Best Practice: Simulation^SM was extremely low.

Table 1. School type, entrant capacity, and number of faculty

B. SBL Use

1) SBL use and equipping a high-fidelity and a medium-fidelity simulator and task trainer use: Concerning the status of SBL use, 346 schools (82.4%) responded that they are using SBL, while 74 schools (17.6%) responded that they are not. Among all schools surveyed, 346 (82.4%) were equipped with a simulator for medical/nursing education, while 74 (17.6%) were not. Concerning simulator types, the most common high-fidelity simulator capable of modelling biological reactions and pathology and performing drug administration and other procedures was the SimMan^®, which was owned by 27 schools (6.4%) followed by a delivery simulator owned by 25 schools (6.0%). SCENARIO was also common (21 schools, 5.0%). SCENARIO is a high-fidelity simulator similar to SimMan^® which hit the market in Japan in 2017 but is cheaper than SimMan^®. Meanwhile, roughly 70% of schools (304 schools, 72.4%) were not equipped with a high-fidelity simulator. There were 361 schools (86.0%) equipped with at least one type of medium-fidelity simulator, allowing control of the creation of biological responses such as breath sounds, consciousness, and blood pressure. Regarding virtual reality (VR), 409 schools (97.4%) did not have VR facilities. VR refers to a computer-generated simulation in which a person can interact within an artificial three-dimensional environment using electronic devices, such as special goggles with a screen or gloves fitted with sensors. In this simulated artificial environment, the user is able to have a realistic-feeling experience. Concerning standardised patient (SP) simulation, 164 schools (39.0%) reported that they used SP while 254 schools (60.5%) did not, as shown in Table 2 (Masuda, 2021).

		n (%)	n (%)
			Foundation	Adult	Pediatric	Maternal	Geriatric	Psychiatric	Home
Using simulation-based education (schools)
	Yes	346 (82.4)
	No	74 (17.6)
Using simulation-based education (specialties)
	Yes	223 (53.1)	58 (16.8)	55 (15.9)	44 (12.7)	48 (13.9)	57 (16.5)	45 (13.0)	39 (11.3)
	No	197 (46.9)	22 (11.2)	27 (13.7)	20 (10.2)	19 (9.6)	40 (20.3)	43 (21.8)	26 (13.2)
Equipped with a simulator for medical/nursing education (schools)
	Yes	346 (82.4)
	No	74 (17.6)
Equipped with a high-fidelity simulator*1 (may select multiple answers)
	SimMan®	27 (6.4)	6 (22.2)	4 (14.8)	4 (14.8)	2 (7.4)	7 (25.9)	3 (11.1)	1 (3.7)
	iStan®	2 (0.5)	0 (0.0)	0 (0.0)	0 (0.0)	1 (50.0)	0 (0.0)	0 (0.0)	1 (50.0)
	Apollo™	1 (0.2)	0 (0.0)	0 (0.0)	0 (0.0)	0 (0.0)	0 (0.0)	0 (0.0)	1 (100.0)
	BabySim®	11 (2.6)	2 (18.2)	1 (9.1)	3 (27.3)	1 (9.1)	2 (18.2)	1 (9.1)	1 (9.1)
	Delivery simulator	25 (6.0)	3 (12.0)	3 (12.0)	1 (4.0)	7 (28.0)	4 (16.0)	3 (12.0)	4 (16.0)
	SCENARIO	21 (5.0)	7 (33.3)	5 (23.8)	2 (9.5)	2 (9.5)	2 (9.5)	3 (14.3)	0 (0.0)
	Equipped with other high-performance simulator	59 (14.0)	13 (22.0)	9 (15.3)	10 (16.9)	5 (8.5)	9 (15.3)	8 (13.6)	5 (8.5)
	Not equipped with a high-fidelity simulator	304 (72.4)	44 (14.5)	45 (14.8)	34 (11.2)	43 (14.1)	54 (17.8)	38 (12.5)	46 (15.1)
Equipped with a medium-fidelity simulator*2 (may select multiple answers)
	Physiko	265 (63.1)	61 (23.0)	50 (18.9)	21 (7.9)	25 (9.4)	44 (16.6)	26 (9.8)	38 (14.3)
	Nursing Anne®	46 (11.0)	7 (15.2)	11 (23.9)	3 (6.5)	6 (13.0)	5 (10.9)	7 (15.2)	7 (15.2)
	SimManALS®	13 (3.1)	4 (30.8)	3 (23.1)	0 (0.0)	0 (0.0)	1 (7.7)	4 (30.8)	1 (7.7)
	Other	37 (8.8)	10 (27.0)	5 (13.5)	9 (24.3)	9 (24.3)	2 (5.4)	1 (2.7)	1 (2.7)
	Not equipped with a medium-fidelity simulator	138 (32.9)	7 (5.1)	11 (8.0)	18 (13.0)	29 (21.0)	28 (20.3)	27 (19.6)	18 (13.0)
Equipped with a task trainer*3
	Yes	331 (78.8)	62 (18.7)	53 (16.0)	42 (12.7)	49 (14.8)	50 (15.1)	30 (9.1)	45 (13.6)
	No	89 (21.2)	7 (7.9)	11 (12.4)	7 (7.9)	9 (10.1)	22 (24.7)	22 (24.7)	11 (12.4)
Equipped with VR
	Yes	11 (2.6)	1 (9.1)	3 (27.3)	2 (18.2)	0 (0.0)	2 (18.2)	1 (9.1)	2 (18.2)
	No	409 (97.4)	68 (16.6)	61 (14.9)	47 (11.5)	58 (14.2)	70 (17.1)	51 (12.5)	54 (13.2)
Using standardised patients in simulation (specialties)
	Yes	164 (39.0)	38 (23.2)	30 (18.3)	16 (9.8)	22 (13.4)	27 (16.5)	10 (6.1)	21 (12.8)
	No	254 (60.5)	31 (12.2)	34 (13.4)	33 (13.0)	35 (13.8)	45 (17.7)	41 (16.1)	35 (13.8)

*1 High-fidelity simulator: The broad range of full body manikins that have the ability to mimic, at a very high level, human body functions (Lioce et al., 2020).
*2 Medium-fidelity simulator: The broad range of full body mannequins with installed human qualities such as breath sounds without chest rise (Smiley, 2019).
*3 Task trainer: A device designed to provide training in just the key elements of the procedure or skill being learned, such as lumbar puncture, or part of total system (Lioce et al., 2020).

Table 2. Simulation-based education use, equipping a high-fidelity and medium-fidelity simulator, and task trainer use

2) Frequency of simulator use and adoption in the curriculum: The application of SBL was infrequent with most schools (98 schools, 23.3%) using such programs once every year, followed by 97 schools (23.1%), which used SBL once every six months, and 72 schools (17.1%) using them once every two to three months, as shown in Table 3 (Masuda, 2021).

Table 3. Frequency of simulator use and adoption in the curriculum

C. The Status of INACSL Standards Application

Awareness of the INACSL Standards was low with over 90% of respondents stating, “I don’t know of them” (383 respondents, 91.2%) and only 8.8% (37 respondents) stating “I know of them.” Nonetheless, looking at specific items concerning simulation design, over half of the respondents reported using learning objectives, scenario design, methods for ensuring fidelity, learner levels and outcomes, and debriefing. However, rates of application were low for all other items, reaching only 20 to 30%, as shown in Supplementary Table S1 (Masuda, 2021).

IV. DISCUSSION

A. The Status of SBL Use

We surveyed the status of SBL use at nursing schools throughout Japan. The results revealed that 82.4% of those surveyed, use simulation-based education, validating that SBL is widespread in undergraduate nursing education. A nationwide survey in the United States in 2015 found the rate to be 99% (Breymier et al., 2015). Thus, although SBL is gaining popularity in Japan, the country continues to trail the United States in overall usage. Concerning the use of SBL by specialty, simulation is commonly used in foundation of nursing, followed by adult nursing and geriatric nursing. Meanwhile, simulation is used by less than 50% of schools for maternal nursing, pediatric nursing, psychiatric nursing, and home nursing. In countries such as the United States and South Korea, it is primarily used for specialties that require clinical practice in hospitals, such as adult nursing, and clinical nursing, but use for psychiatric and home nursing is limited (Kardong-Edgren et al., 2012; Shin et al., 2015). Thus, our findings were similar to those in these previous studies (Kardong-Edgren et al., 2012; Shin et al., 2015).  The less frequent use of psychiatric and home nursing may be due to the inappropriate use of simulators such as SimMan^® for medical field or the limited number of simulators in schools.

Concerning schools being equipped with simulators, roughly only 30% of schools had a high-performance simulator, but in comparison mid-level performance simulators were common, being present at approximately 80% of schools. Previous studies (Hayden, 2010; Smiley, 2019) and a meta-analysis (Kim et al., 2016) have shown that roughly 90% of schools implement programs, using either a high- or medium-fidelity simulator. It seems that Japan is catching up to countries with advanced SBL in terms of access to high- and medium-fidelity simulators. In addition, it is that the results were similar to 77.8% equipped with a task trainer of the schools. Nonetheless, the results of the present survey revealed that the frequency of use remains low. In other words, our findings suggest that although Japanese schools are sufficiently equipped with simulators, they are not optimally used.

Simulators are an expensive instructional tool, but it is not enough to simply purchase them. It is essential to further study the obstacles to their application and prepare an environment in which their use is possible. While high- and medium-fidelity simulators from Laerdal Medical were common, use of SCENARIO—a simulator put on the market by Kyoto Kagaku Co., Ltd. in 2017—is rapidly increasing. This rise may be influenced by the fact that SCENARIO is made in Japan. Moreover, not only is SCENARIO lightweight and relatively cheap, it also features native Japanese instructions reflecting the clinical situation, culture, and background of Japan rather than scenarios and an instruction manual written in English. Regarding simulator access by different specialties, it is only natural that the results matched those for the status of simulator-based education use, with foundation of nursing, adult nursing, and geriatric nursing having the most simulators. The rate of 39% of SP utilisation found in this survey was on par with the rate of 36% in bachelor’s programs in the United States (Kardong-Edgren et al., 2012), and a systematic review of 40 published studies from countries such as the United States, South Korea, UK, Australia, found that 25% used SPs (Kim et al., 2016). On the other hand, our results revealed that VR is virtually unused in undergraduate nursing education in Japan. This lags behind the US results of 25% for use in internet-based virtual hospitals and 34% for virtual intravenous injection programs (Kardong-Edgren et al., 2012).

Regarding the adoption of SBL into the curriculum, over half of the schools surveyed featured SBL in their present curriculum, and this inclusion can be expected to increase further as schools plan to transition to new curriculums shortly. Integration of simulation-based education into the curriculum is already in progress in the United States with studies finding that up to 50% of the time spent on clinical practice could be replaced with simulation-based learning with no change in learning outcomes (Hayden et al., 2014). Studies exploring how much of the curriculum has been replaced are also underway (Breymier et al., 2015; Gore et al., 2012; Hayden, 2010; Smiley, 2019). Based on current regulations of Ministry of Education, Culture, Sports, Science and Technology of Japan, the clinical experience cannot be replaced by simulation but is expected to become possible in the future as curriculum integration progresses. As such, integration of simulation into a curriculum requires a meticulous review of the program of study to identify gaps where simulation best fits to increase learning (Hodge et al., 2008). Additionally, past research on barriers to curriculum integration of simulation-based experience found that lack of time for faculty development was the primary barrier (Adamson, 2015; Sole et al., 2013). Therefore, it will be necessary to account for all obstacles to curriculum integration.

B. The Status of the INACSL Standards Application

Among the theoretical frameworks of SBLs, it was inferred that the Japanese version was the INACSL Standards to be developed and relatively known. Therefore, in this study, we investigated the awareness of the INACSL Standards. As a result, unfortunately, we found that awareness was extremely low and that these standards are not widespread in Japanese undergraduate nursing education. However, the results also suggested that over half of respondents implemented learning objectives, scenario design, methods for ensuring fidelity, learner levels and outcomes, and debriefing on simulation design. A survey of the status of simulation used by the INACSL found the use of conceptual frameworks and theories to be approximately 50% and described this result as extremely low (Beroz, 2017). The results of the present survey found an even lower rate with roughly 90% of respondents stating that they did not know of the INACSL Standards. However, even without having prior knowledge regarding the INACSL Standards, over half of the respondents were following the best practice standards in terms of simulation design. Therefore, it is imperative to correctly understand and apply theories and conceptual frameworks to teaching strategies rather than to know their official names. Nearly half of the educators surveyed were designing their simulation programs according to the INACSL Standards, which is considered a positive result. Despite these positive results for simulation design, other items were found to have low utilisation. It is thought that further popularisation of other items will lead to a comprehensive understanding of simulation program design.

C. Study Limitations

Online surveys are not yet widespread in undergraduate nursing education in Japan. It is also presumed that in Japanese undergraduate nursing educations, the coordinators of the nursing specialties are often held by professors or associate professors, and that the age groups of the subjects were relatively high. Therefore, it is considered that the participants may not be familiar with the online surveys. The inclusion criterion for subjects should have included someone familiar with on-line manipulations such as SurveyMonkey. In addition, it may be a deficiency in due to the lengthiness of the questionnaire, improper selection of respondents, response items or the data collection period has not been sufficiently taken. This aspect was the bias of this survey and may explain the low response rate.

In addition, the development of the questionnaire used in this study is limited to content validation, and no examination of construct validity has been conducted. Therefore, there is a limitation that the validity of the questionnaire is insufficient.

Regarding the classification of simulators, in our study, SimMan ALS^® was included in a medium-fidelity simulator, and SimMan^® was included in a high-fidelity simulator. However, SimManALS^® can also be captured as a high-fidelity simulator. This may have influenced the scoring of “others”. In this study, a completed survey was conducted, and randomisation of confounders was not possible.

Therefore, based on the results obtained in this study, we will plan a more focused survey in the future. We suggest that the results of this study can be used to promote SBL in a more specific method.

Lastly, in 2020-21, opportunities for nursing students to complete practical training in medical facilities are limited as a result of the global pandemic of COVID-19. Hence, it can be said that SBL will play a major role in maintaining learning opportunities and clinical competence in students while ensuring the safety of patients, students, and medical staff. We plan to continue and expand upon our survey research to enable more educational institutions to begin practicing SBL.

V. CONCLUSION

In conclusion, this study provided an initial view of the current status of SBL in Japan. Our results suggested that while many schools are equipped with simulators, they are not properly utilised in the foundation of nursing programs in Japan. Thus, further study of barriers to simulator use is needed. Also, although awareness of INACSL Standards was extremely low, over half of the respondents were designing simulations following the standards, implying that they were making use of some theories or conceptual frameworks in their designs.  Our results act as a foundational resource for studying strategies intended to systemise SBL in Japanese undergraduate nursing education. In the future, it will be necessary to specifically survey awareness and use of theories and conceptual frameworks to recommend methods for increasing their application and use in SBL.

Notes on Contributors

Dr. Mitsumi Masuda, PhD, RN, is an associate professor at the Graduate School of Nursing, Nagoya City University. She reviewed the literature, designed the study, performed data collection, data analysis and wrote the manuscript.

Dr. Machiko Saeki Yagi, MS, RN, is a lecturer at the School of Nursing, Jichi Medical University. She developed the methodological framework for the study, performed data collection, data analysis and gave critical feedback to the writing of the manuscript.

Dr. Fumino Sugiyama, PhD, RN, is an associate professor at the School of Nursing, National College of Nursing. She performed data collection and data analysis. All the authors have read and approved the final manuscript.

Ethical Approval

This research was approved by the research ethics review committee of the Nagoya City University, Graduate School of Nursing (Approval no.: 19019-2).

Data Availability

Datasets generated and/or analysed during the current study are available from the following DOI. https://doi.org/10.6084/m9.figshare.14010755.v1.

Acknowledgement

This study was completed with the support of The Murata Science Foundation (2019 No. 28). In addition, we would like to thank Editage (www.editage.com) for assisting with English language editing. We would like to offer our heartfelt thanks to the nursing schools across the country who participated in this study.

Zarifsanaiey, N., Amini, M., & Saadat, F. (2016). A comparison of educational strategies for the acquisition of nursing student’s performance and critical thinking: Simulation-based training vs. integrated training (simulation and critical thinking strategies). BMC Medical Education, 16(1), 294. https://doi.org/10.1186/s12909-016-0812-0

*Mitsumi Masuda
1, Kawasumi, Mizuho-cho,
Mizuho-ku, Nagoya,
467-8601, Japan.
Tel: +81-52-853-8063
Email: m.masuda@med.nagoya-cu.ac.jp

Submitted: 19 February 2021
Accepted: 17 June 2021
Published online: 4 January, TAPS 2022, 7(1), 33-43
https://doi.org/10.29060/TAPS.2022-7-1/OA2493

Chan Choong Foong*, An Jie Lye*, Che Rafidah Aziz, Wei-Han Hong, Vinod Pallath, Jessica Grace Cockburn, Siti Nurjawahir Rosli, Kuhan Krishnan, Prahaladhan Sivalingam, Noor Filzati Zulkepli & Jamuna Vadivelu

Medical Education & Research Development Unit (MERDU), Faculty of Medicine, University of Malaya, Malaysia

*Joint first authors

Abstract

Introduction: Medical schools universally responded by migrating teaching and learning to virtual learning environments (VLE) due to the impact of the COVID-19 pandemic. The use of virtual problem-based learning (PBL) in lieu of face-to-face sessions seems to be an appropriate response, but its effectiveness was understudied. The study compared the learning experiences of pre-clinical medical students at the beginning and completion of the virtual PBL.

Methods: The study was conducted at the University of Malaya, a public-funded university in Malaysia. A 12-item questionnaire was developed and validated to assess the learning experiences of students conducting virtual PBL sessions. Principal component analysis and test for internal consistency suggested that the questionnaire is valid and reliable. The questionnaire was administered to pre-clinical students (Year 1 and Year 2) twice: at the beginning and the end of the virtual PBL implementation. Their responses were compared for the domains “learning”, “confidence” and “concern”.

Results: Three hundred and forty-four pre-clinical students were recruited but only 275 students (80%) responded to both the initial and final questionnaires. Based on the responses, the learning experiences of students generally improved by the completion of the virtual PBL implementation. Students were most convinced that they obtained and understood the information given during the virtual PBL. However, they continued to be worried about passing the clinical examination and content acquisition.

Conclusion: The study supports the feasibility of virtual PBL as an acceptable alternative to replace face-to-face PBL during the COVID-19 pandemic.

Keywords:           Problem-based Learning, COVID-19, Undergraduate Medical Education, Virtual Learning Environment

Practice Highlights

• The study compared the learning experiences of pre-clinical medical students in virtual Problem Based Learning (PBL).
• The learning experiences of students in virtual PBL generally improved.
• Students obtained and understood the information given during the virtual PBL.
• Students continued to be worried about passing the clinical examination and content acquisition.
• Virtual PBL could be an acceptable alternative to replace face-to-face PBL during the pandemic.

Submitted: 31 August 2020
Accepted: 8 February 2021
Published online: 5 October, TAPS 2021, 6(4), 118-130
https://doi.org/10.29060/TAPS.2021-6-4/OA2481

Isabella E. Supnet, Jose Alvin P. Mojica, Sharon D. Ignacio & Carl Froilan D. Leochico

Department of Rehabilitation Medicine, College of Medicine and Philippine General Hospital, University of the Philippines Manila, Philippines

Abstract

Introduction: In the Philippines, telerehabilitation has been at the forefront of integrating telemedicine into the medical curriculum. However, the course evaluation tool used for traditional classroom-based courses is not appropriate in evaluating the unique teaching-learning tool that is telerehabilitation. This study aimed to develop a questionnaire that will aid in addressing this gap.

Method: A mixed methods study was devised to gather information from medical students exposed to telerehabilitation from the College of Medicine, University of the Philippines Manila as well as the residents from the Department of Rehabilitation Medicine, Philippine General Hospital from October to November 2019. The investigators obtained informed consent from all participants as well as their demographics before undergoing interviews. Themes were identified to create questions under the previously identified constructs, along with items derived from other course evaluation tools and opinions gathered from experts in telerehabilitation.

Results: In total, 26 individuals participated in the study. Most of the respondents had experience or ownership of various communication technologies and were well-versed in communication strategies through these technologies. There were 52 questions formulated from the interviews and review of previous tools.

Conclusion: This study is the first step in providing more research in the student evaluation of telerehabilitation and telemedicine. To match the needs of changing times evaluation of new standards and methods should follow. More research must be done to standardise teaching evaluation tools to validate the data gathered, and allow courses, such as telerehabilitation, an opportunity to adapt and promote further learning.

Keywords:           Assessment, Educational, Education, Medical, Telemedicine, Medical Informatics, Physical and Rehabilitation Medicine

Practice Highlights

• Telemedicine has been used as a tool to educate students on rehabilitation medicine.
• Almost all students at this time have had experience in the usage of information and communication technologies.
• Evaluation tools must be more specific to the method of instruction used.
• Clarity, congruence, and relevance are the most sought-after characteristics in telerehab courses.
• The recipient of the evaluation must account for the effect of the teacher or logistical concerns.

I. INTRODUCTION

A. Background

Telemedicine is defined as the use of advanced telecommunications technologies to exchange health information and provide health care services across geographic, temporal, social and cultural barriers (Myers, 2003). It has been widely used in the treatment and care of patients as we go through the COVID-19 pandemic and has also been vital in improving medical education in the age of online classes and social distancing. Through telemedicine, students are exposed to a variety of specialties, and gain experience in their observation of the management of diseases, such as in the case of telesurgery, or hone their skills in communication and counselling, such as through telerehabilitation (Jumreornvong et al., 2020).

Now more than ever, due to the capacity of students to interact and adapt with digital infrastructure (Pathipati et al., 2016), integration of telemedicine into the medical curriculum is increasingly important and evaluations of telemedicine curricula are necessary to ensure quality and to detect areas for growth and improvement. Previous studies have used general evaluation forms to evaluate their telemedicine programmes or created general surveys from their course objectives, which have yielded some valuable insights (Brockes et al., 2017; Bulik & Shokar, 2010). However, due to the nature of instruction of telemedicine, specialised and validated tools are necessary to provide a comprehensive assessment.

In the Philippines, telerehabilitation has been at the forefront of integrating telemedicine into the curriculum, particularly in the University of the Philippines. Because it has been four years since the start of the initiative, and one year since its implementation, it is important to evaluate the previous courses in order to improve them for the next generation.

Through this research, a preliminary evaluation questionnaire for telerehabilitation as a teaching-learning tool was developed. From this questionnaire, perceptions of students regarding telerehabilitation courses will be more efficiently gathered and evaluated, which will serve to further improve the telerehabilitation curriculum and possibly bring forth interventions to improve medical education in general.

B. Literature Review

1) Teaching programmes in telemedicine: In the process of conducting telemedical evaluations in order to aid patients, clinical training may also be received. Telemedicine has had good acceptance in training institutions abroad, with Neurology trainees agreeing that it should be part of their curriculum and supporting a formalised telemedicine rotation within their residency. Dermatology programmes in particular observed that telemedicine supported rather than detracted from the core competencies required from them (Lee & Nambudiri, 2019). No significant differences were seen between clinical outcomes of patients who underwent surgery through telementoring versus the traditional method, and case supervision using e-mails and voice-over applications afforded similar psychiatry education as compared to rotations in mental health clinics (O′Shea et al., 2015).

The methods through which telemedicine education is given also vary considerably per institution (Waseh & Dicker, 2019). Some schools have telemedicine included primarily as didactic sessions. Others allow their students to take part in patient encounters and interprofessional training. Some institutions also allow scholarly projects to be done in telemedicine.

Because telemedicine in medical education has yet to be explored formally, no evaluation tools have been developed to assess its application. Other institutions used generalised forms and made their own questionnaires (Brockes et al., 2017; Bulik & Shokar, 2010). Literature review for formally developed evaluation materials showed the presence of a Telehealth Usability Questionnaire, with items and domains encompassing many telehealth assessment tools (Parmanto et al., 2016). It was made for use with various types of telehealth systems, including computer-based systems, videoconferencing programmes, and adaptable for progressive innovations, particularly for mobile telemedicine applications. However, this was primarily utilised as an assessment tool between clinicians and patients and does not assess instruction on the usage of the programme or the organisation of the implementation.

2) Teaching programmes in telerehabilitation: In 2015, the Department of Rehabilitation Medicine, College of Medicine and Philippine General Hospital, University of the Philippines Manila (PGH DRM) has initiated education on this aspect of telemedicine, with telerehabilitation used as a teaching-learning tool for medical students in full implementation in the 2018 curriculum. The programme has expanded from its origins since then; from students engaging in telerehabilitation to actual telerehabilitation consultations and teletherapy services with the rural health unit of Alfonso, Cavite, as part of the University of the Philippines Community Health and Development Programme (UP CHDP).

Telerehabilitation as a teaching-learning tool was formally included as part of the curriculum for rehabilitation medicine in 2018 (Philippine General Hospital Department of Rehabilitation Medicine, 2018). The students start their engagement during their third year of medical school (known as Learning Unit 5), their first year of clinical exposure, with the concepts and theories behind telerehabilitation, and do observations of an actual telerehabilitation session. A year later, during Learning Unit 6, they then get to do a simulated telerehabilitation encounter, with guidance and techniques on how to present a rehabilitation case through telemedicine. During their final year of medical education (Learning Unit 7), the student is then evaluated on the presentation, evaluation and management given during an actual patient encounter. All students from the UP College of Medicine receive two sessions of telerehabilitation instruction per year.  On the other hand, Post-Graduate Interns (PGIs), who are students who have received their first four years of medical education in other medical schools and have chosen to spend their last year of medical school in the Philippine General Hospital, receive only one compressed session. Overall, the participants reported the experience to be excellent, and should be explored further (Leochico & Mojica, 2017).

Telerehabilitation was also included in January 2019 as a formal service for training residents in the Department of Rehabilitation Medicine, featuring the same concepts in a more compressed manner, and highlighting the hybridisation of standard rehabilitation practice with telemedicine.

The initiative is currently being evaluated by the students in part through the Course Evaluation by Students (CEBS) given by the University of the Philippines College of Medicine (UPCM) (University of the Philippines Manila College of Medicine, 2005), and through a Devised Telerehabilitation Feedback Form for Students (Philippine General Hospital Department of Rehabilitation Medicine, n.d.), which does not distinguish usability attributes of the telerehabilitation system from the telerehabilitation curriculum itself.

3) The usage of student evaluations: Student evaluation forms are a commonly used tool in determining teacher and course effectiveness in many areas of higher education. These evaluations commonly serve three purposes: to improve teaching quality, to support faculty decisions, and to provide evidence for institutional accountability. Many evaluation measures have been constructed and performed with students as the main and sometimes sole indicator of education quality; however, the implementation, reliability and validity of these methods and instruments have been a source of concern within academic circles (Soto-Estrada et al., 2018). Due to this, student evaluation remains a field of active study.

Although helpful, some caution must be undertaken in interpreting results of student evaluations. A recent meta-analysis by Uttl et al. (2017) argues that studies looking into student evaluations of teaching (SETs) were scant in terms of data to support the equivalence of high student evaluation ratings to student performance, and positive correlations between the two were primarily due to small study size effects. However, it is unfair to assume that SETs have no value whatsoever. Responses to the meta-analysis have been published as well, and aside from critiquing the method Uttl used to form his study, they support the view that, while it is poor practice to use student ratings of instruction alone in evaluating teaching, it remains to be necessary (Ryalls et al., n.d.).

C. Objectives

1) General objective:

• To develop a questionnaire that will evaluate telerehabilitation as a teaching-learning tool for medical students.

2) Specific objectives:

• To determine themes regarding the evaluation of telerehabilitation as a teaching-learning tool from residents of the PGH Department of Rehabilitation Medicine, as well from the medical students who have undergone the telemedicine courses.
• To formulate questionnaire items for the telerehabilitation course evaluation tool.

II. METHODS

A mixed methods study was devised to gather information from medical students exposed to telerehabilitation from the Philippine General Hospital and the College of Medicine, University of the Philippines Manila (with exposures corresponding to Learning Units 5, 6, and 7) as well as the residents from the PGH DRM from October to November 2019 (Figure 1). All participants were greater than 18 years old.

Figure 1. Flowchart of study methodology

Submitted: 18 January 2021
Accepted: 19 May 2021
Published online: 5 October, TAPS 2021, 6(4), 107-117
https://doi.org/10.29060/TAPS.2021-6-4/OA2470

Sarah Brown & Justin L C Bilszta

Department of Medical Education, Melbourne Medical School, University of Melbourne, Australia

Abstract

Introduction: Use of complementary and alternative medicine (CAM) is popular in the general population and medical practitioners may not be fully equipped in their knowledge of CAM to advise patients appropriately. The aim of this paper was to perform a scoping review of current literature describing undergraduate medical student use, attitudes, and knowledge of CAM as a means of better understanding the educational needs of these students.

Methods: A systematic search of Medline, PubMed and the Education Resources Information Center (ERIC) databases with keywords related to “complementary and alternative medicine” and “undergraduate medical students” for relevant articles published until August 2020.  

Results: Of 131 papers identified, 38 underwent full review. It was found 13-80% of medical students use CAM, and overall have a positive attitude towards CAM therapies.  Female medical students and those with religiosity had more positive attitudes towards CAM than their male colleagues and those without a religion. Knowledge of CAM is lacking with approximately only half of students feeling they were knowledgeable about CAM therapies. Popular information resources are the Internet and social media, but students expressed they want more teaching of CAM in the undergraduate medical curriculum.

Conclusion: Evidence suggests high usage of CAM amongst undergraduate medical students, and positive attitudes towards CAM therapies; however, knowledge of CAM is poor, and students want more CAM teaching to upskill them in counselling patients interested in CAM therapies. Further areas for research include a better understanding of resources medical students use for their knowledge and how gender and religiosity influence attitudes towards CAM.  

Keywords:           Undergraduate Medical Student, Complementary and Alternative Medicine, CAM, Attitude, Knowledge, Use

Practice Highlights

• Medical student’s personal use of CAM is significant, with the most popular therapies being massage, meditation and herbal medicine.
• Medical students have a positive attitude towards CAM but potential differences between attitudes of preclinical and clinical student populations exist.
• Medical student’s knowledge of CAM is lacking, and this impacts their ability to advise patients appropriately.
• Medical students want more CAM teaching integrated into their UGME curriculum, and believe conventional western medicine could benefit from CAM methods and ideas.
• Better integration of the principles of EBM rather than teaching related to specific CAM therapies can provide medical students with the skills to critique claims of CAM efficacy.

Complementary and alternative medicine (CAM) encompasses a range of health practices including, but not limited to, acupuncture, naturopathy, chiropractic traditional Chinese medicine (TCM), herbal, vitamin & homeopathic therapies (Zollman & Vickers, 1999). Complementary refers to the practice of something alongside conventional Western medicine, whilst alternative refers to the practice of something instead of conventional Western medicine (Zollman & Vickers, 1999).

In comparison to conventional Western medicine, the eclectic range of CAM is often cited as having a poor-quality evidence-base regarding its efficacy (Australian Medical Association, 2018), yet it remains popular with the general public (Frass et al., 2012). With surveys suggesting 10–76% of the public has used CAM (Ernst, 2000; Harris et al., 2012) the demand for CAM is evident. How knowledgeable medical practitioners are, and their attitudes towards CAM, can influence the advice they might provide to patients who seek information about CAM therapies or want to use CAM in lieu of conventional Western medicine. Janamian et al. (2011) have reported general practitioners feel they do not have the education to appropriately advise patients on CAM (Janamian et al., 2011). This may deter patients from seeking guidance from their healthcare provider and result in a breakdown of communication and lack of congruence over health goals (Xue et al., 2007).

Medical students’ insights into their prevailing attitudes and knowledge of CAM can help determine if CAM teaching within undergraduate medical education (UGME) should be expanded to better meet the needs of growing public demand. The last significant review of medical students’ attitudes and knowledge of CAM was published in 2016 (Joyce et al., 2016), and reported overall medical students believed they lacked knowledge of CAM but were generally positive towards CAM education and thought CAM teaching should be incorporated into UGME. What this review did not explore was the rates of CAM use by medical students, and whether this influenced attitudes towards CAM. Importantly, since the publication of Joyce et al’s review, not only have there been additional studies exploring medical students’ use, attitudes and knowledge of CAM, but the increasing use of the Internet and social media as a source of information means it is timely to re-evaluate the findings of this earlier study and determine whether the results presented remain relevant.

We conducted this scoping review of primary studies to evaluate undergraduate medical student use, attitudes, and knowledge of CAM as a means of better understanding the educational needs of these students. The specific research questions were: 1) what is the usage and knowledge of CAM among medical students?; 2) what are medical students’ attitudes towards CAM?; 3) what factors influence medical students’ attitudes towards CAM, and where do they seek information about CAM? and; 4) what are medical students’ views on the current teaching they receive in UGME about CAM?

II. METHODS

This study adopted the “Preferred Reporting Items for Systematic Reviews and Meta-Analysis Extension for Scoping Reviews” (PRISMA-ScR) protocol (Tricco et al., 2018).

A. Search Strategy

Electronic databases Medline (Ovid), PubMed and ERIC were searched for full text articles describing undergraduate medical students use, attitudes, and knowledge of CAM (see Appendix 1). Additional papers were found through a hand search of reference lists of articles identified through the database search. There was no limit on publication date.

B. Selection of Sources

Papers were included if published in English and reported on undergraduate medical students. Papers were excluded if they were not published in English; opinion pieces or review articles; reported on non-undergraduate medical students or; reported on osteopathic medical students. This group was excluded due to the potential bias these students may have due to their medical training including specific CAM teaching.

C. Key Words and Boolean Operators

The search strategy (see Appendix 1) included Boolean operators (AND, NOT, OR), Medical Subject Headings (MeSH) and specific key words.

D. Data Extraction and Charting

Data extraction was performed using a predetermined checklist, and included

• Article details: first author and publication year
• Research aim: purpose of the study
• Method: methods of data collection
• Sample size: number of students invited to participate and the number who responded
• Student CAM use: number of students using CAM, including use of specific therapies and timeframe of use
• Student attitudes/perceptions to CAM: student attitudes to CAM including desire to undertake further education about CAM, beliefs about CAM efficacy, role of CAM in conventional Western medicine and, sources of CAM information
• Influencing factors: factors reported to influence student’s attitudes/perceptions to CAM
• Limitations: identified study limitations

E. Synthesis of Results

Included studies were described by author, publication year, and characteristics described above. Thematic analysis was conducted to identify commonality between included studies. No inferences were made about students’ use, attitudes, or knowledge of CAM if this was not explicitly stated.

Literature searching, title and abstract screening, full text review & data extraction, and charting were undertaken by the 1st author (SB). Where there was uncertainty regarding the aforementioned, these articles were reviewed independently by the 2nd author (JB) and discussed until consensus was reached. The 2nd author also independently reviewed the data extraction and charting results once this process was completed by the 1st author.

III. RESULTS

From the initial search strategy, 131 articles were obtained for screening based on title and abstract. Following screening, 17 duplicate citations were excluded, leaving 114 papers. Abstract and title screening excluded papers focused on pharmacy or nursing students, and those that did not centre on the research questions. Following this, 58 articles underwent full review. Of these, 21 were excluded as they were not in English, did not provide enough detail regarding medical students specifically or focused heavily on medical curricula. The total number of articles included is 38 (see Figure 1); a full version of the outcome harvesting data can be found at https://doi.org/10.6084/m9.figshare.14471250 (Brown & Bilszta, 2021).

Figure 1: PRISMA Diagram

A. CAM usage

Twenty-two studies assessed CAM usage in medical students (Baugniet et al., 2000; Chaterji et al., 2007; Chez et al., 2001; Ditte et al., 2011; Donald et al., 2010; Flaherty et al., 2015; Frye et al., 2006; Greenfield et al., 2000, 2002; Hegde et al., 2018; Hopper & Cohen, 1998; James et al., 2016; Lie & Boker, 2004, 2006; Nicolao et al., 2010; Oberbaum et al., 2003; Rees et al., 2009; Sadeghi et al., 2016; Samara et al., 2019; Shani-Gershoni et al., 2008; Wong et al., 2006; Yildirim et al., 2010). Rates of usage ranged from 13% to 80% (Chez et al., 2001; Wong et al., 2006). The most common CAM therapy used by medical students was difficult to evaluate due to inconsistent methods of collecting this information with studies either asking students to nominate usage from a predefined list of therapies (Baugniet et al., 2000; Chaterji et al., 2007; Frye et al., 2006; Hegde et al., 2018; James et al., 2016; Lie & Boker, 2004, 2006; Nicolao et al., 2010; Sadeghi et al., 2016; Samara et al., 2019; Shani-Gershoni et al., 2008; Wong et al., 2006; Yildirim et al., 2010); self-report the therapies they had engaged with (Donald et al., 2010; Greenfield et al., 2000, 2002) or; indicating CAM usage without nominating a specific therapy (Chez et al., 2001; Ditte et al., 2011; Flaherty et al., 2015; Hopper & Cohen, 1998; Oberbaum et al., 2003; Rees et al., 2009). The challenge in evaluated CAM usage is highlighted by two separate studies (Lie & Boker, 2004, 2006) which reported a range of practices to be popular including massage, meditation and yoga compared to vitamins, meditation and spirituality, respectively (Lie & Boker, 2004, 2006).

B. CAM perception/attitudes

Overall, the general attitude of medical students towards CAM is positive (Ahmed et al., 2017; Akan et al., 2012; Chaterji et al., 2007; Chez et al., 2001; Ditte et al., 2011; Flaherty et al., 2015; Frye et al., 2006; Furnham & McGill, 2003; Godin et al., 2007; Greenfield et al., 2002, 2006; Hegde et al., 2018; Hopper & Cohen, 1998; James et al., 2016; Lie & Boker, 2004, 2006; Loh et al., 2013; Nicolao et al., 2010; Oberbaum et al., 2003; Perkin et al., 1994; Rees et al., 2009; Sadeghi et al., 2016; Samara et al., 2019; Templeman et al., 2015; Torkelson et al., 2006; Wong et al., 2006; Xie et al., 2020; Yeo et al., 2005) with positive attitudes ranging from 49% to 60% (Ditte et al., 2011; Perkin et al., 1994; Sadeghi et al., 2016). Different methods were used to collect this data including validated measures (for example: (Flaherty et al., 2015; Frye et al., 2006; Rees et al., 2009)), rating scales with various stages of validation (for example: (Ahmed et al., 2017; Oberbaum et al., 2003; Sadeghi et al., 2016; Samara et al., 2019)) or qualitative interviews (for example: (Templeman et al., 2015)).

Differences in the attitude of preclinical and clinical cohorts varied between studies. Akan et al., Furnham & McGill and Syverstad et al. reported preclinical students had a more positive attitude compared to clinical students (Akan et al., 2012; Furnham & McGill, 2003; Syverstad et al., 1999), however others found clinical students were more positive, perhaps due to them receiving more CAM education (Chaterji et al., 2007; Xie et al., 2020). However, multiple studies found no difference between preclinical and clinical cohorts (Ditte et al., 2011; Flaherty et al., 2015; Hopper & Cohen, 1998; Rees et al., 2009). Additionally, all studies bar one (Hübner et al., 2012) asked students to consider attitudes and perceptions in relation to ‘…their clinical practice…’ rather than associated with a specific clinical context or medical condition.

Attitudes towards specific CAM therapies demonstrated acupuncture received consistent positive perceptions ranging from 77% to above 90% (Loh et al., 2013; Torkelson et al., 2006; Yeo et al., 2005). Other therapies with consistent positive perceptions include massage and meditation, with more than 70% of medical students feeling positively towards these practices (Furnham & McGill, 2003; Loh et al., 2013; Torkelson et al., 2006). Conversely, Greenfield et al. and Loh et al. both found homeopathy had the most negative perception from students (Greenfield et al., 2006; Loh et al., 2013). Interestingly, studies indicate ambivalence towards chiropractic practice, with some reporting positive attitudes whilst others reported negative attitudes (Greenfield et al., 2006; Loh et al., 2013). When asked about the integration of CAM with conventional Western medicine, there was overwhelming belief from medical students that benefits may come from applying these practices together. Ahmed et al. and Chez et al. reported a large percentage of their cohorts believed conventional Western medicine could benefit from CAM methods and ideas (67% and 89%, respectively) (Ahmed et al., 2017; Chez et al., 2001). Similar findings have been reported by others, with the percentage of students wanting the integration of CAM and conventional Western medicine ranging from 71% to 92% (Chez et al., 2001; Hopper & Cohen, 1998; Lie & Boker, 2004; Loh et al., 2013; Nicolao et al., 2010; Torkelson et al., 2006; Xie et al., 2020; Yeo et al., 2005).

An important question to consider when evaluating medical student knowledge and attitudes towards CAM is whether they believe CAM works. This question arose frequently in the reviewed literature, with investigators trying to determine if there was congruence between having a positive perception of CAM and belief in its efficacy. When asked if participants found the results of CAM were mainly due to the placebo effect, there were mixed findings with 39-59% of medical students agreeing the CAM efficacy was mainly due to the placebo effect (Baugniet et al., 2000; Nicolao et al., 2010; Yeo et al., 2005; Yildirim et al., 2010). Interestingly, recent studies by Xie et al. and Samara et al. reported only 10% and 35%, respectively, believed CAM efficacy was due to the placebo effect (Samara et al., 2019; Xie et al., 2020). Whether this is simply a cohort effect or a reflection that over time students may be becoming less sceptical of CAM, is unable to be confirmed.

Factors that influence attitudes towards CAM should be considered as they may alter how medical students perceive CAM therapies. The most frequent influence reported was gender, with female medical students overall tending to have a more positive attitude than male colleagues (Akan et al., 2012; Chaterji et al., 2007; Ditte et al., 2011; Donald et al., 2010; Flaherty et al., 2015; Frye et al., 2006; Furnham & McGill, 2003; Godin et al., 2007; Greenfield et al., 2002, 2006; Hübner et al., 2012; Oberbaum et al., 2003; Rees et al., 2009; Schmidt et al., 2005; Xie et al., 2020). Interestingly, Ditte et al. found male medical students feared the social stigma of using CAM more than female students (45% vs 36%) and Donald et al. reported male students were more likely to be more sceptical of CAM than female students (50% vs 44%) (Ditte et al., 2011; Donald et al., 2010).

Religion may also influence attitudes to CAM. Yeo et al. reported nearly two thirds of their cohort believed spiritual/religious beliefs influenced attitudes towards CAM (Yeo et al., 2005). Furnham & McGill reported similar findings as female students and students that rated themselves higher religiously, were more eager for CAM to be on the curriculum compared to males and those who were less religious (Furnham & McGill, 2003). In contrast, Loh et al. reported students with no religion tended to be less interested in CAM (Loh et al., 2013).

C. CAM knowledge

The included studies suggest approximately only half of medical students felt knowledgeable about CAM (Hopper & Cohen, 1998; James et al., 2016; Sadeghi et al., 2016; Shani-Gershoni et al., 2008) and the most common individual therapies students were subjectively knowledgeable in were massage (Baugniet et al., 2000; Chez et al., 2001; Frye et al., 2006; Furnham & McGill, 2003), acupuncture (Akan et al., 2012; Nicolao et al., 2010; Sadeghi et al., 2016; Yeo et al., 2005), chiropractic (Baugniet et al., 2000; Chez et al., 2001; Frye et al., 2006) and herbal medicine (Akan et al., 2012; Baugniet et al., 2000; Chez et al., 2001). It is clear, however, medical students still perceive significant gaps in their knowledge of CAM. Nicolao et al. reported although students indicated acupuncture and homeopathy as their most knowledgeable areas, this was only for a minority of the cohort (34% and 43%, respectively) and the majority of students felt their level of knowledge, and therefore ability to appropriately advise patients, was poor (Nicolao et al., 2010). In one Australian study, investigators found out of ten common CAM therapies (acupuncture, chiropractic, herbal medicine, homeopathy, hypnosis, massage, meditation, naturopathy, reflexology and spiritual healing) 56% of their cohort had no knowledge of any of these therapies (Hopper & Cohen, 1998).

The Internet appears to be a popular avenue for finding CAM information with Lie and Boker reporting 81% of their cohort used online resources as their main information source (Lie & Boker, 2004). Although the specific online resources used were not described, journals, books and health databases were used less frequently in comparison (41%, 38% and 28%, respectively) (Lie & Boker, 2004). A study of Chinese medical students with CAM teaching integrated into their UGME, found their cohort’s main source of information was from teachers, making up 91% with only 2% using the Internet (Xie et al., 2020). Samara et al. reported 73% of their cohort used social media as their main source of information (Samara et al., 2019).

It is clear medical students want more teaching about CAM in their UGME. Flaherty et al. reported just over two thirds of students across all year levels had a desire to study CAM, however, there was a positive correlation between those students who wanted more teaching and a positive attitude towards CAM (Flaherty et al., 2015). Similar findings were reported by Perkins et al. and Frye et al. with approximately 85% of clinical and preclinical students believing they should learn about CAM in UGME (Frye et al., 2006; Perkin et al., 1994). In terms of this would look like in the curriculum, Greenfield et al. reported 67% wanted to study the theory, with Greiner et al. and Yeo et al. finding 72% and 86%, respectively, wanted clinical exposure (Greenfield et al., 2006; Greiner et al., 2000; Yeo et al., 2005). In semi-structured qualitative interviews of Australian medical students, CAM education was considered important regardless of specialisation and CAM literacy was an ethical responsibility (Templeman et al., 2015).  

IV. DISCUSSION

In this Discussion, we provide commentary on our findings, consider how these might be applied to the design of UGME curriculum, and suggest areas of further research.

In the current study, we found a significant number of medical students use CAM, however reported rates varied considerably, and our analysis suggests inconsistency in approaches used to assess CAM use may explain the observed differences. As highlighted, data collection was split between those studies that provided a predefined list of specific CAM therapies, with a limited number of response options, and the exact therapies between papers incongruent; those that allowed students to self-report usage and; those indicating CAM usage without nominating a specific therapy (see Table 1). Cultural differences may also explain usage variation; for example, the high use of yoga in Indian medical students (Hegde et al., 2018), TCM in Chinese medical students (Wong et al., 2006) and, traditional Iranian medicine in Iranian medical students (Sadeghi et al., 2016).

Table 1. Example of the differing methods used to collect student CAM usage data. Data collection of medical student CAM usage was split between those studies that provided a predefined list of specific CAM therapies; those that allowed students to self-report usage and; those indicating CAM usage without nominating a specific therapy. The above examples highlight the inconsistent approaches to collecting this information.

Although medical students believe conventional Western medicine could benefit from CAM methods and ideas, an important caveat to this finding is it does not consider how they feel about specific CAM therapies. For example, would medical students think conventional Western medicine could benefit from integrating herbal medicine or homeopathy approaches, compared to osteopathy or acupuncture? Given the diversity of CAM, it is arguably unfeasible (Wetzel et al., 1998) to teach students about each individual therapy, so an EBM teaching model equipping students with skills needed to critically appraise evidence presents a potential solution, and several medical schools have designed curricular in this way (Bailey et al., 2015; Forjuoh et al., 2003; Hassed, 2004; Jeffries, 2001; Mahapatra et al., 2017; Owen & Lewith, 2001; Perlman & Stagnaro-Green, 2010). This addresses a thematically overwhelming and resource intensive area of medical education where learned knowledge and skills can be adapted for differing CAM therapies.

Only three studies (Lie & Boker, 2004; Samara et al., 2019; Xie et al., 2020) specifically explored how medical students use digital resources to supplement formal CAM teaching, so this is a clear direction for further research. It is important medical students have access to high quality evidence-based information and the skills to determine the validity of evidence presented on digital platforms. Whilst there is evidence to demonstrate resources such as blogs, social networking sites and online support groups are useful platforms for health education, social connection and experience sharing, they are at significant risk of misinformation and mischaracterisation of CAM (Delgado-López & Corrales-García, 2018; Molassiotis & Xu, 2004). Sharma et al. (2016) have underscored the challenges in developing approaches for identifying the reliability of CAM-related information on the Internet, which may not be supported by a reliable evidence-base (Sharma et al., 2016). This means medical students require skills to critique claims related to CAM efficacy for their own education needs, but also so they can appropriately counsel patients. As advocated, an EBM teaching model would provide students with the skills to critique claims of CAM efficacy as well as empower patients to make informed decisions relevant to their health needs.

V. LIMITATIONS OF THE STUDY

As we did not conduct a systematic review of the literature, we are unable to draw any conclusions about the quality of the included studies. We excluded non-English studies, and this potentially creates a gap in the literature and may have altered the findings as CAM use, attitudes and knowledge may be different in non-English speaking countries (for example, the high use of acupuncture or Traditional Chinese medicine in Asian countries). The lack of consistency in data related to medical student use, attitudes and knowledge of CAM therapies is reflected in its heterogeneity. This makes generalisations related to ‘medical students’, even within the same medical school or UGME curriculum, difficult. Despite this significant limitation, the diversity of reported outcomes reflects important contextual differences in medical programs, and medical student cohorts, across the globe.

VI. CONCLUSION

Overall, medical students have a positive attitude towards CAM, with significant influencing factors being gender and religion, and potential differences between preclinical and clinical cohorts. The included literature indicates medical students’ personal use of CAM is significant, with therapies such as massage, meditation and herbal medicine being more popular than others. It is clear medical students’ CAM knowledge is lacking and integration of the principles of EBM may be beneficial in addressing this, ultimately leading to better educated doctors who have better relationships with their patients.

Notes on Contributors

JB: conceptualised the review; JB, SB: designed the search strategy; SB: conducted the search and screened the literature; JB, SB: reviewed the included studies; SB: analysed the data; JB: drafted the manuscript; JB, SB: critically revised the manuscript. All authors had full access to all study data, read and approved the final version of the manuscript.

Ethical Approval

As all the data were retrieved from public databases, this study did not require institutional review board approval.

Data Availability

The data that support the findings of this study are openly available in Figshare repository, https://doi.org/10.6084/m9.figshare.14471250. The data extraction sheet used during the current study is available from the corresponding author on request. All data is based on published studies.

Funding

This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

Declaration of Interest

None of the listed authors have financial and personal relationships with organisations or people that could inappropriately influence their work.

References

Ahmed, S. M., Al-Mansour, M. A., Mohamed, E. Y., Medani, K. A., Abdalla, S. M., & Mahmoud, W. S. (2017). Medical Students’ Opinion Toward the Application of Complementary and Alternative Medicine in Healthcare. Saudi Journal of Medicine & Medical Sciences, 5(1), 20-25. https://doi.org/10.4103/1658-631X.194255

Akan, H., Izbirak, G., Kaspar, E. C., Kaya, C. A., Aydin, S., Demircan, N., Bucaktepe, P. G., Ozer, C., Sahin, H. A., & Hayran, O. (2012). Knowledge and attitudes towards complementary and alternative medicine among medical students in Turkey. BMC Complementary & Alternative Medicine, 12, 115. https://doi.org/10.1186/1472-6882-12-115

Alwhaibi, M., AlRuthia, Y., & Meraya, A. (2019). Gender differences in the prevalence of complementary and alternative medicine utilization among adults with arthritis in the United States. Evidence-Based Complementary & Alternative Medicine: eCAM. https://doi.org/10.1155/2019/8739170

Alwhaibi, M., & Sambamoorthi, U. (2016). Sex differences in the use of complementary and alternative medicine among adults with multiple chronic conditions. Evidence-Based Complementary & Alternative Medicine: eCAM. https://doi.org/10.1155/2016/2067095

Australian Medical Association. (2018). AMA Position Statement, Complementary Medicine. https://ama.com.au/position-statement/ama-position-statement-complementary-medicine-2018

Bailey, M. L., Chudgar, S. M., Engle, D. L., Moon, S. D., Grochowski, C. O., & Clay, A. S. (2015). The impact of a mandatory immersion curriculum in integrative medicine for graduating medical students. Explore (NY), 11(5), 394-400. https://doi.org/10.1016/j.explore.2015.07.003

Barraco, D., Valencia, G., Riba, A., Nareddy, S., Draus, C., & Schwartz, S. (2005). Complementary and alternative medicine (CAM) use patterns and disclosure to physicians in acute coronary syndromes patients. Complementary Therapies in Medicine, 13(1), 34-40. https://doi.org/ 10.1016/j.ctim.2005.02.003

Baugniet, J., Boon, H., & Ostbye, T. (2000). Complementary/alternative medicine: comparing the view of medical students with students in other health care professions. Family Medicine, 32(3), 178-184.

Bell, R., Suerken, C., Grzywacz, J., Lang, W., Quandt, S., & Arcury, T. (2006). Complementary and alternative medicine use among adults with diabetes in the United States. Alternative Therapies in Health and Medicine, 12, 16-22.

Ben-Arye, E., Schiff, E., Vintal, H., Agour, O., Preis, L., & Steiner, M. (2012). Integrating complementary medicine and supportive care: Patients’ perspectives toward complementary medicine and spirituality. Journal of Alternative and Complementary Medicine, 18(9), 824-831. https://doi.org/10.1089/acm.2011.0327

Bishop, F., & Lewith, G. (2010). Who uses CAM? A narrative review of demographic characteristics and health factors associated with CAM use. Evidence-Based Complementary & Alternative Medicine: eCAM, 7(1), 11-28. https://doi.org/10.1093/ecam/nen023

Brown, S., & Bilszta, J. L. C. (2021). Medical students use, attitudes towards, and knowledge of complementary and alternative medicine: A scoping review [Data set]. Figshare. https://doi.org/10.6084/m9.figshare.14471250

Chao, M., Wade, C., & Kronenberg, F. (2008). Disclosure of complementary and alternative medicine to conventional medical providers: Variation by race/ethnicity and type of CAM. Journal of the National Medical Association, 100(11), 1341-1349. https://doi.org/10.1016/s0027-9684(15)31514-5

Chaterji, R., Tractenberg, R. E., Amri, H., Lumpkin, M., Amorosi, S. B., & Haramati, A. (2007). A large-sample survey of first- and second-year medical student attitudes toward complementary and alternative medicine in the curriculum and in practice. Alternative Therapies in Health & Medicine, 13(1), 30-35.

Chez, R. A., Jonas, W. B., & Crawford, C. (2001). A survey of medical students’ opinions about complementary and alternative medicine. American Journal of Obstetrics and Gynecology, 185(3), 754-757. https://doi.org/10.1067/mob.2001.117484

Davis, E. L., Oh, B., Butow, P. N., Mullan, B. A., & Clarke, S. (2012). Cancer patient disclosure and patient-doctor communication of complementary and alternative medicine use: A systematic review. The Oncologist, 17(11), 1475-1481. https://doi.org/10.1634/theoncologist.2012-0223

Delgado-López, P., & Corrales-García, E. (2018). Influence of Internet and social media in the promotion of alternative oncology, cancer quackery, and the predatory publishing phenomenon. Cureus, 10(5), e2617. https://doi.org/10.7759/cureus.2617

Ditte, D., Schulz, W., Ernst, G., & Schmid-Ott, G. (2011). Attitudes towards complementary and alternative medicine among medical and psychology students. Psychology, Health & Medicine, 16(2), 225-237. https://doi.org/10.1080/13548506.2010.532559

Donald, G. K., Mackereth, P., & Tobin, I. (2010). Medical students and acupuncture: A short sharp placement experience! Acupuncture Medicine, 28(1), 12-15. https://doi.org/10.1136/aim.2009.001602

Ellison, C. G., Bradshaw, M., & Roberts, C. A. (2012). Spiritual and religious identities predict the use of complementary and alternative medicine among US adults. Preventive Medicine, 54(1), 9-12. https://doi.org/10.1016/j.ypmed.2011.08.029

Ernst, E. (2000). Prevalence of use of complementary/alternative medicine: A systematic review. Bulletin of the World Health Organization, 78(2), 252-257.

Flaherty, G., Fitzgibbon, J., & Cantillon, P. (2015). Attitudes of medical students toward the practice and teaching of integrative medicine. Journal of Integrative Medicine, 13(6), 412-415. https://doi.org/10.1016/s2095-4964(15)60206-0

Foley, H., Steel, A., Cramer, H., Wardle, J., & Adams, J. (2019). Disclosure of complementary medicine use to medical providers: A systematic review and meta-analysis. Scientific Reports, 9(1573). https://doi.org/10.1038/s41598-018-38279-8

Forjuoh, S. N., Rascoe, T. G., Symm, B., & Edwards, J. C. (2003). Teaching medical students complementary and alternative medicine using evidence-based principles. The Journal of Alternative & Complementary Medicine, 9(3), 429-439.

Frass, M., Strassl, R. P., Friehs, H., Müllner, M., Kundi, M., & Kaye, A. D. (2012). Use and acceptance of complementary and alternative medicine among the general population and medical personnel: A systematic review. Ochsner Journal, 12(1), 45-56.

Frye, A. W., Sierpina, V. S., Boisaubin, E. V., & Bulik, R. J. (2006). Measuring what medical students think about complementary and alternative medicine (CAM): A pilot study of the complementary and alternative medicine survey. Advances in Health Sciences Education, 11(1), 19-32. https://doi.org/10.1007/s10459-005-1587-y

Furnham, A., & McGill, C. (2003). Medical students’ attitudes about complementary and alternative medicine. The Journal of Alternative & Complementary Medicine, 9(2), 275-284. https://doi.org/10.1089/10755530360623392

Godin, G., Beaulieu, D., Touchette, J. S., Lambert, L. D., & Dodin, S. (2007). Intention to encourage complementary and alternative medicine among general practitioners and medical students. Behavioral Medicine, 33(2), 67-77.

Greenfield, S. M., Brown, R., Dawlatly, S. L., Reynolds, J. A., Roberts, S., & Dawlatly, R. J. (2006). Gender differences among medical students in attitudes to learning about complementary and alternative medicine. Complementary Therapies in Medicine, 14(3), 207-212. https://doi.org/10.1016/j.ctim.2005.12.001

Greenfield, S. M., Innes, M. A., Allan, T. F., & Wearn, A. M. (2002). First year medical students’ perceptions and use of complementary and alternative medicine. Complementary Therapies in Medicine, 10(1), 27-32. https://doi.org/10.1054/ctim.2002.0501

Greenfield, S. M., Wearn, A. M., Hunton, M., & Innes, M. A. (2000). Considering the alternatives: A special study module in complementary therapy. Complementary Therapies in Medicine, 8(1), 15-20.

Greiner, K. A., Murray, J. L., & Kallail, K. J. (2000). Medical student interest in alternative medicine. The Journal of Alternative & Complementary Medicine, 6(3), 231-234. https://doi.org/10.1089/acm.2000.6.231

Harris, P. E., Cooper, K. L., Relton, C., & Thomas, K. J. (2012). Prevalence of complementary and alternative medicine (CAM) use by the general population: A systematic review and update. International Journal of Clinical Practice, 66(10), 924–939. https://doi.org/10.1111/j.1742-1241.2012.02945.x

Hassed, C. S. (2004). Bringing holism into mainstream biomedical education. The Journal of Alternative and Complementary Medicine, 10(2), 405-407. https://doi.org/10.1089/107555304323062428

Hegde, S. V., Rao, S. K., Menezes, R. G., Kotian, S. M., & Shetty, S. (2018). Knowledge, attitude, and practice of yoga in medical students: Assessment of anthropometry and lifestyle factors. International Journal of Yoga Therapy, 28(1), 9-14. https://doi.org/10.17761/2018-00005r1

Heller, T., Kloos, C., Mueller, N., Roemelt, J., Keinki, C., Wolf, G., Mueller, U. A., & Huebner, J. (2020). Complementary and alternative medicine is positively associated with religiousness/ spirituality. Journal of Complementary & Integrative Medicine, Advance online publication. https://doi.org/10.1515/jcim-2018-0023

Hopper, I., & Cohen, M. (1998). Complementary therapies and the medical profession: A study of medical students’ attitudes. Alternative Therapies in Health and Medicine, 4(3), 68-73.

Hsiao, A.-F., Wong, M. D., Miller, M. F., Ambs, A. H., Goldstein, M. S., Smith, A., Ballard-Barbash, R., Becerra, L. S., Cheng, E. M., & Wenger, N. S. (2008). Role of religiosity and spirituality in complementary and alternative medicine use among cancer survivors in California. Integrative Cancer Therapies, 7(3), 139-146. https://doi.org/10.1177/1534735408322847

Hübner, J., Wicker, S., & Münstedt, K. (2012). Survey of students regarding their attitudes toward complementary and alternative medicine in oncology. Forschende Komplementarmedizin, 19(6), 319-321. https://doi.org/10.1159/000345973

James, P. B., Bah, A. J., & Kondorvoh, I. M. (2016). Exploring self-use, attitude and interest to study complementary and alternative medicine (CAM) among final year undergraduate medical, pharmacy and nursing students in Sierra Leone: A comparative study. BMC Complementary & Alternative Medicine, 16, 121. https://doi.org/10.1186/s12906-016-1102-4

Janamian, T., O’Rourke, P., Myers, S. P., & Eastwood, H. (2011). Information resource needs and preference of Queensland general practitioners on complementary medicines: Result of a needs assessment. Evidence-Based Complementary & Alternative Medicine: eCAM, https://doi.org/10.1155/2011/810908

Jawahar, R., Yang, S., Eaton, C. B., McAlindon, T., & Lapane, K. L. (2012). Gender-specific correlates of complementary and alternative medicine use for knee osteoarthritis. Journal of Women’s Health, 21(10), 1091-1099. https://doi.org/10.1089/jwh.2011.3434

Jeffries, W. B. (2001). A fourth-year elective course in alternative medicine. Academic Medicine, 76(5), 525-526. https://doi.org/10.1097/00001888-200105000-00060

Joyce, P., Wardle, J., & Zaslawski, C. (2016). Medical student attitudes towards complementary and alternative medicine (CAM) in medical education: A critical review. Journal of Complementary & Integrative Medicine, 13(4), 333-345. https://doi.org/10.1515/jcim-2014-0053

Kristoffersen, A. E., Stub, T., Salamonsen, A., Musial, F., & Hamberg, K. (2014). Gender differences in prevalence and associations for use of CAM in a large population study. BMC Complementary & Alternative Medicine, 14, 463. https://doi.org/10.1186/1472-6882-14-463

Lie, D., & Boker, J. (2004). Development and validation of the CAM Health Belief Questionnaire (CHBQ) and CAM use and attitudes amongst medical students. BMC Medical Education, 4, 2. https://doi.org/10.1186/1472-6920-4-2

Lie, D. A., & Boker, J. (2006). Comparative survey of Complementary and Alternative Medicine (CAM) attitudes, use, and information-seeking behaviour among medical students, residents & faculty. BMC Medical Education, 6, 58. https://doi.org/10.1186/1472-6920-6-58

Loh, K. P., Ghorab, H., Clarke, E., Conroy, R., & Barlow, J. (2013). Medical students’ knowledge, perceptions, and interest in complementary and alternative medicine. The Journal of Alternative & Complementary Medicine, 19(4), 360-366. https://doi.org/10.1089/acm.2012.0014

Mahapatra, S., Bhagra, A., Fekadu, B., Li, Z., Bauer, B. A., & Wahner-Roedler, D. L. (2017). Incorporation of integrative medicine education into undergraduate medical education: A longitudinal study. Journal of Integrative Medicine, 15(6), 442-449. https://doi.org/10.1016/s2095-4964(17)60367-4

McCurdy, E. A., Spangler, J. G., Wofford, M. M., Chauvenet, A. R., & McLean, T. W. (2003). Religiosity is associated with the use of complementary medical therapies by pediatric oncology patients. Journal of Pediatric Hematology Oncology, 25(2), 125-129.

Molassiotis, A., & Xu, M. (2004). Quality and safety issues of web-based information about herbal medicines in the treatment of cancer. Complementary Therapies in Medicine, 12(4), 217-227. https://doi.org/10.1016/j.ctim.2004.09.005

Nicolao, M., Täuber, M. G., Marian, F., & Heusser, P. (2010). Complementary medicine courses in Swiss medical schools: Actual status and students’ experience. Swiss Medical Weekly, 140(3-4), 44-51.

Oberbaum, M., Notzer, N., Abramowitz, R., & Branski, D. (2003). Attitude of medical students to the introduction of complementary medicine into the medical curriculum in Israel. Israel Medical Association Journal, 5(2), 139-142.

Owen, D., & Lewith, G. T. (2001). Complementary and alternative medicine (CAM) in the undergraduate medical curriculum: the Southampton experience. Medical Education, 35(1), 73-77. https://doi.org/10.1046/j.1365-2923.2001.00762.x

Pedersen, C. G., Christensen, S., Jensen, A. B., & Zachariae, R. (2013). In God and CAM we trust. Religious faith and use of complementary and alternative medicine (CAM) in a nationwide cohort of women treated for early breast cancer. Journal of Religion and Health, 52(3), 991-1013. https://doi.org/10.1007/s10943-012-9569-x

Perkin, M. R., Pearcy, R. M., & Fraser, J. S. (1994). A comparison of the attitudes shown by general practitioners, hospital doctors and medical students towards alternative medicine. Journal of the Royal Society of Medicine, 87(9), 523-525.

Perlman, A., & Stagnaro-Green, A. (2010). Developing a complementary, alternative, and integrative medicine course: One medical school’s experience. Journal of Alternative & Complementary Medicine, 16(5), 601-605. https://doi.org/10.1089/acm.2009.0276

Rees, C. E., Wearn, A. M., Dennis, I., Amri, H., & Greenfield, S. M. (2009). Medical students’ attitudes to complementary and alternative medicine: Further validation of the IMAQ and findings from an international longitudinal study. Medical Teacher, 31(2), 125-132. https://doi.org/10.1080/01421590802139724

Sadeghi, M., Rabiepoor, S., Forough, A. S., Jabbari, S., & Shahabi, S. (2016). A survey of medical students’ knowledge and attitudes toward complementary and alternative medicine in Urmia, Iran. Journal of Evidence-Based Integrative Medicine, 21(4), 306-310. https://doi.org/10.1177/2156587215605751

Samara, A. M., Barabra, E. R., Quzaih, H. N., & Zyoud, S. H. (2019). Use and acceptance of complementary and alternative medicine among medical students: A cross sectional study from Palestine. BMC Complementary & Alternative Medicine, 19(1), 78. https://doi.org/10.1186/s12906-019-2492-x

Schmidt, K., Rees, C., Greenfield, S., Wearn, A. M., Dennis, I., Patil, N. G., Amri, H., & Boon, H. (2005). Multischool, international survey of medical students’ attitudes toward “holism”. Academic Medicine, 80(10), 955-963. https://doi.org/10.1097/00001888-200510000-00017

Shani-Gershoni, Z., Freud, T., Press, Y., & Peleg, R. (2008). Knowledge and attitudes of internists compared to medical students regarding acupuncture. Israel Medical Association Journal, 10(3), 219-223.

Sharma, V., Holmes, J., & Sarkar, I. (2016). Identifying complementary and alternative medicine usage information from internet resources. A systematic review. Methods of Information in Medicine, 55(4), 322-332. https://doi.org/10.3414/ME15-01-0154

Syverstad, D., Norheim, A., & Fonnebo, V. (1999). Changes in medical students’ attitudes toward acupuncture: The possible impact of role models. Education for Health: Change in Learning & Practice, 12(2), 195-210.

Templeman, K., Robinson, A., & McKenna, L. (2015). Student identification of the need for complementary medicine education in Australian medical curricula: A constructivist grounded theory approach. Complementary Therapies in Medicine, 23(2), 257-264. https://doi.org/10.1016/j.ctim.2015.02.002

Thomson, P., Jones, J., Evans, J. M., & Leslie, S. L. (2012). Factors influencing the use of complementary and alternative medicine and whether patients inform their primary care physician. Complementary Therapies in Medicine, 20(1-2), 45-53. https://doi.org/10.1016/j.ctim.2011.10.001

Torkelson, C., Harris, I., & Kreitzer, M. J. (2006). Evaluation of a complementary and alternative medicine rotation in medical school. Alternative Therapies in Health and Medicine, 12(4), 30-34.

Tricco, A. C., Lillie, E., Zarin, W., O’Brien, K. K., Colquhoun, H., Levac, D., Moher, D., Peters, M. D. J., Horsley, T., Weeks, L., Hempel, S., Akl, E. A., Chang, C., McGowan, J., Stewart, L., Hartling, L., Aldcroft, A., Wilson, M. G., Garritty, C., … Straus, S. E. (2018). PRISMA extension for scoping reviews (PRISMAScR): Checklist and explanation. Annals of Internal Medicine, 169, 467-473. https://doi.org/10.7326/M18-0850

Wetzel, M. S., Eisenberg, D. M., & Kaptchuk, T. J. (1998). Courses involving complementary and alternative medicine at US medical schools. Journal of the American Medical Association, 280(9). https://doi.org/10.1001/jama.280.9.784 

Wong, W. C., Lee, A., Wong, S. Y., Wu, S. C., & Robinson, N. (2006). Strengths, weaknesses, and development of Traditional Chinese Medicine in the health system of Hong Kong: Through the eyes of future Western doctors. Journal of Alternative & Complementary Medicine, 12(2), 185-189; 191-182. https://doi.org/10.1089/acm.2006.12.185

Xie, H., Sang, T., Li, W., Li, L., Gao, Y., Qiu, W., & Zhou, H. (2020). A survey on perceptions of complementary and alternative medicine among undergraduates in China. Evidence-Based Complementary & Alternative Medicine: eCAM, 2020, 9091051. https://doi.org/10.1155/2020/9091051

Xue, C. C., Zhang, A. L., Lin, V., Da Costa, C., & Story, D. F. (2007). Complementary and alternative medicine use in Australia: A national population-based survey. Journal of Alternative & Complementary Medicine, 13(6), 643-650. https://doi.org/10.1089/acm.2006.6355

Yeo, A. S., Yeo, J. C., Yeo, C., Lee, C. H., Lim, L. F., & Lee, T. L. (2005). Perceptions of complementary and alternative medicine amongst medical students in Singapore – A survey. Acupuncture in Medicine, 23(1), 19-26. https://doi.org/10.1136/aim.23.1.19

Yildirim, Y., Parlar, S., Eyigor, S., Sertoz, O. O., Eyigor, C., Fadiloglu, C., & Uyar, M. (2010). An analysis of nursing and medical students’ attitudes towards and knowledge of complementary and alternative medicine (CAM). Journal of Clinical Nursing, 19(7-8), 1157-1166. https://doi.org/10.1111/j.1365-2702.2009.03188.x

Zollman, C., & Vickers, A. (1999). What is complementary medicine? British Journal of Medicine, 319(7211), 693-696. https://doi.org/10.1136/bmj.319.7211.693

*Justin Bilszta
Department of Medical Education,
Melbourne Medical School,
University of Melbourne, Australia
Parkville, Australia 3052
Tel: +61 3 8344 7826
Fax: +6567775702
Email: jbilszta@unimelb.edu.au

Submitted: 6 November 2020
Accepted: 27 January 2021
Published online: 5 October, TAPS 2021, 6(4), 92-106
https://doi.org/10.29060/TAPS.2021-6-4/OA2459

Ardi Findyartini^1,2, Natalia Widiasih Raharjanti³, Nadia Greviana^1,2, Gregorius Ben Prajogi⁴ & Daniar Setyorini²

¹Department of Medical Education, Faculty of Medicine Universitas Indonesia, Indonesia; ²Medical Education Center, Indonesian Medical Education and Research Institute (IMERI), Faculty of Medicine, Universitas Indonesia, Indonesia; ³Department of Psychiatry, Faculty of Medicine, Universitas Indonesia, Indonesia; ⁴Department of Oncology Radiation, Faculty of Medicine, Universitas Indonesia, Indonesia

Abstract

Introduction: E-portfolio may facilitate the entrustment process from supervisors to residents in postgraduate medical education. The present study was aimed at identifying necessary features of an e-portfolio application and to conduct pilot study in a teaching hospital.

Methods: Six programs participated. Eight interviews with education directors and six focus group discussions with residents and supervisors were completed for the needs analysis stage. The application was developed based on the thematic analysis of the needs analysis stage. The mobile-app e-portfolio pilot was conducted for four weeks and a modified version of the System Usability Scale (SUS) was distributed to participants following the pilot program.

Results: Key features of the e-portfolio were identified. A total of 45 supervisors and 66 residents participated in the pilot study. The residents utilised the application according to the clinical activities and supervision level, and the information was fed to the supervisors as per the application design. Challenges during the pilot study in terms of feedback provision and residents’ workload which influence the e-portfolio use for entrustment decisions are discussed.

Conclusion: Current e-portfolio features were created for supervision and are potential to facilitate the entrustment process in Entrustable Professional Activities (EPA) implementation. The pilot study highlighted challenges of the implementation which should be considered for future improvement.

Keywords:           Postgraduate Medical Education, E-Portfolio, Entrustable Professional Activities, Supervision, Needs Analysis, Pilot Study

Practice Highlights

• Development of assessment application, including e-portfolio, should consider input from the stakeholders.
• The e-portfolio should further be defined to emphasise the documentation, reflection, and feedback processes.
• Consideration of adequate evidence for EPA should be determined based on the aim of the e-portfolio.
• A robust e-portfolio implementation will potentially support the implementation of EPA.
• Challenges in e-portfolio implementation should be aware of and tackled for future improvement.

I. INTRODUCTION

Supervision is a critical component of postgraduate medical education as it allows residents to have a gradual achievement of competencies while still fulfilling patient safety standards. Patient care conducted by residents under adequate supervision can have results comparable to the care provided by more experienced medical doctors (Farnan et al., 2012). To entrust the resident, a supervisor considers several factors, including incidents experienced during supervision, the resident’s characteristics, the results of assessments, and encountered situations (Cianciolo & Kegg, 2013).

Ten Cate et al. (2015) introduced the Entrustable Professional Activities (EPA) concept to facilitate the implementation of a competency-based curriculum in the workplace (Ten Cate et al., 2015). EPAs are observable and measurable units of work that can correspond to competency milestones, allowing for safe and effective performance (Carraccio et al., 2017).  Easily accessible and interpreted information about a resident’s past performance using the EPA concept is, therefore, suggested to facilitate the entrustment process, both for ad-hoc and summative purposes (Hauer et al., 2013; Ten Cate et al., 2015).

Entrustment decision-making refers to a supervisor’s decision to trust a resident to carry outpatient care ‘without supervision’ (Crossley et al., 2011; Ten Cate, 2006; Weller et al., 2014). Given the need for assured patient safety, ‘without supervision’ refers to the reduction of educational supervision and the provision of ‘relational autonomy’, whereby interdependence between the resident, the supervisor, the healthcare team, and the healthcare system becomes critical (Holmboe et al., 2011). However, the amount of data accessible about a resident’s performance should be analysed comprehensively to enable ad-hoc entrustment (Sandhu, 2018).

Logbooks and portfolios have been considered as workplace-based assessment methods that would enable summarising a resident’s performance as well as maintaining assessment results during training (van Tartwijk & Driessen, 2009). Electronic portfolios, or e-portfolios, are believed to increase the accessibility of a portfolio in medical training provided that the purpose of the portfolio development is well defined, such as for learning, assessment, or continuing professional development (Deketelaere et al., 2009, Tepper et al, 2020; van Tartwijk & Driessen, 2009). Generally, an e-portfolio aims to monitor a resident’s competency development and to stimulate the capacity for self-reflection (Meeus et al., 2006).

An e-portfolio may consist of a list of a resident’s performance, the supporting evidence, and the resident’s self-reflection (van der Schaaf, et al., 2017). The content is specific according to the e-portfolio’s purpose and the required activities/competencies for the resident at each level (Mulder et al., 2010). The use of an e-portfolio with a mobile application and updated evidence of residents’ performance potentially enhance informed decision-making in the entrustment process, and hence, it can be embedded in the resident supervision system (Ten Cate et al., 2016). An exploratory study in Australia showed that an EPA-driven e-portfolio model assisted trainees and supervisors in agreeing upon expected trainees’ performance in order to obtain competence level (Bramley et al., 2020).

Implementation of e-portfolio as a supporting system for EPA should consider supporting and hindering factors specific to an institution. Implementation of best practices in this area can be accompanied by an exploration of the most suitable system according to the stakeholders’ needs. Therefore, the current study aimed to explore stakeholders’ needs and develop a mobile-app e-portfolio model in a teaching hospital with various postgraduate medical education programmes located in a resource-limited setting. The study also aims to identify challenges of e-portfolio implementation. The research questions of this study were: (1) What are mobile-app e-portfolio features needed to support EPA implementation in the current teaching hospital? (2) How do stakeholders perceive the use of the developed mobile-app e-portfolio during the pilot study? 

II. METHODS

A. Context

Cipto Mangunkusumo General Hospital (CMGH) is the main teaching hospital for the Faculty of Medicine Universitas Indonesia (FMUI) and is located in Jakarta, Indonesia. All 31 study programmes implement a competency-based medical curriculum and EPA approach. CMGH is obliged to have all study programmes participate in clinical activities, which are gradually entrusted to residents in accordance with the resident’s level. The EPA document in form of a matrix is then used to develop supervision system. Supervisors are trained to provide feedback through compulsory clinical teacher training conducted by the faculty. Assessment documentation has been conducted mostly manually in hardcopies for all study programs; consequently, tracking the residents’ performance to provide formative or summative EPA decisions has been challenging in the current setting.

B. Design

This study is participatory action research with a mixed -method approach. The exploration stage was aimed at exploring the needs and features expected from the mobile application. Following the exploration stage, the research team in collaboration with a professional vendor developed the mobile-app e-portfolio according to the results of the exploration stage. The pilot implementation phase aimed to implement and train residents and supervisors in the use of the application. A follow-up evaluation was conducted to assess users’ perception about the use of the e-portfolio application.

C. Respondents

Respondents of this study were education leaders from the medical school and the teaching hospital. Residents, clinical teachers, and administrators from six study programs representing medicine, surgery, and diagnostic: anaesthesiology and intensive care, obstetrics and gynaecology, psychiatry, oncology radiation, radiology, and internal medicine at CMGH/FMUI were recruited with maximum variety sampling method taking into account gender, age, and length of study (residents) or work (clinical teachers). Detailed information of the study was given to all participants who then provided written consents prior to the interviews and completion of the questionnaire.

D. Data Collection

The exploration stage involved interviews with: (1) residency programme directors; (2) education leaders from the medical school and the teaching hospital. Focus group discussions (FGDs) were also conducted with clinical teachers and residents from the six study programmes. The guiding questions of the interviews and FGDs were developed according to e-portfolio and EPA concepts used in medical education, and are shown in Table 1. The data obtained were used as a reference for further development of the application’s content and features. In addition, the EPA document from each study program was also analysed as baseline data for developing the platform.

Following the development of the application, a four-week-long pilot implementation was conducted in November 2018. After week four, the back-end system data were analysed. Supervisors’ and residents’ perceptions of the use of mobile-app e-portfolio for supporting EPA were collected using a modified questionnaire of System Usability Scale (SUS) (Brooke, 1986). Active commentaries were also added in the questionnaire to obtain users’ suggestions:

Table 1. Guiding questions of interviews and focus groups

E. Data Analysis

The transcripts of interviews and FGDs of the need analysis stage were analysed using thematic analysis. The agreed subthemes and themes and the results of the analysis of EPA documents from each study program were then translated into a computational framework to be developed as mobile application e-portfolio’s features, also considering the best practice and theoretical framework of e-portfolio development and EPA practice.

Back-end data of the activities, case variations, and supervisors’ feedback were descriptively analysed using SPSS IBM 22.0. The words submitted on the reflection-on-action section of each activity were counted along with the feedback recorded by the supervisors. Residents’ reflections were categorised into those with descriptions of difficulties (DD), lessons learned (LL), and action plans (AP). The feedback recorded was classified into descriptive (D), constructive (C), and neither descriptive nor constructive (N) categories.

The modified SUS questionnaires were descriptively analysed using chi-square/fisher exact analysis (SPSS 22.0) to compare residents’ and supervisors’ perceptions. The active commentaries in the questionnaire were also thematically analysed to identify areas of improvement in the e-portfolio development.

III. RESULTS

A. Exploration Stage

Eight in-depth interviews were conducted with each residency program director as well as with the leader of FMUI and with teaching hospital director representatives. Two FGDs were conducted, each with three study programmes faculty members. In addition, three FGDs were conducted, each with two study programmes resident representatives. The number and characteristics of the participants are described in Table 2.

Table 2. Characteristics of Focus Group Discussion (FGD) Participants in Exploration Stage

Four main themes and 22 subthemes were identified during the exploration stage. The main themes identified were (1) resident supervision; (2) current implementation of workplace-based assessments; (3) current implementation of portfolios and logbooks; and (4) development of the mobile application e-portfolio.

	Theme	Subtheme	# of Quotes	Quotations
1	Resident supervision	Constraints in supervision (faculty members, patient/clinical cases)	59	“For example, during the mid-level programme, residents were required to be fully supervised, but when they should be fully supervised was not specifically stated.” (W1, p. 1)
		Form of supervision	46	“…we should report to the supervisor when anything goes wrong to get feedback; otherwise, it will become our fault.” (FGD 2, [R5], p. 2)
		Determination of entrustment (written curriculum, residents and patients factor, workplace, agreement among faculty members)	42	“At some point, we might feel that the resident is eligible to perform the procedure after he/she has observed several times, and then we try to let him/her give it a try, but we are actually risking our trust…” (W7, p. 5)
		Background and determination of supervision (workplace, resident’s educational stage)	26	“Supervision was given hierarchically; the third-year residents attain duty as the chief, supervisor, and the highest form was division of weekly rounds.” (FGD 4, [R1], p. 2)
		Documentation of supervision level	10	“Our logbook contains a very clear explanation about supervision level, such as what cases should be achieved so that we can evaluate and reflect on our progress in one semester.” (FGD 3, [R6], p. 9)
2	The current implementation of workplace-based assessment	Scheduled WBA	17	“During each rotation, residents and consultants were scheduled with case-based discussions or miniCEX; while for surgical DOPS, [it] is not specifically scheduled because it depends on the availability of the case.” (W3, p. 1)
		WBA as stage step-up procedure	6	“In our programme, there are several cases we need to get done at each level, usually cases written in the logbook.” (FGD 2, [R7], p. 3)
		Based on the written curriculum	4	“In our programme, we arrange evaluation at every level. We evaluate every intern according to the required competency they should achieve.” (FGD 4, [R3], p. 7)
3	The current implementation of portfolio and logbook	Technical portfolio and logbook implementation (as monitoring, communication, and assessment instrument)	26	“On a routine basis, the form was filled in each afternoon, when the patient’s information, such as name, weight, and height, are recorded.” (FGD 2, [R7], p. 4)
		Constraints in portfolio and logbook implementation (lack of ownership, faculty member factor)	38	“Due to the enormous workload every day, such as examining patients, attending division rounds, filling out electronic health records, and receiving so many instructions, they couldn’t complete the logbook routinely. Besides, the supervisor might find it unnecessary to write down their supervision in the log book after discussions, rounds, or treatments.” (W2, p. 10)
		Understanding of portfolio and logbook for residents’ development (clinical, academic, non-academic)	24	“Portfolio consists of one’s creativity, innovation, and organisation experiences.” (W8, p. 1)
4	Development of the mobile application-based e-portfolio	Residents’ performance and experience (clinical and academic) tracking	20	“We evaluated one intern’s competency in ultrasound, and the result wasn’t quite satisfying. It turned out that this intern lacked of practice that could’ve been done every day since the cases were quite abundant.” (W6, p. 10)
		User-friendliness and real-time access	18	“The application may have to be ‘consultant-friendly,’ so they can finish it in a click.” (FGD 2, [R3], p. 7)
		Feedback documentation	13	“In my opinion, there has to be some immediate feedback.” (FGD 2, [R5], p. 7)
		Accessibility and confidentiality	11	“As they save the record of the treatment that has been checked by the doctor in charge, they can no longer change it because only the supervisor has the authority to change.”(FGD 4, [R4,] p. 19)
		Encompass achievement of competence and modules within the curriculum	7	“We want it to be comprehensive, so it can be evaluated. This is about clinical skills, but then the related academic skill is also important.” (W4, p. 9)
		Ensuring patients were treated by competent and authorised residents	4	“We need to ensure that the patient is treated by competent, authorised, and certified doctors.” (W7, p. 10)
		Reminder and consequences for undocumented activity	5	“There must be something to force the resident to write down his activity so the next day, he can keep up with the new tasks.” (W2, p. 10)
		Paperless attribute, but printable if needed	6	“The record and the format will be the students’ database as well as the supervisors’. Therefore, it needs to be printable for our benefit.” (FGD 2, [R3], p. 7)
		Integrated with service care system	3	“We have to write down the same thing repetitively in so many books and records.” (FGD 3, [R8], p. 12)
		Collect important evidence of accomplishments (cases and complications, image documentation)	4	“Compiling the number of cases is required, and reporting complications in cases is equally important. Let’s say they are in the third level, but complications occurred in most of the procedures they performed.” (W3, p. 7)

Table 3. Themes and Subthemes Identified in Exploration Stage

Submitted: 19 October 2020
Accepted: 12 April 2021
Published online: 5 October, TAPS 2021, 6(4), 80-91
https://doi.org/10.29060/TAPS.2021-6-4/OA2456

Balakrishnan Ashokka^1,2, Tat Leang Lee¹ & Daniëlle M.L. Verstegen³

¹Department of Anaesthesia, National University Health System, National University Hospital, Singapore; ²Centre for Medical Education (CenMED), Yong Loo Lin School of Medicine, National University of Singapore, Singapore; ³Department of Educational Development and Research, Faculty of Health Medicine and Life Sciences, Maastricht University, Netherlands

Abstract

Introduction: There are certain factors in exam preparedness that are not well studied in the postgraduate medical education context. Non-academic predictors have been extensively researched but usually in isolation.

Methods: The study involved a sequential explanatory mixed methods research design. The study was conducted among anaesthesia postgraduates appearing for high-stake nation-wide primary examination. Data obtained by a questionnaire assessing pre-examination attributes were compared with the students’ reflections through focus group discussions (FGD) after the formal declaration of results. The examination had an overall pass rate of 42.9% (18 out of 42).

Results: The study showed that pre-examination questionnaire could identify attributes and study behaviours in the postgraduates who passed. Passers procrastinated three times lesser, pursuing a timetable-based study (conscientiousness); had higher metacognitive self-regulation (p value<0.05) applying concentrated self-directed learning & effective group study and higher self-efficacy compared to those who failed. The focus group discussions affirmed of these attributes in candidates who ‘breeze through exams’. Postgraduate success required better ‘work-study’ balance, self & cross regulation and peer and faculty support.        

Conclusion: Implementing a composite tool to assess ‘exam preparedness’, we propose, would help the learners and teachers to skim for non-academic factors (metacognitive self-regulation, self-efficacy, conscientiousness) that influence the chances of success. Understanding & predicting this would help educators to identify the ‘candidates with difficulty’ and delegate personalised faculty attention. This could guide the exam candidates to have a ‘reality check’ to plan and pace their effort with peer learning, consolidated study and goal orientation.

Keywords:           Postgraduate Exam Success, Non-Academic Predictors, Self-Regulation

Practice Highlights

• Non-academic attributes impact success in postgraduate examinations.
• Postgraduate exam success necessitates work-study & work-life balance.
• Time on task, self-regulation to task demands is needed when assessments are tougher and high stake.
• Exam preparedness: A collective attribute is proposed with a questionnaire to measure predictability of exam success.
• Shunning away from ‘shame of mock vivas’ spirals down to poor chance of passing.

Submitted: 16 January 2021
Accepted: 17 May 2021
Published online: 5 October, TAPS 2021, 6(4), 65-79
https://doi.org/10.29060/TAPS.2021-6-4/OA2447

Yee Cheun Chan¹, Chi Hsien Tan¹ & Jeroen Donkers²

¹Department of Medicine, National University Health System, Singapore; ²Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, Netherlands

Abstract

Introduction: Reflection is a critical component of learning and improvement. It remains unclear as to how it can be effectively developed. We studied the impact of reflective writing in promoting deep reflection in the context of learning Accreditation Council for Graduate Medical Education (ACGME) competencies among residents in an Internal Medicine Residency programme.

Methods: We used a convergent parallel mixed-methods design for this study in 2018. We analysed reflective writings for categories and frequencies of ACGME competencies covered and graded them for levels of reflection. We collected recently graduated residents’ perceptions of the value of reflective writings via individual semi-structured interviews.

Results: We interviewed nine (out of 27) (33%) participants and analysed 35 reflective writings. 30 (86%) of the writings showed a deep level (grade A or B) of reflection. Participants reflected on all six ACGME competencies, especially ‘patient care’. Participants were reluctant to write but found benefits of increased understanding, self-awareness and ability to deal with similar future situations, facilitation of self-evaluation and emotional regulation. Supervisors’ guidance and feedback were lacking.

Conclusion: We found that a reflective writing programme within an Internal Medicine Residency programme promoted deep reflection. Participants especially used self-reflection to enhance their skills in patient care. We recognised the important role of mentor guidance and feedback in enhancing reflective learning.

Keywords:           Reflective Writing, ACGME Competencies, Internal Medicine, Residency

Practice Highlights

• Reflection is a critical component of learning and improvement.
• Written reflections offer theoretical advantages over other forms of reflections in requiring more commitment and ownership of experience, promoting critical thinking and offering more opportunities for feedback.
• Written reflections can be a record for mentored reflection, included in a portfolio, used in ongoing self-assessment and longitudinal integration of learning.
• Practitioners reported benefits of increased understanding, self-awareness and ability to deal with similar future situations, facilitation of self-evaluation and emotional regulation.
• Supervisors’ guidance and feedback are important for enhancing reflective learning.

I. INTRODUCTION

Medical competencies are developed through experience and application, not just knowledge acquisition (Frank et al., 2010). Kolb (1984) conceptualises experiential learning in a four-stage cyclical process.  An experience triggers a reflection on that experience that leads to the formation of abstract concepts and generalisations. These are then tested in future situations, resulting in new experiences.  Reflection is an essential aspect of the learning experience. It remains unclear how it can be developed most effectively.

Reflection is a complex concept that has been defined in several ways. One definition describes it as the process of engaging self in attentive, critical, exploratory, and iterative interactions with one’s thoughts and actions, and their underlying conceptual frame, with a view on the change itself (Nguyen et al., 2014). Thus, reflection has an iterative dimension which describes a cyclic process with phases triggered by experience, which produces new understanding, and then an intention to act differently in future encounters of similar experience (Mann et al., 2009). There is also a vertical dimension correlating to the depth of reflection. The surface levels are more descriptive and less analytical than the deeper levels. For example, Boud et al. (1985) described iterative phases of returning to experience, attending to feelings, re-evaluation of experience and outcome/resolution. Mezirow (1991) described increasing depth of reflection as habitual action, thoughtful action/understanding, reflection, critical reflection. Evidence suggests that deeper levels of reflections are associated with deep approaches to learning (Leung & Kember, 2003).

Reflective writing is a commonly utilised method in developing reflective learning but evidence for its value remains limited. Theoretically, written reflections offer advantages over other types of reflections e.g. verbal discussions. Creating an artefact by writing involves a commitment to learning, ownership of experience, promotes critical thinking and offers more opportunities for feedback (Aronson, 2011). The writings can be a record for mentored reflection, included in a portfolio, used in ongoing self-assessment and longitudinal integration of learning. A systematic review (Winkel et al., 2017) looking at the impact of reflection in graduate medical education found only three studies (Epner & Baile, 2014; Levine et al., 2008; Winkel et al., 2010) that involved reflective writings. Levine et al. (2008) found that the process of narrative writings encouraged deepening of reflection leading to reconsideration of core values and priorities, improved self-awareness, provided an emotional outlet and motivation to improve. However, the study did not formally gauge the depth of reflections in the writings.

We aimed to further study the impact of reflective writing in promoting reflection and the learning of medical competencies. Better understanding this will guide the development of reflective learning skills in training programmes for medical trainees.

A. Research Question

Does reflective writing promote deep reflection in the context of learning core competencies defined by the Accreditation Council for Graduate Medical Education (ACGME) (Accreditation Council for Graduate Medical Education, 2013)?

II. METHODS

A. Research Paradigm and Design

Our study adopted a phenomenological approach. We used a convergent parallel mixed-methods design (Figure 1). Quantitative data included the tabulation of the categories of ACGME competencies and the frequency they were covered in the reflective writings. Quantitative scoring of levels of reflections in the reflective writings was done using two grading scales. Qualitative data included graduates’ perceptions of the value and effects of reflective writings on learning ACGME competencies. The quantitative and qualitative data were analysed, compared and related together to answer the research question.

Figure 1. Convergent parallel mixed-methods design to study the role of reflective writing in promoting reflective learning of ACGME competencies

B. Study Setting and Subjects

The study setting was the Internal Medicine Residency of a single tertiary university hospital in 2018. We have used reflective writing as a tool for developing reflective learning and practice in our Internal Medicine Residency. Our programme has a competency-based curriculum using the ACGME framework. Residents are encouraged to write their reflections on how an encounter or situation helped them develop one or more of the competencies. They are required to include at least two such reflective writings in their portfolio each year. The reflective writings are not graded but are read by the residents’ supervisors as part of their portfolio’s content during regular reviews and by the competency review committee during 6-monthly meetings. They provide insight into the residents’ competencies development.

We invited all past residents (27) who graduated from the programme one year earlier to participate. We used a convenience sampling method. We determined the final number of participants after data saturation was reached in the analysis of the collected qualitative data.

The study was approved by the National Healthcare Group Domain Specific Review Board (NHG DSRB) (Reference number: 2017/01219). We obtained informed consent from each participant.

C. Data Collection

We collected and analysed reflective writings from the participants’ three years of residency. We used individual semi-structured interviews to gather participants’ perceptions to avoid bias from others’ opinions. One researcher (YCC) conducted, recorded and transcribed the interviews. Box 1 shows the main questions that were asked. An interactive approach was used, and interviews conducted till thematic saturation was reached.

Box 1. Main questions asked during interviews

D. Data Analysis

Reflective writings from participants were analysed for the categories as well as frequencies of ACGME competencies covered. They were graded for levels of reflection using grading rubrics. To reduce possible interpretation bias or conflicts related to confidentiality and power relationships, grading was done by an ‘external’ co-researcher (CHT) who was a faculty member of the Neurology residency programme. Two grading scales were used. The first (Box 2) had a simple grading scale from A to F (Moon, 2004). The other grading rubric provided more categorical details and was based on that used by Tsingos et al. (2015) (Supplementary Table 1). The rubric graded the reflective writings on seven stages of reflection based on the model by Boud et al. (1985) and categories of non-reflector, reflector or critical reflector according to Mezirow’s model (Mezirow, 1991). The co-researcher read through each reflective writing and first determined if stages of ‘returning to experience’, ‘attending to feelings’, ‘association’, ‘integration’, ‘validation’, ‘appropriation’ and ‘outcomes of reflection’ were present. He then assessed if the written content related to these stages fit the descriptors for non-reflector, reflector or critical reflector as given in the rubric. Finally, he graded the reflective writing on the simple grading scale of A to F according to the descriptors given (Box 2).

Submitted: 1 December 2020
Accepted: 5 April 2021
Published online: 5 October, TAPS 2021, 6(4), 49-64
https://doi.org/10.29060/TAPS.2021-6-4/OA2443

Yasushi Matsuyama¹, Hitoaki Okazaki¹, Kazuhiko Kotani², Yoshikazu Asada³, Shizukiyo Ishikawa¹, Adam Jon Lebowitz⁴, Jimmie Leppink⁵ & Cees van der Vleuten⁶

¹Medical Education Center, Jichi Medical University, Japan; ²Center for Community Medicine, Jichi Medical University, Japan; ³Center for Information, Jichi Medical University, Japan; ⁴Department of General Education, Jichi Medical University, Japan; ⁵Hull York Medical School, University of York, United Kingdom; ⁶School of Health Professions Education, Maastricht University, The Netherlands

Abstract

Introduction: Previous studies indicate that professional identity formation (PIF), the formation of a self-identity with the internalised values and norms of professionalism, may influence self-regulated learning (SRL). However, it remains unclear whether a PIF-oriented intervention can improve SRL in clinical education. The aim of this study was to explore whether a PIF-oriented mentoring platform improves SRL in a clinical clerkship.

Methods: A mixed-methods study was conducted. Forty-one students in a community-based clinical clerkship (CBCC) used a PIF-oriented mentoring platform. They articulated the values and norms of professionalism in a professional identity essay, elaborated on future professional self-image, and reflected on their current compared to future selves. They made a study plan while referring to PIF-based self-reflection and completed it. The control group of 41 students completed CBCC without the PIF-oriented mentoring platform. Changes in SRL between the two groups were quantitatively compared using the Motivated Strategies for Learning Questionnaire. We explore how PIF elements in the platform affected SRL by qualitative analysis of questionnaire and interview data.

Results: A moderate improvement in intrinsic goal orientation (p = 0.005, ε² = 0.096) and a mild improvement in critical thinking (p = 0.041, ε² = 0.051) were observed in the PIF-oriented platform group. Qualitative analysis revealed that the PIF-oriented platform fostered professional responsibility as a key to expanding learning goals. Gaining authentic knowledge professionally fostered critical thinking, and students began to elaborate knowledge in line with professional task processes.

Conclusion: A PIF-oriented mentoring platform helped students improve SRL during a clinical clerkship.

Keywords:           Self-Regulated Learning, Professional Identity Formation, Clinical Clerkship

Practice Highlights

• Encourage students to verbalise their future self-image as a medical professional.
• Encourage students to reflect on their current selves compared with their perceived future ones.
• Promote in-depth communication between students and role models to foster self-regulated learning.
• Train mentors to become professional role models as self-regulated learners.

I. INTRODUCTION

Rapid advances in clinical knowledge require medical professionals to update their knowledge autonomously throughout their practice. Self-regulation in life-long learning has therefore become an important competency, and competency-based undergraduate medical education has emphasised students’ self-regulated learning (SRL) (Berkhout et al., 2018; Brydges & Butler, 2012; Frank, 2005; Sandars & Cleary, 2011). SRL is defined as learners’ active participation in their own learning processes from metacognitive, motivational, and behavioural perspectives (Zimmerman, 1989). In undergraduate education, SRL has been related to academic achievements (Artino, Cleary et al., 2014; Artino, Dong et al., 2012; Song et al., 2011; Turan & Konan, 2012), clinical skills (Cleary & Sandars, 2011) and emotional management (van Nguyen et al., 2015).

Several reports have claimed that drastic changes in learning context, from structured learning in preclinical years to less-structured and complex learning in clinical clerkships, may be too challenging for students and lead to insufficient learning (Berkhout et al., 2015, 2018; Cho et al., 2017; van Houten-Schat et al., 2018). This question appears more serious in East Asian countries, including Japan, where strong teacher instruction in pre-university education and teacher-centred curricula are the norm (Iwata & Doi, 2017; Lam & Lam, 2009; Tagawa, 2008). In order to make the typically limited clinical clerkship period a fruitful learning opportunity, remediation for struggling students from the perspective of SRL might be important (Durning et al., 2011; van Houten-Schat et al., 2018).

Several reports have shown that individualised mentoring intervention is effective in fostering SRL in clinical practice. For example, Aho et al. (2015) found that mentor-assisted SRL for surgical habits of residents led to more frequent practice and improved skills compared to peers. In Stuart et al. (2005), individualised guidance on strategies and learning plans raised students’ awareness of the learning process. However, the educational interventions undertaken in this study will focus on another contextual characteristic that may facilitate SRL – Professional Identity Formation (PIF) – defined according to Cruess et al. (2014, p.1447) as “a representation of self, achieved in stages over time during which the characteristics, values, and norms of the medical profession are internalised”.

In response to advances in medical science and the increasingly diverse needs of society, “professional” attributes such as autonomy, self-regulation, and social responsibility have been emphasised, in addition to traditional moral and ethical education emphasising healer roles (Cruess & Cruess, 1997, 2019). Furthermore, formation of professional identity can result in individuals thinking, acting, and feeling like physicians (Cruess et al., 2014; Cruess & Cruess, 2019). During the formation of professional identity, medical students begin to perceive belonging to a professional community and increase attention to role models (Jarvis-Selinger et al., 2012; Kalet, Buckvar-Keltz, Harnik et al., 2017; Kalet, Buckvar-Keltz, Monson et al., 2018). Emulation of role models’ self-regulation in learning behaviour is also expected.

People are more likely to interpret difficult experiences as task important when an accessible identity feels congruent to the task (Oyserman et al., 2017). In the context of this study, growing professional identity as a ‘physician-to-be’ might strengthen the perceived importance of engaging in challenges during clinical clerkships, and in self-regulating learning behaviours. In addition, when physicians perceive their identity as professionals, they begin to view daily learning tasks as high-stakes, and to self-regulate learning behaviours as coping strategies (Matsuyama et al., 2018). Another study has suggested that an explicit future professional self-image in medical students leads to self-reflection, increased attention to learning strategies of professional role models, and diversification of learning strategies (Matsuyama et al., 2019). Given that PIF is associated with motivational states, self-reflection, and diversified learning strategies, SRL may be facilitated by introduction of a PIF-oriented intervention.

This study specifically focused on PIF as a facilitating factor for SRL, because previous studies  have suggested possible benefits of PIF-oriented education even for East Asian medical students, who are generally considered to have less SRL due to influence of pre-university education, with its strong faculty instruction and in-university teacher-centred curricula (Matsuyama et al., 2018, 2019).

The purpose of this study was to examine whether SRL during clinical training can be fostered using a mentorship tool emphasising PIF, in addition to conventional mentorship by faculty members. In the pre- and post-clinical clerkship mentorship, students were asked to elaborate on their future self-image as professionals and compare their current and future selves to strengthen self-reflection under mentor support. Learners were aided in articulating their values and norms of medical professionalism by using the Professional Identity Essay (PIE) (Kalet, Buckvar-Keltz, Harnik et al., 2017; Kalet, Buckvar-Keltz, Monson et al., 2018), a self-administered questionnaire with 9 questions relevant to PIF. This encouraged mentors to understand the developmental stage of each learner’s professionalism and to provide individualised feedback on PIE and their future self-image. The feedback was also aimed at remediation for those whose self-images showed underdeveloped professionalism (low developmental stages in PIE). Study plans in the clinical clerkship were developed with reference to PIE-based self-images. We have named this platform ‘PIF-oriented mentoring platform for SRL (PIF-SRL)’. An overview of PIF-SRL is provided in Figure 1.

Figure 1. Overview of the PIF-SRL and research data collection

This studied centred on two research questions:

1. Does PIF-SRL improve SRL during the period around the clinical clerkship?
2. If so, how does the PIF-oriented elements in PIF-SRL improve SRL?

This study was approved by the ethics committee of Jichi Medical University (reference number: 19-001).

II. METHODS

A. Settings

1) Community-based clinical clerkship in Jichi Medical University: The mission of Jichi Medical University (JMU) is to educate students to become general practitioners competent in rural settings. Students are accepted per a quota system from each of Japan’s 47 prefectures. In the current curriculum at JMU, students complete lectures on almost every basic and clinical medicine area before the end of Year 3. From Year 4 to Year 6, students are permitted to participate in a clinical clerkship during which they receive training centred on taking patient histories and providing physical examination. Previously, most ward placements took place at the University’s affiliated hospital providing little opportunity for in-depth communication with role models in rural settings.

The community-based clinical clerkship (CBCC) was introduced in 1998 (Okayama & Kajii, 2011). For 2 weeks from late August to early September in Year 5, CBCC students stay at a community hospital or clinic in their home prefecture where JMU graduates work. Every year, two to five JMU graduates per prefecture are appointed to be CBCC mentors based on their motivation to teach in their prefectural communities. JMU faculty ensures the instructional quality of mentors by conducting annual face-to-face faculty development sessions. In addition, standards for learning activities are proposed, including ambulatory care, home care, hospital care, placement in mobile clinics, on-call work, rehabilitation, health education, health check-ups, vaccination, day services, and placement in welfare facilities (welfare institutions or nursing homes for the aged) (Okayama & Kajii, 2011).

Prior to the regular CBCC without PIF-SRL, students had several opportunities to communicate with mentors by telephone or e-mail. However, these communications did not provide sufficient opportunity for students to develop an image of future professionalism. We therefore felt that the communication framework in the regular CBCC did not fulfil its potential to stimulate PIF.

2) PIF-SRL for the CBCC: The PIF-SRL platform (Matsuyama et al., 2021) used an online communication platform, Google Forms. Before the CBCC began, mentors were briefed multiple times in writing and verbally on the purpose of the PIF-SRL to ensure their mentorship was PIF-oriented. They were also asked to read a manual which provided specific step-by-step responses from their first interaction with medical students on Google Forms in mid-July to their post-practice reflection in early to mid-September (Figure 1).

In the pre-clerkship phase, participants were asked to write their reflexive PIE. We used PIE because it is useful for helping learners articulate their own values and norms of medical professionalism, and for teachers providing feedback with rubrics based on Kegan’s constructive developmental theory (Kalet, Buckvar-Keltz, Harnik et al., 2017; Kalet, Buckvar-Keltz, Monson et al., 2018; Kegan, 1994). The present study used a Japanese version of the original English-language PIE form. Translation to Japanese was conducted by the main author (YM). To validate the translation accuracy, back-translation to English was conducted by a co-author (AJL), an American professor living in Japan who is literate in both English and Japanese. In accordance with in-depth talks based on PIE contents, students were encouraged to verbalise their future self-image and reflect on their current selves compared with their perceived future ones. The Question 6 of PART 2 in the PIF-SRL asked alumni mentors to describe their present self-image (Matsuyama et al., 2021). Students in PIF-SRL can also refer to this information when verbalising their future self-image.

Additionally, students in PIF-SRL were asked to create study plans for CBCC based on gaps between their current and future selves, and identify one or more learning goals. Referring to these plans, alumni mentors observed students and gave just-in-time feedback. Also, mentors provided students with learning strategies to help them overcome potential future challenges. Apart from these instructions, mentors were essentially independent in their education of the medical students. After the two-week clerkship, students were asked to re-articulate their own future image and received feedback from their mentors by Google Forms (Figure 1).

B. Subjects

First, PIF-SRL mentors were selected. In 2018 and 2019, 94 JMU graduates registered as attending rural physicians for the CBCC. Among them, 20 candidates for PIF-SRL mentors in 2018 and 2019 were randomly selected and informed consent for their contribution to PIF-SRL in this study was requested. Eventually, 17 and 13 JMU alumni agreed to participate in 2018 and 2019, respectively, 8 of whom participated in both years.

Independent of this study, the JMU Center for Community Medicine matched one alumnus with one to three students for the CBCC. The 30 PIF-SRL alumni-mentors were paired with 1 to 3 students each before informed consent was obtained. In this study, students paired with the 30 PIF-SRL alumni were chosen as candidates for the PIF-SRL group subjects. There were 22 and 20 candidates in 2018 and 2019, respectively. One candidate in 2019 declined participation. Eventually, 41 students were registered as subjects in the PIF-SRL group. Simultaneously, 41 control subjects were chosen from the same school year cohort and informed consent to participate was obtained. Control subjects experienced the regular CBCC mentorship without PIF-SRL. Because previous studies have shown that gender (Ray et al., 2003) and academic performance (Lucieer et al., 2016) might independently influence SRL development, participants in both groups were paired by gender and academic ranking from the previous year (Year 4).

C. Procedures

A convergent mixed method was chosen for the first research question ‘Does PIF-SRL improve SRL during the period around the clinical clerkship?’ to identify common data between quantitative and qualitative results (Creswell & Clark, 2017). Next, an explanatory mixed method was used to address the second research question ‘How does the PIF-oriented elements in PIF-SRL improve SRL?’. A rationale for this method is that follow-up qualitative approaches can explain quantitative results (Creswell & Clark, 2017). We conducted this mixed method study in the paradigm of pragmatism, which emphasises solutions to research questions and integrates qualitative and quantitative research results to obtain general findings (Shannon-Baker, 2016).

1) Quantitative approach: Learner SRL levels were measured with a Japanese-language version of the Motivated Strategies for Learning Questionnaire (MSLQ-J) before (mid-July) and after (mid-September) subjects participated in CBCC with or without PIF-SRL. The MSLQ (Pintrich et al., 1991) is composed of 81 items with seven-point Likert scales quantifying levels of 9 types of SRL strategy (rehearsal, elaboration, organisation, critical thinking, metacognitive self-regulation, time and study environment, effort regulation, peer learning, and help seeking), and 6 variables of motivation states (intrinsic goal orientation, extrinsic goal orientation, task value, control of learning beliefs, self-efficacy for learning and performance, and test anxiety). All 81 items of the MSLQ were used as done previously in a medical school context (Cho et al., 2017) because it was believed the 15 SRL-related categories could multi-dimensionally identify differences between the PIF-SRL group and control group. Translation to Japanese was conducted by the main author (YM) and back-translation to English by a co-author (AJL). For the MSLQ validation, the Cronbach alpha and McDonald omega were measured for 15 categories (Matsuyama et al., 2021).

2) Qualitative approach: To explore changes in motivation, strategies and reflective behaviours from self-reflection and study during the clinical clerkship, we created a questionnaire composed of seven questions (Matsuyama et al., 2021). All participants answered the questionnaire within two weeks after post-CBCC PIF-SRL mentoring. In late September 2019, one-on-one interviews were also conducted after intensive qualitative data analysis of the questionnaire from 41 participants in 2018 and 2019. Three interviewers familiar with the CBCC but not engaged in the assessment of Year 5 students conducted interviews in order to encourage interviewees to openly articulate their own perceptions. Twelve students in the 2019 PIF-SRL group consented to participate in interviews conducted in a semi-structured manner using an interview form with similar questions to those in the questionnaire (Matsuyama et al., 2021). The interviewers were instructed beforehand by the main author (YM) to obtain data about changes in perception regarding motivation, strategies and reflective behaviours after experiencing PIF-SRL. After collecting interview data from 10 students, the two main authors (YM and HO) found no additional meaningful codes emerging and, concluding that data saturation had been reached (Hennink et al., 2017), and stopped further interview data collection.

D. Analysis

1) Quantitative approach: The 15 MSLQ-J pre-intervention subcategory scores of the PIF-SRL group and control group were compared using Kruskal-Wallis one-way analysis of variance (ANOVA). After confirming that there were no statistically significant differences between the two groups, subtracted (post-pre) scores in the 15 MSLQ-J subcategories were compared between the two groups using Kruskal-Wallis one-way ANOVA. This non-parametric method was used because of considerable skewness in distribution in the scales of several items and outliers in other scales in MSLQ-J. A p-value < 0.05 was considered statistically significant. The effect sizes for comparisons were also calculated using ε² values, wherein small effect sizes ranged from 0.01 to <0.08, medium effect sizes ranged from 0.08 to <0.26 and large effect sizes ranged from ≥0.26. We used JAMOVI version 1.0.7.0 for statistical analysis.

2) Qualitative approach: Qualitative data from PIE, the questionnaire and interviews were analysed using thematic analysis. Anonymised qualitative data were analysed in accordance with the six phases proposed by Braun and Clarke (2006). Initial coding was conducted by the two Japanese researchers (YM and HO). YM, the lead author, was involved in the development of PIF-SRL and has previous experience with qualitative studies relevant to SRL. HO was not directly engaged in PIF-SRL but has had experience in qualitative studies relevant to SRL. The transcripts were thoroughly analysed using an inductive coding approach until agreement on coding was achieved through repetitive face-to-face meetings between the pair.

The focus was on changes in SRL (motivation, learning strategies, and reflective behaviours), and student opinion of the effects of PIF-SRL on SRL. Representative codes and statements were translated into English by an American professor literate in both English and Japanese (AJL). In the final phase, two other authors (JL and CV; education psychologists familiar with SRL) joined the discussion, and a higher-level synthesis of the codes was developed.

III. RESULTS

A. Quantitative Data

Mean averages, standard deviations, and median averages for fifteen MSLQ-J categories plus gender and academic rank data at pre-intervention are shown in Table 1. No categories significantly differed between the PIF-SRL and control groups.

The subtracted (post- minus pre-intervention) between-group scores in the 15 MSLQ-J categories are shown in Table 2. Improvements in 1. Intrinsic goal orientation and 10. Critical thinking were significantly better in the PIF-SRL group than the control group with ε² values 0.096 (p = .005) and 0.051 (p = .041), respectively. The quantitative data used in this study are accessible (Matsuyama et al., 2021).

A higher-level synthesis of the codes eventually resulted in three major themes corresponding to the second research question, ‘How does the PIF-oriented elements in PIF-SRL improve SRL?’ 

1) Active expansion of learning goals based on professional responsibility: The first theme consisted of subthemes which included learning motivated from responsibility, clear learning goals based on explicit self-images, and wider learning goals based on wider perceptions of professional roles.

Students viewed in-depth communication with mentors through the PIF-SRL platform with PIE helpful for imagining their future professional responsibilities in a positive and objective manner.

‘A lot of opening questions were, for example, what do you expect from work, what is the worst that can happen if you failed to live up to the expectations you have set for yourself, that’s the situation you’re working under when you’re a doctor, and the first time I really felt this was the time I really should be aware of this, it was positive, and actually although I was still just a student, I could objectively assess what I was thinking…’

(Interview, 1D-10)

During PIF-SRL mentoring, they were able to realise that knowledge beyond what they were currently learning in the curriculum was required of them as professionals. They were actively trying to set learning goals that they could connect to what they would need to learn in the future.

 

‘Knowing what skills and knowledge the region expects of you, you can create a working image of your future situation, and this becomes motivation to learn about new areas you weren’t aware of before.’

 (Questionnaire, 2019-19)

Aside from the expansion of perceived learning goals, students also began to see that self-study was required to ensure the authenticity of medical knowledge applicable to their future professional work. This was linked to Theme 3.

‘What I got was that incomplete understanding or recall wasn’t going to cut it when actually applying knowledge in the clinic. I began to train with the awareness that I wouldn’t be useful there if I didn’t understand and remember all I learned about disease.’

(Questionnaire, 2018-21)

2) Elaboration by linking future professional task processes to daily self-study contents: The second theme was comprised of subthemes which included focus on the task processes of role models and relating daily self-study content to future roles.

After PIF-SRL, students began to imagine what they would do as professionals in future medical practice at a task process-based level. Because their vivid future professional image helped them identify deep responsibilities for their professional tasks, they began to carefully observe mentors’ complete professional tasks in the clinical clerkship and articulate task processes. This in turn encouraged students to select learning strategies which linked information in daily self-study materials (e.g. textbooks) to professional task processes, which is referred to as ‘elaboration’ in MSLQ.

‘I started to learn in terms of what I would do if it were me. I also started to think about the details and not just the general flow of things, and to apply them as much as possible to reality.’

(Questionnaire, 2018-8)

‘Since the clerkship, I’ve been able to relate and recall what I actually encountered in the clinical clerkship, and when I was actually reading textbooks in self-study, I was able to align it with my future work content, and if there were things that applied, I would emphasise them… The study method that I use to impress upon myself was strengthened in the mentoring and clerkship…’

(Interview, 1D-3)

3) Critical thinking based on the pursuit of authentic medical knowledge: The third theme was comprised of subthemes which included realisation of the significance of authenticity pursuit, access to a wide variety of educational materials, and critical reflection.

Because they began to perceive that what they were learning would affect the lives of individual patients, they recognised the significance of gaining authentic knowledge that could be applied to each patient, differentiated from textbook-based universal knowledge.

‘It’s really important to learn about disease by studying the texts and gaining comprehensive knowledge, but since that tertiary material is insufficient for responding to clinical questions and individual patient backgrounds, I’m not sure that knowledge is useful in clinical practice. For that, what’s most meaningful are secondary materials like UpToDate, or if you still have doubts then primary material research papers.

(Questionnaire, 2019-1)

After beginning to pursue authenticity of medical knowledge, students found diversity and inconsistency in information from learning materials. To deal with this, they began to formulate inquiries focusing on self-study, a variety of information resources, and different viewpoints. Through this strategic shift, critical thinking emerged in an intrinsic manner.

‘I’ve come to think opinions are going to diverge to some extent as you get down to actually asking opinions of several different doctors, and these are choices you have to make, the stages you go through when studying.’

(Interview, 1C-7)

‘Having the ability to doubt, and because it’s science not taking everything at face value, becoming sceptical, I think that’s necessary.’

(Interview, 3C-14)

IV. DISCUSSION

Regarding the first research question ‘Does PIF-SRL improve SRL during the period around the clinical clerkship?’, our findings of a moderate improvement in intrinsic goal orientation and a mild improvement in critical thinking in the PIF-SRL group compared with the control group suggested that PIF-SRL promotes SRL. The qualitative analysis supports the MSLQ-J results. With regard to intrinsic goal orientation, students’ recognition of their future ‘professional responsibility’ was a key to fostering this. Further, recognition of professional responsibility led to critical thinking — critical evaluation of learning materials or their interpretation — as a means of pursuing authenticity of information for professional task processes. The consistency between quantitative and qualitative data was notable in understanding the outcomes of the PIF-oriented mentoring on SRL.

There was no statistical significance in the difference between the PIF-SRL and the control groups regarding elaboration in MSLQ-J data. However, qualitative analysis illuminated that PIF-SRL students’ attention to professional task processes triggered a strategy toward elaboration of knowledge in accordance with their future professional task processes. Reasons for this discrepancy might include the number of participants and sensitivity of the outcome measurement (Tonkin-Crine et al., 2016). Further research is needed to address these issues. However, we believe that in order to remediate learning strategies necessary for professional tasks or professionalism, it is useful to reflect on daily self-learning in accordance with the process of professional work shown by role models.

Recent research pertaining to educational intervention for SRL emphasises analysing the learning process of a particular task in detail and remediating the individual process (Artino et al., 2014; Durning et al., 2011; Gandomkar et al., 2016). While we generally concur with this, learning tasks that take place in clinical practice are limitless, so it would be beneficial to explore a foundational intervention applicable to a variety of tasks in clinical settings. The PIF-oriented mentoring in this study is presented as a foundational SRL intervention for a variety of clinical settings wherein learners can recognise professional identities and role models can suggest learning strategies.

The major strength of this study is that the subjects were Japanese students, who are considered to engage less in self-regulation than their Western counterparts (Iwata & Doi, 2017; Matsuyama et al., 2018; Tagawa, 2008). We believe that our study can provide educators with evidence that PIF-oriented educational schemes promote better learning behaviours in institutions desiring to promote undergraduate SRL. Another strength is that there were few formal classes or training programmes – i.e., intervening confounders — except for PIF-SRL and CBCCs between pre- and post-data collection (Figure 1). We believe the study schedule, without other educational interventions, optimised learning comparison outcomes between the two groups. For instance, changes in accessing learning materials for critical thinking (e.g. UpToDate) can be attributed mostly to the experiences in PIF-oriented mentoring and the subsequent CBCCs.

This study also has some limitations. First, we did not collect one-on-one interview data from 2018 cohort. Second, we did not examine student SRL changes from the mentor’s perspective, despite the fact that mentors’ perceptions of changes in students’ SRL would be as useful as, or more useful than students’ self-administered data. Third, reference to mentors’ self-images in the PIF-SRL could help students in PIF-SRL construct their professional self-image more clearly; however, this may cause bias student statements in the PIF-SRL form regardless of their real professionalism. Future research needs to measure PIF and SRL in more multifaceted and objective manners. Lastly, we do not have long-term outcomes of PIF-SRL. According to previous studies (Cruess et al., 2014; Cruess & Cruess, 2019; Kalet et al., 2017; Oyserman et al., 2017), identity is flexibly attuned to immediate situations rather than fixed in memory. Plus, frequently and fluently cued identities form stable ones. Accordingly, we speculate that the repetitive use of PIF-SRL could strengthen learners’ SRL. Validation of this notion would require a longitudinal cohort study.

Evaluation of these results suggests that the ability of PIF-SRL to work effectively requires that the quality of mentors be guaranteed. One reason for the success of the relatively random combination of students and mentors in this PIF-SRL group is that all are future rural physicians, and their mentors are also alumni of JMU in rural practice. On the other hand, for medical students from other universities who can follow diverse specialties and career paths, use of the PIF-SRL will require pairing medical students with appropriate mentors who can respond to students’ identities or future images. Also, it is important to pair students not only by their interest in future expertise, but also by looking at the mental and physical traits of individual students as they relate to motivation and student career choices (Henning et al., 2017). Moreover, in-depth conversations that would foster professionalism might not be done only through text messages in the PIF-SRL platform, but also through video calls and in-person meetings that would convey the participants’ voices, facial expressions, and mood. We think it is important for mentors to actively provide opportunities for direct dialogue with students. In light of this challenge, the present study supports PIF-oriented intervention as a method for improvement in SRL.

V. CONCLUSION

Allowing for these limitations and the need for further research, this study indicated that PIF-oriented education in a clinical clerkship with alumni mentors increased immediate intrinsic goal orientation and promoted a shift to SRL. Their SRL was characterised as task process-based elaboration, with critical thinking emerging from the pursuit of authenticity in medical practice.

Notes on Contributors

Yasushi Matsuyama reviewed the literature, designed the study, conducted both quantitative and qualitative data analysis, and wrote the manuscript.

Hitoaki Okazaki conducted qualitative data analysis.

Kazuhiko Kotani designed the study, and collected both quantitative and qualitative data.

Yoshikazu Asada collected both quantitative and qualitative data.

Shizukiyo Ishikawa collected both quantitative and qualitative data.

Adam Jon Lebowitz contributed to Japanese-English translation of data collection tools and qualitative data from questionnaire and interviews.

Jimmie Leppink reviewed the literature, designed the study, conducted both quantitative and qualitative data analysis, and wrote the manuscript.

Cees van der Vleuten reviewed the literature, designed the study, conducted both quantitative and qualitative data analysis, and wrote the manuscript.

All the authors have read and approved the final manuscript.

Ethical Approval

This study was approved by the ethics committee of Jichi Medical University (Reference number: 19-001). Consent was obtained from all participants for the research study.

Acknowledgements

We would like to express our sincere gratitude to Dr. Adina Kalet and Dr. Verna Monson for their consultation and provision of supplementary materials. We would also like to thank Drs. Yasuko Aoyama, Yoshitaka Maeda, and Maiko Watanabe for their support in conducting one-on-one interviews. We would also like to thank Ms. Akemi Watanabe and Yasuko Koguchi for their helpful assistance. 

Funding

This work was supported by JSPS KAKENHI [Grant number JP17K08924]. 

Declaration of Interest

The authors report no conflicts of interest.

Data availability

The quantitative data used in this study, Supplemental files are available at https://doi.org/10.6084/m9.figshare.14312507

Representative qualitative data translated into English are shown in the Result section (Matsuyama et al., 2021). All qualitative data written in Japanese are available from the corresponding author on reasonable request.

A preprint of the previous version of our manuscript, which is not peer-reviewed, is available at https://www.researchsquare.com/article/rs-12667/v1

References

Aho, J. M., Ruparel, R. K., Graham, E., Zendejas-Mummert, B., Heller, S. F., Farley, D. R., & Bingener, J. (2015). Mentor-guided self-directed learning affects resident practice. Journal of Surgical Education, 72(4), 674–679. https://doi.org/10.1016/j.jsurg.2015.01.008o

Artino, A. R., Jr., Cleary, T. J., Dong, T., Hemmer, P. A., & Durning, S. J. (2014). Exploring clinical reasoning in novices: A self-regulated learning microanalytic assessment approach. Medical Education, 48(3), 280–291. https://doi.org/10.1111/medu.12303

Artino, A. R., Jr., Dong, T., DeZee, K. J., Gilliland, W. R., Waechter, D. M., Cruess, D., & Durning, S. J. (2012). Achievement goal structures and self-regulated learning: Relationships and changes in medical school. Academic Medicine, 87(10), 1375–1381. https://doi.org/10.1097/ACM.0b013e3182676b55

Berkhout, J. J., Helmich, E., Teunissen, P. W., van den Berg, J. W., van der Vleuten, C. P., & Jaarsma, A. D. (2015). Exploring the factors influencing clinical students’ self-regulated learning. Medical Education, 49(6), 589–600. https://doi.org/10.1111/medu.12671

Berkhout, J. J., Helmich, E., Teunissen, P. W., van der Vleuten, C., & Jaarsma, A. (2018). Context matters when striving to promote active and lifelong learning in medical education. Medical Education, 52(1), 34–44. https://doi.org/10.1111/medu.13463

Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77-101. https://doi.org/10.1191/1478088706qp063oa

Brydges, R., & Butler, D. (2012). A reflective analysis of medical education research on self-regulation in learning and practice. Medical Education, 46(1), 71–79. https://doi.org/10.1111/j.1365-2923.2011.04100.x

Cho, K. K., Marjadi, B., Langendyk, V., & Hu, W. (2017). Medical student changes in self-regulated learning during the transition to the clinical environment. BMC Medical Education, 17(1), 59. https://doi.org/10.1186/s12909-017-0902-7

Cleary, T. J., & Sandars, J. (2011). Assessing self-regulatory processes during clinical skill performance: A pilot study. Medical Teacher, 33(7), e368–e374. https://doi.org/10.3109/0142159X.2011.577464

Creswell, J. W., & Clark, V. L. P. (2017). Designing and conducting mixed methods research. SAGE Publications.

Cruess, R. L., & Cruess, S. R. (1997). Teaching medicine as a profession in the service of healing. Academic Medicine, 72(11), 941–952. https://doi.org/10.1097/00001888-199711000-00009

Cruess, R. L., Cruess, S. R., Boudreau, J. D., Snell, L., & Steinert, Y. (2014). Reframing medical education to support professional identity formation. Academic Medicine, 89(11), 1446–1451. https://doi.org/10.1097/ACM.0000000000000427

Cruess, S. R., & Cruess, R. L. (2019). The development of professional identity. In T. Swanwick., K. Forrest., B. C. O’Brein (Eds.), Understanding medical education; Evidence, theory, and practice (3^rd ed., pp. 239–254). https://doi.org/10.1002/9781119373780.ch17

Durning, S. J., Cleary, T. J., Sandars, J., Hemmer, P., Kokotailo, P., & Artino, A. R. (2011). Perspective: Viewing “strugglers” through a different lens: How a self-regulated learning perspective can help medical educators with assessment and remediation. Academic Medicine, 86(4), 488–495. https://doi.org/10.1097/ACM.0b013e31820dc384

Frank, J. R. (2005). The CanMEDS 2005 physician competency framework. Better standards. Better physicians. Better care. The Royal College of Physicians and Surgeons of Canada.

Gandomkar, R., Mirzazadeh, A., Jalili, M., Yazdani, K., Fata, L., & Sandars, J. (2016). Self-regulated learning processes of medical students during an academic learning task. Medical Education, 50(10), 1065–1074. https://doi.org/10.1111/medu.12975

Henning, M. A., Krägeloh, C. U., Booth, R., Hill, E. M., Chen, J., & Webster, C. S. (2017). Profiling potential medical students and exploring determinants of career choice. The Asia Pacific Scholar, 2(1), 7-15. https://doi.org/10.29060/TAPS.2017-2-1/OA1019

Hennink, M. M., Kaiser, B. N., & Marconi, V. C. (2017). Code saturation versus meaning saturation: How many interviews are enough? Qualitative Health Research, 27(4), 591–608. https://doi.org/10.1177/1049732316665344

Iwata, K., & Doi, A. (2017). Can hybrid educational activities of team and problem based learning program be effective for Japanese medical students? International Journal of Medical Education, 8, 176–178. https://doi.org/10.5116/ijme.591b.2bc8

Jarvis-Selinger, S., Pratt, D. D., & Regehr, G. (2012). Competency is not enough: Integrating identity formation into the medical education discourse. Academic Medicine, 87(9), 1185–1190. https://doi.org/10.1097/ACM.0b013e3182604968

Kalet, A., Buckvar-Keltz, L., Harnik, V., Monson, V., Hubbard, S., Crowe, R., Song, H. S., & Yingling, S. (2017). Measuring professional identity formation early in medical school. Medical Teacher, 39(3), 255–261. https://doi.org/10.1080/0142159X.2017.1270437

Kalet, A., Buckvar-Keltz, L., Monson, V., Harnik, V., Hubbard, S., Crowe, R., Ark, T. K., Hyuksoon, Song, H. S., & Yingling, S. (2018). Professional Identity Formation in medical school: One measure reflects changes during pre-clerkship training. MedEdPublish, 7(1), 41. https://doi.org/10.15694/mep.2018.0000041.1

Kegan, R. (1994). In over our heads: The mental demands of modern life. Harvard University Press.

Lam, T. P., & Lam, Y. Y. (2009). Medical education reform: The Asian experience. Academic Medicine, 84(9), 1313–1317. https://doi.org/10.1097/ACM.0b013e3181b18189

Lucieer, S. M., Jonker, L., Visscher, C., Rikers, R. M. J. P., & Themmen, A. P. N. (2016). Self-regulated learning and academic performance in medical education. Medical Teacher, 38(6), 585–593. https://doi.org/10.3109/0142159X.2015.1073240

Matsuyama, Y., Nakaya, M., Okazaki, H., Leppink, J., & van der Vleuten, C. (2018). Contextual attributes promote or hinder self-regulated learning: A qualitative study contrasting rural physicians with undergraduate learners in Japan. Medical Teacher, 40(3), 285–295. https://doi.org/10.1080/0142159X.2017.1406074

Matsuyama, Y., Nakaya, M., Okazaki, H., Lebowitz, A. J., Leppink, J., & van der Vleuten, C. (2019). Does changing from a teacher-centered to a learner-centered context promote self-regulated learning: A qualitative study in a Japanese undergraduate setting. BMC Medical Education, 19(1), 152. https://doi.org/10.1186/s12909-019-1550-x

Matsuyama, Y., Okazaki, H., Kotani, K., Asada, Y., Ishikawa, S., Lebowitz, A. J., Leppink, J., & van der Vleuten, C. (2021). Professional identity formation-oriented mentoring technique as a method to improve self-regulated learning: A mixed-method study. [Data set]. Figshare. https://doi.org/10.6084/m9.figshare.14312507

Matsuyama, Y., Okazaki, H., Kotani, K., Asada, Y., Ishikawa, S., Lebowitz, A. J., Leppink, J., & van der Vleuten, C. (2020). Education in professional identity formation enhances self-regulated learning: A mixed-method exploratory study from a community-based clinical clerkship in Japan. [Preprint]. Research Square. January 28, 2020. [cited June 10, 2020] Available from: https://doi.org/10.21203/rs.2.22088/v1

Okayama, M., & Kajii, E. (2011). Does community-based education increase students’ motivation to practice community health care? A cross sectional study. BMC Medical Education, 11(1), 19. https://doi.org/10.1186/1472-6920-11-19

Oyserman, D., Lewis, Jr., N. A., Yan, V. X., Fisher, O., O’Donnell, S. C., & Horowitz, E. (2017). An identity-based motivation framework for self-regulation. Psychological Inquiry, 28(2-3), 139-147. http://doi.org/10.1080/1047840X.2017.1337406

Pintrich, P. R., Smith, D. A. F., Garcia, T., & McKeachie, W. J. (1991). A manual for the use of the Motivated Strategies for Learning Questionnaire (Technical Report 91-B-004). The Regents of the University of Michigan. https://files.eric.ed.gov/fulltext/ED338122.pdf

Ray, M. W., Garavalia, L. S., & Gredler, M. E. (2003, April 21-25). Gender differences in self-regulated learning, task value, and achievement in developmental college students. Annual Meeting of the American Educational Research Association. https://files.eric.ed.gov/fulltext/ED476176.pdf

Sandars, J., & Cleary, T. J. (2011). Self-regulation theory: Applications to medical education: AMEE Guide No. 58. Medical Teacher, 33(11), 875–886. https://doi.org/10.3109/0142159X.2011.595434

Shannon-Baker, P. (2016). Making paradigms meaningful in mixed methods research. Journal of Mixed Methods Research, 10(4), 319-334. https://doi.org/10.1177/1558689815575861

Song, H. S., Kalet, A. L., & Plass, J. L. (2011). Assessing medical students’ self-regulation as aptitude in computer-based learning. Advances in Health Sciences Education: Theory and Practice, 16(1), 97–107. https://doi.org/10.1007/s10459-010-9248-1

Stuart, E., Sectish, T. C., & Huffman, L. C. (2005). Are residents ready for self-directed learning? A pilot program of individualized learning plans in continuity clinic. Ambulatory Pediatrics, 5(5), 298–301. https://doi.org/10.1367/A04-091R.1

Tagawa, M. (2008). Physician self-directed learning and education. The Kaohsiung Journal of Medical Sciences, 24(7), 380–385. https://doi.org/10.1016/S1607-551X(08)70136-0

Tonkin-Crine, S., Anthierens, S., Hood, K., Yardley, L., Cals, J. W., Francis, N. A., Coenen, S., van der Velden, A. W., Godycki-Cwirko, M., Llor, C., Butler, C. C., Verheij, T. J., Goossens, H., Little, P., & GRACE INTRO/CHAMP consortium (2016). Discrepancies between qualitative and quantitative evaluation of randomised controlled trial results: Achieving clarity through mixed methods triangulation. Implementation Science, 11, 66. https://doi.org/10.1186/s13012-016-0436-0

Turan, S., & Konan, A. (2012). Self-regulated learning strategies used in surgical clerkship and the relationship with clinical achievement. Journal of Surgical Education, 69(2), 218–225. https://doi.org/10.1016/j.jsurg.2011.09.003

van Houten-Schat, M. A., Berkhout, J. J., van Dijk, N., Endedijk, M. D., Jaarsma, A. D. C., & Diemers, A. D. (2018). Self-regulated learning in the clinical context: A systematic review. Medical Education, 52(10), 1008–1015. https://doi.org/10.1111/medu.13615

Van Nguyen, H., Laohasiriwong, W., Saengsuwan, J., Thinkhamrop, B., & Wright, P. (2015). The relationships between the use of self-regulated learning strategies and depression among medical students: An accelerated prospective cohort study. Psychology, Health & Medicine, 20(1), 59–70. https://doi.org/10.1080/13548506.2014.894640

Zimmerman, B. J. (1989). A social cognitive view of self-regulated academic learning. Journal of Educational Psychology, 81(3), 329.

*Yasushi Matsuyama
3311-1
Yakushiji, Shimotsuke,
Tochigi, 329-0498
Email: yasushim@jichi.ac.jp

Submitted: 2 November 2020
Accepted: 8 February 2021
Published online: 5 October, TAPS 2021, 6(4), 37-48
https://doi.org/10.29060/TAPS.2021-6-4/OA2425

Stephen Bradley¹, Aaron Ooi², Kerry Stafford³, Shuvayon Mukherjee¹ & Marcus A. Henning⁴

¹Department of Paediatrics, Lakes District Health Board, New Zealand; ²Department of Paediatrics, Waikato District Health Board, New Zealand; ³Department of Paediatrics, Christchurch Hospital, New Zealand; ⁴Centre for Medical and Health Sciences Education, University of Auckland, New Zealand

Abstract

Introduction: The paediatric team handover process is a crucial workplace practice and comprises the transfer of patient information from one shift to another involving medical professionals and students.  A qualitative study was performed to analyse the feasibility, functionality, benefits and limitations of the dramaturgical approach when applied to examining a handover session. 

Methods: Data relating to one handover were collected and analysed from video and audio recordings, notes created by two independent observers and a de-identified copy of the handover sheet. 

Results: The dramaturgical constructs and subsequent findings allowed us to make informed inferences about the dynamics of the handover procedure. The directors/lead actors consisted of a consultant and a registrar.  One consultant was transitory and the remaining 12 attendees were either major support, support or bit actors.  The students (bit actors/audience) were included when a learning point was emphasised.  The script was informal and improvised as the discussion emphasised certain facets of patient care or accentuated learning points.  The staging involved the seating arrangement, a whiteboard, computer screen and ongoing data presentation.  The performance suggested a handover of two halves: one emphasising learning and the other allocation of patient care responsibility. 

Conclusion: We concluded that the real-life drama occurring within a handover was feasibly analysed, with its functionality demonstrated, using the dramaturgical investigative system.  The multifaceted recordings enabled researchers to review the ‘authentic’ handover system without censorship. These findings have implications for educational and organisational research.

Keywords:           Dramaturgical, Handover, Paediatric, Methodology

Practice Highlights

• Dramaturgical methodology provided a unique, authentic and detailed analysis of the handover.
• The dramaturgical research methodology used to evaluate the handover was feasible and functional.
• This research methodology can be used to analyse education within similar team based settings.
• This research methodology can be applied to the team handovers and other complex health meetings.
• This research methodology identifies important clinical/educational roles and dynamics within teams.

I. INTRODUCTION

Hospital team handovers involve effective transfer of information and responsibility from one health professional to another, ensuring continuity of patient care (Australian Medical Association Limited, 2006; Hilligoss & Cohen, 2011). The level of communication needs to be comprehensive, unambiguous and coherent so that patient information is easily understood, thus optimising patient care through the meaningful and efficient transfer of patient information (Fujikawa et al., 2021). This is crucial given the ramifications for optimising patient care and minimising potential treatment error, including miscued transfer of knowledge, insertion of faulty or misleading information, treatment delay, and poor patient outcomes (Arora et al., 2005; Bomba & Prakash, 2005). To explore the nuances occurring in handover practice from organisational behaviour and educational perspectives, different methodological approaches need to be developed.

In this paper, we propose that the dramaturgical approach can optimally analyse handover dynamics, as it is an integrated, pragmatic and multidimensional approach. This approach uses multi-source feedback from video and audio recordings, observer records, and transcripts of analysis. The dramaturgical approach argues that the individuals present in the activity assume different roles that influence the way they communicate and behave (Canary et al., 2008; Goffman, 1959; Henderson, 2005). Using this approach, the handover activity can be analysed much like a drama or element of theatre. The dramaturgical approach has the potential to offer insights into the clinical and educational handover components, much like the insights drawn when applying this approach to analysing healthcare simulations (Crea, 2017), decision-making aspects of an emergency department triage (Forde, 2014), and behaviour of radiographers and their patients (Murphy, 2009). This analytical approach enables the researcher to be present at the moment of the occurrence, rather than relying on retrospective data obtained when subsequently interviewing participants. Interviews can be a powerful means of obtaining information, but require participants to accurately retell their experiences of the activity (DiCicco‐Bloom & Crabtree, 2006).  The dramaturgical approach tells it for what it is, and allows researchers the ability to see and hear the authentic process of communication (Goffman, 1959; Murphy, 2009). We surmised that the dramaturgical approach would be a more comprehensive evaluative system and thus well suited for collecting observational data that could inform training and development initiatives within hospital systems.

The research aim of this study was to explore the feasibility and functionality of the dramaturgical methodological system of analysis not yet applied to the handover procedure. 

The research questions driving this study include:

1. How can the dramaturgical approach feasibly be applied to the handover system?
2. How does the dramaturgical approach describe the functional dynamics of the handover procedure?
3. What are the benefits and limitations associated with applying this research methodology?

 

II. METHODS

A. Phenomenon of Interest

Feasibility, functionality and quality were informed by examples taken from one handover involving team discussion regarding patients admitted to a general paediatric ward (20 beds) and a Special Care Baby Unit (8 cots) in a New Zealand secondary-level hospital (Hensher et al., 2006).

B. Sample/Participants

All the health professionals and medical students involved in one handover were asked to volunteer for the study, with eligibility determined by consent and approval from hospital management. Informed consent was obtained after eligible participants read a detailed information sheet, provided by an administrator, followed by a consent form that they then signed. 

C. Data Collection

Data were obtained from several sources.

1. Five video cameras were situated in the handover room to obtain multiple angles of the handover. Two audio recorders were placed in the room and served as the primary sources of data for transcription.
2. The final transcription of events was checked by all authors using data from the cameras.
3. Notes on the salient aspects of handover interactions were made by two present ‘unknown’ observers (i.e., one medical student and one medical educationalist).

A diagram of the seating positions of each participant was constructed (see Figure 1).

Figure 1: Handover room layout depicting seating arrangements, participants (P1-P15, with original position participants sat in), 2 observers (Ob1 and Ob2) and equipment.

Submitted: 17 October 2020
Accepted: 12 April 2021
Published online: 5 October, TAPS 2021, 6(4), 26-36
https://doi.org/10.29060/TAPS.2021-6-4/OA2420

Chee Yang Chin¹, Si Qi Tan², Swee Leng Kui^1,2, Kurugulasigamoney Gunasegaran¹ & Jill Cheng Sim Lee³

¹Department of Cardiovascular Medicine, National Heart Centre Singapore, Singapore; ²Ministry of Health Holdings, Singapore; ³Department of Obstetrics and Gynaecology, KK Women’s and Children’s Hospital, Singapore

Abstract

Introduction: Sleep deprivation impacts clinical performance. However, literature is conflicting, with insufficient focus on patient outcomes. The aim of this study was to assess if patient satisfaction and prescription errors in outpatient clinics were adversely affected when consulting post-call versus non-post-call registrars.

Methods: This prospective, quantitative study was set in a large teaching hospital in Singapore. Between November 2015 and February 2016, patients from clinics run by a registrar after 24-hour shift were recruited to post-call group. Patients from non-post-call clinics run by the same registrar were controls. Outcome measures were patient satisfaction, using 5-item 4-point Likert scale questionnaire, and prescribing error rate, defined as number of errors over number of orders. Differences were analysed using chi-squared test.

Results: 103 of 106 (97%) patients in 9 post-call clinics and 93 of 105 (90%) patients in 9 non-post-call clinics were recruited. Questionnaire completion rate was 99%. 536 and 526 prescriptions were ordered in post-call and non-post-call groups, respectively. Percentage of top-box responses (greatest satisfaction) was higher in post-call group overall (79.3% versus 62.4%, p<0.001), and for each questionnaire item. There was no significant difference in prescribing errors (1.31% versus 2.28%, p=0.23).

Conclusion: Patient satisfaction and prescribing error rates in outpatient clinics were not detrimentally affected. This provides some objective evidence that patients may safely consult post-call registrars. True impacts of sleep deprivation remain poorly understood, and larger, longer term, multicentre studies would inform generalisability. Qualitative studies of fatigue may shed light on complex interactions of emotions that compensate for tiredness.

Practice Highlights

• Studies of sleep deprivation and work-hour restrictions are conflicting, with few on patient outcome.
• We compared patient satisfaction and prescribing errors of clinics post-call, versus non-post-call.
• We demonstrate quantitative evidence that patients may safely consult post-call registrars in clinic.
• Impairment may be mitigated by adaptive effort to be conscientious and post call euphoria.
• No recommendations were made to change the current practice of registrars running post-call clinics.

I. INTRODUCTION

Sleep deprivation and fatigue are thought to have detrimental effects on trainee doctors’ performance. It was previously shown that the Epworth Sleepiness Scale score in junior doctors was equivalent to that of patients with sleep apnoea and narcolepsy (Mustafa et al., 2005). Indeed, cognitive psychomotor performance after 24 hours of sustained wakefulness was found to be equivalent to a blood alcohol concentration twice the prohibited level of alcohol intoxication (Dawson & Reid, 1997). The performance and safety to practise of fatigued trainee doctors, especially at the end of a long shift, is thus of utmost concern.

Increasing awareness of fatigue-related medical errors has led to reforms in trainee doctor working hours. Work hours were limited by the Accreditation Council for Graduate Medical Education (ACGME) in the USA in 2003 to 80 hours per week, and by the European Working Time Directive (EWTD) in 2009 to 48 hours per week. A subsequent ACGME stipulation in 2017 further restricted work periods to 24 consecutive hours for first‐year residents (Rosenbaum & Lamas, 2012). Still, doubts remain as to the clinical impact of these restrictions (Baldwin et al., 2011; Fletcher et al., 2005). A systematic review of ACGME’s 2011 work hour restrictions found no improvements in patient care or resident well-being, and in fact revealed unintended negative impacts on resident education (Bolster & Rourke, 2015). Indeed, low job satisfaction was found to be associated with a higher rate of attrition from training (Lee et al., 2019). Thus, the optimal working hours and working patterns for trainee doctors remain ill-defined.

Studies on the impact of sleep deprivation and fatigue on trainee doctors’ performance and wellbeing have yielded mixed results. Chronic sleep deprivation was shown to be particularly detrimental to tasks requiring sustained concentration (Bertram, 1988). After an extended night shift, intensive care unit physicians made 36% more serious medical errors (Landrigan et al., 2004), were 61% more likely to suffer a percutaneous injury (Ayas et al., 2006) and 2.3 times more likely to have a motor vehicle collision (Barger et al., 2005). Hostility and anger in trainee doctors were also more prevalent after a night’s sleep loss (Saadat et al., 2016; Samkoff & Jacques, 1991). Cardiologists were particularly prone to sleep deprivation due to frequent call coverage responsibilities, and in a recent 2019 survey of 481 cardiologists, 46% felt work performance hindrance and 8.6% described sleep-deprivation related adverse events (Lobo et al., 2019).

Yet, other studies have not observed a similar impairment of ability amongst fatigued trainee doctors. There were fewer errors in cognitive tests and cardiopulmonary resuscitation (CPR) manoeuvres in residents after on-call duty (Domínguez et al., 2009; Hamui-Sutton et al., 2013), and psychomotor performance and acquisition of laparoscopic and robotic simulator skills in post-call trainee doctors were also found to be no different from non-post-call trainee doctors (Robison et al., 2018; Yi et al., 2013).

Notably, the outcome measures in the vast majority of these published studies were either subjective through interviews with the trainee doctors themselves, which introduces bias in results, or based on performance on robotic surgical simulators, which is not reflective of true patient contact. Few studies focused on actual patient outcome measures.

Trainee doctors (“registrars”) at our institution, a high volume tertiary referral centre for cardiovascular medicine in Singapore, may be expected to run busy outpatient clinics after a 24-hour continuous in-hospital shift. Typically, up to 20 patients are seen within 4 hours in these clinics. Concerns linger regarding the suitability of post-call registrars running these high-intensity clinics, both in terms of psychomotor ability as well as proneness to error.

With the lack of published literature on the impact of fatigue on trainee doctor performance in outpatient clinics, we aimed to study for ourselves whether patient outcomes were adversely affected if they consulted a post-call registrar, versus a non-post-call registrar. To achieve this, we decided on two practical and measurable outcomes: patient satisfaction and prescribing error rates. Quantifying patient satisfaction allowed an assessment of the registrars’ ability to communicate and show empathy despite their fatigued state. Prescribing error rates allowed an objective evaluation of patient safety. Using two very different outcome measures was a means to allow some triangulation of results and to reduce bias.

Our hypothesis was that both patient outcome measures – patient satisfaction and prescribing error rate – would be worse off in the clinics run by registrars when they were post-call versus when they were not post-call.

II. METHODS

A. Study Setting and Population Design

This study was set in a high volume, specialised Cardiology clinic in a large teaching hospital in Singapore. Registrars, who are trainee doctors in the final stages of training prior to specialist accreditation, work overnight in-hospital shifts as part of service and training commitments. “On-call” duties last from 7am to 1pm the following day (30 continuous hours), during which they are responsible for all urgent inpatient and Emergency department Cardiology referrals. “Post-call” refers to the last 6 hours of the continuous 30-hour shift, during which they may be posted to run busy outpatient clinics.

B. Patient Recruitment

Between November 2015 and February 2016, all patients who attended clinics run by a post-call registrar were recruited to the post-call group. Patients who attended clinics run by the same registrar on a non-post-call day were recruited as controls into the non-post-call group. This provided internal control by comparing performances by the same registrar post-call and non-post call. Patients were approached for participation after their clinic consultation so as not to bias the consultation. They were informed of the purpose of the study but not informed whether the registrar they had just consulted was post-call or not, so as not to bias their responses. All patients, whether new or established, were approached.

C. Patient Satisfaction Questionnaire

Patient satisfaction was evaluated using a 5 item questionnaire (Appendix A). These questions were adapted from the Hospital Consumer Assessment of Healthcare Providers And Systems (HCAHPS), a well-established standardised instrument for the measurement of patient perspectives on hospital care (Giordano et al., 2009; Goldstein et al., 2005).

1) During today’s consultation, did the doctor treat you with courtesy and respect?

2) During today’s consultation, did you feel the doctor listened carefully to you?

3) During today’s consultation, were you given the opportunity to ask your questions?

4) During today’s consultation, did the doctor explain things in a way you could understand?

5) How would you rate your overall satisfaction with today’s consultation?

For each item, as per the HCAHPS questionnaire format, patients were asked to rate responses on a 4-point Likert scale. Smiley faces were incorporated to provide visual representations. The questionnaire was in English and Chinese text, which would have been understood by the vast majority of the patient population. Relevant translations in Chinese, Malay or Tamil languages were provided by research assistants for illiterate patients or those who required further clarification, with care not to influence the patients’ responses. Only fully completed questionnaires were included in the data analysis.

D. Prescribing Error Rates

All prescription orders entered electronically were logged. As is standard procedure for our outpatient clinics, all prescriptions are reviewed by the Pharmacy Department prior to dispensing. All suspected prescription errors will be verified via a phone call to the prescribing doctor, who will confirm whether that prescription was intended or if it truly was an error. All verified prescription errors, including medication name, dose or frequency, or omission of a necessary drug, were recorded regardless of severity. The Pharmacy Department conducts regular internal audits to ensure that this process is accurate and consistent.

The total number of medications ordered and total number of errors were retrospectively summed. Prescribing error rate was calculated by dividing total number of prescribing errors by total number of medications ordered.

E. Consent and Ethics

In accordance with local Institutional Review Board (IRB) guidelines, the study protocol was exempted from full IRB review as it was non-interventional and intended primarily for medical education. All recruited patients provided informed consent to participate. All data were anonymised, with no personal identifiers recorded.

F. Data Analysis

For the patient satisfaction questionnaire, the “top-box” response was defined as the most positive response on the Likert scale (for example “all of the time” or “very satisfied”). The proportions of “top-box” responses were calculated and reported for each questionnaire item and overall items. Only “top-box” responses were sought, to target a higher quality of performance.

For both patient satisfaction and prescribing error outcomes, comparisons between post-call and non-post-call groups were tested for statistical significance with p<0.05 using the chi-squared test. All statistical analyses were reviewed by a biostatistician.

III. RESULTS

The recruitment period consisted of 9 outpatient clinics run by 8 post-call registrars. Two of these clinics were run by the same registrar who was post-call on separate days. We then identified a further 9 clinics run by the same registrars during days when they were not post-call. Thus, 7 registrars had one post-call clinic and one non-post-call clinic studied, while 1 registrar had two post-call clinics and two non-post-call clinics studied.

All patients seen were invited to participate. 103 of 106 (97%) patients in the post-call group and 93 of 105 (90%) in the non-post-call group consented. The number of patients seen per clinic ranged from 7 to 20 (median 11) in the post-call group, and 7 to 17 (median 10) in the non-post-call group.

Basic demographic data on the registrars and the distribution of patients in these clinics were summarised in Table 1.

Table 1: Basic demographic data of registrars and distribution of patients seen and number of medications ordered per clinic.

Note: ^{[1] †}Numbers in brackets indicate number of patients who declined participation. *C1 and C2 refer to the same registrar, who had 2 post-call and 2 non-post-call clinics from whom patients were recruited for the study.

Figure 1: Summary of patient questionnaire responses according to registrars’ post-call status. Questionnaire items were shown on the y-axis and percentage of top-box responses were shown on the x-axis.

B. Prescribing Error Rates

536 and 526 medications were ordered in the post-call and non-post-call group respectively. Seven prescribing errors were made in the post-call group and 12 in the non-post-call group (Table 2).

Overall, there was no significant difference in prescribing error rates (1.31% versus 2.28%, p=0.231).

Table 2: Distribution of medications ordered and prescribing errors made according to registrar.

IV. DISCUSSION

To our knowledge, this is one of very few studies on post-call trainee doctor performance in the context of a realistic outpatient setting, centred on patient outcomes. In our study, the results of both patient satisfaction and prescription errors were congruent in demonstrating that these measurable patient outcomes were not adversely affected when patients were seen by a post-call registrar in an outpatient clinic setting.

A. Patient Satisfaction

Patient satisfaction is an important measure as it demonstrates a doctor’s ability to communicate, which is a key element of any patient-doctor encounter. Fatigue leading to increased stress levels and hostility could have negatively influenced this.

In our study, rather than create a novel and unvalidated questionnaire, we adapted the HCAHPS, a well-established standardised instrument for the measurement of patient perspectives on hospital care (Giordano et al., 2009; Goldstein et al., 2005). As per HCAHPS, only the proportions of top-box responses (i.e., the percentage of patients who rated a particular questionnaire item the most positively) were taken into account in the analysis. This was a means of assessing and aiming for the highest quality of care possible.

The results from overall and individual item top-box responses in our questionnaire demonstrated that patient satisfaction was statistically significantly (p-value <0.001) higher in the post-call group. Indeed, 6 out of 8 registrars had higher percentages of overall top-box responses when post-call, adding consistency to this finding. Nonetheless, given the small cohort of registrars surveyed, there were still the likelihood that this was a chance finding, and previous studies in this area had shown only subtle differences in patient satisfaction (Hoellein et al., 2004; Liu & Wissow, 2011). The key take away point was that patient satisfaction in our study was not adversely affected when patients saw a post-call registrar.

There were two other studies examining post-call trainee doctors in outpatient clinics. In Liu and Wissow’s study (2011), 170 pre-recorded paediatric clinic consultations were analysed to reveal that parents were twice as likely to request a post-call doctor to repeat explanations, both post-call doctors and parents used paraphrasing more frequently to check that they understood each other, and trainee doctors were perceived as less willing to help the patient accomplish their goals during the visit. This suggested that communication was less effective when trainee doctors were post-call. The earlier study by Hoellein et al. (2004), using a 7-item 10-point Likert scale questionnaire, found that patients were significantly less satisfied (mean score 8.99) when seen by a post-call trainee doctor than when seen by a non-post-call trainee doctor (mean score 9.31). Differences in study design and clinic setup could explain the discrepancy between our findings and those of these 2 previous studies. In these 2 studies, patient volume was low at 3 to 7 per clinic, trainee doctors and patients were aware that they were being recorded, and some clinics surveyed were the trainee doctors’ “continuity clinics” where visits were more likely returning patients with already established rapport. In our clinic setting, patient volumes were higher, patients were only approached after the consultation, and none of the clinics were “continuity clinics”.

Another possible explanation for the lack of reduction in patient satisfaction in our study was the phenomenon of “post-call euphoria”, where trainee doctors experience an upbeat mood associated with being able to look forward to the end of a long 30-hour shift and finishing work earlier than usual, at 1 p.m. instead of at 6 p.m. They may be emotionally less stressed in clinic than if they had a long day of work ahead, particularly if they were just starting an on-call shift. In Liu and Wissow’s study (2011), trainee doctors reported more anxiety and frustration when running morning clinics when they still had the rest of the day’s work ahead of them. Additionally, they described themselves as tired, but not more unhappy, when they were post-call as opposed to when they had left on time the day before. Furthermore, post-call trainee doctors made attempts to cope with their fatigue, adaptively or maladaptively, and were seemingly able to compensate for compromised abilities. This counter-intuitive psychological effect warrants further study.

B. Prescribing Error Rates

Prescribing errors were evaluated as a marker of patient safety. Errors have been shown to be provoked by situations of high workload, stress and fatigue (Keers et al., 2013; Tully et al., 2009). Overall case volume in particular was associated with increased rate of major diagnostic discrepancies for junior residents (Hanna et al., 2016), which is relevant to our high-volume clinic setting. The questionnaire among anaesthesiologists from Santa Catarina by Erdmann et al. (2016) revealed that most respondents committed more than one error in drug administration, with distraction and fatigue being the highest contributing factor of 64.9%. Interns made substantially more serious medical errors in the intensive care unit when they worked frequent shifts of 24 hours or more, than when they worked shorter shifts (Landrigan et al., 2004). There was only one other study that dealt specifically with medication errors in the post-call setting. In a retrospective study of 8,195 inpatient prescriptions, trainee doctors were found to be significantly more likely to commit an error on-call and post-call, than when they were off-call, by 2.16%, with odds ratio 1.44 (Hendey et al., 2005).

Our study in the outpatient setting revealed no statistically significant difference in prescribing error rates between patients who had seen a post-call registrar (1.31%) and those who had seen a non-post-call registrar (2.28%) across 536 and 526 prescriptions respectively. The overall prescribing error rate seen in our study was comparable to that of Hendey et al. (2005) (1.79% versus 2.16%, respectively), supporting the validity of our results. Importantly, prescribing error rates were low in both groups, providing reassurance for patient safety.

Previous studies on the impact of fatigue on clinical performance show mixed results. Surgical trainees awake the previous night made 20% more errors in stimulators (Taffinder et al., 1998). Gastroenterologists who performed emergent procedures the previous night had a significant 24% decrease in adenoma detection rates on colonoscopies performed post-call (Benson et al., 2014). While long work hours may contribute to mistakes, the lack of supervision, faulty handovers and large patient caseloads were important factors that further compounded these errors.

Yet, other studies have observed that performance was no worse amongst fatigued trainee doctors. Yi et al. (2013) found no significant difference in psychomotor performance of general surgery residents performing laparoscopic simulator tasks after a 24-hour call work shift versus a 12-hour night-float shift. Robison et al. (2018) reported no significant difference in acquisition of robotic simulator skills between pre-call and post-call general surgery residents despite higher levels of fatigue, and Domínguez et al. (2009) and Hamui-Sutton et al. (2013) observed fewer errors in cognitive tests and CPR manoeuvres in residents after on-call duty. This paradox of improved execution might be explained by an increased motivation to execute tasks as efficiently as possible, even if cognitive processes were theoretically impaired by sleep deprivation (Ayas et al., 2006; Barger et al., 2005; Taffinder et al., 1998).

Additionally, it was postulated that alertness and task performance were impaired immediately in the first 2 hours of awakening due to sleep inertia, but these detrimental effects dissipate with time (Jewett et al., 1999). Post-call residents in our study might therefore have had a chance to recover by the time clinic started.

C. Strengths and Limitations

A particular strength of our study was that registrars who had post-call clinics surveyed also had a non-post-call clinic surveyed. This enabled them to act as controls for themselves and reduced potential bias that may be introduced as a result of differences in registrar demographics, such as age, gender and training experience. Second, there was a high patient participation rate in both the post-call group (97%) and non-post-call group (90%), with a high questionnaire return rate (99% in both groups). This increased the validity of our findings. Third, our study measured two different patient outcomes as a means of triangulation, and both were congruent in showing better outcomes in the post-call group. Fourth, we approached patients for their participation only after their clinic consultation, so as not to have influenced their interaction with the doctor. Fifth, patients were not informed if the registrar that they had just consulted was post-call or not, so as not to influence their questionnaire responses.

Nonetheless, being a single-centre study, caution should be exercised when attempting to generalise these findings to other healthcare settings. Second, only 8 of 18 eligible registrars in the department had clinics surveyed during the study period; thus the findings may not be reflective of the entire cohort. Third, there may have been factors outside the doctor-patient encounter that affected overall patient satisfaction, such as waiting times. Fourth, we did not quantify how busy each registrar’s on-call shift was and what amount of rest they obtained; the results might have been influenced by the amount of rest obtained. Fifth, we could not control for the complexity of the patients seen in the clinics; it is perceivable that a clinic with more straightforward patients would have been more easily managed by a fatigued registrar and vice versa.

For future studies, increasing cohort size of registrars over a longer period of time, and at more categories of call status (such as post-call, on-call and neither post- nor on-call) could improve the generalisability of the findings and reveal influences of call status on performance. In addition, incorporating qualitative data on registrars’ emotions and perceptions of their own performances in clinic would improve understanding of potential factors that may influence performance.

V. CONCLUSION

Our study showed that patient satisfaction and prescription error rates were not adversely affected when patients consulted a post-call registrar versus a non-post-call registrar in an outpatient clinic. This provides reassurance that the current practice of post-call trainee doctors running clinics is not harmful to patients. Complex interactions of associated positive and negative emotional factors could possibly compensate for any physical and mental tiredness. Impairment may be mitigated by adaptive effort to be conscientious. In our institution, no recommendations were made to change the current practice of registrars running post-call clinics. Ultimately, the relationship between being post-call with physician performance and patient outcomes remains poorly understood.

Notes on Contributors

Dr. Chin Chee Yang is a Consultant in Cardiology at the National Heart Centre Singapore and Clinical Assistant Professor at Duke-NUS Medical School. He has a Master of Science in Clinical Education. CCY designed and directed the study, implemented the research, data analysis and writing of the manuscript.

Dr. Tan Si Qi is a Medical Officer with the Ministry of Health Holdings, Singapore. TSQ contributed to the data analysis and writing of the manuscript.

Dr. Kui Swee Leng is a senior resident in Cardiology at Singhealth. She is an Adjunct Research Fellow with Singhealth Duke-NUS Cardiovascular Sciences Academic Clinical Programme (ACP) and Singhealth Associate in Education. KSL contributed to design and implementation of the research.

​Assistant Professor Kurugulasigamoney Gunasegaran is a Senior Consultant at the National Heart Centre Singapore. KG contributed to the design and overall supervision of the research.

Dr. Jill Cheng Sim Lee is a Consultant in Obstetrics and Gynaecology and Associate Programme Director to the SingHealth Obstetrics and Gynaecology Residency Programme. She has a Master of Science in Clinical Education. JCSL contributed to the design of the research and writing of the manuscript.

All authors have read and approved the final manuscript.

Ethical Approval

The study was categorised as an educational study hence formal Institutional Board Review (IRB) was exempted. We have attached the Local IRB Exception waiver document.

Funding

No funding source was utilised in this study. 

Declaration of Interest

Authors report no declaration of interest.

Data Availability

In our study, all raw data from questionnaires was collected on hard copies and not scanned into a repository. All other data are presented in the direct manuscript.

References

Ayas, N. T., Barger, L. K., Cade, B. E., Hashimoto, D. M., Rosner, B., Cronin, J. W., Speizer, F. E., & Czeisler, C. A. (2006). Extended work duration and the risk of self-reported percutaneous injuries in interns. Journal of the American Medical Association, 296(9), 1055–1062. https://doi.org/10.1001/jama.296.9.1055

Baldwin, K., Namdari, S., Donegan, D., Kamath, A. F., & Mehta, S. (2011). Early effects of resident work-hour restrictions on patient safety: A Systematic review and plea for improved studies. The Journal of Bone and Joint Surgery-American Volume, 93(2), e5(1)-e5(9). https://doi.org/10.2106/JBJS.J.00367

Barger, L. K., Cade, B. E., Ayas, N. T., Cronin, J. W., Rosner, B., Speizer, F. E., & Czeisler, C. A. (2005). Extended work shifts and the risk of motor vehicle crashes among interns. New England Journal of Medicine, 352(2), 125–134. https://doi.org/10.1056/NEJMoa041401

Benson, M., Grimes, I., Gopal, D., Reichelderfer, M., Soni, A., Benson, H., Austin, K., & Pfau, P. (2014). Influence of previous night call and sleep deprivation on screening colonoscopy quality. The American Journal of Gastroenterology, 109(8), 1133–1137. https://doi.org/10.1038/ajg.2014.28

Bertram, D. A. (1988). Characteristics of shifts and second-year resident performance in an emergency department. New York State Journal of Medicine, 88(1), 10–14.

Bolster, L., & Rourke, L. (2015). The Effect of Restricting Residents’ Duty hours on patient safety, resident well-being, and resident education: An updated systematic review. Journal of Graduate Medical Education, 7(3), 349–363. https://doi.org/10.4300/JGME-D-14-00612.1

Dawson, D., & Reid, K. (1997). Fatigue, alcohol and performance impairment. Nature, 388(6639), 235–235. https://doi.org/10.1038/40775

Domínguez, P., Grosso, M. L., Pagotto, B., Taliercio, V., & Allegri, R. (2009). Efectos de la privación de sueño en el desempeño de los médicos residentes de pediatría [Effects of sleep deprivation on medical performance of pediatric residents]. Archivos Argentinos De Pediatria, 107(3), 241–245.

Erdmann, T. R., Garcia, J. H. S., Loureiro, M. L., Monteiro, M. P., &amp; Brunharo, G. M. (2016). Profile of drug administration errors in anesthesia among anesthesiologists from Santa Catarina. Brazilian Journal of Anesthesiology (Elsevier), 66(1), 105–110. https://doi.org/10.1016/j.bjane.2014.06.011

Fletcher, K. E., Underwood, W., Davis, S. Q., Mangrulkar, R. S., McMahon, L. F., & Saint, S. (2005). Effects of work hour reduction on residents’ lives: A systematic review. Journal of the American Medical Association, 294(9), 1088–1100. https://doi.org/10.1001/jama.294.9.1088

Giordano, L. A., Elliott, M. N., Goldstein, E., Lehrman, W. G., & Spencer, P. A. (2009). Development, implementation, and public reporting of the HCAHPS survey. Medical Care Research and Review, 67(1), 27–37. https://doi.org/10.1177/1077558709341065

Goldstein, E., Farquhar, M., Crofton, C., Darby, C., & Garfinkel, S. (2005). Measuring hospital care from the patients’ perspective: An overview of the CAHPS® hospital survey development process: Measuring hospital care from the patients’ perspective. Health Services Research, 40(6p2), 1977–1995. https://doi.org/10.1111/j.1475-6773.2005.00477.x

Hamui-Sutton, L., Barragán-Pérez, V., Fuentes-García, R., Monsalvo-Obregón, E. C., & Fouilloux-Morales, C. (2013). Efectos de la privación de sueño en las habilidades cognitivas, psicomotoras y su relación con las características personales de los médicos residentes [Sleep deprivation effects on cognitive, psychomotor skills and its relationship with personal characteristics of resident doctors]. Cirugia Y Cirujanos, 81(4), 317–327.

Hanna, T. N., Loehfelm, T., Khosa, F., Rohatgi, S., & Johnson, J. (2016). Overnight shift work: factors contributing to diagnostic discrepancies. Emergency Radiology, 23(1), 41-47. https://doi.org/10.1007/s10140-015-1355-0

Hendey, G. W., Barth, B. E., & Soliz, T. (2005). Overnight and postcall errors in medication orders. Academic Emergency Medicine, 12(7), 629–634. https://doi.org/10.1197/j.aem.2005.02.009

Hoellein, A. R., Feddock, C. A., Griffith, C. H., Wilson, J. F., Barnett, D. R., Bass, P. F., & Caudill, T. S. (2004). Are continuity clinic patients less satisfied when the resident is postcall? Journal of General Internal Medicine, 19(5p2), 562–565. https://doi.org/10.1111/j.1525-1497.2004.30165.x

Jewett, M. E., Wyatt, J. K., Ritz-De Cecco, A., Khalsa, S. B., Dijk, D. J., & Czeisler, C. A. (1999). Time course of sleep inertia dissipation in human performance and alertness. Journal of Sleep Research, 8(1), 1–8. https://doi.org/10.1111/j.1365-2869.1999.00128.x

Keers, R. N., Williams, S. D., Cooke, J., & Ashcroft, D. M. (2013). Prevalence and nature of medication administration errors in health care settings: A Systematic review of direct observational evidence. Annals of Pharmacotherapy, 47(2), 237–256. https://doi.org/10.1345/aph.1R147

Landrigan, C. P., Rothschild, J. M., Cronin, J. W., Kaushal, R., Burdick, E., Katz, J. T., Lilly, C. M., Stone, P. H., Lockley, S. W., Bates, D. W., & Czeisler, C. A. (2004). Effect of reducing interns’ work hours on serious medical errors in intensive care units. New England Journal of Medicine, 351(18), 1838–1848. https://doi.org/10.1056/NEJMoa041406

Lee, J. C. S., Kee, X. L. J., Wiener-Ogilvie, S., Chern, B. S. M., & Chin, C. Y. (2019). What drives attrition amongst obstetrics and gynaecology residents in Singapore. The Asia Pacific Scholar, 4(1), 34–41. https://doi.org/10.29060/TAPS.2019-4-1/OA2005

Liu, C.-C., & Wissow, L. (2011). How post-call resident doctors perform, feel and are perceived in out-patient clinics: Post-call residents seeing outpatients. Medical Education, 45(7), 669–677. https://doi.org/10.1111/j.1365-2923.2010.03912.x

Lobo, A. S., Sandoval, Y., Burke, M. N., Sorajja, P., Mooney, M., Traverse, J., Henry, T. D., Chavez, I., Gössl, M., Lips, D. L., Bradley, S. M., Poulose, A., Wang, Y., & Brilakis, E. S. (2019). Sleep deprivation in cardiology: A multidisciplinary survey. The Journal of Invasive Cardiology, 31(6), 195–198.

Mustafa, M., Erokwu, N., Ebose, I., & Strohl, K. (2005). Sleep problems and the risk for sleep disorders in an outpatient veteran population. Sleep and Breathing, 9(2), 57–63. https://doi.org/10.1007/s11325-005-0016-z

Robison, W., Patel, S. K., Mehta, A., Senkowski, T., Allen, J., Shaw, E., & Senkowski, C. K. (2018). Can fatigue affect acquisition of new surgical skills? A prospective trial of pre- and post-call general surgery residents using the da Vinci surgical skills simulator. Surgical Endoscopy, 32(3), 1389–1396. https://doi.org/10.1007/s00464-017-5820-6

Rosenbaum, L., & Lamas, D. (2012). Residents’ Duty hours — Toward an empirical narrative. New England Journal of Medicine, 367(21), 2044–2049. https://doi.org/10.1056/NEJMsr1210160

Saadat, H., Bissonnette, B., Tumin, D., Thung, A., Rice, J., Barry, N., & Tobias, J. (2016). Time to talk about work-hour impact on anesthesiologists: The effects of sleep deprivation on Profile of Mood States and cognitive tasks. Pediatric Anesthesia, 26(1), 66–71. https://doi.org/10.1111/pan.12809

Samkoff, J. S., & Jacques, C. H. (1991). A review of studies concerning effects of sleep deprivation and fatigue on residents’ performance. Academic Medicine, 66(11), 687–693. https://doi.org/10.1097/00001888-199111000-00013

Taffinder, N., McManus, I., Gul, Y., Russell, R., & Darzi, A. (1998). Effect of sleep deprivation on surgeons’ dexterity on laparoscopy simulator. The Lancet, 352(9135), 1191. https://doi.org/10.1016/S0140-6736(98)00034-8

Tully, M. P., Ashcroft, D. M., Dornan, T., Lewis, P. J., Taylor, D., & Wass, V. (2009). The causes of and factors associated with prescribing errors in hospital inpatients: A Systematic Review. Drug Safety, 32(10), 819–836. https://doi.org/10.2165/11316560-000000000-00000

Yi, W. S., Hafiz, S., & Sava, J. A. (2013). Effects of night-float and 24-h call on resident psychomotor performance. Journal of Surgical Research, 184(1), 49–53. https://doi.org/10.1016/j.jss.2013.03.029

*Chin Chee Yang
National Heart Centre Singapore
5 Hospital Drive, Singapore 169609
Tel: +65 6704 8962
E-mail: chin.chee.yang@singhealth.com.sg