Original Articles Archives - Page 4 of 16 - The Asia Pacific Scholar

Defining undergraduate medical students’ physician identity: Learning from Indonesian experience

Submitted: 16 July 2023
Accepted: 21 December 2023
Published online: 2 April, TAPS 2024, 9(2), 18-27
https://doi.org/10.29060/TAPS.2024-9-2/OA3098

Natalia Puspadewi

Medical Education Unit, School of Medicine and Health Sciences, Atma Jaya Catholic University of Indonesia, Indonesia

Abstract

Introduction: Developing a professional identity involves understanding what it means to be a professional in a certain sociocultural context. Hence, defining the characteristics and/or attributes of a professional (ideal) physician is an important step in developing educational strategies that support professional identity formation. To date, there are still limited studies that explore undergraduate medical students’ professional identity. This study aimed to define the characteristics and/or attributes of an ideal physician from five first-year and three fourth-year undergraduate medical students.

Methods: Qualitative case studies were conducted with eight undergraduate medical students from a private Catholic medical school in Jakarta, Indonesia. The study findings were generated from participants’ in-depth interviews using in vivo coding and thematic analysis. Findings were triangulated with supporting evidence obtained from classroom observations and faculty interviews.

Results: First-year participants modeled their professional identities based on their memorable prior interactions with one or more physicians. They mainly cited humanistic attributes as a part of their professional identity. Fourth-year participants emphasised clinical competence and excellence as a major part of their professional identities, while maintaining humanistic and social responsibilities as supporting attributes. Several characteristics unique to Indonesian’s physician identity were ‘Pengayom’ and ‘Jiwa Sosial’.

Conclusion: Study participants defined their professional identities based on Indonesian societal perceptions of physicians, prior interactions with healthcare, and interactions with medical educators during formal and informal learning activities.

Keywords: Professional Identity Formation, Indonesia Undergraduate Medical Students, Physician Identity

Practice Highlights

Defining the attributes of ideal physicians is important for developing strategies that support PI.
Prior interactions with healthcare and formal/informal learning activities influence PI definition.

I. INTRODUCTION

Supporting the (trans)formation of a medical student’s identity, from a layperson to a professional, is an important process in preparing future physicians (Cruess et al., 2014; Goldie, 2012; Wald, 2015). This process includes professional identity formation (PIF) throughout their medical education continuum. Professional identity (PI) refers to how someone represents their profession’s characteristics, values, and attributes through thoughts, actions, and behaviors (Cruess et al., 2014; Gee, 2003; Luehmann, 2011). It is highly related to professionalism, which influences and shapes one’s identity in a professional context (Forouzadeh et al., 2018). The formation of PI involves developing one’s understanding of their professional roles, responsibilities, and expectations that are socio-culturally dependent (Siebert & Siebert, 2007). Therefore, the process of forming one’s PI also involves developing one’s cultural identity (Forouzadeh et al., 2018).

Studies on PI formation in medical education tend to focus on educational strategies that support PI formation during medical training (Adema et al., 2019; Ahmad et al., 2018; Cruess et al., 2015; Foster & Roberts, 2016). These studies provide insights on how to support PI formation without really addressing what needs to be taught to support medical students’ PI formation. Several theories on identity and PI formation suggest that one’s identity is formed through dialectical conversations that facilitate the acceptance, rejection, or modification of the profession’s characteristics and/or attributes into one’s core identity (Cruess et al., 2015; Gee, 2003; Siebert & Siebert, 2007; Stets & Burke, 2000). These characteristics and/or attributes are usually context-dependent (Cruess et al., 2014). Thus, defining and understanding what it means to be a professional physician in a certain socio-cultural context is as important as finding out how best to facilitate its formation in an educational setting (Wacquant, 2013).

Altruism and humanism are the two most cited values expected from a physician, along with integrity and accountability, honesty, and morality (Cruess et al., 2014; Edgar et al., 2020; Hall, 2021). Additionally, care providers, researchers, and teachers are some professional roles of physicians often mentioned in the literature (Ahmad et al., 2018; Branch & Frankel, 2016; Carlberg-Racich et al., 2018; Hatem & Halpin, 2019). Nevertheless, there might be other roles and characteristics that have yet to be fully elucidated, especially considering that the current literature on PI formation is mainly dominated by the Western representation of the medical profession.

This study aimed to describe the characteristics and/or attributes of ideal (professional) physicians in Indonesia as defined by undergraduate medical students. Undergraduate medical students are unique as they have limited opportunities to interact with real patients in a real workplace. Through this study, we hope to gain new insights from undergraduate medical students on what it means to be a professional physician.

II. METHODS

This was a qualitative phenomenology research using case studies design at a private Catholic medical school in Jakarta, Indonesia. Participants were recruited using a purposive sampling method. Transitional phases in one’s life are often associated with identity renegotiation as they are exposed to changes in their roles, responsibilities, and expectations (Kay et al., 2019). Therefore, we sought to explore how Indonesian undergraduate medical students defined their professional identity at the beginning (first-year) and end (fourth-year) of their preclinical years. Ethical clearance was obtained from the school’s Research Ethics Committee prior to the study.

We set a quota of 5 participants for each study year (with a total of 10 study participants) to account for any possible socioeconomic status, ethnicity, religion, and gender variations. We recruited five first-year and five fourth-year preclinical students at the beginning of the study; however, two of the fourth-year participants dropped out during data collection; hence, only eight case studies constructed to depict the characteristics and/or attributes of an ideal Indonesian physician.

Each case study participant was interviewed twice using semi-structured interviews. The first interview was conducted at the beginning of school semester (August 2021) and the follow up interview was conducted one month after. The purpose of the first interview was to determined participants’ current understanding and views of what it meant to be a physician, while the second interview aimed to determine if there were any changes in their understanding or views and what precipitated the changes. Interview questions include: What kind of physician do you aspire to be? Was there one or more specific moment that prompted you to become a physician (if so, please describe it)? What characteristics and/or attributes should an ideal physician possess? Please explain. At the follow-up interviews, participants were asked to re-describe the characteristics and/or attributes of physicians that they aspired to be and what prompted the changes. Furthermore, participants were also asked to describe any specific learning moments that might influence their understanding of what it means to be a professional physician. Because of the COVID-19 physical distancing policy during the data collection phase, all data were obtained virtually or through electronic exchange via secured online platforms. All interviews were transcribed verbatim and analysed in vivo using abductive thematic analysis with Atlas.Ti 8

In addition to from the interviews, we also conducted several classroom observations. We observed the first- and fourth-year’s large classroom lecture, problem-based learning, and skills laboratory session once, focusing on the teacher-student interactions and made note on how, if any, the faculty member facilitated students’ PI formation in the classroom. We also interviewed several faculty members who interacted with the participants in teaching capacity during the data collection phase. Faculty members were asked to describe what kind of physicians they wanted their students to be based on institutional values and their own beliefs about what constitutes an ideal physician. They were also asked to elaborate on their efforts to facilitate those characteristics and/or attributes in the formal and informal curriculum.

Data obtained from classroom observation and faculty interviews were used to triangulate the findings from the participants’ interviews. Permission was obtained from all related parties to record and use the interviews and classroom interactions in the data analysis. Individual case study reports were generated by combining the data obtained from interviews and field notes. These case study reports were then cross-analysed to find commonalities across the case studies to define the characteristics and/or attributes of an ideal physician that the participants aspired to be at their current stage of education.

III. RESULTS

The majority of participants were either Chinese or of Chinese descent. Five participants were Christian Protestants, one was a Buddhist, and two refused to disclose their ethnicity and religion. Note that the names used in these case studies are pseudonyms.

A. Case Study #1: Celine (First-year Student)

Celine, a female of Chinese-Betawi descent from West Java, was raised in a devout Christian-Protestant family. Being a physician was not her childhood aspiration. Initially, she thought physicians tended to be “rude, bossy, had too much pride, unwilling to listen to suggestions” (Celine, Interview 1, Line 42-43), which contradicted her personal values to being humble and helping others as a form of service and manifestation of her faith. Nevertheless, she developed a new appreciation toward physicians when she found out that there were physicians who gave back to the surrounding community by providing free healthcare (see Appendix No. 1).

Humility, and self-reflectiveness—which Celine called “openness to criticism” (Celine, Interview 1, Line 39-44) were the characteristics she deemed important as a physician. She believed that a physician should engage in social actions and put the patient first. Furthermore, a physician should consider the patient’s personal circumstances while providing individualised healthcare based on the patient’s needs. A good physician should also believe that their most important role is to provide credible health information and educate the community to improve their health and well-being. Good communication skills, including active listening, empathy, building trust, and the ability to break bad news, were essential in supporting this role (see Appendix No. 2).

B. Case Study #2: Dimitri (First-year Student)

Dimitri, a Christian-Protestant female of Chinese descent, was quite familiar with medicine and the medical profession as she was surrounded by people who either worked as or studied to become a physician. Additionally, she helped caring for her visually impaired sibling since she was young, which gave her opportunities to interact with various care providers as she accompanied her sibling for treatment. Being a physician naturally became her aspiration since childhood. Dimitri was appointed as a ‘Dokter Kecil’ (or, ‘little doctor’) in elementary school, assigned to provide first aid treatment to fellow students and promote health efforts conducted by the school. Before entering medical school, Dimitri’s grandfather fell critically ill; therefore, she helped her family to care for him in the hospital. There, she met a cardiologist whom she respected. She recalled that she appreciated the way this cardiologist relayed which information could be shared with her grandfather to keep his spirit up and which information should be disclosed to her family to prepare for the worst possible outcome. She mentioned that her grandfather looked “calm and comfortable” in his last days, which helped the family to accept his departure peacefully (Dimitri, Interview 1, Line 77-80).

Dimitri highlighted a physician’s ability to handle the distribution of information as an important part of her ideal physician identity (See Appendix No. 3). She believed that it was acceptable for a physician to keep certain information from the patient if that information could add unnecessary stress or cause them to stop following the treatment (Dimitri, Interview 1, Line 90-98). Regardless, the physician should disclose all information to the patient’s relatives as the patient’s decision-maker. Dimitri aspired to be a caring and compassionate physician with good communication skills who can be held accountable for her actions. Aside from being a care provider, Dimitri believed that a physician should take on a role as ‘Pengayom’ (protector). She believed that patients were in vulnerable positions due to their health issues, and therefore the physician was responsible for protecting them like a parent would when their child was sick. Implied in the Pengayom role was the leader whose responsibility was to make the best decision for the patient’s health and well-being (See Appendix No. 4).

C. Case Study #3: Faustine (First-year Student)

Faustine, a Christian-Protestant female of Chinese descent, was born and raised in a remote area in Riau province, in the southern part of Sumatra Island. Her interest in biology and life sciences prompted her to browse online videos related to healthcare since she was young. She tended to feel sad if the people closest to her were suffering and she could not do anything to help. She made up her mind to study medicine when one of her high school friends was forced to seek treatment abroad because of limited healthcare access in her region. Prior to this, her father was misdiagnosed with a malignant tumor, which caused tremendous distress for her family. These incidents drove her to be a physician who could provide good quality care, especially to those closest to her (See Appendix No. 5).

Faustine aspired to be an empathetic physician, taking patients’ mental or psychological state into consideration when planning for their treatment. She did not want to be a physician who focused on financial gain at the cost of the patient’s wellbeing. Being aware of her limitations in providing care and continuously updating her knowledge and skills were characteristics she hoped to develop once she became a physician (Faustine, Interview 1, Line 103-115). Faustine also mentioned that a physician was responsible for being a reliable source of information and improving community wellbeing through education (See Appendix No. 6).

D. Case Study #4: Jasmine (First-year Student)

Jasmine originated from Rembang, a small regency on the northeast coast of Central Java. Being a physician had always been her childhood aspiration because she loved helping people and interacting with others. Jasmine tended to her grandmother’s health needs during middle school. This event confirmed her passion and desire to serve others. Putting others’ needs above herself was a value instilled by her father since she was young. She wanted to be a physician who focused on social services, and was driven to help others sincerely without expecting anything in return.

As Jasmine mentioned, an ideal physician should be honest, disciplined, possess high ‘Jiwa Sosial’ (an attitude that shows concern to perform actions that are beneficial for humanity and social community), and always put the patient’s needs first (Jasmine, Interview 1, Line 50-53). Jasmine viewed her work as an extension of her faith, and she wanted to reflect Christian values, particularly the value of servitude, in her professional life (See Appendix No. 7-8).

E. Case Study #5: Rose (First-year Student)

Rose, a Christian-Protestant female of Chinese descent, was born and raised in Ambon city, Maluku province, Eastern Indonesia. She was the oldest child in her family. Rose became interested in medicine when her mother was diagnosed with a serious illness and could not receive appropriate treatment. She disclosed that her mother ignored the early signs and symptoms of her illness until her condition became so severe that she could not be treated fully. From this experience, Rose was motivated to become a physician so that she could take better care of her family (See Appendix No. 9).

Growing up, Rose heard several stories in which a patient did not receive appropriate healthcare due to their socioeconomic status. She aspired to be a competent and non-discriminative physician. Putting the patient’s needs first, being responsible, helpful, patient, disciplined, and continuously improving her knowledge and skills were the characteristics that she hoped to develop by the time she became a physician. Aside from being a care provider, Rose believed that a physician was responsible for improving the wellbeing of the community through education (See Appendix No. 10).

F. Case Study #6: *Anton (Fourth-year Student)

*Anton, a Christian-Protestant male of Chinese-descent, had an interest in biology since childhood. He was dissatisfied with Indonesian healthcare services, particularly with the healthcare workers’ communication skills when treating his father. This incident occurred when he was in middle school. *Anton observed a power imbalance between the patients and physicians, where the healthcare providers held more power over their patients. As a patient, he felt disadvantaged because he could not demand a better quality of care nor asked for a lower cost of the care he received (See Appendix No. 11). He described the two roles of physicians: as a healthcare provider and educator. As a healthcare provider, one should be able to help patients understand what is best for them while still respecting their autonomy. As educators, physicians have the responsibility to provide valid evidence-based information for patients.

For *Anton, an ideal physician’s fundamental values and skills included providing good quality care that kept the patients’ best interest, respecting patients’ autonomy, doing no harm, having all necessary medical competencies as listed in the Competence Standards of Indonesian Physician, the drive to learn for a lifetime, patience, humility, competence, and the ability to engage in interprofessional collaboration (See Appendix No. 12).

G. Case Study #7: *R (Fourth-year Student)

*R is a Chinese Buddhist female from Sintang, central Indonesia. *R wanted to pursue medicine because physician was portrayed as a noble profession in Indonesia and as a ‘role model’ in her family. She wanted to serve marginalised areas in East Indonesia after hearing about the poor health situation in those areas from several alumni and fellow students who served there in various capacities. This experience, along with her formal learning experiences, shaped her ideal physician image, which included being detail-oriented, confident, honest, thorough, and caring. She believed that physicians should be able to fulfill the roles and responsibilities of a healthcare provider, which required good proficiency in medical competencies, based on several fundamental values such as honesty, willingness to serve marginalised and under-served communities, and being sensitive to patients’ needs (See Appendix No. 13).

H. Case Study #8: *Anastasia (Fourth-year Student)

*Anastasia, who identified as a female, wanted to be a physician since elementary school. She did not have a specific motivation to enter a medical school when she first started. Nevertheless, there were several past experiences that she claimed to have influenced her image of ideal physicians. She mentioned feeling comfortable being examined by her pediatrician during her childhood. This made her consider the pediatrician as her role model. She also followed several healthcare professionals’ whom she admired on their social media accounts. She claimed that these figures influenced her to be selfless and put the patients’ needs above her own. She acknowledged the importance of entrepreneurial skills in aiding her goal of being selfless yet still able to make a living for herself. Her ideal physician image is someone who has good communication skills, clinical competence, and willingness to learn continuously. She identified healthcare provider as the essential role of a physician, who was responsible for providing physical and mental healthcare, as well as participating in preventive and promotive healthcare. She particularly considered female medical teachers at her school as her role models because she admired the way these figures divide their time and energy to work professionally–both as healthcare practitioners and teachers–and keeping up with their personal and family time. She aspired to be someone who could divide her focus like these figures once she graduated (See Appendix No. 14).

IV. CROSS-CASE ANALYSIS AND DISCUSSION

Cross-case analysis revealed four major attributes of physician identity as defined by the first- and fourth-year participants (indicated by * behind their pseudonyms), including characteristics, values, roles and responsibilities, and skills. First-year participants drew their ideal image of a physician based on their interactions with one or more healthcare provider whom they met in their earlier lives. These interactions left a significant impression that further strengthened their motivation to study medicine and influenced the kind of values or other things that they held important and were willing to stand for as future physicians.

First-year participants mainly mentioned humanistic and altruistic values as the characteristics and/or attributes that define their professional identity. Honesty, humbleness/humility, accountability, patience, jiwa social, prioritising patients’ needs, empathy, care, and compassion are some of the characteristics mentioned by the first-year participants as characteristics of an ideal physician. These characteristics correspond to society’s expectations of professional physicians to put patient’s interest above all else, which is then further translated into medical professionalism and professional responsibilities (Alrumayyan et al., 2017; Elaine Saraiva Feitosa et al., 2019).

Different from their counterparts, fourth-year participants focused on clinical excellence and competence when citing the ideal characteristics and/or attributes of an ideal physician based on the national Competence Standards for Indonesian Physician. This indicates that fourth-year participants were aware of the standards as well as the ethical principles and physician’s code of conduct that were being enforced in Indonesia (See Appendix 15-16).

The way fourth-year participants described their physician identity aligned with the image of a professional physician painted by the school’s teaching faculty. According to interviews with several key faculty members, meeting the minimal standard of competence, being aware of one’s limitations, practicing evidence-based medicine, honesty, and discipline were some of the fundamental physician attributes/values/characteristics that they tried to instill in their students during education. These institutional values were most notably found in the way first-year participants described their physician identity during their second interview (See Appendix No. 17-18).

The attributes of Indonesian physicians mentioned by all case studies participants closely resemble China’s framework of professionalism, where they emphasise altruism, integrity and accountability, excellence, and religion/moral values (Al-Rumayyan et al., 2017). Possessing jiwa sosial (inherent sense of social responsibility, empathy, and engagement) and being a pengayom (mentor/guardian/protector) are two unique attributes that represent the Indonesian ideal physician.

There were minimal overlaps between the first- and fourth-year participants’ ideal physician images. First-year participants placed humanism/altruism and social responsibility as the focal points of their physician identity, whereas fourth-year participants chose clinical excellence and competence to represent their physician identities. Social interactions play a major role in identity formation (Thomas et al., 2016). This may explain the shift in the first- and fourth-year participants’ definition of an ideal physician. First-year participants modeled their ideal physician identity after their memorable interactions with physicians who provided care for them or their family members. Positive past interactions with healthcare providers shaped the characteristics and/or attributes that participants aspired to be, whereas negative past interactions motivated them to develop the opposite of observed characteristics and/or attributes. Fourth-year participants also integrated the characteristics and/or attributes they identified from the formal and informal learning experiences with their evolving understanding of an ideal physician. In these case studies, fourth-year participants cited clinical competencies and excellence, as well as discipline and honesty—which were emphasised by the teachers during their undergraduate medical training—as the major characteristics and/or attributes that defined their physician identity.

Figure 1. Shift in First-Year and Fourth-Year Participants’ Definition of Physician Identity

The first year of the medical curriculum was indicated to be an important transition point that shaped all participants’ PI. In particular, all participants mentioned the school orientation as one of the learning moments that triggered their identity negotiation. Participants were introduced to the school’s expectations of them as medical students and future physicians. These expectations include the characteristics of self-regulated and life-long learners and those of professional physicians (See Appendix No. 19-20). For example, Jasmine “learned to be disciplined and responsible and she believed that the school orientation “helped shape [her] basic personality as a physician [who needs] to be disciplined and responsible [as well as] trustworthy.” (Jasmine, Interview 1, Line 115-118).

The shifts in participants’ physician identity definition indicated that participants engaged in a dialectical conversation that stimulated them to merge their core or personal identity with the institution’s perception of ideal physicians (“virtual/ideal identity) as interpreted in their curriculum, which was a part of one’s identity negotiation process (Gee, 2003). In the cross-case analysis, we found that participants’ reactions toward the values, characteristics, and attributes instilled by the faculty varied. For example, some participants saw the importance of being on time (‘discipline’) as well as being academically honest by avoiding plagiarism and cheating during exams (‘honesty’), which they accepted as a part of their physician identity. On the other hand, other participants struggled to understand the relevance of being on time and academically honest with their future physician roles or aspirations. This became a major challenge for these participants in incorporating those values into their physician identity. Nevertheless, no participants rejected any characteristics/attributes instilled by the institution even if those characteristics/attributes were distinctly different from their personal beliefs system (See Appendix No. 21-23).

Any new or contradictory characteristics or attributes to one’s core identity pose a professional dilemma that triggers an identity negotiation (Spencer et al., 1997). During this identity negotiation process, the study participants tried to merge their core identity, which was represented by their definition of the ideal physician that they aspired to be, either by accepting, rejecting, or integrating the new characteristics/attributes into their core identity (Cruess et al., 2015).

The acceptance of new characteristics/attributes into one’s physician identity will be easier if it is consistent with one’s core identity; however, it is still possible to instill characteristics/attributes that contradict one’s core identity if they are provided with the long-term benefit of accepting those characteristics/attributes (Guillemot et al., 2022). This underlined the importance of providing students with the relevancy of developing certain characteristics/attributes desired from a professional physician during their educational phase to support their PIF.

V. CONCLUSION

This case study found that first-year participants prioritised humanistic characteristics as the foreground of their professional identity, and medical professionalism as their background. Meanwhile, fourth-year participants developed a projected identity that embodied the general values of the medical profession and those promoted by their institution. The perceived image of ideal physicians as constructed by the Indonesian society’s ideal image of a physician, prior interactions with Indonesian physicians that influenced their decisions to study medicine, and interactions with the medical teachers during formal and informal learning activities influenced the way participants defined their professional identity.

Notes on Contributors

Natalia Puspadewi contributed to the work’s conception and design by developing the study proposal, protocols and instruments, data collection, analysis, and interpretation. Further, Natalia also drafted and revised the manuscript and ensured that all aspects of the work were accountable, and followed all procedures to ensure data security and anonymity.

Ethical Approval

This study was a part of a doctoral dissertation. The University of Rochester acted as the author’s host institution, and Atma Jaya Catholic University of Indonesia, School of Medicine and Health Sciences, was the research site. Ethical approval was provided by the University of Rochester RSRB (a letter of exempt determination was obtained on July 8^th, 2021 for Study ID 00006273) and the Atma Jaya Catholic University of Indonesia, School of Medicine and Health Sciences Ethics Committee (ethical clearance certificate No. 08/07/KEP-FKUAJ/2021).

Data Availability

The data supporting the findings of this study are openly available in the Figshare repository

https://doi.org/10.6084/m9.figshare.23684235. The data were not translated into English to preserve the Indonesian sociocultural nuances captured in the interviews. All data were coded and analysed in vivo in Bahasa Indonesia before being translated into English for presentation in this manuscript.

Acknowledgement

We would like to express our gratitude to those who have contributed to this study and article development: Dr. Rafaella Borasi as the head of the dissertation committee and advisor, Dr. Sarah Peyre as dissertation committee member, and Gracia Amanta, MD and Cristopher David, MD who helped with manuscript organisation and layouts.

Funding

This study was funded by the Atma Jaya Catholic University of Indonesia and American Indonesian Cultural and Education Foundation.

Declaration of Interest

The author has no conflicts of interest to disclose.

References

Adema, M., Dolmans, D., Raat, J. a. N., Scheele, F., Jaarsma, D., & Helmich, E. (2019). Social interactions of clerks: The role of engagement, imagination, and alignment as sources for professional identity formation. Academic Medicine, 94(10), 1567. https://doi.org/10.1097/ACM.0000000000002781

Ahmad, A., Yusoff, M. S. B., Mohammad, W. M. Z. W., & Nor, M. Z. M. (2018). Nurturing professional identity through a community based education program: Medical students experience. Journal of Taibah University Medical Sciences, 13(2), 113–122. https://doi.org/10.1016/j.jtumed.2017.12.001

Alrumayyan, A., Van Mook, W. N. K. A., Magzoub, M., Al-Eraky, M. M., Ferwana, M., Khan, M. A., & Dolmans, D. (2017). Medical professionalism frameworks across non-Western cultures: A narrative overview. Medical Teacher, 39(sup1), S8–S14. https://doi.org/10.1080/0142159X.2016.1254740

Branch, W. T., & Frankel, R. M. (2016). Not all stories of professional identity formation are equal: An analysis of formation narratives of highly humanistic physicians. Patient Education and Counseling, 99(8), 1394–1399. https://doi.org/10.1016/j.pec.2016 .03.018

Carlberg-Racich, S., Wagner, C. M. J., Alabduljabbar, S. A., Rivero, R., Hasnain, M., Sherer, R., & Linsk, N. L. (2018). Professional identity formation in HIV care: Development of clinician scholars in a longitudinal, mentored training program. Journal of Continuing Education in the Health Professions, 38(3), 158–164. https://doi.org/10.1097/CEH.0000000000000214

Cruess, R. L., Cruess, S. R., Boudreau, J. D., Snell, L., & Steinert, Y. (2014). Reframing medical education to support professional identity formation. Academic Medicine, 89(11), 1446-1451. https://doi.org/10.1097/ACM.0000000000000427

Cruess, R. L., Cruess, S. R., Boudreau, J. D., Snell, L., & Steinert, Y. (2015). A schematic representation of the professional identity formation and socialisation of medical students and residents: A guide for medical educators. Academic Medicine, 90(6), 718-725. https://doi.org/10.1097/ACM.0000000000000700

Edgar, L., McLean, S., Hogan, S. O., Hamstra, S., & Holmboe, E. S. (2020). The milestones guidebook. Accreditation Council for Graduate Medical Education (ACGME).

Feitosa, E. S., Brilhante, A. V. M., De Melo Cunha, S., Sá, R. B., Nunes, R. R., Carneiro, M. A., De Sousa Araújo Santos, Z. M., & Catrib, A. M. F. (2019). Professionalism in the training of medical specialists: An integrative literature review. Revista Brasileira De Educação Médica, 43(1), 692–699. https://doi.org/10.1590/1981-5271v43suplemento1-20190143.ING

Forouzadeh, M., Kiani, M., & Bazmi, S. (2018). Professionalism and its role in the formation of medical professional identity. Medical Journal of the Islamic Republic of Iran, 32(1), 765–768. https://doi.org/10.14196/mjiri.32.130

Foster, K., & Roberts, C. (2016). The heroic and the villainous: A qualitative study characterising the role models that shaped senior doctors’ professional identity. BMC Medical Education, 16(1), 206. https://doi.org/10.1186/s12909-016-0731-0

Gee, J. P. (2003). What video games have to teach us about learning and literacy. Computers in Entertainment, 1(1), 20. https://doi.org/10.1145/950566.950595

Goldie, J. (2012). The formation of professional identity in medical students: Considerations for educators. Medical Teacher, 34(9), e641–e648. https://doi.org/10.3109/0142159X.2012.687476

Guillemot, S., Dyen, M., & Tamaro, A. (2022). Vital service captivity: Coping strategies and identity negotiation. Journal of Service Research, 25(1), 66–85. https://doi.org/10.1177/10946705 211044838

Hall, C. (2021). Professional identity in nursing. In R. Ellis & E. Hogard (Eds.), Professional identity in the caring professions: Meaning, measurement and mastery. Routledge. https://www.taylorfrancis.com/books/9781003025610

Hatem, D. S., & Halpin, T. (2019). Becoming doctors: Examining student narratives to understand the process of professional identity formation within a learning community. Journal of Medical Education and Curricular Development, 6. https://doi.org/10.1177/2382120519834546

Kay, D., Berry, A., & Coles, N. A. (2019). What experiences in medical school trigger professional identity development? Teaching and Learning in Medicine, 31(1), 17–25. https://doi.org/10.1080/10401334.2018.1444487

Luehmann, A. (2011). The lens through which we looked. In A. Luehmann & R. Borasi (Eds.), Blogging as change: Transforming science & math education through new media literacies. Peter Lang Inc.

Siebert, D. C., & Siebert, C. F. (2007). Help seeking among helping professionals: A role identity perspective. American Journal of Orthopsychiatry, 77(1), 49–55. https://doi.org/10.1037/0002-9432 .77.1.49

Spencer, M. B., Dupree, D., & Hartmann, T. (1997). A Phenomenological Variant of Ecological Systems Theory (PVEST): A self-organisation perspective in context. Development and Psychopathology, 9(4), 817–833. https://doi.org/10.1017/S0954579497001454

Stets, J. E., & Burke, P. J. (2000). Identity theory and social identity theory. Social Psychology Quarterly, 63(3), 224–237. https://doi.org/10.2307/2695870

Thomas, E. F., McGarty, C., & Mavor, K. (2016). Group interaction as the crucible of social identity formation: A glimpse at the foundations of social identities for collective action. Group Processes & Intergroup Relations, 19(2), 137–151. https://doi.org/10.1177/1368430215612217

Wacquant, L. (2013). Homines in extremis: What fighting scholars teach us about habitus. Body & Society, 20(2), 3-17. https://doi.org/10.1177/1357034X13501348

Wald, H. S. (2015). Professional identity (trans)formation in medical education: Reflection, relationship, resilience. Academic Medicine, 90(6), 701. https://doi.org/10.1097/ACM.00000000000 00731

*Natalia Puspadewi
School of Medicine and Health Sciences,
Atma Jaya Catholic University of Indonesia,
Jl. Pluit Selatan Raya No. 19, Penjaringan,
Jakarta Utara, 14440
Email: natalia.puspadewi@atmajaya.ac.id

Submitted: 1 May 2023
Accepted: 21 December 2023
Published online: 2 April, TAPS 2024, 9(2), 5-17
https://doi.org/10.29060/TAPS.2024-9-2/OA3053

WCD Karunaratne¹, Madawa Chandratilake², Kosala Marambe³

¹Centre for Medical Education, School of Medicine, University of Dundee, United Kingdom; ²Department of Medical Education, Faculty of Medicine, University of Kelaniya, Sri Lanka; ³Department of Medical Education, University of Peradeniya, Sri Lanka

Abstract

Introduction: The literature confirms the challenges of learning clinical reasoning experienced by junior doctors during their transition into the workplace. This study was conducted to explore junior doctors’ experiences of clinical reasoning development and recognise the necessary adjustments required to improve the development of clinical reasoning skills.

Methods: A hermeneutic phenomenological study was conducted using multiple methods of data collection, including semi-structured and narrative interviews (n=18) and post-consultation discussions (n=48). All interviews and post-consultation discussions were analysed to generate themes and identify patterns and associations to explain the dataset.

Results: During the transition, junior doctors’ approach to clinical reasoning changed from a ‘disease-oriented’ to a ‘practice-oriented’ approach, giving rise to the ‘Practice-oriented clinical skills development framework’ helpful in developing clinical reasoning skills. The freedom to reason within a supportive work environment, the trainees’ emotional commitment to patient care, and their early integration into the healthcare team were identified as particularly supportive. The service-oriented nature of the internship, the interrupted supervisory relationships, and early exposure to acute care settings posed challenges for learning clinical reasoning. These findings highlighted the clinical teachers’ role, possible teaching strategies, and the specific changes required at the system level to develop clinical reasoning skills among junior doctors.

Conclusion: The ‘Practice-oriented clinical skills development framework’ is a valuable reference point for clinical teachers to facilitate the development of clinical reasoning skills among junior doctors. In addition, this research has provided insights into the responsibilities of clinical teachers, teaching strategies, and the system-related changes that may be necessary to facilitate this process.

Keywords: Clinical Reasoning, Medical Decision Making, Medical Graduates, Junior Doctor Transition, Hermeneutic Phenomenology, Qualitative Research

Practice Highlights

A safe environment and early healthcare team integration facilitate learning clinical reasoning.
Adopting a comprehensive approach to reasoning can overcome specialty-specific reasoning challenges.
Trainees’ emotional commitment toward patients could help them learn clinical reasoning skills.
Interrupted supervisory relationships and early acute care exposure can hamper learning reasoning.
Ensuring junior doctor training is both service and learning oriented is of paramount importance.

I. INTRODUCTION

Clinical reasoning is composed of cognitive processes, metacognitive processors, and behaviour during the application of critical thinking to a clinical situation and is heavily influenced by numerous contextual factors related to the doctor, patient, and the clinical environment (Durning et al., 2011; Durning et al., 2013; Norman, 2005).

The clinical reasoning of learners evolves along the continuum of medical education with unique challenges associated with major transition phases, the progression from non-clinical to clinical stage, medical graduate to junior doctor, and specialist trainee to medical specialist (Teunissen & Westerman, 2011). Notably, the medical graduate to junior doctor transition presents more pronounced difficulties (Brennan et al., 2010), primarily due to changing roles and responsibilities towards patient care, limited experience in navigating clinical uncertainties, and the need to work within multi-professional teams with limited support. Consequently, these factors have contributed to a steep learning curve for developing clinical reasoning skills (Brennan et al., 2010; Lempp et al., 2005; Prince et al., 2004; Tallentire et al., 2017). The challenges in developing reasoning skills are associated with the reduced applicability of undergraduate training in clinical practice (Cave et al., 2009; Monrouxe et al., 2017), coordinating and organising clinical and administrative responsibilities (Cameron et al., 2014; Teunissen & Westerman, 2011), and dealing with diverse contextual factors in practice. These factors encompass navigating hierarchical relationships and meeting the expectations of seniors, difficulties in recognising disease severity, uncertainty regarding their role, and tension in interpersonal relationships with team members (Cameron et al., 2014; Tallentire et al., 2011, 2017). When these challenges are not resolved, they could boil down to deficits in clinical reasoning and diagnostic error leading to adverse patient outcomes (Graber et al., 2005; Huckman & Barro, 2005; Jen et al., 2009).

The challenging nature of the junior doctor transition is shared across many similar contexts globally (Prince et al., 2000; Teunissen & Westerman, 2011) calling for a coherent approach to facilitate learning clinical reasoning. Concerns around clinical reasoning deficits of doctors continue to soar even today in resourceful developed countries (Health Services Safety Investigation Body, 2022; Huckman & Barro, 2005; Jen et al., 2009), emphasising the need for faculty to take decisive actions to resolve it! Unless for the limited research on clinical reasoning outside the western region (Lee et al., 2021), the situation could have been the same elsewhere.

There is ample evidence of numerous factors that may improve the development of clinical reasoning skills. Accordingly, work experience (Ericsson, 2004; Norman, 2005; Norman et al., 2007), a strong foundation on basic biomedical concepts (Woods, 2007), reflective practice (Mamede et al., 2008, 2012), feedback (Hattie & Timperley, 2007), learning from others during practice, and conducive organisational context for learning (Goldacre et al., 2003; Hattie & Timperley, 2007; Lempp et al., 2005) are found to be central in learning clinical reasoning. This evidence, however, is not specific to junior doctors. The learning needs of junior doctors in transition may vary from other trainee doctors and other health professions staff. Therefore, it has become critical that the clinical reasoning experiences, challenges, and practices of junior doctors as a vulnerable group of trainees are understood well to be able to better support their development of clinical reasoning.

When exploring this period of transition, the five-stage model of adult skill acquisition from novice to expert (Dreyfus, 2004), can help understand how junior doctors progress in relation to these stages. The situated learning theory (S. J. Durning & Artino, 2011; Lave, 1991) can provide the basis for understanding the social nature of learning clinical reasoning. The influence of contextual factors on mediating internal motivation for learning clinical reasoning can be understood through the self-determination theory (Ryan & Deci, 2000; Taylor & Hamdy, 2013). Therefore, to gain a better understanding of the transition experiences from medical graduates to junior doctors, a longitudinal study was designed using the above theoretical models as the conceptual framework to explore the following research questions:

(1) How do junior doctors evaluate their learning experiences of clinical reasoning development?

(2) What adjustments in the application of different educational means into the learning environment are necessary to improve the development of clinical reasoning skills?

II. METHODS

A. Methodology

The methodological approach of hermeneutic phenomenology (Crotty, 1998; Laverty, 2003) was employed in this study (Kafle, 2011; Laverty, 2003). Such an approach to clinical reasoning was adopted by other researchers exploring clinical reasoning (Ajjawi & Higgs, 2007; Langridge et al., 2015; Robertson, 2012).

B. Study Setting

The study was conducted at the North Colombo Teaching Hospital, Ragama, Sri Lanka with ethical clearance (P/11/01/16) from the Faculty of Medicine, University of Kelaniya.

In Sri Lanka, medical undergraduate training is a five-year programme with two pre-clinical and three clinical years. After graduation, medical graduates follow a 12-month internship where they work under a consultant for six months each in any of the two main clinical specialities, namely, Medicine, Surgery, Paediatrics, and Gynaecology & Obstetrics before obtaining full registration as a medical doctor.

C. Study Design and Sampling

The study participants were junior doctors during the 12 months of internship following graduation. Maximum variation sampling (Cohen et al., 2017), which enabled purposefully selecting the widest range of variation on dimensions of interest relevant to learning and practicing clinical reasoning was employed. The concept of ‘information power’ which sought not theoretical saturation but sufficient information to address the research questions informed the sample size (Malterud et al., 2016; Varpio et al., 2017). Hence, junior doctors working in the four main clinical specialties, in both university clinical wards staffed by university clinical academics and other clinical wards composed of medical consultants under the Ministry of Health and according to gender were enrolled in the study following informed consent.

Accordingly, eighteen junior doctors (n=18, males=8, females=10) from the four main clinical specialities (Medicine-4, Surgery-5, Paediatrics-4, Obstetrics and Gynaecology-5) were enrolled in the first stage of the study. The second stage of the study imposed heavy demands on the study participants because it involved recording multiple doctor-patient encounters and subsequent discussions based on stimulated recall. Therefore, out of the initially recruited participants, only the well-articulated consenting participants (n=8), who could proficiently express their thoughts and reasoning to obtain a good insight into the nature of practicing clinical reasoning were enrolled in this stage.

D. Data Collection

The data collection proceeded in two stages.

During the first stage, a combination of individual semi-structured interviews with narrative interviews were conducted. Semi-structured interviews allowed probing where necessary (Cohen et al., 2017), while the narratives allowed participants to tell their stories of clinical reasoning (Muylaert et al., 2014). Each lasted for 45-50 minutes.

The second stage included audio-recording the patient consultations of the selected participants on predefined dates during the first and second six months of their internship. The consultations were replayed, and post-consultation discussions were conducted soon afterward by employing a stimulated recall method, to account for a total of 48 post-consultation discussions. As clinical reasoning is a concept revealed only in action (Charlin et al., 2000), employing such an approach was considered essential during this study.

E. Data Analysis

All interviews and discussions were transcribed verbatim. The data analysis followed phenomenological and hermeneutic strategies, which required a thorough description of lived experiences (Ajjawi & Higgs, 2007) and employing a hermeneutic circle for data interpretation by moving back and forth between the parts and the whole of the experience to reach a deeper understanding of the experience (Laverty, 2003).

Thematic data analysis (Braun & Clarke, 2012) was conducted to generate themes explaining the data set as a whole.

The principal researcher developed two thematic frameworks for the two stages of the study. The two supervisors of the project re-coded selected transcripts from each stage. These independently derived frameworks were discussed, themes refined, and new themes identified until an agreement was reached. The finalised thematic framework was employed to code all the transcripts using the Atlas.ti qualitative data analysis tool.

III. RESULTS

A total of 18 individual interviews and 48 post-consultation discussions were analysed giving rise to seven themes. During analysis, it was noted that the factors that inform the development of clinical reasoning could be condensed together as a model. This is presented later in the text.

Each theme is elaborated below with quotations. When more than one quotation is required to describe a theme, these are presented within a table. Additional supportive quotations are openly available in Figshare at https://doi.org/10.6084/m9.figshare.23536548.v2 (Karunaratne et al., 2023).

A. A Safe and Supportive Working Environment Empowers Junior Doctors to Develop Clinical Reasoning Skills

It was the collective view that a ‘safe’ work environment is characterised by easy access to more experienced doctors, and the presence of a safety net of seniors who review junior doctors’ work and understand their reasoning challenges. It provided junior doctors the opportunity and freedom to practice clinical reasoning independently, learn from errors, and arrive at their own reasoning decisions.

Such a conducive work environment also provided them with opportunities to emulate seniors and receive real-time feedback while actively participating in authentic tasks and applying knowledge and skills acquired during their undergraduate training.

“I’m working in a unit where each admission is clerked by the registrar. So, in that case, we are always in feedback…What I usually do is sometimes I clerk the patients first, and after that, I compare it with the registrar’s clerking. So, in that case, we can easily adapt their clerking.”

(MP3, Medicine, Male, Phase-1)

B. Learning to Reason with Clinical Problems is Situated and Facilitated by Work Experience

Work experience provided the opportunity to learn from repeated exposure to clinical presentations and their variations, learn from seniors, and lapses of reasoning. However, work experience alone is not solely sufficient, and it is the collective influence of many other factors that help learn clinical reasoning. These factors are captured by the model developed from this study.

With work experience, junior doctors’ approach to reasoning changed from a ‘disease-oriented approach’ developed through undergraduate education to a ‘practice-oriented approach’. In the practice-oriented approach, junior doctors actively analyse clinical problems instead of matching them with memorised configurations of disease presentations.

They also developed ‘instincts’ for swift decision-making, sharpened through experience in recognising contextual factors in patient presentations. This was especially valuable for identifying acute cases requiring urgent care. In addition, they recognised the impact of the previous disease burden in formulating differential diagnoses, leading to a broader approach in their clinical reasoning.

Table 1 illustrates participant quotations that shed light on the role of work experience in learning clinical reasoning skills.

“…This approach in the ward is always problem-based. We’re dealing with problems. We try to solve the problems. That approach as a student was trying to fit the history into one of the long cases we have studied…Now we are not worried about that broad category. We will instead deal with the different problems that they have.”

(MP2, Medicine, Male, Phase-1)

“I think it’s just being with the patients. You realise that … it’s not just what’s written in the book…I mean now, if you’re just walking past a patient, you realise that this patient is not well. Whereas initially, you would have to go through the ward round and… go through the records, and then only you’ll see it. I don’t know how you get that but…”

(MP2, Medicine, Male, Phase-1)

“…Once a child with hypovolemic shock came to the ward. I was in the ward alone. I was very afraid at that time as I was in my first week of internship. So, nothing was on my mind, and I called my senior and he asked me to give (fluid) boluses until he came…. (There was another emergency at the same time). An Angioedema child came to the ward. I thought of (laughing)… running away from the ward. Because it was the initial period, it was very difficult, and our clinical knowledge was also poor. But now, we can manage any emergencies until the senior comes.”

(PP, Paediatrics, Female, Phase-1)

(When enquired on the reasons for commencing consultations with comorbidities?)

“… Even the presenting complaint may be related to past medical conditions as well…and even this patient has diabetes… so, they can present in various ways… As an intern, I developed that. As an undergraduate, we are asking for name, age, where are you from, and then go on to take the history first…”

(MP4, Medicine, Female, Phase-2)

Table 1. Quotes illuminating that learning clinical reasoning is situated and facilitated by work experience

C. Internal Motivation and the Ability to Reflect and Employ Self-directed Learning are Powerful Tools for Developing Clinical Reasoning Skills

Learning clinical reasoning necessitated junior doctors to be internally driven for learning. Such internal motivation made them willing to learn from any source and be self-directed in their own learning. These individuals progressed rapidly in learning to reason with clinical problems compared to others who were not internally motivated.

Maintenance of internal motivation throughout the internship necessitated external encouragement even for the motivated particularly from the senior staff. There was a similar effect when the work environment fostered a culture of learning with the inclusion and recognition of junior doctors as a group of learners.

Table 2 presents participant quotations that highlight the significance of internal motivation in developing clinical reasoning skills.

“(reasoning with a complicated presentation) …With this kind of patient, it’ll refresh our memory. Going through how to take the history, how to use the basics, and how you investigate and manage…It is not like people coming with gastritis, or headache. Those are just simple things.
But actually, it’s a pleasure to have these kinds of patients.”

(MP3, Medicine, Male, Phase-2)

“I think you don’t need people who are good at what they do, I mean, you need people who are competent, but er…, you need a pleasant environment. Even if, there are, like 50 patients, if the people you work with are good, you can go through it. But then, if someone is really unpleasant, then that day is ruined.”

(MP1, Medicine, Female, Phase-1)

Table 2. Quotes illuminating internal motivation, reflective practice, and being self-directed as central to learning clinical reasoning skills

D. Caring and Compassionate Attitudes towards Patients Facilitate Developing Clinical Reasoning Skills

The individual caring and compassionate attitudes towards patients and the positive role modeling of senior doctors motivated junior doctors to learn clinical reasoning. Work experience nurtured these attitudes irrespective of gender, reflecting the potential to learn them during practice. However, a heavy workload and orientation towards efficiency in practice hindered the development of such attitudes among junior doctors.

“We’ve realised that although we’re members of a team, even individually, we can always do something for the patients. So, we always try to do something at our level. But we’re always willing to take the feeling from everyone above us to help.”

(MP1, Medicine, Female, Phase-1)

E. Collaborating within a Healthcare Team and Engaging in Ward Activities and Procedures Help Expedite the Development of Clinical Reasoning Skills

Junior doctors learn mostly from registrars, who are the immediate seniors and near-peers. In addition, peers and other healthcare staff contribute to their learning by timely sharing of information and working as a team. Patients’ unique characteristics which demand variation in reasoning also provide learning opportunities.

“I think the main influence is probably the registrars. Because we’re mostly in contact with them…So, in a way through working with them, I think I have learned quite a lot. Different ones will teach you different skills. Some are good at acute medicine and how to do that, and some are very willing to teach us how to do a pleural tap… So, from different people, we have learned different things.”

(MP2, Medicine, Male, Phase-1)

F. The Increasing Recognition of Professional Responsibility and Accountability towards Patient Care Drives Learning Clinical Reasoning

This was a strong theme commonly experienced by all junior doctors. During this transition, junior doctors recognised the patient care responsibilities vested in them and experienced a change of role from an undergraduate to a medical doctor. This led them to internalise their role and work towards meeting these expectations, whilst learning from all opportunities.

“We realise that somehow, we’ve got to do something. It wasn’t like that as students. (Now, as doctors) If we can’t take an ABG (Arterial Blood Gas) once, we will try ten times and somehow take the ABG. We realise- we have that ownership, “This is my patient. I will do something for her.” So, I think that’s a good thing. We didn’t have that as students.”

(MP1, Medicine, Female, Phase-1)

Parallel to the change of role, they were accepted as members of a community of doctors actively involved in providing patient care, which gave them a sense of inclusion and prestige and they worked hard towards meeting the expectations, which in turn helped them learn clinical reasoning.

G. Diversity of Personal, Interpersonal, and Contextual Factors Impede the Development of Clinical Reasoning Skills

Several negative influences on learning clinical reasoning exist.

The personal factors that can diminish learning clinical reasoning are related to a lack of internal motivation to learn and limited use of reflective practice.

In addition, external factors such as lack of encouragement and limited recognition of their contribution as doctors further demotivate junior doctors. Settings supervised by several senior clinicians provide better learning opportunities, but they also expose them to experience individual variations of reasoning due to staff working patterns and hinder their ability to appreciate the continuity of care.

Moreover, as junior doctors, they handle a heavy workload and work under time constraints, which gives them limited opportunity to reflect and learn from experience. Junior doctors also experience the presence of a power gap between juniors and seniors within the healthcare team and maintenance of this hierarchy is a barrier to learning during practice.

Table 3 presents participant quotes highlighting the diversity of contextual factors that hinder learning clinical reasoning skills.

“…usually hiccups occur with failures of… all types of failures… I do not have much knowledge about those things. Actually, I got to know that hiccups occur due to organ failure also, after this patient… (laughs)” (no intentions to learn more expressed)

(SP2, Medicine, Male, Phase-2)

“…here I think, in our unit, because the consultant changes daily, I think that is a negative point. The fact that you don’t have that connection with one person, and the fact that there is no continuity in care…”

(MP1, Medicine, Female, Phase-1)

“…I mean, there are too many admissions some days and you’re just trying to get through from one patient to the next one. So, you don’t really have that much time to analyse the problem as such. I mean, when the ward is less heavy, I’m trying to figure out what’s wrong but some days it’s a little bit… like going through.”

(MP2, Medicine, Male)

Table 3. Quotes illuminating contextual factors that impede the development of clinical reasoning skills

In addition, the discussions with junior doctors revealed that their main goal during the internship was to arrive at a diagnosis and/or manage patients’ clinical problems. No learning-related goals were readily verbalised.

(When enquired about the goals of reasoning during the internship)
“That…..err…is… coming to a final diagnosis and starting the treatment…Basically, we are supposed to recognise life-threatening conditions and treat them.”

(MP3, Medicine, Male, Phase-2)

Similarly, the informal discussions with senior clinicians revealed their limited expectations of the contribution of the internship towards facilitating the development of clinical reasoning skills among juniors. This could be due to the service orientation of the internship leaving ‘learning to happen’ concurrently without being actively encouraged. This is not conducive to learning clinical reasoning.

H. The Construction of the ‘practice-oriented clinical reasoning skills development framework’

Embedded within the seven themes were a multitude of factors that could be clearly categorised as ‘Facilitators’, ‘Drivers’, ‘Sources’, and ‘Challenges’ of developing clinical reasoning skills. These factors helped junior doctors to migrate from a disease-oriented to a practice-oriented approach to clinical reasoning (Figure 1).

The categorisation was informed by how these factors influenced the development of clinical reasoning skills. ‘Facilitators’ actively support learning, while ‘drivers’ exert strong internal pressure to motivate learning clinical reasoning. A ‘source’ is an individual or an activity, that helps learn clinical reasoning through interacting with them. ‘Challenges’ are either internal or external to an individual and negatively influence the development of clinical reasoning skills.

Figure 1. ‘Practice-oriented clinical reasoning skills development framework’ highlighting the factors that influence the development of clinical reasoning skills during the transition from medical graduates to junior doctors

IV. DISCUSSION

Aligned with existing literature (Brennan et al., 2010; Lempp et al., 2005; Prince et al., 2000; Teunissen & Westerman, 2011), this study identified a steep learning curve for junior doctors in developing clinical reasoning skills upon commencing the internship. A ‘disjunction’ (Koufidis et al., 2020) was evident between knowledge acquired during medical undergraduate education and the demands of effective reasoning in clinical practice (Cave et al., 2009; Monrouxe et al., 2017). The ‘practice-oriented clinical reasoning skills development framework’ derived from this study shed light on the factors serving as ‘enhancers’ and ‘silencers’ of learning clinical reasoning skills during this critical period. This classification helps consolidate existing knowledge specific to this period and offers insights for addressing disconnections and facilitating the development of clinical reasoning skills.

In this study, novice doctors initially faced clinical reasoning challenges due to limited contextual understanding and reliance on rule-based reasoning comparable to the Dreyfus model of adult skill acquisition (2004). With increased work experience, they were able to promptly recognise contextual features distinguishing acute from non-acute presentations requiring urgent care. Additionally, they acknowledged the significance of the patient’s past medical history in forming a broader approach to reasoning. Some even acquired instincts for prompt clinical decision-making, a form of non-analytic reasoning identified by clinical experts (Norman et al., 2007) and blending non-analytic reasoning with occasional rule-based confirmation (analytic reasoning). This dual-process approach (Croskerry, 2009; Eva, 2004; Pelaccia et al., 2011), incorporating both analytic and non-analytic reasoning is recognised to overcome challenges associated with each approach. Such development of clinical reasoning skills with work experience is reflective of the advancement of reasoning skills along the first four stages of the Dreyfus model, from novice to proficiency stages. This contrasts with the limited value placed on the internship for developing clinical reasoning skills among some clinical supervisors and needs addressing during staff development initiatives.

It was also noted that junior doctors revert to the novice stage using more analytical rule-based reasoning with uncommon presentations or at the start of a new rotation in another specialty (Groves, 2012). This highlights the complexity of developing clinical reasoning skills, varying with the nature of the presentation and the clinical specialty, requiring more support for its development. This aligns with the ‘context-specific nature’ of clinical reasoning (Eva et al., 1998), the variation of reasoning outcomes of an individual due to contextual factors unique to clinical situations. The study revealed a clear influence of clinical specialty on reasoning, confining the development of clinical reasoning to a few focused clinical problems common to a particular specialty. This limits the overall development of clinical reasoning and hinders the momentum of clinical reasoning development entering a new clinical specialty. Therefore, clinical teachers should promote a comprehensive approach, considering differential diagnoses beyond a single specialty. Given the need for promptly recognising contextual features of disease severity in acute care settings coupled with early internship challenges, delaying trainees’ placement in acute care settings until later in a clinical rotation is a reasonable approach, contrary to current clinical practice.

Work experience was central to developing clinical reasoning skills (Charlin et al., 2007; Schmidt & Rikers, 2007; Schmidt & Boshuizen, 1993), but benefiting from experience required junior doctors to be internally motivated. According to the self-determination theory, when an individual experiences a feeling of being able to do something successfully (competence), when their actions are controlled internally or self-determined (autonomy), and when there is a sense of safety, belonging, and supportive relationships (relatedness), it enhances the intrinsic motivation of an individual (Ryan & Deci, 2000) and this was clearly noted during this study. The ‘drivers’, ‘facilitators’, and ‘sources’ of learning clinical reasoning identified during this study enabled fulfilling these three basic psychological needs required to be motivated to learn clinical reasoning. Hence, the ‘practice-oriented clinical skills development framework’ could serve as a valuable reference for clinical teachers supporting junior doctors in developing clinical reasoning skills during their transition to the workplace.

Echoing the evidence in the field (Ajjawi & Higgs, 2008; Gruppetta & Mallia, 2020), junior doctors recognised the change in their role from student to medical doctor and subsequent absorption into the healthcare team which made them internalise their responsibility and accountability towards patient care. Their engagement in patient care gradually increased to finally becoming valued members of this community, collaborating with other like-minded colleagues to develop a more deliberate understanding of reasoning and methods of using it. This aligns with the principles of legitimate peripheral participation and community of practice of the Situated Learning Theory (O’Brien & Battista, 2020). The community of practice created a safe learning environment, motivating junior doctors to learn clinical reasoning actively. This emphasises the significance of early integration of junior doctors as valued members of the healthcare team. A team-oriented approach to patient care, acknowledging every team member’s contribution, proves more beneficial here than an individual-focused hierarchical approach.

The junior doctors of this study learned through their interactions with senior doctors, peers, and other healthcare staff, as well as by actively participating in ward activities, revealing learning as a dynamic social act. The opportunity to observe, listen to, and emulate senior colleagues as they engaged in clinical reasoning with authentic patient presentations, followed by the application of the newly acquired skills, significantly influenced the development of their clinical reasoning skills. This highlights the continued relevance of apprenticeship as a pedagogical tool today (Dornan, 2005), facilitating the ongoing development of clinical reasoning skills among junior doctors. It also provides a unique opportunity to witness firsthand the decision-making processes of junior doctors operating independently in clinical practice, aligning with the highest level of clinical skills assessment in Miller’s pyramid (Miller, 1990). This presents a potential opening for formative assessment of clinical reasoning, whether conducted formally or informally, as part of junior doctor training.

Junior doctors also constructed knowledge through interpersonal interactions in the workplace by engaging in an iterative process of learning, application, and consolidation of knowledge with each experience contributing to the refinement of their clinical reasoning skills. Learning from these experiences required them to reflect on these experiences and arrive at new understandings by integrating and building on previous knowledge. This is aligned with the principles of experiential learning theory (Morris, 2020; Yardley et al., 2012) and the constructivism learning theory (Olusegun, 2015). This highlights the importance of encouraging reflection by proactively including junior doctors in all pertinent patient-related discussions. Also, the value of implementing a reflective portfolio to acknowledge junior doctors’ learning needs at the outset of the internship, with formative assessments conducted midway and at its conclusion by clinical supervisors. This could also introduce a learning orientation to the already service-focused internship placement.

Junior doctors found collaborative learning, including referrals to other specialties and engaging in those discussions or working in partnerships with peers, beneficial for developing clinical reasoning (Laal & Laal, 2012; Tolsgaard et al., 2016). This highlights the value of involving junior doctors in collaborative work within or across disciplines. Simulation-based training (Khan et al., 2011) offers similar opportunities for collaborative learning within a safe environment, without compromising patient safety. Integrating simulation-based training for junior doctors immediately after graduation or before the internship can equip them with reasoning skills for authentic practice, addressing challenges during their transition to the workplace.

The caring and compassionate attitudes instilled in junior doctors by their seniors and further nurtured through close patient interactions, served as indirect motivators for learning clinical reasoning skills. This is an area not widely discussed in literature. While there is acknowledgment of the potential influence of clinicians’ emotions on clinical reasoning (Kozlowski et al., 2017), the specific impact of emotional closeness in patient care, and whether it aligns with the conventions of a more objective, rule-based healthcare delivery system, remains an area that merits more comprehensive investigation (Dreyfus, 2004). However, the study findings support that the more emotionally closer the junior doctors are to their patients, the more they are invested in learning clinical reasoning to ensure healthier outcomes for their patients. Clinical teachers could nurture such attitudes through role modeling as noted in this study.

The interrupted supervisory relationships due to work rotations of the senior staff challenged learning clinical reasoning. Such system-related factors deprived junior doctors of learning by emulating senior practice. It also hampered their ability to appreciate the continuity of patient care due to individual variations of reasoning among senior staff and prevented developing closer relationships with seniors, which could have been more emotionally satisfying (Ryan & Deci, 2000). This underlines the need to take necessary steps to prevent any adverse effects of staff working patterns on trainee doctors, while simultaneously ensuring extended periods of supervision within a consistent healthcare team.

The collective findings of this study not only confirm but also add valuable insights to the clinical reasoning pathway for teaching clinical reasoning skills (Linn et al., 2012). According to this framework, the teaching of clinical reasoning occurs in three stages through three consultations. Stage 1- Demonstration and deconstruction, Stage 2- Comprehension, and Stage 3- Performance. The transition in focus from the teacher’s approach to the student’s performance occurs in the last stage. In junior doctor training, this framework is ideally applied within a team context during daily clinical ward rounds focusing on selected patient presentations as afforded by the time constraints. The three stages of the framework can be combined, and the reasoning discussions can be brief and can take place within the ward round after the selected presentations with increasing junior doctors’ involvement as they gain experience. This could allow junior doctors to learn from verbalised reasoning from the team, reflect and actively contribute to the discussion, and feel valued as team members. They can apply newly acquired reasoning skills in subsequent patient consultations independently, in addition to the opportunity to demonstrate these during the ward rounds. Based on the study findings, additional considerations for analysing patient presentations could be proposed as enhancements to the clinical reasoning pathway (Linn et al., 2012). These aspects are detailed within the overall structure of this framework in Figure 2.

Figure 2. Proposed additions to the deconstructed consultation according to the clinical reasoning pathway (Linn et al., 2012) for teaching clinical reasoning to junior doctors as part of daily clinical ward rounds

Additions are presented in italics and highlighted. (NAR- non-analytic reasoning)

V. CONCLUSION

The ‘practice-oriented clinical skills development framework’ has brought together factors that act as ‘enhancers’ and ‘silencers’ of learning clinical reasoning specific to this period of transition from medical graduates to junior doctors. These findings offer practical insights that can prove invaluable for clinical educators in their teaching practices to facilitate the development of clinical reasoning skills.

This research also offers insights into the responsibilities of clinical teachers in supporting the development of clinical reasoning skills among junior doctors during their internship. It provides suggestions for teaching these skills in practice and highlights potential system-related changes needed to facilitate this process.

A. Limitations of the Study

The reader needs to determine the applicability of the findings to their context to overcome the limitations of qualitative research. To facilitate this process, the methodology and the data analysis are appropriately detailed.

The study focused on immediate medical graduates, and therefore, it did not delve into the clinical reasoning experiences of junior doctors at different levels of seniority and training, although this could have added to our understanding. This lack of comparative analysis is another limitation of this study.

Notes on Contributors

Dr WCD Karunaratne conceptualised the study, prepared the study proposal, conducted all interviews, analysed them and developed the manuscript for this submission.

Professor Madawa Chandratilake was a supervisor of this study and he contributed to the study design, guided initial interviews, and analysed selected transcripts to finalise the final coding framework for the study. He also reviewed and provided feedback on different versions of the manuscript.

Professor Kosala Marambe was also a supervisor of the study. She contributed to the study design and analysis of selected transcripts to finalise the final coding framework for the study and provided feedback on different versions of the manuscript.

Ethical Approval

Ethical clearance (P/11/01/16) was obtained from the Faculty of Medicine, University of Kelaniya, Sri Lanka.

Data Availability

Institutional ethical clearance was given to maintain the data in the secure storage of the principal investigator of the study. However, additional supportive quotations are deposited in the repository (Karunaratne et al., 2023) and are available from this URL – https://doi.org/10.6084/m9.figshare.23536548.v2.

Readers may contact the principal researcher for additional data and details if required.

Acknowledgement

We would like to acknowledge Dr Pavithra Godamunne (Faculty of Medicine, University of Kelaniya, Sri Lanka) who applied and secured funding for the project and Dr Paul Crampton (Hull York Medical School, UK) who reviewed the manuscript and provided constructive comments.

Funding

University of Kelaniya, Sri Lanka funded this research study (No. RP/03/04/17/01/16).

Declaration of Interest

The authors have no conflicts of interest to disclose.

References

Ajjawi, R., & Higgs, J. (2007). Using hermeneutic phenomenology to investigate how experienced practitioners learn to communicate clinical reasoning. The Qualitative Report, 12(4), 612–638. https://doi.org/10.46743/2160-3715/2007.1616

Ajjawi, R., & Higgs, J. (2008). Learning to reason: A journey of professional socialisation. Advances in Health Sciences Education, 13, 133–150. https://doi.org/10.1007/s10459-006-9032-4

Braun, V., & Clarke, V. (2012). Thematic Analysis. In APA Handbook of Research Methods in Psychology, 2, 57-71. https://doi.org/10.1037/13620-004

Brennan, N., Corrigan, O., Allard, J., Archer, J., Barnes, R., Bleakley, A., Collett, T., & De Bere, S. R. (2010). The transition from medical student to junior doctor: Today’s experiences of tomorrow’s doctors. Medical Education, 44(5), 449–458. https://doi.org/10.1111/j.1365-2923.2009.03604.x

Cameron, A., Millar, J., Szmidt, N., Hanlon, K., & Cleland, J. (2014). Can new doctors be prepared for practice? A review. Clinical Teacher, 11(3), 188–192. https://doi.org/10.1111/tct.12127

Cave, J., Woolf, K., Jones, A., & Dacre, J. (2009). Easing the transition from student to doctor: How can medical schools help prepare their graduates for starting work? Medical Teacher, 31(5), 403–408. https://doi.org/10.1080/01421590802348127

Charlin, B., Boshuizen, H. P. A., Custers, E. J., & Feltovich, P. J. (2007). Scripts and clinical reasoning. Medical Education, 41(12), 1178–1184. https://doi.org/10.1111/j.1365-2923.2007.02924.x

Charlin, B., Tardif, J., & Boshuizen, H. P. A. (2000). Scripts and medical diagnostic knowledge: Theory and applications for clinical reasoning instruction and research. Academic Medicine, 75(2), 182–190. https://doi.org/10.1097/00001888-200002000-00020

Cohen, L., Manion, L., & Morrison, K. (2017). Research methods in education (8th ed.). Routledge. https://doi.org/10.4324/9781315456539

Croskerry, P. (2009). A universal model of diagnostic reasoning. Academic Medicine, 84(8), 1022–1028. https://doi.org/10.1097/ACM.0b013e3181ace703

Crotty, M. (1998). Foundations of social research: Meaning and perspective in the research process (1^st ed.). Routledge. https://doi.org/10.4324/9781003115700

Dornan, T. (2005). Osler, Flexner, apprenticeship and “the new medical education.” Journal of the Royal Society of Medicine, 98(3), 91–95. https://doi.org/10.1177/014107680509800302

Dreyfus, S. E. (2004). The five-stage model of adult skill acquisition. Bulletin of Science, Technology and Society, 24(3), 177–181. https://doi.org/10.1177/0270467604264992

Durning, S., Artino, A. R., Jr., Pangaro, L., van der Vleuten, C. P. M., & Schuwirth, L. (2011). Context and clinical reasoning: Understanding the perspective of the expert’s voice. Medical Education, 45(9), 927–938. https://doi.org/10.1111/j.1365-2923.2011.04053.x

Durning, S. J., & Artino, A. R. (2011). Situativity theory: A perspective on how participants and the environment can interact: AMEE guide no. 52. Medical Teacher, 33(3), 188-199. https://doi.org/10.3109/0142159x.2011.550965

Durning, S. J., Artino, A. R., Jr., Schuwirth, L., & van der Vleuten, C. (2013). Clarifying assumptions to enhance our understanding and assessment of clinical reasoning. Academic Medicine, 88(4), 442-448. https://doi.org/10.1097/acm.0b013e3182851b5b

Ericsson, K. A. (2004). Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Academic Medicine, 79(10 Suppl), S70–S81. https://doi.org/10.1097/00001888-200410001-00022

Eva, K., Neville, J., & Norman, G. (1998). Exploring the etiology of content specificity: Factors influencing analogic transfer and problem solving. Academic medicine, 73(10), S1-S5. https://doi.org/10.1097/00001888-199810000-00028

Eva, K. W. (2004). What every teacher needs to know about clinical reasoning. Medical Education, 39(1), 98–106. https://doi.org/10.1111/j.1365-2929.2004.01972.x

Goldacre, M. J., Davidson, J. M., & Lambert, T. W. (2003). Doctors’ views of their first year of medical work and postgraduate training in the UK: Questionnaire surveys. Medical Education, 37(9), 802–808. https://doi.org/10.1046/j.1365-2923.2003.01611.x

Graber, M. L., Franklin, N., & Gordon, R. (2005). Diagnostic error in internal medicine. Archives of Internal Medicine, 165(13), 1493–1499. https://doi.org/10.1001/archinte.165.13.1493

Groves, M. (2012). Understanding clinical reasoning: The next step in working out how it really works. Medical Education, 46(5), 444–446. https://doi.org/10.1111/j.1365-2923.2012.04244.x

Gruppetta, M., & Mallia, M. (2020). Clinical reasoning: Exploring its characteristics and enhancing its learning. British Journal of Hospital Medicine, 81(10), 1–9. https://doi.org/10.12968/hmed.2020.0227

Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112. https://doi.org/10.3102/003465430298487

Health Services Safety Investigation Body. (2022). Investigation report: Clinical decision making- diagnosis and treatment of pulmonary embolism in emergency departments. Retrieved from https://www.hssib.org.uk/patient-safety-investigations/clinical-decision-making-diagnosis-of-pulmonary-embolism-in-emergency-departments/investigation-report/

Huckman, R. S., & Barro, J. R. (2005). NBER working paper series cohort turnover and productivity: The July phenomenon in teaching hospitals. Retrieved from http://www.nber.org/papers/ w11182

Jen, M. H., Bottle, A., Majeed, A., Bell, D., & Aylin, P. (2009). Early in-hospital mortality following trainee doctors’ first day at work. PLoS ONE, 4(9), Article e7103. https://doi.org/10.1371/jo urnal.pone.0007103

Kafle, N. P. (2011). Hermeneutic phenomenological research method simplified. Bodhi: An Interdisciplinary Journal, 5(1), 181–200. https://doi.org/10.3126/bodhi.v5i1.8053

Karunaratne, D., Chandratilake, M., & Marambe, K. (2023). Annexure-1 Additional supportive quotations (Version 2). [Data set]. Figshare. https://doi.org/10.6084/m9.figshare.23536548.v2

Khan, K., Pattison, T., & Sherwood, M. (2011). Simulation in medical education. Medical Teacher 33(1), 1-3. https://doi.org/10.3109/0142159X.2010.519412

Koufidis, C., Manninen, K., Nieminen, J., Wohlin, M., & Silén, C. (2020). Grounding judgement in context: A conceptual learning model of clinical reasoning. Medical Education, 54(11), 1019–1028. https://doi.org/10.1111/medu.14222

Kozlowski, D., Hutchinson, M., Hurley, J., Rowley, J., & Sutherland, J. (2017). The role of emotion in clinical decision making: An integrative literature review. BMC Medical Education, 17(1), Article 255. https://doi.org/10.1186/s12909-017-1089-7

Laal, M., & Laal, M. (2012). Collaborative learning: What is it? Procedia Social and Behavioral Sciences, 31, 491–495. https://doi.org/10.1016/j.sbspro.2011.12.092

Langridge, N., Roberts, L., & Pope, C. (2015). The clinical reasoning processes of extended scope physiotherapists assessing patients with low back pain. Manual Therapy, 20(6), 745-750. https://doi.org/10.1016/j.math.2015.01.005

Lave, J. (1991). Situating learning in communities of practice. In L. B. Resnick, J. M. Levine, & S. D. Teasley (Eds.), Perspectives on socially shared cognition (pp. 63–82). American Psychological Association. https://doi.org/10.1037/10096-003

Laverty, S. M. (2003). Hermeneutic phenomenology and phenomenology: A comparison of historical and methodological considerations. International Journal of Qualitative Methods, 2(3), 21–35. https://doi.org/10.1177/160940690300200303

Lee, C. Y., Jenq, C. C., Chandratilake, M., Chen, J., Chen, M. M., Nishigori, H., Wajid, G., Yang, P. H., Yusoff, M. S. B., & Monrouxe, L. (2021). A scoping review of clinical reasoning research with Asian healthcare professionals. Advances in Health Sciences Education, 26(5), 1555–1579. https://doi.org/10.1007/s10459-021-10060-z

Lempp, H., Seabrook, M., Cochrane, M., & Rees, J. (2005). The transition from medical student to doctor: Perceptions of final year students and preregistration house officers related to expected learning outcomes. International Journal of Clinical Practice, 59(3), 324–329. https://doi.org/10.1111/j.1742-1241.2005.00 438. x

Linn, A., Khaw, C., Kildea, H., & Tonkin, A. (2012). Clinical reasoning: A guide to improving teaching and practice. Australian Family Physician, 41(1), 18–20. https://search.informit.org/doi/10. 3316/informit.875577247352301

Malterud, K., Siersma, V. D., & Guassora, A. D. (2016). Sample size in qualitative interview studies: Guided by information power. Qualitative Health Research, 26(13), 1753-1760. https://doi.org/10.1177/1049732315617444

Mamede, S., Schmidt, H. G., & Penaforte, J. C. (2008). Effects of reflective practice on the accuracy of medical diagnoses. Medical Education, 42(5), 468–475. https://doi.org/10.1111/j.1365-2923. 2008.03030.x

Mamede, S., van Gog, T., Moura, A. S., de Faria, R. M. D., Peixoto, J. M., Rikers, R. M. J. P., & Schmidt, H. G. (2012). Reflection as a strategy to foster medical students’ acquisition of diagnostic competence. Medical Education, 46(5), 464–472. https://doi.org/10.1111/j.1365-2923.2012.04217.x

Miller, G. (1990). The assessment of clinical skills/competence/performance. Academic Medicine, 65(9), S63–S67. https://doi.org/10.1097/00001888-199009000-00045

Monrouxe, L. V., Grundy, L., Mann, M., John, Z., Panagoulas, E., Bullock, A., & Mattick, K. (2017). How prepared are UK medical graduates for practice? A rapid review of the literature 2009-2014. British Medical Journal Open, 7, Article e013656. https://doi.org/10.1136/bmjopen-2016-013656

Morris, T. H. (2020). Experiential learning – A systematic review and revision of Kolb’s model. Interactive Learning Environments, 28(8), 1064–1077. https://doi.org/10.1080/10494820.2019.15702 79

Muylaert, C. J., Sarubbi, V., Jr., Gallo, P. R., Neto, M. L. R., & Reis, A. O. A. (2014). Narrative interviews: An important resource in qualitative research. Revista Da Escola de Enfermagem Da USP, 48, 184–189. https://doi.org/10.1590/s0080-623420140000800027

Norman, G. (2005). Research in clinical reasoning: Past history and current trends. Medical Education, 39(4), 418–427. https://doi.org/10.1111/j.1365-2929.2005.02127.x

Norman, G., Young, M., & Brooks, L. (2007). Non-analytical models of clinical reasoning: The role of experience. Medical Education, 41(12), 1140–1145. https://doi.org/10.1111/j.1365-292 3.2007.02914.x

O’Brien, B. C., & Battista, A. (2020). Situated learning theory in health professions education research: A scoping review. Advances in Health Sciences Education, 25(2), 483–509. https://doi.org/10.1007/s10459-019-09900-w

Olusegun, S. (2015). Constructivism learning theory: A paradigm for teaching and learning. IOSR Journal of Research & Medical Edication, 5(6), 66–70.

Pelaccia, T., Tardif, J., Triby, E., & Charlin, B. (2011). An analysis of clinical reasoning through a recent and comprehensive approach: The dual-process theory. Medical Education Online, 16(1), Article 5890. https://doi.org/10.3402/meo.v16i0.5890

Prince, K. J. A. H., Van de Wiel, M. W. J., Scherpbier, A. J. J. A., Van der Vleuten, C. P. M., & Boshuizen, H. P. A. (2000). A qualitative analysis of the transition from theory to practice in undergraduate training in a PBL-medical school. Advances in Health Sciences Education, 5, 105–116. https://doi.org/10.1023/a:1009873003677

Prince, K. J. A. H., Van de Wiel, M. W. J., Van der Vleuten, C. P. M., Boshuizen, H. P. A., & Scherpbier, A. J. J. A. (2004). Junior doctors’ opinions about the transition from medical school to clinical practice: A change of environment. Education for Health, 17(3), 323–331. https://doi.org/10.1080/13576280400002510

Robertson, D. (2012). Critical thinking and clinical reasoning in new graduate Occupational therapists: A phenomenological study. [Doctoral Dissertation, Robert Gordan University]. OpenAIR@RGU. http://hdl.handle.net/10059/792

Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist, 55(1), 68–78. https://doi.org/10.1037/0003-066X.55.1.68

Schmidt, H., & Rikers, R. (2007). How expertise develops in medicine: Knowledge encapsulation and illness script formation. Medical Education, 41(12), 1133–1139. https://doi.org/10.1111/j. 1365-2923.2007.02915.x

Schmidt, H. G., & Boshuizen, H. P. A. (1993). On acquiring expertise in medicine. Educational Psychology Review, 5(3), 205–221. https://doi.org/10.1007/BF01323044

Tallentire, V. R., Smith, S. E., Facey, A. D., & Rotstein, L. (2017). Exploring newly qualified doctors’ workplace stressors: An interview study from Australia. British Medical Journal Open, 7, e015890. https://doi.org/10.1136/bmjopen-2017-015890

Tallentire, V. R., Smith, S. E., Skinner, J., & Cameron, H. S. (2011). Understanding the behaviour of newly qualified doctors in acute care contexts. Medical Education, 45(10), 995–1005. https://doi.org/10.1111/j.1365-2923.2011.04024.x

Taylor, D. C. M., & Hamdy, H. (2013). Adult learning theories: Implications for learning and teaching in medical education: AMEE Guide No. 83. Medical Teacher, 35(11), e1561–e1572. https://doi.org/10.3109/0142159X.2013.828153

Teunissen, P. W., & Westerman, M. (2011). Opportunity or threat: The ambiguity of the consequences of transitions in medical education. Medical Education, 45(1), 51–59. https://doi.org/10.1111/j.1365-2923.2010.03755.x

Tolsgaard, M. G., Kulasegaram, K. M., & Ringsted, C. V. (2016). Collaborative learning of clinical skills in health professions education: The why, how, when and for whom. Medical Education, 50(1), 69–78. https://doi.org/10.1111/medu.12814

Varpio, L., Ajjawi, R., Monrouxe, L. V., O’Brien, B. C., & Rees, C. E. (2017). Shedding the cobra effect: Problematising thematic emergence, triangulation, saturation and member checking. Medical Education, 51, 40–50. https://doi.org/10.1111/medu.13124

Woods, N. N. (2007). Science is fundamental: The role of biomedical knowledge in clinical reasoning. Medical Education, 41(12), 1173–1177. https://doi.org/10.1111/j.1365-2923.2007.029 11.x

Yardley, S., Teunissen, P. W., & Dornan, T. (2012). Experiential learning: AMEE Guide No. 63. Medical Teacher, 34(2), e102–e115. https://doi.org/10.3109/0142159X.2012.650741

*WCD Karunaratne
Centre for Medical Education,
University of Dundee
Scotland, DD2 4BF
+44 7594 504928
Email: dilminikarunaratne@gmail.com / Wkarunaratne001@dundee.ac.uk

Submitted: 27 April 2023
Accepted: 17 August 2023
Published online: 2 January, TAPS 2024, 9(1), 36-41
https://doi.org/10.29060/TAPS.2024-9-1/OA3051

Mae Yue Tan^1,2, Zong Jie Koh^1,3, Shoban Krishna Kumar⁴, Rui Min Foo⁵, Rou An Tan⁶, Nisha Suyien Chandran^7,8Jeremy Bingyuan Lin^1,2, Malcolm Mahadevan⁹ & Eng Loon Tng⁵

¹Department of Paediatrics, Khoo Teck Puat – National University Children’s Medical Institute, National University Health System, Singapore; ²Department of Paediatrics & ⁸Department of Medicine, Yong Loo Lin School of Medicine, National University of Singapore, Singapore; ³Department of General Surgery, University Surgical Cluster, National University Health System, Singapore; ⁴Division of Advanced Internal Medicine, Department of Medicine, National University Hospital, National University Health System, Singapore; ⁵Department of Medicine & ⁶Department of Intensive Care Medicine, Ng Teng Fong General Hospital, National University Health System, Singapore; ⁷Division of Dermatology, Department of Medicine, National University Hospital, National University Health System, Singapore; ⁹Emergency Medicine Department, National University Hospital, National University Health System, Singapore

Abstract

Introduction: The night float (NF) system has been instituted in some hospitals in Singapore to improve the working hours and wellbeing of junior doctors. There have been concerns of compromised learning and patient safety with NF. The objective of this study is to compare clinical competency outcomes, based on existing assessment framework, between post-graduate year 1 (PGY1) doctors working on NF versus traditional call (TC) systems. The secondary aim was to explore patient safety outcomes between these groups of PGY1s.

Methods: Data on the formal assessments of PGY1s using the Entrustable Professional Activities (EPAs) and medical errors were prospectively collected between May 2021 and April 2022 from two hospitals that employed different on-call systems. Data was analysed descriptively. Categorical data was analysed using the Chi-square test or Fisher’s exact test where appropriate.

Results: One hundred and ninety-three PGY1s consented to the study. There was no statistically significant difference in the clinical competency of PGY1s in both groups. The number of PGY1s who had ‘needs improvement’ scores in a detailed EPA was not significantly different (9.0% in the NF group versus vs 3.7% in the TC group (p = 0.14)). They nonetheless passed the overall core EPA and no PGY1 failed their postings. No serious reportable medical errors occurred in either group.

Conclusion: PGY1s who worked on NF are equally competent compared to those who worked on TC based on the EPA assessment matrix. Patient safety is not compromised by PGY1s working on NF.

Keywords: Clinical competence, Float, Junior doctor, Patient safety, Wellbeing

Practice Highlights

Night float did not affect competency of post-graduate Year 1 doctors on current assessments.
Night float did not lead to compromised patient safety.
We provide objective data in consideration of restructuring working hours for junior doctors.

I. INTRODUCTION

Post graduate year 1 (PGY1) doctors are fresh graduates who are in their transitional year where they learn clinical decision-making skills and how to perform simple medical procedures independently. In Singapore, following the Medical Registration Act, PGY1s are required to complete 4 months of Internal Medicine and 4 months of General Surgery or Orthopaedic Surgery, with another 4-month posting of their choice in their 12 months as a PGY1 (Ministry of Health Holdings Pte Ltd, 2018). As part of the national PGY1 training framework (Ministry of Health, 2019), each hospital is required to provide training (teaching activities) to allow PGY1s to achieve their learning and competency outcomes. PGY1s are assessed via a standardized matrix which is used across all hospital rotations and institutions. They are expected to achieve competency in situations where senior supervision is less readily available, for example, while working on-call. PGY1 duties are similar regardless of discipline as they focus on core medical competencies common to general medical practice.

In Singapore, some hospitals have instituted the night float (NF) system as a service model where a dedicated team of doctors, including PGY1s, take over the care of patients for consecutive nights in a week. A different team of doctors will resume care of patients the following day. This facilitates timely handovers of patient management, eliminates the need for prolonged post-night call working hours and provides junior doctors with adequate rest, avoiding sleep deprivation. Conversely, in the traditional call (TC) system, junior doctors commence night duties immediately after their daytime routines. Frequently, at the end of their night calls, PGY1s continue with daytime work till mid-day or later, resulting in shifts of up to 36 hours.

The Accreditation Council for Graduate Medical Education (ACGME) guidelines recommended a maximum of 80 hours of duties per week based on extant literature on the impact of prolonged duty hours on burnout and fatigue (Singapore Medical Council, 2017), medical errors and adverse events (Barger et al., 2006; Landrigan et al., 2004; Trockel et al., 2020). Departments that have instituted the NF system have significantly improved their compliance to duty hour recommendations. However, concerns were raised regarding reduced training opportunities, specifically during the after-office hour period due to the reduced working hours with NF as well as potential compromises in patient safety arising from communication lapses associated with frequent handovers (Desai et al., 2013; Sun et al., 2016). These concerns have prevented widespread adoption of NF in Singapore hospital systems.

Two studies in Singapore have shown that residents who worked on NF felt that it did not affect their learning outcomes or compromise patient safety (Loo et al., 2020; Tan et al., 2019). However, these studies assessed the perceptions of junior doctors and provided no objective data for comparison of NF against TC. We thus aimed to compare clinical competency outcomes between PGY1s who worked on NF and those who worked on TC, across similar specialties. The secondary aim was to explore any differences in patient safety and medical errors between these two groups. We chose the PGY1 group for this study for the following reasons. Firstly, PGY1 training outcomes and assessments are standardized nationwide thus allowing for direct comparisons across institutions. Additionally, any policy implementation or change would impact all PGY1s across the board equally. In contrast, physicians in their later years of training (PGY2 and above) are a heterogenous group with their training assessment frameworks dependent on the relevant speciality programmes. Secondly, PGY1s are considered to have the one of the highest risks of burnout among physicians and are likely to benefit most from well-being initiatives. Lastly, the PGY1 year is a compulsory year of formative training with PGY1s subsequently graduating as fully licensed physicians. This decisive and important transitional year from medical training to clinical practice underscores the importance of considerations for their training and competency.

II. METHODS

This study was conducted over three rotations (each PGY1 rotation spans 4 months) from May 2021 to April 2022. PGY1s working in National University Health System cluster (either in National University Hospital where NF is the structure for PGY1s rotating through General Surgery, Internal Medicine, and Orthopaedics (since January 2022), or Ng Teng Fong General Hospital where TC is practiced) were invited to participate via emails. PGY1s could opt out of the study. Ethics approval was obtained [National Healthcare Group Domain Specific Review Board (NHG DSRB), Singapore; Reference No.: 2021/00536].

Prospective data was collected from formal educational and workplace-based assessments. Currently, the Entrustable Professional Activities (EPAs) framework, which describes professional activities that a trainee can be entrusted to perform at varying levels of supervision, is used for this purpose. EPAs are established milestones in post graduate medical education to gauge whether trainees have achieved necessary skill sets that are appropriate for their level of training. In its current form, the local EPA framework for PGY1s includes 7 ‘core EPA’ groups. Each ‘core EPA’ group further encompasses a list of ‘detailed EPAs’. An example of a ‘core EPA’ and its ‘detailed EPAs’ is shown in Table 1. At the end of a rotation during their final evaluation, PGY1s are evaluated by their supervisors for each ‘detailed EPA’ as part of their ‘Learning Needs’. A grading of ‘needs improvement’ for a detailed EPA indicates that the PGY1 has not demonstrated sufficient competence in that specific activity. No input is required if a PGY1 is deemed to have met expectations for the detailed EPA. In performing this evaluation, the supervisor is expected to canvass feedback from other team and faculty members who have worked with the PGY1. At the end of the rotation, should all EPAs (core and detailed) be assessed as competent, the supervisor would grade the posting outcome as a pass, with the opportunity to nominate the PGY1 for the national outstanding PGY1 award.

Core Entrustable Professional Activity (EPA)

Detailed EPAs

Manage patients by obtaining a detailed history, performing appropriate physical examination, requesting & follow-through relevant diagnostic evaluation & therapeutic interventions.

Obtain complete relevant and accurate patient history and perform physical examination.
Propose a reasonable working and differential diagnoses.
Decide on initial diagnostic evaluation and recommend/carry out acute therapeutic interventions.
Demonstrate awareness of costs, risks and benefits.
Articulate and utilize knowledge to diagnose and treat common conditions encountered in the posting.
Maintain patient confidentiality.
Work effectively and respectfully within an inter-professional team.
Demonstrate compassion, empathy, openness & respect for patients &relatives that includes cultural and religious sensitivities.

Table 1. Example of an Entrustable Professional Activity (EPA), with detailed EPAs within the core EPA

Data on medical errors or patient safety issues involving PGY1s, including near-miss events, were also collected from the medical error reporting systems of both hospitals. All data were anonymized prior to data analysis. Data was analysed descriptively and with comparative statistics. Chi-square test or Fisher’s exact test were used for analysis of categorical data where appropriate.

III. RESULTS

A total of 193 PGY1s consented to this study with no opt-outs. There was no statistically significant difference in the proportion of PGY1s with ‘needs improvement’ between the 2 groups: 9.0% of PGY1s in the NF group compared to 3.7% in the TC group had ‘needs improvement’ in any of the ‘detailed EPAs’ (p = 0.16) (Table 2). All PGY1s with ‘needs improvement’ were deemed competent in the ‘core EPA’. Notably, three PGY1s in the NF group who had ‘needs improvement’ in the ‘detailed EPAs’ were nominated for the Outstanding PGY1 Award by the Ministry of Health. All PGY1s in this study passed their postings satisfactorily.

	Night Float (NF)	Traditional On-Call (TC)	P-value
Total number of PGY1s who had ‘needs improvement’ for a detailed EPA / total number of PGY1s (%)	10 / 111 (9.0)	3 / 82 (3.7)	0.16
1^st rotation (%)	3 / 36 (8.3)	2 / 29 (6.9)	0.99
2^nd rotation (%)	2^a / 32 (6.3)	0 / 22 (0.0)	0.51
3^rd rotation (%)	5^b /43 (11.6)	1 / 31 (3.2)	0.39

Table 2. Proportion of post graduate year 1 (PGY1) doctors who had detailed EPAs that were marked as ‘needs improvement’

^aBoth these PGY1s were nominated for the outstanding PGY1 award.

^bOne PGY1 was nominated for the outstanding PGY1 award.

In terms of medical errors, there were 25 reported events in the NF group and 12 in the TC group. To account for the difference in number of PGY1s between both groups, we reported the average event rate per PGY1, i.e. event rate over the number of PGY1s in the rotation. There was no statistically significant difference in the average event rate per PGY1 between the two groups (0.23 vs 0.15, p = 0.20) (Table 3). There was also no statistically significant difference between the two groups when comparison was made for each rotation. All reported errors regardless of NF or TC, were in the ‘No Harm’, or ‘Minor Harm’ categories with no serious reportable events: 24.3% of all errors were needle stick-related events while 24.3% were for wrongly labelled blood tubes.

	Night Float (NF)	Traditional On-Call (TC)	P-value
Average event rate per PGY1	0.23	0.15	0.20
1^st rotation	0.11	0.14	0.99
2^nd rotation	0.25	0.14	0.49
3^rd rotation	0.30	0.16	0.18

Table 3. Event rates for reported events (either medical errors or patient safety issues) in post graduate year 1 (PGY1) doctors over the study period.

IV. DISCUSSION

Our study revealed insights on the effects of the NF system on the clinical training of PGY1s and impact on patient safety. There was no statistically significant difference in objective clinical competency between PGY1s working on NF or TC. This refutes the concern of inadequate training opportunities arising from shorter working hours during after-office-hours shifts. These objective findings are in line with the subjective evaluation and perceptions of junior doctors on the impact of the local night float system on their training (Loo et al., 2020; Tan et al., 2019).

Though the time spent for emergency-related work or ‘on-call’ situations where senior supervision is less readily is reduced on the NF compared to the TC groups, the training outcomes were similar and not compromised. We postulate that this is because of the robust formal training programme during the daytime which is consistent across hospitals given the national framework guidelines. In addition, the high number of emergency cases being admitted after office hours in both the NF and TC groups provide ample learning opportunities for the PGY1s. Regardless of which service model the PGY1s practice in, formal training during daytime work is consistent across hospitals given the national framework guidelines. PGY1s have protected teaching time each week and they are able to catch up with the teaching topics through e-learning modules when they are post call. As such, the NF system does not negatively affect their learning through the formal teaching program. While hours spent on training activities are important, it is just one component that contributes to learning outcomes. Previous literature has shown that shorter training hours can also optimize learning as it allows between-session and post-training learning to occur (Molloy et al., 2012).

Though not statistically significant, the NF group did have a higher number of PGY1s who were flagged for ‘needs improvement’ in their ‘detailed EPA’ as compared to the TC group (9.0% vs 3.7%, p = 0.16). We believe this was a result of the limitations with the EPA assessment matrix rather than poor clinical training in the PGY1s in the NF group. The standardized assessment matrix in the overall evaluation form provides only two options in the grading of each ‘detailed EPA’; PGY1s who are deemed to have failed the task should be graded as ‘needs improvement’ whilst the section is left blank by the supervisors for PGY1s who are deemed to be competent in the task. Thus, the grade of ‘needs improvement’ is in reality that of failure, but is commonly misconstrued as having room for improvement in an already competent PGY1. Hence, supervisors may mistakenly grade the ‘detailed EPA’ domains as ‘needs improvement’ without any intent that the PGY1 is in any way incompetent. Supervisors may even grade the PGY1s as ‘needs improvement’ in an effort to spur them to greater heights, and we suspect this to be the case as seen in the 3 PGY1s who were nominated for the Outstanding Award on their overall assessment despite having a “needs improvement” grade for a detailed EPA. Additionally, as with any assessment matrix, the utility and accuracy of EPAs in reflecting competency is heavily dependent on how well it is used by the supervisor and PGY1, and its refinement and standardization continue to be work in progress.

Our study showed that the NF system does not affect patient safety adversely. There were no major clinical errors or adverse patient events in both groups. There were also no statistically significant differences in the minor clinical errors, categorized as ‘No Harm’ and ‘Minor Harm’, that were made by PGY1s between the two groups (0.23 vs 0.15, p = 0.20). Given the anonymity of data on the medical error reporting systems, we were unable to discern whether errors were made while on call or if certain PGY1s were making repeated errors. As these numbers are small, comparative analysis is not meaningful and thus not performed. There were no medical errors arising from miscommunications because of increased handovers in our study. We believe that PGY1s were able to provide better quality handovers in a timelier fashion due to the structured work-hour cycles in NF. NF reduces PGY1 fatigue and the associations between fatigue and its impact on judgement, competency and risk of mistakes has been demonstrated (Trockel et al., 2020).

V. CONCLUSION

In conclusion, our study supports existing evidence that the NF system does not compromise clinical training for PGY1s or patient safety. The NF system emphasizes the importance of training quality over quantity and enhances PGY1 wellbeing. We believe our study provides objective data for the medical fraternity for consideration in the restructuring of working hours for junior doctors (Abu Baker, 2022).

Notes on Contributors

MYT contributed to design and conception of the work, data analysis and interpretation, drafted the article and revised it critically for important intellectual content. ZJK contributed to design and conception of the work, data analysis and interpretation, drafted the article and revised it critically for important intellectual content. SKK contributed to design and conception of the work, data analysis and interpretation, drafted the article and revised it critically for important intellectual content. RMF contributed to data acquisition and revised the article critically for important intellectual content. RAT contributed to data acquisition and revised the article critically for important intellectual content. NSC contributed to study conception, data interpretation and revised the article critically for important intellectual content. JBL contributed to design and conception of the work and revised the article critically for important intellectual content. MM contributed to conception of work, data interpretation and revised the article critically for important intellectual content. ELT contributed design and conception of the work, data acquisition and revised the article critically for important intellectual content. All authors read and approved the final version, and agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Ethical Approval

This study was performed in line with the ethical standards of the institutional and/or national research committee and with the 1964 Declaration of Helsinki and its later amendments or comparable ethical standards. Ethics approval for the study was obtained from the National Healthcare Group Domain Specific Review Board (NHG DSRB), Singapore; Reference No.: 2021/00536.

Data Availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Acknowledgement

The authors would like to thank the Ministry of Health, Singapore, for supporting the implementation of the night float call system for junior doctors in our institution. We would also like to thank our PGY1s for consenting to this study, and our program coordinators for providing administrative assistance for this project. Lastly, we would like to thank Ms Sheena Nishanti Ramasamy for her assistance in editing and formatting the manuscript for submission.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Declaration of Interest

The authors declare no relevant financial or non-financial competing interests with respect to the research, authorship and/or publication of this article.

References

Abu Baker, J. (2022, March 9). Review of junior doctors’ work hours among steps to improve healthcare workers’ well-being. Channel News Asia. https://www.channelnewsasia.com/singapore /review-junior-doctors-work-hours-among-steps-improve-healthcare-workers-well-being-2551376

Barger, L. K., Ayas, N. T., Cade, B. E., Cronin, J. W., Rosner, B., Speizer, F. E., & Czeisler, C. A. (2006). Impact of extended-duration shifts on medical errors, adverse events, and attentional failures. PLoS Medicine, 3(12), Article e487. https://doi.org/10.1371/journal.pmed.0030487

Desai, S. V., Feldman, L., Brown, L., Dezube, R., Yeh, H. C., Punjabi, N., Afshar, K., Grunwald, M. R., Harrington, C., Naik, R., & Cofrancesco, J., Jr (2013). Effect of the 2011 vs 2003 duty hour regulation-compliant models on sleep duration, trainee education, and continuity of patient care among internal medicine house staff: A randomized trial. JAMA Internal Medicine, 173(8), 649–655. https://doi.org/10.1001/jamainternmed.2013.2973

Landrigan, C. P., Rothschild, J. M., Cronin, J. W., Kaushal, R., Burdick, E., Katz, J. T., Lilly, C. M., Stone, P. H., Lockley, S. W., Bates, D. W., & Czeisler, C. A. (2004). Effect of reducing interns’ work hours on serious medical errors in intensive care units. The New England Journal of Medicine, 351(18), 1838–1848. https://doi.org/10.1056/NEJMoa041406

Loo, B. K. G., Ng, C. L., Chin, R. T., Davies, L. J., Yong, J., Ang, A. E. L., Chong, Y. W., & Tambyah, P. A. (2020). Nationwide survey comparing residents’ perceptions of overnight duty systems in Singapore: Night float versus full overnight call. Singapore Medical Journal, 61(10), 559–562. https://doi.org/10.11622/sme dj.2020149

Ministry of Health Holdings Pte Ltd. (2018). Medical service career path. https://www.physician.mohh.com.sg/medicine/medi cal-service-career-path

Ministry of Health Singapore. (2019, January 16). The national PGY1 training and assessment framework. https://www.moh.gov.sg/hpp/all-healthcareprofessionals/news/NewsArticleDetails/nationalpgy1_training_assessment_framework

Molloy, K., Moore, D. R., Sohoglu, E., & Amitay, S. (2012). Less is more: Latent learning is maximized by shorter training sessions in auditory perceptual learning. PloS One, 7(5), Article e36929. https://doi.org/10.1371/journal.pone.0036929

Singapore Medical Council. (2017). SMC circular – Guidelines on Postgraduate Year 1 (PGY1) training and postings for the accreditation of PGY1 training posts.

Sun, N. Z., Gan, R., Snell, L., & Dolmans, D. (2016). Use of a night float system to comply with resident duty hours restrictions: Perceptions of workplace changes and their effects on professionalism. Academic Medicine: Journal of the Association of American Medical Colleges, 91(3), 401–408. https://doi.org/10.1097/ACM.0000000000000949

Tan, B. Y., Ngiam, N. J., Chang, Z. Y., Tan, S. M. Y., Shen, X., Mok, S. F., Subramanian, S., Ooi, S. B. S., & Kee, A. C. (2019). Perceptions of a night float system for intern doctors in an internal medicine program: An Asian perspective. Korean Journal of Medical Education, 31(3), 271–276. https://doi.org/10.3946/kjme.2019.137

Trockel, M. T., Menon, N. K., Rowe, S. G., Stewart, M. T., Smith, R., Lu, M., Kim, P. K., Quinn, M. A., Lawrence, E., Marchalik, D., Farley, H., Normand, P., Felder, M., Dudley, J. C., & Shanafelt, T. D. (2020). Assessment of physician sleep and wellness, burnout, and clinically significant medical errors. JAMA Network Open, 3(12), Article e2028111. https://doi.org/10.1001/jamanet work open.2020.28111

*Dr Mae Yue TAN
Department of Paediatrics,
Khoo Teck Puat-National University
Children’s Medical Institute,
National University Health System, Singapore
Department of Paediatrics,
Yong Loo Lin School of Medicine,
National University of Singapore, Singapore
NUHS Tower Block Level 12,
1E Kent Ridge Rd, Singapore 119228
Email: mae_yue_tan@nuhs.edu.sg

Submitted: 28 January 2023
Accepted: 17 August 2023
Published online: 2 January, TAPS 2024, 9(1), 3-19
https://doi.org/10.29060/TAPS.2024-9-1/OA2947

Thun How Ong¹, Hwee Kuan Ong², Adrian Chan¹, Dujeepa D. Samarasekera³& Cees Van der Vleuten⁴

¹Department of Respiratory and Critical Care Medicine, Singapore General Hospital, Duke-NUS Medical School, Singapore; ²Department of Physiotherapy, Singapore General Hospital; ³Centre for Medical Education, Yong Loo Lin School of Medicine, National University of Singapore, Singapore; ⁴Department of Educational Development and Research, Maastricht University, Maastricht, The Netherlands

Abstract

Introduction: The mini-Clinical Evaluation Exercise (CEX) is meant to provide on the spot feedback to trainees. We hypothesised that an ultra-short assessment tool with just one global entrustment scale (micro-CEX) would encourage faculty to provide better feedback compared to the traditional multiple domain mini-CEX.

Methods: 59 pairs of faculty and trainees from internal medicine completed both the 7-item mini-CEX and a micro-CEX and were surveyed regarding their perceptions of the 2 forms. Wordcount and specificity of the feedback was assessed. Participants were subsequently interviewed to elicit their views on factors affecting the utility of the CEX.

Results: Quantity and quality of feedback increased with the micro-CEX compared to the mini-CEX. Wordcount increased from 9.5 to 17.5 words, and specificity increased from 1.6 to 2.3 on a 4-point scale, p < 0.05 in both cases. Faculty and residents both felt the micro-CEX provided better assessment and feedback. The micro-CEX, but not the mini-CEX, was able to discriminate between residents in different years of training. The mini-CEX showed a strong halo effect between different domains of scoring. In interviews, ease of administration, immediacy of assessment, clarity of purpose, structuring of desired feedback, assessor-trainee pairing and alignment with trainee learning goals were identified as important features to optimize utility of the (mini or micro or both) CEX.

Conclusions: Simplifying the assessment component of the CEX frees faculty to concentrate on feedback and this improves both quantity and quality of feedback. How the form is administered on the ground impacts its practical utility.

Keywords: Workplace Based Assessment, Mini-CEX, Micro-CEX, Feedback, Assessment

Practice Highlights

Simplifying the assessment component of the CEX frees faculty to concentrate on feedback.
A simpler form can result in better and more feedback.
Making it easy for faculty to use the form is important and increases its utility in providing feedback and assessment.

I. INTRODUCTION

The Mini-CEX is one of the most widely used work-placed based assessment (WBA) tools and is supported by a large body of theoretical and empirical evidence which have shown that when used in the context of repeated sampling, it is both a valid assessment tool and is also an effective education tool in giving feedback to the trainee (Hawkins et al., 2010; Norcini et al., 2003). However, in practice, the educational value of the mini-CEX, as measured chiefly by trainee and faculty perceptions and satisfaction, varied significantly (Lorwald et al., 2018). Factors affecting the educational value have been described by Lorwald et al. and categorised into context of usage, and user, implementation and outcome factors (Lorwald et al., 2018).

Context refers to the situation in which the mini-CEX is executed, and factors which impact its actual usage, such as time needed for conducting the Mini-CEX, or the usability of the tool. Time constraint on the part of both the residents and the assessors is an especially frequent issue across multiple studies (Bindal et al., 2011; Brazil et al., 2012; Castanelli et al., 2016; Lörwald et al., 2018; Morris et al., 2006; Nair et al., 2008; Yanting et al., 2016). The mini-CEX was conceived as a 30-minute exercise of directly observed assessment, and there are 6 or 7 domains which faculty are expected to assess (Norcini et al., 2003). In a busy clinical environment however, what actually occurs is often a brief clinical encounter of 10-15 minutes or even less where only a few of the mini-CEX’s domains were assessed (Berendonk et al., 2018).

User factors refers to trainee and faculty knowledge of the mini-CEX and their perceptions of its use. Studies have found that the mini-CEX is frequently regarded as a check box exercise (Bindal et al., 2011; Sabey & Harris, 2011). Assessor’s and trainee’s training and attitudes, or unfamiliarity with the WBA tools also negatively impact the educational value of the mini-CEX (Lörwald et al., 2018). Reports have shown that educating faculty on the formative intent of mini-CEX can improve feedback provided (Liao et al., 2013).

Implementation factors refer to how the mini-CEX is actually executed on the ground. Some studies have reported that the mini-CEX often occurs without actual direct observation (Lörwald et al., 2018) or feedback provided (Weston & Smith, 2014). Implementation in turn affected outcome, which refers to the trainees appraisal of the feedback received (Lörwald et al., 2018).

One way of improving the educational value of the mini-CEX then might be to improve the context of its usage, by redesigning the mini-CEX to better fit the realities of the clinical workplace. In different clinical encounters, specific domains of performance are more easily and obviously observed and assessed than others (Crossley & Jolly, 2012). Reducing the number of dimensions the assessors are asked to rate was shown to decrease measured cognitive load and improved interobserver reliability (Tavares et al., 2016). It has also been shown that using rating scales that align with the clinician’s cognitive schema perform better, for instance, scales that ask the clinician assessors about the trainees ability to practice safely with decreasing levels of supervision (i.e. entrustability) showed better discrimination and higher reliability (Weller et al., 2014). Compared to multidimension rating scales, global rating scales have greater reliability and validity in assessing candidates in OSCE examinations (Regehr et al., 1998), assessing technical competence in procedures (Walzak et al., 2015) and in simulation-based training (Ilgen et al., 2015).

We proposed therefore to replace the multiple domains with a single rating asking faculty what level of supervision the resident would require in performing a similar task, i.e. a global entrustment scale. The shorter assessment task should refocus the faculty on the feedback component, whilst still retaining the ability to identify trainee progression. One such form has been proposed by Kogan and Holmboe (2018), and we designated this the micro-CEX.

We hypothesised that these changes would improve the usability (“context” as described by Lorwald et al.) and hence improve the educational value of the assessment, measured in this study by the specificity and quality of the feedback given by faculty.

Our study aims to show therefore that the shorter micro-CEX can provide better feedback than the usual mini-CEX. We also sought to find out, from the perspective of the end-users, what other adjustments to the implementation and design of the mini or micro-CEX can be made to improve its acceptability, educational value and validity.

The study focussed on the following questions:

Does the micro-CEX stimulate faculty to provide more specific and actionable feedback compared to the mini-CEX?

Can the micro-CEX provide discriminatory assessment for residents across different years of practice?

What are the perceptions of the faculty and residents regarding the factors affecting utility of the assessment instrument in providing feedback and assessment?

II. METHODS

A. Setting and Subjects

The study was conducted in the division of Internal Medicine in a 1700 bed hospital in Singapore between September and December 2018. All faculty and residents rotating through internal medicine were invited to participate via e-mail, and agreeable faculty and residents paired up. In usual practice, residents must complete at least 2 mini-CEX covering standard inpatient or outpatient encounters during each three-month internal medicine posting, hence both residents and faculty are familiar with the usual mini-CEX.

B. Design

In order to evaluate for any participant reactivity affecting the CEX data (i.e. a Hawthorne effect) (Paradis & Sutkin, 2017), a baseline sample of 30 of the usual mini-CEX performed in the 3 months prior to the study was randomly selected and deidentified (from June to August 2018) . The quantity and specificity of feedback in these was evaluated as detailed below.

For the study itself, faculty and residents used the usual mini-CEX as the first assessment in the first 2 weeks of the month, followed by a second assessment using the micro-CEX in the next 2 weeks. This sequence was chosen as performing the micro-CEX first might affect how the subsequent mini-CEX was performed. Cases chosen for the mini-CEX and micro-CEX were inpatient or outpatient internal medicine encounters, and faculty were simply instructed to choose cases that represented typical cases of average difficulty with no restrictions on the exact cases to be chosen.

Faculty and residents completed an anonymised survey on their experiences at the end of the study and were invited to participate in a semi-structured group interview to elicit their views regarding which aspects of the mini-CEX exercise influenced feedback and assessment (Appendix 2). Both faculty and residents were informed that the survey and interviews were part of this study and participation in either was taken to be implied consent. The workflow of the study is seen in Figure 1.

Figure 1. Study workflow

C. Instruments

The mini-CEX used in the program is based on the one described by Norcini (Norcini et al., 2003).This form was hosted on the internet ( New Innovations, Ohio, USA) and could be accessed by faculty from their mobile devices or their email. The micro- CEX was hosted on an opensource online survey tool (LimeSurvey GmbH, Hamburg, Germany) and can be accessed from mobile devices. A copy of both forms is available in Appendix 1.

D. Analysis of Feedback

The quality of feedback was assessed firstly by a word count, and then by grading the specificity of the feedback on a three-point scale (Pelgrim et al., 2012) (Appendix 3) and finally by the presence or absence of an actionable plan for improvement. In order to avoid rater bias, the assessor for the specificity of the feedback was blinded to the source of feedback (mini or Micro CEX). The first 20 forms were independently graded by two separate assessors (OTH and AC) using the above criteria, achieving a kappa coefficient of 0.852; all subsequent forms were graded by OTH, with any uncertainty resolved by discussion between AC and OTH. Word count and specificity, as well as faculty and resident preferences between the forms, were analysed using paired samples T-test. Proportion of Feedback which showed an actionable plan was compared using a Chi-Square test.

E. Semi-structured Interviews

Faculty and residents were interviewed separately. 21 residents and 6 faculty were interviewed over 8 sessions lasting between 20 to 30 minutes each. Interviews were conducted by the investigator (OTH). The interviews were audiotaped and transcribed verbatim. Data collection ended when saturation was reached. Member checking of the transcripts was carried out.

The inductive template analysis as described by Nigel King was used to analyse the interview transcripts (King, 2012). Two transcripts were studied and coded separately by the investigator (OTH) and a collaborator (OHK). A priori themes of assessment, feedback and administration were used to structure the data so that the research question could be answered. Codes were discussed between OTH and OHK until a consensus was reached, and a codebook was created. The subsequent transcripts were coded by OTH. OHK, AC and OTH subsequently met to discuss the categories and emerging themes. NVivo 12 was used to store and manage the codes and transcripts. Results were triangulated with data from the quantitative surveys.

For all quantitative data, an alpha of 0.05 was used as the cut-off for significance. IBM SPSS 25 (IBM SPSS Statistics for Windows, Version 25.0. Armonk, NY) was used for calculations.

III. RESULTS

The data that support the findings of this study are openly available in Figshare repository, at https://doi.org/10.60 84/m9.figshare.21862068.v1 (Ong, 2023).

There were 33 internal medicine residents during the study period and 32 (97%) participated in the study; one resident declined to participate. They were paired with 39 different faculty over the three months of the study. 59 unique faculty- resident pairs completed both sets of CEX. 30.5% of the residents were in their first year of residency (R1), 47.9% in second year (R2), and 22.0% were in their third year (R3). The residents completed an average of 1.96 pairs of mini and micro CEX each. Time taken to complete the assessments was estimated by faculty to be 11.33 +/- 6.56 min for mini-CEX vs 9.42 +/-5.51 min for the micro-CEX (p = 0.02).

A. Evaluation of Feedback in the Mini-CEX: Baseline and During Study

30 de-identified mini-CEX were extracted randomly from the 3 months preceding the initiation of the study. These served as a baseline control and were compared to the feedback from the first, traditional mini-CEX done during the study (Table 1). During the period of the study, faculty using the same mini-CEX provided feedback that was more specific. Proportion of actionable feedback provided was much more in the mini-CEX done as part of the study compared to baseline controls (Table 1: 3.3% controls vs 28% study mini-CEX, p = 0.005).

	Mini-CEX vs prior baseline control			Mini-CEX vs Micro-CEX
	Prior baseline control mini-CEX (mean+/-SD)	Study Mini-CEX (mean +/- SD)	p value	Mini-CEX (mean +SD)	Micro CEX (mean +SD)	p value
Q1 in which areas did the resident do well
Word count	12.1 +/-14.1	9.5 +/- 7.0	0.93	9.5 +/- 7.0	17.5 +/- 10.3	<0.001
Specificity*	1.2 +/- 1.0	1.6 +/- 0.90	0.08	1.6+/- 0.9	2.3 +/- 0.7	<0.001
Q2/3 Areas needing improvement/ recommendations for future improvement
Word count	3.8 +/- 6.8	5.7 +/- 7.3	0.06	5.7 +/- 7.3	19.3 +/- 15.1	<0.001
Specificity*	0.5 +/- 0.7	1.1 +/-1.1	0.01	1.1 +/- 1.0	1.8 +/- 0.9	<0.001
Actionable	1/30 (3.3%)	17/59(28.8%)	0.005	17/59 (28.8%)	18/59(30.5%)	0.84

Table 1. Quality and quantity of feedback in prior baseline control vs study mini-CEX, and in mini vs Micro-CEX

B. Evaluation of Feedback in the Micro and Mini-CEX During Study

Comparison of the feedback given in the mini and micro-CEX during the study is shown in Table 1. Feedback wordcount increased and was more specific with micro-CEX compared to the contemporaneous mini-CEX done by the same pair. However, there were no differences in the proportion of actionable feedback given in both forms.

C. Discrimination Between Residents in Different Years of Training

The micro-CEX was able to show progression between the years of training, with a significant rise in the resident’s mean score across the three years of training. On a 4 point score the mean entrustment score increased from 2.45 in the first year of training to 3.30 by the third year (p<0.05). (Figure 2)

Figure 2. Level of entrustment vs Year of training

Correlation of residents’ year of training and grading of the mini-CEX domains was moderate (kappa 0.39 to 0.60). There was high correlation between all seven questions in the mini-CEX (kappa 0.7 to 0.8) (see appendix 4), implying that the resident’s score in one domain heavily influenced the score in other domains i.e. a halo effect.

D. Faculty and Resident Preferences

21 (out of total 32 participating residents, 65% response rate) residents and 25 (out of total 39 participating faculty, 64.9% response rate) responded to the survey. Faculty and residents felt that the Micro -CEX had better value for both assessment and feedback compared to the Mini-CEX (Table 2).

	Mini-CEX Mean + SD	Micro CEX Mean + SD	p value	t-Stats	Cohen’s d
Usefulness for assessment
Faculty	6.04 +/- 1.34	6.57 +/- 0.95	0.04	2.23	0.46
Residents	6.00 +/- 1.62	6.9 +/- 0.91	0.03	-2.31	0.52
Usefulness for feedback
Faculty	6.00 +/- 1.35	6.87 +/- 1.10	0.01	-3.07	0.64
Residents	5.43 +/- 1.40	6.81 +/- 1.57	0.09	-3.82	0.83

Table 2. Perceptions of faculty and residents regarding usefulness of mini and micro-CEX for assessment and feedback

*Scoring is on a 9-point Likert scale, with 1 = not useful at all …. 9 = very useful

E. Qualitative Data

Qualitative data from the semi-structured interviews was analysed to better understand what the features of the micro-CEX driving this preference were, and to look for helpful features in the CEX. Themes from the semi-structured interviews were distilled into 6 themes (Table 3):

1) Make it easy: A recurrent theme was that the micro-CEX was easier to use and the short form could be used by the bedside, on resident or faculty mobile devices.

2) Immediacy is important: Faculty and residents both prized the ability to integrate the assessment into their daily routines, and this immediacy was very important in enhancing the value of the feedback.

3) Tell us what it’s for: Faculty and residents both expressed that the intended purpose of the forms needed to be explicit. Uncertainty in purpose of the form resulted in a perception of redundancy with the other assessments, and confusion about summative vs formative intent of the assessment inhibited honest feedback and assessment.

4) Structure the form so we know what you want: Structuring the form with specific areas to remind them to provide narrative feedback, and what specific areas to provide feedback in, was useful. Faculty and residents both felt that the micro-CEX had better learning value than the mini-CEX.

5) Choice of assessor matters depending on objective of the tool: Faculty and residents agreed that assessments were frequently affected by the prior experiences between the two, impacting the objectivity of assessments via both the micro-and mini- CEX. Prior engagement with the resident facilitated provision of feedback. However, for assessment purposes, residents felt that a faculty with no prior knowledge of the trainee might be more objective.

6) Align assessment with learning goals: Many of the residents were preparing for their postgraduate medical examinations, and they found the mini-CEX exercise especially useful if it was conducted in a way similar to their examinations (the Royal College of Physicians PACES exam) – in other words, the utility of the exercise increased greatly when the assessment was aligned with the residents’ own learning goals.

S/N	Themes	Quotations
1.	Make it easy to do	The micro-CEX was “more succinct. So, it’s, it’s much easier to administer” -F “If it’s a shorter form, even though the quantity may be less maybe the fact that the quality of whatever feedback we’re given is better because they’re really giving the one or two points that really stood out to them that we need to improve on or the one or two things that we really did well” -R (Regarding the mini-CEX) “The fact that it’s more detailed actually maybe reduces the quality of the feedback because … if you ask me for additional remarks for every single domain, then they just put nil, nil, nil because there’s no time” -F
2.	Immediacy is important	“Memory is also fresh because you’ve just done the case and so I think the learning value’s a lot better” -R “I think looking at it in terms of like a learning experience also, um, when we have that micro-CEX on the spot, ah, not only can we address, like all the points immediately, like what the resident should, um, but at the same time, ah, you can kinda go through certain topics at the same setting as well” -F
3.	What is this for	“I think we need clear goals as to why we do these, rather than to simply check boxes.” -R “The form should come with what is the expectation of this, uh, assessment, whether it’s for assessing, or it’s for a feedback, or it’s …. whether this person can work as a HO. I mean, the intention will drive how I assess” -R “We have a lot of forms, the 360 and the mini-cex and all. Sometimes maybe I personally don’t really see what the difference is sometimes or how it can help to change assessment. I think it’s just extra admin for everyone and everyone just gets fed up doing it” -R “I think the assessor, sometimes they’re very nice, they know it affects your, your grading or your, your overall performance in the residency, so they try not to be too strict” -R
4.	Be specific about what you want to know	The micro-CEX had “I think several features currently that are really quite useful. Number one is that there is the mandatory open-ended field, um, for areas that need improvement and areas that need to be reinforced” -F “I find the comments, uh, quite useful. Maybe not the grades itself, because usually people would just give, like, mod- middle-grade. But, the written comments are actually quite useful” -R
5.	Choice of assessor matters depending on objective for the tool	“It’s quite easy for me to, to, to, remember each of them and give them dedicated feedback” -F “It should be someone that you don’t really know, but maybe in the same department. So, that it can be like, really, like a proper case scenario, yeah. Instead of grading you based on what their other impressions are” -R
6.	Align assessment with learning goals	“So I had this one particular case, that was a very good PACES case, that I clerked in the morning, and, we impromptu made it into a mini-CEX kind of session and, and we went in quite in depth into the discussion, and PACES that sort of stuff, and I thought that was very useful.” -R

Table 3. Themes with supporting quotations

*1 PACES = Membership of Royal Collage Physicians clinical examination, a required exit certification for the residents.

IV. DISCUSSION

The most striking result from this study is that even without specific faculty training or other intervention, simplifying the assessment task alone led faculty to write longer, and more specific feedback. Faculty and residents also perceived that the feedback was better. By simplifying the assessment, the faculty’s attention was shifted from grading the resident in multiple domains toward qualitatively identifying good and bad points in the encounter, providing feedback for the residents.

Proportion of actual actionable feedback in the two forms, however, was not different. This is perhaps because there was no specific faculty training for the study as we felt that the additional training itself would impact results. Specific faculty training may be needed to improve this aspect.

A Hawthorne effect was noticed in the study (Adair, 1984). The proportion of actionable feedback provided was much more in the mini-CEX done as part of the study compared to baseline controls (Table 1: 3.3% controls vs 28% study mini-CEX, p = 0.005). Word count and specificity also increased. However, despite this, we were still able to show that the micro-CEX induced faculty to provide more and better feedback.

From the global entrustment scale used in the micro-CEX, it was possible to demonstrate progression from first year to third year of residency (Figure 2). One potential concern is loss of granularity in assessment of different domains, i.e., that we might lose the ability to identify the specific domain in which the resident is weak if we do not ask faculty to score physical examination, history taking, management etc. separately. However, we found a high correlation between the scores in all domains in the mini-CEX (kappa ranged from 0.7 to 0.8, see appendix 4), indicating a strong halo effect. This suggests that in practice, faculty are making a global assessment anyway rather than a separate assessment of separate domains. Faculty and residents perceived that the single global assessment with the micro-CEX provided better assessment.

The messages from faculty and residents about what they perceive to be important in making the CEX work for them speak for themselves. The importance of making the form easy to administer is very intuitive; the bureaucratic impracticality of paper portfolios was pointed out long ago and e-portfolios were touted as the preferred solution (Van Tartwijk & Driessen, 2009) but the message here is that administrative details have significant impact on the utility of the CEX – many of the issues cited such as the number of assessments an individual assessor has to make, whether the assessor is equipped to do the assessment on the spot, or whether the assessor has prior exposure to the resident or not – are administrative and educational design details that faculty training alone cannot solve.

Our study had several limitations. Variations in the clinical environment such as ward vs ambulatory clinic, variable workload or competing responsibilities of the faculty and residents might have affected how the CEX was administered. However, distractions in the ward do affect the performance of CEX in real life as well.

We also note that in this study design, the mini-CEX was performed before the micro-CEX. This was deliberate as the residents and faculty were used to doing the mini-CEX on an ongoing basis so the first mini-CEX would be a “usual” assessment followed by the new assessment. Performing the micro-CEX first might affect how the subsequent mini-CEX was performed.

In this study, we did not attempt to make judgements about reliability and validity of the micro-CEX as only one specific data point was obtained for each trainee. The mini-CEX is validated to be reliable mainly in the context of repeated tests , and preferably in the context of a coherent program of assessment (van der Vleuten & Schuwirth, 2005). Whether the micro-CEX is able to provide equivalent robust and valid assessment compared to the mini-CEX depends on how it is used and is an area ripe for future study.

V. CONCLUSION

Our study demonstrated that the micro-CEX has a high rate of acceptability amongst faculty and residents, as well as a measurable improvement in feedback characteristics compared to the usual mini-CEX. The context in which the form is administered in actual practice has significant impact on its utility for feedback and assessment.

Ethical Approval

The study protocol was reviewed by the hospital Institutional Review Board, who deemed this as an educational quality improvement project which did not require IRB approval (Singhealth CIRB Ref: 2018/2696).

Notes on Contributors

Thun How Ong conceptualised and designed the study, administered the interviews, analysed the data and wrote the manuscript.

Hwee Kuan Ong participated in data analysis and coding of the qualitative data.

Adrian Chan participated in data analysis and in grading of the feedback specificity.

Dujeepa D. Samarasekera provided input on initial study design and reviewed the manuscript.

C. P. M. van der Vleuten provided guidance and input at all stages of the study, from initial study design to data analysis and manuscript writing.

Data Availability

The data that support the findings of this study are openly available in Figshare repository, at

https://doi.org/10.6084/m9.figshare.21862068.v1

Acknowledgement

The authors would like to acknowledge the contributions of the following:

Tan Shi Hwee and Nur Suhaila who provided the administrative support that made the whole project feasible.

The Faculty and Residents who were willing to do the extra CEX and the interviews, and who labour daily in pursuit of the ultimate goal of providing better care for our patients.

Funding

No funding was obtained for this study.

Declaration of Interest

All authors have no declaration of interest.

References

Adair, J. G. (1984). The Hawthorne effect: A reconsideration of the methodological artifact. Journal of Applied Psychology, 69(2), 334-345. https://doi.org/10.1037/0021-9010.69.2.334

Berendonk, C., Rogausch, A., Gemperli, A., & Himmel, W. (2018). Variability and dimensionality of students’ and supervisors’ mini-CEX scores in undergraduate medical clerkships – A multilevel factor analysis. BMC Medical Education, 18(1), 100. https://doi.org/10.1186/s12909-018-1207-1

Bindal, T., Wall, D., & Goodyear, H. M. (2011). Trainee doctors’ views on workplace-based assessments: Are they just a tick box exercise? Medical Teacher, 33(11), 919-927. https://doi.org/10.3109/0142159X.2011.558140

Brazil, V., Ratcliffe, L., Zhang, J., & Davin, L. (2012). Mini-CEX as a workplace-based assessment tool for interns in an emergency department – Does cost outweigh value? Medical Teacher, 34(12), 1017-1023. https://doi.org/10.3109/0142159X.2012.719653

Castanelli, D. J., Jowsey, T., Chen, Y., & Weller, J. M. (2016). Perceptions of purpose, value, and process of the mini-Clinical Evaluation Exercise in anesthesia training. Canadian Journal of Anaesthesia, 63(12), 1345-1356. https://doi.org/10.1007/s12630-016-0740-9

Crossley, J., & Jolly, B. (2012). Making sense of work-based assessment: Ask the right questions, in the right way, about the right things, of the right people. Medical Education, 46(1), 28-37. https://doi.org/10.1111/j.1365-2923.2011.04166.x

Hawkins, R. E., Margolis, M. J., Durning, S. J., & Norcini, J. J. (2010). Constructing a validity argument for the mini-Clinical Evaluation Exercise: A review of the research. Academic Medicine, 85(9), 1453-1461. https://doi.org/10.1097/ACM.0b013 e3181eac 3e6

Ilgen, J. S., Ma, I. W., Hatala, R., & Cook, D. A. (2015). A systematic review of validity evidence for checklists versus global rating scales in simulation-based assessment. Medical Education, 49(2), 161-173. https://doi.org/10.1111/medu.12621

King, N. (2012). Doing template analysis. Sage Knowledge, 426-450. https://doi.org/10.4135/9781526435620.n24

Kogan, J. R., & Holmboe, E. (2018). Practical guide to the evaluation of clinical competence appendix 4.6 (2nd ed.). Elsevier.

Liao, K. C., Pu, S. J., Liu, M. S., Yang, C. W., & Kuo, H. P. (2013). Development and implementation of a mini-Clinical Evaluation Exercise (mini-CEX) program to assess the clinical competencies of internal medicine residents: From faculty development to curriculum evaluation. BMC Medical Education, 13, 31-31. https://doi.org/10.1186/1472-6920-13-31

Lörwald, A. C., Lahner, F. M., Greif, R., Berendonk, C., Norcini, J., & Huwendiek, S. (2018). Factors influencing the educational impact of Mini-CEX and DOPS: A qualitative synthesis. Medical Teacher, 40(4), 414-420. https://doi.org/10.1080/0142159X.2017 .1408901

Lorwald, A. C., Lahner, F. M., Nouns, Z. M., Berendonk, C., Norcini, J., Greif, R., & Huwendiek, S. (2018). The educational impact of mini-Clinical Evaluation Exercise (mini-CEX) and Direct Observation of Procedural Skills (DOPS) and its association with implementation: A systematic review and meta-analysis. PLoS One, 13(6), Article e0198009. https://doi.org/10.1371/jour nal.pone.0198009

Morris, A., Hewitt, J., & Roberts, C. M. (2006). Practical experience of using directly observed procedures, mini Clinical Evaluation Examinations, and peer observation in pre-registration house officer (FY1) trainees. Postgraduate Medical Journal, 82(966), 285-288. https://doi.org/10.1136/pgmj.2005.040477

Nair, B. R., Alexander, H. G., McGrath, B. P., Parvathy, M. S., Kilsby, E. C., Wenzel, J., Frank, I. B., Pachev, G. S., & Page, G. G. (2008). The mini clinical evaluation exercise (mini-CEX) for assessing clinical performance of international medical graduates. Medical Journal of Australia, 189(3), 159-161.

Norcini, J. J., Blank, L. L., Duffy, F. D., & Fortna, G. S. (2003). The mini-CEX: A method for assessing clinical skills. Annals Internal Medicine, 138(6), 476-481.

Ong, T. H., Ong, H. K., Chan, A., Samarasekera, D. D., van der Vleuten, C. (2023). Micro CEX vs Mini CEX: Less can be more [Dataset]. Figshare. https://doi.org/https://doi.org/10.6084/m9.Fig share.21862068.v3

Paradis, E., & Sutkin, G. (2017). Beyond a good story: From Hawthorne Effect to reactivity in health professions education research. Medical Education, 51(1), 31-39. https://doi.org/10.1111/medu.13122

Pelgrim, E. A. M., Kramer, A. W. M., & Van der Vleuten, P. M. (2012). Quality of written narrative feedback and reflection in a modified mini-Clinical Evaluation Exercise: An observational study. BMC Medical Education, 12(1), 97. https://doi.org/10.1186/1472-6920-12-97

Regehr, G., MacRae, H., Reznick, R. K., & Szalay, D. (1998). Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination. Academic Medicine, 73(9), 993-997.

Sabey, A., & Harris, M. (2011). Training in hospitals: What do GP specialist trainees think of workplace-based assessments? Education for Primary Care, 22(2), 90-99.

Tavares, W., Ginsburg, S., & Eva, K. W. (2016). Selecting and simplifying: Rater performance and behavior when considering multiple competencies. Teaching and Learning in Medicine, 28(1), 41-51. https://doi.org/10.1080/10401334.2015.1107489

van der Vleuten, C. P., & Schuwirth, L. W. (2005). Assessing professional competence: From methods to programmes. Medical Education, 39(3), 309-317. https://doi.org/10.1111/j.1365-2929. 2005.02094.x

Van Tartwijk, J., & Driessen, E. W. (2009). Portfolios for assessment and learning: AMEE Guide no. 45. Medical Teacher, 31(9), 790-801. https://doi.org/10.1080/01421590903139201

Walzak, A., Bacchus, M., Schaefer, J. P., Zarnke, K., Glow, J., Brass, C., McLaughlin, K., & Ma, I. W. (2015). Diagnosing technical competence in six bedside procedures: Comparing checklists and a global rating scale in the assessment of resident performance. Academic Medicine, 90(8), 1100-1108. https://doi.org/10.1097/acm.0000000000000704

Weller, J. M., Misur, M., Nicolson, S., Morris, J., Ure, S., Crossley, J., & Jolly, B. (2014). Can I leave the theatre? A key to more reliable workplace-based assessment. British Journal of Anaesthesia, 112(6), 1083-1091. https://doi.org/10.1093/bja/aeu052

Weston, P. S. J., & Smith, C. A. (2014). The use of mini-CEX in UK foundation training six years following its introduction: Lessons still to be learned and the benefit of formal teaching regarding its utility. Medical Teacher, 36(2), 155-163. https://doi.org/10.3109/0142159X.2013.836267

Yanting, S. L., Sinnathamby, A., Wang, D., Heng, M. T. M., Hao, J. L. W., Lee, S. S., Yeo, S. P., & Samarasekera, D. D. (2016). Conceptualizing workplace-based assessment in Singapore: Undergraduate mini-Clinical Evaluation Exercise experiences of students and teachers. Tzu-Chi Medical Journal, 28(3), 113-120. https://doi.org/10.1016/j.tcmj.2016.06.001

*Ong Thun How
Academia, 20 College Road,
Singapore 168609
97100638
Email: ong.thun.how@singhealth.com.sg

Submitted: 5 April 2023
Accepted: 17 August 2023
Published online: 2 January, TAPS 2024, 9(1), 28-35
https://doi.org/10.29060/TAPS.2024-9-1/OA3035

Aminda Faizura Omar, Tun Yasmin Iffah Mohd Suria Affandi, Mohd Isyrafuddin Ismail, Mas Suryalis Ahmad& Ilham Wan Mokhtar

Comprehensive Care Centre of Studies, Faculty of Dentistry, Universiti Teknologi MARA, Malaysia

Abstract

Introduction: This study was undertaken to investigate students’ perceptions of innovation-integrated learning, adopted as an alternative approach for special care dentistry (SCD) training.

Methods: Ninety final-year dental students from the 2021 cohort were divided into eight groups to complete an innovation project dedicated to eight categories of patients with special health care needs. Discussion and final presentations, involving related experts were conducted remotely via an online platform. Then, students were invited to answer a validated online feedback survey on their perceptions of the learning approach.

Results: The response rate for the feedback survey was 91.1% (n=82). Most students agreed that the activity was interesting, improved their knowledge and understanding of SCD, allowed engagement between peers, supported sharing of ideas and experiences, encouraged student–lecturer interactions, and enhanced knowledge integration and application. Students also expressed that the activity enhanced creativity and innovation, instilled an interest and positive attitude toward learning SCD, and encouraged teamwork. However, a few students noted facing some limitations in completing their projects (i.e., technological challenges and reduced physical access to purchase materials). Around half felt neutral about having an increased workload from this activity. Moreover, perceptions differed regarding time and financial commitments, as well as supervisors’ and patients’ involvement during the project development process.

Conclusion: Students perceived that the innovation-oriented learning activity, was beneficial in multiple aspects of SCD training.

Keywords: Innovation, Learning, Dental Students, Perception

Practice Highlights

We have developed a new integrated teaching method via the utilisation of innovation development for special needs groups.
Students’ perceptions of this learning method were explored using both quantitative and qualitative methods via an online feedback questionnaire.
Overall, students positively responded to the teaching method when it came to improving their knowledge and understanding of SCD, engagement between peers, sharing of ideas and experiences, student–lecturer interactions, and knowledge integration and application.
This novel teaching method achieves the highest level of Bloom’s Taxonomy which is to create something new.

I. INTRODUCTION

Special care dentistry (SCD) involves managing patients with special health care needs, such as those with physical, intellectual, psychological impairments as well as complex medical conditions (Royal Australasian College of Dental Surgeons, 2021). At the undergraduate level, many teaching and learning activities have been conducted to provide students with learning experience in this area of patient care. These activities include lectures, community-based learning, simulation exercises and clinical training (Ahmad, Mokhtar, et al., 2020; Mohamed Rohani & Mohd Nor, 2021). Students who received training in SCD were found to have demonstrated better comfort and attitudes in managing patients with special health care needs (Mohamed Rohani et al., 2021).

However, these learning activities may be compromised by situations that limit face-to-face interactions, either among students, between teacher and learners, as well as with the patients. Such situations may happen due to disease (such as the recent COVID-19 pandemic), geographical barriers and accessibility issues (Amir et al., 2020). It was found that limitations of dental training that requires physical contact may compromise students’ learning experience, thus affecting their competency, knowledge and confidence in patient care (Chang et al., 2021).

Meanwhile, it has become incumbent on dental education providers to prepare future graduates with essential skills that are deemed beneficial for practicing health care professionals. These skills include innovation, leadership, critical thinking and enterpreneurial abilities (Malaysian Dental Council, 2021). Acquisition of these abilities have been associated with many aspects of personal and professional development including emotional intelligence and practice management (Saleem et al., 2019; Wang et al., 2020).

In view of the various challenges and demands of the current (and future) educational environment and healthcare needs, an innovation-integrated educational approach was developed and adopted as an alternative teaching and learning method to teach SCD. This approach, undertaken mostly remotely using an online learning platform, involved the integration of students’ cognitive function, psychomotor ability, and affective skills through the development of innovative products to address oral health issues experienced by the different categories of patients with special health care needs. As a novel educational approach, assessment of its effectiveness is therefore essential. This study was undertaken to investigate students’ perceptions of the innovation-integrated learning, conducted remotely via an online platform, developed during the COVID-19 pandemic, as an alternative approach for SCD training.

II. METHODS

A. Study design

This was a descriptive, cross-sectional, quantitative and qualitative study, using a validated online feedback questionnaire involving the final-year undergraduate students of the Bachelor of Dental Surgery program of the Faculty of Dentistry, Universiti Teknologi MARA.

B. Study Population

Because this module is a compulsory component of the final-year undergraduate dental curriculum (Faculty of Dentistry, 2022), all students were required to be involved with the group innovation project. However, participation in the online feedback survey was voluntary. At the beginning of the semester, the students were made aware of the study and that their participation in the innovation program and the research were treated separately.

All students who agreed to participate in the online feedback survey provided informed consent, attached to the information sheet supplied to all potential participants via their official university email. To avoid coercion, students were provided with an explanation in the information sheet that their participation was entirely voluntary and anonymous. Those who wished not to participate in the online feedback survey would not be penalised, and they could withdraw their participation at any point without having to provide a justification.

C. Study Instrument

The questionnaire used for the online feedback survey was adopted from the study by Rowe and Wood (2008). It was content-validated by a panel of 2 experts in dental education before its use in the main survey. From the item-level content validatiy index (I-CVI) for each item as well as a summation of the I-CVI divided by the number of average (S-CVI/Ave), a score of 1 was obtained for both indices as the panelists rated all items with a score of 3 or 4. The questionnaire consisted of nine items about the perception of students towards this teaching method, measured using a 5-point Likert scale (strongly disagree, disagree, neutral, agree, and strongly agree). Open-ended questions were included at the end of the questionnaire to further explore students’ perceptions of the learning activities.

D. Study Conduct

For the group innovation project, all final-year students (n=90) were divided into eight groups, based on their assigned clinical groupings. Each group was assigned to an academic or clinical specialist in SCD, who acted as a supervisor. Eight categories of patients with special health care needs were identified and randomly allocated to the different groups. The categories of patients were 1) hearing disability, 2) speech disability, 3) visual disability, 4) geriatric, 5) Down syndrome, 6) autism, 7) cerebral palsy, and 8) dementia.

Students were given three months to develop innovative products. They were expected to engage with their supervisors during the whole development process, which involved 1) identification of health complications related to the condition and its impact on oral health, 2) understanding of issues related to patients’ oral health care; 3) creation of a prototype to address the related oral health–related complications. Because all students were at the time subjected to restrictions in having physical classes due to COVID-19 (Samat et al., 2020), large-group discussions and final presentations were undertaken remotely online.

At the end of this group activity, an online presentation, judged by experts in related fields, was conducted as a form of assessment, as well as constructive feedback on improvements and the marketability of the product. Students were then invited to participate in the online feedback questionnaire, which was distributed via their official university email. An English language questionnaire was used as the respondents were undergraduate dental students with a good command and understanding of the language. A reminder email was sent a week later, with another reminder sent the following week.

E. Data Analysis

To maximise the outcome, quantitative data were simplified into a 3-point Likert scale (agree, neutral, and disagree). Using SPSS version 27, the frequency of the data was analysed to compare students’ responses for each item. For analysis of the benefits, the nine items of the questionnaire were further categorised into different themes: Theme 1—Acceptance of the teaching method concept (Item 1), Theme 2—Effectiveness of teaching method (Items 2, 3, 6, and 7), Theme 3—Important role of supervisors (Items 4, 5, and 8), and Theme 4—Impact on students’ workload (Item 9).

For the qualitative component, data were analysed via thematic analysis involving open and closed coding. Emerging themes were then identified based on these codes, which were then validated through comparisons amongst researchers.

F. Ethical Approval

Ethics approval was obtained from the Research Ethics Committee (REC) Universiti Teknologi MARA, Malaysia, in accordance with the Declaration of Helsinki (REC12/2021 [MR 1004]).

III. RESULTS

A total of 82 out of 90 students returned the questionnaire, giving a response rate of 91.1%. The frequency of responses for each question or item, categorised into the different themes, is shown in Table 1.

Items	Questions	Themes	Negative (%)	Neutral (%)	Positive (%)
Item 1	The program is an interesting new teaching method to learn about Special Care Dentistry.	Acceptance of teaching methods/ concept	1.2	8.5	91.3
Item 2	The program has provided me a clear understanding of the Special Care Dentistry concept.	Effectiveness of teaching method	1.2	9.8	89.0
Item 3	The teaching method managed to integrate the knowledge and clinical implication in Special Care Dentistry.		1.2	4.9	93.9
Item 6	The method of teaching allows me to interact effectively with my fellow group mate and the rest of the batch.		2.4	7.3	91.3
Item 7	The teaching method supports ideas and experience sharing between students		0.0	2.4	97.6
Item 4	The supervisors encourage and accept different opinion.	Important role of supervisors	0.0	3.7	96.3
Item 5	The teaching method provide opportunities for discussion between students and supervisors.		2.4	2.4	95.2
Item 8	The supervisors explained the purpose of the program well and structured.		0.0	7.3	92.7
Item 9	The teaching method increase the workload of students compared to classroom learning.	Impact on students’ workload	7.3	53.7	39.1

Table 1. Frequency of responses for items, categorised into different themes.

Overall, most students provided positive responses regarding activity acceptance and effectiveness and the supporting role of supervisors. However, slightly more than half of the students felt neutral about the learning activity causing an increased workload compared to classroom learning.

The thematic analysis of qualitative data yielded multiple themes that can be categorised into three main areas: 1) benefits of this education method, 2) limitations of this learning intervention, and 3) differing opinions on the integration of the group innovation project into the dental curriculum.

A. Benefits of The Study

Within this area, the students expressed four main themes.

1) Enhances creativity and innovation: Most students felt that this teaching method had enhanced their creativity in finding methods to manage patients with special dental needs as well as in developing innovations for the group of interest.

“This integrated learning helps me with my creativity and critical thinking so we could create an innovation that is appropriate with the current technology.”

Student 10

2) Improves knowledge and understanding of SCD: Using innovation in teaching made students delve into the subject matter, thus improving their knowledge of their patients and gaining a deeper understanding of SCD.

“A good approach indeed. I have a chance to challenge my understanding and gain new information from my friends, supervisors, and postgraduate students. This helped me develop a new way of thinking and find good literature on SCD.”

Student 35

Subtheme 1: Encourages independent information-gathering about SCD

Some students noted that they had to rely on the resources they had and work by themselves due to the pandemic, which made them more independent when researching topics about SCD.

“In my opinion, this teaching method is helpful to help the students to understand the special needs module by asking the student to develop products according to the difficulties of the disabilities required.”

Student 25

Subtheme 2: Development of critical thinking and problem-solving skills in SCD

By giving students the task of developing information, they were forced to solve problems the special interest group faced. Therefore, students believed this made them learn better critical thinking and problem-solving skills.

“Good to brainstorm ideas and put us in their shoes in order to address their different needs.”

Student 74

3) Instils interest and positive attitudes towards learning SCD: Due to their minimal patient experience, most students felt this task made them more interested in their special interest group, which improved their attitudes to learning SCD.

“The integrated learning approach gives me ideas on how situations should be handled in real life on aspects like the expectation from special care patients and preparedness as a future dentist to have a special care patient.”

Student 12

4) Encourages teamwork: Some students noted that, even with the isolation, they felt that this type of teaching method had encouraged them to work more closely with not only their teammates but supervisors as well.

“Innovative program. Good. Made students and supervisors unite as teammates.”

Student 62

B. Limitations of this Learning Intervention

Regarding limitations, two themes were noted.

1) Technological challenges: Several students experienced poor internet connection throughout the process, which hindered their efficiency.

”Very helpful, but internet connection and environment are some of the barriers for me for not study effectively, I prefer face-to-face learning approach. ”

Student 72

2) Reduced physical access to purchasing of materials: The movement control order that was implemented throughout most of the pandemic left most students restricted from being able to access appropriate materials from local or distant suppliers.

“We faced problems buying materials due to the movement control order.”

Student 77

C. Differing Opinions on the Integration of the Group Innovation Project into the Dental Curriculum

Students had differing opinions on the following aspects.

1) Time commitment and workload: Despite the lack of face-to-face sessions, some students believed they were given adequate time to create and execute ideas comfortably.

“Since the project is given so much earlier than the deadline, we are able to create a better project without feeling any burden in completing the module.”

Student 36

However, a handful of students felt they needed more time to be able to have performed better for the task.

“It is an interesting method and helpful to learn special care dentistry, but it is quite time-consuming, and we have restrictions due to COVID-19.”

Student 65

2) Financial demands: Depending on the type of innovation they designed, some students felt that the funding provided was not sufficient for them to prepare a good product.

“Some innovations need to create prototype only because low budget, but it is understandable.”

Student 74

However, a few students believed the financial assistance was enough for them to complete their innovation.

“The funding provided was sufficient.”

Student 43

3) Guidance from supervisors: Many students noted that their supervisor provided ample guidance throughout the project.

“Excellent. Students and supervisors can communicate and deliver knowledge well through this method.”

Student 19

“Our supervisor introduced us to some deaf patients. So we were able to get the patient’s feedback on our product.”

Student 52

However, some students believed the transition from face-to-face to online consultations was not ideal for the guidance they needed.

“However, there is lacking in communication and discussion between students and supervisors as this is still new to some students.”

Student 47

4) Exposure to SCD practice: Because the students were supposed to have exposure to special care patients in their final year, they believed that this teaching module was sufficient to help them understand patients.

“I think this method actually gives a realistic overview of how to work with patients.”

Student 43

On the other hand, a few felt that clinical sessions were more beneficial for them to apply the knowledge they have.

“In my opinion, direct exposure to SND patients would be better so we can apply the knowledge and innovation.”

Student 76

IV. DISCUSSION

This study involved the undergraduate teaching of SCD for students in a Malaysian dental school. In Malaysia, SCD education has yet to become a compulsory component of the five-year undergraduate curriculum (Naimie et al., 2020). This may be related to various barriers to providing training in this area of dentistry, including a lack of clinical expertise and educational resources (Ahmad et al., 2014). Despite this, Universiti Teknlologi MARA, Malaysia has taken a proactive approach to provide training in SCD, at both undergraduate and postgraduate levels (https://dentistry. uitm.edu.my/index.php/en/). For undergraduate students, teaching and learning of SCD are provided in the final year, as part of the Comprehensive Care Dentistry course (Faculty of Dentistry, 2022). Students undergo a series of lectures, hands-on demonstrations, problem-based learning, tutorials, a clinical rotation in the SCD specialist clinic, and a community engagement activity project (Faculty of Dentistry, 2022).

Although various teaching and learning interventions have been successfully conducted in the undergraduate training of SCD in previous years (Naimie et al., 2020), the transition from face-to-face training to online distance learning imposed on higher learning institutions in this country during the COVID-19 pandemic (Halim et al., 2021) may lead to compromised learning experiences. This is especially pertinent for SCD training, which involves clinical exposure to various groups of patients (Dougall et al., 2014). As clinical training was limited at that point of time, alternative teaching and learning approaches were designed to ensure students were exposed to issues in managing patients with special health care needs, which in this case was achieved by developing an innovative product to address patients’ difficulty in maintaining satisfactory oral health.

This study found that students demonstrated favorable responses toward acceptance of such an education intervention. With dental students experiencing physical and mental health deterioration with online distance learning during the COVID-19 pandemic (Lestari et al., 2022), it was encouraging to observe that students in this study found the learning activities interesting. They also opined that the educational method enhanced creativity and innovative skills, which are the highest level of learning domains (Bloom et al., 1964; Miller, 1990). Therefore, acceptance of this training approach has positive implications for its future development and implementation in the undergraduate dental curriculum.

Various benefits of this learning activity, as reported by students in this study, also further supports the use of innovation-integrated learning for teaching SCD. Participants cited improved knowledge and understanding of SCD, which has been reported as lacking among members of the dental fraternity around the world (Ahmad et al., 2015; Borromeo et al., 2018; Mandasari et al., 2021). They also opined that this learning intervention allowed integration of SCD theory and clinical practice, supporting its effectiveness, which is comparable to a similar approach previously undertaken via face-to-face learning (Ahmad et al., 2020).

The enhanced student attitude and interest in learning SCD reported in this study suggest a positive impact of learning that could potentially be translated into practice. Those with educational experience in SCD have been found to feel more comfortable and positive in managing patients with special health care needs (Mohamed Rohani et al., 2021). Dental practitioners who received exposure to SCD during university education were also more likely to have provided care to these patients, in comparison with their colleagues without such training experience (Alumran et al., 2018).

To further enhance students’ attitudes and interest in learning, the role of academicians as supervisors is integral. Academicians who play an effective role could influence students’ active engagement in learning, guide students in achieving learning goals, and enhance their overall learning experience (Sølvik & Glenna, 2021). It was evident in the present study that despite being conducted virtually, this learning activity was successful in encouraging engagement between students and supervisors. Some students also reported receiving assistance from their supervisors in gaining access to patients, whose feedback was sought for improvement of their products. This is an encouraging finding, given previous studies reporting negative implications of virtual and remote learning on student–teacher relationships (Halim et al., 2021). To ensure that all students receive quality supervision, the faculty must monitor the roles and engagement of every academician, especially during learning activities that involve remote delivery.

Besides improving student–teacher engagement, this learning activity was also beneficial in enhancing peer interactions and teamwork, while allowing sharing of experiences and ideas between students. Although social interaction and group dynamics in learning have been reportedly compromised during the COVID-19 pandemic (Walker & Koralesky, 2021), this education activity was perceived to have overcome this issue. Enhanced teamwork and communication brought about by this activity could provide opportunities for students to develop professionalism, which is integral to the practice of dentistry. Teamwork and effective communication are especially important in the practice of SCD, where multidisciplinary collaboration is mandatory in addressing patients’ complex health care needs (Glassman et al., 2016). The development of skills (e.g., independent information-gathering, critical thinking, problem-solving skills) reported by the study participants is also beneficial in preparing them as competent practitioners who can function at a high level of professionalism.

Challenges reported by students while undergoing this learning activity (technological difficulties and reduced physical access to purchase materials) demand the attention of faculty and supervisors to be more proactive in addressing the related issues. Nevertheless, with allowance of physical classes recently, this issue would not compromised students’ learning experience should the faculty decide to continue such this approach. Differing opinions about time constraints, heavy workloads, and increased financial demands expressed by students in this study require further investigation and planning by the faculty to ensure that the students receive adequate assistance to support their learning process. Feedback from patients regarding the developed innovative products could be gained through the conduct of proper research that assesses the effectiveness and benefit of the intervention. Consequently, students’ innovative ideas could be further developed for commercialisation, thus equipping them with entreperneurial skills, which is highly recommended to be taught in the higher education curriculum (Tuononen et al., 2022).

This study also indicates that integrating innovation in the teaching and learning of SCD supports many aspects of student learning that targets the recommended core contents as recommended in the IADH guidelines. These contents include 1) Identifying and addressing access and barriers to oral health faced by people with special needs (Domain 2), 2) Demonstrating appropriate communication skills when managing patients with special needs (Domain 4), 3) Understanding the impact of impairments, disabilities and systemic conditions on oral health and functions (Domain 5), as well as 4) Developing and applying effective methods when managing patient with special needs, including in planning oral health education program and designing interventions for clinical treatment (Domain 6) (Daly et al., 2014).

This study is limited by the non-participation of some students, whose responses were not included in the investigation. Nevertheless, the high response rate indicates the representativeness of the study population, hence supporting the validity of the study findings.

The results of this study provided important information for program developers, faculty, and the dental education fraternity at large. It identified the potential of innovation-integrated learning, for providing SCD training to undergraduate students, including in circumstances that may limit physical access to education.

V. CONCLUSION

Students’ acceptance of this teaching method proves it is sufficient and beneficial in improving their creativity and innovation, enhancing their knowledge and understanding of SCD, instilling interest and positive attitudes in learning SCD, and encouraging teamwork and effective communication between peers and supervisors. However, they provided differing opinions on the impact of the learning activity on their workload, time and financial demands, and exposure to SCD practice. Some faced technological challenges and reduced physical access to purchase materials.

Notes on Contributors

Aminda F. Omar was involved in literature search, data acquisitions and analysis and the manuscript preparation.

Tun Yasmin Iffah Mohd Suria Affandi was involved in data acquisitions and analysis.

Mohd Isyrafuddin Ismail was involved indata acquisition.

Mas S. Ahmad was involved in study conception and design, data analysis, manuscript editing and review.

Ilham W. Mokhtar was involved in the study conception and design and manuscript review.

All the authors approved the final version of the manuscript publication.

Ethical Approval

Ethics approval was obtained from the Research Ethics Committee (REC) Universiti Teknologi MARA, Malaysia in accordance with the Declaration of Helsinki (REC12/2021 [MR 1004]).

Data Availability

As per the requirement of the local ethics, data will be kept in an enclosed and dedicated facility in the faculty building.

Acknowledgement

The authors wish to thank the dental students from Universiti Teknologi MARA, Malaysia for participating in this study. The authors confirm that no financial support was received to fund this work. The authors also confirm they have no interests to declare.

Funding

No funding was provided for this study.

Declaration of Interest

There are no conflicts of interests.

References

Ahmad, M. S., Mokhtar, I. W., & Khan, N. L. A. (2020). Extramural oral health educational program involving individuals with disabilities: Impact on dental students’ professionalism. Journal of International Society of Preventive & Community Dentistry, 10(3), 323-328. https://doi.org/10.4103/jispcd.JISPCD_74_20

Ahmad, M. S., Radhi, D. S. M., Rusle, F. F., Zul, M. F., Jalaluddin, J., & Baharuddin, I. H. (2020). Photodentistry – An innovative approach to improving students’ empathy and learning experiences in comprehensive patient care. Journal of Dental Education, 84(11), 1219-1229. https://doi.org/10.1002/jdd.12295

Ahmad, M. S., Razak, I. A., & Borromeo, G. L. (2014). Undergraduate education in special needs dentistry in Malaysian and Australian dental schools. Journal of Dental Education, 78(8), 1154-1161.

Ahmad, M. S., Razak, I. A., & Borromeo, G. L. (2015). Special needs dentistry: Perception, attitudes and educational experience of Malaysian dental students. European Journal of Dental Education, 19(1), 44-52. https://doi.org/10.1111/eje.12101

Alumran, A., Almulhim, L., Almolhim, B., Bakodah, S., Aldossary, H., & Alakrawi, Z. (2018). Preparedness and willingness of dental care providers to treat patients with special needs. Clinical, Cosmetic and Investigational Dentistry, 10, 231-236. https://doi.org/10.2147/CCIDE.S178114

Amir, L. R., Tanti, I., Maharani, D. A., Wimardhani, Y. S., Julia, V., Sulijaya, B., & Puspitawati, R. (2020). Student perspective of classroom and distance learning during COVID-19 pandemic in the undergraduate dental study program Universitas Indonesia. BMC Medical Education, 20(1), Article 392. https://doi.org/10.1186/s12909-020-02312-0

Bloom, B. S., College, C. O., & Examiners, U. (1964). Taxonomy of educational objectives (Vol. 2). Longmans.

Borromeo, G. L., Ahmad, M. S., Buckley, S., Bozanic, M., Cao, A., Al-Dabbagh, M., & Athale, A. (2018). Perception of Special Needs Dentistry education and practice amongst Australian dental auxiliary students. European Journal of Dental Education, 22(3), e321-e326. https://doi.org/10.1111/eje.12296

Chang, T.-Y., Hsu, M.-L., Kwon, J.-S., Kusdhany, M. F. L. S., & Hong, G. (2021). Effect of online learning for dental education in Asia during the pandemic of COVID-19. Journal of Dental Sciences, 16(4), 1095-1101. https://doi.org/10.1016/j.jds.2021.06. 006

Daly, B., Boyle, C., Mc Pherson, A., & Thompson, S. (2014). Core content for a curriculum in Special Care Dentistry at the undergraduate level mapped to the learning outcomes for “Preparing for practice” (GDC-UK, 2011). British Society for Disability and Oral Health, March, 1–23.

Dougall, A., Thompson, S. A., Faulks, D., Ting, G., & Nunn, J. (2014). Guidance for the core content of a curriculum in Special Care Dentistry at the undergraduate level. European Journal of Dental Education, 18(1), 39-43. https://doi.org/10.1111/eje.12054

Faculty of Dentistry. (2022). DS240- Bachelor of Dental Surgery Program Structure. Retrieved from https://dentistry.uitm.edu.my/index.php/en/programme/programme-offered/2-uncategorised/67-ds240-programme-structure

Glassman, P., Harrington, M., Namakian, M., & Subar, P. (2016). Interprofessional collaboration in improving oral health for special populations. Dental Clinics of North America, 60(4), 843-855. https://doi.org/10.1016/j.cden.2016.05.004

Halim, M., Noorani, T., Karobari, M., & Kamaruddin, N. (2021). COVID-19 and dental education: A Malaysian perspective. Journal of International Oral Health, 13(3), 201-206. https://doi.org/10.4103/jioh.jioh_8_21

Lestari, W., Yazid, N. H., Azhar, Z. N., Ismail, A., & Sukotjo, C. (2022). Impact of COVID-19 on Malaysian dental students’ physical, mental, financial and academic concerns. BMC Oral Health, 22(1), Article 46. https://doi.org/10.1186/s12903-022-02081-w

Malaysian Dental Council. (2021). Code of Practice for Programme Accreditation – Undergraduate dental degree. (2019, effective, April 1, 2021). http://mdc.moh.gov.my/modules/mastop_publish/?tac=COPPA

Mandasari, M., Rahmayanti, F., Derbi, H., & Wimardhani, Y. S. (2021). Special care dentistry perception among dentists in Jakarta: An online survey study. PloS ONE, 16(4): e0249727. https://doi.org/10.1371/journal.pone.0249727

Miller, G. E. (1990). The assessment of clinical skills/competence /performance. Academic Medicine, 65(9), S63-S67. https://doi.org/10.1097/00001888-199009000-00045

Mohamed Rohani, M., Ahmad Fuad, N., Ahmad, M. S., & Esa, R. (2021). Impact of the special care dentistry education on Malaysian students’ attitudes, self-efficacy and intention to treat people with learning disability. European Journal of Dental Education, 26(4), 741-749. https://doi.org/10.1111/eje.12756

Mohamed Rohani, M., & Mohd Nor, N. A. (2021). Dental students’ perception on Disability Equality Training as part of the special care dentistry curriculum. Journal of Dental Education, 85(5), 690-698. https://doi.org/10.1002/jdd.12528

Naimie, Z., Ahmad, N., Shoaib, L., Safii, S., & Mohamed Rohani, M. (2020). Curriculum for Special Care Dentistry: Are we there yet? Journal of International Oral Health, 12(1), 1-7. https://doi.org/10.4103/jioh.jioh_146_19

Rowe, A. D., & Wood, L. N. (2008). Student perceptions and preferences for feedback. Asian Social Science, 4(3), 78-88. https://doi.org/10.5539/ass.v4n3p78

Royal Australasian College of Dental Surgeons. (2021). Specialist Dental Practice. https://racds.org/sdp/

Saleem, Z., Sethi, A., Hassan, M., & Wajahat, M. (2019). Assessments of dental students’ entrepreneurial self-efficacy to aid practice management. Health Professions Educator Journal, 2(2), 54-59. https://doi.org/10.53708/hpej.v2i2.105

Samat, M. F., Awang, N. A., Hussin, S. N. A., & Mat Nawi, F. A. (2020). Online distance learning amidst Covid-19 pandemic among university students. Asian Journal of University Education 16(3), 220-233. https://doi.org/10.24191/ajue.v16i3.9787

Sølvik, R. M., & Glenna, A. E. H. (2021). Teachers’ potential to promote students’ deeper learning in whole-class teaching: An observation study in Norwegian classrooms. Journal of Educational Change, 23, 343-369. https://doi.org/10.1007/s10833-021-09420-8

Tuononen, T., Hyytinen, H., Kleemola, K., Hailikari, T., Männikkö, I., & Toom, A. (2022). Systematic review of learning generic skills in higher education—Enhancing and impeding factors. Frontiers in Education, 7, 1-13. https://doi.org/10.3389/feduc.2022.885917

Walker, K. A., & Koralesky, K. E. (2021). Student and instructor perceptions of engagement after the rapid online transition of teaching due to COVID‐19. Natural Sciences Education, 50(1), e20038. https://doi.org/10.1002/nse2.20038

Wang, J., Peng, B., Zhou, H., & Zhang, J. H. (2020). Dentists’ entrepreneurial intention and associated factors in public hospitals in major cities in Guangdong (South China): A cross-sectional study. BMC Oral Health, 20, 334 https://doi.org/10.1186/s12903-020-01331-z

*Ilham Wan Mokhtar
Faculty of Dentistry, Universiti Teknologi MARA,
47000, Jln Hospital, Sungai Buloh
Selangor, Malaysia
Email: ilham@uitm.edu.my

Submitted: 30 January 2023
Accepted: 2 August 2023
Published online: 2 January, TAPS 2024, 9(1), 20-27
https://doi.org/10.29060/TAPS.2024-9-1/OA2999

Mitsumi Masuda, Naomi Kimura & Akemi Nakagaki

Graduate School of Nursing, Nagoya City University, Japan

Abstract

Introduction: The level of BLS achievement at the end of basic education is not clear, so to develop a BLS training program using QCPR Learner^Ⓡ and to verify its effectiveness.

Methods: A quantitative descriptive study design was used. The developed BLS program was implemented for 81 fourth-year students at the school of nursing. The program consisted of 2 minutes of training per student and real-time feedback by QCPR Learner^Ⓡ as well as feedback among students and from faculty members and was conducted twice for each student. Evaluation was conducted from three aspects: learner response and program evaluation both using questionnaires, learning achievement of QCPR score.

Results: The mean values of learners’ responses before and after the program were: attention, 4.32 before and 4.59 after; relevance, 4.48 before and 4.75 after; confidence, 2.32 before and 3.78 after; and satisfaction, 2.12 before and 3.41 after, which increased significantly after the program. The QCPR Learner^Ⓡ scores increased significantly from 74.08 for the first time to 86.76 for the second time. In the free response, some respondents stated that “visualisation of the procedure by the application improved my skills.”

Conclusion: The results of the evaluation from three aspects showed that this program was effective in improving students’ motivation to learn and skills.

Keywords: Basic Life Support, Real-Time Feedback, Basic Nursing Education, Simulation Training, QCPR Learner

Practice Highlights

Real-time feedback application was effective in increasing motivation to learn and BLS skills.
QCPR Learner^Ⓡ can be easily installed on the learner’s device, making it suitable for self-learning.

I. INTRODUCTION

A “chain of life support” is necessary to save the lives of individuals in life-threatening situations such as cardiac arrest or asphyxiation, or in imminent danger of such situations, and to lead them back to society (Japan Resuscitation Council, 2020). When a person is unresponsive, it is important to immediately report suspected cardiac arrest and provide basic life support (BLS), which involves a series of procedures to support breathing and circulation including cardiopulmonary resuscitation (CPR) with chest compressions and artificial respiration and the use of an automated external defibrillator (AED). Prompt initiation of CPR by bystanders is essential to improve the survival rate of individuals in cardiac arrest and prompt BLS plays a major role in their reintegration into society. The international consensus on CPR has recently been revised after five years (Nolan et al., 2015), with the updated Japan Resuscitation Council Resuscitation Guidelines published in 2020 (Japan Resuscitation Council, 2020). This guideline includes detailed guidance on compression depth and rhythm of cardiac massage, with improved quality of cardiac massage highlighted as particularly important in increasing survival following cardiac arrest. Effective education and training are important as quality CPR requires accurate compression depth, rhythm, and ventilation volume.

Studies of BLS simulations for nursing students have reported learning effects in students who had prior knowledge of BLS (Requena-Mullor et al, 2021). However, visual assessment is reported to be inaccurate in assessing quality of CPR (Brennan et al., 2016). Accordingly, the Resuscitation Guidelines 2020 include game-based learning, virtual reality, full mastery learning, iterative learning, and on-site simulation, which did not receive significant attention in the Resuscitation Guidelines 2015 (Japan Resuscitation Council, 2015). The new learning styles advocated by the revised Resuscitation Guidelines are expected to further promote the development and effectiveness of teaching materials. In educational settings, devices that can objectively evaluate CPR skills have been developed as a means of increasing educational effectiveness, with teaching materials that provide real-time feedback increasingly being used. In recent years, the use of feedback devices in BLS training for medical personnel has been shown to improve CPR skills (Abella et al., 2007; Lin et al., 2018; Otero-Agra et al., 2019; Semeraro et al., 2019); however, feedback devices for CPR have not previously been evaluated in prelicensure nursing students.

Accordingly, we focused on QCPR Little Anne, a cardiopulmonary resuscitation simulator that incorporates a feedback device into CPR training, and QCPR Learner^Ⓡ, a dedicated application that allows learners to check their performance. QCPR Learner^Ⓡ can be used in conjunction with QCPR Little Anne to obtain objective feedback on the quality of CPR from a smartphone or tablet PC. In the basic nursing curriculum, students are required to acquire BLS skills by the time they graduate from　the program. The present study therefore aimed to develop a BLS training program with real-time feedback incorporating QCPR Little Anne and QCPR Learner^Ⓡ to enhance emergency response skills in a basic nursing education program. The BLS training program was evaluated according to three aspects: learner response including satisfaction, learning achievement, and overall program evaluation. QCPR is not a skill that can be experienced routinely in clinical practice. In addition, for nursing students, it is a skill that they may not always be able to experience during clinical practice. Objective feedback on the quality of QCPR would help students to further improve their skills through repeated self-leaning.

II. METHODS

A. Program Overview

In the BLS training program, participants were required to study until they achieved a score of 100 on a test of items related to BLS in the online educational tool Nursing Skills as a prerequisite for program participation. This is because in simulation education, it is important to learn practically based on an understanding of knowledge. Those who fulfilled the conditions for participation participated in the program, and a pretest and simulation training assignments were conducted.

B. Study Participants

The present study comprised fourth-year undergraduate nursing students in the School of Nursing at Nagoya City University in Japan. All students consented to participation in the present study.

C. Training Program Development

The development process of the program used in the present study is described below. CPR is one of the essential skills for both nursing students and health care providers. In clinical settings, BLS training has been shown to be effective in allowing trainees to respond quickly in situations requiring BLS (Kuyt et al., 2021; Shin et al., 2015). However, the effectiveness of BLS education in basic nursing curricula has not been adequately examined despite BLS being a requirement for graduation. In recent years, teaching materials have been developed that allow real-time visualisation of performance using cardiopulmonary resuscitation simulators that incorporate feedback devices. As BLS education is required by nursing students before graduation, visualisation of skills through real-time feedback may represent a useful addition to BLS training.

The following learning objectives for the program were set with reference to the skills required of nursing students at graduation (Ministry of Health, Labor and Welfare, 2021) and the BLS provider course conducted by the ACLS Association of Japan (American Heart Association, 2016). Goal 1: Be able to perform an initial assessment of an injured or unwell individual following a witnessed collapse. Goal 2: Be able to request support from the emergency response system. Goal 3: Be able to immediately perform CPR. Goal 4: Be able to promptly use an automated external defibrillator (AED). The program focused on how to respond as a bystander following a witnessed collapse of a patient and included BLS training for in-hospital cardiac arrest as an algorithmic simulation that assumes an actual situation. During the program design, interviews were conducted with nurses with extensive BLS teaching experience and experts in simulation education to confirm the appropriateness and validity of the content. In addition, a pilot test was conducted by researchers and nursing faculty with experience in BLS education for nursing students.

The specific program flow was as follows. An orientation and pre-test were given to participants who met the prerequisites. The simulation used one QCPR Little Anne per group of three to four participants. Each facilitator was assigned one or two groups. After a 10-minute briefing, the participants were divided into groups. Group members took turns and each person performed two 5-minute simulations. Each simulation consisted of preparation, practical (two minutes), and feedback. QCPR Learner^Ⓡ real-time feedback consisted of overall score, compression depth, and ventilation rate (Laerdal, 2022). Performers were also advised of the observations of the group members and facilitators. A 20-minute debriefing was held after all participants had completed the simulation. The evaluation was conducted from three aspects: learner response including satisfaction and program evaluation, as Level 1 response of Kirkpatrick’s four-level evaluation method (Gagne et al., 2005), and learning achievement, as Level 2 learning.

D. Data Collection

1) Survey Contents:

As a pre-test, a test consisting of five questions was evaluated the learners’ knowledge before the program. The pre-test was created based on a test of items related to BLS in the online educational tool Nursing Skills, with correct answer scoring one point. Learner responses were assessed before and after the program using a self-administered, anonymous learner response sheet. The response sheet consisted of questions regarding attention, relevance, confidence, and satisfaction on a 5-point Likert scale with reference to previous research (Hirakawa et al., 2018) on the Kirkpatrick’s Four-Level Training Evaluation Model (Alzahrani, 2016) was originally developed. Participants were also given the opportunity to provide free-response statements. As a measure of learning achievement, the overall score for each applicant was used in the study analysis. A Class Evaluation Scale for Nursing Skills Practice (Mochizuki & Nagano, 1999) was administered after the program to allow evaluation of the overall program. The class evaluation scale is a scale consisting of 39 questions on six subscales.

2) Survey Method:

Prior to the start of the exercise, the pre-tasks were reviewed, and a pre-test and response sheet were administered. After the program, response sheets and class evaluation scales were distributed to all participants. The response sheets and class evaluation scales were collected anonymously. Data collection was conducted in September 2020.

E. Data Analysis

For quantitative data, the correspondence t-test was used to compare test scores and an independent t-test was used to compare data from learner response sheets. Descriptive statistics were used for program evaluations. For qualitative data, free-response statements and class evaluation scores were categorised to maintain semantic content. Codes, subcategories, and categories were checked between researchers to enhance the veracity of the analysis. Quantitative data were analysed using SPSS ver. 28.0.

III. RESULTS

A. Overview of Study Participants

A total of 81 students participated in the present study after provided informed consent. Valid responses were obtained from 78 students (96.2%) for the first response sheet, from 80 students (98.8%) for the second response sheet, and from 78 students (96.2%) for the overall class evaluation form. The mean score on pre-test was 4.57 ± 0.67.

B. Learner Reactions

Comparisons of scores from the pre- and post-program response sheets are shown in Table 1 (Masuda, 2023). Significantly increases in all four assessed areas were observed after the program: interest (t= −2.74; P = 0.01), relevance (t= −2.926; P = 0.00), confidence (t= −10.45; P = 0.00), and satisfaction (t=−8.12; p = 0.00). The four items on the response sheets were strongly and positively correlated with interest and satisfaction (pre: r = 0.51; post: r = 0.58) and confidence and satisfaction (pre: r = 0.87; post: r = 0.74) before and after the program, respectively.

	Pre-program group (n=78)		Post-Program group (n=80)		t	p
	M	SD	M	SD	t	p
Attention to learning BLS	4.32	.69	4.59	.52	-2.73	.01
Are relevance to learning BLS	4.48	.66	4.75	.49	-2.92	.00
I am confident in my BLS skills	2.32	1.02	3.78	.71	-10.45	.00
I am satisfied with my current BLS skills	2.12	1.08	3.41	.92	-8.12	.00

Table 1. Learners’ responses to BLS in the pre- and post-program

Independent t-test: M = mean, SD = standard deviation

The free-response statements were summarised resulting in 116 codes, 17 subcategories, and four categories. Responses related to subcategories and “categories” are shown below (Table 2: Masuda, 2023). The most frequent positive responses were those related to the effectiveness of QCPR Learner. Examples of statements representative of categories and subcategories are provided below.

Visualisation of the appropriateness of the technique clarified my issues, I can improve by checking my current level with the application and “Visualisation of the technique with the application improved my skills”. Comments related to the effectiveness of QCPR Learner informed the “I improved my skills through repetitive practice training” category consisting of subcategory statements such as I improved my skills through repetitive training, and my skills improved through practical training. The category “Learned the elements of techniques necessary for QCPR” consisted of statements informing subcategories such as using the back mask was difficult and I was able to learn a series of BLS techniques. The category “I realised that the effectiveness of BLS is enhanced through group cooperation” included the subcategory “I was able to enhance my skills with my group members, including encouragement and feedback from members”. Statements informing the “Increased motivation to learn” category included I enjoyed improving my skills, I am glad I participated, I gained confidence, and if something happens, I will be able to implement BLS.

Subcategory	Category
Visualising the appropriateness of the technique helped clarify issues	Skills improved through visualisation of techniques by application
Enjoyed learning while improving the accuracy of skills
The use of the app increased my motivation
The application allowed me to see my current level of skill so that I can improve
Practical training improved my skills	Skills improved through repetitive practice training
The content was designed for practical use
I was able to acquire skills through repetitive training
Skills improved through repetitive training
Correct use of BM is difficult	Learned the elements of techniques required for QCPR
Learned how to use the back mask
I was able to learn a series of BLS techniques
Realised that BLS requires physical strength
I was able to perform high-quality CPR by working together as a group.	Realised that BLS is more effective when working in a group
I was able to enhance my skills with my group members
I learned that BLS is more effective when we work together as a group
I was able to improve my skills in a fun way	Increased motivation to learn
If something happened, I would be able to perform it.
If something happens, I will be confused, but I will be able to cooperate with others
I gained confidence
I am glad to have participated

Table 2. Categories related to learner response

C. Learning Achievement

The second test scores on QCPR Learner were significantly higher than the first test scores (t = −4.78; P = 0.00) (Table3).

D. Program Evaluation

Regarding the class evaluation scale administered after the program, the subscale “Time allocation and difficulty of content” scored 59.49 ± 6.12, “Delivery and guidance/advice” scored 33.77± 2.21, “Use and innovation of teaching materials” scored 9.27 ± 1.02, “Demonstration” scored 26.21 ± 4.71, “Interaction among students” scored 9.64 ± 0.84, and “Attitude and response to students and exercises” scored 43.29 ± 3.30 (Table 4).

						n=78
	1st QCPR		2nd QCPR		t	P
	M	SD	M	SD	t	P
QCPR score	74.08	23.53	86.76	14.29	-4.78	0.00

Table 3. Pre- and post-QCPR score changes

Paired-samples t-test: M = mean, SD = standard deviation

		n=79
	M	SD
Time allocation and difficulty of the content	59.49	6.12
Delivery and guidance/advice	33.77	2.21
Use and innovation of teaching materials	9.27	1.02
Demonstration	26.21	4.71
Interaction among students	9.64	.84
Attitude and response to students and exercises	43.29	3.30

Table 4. Evaluation of a class evaluation scale for nursing skills practice

Subjects were asked to respond to an open-ended question regarding their evaluation of the program, resulting in 29 codes, 10 subcategories, and six categories. Responses related to subcategories and “categories” are shown below (Table 5). Statements related to evaluation of the program content and facilitators informed the category “The facilitator’s involvement was good” consisting of statements representative of the subcategories the entire program was clear and concise, and the facilitator created an easygoing atmosphere. The category “The program time and content were just right” consisted of subcategories informed by statements such as the time of the program was just right, and the amount of the program was just right. There were a few opinions regarding the content of the program and the instructors such as there were some problems with the application, more specific explanations would have been better, and there were too many hours spent on the exercise.

Subcategory	Category
The entire program was clear and concise	The facilitators involvement was good
The Facilitators created an easygoing atmosphere	The facilitators involvement was good
The time of the program was just right	The program time and content were just right
The amount of program was just right	The program time and content were just right
There were some problems with the application	There was a glitch in the application
More specific explanations would have been better	More specific explanations would have been better
There were too many hours spent on the exercise	More specific explanations would have been better
The time commitment was too much	The time commitment was too much

Table 5. Categories related to evaluation of the program

IV. DISCUSSION

A. Evaluating Programs from the Perspective of Learner Responses

The results of the response sheets administered before and after the program demonstrated a significant increase in all items. BLS skills are technical items in the nursing education curriculum designated by the Ministry of Health, Labor and Welfare (Ministry of Health, Labor and Welfare, 2021) that may be required in clinical practice. The implementation of simulations of potential future scenarios promotes student motivation (Hae-Kyoung, 2021). and simulations are known to be an effective learning method for putting theoretical knowledge into practice and reducing fear before encountering real situations (Carrero- Planells et al, 2021). The students were highly satisfied with the simulation in the present study as they had completed their clinical training, had an image of the clinical situation, and recognised that it was a necessary skill for post-graduates.

In addition, a correlation was observed between interest in BLS and satisfaction with BLS, and between confidence and satisfaction with BLS. The findings indicate that it is necessary to select subjects that are already interested in BLS and likely to have greater satisfaction with BLS training and to structure BLS programs in a way that leads to confidence and satisfaction with BLS skills.

Many participants in the present study gave positive evaluations of the program, such as “Visualisation of the technique by the application helped me improve my skills.” The effectiveness of visual feedback has been shown to be effective in improving skills in previous studies in other fields (Unell et al., 2021; Yamamoto et al., 2019). The QCPR Learner^Ⓡ used in this study also make it clear at a glance whether the current skill of the self is appropriate, due to is visualised the CPR situation in real time. For example, by following the application parameters while performing chest compressions, the learner was able to learn how to utilise his/her own body functions accurately while performing the CPR. Thus, we believe that the real-time visual feedback was a challenge for the learners and helped them to improve their skills.

The learners also realised that visualisation and repetitive training using the QCPR Learner^Ⓡ helped them learn the elements of the technique necessary for high-quality CPR and that the effectiveness of BLS was enhanced through group cooperation. Furthermore, this may have led to increased motivation to learn. The ARCS model (Keller, 2010), a model for improving motivation to learn proposed by John Keller in 1983 (Keller, 1987), describes the following actions that educators should take to improve and maintain learner motivation: attention, relevance, confidence, and satisfaction. From the components of the “Increased motivation to learn” category identified in the present study, the following subcategories were developed: I enjoyed improving my skills, if something happens, I will be able to implement BLS, I gained confidence, and I am glad I participated.

Considering the four aspects of the ARCS model, “Enjoyed” corresponds to attention, “Will be able to implement” to relevance, “Gained confidence” to confidence, and “Glad to have participated” to satisfaction. A key finding of the present study was the importance of simulating all four aspects of the ARCS model, as evidenced by the increased motivation to learn among participating nursing students.

B. Learning Effectiveness in terms of Learning Attainment

Our program used QCPR Little Anne and QCPR Learner^Ⓡ to improve BLS skills, with feedback given according to the application score after the first implementation. As a result, significantly higher scores were obtained in the second session. Previous studies have reported that visual feedback enables students to perform challenging procedures such as ensuring the correct depth of chest compressions (Baldi et al., 2017). In our program, students were able to visually perceive them own BLS skills as numerical values indicating weaknesses and areas for improvement, thereby leading to improved BLS skills.

C. Program Evaluation

Program feedback, time allocation and difficulty level of the content, communication of significance and objectives, guidance and advice, use and innovation of teaching materials, demonstration, interaction among students, and attitude and response to students and exercises were all highly evaluated in the present study. The program was conducted in small groups after the students watched a demonstration video with clearly stated objectives and goals. We believe this approach increased student motivation to learn BLS skills. The program also received positive feedback from the facilitators and group members during individual training and the facilitators provided timely advice, which led to skill improvement. These results indicate that the program provided appropriate conditions for the acquisition of BLS skills.

In the results of the free description, most of the participants were positive about the facilitator’s involvement and the duration and content of the program. On the other hand, some participants felt that the program was too time-consuming, albeit only slightly. This may be attributable to the extra time required to change clothes due to coronavirus precautions. In addition, some participants expressed a desire for more practice. The QCPR Learner^Ⓡ used in this program can be easily installed on the learner’s own device. We therefore believe that methods that allow self-study at any time may satisfy the desire to learn and lead to skill improvement. Regarding the device, one participant stated “There was a problem with the application”; however, this was expected and could be avoided as a spare device had been prepared in advance. We were able to confirm once again that preparing in advance for anticipated problems and responses to them is an important factor for success when using such devices.

V. CONCLUSION

The results of the present study demonstrate that visualisation of skill improvement using an application that provides real-time feedback was effective in increasing motivation to learn and improving BLS skills. This led some of the learners to request further training. Although it is possible to increase the training time per person by increasing the program duration or the number of simulations, this is limited by cost and manpower. Accordingly, it is necessary to establish a self-learning system that enables learning at any time.

Notes on Contributors

Mitsumi Masuda, Naomi Kimura, and Akemi Nakagaki contributed to all process of this research and read and approved the final manuscript.

Dr. Mitsumi Masuda, PhD, RN, is an associate professor at the Graduate School of Nursing, Nagoya City University. She reviewed the literature, designed the study, performed data collection, data analysis and wrote the manuscript.

Dr. Naomi Kimura, PhD, RN, MW, is an assistant professor at the Graduate School of Nursing, Nagoya City University. She developed the methodological framework for the study, performed data collection, data analysis and gave critical feedback to the writing of the manuscript.

Dr. Akemi Nakagaki, PhD, RN, MW, is an associate professor at the Graduate School of Nursing, Nagoya City University. She performed data collection and data analysis. All the authors have read and approved the final manuscript.

Ethical Approval

The present study was conducted with the approval of the Research Ethics Review Committee of Nagoya City University (approval number: 20011-3). The research subjects were informed orally and in writing of the purpose, methods, protection of personal information, respect for their free will, that the submitted assignments would be processed for research after class evaluation so that individuals would not be identified, and that they would not be involved in any individual class evaluation.

Data Availability

Datasets generated and/or analysed during the current study are available from the following URL:

https://doi.org/10.6084/m9.figshare.21918864.v1

Acknowledgement

We would like to thank all the nursing students who willingly cooperated in the present study.

Declaration of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interests.

Funding Statement

This study was completed without the support of the funding.

References

Abella, B. S., Elizabeth, D. P., Myklebust, H., Barry, A. M., O’Hearn, N., Vanden Hoek, T. L., & Becker, L. B. (2007). CPR quality improvement during in-hospital cardiac arrest using a real-time audiovisual feedback system. Resuscitation, 73(1), 54-61. https://doi.org/10.1016/j.resuscitation.2006.10.027

Alzahrani, A. R. (2016). Kirkpatrick’s four-level training evaluation model. International Journal of Scientific and Engineer ing Research, 7(5), 66-69.

American Heart Association. (2016). Basic life support instructor manual for instructor-led training. American Heart Association.

Baldi, M. E., Cornara, S., Contri, E., Epis, F., Fina, D., Zelaschi, B., Dossena, C., Fichtner, F., Tonani, M., Di Maggio, M., Zambaiti, E., & Somaschini, A. (2017). Real-time visual feedback during training improves laypersons’ CPR quality: A randomized controlled manikin study. Canadian Journal of Emergency Medi- cine, 19(6), 480-487. https://doi.org/10.1017/cem.2016.410

Brennan, E. E., McGraw, R. C., & Brooks, S. C. (2016). Accuracy of instructor assessment of chest compression quality during simulated resuscitation. Canadian Journal of Emergency Medicine, 18(4), 276-282. https://doi.org/10.1017/cem.2015.104

Carrero-Planells, A., Pol-Castaneda, S., Candida, M., Alamillos-Guardiola, M. C., Prieto-Alomar, M., Tomas-Sanchez, M., & Moreno-Mulet, C. (2021). Students and teachers’ satisfaction and perspectives on high-fidelity simulation for learning fundamental nursing procedures: A mixed-method study. Nurse Education Today, 104, Article 104981. https://doi.org/10.1016/j.nedt.2021.104981

Gagne, R. M., Wager, W. W., Golas, K. C., Keller, J. M., & Russell, J. D. (2005). Principles of instructional design (5th ed.). Cengage Learning.

Hae-Kyoung, S., & Son, H. K. (2021). The effects of simulation problem-based learning on the empathy, attitudes toward caring for the elderly, and team efficacy of undergraduate health profession students. International Journal of Environmental Research and Public Health, 18(18), 1-13. https://doi.org/10.3390/ijerph18189658

Hirakawa, Y., Yahiro, Y., Fukano, K., Nakamura, M., Maruyama, T., Usui, Y., & Aoki, N. (2018). Student’s evaluation on simulation learning in perioperative nursing. Journal of Japan Association for Simulation-Based Education in Healthcare Professionals, 6, 36-40.

Japan Resuscitation Council. (2015). JRC resuscitation guideline 2015. IGAKU-SHOIN Ltd.

Japan Resuscitation Council. (2020). JRC Resuscitation Guideline 2020. IGAKU-SHOIN Ltd.

Keller, J. M. (2010). Motivational design for learning and performance: The ARCS model approach (1st ed.). Springer.

Keller, J. M. (1987). Development and use of the ARCS model of instructional design. Journal of Instructional Development, 10(3), 2-10.

Kuyt, K., Mullen, M., Fullwood, C., Chang, T. P., Fenwick, J., Withey, V., & MacKinnon, R. J. (2021). The assessment of a manikin-based low-dose, high-frequency cardiac resuscitation quality improvement program in early UK adopter hospitals. Advances in Simulation, 6(14), 1-12. https://doi.org/10.1186/s410 77-021-00168

Laerdal. (2022). Products & services. Laerdal. Retrieved November 1, 2022, from https://laerdal.force.com/HelpCenter/s/ article/QCPR-App-feedback-symbols-and-scoring

Lin, Y., Cheng, A., Grant, V. J., Currie, G. R., & Hecker, K. G. (2018). Improving CPR quality with distributed practice and real-time feedback in pediatric healthcare providers – A randomized controlled trial. Resuscitation, 130, 6-12. https://doi.org/10.1016/ j.resuscitation.2018.06.025

Masuda, M. (2023). Real-time feedback in basic nursing education: Evaluation of basic life support training programs [Data set]. Figshare. https://doi.org/10.6084/m9.figshare. 2191886 4.v1

Mochizuki, M., & Nagano, M. (2015). Classroom process evaluation scale for nursing skills exercises. IGAKU Ltd.

Ministry of Health, Labour and Welfare. (2021). Operational guidelines for the technical intern training program relating to specific occupations and operations regarding the standards for the occupation ‘care worker’. Retrieved November 30, 2022, from https://www.jspcun.or.jp/wp/wp-content/uploads/2021/03/guideli ne_2020.11.26.pdf

Nolan, J. P., Hazinski, M. F., Aickin, R., Bhanji, F., Billi, J. E., Callaway, C. W., Castren, M., de Caen, A. R., Ferrer, J. M., Finn, J. C., Gent, L. M., Griffin, R. E., Iverson, S., Lang, E., Lim, S. H., Maconochie, I. K., Montgomery, W. H., Morley, P. T., Nadkarni, V. M., & Zideman, D. A. (2015). Part 1: Executive summary: 2015 international consensus on cardiopulmonary resuscitation and emergency cardiovascular care science with treatment recommendations. Resuscitation, 95, e1-31. https://doi.org/10.10 16/j.resuscitation.2015.07.039

Otero-Agra, M., Barcala-Furelos, R., Besada-Saavedra, I., Peixoto-Pino, L., Martiínez-Isasi, S., & Rodriíguez-Nunúñez, A. (2019). Let the kids play: Gamification as a CPR training methodology in secondary school students. A quasi-experimental manikin simulation study. Emergency Medicine Journal, 36(11), 653-659. https://doi.org/10.1136/emermed-2018-208108

Requena-Mullor, M. D. M., Alarcón-Rodríguez, R., Ventura-Miranda, M. I., & García-González, J. (2021). Effects of a clinical simulation course about basic life support on undergraduate nursing students’ learning. International Journal of Environmental Research and Public Health, 18(4), Article 1409. https://doi.org/10.3390/ijerph18041409

Semeraro, F., Frisoli, A., Loconsole, C., Mastronicola, N., Stroppa, F., Ristagno, G., Scapigliati, A., Marchetti, L., & Cerchaiari, E. (2019). Kids (learn how to) save lives in the school with the serious game Relive. Resuscitation, 36(11), 653-659. https://doi.org/10.1016/j.resuscitation.2017.04.038

Shin, N., Tonaka, K., Dosaka, A. (2015). Study of educational effects of BLS training for each number of attempts – from the result of analysis of a participant questionnaire. The Bulletin of Keio University Health Center, 33, 115-121.

Unell, A., Eisenstat, Z. M., Braun, A., Gandhi, A., Gilad‑Gutnick, S., Ben‑Ami, S., & Sinha, P. (2021). Influence of visual feedback persistence on visuo‑motor skill improvement. Scientific Reports, 11, Article 17347. https://doi.org/10.1038/s41598-021-9

Yamamoto, R., Akizuki, K., Kanai, Y., Nakano, W., Kobayashi, Y., Ohashi, Y. (2019). Differences in skill level influence the effects of visual feedback on motor learning. Journal of Physical Therapy Science, 31(11), 939–945. https://doi.org/10.1589/jpts.31.939

*Mitsumi Masuda
Nagoya City University,
1, Kawasumi, Mizuho-cho,
Mizuho-ku, Nagoya, 467-8601, Japan
+81-52-853-8063
E-mail: m.masuda@med.nagoya-cu.ac.jp

Submitted: 9 February 2023
Accepted: 15 May 2023
Published online: 3 October, TAPS 2023, 8(4), 23-35
https://doi.org/10.29060/TAPS.2023-8-4/OA3006

Sethapong Lertsakulbunlue¹, Kaophiphat Thammasoon², Kanlaya Jongcherdchootrakul³, Boonsub Sakboonyarat³ & Anupong Kantiwong¹

¹Department of Pharmacology, Phramongkutklao College of Medicine, Thailand; ²Department of Personnel Administration Division, Phramongkutklao College of Medicine, Thailand; ³Department of Military and Community Medicine, Phramongkutklao College of Medicine, Thailand

Abstract

Introduction: Although medical research (MR) is constantly promoted, a global deficit of medical researchers has been noted. We aimed to explore the relationship among practice, perceptions, attitudes, barriers and motivation toward MR and its impacts on MR publication.

Methods: A cross-sectional study included 262 senior medical students and interns. An electronic, standardised Likert scale questionnaire was used to collect the data. Binary logistic regression was used to determine the odds ratio between characteristics and MR publication. Confirmatory factor analysis was used to confirm the loading factor of each question, and structural equation modeling (SEM) was used to investigate the relationship between latent variables and MR publication.

Results: Cronbach’s alpha revealed a good internal reliability of 0.93. The accumulated grade point average did not differ between those who had published and those who had not. MR presentations were strongly associated with MR publication. SEM showed that attitudes (0.71, p<0.001) and perceptions (0.27, p<0.001) had a direct effect on practices. Practices (0.49, p<0.001) and attitudes (0.30, p<0.001) had a direct effect on motivation, while motivation had a total effect = 0.36, p<0.001 on MR publication through MR presentation as a mediator.

Conclusion: Positive attitudes and perceptions might lead to positivity in the intention to practice MR, which would lead to motivation and finally increase the odds of MR publication. Different approaches to promote excitement and perceptions in MR learning should be encouraged by teachers and faculty members.

Keywords: Medical Research, Students, Perceptions, Attitudes, Barriers, Motivation

Practice Highlights

Enjoyment and excitement should be promoted while learning medical research.
Medical research experiences enhanced publication, particularly medical research presentations.
Extracurricular medical research activities should be routinely promoted.
Facilitators in medical research might be tailor-made for each individual.
Regular meetings regarding medical research with mentors or role models should be held.

I. INTRODUCTION

Health-related research is constantly promoted and has gained great importance over time (Sobczuk et al., 2022). However, a global shortage of medical researchers was noted despite an increasing demand for them (Funston et al., 2016). For example, in the US, the proportion of medical researchers has declined from approximately 4.7% to 1.5% in the 1980s and 2014, respectively (Carberry et al., 2021; Davila, 2016; Puljak, 2007). Several barriers toward conducting medical research (MR) have been reported among undergrads and postgraduates. For instance, lack of allotted time, lack of physician engagement in research early during medical students’ training, and lack of mentoring and guidance (Bonilla-Escobar et al., 2017; El Achi et al., 2020; Habineza et al., 2019; Okoduwa et al., 2018). To resolve these problems, medical education has globally incorporated research methods and epidemiology into its curriculum (Carberry et al., 2021). Nevertheless, only a minority of medical students had reached the primary goal of research, namely publishing (Bonilla-Escobar et al., 2017; Carberry et al., 2021; Laidlaw et al., 2012).

Factors associated with MR publication have been identified. Students from highly ranked undergraduate institutions were more likely to achieve publication. Mentors also played an important role in increasing the likelihood of publication. For example, a student working with a mentor with a PhD degree or a mentor with prior publication(s) with prior mentee(s) increases the chance of achieving publication (Parker et al., 2021). Medical students participating in an extracurricular scientific activity, such as the Scientific Society of Medical Students, or who take a scientific writing skills course, were also associated with greater odds of producing a scientific publication (Valladares-Garrido et al., 2022).

One of the main reasons researchers conduct a study is because of what they believe (Lev et al., 2010). Attitudes toward and barriers against health research influence research success (Lev et al., 2010; Memarpour et al., 2015; Osman, 2016). Attitudes and motivations toward a particular type of study also showed a positive relationship with achievement (Ma & Xu, 2004; Özer, 2020; Simpson & Oliver, 1990). Furthermore, a theory of success describes perception leading to passion, and the continuation of passion leading to obsession, which drives an individual to succeed (Dange, 2016; Schellenberg et al., 2022).

Previously, several studies reported descriptive data on attitudes, practices, knowledge levels, perceptions, motivation and barriers involving research among medical and science students (AlGhamdi et al., 2014; Al-Shalawy & Haleem, 2015; Arif et al., 2018; El Achi et al., 2020; Habineza et al., 2019; Memarpour et al., 2015; Osman, 2016; Pallamparthy & Basavareddy, 2019). Even though these factors are known to affect one’s behaviour, to our knowledge, research on whether these factors are associated with research publications among medical students is scarce.

Without research, no breakthroughs can be achieved in managing disease. Therefore, strategies to enhance medical students’ appreciation of research and publication should be promoted. As a step toward this goal, our study aimed to determine the relations between MR practices, perceptions, attitudes, barriers and motivation toward medical research and their effect on MR publications among senior medical students and internists graduated from Phramongkutklao College of Medicine, Thailand. Furthermore, we also explore the differences in the characteristics and MR experiences of the participants between the publishing group and the non-publishing group. The goal involved influencing future research and actions to increase research publications among physicians in the country and contribute to medical practices.

II. METHODS

A. Study Design and Subjects

A cross-sectional study based on a self-administered survey was conducted at Phramongkutklao College of Medicine (PCM), Bangkok, Thailand. The survey was distributed among senior medical students, including fourth-, fifth-, and sixth-year students and internists who graduated from PCM. The total number of senior medical students and internists was 292 and approximately 250 interns, respectively. The curriculum at PCM is spread out over six years, with the first three being pre-clinical years spent studying basic science and the last three being clinical years spent developing clinical experiences. An introductory module about MR is mandatory in three years of the curriculum in the third, fourth, and sixth years of medical school. Firstly, third-year medical students learn the basics of MR, such as basic study designs focusing on quantitative methods, data analysis, and research proposal development. Furthermore, students learn about public health aspects, including community participation. Secondly, fourth-year medical students learn more about advanced study designs and are more focused on conducting a study and multivariate analysis. Fourth-year students were divided into eight groups of approximately twelve to conduct a community-based research proposal before finalizing their project as a report. Finally, sixth-year medical students were divided into pairs or a group of three to conduct medical research to improve medical care in a community hospital setting. Then the research is presented, and a manuscript report is submitted. However, an MR publication was not mandatory. After they graduated, the students were sent to both Thai Army hospitals and government hospitals run by the Ministry of Public Health to work as interns.

The present study included senior medical students and internists due to their similar MR experiences. First, both groups were enrolled within the same curriculum paradigm. Second, the MR presentation and publications are according to the willingness of the student, as MR publications and presentations are not mundane. Finally, almost all the published research among the population is from projects developed during their fourth and sixth years as medical students. Therefore, several projects were published during the internship.

B. Data Collection

We used an electronic standardised questionnaire, including six parts: short answer questions for obtaining demographic data; a 5-score Likert scale questionnaire on practice, perceptions, attitudes, barriers, and motivation toward MR. The questionnaire was translated using related published work that is relevant to this paper, as well as the investigators’ experience and context of PCM (El Achi et al., 2020; Funston et al., 2016; Habineza et al., 2019; Ichsan et al., 2018; Kamwendo, 2002; Okoduwa et al., 2018; Pallamparthy & Basavareddy, 2019). In addition, five expert professors examinedthe content validity and reliability of the questionnaire; pilot testing was conducted among 66 first year medical students and Cronbach’s alpha score ranged from 0.74-0.93.Then the questionnaire was launched in November and December of 2022 as a Google Form and advertised via social media to the study population. Information sheets, objectives, and methods of the study were provided on the first page of the Google Form, which participants were asked to read carefully before agreeing to participate. The questionnaire was then self-completed and took, on average, about 10–15 minutes to complete. The finalised Cronbach’s alphas were 0.83, 0.84, 0.74, 0.89, 0.88, and 0.93 for practices, perceptions, attitudes, barriers, motivation and overall questions, respectively.

Practice was defined as their willingness or intention to practice MR (El Achi et al., 2020). Perceptions are how the student perceives the importance of MR, while attitude is how they feel about conducting MR (El Achi et al., 2020; Funston et al., 2016). Barriers are defined as what the students perceive as being resistant to conducting MR; on the other hand, motivations are what they perceive as facilitating conducting MR (Habineza et al., 2019; Okoduwa et al., 2018).

C. Statistical Analysis

All data were downloaded from Google Forms, and data analyses were performed using StataCorp, 2021, Stata Statistical Software: Release 17. College Station, TX: StataCorp LLC. A frequency distribution of demographic characteristics was performed to describe the study subjects. Categorical data were presented as percentages, and continuous variables were presented as means and standard deviations (SD). Univariable and multivariable logistic regression analysis was used to determine the odds ratio (OR) and adjusted odds ratio (AOR) with a 95% confidence interval (CI) of the association between the characteristics and MR experiences of the participants and the MR publication. All statistical tests were two-sided, and a p-value less than 0.05 was considered statistically significant.

The structural equation modeling (SEM) using maximum likelihood extraction was done to find out how the latent variables, including practice, perceptions, attitudes, barriers, and motivation, were related and what effect it had on MR publications. The procedure is comprised of two steps. The first is validating the measurement model, which is carried out primarily using confirmatory factor analysis (CFA), and the other is fitting the structural model, which is achieved primarily through path analysis of latent variables. CFA was applied to demonstrate the impact of each question (observed variable) on different beliefs toward MR (latent variable) and presented as a lambda. Questions with a low factor loading (below 0.60) were excluded from the SEM. During the SEM construction, questions with factor loadings below 0.60 were also removed. In the final model, there are 17 observed variables included within the SEM. The samples/observed variable were 15.41, which is more than 10, indicating adequate sample size for SEM (Wolf et al., 2013). The SEM was carried out to investigate the relationship among latent variables and their impact on MR publication in our study population.The six following indices were used to evaluate model fit: (1) the chi-square test, χ2; (2) the chi-square test over degree of freedom (df), χ2/df (3) the comparative fit index, CFI; (4) the Tucker–Lewis index, TLI (5) the root-mean square error of approximation, RMSEA; and (6) the root-mean square residual, (SRMR). All these indices indicated a proposed fit for SEM data. A χ2/df lower than 2, CFI greater than 0.95, TLI greater than 0.95, RMSEA less than 0.06 and SRMR less than 0.06 each indicated a good fit between the data and the hypothesised model.

III. RESULTS

A. Characteristic of Participants

Table 1 demonstrates the characteristics of participants stratified by MR publishing. A total of 139 senior medical students and 123 interns participated in the survey. The response rate was 47.6% and 49.2% for senior medical students and interns, respectively. Over one-fifth (22.1%) of the participants had published MR and were mostly internists (81.0%). Approximately 60% of the participants were male, corresponding to an enrolment at PCM of 60 male and 40 female students. The accumulated grade point average (GPAX) was approximately the same at 3.4±0.3 among both published and those who had not published. Regarding, MR experience or roles served during medical student years, being a group leader (AOR: 2.12, 95% CI: 0.97 to 4.64, p=0.06) was associated with MR publishing. Finally, those having experience in MR presentation, whether oral or poster, and international or national presentation, were strongly associated with MR publishing, with adjusted odds ratios of 4.34 (p<0.001) shown in Table 2.

Characteristics	Non-Published	Published
Characteristics	n (% of 204)	n (% of 58)
Demographics
Sex
Male	119 (58.3)	37 (63.8)
Female	85 (41.7)	21 (36.2)
Educational level
Clinical year	128 (62.7)	11 (19.0)
Intern	76 (37.3)	47 (81.0)
Accumulated grade point average (GPAX)
Mean ± SD	3.4±0.3	3.4±0.3
Medical research participation
MR elective	59 (28.9)	15 (25.9)
Time spent on MR (hours/week)
<1	119 (58.3)	27 (46.6)
1-5	56 (27.5)	17 (29.3)
5-10	15 (7.4)	7 (12.1)
>10	14 (6.9)	7 (12.0)
Extra curriculum research activity	39 (19.1)	23 (39.7)
Academic club activity	30 (14.7)	13 (22.4)
MR experience during medical student
Group leader	45 (22.1)	24 (41.4)
Design MR	89 (43.6)	34 (58.6)
Proposal writing	142 (69.6)	45 (77.6)
Data enter	126 (51.5)	41 (70.7)
Data analysis	99 (48.5)	34 (58.6)
Literature review	125 (61.3)	42 (72.4)
Manuscript writing	76 (37.3)	33 (56.9)
None	7 (3.4)	2 (3.4)
National MR presentation
Oral presentation	23 (11.3)	22 (37.9)
Poster presentation	22 (10.8)	22 (37.9)
International MR presentation
Oral presentation	7 (3.4)	8 (13.8)
Poster presentation	9 (4.4)	15 (25.9)
Published national MR	0 (0)	29 (50.0)
Published international MR	0 (0)	37 (63.8)

MR: Medical Research

Table 1. Characteristics of participants stratified by medical research publishing experience (N=262)

Characteristics	Non-Published	Published	OR	95% CI	p-value	AOR	95% CI	p-value
Characteristics	n (% of 204)	n (% of 58)	OR	95% CI	p-value	AOR	95% CI	p-value
Sex
Male	119 (58.3)	37 (63.8)	ref			ref
Female	85 (41.7)	21 (36.2)	0.79	0.43-1.45	0.455	0.76	0.37-1.56	0.458
Educational level
Clinical year	128 (62.7)	11 (19.0)	ref			ref
Intern	76 (37.3)	47 (81.0)	3.09	1.74-5.50	<0.001	6.67	3.12-14.28	<0.001
Accumulated grade point average (GPAX)
Mean±SD	3.4±0.3	3.4±0.3	1.02	0.40-2.59	0.971	0.417	0.12-1.41	0.159
Extra curriculum research activity	39 (19.1)	23 (39.7)	2.78	1.48-5.23	0.002	1.47	0.62-3.46	0.379
MR experience during medical student
Group Leader	45 (22.1)	24 (41.4)	2.49	1.34-4.63	0.004	2.12	0.97-4.64	0.060
MR presentation
No	170 (83.3)	27 (46.6)	ref			ref
Yes	34 (16.7)	31 (53.5)	5.74	3.05-10.82	<0.001	4.34	1.99-9.47	<0.001

MR: Medical Research, OR: Odds Ratio, AOR: Adjusted Odds Ratio, CI: Confidence interval

Table 2. Univariable and multivariable analysis of characteristics and medical research experiences by medical research publishing experience (N=262)

B. Confirmatory Factor Analysis of Practices, Perceptions, Attitudes, Barriers and Motivation toward Medical Research

Table 3 demonstrates the confirmatory factor analysis (CFA) of practices, perceptions, attitudes, barriers and motivation Likert scores and MR publishing experience. In the practice section, all questions had a loading factor of approximately 0.80. The loading factors for perception range from 0.74 to 0.79, except for two questions: (1) research or publication should be mandatory and (2) research experience should be a criterion for residency training. For the attitude section, the CFA found that MR is exciting and MR is enjoyable, with high impacts of 0.89 and 0.87, respectively, followed by MR being valuable and essential for the medical profession, with loading factors of 0.58 and 0.54, respectively. However, for the questions where MR is complicated and time-consuming, the loading factor was relatively low, under 0.30. Regarding barriers, lack of exposure and opportunities, training and support, mentoring and guidance, and lack of personal knowledge of the research process all had a high loading factor of over 0.70. Lack of statistical support, funding, and lack of rewards or motivations had relatively lower loading factors between 0.60 and 0.69. For motivation, pursuit of further education, pursuit of personal interest, improving their potential in research skills, having mentor guidance/role model and to be a part of solving medical problems in society had high loading factors over 0.70.

Table 3. Confirmatory factor analysis of practice, perception, attitude, barrier and motivation Likert-score and medical research publishing experience

C. SEM of Practices, Perceptions, Attitudes, Barriers and Motivation and MR Publishing

The SEM is developed from five latent variables, leading to the outcome, including medical research presentation and publication (Figure 1). We found that perception has a direct effect on both practices (0.27, p<0.001) and motivation (0.12, p= 0.087). Perceptions and attitudes also correlated (0.57, p<0.001). Practices and attitudes have a direct effect on motivation, 0.49 and 0.30, respectively (p<0.001 for both). The indirect effect of attitudes through practices on motivation was 0.71 * 0.49 = 0.35, all coefficients p<0.001. Practices also exhibited a direct negative effect on barriers (-0.13, p= 0.072). Regarding our primary outcome, both motivation and, surprisingly, barriers also revealed a positive direct effect on MR publishing (0.28, p<0.001 for motivation and 0.15, p= 0.014 for barriers). The MR presentation experience also showed a direct effect on MR publication (0.26, p<0.001). Furthermore, MR presentation also acted as a mediator for motivation, with an indirect effect of 0.08 and a total effect of motivation on MR publication of 0.36. The SEM model provided a good fit for the data (χ²/df= 1.67, CFI= 0.96, TLI= 0.95, RMSEA= 0.05, SRMR= 0.05).

*= P<0.05

Figure 1. SEM of practices, perceptions, attitudes, barriers and motivation and MR publishing

IV. DISCUSSION

We successfully enrolled 139 PCM senior medical students, and 123 interns graduated from PCM. This study is the first to formulate a SEM on the relationship between practices, perceptions, attitudes, motivation and barriers to MR publication and presentation. We found that attitudes, practices and motivation significantly contribute to MR publication and presentation. The roles and experiences that medical students have in medical research during their medical student years are also important to the success of MR publications. However, because our population only includes those who have studied or are studying at PCM, additional external validation may be required.

This study described how baseline characteristics and MR experiences were associated with MR publication. Unsurprisingly, a higher proportion of interns had published MR. GPAX, on the other hand, were not associated with MR publication, which is a common factor in a relative study with positive associations toward perceptions, attitudes and practices (El Achi et al., 2020). A large study in China also reported that research engagement was associated with improving overall learning outcomes (Zhang et al., 2022). This controversy may be explained by two reasons. First, the time required for developing MR and publications is large and might interfere with normal curriculum activity. A study in Colombia noted that their students believed that they could obtain higher GPAs if they were not involved in research (Bonilla-Escobar et al., 2017). The latter is that MR skills and academic skills might not completely overlap. While MR engagement might improve science, scholarship, and professionalism, other domains, such as clinical practice, require more time and effort to learn (Zhang et al., 2022).

MR experiences and roles during MR learning also play an important part in MR publication. Experience in MR presentation was strongly associated with MR publication. This may be partially explained by the student’s readiness before the research presentation; students need to be highly knowledgeable about their own research, and knowledge is a key to success and behavioural change (Bettinghaus, 1986; Pengpid et al., 2016). Furthermore, presentation requires planning, preparation, creating visual aids, and practicing one’s presentation skills. The effort and time spent on this process might be motivation to achieve a higher goal, namely publication. Our study demonstrates that MR presentations, whether nationally, internationally, oral, or poster, are highly associated with MR publication. Thus, MR presentations should be promoted.

Those who had been group leaders had a higher chance of MR publication. During the PCM curriculum, the group leader for MR conduct was never assigned and was elected in each group. Northouse mentioned two forms of leadership: assigned and emergent. Those assigned leadership positions were given the role of group leader. When an individual is perceived as one of the most influential members of a group or organization, that person is exhibiting emergent leadership (Northouse, 2021). Medical leadership development was seen to improve outcomes at the individual, organizational and clinical levels (Lyons et al., 2021). Several leadership training programs in medicine and clinical practices were widely visible. However, to our knowledge, no curriculum focusing on medical research leadership was available.

Perception is described as a method for identifying and interpreting the environment and the meaning of sensual motivations. Cognition may influence perception, which can also occur subconsciously and without cognizance (Saini et al., 2020). Some literature has been carried out showing perceptions, attitudes, and motivation toward research among medical students, in which those with positive perceptions mostly had positive attitudes and motivation (AlGhamdi et al., 2014; El Achi et al., 2020; Osman, 2016). These were similar to our research showing that perception impacts positively on practices and motivation.

In line with the current study, the theory of planned behaviour describes that one’s attitude and how they perceive the behaviour directly affect their intention to perform the behaviour (Bosnjak et al., 2020). The SEM also demonstrated that perception and attitude directly affect the population’s intention to conduct MR (practice). Moreover, a study in Turkey on the predictiveness of attitude and motivation on achievement (vocational English course scores) found a significant positive relationship between attitude and motivation, attitude and achievement, and motivation and achievement (Özer, 2020). Therefore, a positive attitude may positively impact their motivation and their willingness to practice MR. The feeling that MR is exciting and enjoyable had the greatest impact on the attitude domain. Hence, activities that increase the excitement and joy of conducting MR should be encouraged.

The main factors impacting MR publication are motivation, practices, attitudes and perceived barriers. Based on the health-belief model (HBM), providing motivation as needed might help students overcome the triggers of MR barriers so that correct thoughts and perceptions about MR will arise. Thus, techniques derived from motivational interviews might be a useful option for encouraging students toward MR publication (Tober, 2013). One of the most important motivations in our study is having a mentor or role model, and the lack of a mentor constitutes a high-impact barrier. Therefore, mentors should play an important role in guiding their students toward research success. Based on motivational interview techniques, active listening might be the key skill for mentors to better understand their students’ motivations and empower them toward success (Rollnick et al., 2010).

Surprisingly, our study showed that perceived barriers had a positive impact on MR publication. The perception of a barrier greatly influences the likelihood of an individual’s uptake of that behaviour (Becker & Maiman, 1975). Usually, a barrier is a resistance to achieving a goal, which negatively affects achievement. However, the barriers included in our study consisted of a lack of exposure and opportunities, training and support, mentoring and guidance, and statistical support. As a result, those who had not yet published any medical research may not have had the prestige of encountering these barriers, which is why they are perceived as insignificant.

The practice domain included three questions: (1) willingness to participate in any research-related task; (2) willingness to spend more than two months on a research project; and (3) willingness to devote the same amount of time to medical research as they did to their university studies, indicating a willingness to practice medical research. Willingness is the quality or state of being prepared to accomplish something. A study in China about speaking English demonstrated that the willingness to communicate is directly related to motivation and mastery approach (Turner et al., 2021). Furthermore, passion and obsession are what drive an individual to succeed (Dange, 2016).

In view of all the factors presented, mentors have an important role to play in guiding and facilitating the students’ acquisition of adequate experience in medical research during their medical school years. A good extracurricular MR learning environment might be needed to ensure statistical confidence and exposure to conducting research. Actively listening to students and empowering and motivating them to break through barriers may result in successful MR publications. In addition, a different approach to learning MR might be needed to promote attitudes, perceptions and motivations toward MR. According to Self-Determination Theory (SDT), when students perceive that the primary purpose of learning is to obtain external rewards, such as exam grades, they may perform less well due to a detrimental effect on their intrinsic motivation (Deci et al., 1999).

SDT revealed that three basic needs must be fulfilled to empower one’s attitude and motivation, including autonomy, competence, and a feeling of belonging (Deci et al., 1999). To promote these basic needs, faculty members could provide extracurricular research time, give choice to research topics and mentors, and hire students to be research assistants, if possible, to promote their autonomy (Rosenkranz et al., 2015). Competence could be enhanced by early research skills introduction and practical training (Rosenkranz et al., 2015). Research mentors may play a crucial role in fostering a sense of belonging toward MR by promoting well-functioning group work through guidance and evaluation (Dorrance et al., 2008). Furthermore, portfolios, logbooks and reflective journals are particularly useful to improve the students’ attitude and motivation (Taylor & Hamdy, 2013). Constant positive feedback from mentors during MR activities is encouraged to improve the learning environment (Peifer et al., 2020). A different approach to learning MR might also benefits the MR learning environment, for example, through game-based learning and other collaborative learning models (Blakely et al., 2009).

The present study encountered several limitations. First, the study included only senior medical students and interns who graduated from PCM, so the model might not be accurately applied to other universities with different curricula and further validation might be needed. Even though most participants who had published a MR were interns (81%), the medical research published was composed while they were medical students. Second, the study was a cross-sectional study, and causal relationships were unavailable. However, according to PCM curricula, for most participants, the MR presentation and their experience with medical research came before the MR publication. Furthermore, personal beliefs change over time, and recall bias might have affected the study results. (Seitz et al., 2017). The beliefs elicited by the questionnaire comprised the participants’ current beliefs, rather than beliefs formed during their participation in medical research publications. As a result, our study investigated only the participants’ current beliefs and their impact on the publication of medical research. A further prospective cohort or qualitative study on whether the students’ current beliefs toward MR are related to successful MR publication is encouraged. Finally, because only participants who volunteered to take part in the study were included, selection bias may also be a significant limitation of this study. Our study had considerable strengths, there had been no reports describing practices, perceptions, attitudes, motivation and barriers toward medical research. However, this is the first study to formulate a SEM model displaying factors related to MR publication.

V. CONCLUSION

Medical research experience and positive practices or willingness, perception, attitude, and motivation in medical research might pave the road to a successful MR publication. Medical research experience and extracurricular activities should be supported by both teachers and faculties through active policies. A different approach to medical research learning might also be needed to promote enjoyment and excitement. Finally, external validation needs to be explored to generalise the model.

Notes on Contributors

SL reviewed the literature, designed the study, collected the data, data analysis and wrote the manuscript. KT, KJ and BS collected the data, developed the methodology framework and developed the manuscript. AK reviewed the literature, designed the study, data analysis and wrote the first draft.

Ethical Approval

The study was approved by the Medical Department Ethics Review Committee for Research in Human Subjects, Institutional Review Board, Royal Thai Army (Approval no. S060q/65_Exp), according to the international guidelines including the Declaration of Helsinki, the Belmont Report, CIOMS Guidelines, and the International Conference on Harmonization of Technical Requirements for Registration of Pharmaceuticals for Human Use – Good Clinical Practice (ICH-GCP). A documentation of informed consent was used, and was granted permission by the Institutional Review Board, RTA Medical Department.

Data Availability

The datasets used and/or analysed during the current are available from https://doi.org/10.6084/m9.figshare.22128725

Acknowledgement

We thank professors Mathirut Mungthin, Ram Rangsin Panadda Hattachote, Phunlerd Piyaraj and Picha Suwannahitatorn for validating our questionnaire and providing support. This work would not have been possible without the active support of Phramongkutklao College of Medicine and its academic leaders.

Funding

The authors report that there is no funding associated with the work featured in this article.

Declaration of Interest

The authors declare they have no competing interests.

References

AlGhamdi, K. M., Moussa, N. A., AlEssa, D. S., AlOthimeen, N., & Al–Saud, A. S. (2014). Perceptions, attitudes and practices toward research among senior medical students. Saudi Pharma- ceutical Journal, 22(2), 113–117. https://doi.org/10.1016/j.jsps.2013.02.006

Al–Shalawy, F. A.-N., & Haleem, A. (2015). Knowledge, attitudes and perceived barriers towards scientific research among undergraduate health sciences students in the central province of Saudi Arabia. Education in Medicine Journal, 7(1), e16-e21. https://doi.org/10.5959/eimj.v7i1.266

Arif, A., Siddiqui, M. A., Aziz, K., Shahid, G., Shiekh, A., & Fahim, M.F. (2018). Perception towards research among undergraduate physical therapy students. Biometrics & Biostatistics International Journal, 7(3), 171–175. https://doi.org/10.15406/bbij.2018.07.00206

Becker, M. H., & Maiman, L. A. (1975). Sociobehavioral determinants of compliance with health and medical care recommendations. Medical Care, 13(1), 10–24. https://doi.org/10.1097/00005650–197501000–00002

Bettinghaus, E. P. (1986). Health promotion and the knowledge–attitude–behavior continuum. Preventive Medicine, 15(5), 475–491. https://doi.org/10.1016/0091–7435(86)90025–3

Blakely, G., Skirton, H., Cooper, S., Allum, P., & Nelmes, P. (2009). Educational gaming in the health sciences: Systematic review. Journal of Advanced Nursing, 65(2), 259–269. https://doi.org/10.1111/j.1365-2648.2008.04843.x

Bonilla–Escobar, F. J., Bonilla–Velez, J., Tobón–García, D., & Ángel–Isaza, A. M. (2017). Medical student researchers in Colombia and associated factors with publication: A cross–sectional study. BMC Medical Education, 17(1), Article 254. https://doi.org/10.1186/s12909–017–1087–9

Bosnjak, M., Ajzen, I., & Schmidt, P. (2020). The theory of planned behavior: Selected recent advances and applications. Europe’s Journal of Psychology, 16(3), 352–356. https://doi.org/10.5964/ejop.v16i3.3107

Carberry, C., McCombe, G., Tobin, H., Stokes, D., Last, J., Bury, G., & Cullen, W. (2021). Curriculum initiatives to enhance research skills acquisition by medical students: A scoping review. BMC Medical Education, 21(1), Article 312. https://doi.org/10.1186/s12909–021–02754–0

Dange, J. K. (2016). Perception, passion and obsession: The three elements of theory of success. International Journal of Advanced Education and Research, 1(7), 1–3.

Davila, J. R. (2016). The physician–scientist: Past trends and future directions. Michigan Journal of Medicine, 1(1), 66-73. https://doi.org/10.3998/mjm.13761231.0001.112

Deci, E. L., Koestner, R., & Ryan, R. M. (1999). A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation. Psychological Bulletin, 125(6), 627–668. https://doi.org/10.1037/0033-2909.125.6.627

Dorrance, K. A., Denton, G. D., Proemba, J., La Rochelle, J., Nasir, J., Argyros, G., & Durning, S. J. (2008). An internal medicine interest group research program can improve scholarly productivity of medical students and foster mentoring relationships with internists. Teaching and Learning in Medicine,20(2), 163–167. https://doi.org/10.1080/10401330801991857

El Achi, D., Al Hakim, L., Makki, M., Mokaddem, M., Khalil, P. A., Kaafarani, B. R., & Tamim, H. (2020). Perception, attitude, practice and barriers towards medical research among undergraduate students. BMC Medical Education, 20(1), Article 195. https://doi.org/10.1186/s12909–020–02104–6

Funston, G., Piper, R. J., Connell, C., Foden, P., Young, A. M. H., & O’Neill, P. (2016). Medical student perceptions of research and research–orientated careers: An international questionnaire study. Medical Teacher, 38(10), 1041–1048. https://doi.org/10.3109/0142159X.2016.1150981

Habineza, H., Nsanzabaganwa, C., Nyirimanzi, N., Umuhoza, C., Cartledge, K., Conard, C., & Cartledge, P. (2019). Perceived attitudes of the importance and barriers to research amongst Rwandan interns and pediatric residents – A cross–sectional study. BMC Medical Education, 19(1), Article 4. https://doi.org/10.1186/s12909–018–1425–6

Ichsan, I., Wahyuniati, N., McKee, R., Lobo, L., Lancaster, K., & Redwood–Campbell, L. (2018). Attitudes, barriers, and enablers towards conducting primary care research in Banda Aceh, Indonesia: A qualitative research study. Asia Pacific Family Medicine, 17(1), Article 8. https://doi.org/10.1186/s12930–018–0045–y

Kamwendo, K. (2002). What do Swedish physiotherapists feel about research? A survey of perceptions, attitudes, intentions and engagement. Physiotherapy Research International, 7(1), 23–34. https://doi.org/10.1002/pri.238

Laidlaw, A., Aiton, J., Struthers, J., & Guild, S. (2012). Developing research skills in medical students: AMEE Guide No. 69. Medical Teacher, 34(9), 754–771. https://doi.org/10.3109/0142159X.2012.704438

Lev, E. L., Kolassa, J., & Bakken, L. L. (2010). Faculty mentors’ and students’ perceptions of students’ research self-efficacy. Nurse Education Today, 30(2), 169–174. https://doi.org/10.1016/j.nedt.2009.07.007

Lyons, O., George, R., Galante, J. R., Mafi, A., Fordwoh, T., Frich, J., & Geerts, J. M. (2021). Evidence–based medical leadership development: A systematic review. BMJ Leader, 5(3), 206–213. https://doi.org/10.1136/leader–2020–000360

Ma, X., & Xu, J. (2004). Determining the causal ordering between attitude toward mathematics and achievement in mathematics. American Journal of Education, 110(3), 256–280. https://doi.org/10.1086/383074

Memarpour, M., Fard, A. P., & Ghasemi, R. (2015). Evaluation of attitude to, knowledge of and barriers toward research among medical science students. Asia Pacific Family Medicine, 14(1), Article 1. https://doi.org/10.1186/s12930–015–0019–2

Northouse, P. G. (2021). Leadership: Theory and practice (9^th ed.). SAGE publications.

Okoduwa, S. I. R., Abe, J. O., Samuel, B. I., Chris, A. O., Oladimeji, R. A., Idowu, O. O., & Okoduwa, U. J. (2018). Attitudes, perceptions, and barriers to research and publishing among research and teaching staff in a Nigerian research institute. Frontiers in Research Metrics and Analytics, 3, Article 26. https://doi.org/10.3389/frma.2018.00026

Osman, T. (2016). Medical students’ perceptions towards research at a Sudanese University. BMC Medical Education, 16(1), Article 253. https://doi.org/10.1186/s12909–016–0776–0

Özer, S. (2020). The predictiveness of attitude and motivation on achievement in ESP: The mediating role of anxiety. International Journal of Curriculum and Instruction, 12(2), 25-41.

Pallamparthy, S., & Basavareddy, A. (2019). Knowledge, attitude, practice, and barriers toward research among medical students: A cross–sectional questionnaire–based survey. Perspectives in Clinical Research, 10(2), 73.

Parker, S. M., Vona-Davis, L. C., & Mattes, M. D. (2021). Factors predictive of publication among medical students participating in school-sponsored research programs. Cureus, 13(9), e18176. https://doi.org/10.7759/cureus.18176

Peifer, C., Schönfeld, P., Wolters, G., Aust, F., & Margraf, J. (2020). Well done! Effects of positive feedback on perceived self-efficacy, flow and performance in a mental arithmetic task. Frontiers in Psychology,11, Article 1008. https://doi.org/10.3389/fpsyg.2020.01008

Pengpid, S., Peltzer, K., Puckpinyo, A., Tiraphat, S., Viripiromgool, S., Apidechkul, T., Sathirapanya, C., Leethongdee, S., Chompikul, J., & Mongkolchati, A. (2016). Knowledge, attitudes, and practices about tuberculosis and choice of communication channels in Thailand. Journal of Infection in Developing Countries, 10(7), 694-703. https://doi.org/10.3855/jidc.6963

Puljak, L. (2007). An overlooked source of physician–scientists. Journal of Investigative Medicine, 55(8), 402–405. https://doi.org/10.2310/6650.2007.00029

Rollnick, S., Butler, C. C., Kinnersley, P., Gregory, J., & Mash, B. (2010). Motivational interviewing. BMJ, 340, c1900. https://doi.org/10.1136/bmj.c1900

Rosenkranz, S. K., Wang, S., & Hu, W. (2015). Motivating medical students to do research: A mixed methods study using self-determination theory. BMC Medical Education, 15, Article 95. https://doi.org/10.1186/s12909-015-0379-1

Saini, M., Kumar, A., & Kaur, G. (2020). Research perception, motivation and attitude among undergraduate students: A factor analysis approach. Procedia Computer Science, 167, 185–192. https://doi.org/10.1016/j.procs.2020.03.210

Schellenberg, B. J. I., Gaudreau, P., & Bailis, D. S. (2022). Lay theories of obsessive passion and performance: It all depends on the bottom line. Personality and Individual Differences, 190, Article 111528. https://doi.org/10.1016/j.paid.2022.111528

Seitz, R. J., Paloutzian, R. F., & Angel, H.-F. (2017). Processes of believing: Where do they come from? What are they good for? F1000Research, 5, Article 2573. https://doi.org/10.12688/f1000research.9773.2

Simpson, R. D., & Oliver, J. S. (1990). A summary of major influences on attitude toward and achievement in science among adolescent students. Science Education, 74(1), 1–18. https://doi.org/10.1002/sce.3730740102

Sobczuk, P., Dziedziak, J., Bierezowicz, N., Kiziak, M., Znajdek, Z., Puchalska, L., Mirowska–Guzel, D., & Cudnoch–Jędrzejewska, A. (2022). Are medical students interested in research? – Students’ attitudes towards research. Annals of Medicine, 54(1), 1538–1547. https://doi.org/10.1080/07853890.2022.2076900

Taylor, D. C. M., & Hamdy, H. (2013). Adult learning theories: Implications for learning and teaching in medical education: AMEE Guide No. 83. Medical Teacher, 35(11), e1561–e1572. https://doi.org/10.3109/0142159X.2013.828153

Tober, G. (2013). Motivational interviewing: Helping people change. Alcohol and Alcoholism, 48(3), 376–377. https://doi.org/10.1093/alcalc/agt010

Turner, J. E., Li, B., & Wei, M. (2021). Exploring effects of culture on students’ achievement motives and goals, self–efficacy, and willingness for public performances: The case of Chinese students’ speaking English in class. Learning and Individual Differences, 85, Article 101943. https://doi.org/10.1016/j.lindif.2020.101943

Valladares–Garrido, M. J., Mejia, C. R., Rojas–Alvarado, A. B., Araujo–Chumacero, M. M., Córdova–Agurto, J. S., Fiestas, J., Rojas–Vilar, F. J., & Culquichicón, C. (2022). Factors associated with producing a scientific publication during medical training: Evidence from a cross–sectional study of 40 medical schools in Latin America. F1000Research, 9, Article 1365. https://doi.org/10.12688/f1000research.26596.2

Wolf, E. J., Harrington, K. M., Clark, S. L., & Miller, M. W. (2013). Sample size requirements for structural equation models: An evaluation of power, bias, and solution propriety. Educational and Psychological Measurement, 76(6), 913–934. https://doi.org/10.1177/0013164413495237

Zhang, G., Wu, H., Xie, A., & Cheng, H. (2022). The association between medical student research engagement with learning outcomes. Medical Education Online, 27(1), Article 2100039. https://doi.org/10.1080/10872981.2022.2100039

*Anupong Kantiwong
317 Ratchawithi Rd,
Thung Phaya Thai,
Ratchathewi, Bangkok 10400
+66909838338
E-mail: anupongpcm31@gmail.com

Submitted: 2 December 2022
Accepted: 24 July 2023
Published online: 3 October, TAPS 2023, 8(4), 13-22
https://doi.org/10.29060/TAPS.2023-8-4/OA3093

Julie Yun Chen^1,2, Tai Pong Lam¹, Ivan Fan Ngai Hung³, Albert Chi Yan Chan⁴, Weng-Yee Chin¹, Christopher See⁵ & Joyce Pui Yan Tsang¹

¹Department of Family Medicine and Primary Care, University of Hong Kong, Hong Kong; ²Bau Institute of Medical and Health Sciences Education, University of Hong Kong, Hong Kong; ³Department of Medicine, University of Hong Kong, Hong Kong; ⁴Department of Surgery, University of Hong Kong, Hong Kong; ⁵School of Biomedical Sciences, The Chinese University of Hong Kong, Hong Kong

Abstract

Introduction: Medical students have long provided informal, structured academic support for their peers in parallel with the institution’s formal curriculum, demonstrating a high degree of motivation and engagement for peer teaching. This qualitative descriptive study aimed to examine the perspectives of participants in a pilot peer teaching programme on the effectiveness and feasibility of adapting existing student-initiated peer bedside teaching into formal bedside teaching.

Methods: Study participants were senior medical students who were already providing self-initiated peer-led bedside clinical teaching, clinicians who co-taught bedside clinical skills teaching sessions with the peer teachers and junior students allocated to the bedside teaching sessions led by peer teachers. Qualitative data were gathered via evaluation form, peer teacher and clinician interviews, as well as the observational field notes made by the research assistant who attended the teaching sessions as an independent observer. Additionally, a single Likert-scale question on the evaluation form was used to rate teaching effectiveness.

Results: All three peer teachers, three clinicians and 12 students completed the interviews and/or questionnaires. The main themes identified were teaching effectiveness, teaching competency and feasibility. Teaching effectiveness related to the creation of a positive learning environment and a tailored approach. Teaching competency reflected confidence or doubts about peer-teaching, and feasibility subthemes comprised barriers and facilitators.

Conclusion: Students perceived peer teaching effectiveness to be comparable to clinicians’ teaching. Clinical peer teaching in the formal curriculum may be most feasible in a hybrid curriculum that includes both peer teaching and clinician-led teaching with structured training and coordinated timetabling.

Keywords: Peer Teaching, Undergraduate Medical Education, Bedside Teaching, Medical Students

Practice Highlights

Peer-led teaching environment facilitates questions and answers from learners to strengthen learning.
Training on specific skills and pre-case preparation can help improve peer teacher effectiveness.
Clear understanding of the logistics and expectations is necessary to optimise the process.
Formal peer teacher training may help quality assurance and encourage more participation.

I. INTRODUCTION

In accordance with the longstanding apprenticeship model of medical training, senior doctors and trainees have been responsible for teaching their junior colleagues across the continuum of medical education. Despite this accepted practice, peer teaching has not become widely formalised in undergraduate medical curricula.

Peer teaching has been shown to be beneficial at multiple levels. For students who are being taught by peers, learning is enabled by social and cognitive congruence because of the near-peer demographic which allows for a more comfortable learning environment for free flow of discussion and better understanding of the learner’s challenges including awareness of the primacy for exam success (Benè & Bergus, 2014; Rees et al., 2016). The peer teacher develops and hones teaching skills that will be useful in internship (Haber et al., 2006) and through teaching, develops higher motivation and deeper understanding of concepts and perhaps also improve their own exam performance (Burgess et al., 2014). The institution derives some practical benefit from the supplementary manpower (Tayler et al., 2015) due to the comparable effectiveness of peer teachers in teaching in certain areas such as physical examination and communication skills (Rees et al., 2016) but perhaps most importantly, it benefits from building a collaborative relationship with students in their learning process. Though the benefits of peer teaching have been noted, students remain an untapped resource as training provided for students to serve as teachers is inconsistent (Soriano et al., 2010).

Undergraduate medical curricula aim to provide a foundation for future training and the framework for such curricula are guided by the recognition that medical students must achieve certain outcomes, including being able to teach, to be prepared for future practice. Well-accepted frameworks such as the ‘Outcomes for Graduates’, from the UK General Medical Council (2015) and the ‘CanMEDS Framework’ from the Royal College of Physicians and Surgeons of Canada (2015) expect medical graduate to teach others. In Hong Kong, similar guidance is provided in the document ‘Hong Kong Doctors’ published by the Medical Council of Hong Kong, which states that undergraduate medical education must prepare graduates to fulfil the roles of ‘medical practitioner, communicator, educator…’ (Medical Council of Hong Kong, 2017).

It is common in medical schools to have informal peer teaching, where senior students coach junior students on an ad hoc basis or organise revision sessions before exams. Zhang et al. (2011) revealed that a majority of medical students believed that informal learning approaches, including the use of past student notes, and participation in self-organised study groups and peer-led tutorials, helped them pass examinations and be a good doctor. Similarly, in our institution, these kinds of informal peer teaching are popular among students and include sharing sessions on study and exam tips, bedside sessions, and sharing of organised study notes. These activities are not subject to any formal oversight.

With the documented benefits of peer teaching, the availability of enthusiastic senior students who are willing to coach their junior peers, and the demand from junior students to learn from their seniors, there is an opportunity to harness the potential peer teaching that is already taking place. This pilot project is important as it aimed to adapt existing student-initiated peer bedside teaching into the formal bedside teaching curriculum and to examine the perspectives of participants on the effectiveness and feasibility of this initiative. It will be helpful to understand the benefits and drawbacks of formal peer bedside teaching in order to further develop this pedagogical approach in medical education.

II. METHODS

This was a descriptive qualitative study of participants in a pilot peer-teaching initiative for bedside teaching implemented in the first clinical year of study for medical students.

A. Setting

1) Small group bedside teaching for Year 4 medical students in the Clinical Foundation Block: The 11-week Clinical Foundation Block (CFB) of the MBBS Year 4 curriculum at The University of Hong Kong runs from August to October and is the first block of the first clinical year of study. It serves to prepare students for the ward- and clinic-based teaching to follow in the clinical clerkships (Figure 1). Year 4 medical students were selected for the study because it is the first clinical year of study when clinical bedside teaching begins. In addition, as the most junior clinical students, they would benefit most from learning from their senior peers. During the CFB, all Year 4 students learn basic history taking, physical examination and clinical skills as well as common clinical problems of 10 key specialty disciplines. In internal medicine, students attend whole class sessions in which the proper clinical examination of each body system is demonstrated followed by seven small group sessions at the bedside for hands-on practice led by a clinician.

Figure 1. Teaching activities under Medicine within the Clinical Foundation Block in the medical curriculum

Each small group bedside teaching session is comprised of six to eight CFB students who follow the same clinical teacher to examine 3 pre-selected ward patients over a two-hour period. In this pilot study, a peer teacher joined the clinical teacher for the bedside teaching with the first patient case taught by the clinician, the second case taught by the peer teacher under the supervision of the clinician and the final case taught by the peer teacher alone.

2) Peer teaching recruitment and training: Over the years, medical students have been organising bedside peer-teaching on their own and we identified these peer-teaching leaders to help recruit peer teachers for this initiative. Peer teachers recruited in July 2018 and comprised Year 5 students in Senior Clerkship, who were enthusiastic in teaching, and were available to join the training tutorial and take up a subsequent Year 4 CFB bedside teaching session. During the 2.5-hour tutorial, the CFB Coordinator explained the project, and three clinicians then provided a briefing on cardiovascular, neurological, respiratory and abdominal physical examination, common pitfalls, and how to give feedback. There was also time for students to raise questions both on the project and bedside teaching techniques.

B. Participants

The target participants included the three peer teachers who were recruited for this study, together with the three clinician partners and the 24 CFB students in the corresponding three bedside teaching groups. Written informed consent was obtained from all participants before data collection.

C. Data Collection

The qualitative data were collected using a dual subjective (peer teachers, clinicians and students) and objective (independent observer) approach was taken to provide a more holistic perspective of the peer teaching experience. A research assistant not involved in the teaching followed one (of the three) peer teachers as the independent observer. All peer teachers and clinicians were interviewed in-person, by phone or by email, using an interview guide (Appendix 1) by the research assistant after the session where field notes were taken and transcribed. CFB students were invited to complete an evaluation form comprised of open-ended questions and a single Likert-scale question (Appendix 2) immediately after the bedside session, to rate effectiveness and to give general feedback about the peer teaching session.

D. Data Analysis

The qualitative data comprising interview field notes, interview transcripts, email transcripts and open-ended questions from the evaluation form collected from CFB students were analysed thematically by the authors JC and JPYT. The Likert-scale question from the evaluation form was analysed using descriptive statistics. All data were anonymised.

III. RESULTS

All three peer teachers and three clinicians who participated in the pilot peer teaching sessions were interviewed. Eighteen out of 24 CFB students consented to participate and 12 completed questionnaires were collected. Three main themes were identified with two corresponding subthemes for each.

A. Teaching Effectiveness

Peer teachers were rated favourably in terms of their teaching effectiveness. From the evaluation form completed by CFB students, the mean peer teaching effectiveness rating was 4.5/5. While a few students felt the teaching effectiveness of clinicians and peer teachers was comparable, many of them felt less intimidated being taught by the peer teachers. Students also appreciated that the peer teachers understood their current level of understanding and therefore were able to make the teaching more effective by tailoring it to their needs. Students found the experience-sharing by the peer teachers an added-value as shown in Table 1 (Item 1-4). All clinicians agreed that the CFB students appeared more relaxed while the peer teachers were teaching, and the peer teachers met their standard of professionalism as shown in Table 1 (Item 3).

Subtheme: Learning environment

1. ‘I was more willing to ask questions.’ – CFB Student 8

2. ‘I felt more comfortable and less intimidate[ed] with the peer teacher.’ – CFB Student 12

3.‘I think it is pretty well received among the CFB students – they looked like they are more comfortable and less stressed.’ – Clinician B

Subtheme: Tailoring to needs

4.‘We were told her past experience.’ – CFB Student 9

5.‘More exam advice from peer tutor.’ – CFB student 10

Table 1. Exemplar quotes from participants on teaching effectiveness

These comments were congruent with the observations of the independent observer. When the clinician was teaching, students appeared to be cautious when performing physical examination and answering questions from the clinician. On the other hand, when the peer teacher was teaching, students were asking for reassurance while performing physical examination, and appeared less hesitant when attempting to answer the questions. The peer teacher sometimes also asked the students how they would do a certain examination before they actually performed it. He also shared his own bedside experience. After the clinician ended the bedside session and left, the peer teachers stayed behind and answered further questions from the students regarding physical examination skills and examination tips.

B. Teaching Competence

For students, the teaching on physical examination skills by peer teacher appeared to be comparable to that by clinicians, with the perceived benefit of tailored instructions to student’s current level, and additional personal experience sharing as shown in Table 2 (Item 1-2).

After co-teaching with the peer teacher, clinicians had different opinions about the competency of an undergraduate student as a formal peer teacher. Two stated that it was more appropriate for senior students to do sharing instead of teaching, while the other was satisfied with the ability of the peer teachers to teach, and appreciate the opportunity to exchange ideas with peer teachers. One clinician also suggested that peer teachers might need more practice on teaching to build up confidence as shown in Table 2 (Item 3, 6 and 7).

On the other hand, all the peer teachers expressed that they felt stressed being observed by the clinicians. Two of them felt confident to teach, while one was less confident and prefer to co-teach with a clinician as shown in Table 2 (Item 4, 5 and 8).

The peer teachers also questioned their role as a peer teacher in the regular curriculum. They were unsure to teach in place of clinicians in the regular bedside sessions for the CFB students, yet were more comfortable to co-teach with the clinicians, or to teach in unofficial or supplementary peer-led sessions as shown in Table 2 (Item 4, 8 and 9).

Subtheme: Confidence in teaching competence

1. ‘Very comprehensive teaching; detailed explanation on how to report findings.’ – CFB Student 1

2. ‘Senior students know what we need to know and what we don’t know at this stage.’ – CFB Student 5

3. ‘The peer teacher was sufficiently prepared on content knowledge and teaching skills.’ – Clinician A

4. ‘I am confident with my knowledge and teaching skills. The CFB cases were easy enough for me to handle. I have been teaching student-initiated sessions anyway.’ – Peer Teacher A

5. ‘Are we going to replace the clinicians? The student-initiated sessions worked just fine.’ – Peer Teacher B

Subtheme: Doubts on teaching competence

1. ‘It is too early for the current peer teachers to teach as they lack competency and confidence in teaching.’ – Clinician B

2. ‘Tutors should be at least medical graduates who have shown evidence of proficiency and knowledge in the areas that they teach. Senior students can share their experience of learning, but not to teach.’ – Clinician C

3. ‘The clinicians are definitely better at teaching and has better skills… It would work better if I was to co-teach with a clinician but not to teach solo.’ – Peer Teacher C

4. ‘It isn’t appropriate to take away the proper learning opportunity to be taught by clinicians from the students.’ – Peer Teacher C

Table 2. Exemplar quotes from participants on teaching competency

C. Feasibility

1) Barriers: One of the peer teachers was disappointed that the session did not go as planned. He suspected that the clinicians may not truly understand the purpose and the plan for the project, and hence sometimes took the lead when the peer teachers were supposed to be teaching as shown in Table 3 (Item 1).

They also mentioned that timetabling conflicts between CFB and Senior Clerkship were also an issue. For all groups, the session overran and resulted in peer teachers missing their own class, which was scheduled immediately following the intended finishing time of this bedside session.

Peer teachers also commented that there was no concrete incentive for them to join the project. With the added pressure of being observed by clinicians, most peer teachers were hesitant to volunteer again.

2) Facilitators: One peer teacher considered it as an extra learning opportunity as shown in Table 3 (Item 2). Clinicians also believed that the peer teachers could benefit since these were essentially extra tutorials and bedside exposure for them outside of the regular curriculum although students thought that the cases used for CFB were too easy for them to learn anything new. Both peer teachers and clinicians agreed that more practical training on physical examination would be beneficial to boost the confidence and competence of the peer teachers in teaching. Peer teachers suggested that to make the session more efficient, they would prefer to clerk the case themselves before the session, to be better prepared to recognise abnormal physical signs shown in Table 3 (Item 3). A pre-meeting between the peer teacher and the partner clinician would be helpful to clarify expectations and understanding of the process since the training tutorial was conducted by a different clinician. A clinician pointed out that an open call should be made for the recruitment to allow all interested students to participate.

Barriers

1. ‘I felt like the clinician did not want to let me teach solo. Maybe he did not understand the project.’ – Peer Teacher A

Facilitators

2. ‘The organisation of the curriculum is weird – there were a lot to learn in the Medicine Block of the Junior Clerkship, but not much in that of Senior Clerkship. There was also a large gap of time where there was no supervised physical examination at bedside. This is a good refresher session for me.’ – Peer Teacher C

3. The students and I all saw the case for the first time during the session. I felt a bit unprepared and can only comment on the physical examination skills of the students. There is no way to tell if they reported the correct findings. It would help if the peer tutors can clerk the case before the session.’ – Peer Teacher C

Table 3. Exemplar quotes from participants on barriers and facilitators

IV. DISCUSSION

This pilot project aimed to examine the effectiveness and feasibility of adapting peer bedside teaching into the formal curriculum. Student rating has been used as the primary measure of teaching effectiveness in many schools (Chen & Hoshower, 2003). In this project, we triangulated student ratings with clinician viewpoint and also that of an independent observer to assess teaching effectiveness. All found the teaching by the peer teachers was professional and comparable to clinicians.

Their views were also congruent to the observation that peer teaching provided a more relaxed learning environment as cited in the literature (Tai et al., 2016). This is reflected in a study on problem-based learning (PBL) that showed student tutor-led tutorials were rated more highly in group functioning and supportive atmosphere, compared with faculty-led sessions (Kassab et al., 2005).

Sharing from peer teachers was also identified as a bonus feature of bedside peer teaching in our study. Sharing from senior students not only provide junior student with practical exam and ward survival tips, but also served as inspiration and motivation for students to learn. Again this has also been observed in other studies such as one in which students whose peer teachers shared real life experiences performed better in a post-training CPR knowledge test, and demonstrated more confidence and learning motivation (Souza et al., 2022).

In the next incarnation of peer teaching the barriers and facilitators noted by stakeholders need to be addressed. The difficulty in scheduling can be overcome by engaging senior students who are already on the ward to teach by embedding this requirement as part of their usual work. A clinical peer-assisted learning programme by Nikendei, et al. (Nikendei et al., 2009) had demonstrated a successful peer teaching programme at the bedside with final year medical students who were working in the wards as tutors. The comment among peer teachers that there is no ‘concrete incentive’ to being a peer teacher may be due a lack of awareness of the appreciation from peer learners as well as from faculty teachers. More regular and deliberate sharing of learner feedback and role modelling the enjoyment of teaching by teachers and experienced peer teachers can help. Reflecting on the benefits of the learning process undertaken through the preparation and ‘paying forward’ the efforts from other teachers are also less tangible (but important!) factors to emphasise to encourage future students to undertake peer-teaching.

Peer teachers and clinicians should meet before the teaching session to clarify aims and logistics, and match their expectations. To improve peer teacher confidence and to alleviate clinician concern about their competency to teach, more extensive and formal training can be provided to peer teachers, including both theoretical and practical training on physical examination, and on teaching skills. Burgess et al. (2017) had developed and implemented an interprofessional Peer Teaching Training (PTT) programme for medicine, pharmacy and health sciences students, which aimed to develop students’ skills in teaching, assessment and feedback for peer assisted learning and future practice. The PTT course design was adapted by Karia et al. (Karia et al., 2020) for medical students only. Both programmes were shown to be effective in improving students’ confidence and competence in peer teaching, and increasing intention to participate in teaching. This is encouraging and we are also developing a structured peer teaching training programme to fill this gap. Nevertheless, when attempting to include peer teachers in the formal curriculum as a complement to formal teaching by the faculty care must be taken to not over-formalise the process which may undermine the unique benefits of peer teaching (Tong & See, 2020).

A. Strengths and Limitations

This was a small-scale pilot study and the evaluation of the impact was limited to perceptions and feedback from stakeholders and did not include tangible outcomes such as academic performance and clinical competency of participants. However, the objective contemporaneous observations made during the teaching sessions by a third-party researcher strengthened the trustworthiness of the data. A 360-degree evaluation including feedback from patients and ward staff could also provide a more comprehensive evaluation.

V. CONCLUSION

This study examined the perspectives of clinicians, peer teachers and students on the effectiveness and feasibility of peer-led bedside teaching in the formal curriculum and the benefits are encouraging. Peer teaching effectiveness was comparable to clinicians with the added benefit that peer-teachers are better able to understand and meet students’ needs while creating a friendlier environment conducive to constructive learning. Concerns about peer teaching competency were expressed by clinicians and peer-teachers and all participants did not wish to have peer-teaching replace clinician-led teaching. Clinical peer teaching in the formal curriculum may be most feasible in a hybrid curriculum that includes both peer teaching and clinician-led teaching. It can be accomplished with more structured training and overcoming practical barriers of timetabling and preparation. The benefits of peer teaching and promising responses from all stakeholders support further initiatives in clinical peer teaching.

Notes on Contributors

JY Chen designed the study, performed data collection and data analysis, drafted the manuscript and approved the final manuscript.

TP Lam designed the study, gave critical feedback, read and approved the final manuscript.

IFN Hung designed the study, gave critical feedback, read and approved the final manuscript.

ACY Chan designed the study, gave critical feedback, read and approved the final manuscript.

WY Chin designed the study, gave critical feedback, read and approved the final manuscript.

JPY Tsang performed data collection and data analysis, drafted the manuscript and approved the final manuscript.

C See designed the study, gave critical feedback, read and approved the final manuscript.

Ethical Approval

This study was approved by the Institutional Review Board of the University of Hong Kong/ Hospital Authority Hong Kong West Cluster (Reference number: UW 18-439).

Data Availability

The data of this qualitative study are not publicly available due to confidentiality agreements with the participants.

Acknowledgement

We would like to thank the peer teachers, students and clinicians of HKUMed for participating in the study.

Funding

This work was supported by a Teaching Development Grant funded by The University of Hong Kong (Ref No:. N/A).

Declaration of Interest

The authors declare that there is no conflict of interest.

References

Benè, K. L., & Bergus, G. (2014). When learners become teachers: A review of peer teaching in medical student education. Family Medicine, 46(10), 783-787.

Burgess, A., McGregor, D., & Mellis, C. (2014). Medical students as peer tutors: A systematic review. BMC Medical Education, 14(1), 115.

Burgess, A., Roberts, C., van Diggele, C., & Mellis, C. (2017). Peer teacher training (PTT) program for health professional students: Interprofessional and flipped learning. BMC Medical Education, 17(1), Article 239.

Chen, Y., & Hoshower, L. B. (2003). Student evaluation of teaching effectiveness: An assessment of student perception and motivation. Assessment Evaluation in Higher Education, 28(1), 71-88.

General Medical Council. (2015). Outcomes for graduates (Tomorrow’s Doctors). Retrieved July 18, 2022 from https://www.gmc-uk.org/-/media/documents/Outcomes_for_graduates_jul_15_1216.pdf_61408029.pdf

Haber, R. J., Bardach, N. S., Vedanthan, R., Gillum, L. A., Haber, L. A., & Dhaliwal, G. S. (2006). Preparing fourth‐year medical students to teach during internship. Journal of General Internal Medicine, 21(5), 518-520. https://doi.org/10.1111/j.1525-1497. 2006.00441.x

Karia, C., Anderson, E., Hughes, A., West, J., Lakhani, D., Kirtley, J., Burgess, A., & Carr, S. (2020). Peer teacher training (PTT) in action. Clinical Teacher, 17(5), 531-537.

Kassab, S., Abu-Hijleh, M. F., Al-Shboul, Q., & Hamdy, H. (2005). Student-led tutorials in problem-based learning: Educational outcomes and students’ perceptions. Medical Teacher, 27(6), 521-526.

Medical Council of Hong Kong. (2017). Hong Kong Doctors. Retrieved July 18, 2022 from https://www.mchk.org.hk/english/publications/hk_doctors.html

Nikendei, C., Andreesen, S., Hoffmann, K., & Jünger, J. (2009). Cross-year peer tutoring on internal medicine wards: Effects on self-assessed clinical competencies–A group control design study. Medical Teacher, 31(2), e32-e35.

Rees, E. L., Quinn, P. J., Davies, B., & Fotheringham, V. (2016). How does peer teaching compare to faculty teaching? A systematic review and meta-analysis. Medical Teacher, 38(8), 829-837.

Royal College of Physicians and Surgeons of Canada. (2015). CanMEDS Framework. Retrieved July 18, 2022 from http://www.royalcollege.ca/rcsite/canmeds/canmeds-framework-e

Soriano, R. P., Blatt, B., Coplit, L., CichoskiKelly, E., Kosowicz, L., Newman, L., Pasquale, S. J., Pretorius, R., Rosen, J. M., & Saks, N. S. (2010). Teaching medical students how to teach: a national survey of students-as-teachers programs in US medical schools. Academic Medicine, 85(11), 1725-1731.

Souza, A. D., Punja, D., Prabhath, S., & Pandey, A. K. (2022). Influence of pretesting and a near peer sharing real life experiences on CPR training outcomes in first year medical students: A non-randomized quasi-experimental study. BMC Medical Education, 22(1), 1-11.

Tai, J., Molloy, E., Haines, T., & Canny, B. (2016). Same‐level peer‐assisted learning in medical clinical placements: A narrative systematic review. Medical Education, 50(4), 469-484.

Tayler, N., Hall, S., Carr, N. J., Stephens, J. R., & Border, S. (2015). Near peer teaching in medical curricula: Integrating student teachers in pathology tutorials. Medical Education Online, 20(1), 27921.

Tong, A. H. K., & See, C. (2020). Informal and formal peer teaching in the medical school ecosystem: Perspectives from a student-teacher team. JMIR Medical Education, 6(2), e21869.

Zhang, J., Peterson, R. F., & Ozolins, I. Z. (2011). Student approaches for learning in medicine: What does it tell us about the informal curriculum? BMC Medical Education, 11(1), Article 87.

*Julie Chen
4/F William MW Mong Block
Faculty of Medicine Building
21 Sassoon RoadMarrakesh, Marrakesh-Safi,
Pokfulam, Hong Kong
Email address: juliechen@hku.hk

Submitted: 28 September 2022
Accepted: 2 March 2023
Published online: 3 October, TAPS 2023, 8(4), 5-12
https://doi.org/10.29060/TAPS.2023-8-4/OA2883

Soumia Merrou¹, Abdellah Idrissi Jouicha², Abdelmounaim Baslam³, Zakaria Ouhaz³ & Ahmed Rhassane El Adib¹

¹Health Sciences Research Centre (HSRC), Faculty of Medicine and Pharmacy of Marrakech, Cadi Ayyad University, Morocco; ²Health Sciences Research Centre (HSRC), Faculty of Science Semlalia, Cadi Ayyad University, Morocco; ³Pharmacology, neurobiology and behaviour Lab, Faculty of Science Semlalia, Cadi Ayyad University, Morocco

Abstract

Introduction: A deep understanding of physiology, physiopathology, pharmacology, and the management of pain is crucial for nurse anaesthetists to ensure the well-being of their patients. Thus, the teaching strategies should enhance the transition from acquiring the fundamental pain phenomena, to developing translational and critical thinking. The aim of the study is to determine if the flipped classroom that is considered an active learning approach is most effective compared to the traditional method in teaching pain management and if it improves students’ academic performance.

Methods: This study was quasi experimental, at a higher institute of nursing professions, among third-year anaesthesia resuscitation nursing students. participants were randomly allocated into either: the flipped classroom group where PBL was used (FG, n = 19), or the traditional lecture-based classroom group (TG, n = 19). The results and impact of the above approach were appreciated via the analysis of the summative assessment of the class group and from the questionnaire submitted to students.

Results: The present study revealed that in the midterm exam, the mean score of the flipped classroom group (14.0) which is significantly higher (p<0.01) than the traditional lecture group (11.9). Moreover, the standard deviation of this latter is slightly higher (2.41) which indicates scores far from the average. Also, a significant difference between the averages of the two approaches in favor of flipped classroom Group was revealed (p<0.01).

Conclusion: The assessment of student’s grades and their appreciation of both teaching approaches showed a preference for the PBL.

Keywords: Flipped Classroom, Nursing Education, Pain Management, Problem-Based Learning

Practice Highlights

Flipped classroom showed advantageous results on nursing students’ grades.
Flipped classroom endorsed positive results on course comprehension by nursing students.
Flipped classroom has shown to effectively support content learning.

I. INTRODUCTION

Flipped classroom is a pedagogical approach defined as: “What was previously completed as homework is now finished in class, and what was previously completed in class is now completed at home” (Dong, 2016). Using this approach, traditional classroom time is spent on active learning strategies such as problem-based learning, games, or practice questions to allow teachers to guide students in developing strategies. critical thinking (Dong, 2016). Flipped classrooms are used as the main teaching method in the courses of health professions such as nursing theory, statistics and pharmacology (Hanson, 2016; Immekus, 2019; Peisachovich et al., 2016). In fact, there is evidence that students’ academic performance improved in midterm exams while using flipped classroom approach (Geist et al., 2015).

Despite feeling that this method increased their knowledge, nursing students said they preferred traditional lectures to the use of a flipped classroom (Hanson, 2016). It is not uncommon for students to prefer lectures to the flipped classroom method, which may be related to how much work they feel they have to do or insecurity of exam preparation or both of them (Dong, 2016; Tune et al., 2013). The use of the flipped classroom in nursing was supported by evidence that showed lecturers were enthusiastic about this method. The most effective method for implementing and assessing this strategy in nursing education, though, is not consistently supported by the available data (Barranquero-Herbosa et al., 2022; Dong, 2016; Njie-Carr et al., 2017).

Contextual learning can encourage the growth of critical reasoning, which enables students to pick out the top nursing concerns for patients from a long list of problems, ultimately fostering the development of problem-based nursing analysis in line with Benner’s model (Dong, 2016). Problem-based learning (PBL) uses problem scenarios to develop knowledge and understanding learning objectives (Wood, 2003). Among the strategies used in a flipped classroom, the PBL has been used in nursing education, in courses such as pharmacology, mental health nursing and critical care nursing (Alton, 2016; Gholami et al., 2016). Any teaching strategy that involves students in the learning process is considered to be an active learning strategy, which includes PBL (Peisachovich et al., 2016).

Despite the introduction of pain management in health professions education, pain is still undertreated. It affects 80%-90% of patients in medicine, surgery, and cancer units (Gerbershagen et al., 2009; Gianni et al., 2010). Previous research also highlighted that 43% to 51% of patients received inadequate or insufficient analgesic treatment and only 14% of patients who received analgesia benefit from reassessment (Deandrea et al., 2008; Manias et al., 2005). To effectively manage pain, nurses are crucial. Therefore, it is crucial that they receive effective training to ensure better pain management (Teike Lüthi et al., 2015).

In this direction, in order to encourage students’ acquisition strategies, nursing science professors must implement effective teaching techniques. Training typically aims to increase knowledge, which is insufficient in this case; as a result, skills development is a top priority (Kerner et al., 2013). While prior research emphasised the value of nurse-patient interactions in pain management, it undervalued the impact of nurses’ scientific knowledge of pain mechanisms and pharmacology. It is interesting to note that a recent study highlighted the significance of the classroom setting and instructional methods in approaching pain management in a novel manner (Teike Lüthi et al., 2015).

However, a need for a rigorous evaluation of learning strategies is crucial for best practices in nursing education (Barranquero-Herbosa et al., 2022; Njie-Carr et al., 2017). The present study provides an assessment of PBL as a model of applied learning in a flipped classroom of anesthesia nursing students in the context of a pain management course.

The main purpose of the study was to determine if the flipped classroom is more effective than traditional learning in teaching pain management by assessing students’ academic performance and determine their perceptions about the flipped classroom approach. In that capacity, the research questions of the study are:

Is there a significant difference in students’ academic performance between the traditional and flipped classroom approaches on declarative knowledge?
Is there a significant difference in students’ academic performance between the traditional and flipped classroom approaches on conditional knowledge?
What are anesthesia and resuscitation nursing students’ perceptions of PBL impact on the acquisition and application of pain management knowledge?
What are anesthesia and resuscitation nursing students’ perceptions of PBL as a model for learning in pain management?

II. METHODS

A. Research Design and Samples

This study is quasi experimental, and was conducted from September at a higher institute of nursing professions. The participants are third-year anaesthesia resuscitation nursing students. Participation in the study was voluntary and anonymous. Oral consent of all participants was obtained. These participants were randomly allocated into either: The flipped classroom group where PBL was used (FG, n = 19), or the traditional lecture-based classroom group (TG, n = 19). Both classroom groups had the same professor.

B. Curriculum Description

The “pain management” course (50h) is taught during the third year of nursing studies in the institute. It is composed of three parts: the pathophysiology of pain; the evaluation of pain, and the pain management.

C. Problem Based Learning on Flipped Classroom Approach

The problem-based template was designed by the professor who teaches the course, by using small groups of 5 to 6 students. The students were the facilitators of the discussion; they meet in group work to discuss a case for an hour. The objective is to identify the type of pain or to choose the best pain assessment tool for the case. The group must then suggest a drug treatment protocol and design appropriate nursing interventions. The role of the professor was to provide immediate and specific feedback during the discussion.

All cases were written by the professor. The objectives were the acquisition of knowledge and the development of clinical reasoning. Each case contained 300 words and included key patient data. Each of these cases included information that could be analysed to provide priority elements to the discussed case.

D. Data Collection and Statistical Analyses

The results and impact of the above approach were extracted via an analysis of the summative assessment of the class group and from the questionnaire submitted to students.

1) Summative assessment (exam):

Students in both groups went through two exams: midterm exam (ME) which took place in the middle of the course in the 6^th week in order to assess the students’ declarative knowledge, and a final exam (FE) which took place at the end of the course, to assess conditional knowledge. The tests were graded from zero to twenty. The final score (FS) was obtained by the following equation:

FS=(ME+FE)/2

2) Questionnaire:

At the end of the course, the FG students were asked to fill out an anonymous questionnaire divided into two sections. The questions were developed in the first section of the questionnaire to determine students’ perceptions of knowledge acquisition. Elements evoked in the questionnaire were created with a language that demonstrates perceived ability and related to self-efficacy (Tune et al., 2013). The second set of items was created to determine students’ perceptions of the cases used in the course. The statements began, for example, with “Participating in the group discussions made me more confident for…”. Likert scale was used to measure the responses. The scale is presented as follows:

1 = Strongly disagree, 2 = Disagree, 3 = Neither agree nor disagree, 4 = Agree, and 5 = Strongly agree.

3) Statistical analyses:

Data analysis was performed using MS Excel (21), background variables of the study participants were calculated, and the results are presented as frequency distribution, percentages, mean, and standard deviation, statistical significance when p <0.05.

III. RESULTS

The data that support the findings of this RCT study are openly available at https://doi.org/10.6084/m9.figshare.22639279 (Merrou et al., 2023).

A. Demographics

The number of participants in the study was 38 students, 19 per group. Female students represented 79% of the study participants, whereas 21% were male.

B. Students’ Grades

Based on the data obtained, statistical analysis was done to analyse the influence of the teaching approach and the type of examination on learners’ results. The obtained findings have been presented in Tables 1 and 2. They indicate the average performance of learners in both exams: midterm (ME) and final exam (FE) where conditional knowledge is measured for both teaching approaches.

Type of exam	Teaching approach	M	Sd	Inf born	Sup born	P value
ME	TG	11.9	2.41	7.38	16.1	<0.01
ME	FG	14.0	1.94	9.0	16.5	<0.01
FE	TG	11.9	3.28	6.09	16	<0.01
FE	FG	14.1	1.96	10	16	<0.01

Table 1. Descriptive statistics by exam type for each teaching approach.

According to Table 1, it is noted that in the midterm exam (ME), the mean score is significantly higher (p<0.01) in the FG (14.0) compared to the TG (11.9), also, with this latter, there is a slightly high standard deviation of 2.41 which indicates scores far from the average. FG, on the other hand, dressed a lower standard deviation (1.94) which indicates that the scores are more grouped around the mean (14.0). The application of the PBL on flipped classroom approach has, as it appears, improved the grades and reduced the gap between them.

For the final exam, with the traditional approach, the dispersion increased (Sd=3.28). On the other hand, PBL approach has improved student outcomes and widened the gap between them compared to TG (p<0.01). Figure 1 highlights the dispersion of the continuous and final control data (before for the traditional approach and after for PBL).

Figure 1. Students’ performance during the midterm exam (ME) and final exam (FE)

The ME grades were improved using PBL. As the number of compared participants is limited, a nonparametric test was carried out (Paired Mann-Whitney test) which revealed that the average grade of MEs is significantly different (p <0.01), between the traditional approach and the PBL. Similarly, an improvement in FE grades is observed when using the PBL approach. This approach allowed an improvement in the means as well as the dispersion. This leads us to state that the teaching approach based on case studies in the context of a flipped classroom (PBL), may improve both declarative and conditional knowledge on students’ outcomes.

Teaching approach

Standard deviation

Δ mean

11.9

3.30

2.21

0.01

14.1

1.95

Table 2: Descriptive statistics by teaching approach for the different types of controls.

From Table 2, there is a difference between the averages of the two approaches in favor of FG (p<0.01). This means that students who have taken the PBL approach had increased grades compared to those who have taken the traditional approach. To check if these differences are significant, a Paired Mann-Whitney test was used. This one demonstrated that the average rank of the grades is significantly different between the two studied approaches (p = 0.01).

The mean score and the standard deviation for each question in the questionnaire was determined. Average responses to the 12 items that referred to acquisition and application of knowledge related to the pain management ranged from 3.8 to 4.5 (See Table 3).

Statements	Average score (Sd)
1. I am confident in my ability to read a case and select the patient’s key factors that may impact their care.	4.3 (0.7)
2. I am confident in my ability to identify the presence of pain in a given patient.	4.1 (0.6)
3. I feel confident in determining the type of pain from the etiology involved.	4.2 (0.6)
4. I feel confident in determining the type of pain from the descriptive semiology used by a patient.	4.2 (0.7)
5. I am confident in my ability to choose the right pain assessment test for a given patient.	4.5 (0.5)
6. I am confident in my ability to use pain assessment tests with a given patient.	4.3 (0.6)
7. I am confident in my ability to understand the mechanism of action of an analgesic according to its pharmacological class.	4.1 (0.6)
8. I feel confident in my ability to relate the therapeutic benefit of a drug to its mechanism of action.	4.1 (0.5)
9. I am confident in my ability to determine the oxidative, supra-additive, or sub-additive effects of painkillers.	4.1 (0.8)
10. I feel more sensitive to the importance of pain management.	4 (0.7)
11. I feel better prepared at the clinic after participating in clinical case discussions as part of the flipped classroom.	4.1 (0.5)
12. I feel better prepared to act as an advocate for my patient’s interests to ensure comfort.	3.8 (0.8)

Table 3. Acquisition and application of knowledge

The statement “I feel better prepared to act as an advocate for my patient’s interests to ensure comfort” received a response rate of 3.8. This statement has the least satisfaction rate compared to all statements in the corresponding section.

Statements	Average score (Sd)	P value
1. The cases were relevant and interesting.	4.7 (0.4)	Ns*
2. I was nervous at the beginning of the module, but I gained confidence in myself as the course progressed.	4.3 (0.8)
3. Participating in the group discussions made me more confident in analysing key pain-related data.	4.1 (0.9)
4. I find that discussions have helped me learn more effectively than lectures.	4.5 (0.6)
5. I found that the group discussions helped my learning more effectively than the manual (handout).	4.6 (0.7)
6. I would recommend case-based seminar discussions as a tool for other courses.	4 (0.9)

*mean comparison of each item

Table 4. Perception of LPLs as a teaching/learning model

Average responses to the six questionnaire items that referred to cases as a learning model ranged from 4 to 4.7. None of the average responses differed significantly (p>0.05) from the other responses in this section of the questionnaire. The average response to the question “Were the cases relevant and interesting?” was 4.7, which was higher than all other answers. The statement “I would recommend the PBL format (no lectures, only case studies with assigned readings) as a tool for other courses” has a response average of 4, which is lower than all other responses in the corresponding category.

IV. DISCUSSION

Nursing students must grasp intricate concepts of basic physiology, pathophysiology, pharmacology, and more. Employing effective teaching methods with active learning can foster critical thinking abilities and uphold patient safety in complex care scenarios (Dong, 2016; Forsgren et al., 2014; Wood, 2003). Nowadays, nursing education has embraced the use of the flipped classroom as it offers a rich learning environment (Dong, 2016; Hanson, 2016; Immekus, 2019; Missildine et al., 2013; Ndosi & Newell, 2009; Peisachovich et al., 2016; Wood, 2003). Problem-based learning (PBL) is a frequently employed active learning approach in flipped classroom scenarios (Dong, 2016; Geist et al., 2015). PBL has been demonstrated to enhance the capacity of nursing students to evaluate patient information and arrive at more contemplative clinical judgments (Forsgren et al., 2014; Njie-Carr et al., 2017). When nursing students engage in discussions within small groups, they open themselves up to a wealth of interactive learning opportunities that are guided by their professor. This active learning situation is far more advantageous than the traditional lecture format as it promotes critical thinking skills and fosters independent learning. By participating in small group discussions, nursing students are able to delve deeper into the subject matter, ask questions, and engage in meaningful dialogue with their peers and instructor. This type of collaborative learning environment encourages students to take ownership of their education and empowers them to become more confident and competent healthcare professionals. Therefore, it is crucial that nursing programs prioritise small group discussions as a key component of their curriculum (Bailey, 2017; Carvalho et al., 2017; Kong et al., 2014; Teike Lüthi et al., 2015; Wood, 2003). High-level thinking and independent learning are enhanced with the use of interactive small groups (Alton, 2016; Gholami et al., 2016). We note from a review of the literature that a limited number of studies have examined the use of PBL in nursing (Bailey, 2017; Forsgren et al., 2014).

The current study revealed that regardless of the nature of the exam, student learning outcomes significantly improved with the flipped classroom method. Furthermore, the students participating in this study consider this method as a useful model to improve their learning and be more engaging. In fact, active learning allows effective knowledge acquisition (Arrue et al., 2017) and the development of critical thinking skills on nursing students as well as the improvement of metacognitive skills (Bailey, 2017; Carvalho et al., 2017; Domínguez, 2012). Furthermore, the students participating in this study consider this method as a useful model to improve their learning and be more engaging (Schlairet et al., 2014). Consequently, alternating between lectures and PBL approach may be a better option for health science courses (Alexandre & Wright, 2013). Greater confidence is demonstrated in acquiring and applying knowledge (practice) related to pain management.

Participation in this approach was considered a positive learning strategy, regardless of course content, the flipped classroom has shown to effectively support content learning (Hanson, 2016). When students were asked to consider whether it helped them learn more effectively than lectures, a higher response was obtained, and the response was very positive. This conclusion is in line with one from a study conducted in Portugal, which found that using this method in a second-year pathophysiology course led to higher levels of student satisfaction (Marques & Correia, 2017). Although some discomfort may be reported students are uncertain about the content and will attend classes on the assumption that it will help them understand exactly what they need to do and what they hope to achieve. This result confirms that student satisfaction does not always accurately reflect their learning (Dong, 2016). Further evaluation of this strategy and other learning tools is needed to establish best practices in nursing education (Barranquero-Herbosa et al., 2022; Njie-Carr et al., 2017).

A. Limitations

The small number of participants may affect the validity of the study. The results of this study cannot be generalised because participants belonged to one track only, so they are not representative to all nursing students. In addition, the small sample size of the study participants and the small number of available academic levels covered by the study.

B. Implications for Teaching and Future Research

Future studies could be considered to compare different learning strategies (e.g., games, medication card design, and practice problems) to determine the best practices for active learning strategies that support learning in a professional education setting and support flipped classroom learnings in nursing education.

V. CONCLUSION

Nursing education is about the development of professional skills; hence it is important to adopt active teaching strategies that promote critical thinking and knowledge transfer. However, the time constraint often pushes teachers to adopt the magistral lectures, the traditional form of knowledge delivery which mostly lacks the element of interactivity which is an issue recognised among many researchers worldwide.

The flipped classroom, in our case, is a solution to the time management problem. It allowed us to free up time in class which was beneficial to give space for interactive activities and active animation techniques such as case studies. In addition, in this study, we were able to compare the impact of the flipped classroom with the traditional model on two groups of students enrolled to the same course: pain management. The comparison results were mainly based on the acquisition of knowledge by students. We also measured students’ satisfaction with the proposed model as well as their sense of self-efficacy.

Students’ grades were clearly in favour of the PBL model in the flipped classroom. The students were also mostly satisfied with the proposed model and confirmed the development of their sense of self-efficacy regarding the pain management course.

Our perspective is the improvement of our teaching which, in our opinion, must be constantly corrected and enriched to face new conditions and situations. In this direction, the present study could constitute a roadmap for further in-depth studies to bring more to the PBL-based teaching model in the flipped classroom.

Notes on Contributors

Soumia Merrou is involved in the conceptualisation, methodology, data curation, writing and original draft preparation.

Abdellah Idrissi Jouicha helped in the methodology, participated in data curation and software, helped in writing – reviewing and editing.

Baslam Abdelmounaim participated in writing the original draft preparation, performed statistical analyses, helped in reviewing and editing corrections.

Zakaria Ouhaz was involved in visualisation, participated in data collection, helped writing and reviewed the manuscript.

Ahmed Rhassane El Adib was central to the conceptualisation and methodology, validated the design study, and supervised work progress. All authors have read and approved the final manuscript.

Ethical Approval

Participation in the study was voluntary and anonymous. Oral consent of all participants was obtained and the research was approved by the Institutional Ethical committee (CCBE-FSA Ref. No: ER-CS-10/2022-000).

Data Availability

The data that support the findings of this study are openly available in Figshare repository, https://doi.org/10.6084/m9.figshare.21385446.

Acknowledgement

We acknowledge the efforts of both professor and participants.

Funding

The study received no funding.

Declaration of Interest

The authors declare that they have no conflict of interest.

References

Alexandre, M. S., & Wright, R. R. (2013). Flipping the classroom for student engagement. International Journal of Nursing Care, 1(2), 100.

Alton, S. (2016). Learning how to learn: Meta-learning strategies for the challenges of learning pharmacology. Nurse Education Today, 38, 2–4. https://doi.org/10.1016/j.nedt.2016.01.003

Arrue, M., Ruiz de Alegría, B., Zarandona, J., & Hoyos Cillero, I. (2017). Effect of a PBL teaching method on learning about nursing care for patients with depression. Nurse Education Today, 52, 109–115. https://doi.org/10.1016/j.nedt.2017.02.016

Bailey, L. A. (2017). Adaptation of know, want to know, and learned chart for problem-based learning. Journal of Nursing Education, 56(8), 506–508. https://doi.org/10.3928/01484834-20170712-11

Barranquero-Herbosa, M., Abajas-Bustillo, R., & Ortego-Maté, C. (2022). Effectiveness of flipped classroom in nursing education: A systematic review of systematic and integrative reviews. International Journal of Nursing Studies, 105, Article 104327. https://doi.org/10.1016/j.ijnurstu.2022.104327

Carvalho, D. P. S. R. P., Azevedo, I. C., Cruz, G. K. P., Mafra, G. A. C., Rego, A. L. C., Vitor, A. F., Santos, V. E. P., Cogo, A. L. P., & Ferreira Júnior, M. A. (2017). Strategies used for the promotion of critical thinking in nursing undergraduate education: A systematic review. Nurse Education Today, 57, 103–107. https://doi.org/10.1016/j.nedt.2017.07.010

Deandrea, S., Montanari, M., Moja, L., & Apolone, G. (2008). Prevalence of undertreatment in cancer pain. A review of published literature. Annals of Oncology, 19(12), 1985–1991. https://doi.org/10.1093/annonc/mdn419

Domínguez, R. G. (2012). Participatory Learning. In N. M. Seel (Ed.), Encyclopedia of the Sciences of Learning (pp. 2556–2560). Springer.

Dong, X. (2016). Application of flipped classroom in college english teaching. Creative Education, 7(9), 1335–1339. https://doi.org/10.4236/ce.2016.79138

Forsgren, S., Christensen, T., & Hedemalm, A. (2014). Evaluation of the case method in nursing education. Nurse Education in Practice, 14(2), 164–169. https://doi.org/10.1016/j.nepr.2013.08.003

Geist, M. J., Larimore, D., Rawiszer, H., & Al Sager, A. W. (2015). Flipped versus traditional instruction and achievement in a baccalaureate nursing pharmacology course. Nursing Education Perspectives, 36(2), 114-115. https://doi.org/10.5480/13-1292

Gerbershagen, K., Gerbershagen, H. J., Lutz, J., Cooper-Mahkorn, D., Wappler, F., Limmroth, V., & Gerbershagen, M. (2009). Pain prevalence and risk distribution among inpatients in a German teaching hospital. The Clinical Journal of Pain, 25(5), 431–437.

Gholami, M., Moghadam, P. K., Mohammadipoor, F., Tarahi, M. J., Sak, M., Toulabi, T., & Pour, A. H. H. (2016). Comparing the effects of problem-based learning and the traditional lecture method on critical thinking skills and metacognitive awareness in nursing students in a critical care nursing course. Nurse Education Today, 45, 16–21. https://doi.org/10.1016/j.nedt.2016.06.007

Gianni, W., Madaio, R., Cioccio, L., D’Amico, F., Policicchio, D., Postacchini, D., Franchi, F., Ceci, M., Benincasa, E., Gentili, M., & Zuccaro, S. (2010). Prevalence of pain in elderly hospitalized patients. Archives of Gerontology and Geriatrics, 51(3), 273-276. https://doi.org/10.1016/j.archger.2009.11.016

Hanson, J. (2016). Surveying the experiences and perceptions of undergraduate nursing students of a flipped classroom approach to increase understanding of drug science and its application to clinical practice. Nurse Education in Practice, 16(1), 79–85. https://doi.org/10.1016/j.nepr.2015.09.001

Immekus, J. C. (2019). Flipping statistics courses in graduate education: Integration of cognitive psychology and technology. Journal of Statistics Education, 27(2), 79–89. https://doi.org/10.1080/10691898.2019.1629852

Kerner, Y., Plakht, Y., Shiyovich, A., & Schlaeffer, P. (2013). Adherence to guidelines of pain assessment and intervention in internal medicine wards. Pain Management Nursing, 14(4), 302–309. https://doi.org/10.1016/j.pmn.2011.06.005

Kong, L.-N., Qin, B., Zhou, Y., Mou, S., & Gao, H.-M. (2014). The effectiveness of problem-based learning on development of nursing students’ critical thinking: A systematic review and meta-analysis. International Journal of Nursing Studies, 51(3), 458–469. https://doi.org/10.1016/j.ijnurstu.2013.06.009

Manias, E., Bucknall, T., & Botti, M. (2005). Nurses’ strategies for managing pain in the postoperative setting. Pain Management Nursing: Official Journal of the American Society of Pain Management Nurses, 6(1), 18–29. https://doi.org/10.1016/j.pmn.2004.12.004

Marques, P. A. O., & Correia, N. C. M. (2017). Nursing education based on “hybrid” problem-based learning: The impact of PBL-based clinical cases on a pathophysiology course. Journal of Nursing Education, 56(1), 60. https://doi.org/10.3928/01484834-20161219-12

Merrou, S., Jouicha, A. I., Baslam, A., Ouhaz, Z., & El Adib, A. R. (2023). Problem-based learning method in the context of a flipped classroom: Outcomes on pain management course acquisition [Data set]. Figshare. https://doi.org/10.6084/m9.figshare.22639279

Missildine, K., Fountain, R., Summers, L., & Gosselin, K. (2013). Flipping the classroom to improve student performance and satisfaction. Journal of Nursing Education, 52(10), 597-599.

Ndosi, M. E., & Newell, R. (2009). Nurses’ knowledge of pharmacology behind drugs they commonly administer. Journal of Clinical Nursing, 18(4), 570–580. https://doi.org/10.1111/j.1365-2702.2008.02290.x

Njie-Carr, V. P. S., Ludeman, E., Lee, M. C., Dordunoo, D., Trocky, N. M., & Jenkins, L. S. (2017). An integrative review of flipped classroom teaching models in nursing education. Journal of Professional Nursing, 33(2), 133–144. https://doi.org/10.1016/j.profnurs.2016.07.001

Peisachovich, E. H., Murtha, S., Phillips, A., & Messinger, G. (2016). Flipping the classroom: a pedagogical approach to applying clinical judgment by engaging, interacting, and collaborating with nursing students. International Journal of Higher Education, 5(4), 114. https://doi.org/10.5430/ijhe.v5n4p114

Schlairet, M. C., Green, R., & Benton, M. J. (2014). The flipped classroom. Nurse Educator, 39(6), 321–325. https://doi.org/10.1097/nne.0000000000000096

Teike Lüthi, F., Gueniat, C., Nicolas, F., Thomas, P., & Ramelet, A.-S. (2015). Les obstacles à la gestion de la douleur perçus par les infirmières: Étude descriptive au sein d’un hôpital universitaire Suisse. [Barriers to pain management as perceived by nurses: A descriptive study in a Swiss University Hospital.] Douleur et Analgesie [Douleur & Analgésie], 28, 93-99. https://doi.org/10.1007/s11724-015-0414-3

Tune, J. D., Sturek, M., & Basile, D. P. (2013). Flipped classroom model improves graduate student performance in cardiovascular, respiratory, and renal physiology. Advances in Physiology Education, 37(4), 316–320. https://doi.org/10.1152/advan.00091.2013

Wood, D. (2003). Problem based learning. British Medical Journal, 326, 328–330. https://doi.org/10.1136/bmj.326.7384.328

*Abdellah Idrissi Jouicha
Marrakesh, Marrakesh-Safi,
40000, Morocco
Email: abdellah.idrissi@ced.uca.ac.ma

Submitted: 30 May 2022
Accepted: 7 December 2022
Published online: 4 July, TAPS 2023, 8(3), 35-44
https://doi.org/10.29060/TAPS.2023-8-3/OA2876

Rachel Jiayu Lee¹*, Jeannie Jing Yi Yap¹*, Abhiram Kanneganti¹, Carly Yanlin Wu¹, Grace Ming Fen Chan¹, Citra Nurfarah Zaini Mattar^1,2, Pearl Shuang Ye Tong^1,2, Susan Jane Sinclair Logan^1,2

¹Department of Obstetrics and Gynaecology, National University Hospital, Singapore; ²Department of Obstetrics and Gynaecology, Yong Loo Lin School of Medicine, National University of Singapore, Singapore

^*Co-first authors

Abstract

Introduction: Disruptions of the postgraduate (PG) teaching programmes by COVID-19 have encouraged a transition to virtual methods of content delivery. This provided an impetus to evaluate the coverage of key learning goals by a pre-existing PG didactic programme in an Obstetrics and Gynaecology Specialty Training Programme. We describe a three-phase audit methodology that was developed for this

Methods: We performed a retrospective audit of the PG programme conducted by the Department of Obstetrics and Gynaecology at National University Hospital, Singapore between January and December 2019 utilising a ten-step Training Needs Analysis (TNA). Content of each session was reviewed and mapped against components of the 15 core Knowledge Areas (KA) of the Royal College of Obstetrics & Gynaecology membership (MRCOG) examination syllabus.

Results: Out of 71 PG sessions, there was a 64.9% coverage of the MRCOG syllabus. Four out of the 15 KAs were inadequately covered, achieving less than 50% of knowledge requirements. More procedural KAs such as “Gynaecological Problems” and those related to labour were poorly (less than 30%) covered. Following the audit, these identified gaps were addressed with targeted strategies.

Conclusion: Our audit demonstrated that our pre-pandemic PG programme poorly covered core educational objectives i.e. the MRCOG syllabus, and required a systematic realignment. The COVID-19 pandemic, while disruptive to our PG programme, created an opportunity to analyse our training needs and revamp our virtual PG programme.

Keywords: Medical Education; Residency; Postgraduate Education; Obstetrics and Gynaecology; Training Needs Analysis; COVID-19; Auditing Medical Education

Practice Highlights

Regular audits of PG programmes ensure relevance to key educational objectives.
Training Needs Analysis facilitates identification of learning goals, deficits & corrective change.
Mapping against a milestone examination syllabus & using Delphi technique helps identify learning gaps.
Procedural-heavy learning goals are poorly served by didactic PG and need individualised assessment.
A central committee is needed to balance the learning needs of all departmental CME participants.

I. INTRODUCTION

Postgraduate medical education (PG) programmes are an important aspect in meeting core Specialty Trainees’ (ST) learning goals in addition to other modalities of instruction such as practical training (e.g. supervised patient-care or simulator-based training) (Bryant‐Smith et al., 2019) and workplace-based assessments (e.g. case-based discussions and Objective Structured Clinical Examinations [OSCEs] (Chan et al., 2020; Parry-Smith et al., 2014). In academic medical centres, PG education may often be nestled within a wider departmental or hospital Continuing Medical Education (CME) programme. While both PG and CME programmes indirectly improve patient outcomes by keeping clinicians abreast with the latest updates, reinforcing important concepts, and changing practice (Burr & Johanson, 1998; Forsetlund et al., 2021; Marinopoulos et al., 2007; Norman et al., 2004; Raza et al., 2009; Sibley et al., 1982), it is important to balance the learning needs of STs with that of other learners (E.g. senior clinicians, scientists and allied healthcare professionals). This can be challenging as multiple objectives need be fulfilled amongst various learners. Nevertheless, just as with any other component of good quality patient care, it is amenable to audit and quality improvement initiatives (Davies, 1981; Norman et al., 2004; Palmer & Brackwell, 2014).

The protracted COVID-19 pandemic has disrupted the way we deliver healthcare and conduct non-clinical services (Lim et al., 2009; Wong & Bandello, 2020). In response, the academic medical community has globally embraced the use of teleconferencing platforms such as Zoom^Ⓡ, Microsoft Teams^Ⓡ and Webex^Ⓡ(Kanneganti, Sia, et al., 2020; Renaud et al., 2021) as well as other custom-built solutions for the synchronous delivery of didactics and group discourse (Khamees et al., 2022). While surgical disciplines have suffered a decline in the quality of “hands-on” training due to reduced elective surgical load and safe distancing (English et al., 2020), the use of simulators (Bienstock & Heuer, 2022; Chan et al., 2020; Hoopes et al., 2020; Xu et al., 2022), remote surgical preceptorship, and teaching through surgical videos (Chick et al., 2020; Juprasert et al., 2020; Mishra et al., 2020) have helped mitigate some of these. Virtual options that that have been reproducibly utilised during the pandemic and will be a part of the regular armamentarium of post-graduate medical educationists include online didactic lectures, livestreaming or video repositories of surgical procedures, (Grafton-Clarke et al., 2022) and virtual case discussions and grand ward rounds (Sparkes et al., 2021). Notably, they facilitate the inclusion of a physically wider audience, be it trainer or trainee, and allow participants to tune in from different geographical locations.

At the Department of Obstetrics and Gynaecology, National University Hospital, Singapore, the forced, rapid transition to a virtual CME format (vCME) (Chan et al., 2020; Kanneganti, Lim, et al., 2020) provided an impetus to critically review and revamp the didactic component of our PG programmes. A large component of this had been traditionally baked into our departmental CME programme which comprises daily morning meetings covering recent specialty and scientific updates, journal clubs, guideline reviews, grand round presentations, surgical videos, exam preparation, topic modular series, and research and quality improvement presentations. The schedule and topics were previously arbitrarily decided by a lead consultant one month prior and were presented by a supervised ST or invited speaker. While attendance by STs at these sessions was mandatory and comprised the bulk of protected ST teaching time, no prior attempt had been made to assess its coverage of core ST learning objectives and in particular, the syllabus for milestone ST exams.

Our main aim was to conduct an audit on the coverage of our previous PG didactic sessions on the most important learning goals with the aim of subsequently restructuring them to better meet these goals.

II. METHODS

We audited and assessed our departmental CME programme’s relevance to the core learning goals of our STs by utilising a Training Needs Analysis (TNA) methodology. While there are various types of TNA used in healthcare and management (Donovan & Townsend, 2019; Gould et al., 2004; Hicks & Hennessy, 1996, 1997; Johnston et al., 2018; Markaki et al., 2021), in general they represent systematic approaches towards developing and implementing a training plan. The common attributes can be distilled into three common phases (Figure 1). Importantly for surgical and procedurally-heavy disciplines, an dimension that is not well covered by didactic sessions alone are assessments for procedural skill competency. These require separate attention that is beyond the scope of this audit.

Figure 1. A simplified three phase approach to blueprinting, mapping, and auditing a Postgraduate (PG) Education Programme

A. Phase 1: Identifying Organisational Goals and Specific Objectives

The overarching goal of a specialty PG education programme is to produce well-balanced clinicians with a strong knowledge base. Singapore’s Obstetrics and Gynaecology specialty training programmes have adopted the membership examinations for the Royal College of Obstetricians and Gynaecologists (MRCOG) (Royal College of Obstetricians and Gynaecologists, 2021) of the United Kingdom as the milestone examination for progression from junior to senior ST.

First, we adapted a ten-step TNA proposed by Donovan & Townsend (Table 1) to crystallise our our core learning goals, identify deficiencies, and subsequently propose steps to address these gaps in a systematic fashion that is catered to our specific context. While most aspects were followed without change, we adapted the last aspect i.e. Cost Benefit Analysis. As a general organisational and management tool, the original TNA primarily looked at the financial costs of implementing a training programme. At an academic medical institution, the “cost” is mainly non-financial and mainly refers to time taken away from important clinical service roles.

As part of formulating what were deemed to be core learning goals of an ideal PG programme (i.e. Steps 1 to 4), we had a focused group discussion comprising key stakeholders in postgraduate education, including core faculty (CF), physician faculty (PF), and representative STs. The discussions identified 18 goals specific to our department. We then used a modified Delphi method (Hasson et al., 2000; Humphrey-Murto et al., 2017) to distil what CF, PFs, and STs felt were important priorities for grooming future specialists. Three rounds of priority ranking were undertaken via an anonymised online voting form. At each round, these 18 goals were progressively ranked and distilled until five remained. These were then ranked from highest to lowest priority and comprised 1) exam preparedness, 2) clinical competency, 3) in-depth understanding of professional clinical guidelines, 4) interpretation of medical research literature, and 5) ability to conduct basic clinical research and audits.

Training Needs Analysis
1	Strategic objectives	Competent O&G clinicians
2	Operational outcome	Specialist trainees – preparation and passing of exams (MRCOG, CREOG), achieving ACGME training requirements Existing clinicians: Maintaining knowledge and competence
3	Employee Behaviours	Be familiar with MRCOG syllabus Be familiar with updates in clinical guidelines, keep up with progress/advancements in scientific research
4	Learnable Capabilities	Completed Part 1 exam before entering specialist training Knowledge, procedural skills, and competency Achieving ACGME milestones
5	Gap Assessment	Blueprinting of PG programme to identify deficiencies in teaching Survey to STs/clinicians Self-assessment Tests (MRCOG, CREOG) Performance evaluation
6	Prioritise Learning and Training Needs	Restructure PG programme – in terms of breadth & width of topics Identify who needs training – STs taking exams
7	Learning Approaches	In various methods: didactics, lecture (invited speaker), e-learning, conferences, journal club, scientific research meeting, on the job training, surgical videos, panel discussions Transition to virtual platforms, webinars Suspension of simulation/hands-on workshops
8	Roll-out Plan	Virtual didactic PG programme – 3-4 times per week
9	Evaluation Criteria	Survey 1 year post implementation Assessment form post teaching
10	Cost Benefit Analysis	Points in consideration: content development time, lost productivity from time spent in training, delivery method (Zoom^®)

Table 1. 10-step Training Needs Analysis

Table adapted from Donovan, Paul and Townsend, John, Learning Needs Analysis (United Kingdom, Management Pocketbooks, 2019)

MRCOG: Member of the Royal College of Obstetricians and Gynaecologists, O&G: Obstetrics and Gynaecology

CREOG: Council on Resident Education in Obstetrics and Gynecology

ACGME: Accreditation Council for Graduate Medical Education,

PG: Post-Graduate Education

B. Phase 2: Identifying a Standard and Assessing for Coverage against This Standard

As with any audit, a “gold-standard” should be identified. As the focus group discussion and Delphi method identified exam preparedness as the highest priority, we created a “blueprint” based on the syllabus of the MRCOG examination (Royal College of Obstetricians and Gynaecologists, 2019). This comprised more than 200 Knowledge Requirements organised more than 200 knowledge requirements into 15 Knowledge Areas (KAs) (Table 2). We mapped the old CME programme against this blueprint to understand the extent of coverage of these KAs. We analyse the session contents between January and December 2019. We felt the best way to ensure systematic coverage of these KAs would be through sessions with pre-identified areas of topical focus conducted during protected teaching time as opposed to opportunistic and voluntary learning opportunities that may not be widely available to all STs. In our department, this applied to morning CME sessions which indeed formed the bulk of protected teaching time for STs, required mandatory attendance, and comprised sessions covering pre-defined topics. Thus, we excluded didactic sessions where 1) the content of the presentations was unavailable for audit, 2) they covered administrative aspects and did not have a pre-identified topical focus where learning was opportunistic (e.g. risk management meetings, labour ward audits), and 3) where the attendance was optional.

Mapping was conducted independently by two members of the study team (JJYY and CYW) with conflict resolved by a third member (RJL). The number of knowledge requirements fulfilled within a KA were expressed as a percentage.

Core knowledge areas

Clinical skills

Teaching and research

Core surgical skills

Post operative care

Antenatal care

Maternal Medicine

Management of Labour

Management of delivery

Postpartum problems

Gynaecological problems

Subfertility

Sexual and reproductive health

Early pregnancy care

Gynaecological Oncology

Urogynaecology & pelvic floor problems

Table 2. RCOG Core Knowledge Areas (Royal College of Obstetricians and Gynaecologists, 2019)

C. Phase 3: Restructuring a PG Programme

The final phase i.e. the restructuring of a PG programme, is directed by responses to Steps 7-10 of the 10-step TNA (Table 1). As the focus of our article is on the methodology of auditing the extent of coverage of our departmental didactic sessions over our core ST learning goals i.e. the MRCOG KAs, these subsequent efforts are detailed in the discussion section.

III. RESULTS

Altogether, 71 presentations were identified (Table 3) of which 12 CME sessions (16.9%) were unavailable and, thus, excluded from the mapping exercise. The most common types of CME sessions presented clinical updates (31.0%), original research (29.6%), journal clubs (16.9%), and exam-preparation sessions (e.g. Case Based Discussion and OSCE simulations) (12.6%). The overall coverage of the entire syllabus was 64.9% (Figure 2). The KAs demonstrating complete coverage (i.e. 100% of all requirements) were “Teaching and Research”, “Postoperative Care” and “Early Pregnancy Care”. Three KAs had a coverage of 75-100% in the CME programme i.e. “Clinical Skills” (89%), “Urogynaecology and Pelvic Floor” (82%), and “Subfertility” (77%) while three were covered below 50% i.e. “Management of Labour”, “Management of Delivery”, “Postpartum Problems”, and “Gynaecological Problems”. These were more practical KAs that were usually covered during ward covers, operating theatre, clinics, and labour ward as well as during practical skills training workshops and grand ward rounds where clinical vignettes were opportunistically discussed depending on in-patient case mix. Nevertheless, this “on-the-ground” training is often unplanned, unstructured and ‘bite-sized’, thus complicating integration with the deep and broad guideline and knowledge proficiency that may be needed to train STs to adapt to complex situations.

Type of presentation	Number of sessions	Percentage breakdown
Clinical Updates	22	31.0%
Presentation of Original Research	21	29.6%
Journal Club	12	16.9%
Case Based Discussion	5	7.0%
OSCE practice	4	5.6%
Others*	2	2.8%
Audit	2	2.8%
Workshops	3	3.0%
Total	71	100%
*Others: ST Sharing of Overseas Experiences and Trainee Wellbeing

Table 3. Type of CME presentations

Figure 2. Graph showing the percentage coverage of knowledge areas

IV. DISCUSSION

Our audit revealed a relatively low coverage of the MRCOG KAs with only 64.9% of the syllabus covered. While the morning CME programme caters to all members of the department, the sessions are an important didactic component for ST education and exam preparation as they are deemed “protected” teaching time. There had been no prior formal review assessing whether it catered to this very important section of the department’s workforce. We were also able to recognise those KAs which had exceptionally low coverage were those with a large amount of practical and “hands-on” skills (i.e. “Gynaecological Problems”, “Management of Labour”, “Management of Delivery”, and “Postpartum Problems”). As a surgical discipline, this highlighted that these areas needed directed solutions through other forms of practical instruction and evaluation. In the pandemic environment, this may involve virtual or home-based means (Hoopes et al., 2020). These “hands-on” KAs likely require at least semi-annual individualised assessment by the CF through verified case logs, Objective Structured Assessment of Technical Skills, Direct Observation of Procedural Skills, and Non-Technical Skills for Surgeons (NOTSS) (Bisson et al., 2006; Parry-Smith et al., 2014). This targeted assessment was even more crucial during the recovery “catch-up” phase due to de-skilling because of reduced elective surgical caseload (Amparore et al., 2020; Chan et al., 2020) and facilitated the redistributing of surgical training material to cover training deficits.

While there is significant literature on how to organise a robust PG didactic programme (Colman et al., 2020; Harden, 2001; Willett, 2008), little has been published on how to evaluate an established didactic programme’s coverage of its learner’s educational requirements (Davies, 1981). Most studies evaluating the efficacy of such programmes typically assess the effects of individual CME sessions on physician knowledge or performance and patient outcomes after a suitable interval (Davis et al., 1992; Mansouri & Lockyer, 2007), with most citing a small to medium effect. We believe, however, that our audit process permits a more holistic, reproducible, and structured means of evaluating an existing didactic programme and finding deficits that can be improve upon to brings value to any specialty training programme.

At our institution, safe distancing requirements brought on by the COVID-19 pandemic required a rapid transition to a video-conferencing-based approach i.e. vCME. As milestone examinations were still being held, the first six months were primarily focused on STs as examination preparation remained a high and undisputed priority and learning opportunities had been significantly disrupted by the pandemic. During this phase, our vCME programme was re-organised into three to four sessions per week which were peer-led and supervised by a faculty member. Video-conferencing platforms encouraged audience participation through live feedback, questions posed via the chat box, instantaneous online polling, and directed case-based discussions with ST participants. These facilitated real time feedback to the presenter in a way that was not possible in previous face-to-face sessions due to reasons such as shyness and difficulty conducting polls. Other useful features included being able to record presentations for digital storage in a hospital-based server for access on-demand for revision purposes by STs.

A previously published anonymised questionnaire within our department (Chan et al., 2020) found very favourable opinions of vCME as an effective mode of learning amongst 28 junior doctors (85.7%) and nine presenters (100%) with 75% hoping for it to continue even after normalisation of social distancing policies. Nevertheless, common issues reported included a lack of personal interaction, difficulties in engaging with speakers, technical difficulties, and inaccurate attendance confirmation as shared devices for participating on these vCME sessions sometimes failed to identify who was present. While there is altered teacher and learner engagement due to physical separation across a digital medium, studies have also found that the virtual platform provided a useful means of communication and feedback and created a psychologically safe learning environment (Dong et al., 2021; Wasfy et al., 2021).

While our audit focused primarily on STs, departmental CME programmes need to find balanced in catering to the educational outcome of various groups of participants within a clinical department (e.g. senior clinicians, nursing staff, allied healthcare professionals, clinical scientists). Indeed, as these groups started to return to the CME programme after about six months following the vCME transition, we created a core postgraduate committee comprising members representing the learning interests of each party i.e. Department research director, ST Programme Director and Assistant Director, and a representative senior ST in the fifth or sixth year of training, so that we could continue to meet the recommendations set in our TNA while rebalancing the programme to meet the needs of all participants. Out of an average of 20 CME sessions per month, four were dedicated to departmental and hospital grand rounds each. Of the remaining 12 sessions, two were dedicated towards covering KAs, four scientific presentations, three clinical governance aspects, and one journal club. The remaining two sessions would be “faculty wildcard” sessions to be used at the committee’s discretion of the committee to cover poorly covered, more contemporary, “breaking news” topics, or serve as a buffer in the event of cancellations of other topics. Indeed, the same TNA-based audit methodology can be employed any other group of CME participants.

A key limitation in our audit method is that it focused on the breadth of coverage of learning objectives, but not the quality of the teaching and its depth. Teaching efficacy is also important in the delivery of learning objectives (Bakar et al., 2012) and needs more specific assessment tools (Metheny et al., 2005). Evaluating the quality of PG training could take several forms and may be direct e.g. an evaluation by the learner (Gillan et al., 2011), or could be indirect e.g. charting the learner’s progress through OSCEs and CEXs, scheduled competency reviews, and ST examination pass rates (Pinnell et al., 2021). Importantly, despite the rise of virtual learning platforms, there is little consensus on the best way to evaluate e-learning methods (De Leeuw et al., 2019). Nevertheless, our main audit goal was to assess the extent of coverage of the MRCOG syllabus which is a key training outcome. Future audits, however, should incorporate this element to provide additional qualitative feedback to assess this dimension as well. Further research should be carried out in terms of evaluating the effects of optimising a PG didactic programme on key outcomes such as ST behaviour, perceptions, and objective outcomes such as examination results.

Finally, while these were the results of an audit conducted in a single hospital department and used a morning CME programme as a basis for evaluation, we believe that this audit methodology based on a ten-step TNA and also utilising the Delphi method and syllabus mapping techniques (Harden, 2001) can be reproduced to any academic department that has a regular didactic programme as long as a suitable standard is selected. The Delphi method can easily be conducted via online survey platforms (e.g. Google Forms) to crystallise the PG programme goals. Our audit shows that without a systematic evaluation of past didactic sessions, it is possible for even mature CME programme to fall significantly short of ameeting the needs of its learners and that PG didactic sessions need deliberate planning.

V. CONCLUSION

Just as any other aspect of healthcare delivery, CME and PG programmes are amenable to audits and must adjust to an ever-changing delivery landscape. Rather than curse the darkness during the COVID-19 pandemic, we explored the potential of reformatting the PG programme and adjusting course to better suit the needs of our STs. We demonstrate a method of auditing an existing programme, distilling important learning goals, comparing it against an appropriate standard (i.e. coverage of the MRCOG KAs), and implementing changes utilising reproducible techniques such as the Delphi method (Humphrey-Murto et al., 2017). This process should be a regular mainstay of any mature ST programme to ensure continued relevancy. As continual outbreaks, even amongst vaccinated populations (Amit et al., 2021; Bar-On et al., 2021; Bergwerk et al., 2021) auger a future of COVID-19 endemicity, we must accept a “new-normal” comprising of intermittent workplace infection control policies such as segregation, shift work, and restrictions for in-person meetings (Kwon et al., 2020; Liang et al., 2020). Through our experience, we have shown that this auditing methodology can also be applied to vCME programmes.

Notes on Contributors

Rachel Jiayu Lee participated in the data collection and review, the writing of the paper, and the formatting for publication.

Jeannie Jing Yi Yap participated in the data collection and review, the writing of the paper, and the formatting for publication.

Carly Yanlin Wu participated in data collection and review.

Grace Chan Ming Fen participated in data collection and review.

Abhiram Kanneganti was involved in the writing of the paper, editing, and formatting for publication. Citra Nurfarah Zaini Mattar participated in the editing and direction of the paper.

Pearl Shuang Ye Tong participated in the editing and direction of the paper.

Susan Jane Sinclair Logan participated in the editing and direction of the paper.

Ethical Approval

IRB approval for waiver of consent (National Healthcare Group DSRB 2020/00360) was obtained for the questionnaire assessing attitudes towards vCME.

Data Availability

There is no relevant data available for sharing in this paper.

Acknowledgement

We would like to acknowledge the roles of Mr Xiu Cai Wong Edwin, Mr Lee Boon Kai and Ms Teo Xin Yue in the administrative roles behind auditing and reformatting the PG medical education programme.

Funding

There was no funding for this article.

Declaration of Interest

The authors have no conflicts of interest in connection with this article.

References

Amit, S., Beni, S. A., Biber, A., Grinberg, A., Leshem, E., & Regev-Yochay, G. (2021). Postvaccination COVID-19 among healthcare workers, Israel. Emerging Infectious Diseases, 27(4), 1220. https://doi.org/10.3201/eid2704.210016

Amparore, D., Claps, F., Cacciamani, G. E., Esperto, F., Fiori, C., Liguori, G., Serni, S., Trombetta, C., Carini, M., Porpiglia, F., Checcucci, E., & Campi, R. (2020). Impact of the COVID-19 pandemic on urology residency training in Italy. Minerva Urologica e Nefrologica, 72(4), 505-509. https://doi.org/10.23736/s0393-2249.20.03868-0

Bakar, A. R., Mohamed, S., & Zakaria, N. S. (2012). They are trained to teach, but how confident are they? A study of student teachers’ sense of efficacy. Journal of Social Sciences, 8(4), 497-504. https://doi.org/10.3844/jssp.2012.497.504

Bar-On, Y. M., Goldberg, Y., Mandel, M., Bodenheimer, O., Freedman, L., Kalkstein, N., Mizrahi, B., Alroy-Preis, S., Ash, N., Milo, R., & Huppert, A. (2021). Protection of BNT162b2 vaccine booster against Covid-19 in Israel. New England Journal of Medicine, 385(15), 1393-1400. https://doi.org/10.1056/NEJMoa2114255

Bergwerk, M., Gonen, T., Lustig, Y., Amit, S., Lipsitch, M., Cohen, C., Mandelboim, M., Levin, E. G., Rubin, C., Indenbaum, V., Tal, I., Zavitan, M., Zuckerman, N., Bar-Chaim, A., Kreiss, Y., & Regev-Yochay, G. (2021). Covid-19 breakthrough infections in vaccinated health care workers. New England Journal of Medicine, 385(16), 1474-1484. https://doi.org/10.1056/NEJMoa2109072

Bienstock, J., & Heuer, A. (2022). A review on the evolution of simulation-based training to help build a safer future. Medicine, 101(25), Article e29503. https://doi.org/10.1097/MD.0000000000029503

Bisson, D. L., Hyde, J. P., & Mears, J. E. (2006). Assessing practical skills in obstetrics and gynaecology: Educational issues and practical implications. The Obstetrician & Gynaecologist, 8(2), 107-112. https://doi.org/10.1576/toag.8.2.107.27230

Bryant‐Smith, A., Rymer, J., Holland, T., & Brincat, M. (2019). ‘Perfect practice makes perfect’: The role of laparoscopic simulation training in modern gynaecological training. The Obstetrician & Gynaecologist, 22(1), 69-74. https://doi.org/10.1111/tog.12619

Burr, R., & Johanson, R. (1998). Continuing medical education: An opportunity for bringing about change in clinical practice. British Journal of Obstetrics and Gynaecology, 105(9), 940-945. https://doi.org/10.1111/j.1471-0528.1998.tb10255.x

Chan, G. M. F., Kanneganti, A., Yasin, N., Ismail-Pratt, I., & Logan, S. J. S. (2020). Well-being, obstetrics and gynaecology and COVID-19: Leaving no trainee behind. Australian and New Zealand Journal of Obstetrics and Gynaecology, 60(6), 983-986. https://doi.org/10.1111/ajo.13249

Chick, R. C., Clifton, G. T., Peace, K. M., Propper, B. W., Hale, D. F., Alseidi, A. A., & Vreeland, T. J. (2020). Using technology to maintain the education of residents during the COVID-19 pandemic. Journal of Surgical Education, 77(4), 729-732. https://doi.org/10.1016/j.jsurg.2020.03.018

Colman, S., Wong, L., Wong, A. H. C., Agrawal, S., Darani, S. A., Beder, M., Sharpe, K., & Soklaridis, S. (2020). Curriculum mapping: an innovative approach to mapping the didactic lecture series at the University of Toronto postgraduate psychiatry. Academic Psychiatry, 44(3), 335-339. https://doi.org/10.1007/s40596-020-01186-0

Davies, I. J. T. (1981). The assessment of continuing medical education. Scottish Medical Journal, 26(2), 125-134. https://doi.org/10.1177/003693308102600208

Davis, D. A., Thomson, M. A., Oxman, A. D., & Haynes, R. B. (1992). Evidence for the effectiveness of CME: A Rreview of 50 randomized controlled trials. Journal of the American Medical Association, 268(9), 1111-1117. https://doi.org/10.1001/jama.1992.03490090053014

De Leeuw, R., De Soet, A., Van Der Horst, S., Walsh, K., Westerman, M., & Scheele, F. (2019). How we evaluate postgraduate medical e-learning: systematic review. JMIR medical education, 5(1), e13128.

Dong, C., Lee, D. W.-C., & Aw, D. C.-W. (2021). Tips for medical educators on how to conduct effective online teaching in times of social distancing. Proceedings of Singapore Healthcare, 30(1), 59-63. https://doi.org/10.1177/2010105820943907

Donovan, P., & Townsend, J. (2019). Learning Needs Analysis. Management Pocketbooks.

English, W., Vulliamy, P., Banerjee, S., & Arya, S. (2020). Surgical training during the COVID-19 pandemic – The cloud with a silver lining? British Journal of Surgery, 107(9), e343-e344. https://doi.org/10.1002/bjs.11801

Forsetlund, L., O’Brien, M. A., Forsén, L., Mwai, L., Reinar, L. M., Okwen, M. P., Horsley, T., & Rose, C. J. (2021). Continuing education meetings and workshops: Effects on professional practice and healthcare outcomes. Cochrane Database of Systematic Reviews, 9(9), CD003030. https://doi.org/10.1002/14651858.CD003030.pub3

Gillan, C., Lovrics, E., Halpern, E., Wiljer, D., & Harnett, N. (2011). The evaluation of learner outcomes in interprofessional continuing education: A literature review and an analysis of survey instruments. Medical Teacher, 33(9), e461-e470. https://doi.org/10.3109/0142159X.2011.587915

Gould, D., Kelly, D., White, I., & Chidgey, J. (2004). Training needs analysis. A literature review and reappraisal. International Journal of Nursing Studies, 41(5), 471-486. https://doi.org/10.1016/j.ijnurstu.2003.12.003

Grafton-Clarke, C., Uraiby, H., Gordon, M., Clarke, N., Rees, E., Park, S., Pammi, M., Alston, S., Khamees, D., Peterson, W., Stojan, J., Pawlik, C., Hider, A., & Daniel, M. (2022). Pivot to online learning for adapting or continuing workplace-based clinical learning in medical education following the COVID-19 pandemic: A BEME systematic review: BEME Guide No. 70. Medical Teacher, 44(3), 227-243. https://doi.org/10.1080/0142159X.2021.1992372

Harden, R. M. (2001). AMEE Guide No. 21: Curriculum mapping: A tool for transparent and authentic teaching and learning. Medical Teacher, 23(2), 123-137. https://doi.org/10.1080/01421590120036547

Hasson, F., Keeney, S., & McKenna, H. (2000). Research guidelines for the Delphi survey technique. Journal of Advanced Nursing, 32(4), 1008-1015. https://doi.org/10.1046/j.1365-2648.2000.t01-1-01567.x

Hicks, C., & Hennessy, D. (1996). Applying psychometric principles to the development of a training needs analysis questionnaire for use with health visitors, district and practice nurses. NT Research, 1(6), 442-454. https://doi.org/10.1177/174498719600100608

Hicks, C., & Hennessy, D. (1997). The use of a customized training needs analysis tool for nurse practitioner development. Journal of Advanced Nursing, 26(2), 389-398. https://doi.org/https://doi.org/10.1046/j.1365-2648.1997.199702 6389.x

Hoopes, S., Pham, T., Lindo, F. M., & Antosh, D. D. (2020). Home surgical skill training resources for obstetrics and gynecology trainees during a pandemic. Obstetrics and Gynecology 136(1), 56-64. https://doi.org/10.1097/aog.0000000000003931

Humphrey-Murto, S., Varpio, L., Wood, T. J., Gonsalves, C., Ufholz, L.-A., Mascioli, K., Wang, C., & Foth, T. (2017). The use of the Delphi and other consensus group Methods in medical education research: A review. Academic Medicine, 92(10), 1491-1498. https://doi.org/10.1097/ACM.0000000000001812

Johnston, S., Coyer, F. M., & Nash, R. (2018). Kirkpatrick’s evaluation of simulation and debriefing in health care education: a systematic review. Journal of Nursing Education, 57(7), 393-398. https://doi.org/10.3928/01484834-20180618-03

Juprasert, J. M., Gray, K. D., Moore, M. D., Obeid, L., Peters, A. W., Fehling, D., Fahey, T. J., III, & Yeo, H. L. (2020). Restructuring of a general surgery residency program in an epicenter of the Coronavirus Disease 2019 pandemic: Lessons from New York City. JAMA Surgery, 155(9), 870-875. https://doi.org/10.1001/jamasurg.2020.3107

Kanneganti, A., Lim, K. M. X., Chan, G. M. F., Choo, S.-N., Choolani, M., Ismail-Pratt, I., & Logan, S. J. S. (2020). Pedagogy in a pandemic – COVID-19 and virtual continuing medical education (vCME) in obstetrics and gynecology. Acta Obstetricia et Gynecologica Scandinavica, 99(6), 692-695. https://doi.org/10.1111/aogs.13885

Kanneganti, A., Sia, C.-H., Ashokka, B., & Ooi, S. B. S. (2020). Continuing medical education during a pandemic: An academic institution’s experience. Postgraduate Medical Journal, 96, 384-386. https://doi.org/10.1136/postgradmedj-2020-137840

Khamees, D., Peterson, W., Patricio, M., Pawlikowska, T., Commissaris, C., Austin, A., Davis, M., Spadafore, M., Griffith, M., Hider, A., Pawlik, C., Stojan, J., Grafton-Clarke, C., Uraiby, H., Thammasitboon, S., Gordon, M., & Daniel, M. (2022). Remote learning developments in postgraduate medical education in response to the COVID-19 pandemic – A BEME systematic review: BEME Guide No. 71. Medical Teacher, 44(5), 466-485. https://doi.org/10.1080/0142159X.2022.2040732

Kwon, Y. S., Tabakin, A. L., Patel, H. V., Backstrand, J. R., Jang, T. L., Kim, I. Y., & Singer, E. A. (2020). Adapting urology residency training in the COVID-19 era. Urology, 141, 15-19. https://doi.org/10.1016/j.urology.2020.04.065

Liang, Z. C., Ooi, S. B. S., & Wang, W. (2020). Pandemics and their impact on medical training: Lessons from Singapore. Academic Medicine, 95(9), 1359-1361. https://doi.org/10.1097/ACM.0000000000003441

Lim, E. C., Oh, V. M., Koh, D. R., & Seet, R. C. (2009). The challenges of “continuing medical education” in a pandemic era. Annals Academy of Medicine Singapore, 38(8), 724-726. https://www.ncbi.nlm.nih.gov/pubmed/19736579

Mansouri, M., & Lockyer, J. (2007). A meta-analysis of continuing medical education effectiveness. Journal of Continuing Education in the Health Professions, 27(1), 6-15. https://doi.org/10.1002/chp.88

Marinopoulos, S. S., Dorman, T., Ratanawongsa, N., Wilson, L. M., Ashar, B. H., Magaziner, J. L., Miller, R. G., Thomas, P. A., Prokopowicz, G. P., Qayyum, R., & Bass, E. B. (2007). Effectiveness of continuing medical education. Evid Rep Technol Assess (Full Rep), 149, 1-69. https://www.ncbi.nlm.nih.gov/pubmed/17764217

Markaki, A., Malhotra, S., Billings, R., & Theus, L. (2021). Training needs assessment: Tool utilization and global impact. BMC Medical Education, 21(1), Article 310. https://doi.org/10.1186/s12909-021-02748-y

Metheny, W. P., Espey, E. L., Bienstock, J., Cox, S. M., Erickson, S. S., Goepfert, A. R., Hammoud, M. M., Hartmann, D. M., Krueger, P. M., Neutens, J. J., & Puscheck, E. (2005). To the point: Medical education reviews evaluation in context: Assessing learners, teachers, and training programs. American Journal of Obstetrics and Gynecology, 192(1), 34-37. https://doi.org/10.1016/j.ajog.2004.07.036

Mishra, K., Boland, M. V., & Woreta, F. A. (2020). Incorporating a virtual curriculum into ophthalmology education in the coronavirus disease-2019 era. Current Opinion in Ophthalmology, 31(5), 380-385. https://doi.org/10.1097/icu.0000000000000681

Norman, G. R., Shannon, S. I., & Marrin, M. L. (2004). The need for needs assessment in continuing medical education. BMJ (Clinical research ed.), 328(7446), 999-1001. https://doi.org/10.1136/bmj.328.7446.999

Palmer, W., & Brackwell, L. (2014). A national audit of maternity services in England. BJOG, 121(12), 1458-1461. https://doi.org/10.1111/1471-0528.12973

Parry-Smith, W., Mahmud, A., Landau, A., & Hayes, K. (2014). Workplace-based assessment: A new approach to existing tools. The Obstetrician & Gynaecologist, 16(4), 281-285. https://doi.org/10.1111/tog.12133

Pinnell, J., Tranter, A., Cooper, S., & Whallett, A. (2021). Postgraduate medical education quality metrics panels can be enhanced by including learner outcomes. Postgraduate Medical Journal, 97(1153), 690-694. https://doi.org/10.1136/postgradmedj-2020-138669

Raza, A., Coomarasamy, A., & Khan, K. S. (2009). Best evidence continuous medical education. Archives of Gynecology and Obstetrics, 280(4), 683-687. https://doi.org/10.1007/s00404-009-1128-7

Renaud, C. J., Chen, Z. X., Yuen, H.-W., Tan, L. L., Pan, T. L. T., & Samarasekera, D. D. (2021). Impact of COVID-19 on health profession education in Singapore: Adoption of innovative strategies and contingencies across the educational continuum. The Asia Pacific Scholar, 6(3), 14-23. https://doi.org/10.29060/TAPS.2021-6-3/RA2346

Royal College of Obstetricians and Gynaecologists. (2019). Core Curriculum for Obstetrics & Gynaecology: Definitive Document 2019. https://www.rcog.org.uk/media/j3do0i1i/core-curriculum-2019-definitive-document-may-2021.pdf

Sibley, J. C., Sackett, D. L., Neufeld, V., Gerrard, B., Rudnick, K. V., & Fraser, W. (1982). A randomized trial of continuing medical education. New England Journal of Medicine, 306(9), 511-515. https://doi.org/10.1056/NEJM198203043060904

Sparkes, D., Leong, C., Sharrocks, K., Wilson, M., Moore, E., & Matheson, N. J. (2021). Rebooting medical education with virtual grand rounds during the COVID-19 pandemic. Future Healthc J, 8(1), e11-e14. https://doi.org/10.7861/fhj.2020-0180

Wasfy, N. F., Abouzeid, E., Nasser, A. A., Ahmed, S. A., Youssry, I., Hegazy, N. N., Shehata, M. H. K., Kamal, D., & Atwa, H. (2021). A guide for evaluation of online learning in medical education: A qualitative reflective analysis. BMC Medical Education, 21(1), Article 339. https://doi.org/10.1186/s12909-021-02752-2

Willett, T. G. (2008). Current status of curriculum mapping in Canada and the UK. Medical Education, 42(8), 786-793. https://doi.org/10.1111/j.1365-2923.2008.03093.x

Wong, T. Y., & Bandello, F. (2020). Academic ophthalmology during and after the COVID-19 pandemic. Ophthalmology, 127(8), e51-e52. https://doi.org/10.1016/j.ophtha.2020.04.029

Xu, J., Zhou, Z., Chen, K., Ding, Y., Hua, Y., Ren, M., & Shen, Y. (2022). How to minimize the impact of COVID-19 on laparoendoscopic single-site surgery training? ANZ Journal of Surgery, 92, 2102-2108. https://doi.org/10.1111/ans.17819

*Abhiram Kanneganti
Department of Obstetrics and Gynaecology,
NUHS Tower Block, Level 12,
1E Kent Ridge Road,
Singapore 119228
Email: abhiramkanneganti@gmail.com

Announcements

Best Reviewer Awards 2024
TAPS would like to express gratitude and thanks to an extraordinary group of reviewers who are awarded the Best Reviewer Awards for 2024.
Refer here for the list of recipients.
Most Accessed Article 2024
The Most Accessed Article of 2024 goes to Persons with Disabilities (PWD) as patient educators: Effects on medical student attitudes.
Congratulations, Dr Vivien Lee and co-authors!
Best Article Award 2024
The Best Article Award of 2024 goes to Achieving Competency for Year 1 Doctors in Singapore: Comparing Night Float or Traditional Call.
Congratulations, Dr Tan Mae Yue and co-authors!
Fourth Thematic Issue: Call for Submissions
The Asia Pacific Scholar is now calling for submissions for its Fourth Thematic Publication on “Developing a Holistic Healthcare Practitioner for a Sustainable Future”!
The Guest Editors for this Thematic Issue are A/Prof Marcus Henning and Adj A/Prof Mabel Yap. For more information on paper submissions, check out here!
Best Reviewer Awards 2023
TAPS would like to express gratitude and thanks to an extraordinary group of reviewers who are awarded the Best Reviewer Awards for 2023.
Refer here for the list of recipients.
Most Accessed Article 2023
The Most Accessed Article of 2023 goes to Small, sustainable, steps to success as a scholar in Health Professions Education – Micro (macro and meta) matters.
Congratulations, A/Prof Goh Poh-Sun & Dr Elisabeth Schlegel!
Best Article Award 2023
The Best Article Award of 2023 goes to Increasing the value of Community-Based Education through Interprofessional Education.
Congratulations, Dr Tri Nur Kristina and co-authors!
Volume 9 Number 1 of TAPS is out now! Click on the Current Issue to view our digital edition.

Best Reviewer Awards 2022
TAPS would like to express gratitude and thanks to an extraordinary group of reviewers who are awarded the Best Reviewer Awards for 2022.
Refer here for the list of recipients.
Most Accessed Article 2022
The Most Accessed Article of 2022 goes to An urgent need to teach complexity science to health science students.
Congratulations, Dr Bhuvan KC and Dr Ravi Shankar.
Best Article Award 2022
The Best Article Award of 2022 goes to From clinician to educator: A scoping review of professional identity and the influence of impostor phenomenon.
Congratulations, Ms Freeman and co-authors.
Volume 8 Number 3 of TAPS is out now! Click on the Current Issue to view our digital edition.

Best Reviewer Awards 2021
TAPS would like to express gratitude and thanks to an extraordinary group of reviewers who are awarded the Best Reviewer Awards for 2021.
Refer here for the list of recipients.
Most Accessed Article 2021
The Most Accessed Article of 2021 goes to Professional identity formation-oriented mentoring technique as a method to improve self-regulated learning: A mixed-method study.
Congratulations, Assoc/Prof Matsuyama and co-authors.
Best Reviewer Awards 2020
TAPS would like to express gratitude and thanks to an extraordinary group of reviewers who are awarded the Best Reviewer Awards for 2020.
Refer here for the list of recipients.
Most Accessed Article 2020
The Most Accessed Article of 2020 goes to Inter-related issues that impact motivation in biomedical sciences graduate education. Congratulations, Dr Chen Zhi Xiong and co-authors.

Defining undergraduate medical students’ physician identity: Learning from Indonesian experience

Announcements

Disclaimer