Submitted: 31 July 2024
Accepted: 24 February 2025
Published online: 1 July, TAPS 2025, 10(3), 37-48
https://doi.org/10.29060/TAPS.2025-10-3/OA3473

Kevin Xuan Hong Tang¹, Koon Kee Teo¹ & Kye Mon Min Swe²

¹Department of Medicine, Faculty of Medicine, Newcastle University Medicine Malaysia (NUMed), Malaysia; ²Department of Research, Faculty of Medicine, Newcastle University Medicine Malaysia (NUMed), Malaysia

Abstract

Introduction: Every medical graduate is expected to fulfil the teaching responsibilities stated by the General Medical Council (GMC). It is beneficial to nurture both teaching motivation and skills early in the undergraduate program. This study aims to evaluate the outcomes of final-year medical students as near-peer teachers in a student-led near-peer teaching program and their fulfilment of the educational responsibilities stated by the GMC.

Methods: A cross-sectional study was conducted among the year 5 medical students who participated in the Peer Teaching Program. A structured post-participation 6-point Likert scale questionnaire with written consent was distributed to the near-peer teachers to assess their perspectives on skills enhancement, motivation, and career direction. Additionally, the Peer Tutor Assessment Instrument questionnaires were distributed to the near-peer students to evaluate the performance of the near-peer teachers in five areas: responsibility and respect, information processing, communication, critical analysis, and self-awareness.

Results: There were 28 near-peer teachers, and 49 near-peer students participated in the study.  The near-peer teachers score the highest in skills (5.36 ± 0.53), followed by motivation (5.16 ± 0.60) and career direction (4.79 ± 0.82). Three quarters of the near-peer teachers considered teaching to be their future primary career path after experiencing this teaching experience (4.36 ±1.34). Generally, the near-peer teachers were highly evaluated by the near-peer students across all domains (5.06 ± 0.51).

Conclusion: Overall, the near-peer teaching programme likely improved the final-year medical students in fulfilling the “Doctors as Teachers” responsibilities outlined by the GMC.

Keywords:           Near-peer Teaching, Medical Students, Undergraduate Medical Education, General Medical Council, Doctors as Teachers

Practice Highlights

• Near-peer teachers are likely improved in skills enhancement, motivation and career direction.
• Sex and students’ background are not associated with the perceived outcomes of near-peer teachers.

I. INTRODUCTION

One of the aspects of Good Medical Practice outlined by the General Medical Council (GMC) for all medical professionals is to “be willing to offer professional support to colleagues, including students, through teaching” (General Medical Council, 2023). The role of doctors as teachers has been widely recognised as they need to teach and educate juniors, students and even patients (General Medical Council, 2011, 2015, 2023). On average, junior doctors spend around 80 minutes per day teaching medical students (Busari et al., 2002). This task is daunting for every new medical graduate who has just begun their UK foundation programme. They need to assume this responsibility with minimal formal training and preparations (Pierce et al., 2024; Qureshi et al., 2013).

Therefore, it is beneficial to motivate medical graduates to teach and equip them with appropriate teaching skills as early as their undergraduate programme (General Medical Council, 2011; Knobloch et al., 2018). Near-peer teaching involves students one or more academic years ahead teaching their peers or junior students (De Menezes & Premnath, 2016; Ten Cate & Durning, 2007; Yu et al., 2011). This has long been thought of as a programme to be incorporated into the medical curriculum to optimise teaching qualities and to produce more competent and knowledgeable doctors in the future (Botelho et al., 2022; Burgess et al., 2014; Zheng & Wang, 2022). Generally, medical schools provide a safe space for medical students to practice, correct and improve their teaching and pedagogical skills (Hardie et al., 2022). Most medical students feel less daunted and more supported involved in teaching their near-peer students (Yu et al., 2011).

To address this gap of insufficient teaching opportunities, most medical schools provide near-peer teaching programmes for their medical student (Frearson & Gale, 2017). However, most of the near-peer teaching programmes are carried out formally with structured guidance and training (General Medical Council, 2011), be it in the form of the Peer Assisted Learning Scheme (PALS) student-selected components (SSC) (Furmedge et al., 2014; Hettle & Morgan, 2019; Ross & Cameron, 2007; Ten Cate & Durning, 2007) or Doctors as Teachers and Educators training course (Cook et al., 2010; General Medical Council, 2011). Little is known about the outcomes of the student-led, student-run, near-peer teaching (NPT) programme in medical schools.

In the academic year 2023/2024, the Newcastle University Medicine Malaysia (NUMed) final-year medical students were involved as near-pear teachers in an NPT programme. This study thus aims to evaluate the outcomes of participation of the final year medical students as near-peer teachers in the student-led near-peer teaching programme and to determine whether the soon-to-be medical graduates can fulfil the “Doctors as Teachers” responsibilities stated by the GMC.  

II. METHODS

The NPT Programme was a purely student-led, student-run 3-month teaching programme which provided additional focus on the learning outcomes of the third-year medical curriculum. This programme functioned as an adjunct to the formal curriculum and provided precious opportunities for final-year medical students to improve their teaching skills.

Before the academic year started, invitation email was sent out to recruit final-year medical students to participate voluntarily as the near-peer teachers and the year 3 medical students as the near-peer students. A total of 51 final-year medical students and 100 year 3 medical students signed up for this programme. The near-peer students were randomly assigned to groups of 5 to 6 each, and each group was guided by 3 near-peer teachers. Before the programme commenced, all near-peer teachers were required to attend a mandatory online training course conducted by the lecturers to enhance their presentation and teaching skills.

This programme comprised a total of twelve teaching sessions spanning twelve weeks, covering four sessions of Essential Clinical Placement teaching sessions, four sessions of Case-Based Discussion, one surgical teaching topic and case-based session, one Single Best Answer practice session, one Written Prescribing Exam (WRISKE) session and one Objective Structural Clinical Examination (OSCE) session. Most sessions were delivered virtually (Zoom) or physically, depending on students’ preferences, except for the OSCE session, which was always conducted physically. The teaching materials were prepared by the near-peer teachers beforehand and distributed to the students after each teaching session. The NPT programme coordinator supervised and provided necessary support to both near-peer teachers and near-peer students throughout the entire programme.

The near-peer teachers and near-peer students consented to participate in this study via written consent. A structured post-participation Likert 6-point scale “Peer Tutors Own Assessment” questionnaire with written consent, which was adopted from Liew et al. (2015), was sent to the near-peer teachers after this Near-Peer Teaching Programme via Google form to explore their perceived benefits in three components which are 1) Skills Enhancement, 2) Motivation and 3) Career Direction [Expectation]. This questionnaire (Appendix 1) contains 14 items, with responses scale from strongly disagree (1) to strongly agree (6). Cronbach’s alpha was 0.801, 0.714, and 0.814 for the domains of skills enhancement, motivation, and career direction.

Similarly, all the near-peer students who participated in this near-peer teaching programme were given the Peer Tutor Assessment questionnaire adopted from Liew et al. (2015), to fill in via Google form (Appendix 2). This questionnaire is to assess the acceptability of the teachings of the near-peer teachers. It contains 16 items that evaluate five domains: (1) Responsibility and Respect, (2) Information Processing, (3) Communication, (4) Critical Analysis, and (5) Self-Awareness. Subgroup analyses were conducted to evaluate whether sex and student background affect self-perceived outcomes of near-peer teachers using independent T-test. Each participant was given 3 weeks to complete the questionnaire. Several reminders were sent via email throughout these 3 weeks to each participant to encourage them to fill in the questionnaire. Both near-peer teachers and near-peer students’ data were checked for normality. The asymmetry fell between -1 and +1 and assumed relatively symmetrical and mesokurtic.

III. RESULTS

A total of 51 medical students in their final year signed up as near-peer teachers. Of these, 28 near-peer teachers completed the questionnaire (response rate 54.9%), while 49 out of 100 near-peer students who joined this NPT programme responded in this study (response rate 49.0%). Among those near-peer teachers who responded, there were 9 (32.1%) males and 19 (67.9%) females. More local students responded in this study than international students (75% vs 25%). The overall mean age ± SD of the near-peer teachers is 23.75 ±1.21 years old.  For the near-peer students, the overall mean age ± SD is 21.69 ± 0.74 years old. The number of international and local near-peer students who responded was similar. The data of the responses of both near-peer teachers and near-peer students that supports the findings of this study  is openly available at Figshare https://doi.org/10.6084/m9.figshare.26886517.v1 (Tang et al., 2024a) and https://doi.org/10.6084/m9.figshare.26886514.v1 (Tang et al., 2024b).

Table 1 showed peer review findings to ensure their voices were represented. To ensure the trustworthiness of the findings, actions were taken to address credibility, dependability, confirmability, transferability, and reflexitivity are outlined in Table 2. Figure 1 indicates the near-peer teachers view of the benefits of involving in the student-led near-peer teaching program.

Table 1. Demographic data of the near-peer teachers

Table 2. The mean score (± SD) for the self-evaluation of near-peer teachers in relation to (1) skills enhancement, (2) motivation and (3) career direction (expectation)

Figure 1. Near-peer teachers view of the benefits of involving in the student-led NPT program: Analysis of post-participation questions responses (n=28) in Likert 6-point scale in relation to (a) skills enhancement, (b) motivation and (c) career direction (expectation)

Based on the responses to the questionnaire, the near-pear teachers widely agreed that the NPT programme positively impacted them (5.11 ± 0.58), with the highest score in the domain of skills enhancement (5.36 ± 0.53), followed by motivation (5.16 ± 0.60) and career direction (4.79 ± 0.82). Most of the near-peer teachers considered this programme improved their skills in terms of teaching, organising, communicating and understanding educational principles (Table 2). 100% of them agreed that this NPT programme increased their confidence in speaking to groups and enhanced their planning and organising skills.  Furthermore, all 28 respondents (100%) reported being more motivated to revise their own clinical skills and focus more on their practical skills after teaching near-peer students to address the

knowledge gap. A significant proportion of near-peer teachers felt more encouraged to participate in more teaching programmes in the future (n=27, 96.43%) and more inclined to help fellow students next time (n=25, 89.28%).  It is noteworthy that 96.43% of near-peer teachers agreeing that this NPT programme helped in deciding their career direction and motivated them to undertake more trainings. Besides, three quarters of them would consider teaching as their major career pathway in the future, with a mean score of 4.36 ± 1.34. Lastly, all 28 respondents (100%) agreed that joining the NPT programme had enhanced their curriculum vitae (100%).

Table 3 shows the mean score of near-peer students’ evaluation of near-peer teachers in five domains after joining the near-peer teaching programme.

Table 3. The mean score (± SD) for the near-peer students’ evaluation of near-peer teachers after the NPT programme in relation to (1) responsibility and respect, (2) information processing, (3) communication, (4) critical analysis and (5) self-awareness

When asked to evaluate the teaching of their near-peer teachers, the near-peer students considered the near-peer teachers demonstrating positive outcomes in all five domains (5.06 ± 0.51). The near-peer teachers were thought to have a high degree of professionalism in terms of responsibility and respect (5.15 ± 0.57) and self-awareness (5.11 ± 0.62). The most outstanding attribute demonstrated was the ability to complete assigned tasks appropriately (5.27 ± 0.73). Besides, the near-peer teachers performed satisfactorily to process information (5.12± 0.66), communicate (5.01± 0.65) and analyse critically (4.93 ± 0.66). However, the near-peer teachers were identified to score slightly lower in making non-confusing comments and responses (4.92 ± 0.84) as well as giving a summary of the session (4.90 ± 0.82).

Variables	Independent T- test
	Mean difference (95% CI)	P-value
Sex (Male vs Female)
Skills	-0.101 (-0.549, 0.348)	0.649
Motivation	0.118 (-0.385, 0.621)	0.634
Career Direction (Expectation)	0.507 (-0.154, 1.169)	0.127
Students’ background (Local vs International)
Skills	-0.210 (-0.688, 0.268)	0.376
Motivation	-0.276 (-0.810, 0.258)	0.297
Career Direction (Expectation)	0.280 (-0.439, 1.034)	0.414

Table 4. Comparison of self-perceived outcomes of near-peer teachers between male and female, local and international students (Independent t-test)

The independent t-test were performed to find out the association between sex and perceived outcomes of the near-peer teachers. Our study revealed that it is statistically insignificant between male and female in the perceived benefits for skills (P = 0.649), motivation (P = 0.549) and career direction (P = 0.127).

Besides, there is no correlation between students’ background and the three measured outcomes. There is statistically insignificant between local and international students in term of skills (P = 0.376), motivation (P = 0.397) and career direction (P = 0.414).

IV. DISCUSSION

This study provides valuable insights into the background and characteristics of the final-year medical students who voluntarily participated in a student-led, student-run NPT programme. The outcomes of their participation concerning their perceived benefits in terms of skills, motivation, and expectations were investigated. Overall, the near-peer teachers reported that this NPT programme helped them tremendously to improve their skills in terms of learning and teaching, which might be driven by their primary motive for joining this programme. This finding was similar to previous studies, which showed skills enhancement in volunteer near-peer teachers (Buckley & Zamora, 2007; Liew et al., 2015). Our study further reaffirms the plausibility of a student-run NPT programme to enhance teaching and learning skills. However, due to the voluntary nature of participation in this near-peer teaching programme, the students who are likely most in need of skill enhancement may have been omitted from this programme, and they might be less equipped to teach after they graduate. Some studies recommended more incentives to be given to such students to encourage them to make use of the opportunities offered (Buckley & Zamora, 2007).

The motivation evaluated includes both self-actualising inner motivations to improve their clinical and practical skills and the external, tangible desire to help fellow students. The high motivation score suggests the reinforcement of a desirable attitude towards future educational and teaching responsibilities, which matches the GMC’s emphasis on the teaching role of doctors (General Medical Council, 2015). A couple of reasons may explain this: firstly, the near-peer teachers are final-year medical students, who will sit for their final examinations very soon and are desperately finding ways to improve their learning. The process of teaching, which requires extensive preparation, a comprehensive understanding of the content, dynamic synthesis, and anticipation of the questions that may be asked of them, forms an efficient learning strategy. Secondly, the near-peer teachers, inspired by the excellent teaching they once had, wish to impart good teaching to the near-peer students going through the same journey.

Although many near-peer teachers are more motivated to be involved in more teaching and even take up teaching training courses in the future, the influence is not apparent in the long-term career direction. This could be explained well by the brief intervention of this NPT programme that lasted 3 months. However, the lucrative income opportunities in other medical specialities and the limited exposure to medical education pathways in undergraduate medical schools are some factors that may sway them away from considering medical education as their primary career pathway (Puri et al., 2021; Sarikhani et al., 2021). Therefore, more effort should be directed to increase teaching opportunities and to raise awareness of medical education career options in the undergraduate medical school programme. This includes developing Student Selected Components focusing on medical education and giving opportunities to shadow clinical teaching staff (Liew et al., 2015; Wilson et al., 2008).

After participating in this study, the near-peer students evaluated the near-peer teachers highly in all the domains. This provides a strong indication of the recognition and acceptance of the teaching skills of the near-peer teachers. In addition, the ability of the near-peer teachers to demonstrate responsibility and respect throughout this programme shows their preparedness to work under the GMC with desirable attitudes and professionalism. Previous studies have also shown that near-peer teachers gain more subjectively and objectively than students (Liew et al., 2015; Ten Cate & Durning, 2007). This can be related to the underpinning of the psychological and social theories behind the dynamics between near-peer teachers and near-peer students (Loda et al., 2019). The theoretical model of cognitive and social congruence explains the positive evaluation of the near-peer teachers (Loda et al., 2019; Rollmann et al., 2023). The proximity of age between the near-peer teachers and the near-peer students enables them to share similar knowledge frameworks, language and social roles.  Besides, near-peer teachers are perceived to be more approachable and understanding of the needs and struggles of the near-peer students. This may be because the near-peer teachers have had similar experiences themselves. Therefore, near-peer teachers are better able to process difficult concepts and frameworks, emphasize the key points, and communicate the information using familiar and non-confusing language to ensure that near-peer students comprehend better (Loda et al., 2019; Loda et al., 2020). The perceived barrier to providing feedback to near-peer teachers is also lower compared to faculty-led staff, as the age difference between them is much smaller. This might suggest why near-peer teachers feel less offended by criticism and are more likely to accept and respond to criticism gracefully (Loda et al., 2019). Nevertheless, the near-peer students thought that some near-peer teachers experienced some difficulties in giving relevant input and summary of the sessions, thus necessitating more structured pedagogical training for near-peer teachers in this aspect. Mastering these teaching skills would allow the near-peer students to appreciate better the big picture and key takeaway points of each lesson (Khaw & Raw, 2016).

One of the focuses of this study is to analyse the sex-specific difference of the perceived outcomes of the near-peer teachers. Although there is an appropriate twice female near-peer teachers who responded to this study compared to male near-peer teachers, the results shows that the sex-specific difference in the perceived improvement in motivation, skills and career direction is not significant. There is no sex-specific difference in term of enthusiasm and motivation to involve in near-peer teaching (Messerer et al., 2021).  Throughout the whole process, they receive similar gender-equitable support and guidance without any discrimination.

V. LIMITATIONS AND RECOMMENDATIONS

As few studies have reported outcomes for a purely student-led student-run NPT programme, this study offers valuable insights concerning the perceived benefits for near-peer teachers. However, this study has several limitations. Firstly, given the relatively small sample size of the near-peer teachers and the subjective nature of the self-reported questionnaire, the results may benefit from further objective testing, such as correlation with examination results. Secondly, this study is only carried out in a single medical school, with a slight variation in the implementation of the NPT compared to other institutions. Verification of these results across various medical schools would strengthen these findings. This study thus calls for more structured student-led peer teaching programmes to be implemented in more medical schools and to be assessed longitudinally to evaluate the association between the student-led peer teaching programme and the outcomes of participation of the near-peer teachers. It may also be worthwhile to investigate and assess the perspective of near-pear teachers who demonstrated interest in medical education, as well as to evaluate the long-term outcomes in career direction for medical graduates who once participated in near-peer teaching programmes.

VI. CONCLUSION

In conclusion, this purely student-led, student-run near-peer teaching programme likely improved the final-year medical students in fulfilling the “Doctors as Teachers” responsibilities outlined by the GMC. Besides, the near-peer teachers also reported having positive outcomes in their skills and career direction.  Likewise, from the perspective of the near-peer students, the near-peer teachers demonstrated outstanding skills and professionalism in all five domains: responsibility and respect, information processing, communication, critical analysis and self-awareness. Possessing the skills and professionalism fulfils the expectations of GMC for healthcare professionals to provide the right care at the right time with the right skills for the good of patients.

Notes on Contributors

Kevin Xuan Hong Tang was a final year medical student who conceptualised and designed the study, reviwed the literature, conducted the data collection and analysis, prepared the figures and wrote the manuscript.

Koon Kee Teo was a final year medical student who reviewed the literature, collected and analysed the data, prepared the figures and helped in writing the manuscript.

Kye Mon Min Swe is an Associate Professor in Education Research in NUMed. She participated in conceptualising the study, performed statistical analysis and drafted, reviewed and edited the manuscript. All the authors have read and approved the final manusript.

Ethical Approval

 This study titled “Embracing Doctors as Teachers: Evaluating the Student-led Near-Peer Teaching at Transnational Campus” was approved by the Research Management Committee and the Newcastle University Ethics Committee (Approval number 45070/2023).

Data Availability

The data that support the findings of this study are openly available in Figshare repository, as below,

https://doi.org/10.6084/m9.figshare.26886517.v1 (Tang et al., 2024a) and

https://doi.org/10.6084/m9.figshare.26886514.v1 (Tang et al., 2024b).

Acknowledgement

The authors would like to thank the NUMed Medical Education Society for carrying out this programme. The authors are grateful for Professor Vishna Devi Nadarajah for her support and feedback in this research.

Funding

The authors declare that there is no funding received in this study.

Declaration of Interest

The authors declare that there is no conflict of interest.

References

Botelho, M. G., Boubaker, B., & Wong, I. B. (2022). Near‐peer teaching for learning clinical photography skills: Perceptions of students. European Journal of Dental Education, 27(3), 477–488. https://doi.org/10.1111/eje.12831

Buckley, S., & Zamora, J. (2007). Effects of participation in a cross year peer tutoring programme in clinical examination skills on volunteer tutors’ skills and attitudes towards teachers and teaching. BMC Medical Education, 7(1), Article 20. https://doi.org/10.1186/1472-6920-7-20

Burgess, A., McGregor, D., & Mellis, C. (2014). Medical students as peer tutors: A systematic review. BMC Medical Education, 14(1), Article 115. https://doi.org/10.1186/1472-6920-14-115

Busari, J. O., Prince, K. J., Scherpbier, A. J., Van Der Vleuten, C. P., & Essed, G. G. (2002). How residents perceive their teaching role in the clinical setting: A qualitative study. Medical Teacher, 24(1), 57-61. https://doi.org/10.1080/00034980120103496

Cook, V., Fuller, J. H., & Evans, D. E. (2010). Helping students become the medical teachers of the future – The Doctors as Teachers and Educators (DATE) programme of Barts and the London School of Medicine and Dentistry, London. Education for Health, 23(2), 415. https://doi.org/10.4103/1357-6283.101487

De Menezes, S., & Premnath, D. (2016). Near-peer education: A novel teaching program. International Journal of Medical Education, 7, 160–167. https://doi.org/10.5116/ijme.5738.3c28

Frearson, S., & Gale, S. (2017). Educational opportunities on a ward round; utilising near-peer teaching. Future Healthcare Journal, 4(1), 19–22. https://doi.org/10.7861/futurehosp.4-1-19

Furmedge, D. S., Iwata, K., & Gill, D. (2014). Peer-assisted learning – Beyond teaching: How can medical students contribute to the undergraduate curriculum? Medical Teacher, 36(9), 812–817. https://doi.org/10.3109/0142159x.2014.917158

General Medical Council. (2011, February). Developing teachers and trainers in undergraduate medical education. Retrieved June 2, 2024, from https://www.gmc-uk.org/-/media/documents/developing-teachers-and-trainers-in-undergraduate-medical-education—guidance-0815_pdf-56440721.pdf

General Medical Council. (2015, July). Outcomes for graduate (Tomorrow’s doctors). Retrieved June 2, 2024, from  https://www.gmcuk.org//media/documents/Outcomes_for_graduates_ Jul_15_1216.pdf_61408029.pdf

General Medical Council. (2023, August 22). Good medical practice. Retrieved June 2, 2024, from https://www.gmc-uk.org/-/media/documents/good-medical-practice-2024—english-102607294.pdf

Hardie, P., O’Donovan, R., Jarvis, S., & Redmond, C. (2022). Key tips to providing a psychologically safe learning environment in the clinical setting. BMC Medical Education, 22(1), Article 816. https://doi.org/10.1186/s12909-022-03892-9

Hettle, D., & Morgan, J. (2019 December 2-3). “Teaching and Training for the future”: Final-year students’ perspectives on their role as near-peer teachers” [Conference presentation abstract]. Developing Excellence in Medical Education, Manchester. https://www.researchgate.net/publication/344439487_Teaching_and_Training_for_the_future_Finalyear_students’_perspectives_on_their_role_as_near-peer_teachers

Khaw, C., & Raw, L. (2016). The outcomes and acceptability of near-peer teaching among medical students in clinical skills. International Journal of Medical Education, 7, 189–195. https://doi.org/10.5116/ijme.5749.7b8b

Knobloch, A. C., Ledford, C. J., Wilkes, S., & Saperstein, A. K. (2018). The impact of near-peer teaching on medical students’ transition to clerkships. Family Medicine, 50(1), 58–62. https://doi.org/10.22454/fammed.2018.745428

Liew, S., Sow, C., Sidhu, J., & Nadarajah, V. D. (2015). The near-peer tutoring programme: embracing the ‘doctors-to-teach’ philosophy – A comparison of the effects of participation between the senior and junior near-peer tutors. Medical Education Online, 20(1), 27959. https://doi.org/10.3402/meo.v20.27959

Loda, T., Erschens, R., Loenneker, H., Keifenheim, K. E., Nikendei, C., Junne, F., Zipfel, S., & Herrmann-Werner, A. (2019). Cognitive and social congruence in peer-assisted learning – A scoping review. PloS One, 14(9), e0222224. https://doi.org/10.1371/journal.pone.0222224

Loda, T., Erschens, R., Nikendei, C., Giel, K., Junne, F., Zipfel, S., & Herrmann-Werner, A. (2020). A novel instrument of cognitive and social congruence within peer-assisted learning in medical training: construction of a questionnaire by factor analyses. BMC Medical Education, 20(1), Article 214. https://doi.org/10.1186/s12909-020-02129-x

Messerer, D. A. C., Kraft, S. F., Horneffer, A., Messerer, L. A. S., Böckers, T. M., & Böckers, A. (2021). What factors motivate male and female Generation Z students to become engaged as peer teachers? A mixed‐method study among medical and dental students in the gross anatomy course. Anatomical Sciences Education, 15(4), 650–662. https://doi.org/10.1002/ase.2114

Pierce, H., Kermack, A., & Foley, V. (2024). What training do foundation doctors feel they require to develop as clinical teachers? Journal of Education and Training Studies, 12(2), 106–113. https://doi.org/10.11114/jets.v12i2.6714

Puri, P., Landman, N., Smoldt, R. K., & Cortese, D. (2021). Quantifying the financial value of clinical specialty choice and its association with competitiveness of admissions. Curēus, 13(2), e13272. https://doi.org/10.7759/cureus.13272

Qureshi, Z., Gibson, K., Ross, M., & Maxwell, S. (2013). Perceived tutor benefits of teaching near peers: Insights from two near peer teaching programmes in South East Scotland. Scottish Medical Journal, 58(3), 188–192. https://doi.org/10.1177/0036933013496935

Rollmann, I., Lauter, J., Kuner, C., Herrmann-Werner, A., Bugaj, T. J., Friederich, H., & Nikendei, C. (2023). Tutors ́ and students’ agreement on social and cognitive congruence in a Sonography peer-assisted-learning scenario. Medical Science Educator, 33(4), 903–911. https://doi.org/10.1007/s40670-023-01814-y

Ross, M. T., & Cameron, H. S. (2007). Peer assisted learning: A planning and implementation framework: AMEE Guide no. 30. Medical Teacher, 29(6), 527–545. https://doi.org/10.1080/01421590701665886

Sarikhani, Y., Ghahramani, S., Bayati, M., Lotfi, F., & Bastani, P. (2021). A thematic network for factors affecting the choice of specialty education by medical students: A scoping study in low-and middle-income countries. BMC Medical Education, 21(1), Article 99. https://doi.org/10.1186/s12909-021-02539-5

Tang, K. X. H., Teo, K. K., & Swe, K. M. M. (2024a). The evaluation of near-peer teachers in five domains completed by near-peer students after joining peer teaching program (Version 1) [Data set].  Figshare. https://doi.org/10.6084/m9.figshare.26886517.v1

Tang, K. X. H., Teo, K. K., & Swe, K. M. M. (2024b). Near-peer teachers self-evaluation data after joining peer teaching program (Version 1) [Data set]. Figshare. https://doi.org/10.6084/m9.fig share.26886514.v1

Ten Cate, O., & Durning, S. (2007). Dimensions and psychology of peer teaching in medical education. Medical Teacher, 29(6), 546–552. https://doi.org/10.1080/01421590701583816

Wilson, S., Denison, A. R., & McKenzie, H. (2008). A survey of clinical teaching fellowships in UK medical schools. Medical Education, 42(2), 170–175. https://doi.org/10.1111/j.1365-2923.2007.02933.x

Yu, T. C., Wilson, N. C., Singh, P. P., Lemanu, D. P., Hawken, S. J., & Hill, A. G. (2011). Medical students-as-teachers: A systematic review of peer-assisted teaching during medical school. Advances in Medical Education and Practice, 2, 157–172. https://doi.org/10.2147/amep.s14383

Zheng, B., & Wang, Z. (2022). Near-peer teaching in problem-based learning: Perspectives from tutors and tutees. PloS One, 17(12), e0278256. https://doi.org/10.1371/journal.pone.0278256

*Kye Mon Min Swe
1, Jalan Sarjana 1, Educity,
79200 Iskandar Puteri, Johor, Malaysia
Email: kye-mon.min-swe@newcastle.edu.my

Submitted: 13 June 2024
Accepted: 4 February 2025
Published online: 1 July, TAPS 2025, 10(3), 26-36
https://doi.org/10.29060/TAPS.2025-10-3/OA3439

Humairah Zainal¹, Xin Xiaohui¹, Julian Thumboo^1,2,3 & Fong Kok Yong^2,3

¹Health Services Research Unit, Singapore General Hospital, Singapore; ²Department of Rheumatology and Immunology, Singapore General Hospital, Singapore, ³Duke-NUS Medical School, National University of Singapore, Singapore

Abstract

Introduction: As healthcare becomes increasingly digital, undergraduate medical students receive limited formal training in digital healthcare technologies (DHTs). Additionally, stakeholders’ perspectives on essential digital health competencies (DHCs) across generations remain underexplored. This comparative study aims to identify knowledge gaps, inform evolving expectations, and promote continuous learning by comparing medical students’ and senior clinicians’ views on essential DHCs in Singapore.

Methods: Individual semi-structured interviews were conducted with medical students, clinical educators, and clinical teachers. Clinical-year medical students from all three local medical schools were recruited using convenience and snowball sampling, while purposive sampling ensured diverse representation of clinicians across Singapore’s public healthcare organisations, focusing on those engaged in education-related activities. Data were collected from September 2020 to February 2023 until thematic saturation was achieved, and analysed using qualitative thematic analysis.

Results: Sixty-three participants took part, including 30 medical students, 12 clinical educators and 21 clinical teachers. All clinicians had more than 10 years of clinical experience and came from 24 different disciplines and 12 healthcare institutions. The findings showed unanimous support for integrating electronic health records in patient communication, and training in DHTs like telemedicine and point-of-care ultrasound. Discrepancies emerged; clinicians prioritised physical examination, while students viewed it as potentially replaceable by DHTs. Furthermore, while students valued healthcare-related smartphone applications in patient care, few clinicians shared this view, citing privacy and security concerns.

Conclusion: Evaluating stakeholders’ perspectives highlights evolving clinical skills and essential DHCs for medical students, potentially informing global DHC training programmes.

Keywords:           Technology, Medical Education, Curriculum, Clinical Competence, Digital Competence, Qualitative, Singapore

Practice Highlights

• An inter-generational perspective in medical education leverages the strengths of both medical students and senior clinicians, ensuring a balanced approach that keeps pace with technological advancements while maintaining the core principles of patient care.
• The digital health competencies deemed essential for future clinical practice are consistent across all stakeholders.
• The study highlights the emerging need for additional competencies among medical trainees so as to meet the evolving demands of healthcare delivery.
• Regularly comparing the perspectives of various stakeholders in the healthcare system enables educational institutions to adapt and refine their curricula continuously.

I. INTRODUCTION

Digital healthcare, which leverages advanced technologies to enhance, complement, or replace traditional healthcare services, is becoming increasingly common in clinical practice (Alkire (née Nasr) et al., 2020). Digital healthcare technologies (DHTs) such as electronic health records (EHR), telemedicine, and Artificial Intelligence (AI) have significantly improved data management, diagnostics, and patient care (Clay-Williams et al., 2023; Food and Drug Administration, 2019; Welcher et al., 2018;). However, many studies and reviews have highlighted a global lack of formal and systematic training in DHTs for undergraduate medical students (Aungst & Patel, 2020; Edirippulige et al., 2018; Tudor Car et al., 2021). While recommendations for integrating digital health education (DHE) exist, they often propose standalone courses focused on specific DHTs rather than embedding DHE into the core medical curriculum (Tudor Car et al., 2021; Zainal et al., 2023a; Zainal et al., 2023b).  Barriers to consistently incorporating DHE into medical curricula, as reported in countries such as Canada, Germany, the United Kingdom (U.K.), and the United States (U.S.) include limited faculty expertise, curriculum overload and fragmented implementation efforts (Gillissen et al., 2022; Hurley et al., 2011; Machleid et al., 2020; Sit el al., 2020; Sorg et al., 2022).

Furthermore, there is limited research exploring the perspectives of both medical students and senior clinicians on the relevant digital health competencies (DHCs) required for clinical care. DHCs encompass the knowledge, skills, and attitudes essential for assessing, diagnosing, and treating patients in the digital age. An inter-generational comparison of these perspectives is crucial for identifying gaps in the existing curricula and ensuring medical education aligns with the rapidly evolving healthcare landscape. Medical students can offer fresh insights into emerging tools and platforms while senior clinicians can contribute perspectives on clinical skills given their wealth of clinical experience.

Despite Singapore’s advanced digitalisation, challenges remain in standardising DHE across its three medical schools: Yong Loo Lin School of Medicine at National University of Singapore (NUS), Lee Kong Chian School of Medicine at Nanyang Technological University (NTU) and Duke-NUS Medical School (Duke-NUS) (Zainal et al., 2023a). Although these institutions boast state-of-the-art facilities, a disconnect persists between medical training and clinical practice due to various institutional and structural barriers (Zainal et al., 2023a). While efforts to integrate DHE, such as virtual reality courses, have been initiated, there are variations in content and duration across institutions, underscoring the lack of a standardised approach (Zainal et al., 2023a).

This study uses Singapore as a case study to explore the perceptions of medical students, clinical educators, and teachers regarding DHCs, with the aim of improving DHE integration into undergraduate medical curricula. Specifically, it compares the views of these stakeholders in identifying any digital competencies that may be relevant for clinical practice in an increasingly digital healthcare landscape. By focusing on Singapore, this study not only sheds light on gaps in the local curricula but also provides insights that can inform global efforts to strengthen DHE integration in medical education.

II. METHODS

A. Setting and Sample

A qualitative study was conducted using individual semi-structured interviews with medical students, clinical educators, and clinical teachers. Participants were identified by Principal Investigator (P.I.) FKY, based on their year of study and clinical roles respectively, ensuring relevant knowledge and experience.

The study employed a combination of convenience sampling and snowballing technique to recruit students from all three local medical schools. Clinical-year students (third to fifth years) were included due to their potential exposure to DHTs during clinical rotations, while pre-clinical students were excluded. Convenience sampling was initially used for ease of access, and snowballing technique was subsequently employed to expand participation and capture diverse perspectives.

To ensure diverse representation among clinicians, purposive sampling was applied across Singapore’s three public healthcare organisations. Clinical educators dedicating over 20 per cent of their time to education-related activities (e.g., teaching, curriculum planning, research, and administration) and clinical teachers contributing less than 20 per cent of their time to such roles were included. Clinicians not involved in educational roles were excluded from the study.

This study was classified as a quality improvement (QI) project on medical education curricula by the SingHealth Research Integrity, Compliance, and Ethics (RICE) committee. As such, it did not meet the criteria for human subject research and was granted an ethical waiver by SingHealth Institutional Review Board (Reference Number: 2020/2880). Nonetheless, this research adhered strictly to the ethical principles of the Declaration of Helsinki and institutional guidelines.

Data collection occurred from September 2020 to February 2023. Participants were invited by the P.I. via email detailing the study’s purpose, procedures, potential risks, and benefits, with consent obtained before the interviews. To address potential biases due to the P.I.’s professional relationships, interviews were conducted by a research fellow without prior connections to participants. Verbal consent was recorded at the start of each Zoom session, and participants were reminded of their right to withdraw, with pre-withdrawal data retained for comprehensive analysis.

To ensure anonymity, participants were assigned coded identifiers (‘MS’ for medical students, ‘CL’ for clinicians). Identifying information and audio recordings were securely stored and separated from the main dataset. Results were reported with care to avoid identifying individuals, and the data was exclusively used for curriculum development.

B. Data Collection

The interview guide followed Kallio et al.’s (2016) framework, including pre-requisites for semi-structured interviews, literature review, expert consultation, preliminary guide development, pilot testing with five participants, and finalisation. Questions addressed clinical competencies, their relevance to digital healthcare, and ways medical schools can better prepare students for digitalisation (Tables 1 and 2). Interviews included medical students and clinicians across specialties to ensure diverse perspectives.

The study involved 63 participants, with sample size guided by theoretical and practical considerations. Data collection continued until thematic saturation was reached, ensuring all relevant perspectives were captured. Practical constraints like time and resources influenced the participant number, but the focus remained on data richness and diversity for a comprehensive exploration of the research questions.

Sixty interviews were conducted via Zoom due to COVID-19 restrictions, with three in-person interviews held under local health guidelines. Each session lasted about 40 minutes and was audio-recorded. Transcriptions were generated using Otter AI and reviewed for accuracy by the P.I. and research fellow.

Table 2. Interview questions for clinical educators and teachers

C. Data Analysis

Thematic analysis followed Braun and Clarke’s six-step framework to explore curricular gaps (Braun & Clarke, 2006). Two researchers independently coded the data, resolving discrepancies through discussion to refine the codebook and ensure consistency. Credibility was enhanced by triangulating findings across participants from various medical schools and healthcare clusters, capturing diverse perspectives and minimising bias.

To contextualise the findings, comparisons were made with studies from other high-income countries undergoing similar healthcare digitalisation. Additionally, recent data from stakeholders, including medical school leaders, were analysed to understand the digital competencies needed for future clinical practice. Reporting adhered to the Standards for Reporting Qualitative Research (O’Brien et al. 2014).

III. RESULTS

Sixty-three participants took part in our study. Thirty of them were medical students who were undergoing clinical rotations in various disciplines, with 10 coming from each school. Twelve were clinical educators and 21 were clinical teachers. The clinicians were aged between 44 and 70 years. All had more than 10 years of clinical experience and came from 24 different disciplines and 12 healthcare institutions. A summary of their demographics is reported in Table 3 and illustrative excerpts are found in Table 4.

Table 3. Demographics of participants

Four major themes were derived from the data.

A. Limited and Inconsistent Exposure to DHTs

Students reported limited exposure to DHTs in core medical curricula, with training mainly through electives, select programmes, and student-driven initiatives. Exposure to health informatics and telemedicine was brief, with telemedicine training primarily prompted by the COVID-19 pandemic. Other DHTs like robotic surgery, virtual reality, and POCUS were inconsistently incorporated across medical schools. Some clinicians also reiterated the need for DHT exposure in public healthcare institutions, as doing so could inspire medical students to engage in innovation, research, and development in cutting-edge fields like robotics.

B. Unified Views among Clinicians and Students on Essential Competencies

Clinicians and students shared unified perspectives on incorporating key DHCs like EHR, telemedicine, and POCUS into medical curricula. These competencies are viewed as crucial for preparing students for contemporary clinical practice.

Participants emphasised the need to balance EHR use with effective patient interaction. M25, a third-year student, highlighted the importance of maintaining empathy and humanistic values, while CL27, a rheumatologist, stressed integrating EHR data into patient discussions to improve engagement and understanding of health conditions.

All groups also agreed on the importance of telemedicine training. Students expressed interest in learning how to identify conditions suitable for virtual consultations, build patient rapport, and conduct comprehensive assessments via tele-platforms. MS5 noted the challenges of teleconsultation, such as interpreting body language and addressing legal responsibilities. Clinicians like CL14 underscored the need to teach students how to select patients for teleconsultation and identify red flags effectively.

Additionally, both clinicians and students strongly supported incorporating POCUS into the curriculum. Students acknowledged its utility in bedside diagnostics, while clinicians like CL9 and CL18 highlighted its value in resource-limited settings and its role in enhancing diagnostic accuracy. They advocated for routine training in ultrasound, considering its increasing use in general practice and as a critical tool in diverse healthcare settings. These unified views suggest the need to prioritise these DHCs in medical education to meet the evolving demands of healthcare delivery.

C. Divergent Opinions on Traditional Clinical Skills

Discrepancies in opinions emerged regarding traditional clinical skills, particularly physical examination. Students often viewed diagnostic imaging and tests as superior to these skills, perceiving physical exams as less critical. For instance, M11, a Year 4 student, argued that ultrasonography provides more precise diagnoses, making physical examination less relevant.

Clinicians, however, emphasised preserving traditional skills like history-taking and physical examination, especially for resource-limited settings. CL3, an ophthalmologist, expressed concern that reliance on diagnostic tools has led to the erosion of these essential skills.

D. Emerging Technologies and Concerns

The perceived competencies for medical education varied between students and senior clinicians. Students valued wearables and healthcare apps for patient empowerment, personalised care, and strengthening doctor-patient trust. For instance, MS30, a fourth-year student, suggested that wearable data could help create personalised care plans and enhance clinical practice.

Clinicians, however, expressed concerns about the privacy and security of patient data in these apps. They emphasised the responsibility to protect patient information and were cautious about using apps that did not meet security standards. These differing opinions highlight the need for a DHE curriculum that addresses both student interests and clinician concerns.

In our sample selection, we included both clinical educators and clinical teachers to capture a range of perspectives on the integration of DHCs into medical education. However, our analysis did not reveal significant differences in the views expressed by these two groups. Both shared similar opinions on the importance of teaching DHCs such as EHR, telemedicine and POCUS. This alignment may be attributed to their shared roles in preparing medical students for clinical practice, regardless of their specific titles or responsibilities. As such, the findings reflect a unified perspective among clinicians involved in medical education, despite the initial distinction made in the sample selection.

Table 4. Illustrative quotes from interviews with participants

IV. DISCUSSION

This study contributes to the literature by comparing the views of medical students, clinical educators, and clinical teachers on essential competencies for medical school curricula. It highlights a consensus on the need for training in EHR, telemedicine, and POCUS, while also identifying emerging needs like interpreting data from healthcare devices and engaging patients with this data. Unlike previous studies that focused on either clinicians or students (Blacketer et al., 2021; Foadi et al., 2021; Gillissen et al., 2022; Hersh et al., 2017; Liu et al., 2022; Machleid et al., 2020; Pontefract & Wilson, 2019), this research explores inter-generational differences, revealing that younger trainees prioritise patient-centred care and digital health literacy. This emphasis on inclusivity and data literacy is crucial given the global challenge of inadequate health literacy, particularly among older populations (Sørensen et al., 2021; Wittink & Oosterhaven, 2018). Integrating these competencies into curricula can foster ongoing improvement in medical education.

In comparing the views of medical students and senior clinicians regarding DHCs, it is important to acknowledge that the latter group represents a diverse range of roles within the clinical environment. While this study focused specifically on clinicians involved in medical education, senior clinicians may also take on roles as clinical researchers, administrators, or leaders in healthcare innovation. These differing responsibilities could shape their perspectives on the DHCs that should be emphasised in medical curricula. The results of this study should be interpreted within the context of the clinical educators’ primary role in teaching medical students, which may prioritise practical and immediate clinical skills over broader administrative or research-based digital competencies.

The integration of EHR systems into medical education has been recognised as essential. Successful models in the U.S. and U.K. have embedded EHR systems into training, such as allowing students to write notes at primary clinical sites or use de-identified patient data for practice (Pereira et al., 2018; Pontefract & Wilson, 2019; Welcher et al., 2018). These approaches ensure hands-on learning and prepare students for modern healthcare environments. For broader applicability, partnerships with local healthcare institutions and technology providers can enable the development of localised EHR modules tailored to different healthcare contexts. However, over-reliance on EHR systems may reduce critical thinking or direct patient interaction. Mitigation strategies include embedding reflective practices into training and balancing EHR use with patient engagement activities. When successfully implemented, EHR training enhances data management skills and promotes patient interaction.

Telemedicine training varies globally, often focusing on rural healthcare placements, as seen in Australia and the U.K. (Rienits et al., 2016; Wootton, 1999). In Singapore, telemedicine exposure arose mainly during the COVID-19 pandemic’s social distancing requirements, highlighting its potential in urban healthcare. To ensure comprehensive telemedicine training, curricula should include simulated consultations, role-playing exercises, and coverage of diverse scenarios, such as breaking bad news over digital platforms. To address the challenge of interpreting body language, medical schools should incorporate modules that focus on interpreting non-verbal cues, such as facial expressions and gestures, which are essential for building rapport and understanding patients’ concerns. Courses that address the legal responsibilities in telemedicine, including patient confidentiality, informed consent, and documentation requirements, should also be integrated to ensure students are aware of the regulatory frameworks guiding virtual care. Case studies and real-life examples that discuss the legal implications of telemedicine can also help students navigate potential legal challenges. With regard to patient selection, it is crucial for professional bodies to develop frameworks that guide clinicians in determining when in-person care is more appropriate, especially for complex or high-risk cases. Additionally, the curricula should provide guidance on how to recognise red flags such as signs of mental health distress, urgent medical conditions, or patient non-compliance with digital tools, and escalate them appropriately. Providing continuing education opportunities for practising clinicians is also important to keep them updated on telemedicine advancements. The healthcare system can then pair students with experienced telemedicine practitioners who can provide the necessary mentorship and guidance.

POCUS has become an essential component of undergraduate medical education (Glass et al., 2021; Sheppard et al., 2023). However, its application is often limited to specific specialties, such as emergency medicine, obstetrics and gynaecology, and radiology (Glass et al., 2021). In contrast, our study advocates for a longitudinal POCUS curriculum spanning all years and specialties. Integrating POCUS training into medical curricula improves diagnostic accuracy, enhances bedside decision-making, and builds clinical confidence. A structured approach, such as Kern’s six-step model, provides a robust framework for designing and implementing a standardised curriculum with consistent competency assessments (Kern, 1998). Innovative teaching methods, including handheld devices and near-peer instruction, have been shown to improve ultrasound proficiency and student performance, as evidenced by successful implementations at the University of Connecticut School of Medicine (Boivin et al., 2022). While improper use of POCUS can result in diagnostic errors, these risks can be mitigated through routine competency evaluations and guided practice, ensuring students develop the necessary skills safely and effectively.

Addressing differing opinions regarding DHC priorities in schools requires collaboration among medical educators, technology developers, and healthcare providers. Medical curricula should include training on the responsible use of DHTs, and evaluating the reliability of apps. Developers should design apps that address clinicians’ data security concerns. Moreover, healthcare organisations should establish guidelines and protocols for healthcare-related app use to ensure patient safety and privacy. Integrating these competencies into medical curricula will better prepare future healthcare professionals for evolving demands of healthcare delivery.

Implementing DHC also requires consideration of diverse healthcare systems and resource availability. Local adaptations, coupled with ongoing evaluation and feedback, ensure curricula remain relevant and scalable. Training programmes must balance global standards with local needs, enabling students to navigate evolving healthcare landscapes effectively. By addressing the implementation strategies, risks, and benefits of DHTs, this study provides actionable insights for advancing DHE.

A. Strengths and Limitations

This qualitative study highlights the DHCs needed in the core medical curricula from the perspectives of students, clinical educators and teachers. The diverse sample of institutions and specialties enriched the data. Student feedback provided up-to-date information on curricula gaps, while senior clinicians’ experience enhanced credibility. Interviewing participants from different generations ensured medical education remains relevant to current and future healthcare needs.

A perceived limitation of this study is its small sample size, which may affect its generalisability, validity and reliability. With a small sample, the findings might not represent the broader population and could miss the population’s diversity (Sandelowski, 1995). Additionally, small sample sizes may not adequately capture the diversity within the population of interest (Sandelowski, 1995). Moreover, the unique socio-political, cultural and economic context of Singapore may limit the generalisability of our findings to other settings. Singapore’s relatively small population and centralised governance create conditions that may differ from other countries. Consequently, while the insights from our study provide valuable lessons, they should be interpreted with caution when applying them to contexts with different governance structures or cultural dynamics.

Despite the limitations, this qualitative study provides a contextualised understanding of participants’ expectations and concerns through in-depth narrative analysis. While not entirely generalisable, the findings have significant implications for medical curricula in other developed countries. They highlight valued competencies, informing curriculum design, training programmes and continuing education initiatives.

Furthermore, this study focused on clinicians from public healthcare institutions, which may influence their perspectives on DHCs. Public sector clinicians often work within more structured systems that prioritise certain competencies differently than in the private sector. While this focus provides insights specific to public healthcare education, it is important to note that the views of private-sector clinicians, who may face different digital health challenges, were not captured in this study. Future work comparing clinicians across both sectors could provide a more comprehensive view. The perspectives of clinicians from the private healthcare sector may also differ due to the distinct operational and financial frameworks in which they practise. 

Moreover, future research could explore the perspectives of clinicians in various roles beyond medical education, such as clinical researchers, healthcare administrators, or those involved in clinical governance. By comparing these viewpoints with those of clinical educators, a more comprehensive understanding of the DHCs required across different professional pathways in healthcare can be developed. Such studies could further inform curriculum development to ensure that medical students are adequately prepared not only for clinical practice but also for the evolving demands of digital health in research, administration, and leadership.

V. CONCLUSION

In conclusion, an inter-generational perspective in medical education leverages the strengths of both medical students and senior clinicians, ensuring a balanced approach that keeps pace with technological advancements while maintaining the core principles of patient care. This collaborative model prepares future healthcare professionals to navigate the evolving landscape of healthcare technology effectively. As the findings have shown, despite their varying levels of experience and diverse medical backgrounds, the DHCs deemed essential for future clinical practice are consistent across all stakeholders. While all stakeholder groups considered most basic clinical skills important, medical students viewed physical examination as replaceable by DHTs. In contrast, the clinical educators and teachers thought otherwise. To address the needs and concerns of all groups, DHCs should complement instead of replace traditional clinical skills.

We acknowledge that the DHTs deemed important in other countries may vary from those reported in this study. However, the type of DHTs to be introduced should not distract us from the main goal of ensuring that the curricula of medical schools around the world remain updated and relevant to current and future healthcare needs. Regularly comparing the perspectives of various stakeholders in the healthcare system is crucial, as it enables educational institutions to adapt and refine their curricula continuously. This iterative process ensures that the curricula remain responsive to the dynamic nature of clinical work.

Notes on Contributors

Humairah Zainal collected and analysed the data, wrote the initial drafts, revised the manuscript for publication and approved the submitted copy of the manuscript.

Xin Xiaohui reviewed, edited, read and approved the submitted copy of the manuscript.

Julian Thumboo reviewed, edited, read and approved the submitted copy of the manuscript.

Fong Kok Yong was involved in the conception and design of the study, reviewed, edited, read and approved the submitted copy of the manuscript.

Ethical Approval

Waiver for ethical approval was granted by SingHealth Institutional Review Board (Reference Number: 2020/2880). 

Data Availability

The data that support the findings of this study are available from the first author upon reasonable request. The data cannot be uploaded to the repository as it contains some confidential views of participants.

Acknowledgements

The authors extend their gratitude to the reviewers for their insightful feedback on the manuscript drafts. Special thanks go to Anisa Muhammad for transcribing the interviews, as well as to Dr. Warren Fong Weng Seng and Dr. Ng Qin Xiang for their assistance in recruiting some of the medical students. The authors also deeply appreciate the invaluable contributions of the study participants.

Funding Statement

This study was supported by SingHealth Duke-NUS Medicine Academic Clinical Programme under Seah Cheng Siang Distinguished Professorship in Medicine.

Declaration of Interest

There are no relevant financial or non-financial competing interests to report.

References

Alkire (née Nasr), L., O’Connor, G. E., Myrden, S., & Köcher, S. (2020). Patient experience in the digital age: An investigation into the effect of generational cohorts. Journal of Retailing and Consumer Services, 57, Article 102221. https://doi.org/10.1016/j.jretconser.2020.102221

Aungst, T. D., & Patel, R. (2020). Integrating digital health into the curriculum – Considerations on the current landscape and future developments. Journal of Medical Education and Curriculum Development, 7, Article 2382120519901275. https://doi.org/10.1177/2382120519901275

Blacketer, C., Parnis, R., B. Franke, K., Wagner, M., Wang, D., Tan, Y., Oakden-Rayner, L., Gallagher, S., Perry, S. W., Licinio, J., Symonds, I., Thomas, J., Duggan, P., & Bacchi, S. (2021). Medical student knowledge and critical appraisal of machine learning: A multicentre international cross-sectional study. Internal Medicine Journal, 51(9), 1539-1542. https://doi.org/10.1111/imj.15479

Boivin, Z., Carpenter, S., Lee, G., Chimileski, B., Harrison, J., Choudhary, D., & Herbst, M. (2022). Evaluation of a required vertical point-of-care ultrasound curriculum for undergraduate medical students. Cureus, 14(10), e30002. https://doi.org/10.7759/cureus.30002

Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa

Clay-Williams, R., Hibbert, P., Carrigan, A., Roberts, N., Austin, E., Fajardo Pulido, D., Meulenbroeks, I., Nguyen, H. M., Sarkies, M., Hatem, S., Maka, K., Loy, G., & Braithwaite, J. (2023). The diversity of providers’ and consumers’ views of virtual versus inpatient care provision: A qualitative study. BMC Health Services Research, 23(1), 724. https://doi.org/10.1186/s12913-023-09715-x

Edirippulige, S., Brooks, P., Carati, C., Wade, V. A., Smith, A.C., Wickramasinghe, S., & Armfield, N. (2018). It’s important, but not important enough: eHealth as a curriculum priority in medical education in Australia. Journal of Telemedicine and Telecare, 24(10), 697-702. https://doi.org/10.1177/1357633×18793282

Food and Drug Administration. (2019). Digital health center of excellence. https://www.fda.gov/medical-devices/digital-health-center-excellence

Foadi, N., Koop, C., Mikuteit, M., Paulmann, V., Steffens, S., & Behrends, M. (2021). Defining learning outcomes as a prerequisite of implementing a longitudinal and transdisciplinary curriculum with regard to digital competencies at Hannover Medical School. Journal of Medical Education and Curricular Development, 8, 23821205211028347. https://doi.org/10.1177/23821205211028347

Gillissen, A., Kochanek, T., Zupanic, M., & Ehlers, J. (2022). Medical students’ perceptions towards digitization and Artificial Intelligence: A mixed-methods study, Healthcare (Basel), 10(4), 723-737. https://doi.org/10.3390/healthcare10040723

Glass, C., Sarwal, A., Zavitz, J., Nitsche, J., Joyner, J., Johnson, L. L., Garcia-Vargas, J., & O’Brien, M. C. (2021). Scoping review of implementing a longitudinal curriculum in undergraduate medical education: The wake forest experience. The Ultrasound Journal, 13(1), 23. https://doi.org/10.1186/s13089-021-00206-w

Hersh, W., Biagioli, F., Scholl, G., Gold, J., Mohan, V., Kassakian, S., Kerns, S., & Gorman, P. (2017). From competencies to competence: Model, approach, and lessons learned from implementing a clinical informatics curriculum for medical students. In Shachak, A., Borycki, E.M., & Reis, S.P. (Eds.),  Health professionals’ education in the age of clinical information systems, mobile computing and social networks (pp. 269-287). Academic Press. https://doi.org/10.1016/B978-0-12-805362-1.00013-9    

Hurley, K. F., Taylor, B., Postuma, P., & Paterson, G. (2011). What are Canadian medical students learning about health informatics. Electronic Journal of Health Informatics, 6(4), np.

Kallio, H., Pietilä, A.-M., Johnson, M., & Kangasniemi, M. (2016). Systematic methodological review: Developing a framework for a qualitative semi-structured interview guide. Journal of Advanced Nursing., 72(12), 2954–2965. https://doi.org/10.1111/jan.13031

Kern, D. E. (1998). Overview: A six-step approach to curriculum development. In P.A., Thomas, D.E., Kern, M.T., Hughes & B.Y. Chen (Eds.), Curriculum Development for Medical Education: A Six-Step Approach (pp. 5-10). Johns Hopkins University Press.

Liu, D. S., Sawyer, J., Luna, A., Aoun, J., Wang, J., Boachie, L., Halabi S., & Joe B. (2022). Perceptions of US medical students on Artificial Intelligence in medicine: Mixed methods survey study. JMIR Medical Education, 8(4), e38325. https://doi.org/10.2196/38325

Machleid, F., Kaczmarczyk, R., Johann, D., Balčiūnas, J., Atienza-Carbonell, B., von Maltzahn, F., & Mosch, L. (2020). Perceptions of digital health education among european medical students: Mixed methods survey. Journal of Medical Internet Research, 22(8), e19827. https://doi.org/10.2196/19827

O’Brien, B. C., Harris, I. B., Beckman, T. J., Reed, D. A., & Cook, D. A. (2014). Standards for reporting qualitative research: A synthesis of recommendations. Academic Medicine, 89(9), 1245–1251. https://doi.org/10.1097/ACM.0000000000000388

Pereira, A. G., Kim, M., Seywerd, M., Nesbitt, B., & Pitt, M. B. (2018). Collaborating for competency – A model for single electronic health record onboarding for medical students rotating among separate health systems. Applied Clinical Informatics, 9(1), 199-204. https://doi.org/10.1055/s-0038-1635096

Pontefract, S. K., & Wilson, K. (2019). Using electronic patient records: Defining learning outcomes for undergraduate education. BMC Medical Education, 19(1), 30. https://doi.org/10.1186/s12909-019-1466-5

Rienits, H., Teuss, G., & Bonney, A. (2016). Teaching telehealth consultation skills. The Clinical Teacher, 13(2), 119–123. https://doi.org/10.1111/tct.12378

Sandelowski, M. (1995). Sample size in qualitative research. Research in Nursing & Health, 18(2), 179–183. https://doi.org/10.1002/nur.4770180211

Sheppard, G., Williams, K.-L., Metcalfe, B., Clark, M., Bromley, M., Pageau, P., Woo, M., Yi, Y., Devasahayam, A. J., & Dubrowski, A. (2023). Using Kane’s framework to build an assessment tool for undergraduate medical student’s clinical competency with point of care ultrasound. BMC Medical Education, 23(1), 43. https://doi.org/10.1186/s12909-023-04030-9

Sørensen, K., Levin-Zamir, D., Duong, T. V., Okan, O., Brasil, V. V., & Nutbeam, D. (2021). Building health literacy system capacity: A framework for health literate systems. Health Promotion International, 36(Supp 1), i13–i23. https://doi.org/10.1093/heapro/daab153

Tudor Car, L., Kyaw, B. M., Panday, R. S. N., van der Kleij, R., Chavannes, N., Majeed, A., & Car, J. (2021). Digital health training programs for medical students: Scoping review. JMIR Medical Education, 7(3), e28275. https://doi.org/10.2196/28275

Welcher, C. M., Hersh, W., Takesue, B., Stagg Elliott, V., & Hawkins, R. E. (2018). Barriers to medical students’ electronic health record access can impede their preparedness for practice. Academic Medicine, 93(1), 48–53. https://doi.org/10.1097/ACM.0000000000001829

Wittink, H., & Oosterhaven, J. (2018). Patient education and health literacy. Musculoskeletal Science and Practice, 38, 120–127. https://doi.org/10.1016/j.msksp.2018.06.004

Wootton, R. (1999). Telemedicine and isolated communities: A UK perspective. Journal of Telemedicine and Telecare, 5(2), 27–34. https://doi.org/10.1258/1357633991933495

Zainal, H., Xiaohui, X., Thumboo, J., & Kok Yong, F. (2023a). Digital competencies for Singapore’s national medical school curriculum: A qualitative study. Medical Education Online, 28(1), 2211820. https://doi.org/10.1080/10872981.2023.2211820

Zainal, H., Xiaohui, X., Thumboo, J., & Kok Yong, F. (2023b). Exploring the views of Singapore junior doctors on medical curricula for the digital age: A case study. PLOS ONE, 18(3), e0281108. https://doi.org/10.1371/journal.pone.0281108

*Fong Kok Yong
10 Hospital Boulevard
Singapore 168582
+65 6908 8949
Email: fong.kok.yong@singhealth.com.sg

Submitted: 29 February 2024
Accepted: 5 February 2025
Published online: 1 July, TAPS 2025, 10(3), 15-25
https://doi.org/10.29060/TAPS.2025-10-3/OA3259

Zaw Phyo¹, Titi Savitri Prihatiningsih², Ye Phyo Aung¹ & Tun Tun Naing¹

¹Department of Medical Education, Defence Services Medical Academy, Yangon, Myanmar; ²Public Health and Nursing Universitas Gadjah Mada, Yogyakarta, Indonesia

Abstract

Introduction: The effective implementation of changes in medical school curricula requires modifications to assessments alongside teaching strategies. The World Federation for Medical Education emphasises the need for assessment policies to align with educational outcomes. In Myanmar, the Defence Services Medical Academy (DSMA) has adopted an outcome-based curriculum since 2017, but the standard setting in assessments remains pending. This study explores concerns and challenges for enhancing assessment practices.

Methods: An explanatory sequential design was used. For a quantitative approach, 36 assessment committee members were surveyed using questionnaires, and pre-and post-test analyses were conducted using SPSS Statistics Version 25. Qualitatively, eight committee members were selected through purposive sampling for in-depth interviews using interpretative methodology with thematic analysis by MAXQDA Software 2022.

Results: It shows that most committee members acknowledge the advantages of standard settings, such as improved teaching and learning defect identification, staff knowledge application, enhanced educational programs, and personal contributions to education quality. Half of the committee members expressed concerns about the increased workload and administrative burden, while another half-supported standard-setting implementation. They all appreciate the integration of standard setting in the outcome-based curriculum, recognising its positive impact on student quality and accreditation standards. Challenges include limited human resources, resistance to change, time-consuming, and an increased workload.

Conclusion: Committee members suggested that targeted interventions are needed to improve awareness, collaboration, and successful implementation. These should consider perspectives, enhance understanding, define roles, and address resources and resistance. This will optimise standard setting, ensure educational quality, and meet accreditation standards.

Keywords:           Assessment, Standard Setting, Integrated Curriculum, Awareness, Perception, Attitude, Lifelong Learning

Practice Highlights

• Members of the committee think that having standards in DSMA’s curriculum is essential for raising the quality of students and the school’s standards.
• Most members know how important it is to set standards, especially for pass/fail exams, and stress how important it is to consider culture and institutional factors.
• Setting standards is thought to help with aligning the curriculum, giving students relevant tests, and meeting accreditation requirements.
• For implementation to go well, all stakeholders must be involved, the process must be in line with real evaluation principles, and teachers must continue to learn and grow.
• Not enough resources, resistance to change, limited time, and more work to do are problems that need solutions based on educational ideas.

I. INTRODUCTION

In the last 30 years, there have been many suggestions for changing the curriculum, but only a few medical schools have made significant changes (Supe & Burdick, 2006). Without altering assessments, modifying curricula or teaching strategies will be ineffective. In the global standards for quality improvement area of the World Federation for Medical Education (WFME), there are clear rules about primary medical education in chapter 3.1 of Assessment Policy and System:

“An assessment policy with a centralised system that guides and supports its implementation will entail using multiple summative and formative methods to acquire the knowledge, clinical skills, and behaviors needed to be a doctor. The policy and the system should be responsive to the school’s mission, specified educational outcomes, available resources, and the context” (World Federation for Medical Education, 2020).

In current Medical Education, with competency-based education becoming more popular, healthcare educational institutions and assessment bodies are being looked at more closely than ever before to make sure they are fair, transparent, and defensible when setting the expected level of performance and rating students’ performance to make decisions about their next steps (Lane et al., 2020). With competency-based assessment, the way the test is done must align with what the student has learned (Crespo et al., 2010).

Regularly setting, keeping, and evaluating assessment standards is essential in medical education. It is not an exact science to set standards. Most educational test validity arguments focus on how well the test matches the curriculum and how valuable the test’s content is (Barman, 2008).

Standards are an expression of values, so the ways to set them are systematic ways to gather people’s opinions on values, make agreements, and show that agreement as a single score on a test (Norcini, 2003). “Standard Setting” is a process used to make precise lines between things like “pass” and “fail” or “honors,” “proficient,” and “needs improvement.” Setting standards is “central to giving meaning to test results and, as such, is at the heart of validity arguments” (Wiliam, 1996).

There are two primary methods in which standards are set: (1) criterion-referenced or absolute methods, in which standards setting is not based on test results, and (2) norm-referenced or relative methods, in which setting standards is based on test results. No golden rule exists (Cohen-Schotanus & Van Der Vleuten, 2010).  In health professions education, it can be challenging to develop credible, defendable, and acceptable passing or cut-off scores for exams (Norcini & Guille, 2002).

In Myanmar, there are currently no private medical schools, although there are five public medical schools and one military medical school (Defence Services Medical Academy). Myanmar medical schools have used traditional lecture-based teaching methods, and many eminent physicians have emerged from teacher-centered educational programs (Myint, 2020).

Defence Services Medical Academy (DSMA), listed in the World Directory of Medical Schools, was founded in Mingalardon, Yangon, on November 19, 1992 (Naing et al., 2022). Since 2017, DSMA has been the first medical school in Myanmar to implement an outcome-based, integrated curriculum. According to the Dundee three circle models, the graduates of DSMA are supposed to be scholars and scientists, as well as practitioners and practitioners professionals (Harden et al., 1999).  The new DSMA curriculum is meant to encourage students to keep learning and give them more responsibility for their learning. It is presently focusing on meeting the criteria of WFME accreditation.

In DSMA, the assessment methods (written, practical, oral, and structured and unstructured observations) include accurate and standardised patients, objective structured clinical examination, patient management problems or simulations, etc. The assessments include continuous assessment, formative assessment, and summative assessment. However, there is no Standard Setting in DSMA, and it will be implemented soon (after training standard setting experts) to set the required basic minimum standards.

Even though accrediting organisations advise medical schools in general what the curriculum should cover and how it should be tested, most medical schools decide how and what to test (Epstein, 2007). There are five ways to figure out how valuable a method of assessment is: reliability (how accurate and repeatable the measurement is), validity (whether the assessment measures what it says it measures), impact on future learning and practice, acceptability to students and teachers, and costs (Van Der Vleuten, 1996). Therefore, the Assessment Committee’s DSMA meetings establish the process and procedures for conducting assessments throughout the undergraduate medical program. The Assessment committee must collect and evaluate feedback regularly, and faculty feedback revealed issues with the assessment process in DSMA.

 For a successful, acceptable, and feasible assessment to implement an outcome-based, integrated curriculum, it is essential to explore how students, faculty, and assessment committee members perceive the implication of the standard setting for the assessment. Therefore, this study investigated what the Assessment Committee members think about setting standards in medical education and the most significant problems.

II. METHODS

A. Research Design

Figure 1 shows explanatory sequential mixed methods design (also called a two-phase model (Creswell & Clark, 2017)): Quantitative followed by qualitative was used to explore the views, opinions, beliefs, and feelings of the assessment committee members (Ivankova & Stick, 2007). For the quantitative, all assessment committee members were asked using questionnaires (Likert scale), and qualitatively, Individual In-depth Interviews (IDI) were asked of the selected assessment committee members.

Figure 1. Explanatory sequential mixed methods design

The main benefit of mixed methods research is that they make a potent mix when quantitative and qualitative data are combined (Miles & Huberman, 1994). In addition, a “complex” image of a social phenomenon might be created, for instance, by evaluating both the quantitative and qualitative aspects of a study’s results (Greene, 2006).

B. Data Collection

For Phase 1 (Quantitative): The survey consists of four parts: (1) Collecting demographic information; (2) Investigating people’s perception of standard-setting procedures; (3) Examining attitudes towards these procedures; and (4) Evaluating these procedures using five-point Likert scale items ranging from 5 (strongly agree) to 1 (strongly disagree).

For Phase 2 (Qualitative): An in-depth interview (IDI) with semi-structured questions was used to collect the data. The researcher made an interview guide, which helped shape the interview and get detailed comments from the assessment committee members. The IDIs lasted 30-45 minutes, and the researcher informed participants that their names and information would be kept private.

The researcher performed all the interviews in the participants’ local language (Myanmar). The IDI explored the themes of the conceptual framework. A video recording was made for continuous recording in a quiet, private section of the medical education department office. The researcher conducted open discussions with the assessment committee members and kept them confidential, especially the students. Each interviewer uses a different set of questions (10–12), and the sessions last between 45 and 60 minutes. The researcher kept doing interviews until it became clear that the stories, themes, and problems had been covered to saturation point. The researchers utilised constructivist reflexivity throughout the entire study procedure, as advised by Alvesson and Sköldberg (2017).

C. Sampling Procedure

In DSMA, there are 38 members of the assessment committee. The assessment committee comprises three groups: Discussant members, voting members, and secretariat.

For Phase 1 (Quantitative): All the assessment committee members of DSMA were recruited. Among the 38 eligible persons who met eligibility, approximately 95% of the population (36 participants) responded to the evaluation survey.

For Phase 2 (Qualitative): By using the purposes sampling methods, individual in-depth interviews (IDIs) with semi-structured questions were given to the selected eight members of the assessment committee who are information-rich persons (two discussant Members, five voting members, and one secretariat) to find out what they thought, what they had done, and how they did it.

D. Data Analysis

The survey data was stored in an Excel file and processed using SPSS software (version 25). Descriptive analysis explored assessment committee members’ perceptions, attitudes, and practices.

The MAXQDA 2022 program extracted themes and sub-themes (Côté & Turgeon, 2005). The recorded interviews were transcribed to do content analysis. First, the data were summarised, proofread, and translated into English. Second, for the data analysis, themes, sub-themes, and clusters were identified (Cresswell, 2012). Finally, conceptualisation and explanation throughout the later stages led to the development of concrete abstracts. Analytic coding was done while reading the transcripts, and emerging issues were thought to be included in the thematic statements.

E. Ethical Considerations

The study was done according to the guidelines published by the Ethical and Research Committee of DSMA. Participation in this study is entirely voluntary.

F. Informed Consent

Before conducting the study, informed verbal consent was obtained for respondents’ autonomy to participate in this study.

III. RESULTS

Quantitatively, thirty-seven questions were asked about each participant’s background, knowledge, perception, attitude, and practices regarding the standard-setting. The survey received responses from 36 participants. The data supporting this study’s findings are openly available in the Figshare repository, https://doi.org/10.6084/m9.figshare.25657875.v1 (Phyo et al., 2024).

A. Demographic Characteristics

According to an analysis of the committee’s demography (see Table 1), two-thirds of the members were older than 25, and males dominated. Faculty members (64%) were primarily doctoral holders on the evaluation committee. More than ten years of teaching experience can be found among half of the assessment committee members.

Table 1. Personal background information of the assessment committee members

B. Awareness of Standard Setting

In Table 2, most assessment committee members believe that DSMA has been implementing a standard setting in the assessment process, but it is still in the process of being fully implemented. However, 75% know the purpose behind establishing standard settings in DSMA. Furthermore, faculty members are debating whether DSMA is a precise standard-setting approach for standardisation and how DSMA intends to implement whatever assessment model.

No	Question	Answers	Total (n=36)
			Frequency	Percentage
1	Does DSMA have a standard Setting in the MBBS Programme?	Yes	23	63.9
		No	11	30.6
		Uncertain	2	5.6
2	Who Is the most Responsible person in the Implementation standard setting within DSMA?	Rector	9	25.0
		Assessment Committee	6	16.7
		All Staff	20	55.6
		Uncertain	1	2.8
3	Why did DSMA introduce a Standard setting?	Rector’s foresight and planning	8	22.2
		Requirement of MMCAC guideline	27	75.0
		Uncertain	1	2.8
4	How many types of standard settings?	One	6	16.7
		Two	22	61.1
		Three	8	22.2
5	The assessment system of the DSMA is designed by applying a ready-made model like programmatic assessment.	Yes	12	33.3
		No	12	33.3
		Uncertain	12	33.3
6	Does DSMA have an exact standard-setting method to standardise the assessment system?	Yes	17	47.2
		No	10	27.8
		Uncertain	9	25.0

Table 2. Awareness of standard setting

Table 3. Perception, attitude, and practice towards standard setting

The attitudes section indicates lower ratings (2.0–3.0) with worries about increasing workload (59% agreement) and administrative burden (56% agreement), revealing resistance or uncertainty among some staff members towards the practical implications of standard setting. They also feel that implementing standards creates an uncomfortable academic environment, but 81% think the standard-setting process is successful (4.0 ± 0.7).

The high mean scores (4.0–4.4) and agreement percentages (up to 88%) in sections on perceptions and practices indicate strong support for standard setting among respondents. This suggests a recognition of its value in improving educational quality.

The data suggests that standard settings are supported, but there’s a need for more engagement and clear communication about its processes. Apprehension about workload and administrative tasks may stem from a lack of understanding or poor implementation strategies. Institutions could benefit from transparent, inclusive, and efficient management of standard-setting processes, including regular updates, training sessions, and support systems. A balanced approach is needed to address apprehensions and ensure positive outcomes for educational quality and student success.

D. Thematic Analysis Outcomes

The participants were Assessment Committee Members (n=8) from the Assessment Committee of the DSMA who were involved in developing the Standard setting for the Assessment.  The data collected from IDIs were organised into five themes and 19 sub-themes in Figure 2.

Figure 2. Showing the main themes and sub-themes

1) Perceptions of the assessment committee members: The assessment committee members expressed appreciation for the incorporation of standard setting in the Outcome-based Integrated curriculum at our university. Most committee members showed enthusiasm for using this method, which they believed was fitting for assessing students in this program. Additionally, the committee aimed to enhance the assessment techniques used in the Outcome-based Integrated curriculum. They were convinced that implementing standard settings would improve student and DSMA quality.

“An outcome-based integrated curriculum has replaced DSMA’s old curriculum. The product of an outcome-based curriculum is not produced. So, we must prepare for that product to be assessed using the standard setting. It is a transitional period for our DSMA from the old curriculum to the outcome-based curriculum. So, the state holders and teachers from DSMA must understand the standard setting. We must set the standard for the assessment of DSMA, but slowly and steadily.”

(Participant 1)

“If the integrated curriculum is changed and outcomes are improved, the quality of the university will improve. Local or international qualifications will be enhanced, not only by students but also by the quality of the DSMA.” 

(Participant 3)

2) Understanding of standard settings: It was evident that members emphasised the importance of standard setting, particularly in the context of Pass/Fail exams for students. Furthermore, committee members demonstrated knowledge regarding various types of standard setting. They were able to identify the most suitable method for DSMA, taking into consideration factors such as cultural context and institutional requirements.

“The standard setting is a method designed to determine a student’s pass or fail scores, and it is also a technique for determining whether students will pass or fail a test.”

(Participant 4)

“DSMA has to set standards for the assessment according to our culture, values, and conditions. We have taken the standard setting of another prestigious university as a reference. It is not suitable to apply the ready-made model.”

(Participant 1)

3) Benefits of implementing standard settings: The assessment committee members recognised Standard Settings are essential for effective curriculum alignment, meaningful student assessment, and accreditation. DSMA can demonstrate its commitment to delivering high-quality education and meeting accreditation standards by aligning the curriculum, teaching practices, and assessments through Standard Settings. The committee members believe Standard Settings play a crucial role in enhancing the quality of education DSMA provides.

“It is beneficial for constructive alignment. So, we adopted and prepared the standard setting for assessing DSMA for accreditation. It is also essential for the teaching and learning process and method. We can check the teaching and learning process, and if errors occur, we can fix them and circle again for the following year.”

(Participant 1)

“I think the outcomes that come out improve not only the student’s quality but also the DSMA’s quality. If the integrated curriculum is changed and outcomes are improved, the quality of the university will improve. Local or international qualifications will be enhanced, not only by students but also by the quality of the DSMA.”

(Participant 3)

4) Roles and responsibilities while implementing standard setting in DSMA: The Assessment Committee members recognised the critical roles and responsibilities of implementing standard setting. They emphasised the need for an inclusive approach involving all stakeholders. Furthermore, they highlighted the importance of aligning standard setting with authentic assessment principles and providing continuous professional development for faculty members. By considering these factors, the committee members aimed to enhance the quality of assessment and promote meaningful learning outcomes for students.

“The standard setting must be built with support; all teachers at DSMA are responsible for it because only the medical education department knows about the standard setting, which is ineffective. Teachers in all departments, including professors, must know about the standard setting. That’s why all the teachers at the university need to understand when setting standards.”

(Participant 2)

“The assessment committee is in charge of the standard setting in DSMA, including the medical education committee, the curriculum committee, the quality assurance committee, the assessment committee, etc. The assessment committee can emphasise its role and responsibility, including implementing the standard setting. Thus, it is more reasonable and practical.”

(Participant 6)

5) Challenges and weaknesses of implementing the standard-setting: The Assessment committee members identified several challenges in implementing standard setting. These challenges include limited human resources, resistance to change, time-consuming processes, and increased workload. Understanding these challenges through the lens of educational theories can guide the development of strategies to address them effectively and ensure the successful implementation of standard-setting practices.

“Human resources are our main weakness. There are very few human resources in DSMA to apply the standard setting. Time-consuming is another area for improvement.”

(Participant 1)

“Most importantly, students don’t know what a “standard” setting is, and some teachers don’t know either. They need to know what a “standard” setting is.”

(Participant 7)

IV. DISCUSSION

The outcomes from the study at the Defence Service Medical Academy (DSMA) assessment committee shed light on essential aspects of standard-setting in medical education. In institutions like DSMA, which implement the outcome-based integrated curriculum, medical students must remember what they learned in their preclinical training to make sound clinical decisions based on reliable scientific principles (Lazić et al., 2006). In medical education, it is crucial to establish, maintain, and frequently review assessment criteria (Senanayake & Mettananda, 2005).

The study revealed a predominantly male composition within the assessment committee, highlighting the need for diversity and expertise among committee members. While increasing the diversity of academic medical leadership to address the underrepresentation of minorities and women in medicine has been proposed, there has been limited investigation into this topic (Meadows et al., 2023).  Encouraging diversity can contribute to robust discussions and decision-making processes related to standard-setting. While committee members demonstrate a solid grasp of standard-setting, the results indicate a requirement for continual education to translate this knowledge into practical application (Norcini & McKinley, 2007). Therefore, targeted interventions and professional development programs must enhance their knowledge and awareness.

Committee members show favorable attitudes towards standard-setting, but there is a discrepancy between their perceptions and actual practices. Implementing open communication strategies consistent with Downing’s (2005) emphasis on the value of clarity in assessment practices could close this gap. Moreover, the lack of active participation in the implementation process may indicate a more significant engagement problem within academic committees (Van der Vleuten et al., 2010).

The study highlights issues that align with existing literature on medical education reform, citing resource limitations and reluctance to change as typical obstacles to adopting innovative instructional approaches (Swing, 2007). The committee’s alignment with constructivist and cognitive theories advocates for a transition to more interactive and participatory learning methods, which a parallel change in evaluation methods should accompany (Harden & Crosby, 2000). Committee members emphasised the need for an inclusive approach involving all stakeholders and aligning standard-setting with authentic assessment principles, resonating with sociocultural theories emphasising social interaction and collaboration in the learning process.

The findings offer crucial insights into the committee’s perceptions and experiences with standard-setting in the integrated outcome-based curriculum. Aligned with educational theories, these insights underscore the importance of lifelong learning and can guide future research and targeted interventions to continuously enhance standard-setting practices.

V. CONCLUSION

In conclusion, the findings from both the quantitative and qualitative research provide valuable insights into the assessment committee’s knowledge, perceptions, attitudes, and practices regarding standard-setting in medical education at the Defence Service Medical Academy (DSMA). While there is a positive attitude towards the potential benefits of standard-setting, there are also concerns and doubts among some committee members.

The study emphasises the importance of targeted interventions in medical education to improve awareness, address concerns, foster collaboration, and support the successful implementation of standard-setting. The Assessment Committee Members appreciate the incorporation of standard setting in the outcome-based integrated curriculum and recognise its importance during the transitional period. They demonstrate a strong understanding of standard settings, particularly in Pass/Fail exams, and emphasise the need to consider cultural and institutional requirements. Standard settings can enhance curriculum alignment, meaningful student assessment, and accreditation purposes.

The committee members emphasise the roles and responsibilities associated with implementing standard setting, including an inclusive approach, continuous professional development for faculty members, and addressing challenges such as limited human resources, resistance to change, time-consuming processes, and increased workload. Strategies informed by educational theories are needed to improve human resource capacity, provide training on standard setting, and create awareness among students and teachers.

These findings underscore the importance of supporting the assessment committee in their understanding, implementation, and continuous improvement of standard-setting practices. By addressing concerns, providing necessary resources, and promoting collaboration, DSMA can optimise the benefits of standard-setting in medical education and ensure the quality of its educational programs. Incorporating lifelong learning fosters continuous professional development, keeping DSMA at the forefront of educational excellence and adaptability.

Notes on Contributors

Dr. Zaw Phyo contributed significantly to the manuscript development process, including reviewing existing literature, identifying research gaps, and addressing questions and goals. He helped draft and revise the manuscript, emphasising its importance for medical education.

Titi Savitri Prihatiningsih contributed to shaping the research question and framework, ensuring a systematic approach. She applied her expertise in evaluation to refine the research tools and meticulously reviewed the thesis and manuscript for accuracy.

Dr. Ye Phyo Aung engaged in the study’s methodology phase, endorsing the chosen research design, sampling strategies, and data collection techniques, adding valuable insights to the research execution.

Dr. Tun Tun Naing assisted in the meticulous proofreading of the article, ensuring clarity, coherence, and compliance with academic standards.

Ethical Approval

The Ethical Review Committee of the Defence Services Medical Academy,  Yangon, Myanmar, granted ethical approval. (2 / Ethics/ 2021).

Data Availability

The data supporting this study’s findings are openly available in the Figshare repository, https://doi.org/10.6084/m9.figshare.25657875.v1 (Phyo et al., 2024).

Acknowledgment

I would also like to thank the FAIMER Regional Institute of Indonesia for Educational Development and Leadership (FRIENDSHIP) for helping me finish this research project at the Faculty of Medicine, Public Health, and Nursing at Universitas Gadjah Mada (Indonesia).

I would also like to thank my supervisor, the head of the medical education department (DSMA), and my colleagues. Without their help, I would not have been able to survey my research. I am grateful to all the assessment committee members from DSMA, Myanmar, who helped me complete my survey by giving me their valuable opinions and experiences.

Funding

The research was carried out during the study period at the FRIENDSHIP-FAIMER Regional Institute of Indonesia for Educational Development and Leadership. No financing is associated with this publication.

Declaration of Interest

There is no conflict of interest in the current research.

References

Alvesson, M., & Sköldberg, K. (2017). Reflexive methodology: New vistas for qualitative research. Sage publications.

Barman, A. (2008). Standard setting in student assessment: Is a defensible method yet to come? Annals Academy of Medicine Singapore, 37(11), 957-963. https://doi.org/10.47102/annals-acadmedsg.V37N11p957

Cohen-Schotanus, J., & Van Der Vleuten, C. P. (2010). A standard-setting method with the best-performing students as a point of reference: Practical and affordable. Medical Teacher, 32(2), 154-160. https://doi.org/10.3109/01421590903196979

Côté, L., & Turgeon, J. (2005). Appraising qualitative research articles in medicine and medical education.  Medical Teacher, 27(1), 71–75. https://doi.org/10.1080/01421590400016308

Crespo, R. M., Najjar, J., Derntl, M., Leony, D., Neumann, S., Oberhuemer, P., Totschnig, M., Simon, B., Gutierrez, I., & Delgado Kloos, C. (2010). Aligning assessment with learning outcomes in outcome-based education.  Proceedings of the IEEE International Engineering Education Conference (EDUCON) 2010. https://doi.org/10.1109/EDUCON.2010.5492385 

Cresswell, J. W. (2012). Educational research: Planning, conducting, and evaluating quantitative and qualitative research (4th ed.). Pearson.

Creswell, J. W., & Clark, V. L. P. (2017). Designing and conducting mixed methods research (3^rd ed.). Sage.

Downing, S. M. (2005). The effects of violating standard item writing principles on tests and students: The consequences of using flawed test items on achievement examinations in medical education. Advances in Health Sciences Education, 10(2), 133-143. https://doi.org/10.1007/s10459-004-4019-5

Epstein, R. M. (2007). Assessment in medical education. New England Journal of Medicine, 356(4), 387-396. https://doi.org/10.1056/nejmra054784

Greene, J. C. (2006). Toward a methodology of mixed methods social inquiry. Research in the Schools, 13(1), 93-98. https://www.researchgate.net/publication/228968099

Harden, R. M., & Crosby, J. (2000). AMEE Guide No 20: The good teacher is more than a lecturer – The twelve roles of the teacher. Medical Teacher, 22(4), 334–347. https://doi.org/10.1080/014215900409429

Harden, R. M., Crosby, J., Davis, M., & Friedman, R. M. (1999). AMEE Guide No. 14: Outcome-based education: Part 5-From competency to meta-competency: A model for the specification of learning outcomes. Medical Teacher, 21(6), 546–552. https://doi.org/10.1080/01421599978951

Ivankova, N. V., & Stick, S. L. (2007). Students’ persistence in a distributed doctoral program in educational leadership in higher education: A mixed methods study. Research in Higher Education, 48(1), 93-135. https://doi.org/10.1007/s11162-006-9025-4

Lane, A. S., Roberts, C., & Khanna, P. (2020). Do we know who the person with the borderline score is, in standard-setting and decision-making. Health Professions Education, 6(4), 617-625. http://doi.org/10.1016/j.hpe.2020.07.001

Lazić, E., Dujmović, J., & Hren, D. (2006). Retention of basic sciences knowledge at clinical years of medical curriculum. Croatian medical journal, 47(6), 880-882. https://pubmed.ncbi. nlm.nih.gov/17167861

Meadows, A. M., Skinner, M. M., Hazime, A. A., Day, R. G., Fore, J. A., & Day, C. S. (2023). Racial, ethnic, and sex diversity in academic medical leadership. JAMA Network Open, 6(9), e2335529. https://doi.org/10.1001/jamanetworkopen.2023.35529

Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook (2nd ed.) Sage Publications.

Myint, K. M. (2020). Curriculum reform and teaching methodologies: A survey on teachers’ perspectives in the university of medicine 1, Yangon. Myanmar Medical Journal, 62(1), 27-33.

Naing, T. T., Minamoto, Y., Aung, Y. P., & Than, M. (2022). Faculty development of medical educators: Training evaluation and key challenges. The Asia Pacific Scholar, 7(3), 23-32. https://doi.org/10.29060/TAPS.2022-7-3/OA2742

Norcini, J. J. (2003). Setting standards on educational tests. Medical Education, 37(5), 464-469. https://doi.org/10.1046/j.1365-2923.2003.01495.x

Norcini, J. J., & Guille, R. (2002). Combining tests and setting standards. In G. R. Norman, C. P. M. van der Vleuten, D. I. Newble, D. H. J. M. Dolmans, K. V. Mann, A. Rothman, & L. Curry (Eds.), International Handbook of Research in Medical Education (pp. 811-834). Springer. https://doi.org/10.1007/978-94-010-0462-6_30

Norcini, J. J., & McKinley, D. W. (2007). Assessment methods in medical education. Teaching and teacher education, 23(3), 239-250. https://doi.org/10.1016/j.tate.2006.12.021

Phyo, Z., Aung, Y, P., Naing, T. T., & Prihatiningsih, T. S. (2024). Perception of the assessment committee on the standard setting in medical education [Data set]. Figshare. https://doi.org/10.6084/m9.figshare.25657875.v1

Senanayake, M. P., & Mettananda, D. S. (2005). Standards medical students set for themselves when preparing for the final MBBS examination. Annals of the Academy of Medicine, Singapore, 34(8), 483–485.

Supe, A., & Burdick, W. (2006). Challenges and issues in medical education in India. Academic Medicine, 81(12), 1076-1080. https://doi.org/10.1097/01.acm.0000246699.94234.ab

Swing, S. R. (2007). The ACGME outcome project: Retrospective and prospective. Medical Teacher, 29(7), 648–654. https://doi.org/10.1080/01421590701392903

Van der Vleuten, C., Schuwirth, L., Scheele, F., Driessen, E., & Hodges, B. (2010). The assessment of professional competence: Building blocks for theory development. Best Practice & Research Clinical Obstetrics & Gynaecology, 24(6), 703-719. https://doi.org/10.1016/j.bpobgyn.2010.04.001

Van Der Vleuten, C. P. M. (1996). The assessment of professional competence: Developments, research and practical implications. Advances in Health Sciences Education, 1(1), 41-67. https://doi.org/10.1007/bf00596229

Wiliam, D. (1996). Meanings and consequences in standard setting. Assessment in Education: Principles, Policy & Practice, 3(3), 287-308. https://doi.org/10.1080/0969594960030303

World Federation for Medical Education. (2020). Basic medical education WFME global standards for quality improvement – The 2020 revision. https://wfme.org/wp-content/uploads/2022/03/WFME-BME-Standards-2020.pdf  

*Zaw Phyo
No.94, Pyay Road, Mingaladon Township,
Postal code – 1102
Yangon, Myanmar
Phone: 95 92032754
Email: dr.zawphyoo@gmail.com

Submitted: 6 April 2024
Accepted: 10 December 2025
Published online: 1 April, TAPS 2025, 10(2), 57-64
https://doi.org/10.29060/TAPS.2025-10-2/OA3332

Chatchai Kreepala¹, Srunwas Thongsombat², Krittanont Wattanavaekin³, Taechasit Danjittrong⁴, Nattawut Keeratibharat⁵ & Thitikorn Juntararuangtong¹

¹School of Internal Medicine, Institute of Medicine, Suranaree University of Technology, Thailand; ²Department of Orthopedics, Faculty of Medicine, Prince of Songkla University, Thailand; ³Department of Surgery, Faculty of Medicine, Ramathibodi Hospital, Mahidol University, Thailand; ⁴Department of Anesthesiology, Chulabhorn Hospital, Thailand; ⁵School of Surgery, Institute of Medicine, Suranaree University of Technology, Thailand

Abstract

Introduction: Factors affecting decision-making duration in MCQs can enhance assessment effectiveness, ensuring they accurately measure the intended objectives and address issues related to incomplete exams due to time constraints. The authors aimed to explore the aspects of medical student’s perspective regarding the factors influencing their decision making on MCQ assessments.

Methods: A mixed-methods explanatory sequential design was utilised. Initial surveys were conducted using percentages, mean and non-parametric analysis obtained via online questionnaires from the sample group: all 2^nd – 5^th year medical students from SUT, Thailand. The validity of the questionnaires was verified by three independent reviewers (IOC=0.89). This was followed by semi-structured group interviews to explore student’s perspective on the factors affecting their decision. Qualitative analysis was conducted to explore detailed information until data saturation was achieved.

Results: Data from the quantitative analysis identified four factors that students believe affect the duration of the exam: the total word count of each question, test difficulty, and images in tests. Meanwhile, the qualitative analysis provided additional insights on factors such as the examination atmosphere affecting their decisions.

Conclusion: This report indicated that data acquired from a comprehensive reading question should be distinguished from those requiring decisive reading. Apart from text length, question taxonomy-such as recall or application- and questions with given images and tables should be considered as factors determining time allocation for an MCQ. Future research based on these results should aim to develop a mathematical formula to calculate exam duration, accounting for question difficulty and length.

Keywords:           MCQ, Medical Assessment, Medical Education, Testing Time Estimation, Qualitative Research, Students’ Perspective

Practice Highlights

• The multiple-choice question (MCQ) stands as one of the objective assessment methods, widely regarded as the most utilised form of assessment.
• The word-length effect has been proposed to determine the length of each examination.
• Educational theories on decision-making have posited that decision-making is a dynamic process stemming from prior experiences.
• The authors were interested in exploring the aspects of the medical student’s perspective about the factors affecting their decision on MCQs answering.

I. INTRODUCTION

The multiple-choice question (MCQ) stands as one of the available objective assessment methods, widely regarded as the most utilised form of assessment, particularly within the fields of medical sciences and technology. Evidence suggests that the recall of short words often surpasses that of longer words (Tehan & Tolan, 2007). This observation is frequently analysed within the framework of a working memory model and the role of the phonological loop in immediate recall. However, the word-length effect has also been observed in delayed tests and in lists that surpass the memory span, thereby challenging the working memory interpretation of the phenomenon. Three alternative interpretations of the word-length effect have been proposed to explain how an exam length should be determined (Arif & Stuerzlinger, 2009; Kumar et al., 2021).

Educational theories on decision-making have posited that decision-making is a dynamic process stemming from prior experiences (Phillips et al., 2004) and meaningful learning (Foley, 2019). As a result, the ability to comprehend text while reading does not automatically equate to reading for decision-making or answering questions. From the literature, the context of factors influencing medical students’ decisions on MCQs includes 1) Length or number of words: The time students need to read to gather information before making a decision on an answer (Arif & Stuerzlinger, 2009). 2) Difficulty of the questions: analytical thinking, especially calculations are involved, may increase decision-making time. This depends on the students’ prior learning experiences before the exam (González et al., 2008). 3) Language comprehension: since exams in medical schools are often in English, non-native speakers may take longer to read and understand the questions (Schenck, 2020). 4) Visuals and tables: these serve as symbols that help students retrieve information from their prior learning experiences more easily (Ziefle, 1998). It is certain that teachers want academic assessment tests, such as MCQs, to be used to distinguish between high-performing and low-performing students and to assess the knowledge and understanding they have acquired. However, these objectives may be undermined by issues such as students running out of time and resorting to guessing. This inevitably reduces the reliability of the test.

The authors were interested in exploring medical student’s aspect regarding factors affecting their decision on MCQs answering. Previous studies focused on duration required for question comprehension and understanding but not for analysis. These were also mostly done in native Englisher speakers. This study builds upon previous studies but with an emphasis on factors affecting non-native English speakers’ decision making after analysis of the provided questions to answer MCQs in English. This research should be approached from the perspective of the student to obtain appropriate data. Semi-structured qualitative interviews were analysed in conjunction with quantitative data to identify and clarify the reasons and factors that students believe influence their performance on exams.

II. METHODS

A. Study Population

The research participants were second to fifth-year Thai medical students who had taken MCQ tests during their preclinical and clinical years between the academic years 2021-2022. Questionnaires were sent to all students without sampling.

To minimise data artifacts caused by recall bias, the online questionnaires were distributed the first week after each MCQ test before the study to the students who completed the exams. All examinations in this study were computer-based, closed book, single best answer MCQs written in English. The participants were non-native English speakers of Thai nationality (as detailed in Definition of Terms). An online survey or questionnaire-based study was used to collect information from participants in this study. If the data was unsaturated, triangulated data from a group of interviews consisting of students from different rotations was included to receive as much information from students’ perspectives as possible.

B. Study Design and Data Collection

The authors employed a mixed method study comprising a quantitative approach and a sequential, explanatory approach. The literature review unveiled several factors influencing MCQ test duration, including the number of questions, question types (recall or comprehension), subject matter difficulty, calculation items, and picture identification, as outlined in the questionnaire (O’Dwyer, 2012). 

An online survey or questionnaire-based study was used to collect information from participants with minimal disruption to their learning activities. The quantitative research section was managed by CK, NK and TJ.  Students completed the questionnaire once, based on their experiences in medical school. This, therefore, necessitates the researcher to summarise the responses and, if required, categorise interviews into groups according to their year of study. Open-ended questions were included in the last section of the questionnaire. The open-ended questions about the factors that, in the student’s opinions, were helpful information about the other factors affecting MCQ time (Lertwilaiwittaya et al., 2019). Survey research was employed as a quantitative method, while semi-structured group interviews were utilised in qualitative data collection to gather insights from medical students’ perspectives. The interview questions were designed to investigate whether students possessed any additional insights regarding the factors influencing MCQ test duration (Carnegie Mellon University, 2019; Schenck, 2020; Wang, 2019).

There were three sections in the questionnaire. Part I consisted of the instruction and informed consent. Part II consisted of general information of the participants, including sex, age, and academic year. Part III consisted of the questionnaires covering all four constructive domains previously mentioned (the domains affecting MCQ time from the literature included: 1) the number and total word count, 2) English language questions, 3) calculation questions, 4) the analytical thinking questions and open-ended questions about the factors that, in the students’ opinions, were helpful information about the other factors affecting MCQ time. After the questions in Part One were completed, they would be taken away so that the researchers would not be able to identify whose students have answered Part II and Part III.

To prevent neutral opinions from students, each questionnaire item featured a four-point Likert scale corresponding to levels of agreement: ‘Strongly disagree,’ ‘Disagree,’ ‘Agree,’ and ‘Strongly agree.’ The researcher wanted clear opinion whether the students were trending towards which side, hence the four-point Likert scale to prevent neutral opinion which may complicate statistical analysis. Validity of the questionnaires were verified by three independent reviewers with an Index of Item-Objective Congruence (IOC) value of 0.89.

Semi-structured group interviews were adopted into this study as insufficient flexibility is provided by a structured interview, whereas unstructured interviews would be too flexible. Semi-structured group interviews were the combination of formal and informal interviews focusing on personal experience; this often leads to unexpected results, enhancing the quality of data collected.

These interviews would take place after class by independent interviewers without any conflict of interest. Two facilitators were present in each session, CK facilitated the conversation and NK contributed ideas. The two facilitators were known by the student participants as faculty members, but they were not actively engaged in their academic learning.  Audio and written recording would be coded then decoded by the researchers (SK, KW and TD).

The interview would take around 30-45 minutes per group, with each group consisting of five to eight people. Analysis would be done after the first three groups using relevant domain analysis and further analysis done after new interviews until data saturation was achieved.  Coding, theme identification, and triangulation would be undertaken following the analysis and evaluation of the quantitative and qualitative data of which the analysis could be extrapolated to form a conclusion of the study. In this study, the open-end question would be analysed, and the semi-structured interview would be done.

Triangulation helped to provide meaning and helped to gain broader and more precise understanding. It could help increase validity. Triangulation was undertaken following the analysis and evaluation of the quantitative and qualitative data of which the analysis was extrapolated to form a conclusion of the study.

C. Definition of Terms

1) Multiple choice question (MCQ): This paper exclusively focused on the Single Best Answer (SBA) Multiple Choice Questions (MCQs), which were structured as questions followed by 4 or 5 potential answers, with only one correct response per question (Coughlin & Featherstone, 2017).

2) Taxonomy MCQ: MCQs were formulated based on two assumptions: that they could be categorised into higher or lower orders according to Bloom’s taxonomy (Stringer et al., 2021). This study sought to comprehend students’ approaches to questions by examining variances in their perceptions of the Bloom’s level of MCQs regarding their knowledge and confidence. The authors employed Bloom’s taxonomy in this study, classifying questions as “recall,” “comprehension,” and “application” (Stringer et al., 2021).

3) Non-native English speakers: The term non-native English speakers was defined as those students who spoke a language other than English domestically. Non-native English speakers were inclusive of both competent bi-literate and limited English proficiency students. In addition, it is also defined as students who learn the language as older children or adults (Cassels & Johnstone, 1984).

D. Statistical Analysis

Statistical analyses were performed for quantitative analysis with SPSS Statistics for Windows, Version 18.0 (SPSS Inc., Chicago, Illinois, USA). Information in the quantitative section was elaborated and displayed in and counts percentage. The qualitative data was analysed by code grouping of text fragments based on content. Subsequently, the codes were reorganised and grouped, main themes and subthemes were identified, and illustrative quotations were selected. The authors assigned other three medical teachers to undertake independent coding of the transcripts for each interview. The final coding and discussions continued until the frameworks were agreed upon and new themes were derived (CK, SK, KW and TD).

III. RESULTS

A. Demographic Information

The questionnaire was done online by the participants from second to fifth-year medical students in the academic year 2021-2022. There were 93 second-year medical students, 92 third-year medical students, 92 fourth-year medical students, and 93 fifth-year medical students, respectively, with 370 participants in total. It was found that there were 298 respondents (a return rate of 81%). 73 second-year medical students (78% response rate) answered the questions, while 70 third year (76%), 75 fourth year (81%), and 80 fifth year medical students (86%) answered the questions respectively as shown in Table 1.

D. The Number Tests and Total Word Count

The exam questions, according to some students, were challenging and time-consuming, and the answer options were likewise lengthy. It was shown that not only the number of tests, but the length of each test item also affected the testing time.

Quote: Student B1F*; “The questions were too long. I can’t complete them in time.”

Quote: Student A2M*; “If there are too many questions in the exam, I wouldn’t be able to finish it”

* student’s code

English Language Questions and Examiners (Native Versus Non-Native English Speakers): The respondents, who were not native English speakers, believed that the English-language test took longer to finish than the Thai-language test. Accordingly, they decided to guess or answer each question slowly since they could not understand the questions. They believed that the English-language tests took longer to finish than the Thai-language tests. Accordingly, they decided to guess or answer each question slowly since they could not fully understand the English questions.

Quote: Student D1F*; “I’m not good at reading English. Sometimes I just have to guess on the exam.”

Quote: Student C1M*; “The language in the test is too hard to understand.”

* student’s code

E. Test Difficulty Determining Time Allocation

For the analysis of coding, grouping, and generating themes, the author found that the medical students paid attention to the difficulty level of the questions which affected the decision to answer the questions.

1) The Calculation and Analytical Thinking: The calculation and analytical thinking tests took students longer to read. Additionally, students believed that examinations they had never taken before or exams that required knowledge application took longer to complete, such as exams that included questions requiring the students to diagnose patients by themselves which occasionally left them unsure of how to respond.

Quote: Student C2M*; “Calculation tests take a long time to get the answers.”

* student’s code

2) Recall Question Leads to Quick Answers: Students commented that recall-type questions, including tests from previous academic years, contained duplicated sentences, pictures, or messages from textbooks that students remembered. This led to students being able to complete the test in a short thinking time.

Quote: Student K1M*; “If the teacher copied the exact words from the course sheet, I would remember and answer questions quickly.”

Quote: Student L1M*; “If the questions are the same as in the sheet provided, I can answer them.”

*student’ code

This information indicated that the taxonomy of the test (recall -compression-application) had a large effect on decision time. Applied questions, not direct or calculated questions, required more attention and time for decision-making when compared to comprehensive questions (questions about knowledge understanding). In contrast, recall questions required the least decision-making time.

F. The Visual Image and Atmosphere of the Examination: the New Derived Domains Recognised by Qualitative Analysis

1) Questions with images, graphs, or tables serve as key guides for decision-making: The students thought that the exams that consisted of graphs and tables helped them understand the questions and were better than the questions that only had descriptions. That would lead to less time consumed.

Quote: Student L2M*; “If the test got the exact same summary table from the book, I could remember and get the answers right away.”

* student’s code

2) The Atmosphere of the Examination: The environment and atmosphere of the exam were also mentioned. The student’s response time was slowed down by the distractions during the exam. The environment such as brightness, temperature, and examination devices affected the concentration of the students.

Quote: Student H1F*; “The atmosphere in the exam venue, noise, and the air quality in the room affect the exam results.”

*student’s code

IV. DISCUSSION

The results revealed that students perceived lengthy exam content or a large number of questions as time-consuming, particularly when exams were conducted in English. Studies indicated that English speakers could read up to 150 words per minute (Trauzettel-Klosinski et al., 2012). However, for non-native English speakers, the expected reading time for exams was longer. Hence, using the English reading rate as a basis for determining exam duration was deemed unsuitable for Thai students, given that English was not their primary language of communication. When compared with a previous study (Trauzettel-Klosinski et al., 2012), the increased duration may result from decision making, thus this implicates reading for decision making requires more time than reading for the context which is cumulatively longer for non-English native speakers.

Qualitative findings indicated that irrespective of the exam duration set by the administering professor, students generally completed exams within the allotted time frame. This often entailed guessing answers towards the end of the exam period, as students might not have adequate time to complete the exam thoroughly. It was observed that students tended to resort to guessing exam questions approximately five minutes before the exam conclusion, thereby minimising threats to validity posed by guessing due to time constraints during exam (Foley, 2019).

There may be limitations if the exam questions contain lengthy content that cannot be comprehended and decided upon within one minute. Furthermore, the difficulty level of the exam questions is often established as a passing criterion, prioritising validity considerations in terms of content format and achieving the intended objectives. Moreover, students naturally desire to obtain the highest possible score on the exam, regardless of the level of difficulty or length of the exam. Therefore, it is important for students to manage their time effectively to ensure they can complete all the exam questions within the given timeframe.

The qualitative results indicated that regardless of the exam duration set by the administering professor, students ultimately would complete the exam within the allotted time frame. Additionally, students agreed that application and calculation questions on the exam require more time to read and decide upon, as opposed to questions with figures and tables that aided in faster decision making. Based on these findings, it could be concluded that comprehensive reading rates may not be a reliable indicator of decision-making reading rates, particularly in the context of medical school exams. Therefore, studying decision-making reading rates within the context of medical school exams was crucial.

The researcher, therefore, examined the domain and specific factors on the characteristics of the MCQ test. Additionally, the study scope was limited to English tests administered to non-native English speakers and onsite computer-based tests, thereby eliminating unrelated factors that could impact exam duration. The analysis yielded the following results: Firstly, factors positively correlated with exam duration (negatively correlated with decision-making) included the number of questions, total word count, calculation questions, and analytical thinking questions. Secondly, factors negatively correlated with exam duration (positively correlated with decision-making) were recall questions, questions with provided images, and tables.

A factor contributing to longer reading times for decision-making purposes was when the exam contained a higher proportion of application or calculation questions, comprising over 33% of the exam questions, as evidenced by qualitative data from students. Therefore, analysing exam completion time based on reading comprehension data for decision-making purposes is not recommended. Moreover, it should be noted that these factors present internal threats to validity, but they can be managed to ensure that examination tools are effectively used and aligned with intended objectives. Incorporating data from research can lead to the identification of new themes related to factors influencing examination time.

Five constructive domains were identified: 1) the number and total word count, 2) positive difficulty factors (application/calculation questions), 3) negative difficulty factors (recall questions), 4) examiners (non-native English speakers or not), and 5) pictures/symbols in tests.

A distinctive aspect of this study was its targeted focus on Thai medical students who were non–native English speakers. While many studies have examined MCQ performance across broad and diverse populations, this research concentrated on a specific demographic, enabling a more in-depth exploration of how cultural and linguistic factors influence test-taking behaviour. The study uniquely combined quantitative survey data with qualitative insights from semi-structured group interviews. While some research utilised either quantitative or qualitative methods, this study’s integration of both provided a more holistic understanding of student perspectives and experiences (Lertwilaiwittaya et al., 2019). This methodological triangulation strengthened the validity of the findings by cross-verifying quantitative data with qualitative insights. In contrast to many existing studies that focused predominantly on performance metrics (such as scores or pass rates), this research examined the cognitive processes and decision-making strategies students employed while answering MCQs. It investigated how elements like question difficulty, language comprehension, and prior experiences shaped students’ approaches to test questions—a dimension less frequently explored in previous literature.

In conjunction with examination-related factors, students also recognised the importance of considering the test environment within the examination room which was a new finding found using qualitative analysis from this research. This was crucial for promoting student concentration and facilitating accurate response selection in line with assessment tool objectives. It aligned with existing literature, which suggested that the test environment poses a construct irrelevant threat to the validity of educational measurement. The findings from this study may have led to future research on developing a mathematical formula to tailor the exam duration for different sets of questions. This would have involved analysing factors such as the number of words, length, difficulty, and the presence of images and tables in the exam. Additionally, the impact of language proficiency on reading and decision-making time should have been considered, as there may have been differences between native and non-native speakers. The study suggested that the future research direction should include diverse populations of non-native English speakers from different countries and educational contexts. This could help identify whether the findings are consistent across various cultural backgrounds and educational systems. Moreover, conducted longitudinal studies should be used to track students’ performance and decision-making processes over time. This approach could provide insights into how experiences and familiarity with MCQs influence their strategies and confidence levels throughout their medical education.

A major limitation of this research was the variation in learning experiences, exam-taking skills, and analytical thinking among medical students at different year levels, which might lead to differing opinions. Therefore, the researcher needed to conduct qualitative analysis to examine the reasons behind these differences. However, the diversity of experiences might also introduce bias due to varying familiarity with different types of exams. The online format restricted the depth of responses, as students often did not fully articulate their thoughts without immediate follow-up questions, which limited the richness of the qualitative data. Additionally, the focus on Thai medical students constrained the applicability of the findings to other populations or contexts, thereby limiting broader conclusions about non-native English speakers in different educational settings.

V. CONCLUSION

Based on the student’s perspective, data showed questions with lengthy content required more time whilst those with tables or diagrams required less time. This report indicated that the data acquired from a comprehensive reading examination should be distinguished from a decisive reading examination.

In addition to the number of questions and the length of text, factors that should be positively correlated with the duration of the exam include the number of questions, word count, calculation-based questions, and analytical thinking questions. These factors should be considered for additional time allocation beyond the regular exam duration, particularly when the proportion of analytical thinking questions exceeds one-third of the total question set. On the other hand, recall questions, as well as questions accompanied by images and tables, should be taken into account to ensure a balanced distribution of exam time, as they can be answered more easily and quickly in terms of decision-making compared to general questions.

Notes on Contributors

CK conceived of the presented idea, developed the theory, and performed the computations and discussed the results and contributed to the final manuscript. ST, KW, and TD. discussed the results and wrote the manuscript with support from CK, NK, and TJ, designed the model and the computational framework and analysed the data.

Ethical Approval

All participants voluntarily signed a consent form prior to participating in the study. The participation protocol was approved by the Human Research Ethics Committee, Suranaree University of Technology (Issue # EC-64-102).

Data Availability

Institutional ethical clearance was given to maintain the data in the secure storage of the principal investigator of the study. The data to this study may be provided upon reasonable request to the corresponding author. A preprint of our manuscript, which is not peer-reviewed, is available at https://www.researchsquare.com/article/rs-3019852/v1

Acknowledgement

The authors would like to thank the participants of this study, the medical students in the Institute of Medicine, Suranaree University of Technology. Without their passionate participation and input, the validation survey could not have been successfully conducted.

Funding

This work was supported by the Grant of Suranaree University of Technology (contract number SUT-602-64-12-08(NEW)).

Declaration of Interest

The authors have no conflicts of interest to disclose.

References

Arif, A. S., & Stuerzlinger, W. (2009). Analysis of text entry performance metrics [Conference presentation]. 2009 IEEE Toronto International Conference Science and Technology for Humanity (TIC-STH), Canada. https://doi.org/10.1109/TIC-STH.2009.5444533

Cassels, J., & Johnstone, A. (1984). The effect of language on student performance on multiple choice tests in chemistry. Journal of Chemical Education, 61(7), 613. https://doi.org/10.1021/ed061p613

Coughlin, P., & Featherstone, C. (2017). How to write a high-quality multiple-choice question (MCQ): A guide for clinicians. European Journal of Vascular and Endovascular Surgery, 54(5), 654-658. https://doi.org/10.1016/j.ejvs.2017.07.012

Carnegie Mellon University. (2019). Creating exams Eberly Center. https://www.cmu.edu/teaching/assessment/assesslearning/creatingexams.html

Foley, B. P. (2019). Getting lucky: How guessing threatens the validity of performance classifications. Practical Assessment, Research, and Evaluation, 21(1), 3. https://doi.org/10.7275/1g6p-4y79

González, H. L., Palencia, A. P., Umaña, L. A., Galindo, L., & Villafrade M, L. A. (2008). Mediated learning experience and concept maps: A pedagogical tool for achieving meaningful learning in medical physiology students. Advances in Physiology Education, 32(4), 312-316. https://doi.org/10.1152/advan.00021.2007

Kumar, D., Jaipurkar, R., Shekhar, A., Sikri, G., & Srinivas, V. (2021). Item analysis of multiple choice questions: A quality assurance test for an assessment tool. Medical Journal Armed Forces India, 77, S85-S89. https://doi.org/10.1016/j.mjafi.2020.11.007

Lertwilaiwittaya, P., Sitticharoon, C., Maikaew, P., & Keadkraichaiwat, I. (2019). Factors influencing the National License Examination step 1 score in preclinical medical students. Advances in Physiology Education, 43(3), 306-316. https://doi.org/10.1152/advan.00197.2018

O’Dwyer, A. (2012). A teaching practice review of the use of multiple-choice questions for formative and summative assessment of student work on advanced undergraduate and postgraduate modules in engineering. All-Ireland Journal of Teaching and Learning in Higher Education, 4(1). https://doi.org/10.21427/D7C03R

Phillips, J. K., Klein, G., & Sieck, W. R. (2004). Expertise in judgment and decision making: A case for training intuitive decision skills. Blackwell Handbook of Judgment and Decision Making, 297, 315. https://doi.org/10.1002/9780470752937.ch15

Schenck, A. (2020). Examining the influence of native and non-native English-speaking teachers on Korean EFL writing. Asian-Pacific Journal of Second and Foreign Language Education, 5(1), 2. https://doi.org/10.1186/s40862-020-00081-3

Stringer, J., Santen, S. A., Lee, E., Rawls, M., Bailey, J., Richards, A., . . . Biskobing, D. (2021). Examining Bloom’s taxonomy in multiple choice questions: Students’ approach to questions. Medical Science Educator, 31(4), 1311-1317. https://doi.org/10.1007/s40670-021-01305-y

Tehan, G., & Tolan, G. A. (2007). Word length effects in long-term memory. Journal of Memory and Language, 56(1), 35-48. https://doi.org/10.1016/j.jml.2006.08.015

Trauzettel-Klosinski, S., Dietz, K., & Group, I. S. (2012). Standardized assessment of reading performance: The new international reading speed texts IReST. Investigative Ophthalmology & Visual Science, 53(9), 5452-5461. https://doi.org/10.1167/iovs.11-8284

Wang, A. (2019, July 15). How to determine the best length for your assessment. Pear Deck Learning. https://www.peardeck.com/blog/how-to-determine-the-best-length-for-your-assessment

Ziefle, M. (1998). Effects of display resolution on visual performance. Human Factors, 40(4), 554-568. https://doi.org/10.1518/001872098779649355

*Assoc. Prof. Chatchai Kreepala, M.D.
Institute of Medicine
Suranaree University of Technology
Thailand
+66(93)3874665
Email: chatchaikree@gmail.com

Submitted: 17 April 2024
Accepted: 18 December 2025
Published online: 1 April, TAPS 2025, 10(2), 65-70
https://doi.org/10.29060/TAPS.2025-10-2/OA3336

Rachael Tufui Masilomani¹, Sophaganie Jepsen¹, Maria Lourdes Villaruel¹, Aying Wang¹, Alena Kotoiwasawasa¹, Lusiana Naikawakawavesi¹, Norman Bartolome¹, Claudia Paterson², Andrew Hill² & Maria Concepcion Bartolome¹

¹Basic Clinical Medicine, Fiji National University, Fiji; ²Department of Surgery, Middlemore Hospital, University of Auckland, New Zealand

Abstract

Introduction: The MBBS programme at Fiji National University reduced its teaching weeks from 18 to 14 weeks in 2018. The purpose of this study was to assess student perceptions of learning and teachers following the reduction in the number of teaching weeks from 18 to 14 weeks.

Methods: A questionnaire was created using a modified Dundee Ready Education Environment Measure (mDREEM) tool (23 items). This was comprised of two subscales; Students Perception of Learning (SPL – 12 items) and Students Perception of Teachers (SPT – 11 items). This was circulated to Year 5 MBBS students through an online survey in 2020.

Results: The response rate was 96%. The students regarded their educational environment as positive in both length of teaching weeks, with an overall mDREEM mean score of 63.29 in 18 weeks and 62.03 in 14 weeks. No statistically significant differences were found between 18 week and 14 week scores across mDREEM scores, SPL scores or SPT scores. The highest scoring item over both was ‘teachers are knowledgeable’.

Conclusion: A positive perception was noted for both lengths of teaching weeks. Reducing the teaching weeks from 18 to 14 did not statistically significantly reduce students’ perception of their educational environment. Items identified with low scores will give a window of opportunity for lecturers and to improve these areas. Future studies may explore the use of the five subscales of the DREEM tool and this study can be integrated into further evaluations of educational environment at Fiji National University.

Keywords:           Medical Students, Educational Learning Environment, DREEM Tool, Fiji National University, Teaching, Medical Education

Practice Highlights

• mDREEM scores did not significantly differ between 18 weeks and 14 weeks of teaching.
• The highest scoring item over both weeks was ‘teachers are knowledgeable’.

I. INTRODUCTION

The Fiji National University (FNU) was founded in 2010 by the merging of six academic institutions in the Fiji Islands, including the Fiji School of Medicine (FSM). FNU has continued the FSM’s tradition of educating and training a diverse population of students from Fiji and neighbouring Pacific Island nations.  The Bachelor of Medicine and Bachelor of Surgery (MBBS) programme is a six-year course at FNU. As part of the academic year, each pre-clinical year group have a teaching week block course. During this time, students receive 2-4 hours of scheduled lectures, 2-hour tutorials twice weekly, as well as 2 hours of clinical skills and 2 hours of anatomy, 2 hours of laboratory sessions and a health centre attachment 4 hours a week.

In 2018, FNU reduced the teaching week block course in the MBBS programme from 18 weeks to 14 weeks. The teaching weeks were shortened due to the decision to move specialty courses such as Psychiatry and Public Health to their respective clinical blocks from Years 4 to 6. This adaptation was challenging for the lecturers, who had to reformat their teaching sessions, in addition to providing resources onto an online Moodle platform. It is well established that the educational environment plays an important role in determining the academic success of medical students (Prosser et al., 1999; Ramsden 2003). Therefore, it is important to evaluate the impact of any major changes to the educational environment, such as a reduction in teaching weeks from 18 weeks to 14 weeks.

Previous research has demonstrated that the duration of clinical rotations has been able to be decreased without adversely affecting the academic success of medical students. For example, one group demonstrated that a shortened four-week clinical rotation in Obstetrics and Gynaecology provided enough opportunity for final year medical students to undertake a quality improvement project in the curriculum (Kool et al., 2017).

The Dundee Ready Education Environment Measure (DREEM) is a quantitative tool used to evaluate students’ perceptions of the educational environment in medical schools. The DREEM tool consists of 50 items, each rated on a scale from 0 to 4. It has five domains, allowing for a maximum score of 200. Higher scores indicate a positive perception of the learning environment (Roff et al., 1997).

A systematic review published in 2012 analysed 40 studies reporting DREEM scores from 20 countries. This review demonstrated that DREEM had been used internationally for various purposes, including diagnostic assessments and comparative studies across different groups (Miles et al., 2012). Five studies focused on investigating the impact of a changed curriculum, which was our area of interest (Demirören et al., 2008, Edgren et al., 2010, O’Brien et al., 2008, Riquelme et al., 2009, Till et al., 2004). We identified three key themes: 1) the DREEM tool was able to highlight areas of concern and/or remediation among students (Riquelme et al., 2009 Till et al., 2004), 2) DREEM scores were different in different phases of medical education, with year 3 students having the highest scores, and year 5 students having the lowest scores (Demirören et al., 2008, Riquelme et al., 2009), and 3) the DREEM tool identified clusters of students based on how positively they perceived the curriculum (O’Brien et al., 2008).

The aim of this research was to compare students’ perception of learning and teachers between 18 weeks and 14 weeks, using a modified DREEM (mDREEM) tool which utilises two of the five domains from the original DREEM tool. The authors’ hypothesis was that students’ perception of learning and teaching would be reduced in with a decrease in teaching weeks to 14 weeks compared to 18 weeks.

II. METHODS

A. Participants

An online survey was developed by the authors. This was distributed via email to eligible participants using Google Forms from 12^th December to 17^th December 2020. Eligible participants included the Year 5 MBBS cohort of 2021, as this cohort was exposed to both 18 week and 14 week teaching programmes. Participation in the survey was optional and results were anonymous. Submission of a completed survey was taken as providing informed consent to participate in this research. Full ethics approval was provided through FNU’s College Human Health Ethics Committee (ID: 292.20). Facility approval was granted to conduct the research.

B. mDREEM Tool

The authors selected two of the five domains of the DREEM tool to be included in the survey. The selected two domains included students’ perceptions of learning (SPL) and students’ perceptions of teachers (SPT). The rationale for using only these two domains was that they were the two domains of interest for the teachers, and it was thought that a survey with less questions would be more likely to be filled to completion by more of the students. Using a Likert scale, each item was scored from 0 to 4, with 4 = strongly agree, 3 = agree, 2 = not sure, 1 = disagree and 0 = strongly disagree. Six of the 23 statements in the mDREEM tool were negative statements: 1) the teaching over-emphasised factual learning; 2) the teaching is too teacher-centred; 3) teachers ridicule the students; 4) the teachers are authoritarian; 5) teachers get angry in class and 6) the students irritate the teachers. These were scored in a reverse manner.

The mDREEM tool had a maximum score of 92. The SPL domain included 12 items, with a maximum score of 48. The SPT domain included 11 items, with a maximum score of 44. This research used the following guide to interpret the overall scores:

• 0–23 = Very poor environment
• 24 – 46 = A large number of problems in the environment
• 47–69 = More positive than negative environment
• 70–92 = Excellent

C. Statistical Analysis

Analysis of the collected data was by using R version 4.3.1. Mean scores were reported with standard deviations. Paired t-tests were performed to compare aspects of the mDREEM scores over 18 weeks and 14 weeks, with a statistical significance threshold of p<0.05. Reliability analysis of the mDREEM tool was assessed using Cronbach’s Alpha test, where >0.7 was deemed acceptable internal consistency.

III. RESULTS

Seventy-eight out of eighty-one (96%) MBSS Year 5 students participated in the online survey. Fifty-one (65%) were females and 51% of participants were aged between 23 and 25 years old. Fijian of Indian descent students made up the majority of the responders (67%) followed by other ethnicities (18%) and i-Taueki students (15%). The majority of students resided at the FNU Hostel (69%) while 28% lived within Suva and only 3% lived outside Suva.

Table 1 illustrates the 23 individual mDREEM items with mean scores across 18 weeks and 14 weeks. The item ‘the teaching overemphasised factual learning’ scored the lowest for both 18 and 14 weeks. The item ‘teachers are knowledgeable’ scored the highest for both 18 and 14 weeks.

Items	mDREEM item	Code	18 weeks		14 weeks
			Mean	SD	Mean	SD
1	I am encouraged to participate in class	SPL	3.13	0.91	2.78	1.03
2	The teaching is often stimulating	SPL	2.79	1.02	2.65	0.94
3	The teaching is student-centred	SPL	2.97	0.95	2.79	0.92
4	The teaching helps to develop my competence	SPL	3.06	0.82	2.63	0.92
5	The teaching is well focused	SPL	2.67	1.00	2.55	1.03
6	The teaching helps to develop my confidence	SPL	2.88	1.01	2.79	0.94
7	The teaching time is put to good use	SPL	2.81	1.12	2.60	1.00
8	The teaching over-emphasised factual learning	SPL	1.85	0.92	1.91	0.79
9	I am clear about the learning objectives of the course	SPL	2.77	1.02	2.78	0.98
10	The teaching encourages me to be an active learner	SPL	2.86	1.16	2.97	0.88
11	Long term learning is emphasised over short term learning	SPL	2.59	1.13	2.56	0.98
12	The teaching is too teacher-centred	SPL	2.85	0.80	2.74	0.78
13	The teachers are knowledgeable	SPT	3.18	0.83	3.13	0.80
14	The teachers are patient	SPT	3.01	0.73	2.74	0.89
15	The teachers ridicule the students	SPT	2.49	0.97	2.56	0.97
16	The teachers are authoritarian	SPT	2.49	0.96	2.58	0.91
17	The teachers have good communication skills	SPT	3.04	0.90	2.99	0.71
18	The teachers are good at providing feedback to students	SPT	2.54	1.15	2.86	0.96
19	The teachers provide constructive criticisms	SPT	2.85	0.90	2.92	0.84
20	The teachers give clear examples during class	SPT	2.78	0.91	2.79	0.84
21	The teachers get angry in class	SPT	2.63	1.01	2.63	0.85
22	The teachers are well prepared for their classes	SPT	2.87	0.90	2.92	0.81
23	The students irritate the teachers	SPT	2.23	0.95	2.23	0.83

The mean SPL scores over 18 weeks and 14 weeks were 33.23 (SD 7.38) and 31.74 (SD 7.52), respectively, out of a maximum of 48 (SPL 18 weeks: 69.33%; SPL 14 weeks: 66.13%). The mean SPT scores over 18 weeks and 14 weeks were 30.06 (SD 6.34) and 30.28 (SD 5.74), respectively, out of a maximum of 44 (SPT 18 weeks: 68.32%; SPT 14 weeks: 68.82%). The mean mDREEM scores over 18 weeks and 14 weeks were 63.29 (SD 12.58) and 62.03 (SD 12.01), respectively, out of a maximum of 92 (mDREEM 18 weeks: 68.80%; mDREEM 14 weeks: 67.42%). These results are presented in Table 2.

	SPL 18	SPL 14	SPT 18	SPT 14	mDREEM 18	mDREEM 14
Mean	33.23	31.74	30.06	30.28	63.29	62.03
SD	7.38	7.52	6.34	5.74	12.58	12.01

Table 2. Descriptive summary statistics for 18 and 14 teaching weeks

The mean difference in SPL scores between 18 weeks and 14 weeks was 1.48. This difference was not statistically significant (t (77) = 1.61, p = 0.11). The mean difference in SPT scores between 18 weeks and 14 weeks was -0.22, and this was also not statistically significant (t (77) = -0.43, p = 0.67). The mean difference in overall mDREEM scores between 18 weeks and 14 weeks was 1.27, which was also not statistically significant (t (77) = 1.04, p = 0.30).

The reliability analysis for both 18 and 14 teaching weeks found a Cronbach’s Alpha Test of 0.58 for SPL, which was less than the threshold of 0.7 and 0.84 for SPT, which was greater than the threshold of 0.7. For mDREEM, the Cronbach’s Alpha Test was 0.77, which was greater than the threshold of 0.7 and confirmed acceptable internal consistency for the mDREEM tool.

IV. DISCUSSION

This study has demonstrated that a reduction in teaching from 18 to 14 weeks did not result in a significant decrease in mDREEM, SPL or SPT scores among Year 5 MBBS students at FNU in 2020. This finding refuted the authors’ hypothesis of a reduction in teaching weeks leading to student dissatisfaction of the educational environment.

Both 18 weeks and 14 weeks scored within the range of 47 to 69 out of 92, indicating a ‘more positive than negative environment’ based on predefined thresholds outlined in the methods section. The ‘excellent’ threshold of 70+ out of 92 was not obtained, indicating room for improvement. In particular, the item ‘teachers are knowledgeable’ scored the highest for both 18 and 14 weeks indicating that students perceived their teachers to have high levels of knowledge despite the reduction in teaching weeks. In contrast, ‘the teaching overemphasised factual learning’ was identified as the most significant negative statement and provides an area of focus for the lecturers.

One group used the DREEM tool to assess curriculum changes in an Irish medical school and that found that the new curriculum was associated with students reporting higher mean DREEM scores (Finn et al., 2014). While FNU’s curriculum change did not result in higher mDREEM scores, the maintenance of mDREEM scores is encouraging and provides a framework for ongoing work towards improving students’ perception of their educational environment.

This finding was similar to a previous study reviewing shortened medical education rotations previously discussed in the introduction section of this paper (Kool et al., 2017). Given the concordance with other similar research findings, the authors are of the belief that the results from this study are largely generalisable to MBBS students and lecturers worldwide, despite only being a single-centre study.

A major strength of this study was the high response rate of 96%. This ensures that data collected as part of this research is representative of the population of interest as compared to several studies with lower response rates (Al-Ansari et al., 2015; Ali et al., 2012; Hyde et al., 2018). Moreover, the results are comparable to the high response rates of other DREEM studies conducted internationally (Alraawi et al., 2020; Stormon et al., 2019; Till et al., 2004).

One limitation of this study was using only two domains of the DREEM tool, neglecting the three domains of Students’ Academic Self-Perception (SAP), Students’ Perception of Atmosphere (SPA) and Students’ Social Self-Perception (SSP). The authors selected SPT and SPL as the two key domains for this research, and thought that by selecting the two most relevant domains, that this would shorten the questionnaire and improve completion of the questionnaire amongst participants. Future research at FNU should trial the use of all five domains of the DREEM tool to assess whether this provides further insights into how teaching weeks can be improved for MBBS students. It will be of interest to see whether response rates are reduced with the use of the full DREEM tool in a questionnaire. Furthermore, future surveys should investigate perspectives of MBBS students over a range of year groups, given previous research suggesting that Year 3 MBBS students have higher DREEM scores than Year 5 MBBS students. A second limitation of this study was that the alpha value for SPL failed to achieve the threshold of >0.7, making it concerning that this domain was unable to achieve acceptable internal consistency. However, the authors note that SPT and mDREEM both achieved acceptable internal consistency. A final limitation of this study was the fully quantitative nature of the survey – the authors did not provide an option for students to add comments to this survey. Future surveys should provide an option for students to add comments, in order to provide more insights into the perspectives of MBBS students.

V. CONCLUSION

In conclusion, the students rated FNU’s MBBS programme educational environment positively. A reduction in teaching weeks from 18 to 14 did not result in a statistically significant decrease in SPL, SPT or mDREEM scores. This study identified valuable information for the authors regarding the improvement of educational environment for medical students. Utilisation of these results to guide educational development in these areas needing improvement will be of help in shaping the delivery of quality education to medical students. In addition, this research may open a door for further studies to investigate challenges faced by tutors and link it to the perceptions of medical students in their educational environment. Likewise, this study is important for future studies in evaluating the educational climate for FNU and other local and international universities.

Notes on Contributors

Dr Maria Bartolome is the corresponding author for this research. She is a PBL (problem based learning) lecturer at the School of Medicine, Fiji National University. She was involved in conceptualisation, methodology, investigation, formal analysis, and writing the original draft.

Dr Norman Bartolome is a PBL lecturer at the School of Medicine,  Fiji National University. He was involved in conceptualisation, methodology, investigation, and writing the original draft.

Dr Alena Kotoiwasawa is a PBL lecturer at the School of Medicine,  Fiji National University, and was involved in  conceptualisation, methodology, and investigation.

Dr Rachael Masilomani is a former PBL lecturer at the School of Medicine,  Fiji National University. She was involved in conceptualisation, methodology, investigation, and formal analysis.

Dr Lusiana Naikawakawavesi is a PBL lecturer at the School of Medicine,  Fiji National University, and was involved in conceptualisation, methodology, and investigation.

Dr Maria Villareul is a PBL lecturer at the School of Medicine, Fiji National University. She was involved in conceptualisation, methodology, and investigation.

Dr Sophaganie Jepson is a PBL lecturer at Fiji National University. She was involved in conceptualisation, methodology, investigation.

Dr Aying Wang is a former PBL Lecturer at Fiji National University. He was involved in conceptualisation, methodology, investigation.

Dr Claudia Paterson is a research fellow at The University of Auckland, New Zealand. She was involved in formal analysis, reviewing and editing.

Professor Andrew Hill is a colorectal surgeon and Professor of Surgery at Middlemore Hospital, Auckland, New Zealand. He was involved in supervision, reviewing and editing.

Ethical Approval

Ethical approval was provided through FNU’s College Human Health Ethics Committee (CHHREC) – reference ID: 292.20.

Data Availability

The corresponding author is able to provide researchers access to our anonymised dataset, on reasonable request.

Acknowledgements

The authors acknowledge the efforts of the students in participating in this study.

Funding

No funding was used for this study.

Declaration of Interest

The authors do not have any conflicts of interest to disclose.

References

Al-Ansari, A. A., & El Tantawi, M. M. A. (2015). Predicting academic performance of dental students using perception of educational environment. Journal of Dental Education, 79(3), 337–344. https://doi.org/10.1002/j.00220337.2015.79.3.tb05889.x

Ali, K., Raja, M., Watson, G., Coombes, L., & Heffernan, E. (2012). The dental school learning milieu: Students’ perceptions at five academic dental institutions in Pakistan. Journal of Dental Education, 76(4), 487–494. https://doi.org/10.1002/j.0022-0337.2012.76.4.tb05281.x

Alraawi, M. O., Baris, S. A., Ahmari, N. A., Alshadidi, A. B., Abidi, N. A., & Al Moaleem, M. (2020). Analyzing students’ perceptions of educational environment in new dental colleges, Turkey using DREEM Inventory. Bioscience Biotechnology Research Communications, 13(2), 556–564. http://dx.doi.org/10.21786/bbrc/13.2/29

Demirören, M., Palaoğlu, Ö., Kemahlı, S., Özyurda, F., & Ayhan, İ. H. (2008). Perceptions of students in different phases of medical education of educational environment: Ankara University Faculty of Medicine. Medical Education Online, 13, Article 8. https://doi.org/10.3885/meo.2008.Res00267

Edgren, G., Haffling, A. C., Jakobsson, U., McAleer, S., & Danielsen, N. (2010). Comparing the educational environment (as measured by DREEM) at two different stages of curriculum reform. Medical Teacher, 32(6), e233–e238. https://doi.org/10.3109/01421591003706282

Finn, Y., Avalos, G., & Dunne, F. (2014). Positive changes in the medical educational environment following introduction of a new systems-based curriculum: DREEM or reality? Irish Journal of Medical Science, 183(2), 253–258. https://doi.org/10.1007/s11845-013-1000-4

Hyde, S., Hannigan, A., Dornan, T., & McGrath, D. (2018). Medical school clinical placements – The optimal method for assessing the clinical educational environment from a graduate entry perspective. BMC Medical Education, 18, Article 7. https://doi.org/10.1186/s12909-017-1113-y

Kool, B., Wise, M. R., Peiris-John, R., Sadler, L., Mahony, F., & Wells, S. (2017). Is the delivery of a quality improvement education programme in obstetrics and gynaecology for final year medical students feasible and still effective in a shortened time frame? BMC Medical Education, 17, Article 91. https://doi.org/10.1186/s12909-017-0927-y

Miles, S., Swift, L., & Leinster, S. J. (2012). The Dundee Ready Education Environment Measure (DREEM): A review of its adoption and use. Medical Teacher, 34(9), e620–e634. https://doi.org/10.3109/0142159X.2012.668625

O’Brien, A. P., Chan, T. M. F., & Cho, M. A. A. (2008). Investigating nursing students’ perceptions of the changes in a nursing curriculum by means of the Dundee Ready Education Environment Measure (DREEM) Inventory: Results of a cluster analysis. International Journal of Nursing Education Scholarship, 5(1), Article 25. https://doi.org/10.2202/1548-923X.1503

Prosser, M., & Trigwell, K. (1999). Understanding learning and teaching: The experience in higher education. McGraw-Hill Education.

Ramsden, P. (2003). Learning to teach in higher education (2nd ed.). Routledge.

Riquelme, A., Oporto, M., Oporto, J., Méndez, J. I., Viviani, P., Salech, F., Chianale, J., Moreno, R., & Sánchez, I. (2009). Measuring students’ perceptions of the educational climate of the new curriculum at the Pontificia Universidad Católica de Chile: Performance of the Spanish translation of the Dundee Ready Education Environment Measure (DREEM). Education for Health, 22(1), Article 112.

Roff, S., McAleer, S., Harden, R. M., Al-Qahtani, M., Ahmed, A. U., Deza, H., Groenen, G., & Primparyon, P. (1997). Development and validation of the Dundee Ready Education Environment Measure (DREEM). Medical Teacher, 19(4), 295–299.

Stormon, N., Ford, P. J., & Eley, D. S. (2019). DREEM-ing of dentistry: Students’ perception of the academic learning environment in Australia. European Journal of Dental Education, 23(1), 35–41. https://doi.org/10.1111/eje.12384

Till, H. (2004). Identifying the perceived weaknesses of a new curriculum by means of the Dundee Ready Education Environment Measure (DREEM) Inventory. Medical Teacher, 26(1), 39–45. https://doi.org/10.1080/01421590310001642948

*Dr Maria Concepcion Bartolome
Basic Clinical Medicine Department
School of Medical Sciences (SMS)
College of Medicine, Nursing and Health Sciences (CMNHS),
Fiji National University
Hoodless House, Brown St. Suva, Fiji Islands
+679 3311700
Email: maria.bartolome@fnu.ac.fj

Submitted: 5 June 2024
Accepted: 30 October 2024
Published online: 1 April, TAPS 2025, 10(2), 71-81
https://doi.org/10.29060/TAPS.2025-10-2/OA3424

Mary Xiaorong Chen¹, Meredith Tsz Ling Yeung¹, Nur Khairuddin Bin Aron², Joachim Wen Jie Lee³ & Taylor Yutong Liu⁴

¹Health and Social Sciences Cluster, Singapore Institute of Technology, Singapore; ²Rehabilitation Department, Jurong Community Hospital, Singapore; ³Rehabilitation Medicine, Singapore General Hospital, Singapore; ⁴Clinical Support Services Department, National University Hospital, Singapore

Abstract

Introduction: Transitioning from a novice physiotherapist (NPT) to an independent practitioner presents significant challenges. Burnout becomes a risk if NPTs lack adequate support for learning and coping. Despite the importance of this transition, few studies have explored NPTs’ experiences in Singapore. This study aims to investigate the transitional journey of NPTs within this context.

Methods: Conducted as a descriptive phenomenological study, researchers collected data through semi-structured online interviews with eight NPTs from six acute hospitals across Singapore. Simultaneous data analysis during collection allowed for a reflexive approach, enabling the researchers to explore new facets until data saturation. Thematic analysis was employed and complemented by member triangulation.

Results: The challenges NPTs encountered include seeking guidance from supervisors, managing fast-paced work and patients with complex conditions. Additionally, NPTs grappled with fear of failure, making mistakes and self-doubt. They adopted strategies such as assuming responsibility for learning, developing patient-focused approaches, and emotional resilience. However, a concerning trend emerged with the growing emotional apathy and doubts about their professional choice.

Conclusion: This study provides a nuanced understanding of the challenges faced by NPTs during their transition. The workplace should be viewed as a learning community, where members form mutual relationships and support authentic learning. Recommendations include augmenting learning along work activities, fostering relationships, ensuring psychological safety, and allowing “safe” mistakes for comprehensive learning.

Keywords:           Novice Physiotherapist Transition in Practice, Clinical Learning and Supervision, Mentoring, Emotional Resilience and Support, Safe Learning Environment

Practice Highlights

• Gradual assumption of responsibilities helps Novice Physiotherapists (NPTs) build competence.
• Learning should be augmented along with work activities.
• It is important to support NPTs to overcome the fear of failure and self-doubt.
• NPTs’ ability to negotiate learning and emotional resilience are essential.
• Trusting relationships and a safe learning environment are essential to NPTs’ learning.

I. INTRODUCTION

Novice Physiotherapists (NPTs) are physical therapy graduates with two years or less of clinical practice, and during this transition to independent practitioners in clinical settings, they face significant challenges (Martin et al., 2020; Wright et al., 2018). Despite the expectation  of competence, concerns persist regarding NPTs’ abilities in various aspects of their practice.

It was reported that the persistent challenges faced by NPTs include managing workload, handling patients with complex conditions, seeking adequate guidance, and navigating relationship dynamics (Latzke et al., 2021; Mulcahy et al., 2010). One critical issue is the oversight of NPTs’ “new” status, leading to their assignment of patient loads comparable to experienced practitioners. Consequently, NPTs find themselves under tremendous stress in managing patients with complex conditions and diverse sociocultural backgrounds beyond their abilities (Stoikov et al., 2021; Wells et al., 2021). Workloads and time constraints hinder the development of meaningful connections between NPTs and supervisors, affecting teaching and coping abilities (Rothwell et al., 2021). In the busy clinical environment, NPTs cannot solely rely on their assigned supervisors, the support from senior colleagues around them along their developmental journey is necessary. Unfortunately, studies suggest that inadequate support and guidance from senior colleagues exacerbate these challenges (Forbes et al., 2021; Jones et al., 2021; Phan et al., 2022; Stoikov et al., 2020; Te et al., 2022).

Additionally, as NPTs are inexperienced, communicating with patients, their families, and other healthcare professionals present a significant hurdle in clinical decision-making (Atkinson & McElroy, 2016). The pressure to make informed clinical decisions, drawing upon extensive knowledge and experience, contributes to job-related stress and feelings of inadequacy among NPTs (Adam et al., 2013).

Job stress-related symptoms, including exhaustion, self-doubt, and depression, further impact NPTs’ well-being. These symptoms, akin to burnout, result from a mismatch between the worker’s performance and job expectations (Brooke et al., 2020; Pustułka-Piwnik et al., 2014). Studies reveal that burnout affects approximately 65% of physiotherapists in Spain (Carmona-Barrientos et al., 2020). This is a concern as burnout was found to be correlated positively with intentions to leave the profession (Cantu et al., 2022), leading to low morale, and compromised patient service quality (Evans et al., 2022; Lau et al., 2016).

Studies suggest that ill-prepared PTs may feel inadequate and lack confidence in making decisions which can negatively influence their clinical management and support for patients’ needs. For example, PTs who lack the ability to adopt a person focused approach might not be able to manage patients with chronic lower back pain effectively (Gardner et al., 2017). Furthermore, such impacts are subtle, difficult to pinpoint, and can result in poor care quality, low patient satisfaction and staff morale (Gardner et al., 2017; Holopainen et al., 2020; Marks et al., 2017).

In Singapore, the healthcare system is bifurcated into public and private sectors. Public hospitals, which fall under government ownership (Ministry of Health, 2023), are pivotal in delivering healthcare services. These hospitals are organised into three distinct clusters, each serving specific regions within the country. Table 1 for a comprehensive list of public hospitals categorised by their respective clusters.

Submitted: 25 January 2024
Accepted: 20 December 2025
Published online: 1 April, TAPS 2025, 10(2), 46-56
https://doi.org/10.29060/TAPS.2025-10-2/OA3228

Chhaya Divecha¹, Miriam Simon² & Ciraj Mohammed³

¹Department of Paediatrics, College of Medicine and Health Sciences, National University of Science and Technology, Oman; ²Department of Psychiatry and Behavioural Science, College of Medicine and Health Sciences, National University of Science and Technology, Oman; ³Department of Medical Education, College of Medicine and Health Sciences, National University of Science and Technology, Oman

Abstract

Introduction: Paediatric milestones provide a structured method for observing and monitoring a child’s progress and should be part of core paediatric curriculum. However, a literature review reveals that primary care physicians and pediatricians feel inadequate about their knowledge and practice of developmental paediatrics, thus exposing the lacunae in training.

Methods: An intervention was planned amongst final-year medical undergraduate students in Oman during their paediatric rotation. A 90-minute multimodal active learning module incorporating diverse learning orientations was planned and administered as a skill-lab session. Its effectiveness in learner motivation, engagement, and faculty participation was evaluated using a questionnaire based on the ICAP (Interactive, Constructive, Active, and Passive) framework, administered to students at the end of the session.

Results: Responses of the 62 participants indicated a significant association between their overall experience and tasks related to the active, constructive, and interactive elements of the module (p=0.001). The faculty’s role in facilitating the session significantly contributed to students’ overall experience (p=0.000). On linear regression, active, constructive, and interactive components of the module were moderate to high predictors of the participants’ overall learning experience.

Conclusion: It was beneficial to base the teaching module on established learning theories. Active learning strategies proactively fostered student engagement and self-directed learning during the session. Faculty played an important role in planning and customising the content, flow, and delivery to maximise meaningful learning. Such interactive collaboration, especially for theoretical concepts in medicine, enables better student engagement, providing enhanced opportunities for learning, practice, and feedback.

Keywords:           Active Learning, Child Development, Undergraduate Medical Education, Student Engagement, ICAP Framework

Practice Highlights

• Active learning strategies can foster student engagement in teaching paediatrics.
• The use of interactive collaboration for theoretical concepts in medicine is effective.
• The role of faculty is crucial to maximise meaningful student learning.
• Utilisation of learning theories to design modules is beneficial for successful content delivery.

I. INTRODUCTION

Clinical curriculum while preparing students for patient care frequently focuses on abnormal pathologies with normal development or physiology often being neglected or underemphasised (Densen, 2011). Developmental and behavioral paediatrics are integral components of pediatric clinical practice. Paediatric milestones provide a structured method to monitor a child’s progress with a comprehensive understanding of development across all domains (gross motor, fine motor, language, and social) and thus must be adequately addressed in the core paediatrics curriculum. However, review of literature reveals that primary care physicians and paediatricians feel inadequate about their knowledge and practice of developmental paediatrics, thus exposing the gaps in education and training (Bauer et al., 2009; Beggs et al., 2005; Bright et al., 2019; Chödrön et al., 2021; Comley et al., 2008; Honigfeld et al., 2012).

Large group didactic classrooms often involve passive reception, leading to lack of engagement among learners (Huggett & Jeffries, 2021). Undergraduate medical curriculum is transitioning from a passive, teacher-centered approach to a learner-centered, active learning strategy, which has demonstrated enhanced students’ understanding, retention of complex concepts, improved student motivation, and overall course satisfaction (McCoy et al., 2018; O’Sullivan et al., 2022). Innovative active learning approaches on developmental milestones largely employ live visits to community resource sites along with instructional videos, reflective reports, observing live parent-child pairs, and use of short video clips (Clark et al., 2012; Comley et al., 2008). In addition, interactive DVDs containing animated cartoons and questions, live interviews, case vignettes, case write-ups, stimulus videos, observation experiences, discussion groups, field trips, and personal experiences have been documented as teaching-learning strategies (Fox et al., 2007; Leiner et al., 2011). Digital resources such as “Beyond Milestones”, developed as free online material for medical professionals using real-life developmental assessments, have shown improved scores on knowledge, observational expertise, confidence, and learner satisfaction (Connolly et al., 2014).

Though observing children in real-life scenarios (including wards, outpatient departments, well-baby clinics, schools, community centers, etc.) creates opportunities for interactive and authentic learning, restrictions in availability of exposure, time, and faculty, especially during the COVID pandemic have compounded the woes of clinical training. The aforementioned reasons prompted us to develop an active learning module using the ICAP framework to introduce developmental milestones in the paediatric curriculum. This framework identifies four modes of cognitive engagement in active learning: Interactive (I), Constructive (C), Active (A), and Passive (P). Passive modes of cognitive engagement involve receipt of information, compared to active modes which require learners to physically manipulate information provided (Chi & Wylie, 2014). Engagement further increases in the constructive mode as students generate diagrams, questions, etc., and is maximum in the interactive mode where peers collaborate and co-construct knowledge through the process of questioning and responding during a conversation. Research reveals that learning achievement is lowest at P and increases in the order of A, C, and I (Chi & Wylie, 2014). Given that the ICAP framework involves both interactive and active learning, we hypothesised that its application to the education of developmental milestones would further promote and expand learning and performance of undergraduate medical students. Additionally, it would help identify and address gaps in their knowledge and understanding of developmental milestones.

In this study, we developed an innovative learning module for developmental milestones using the ICAP model. The module was active, interactive, experiential, and grounded in the major theories of learning (behaviorism, cognitivism, constructivism, humanism, and social learning theories) to maximise opportunities for learning. This pilot study aimed at testing the effectiveness of the module in terms of learner motivation, engagement, and faculty participation.

II. METHODS

The current study was conducted at a private medical college in Oman. It was a cohort study with a quantitative survey and qualitative component. Final-year undergraduate medical students doing their 6-week clinical rotation in Paediatrics were introduced to the teaching module during their skill-lab session (in groups of 6 to 8 students). Verbal consent was obtained for learner feedback.

The learning session was planned and conducted by faculty researchers with expertise in paediatrics, developmental psychology, and medical education. The learning outcomes of this module on developmental milestones were:

1. To identify age-appropriate milestones in children from birth to 5 years of age.
2. To apply knowledge of milestones for various domains (gross motor, fine motor, language, and social/cognitive) to assess development in various age groups (birth to 5 years).
3. To differentiate between normal and delayed development in children.

The specific learning outcomes were addressed through various strategies as summarised in Figure 1.

Figure 1. Flow chart depicting the sequence of activities mapped with specific learning objectives

Figure 2. Sample of bingo ticket with milestones

F. Scaffolding

The faculty would wrap up the case and summarise major learning points thus consolidating the knowledge about milestones. Throughout the learning session, the facilitator provided cognitive scaffolding by leading the students through different questions, prompts, tasks, and structured interactions enabling them to learn more about developmental milestones. Students had opportunities to work with their existing knowledge and build further on it through the various learning activities, group discussions and faculty-led facilitation.

G. Reflection on Action

Students were divided into groups (2 or 3 students per group) to reflect on their entire learning experience using Borton’s model of reflection (Rolfe, 2014). Their reflections elucidated the “what”- their experience of the activities, “so what”- how the module improved on their prior knowledge and understanding about the topics and “now what”- providing suggestions for improvement and preference for similar active learning strategies in future sessions. Individual sharing within groups was followed by sharing between groups via their representatives.

H. Data Collection and Analysis

Students were asked to complete a feedback form about their learning experience during the activity via an anonymised electronic feedback form administered at the end of the session. The form had 15 items evaluating their learning experience – 11 quantitative (Likert scale-based) and 4 qualitative questions. The quantitative feedback responses were analysed based on the four domains of the ICAP model- questions were framed for Interactive, Constructive, Active, and Passive engagement of students during the active learning strategies. There were 4 items (Max score=20) about the interactive components of the module, 3 items (Max score=15) based on constructive elements, and two items each (Max score=10) on the active and passive engagement of students during the module. In addition, 4 quantitative items assessed feedback on faculty involvement and students’ overall experience.

Data was analysed using IBM’s Statistical Package for Social Studies (SPSS 22; IBM Corp., Armonk, NY, USA). Descriptive statistical methods such as percentages, mean and standard deviation were employed. Cronbach’s alpha method was used to assess internal consistency/test reliability. The Shapiro-Wilk method was used to test normality. Inferential statistical methods such as the Spearman’s correlation analysis and Chi-square test were used to explore significant associations between variables. Linear regression was also used to explore various predictors of the participants’ learning experience. Thematic analysis was performed for qualitative feedback.

III. RESULTS

A. Overview of Study Participants

A total of 62 students from the final clinical year of the MD program participated in this study; 91% were females and 9% were males. The average age of participants was 24.4 years (SD=0.707).

Results indicate high internal consistency for the survey items developed. Full scale (15 items) Cronbach’s alpha value of 0.808 was obtained. Results of the Shapiro-Wilk test of normality for all survey items (p=0.000) indicate that participant’s responses were not normally distributed. Non-parametric tests were thus employed for data analyses.

B. Analysis of Learner Feedback

In general, participants agreed that the session on developmental milestones was enjoyable.

The preference for game-based interactive/collaborative learning was high. The learner responses categorised item wise are provided in Table 1.

Item	Strongly Agree	Agree	Can’t say	Disagree	Strongly Disagree	Mean	SD
The pre-reading material on developmental milestones was easily understandable	59.7%	35.5%	4.8%	0	0	4.548	0.591
The session established clear learning outcomes and objectives.	91.9%	8.1%	0	0	0	4.919	0.274
The case-based activity on developmental milestones provided relevant opportunity to witness theory in practice.	83.9%	16.1%	0	0	0	4.839	0.370
Presentation of real-world contexts followed by discussion in groups helped me learn better.	88.7%	9.7%	1.6%	0	0	4.871	0.383
The game-based activity enabled me to build an emotional connection to learning and the subject matter.	82.3%	16.1%	0	1.6%		4.790	0.516
This session provided me opportunities for feedback and practice	83.9%	16.1%	0	0	0	4.839	0.370
I enjoyed the session on developmental milestones as it actively engaged me with the course material through case study and discussion.	87.1%	12.9%	0	0	0	4.871	0.338
The session assisted us in organising our thoughts, reflecting on our understanding, and finding gaps in our reasoning.	74.2%	21%	4.8%	0	0	4.694	0.560
I prefer similar sessions to learn theoretical concepts in paediatrics and medicine.	88.7%	9.7%	1.6%	0	0	4.871	0.383
Today’s session was well-planned and helped me work on my own to accomplish learning goals independently.	82.3%	17.7%	0	0	0	4.823	0.385
The faculty supported students in the learning process during the session.	98.4%	1.6%	0	0	0	4.984	0.127
The faculty regulated the level of information and maintained my ability to be responsible for my learning.	87.1%	11.3%	1.6%	0	0	4.855	0.398
The session made me identify psychosocial issues associated with developmental delays	69.4%	21%	8.1%	1.6%	0	4.581	0.714
I found myself motivated, engaged and self-directed during the entire session.	82.3%	14.5%	3.2%	0	0	4.790	0.483
Using the interactive online module to explore developmental milestones helped improve my learning in this area.	66.1%	25.8%	6.5%	1.6%	0	4.565	0.692

Table 1. Analysis of survey responses based on dimensions of the ICAP Model

 

C. Analysis of Feedback on Faculty Involvement and Overall Experience

The mean score on the participants’ feedback on the involvement of faculty members during the session was 9.839 (SD=0.450) and their overall experience was 9.710 (SD=0.686).

D. Association among Various Survey Dimensions

Results indicated significant associations between learner’s overall experience and tasks related to the active component of the session (p=0.000), constructive engagement (p=.000), and interactive collaboration with peers (p=0.001). Results also indicated significant associations between the interactive and constructive components of the session (p=0.000), faculty’s role in facilitating the session and students’ overall experience (p=0.000), and faculty involvement with the passive component (designing the initial reading material) (p=0.000).

E. Spearman’s Correlation

Spearman’s correlation across the various survey dimensions based on the ICAP model indicated high positive inter-dimension correlation. A significant correlation was also seen between the survey dimensions and the full-scale scores.

F. Linear Regression

Linear regression analysis was carried out to explore various predictors of the participants’ learning experience. The active (R²=0.438), constructive (R²=0.718), and interactive components (R²=0.644) are moderate to high predictors of the participants’ overall learning experience.

G. Thematic Analysis of Qualitative Items

The following themes and elaborations emerged on analysis of student feedback relating to their experience during the active learning module on developmental milestones which are summarised in Table 2.

Themes	Elaboration of theme	Samples of students’ feedback
(i) Elevated learning experience	Participants expressed positive feedback regarding all components of the active learning session. They enjoyed the game-based activities, technology-integrated tasks, while at the same time enhancing their knowledge on developmental disorders from a clinical perspective.	“The bingo game brought a surprisingly fun twist to a boring topic and the competitiveness of the group as well as the doctors fun proctoring made it an overall pleasant and enjoyable experience. In short, this was a session I’m glad I got out of bed for.” “The smart board table was an active way of remembering.” “Encourage us to learn more and attend to skill lab. It was wonderful.” “A lot better than what I thought I would feel compared to a normal session. Confident to answer any questions related to developmental milestones.”
(ii) Efficacious collaborative interactions	Students reported increased interest in learning as the session involved working together with their peers. Participants felt that continual interaction throughout the session strengthened learning.	“The group interactions really helped me retain the information. It was a fun experience and something new and out of the ordinary.” “Interactions made the information much easier to understand.” “It was interactive which made it interesting.”  “I interacted throughout the session, learned quite a lot of information, very helpful.”
(iii) Reflective outlook to patient care	Participants appreciated the inclusion of humanism in the learning experience. The incorporation of reflective practice in patient and caregiver experiences is a vital component that may impact future outcomes related to patient care.	“Playing the video which wasn’t pure scientific as usual, it is emotional also, so it touches both the doctor and the human inside me.” “Watching the video of cerebral palsy and then reflecting over why development is very important.”
(iv) Supportive learning environment	Student feedback highlighted the presence of a positive, non-judgmental environment that ensured improvement of learning in a relaxed/ stress-free setting.	“We were able to make mistakes and learn from them without the fear of being criticised” “It was very fun and interactive. No pressure was there and not scary.” “It was interactive and very useful and most important comfortable and didn’t feel like we were being pressured and that surely helped us learn way better.” “Very happy and I enjoyed it a lot. My favourite session since the beginning of the year… no pressure was put on us, so we were able to actually learn instead of being terrified.”

Table 2. Thematic analysis of qualitative feedback

IV. DISCUSSION

Monitoring a child’s development over time via milestones is a core part of paediatric curriculum and practice, as it enables early identification of delay or atypical progress, allowing timely referral. As an alternative to traditional learning through lectures as practiced routinely, we utilised active learning strategies that addressed multiple learning orientations for enhanced student engagement and learning.

A. Designing the Module and Active Learning Strategies based on Learning Theories

The developmental milestone module designed to involve multimodal active learning strategies has been described in the methods section. The session began with a novel “lid-opener” showing a real-life video of a child with developmental delay and the psychosocial impact of the disability on the child and family, followed by an active learning strategy; think pair and share (TPS). We explored the utility of “cognitive” orientations to learning by positioning learners to seek and understand the structure of knowledge for meaningful learning. TPS has been known to enhance the process of clinical learning by allowing students to experience different viewpoints on a particular problem and express as well as to listen to others as compared to a traditional classroom lecture (Ganatra et al., 2021; Linsenmeyer, 2021).

A significant proportion (91%) of students agreed that using the interactive module to explore milestones helped improve their learning on the topic which is also reflected in the qualitative feedback (Table 2). The interactive smart board allowed students to explore milestones in a self-directed activity and enabled higher-order cognition through active engagement with the course content. Technology in learning is stimulating for millennial learners and allows them to use online novel educational tools to maneuver their learning process (George & Dreibelbis, 2021). The wide availability of mobile devices, laptops, smart boards, etc. provides the teacher opportunities to use these new technologies effectively to transform learning into a more collaborative, personalised, and empowering experience that is rooted in connectivism- the learning theory of the digital era. We employed a multimodal approach integrating diverse activities such as videos, interactive smart boards, and gamification (via the BINGO game). Integrating many information sources nurtures the learner’s ability to reflect on connections between fields, ideas, and concepts, a core skill linked to the theory of connectivism. (Goldie, 2016).

Case-based learning (CBL) effectively links theory and practice and prepares students for actual clinical application of knowledge through inquiry (Thistlethwaite et al., 2012). The hypothetico-deductive reasoning inherent to a “constructivist” framework was applied while designing this approach (Kalinowski & Pelakh, 2024). It increases the learner’s engagement and motivation for learning through creativity, challenge, interest, and enjoyment afforded through the case-study method. An important part of CBL involves problem-solving through free discussions and with often no correct or incorrect answers, thus broadening the students’ reasoning process (Thistlethwaite et al., 2012). We used a CBL trigger in the form of a video demonstrating an assessment of a child’s development, allowed students analyse the video individually followed by buzz group discussions to share their understanding. All our students unanimously agreed that the case-based activity gave them ample opportunity to witness theory in practice. Buzz groups also provide an opportunity to apply contextual thinking to actual practice and develop collaborative skills. Studies have shown that buzz groups improve clinical reasoning and learning, promote teamwork, increase motivation to learn, improve academic performance through a conducive learning environment, and provide a chance for all members to participate and share their opinions within their sub-group (Abbasi et al., 2017; Balslev et al., 2015; Shrivastava, & Shrivastava, 2018). The elements of exploration and experimentation ensured that the theory of constructivism rooted in the connection of new knowledge to pre-existing knowledge was effectively utilised in the CBL process. Co-creation of this knowledge with peer interactions can be attributed to social learning theories of cognitivism and constructivism.

Selective and purposeful gamification has been known to increase learner motivation and engagement and, ultimately, learning (Rutledge et al., 2018). This was evident in our study where a vast majority (98%) of students agreed that the game-based activity helped them build an emotional connection with the process of learning and concepts being learned, which is also reflected in the qualitative feedback (Table 2). Game-based learning (GBL) builds emotional connections by immersing learners in engaging and interactive environments. These environments evoke emotions such as curiosity, excitement, and empathy, which enhance motivation and deepen the learning experience (Loderer et al., 2020). Based on constructivist theory, the game-based activity integrated their knowledge of age-appropriate milestones with a fun-based BINGO theme. The instant feedback that served as formative assessment in this segment had a behaviorist orientation that is specific and quantifiable.

B. Student Engagement using Multimodal Active Learning Strategies

Studies have noted a preference for multi-modal learning strategies by medical students which have been shown to improve academic performance, and promote deeper learning and enquiry (Alkhasawneh et al., 2008; James et al., 2011; Lujan & DiCarlo, 2006; Nicholson et al., 2016). Different learners have different learning styles and use of a variety of strategies can engage learners synergistically. We used various active learning strategies to facilitate student engagement as shown in Figure 3 which incorporated the ICAP framework of Interactive, Constructive, Active, and Passive activities. In addition, the use of learning theories to rationalise the framing of learning objectives, selection of specific teaching-learning methods/strategies, and design of appropriate evaluation strategies ensured student engagement and meaningful learning.

Figure 3. Summary of active learning strategies used in the module based on ICAP framework

The module had more activities involving interactive and constructive domains as compared to the active and passive modes. It is known that student learning and engagement improve as they move from passive to active to constructive to interactive modes (Chi & Wylie, 2014). Significant association between overall experience and interactive collaboration with peers (p=0.001) as well as constructive engagement (p=0.000) was observed. The feedback was also structured to measure student engagement in all four behaviors. We observed high mean scores in interactive and constructive domains. Learners agreed on enough opportunities for feedback, and practice and appreciated the engagement through case studies and discussions. Effective use of multiple active learning strategies thus enabled self-directed learning and students felt that they could accomplish the learning goals independently. The results underpin the utility of behaviorism and humanism as orientations for the learning process in such activities.

C. Individual and Collaborative Learning

The combination of active learning strategies facilitated engagement at both individual and collaborative levels. Students were involved in individual learning through pre-reading, lid-opener, discourse on development, case-based learning, and game (BINGO). These activities provided opportunities to self-regulate and moderate their learning. It also allowed them to organise their thoughts and reflect on their understanding. Active learning has a positive impact on memory and knowledge transfer and hence, individual self-studying before discussion improves the effectiveness of collaboration (Beggs et al., 2005). Activities initiated with an individual thinking process were balanced by subsequent group discussions. Collaborative activities in the form of Think pair share (TPS) after lid-opener, Buzz groups after CBL learning, and interactive table enabled constructive interaction among students and exchange of information and concepts. The collaborative activities helped students work together towards problem-solving, observe both their own and another’s learning process, discover different ways of approaching a situation, and find gaps in their reasoning. It also helped to build on their team skills, leadership, clinical competence, and interpersonal communication which are core attributes of professionalism in our curriculum (Branch, 2015).

D. Faculty Involvement

Though activities in the module were planned to encourage self-directedness, faculty role in the preparation and implementation of these strategies cannot be undermined. We adopted an amalgamation of humanistic and social cognitive orientation to strengthen the relationship between quality of instructional design and outcomes, thereby influencing learning directly. Perhaps for the same reasons we found a significant association between faculty facilitation with overall learning experience (p=.000). Students felt that the faculty supported them through the learning process throughout the session (100%) and regulated information besides allowing students to be responsible for their learning (98%). There was also a significant association between faculty involvement and the passive component (pre-reading material, discourse) (p=0.00); students agreed that pre-reading material was easily understandable (95%) and clear objectives and outcomes laid down before the session facilitated their understanding (100%). It must be noted that faculty can personalise the module based on the strengths, interests, cultural competencies, and time restraints of the student. Thus, rather than delivering a “one size fits all” module, the faculty can customise the content based on the student’s needs and limitations.

Involving various sensory processes enables better processing and retention of information; thus, enhancing the learning process (Friedlander et al., 2011). As more methods are employed, they are likely to engage more senses thus improving learning. Neuroscientific rationale for constructivist cognition may be seen as a possible explanation (Dennick, 2016).

While the use of active learning techniques in undergraduate medical education is not entirely novel, our approach is distinctive in that we’ve designed an entirely active learning module addressing various orientations to student learning, which ensures individual and collaborative student engagement. The study’s primary advantage is the creation and application of a targeted educational resource to teach developmental paediatrics to undergraduate students. This not only addresses the gap in effectively integrating developmental paediatrics into the core paediatrics curriculum but also demonstrates the feasibility of using a fully active learning approach in other clinical topics. Our detailed methodology aims to assist other educators in transforming their content into interactive teaching modules. The module not only meets the institutional requirement for increasing active learning sessions in the clinical years but also addresses a broader need by offering a framework and learning strategies that can be effectively applied to different topics and courses.

We could only assess student engagement based on their perception, i.e. level 1 of the New World Kirkpatrick Model (reaction), and have not quantified their ability to learn, understand, and apply their learning which constitutes higher levels of the model (Liao & Hsu, 2019). Additional limitations include the fact that this is a pilot study, tailored to a specific context and curriculum, which may restrict its generalisability. It also does not directly compare with other learning methods and lacks long-term tracking of students.

V. CONCLUSION

Paediatric topics such as developmental paediatrics are delivered mainly through didactic orientations and fail to capture student engagement leading to poor comprehension. Our study demonstrates that medical students enjoy sessions involving multimodal active learning strategies, particularly while discussing theoretical concepts that provide opportunities for practice and feedback. Heterogeneous learning strategies which underpin various learning theories and constructs have been shown to increase student motivation and engagement, thus contributing towards retention and deep learning. The faculty have an important role in planning such modules to customise the content and delivery for successful student engagement and effective learning thereafter. The framework and active learning strategies presented in the module can be applied more broadly to other subjects adapting to the needs of other undergraduate faculty in their teaching, thus making active teaching strategies easily transferable. We recommend future research be planned to include pre- and post-session assessments or a crossover study with a control group for comprehensive evaluation. Furthermore, the implementation of active learning strategies to create entirely active modules in other courses within the undergraduate medical program can be explored to assess its potential for broader applicability.

Notes on Contributors

Dr. Chhaya Divecha, Associate Professor of Paediatrics was involved in the conceptualisation and design of the study, literature search, data collection and drafting the manuscript.

Dr. Miriam Simon, Associate Professor of Behavioral Science was involved in the design of the study, analysis, interpretation of results and drafting the manuscript.

Dr. Ciraj Mohammed, Professor of Medical Education was involved in the design of the study and revised the manuscript for scientific content.

All authors have read and approved the final manuscript.

Ethical Approval

Approval to conduct this study was obtained from the institution’s Ethics and Biosafety Committee (NU/COMHS/EBC0036/2022).

Data Availability

Data will be made available by the authors on acceptance of the manuscript.

Acknowledgement

The authors wish to thank the participants for their cooperation in the study.

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Declaration of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interests.

References

Abbasi, F., Mofrad, M. N., Borhani, F., & Nasiri, M. (2017). Evaluation of the effect of training by buzz group method on nursing diagnostic skills of nursing students. Research Journal of Pharmacy and Technology, 10(1), 213-218.  https://doi.org/10.5958/0974-360x.2017.00045.2

Alkhasawneh, I. M., Mrayyan, M. T., Docherty, C., Alashram, S., & Yousef, H. Y. (2008). Problem-based learning (PBL): Assessing students’ learning preferences using VARK. Nurse Education Today, 28(5), 572–579. https://doi.org/10.1016/j.nedt.2007.09.012

Balslev, T., Rasmussen, A. B., Skajaa, T., Nielsen, J. P., Muijtjens, A., De Grave, W., & Van Merriënboer, J. (2015). Combining bimodal presentation schemes and buzz groups improves clinical reasoning and learning at morning report. Medical Teacher, 37(8), 759–766. https://doi.org/10.3109/0142159X.2014.986445

Bauer, S. C., Smith, P. J., Chien, A. T., Berry, A. D., & Msall, M. (2009). Educating pediatric residents about developmental and social–emotional health. Infants & Young Children, 22(4), 309-320. https://doi.org/10.1097/IYC.0b013e3181bc4da0

Beggs, S., Sewell, J., Efron, D., & Orkin, C. (2005). Developmental assessment of children: A survey of Australian and New Zealand paediatricians. Journal of Paediatrics and Child Health, 41(8), 444–448. https://doi.org/10.1111/j.1440-1754.2005.00664.x

Branch, W. T. (2015). Teaching professional and humanistic values: Suggestion for a practical and theoretical model. Patient Education and Counseling, 98(2), 162–167. https://doi.org/10.1016/j.pec.2014.10.022

Bright, M. A., Zubler, J., Boothby, C., & Whitaker, T. M. (2019). Improving developmental screening, discussion, and referral in pediatric practice. Clinical Pediatrics, 58(9), 941–948. https://doi.org/10.1177/0009922819841017

Chi, M. T. H., & Wylie, R. (2014). The ICAP framework: Linking cognitive engagement to active learning outcomes. Educational Psychologist, 49(4), 219–243. https://doi.org/10.1080/00461520 .2014.965823

Chödrön, G., Barger, B., Pizur-Barnekow, K., Viehweg, S., & Puk-Ament, A. (2021). “Watch Me!” Training increases knowledge and impacts attitudes related to developmental monitoring and referral among childcare providers. Maternal and Child Health Journal, 25(6), 980–990. https://doi.org/10.1007/s10995-020-03097-w

Clark, B., Andrews, D., Taghaddos, S., & Dinu, I. (2012). Teaching child development to medical students. The Clinical Teacher, 9(6), 368–372. https://doi.org/10.1111/j.1743-498X.2012.00581.x

Comley, L., Janus, M., Marshall, D., & Niccols, A. (2008). The early years: Child development in undergraduate medical school training. Canadian Family Physician, 54(6), 876.e1-4.

Connolly, A. M., Cunningham, C., Sinclair, A. J., Rao, A., Lonergan, A., & Bye, A. M. (2014). ‘Beyond Milestones’: A randomised controlled trial evaluating an innovative digital resource teaching quality observation of normal child development. Journal of Paediatrics and Child Health, 50(5), 393–398. https://doi.org/10.1111/jpc.12485

Dennick, R. (2016). Constructivism: Reflections on twenty-five years teaching the constructivist approach in medical education. International Journal of Medical Education, 7, 200–205. https://doi.org/10.5116/ijme.5763.de11

Densen, P. (2011). Challenges and opportunities facing medical education. Transactions of the American Clinical and Climatological Association, 122, 48–58.

Fox, G., Katz, D. A., Eddins-Folensbee, F. F., & Folensbee, R. W. (2007). Teaching development in undergraduate and graduate medical education. Child and Adolescent Psychiatric Clinics of North America, 16(1), 67-94. https://doi.org/10.1016/j.chc.2006.07.006

Friedlander, M. J., Andrews, L., Armstrong, E. G., Aschenbrenner, C., Kass, J. S., Ogden, P., Schwartzstein, R., & Viggiano, T. R. (2011). What can medical education learn from the neurobiology of learning? Academic Medicine, 86(4), 415–420. https://doi.org/10.1097/ACM.0b013e31820dc197

Ganatra, S., Doblanko, T., Rasmussen, K., Green, J., Kebbe, M., Amin, M., & Perez, A. (2021). Perceived effectiveness and applicability of Think-Pair-Share including storytelling (TPS-S) to enhance clinical learning. Teaching and Learning in Medicine, 33(2), 184–195. https://doi.org/10.1080/10401334.2020.1811094

George, D. R., & Dreibelbis, T. D. (2021). Active learning in ‘The cloud’: Using “social” technologies to expand the medical classroom. In A. Fornari & A. Poznanski (Eds.), How-to guide for active learning (pp. 85-99). Springer.

Goldie, J. G. (2016). Connectivism: A knowledge learning theory for the digital age? Medical Teacher, 38(10), 1064–1069. https://doi.org/10.3109/0142159X.2016.1173661

Honigfeld, L., Chandhok, L., & Spiegelman, K. (2012). Engaging pediatricians in developmental screening: The effectiveness of academic detailing. Journal of Autism and Developmental Disorders, 42(6), 1175–1182. https://doi.org/10.1007/s10803-011-1344-4

Huggett, K. N., & Jeffries, W. B. (2021). Overview of active learning research and rationale for active learning. In A. Fornari & A. Poznanski (Eds.), How-to guide for active learning (pp. 1-7). Springer.

James, S., D’Amore, A., & Thomas, T. (2011). Learning preferences of first year nursing and midwifery students: Utilising VARK. Nurse Education Today, 31(4), 417–423. https://doi.org/10.1016/j.nedt.2010.08.008

Kalinowski, S. T., & Pelakh, A. (2024). A hypothetico-deductive theory of science and learning. Journal of Research in Science Teaching, 61(6), 1362–1388. https://doi.org/10.1002/tea.21892

Leiner, M., Krishnamurthy, G. P., Blanc, O., Castillo, B., & Medina, I. (2011). Comparison of methods for teaching developmental milestones to pediatric residents. World Journal of Paediatrics, 7(2), 161–166. https://doi.org/10.1007/s12519-011-0269-5

Liao, S. C., & Hsu, S. Y. (2019). Evaluating a continuing medical education program: New world Kirkpatrick model approach. International Journal of Management, Economics and Social Sciences, 8(4), 266-279.

Linsenmeyer, M. (2021). Brief activities: Questioning, brainstorming, think-pair-share, jigsaw, and clinical case discussions. In A. Fornari & A. Poznanski (Eds.), How-to guide for active learning (pp. 39-66). Springer.

Loderer, K., Pekrun, R., & Plass, J. L. (2020). Emotional foundations of game-based learning. In J. L. Plass, R. E. Mayer, & B. D. Homer (Eds.), Handbook of game-based learning (pp. 111–151). MIT Press.

Lujan, H. L., & DiCarlo, S. E. (2006). First-year medical students prefer multiple learning styles. Advances in Physiology Education, 30(1), 13–16. https://doi.org/10.1152/advan.00045.2005

McCoy, L., Pettit, R. K., Kellar, C., & Morgan, C. (2018). Tracking active learning in the medical school curriculum: A learning-centered approach. Journal of Medical Education and Curricular Development, 5, 2382120518765135. https://doi.org/10.1177/2382120518765135

Nicholson, L. L., Reed, D., & Chan, C. (2016). An interactive, multi-modal anatomy workshop improves academic performance in the health sciences: A cohort study. BMC Medical Education, 16, Article 7. https://doi.org/10.1186/s12909-016-0541-4

O’Sullivan, S., Campos, L. A., & Baltatu, O. C. (2022). “Involve Me and I Learn”: Active learning in a hybrid medical biochemistry first year course on an American-style MD program in the UAE. Medical Science Educator, 32(3), 703–709. https://doi.org/10.1007/s40670-022-01545-6

Rolfe, G. (2014). Reach touch and teach: Terry Borton. Nurse Education Today, 34(4), 488–489. https://doi.org/10.1016/j.nedt.2013.11.003

Rutledge, C., Walsh, C. M., Swinger, N., Auerbach, M., Castro, D., Dewan, M., Khattab, M., Rake, A., Harwayne-Gidansky, I., Raymond, T. T., Maa, T., & Chang, T. P. (2018). Gamification in action: Theoretical and practical considerations for medical educators. Academic Medicine, 93(7), 1014–1020. https://doi.org/10.1097/ACM.0000000000002183

Shrivastava, S. R., & Shrivastava, P. S. (2018). Exploring the scope of introducing buzz group in teaching community medicine to undergraduate medical students. MAMC Journal of Medical Sciences, 4(2), 109.

Thistlethwaite, J. E., Davies, D., Ekeocha, S., Kidd, J. M., MacDougall, C., Matthews, P., Purkis, J., & Clay, D. (2012). The effectiveness of case-based learning in health professional education. A BEME systematic review: BEME Guide No. 23. Medical Teacher, 34(6), e421–e444. https://doi.org/10.3109/0142159X.2012.680939

Woolfson, L. (2004). Family well-being and disabled children: A psychosocial model of disability-related child behaviour problems. British Journal of Health Psychology, 9(1), 1–13. https://doi.org/10.1348/135910704322778687

*Dr. Miriam Simon
Department of Psychiatry and Behavioral Science
College of Medicine and Health Sciences
National University of Science and Technology
PO Box 391; PC 321
Sohar, Sultanate of Oman
+96826852039
Email: miriamsimon@nu.edu.om

Submitted: 2 April 2024
Accepted: 12 September 2024
Published online: 7 January, TAPS 2025, 10(1), 27-35
https://doi.org/10.29060/TAPS.2025-10-1/OA3247

Estivana Felaza¹, Ardi Findyartini^1,2, Rita Mustika¹, Jenny Bashiruddin¹, Joedo Prihartono³, Lucia RM Royanto⁴ & Subha Ramani⁵

¹Department of Medical Education, Faculty of Medicine Universitas Indonesia, Indonesia; ²Medical Education Centre, Indonesia Medical Education and Research Institute (IMERI), Faculty of Medicine Universitas Indonesia, Indonesia; ³Department of Public Health, Faculty of Medicine Universitas Indonesia, Indonesia; ⁴Faculty of Psychology Universitas Indonesia, Indonesia; ⁵Brigham Women and Child Hospital, Harvard Medical School, USA

Abstract

Introduction: Professional identity formation requires students’ ability to reflect, which can be supported by constructive feedback from teachers. Feedback practice faces many challenges, including those related to sociocultural aspects. Feedback can be optimised by implementing a model suitable to the attributes of students, teachers, and the clinical environment. This study aimed to develop a suitable feedback model and test its effectiveness in promoting reflection.

Methods: The model was designed based on the theoretical framework and identified feedback-related attributes in FMUI. Expert reviews and cognitive interviews were conducted before the model was tested. The model’s effectiveness was tested using a quasi-experimental design involving 74 students in the Paediatric Module. Students were asked to reflect on their mini-CEX experience after receiving feedback from teachers, and the depth of reflection was compared between students in the intervention and control group. The depth of reflection was measured using Kember scoring, and bivariate analysis was conducted using SPSS 20.0.

Results: The model consists of 5 steps including: Rapport building, Acknowledge students’ strengths, Identify aspect(s) that need improvement, Share teachers’ experiences, and Establish a plan to improve. Deep reflection was more frequently found in the intervention group. Mann-Whitney test showed that the difference between groups was statistically significant (Z=2.964, p=0.003), indicating that reflection in the intervention group was deeper than the control group.

Conclusion: The feedback model formulated based on feedback-related attributes in FMUI was named the RAISE model. Upon testing, students receiving the model could reflect more deeply compared to those receiving the standard model.

Keywords:           Feedback, Clinical, Reflection, Sociocultural

Practice Highlights

• Constructive feedback is needed for medical students to optimally form their professional identity, especially in clinical settings
• The RAISE model which was developed based on feedback-related attributes can be implemented in clinical education
• The RAISE model consists of five steps: Building rapport, Acknowledge students’ strengths, Identify aspect(s) to improve, Share teachers’ experience, and Establish a plan to improve
• The RAISE model promotes students’ ability to reflect on their experiences

I. INTRODUCTION

Medical students’ professional identity is mostly formed during their clinical education (Cruess et al., 2015). Professional identity formation is a long and complex process that occurs through the integration of values and norms of the profession. Socio-constructivist theory can explain how learning occurs in clinical settings through the involvement of students in a medical community of practice. As students immerse themselves in the clinical environment, they engage in numerous interactions with their surroundings. The interactions expose them to professional values, which are gradually internalised as students develop their new identity as medical doctors (Cruess et al., 2015).

Optimal professional identity formation requires students’ ability to reflect on their experiences. Reflection enables students to extract meaning from their hectic day-to-day activities, and create plans to continuously improve themselves (Mann et al., 2009). The ability to reflect is also essential in their future profession as doctors. Reflection was defined as critically analysing one’s own experience to reach new and deeper comprehension and insight. Experience become the object to reflect, and the process of recalling and analysing the experience was conducted with the intention of improving future performance (Mann et al., 2009). Reflection also becomes the media to process feelings and emotions, and create plans to improve performance, which leads to improved clinical decision-making ability (Rozental et al., 2021). The importance of reflective practice made it necessary to provide students with opportunity to become skilful in conducting reflection.

The analysis of reflection should include many perspectives to get comprehensive views and insights. One of the ways to support reflection is through the provision of constructive feedback (Husebo et al., 2015; Sandars, 2009). Feedback has been shown to be beneficial in helping students improve their performance (Choi et al., 2020; Johnson et al., 2020; Trehan et al., 2015). Information given through feedback assists students in identifying the improvement needed and the strategies they can use. Feedback can initiate reflections by pointing out the gap between the intended outcome and the current performance; and it can also strengthen reflection by confirming students’ judgment and promoting insights into their performance (Nicol & MacFarlane-Dick, 2006).

Many problems are still found in the practice of feedback in clinical settings, such as the message not being specific enough and the limited time available for interaction. In addition, the Indonesian culture adds certain unique challenges to feedback practice, as identified in our previous study (Felaza et al., 2023). These challenges can be associated with the cultural dimensions of high-power distance and collectivism in Indonesian society. Teachers, experienced and have expertise, are believed by students to know what is best. In a busy clinical environment, the hierarchical gap creates a preference of both teachers and students to engage in directive feedback. In receiving feedback, students tend to appear receptive even when not fully agreeing with the feedback. Being a collectivistic society, group approaches in seeking, providing, and processing feedback practices were commonly found. Students share their feedback experiences with their peers, resulting in collective perception of teacher that they had interacted with (Felaza et al., 2023). Similar challenges have been reported in other studies conducted in Indonesia and Thailand (Areemit et al., 2020; Suhoyo et al., 2018).

Considering the importance of feedback in facilitating students’ professional identity formation, and the way feedback practice is affected by sociocultural aspects, there is a need to develop a feedback model suitable to the attributes of students, teachers, and the clinical environment in Indonesia. This study aims to develop a feedback model relevant to our setting and assess its effectiveness in supporting students’ reflective ability. This study proposes two research aims:

1. To develop a feedback model suitable for the Indonesian clinical education setting.
2. To evaluate the effectiveness of the feedback model in supporting students’ reflective ability.

 

II. METHODS

This quantitative research was conducted using a design-based research (DBR) approach, comprising the following steps of identifying the problem, designing a solution, evaluating its effectiveness in a real-life setting, and reflecting on the outcomes (Scott et al., 2020). The first step of our DBR was the identification of feedback-related attributes in FMUI, in a study conducted in September 2023. The next steps, described in detail below, included development of a feedback model within our institution based on data from the previous study, followed by evaluation of its effectiveness in promoting reflection.

A. Feedback model development

The development of the model was guided by a theoretical framework from literature and findings of students’, teachers’, and environmental attributes in our setting. Socio-constructivism, as the theoretical framework, perceives learning as a process of making meanings from experiences through social interactions. Discourses between teacher and students exposes students to thought process and professional norms and values which later can be internalised within themselves. The attributes identified were environmental attributes (limited time, collectivistic, hierarchical, and the need for a safe environment), teachers’ attributes (busy, expert, and having the skill to provide feedback), and students’ attributes (dependent on feedback, less able to read between the lines, perceived to be less resilient, appear to be receptive of feedback, critical, and values teacher’s experience). These findings have been published on September 2023 (Felaza et al., 2023). Along with the socio-constructivist approach as the theoretical framework, these attributes would become the foundation in the development of the model.

Review by experts and cognitive interviews were conducted to ensure the model’s readiness for implementation. Review panel included experts on medical education and communication, program coordinator, and module coordinators. The model was sent via e-mail, and experts were asked to provide feedback on each step and the overall applicability of the model. Cognitive interviews involved clinical teachers from surgery-related and non-surgery-related specialties as future users of the model. After receiving explanation of the model, the teachers practiced the model on their clinical teaching activities. Interviews were held with each teacher to gain input on the experience. Inputs from expert review and cognitive interviews were used to refine the model. The final version of the model was taught to clinical teachers in Paediatric Module as a preparation to test the model. The model was tested during Mini-Clinical Evaluation Exercise (Mini-CEX) by comparing its effectiveness with the Feedback Sandwich method as the standard model being used in FMUI. Mini-CEX is a workplace-based assessment which evaluate students’ performance of clinical skills in real setting. The assessment was originally intended for formative purposes as the clinical teacher observes the performance and provide feedback for students (Hejri et al., 2019). The sandwich model was found to be the common method used for feedback provision in FMUI, and it starts with informing students of the positive aspects of their performance, followed by the negative ones, and closed with another positive aspect (Jug et al., 2019).

B. Testing the effectiveness of the model

The design used was quasi-experimental, involving 5^th-year FMUI medical students during their clinical rotation in Paediatric Module. There were 37 students in each group and the data collection took place in May 2022–March 2023. Students in the intervention group received feedback using the new model, while their peers in the control group received feedback using the Sandwich model (Jug et al., 2019). Upon completing data collection in the control group, authors conducted training for teachers on how to utilise the new model, followed by data collection in the intervention group.

Data was collected by asking students to record their voices as they reflect by thinking out loud. FMUI students have been introduced to reflection since the beginning of their academic years, and have experienced completing reflective writing assignments in several modules. For this study, they were instructed to reflect on their experience during mini-CEX. The depth of the reflection was then scored using Kember’s scoring. The scoring was developed by Kember based on Mezirow’s four levels of reflection, and it has been adapted to Bahasa Indonesia (Soemantri et al., 2022). Mezirow divided reflection to 4 levels based on its depth, starting from habitual action which showed no reflection, thoughtful action/understanding, reflection, and critical reflection (Kember et al., 2000). The scoring process was conducted separately by the first author (EF) and two co-authors (AF & RM) and later compared in team discussions to reach an agreed score. Bivariate analysis to test to compare the depth of the reflection between the intervention and the control group was conducted using SPSS 20.0.

III. RESULTS

The criteria for the feedback model were developed based on attributes found from the previous stage (as shown in Table 1 and Table 2).

Table 1. Attributes of Factors Affecting Feedback and The Criteria of the Feedback Model

Table 2. Steps Proposed and How it Accommodates the Criteria of the Feedback Model

The model was named RAISE as an abbreviation of the first letter in each step. The RAISE model was reviewed by an expert panel, consisting of 3 experts in medical education, 2 experts in communication, and 4 clinical program coordinators. Overall, the RAISE model was perceived as suitable to be implemented in the clinical settings by the expert panel (Figure 1). The crucial aspects that needed to be ensured were teachers’ comprehension of the reasoning behind each step, and the adjustability of the model to be applied in limited time and variability of contexts. Rapport building was considered an essential initial step to start feedback interaction. For the second step, it was advised that teachers ask the student to state positive aspects of performance first before confirmation based on their judgment. The focus of the third step was the areas for improvement, a term that experts emphasised. The step which was initially named ‘invite student to reflect’ was changed into ‘identify aspect that needs improvement’. This revision was made to prevent the misconception that reflection was exclusively covered in the 3^rd step. In the fourth step, teachers share their experiences from which students can learn and develop ideas to improve their performance. The information shared should be relevant and can be delivered in a manner that is non-threatening and relevant for the students. The final step in the RAISE model focuses on the action plan for improvement. The agreed plan should be feasible and ready to be implemented.

Figure 1. RAISE Model After Expert Review

The model was then utilised by four clinical teachers in various teaching sessions, and cognitive interviews were conducted to gain insight into the experiences. They reported that the model was relatively easy to implement and able to accommodate the limited time available. The model helped develop rapport between teacher and students and enable the two of them to interact more effectively. Continuation between each step was seamless and the conversation did not have to be interrupted due to the movement from one step to another.

The final version was then taught to 22 clinical teachers in August 2022. The training was organised by the first author and two of the co-authors, who had expertise in medical education and experience in conducting faculty development. Various activities were held in the training (video appraisal, discussions, role play, and reflection) to ensure teachers’ ability to perform feedback provision using the RAISE model.

The final version was compared with the Sandwich model, the standard model being used in FMUI, to evaluate its effectiveness. The comparison between the steps of the two models can be seen in Table 3. As shown in the comparison, the strength of the RAISE model lies in its effort to establish rapport from the start of the feedback conversation, its way of facilitating reflection, and the opportunity it provides for teachers to share their experiences.

Table 3. Comparison Between the RAISE Model and the Standard Model

The effectiveness of the model was then tested during mini-CEX in Paediatric Module by comparing the depth of reflection in the experimental and control group. There were 37 students in each group with majority of students in experimental and control groups (30 and 28 students respectively) were female and belonged in the year 2018 batch.

Students reflected on the feedback they received, and the depth of the reflection was rated using the Kember score with the score ranging from 1-4 (1: no reflection; to 4: deep reflection). The distribution of the depth of students’ reflection is shown in Table 4.

Table 4. Distribution of the Depth of Students’ Reflection

The distribution showed most of the students’ reflections in the intervention group were scored 3 based on the Kember score (26 students or 70.2%); while the score in the control group was almost evenly divided between 2 (16 students or 43.2%) and 3 (17 students or 45.9%). Deep reflection was more frequently found in the intervention group (9 students or 24.4%) compared to the control group (4 students or 11.9%).

Mann-Whitney test was used to compare the depth of students’ reflection between the intervention and the control group. The test showed that the mean rank in the intervention group was 30.61 and in the control group was 44.39. The difference between the two groups was statistically significant (Z = 2.964, p = 0.003), indicating that students’ reflections in the intervention group were deeper than their peers in the control group.

IV. DISCUSSION

This study aimed to develop a culturally relevant feedback model that nurtured reflective ability of medical students and to evaluate its effectiveness. The results indicated that students receiving feedback using the RAISE model could reflect more deeply compared to their peers receiving feedback using the standard model. The RAISE model was designed to enable feedback practice in an interactive manner, while feedback using Sandwich model was more directive. The RAISE model starts by building rapport between teacher and student in order to create a safe environment which helps student to optimally conduct reflection. The 2^nd and 3^rd steps of the model guide students’ reflection systematically, covering both the positive and negative aspects of the performance. On the 4^th step, teachers’ sharing of their experiences helps to enrich students’ reflection, enabling them to learn from the experience, as well as strengthening and further maintaining the rapport between them. The final step concludes the interaction with a plan that students can implement to improve. Each of these steps portrays how reflection is guided and supported by the teacher through the feedback interaction.

The RAISE model was developed based on the socio-constructivist theory that describes learning as a way of constructing meaning from experiences. In clinical settings, students are exposed to a variety of experiences to build comprehension, acquire skill proficiency, and integrate professional norms and values. Effective feedback interactions between teachers and students facilitate students’ reflective process and support them in identifying effective ways to improve their performance (Wenger, 2020). Through these interactions, students are exposed to the teacher’s thought process, as well as their professional norms and values. The process that starts as inter-mental between teacher and student gradually becomes internalised into intra-mental within the student (Amineh & Asl, 2015; Tappan, 2010). This process enables teachers to guide students to reach their zone of proximal development (ZPD). Students are supported at each stage of their performance through the concept of scaffolding, and the level of support given would be decreased gradually as students gain the ability for independent clinical practice (Fani & Ghaemi, 2011).

Our previous study highlighted the need for rapport development and the act of sharing teachers’ own experiences in developing the feedback model (Felaza et al., 2023). Rapport was considered essential to create a safe environment for students. Interactions that empowered students in an authentic and safe environment, and teachers as role models in practicing reflection, were needed for creating reflective habits among students (Gathu, 2022; Mann et al., 2009). A sense of safety in feedback interaction was built upon a good relationship between teacher and students, mutual goals, and students’ perception of the teacher’s credibility (Sargeant et al., 2017). In the RAISE model, the safe environment has been developed since the beginning by building rapport between teacher and students. When students reflect, they would need to be open in analysing their experience. Destructive feedback would create reluctance for students to share their reflections (Chan & Lee, 2021; Khoiriyati & Sari, 2021). The model started by asking students’ perceptions of their performance, and their previous experience with similar situations; encouraging them to be open and reflective. The questions were relatively easy to answer and able to lighten the situation. Safety was continuously built through constructive dialogue between teachers and students which created an environment that was safe for reflection (MacNeil et al., 2020; Telio et al., 2015). Mutual goals also can implicitly be seen in students’ perception of the teacher’s intention. When the students believe that feedback was given to help them improve, they would be more willing to openly reflect (Blakey et al., 2019; Eva et al., 2012).

The reflection process is initiated when the condition cannot be handled simply by the automatic action the individual usually conducts or habitual act (Mann et al., 2009). The initiation of reflection might also be triggered by feedback. The feedback information made the individuals realise the quality of their performance, highlighting the strength and weaknesses that requires improvement (Husebo et al., 2015; Mann et al., 2009). When students receive feedback that is aligned with their perception regarding the quality of their performance, reflection usually occurs faster. However, for feedback information that is not aligned, students need to first manage the negative emotion it might evoke, analyse the message, and contemplate what action plan should be taken (Sargeant et al., 2009).

Teachers, perceived as knowledgeable figures, have numerous valuable experiences they can share. By incorporating the act of sharing experience, every feedback interaction would provide lessons from which students can extract meaningful understanding. This particular approach has not been seen explicitly in other feedback models. Other than enriching students’ comprehension, the sharing also provides a sense of relatedness with the teacher, and a role model on how to become reflective (Burgess et al., 2020). A role model in practicing reflection was an important aspect of strengthening students’ motivation. Teachers who practice self-reflection confirm to students that such practice can be done in their profession as doctors. Teachers’ reflective behaviour also showed the students that admitting weaknesses was not an embarrassing act, and on the contrary, it was an important skill to master to improve (Gishen & Chakrabarti, 2022). In the RAISE model, the opportunity for teachers to demonstrate reflection was mostly done in the 4^th step. In this step, teachers can share their experiences that was relevant to the situation the students faced. Upon sharing the experience, the teacher can explain the lesson learned from that experience. Students receiving feedback using the RAISE model claimed that this step exposed them to teachers’ openness and minimised the power distance between them. Students also felt more motivated by listening to the experiences. Sense of relatedness was one of the factors that increased intrinsic motivation. The teachers’ willingness to share their experiences creates a connection between them and students, and strengthens students’ internal motivation (Kusurkar et al., 2011).

Other than being influenced on a systemic level by cultural aspects, such as power distance and collectivism; reflection is also affected in individual level by students’ and teachers’ reflective ability, teachers’ ability in facilitating reflection, and the relationship between teacher and students (Chan & Lee, 2021). The RAISE model supports teachers in facilitating students’ reflection through the use of reflective questions and the opportunity to show students how reflection was practiced by themselves as they shared their experiences. The rapport built during the interactions also strengthens the teacher-student relationship which further enables students to be more open and trusting in conducting their reflection.

Students’ reflective ability was crucial in their education process and later as they became member of the medical profession. Experiences in the clinical learning environment, which depicts the authentic professional world that students would enter, expose students to norms and values. Through reflection, students create meanings from these experiences which facilitate the formation of their professional identity as medical doctors (Winkel et al., 2017).

This study showed that implementation of the RAISE model, a feedback model developed based on the feedback-related attributes in Indonesian settings, enabled students to reflect more deeply on their performance. Findings from the study suggested that the RAISE model was suitable for supporting students’ reflection in a setting with high power distance and collectivistic background. However, it should be noted as a limitation of this study that reflection was an internal process and the effect might not be visible immediately. The changes in comprehension, perspectives, and even students’ attitude occurs gradually along the reflection process that was taking place. Therefore, further research is needed to observe the effect of reflection which might require longitudinal observation of students’ performance. The study involved a single institution in Jakarta, and therefore research and certain adjustments might be needed before the model can be implemented in different settings.

V. CONCLUSION

The feedback model formulated based on the feedback-related attributes in our clinical setting was named the RAISE model. It consists of 5 steps, which are Rapport building, Acknowledge student’s strengths, Identify aspects that need improvement, Share teacher’s experience and tips, and Establish a plan to improve. Our data indicate that students receiving feedback using the RAISE model were able to reflect more deeply compared to their peers receiving the standard model. We conclude that the model’s ability to create a safe environment, build rapport between teachers and learners, and teachers’ role-modelling of reflective practice can encourage students to be more self-reflective of their clinical performance.

Notes on Contributors

Estivana Felaza, Ardi Findyartini, Rita Mustika, Jenny Bashiruddin, Joedo Prihartono, Lucia RM Royanto, and Subha Ramani contributed to all processes of this research, and have read and approved this manuscript.

Dr. dr. Estivana Felaza, MPdKed is a Lecturer in Faculty of Medicine, Universitas Indonesia. She reviewed the literatures, designed the study, performed data collection and analysis, and wrote the manuscript.

Prof. dr. Ardi Findyartini, PhD is a Professor of Medical Education in the Faculty of Medicine, Universitas Indonesia. She developed the methodological framework for this research, involved in scoring the students’ reflections (data), provided feedback on the data collection, data analysis, and the manuscript writing process.

Dr. dr. Rita Mustika, M.Epid is an expert in Medical Education and a Lecturer in the Faculty of Medicine, Universitas Indonesia. She developed the methodological framework for this research, was involved in scoring the students’ reflections (data), provided feedback on the data collection, data analysis, and manuscript writing process.

Prof. Dr. dr. Jenny Bashiruddin, SpTHT-KL(K) is a Professor in Otolaryngology and an expert in Medical Education. She provided feedback on the theoretical and methodological framework of this research, data collection, data analysis, and the manuscript writing process.

Dr. dr. Joedo Prihartono, MPH is an expert in Statistics and a Senior Lecturer in Community Medicine, Faculty of Medicine, Universitas Indonesia. He provided feedback on the methodological framework of this research, data collection, data analysis, and the manuscript writing process.

Dr. Lucia RM Royanto, M.Si, Sp.Ed is an expert in Educational Psychology and a Senior Lecturer in the Faculty of Psychology, Universitas Indonesia. She provided feedback on the theoretical framework of this research, data collection, data analysis, and the manuscript writing process.

Dr. Subha Ramani, MBBS, MPH, PhD, FAMEE is an expert in Medical Education and an Associate Professor of Medicine in Harvard Medical School. She provided feedback on the theoretical and methodological framework of this research, data collection, data analysis, and the manuscript writing process.

Ethical Approval

The study was approved by the Research Ethics Review Committee of the Faculty of Medicine, Universitas Indonesia (approval number: KET-1114/UN2.F1/ETIK/ PPM.00.02/2021). The subjects of this study were informed verbally and in writing of the purpose, the methods, and the confidentiality of the data being collected. Each subjects’ involvement in this research were voluntary, and consent were signed upon receiving the necessary information.

Data Availability

Datasets generated and/or analysed in this study are available upon request via electronic mail to the corresponding author.

Acknowledgement

The authors would like to acknowledge the contributions of the students and clinical teachers who willingly involved and cooperated in this study. 

Funding

No funding was obtained for this study.

Declaration of Interest

All authors have no declaration of interest.

References

Amineh, R. J., & Asl, H. D. (2015). Review of constructivism and social constructivism. Journal of Social Science, Literature & Language, 1(1), 9–16.

Areemit, R. S., Cooper, C. N., Wirasorn, K., Paopongsawan, P., Panthongviriyakul, C., & Ramani, S. (2020). Hierarchy, “Kreng Jai”, and feedback: A grounded theory study exploring perspectives of clinical faculty and medical students in Thailand. Teaching and Learning in Medicine, 33(3), 235–244. https://doi.org/10.1080/10401334.2020.1813584

Blakey, A. G., Smith-Han, K., Anderson, L., Collins, E., Berryman, E., & Wilkinson, T. (2019). “They cared about us students”: Learning from exemplar clinical teaching environments. BMC Medical Education, 19, 119–128. https://doi.org/10.1186/ s12909-019-1551-9

Burgess, A., van Diggele, C., Roberts, C., & Mellis, C. (2020). Feedback in the clinical settings. BMC Medical Education, 20(Suppl 2), 458–463. https://doi.org/10.1186/s12909-020-02280-5

Chan, C. K. Y., & Lee, K. K. W. (2021). Reflection literacy: A multilevel perspective on the challenges of using reflections in higher education through a comprehensive literature review. Educational Research Review, 32(2), Article 100376. https://doi.org/10.1016/j.edurev.2020.100376

Choi, S., Oh, S., Lee, D. H., & Yoon, H. S. (2020). Effects of reflection and immediate feedback to improve clinical reasoning of medical students in the assessment of dermatologic conditions: A randomized controlled trial. BMC Medical Education, 20, 146–154. https://doi.org/10.1186/s12909-020-02063-y

Cruess, R. L., Cruess, S. R., Boudreau, J. D., Snell, L., & Steinert, Y. (2015). A schematic representation of the professional identity formation and socialization of medical students and residents: A guide for medical educators. Academic Medicine, 90(6), 718–725. https://doi.org/10.1097/acm.0000000000000700

Eva, K. W., Armson, H., Holmboe, E., Lockyer, J., Loney, E., Mann, K., & Sargeant, J. (2012). Factors influencing responsiveness to feedback: On the interplay between fear, confidence, and reasoning process. Advance in Health Science Education, 17, 15–26. https://doi.org/10.1007/s10459-011-9290-7

Fani, T., & Ghaemi, F. (2011). Implications of Vygotsky’s zone of proximal development in teacher education. Social Behavioral Science, 29, 1549–1554. https://doi.org/10.1016/j.sbspro.2011.11. 396

Felaza, E., Findyartini, A., Mustika, R., Bashiruddin, J., Royanto, L. R. M., Prihartono, J., & Ramani, S. (2023). Deeper look into feedback practice in an Indonesian context: Exploration of factors in undergraduate clinical setting. Korean Journal of Medical Education, 35(3), 263–273. https://doi.org/10.3946/kjme.2023.264

Gathu, C. (2022). Facilitators and barriers of reflective learning in postgraduate medical education: A narrative review. Journal of Medical Education and Curriculum Development, 9, 1–8. https://doi.org/10.1177/23821205221096106

Gishen, F., & Chakrabarti, R. (2022). Medical students’ perceptions of reflective practice in the undergraduate curriculum. MedEd Publish, 12, 53–67. https://doi.org/10.12688/mep.19211.3

Hejri, S. M., Jalili, M., Masoomi, R., Shirazi, M., Nedjat, S., & Norcini, J. (2019). The utility of mini clinical examination exercise in undergraduate and postgraduate medical education: A BEME review: BEME guide no. 59. Medical Teacher, 42(2), 1–18. https://doi.org/10.1080/0142159x.2019.1652732

Husebo, S. E., O’Reagen, S., & Nestel, D. (2015). Reflective practice and its role in simulation. Clinical Simulation in Nursing, 11, 368–375. http://doi.org/10.1016/j.ecns.2015.04.005

Johnson, C. E., Weerasuria, M. P., & Keating, J. L. (2020). Effect of face-to-face verbal feedback compared with no alternative feedback on the objective workplace task performance of health professionals: A meta-analysis and systematic review. BMJ Open, 10(3), Article e030672. https://doi.org/10.1136/bmjopen-2019-030672

Jug, R., Jiang, X., & Bean, S. M. (2019). Giving and receiving effective feedback. Archives of Pathology and Laboratory Medicine, 143, 244–250. https://doi.org/10.5858/arpa.2018-0058-RA

Kember, D., Leung, D. Y. P., Jones, A., Loke, A. Y., MacKay, J., Sinclair, K., Tse, H., Webb, C., Wong, F. K. Y., Wong, M., & Yeung, E. (2000). Development of a questionnaire to measure the level of reflective thinking. Assessment and Evaluation in Higher Education, 25(4), 381–395. http://doi.org/10.1080/713611442

Khoiriyati, A., & Sari, N. K. (2021). Reflective practice on nursing students: A qualitative study. Indonesian Nursing Journal of Education and Clinic, 6(2), 201–209. http://doi.org/10.24990/in jec.v6i2.391

Kusurkar, R., Croiset, G., & Ten Cate, O. (2011). Twelve tips to stimulate intrinsic motivation in students through autonomy-supportive classroom teaching derived from self-determination theory. Medical Teacher, 33, 978–982. https://doi.org/10.3109/01 42159x.2011.599896

MacNeil, K., Cuncic, C., Voyer, S., Butler, D., & Hatala, R. (2020). Necessary but not sufficient: Identifying conditions for effective feedback during internal medicine residents’ clinical education. Advances in Health Science Education, 25, 641–654. https://doi.org/10.1007/s10459-019-09948-8

Mann, K., Gordon, J., & MacLeod, A. (2009). Reflection and reflective practice in health professions education: A systematic review. Advances in Health Science Education, 14, 595–621. https://doi.org/10.1007/s10459-007-9090-2

Nicol, D., & MacFarlane-Dick, D. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 31(2), 199–218. https://doi.org/10.1080/03075070600572090

Rozental, L., Meitar, D., & Karnielli-Miller, O. (2021). Medical students’ experiences and needs from written reflective journal feedback. Medical Education, 55, 505–517. https://doi.org/10.1111/medu.14406

Sandars, J. (2009). The use of reflection in medical education: AMEE Guide no 44. Medical Teacher, 31, 685–695. https://doi.org/10.1080/01421590903050374

Sargeant, J. M., Mann, K. V., van der Vleuten, C. P., & Metsemakers, J. F. (2009). Reflection: A link between receiving and using assessment feedback. Advances in Health Science Education, 14, 399–410. https://doi.org/10.1007/s10459-008-9124-4

Scott, E. E., Wenderoth, M. P., & Doherty, J. H. (2020). Design-based research: A methodology to extend and enrich biology education research. CBE Life Science Education, 19(3), 1–12. https://doi.org/10.1187/cbe.19-11-0245

Soemantri, D., Mustika, R., & Greviana, N. (2022). Inter-rater reliability of reflective writing assessment in an undergraduate professionalism course in medical education. Education in Medicine Journal, 14(1), 87–97. http://doi.org/10.21315/eimj2022.14.1.8

Suhoyo, Y., Schönrock-Adema, J., Emilia, O., Kuks, J. B. M., & Cohen-Schotanus, J. (2018). Clinical workplace learning: Perceived learning value of individual and group feedback in a collectivistic culture. BMC Medical Education, 18, 79–84. https://doi.org/10.1186/s12909-018-1188-0

Tappan, M. (2010). Sociocultural psychology and caring pedagogy: Exploring Vygotsky’s hidden curriculum. Educational Psychology, 33(1), 23–33. http://doi.org/10.1207/s15326985ep3301_2

Telio, S., Ajjawi, R., & Regehr, G. (2015). The “Educational Alliance” as a framework for reconceptualizing feedback in medical education. Academic Medicine, 90(5), 609–614. https://doi.org/10.1097/acm.0000000000000560

Trehan, A., Barton-Vaness, A., Carty, M. J., McCulloch, P., & Maruthappu, M. (2015). The impact of feedback of intraoperative technical performance in surgery: A systematic review. BMJ Open, 5, Article e006759. http://doi.org/10.1136/bmjopen-2014-006759

Wenger, E. (2020). Communities of practice and social learning systems: The career of a concept. Organization, 7(2), 225–246. http://doi.org/10.1177/135050840072002

Winkel, A. F., Yingling, F., Jones, A. A., & Nicholson, J. (2017). Reflection as a learning tool in graduate medical education: A systematic review. Journal of Graduate Medical Education, 5, 430–439. https://doi.org/10.4300/jgme-d-16-00500.1

*Estivana Felaza
Faculty of Medicine Universitas Indonesia,
Jalan Salemba Raya No 6,
Jakarta 10430, Indonesia
Email: estivanafelaza@gmail.com;
estivana.felaza01@ui.ac.id

Submitted: 13 June 2024
Accepted: 14 October 2024
Published online: 7 January, TAPS 2025, 10(1), 36-47
https://doi.org/10.29060/TAPS.2025-10-1/OA3438

Sivapalan Sanchayan¹, Sisira Dharmaratne², Shalika Pathirana³, Pavithra Godamunne³ & Madawa Chandratilake³

¹Department of Medical Education, Faculty of Medicine, University of Jaffna, Sri Lanka; ²Translational Health Research Institute, Western Sydney University, Australia; Department of Medical Education, Faculty of Medicine, University of Kelaniya, Sri Lanka

Abstract

Introduction: Selection for basic medical training is highly contextual. The use of cognitive aptitude tests, which commonly supplement measures of prior academic achievement in the selection process of medical schools internationally, is rarely reported from resource-constrained settings in South Asia. We report on the psychometric properties of an aptitude test designed based on the UCAT format, administered to first-year medical undergraduates to determine its utility.

Methods: The aptitude test was administered online to first-year medical students shortly after their admission to two medical faculties in Sri Lanka (n=328). The reliability of the test was determined using Cronbach’s alpha. Overall and subtest scores were computed, and the scores of different demographic groups were compared using the t-test. Factor analysis of the subtests was performed.

Results: The internal consistency of the test was 0.63. The difficulty and discrimination indices were within the acceptable range. The mean score of the aptitude test (AT) was 70.9/100 (SD 8.88). The mean score for females was higher than for males (p=0.04). No statistically significant differences in AT scores were observed between different ethnicities or religions. The entry academic scores demonstrated a weak correlation with the overall AT score (Pearson’s correlation coefficient r=0.27), verbal reasoning (r=0.24), the human body (r=0.19), quantitative reasoning (r=0.18), and situational judgement (r=0.128). Factor analysis indicated items that need revision.

Conclusion: The results indicate the potential value of aptitude tests in Sri Lanka and in similar jurisdictions where selection criteria have yet to expand beyond prior academic performance. A way forward in introducing such tests has been outlined.

Keywords:           Medical Student Selection, Medical Student Admission, Selection Criteria

Practice Highlights

• Aptitude tests assess cognitive and non-cognitive attributes desirable for healthcare professionals.
• The use of aptitude tests is rarely reported from resource constrained settings in South Asia.
• Sri Lanka uses only the national high school examination performance for medical school admission.
• This study indicates the potential value of aptitude tests in Sri Lanka and in similar jurisdictions.

I. INTRODUCTION

Selection for basic medical training is highly contextual and influenced by educational, geographical, socioeconomic, and political factors (Dharmaratne & Ponnamperuma, 2020; Patterson et al., 2018). High academic achievement has long been considered an appropriate criterion for ranking candidates for medical school (McManus et al., 2013; Patterson et al., 2017). However, this approach has been contested due to the low predictive validity of pre-university performance for medical school success (McManus et al., 2013; Prideaux et al., 2011).

Aptitude tests have emerged to assess cognitive and non-cognitive traits desirable for medical and healthcare professions (Gliatto et al., 2016; Greatrix & Dowell, 2020; Powis et al., 2021). Despite evidence on their potential to improve selection processes (Bala et al., 2022; Greatrix et al., 2021), most resource-constrained jurisdictions, including Sri Lanka, rely solely on prior academic achievement (Soemantri et al., 2020; University Grants Commission, 2023). This study represents an initial step in Sri Lanka toward introducing aptitude tests and reports on the psychometric properties of a test developed to measure cognitive attributes.

The Sri Lankan medical school admission system uses national high school examination results (G.C.E. AL) as the only criterion for admission (Dharmaratne & Ponnamperuma, 2020; University Grants Commission, 2023). The results of three AL subjects (Chemistry, Physics, and Biology) are considered, and an average Z score is used to rank individuals (University Grants Commission, 2023). A minimum of two credit passes and one simple pass is required for eligibility. A few candidates with foreign examination results equivalent to the G.C.E. AL Examination and meeting equitable minimum requirements are also admitted (University Grants Commission, 2023).

G.C.E. AL examination results have shown a mild correlation with subsequent undergraduate performance, with a maximum predictive correlation coefficient of 0.37 (de Silva et al., 2004, 2006; Hewage et al., 2011; Mettananda et al., 2006). These low figures may underestimate the true relationship, as range restriction among high-achieving candidates can affect the statistical accuracy of predictive validity measures (Zimmermann et al., 2017). Studies in the UK, addressing range restriction, found that prior academic performance is the best predictor of subsequent performance, accounting for 65% of the variance in undergraduate performance (McManus et al., 2013). However, this leaves 35% of the variance unexplained.

Aptitude tests are used worldwide to support medical student selection. The United Kingdom Clinical Aptitude Test (UKCAT), now named UCAT, has been used for nearly two decades (Greatrix & Dowell, 2020). The UCAT consists of 200 items in five subtests delivered over two hours (Paton et al., 2022). Results of the UKCAT and UCAT have been used to discriminate between high-achieving students during selection and predict undergraduate and postgraduate performance (Bala et al., 2022; MacKenzie et al., 2016; McManus et al., 2013; Paton et al., 2022).

However, there are additional qualities beyond academic readiness (prior academic performance and cognitive traits) that are necessary qualities of medical professionals that include desirable behaviours and attitudes that could indicate successful development as healthcare practitioners in the future (Patterson et al., 2018). The Ottawa consensus statement recommends that best practice selection should involve the use of selection criteria based on multi-source multi-method job analysis studies (Patterson et al., 2018). This necessitates the use of other selection processes such as structured interviews or multiple mini interviews (MMI).

This study aims to evaluate the psychometric properties of an aptitude test based on the UCAT format, administered to first-year medical undergraduates at two medical faculties in Sri Lanka. Specifically, the objectives are to evaluate the psychometric properties of the test, determine the relationship between Z scores at the AL examination and aptitude test scores, and compare the performance of different demographic groups in the aptitude test.

II. METHODS

A cross-sectional observational study was conducted in the faculties of medicine at the universities of Kelaniya and Jaffna, Sri Lanka. Including these two faculties allowed for the recruitment of an adequate number of students from the three main ethnic groups (Sinhala, Tamil, and Sri Lankan Moors) and a wider range of Z scores. All first-year students in 2023 from these faculties were invited to participate. Students admitted through selection criteria other than Sri Lankan advanced level performance were excluded. We obtained a sample size of 328 students who consented to participated in the study (response rate of 80.8%).

The aptitude test comprised 50 items across six subtests: abstract reasoning (8 items), the human body (8 items), quantitative reasoning (8 items), decision-making (8 items), verbal reasoning (8 items), and situational judgment (10 items). Except for the “the human body” subtest, all others were modelled after the UKCAT examination. The “human body” subtest was designed to assess advanced-level knowledge of human biology from the Sri Lankan school curriculum. Test items were selected by a panel of experts in psychology and medical education from the Faculty of Medicine, University of Kelaniya with the help of disability studies experts who use a similar test for selection to their course. These experts, being native Sri Lankans, ensured the items were culturally relevant and conceptually and semantically equivalent (Hambleton & Zenisky, 2010). Similar methods were used to finalise the Sinhala and Tamil versions of the test, translated by competent translators. Appendix 1 shows sample test items from each subtest. The test was piloted on 15 students and identified issues were corrected.

The test was administered via the online learning management system (LMS) of the respective faculties under examination conditions, with invigilators ensuring adherence to university examination protocols. Students had one hour to complete the test, available in English, Sinhala, and Tamil. Participants could choose the language they were most comfortable with to simulate AL examination conditions and to minimise interpretation errors.

Descriptive statistics and internal consistency (Cronbach’s alpha) were calculated for the test and each subtest and item analysis (Paniagua & Swygert, 2016) was performed. Correlations between the aptitude test score, subtest scores, and Z scores were calculated using Pearson’s correlation coefficient, and relationships to other demographic factors were analysed using Student’s t-test. Factor analysis was performed using R statistical package.

Written informed consent was obtained from the students before the test administration. Ethics approval was granted by the ethics review committee of the Faculty of Medicine, University of Kelaniya, and permission for the study was obtained from the deans of the respective medical faculties.

III. RESULTS

328 students participated in the study, yielding an overall response rate of 80.8%. The descriptive statistics of these students are presented in Table 1. The male-to-female ratio of the sample (31.7% to 68.3%) closely mirrors the demographics of the national cohort of medical undergraduates from the previous academic year (34% to 66%), as reported by the University Grants Commission, Sri Lanka (University Grants Commission, 2022).

Table 1. Descriptive statistics of the demographic variables of the sample

The overall mean aptitude test (AT) score was 70.9 (SD 8.88), with a median of 72. Female students had a higher mean AT score than male students (p=0.044, Table 2), while there was no statistically significant difference in the Z scores between genders (p=0.122). No significant differences in mean AT scores were found between ethnicities and religions. The Faculty of Medicine, University of Kelaniya, had a higher mean AT score (73.7, SD 8.7) than the Faculty of Medicine, Jaffna (68.8, SD 8.4, p=0.000). Similarly, the mean Z score at Faculty of Medicine, University of Kelaniya (2.10, SD 0.07) was higher than at Jaffna (1.89, SD 0.13, p=0.000).

Table 2. Comparison of means of AT scores and Z scores between males and females

The internal consistency of the test was 0.628 and the subscales ranged between 0.2 to 0.47 (Table 3).

Table 3. Data regarding the subtests of the aptitude test

Difficulty and discrimination indices for each test item were calculated (Table 4). The difficulty index represents the proportion of students who answered each item correctly (Paniagua & Swygert, 2016). The mean difficulty index was 70.9% (SD 23.5), ranging from 19.2% to 99.4%. The discrimination index was calculated as the difference between the proportion of students who answered an item correctly in the top group (those who obtained the top 27% in the overall score) and bottom group (bottom 27% scorers) and measures an item’s ability to differentiate between students who performed best and those who performed worst on the test (Paniagua & Swygert, 2016). The mean discrimination index was 19% (SD 12) and ranged from 0 to 50%. There was only one item with the lowest discrimination index of 0. It had a difficulty index of 99.4% indicating that most students had scored it correctly. Items that adversely affected the reliability of the subtests were identified (Appendix 2).

Table 4. Distribution of the difficulty indices of the aptitude test questions

The correlation analysis revealed several key findings regarding the relationship between Z and AT scores. The correlation coefficient between Z score and AT score was 0.268 (p=0.000), indicating a weak correlation. The coefficient of determination (r²) was calculated to be 0.072, suggesting that only 7.2% of the variability in AT scores could be explained by Z scores. This signifies a limited association between Z scores and overall AT performance. Further analysis was conducted on the correlation between Z scores and subtest scores. There was a weak but statistically significant correlation between Z scores and the subtests of verbal reasoning (r=0.24, p=0.000), human body (r=0.19, p=0.001), quantitative reasoning (r=0.18, p=0.001), and situational judgment (r=0.128, p=0.021). However, the correlations between Z scores and the subtests of abstract reasoning and decision-making were not statistically significant.

Factor analysis was performed using the most prominent two factors of each of the six subtests separately as they each measure well defined and different cognitive domains. The factor loading of abstract reasoning subtest is shown in figure 1. “Question 3” shows a strong positive loading on Factor 1, suggesting that this item heavily measures the trait most represented by this factor whilst “Question 5” shows a slight negative loading on Factor 1, indicating it measures a trait that is perhaps inversely related to what is captured by this factor. “Question 1” and “Question 2” are both located positively on Factor 2 but with minimal loading on Factor 1. This suggests that they are measuring traits more aligned with Factor 2. The items “Question 1”, “Question 2″ and “Question 3”, appear to be critical for measuring distinct aspects of abstract reasoning as shown by their stronger loadings. “Question 5” shows a strong negative loading on Factor 2, suggesting that it measures a trait that contrasts with what Factor 2 represents. “Question 4”, ” Question 6″, “Question 7”, and “Question 8” are clustered around the origin. This indicates that these items have weaker loadings on both factors, meaning they may not strongly measure the traits represented by either factor, or they could be measuring a balanced mix of both traits. While these questions contribute to the test, they might need to be reviewed to enhance their discriminative power or clarity in measuring specific abstract reasoning skills.

Figure 1. Two factor loading of the abstract reasoning subtest

The factor loadings of the other five subtests are shown in Appendix 3. Regarding the human body subtest, Factor 1 might represent knowledge or understanding of structural aspects of the human body, given the strong loading by “Question 11” and reasonable loading by other items. This factor could reflect knowledge regarding human biology whilst Factor 2 seems to capture a different dimension, suggested by the negative loading of “Question 15”. In quantitative reasoning, most questions are clustered near the centre with a slight spread along Factor 1, indicating moderate influence by this factor across the items. “Question 19” and “Question 22” are slightly farther along Factor 1 suggesting a stronger influence by the trait measured by this factor. Regarding the subtests of decision making, verbal reasoning and situational judgment, Factor 1 and Factor 2 seemed to be capturing different dimension of these subtests thereby suggesting the need for revision of their items.

IV. DISCUSSION

This study aimed to design an aptitude test and investigate its psychometric properties among first-year medical students from two selected faculties of Medicine in Sri Lanka. The results indicate that overall internal reliability of the aptitude test was marginal (Cohen et al., 2018). However, the reliability of the UKCAT has been 0.86 or above overall and between 0.58 – 0.86 for the subtests (MacKenzie et al., 2016). These findings demonstrate the value of evaluating and reviewing aptitude tests locally. Further, reviewing test items that adversely affected the internal consistency of the subtests for clarity and improved alignment with the construct, could improve the overall validity. It is likely that the low number of test items has further contributed to the low-reliability score (Taber, 2018). The UKCAT consists of 200 test items, as opposed to the 50 items in the aptitude test employed by the current study. The item analysis indicated that most of the questions were easy, with 60% having a difficulty index of more than 70%. The discriminating capacity of the items was low (mean discrimination index was 19%). This could be attributed to the aptitude test being administered to students who have performed well in the advanced level examination and already have been selected to medical school.

There was a weak correlation between the aptitude test score and the Z score (0.27), indicating that the test assessed a significantly different aspect of cognitive skills compared to traditional AL examinations. This observation encourages exploring how the aptitude test may be incorporated into the selection process. Different selection authorities use aptitude scores differently (Greatrix & Dowell, 2020). Overall, among the universities which employ UKCAT, the weightage given for school leaver level academic performance in the selection process has increased, while the use of aptitude test scores as an absolute or relative cutoff to be achieved also has increased over the period of 2008-2015 (Greatrix & Dowell, 2020). However, empirical evidence on a best way of using UKCAT or UCAT in the selection process is not available (Greatrix & Dowell, 2020). Therefore, implementing a valid aptitude test as a supplementary test and exploring predictive validities of such a test across a national sample is likely to provide more robust evidence of the utility of aptitude tests in the Sri Lankan context.

This study used an aptitude test developed and deployed systematically and modelled on a widely tested international aptitude test format for medical school selection. The sample was representative of demographics of the national cohort of medical students and included all three major ethnic groups and students with a wide range of Z scores. However, the limited number of test items may have offset these strengths. The authors did not have access to participants’ raw scores at the advanced level examination, which may have provided a more appropriate measure of performance for statistical manipulation. In addition, the aptitude test was administered to students who have already been selected to medical school, whereas ideally, it should have been administered to medical school applicants. Despite the limitations, the aptitude test returned an acceptable internal reliability score, and further analysis demonstrated a way forward.

V. CONCLUSION

This study was the first time in the literature that an aptitude test was evaluated in medical education settings in the Sri Lankan context. Using an aptitude test to select students for medical school in Sri Lanka and similar settings, where admission processes have not diversified, is potentially valuable. However, further studies are necessary to establish evidence to support the adoption of such a test in the Sri Lankan and other similar resource-constrained contexts where prior educational achievement primarily informs selection decisions. In the Sri Lankan context, further evaluating the psychometric properties and the predictive validity of an improved aptitude test using a national cohort is recommended as the way forward.

Notes on Contributors

Dr. Sivapalan Sanchayan contributed to the development of the aptitude test, administering the test, collection of data, analysis and writing of the manuscript.

Dr. Sisira Dharmaratne contributed to the conceptualisation, study design, development of the aptitude test, data analysis and writing of the manuscript.

Ms. Shalika Pathirana took part in developing and administering the test, data collection and reviewing the manuscript.

Dr. Pavithra Godamunne contributed to the conceptualisation, study design, development of the aptitude test, data collection, analysis and revision of the manuscript.

Prof. Madawa Chandratilake took part in the conceptualisation, study design, data analysis and revision of the manuscript

Ethical Approval

Ethical clearance (P/223/12/2018) was obtained from the Faculty of Medicine, University of Kelaniya, Sri Lanka.

Data Availability

The datasets generated and/or analysed during the current study are not publicly available to preserve the confidentiality of the participants in this study but are available from the corresponding author on reasonable request.

Acknowledgement

We acknowledge the contribution of Ms. B. N. Jayasena (former lecturer, Department of Disability Studies, Faculty of Medicine, University of Kelaniya) and Ms. L. K. E. Chathurika (former lecturer, Department of Disability Studies, Faculty of Medicine, University of Kelaniya) in the development of the aptitude test. We also acknowledge the contribution of Dr. Rasiah Thayakaran (Research fellow in Health Informatics, Institute of Applied Health Research, University of Birmingham) in performing factor analysis.

Funding

The research received no specific grant from any funding agency.

Declaration of Interest

The authors have no conflicts of interest to disclose.

References

Bala, L., Pedder, S., Sam, A. H., & Brown, C. (2022). Assessing the predictive validity of the UCAT-A systematic review and narrative synthesis. Medical Teacher, 44(4), 401–409. https://doi.org/10.1080/0142159X.2021.1998401

Cohen, L., Manion, L., & Morrison, K. (2018). Research Methods in Education. Routledge.

de Silva, N. R., Pathmeswaran, A., & de Silva, H. J. (2004). Selection of students for admission to a medical school in Sri Lanka. The Ceylon Medical Journal, 49(3), 81–85. https://doi.org/10.4038/cmj.v49i3.3245

de Silva, N. R., Pathmeswaran, A., de Silva, N., Edirisinghe, J. S., Kumarasiri, P. V., Parameswaran, S. V., Seneviratne, R., Warnasuriya, N., & de Silva, H. J. (2006). Admission to medical schools in Sri Lanka: Predictive validity of selection criteria. The Ceylon Medical Journal, 51(1), 17–21. https://doi.org/10.4038/cmj.v51i1.1371

Dharmaratne, S., & Ponnamperuma, G. (2020). Selection for health professions education. In M. Chandratilake & A. Olupeliyawa (Eds.), 14 topics in health professions education (pp. 120–132). College of Medical Educationists, Sri Lanka.

Gliatto, P., Leitman, I. M., & Muller, D. (2016). Scylla and charybdis: The MCAT, USMLE, and degrees of freedom in undergraduate medical education. Academic Medicine, 91(11), 1498–1500. https://journals.lww.com/academicmedicine/Fulltext/ 2016/11000/Scylla_and_Charybdis__The_MCAT,_USMLE,_and_Degrees.16.aspx

Greatrix, R., & Dowell, J. (2020). UKCAT and medical student selection in the UK – what has changed since 2006? BMC Medical Education, 20(1), 1–12. https://doi.org/10.1186/s12909-020-02214-1

Greatrix, R., Nicholson, S., & Anderson, S. (2021). Does the UKCAT predict performance in medical and dental school? A systematic review. BMJ Open, 11, Article e040128. https://doi.org/10.1136/BMJOPEN-2020-040128

Hambleton, R. K., & Zenisky, A. L. (2010). Translating and adapting tests for cross-cultural assessments. In D. Matsumoto, F. J. R. van de Vijver (Eds.), Cross-cultural research methods in psychology (pp. 46–70). Cambridge University Press.

Hewage, S. N., Salgado, L. S. S., Fernando, G. M. O., Liyanage, P., Pathmeswaran, A., & De Silva, N. R. (2011). Selection of medical students in Sri Lanka: Time to re-think criteria? Ceylon Medical Journal, 56(1), 22–28.

MacKenzie, R. K., Cleland, J. A., Ayansina, D., & Nicholson, S. (2016). Does the UKCAT predict performance on exit from medical school? A national cohort study. BMJ Open, 6, Article e011313. https://doi.org/10.1136/BMJOPEN-2016-011313

McManus, I. C., Dewberry, C., Nicholson, S., & Dowell, J. S. (2013). The UKCAT-12 study: Educational attainment, aptitude test performance, demographic and socio-economic contextual factors as predictors of first year outcome in a cross-sectional collaborative study of 12 UK medical schools. BMC Medicine, 11, Article 244. https://doi.org/10.1186/1741-7015-11-244

McManus, I. C., Dewberry, C., Nicholson, S., Dowell, J. S., Woolf, K., & Potts, H. W. W. (2013). Construct-level predictive validity of educational attainment and intellectual aptitude tests in medical student selection: Meta-regression of six UK longitudinal studies. BMC Medicine, 11, Article 243. https://doi.org/10.1186/1741-701 5-11-243

Mettananda, D. S. G., Wickramasinghe, V. P., Kudolugoda Arachchi, J., Lamabadusuriya, S. P., Ajanthan, R., & Kottahachchi, D. (2006). Suitability of selection criteria as a measure of outcome of medical graduates: University of Colombo. The Ceylon Journal of Medical Science 49, 1–12.

Paniagua, M. A., & Swygert, K. A. (2016). Constructing written test questions for the basic and clinical sciences (4th ed.). National Board of Medical Examiners.

Paton, L. W., McManus, I. C., Cheung, K. Y. F., Smith, D. T., & Tiffin, P. A. (2022). Can achievement at medical admission tests predict future performance in postgraduate clinical assessments? A UK-based national cohort study. BMJ Open, 12, Article e056129. https://doi.org/10.1136/bmjopen-2021-056129

Patterson, F., Cousans, F., Edwards, H., Rosselli, A., Nicholson, S., & Wright, B. (2017). The predictive validity of a text-based situational judgment test in undergraduate medical and dental school admissions. Academic Medicine, 92(9), 1250–1253. https://doi.org/10.1097/ACM.0000000000001630

Patterson, F., Roberts, C., Hanson, M. D., Hampe, W., Eva, K., Ponnamperuma, G., Magzoub, M., Tekian, A., & Cleland, J. (2018). 2018 Ottawa consensus statement: Selection and recruitment to the healthcare professions. Medical Teacher, 40(11), 1091–1101. https://doi.org/10.1080/0142159X.2018.149 8589

Powis, D. A., Munro, D., & Bore, M. R. (2021). Is the UCAT appropriate for selecting undergraduate medical students? Medical Journal of Australia, 214(2), 77–78. https://doi.org/10.5694/mja2.50906

Prideaux, D., Roberts, C., Eva, K., Centeno, A., Mccrorie, P., Mcmanus, C., Patterson, F., Powis, D., Tekian, A., & Wilkinson, D. (2011). Assessment for selection for the health care professions and specialty training: Consensus statement and recommendations from the Ottawa 2010 conference. Medical Teacher, 33(3), 215–223. https://doi.org/10.3109/0142159X.2011.551560

Soemantri, D., Karunathilake, I., Yang, J. H., Chang, S. C., Lin, C. H., Nadarajah, V. D., Nishigori, H., Samarasekera, D. D., Lee, S. S., Tanchoco, L. R., & Ponnamperuma, G. (2020). Admission policies and methods at crossroads: A review of medical school admission policies and methods in seven Asian countries. Korean Journal of Medical Education, 32(3), 243–256. https://doi.org/10.3946/KJME.2020.169

Taber, K. S. (2018). The use of Cronbach’s Alpha when developing and reporting research instruments in science education. Research in Science Education, 48(6), 1273–1296. https://doi.org/10.1007/ s11165-016-9602-2

University Grants Commission. (2022). Sri Lanka university statistics 2022. Ministry of Higher Education. https://www.ugc.ac.lk/downloads/statistics/stat_2022/Chapter%202.pdf

University Grants Commission. (2023). Admission to undergraduate courses of the universities in Sri Lanka – 2022/2023. Ministry of Higher Education. https://www.ugc.ac.lk/downloads/admissions/Handbook_2022_23/student_handbook_english.pdf

Zimmermann, S., Klusmann, D., & Hampe, W. (2017). Correcting the predictive validity of a selection test for the effect of indirect range restriction. BMC Medical Education, 17, Article 246. https://doi.org/10.1186/s12909-017-1070-5

*S. Sanchayan
Faculty of Medicine,
University of Jaffna,
Adiyapatham Road,
Kokuvil West, Kokuvil, Sri Lanka
Email: sanch.s6@gmail.com,
sanchayan@univ.jfn.ac.lk

Submitted: 15 November 2023
Accepted: 1 August 2024
Published online: 7 January, TAPS 2025, 10(1), 17-26
https://doi.org/10.29060/TAPS.2025-10-1/OA3146

Eugene Kua¹, Sinead Kado⁴, Valsan Philip Verghese⁵, Pauline Wake⁶ & Amy Gray^1,2,3

¹Murdoch Children’s Research Institute, The Royal Children’s Hospital, Melbourne, Australia; ²The Department of Paediatrics, Melbourne Medical School, University of Melbourne, Melbourne, Australia; ³The Royal Children’s Hospital Melbourne, Parkville, Victoria, Australia; ⁴Division of Health Professions Education, School of Allied Health, University of Western Australia, Perth, Western Australia; ⁵Department of Paediatric Infectious Diseases, Christian Medical College, Vellore, India; ⁶School of Medicine and Health Sciences, University of Papua New Guinea, National Capital District, Port Moresby, Papua New Guinea

Abstract

Introduction: COVID-19 pandemic disruptions to medical educational programs have been felt worldwide. However, little is understood about the experience of Low to Middle Income Countries’ (LMIC) and the impact of ensuing disruptions on medical education. We aimed to develop a collective understanding of this through the perspectives of individual LMIC educators, exploring how they continued to deliver undergraduate and postgraduate medical education in the context of pandemic barriers, by reflecting on their successes and shortcomings.

Methods: A phenomenological study was conducted with ten purposefully recruited educators involved in medical education from mainly the Asia-Pacific region. In-depth interviews via web-conferencing were conducted, and data analysed using Braun & Clarke’s (2006) framework. 

Results: Four major themes were identified: ‘Navigating Change, ‘Altered and Divided Teaching Roles and Spaces’, ‘Curricula Impact’ and ‘Challenges and Affordances of Remote Learning’. Educators described major disruptions to delivering medical education, particularly clinical education. Remote learning was the mainstay of facilitating continued education, often requiring investment into infrastructure that was non-existent pre-pandemic.

Conclusion: LMIC educators’ pandemic response have been robust in pursuit of continued medical education. Necessary adoption of online teaching approaches has provided avenues to close healthcare education gaps by facilitating improved reach and quality of medical education in these regions. Building on this requires strategies that meet LMICs areas of need, focusing on capacity building that sustains and grows new pedagogical practices.  We must ensure educational advances are equitable and accessible for all, identifying and supporting countries and contexts at risk of being left behind.

Keywords:           Low to Middle Income Countries, COVID-19, Medical Education, Remote Learning, Qualitative

Practice Highlights

• Educational ramifications from COVID-19 were most acutely discernible within the clinical domain.
• LMICs demonstrated resilient responses, despite their comparatively limited resource availability.
• The pandemic necessitated remote learning and heralded investment into infrastructure.
• Remote learning may provide a means of narrowing health disparities.
• COVID-19 practices of remote learning should be leveraged and supported in LMICs.

I. INTRODUCTION

Declared a global pandemic by the World Health Organisation in March 2020, COVID-19 has had significant global influences, impacting the social fabric and operations of everyday living (Cucinotta & Vanelli, 2020). Governments and institutions were required to re-examine and enforce changes to social procedures in light of the need for infection control, whilst also finding ways to minimise the negative impacts of disruptions to healthcare provision and medical education (Kaul et al., 2021).

The United Nations Secretary-General warned that COVID-19 disruptions to education threaten to cause a ‘generational catastrophe’, widening existing inequalities and stifling what established and evolving progress has been made (UNESCO, 2020). Medical education was not exempt. Indeed, in a global survey of 424 universities and higher education institutes by the International Association of Universities in early 2020, 59% had ceased campus activities and closed, with indications that LMICs were being hit harder, struggling and lagging in their responses and abilities to adapt compared to high income countries (HIC) counterparts (Marinoni et al., 2020).

Adaptations, challenges and impacts to medical education during the pandemic are well described in HICs (Dedeilia et al., 2020; Gallagher & Schleyer, 2020; Gill et al., 2020; Kachra & Brown, 2020; Kaul et al., 2021; Rose, 2020). LMIC narratives in comparison have been relatively isolated, and often represented by single institutions (Adesunkanmi et al., 2021; Aslan & Sayek, 2020; Cecilio-Fernandes et al., 2020; Fiorillo & Javed, 2021; Kalayasiri & Wainipitapong, 2021; Kanmounye & Esene, 2020; Sahi et al., 2020; Sandal et al., 2021; Tokuç & Varol, 2020). Collectively, what has happened to medical education in LMICs, particularly in contexts with fever resources, already stretched healthcare systems and concerns for slower transitions (Aishat et al., 2020; Cecilio-Fernandes et al., 2020; Gill et al., 2020; Sandal et al., 2021)?

This study aimed to answer this question by highlighting the experiences of educators in LMICs during the COVID-19 pandemic, through distilling common themes across various contexts. Through understanding their narratives of successes and shortcomings, we hope to advocate for future needs and highlight opportunities.

II. METHODS

A. Study Design

We conducted a qualitative study using a phenomenological approach involving semi-structured interviews to evaluate the ‘lived experience’ of medical educators in LMICs as they aimed to change, adapt and sustain the delivery of medical education during the pandemic. Interviews were conducted between 7^th July and 21^st October 2021. We sought to capture a range of contexts rather than an in-depth experience of a single site.

B. Inclusion and Exclusion

LMIC educators involved in undergraduate and postgraduate medical programs were the target population. We included educators from tertiary (pre-vocational) and clinical (post-graduate) settings in primary roles of teaching, curriculum development and education oversight. We defined LMIC using the economic bands of low, lower-middle or upper-middle as described by The World Bank for the financial years of 2020 and 2021(World Bank Group, 2021).

C. Data Collection and Analysis

Participants were identified using existing professional networks through purposeful (Côté & Turgeon, 2005; Giacomini & Cook, 2000) and convenience sampling (Etikan et al., 2016). Subsequently, snowball sampling was used whereby existing participants identified future participants from their own professional network. Recruitment was ongoing during the analysis phase until data saturation was achieved. Data saturation was determined through continuous analysis of data and iterative development and review of themes until no new concepts were interpreted.

Invitations to participate were sent via email including a written description of the research goals, study design, and a consent form. Ten consenting participants were scheduled for an interview at a time convenient to them. 

Consistent with phenomenological research (Van Manen, 2016), data was collected via in-depth semi-structured interviews.  Individual interviews were conducted using a guide comprising open ended questions in combination with prompts to ensure consistency of scope while maintaining flexibility to explore experiences raised by participants (Table 1). Key areas of focus included the impacts of the pandemic on medical education at undergraduate and postgraduate levels, strategies used to adapt education, their outcomes and the factors influencing success, and future opportunities.

All interviews were conducted via a password-protected, video-conferencing platform (Zoom, https://zoom.us/) by the principal investigator (EK) who had no pre-existing relationships with participants. Video interviews were conducted and audio-recorded for verbatim transcription. Initial impressions were recorded in the researcher’s journal during or immediately after the interview. Participants were given the opportunity to review written copies of their transcripts on request. There were no withdrawals or repeated interviews.

Thematic analysis of data drew upon both essentialist and constructionist approaches described by Braun and Clarke (Braun & Clarke, 2006), illustrating the lived realities whilst also exploring the impact of the COVID-19 pandemic on institutional operations and subsequent educational outcomes. Interview data was coded by the principal investigator and subsequently reviewed by the chief investigator (AG) to enhance reliability and reduce biases. Theme generation occurred using an inductive approach whereby theoretical concepts were derived from data without a pre-existing framework (Braun & Clarke, 2006). Themes were developed, analysed and refined through iterative consensus meetings between the principal and chief investigators. A selection of interviewees (PW, VV, SK) were subsequently invited to peer review findings to ensure their voices were represented. To ensure trustworthiness of findings, actions taken to address credibility, dependability, confirmability, transferability and reflexivity are outlined in Table 2.

Table 1. Question guide

Table 2. Ensuring trustworthiness

III. RESULTS

A total of 10 educators participated, representing perspectives from eight countries: Vietnam, India, Indonesia, Fiji, Solomon Island, Papua New Guinea, People’s Democratic Republic (PDR) of Lao and Nigeria. Educators had backgrounds in paediatrics, community medicine, internal medicine, anaesthesiology, public health/hospital management and educational leadership.

Four major themes were identified. These were ‘Navigating Change’, ‘Altered and Divided Teaching Roles and Spaces’, ‘Curricular Impact’ and ‘Challenges and Affordances of Remote Learning’.

A. Navigating Change

Educators described operating in a volatile landscape, where the fluctuations of the COVID-19 pandemic forced a dynamic interplay between health policy, health systems and educational institutions. Efforts to control infection rates meant lockdowns, mask wearing, travel restrictions, prevention of mass gatherings, curfews and social distancing precautions were universal experiences. Geographical variability existed in responses within countries to local COVID-19 severity, resulting in a heterogenous distribution of restrictions, and impacts on education.

“Because they [other Pacific Island nations] shut their borders so early, I think most of them have carried on as normal… So don’t think it’s been such a big impact for the other smaller Pacific Islands.”

(Fiji)

Frequent changes in hospital policy, operations and organisational structure meant educators and learners were operating in dynamic environments, at the mercy of redirection of workforce redistributions and space divisions. The dependence of medical curriculum on healthcare environments meant changes invariably influenced the provision of education, with clinical learning hit hardest. How this occurred varied between LMICs over time waves of the pandemic.

“So what happened is you have less number of inpatients, even the consults that we got called to see reduced… And now our (COVID-19) numbers have started coming down so we are going back to normal. So it’s like a cycle as far as numbers and clinical teaching is concerned.”

(India)

B. Altered and Divided Teaching Roles and Spaces

Educators described periods of restricted access to university campuses. Continuing face-to-face teaching was difficult, with examples of existing architectural spaces poorly adaptable to COVID-19-safe measures.

“…ventilation is not very good…it’s not set up for social distancing. Students are backed up in a room …20 centimetres distance between each other.”

(Indonesia)

Clinically, most educators described a spectrum of hospital access, from periods marked by complete cancelation of bedside teaching, to limited access necessitating modified clinical activities, reduced time and crowd limitations.

The move towards remote learning resulted in the delivery of education from home spaces. Here educators described major challenges regarding infrastructure. Successful initiatives included faculty-initiated needs surveys with subsequent provision of required infrastructure for the home. Alternatively, some universities provided spaces fully equipped with remote teaching equipment for use.

 “…we have a survey to all of the lecturers, what they need so that they can give the lecture from their home.”

(Indonesia)

Social distancing necessitated investment into new infrastructure. In PNG, educators described procuring screens to facilitate telehealth to aid medical practice both internally within hospitals and remotely with peripheral healthcare centres. Such technology would then be used for enhancing educational experiences.

“So we now have screens in all the classrooms where we can connect directly to internet ….So I can honestly tell you, we used to ask for it, it never happened. And when COVID-19 came it forced them to make it happen and we’re happy.”

(Papua New Guinea)

For educators with clinical responsibilities, pandemic medicine pulled human resources away from education, driven by re-deployment and personal illness. Capacity was impacted further by fatigue and psychological stress. Educators reported the absence of international trainees normally present to buffer workforce. However, educators employed by universities felt at times underused on the clinical side.

 “The teaching is still ongoing, but it’s not regular …for example, for the Department of Infectious Disease, the teacher has to take responsibility of the COVID-19 ward. So at the time bedside teaching is not that easy to do.”

(Lao PDR)

C. Curricular Impact: Deficiencies in Clinical Content, Assessment and Trusting the New Workforce, and Compensation in Delivery

1) Deficiencies in clinical content: Theoretical content lent itself well to remote learning approaches and was delivered without much disruption to academic calendars. In contrast, clinical education, dependent on availability of clinical spaces and patients, suffered and often occurred over reduced time frames in settings where the content landscape was different and deficient.

“So by the end of January, we called the students in for clinics, and they had their paediatric posting …. I think we had to cram everything into a period of approximately four weeks …it was a struggle to make sure that we had those cases.”

(India)

Cancellation of elective surgical cases resulted in a heavy skew towards emergency cases. Common medical presentations became rare and patient numbers shrank due to travel restrictions and reduced health-seeking behaviour, fostering both clerical and procedure poor environments. At both undergraduate and post-graduate levels, educators lamented this loss of clinical breadth, opportunities and authentic workplace experiences normally afforded to learners. Many expressed concerns about the effects on job-preparedness and skill acquisition of new graduates and those in specialty training.

“…my students couldn’t see like severe malnutrition or TB with spondylosis. Actually, we still have a lot, but I don’t know where these patients are! They disappeared!”

(Indonesia)

“For the current batch of interns, if someone says there’s no spleen, no liver palpable, you don’t trust. You know, it’s quite possible that they’ve missed it completely, because the only exposure they’ve had is anywhere between two to four weeks…”

(India)

The COVID learning environment was not without its own merits. Educators described those at the forefront of COVID-19 care experiencing increased learning opportunities and greater depth of learning via involvement and active management of cases on the clinical floor. Furthermore, it was also viewed by some as a valuable experience for future pandemic preparedness.

“The pandemic will teach the students what we have never taught …they become better clinicians than us when they are exposed to the similar kinds of situations in the future.”

(Papua New Guinea)

2) Assessment and trusting the new workforce: The aforementioned impacts of COVID-19 had flow-on effects on assessments resulting in postponement, modification, replacement by other assessment modalities or cancellation. Irrespective of this, ensuring a continued cycle of new graduates or subspecialists was paramount, particularly in areas of need both locally and regionally, and meant that regardless of the quality and quantity of education learners received, transitions (e.g., undergraduate to postgraduate, postgraduate to subspeciality) were pushed through. Whilst this ensured continuity of workforce, educators voiced concerns regarding clinical competencies of COVID-era cohorts.

“…we can’t delay graduation because we need interns. We need the workforce. …if we don’t conduct exams and we don’t take the next batch … after three years, we actually have nobody.”

(India)

3) Compensation in delivery: Endeavoring to minimise faced shortcomings, educators compensated by trialing and implementing various low to high resource strategies. Delivering education by aligning judicious use of space, crowd management, session timetabling and utilising windows of ‘normality’ was commonplace. Collaborative interdisciplinary relationships were fostered to account for reduced patient pools available for clinical learning/interactions and simulation was used in a couple of settings to make up for the paucity in procedural opportunities.

“We were sharing cases between units. If we had one heart disease, that child would have three clinics in a day by three different sets of people.”

(India)

Evaluating risk of disease versus the benefits of learning, vaccinated students in Papua New Guinea (PNG) who consented were able to continue clinical clerkship schedules with a degree of normality. Recognising a need for increased senior supervision of the post COVID-era workforce, periods of extended internship were also discussed allowing for a longer period of supervised upskilling.

“So what we’ve said to the other clinicians is that don’t worry, everyone’s in the same boat after COVID-19. So the Ministry of Health has said, okay, we’re going to make it a two year internship now, so that they get bit more supervision.”

(Fiji)

D) Challenges and Affordances of Remote Learning

1) Challenges: Educators described limited pre- COVID-19 experiences with remote learning, with little to no incorporation of online teaching in medical education curricula. Furthermore, pedagogical knowledge was limited, with educators often learning how to teach via process of trial and error and little feedback. Time to self-educate was identified as a limiting factor. For those who received support, utility was mixed from helpful to lacking expert guidance.

“…we don’t get any training, on how to use the IT and the tools that can help us teach. This is something that I had to learn on my own, to figure out how to do. So I guess that’s the other barrier, not having that support to help with the students’ virtual learning.”

(Fiji)

Educators describe initial apprehension amongst colleagues with regards to online education, rooted in discomforts around change and pitted against the inertia of entrenched concepts of what constitutes proper teaching. Many had persistent concerns about quality of education provided by online methods and expressed preferences for face-to-face learning – ‘…nothing is equal to the human touch’. Common challenges included limited ability to interact with students, gauge understanding and assess competency. Video cameras were often off due to challenges with bandwidth, and lecture slides were commonly viewed on small smartphone screens.

“But I’ve definitely got the negative people, you know, the cohort of the negatives who decide this is just too hard, you shouldn’t be teaching online …they keep quoting to me Osler …‘you can’t learn medicine without books, but neither can you learn just from the books, you need to have some patient contact.’”

(Fiji)

Despite most learners having some sort of device that could facilitate e-learning, internet reliability was a constraint, alongside prohibitive costs of required data. Acknowledging existing socioeconomic inequities affecting access, provisions for infrastructure, data and reliable connectivity were provided by some educational institutes for learners and teachers.

2) Affordances: For educators, the ability to rehearse and edit lectures was felt to improve the quality of teaching. Many appreciated flexibilities in scheduling and abilities to extend educational sessions beyond the limitations of time and place normally associated with face-to-face teaching. Increased geographical reach provided opportunities for individuals and communities, particularly those limited by location, to access education and expertise previously not available. Accessibility to both domestic and international expertise also helped improve learner engagement.

“… technology has become something that’s sort of changed in our practice …and still plays a vital role in in medical education in our country. And we are now able to reach people we’ve never reached before; we could not reach before. And that’s a good thing.”

(Papua New Guinea)

The ability to bridge international borders provided further opportunities to support and improve programs of domestic education and assessment through international collaboration. Through existing professional relationships, the anaesthetics department in PNG was able to organise fortnightly online teaching sessions to assist with the burden of teaching, as well as invite subspecialty specialists via videoconference to help examine their trainee candidates in viva assessment exams.

IV. DISCUSSION

This qualitative study investigated the impact of COVID-19 on medical education in LMICs from the perspective of medical educators who described how the pandemic transformed their educational landscapes. Curricular impacts were felt most in the clinical domain, and educators conveyed concerns regarding the quality of education during this time and what this meant for the capabilities of future healthcare workers and service provision. In the face of social distancing and contextual barriers unique to LMIC settings, educators looked towards remote learning as a largely successful bridging solution despite some difficulties in establishing and supporting this approach. In recognising remote learning affordances as we emerge from the pandemic, positive changes can be and have been made – how we continue to best support this is currently unclear.

Medical educators early on anticipated challenges related to educational continuity during COVID-19 (Rose, 2020), with narratives from study participants reflecting both the projected impacts and consequential experiences reported by other individual LMICs (Adesunkanmi et al., 2021; Aslan & Sayek, 2020; Cecilio-Fernandes et al., 2020; Connolly & Abdalla, 2022; Fiorillo & Javed, 2021; Kalayasiri & Wainipitapong, 2021; Kanmounye & Esene, 2020; Sahi et al., 2020; Sandal et al., 2021; Tokuç & Varol, 2020). Our study adds by highlighting baseline gaps in pedagogical knowledge of more modern educational techniques, exemplified by the struggles encountered during implementation of remote learning. Where needs for educator instruction were recognised, the extent to which these occurred was heterogeneous, relying on funding and available expertise, connections or partnerships.

Educators without existing collaborations were and continue to be relatively isolated in educational endeavours, whereas those with existing partnerships were enabled to capitalise on opportunity. Collaborations between LMIC educators are powerful initiating tools for grassroots change and for local educators to play leading roles in the development and sustainment of their own competencies and capacities (Al Shorbaji et al., 2015).  Common challenges faced by educators are best addressed by connecting, sharing and learning from each other. This can be facilitated by prioritising ongoing investment in communities of practice that ensures sustainability through fostering continued learning, growth and independence (Ramani et al., 2020). We found that despite societal schisms engendered by the pandemic, proximity among individuals and communities traditionally separated have perhaps closed owing to the pronounced emphasis on online interactions, enabling collaborations that otherwise may not have occurred.

Whilst it would be safe to assume that barriers introduced by pandemics are not general considerations in curriculum design, the COVID-19 pandemic has highlighted how deficiencies in the use of modern educational practices and the consequent severer interruptions to medical education in LMICs compared to HICs, have the potential to widen existing gaps in health outcomes (Chahine et al., 2018; Dattani et al., 2013; Hunter & Reddy, 2013; Marinoni et al., 2020; Mossialos et al.; Roser & Ritchie, 2013). The compensations and affordances – the success stories – described here by LMIC educators serve to highlight how familiarity with technology-assisted education can safeguard education particularly in times of social disorder.

COVID-19 crisis has thus also been a transformative moment for education in LMICs, particularly with the widespread adoption of remote learning, previously under-exploited in many regions (Al Shorbaji et al., 2015). Where the benefits of remote learning have been well described – cost effectiveness, accessibility, portability, coverage, scalability, adaptability and diversity (Al Shorbaji et al., 2015; Barteit et al., 2020; Papapanou et al., 2022) – its potential for ‘revolutionary’ change for medical education in LMICs depends on how changes are taken forward beyond the pandemic. There is an opportunity to close pre- and post-COVID-19 educational and health gaps through the affordances of remote learning via improving educational equity in regions where access to ongoing professional development and education programs have been limited both in quantity and quality (Barteit et al., 2020; Mack et al., 2017). In doing so we must consider context-specific strategies that foster fidelity, adaptability, longevity and accessibility (McLean et al., 2017).

Momentum is key.  We know that attitudes in the broader literature about the longevity of remote learning practices post-COVID 19 are mixed (Adesunkanmi et al., 2021; Motte-Signoret et al., 2021). Indeed, gaps in technology-aided education could widen as countries where remote learning was done well continue to consolidate and innovate such approaches, while those who struggled to adapt may revert to more traditional practices. Nevertheless, there is room for optimism as educators in our study, particularly those of the younger generation, were enthusiastic about the place of remote learning in the future of medical curricula. How then do we support these individuals in capitalising, championing, and sustaining forward momentum? We provide six considerations.

A. Invest in Technology Infrastructure

Prioritise investing in technology infrastructure to ensure reliable and equitable access to remote learning resources. This includes internet connectivity, hardware and spaces, and allocating funds to support remote learning initiatives.

B. Pedagogical Training

Pedagogical training and support that meets educators in LMICs to upskill and adapt teaching methods for remote learning practice, emphasising ongoing efforts to enhance local expertise.

C. Innovating Curriculum

Develop long-term strategies for integrating technology-assisted education into curricula. Remote learning should not be seen as a temporary solution but as a valuable tool for enhancing educational quality, accessibility and ultimately quality healthcare provision.

D. Foster Collaboration

Encourage collaboration and sharing between LMICs, and with HICs on knowledge, resources, and best practices in education. These partnerships can help bridge knowledge gaps and provide mentorship opportunities that link experienced educators with younger generations to champion change.

E. Communities of Practice

Development of communities of practice among educators particularly among LMIC counterparts. These groups can serve as platforms for sharing experiences, challenges, and innovative solutions, thus promoting ongoing learning and growth, which in turn promotes self-sustainability, local expertise and resourcefulness.

F. Stakeholder Engagement

Engage stakeholders at individual, institutional, national, and international levels to commit to and support remote learning initiatives. Collaboration between governments, educational institutions, and non-governmental organisations is essential.

This study has limitations. Whilst our aim is to provide an initial window of understanding into the collective experience of LMICs, we acknowledge that this study is non-exhaustive in capturing all variations in experience. Educators were recruited via convenience, purposeful and snowballing sampling resulting in an Asia-Pacific sample bias. Furthermore, we appreciate that not all medical educational institutes nor specialty departments are necessarily equal in resources and varied experiences may exist domestically. However, the key themes we identified were broad, recurred across all interviews and may be applicable to other LMIC jurisdictions through reader generalisability, that is, where themes resonate with the others. Another limitation was that student perspectives were not included our study. We acknowledge that educational experiences exist in duality between educator and learner and that understanding both will be paramount in developing high fidelity educational strategies (Kachra & Brown, 2020; Zaharias & Poylymenakou, 2009).

V. CONCLUSION

This study gives insight into the degree and scope of disruptions the COVID-19 pandemic had on the medical education landscape and the adaptive changes to medical educational systems implemented by educators in LMIC settings. Despite reduced capacity and infrastructure, LMICs demonstrated resourcefulness and robustness in continuing medical education in a time of health crisis. Our study brings to light the enthusiasm and commitment educators in LMICs have in improving the education that they provide.  With the quality of education intrinsically tied to health outcomes there remains a risk that countries that did not succeed in embedding online learning and teaching approaches will now fall behind on the educational landscape.  There is not just an opportunity but an onus to build on educational progress triggered by COVID-19, particularly that of remote learning methods, and ensure it is grown and sustained, but at the same time identify and assist those at risk of falling behind. 

Notes on Contributors

Dr. Eugene Kua contributed to study design, interviewing of study participants, coding of data, subsequent thematic analysis and paper write up.

Dr. Sinead Kado contributed by means data analysis through peer review, with supplementary contributions in the form of additional content enhancements to the paper.

Prof. Valsan Philip Verghese contributed by means data analysis through peer review, with supplementary contributions in the form of additional content enhancements to the paper.

Dr. Pauline Wake contributed by means data analysis through peer review, with supplementary contributions in the form of additional content enhancements to the paper.

A/Prof. Amy Gray contributed to study design, subsequent thematic analysis and paper write up and review.

Ethical Approval

This project was approved by The Royal Children’s Hospital Melbourne Human Research Ethics Committee (reference number 75258).

Data Availability

No consent was given to share transcript data.

Funding

No funding was provided in conducting this research.

Declaration of Interest

There is no competing interest to be declared.

References

Adesunkanmi, A. O., Ubom, A. E., Olasehinde, O., Wuraola, F. O., Ijarotimi, O. A., Okon, N. E., Ikimalo, J. I., Fasubaa, O. B., & Adesunkanmi, A. R. K. (2021). Impact of the COVID-19 pandemic on surgical residency training: Perspective from a low-middle income country. World Journal of Surgery, 45(1), 10-17. https://doi.org/10.1007/s00268-020-05826-2

Aishat, T. O., Oluwayemisi, T. F., Eniola, I. A., Omolola, O. A., Oluwatobiloba, D. O., & Adeseye, M. A. (2020). Challenges with medical education in Nigeria in the COVID-19 era. Pan African Medical Journal, 37(223). https://doi.org/10.11604/pamj.2020.37.223.26418

Al Shorbaji, N., Atun, R., Car, J., Majeed, A., Wheeler, E. L., Beck, D., Belisario, J. M., Cotič, Ž., George, P. P., & Hirvonen, H. (2015). eLearning for undergraduate health professional education: A systematic review informing a radical transformation of health workforce development. World Health Organisation.

Aslan, D., & Sayek, İ. (2020). We need to rethink on medical education for pandemic preparedness: lessons learnt from COVID-19. Balkan Medical Journal, 37(4), 178-179. https://doi.org/10.4274/balkanmedj.galenos.2020.2020.4.002  

Barteit, S., Guzek, D., Jahn, A., Bärnighausen, T., Jorge, M. M., & Neuhann, F. (2020). Evaluation of e-learning for medical education in low- and middle-income countries: A systematic review. Computers & Education, 145, Article 103726. https://doi.org/10.1016/j.compedu.2019.103726  

Braun, V., & Clarke, V. (2006). Using thematic analysis in pyschology. Qualitative Health Research in Psychology, 3(2), 77-101. https://doi.org/10.1191/1478088706qp063oa  

Cecilio-Fernandes, D., Parisi, M., Santos, T., & Sandars, J. (2020). The COVID-19 pandemic and the challenge of using technology for medical education in low and middle income countries [version 1]. MedEdPublish, 9(74). https://doi.org/10.15694/mep.2020.000074.1

Chahine, S., Kulasegaram, K., Wright, S., Monteiro, S., Grierson, L. E. M., Barber, C., Sebok-Syer, S. S., McConnell, M., Yen, W., De Champlain, A., & Touchie, C. (2018). A call to investigate the relationship between education and health outcomes using big data. Academic Medicine, 93(6), 829-832. https://doi.org/10.1097/acm.0000000000002217

Connolly, N., & Abdalla, M. E. (2022). Impact of COVID-19 on medical education in different income countries: a scoping review of the literature. Medical Education Online, 27(1), 2040192. https://doi.org/10.1080/10872981.2022.2040192  

Côté, L., & Turgeon, J. (2005). Appraising qualitative research articles in medicine and medical education. Medical Teacher, 27(1), 71-75. https://doi.org/10.1080/01421590400016308

Cucinotta, D., & Vanelli, M. (2020). WHO declares COVID-19 a pandemic. Acta bio-medica : Atenei Parmensis, 91(1), 157-160. https://doi.org/10.23750/abm.v91i1.9397  

Dattani, S., Spooner, F., Roser, M., & Ritchie, H. (2013). Child and Infant Mortality. Our World in Data. Retrieved January 9, 2023, from https://ourworldindata.org/child-mortality [Online Resource]

Dedeilia, A., Sotiropoulos, M. G., Hanrahan, J. G., Janga, D., Dedeilias, P., & Sideris, M. (2020). Medical and surgical education challenges and innovations in the COVID-19 era: a systematic review. In Vivo, 34(3 Suppl), 1603-1611. https://doi.org/10.21873/invivo.11950

Etikan, I., Musa, S. A., & Alkassim, R. S. (2016). Comparison of convenience sampling and purposive sampling. American Journal of Theoretical and Applied Statistics, 5(1), 1-4.

Fiorillo, A., & Javed, A. (2021). Education and training in psychiatry in low‐ and middle‐income countries during and after the COVID‐19 pandemic. Asia Pacific Psychiatry, 13(4), Article e12500. https://doi.org/10.1111/appy.12500  

Gallagher, T. H., & Schleyer, A. M. (2020). “We signed up for this!” — student and trainee responses to the Covid-19 pandemic. New England Journal of Medicine, 382(25), Article e96. https://doi.org/10.1056/NEJMp2005234  

Giacomini, M. K., & Cook, D. J. (2000). Users’ guides to the medical literature: XXIII. Qualitative research in health care A. Are the results of the study valid? Evidence-Based Medicine Working Group. Journal of the American Medical Association, 284(3), 357-362. https://doi.org/10.1001/jama.284.3.357  

Gill, D., Whitehead, C., & Wondimagegn, D. (2020). Challenges to medical education at a time of physical distancing. Lancet, 396(10244), 77-79. https://doi.org/10.1016/S0140-6736(20)31368-4

Hunter, D. J., & Reddy, K. S. (2013). Noncommunicable diseases. New England Journal of Medicine, 369(14), 1336-1343. https://doi.org/10.1056/NEJMra1109345  

Kachra, R., & Brown, A. (2020). The new normal: Medical education during and beyond the COVID-19 pandemic. Canadian Medical Education Journal, 11(6), e167-e169. https://doi.org/10.36834/cmej.70317  

Kalayasiri, R., & Wainipitapong, S. (2021). Training of psychiatry and mental health in a low- and middle-income country: Experience from Thailand before and after COVID-19 outbreak. Asia-Pacific Psychiatry, 13(4), Article e12493. https://doi.org/10.1111/appy.12493  

Kanmounye, U. S., & Esene, I. (2020). COVID-19 and neurosurgical education in Africa: Making lemonade from lemons. World Neurosurgery, 139, 732-733. https://doi.org/10.1016/j.wneu.2020.05.126

Kaul, V., Gallo de Moraes, A., Khateeb, D., Greenstein, Y., Winter, G., Chae, J., Stewart, N. H., Qadir, N., & Dangayach, N. S. (2021). Medical education during the COVID-19 pandemic. Chest, 159(5), 1949-1960. https://doi.org/10.1016/j.chest.2020.12.026

Mack, H. G., Golnik, K. C., Murray, N., & Filipe, H. P. (2017). Models for implementing continuing professional development programs in low-resource countries. MedEdPublish, 6(1).

Marinoni, G., Van’t Land, H., & Jensen, T. (2020). The impact of COVID-19 on higher education around the world. IAU global survey report, 23, 1-17.

McLean, M., McKimm, J., & Gibbs, T. (2017). Medical education in difficult circumstances: A global responsibility to contribute. Medical Teacher, 39(1), 4-6.

Motte-Signoret, E., Labbé, A., Benoist, G., Linglart, A., Gajdos, V., & Lapillonne, A. (2021). Perception of medical education by learners and teachers during the COVID-19 pandemic: A cross-sectional survey of online teaching. Medical Education Online, 26(1), 1919042. https://doi.org/10.1080/10872981.2021.1919042  

Papapanou, M., Routsi, E., Tsamakis, K., Fotis, L., Marinos, G., Lidoriki, I., Karamanou, M., Papaioannou, T. G., Tsiptsios, D., Smyrnis, N., Rizos, E., & Schizas, D. (2022). Medical education challenges and innovations during COVID-19 pandemic. Postgraduate Medical Journal, 98(1159), 321. https://doi.org/10.1136/postgradmedj-2021-140032

Ramani, S., McKimm, J., Thampy, H., O’Sullivan, P. S., Rogers, G. D., Turner, T. L., Chisolm, M. S., Kusurkar, R. A., Hays, R., Fornari, A., Kachur, E. K., Wilson, K. W., Filipe, H. P., & Schumacher, D. J. (2020). From clinical educators to educational scholars and leaders: Strategies for developing and advancing a career in health professions education. The clinical teacher, 17(5), 477-482.

Rose, S. (2020). Medical student education in the time of COVID-19. JAMA, 323(21), 2131-2132. https://doi.org/10.1001/jama.2020.5227

Sahi, P. K., Mishra, D., & Singh, T. (2020). Medical education Amid the COVID-19 pandemic. Indian Pediatrics, 57(7), 652-657. https://doi.org/10.1007/s13312-020-1894-7  

Sandal, S., Boyarsky, B. J., & Cantarovich, M. (2021). The higher impact of the COVID‐19 pandemic on resident/fellow training in low‐ and middle‐income countries. Transplant International, 34(5), 988-989. https://doi.org/10.1111/tri.13853  

Tokuç, B., & Varol, G. (2020). Medical education in Turkey in time of COVID-19. Balkan Medical Journal, 37(4), 180-181. https://doi.org/10.4274/balkanmedj.galenos.2020.2020.4.003 

UNESCO. (2020). Policy brief: education during COVID-19 and beyond. https://unsdg.un.org/sites/default/files/2020-08/sgpolicy_brief_covid-19_and_education_august_2020.pdf

Van Manen, M. (2016). Researching lived experience: Human science for an action sensitive pedagogy. Routledge.

World Bank Group. (2021). World Bank Country and Lending Groups – World Bank Data Help Desk. Retrieved December 16. 2021 from https://datahelpdesk.worldbank.org/knowledgebase/articles/906519-world-bank-country-and-lending-groups

Zaharias, P., & Poylymenakou, A. (2009). Developing a usability evaluation method for e-learning applications: beyond functional usability. International Journal of Human–Computer Interaction, 25(1), 75-98. https://doi.org/10.1080/10447310802546716  

*Eugene Kua
Royal Children’s Hospital,
50 Flemington Rd,
Parkville VIC 3052
(+61) 411 224 125
Email: ekua12@gmail.com