Submitted: 17 February 2024
Accepted: 25 March 2025
Published online: 1 July, TAPS 2025, 10(3), 65-74
https://doi.org/10.29060/TAPS.2025-10-3/SC3251

Dani R Firman^1,2, Friedo W Dekker^2,3, Eriska Riyanti¹, Sunardhi Widyaputra¹ & Peter GM de Jong²

¹Faculty of Dentistry, Universitas Padjadjaran, Indonesia; ²Center for Innovation in Medical Education, Leiden University Medical Center, the Netherlands; ³Department of Clinical Epidemiology, Leiden University Medical Center, the Netherlands

Abstract

Introduction: To explore the current situation regarding teaching evidence-based practice (EBP) in Indonesian dental schools as a low-middle-income country (LMIC) setting.

Methods: Two nation-wide surveys were administered to all Indonesian dental schools (N=32) to capture the perceived value of the deans about EBP skills and teaching with its resource-related situation, as well as the actual teaching implementation and its challenges by their curriculum teams. The analysis was performed descriptively as national aggregate data.

Results: The response rates were 96.9% and 90.6%, respectively. The deans considered EBP skills to be very important (61.3%) or important (38.7%). However, only 16.1% of deans reported that the implementation of teaching EBP met their expectations. Most schools (75.9%) considered providing training for teachers as the priority plan. Most schools (62.1%) had an independent EBP course in their curriculum, while 86.2% incorporated EBP teaching (also) into other courses. All dental schools employed a lecture strategy for teaching EBP topics, but also used other strategies such as interactive strategies with a wide range of percentages being utilised (10.3 – 89.7%). EBP-specific topics were generally taught with a level of implementation varying from 34.5% to 96.6% in the pre-clinical and from 10.3% to 62.1% in the clinical programme.

Conclusion: The used approach can be considered an effective strategy to provide opportunities for schools to reflect their EBP teaching. This study confirmed the necessity for improvement, especially the need to support faculty development. The deans and their curriculum teams highly value collaborative improvement plans.

Keywords:           National Survey, Evidence-based Practice, Faculty Perception, Low-middle-income Country

I. INTRODUCTION

Evidence-based practice (EBP) is emerging as the standard in clinical dentistry. It integrates the best research evidence with patients’ values, preferences, and clinical experiences (Young et al., 2014). The EBP process involves five key steps in sequence (ask, acquire, appraise, apply, and evaluate). The appraise or critical appraisal skill as part of the steps is related to research capacity still considered as crucial gaps in the LMIC setting compared to high-income country (HIC) setting, and known to hinder EBP development and implementation (Gill et al., 2021). Furthermore, the suboptimal practice of EBP among dental professionals in lower and middle-income countries (LMIC) was suspected due to limited access of web-based subscription database and scarcity of experts as mentor (Minja & Lwoga, 2021).

Potentially, dental schools can play a significant role in promoting EBP implementation in dentistry of such setting by providing the access to evidence, supporting professional development and integrate the EBP teaching and learning in the curriculum. However, institutionalising EBP requires consistent policy and strong administrative support. To understand the current situation of EBP teaching in Indonesian dental schools, we conducted a study exploring the deans’ perceptions about its importance and the need for improvement. We also determined the actual level of implementation as reported by their curriculum teams. The study’s outcomes could inform future interventions to enhance EBP teaching in Indonesian dental schools.

II. METHODS

At the time of the study, in September 2022, there were 32 dental schools in Indonesia, all of which ran a bachelor’s programme (year 1-4) and a clinical programme (year 5-6). A descriptive survey study was conducted online using two sets of Indonesian-language questionnaires in sequence.

The first survey was launched during an in-person national meeting in Surabaya. Only the principal investigator himself participated via Zoom. This first survey aimed to gauge deans’ perceived importance of EBP skills and teaching, along with necessary resource improvements. The questionnaire included demographic items, five-point of Likert-scale questions on EBP teaching value, resources-related, an open-ended item on EBP teaching challenges, and a section for additional comments. Most deans completed the first online survey during that meeting, while those who did not attend the meeting were received the survey’s link via email.

Upon completion for each school by the deans, a second survey was sent to vice dean for academic affairs as corresponding contact using the email address provided by the deans in the previous survey. In the introduction letter, we specifically requested the involvement of their curriculum team of the bachelor and clinical programme as a group response for each school. This second survey gathered data on EBP teaching implementation and challenges more in details. It was translated and adapted from a previous survey used by Gorgon et al. (2013) which was considered relevant to all health professions, including dentistry. The adaptation included a pilot test beforehand to ensure its relevance and clarity.

The study protocol was approved by the Educational Research Review Board of the LUMC (Registry number: OEC/ERRB/20220913/1). Participants electronically consented to both surveys after reviewing the provided information. Respondents with missing or unclear data were contacted for clarification. Five-point Likert scale items, checklist items, and multiple-choice items were analysed descriptively. The open-ended item responses and additional comments were analysed thematically.

III. RESULTS

The first survey’s responses were received from 31 dental schools across Indonesia (96.9%). The schools were located on Java (18), Sumatera (6), Sulawesi (3), and Kalimantan and Bali (2 each). For the second survey, which targeted the curriculum team for each school, the responses were received from 29 schools (90.6%).

A. The Perceived Value of EBP Teaching (The 1^st Survey, n=31)

The deans perceived EBP skills as very important/essential (61.3%) or important (38.7%) for students. Incorporating EBP in the curriculum was seen as very important/essential (51.6%) or important (48.4%). Improving teachers’ EBP teaching expertise was deemed very necessary (54.8%) or necessary (38.7%). Five deans (16.1%) felt their schools met faculty expectations for EBP teaching, while others observed room for improvement (64.5% necessary, 19.4% very necessary). Resources like database access and librarian expertise varied in satisfaction. Most respondents (71%) valued collaboration with other schools on EBP training and implementation (See Appendix 1).

B. The Implementation of EBP Teaching (The 2^nd Survey, n=29)

Over half (62.1%) of participated schools have a standalone EBP course, and 86.2% incorporate EBP in other courses. Lecture-based teaching is the most common EBP strategy, followed by thesis discussions, journal readings, colloquiums, and seminars. Small group discussions and problem-based tutorials are also used. “Critically appraised topics” (CATs) are least used, with only three schools employing this active learning method. Both summative and formative assessments are used. Most schools use a blended approach for EBP courses, with one school offering a fully online course (See Appendix 2.a).

C. Available Resources for EBP Teaching (The 1^st and 2^nd Survey, n=31 and n=29)

Of the participating institutions, only 8 out of 29 dental schools have teachers formally-trained in EBP. Most schools have access to scientific databases, with ScienceDirect being the most accessible (58.6%). Other databases include Web of Science, Cochrane Library, EBSCO Host, CINAHL, and UpToDate, with some schools also subscribing to Springer Link, Wiley, Emerald, and SAGE (See Appendix 2.b). Database access is mostly provided by the university (80.6%), with some funded by faculty budgets (29%) or government institutions (9.7%). Only one school lack subscribed database access (See Appendix 3.a).

D. The EBP Specific Topics Coverage (The 2^nd Survey, n=29)

The five EBP steps (ask, acquire, appraise, apply, evaluate) are taught in Indonesian dental schools, with varying implementation levels from 34.5% to 96.6% in pre-clinic programmes and 10.3% to 62.1% in clinic programmes. The ‘apply’ and ‘evaluate’ steps are least taught in pre-clinic programmes. Interpreting forest plots in systematic reviews is the least covered topic, with only 10 out of 29 schools teaching it in pre-clinic levels, 3 in clinical levels, and 19 not covering it at all. Over a third of the schools (37.9%) do not teach critical appraisal of a systematic review. See Table 1 below for details.

	Pre-clinic		Clinic		Not taught in both programme**
EBP steps and its specific topics coverage (n=29)	Taught*	Not taught*	Taught*	Not taught*
I. Asking clinical question
(1) Formulating clinically questions using the PICO format	22 (75.9)	7 (24.1)	12 (41.4)	17 (58.6)	5 (17.2)
(2) Guided literature search by the clinical questions	20 (69.0)	9 (31.0)	18 (62.1)	11 (37.9)	5 (17.2)
II. Search the evidence
(3) Constructing a focused search with Boolean operator in a database	23 (79.3)	6 (20.7)	7 (24.1)	22 (75.9)	6 (20.7)
(4) Locating clinical evidence using electronic databases	26 (89.7)	3 (10.3)	8 (27.6)	21 (72.4)	3 (10.3)
(5) Utilising a reference manager software	25 (86.2)	4 (13.8)	9 (31.0)	20 (69.0)	4 (13.8)
III. Critically appraising evidence
(6) Study designs and their major strength and limitations	27 (93.1)	2 (6.9)	7 (24.1)	22 (75.9)	2 (6.9)
(7) Assessing the relevance of study design to the question asked	22 (75.9)	7 (24.1)	5 (17.2)	24 (82.8)	7 (24.1)
(8) Hierarchy or levels of evidence	23 (79.3)	6 (20.7)	6 (20.7)	23 (79.3)	5 (17.2)
(9) Difference between narrative, systematic review, and meta-analysis	21 (72.4)	8 (27.6)	4 (13.8)	25 (86.2)	7 (24.1)
(10) Difference between clinical and statistical significance	22 (75.9)	7 (24.1)	9 (31.0)	20 (69.0)	6 (20.7)
(11) Interpreting t tests, chi-square tests	28 (96.6)	1 (3.4)	6 (20.7)	23 (79.3)	1 (3.4)
(12) Interpreting p-value, confidence interval	28 (96.6)	1 (3.4)	6 (20.7)	23 (79.3)	1 (3.4)
(13) Understanding sensitivity and specificity, number needed to treat, odds ratio	28 (96.6)	1 (3.4)	6 (20.7)	23 (79.3)	1 (3.4)
(14) Understanding intention to treat analysis and power calculation	20 (69.0)	9 (31.0)	3 (10.3)	26 (89.7)	9 (31.0)
(15) Use of appraisal tool(s) to assess validity	22 (75.9)	7 (24.1)	7 (24.1)	22 (75.9)	7 (24.1)
(16) Ways in which study validity can be threatened	25 (86.2)	4 (13.8)	5 (17.2)	24 (82.8)	4 (13.8)
(17) Difference between internal and external validity	21 (72.4)	8 (27.6)	3 (10.3)	26 (89.7)	8 (27.6)
(18) Critical appraisal of systematic reviews	18 (62.1)	11 (37.9)	6 (20.7)	23 (79.3)	11 (37.9)
(19) Interpreting forest plots in systematic reviews	10 (34.5)	19 (65.5)	3 (10.3)	26 (89.7)	19 (65.5)
(20) Critical appraisal of studies about intervention (RCTs and clinical trials)	18 (62.1)	11 (37.9)	7 (24.1)	22 (75.9)	9 (31.0)
(21) Critical appraisal of studies about diagnosis (cohort, case-control studies)	21 (72.4)	8 (27.6)	5 (17.2)	24 (82.8)	7 (24.1)
(22) Critical appraisal of studies about prognosis (longitudinal studies)	18 (62.1)	11 (37.9)	5 (17.2)	24 (82.8)	10 (34.5)
IV. Implementation and communication
(23) Communicating the recommendation from EBP process to the patient	14 (48.3)	15 (51.7)	9 (31.0)	20 (69.0)	12 (41.4)
(24) Deciding the clinical decision considering the patient’s value	12 (41.4)	17 (58.6)	11 (37.9)	18 (62.1)	11 (37.9)
V. Evaluation
(25) Evaluation of the EBP implementation (The patient’s outcome and student’s performance through self-reflection)	13 (44.8)	16 (55.2)	9 (31.0)	20 (69.0)	13 (44.8)

Table 1. EBP steps and its specific topics coverage for each programme level (n=29)^b

* n (%)

** n (%) Overlap calculation with the number of not taught on each programme

^b From the 2^nd survey to the curriculum teams

E. Challenges (The 1^st Survey, n=31 and The 2^nd Survey, n=29)

The top three challenges in Indonesian dental schools, as reported by curriculum teams, are a lack of EBP experts (72.4%), insufficient accessible EBP courses for faculty (69%), and students’ inadequate knowledge of statistics and research methodology (65.5%) (See appendix 4). In the open-ended item section, most of the deans expressed the challenges of EBP teaching. Teachers-related factors were considered the most challenging, such as limitation in knowledge and skills, and an overloaded work situation.

F. Action Plans (The 2^nd Survey, n=29)

Most respondents (75.9%) highly valued teacher training. They also preferred incorporating EBP skills into the curriculum (48.3%) and holding faculty meetings to discuss EBP programme development (44.8%) (See appendix 3.b).

IV. DISCUSSION

To our knowledge, this is the first nation-wide survey with very high response rate to report insights of EBP teaching in the South-east Asia region, more specifically in Indonesian dental education setting. Targeting both deans and curriculum teams yielded rich data and stakeholder involvement, potentially fostering acceptance of future EBP improvements at institutional or national levels.

There is a slight gap between the perceived importance of EBP skills for students and the importance of incorporating such skills into the curriculum, which may be related to limited resources. The deans considering room for improvement, especially in teacher expertise. This is echoed by curriculum teams, as less than a third of Indonesian dental schools’ teachers have formal training experience, i.e. training from a well-known academic-based institution or recognised training organisation. This lack of EBP teachers is a top challenge identified by respondents. Deans and curriculum teams agree on the need for improved teacher skills.

The majority of Indonesian dental schools are teaching EBP topics in stand-alone EBP courses as well as incorporated in other regular courses using both face-to-face and online delivery. Lecture-based is the most used teaching strategy among the schools. Our findings suggest that there is room for enhancing the teaching strategy of EBP in Indonesian dental schools towards a more multi-facet, interactive and more authentic setting in the clinical environment as suggested by Howard et al. (2022).

Regarding critical appraisal topics based on the study design (See Table 1, item 18, 19-21), there were very few schools implementing a Critically Appraised Topics (CATs) teaching strategy (See Appendix 2.a). CATs, one-page summaries of evidence on a clinical question, offer active, authentic learning and being recommended in the literature to provide both a critique of the research and a statement of the clinical relevance of its findings. It is crucial to teach critical appraisal skills using the teaching strategy that is supported by the evidence to ensure sufficient development of EBP skills. Many Indonesian dental educators may not be familiar with this CATs teaching method.

The survey reveals that the ‘apply’ and ‘evaluate’ steps of EBP are less addressed than the ‘ask’, ‘acquire’, and ‘appraise’ steps. These five-steps, identical to those in Evidence-Based Medicine (EBM), form the basis of clinical practice and teaching. Key EBM developments like shared-decision making skills, part of the ‘apply’ step, are often overlooked. Hence, it’s vital to teach all EBP steps, ideally in a more authentic clinical setting.

The need for more EBP-trained faculty in most schools highlights the importance of a faculty development programme supported by national collaboration. As a lower-resource country, Indonesia faces challenges in dental education, including limited access to databases and librarian expertise, primarily due to financial constraints, inadequate institutional support, and the absence of national policies prioritising scientific resources. These limitations hinder faculty and students from fully implementing evidence-based practices, widening disparities with high-income countries. To address these issues, a policy brief should be directed to high-level stakeholders, advocating for essential infrastructure, more funding and resource investments for both private and government-funded schools.

This study has limitations. Due to the five-point Likert scale in the first questionnaire were not constructed on the same concept, the reliability test was not feasible. This is also applied for the second questionnaire, which adapted from a previous study, was mostly factual and checklist-based. The value of EBP teaching, reported by deans and curriculum teams, might be socially desirable. However, the results underscore the importance of EBP teaching and the need for improvement. It is worth noting that after the data collection was completed, Indonesia formally transitioned from a lower-middle-income to an upper-middle-income country as of July 2023. This transition suggests a more promising future, with potential improvements in the financial capacity, but it will require the national policy support to be fully implemented.

V. CONCLUSION

This first national survey reveals the state of EBP teaching in Indonesian dental schools, representing Southeast Asian countries. It highlights the need for faculty development and more active, authentic learning experiences but further analysis is required to adopt and adapt these interventions for lower-resource settings. The deans and their curriculum teams highly value collaborative improvement plans. This supports EBP’s vision of using current research to improve Indonesian oral healthcare.

Notes on Contributors

DF is the principal investigator. He designed, analysed, prepared and wrote the manuscript collaboratively with co-authors.

SW and ER made substantial contributions to preparation of the instruments, data collection and editing the final manuscript.

FD and PJ made substantial contribution to the design, analysis, preparation and editing the final manuscript.

All authors read and approved the final version.

Ethical Approval

The study protocol was approved by the Educational Research Review Board of the LUMC (Registry number: OEC/ERRB/20220913/1).

Data Availability

Due to confidentiality of the responses and commitments made with the study participants, also considering the small size data set with its unique data for some variables that might still be traceable to the respondent. Data are available on reasonable request by email to the corresponding author.

Acknowledgement

The authors express gratitude to all participants and schools involved in this study, AFDOKGI and its members for their support and data access, the Research group at the OEC LUMC for their constructive feedback, Universitas Padjadjaran for sponsoring the first author’s PhD scholarship, and Edward Gorgon for permitting the modification and use of the questionnaire.

Funding

This research is part of the first author’s (DF) PhD study, funded by a scholarship from Universitas Padjadjaran, Indonesia (Registry number 3781/UN.6.WR2/KP.10/2021).

Declaration of Interest

The first author (DF), an assistant professor at the Faculty of Dentistry, Universitas Padjadjaran, which participated in this study. DF also had ties until 2020 with AFDOKGI, the association of Indonesian dental schools, as a dental education consultant. 

References

Gill, P. J., Ali, S. M., Elsobky, Y., Okechukwu, R. C., Ribeiro, T. B., Soares dos Santos Junior, A. C., Umpierre, D., & Richards, G. C. (2021). Building capacity in evidence-based medicine in low-income and middle-income countries: Problems and potential solutions. BMJ Evidence-Based Medicine, 26(3), 82–84. https://doi.org/10.1136/bmjebm-2019-111272

Gorgon, E. J. R., Basco, M. D. S., & Manuel, A. T. (2013). Teaching evidence-based practice in physical therapy in a developing country: A national survey of Philippine schools. BMC Medical Education, 13(1), Article 154. https://doi.org/10.1186/1472-6920-13-154

Howard, B., Diug, B., & Ilic, D. (2022). Methods of teaching evidence-based practice: A systematic review. BMC Medical Education, 22(1), Article 742. https://doi.org/10.1186/s12909-022-03812-x

Minja, I. K., & Lwoga, E. T. (2021). Evidence based dentistry among dentists in low- and middle-income countries: A systematic review. The East African Health Research Journal, 5(2), 129–136. https://doi.org/10.24248/eahrj.v5i2.662

Young, T., Rohwer, A., Volmink, J., & Clarke, M. (2014). What are the effects of teaching evidence-based health care (EBHC)? Overview of systematic reviews. PLOS ONE, 9(1), e86706. https://doi.org/10.1371/journal.pone.0086706

*Dani R Firman
Jalan Sekeloa Selatan I,
Kota Bandung, Jawa Barat 40132,
Indonesia
+62222534985
Email: dani.firman@unpad.ac.id / d.r.firman@lumc.nl

Submitted: 30 April 2024
Accepted: 25 September 2024
Published online: 1 April, TAPS 2025, 10(2), 82-85
https://doi.org/10.29060/TAPS.2025-10-2/SC3345

Sethapong Lertsakulbunlue & Anupong Kantiwong

Department of Pharmacology, Phramongkutklao College of Medicine, Thailand

Abstract

Introduction: Very Short Answer Questions (VSAQs) minimise cueing and simulate actual clinical practice more accurately than Single Best Answer Questions, as multiple-choice options might not be realistic. Phramongkutklao College of Medicine has developed a Self-Marked VSAQ (SM-VSAQ) for formative assessments. This study determines the validity and reliability of the SM-VSAQs.

Methods: Ninety-four third-year pre-clinical students took two occasions of 10-question SM-VSAQ exams regarding cardiovascular drugs. Each question consisted of two steps: (1) clinical vignettes with questions and (2) expected answers with scores, self-marking, and feedback comprehension. Scores ranged from 0.00 to 1.00 in 0.25 increments, though not every increment was applied to all questions. A distribution of the rating agreement between students’ and teacher’s ratings was presented to determine criterion-related validity and inter-rater reliability.

Results: Criterion-related validity revealed 90.64% and 93.19% of the ratings demonstrated exact agreement between students’ and teachers’ ratings, with an inter-rater reliability of 0.972 and 0.977 for the first and second occasions, respectively (p=0.001). The exact agreement was relatively lower on the first occasion for questions with more diverse expected answers (85.11%, r=0.867, p=0.001) and drugs requiring their specific full names for a perfect mark (74.47%, r=0.849, p=0.001). While questions with specific guides do not require complex answers, they received a higher exact agreement.

Conclusion: The SM-VSAQ format effectively combines guided answers with the VSAQ model. The agreement with teacher-rated is excellent. Marking discrepancies rooted in misconceptions underscores the importance of teacher feedback in improving self-grading in formative assessments. Regular self-assessment practice is recommended to enhance grading accuracy.

Keywords:           Very Short Answer Question, Self-assessment, Medical Education, Undergraduate, Pharmacology

I. INTRODUCTION

Very Short Answer Questions (VSAQs) emerge as a relatively novel assessment format, addressing the constraints of traditional examination methods like Single Best Answer Questions (SBAQs), Constructed Response Questions (CRQs), and Modified Essay Questions (MEQs) (Sam et al., 2018). Although SBAQs are widely adopted in medical education globally, they are prone to cueing effects, leading examinees to depend on contextual clues, promoting a recognition-based learning approach (Sam et al., 2018). Moreover, the absence of multiple-choice options in real-life scenarios diminishes the relevance of SBAQs to medical practice.

Conversely, while CRQs and MEQs better mimic real-life situations, they suffer from rater dependency and significant evaluation time. Whereas VSAQs, free-response questions with 1–5 word answers, lessen rater dependency and evaluation time. Evidence indicates that VSAQs outperform SBAQs in discrimination, validity, and reliability in undergraduate assessments. Their open-ended nature prevents recognition-based learning and cueing. Additionally, VSAQs adeptly pinpoint common errors, often missed by SBAQs, and offer valuable feedback opportunities for educators (van Wijk et al., 2023).

Feedback is crucial for supporting and enhancing learning. Despite its longstanding importance in medical education, effective feedback is frequently deemed insufficient (Kuhlmann Lüdeke & Guillén Olaya, 2020). Self-assessment, enabled by formative exams, allows learners to identify their learning needs (Gedye, 2010). To improve feedback in formative assessments, Phramongkutklao College of Medicine (PCM) developed the Self-marked VSAQ (SM-VSAQ) format, which pairs a VSAQ with possible answers and a marking guide. Students may assess their understanding and pinpoint study areas through SM-VSAQ, enhancing feedback. Although VSAQs offer several benefits, challenges remain in grading the tests, as they may require a longer time. The self-graded format could address this issue in low-stakes examinations. This study assesses whether the SM-VSAQ with partial credit format, utilizing the marking guide, would achieve valid and reliable ratings compared with the teachers.

II. METHODS

Ninety-four third-year pre-clinical students participated in two 10-item SM-VSAQ during a cardiovascular pharmacology course. The exams covered antihypertensive, antiarrhythmic, antianginal, antithrombotic drugs, heart failure drugs, rational drug use, dyslipidaemia treatments, and drugs for atherosclerotic cardiovascular disease (ASCVD). The second SM-VSAQ sessions vary by changing the clinical vignette, the question, or both while maintaining the same underlying blueprint as the first session. Difficulty levels align with the Thai Medical Competency Assessment Criteria. Students had attended lectures on these drug groups before the exams. The VSAQ was content-validated by three professors for relevance, difficulty, feasibility, and simplicity using the Item Objective Congruence method with all over 0.67 of 1.00, indicating acceptable content validity. This approach ensured comparable difficulty.

The formative test was administered through Google Forms under examination conditions within a one-hour timeframe. Ethical approval was obtained from the Institutional Review Board, Royal Thai Army, and the waiver of the requirement for participant consent was deemed unnecessary following national regulations. An information sheet was provided on the first page of the Google Form. This initial test was conducted a day after they completed all lectures. After receiving teacher-led feedback and having time to review, students took a second parallel formative test ten days before the summative exam.

The SM-VSAQs featured four components for each question: clinical vignettes and questions on the first page, answers with scoring guidelines on the next page after they’ve answered, and a self-scoring option with feedback on answer comprehension. Scores ranged from 0.00 to 1.00 in 0.25 increments, though not every increment was applied to all questions. After the students completed the exam, they provided open-ended feedback on the pros and cons of the format. Examples of the format are shown in supplementary figures 1 and 2.

The self-rated, according to the marking guide, were exported into a Microsoft Excel spreadsheet to facilitate teacher ratings of the VSAQ answers. Using the ‘filter’ function in Microsoft Excel, the range of answers for each question was examined, and marks were awarded (Sam et al., 2018). Minor misspellings or alternative correct spellings were considered correct. Three pharmacology professors, who assigned scores, reviewed student answers that fell outside the guide. Consensus-determined scores require agreement from at least two of the three professors.

The data analyses were performed using StataCorp, 2021, Stata Statistical Software: Release 17. College Station, TX: StataCorp LLC. Consistency reliability was analysed using Cronbach’s alpha. Criterion-related validity was demonstrated by the distribution of the rating agreement between student and teacher ratings, presented as frequency and percentages. Inter-rater reliability was calculated using Pearson’s correlation.

III. RESULTS

Cronbach’s alpha for the SM-VSAQ was 0.741 and 0.721 on the first and second occasions, respectively. The teacher-rated alpha was 0.766 initially and 0.735 on the second. Criterion-related validity was assessed through agreement analysis (Supplementary Tables 1 and 2). Table 1 summarises the results of the agreement analysis. 90.6% and 93.19% of the ratings showed exact agreement between the students’ and teachers’ ratings, with an inter-rater reliability of 0.972 and 0.977 for the first and second occasions, respectively. The exact agreement is relatively low on the first occasion of Drugs used in heart failure (85.11%) and Anti-angina drugs (74.47%). Conversely, antithrombotics and drugs used in ASCVD received a high exact agreement of 96.81%. Example of questions with high and low agreement is demonstrated in supplementary figures 1 and 2. Additionally, content analysis of student’s feedback revealed that they perceived that the format helps identify knowledge gaps, encourages review of missed topics, and aids in recognizing their current knowledge level (Supplementary Table 3).

Submitted: 30 April 2024
Accepted: 25 September 2024
Published online: 1 April, TAPS 2025, 10(2), 86-90
https://doi.org/10.29060/TAPS.2025-10-2/SC3551

Ozlem Tanriover¹, Sukran Peker², Seyhan Hidiroglu², Dilek Kitapcioglu³ & M. Ali Gülpınar¹

¹Department of Medical Education, School of Medicine, Marmara University, Türkiye; ²Department of Public Health, School of Medicine, Marmara University, Türkiye; ³Department of Medical Education, School of Medicine, Acibadem Mehmet Ali Aydinlar University, Türkiye

Abstract

Introduction: This study aimed to identify the emotions experienced by medical students during the quarantine period, explore their coping strategies, and determine whether the pandemic affected their learning behaviours.

Methods: In this descriptive study, data were collected through an online questionnaire. The study included quarantined medical students enrolled in a distance education program at a medical school during the pandemic. The Positive and Negative Affect Scale was used to assess the emotional spectrum of the participants.

Results: The majority of medical students in the sample experienced predominantly negative emotions. Students with higher negative affect scores struggled with stress management and were less able to employ emotional regulation strategies, which adversely affected their learning behaviours.

Conclusion: Emotions should be explicitly addressed in medical education, and students should be supported in managing their emotions as part of their professional development and well-being.

Keywords:           Coping, Emotions, Emotion Regulation, Medical Student, Pandemic, Stress

I. INTRODUCTION

Medical education has long been a significant source of stress for students. The COVID-19 pandemic introduced additional challenges with social distancing, home confinement, and online learning, further intensifying this burden (Fegert et al., 2020). In Turkey, the Council of Higher Education suspended face-to-face education on March 26, 2020, shifting all theoretical and some practical medical courses online. This move removed students from campuses and hospital settings to protect public health. As a result, medical students from all years, including those close to graduation, were quarantined and separated from their academic routines.

Quarantine, a measure to prevent disease spread, often leads to frustration, boredom, and heightened distress (Brooks et al., 2020). Medical education is already emotionally taxing, and the pandemic amplified these challenges. While many studies have explored the emotions of healthcare workers and students on the front lines, little is known about how medical students in Turkey, confined at home, experienced and coped with this period. This study aims to explore their emotional experiences, coping mechanisms, and changes in learning behaviours during quarantine.

II. METHODS

This descriptive study utilised an online questionnaire sent via Microsoft Forms to medical students from a university in Istanbul. Participants included both preclinical and clinical students, all engaged in distance learning due to the pandemic. The survey was distributed in June 2020, with participation being voluntary and anonymous. The study ran from mid-June to the end of July 2020.

A. Ethical Approval

The study was approved by the School of Medicine Ethical Committee (No. 09.2020.657). Consent was obtained from all participants before data collection.

B. Data Collection

The questionnaire gathered sociodemographic data and inquired about online education experiences. The Positive and Negative Affect Scale (PANAS), developed by Watson et al. (1988) and adapted to Turkish by Gençöz (2000), was used to assess emotions. The PANAS contains 20 adjectives rated on a five-point Likert scale, with scores ranging from 10 to 50 for both positive and negative emotions. High positive scores reflect engagement and energy, while high negative scores indicate distress.

C. Statistics

Statistical analysis was conducted using SPSS 20.0. Normality was tested with Kolmogorov-Smirnov and Shapiro-Wilk tests. Descriptive statistics were presented as medians (IQR) for non-normal data and frequencies for categorical data. The Mann-Whitney U test was used for two-group comparisons, and the Kruskal-Wallis test for comparisons across more than two groups. A p-value < 0.05 was considered statistically significant.

III. RESULTS

A total of 145 students participated in the survey, with a slight female majority (57%, n=83). The participants’ ages ranged from 20 to 27 years (mean=21.4, SD=1.99), and most (93%, n=135) lived at home with their families. Students attended an average of 4-6 hours of online lectures per day during quarantine. Emotions such as “interested,” “alert,” and “enthusiastic” had low scores, while 63.4% reported feeling distressed, followed by 40.7% upset and 39.3% irritable.

The PANAS total score comparison showed that students aged 22-25 scored significantly higher than those aged 18-21 (51.7 vs 48.1, p=0.028). Similarly, positive affect (PA) scores were higher in older students (p<0.001). Students living alone had higher total PANAS scores (59 vs 49, p=0.008) and PA scores (31.7 vs 23.6, p<0.001) than those living with their families.

A. Coping Strategies

Most students (65.5%) coped with stress using distraction activities like watching TV, while 8.3% sought emotional support. A minority engaged in substance use or denial (4.9%), and 8.3% reported being unable to cope. Those who made an effort to cope had higher PA scores (24.6 vs 21.2, p=0.036) and lower negative affect (NA) scores (25.3 vs 29.5, p=0.01).

B. Changes in Learning Behaviour

About 74.5% of students reported changes in their study methods during the pandemic, though no significant differences in PANAS scores were observed. When asked about adapting to online learning, 33.7% of students reported difficulty adjusting, while 17.3% used self-motivation, and 11.5% created study programs with peers.

C. Emotion Regulation

Cognitive reappraisal was the most common strategy used by 65.5% of students to manage emotions, while 8.3% used expressive suppression. Nearly 25% reported no effective emotion regulation strategy, and 1.4% felt they didn’t need one. Those who used emotion regulation strategies had significantly lower NA scores (p=0.017).

Submitted: 6 May 2024
Accepted: 12 September 2024
Published online: 7 January, TAPS 2025, 10(1), 48-52
https://doi.org/10.29060/TAPS.2025-10-1/SC3349

Craig S. Webster^1,2, Antonia Verstappen¹, Jennifer M. Weller¹ & Marcus A. Henning¹

¹Centre for Medical and Health Sciences Education, School of Medicine, University of Auckland, Auckland, New Zealand; ²Department of Anaesthesiology, Faculty of Medical and Health Sciences, University of Auckland, Auckland, New Zealand

Abstract

Introduction: We aimed to determine the extent to which non-technical skills and attitudes acquired during undergraduate interprofessional simulation in an Advanced Cardiac Life Support (ACLS) course translated into clinical work.

Methods: Following ACLS simulation training for final-year nursing and medical students, we conducted a 1-year follow-up survey, when graduates were in clinical practice. We used the Readiness for Interprofessional Learning Scale (RIPLS – higher scores indicate better attitudes to interprofessional practice), and nine contextual questions with prompts for free-form comments. RIPLS scores underwent repeated-measures between-groups (nurses vs doctors) analysis at three timepoints (pre-course, post-course and 1-year).

Results: Forty-two surveys (58% response) were received, demonstrating translation of non-technical skills and attitudes to clinical practice, including insights into the skills and roles of others, the importance of communication, and improved perceptions of preparedness for clinical work. However, RIPLS scores for doctors decreased significantly upon beginning clinical work, while scores for nurses continued to increase, demonstrating a significant interaction effect (reduction of 5.7 points to 75.7 versus an increase of 1.3 points to 78.1 respectively – ANOVA, F(2,76)=5.827, p=0.004). Responses to contextual questions suggested that reductions in RIPLS scores for doctors were due to a realisation that dealing with emergency life support was only a small part of their practice. However, the prevailing work cultures of nurses and doctors in the workplace may also play a part.

Conclusion: We demonstrated the translation of non-technical skills and attitudes acquired in undergraduate simulation to the clinical workplace. However, results are tempered for junior doctors beginning practice.

Keywords:           Work Culture, Translation, RIPLS, Simulation, Advanced Cardiac Life Support, Undergraduate Education, Skills and Attitudes, Patient Safety

I. INTRODUCTION

Preparing undergraduate healthcare students for their future roles in the clinical workplace is a central concern for modern healthcare educators and is of critical importance for the maintenance of adequate healthcare services throughout the world (Barnes et al., 2021). Modern healthcare is inherently multidisciplinary, yet much of the training received by healthcare practitioners remains siloed within professional groups, and this is particularly the case at the undergraduate level. The use of simulation in healthcare has become increasingly important in recent years as a way to offer safe and immersive training. Conducting simulation with interprofessional healthcare teams allows those who will work together to be trained together, and can have the double benefit of promoting the acquisition of technical and non-technical skills in participants, while also allowing insight into the skills, roles and knowledge of other team members from different professional groups (Jowsey et al., 2020).

We previously reported on the development and evaluation of an interprofessional Advanced Cardiac Life Support (ACLS) course for undergraduate nursing and medical students in their final year at the University of Auckland, aimed at increasing technical resuscitation and non-technical teamwork skills (Webster et al., 2018). The evaluation study, using a mixed-methods design and recruiting 69% of the entire year’s student cohort, demonstrated significant improvements in scores on the Readiness for Interprofessional Learning Scale (RIPLS) over the course of the training day, and important interprofessional and attitudinal insights into the skills and knowledge of other team members related to communication, teamwork, leadership, realism, and professional roles. Medical and nursing students both reported that such insights would not have occurred during uniprofessional simulation and felt that the course had better prepared them for work in the clinical context. At the end of the training day we invited participants to take part in a further follow-up survey timed to occur approximately one year later, at a time when participants would typically be working clinically.

Our aim in the present study was to determine the extent to which the non-technical skills and attitudes acquired during the undergraduate interprofessional ACLS simulation course translated into the clinical work of the former course participants.

II. METHODS

We conducted a 1-year follow-up survey comprising a further RIPLS questionnaire and nine additional contextual questions, with quantitative response scales and prompts for explanatory free-form comments (see Supplementary Table 1). The survey was mailed to participants who had elected to supply their contact information, along with a post-paid return envelope. All participants gave written informed consent to participate. One postal and one email reminder was also sent if a reply was not forthcoming.

The RIPLS is a validated questionnaire comprising 19 questions using 5-point Likert response scales (anchors, 1=strongly disagree to 5=strongly agree), and yielding a possible total score from 19 to 95 points where higher scores indicate a greater willingness to engage in interprofessional practice (Parsell & Bligh, 1999). In the present analysis, RIPLS responses from each participant in the 1-year follow-up survey were paired with their own corresponding RIPLS scores at two previous time points and underwent repeated-measures between-groups (nurses vs doctors) analysis at three timepoints (pre-course, post-course and 1-year). Responses to quantitative ratings on contextual questions used identical 5-point Likert scales and were summarised along with exemplar quotations from the free-form comments (Supplementary Table 1).

III. RESULTS

Between August 2014 and November 2015, 42 survey responses were received, representing a 58% response rate from the 73 participants who elected to give contact information for the follow-up survey. Two nurses were not working clinically at the time of the survey, and their responses were excluded from analysis – resulting in a total of 14 nurses and 26 doctors being included in the present study. All doctors were working in hospitals at the time of the 1-year survey, as were 71% of nurses. The remaining nurses were working in primary healthcare or general practice. RIPLS data did not significantly depart from a normal distribution (Shapiro-Wilk test, p=0.22), therefore parametric analysis was conducted using SPSS v.27 (IBM SPSS Statistics, Armonk, New York).

A one-way repeated measures ANOVA demonstrated a significant interaction effect between time point and professional group (F(2, 76)=5.827, p=0.004), demonstrating that at the 1-year time point mean RIPLS scores for doctors fell significantly by 5.7 points, while mean RIPLS scores for nurses continued to increase by 1.3 points (Figure 1).

 

Figure 1. RIPLS scores for nurses and doctors paired over three time points

The results of the contextual questions in the present study (1-year time point) demonstrated strong support by nurses and doctors for the value of the interprofessional ACLS course in general terms and more specifically in terms of feeling part of the team, better understanding the skills and roles of others, and feeling more confident in clinical practice – with all mean responses ranging from high 3’s to >4 (see Supplementary Table 1 for complete summary). Participants strongly agreed that the interprofessional ACLS course should continue to be offered (with an overall mean score of 4.68 out of 5). The single reverse-scored question asking whether ACLS training would have been more effective if conducted uniprofessionally demonstrated strong disagreement with an overall mean score of 1.65. Exemplar quotations from free-form comments provided a context for the quantitative results in terms of demonstrating that the ACLS training better prepared doctors and nurses for emergencies, helped to improve their communication, and was a realistic form of training – for example, stating “Much more ‘real life’ when other professions involved” (doctor) and “Interdisciplinary teamwork is huge in the real world…” (nurse).

Despite the largely positive findings, exemplar quotations also allowed some insight into why doctors’ RIPLS scores were high at the end of the ACLS course, but then fell significantly upon entry into clinical practice at the 1-year time point. Exemplar quotations suggested that once in the clinical workplace junior doctors better appreciated that the technical skills in the ACLS course made up only a small part of their scope of practice, stating that there “are many things… you are unable to do and it is important to know what level of knowledge and ability other individuals may have” and that ACLS “does not make up a large part of my clinical practice” (Supplementary Table 1).

IV. DISCUSSION

Our results demonstrate the translation of non-technical skills and attitudes acquired during undergraduate interprofessional simulation training to the clinical workplace. Our findings show particular benefits for nurses, and reinforce the value of the interprofessional ACLS course as an important part of the undergraduate curriculum. While the overall evaluation of the ACLS course was positive, the differential response in RIPLS scores between nurses and doctors upon entry into the clinical workplace is an intriguing result which clearly warrants further research.

We know of no previous study that has followed the same cohort of undergraduate participants after an interprofessional simulation course up to the point where they have entered the clinical workplace. The ability to pair responses for the same participants across all three time points in our study is a strength, as this avoids the variability that would be present if there were different participants at each time point, and so gives us more confidence in our findings.

Our results suggest that the significant reduction in RIPLS scores upon entry into the clinical workplace for junior doctors may be due to a realisation that the technical skills learnt in the ACLS course make up only a small part of a doctor’s domain of practice. However, recent research into the experiences of junior doctors during interprofessional collaboration suggests that the interaction effect in RIPLS scores across professional groups may also be a consequence of the different work cultures of nurses and doctors. Evidence suggests, including from our own University, that doctors typically believe that they should take individual responsibility for their clinical work, while nurses have a more collective view of patient care (Horsburgh et al., 2006; van Duin et al., 2022). Thus, the prevailing workplace cultures could reinforce and promote nurses’ willingness to work interprofessionally (hence explaining the increase in their RIPLS scores), while for doctors the prevailing individualistic work culture may reduce their willingness to work interprofessionally (hence contributing to the reduction in their RIPLS scores, Figure 1).

Further work to investigate this intriguing interaction effect, and the dynamics of work cultures and professional identity formation, would likely involve mixed-method research, perhaps using observation, interviews or focus groups and quantitative measures such as RIPLS (Jowsey et al., 2020). In addition, such studies conducted with clinicians at various levels of experience within a hospital could potentially yield insight into the state of the prevailing clinical work cultures and may allow some estimate of whether incoming graduates with interprofessional training could change these cultures, and when a critical mass of such graduates may allow this to happen. In the meantime, our results suggest that prevailing work cultures may represent a challenge for interprofessional teamwork initiatives, at least in medicine.

V. CONCLUSION

Our follow-up study demonstrated the translation of the non-technical skills and attitudes acquired during undergraduate interprofessional simulation training to the clinical workplace in terms of insights into the skills and roles of others, the importance of communication, and perceptions of preparedness to deal with emergencies. However, these results appear to be tempered for junior doctors beginning clinical work likely due to realisations around the applicability of ACLS training to their scope of practice and the influences of their prevailing workplace culture.

Notes on Contributors

Craig S. Webster was involved in the conceptualisation of this study, data analysis, writing and revision.

Antonia Verstappen was involved in data collection and analysis, writing and revision.

Jennifer M. Weller was involved in the conceptualisation of this study, writing and revision.

Marcus A. Henning was involved in the writing and revision of this paper.

Ethical Approval

This study was carried out in accordance with all regulations of the host organisation and with the approval of the Human Participants Ethics Committee of the University of Auckland (reference number 9073). All participants gave written informed consent to participate.

Data Availability

The complete data set for this study is openly available on the Figshare repository, https://doi.org/10.6084/m9.figshare.25750230

Funding

This study was conducted without funding.

Declaration of Interest

All authors have no potential conflicts of interest.

References

Barnes, T., Yu, T. W., & Webster, C. S. (2021). Are we preparing medical students for their transition to clinical leaders? A national survey. Medical Science Educator, 31(1), 91-99. https://doi.org/10.1007/s40670-020-01122-9

Horsburgh, M., Perkins, R., Coyle, B., & Degeling, P. (2006). The professional subcultures of students entering medicine, nursing and pharmacy programmes. Journal of Interprofessional Care, 20(4), 425-431. https://doi.org/10.1080/13561820600805233

Jowsey, T., Petersen, L., Mysko, C., Cooper-Ioelu, P., Herbst, P., Webster, C. S., Wearn, A., Marshall, D., Torrie, J., Lin, M. P., Beaver, P., Egan, J., Bacal, K., O’Callaghan, A., & Weller, J. (2020). Performativity, identity formation and professionalism: Ethnographic research to explore student experiences of clinical simulation training. PLoS One, 15(7), e0236085. https://doi.org/10.1371/journal.pone.0236085

Parsell, G., & Bligh, J. (1999). The development of a questionnaire to assess the readiness of health care students for interprofessional learning (RIPLS). Medical Education, 33(2), 95-100. https://doi.org/10.1046/j.1365-2923.1999.00298.x

van Duin, T. S., de Carvalho Filho, M. A., Pype, P. F., Borgmann, S., Olovsson, M. H., Jaarsma, A. D. C., & Versluis, M. A. C. (2022). Junior doctors’ experiences with interprofessional collaboration: Wandering the landscape. Medical Education, 56(4), 418-431. https://doi.org/10.1111/medu.14711

Webster, C. S., Hallett, C., Torrie, J., Verstappen, A., Barrow, M., Moharib, M. M., & Weller, J. M. (2018). Advanced cardiac life support training in interprofessional teams of undergraduate nursing and medical students using mannequin-based simulation. Medical Science Educator, 28(1), 155-163. https://doi.org/10.1007/s40670-017-0523-0

*Craig Webster
Centre for Medical and Health Sciences Education
School of Medicine, University of Auckland
Private Bag 92-019
Auckland 1142, New Zealand.
Email: c.webster@auckland.ac.nz

Submitted: 17 January 2024
Accepted: 14 May 2024
Published online: 1 October, TAPS 2024, 9(4), 57-60
https://doi.org/10.29060/TAPS.2024-9-4/SC3234

Nicole Huiyi Ong¹, Boon Chong Kwok^2,3, Siti Munirah Aman⁴, Steven Michael Dans Macabasco⁵ & Wai Pong Wong²

¹Department of Physiotherapy, Singapore General Hospital, Singapore; ²Health and Social Sciences, Singapore Institute of Technology, Singapore; ³Rehabilitation, Clinical Pilates Family Physiotherapy, Singapore; ⁴Department of Rehabilitation, Outram Community Hospital, Singapore; ⁵Rehabilitation, NTUC Eldercare, Singapore

Abstract

Introduction: Clinical practice education is an important curriculum in developing physiotherapy students to professionals. This curriculum is largely dependent on clinical educators. Although traits of effective clinical educators had been studied in Western population, it was unclear if the Asian population would yield similar traits. Hence, we studied perspectives of students and academic educators to identify top five core traits of an effective clinical educator. The findings will then help guide policy development for clinical educator training and development.

Methods: A survey was circulated to physiotherapy students and academic educators through convenience sampling. We studied 16 traits – enthusiasm, availability, clarity, clinical competence / knowledge, feedback skills, organisational skills, professionalism, well prepared, scholarly activity, non-judgemental, respect students’ autonomy / independence, sincerity, listening skills, evidence-based practice, role model, and awareness of students’ learning needs.

Results: The top five traits identified by students and academic educators were similar in four traits – clinical competency / knowledge, clarity, feedback skills, and enthusiasm, and differed in the choice of “non-judgemental” by students and “awareness of students’ learning needs” by academic educators. Interestingly, when the top six traits were considered instead of five, students and academic educators identified the same traits but in different ranked orders.

Conclusion: The study found six common core traits of an effective clinical educator from the perspectives of students and academic educators. Continuing education development for clinical educators should focus on these six traits prior to other 10 sub-traits.

Keywords:           Training, Development, Education, Allied Health, Entrustable Professional Activities, Mentoring

I. INTRODUCTION

Clinical Practice Education (CPE) is a core component in undergraduate allied health studies, partnering clinicians with additional role as a Clinical Educator (CE) (Gibson et al., 2019). A study evaluating effective residency teaching for doctors identified 15 traits that a CE should possess, which are enthusiasm, competence-knowledge, clarity, availability, non-judgemental, independence, feedback skills, role model, professionalism, sincere, organised, well-prepared, evidence-based practice, scholarly activity and listening skills (Buchel & Edwards, 2005). A recent study found that being aware of the learning needs of students could be the 16^th trait a CE should possess (Francis et al., 2016). Considering recent shift towards the use of the Entrustable Professional Activity framework for work place competency review at student and professional levels (Kwok et al., 2023), it is important to consider which top five traits are considered core competence of a CE as defined by the pinnacle of entrustment at Level 5.

Perspectives of critical traits of an effective CE might differ between students, CEs and academic educators (AEs). Students in early years of undergraduate study and CEs identified similar top five traits of an ideal CE (Cross, 1995). In the study by Cross (1995), CEs should be approachable, possess good communication skills, be knowledgeable and demonstrate enthusiasm in teaching, including clinical competency (Cross, 1995). Competency of CEs was similarly desired among academic educators, alongside “being approachable” and “good communicator”, whereas two differences were noted as compared to CEs and students, which were “interested in learning process of the students” and “concerned about patient care” (Cross, 1995). It is possible that among the Asian population, competency of the CEs would be similarly desired. Recent studies might have considered different trait terms as compared to the classic study, for instance, “learning process” was likely subsumed under “learning needs” (Francis et al., 2016). It is unclear whether the Asian population will present with different trait ranking as compared to Western population.

Therefore, this study was undertaken to identify the top five essential traits of an effective CE. In view of notable differences between perspectives of students and AEs in past literature, this study explored similarities and differences of perspectives between students and AEs. In addition, we explored what gaps might exist in developing CEs. In this study, the AEs, all of whom used to be CEs, could provide useful and important inputs on the traits required of a CE. The findings would then provide directions for future policy development pertaining to clinical education and professional supervision.

II. METHODS

A. Study Design

A survey was conducted through convenience sampling. The study was conducted between March to December 2022. The study was approved by the Institutional Review Board of the Singapore Institute of Technology (SIT-IRB-2022031).

B. Participants

Students in the third year of undergraduate physiotherapy programme and AEs of the physiotherapy faculty (as of April 2022) were involved in the survey. Large variations in response rate exists in convenience sampling, so we projected at least 50% response rate for the students and also the AEs. Students who had withdrawn from the programme were excluded. Associate faculty educators were excluded because they were only involved in ad hoc teaching sessions. Informed consent was sought at the beginning of the survey and digitally recorded.

C. Procedures

The survey described the 16 traits that contribute to an effective CE, which provided participants with knowledge about each trait, which was externally validated by five physiotherapists from the public and private sectors with experience ranging from eight to 25 years. The demographic data included student identification numbers to ensure that each response was unique. Thereafter, the data was de-identified to ensure anonymity during data analysis. As the sample size for the AEs was small, no personal data was collected to ensure anonymity. A survey invitation including a standardised survey link was distributed via email to all eligible participants. Participants were then asked to rank the 16 traits from 1 to 16, with 1 being the most important trait and 16 being the least important. Survey items were not randomised. The online survey was hosted on the secured Qualtrics Survey System (Qualtrics XM, 333 W. River Park Drive, Provo, UT 84604 USA). Student participants were first notified via email, followed by four reminder notifications, two months apart, via the cohort group chat on WhatsApp. AEs were first notified via email, followed by two reminder notifications via WhatsApp three months apart.

D. Statistical Analysis

Data analyses were performed using IBM SPSS Statistics for Windows, version 26 (IBM Corp., Armonk, N.Y., USA). Continuous variables are presented with means (standard deviations) and categorical variables with counts (percentages). Friedman test was used to analyse the mean rank of each trait. Then Kendall’s coefficient of concordance (Kendall’s W) was used to test the level of agreement among students for the mean ranks of the traits, as well as among AEs. The level of agreement was interpreted based on common recommendation and should yield statistical significance at p < 0.05.

III. RESULTS

A total of 81 students completed the survey (response rate of 46.6%) and a total of five AEs completed the survey (response rate of 41.7%). The average age, mean (standard deviation), of the students is 24.5 (4.1) years, and 52 of them are female (64.2%). The traits were ranked and presented in Table 1. Among the top five traits, there were four similar traits found between students and AEs, which were clinical competency / knowledge, clarity, feedback skills, and enthusiasm. The students ranked “non-judgemental” within the top five traits, whereas the AEs identified “awareness of students’ learning needs”. If the top six traits were considered, both students and AEs yielded similar traits. In the traits ranking, the students showed fair level of agreement, approximating closely to moderate agreement, Kendall’s W = 0.38, p < 0.001, while AEs showed moderate level of agreement, Kendall’s W = 0.51, p = 0.001.

Submitted: 19 December 2023
Accepted: 26 April 2024
Published online: 1 October, TAPS 2024, 9(4), 50-56
https://doi.org/10.29060/TAPS.2024-9-4/SC3194

Craig S. Webster¹, Jack Forsythe², Antonia Verstappen¹, Phillippa Poole³, Tim Wilkinson⁴ & Marcus A. Henning¹

¹Centre for Medical and Health Sciences Education, School of Medicine, University of Auckland, Auckland, New Zealand; ²Auckland District Health Board, Te Whatu Ora, Auckland, New Zealand; ³School of Medicine, University of Auckland, Auckland, New Zealand; ⁴Education Unit, University of Otago, Christchurch, New Zealand

Abstract

Introduction: A valid, longitudinal approach is critical for service planning in healthcare and to understand career choice in medical graduates.

Methods: We quantitatively analysed self-reported influences underlying career choice in a national cohort of medical graduates over the first five years of their careers. Participants rated career influences on importance across 26 items using a 5-point Likert scale (1=not at all, 5=a great deal).

Results: We included 659 New Zealand medical graduates (mean 25.4 years old, 376 F, 283 M) from the University of Auckland and the University of Otago, graduating in 2012 and 2013 (85% response rate). Responses were linked longitudinally over their post-graduate years 1, 3 and 5, and underwent principal component analyses. At graduation the factor rated as the most important in career choice had a mean (SD) item score of 3.9 (0.7) and comprised: Medical School Experiences; Specialty Experience; Mentors; and Self-Appraisal – consistent with graduates securing initial employment. Factors which explained the most variance in career choice over the five years after graduation indicated that the costs of medical school and further training were consistently rated as the least important in career choices, while flexibility in working hours were consistently rated as the most important. Factors remained relatively stable over time, showing variation in scores of only a median of 0.5 Likert points, indicating further opportunities for career choice research.

Conclusion: Our results regarding costs of medical training are reassuring, and suggest that greater flexibility in working hours may attract graduates to underserved specialties.

Keywords:           Medical Education, Career Choice, Career Influences, Cost, Debt, Measurement, Medical Graduates, National Longitudinal Study, Working Hours, Work Culture

I. INTRODUCTION

A common strategic aim of medical schools throughout the world is to supply the range of graduates who will best meet the healthcare needs of their communities (Gorman, 2018). However, fulfilling this aim is far from straight-forward, with perhaps the most critical difficulty involving understanding the influences that underlie career choice in medical graduates and how these vary over time.

The financial burden of completing medical school and further training has been one of the most widely studied influences underlying career choice, with suggestions that rising costs may encourage graduates to pursue specialties perceived to be more highly paid, often in cities, thus undersupplying primary healthcare and rural locations. However, in New Zealand medical graduates are otherwise free to choose their preferred career path and choices may be influenced by many things other than training costs and debt, including personal interest, employment conditions, specialty availability and lifestyle preferences (Webster et al., 2020; Webster et al., 2017).

Therefore, in the following, we analyse the self-reported influences underlying career choice in a national cohort of medical school graduates during the first five years of their careers using quantitative data drawn from the existing longitudinal Medical Schools Outcomes Database (MSOD).

II. METHODS

The MSOD project is a bi-national longitudinal questionnaire study that aims to improve healthcare delivery in Australia and New Zealand (Poole et al., 2019). At graduation and in postgraduate years (PGY), graduates are asked to specify their preferred area(s) of medicine, and complete a schedule of influencing items, indicating the degree to which each was important in their choice, using a 5-point Likert scale with anchors of 1 (not at all) to 5 (a great deal) – see Supplementary Table 1 for full question set.

A. Data Analysis

We conducted a series of principal component analyses (SPSS v27, IBM Corporation, New York) of the responses to the 26 influencing item questions at each time point to identify factors within responses, and describe them over time.

III. RESULTS

Data from a national cohort of 659 New Zealand medical school graduates who had graduated from the University of Auckland and the University of Otago in 2012 and 2013 were included. The response rate for completed questionnaires in the Exit cohort was 85% – representing a sampling margin of error of only 1% at the 95% level of confidence. The mean (SD) age of participants in the cohort was 25.4 (2.7) years, with a higher proportion of female graduates (376 F vs 283 M). Over the next five years, this Exit cohort self-reported on the same set of influences underlying career choice at PGY1, PGY3 and PGY5 – maintaining a response rate between 53% and 56%, and a sampling margin of error of 3%.

We used conventional settings during analysis, comprising varimax rotation and suppression of loadings below 0.3. The Kaiser-Meyer-Olkin measure of sampling adequacy across time points demonstrated a median (range) of 0.77 (0.75 to 0.82), indicating distinct and reliable factors at each time point. In addition, Bartlett’s test of sphericity was highly significant at each time point, (2338<c²<3498, p<0.0001), demonstrating correlation with little redundancy in items (Kaiser, 1974).

Submitted: 12 December 2023
Accepted: 25 March 2024
Published online: 1 October, TAPS 2024, 9(4), 40-49
https://doi.org/10.29060/TAPS.2024-9-4/SC3254

Vivien Lee¹, Jeffrey Jiang¹, Anna Szücs¹, V Vien Lee¹, Low Si Hui², Faith Teo¹, Jose M Valderas¹ & Victor Loh¹

¹Division of Family Medicine, Yong Loo Lin School of Medicine, National University of Singapore, Singapore; ²National University Polyclinics, Singapore

Abstract

Introduction: Globally, persons with disabilities (PWD) face structural and social barriers to inclusive healthcare. Medical schools, as crucibles of medical professional identity formation, have the responsibility to foster person-centredness toward all patients, including PWD, among her graduates. We co-designed with PWD a “Communications with Persons with Disabilities” workshop and evaluated its impact.

Methods: The workshop enlisted PWD as patient educators, occurred within the third-year undergraduate Family Medicine posting, and was designed to positively impact communications skills and attitudes toward PWD. Students (n=135) were surveyed pre- and post- workshop following a mixed-methods approach (demographic data, Attitude Towards Disabled Persons (ATDP)-O scale, feedback questions, and post-workshop free reflections). Descriptive analysis was used for demographic and feedback questions, and thematic analysis for reflections. Paired t-test was used to evaluate change in ATDP-O scores.

Results: There were 69 survey respondents (51.11%). Most students agreed that communications training with PWD could be strengthened in medical school, and that the workshop was relevant to their future medical career (n=68, 98.55%). Attitudes towards PWD significantly improved after the workshop (ATDP-O change: +9.29 points (12.7%), p <0.001). Student reflections included attitudes, such as the importance of seeing the person beyond the disability, and a call for action towards inclusivity and accessibility of care for PWD.

Conclusion: Students’ attitudes were positively affected by involving PWD in the curriculum. Further research is needed for assessing the impact of how a longitudinal PWD curriculum could affect medical trainees and improve social inclusiveness in healthcare practice.

Keywords:           Medical Education, Communication, Persons with Disabilities, Disability, Social Inclusiveness, Student Attitudes, Person Centred Care, Active Learning

I. INTRODUCTION

Globally, persons with disabilities (PWD) face structural and social barriers to inclusive healthcare. In Singapore, few are comfortable with direct personal interaction with PWD (National Council of Social Service Singapore, 2019).

With respect to medical education, different disability curriculum from didactic lectures to longitudinal patient experiences have been described (Ali et al., 2023). Overall evidence suggests that more frequent and positive contact with PWD increased positive attitudes (Ali et al., 2023; National Council of Social Service Singapore, 2019).

Medical schools, as crucibles of medical professional identity formation, have the responsibility to foster person-centredness toward all patients, including PWD, among her graduates. In July 2022, the National University of Singapore’s Division of Family Medicine developed with PWD, piloted and assessed a four-hour experiential workshop for third-year medical students called “Communication with PWD” in July 2022.

The four-hour workshop was prefaced by pre-event reading materials. A plenary started on site learning, followed by fishbowl discussions where student volunteers took turns to perform a clinical consultation with PWD who enacted scripted role-plays. There was a ratio of one tutor and PWD to a group of ten to fifteen students. Each student would encounter three broad categories of disability (vision, hearing, mobility) during the fishbowl sessions. Students interacted with PWD after each role play. Reflections and debrief by tutors would close the workshop.

This study aimed to evaluate the workshop’s impact on students’ attitudes towards PWD for: (Q1) differences in pre-workshop attitudes towards PWD between male and female medical students; (Q2) within-person changes in students’ overall attitudes towards PWD pre-post workshop; (Q3) associations between pre-workshop attitudes towards PWD and personal experience with PWD; and (Q4) exploration of how the workshop has affected students’ attitudes towards caring for PWDs.

II. METHODS

A. Study Design

This study had a mixed methods design including a pre- and post-intervention survey (quantitative), and reflections (qualitative). Students attending the 3^rd and 4^th Family Medicine Rotation (January to March 2023) of Academic Year 2022/23 were invited to participate in the study. The questionnaire was administered in person. Students completed the questionnaires independently. A waiver of consent was applied for reflections. 

B. Survey

The pre-workshop questionnaire comprised demographic information and the 20-item Attitude Towards Disabled Persons (ATDP)-O scale (Yuker et al., 1960). The questions were scored on six-point Likert scale from “I disagree very much” (-3) to “I agree very much” (+3). The post-workshop questionnaire comprised the ATDP-O scale and feedback questions.

We used the original ATDP-O form as it had been used in other studies on medical students (Arabi et al., 2021) and was deemed reliable (reliability coefficient .80) and valid by prior psychometric analyses (Yuker et al., 1960).  Performance in local context was assessed through cognitive testing with five randomly selected medical students who were classmates of study participants and excluded from the study sample.

C. Reflections

Student reflection sheets included a single open-ended statement: “Please pen a question or a reflection you’d like to share from the session today”.  Reflections were transcribed verbatim and de-identified before analysis. All reflections were used to limit selection bias.

D. Data Analysis

Numeric data was analysed by R software (version 4.31). Internal consistency of the scale in our study sample was measured with Cronbach’s alpha. We used two samples independent unpaired t-test to evaluate Q1 (differences in baseline ATDP-O scores between males and females), two samples paired t-test to evaluate Q2 (change in pre-post workshop ATDP-O scores), and two samples independent unpaired t-tests, Welch’s t-tests and Spearman’s correlation tests to evaluate Q3 (difference in baseline ATDP-O scores across different types/levels of contact with PWD). In the evaluation of Q3, two samples independent unpaired t-test was used when two groups have similar sample sizes or similar variances while Welch’s t-test was used when there’s unequal variances or disparate sample sizes between two groups. Spearman’s correlation was used to analyse associations with ordinal variables.

Student reflections were analysed thematically. Four researchers (VLE, VVL, AS, FT) reviewed the data and generated initial codes independently. Multiple discussions were conducted to explore different perspectives and increasingly develop codes. Based on codes generated, potential themes and subthemes were developed over several iterations. Final themes and subthemes were reviewed in relation to the entire dataset.

III. RESULTS

Of 135 students invited, 69 students consented to the study and completed both the pre- and post-workshop questionnaire. The results are summarised (Table 1a and 1b).

Cognitive testing confirmed that students had no difficulties understanding questions nor suggested any edits for clarification. The internal consistency of the scale was α = 0.72 (pre-workshop) and α = 0.78 (post-workshop).

(Q1) Female gender had a significantly higher baseline ATDP-O scores (p = 0.028).

(Q2) There was a significant improvement of 9.29 points (12.7%) in the ATDP-O score post workshop (p <0.001).

(Q3) Previous experience of training to care for PWD (p<0.001) and previous experience caring for PWD (p = 0.033) had significantly lower baseline ATDP-O scores.

 

Results	Number (%)	ATDP-O baseline score Mean (SD)			Statistics
Overall	N=69	72.71 (10.46)
By age (Mean = 21.59, SD = 0.96)					Pearson’s r(67) = – 0.02, p = .857
By gender
Female	37 (53.62%)	75.32 (8.61)			t(67)  = 2.30, p = .025*
Male	32 (46.38%)	69.69 (11.69)
By ethnicity
Chinese	56 (81.16%)	73.18 (10.20)			F(2, 66) = 1.64, p = .201
Indian	12 (17.39%)	69.33 (11.18)
Others	1 (1.45%)	87.00 (.)
Self-reported disability
No	66 (95.65%)	72.74 (10.59)			t(2) = 0.14, p = .900
Yes	3 (4.35%)	72.00 (8.89)
Self-reported training to care for PWD
No	67 (97.10%)	73.06 (10.42)			t(7) = 7.45, p < .001*
Yes	3 (4.35%)	61.00 (1.41)
Self-reported frequency of care for PWD (ranked variable ranging from never = 1 to often = 4)
Never	34 (49.28%)	75.7 (9.5)			Spearman’s rho(67) = – 0.26, p = .033*
Rarely	22 (31.88%)	69.8 (11.8)
Sometimes	12 (17.39%)	69.5 (9)
Often	1 (1.45%)	75 (.)
Self-reported knowledge of PWD who is important to them
No	53 (76.81%)	73.57 (10.95)			t(67) = 1.24, p = .219
Yes	16 (23.19%)	69.88 (8.34)
Self-rated likelihood of pursuing career in Family Medicine (ranked variable ranging from very unlikely = 1 to very likely = 5)
Very likely	9 (13.04%)	72.1 (10)			Spearman’s rho(67) = 0.03 , p = .803
Likely	18 (26.09%)	73.2 (11.5)
Undecided	37 (53.62%)	73.5 (10.3)
Unlikely	4 (5.80%)	63.5 (8.9)
Very unlikely	1 (1.45%)	78 (.)
Do you feel your education about persons with disabilities in medical school has been adequate so far?
Agree	1 (1.45%)		–		–
Neither disagree nor agree	19 (27.54%)		–		–
Disagree	42 (60.87%)		–		–
Strongly disagree	7 (10.14%)		–		–
Would you find it important to include education and training on persons with disabilities?
Strongly agree	23 (33.33%)			–	–
Agree	39 (56.52%)			–	–
Neither disagree nor agree	5 (7.25%)			–	–
Disagree	1 (1.45%)			–	–
Strongly disagree	1 (1.45%)			–	–
Do you feel that the education provided during the Communications with Persons with Disabilities workshop was relevant to your future career as doctors?
Strongly agree	45 (65.22%)			–	–
Agree	23 (33.33%)			–	–

Table 1a. Baseline ATDP-O scores and curriculum feedback

 

Results	ATDP-O score Mean (SD)	Pre-Post workshop comparison statistics
Descriptive statistics
Pre-workshop ATDP-O score	72.71 (10.46)	t(68)  = 8.69, p < .001*
Post-workshop ATDP-O score	82.00 (11.46)
Difference in ATDP-O scores
Overall sample	9.29 (8.88)	t(68)  = 8.69, p < .001
By age (Mean = 21.59, SD = 0.96)		Pearson’s r(67) = – 0.06, p = .643
By gender
Female	8.62 (9.33)	t(67) = – 0.67, p = .505
Male	10.06 (8.41)
By ethnicity
Chinese	9.02 (9.45)	F(2, 66) = 0.15, p = .859
Indian	10.33 (6.21)
Others	12.00 (.)
Self-reported disability
No	9.50 (8.06)	t(2) = 0.36, p = .752
Yes	4.67 (23.12)
Self-reported training to care for PWD
No	9.09 (8.92)	t(1) = – 2.17, p = .230
Yes	16.00 (4.24)
Self-reported frequency of care for PWD (ranked variable ranging from never = 1 to often = 4)
Never	8.8 (6.1)	Spearman’s rho(67) = 0.05, p = .678
Rarely	11.5 (9.1)
Sometimes	9.4 (10.8)
Often	-22 (.)
Self-reported knowledge of PWD who is important to them
No	10.72 (7.61)	t(67) = 2.52, p = .014*
Yes	4.56 (11.20)
Self-rated likelihood of pursuing career in Family Medicine (ranked variable ranging from very unlikely = 1 to very likely = 5)
Very likely	10.4 (9.2)	Spearman’s rho(67) = 0.07, p = .547
Likely	10.9 (7.8)
Undecided	7.9 (9.5)
Unlikely	11.5 (9.3)
Very unlikely	11 (.)

Table 1b. Difference in ATDP-O scores post workshop

Most students found the workshop a positive experience and reported feelings of thankfulness and inspiration towards patient educators. Four themes emerged from the students’ post-workshop reflections, of which we highlight those related to attitudes – Humanizing mindsets towards PWD (Theme 1) and Call for action towards inclusivity and accessibility (Theme 2). The other themes were Knowledge and Skills, as students included knowledge and communication skills they acquired during the workshop (Appendix 1 and 2).

A. Theme 1 – Humanising Mindset towards Persons with Disabilities

Students expressed attitudes of empathy, sensitivity, and humanising care through viewing PWD as individuals beyond their disabilities. A student reflected

“Being able to speak to members of these respective communities helped me to understand how empowered PWD can be… It is essential that [PWD] are treated as exactly that, [and] not characterised by their disabilities.”

On providing care for PWD, students noted the distinction between empathy and sympathy, and highlighted the importance of empathy instead of pity. As summed up by a student,

“Not to sympathise but to empathise. Not to see them and pity them”

B. Theme 2 – Call for Action towards Inclusivity and Accessibility

Concurrent with attitudes expressed in the reflections, students discussed the need to improve inclusivity and accessibility from an individual and societal perspective. One student wrote,

“As doctors, we need to try our best to alleviate their disease burden, and at the same time, make their journey to accessing healthcare more seamless with less barriers.”

Beyond self-improvement, students were aware of the need to shift their own perspectives of disability from a medical to a social model. Students also reflected the need for greater national efforts towards societal inclusivity for PWD. A student reflected,

“I think policymakers and organisations should periodically involve PWD to [better] take their needs into consideration as they have the right to integrate into our society just as much as everyone else.”

IV. DISCUSSION

Our study showed positive change in attitudes towards PWD among medical students after a workshop with improvement of 9.29 points in the ATDP-O score (Table 1b), reinforced by their post-workshop reflections. To our knowledge, this is the first study in Asia evaluating an educational intervention focusing on communications skills training with PWD. Based on student reflections, the communications workshop was effective in developing knowledge and communication skills, prompting the key attitudes of humanizing care, and a call for action towards societal inclusivity and accessibility towards PWD.

We intentionally had PWD co-designing the workshop and playing key roles as patient educators. Student reflections support these curriculum design decisions. Our work coheres with literature that a contact-based approach with PWD would be impactful: reducing anxiety and improving empathy levels, rather than more theoretical approaches to promote attitudes towards PWD (Arabi et al., 2021).

Our study corroborates the widely reported result that female gender positively influenced baseline attitudes towards PWD in medical students (Satchidanand et al., 2012). A possible explanation is females having a propensity to exhibit more empathetic feelings toward others. There is a greater change in attitudes in males than females after the workshop, but this change is not significant.

In contrast to prior research that consistently associated increased contact with PWD with more positive attitudes (Satchidanand et al., 2012), we found that having previous experience with PWD influenced baseline ATDP-O scores negatively (Table 1a). Moreover, there was a significantly smaller change in ATDP-O scores if they knew a PWD who is important to them (Table 1b). A possible explanation is “caregiver fatigue” due to unavailability of support to allow adequate care for PWD (Arabi et al., 2021). The type of previous experience may thus be key factors in shaping current attitudes towards PWDs.

This study has several limitations. We cannot determine if positive change in attitudes post-workshop would translate into changes in clinical practice in the future. The survey was not compulsory for students, potentially leading to selection bias. We tried to overcome the selection bias by including all student reflections.

V. CONCLUSION

There was significant improvement in students’ attitudes towards PWD post-workshop, as shown through the ADTP-O score and reflection analysis. Medical students benefit particularly from the fishbowl discussions which allows students to interact with persons with disability, growing our future generations of healthcare professionals who humanise care.

Notes on Contributors

Author VLE conceptualised the study, developed the methodology, conducted the investigation, did the formal analysis and visualization of data, wrote the original draft, reviewed and edited the manuscript. Author JJ conceptualized the study, developed the methodology and original draft, reviewed and edited the manuscript. Author AS developed the methodology, did the formal analysis of data, reviewed and edited the manuscript. Author VVL did the formal analysis of data, wrote the original draft, reviewed and edited the manuscript. Author LSH did the formal analysis and visualization of data. Author FT did the formal analysis, reviewed and edited the manuscript. Author JMV supervised the conceptualization and methodology of the study, reviewed and edited the manuscript. VLO supervised the conceptualization of the study, developed the methodology, reviewed and edited the manuscript. All authors have read and agreed to the published version of the manuscript.

Ethical Approval

The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Institutional Review Board of the National University of Singapore on 11 November 2022 (NUS-IRB-2022-608).

Data Availability

Quantitative data repository can be freely accessed at: https://doi.org/10.6084/m9.figshare.24013134  

Qualitative data repository can be freely accessed at: https://doi.org/10.6084/m9.figshare.24051195  

Funding

There was no funding for this research study.

Declaration of Interest

Authors VLE, JJ and VLO are tutors in the “Communications with Persons with Disabilities” workshop. There are otherwise no other conflicts of interest.

References

Ali, A., Nguyen, J., Dennett, L., Goez, H., & Rashid, M. (2023). A scoping review for designing a disability curriculum and its impact for medical students. Canadian Medical Education Journal, 14(3), 75–86. https://doi.org/10.36834/cmej.74411

Arabi, H., Adarmouch, L., & Ahmed Eladip, G. (2021). The assessment of student doctors’ attitude towards disabled people after teaching them a module. Acta Bio-Medica: Atenei Parmensis, 92(2), e2021059. https://doi.org/10.23750/abm.v92i2.9547

National Council of Social Service Singapore. (2019). Public Attitudes Towards Persons with Disabilities 2019. https://www.ncss.gov.sg/docs/default-source/ncss-publications-doc/pdfdocument/public-attitudes-towards-persons-with-disabilities-2019-infographics.pdf

Satchidanand, N., Gunukula, S. K., Lam, W. Y., McGuigan, D., New, I., Symons, A. B., Withiam-Leitch, M., & Akl, E. A. (2012). Attitudes of healthcare students and professionals toward patients with physical disability: A systematic review. American Journal of Physical Medicine & Rehabilitation, 91(6), 533–545. https://doi.org/10.1097/PHM.0b013e3182555ea4 

Yuker, H. E., Block, J. R., & Campbell, W. J. (1960). A scale to measure attitudes toward disabled persons. In Human Resources Study Number 5. Human Resources Foundation.

*Vivien Lee
1E Kent Ridge Rd,
Singapore 119228
Email: vivienl@nus.edu.sg

Submitted: 3 July 2023
Accepted: 18 June 2024
Published online: 1 October, TAPS 2024, 9(4), 61-64
https://doi.org/10.29060/TAPS.2024-9-4/SC3071

Victoria Scudamore, Sze Yi Beh, Adam Foster & Michaela Goodson

School of Medicine, Newcastle University Medicine Malaysia, Malaysia

Abstract

Introduction: This study compares online and in-person delivery of a weekly clinical reasoning seminar for fourth-year medical students at a Malaysian medical school. During the easing of COVID-19 pandemic restrictions, the initial eight seminars took place online, followed by eleven in-person seminars. This study looks at student preference for online or in-person delivery and how these reasons differ due to gender.

Methods: An online questionnaire was sent to fourth-year medical students after returning to in-person seminars. The response rate was 60/128 (46.88%) and the data was analysed using SPSS software.

Results: 65% of students preferred in-person seminars and a larger proportion of female students (71.43%) preferred in-person sessions compared to male students (50.00%), although this was not statistically significant (p=0.11). A significantly larger proportion of female students preferred in-person seminars for the following reasons compared to male students: enjoyment (p=0.041), developing history-taking skills (55.56%) and for formulating differential diagnoses (p=0.046). Students were asked whether online or in-person seminars were most appropriate for eighteen reasons, they felt in-person seminars were most appropriate for 16/18 of these reasons.

Conclusions: More students preferred in-person clinical reasoning seminars and a higher proportion of these students were female. A significantly larger proportion of female students felt in-person seminars were better for; enjoyment and for developing history-taking skills and formulating differential diagnoses, compared to male students. The students preferred online seminars for home comforts and ease of travel, but they preferred in-person seminars for the other 16/18 reasons listed including all reasons linked to learning skills and interreacting with others.

Keywords:            Online Teaching, In-person Teaching, Gender, Clinical Reasoning, Medical Students

I. INTRODUCTION

Fourth-year medical students at Newcastle Medical School Malaysia attend weekly clinical reasoning seminars as part of their ‘Clinical Decision Making’ (CDM) module. Each week of CDM covers a different medical speciality and students attend a seminar where the group works through a presentation with patient cases and they discuss how to diagnose, investigate and manage the patient. The sessions are attended by eleven students and the groups remain the same throughout the year. In 2021-22 the initial eight seminars took place online using Zoom video conferencing software and as COVID-19 restrictions eased in Malaysia the final eleven sessions took place in-person.

The academic performance of students undertaking online and in-person clinical reasoning seminars has been researched and third-year medical student academic performance was comparable in both settings (Babenko et al., 2022). However, there is currently no research regarding medical student preference for online or in-person delivery of clinical reasoning seminars. Medical student preference for online or in-person delivery of all parts of the curriculum has been analysed and second-year medical students at a US medical school had a preference for online lectures and there was a correlation between these students and those who felt online lectures reduced stress (Altaf et al., 2022).

A cohort study analysed participation in a teaching programme for US graduate physicians and this showed female students asked and answered less questions during in-person sessions compared to online sessions (Cromer et al., 2022). The results of this study are contrasting with my observations whereby female students participated less in online seminars and their participation increased when seminars returned to an in-person setting. This could be due to differences in the research environments or due to the group of observed students being small with less statistical significance.

My first research question was to understand medical student preference and reason for preference of online or in-person delivery of clinical reasoning seminars. My second research question was to establish if student preference differed due to gender and why.

II. METHODS

The data was collected using survey methodology with a self-developed questionnaire made using Microsoft forms. The questionnaire was emailed to all fourth-year medical students after they had experienced both session deliveries. Students were provided with a consent form and informed the research project was optional and were asked to provide voluntary consent before participating. Participants were informed they could withdraw from the project at any time up until the data was anonymised during data collection.

The survey response rate was 60/128 (46.88%), the low participation numbers are likely due to the data collection being optional and undertaken in the students own time. This could lead to a nonresponse bias, as it is likely the more engaged students participated and students with less motivation who did not participate may have responded differently. The data was analysed using SPSS software. Chi-squared tests were used to cross-tabulate the results and to calculate p-values to indicate data with statistical significance.

III. RESULTS

Overall 65% of students preferred in-person seminars and 71.43% (30/42) of female students preferred in-person sessions compared to 50.00% (9/18) of male students (p=0.11). The students were asked if they felt online or in-person seminars were best for eighteen different reasons (see table 1). There were three statistically significant reasons female students preferred in-person sessions more than male students (p<0.05). These were Enjoyment (p=0.041), developing history-taking skills (p=0.011) and formulating differential diagnoses (p=0.046).

The students felt in-person sessions were most appropriate for 16/18 of the reasons listed in (table 1). The reasons with the highest proportion of students feeling in-person were the most appropriate were; interaction with friends (95.00%), interaction with the facilitator (91.67%), and developing clinical reasoning skills (91.67%). There were only two reasons students felt online sessions were most appropriate, these were home comforts (98.33%) and ease of travel (91.67%).

Original data can be accessed in Figshare at https://doi.org/10.6084/m9.figshare.23616627.v1 and https://doi.org/10.6084/m9.figshare.23616630.v1

Results are ranked from reasons with the highest proportion of students thinking in-person was most appropriate for that reason. P-values have been calculated to establish if there is statistical significance between the results for male and female students, significant results are highlighted in bold.

	All students	Female students	Male students	P-value
Interaction with friends	57/60 (95.00%)	39/42 (92.86%)	18/18 (100.00%)	0.245 (>0.05)
Interaction with the facilitator	55/60 (91.67%)	39/42 (92.86%)	16/18 (88.89%)	0.610 (>0.05)
Developing clinical reasoning skills	55/60 (91.67%)	39/42 (92.86%)	16/18 (88.89%)	0.610 (>0.05)
Learning from the facilitator	51/60 (85.00%)	35/42 (83.33%)	16/18 (88.89%)	0.581 (>0.05)
Flow	47/60 (78.33%)	33/42 (78.57%)	14/18 (77.78%)	0.945 (>0.05)
Developing history-taking skills	46/60 (76.67%)	36/42 (85.71%)	10/18 (55.56%)	0.011 (<0.05)
Developing knowledge	45/60 (75.00%)	33/42 (78.57%)	12/18 (66.67%)	0.329 (>0.05)
Ease of sharing opinions	45/60 (75.00%)	33/42 (78.57%)	12/18 (66.67%)	0.329 (>0.05)
Enjoyment	44/60 (73.33%)	34/42 (80.95%)	10/18 (55.56%)	0.041 (<0.05)
Learning from peers	41/60 (68.33%)	28/42 (66.67%)	13/18 (72.22%)	0.672 (>0.05)
Formulating differential diagnoses	41/60 (68.33%)	32/42 (76.19%)	9/18 (50.00%)	0.046 (<0.05)
Interpreting clinical data	40/60 (66.67%)	29/42 (69.05%)	11/18 (61.11%)	0.550 (>0.05)
Formulating management plans	39/60 (65.00%)	29/42 (69.05%)	10/18 (55.56%)	0.315 (>0.05)
Better for mental health	36/60 (60.00%)	26/42 (61.90%)	10/18 (55.56%)	0.645 (>0.05)
Developing communication skills	35/60 (58.33%)	22/42 (52.38%)	13/18 (72.22%)	0.153 (>0.05)
Interpreting investigations	35/60 (58.33%)	27/42 (64.29%)	8/18 (44.44%)	0.153 (>0.05)
Ease of travel	5/60 (8.33%)	5/42 (11.90%)	0/18 (0.00%)	0.126 (>0.05)
Home comforts	1/60 (1.67%)	1/42 (2.38%)	0/18 (0.00%)	0.509 (>0.05)
Table 1. The proportion of medical students who felt in-person sessions were the most appropriate for the listed reasons.

IV. DISCUSSION

Overall, more students in our cohort preferred in-person clinical reasoning seminars and a larger proportion of these students were female than male, however, the difference in preference based on gender did not show statistical significance. This could be due to a smaller cohort of male respondents (18/60) compared to female respondents (42/60). However, even in a study of 488 medical and dental students there was no significant difference in preference for online or in-person delivery when asked about all sessions in the curriculum (Al-Azzam et al., 2020). A larger sample size of medical students will need to be analysed to establish if gender significantly influences student preference for online or in-person delivery of clinical reasoning seminars.

Students felt in-person seminars were better for 16/18 of the listed reasons. This included all reasons pertaining to interaction with other students and staff and all reasons regarding learning a variety of skills. The only two reasons students felt online sessions were better were ease of travel and home comforts. Therefore, this data suggests the only reasons the medical students preferred online seminars were due to the convenience of the setting, and they felt the learning and interaction were superior in in-person seminars.

Of these eighteen reasons, three reasons showed a significant difference in response based on gender, with more female students preferring in-person seminars for the following reasons: enjoyment, development of history-taking skills and formulating differential diagnoses. History-taking and formulating differential diagnoses are more commonly undertaken by doctors within in-person environments. Therefore, female medical students may have a stronger preference for learning skills in the same setting they will be undertaken in when they are doctors.

This study helps to identify the components of clinical reasoning seminars male or female students prefer to undertake online or in-person. Future research could try to identify the reasons for these preferences and to establish if female students have a stronger preference for learning a skill in the same environment it would be undertaken in when they become a doctor.

This research will have most transferability to educators designing clinical reasoning modules to undergraduate students. It may also have some transferability to any undergraduate seminars and to postgraduate medical education. Also, understanding the environment each gender prefers to learn in and why, could help to designing future educational programmes. Especially if these programmes have previously shown differing participation or attainment based on gender.

V. CONCLUSION

In conclusion, students preferred in-person clinical reasoning seminars compared to online seminars and a higher percentage of female students preferred in-person compared to male students, although this was not statistically significant. Students had the strongest preference for in-person sessions due to interaction with friends and the facilitator and for developing clinical skills. Students had the strongest preference for online sessions due to home comforts and ease of travel. Female students preferred in-person seminars compared to male students for the following statistically significant reasons: enjoyment, developing history-taking skills and formulating differential diagnoses.

Notes on Contributors

Dr Victoria Scudamore was involved in data collection, data analysis and wrote this manuscript in full.

Dr Sze Yi Beh was involved in data collection and data analaysis.

Dr Adam Foster was involved in data collection and data analysis.

Dr Michaela Goodson supervised and advised on data collection and analysis.

Ethical Approval

Research and ethics approval was granted by the research committee at Newcastle University Medicine Malaysia on 08/02/2022 (Approval number: 18547/2022).

Data Availability

The data used in this paper is available in the Figshare repository through the following links with associated DOI’s https://doi.org/10.6084/m9.figshare.23616627.v1 and https://doi.org/10.6084/m9.figshare.23616630.v1. The data is shared on the Figshare repository under the CC0 licence (no rights reserved) as there is no copywritten information included.

Funding

No additional funding was used to undertake this project.

Declaration of Interest

There are no potential conflicts of interest to declare.

References

Al-Azzam, N., Elsalem, L., & Gombedza, F. (2020). A cross-sectional study to determine factors affecting dental and medical students’ preference for virtual learning during the COVID-19 outbreak. Heliyon, 6(12), 4. https://doi.org/10.1016/j.heliyon.2020.e05704

Altaf, R., Kling, M., Hough, A., Baig, J., Ball, A., Goldstein, J., Brunworth, J., Chau, C., Dybas, M., & Jacobs, R. J. (2022). The association between distance learning, stress level, and perceived quality of education in medical students after transitioning to a fully online platform. Cureus, 14(4), 3. https://doi.org/10.7759/cureus.24071

Babenko, O., Ding, M., & Lee, A. S. (2022). In-person or online? The effect of delivery mode on team-based learning of clinical reasoning in a family medicine clerkship. Medical Sciences, 10(3), 41. https://doi.org/10.3390/medsci10030041

Cromer, S. J., D’Silva, K. M., Phadke, N. A., Lord, E., Rigotti, N. A., & Baer, H. J. (2022). Gender differences in the amount and type of student participation during in-person and virtual classes in academic medicine learning environments. JAMA Network Open, 5(1), 1. https://doi.org/10.1001/jamanetworkopen.2021.43139

*Dr Victoria Scudamore
Newcastle University Medicine Malaysia
No. 1 Jalan Sarjana 1,
Kota Limu, EduCity@Iskandar,
Iskandar Puteri, Johor, Malaysia, 79200
Email: victoria.scudamore@nhs.net

Submitted: 8 August 2023
Accepted: 9 January 2024
Published online: 2 July, TAPS 2024, 9(3), 50-54
https://doi.org/10.29060/TAPS.2024-9-3/SC3107

Chuu Ling Chan¹, Russell Lee², Lih Ing Goh¹, Nathanael Hao Kai Chong¹, Li Neng Lee² & Jun-Hong Ch’ng¹

¹Department of Microbiology and Immunology, Yong Loo Lin School of Medicine, National University of Singapore, Singapore; ²Department of Psychology, Faculty of Arts and Social Sciences, National University of Singapore, Singapore

Abstract

Introduction: We frequently associate microbes with infection, rarely expounding on their usefulness and importance to healthy development. For humanity to leverage these microbial “super powers”, learners from all backgrounds need to appreciate their utility and consider how microbes could help solve some of the most critical problems we face. However, learners are frequently uninterested or intimidated by microbiology. The card game “No Guts No Glory” was created to engage students by piquing curiosity and encouraging informal learning to change perceptions and advocate the value of microbes to good health.

Methods: Undergraduates from various faculties in the National University of Singapore were invited to play and give feedback on accessibility, engagement and self-reported learning gains.

Results: The game was well-received across disciplinary backgrounds with positive feedback (5-point scale) on game mechanics being fun (4.17±0.63), attractive artwork (3.83±1.00) and scientific snippets (3.79±1.04), positive re-playability (3.46±0.84), player engagement for those without foundational knowledge in microbiology (3.63±1.04), and usefulness of knowledge taught (3.54±1.10). Areas for improvement evidenced from feedback included unclear instructions (2.74±0.98), limited content taught (2.76±0.93), not generating interest to attend formal microbiome classes (2.88±1.17) and not prompting lifestyle changes (1.98±1.11).

Conclusion: This pilot study provided valuable insights from the target demographic, with concrete ideas on how to improve the educational potential of “No Guts No Glory”. Findings further lay the groundwork for the design of future instruments to objectively quantify learning gains from gameplay.

Keywords:           Game-based Learning, Microbiome, Microbiology, Card Game, No Guts No Glory

I. INTRODUCTION

Though responsible for infection and disease, microbes are also necessary for promoting and maintaining good health and are remarkably useful in many industries. Microbes are crucial and versatile tools which can be used to tackle some of the world’s most complex problems ranging from pandemics and climate change to sustainable foods and environmental remediation (Akinsemolu, 2018). Recognising the true potential of microbes could encourage their use in impactful problem-solving. However, students may perceive microbiology as a difficult subject and not associate microbes with anything positive.  To address this, we developed the card game “No Guts No Glory” that focuses on the positive aspects of microbes, particularly in the context of the gut microbiome which is closely tied to many aspects of our health and development (Lynch & Pedersen, 2016). We sought to engage learners from diverse disciplines by reducing the psychological barrier to learning microbiology, sparking curiosity and encouraging self-directed exploration.

Previous studies on card games, including one involving immunology (a related and equally complicated subject), have demonstrated the benefits of game-based learning (Barnes, 2022; Su et al., 2014): (a) games provide an attractive, innocuous entry point for individuals to explore complex subjects in a fun and enjoyable manner; (b) they remove barriers associated with traditional learning approaches, making it easier for learners to get started and actively participate; (c) additionally, game-based learning promotes active engagement; (d) through interactive gameplay, players become immersed in theoretical concepts which fosters deeper understanding of the subject and enhances retention.

In this study, we examined the impact of playing “No Guts No Glory” on participants’ interest in microbiology and garnered feedback for optimising game mechanics, instructions and artwork in preparation for future studies on the game’s impact on learning gains.

II. METHODS

Undergraduate students from various academic backgrounds were invited to participate in this pilot study on version 1 of the microbiome card game “No Guts No Glory”. Students were briefed on the details of the study and implied consent was given with their participation. Documentation of informed consent was waived as the collection of personal, sensitive information was kept to a minimum. A total of 41 participants were recruited – 29% from Medicine, 46% from Life Sciences, 20% from Psychology and 5% from other majors. Although not enforced, most participants played the game with others from the same discipline due to the way participants were recruited and how gameplay sessions were organised.

Participants first read through the game instructions and provided feedback on the instructions before playing two rounds of the game in groups of three or four. During the first round of gameplay, participants discussed and negotiated the rules among themselves based on what they had read, without input from the session facilitator. There was a break between the two rounds when the session facilitator highlighted deviations from the intended gameplay and answered questions about the game, before proceeding to the second round of gameplay. Participants then gave feedback via Qualtrics on their experience with the game. Throughout the two rounds of gameplay, the session facilitator also recorded observations of questions about instructions, disagreements about rules, deviations in gameplay and comments about game mechanisms, artwork, experiences and learning gains.

For quantitative feedback, participants rated the clarity and presentation of the instructions, the accessibility of the game, player engagement level and perceived learning gains, based on 5-point rating scales. Open-ended qualitative questions included: 1) suggestions to improve the instructions or the game, 2) elaboration on likely lifestyle/behavioural changes after playing and 3) key ideas they had learnt about the gut microbiome.

III. RESULTS

Quantitative feedback from participants after playing “No Guts No Glory” is summarised in Table 1 and qualitative feedback (individual comments and suggestions) is accessible at https://doi.org/10.6084/m9.figshare.23735211

Table 1. Participants’ quantitative feedback (5-point rating scales) on the microbiome card game “No Guts No Glory”

A. Instructions

Participants rated the ease of understanding and the presentation of the instructions near average (2.74±0.98 and 3.11±1.09 respectively). Qualitative feedback on the instructions revealed that many participants felt uncertain of game components, rules and mechanics (19 participants), that phrasing and overall wordiness of the instructions could be improved (14 participants), and that the instructions lacked emphasis on the goals of the game (3 participants). They suggested the need for more examples, visuals or even a demonstration video (10 participants) and reformatting the instructions (3 participants).

B. Game Accessibility and Player Engagement

Participants agreed that the length of one game was just right (3.02±0.42) and felt that the game was fun (4.17±0.63). They also had a good impression of the artwork (3.83±1.00) and scientific snippets included on the game cards (3.79±1.04), although 7 participants did not manage to read these snippets. Most participants responded positively regarding the replayability of the game, with a likelihood of playing the game if they owned it scoring 3.46±0.84, at an average frequency of 2.80±0.8. They perceived that players without any prior microbiome knowledge would be able to play and find the game engaging (3.63±1.04). Suggestions were given to enhance the game by changing the game mechanics to enrich learning (17 participants), improving the quality of game components (7 participants), including visual aids in the instructions (6 participants), and refining the artwork aesthetics (4 participants).

C. Experiences and Learning Gains

Gameplay helped participants to recognise the usefulness of microbiome knowledge (3.54±1.10), and to a lesser degree, connect topics (3.05±0.92) and raise curiosity (3.00±1.02). Participants who indicated more prior knowledge reported that the game was moderately effective in reinforcing existing content (3.19±1.05) while those with less prior knowledge indicated that the game was less effective in teaching content (2.76±0.93). In the qualitative feedback, suggestions for improving learning gains involved linking the scientific snippets found on the cards to gameplay (4 participants), and including a quiz component in the game mechanics (3 participants). Participants showed little interest in taking formal microbiome classes after gameplay (2.88±1.17), and only half (51.2%) indicated potential behavioural or lifestyle changes: 14 mentioned changes in their diet, two mentioned an increased curiosity in microbiome-related topics while one reflected on blindly chasing lifestyle/dietary trends.

D. Key Lessons Learnt by Students from Playing the Game

Drawing on open-ended feedback from participants, the top three ideas drawn from gameplay were the importance of the gut microbiome in health (8 out of 36 responses, 22%), how our microbiome is affected by multiple factors (19.4%), and the importance and definitions of pre/pro/syn-biotics (16.6%).

IV. DISCUSSION

Based on the feedback received, the game was enjoyable, with the inclusion of both attractive artwork and intriguing scientific snippets being crucial in generating interest. Participants acknowledged the value of the information presented in the game, which could inspire them to further explore specific areas of interest on their own. Participants also agreed that the game’s entry barrier was low enough, such that even individuals without a microbiology background could comprehend the gameplay and find it engaging. Positive replayability of the game could aid learning through increased exposure to information on the cards and instructions. Notably, the game’s effectiveness in reinforcing existing knowledge received higher ratings compared to its ability to teach new content in this study, aligning with similar findings published by others (Spandler, 2016; Su et al., 2014). While positive, these outcomes need to be viewed with caution as some of the students were likely to have been from classes taught by the researchers. Although feedback was anonymous and collected in the absence of their teachers, we acknowledge that this student-teacher/researcher relationship may have inadvertently introduced bias in this study.

This study also revealed four shortcomings of the game: 1) unclear instructions, 2) ineffectiveness at teaching new content, 3) generating little interest to enrol in microbiology classes and 4) low possibility of changing lifestyles. The latter three points allude to current game mechanics being ineffective in highlighting the relevance of microbiomes to students’ lives. To assess the concordance between the game’s intended learning objectives and students’ actual learning outcomes, self-reported key takeaways were examined. Although many of the mentioned themes corresponded to the learning objectives that were established during the game development phase, it was evident that certain learning objectives were insufficiently emphasised.

With valuable suggestions provided by participants to enhance learning impact, we anticipate that the revised game (version 2), which further integrates learning outcomes with game mechanics, will better showcase the importance and relevance of microbiomes. Furthermore, student insights from the current study have facilitated the development of assessment tools for quantifying learning gains in future studies through pre- and post-play testing methodologies. Also, since “No Guts No Glory” emphasises the beneficial impact of microbes on our health, future studies could compare the impact of this game to others which emphasise disease-causing pathogens and antimicrobial resistance, especially in how they shape perceptions about microbes.

V. CONCLUSION

In conclusion, our pilot study of “No Guts No Glory” highlighted its strengths in terms of accessibility and player engagement. However, it also brought to attention areas requiring improvement. These include simplifying the instructions to enhance clarity, ensuring that learning is effectively integrated with gameplay and better aligning game mechanics with the science.

Furthermore, we have identified key learning outcomes from unsupervised gameplay which will guide the development of assessment instruments in future studies, via pre- and post-play testing. Such testing will allow us to evaluate learning gains more effectively in subsequent studies involving both microbiology and non-microbiology undergraduates.

Notes on Contributors

Chuu Ling Chan was involved in conceptualisation, methodology, project administration and supervision, data analysis and writing (original draft and editing).

Russell Lee was involved in conceptualisation, methodology, project administration and supervision, data analysis and writing (editing).

Lih Ing Goh was involved in methodology, project administration and supervision, data analysis and writing (editing).

Nathanael Hao Kai Chong was involved in project administration and supervision.

Li Neng Lee was involved in conceptualisation, methodology and writing (editing).

Jun-Hong Ch’ng was involved in conceptualisation, methodology, data analysis and writing (editing).

Ethical Approval

This study was reviewed by the Learning and Analytics Committee on Ethics- Departmental Ethics Review Committee (LACE-DERC) from the National University of Singapore (NUS) Institute for Applied Learning Sciences and Educational Technology (ALSET) and Centre for Development of Teaching & Learning (CDTL), with an exemption from IRB review and the approval to conduct research at NUS (LACE Reference Code: L2021-12-01).

Data Availability

Qualitative study data can be accessed at https://doi.org/10.6084/m9.figshare.23735211.

Acknowledgement

We would like to acknowledge all the students who participated in this study.

Funding

This study is funded by the Teaching Enhancement Grant (TEG FY2023) from the Centre for Development of Teaching and Learning (CDTL), National University of Singapore (E-571-00-0001-01).

Declaration of Interest

The authors declare no conflicts of interest.

References

Akinsemolu, A. A. (2018). The role of microorganisms in achieving the sustainable development goals. Journal of Cleaner Production, 182, 139–155. https://doi.org/10.1016/j.jclepro.2018. 02.081 

Barnes, R. L. (2022). A protein purification card game develops subject knowledge and transferable skills. Journal of Biological Education, 56(4), 365–375. https://doi.org/10.1080/00219266.20 20.1799844

Lynch, S. V., & Pedersen, O. (2016). The human intestinal microbiome in health and disease. New England Journal of Medicine, 375(24), 2369–2379. https://doi.org/10.1056/NEJMra16 00266

Spandler, C. (2016). Mineral supertrumps: A new card game to assist learning of mineralogy. Journal of Geoscience Education, 64(2), 108–114. https://doi.org/10.5408/15-095.1

Su, T., Cheng, M.-T., & Lin, S.-H. (2014). Investigating the effectiveness of an educational card game for learning how human immunology is regulated. CBE Life Sciences Education, 13(3), 504–515. https://doi.org/10.1187/cbe.13-10-0197

*Jun-Hong Ch’ng
MD4, 5 Science Drive 2,
Singapore 117545
Email: micchn@nus.edu.sg

Submitted: 19 August 2023
Accepted: 21 December 2023
Published online: 2 July, TAPS 2024, 9(3), 41-49
https://doi.org/10.29060/TAPS.2024-9-3/SC3111

Rosy Swee Cheng Tay¹, Debby Regina Wegner², Li Siong Lim³, Joshua Ting⁴ & Shu Ting Ong⁵

¹University of Glasgow Singapore, Singapore; ²Singapore Institute of Technology, Singapore; ³Singapore Institute of Technology, Singapore; ⁴Alexandra Health, Singapore; ⁵Gleaneagles Hospital, Singapore

Abstract

Introduction: The Singapore Institute of Technology-University of Glasgow (SIT-UofG) Nursing Programme has traditionally taken a didactic teaching approach in the delivery of the Evidence-Based Practice (EBP) module.  A hybrid approach was introduced using Game-Based Learning (GBL) to encourage active learning through gameplay. 

Methods: A Randomised Controlled Trial (RCT) was undertaken encompassing a cohort of 100 Nursing students taking the EBP module in their first year at the Singapore Institute of Technology (SIT) in the 2021/22 academic year. The experimental group (n=27) worked through the online GBL intervention and the EBP module, while the control group (n=27) took the EBP module alone. The GBL included five Learning Quests and three case studies.

Results: High levels of satisfaction were reported by both the experimental group (n=22) and the control group (n=15) on the traditional content and delivery of the EBP module. High levels of engagement were reported by the experimental group on the GBL intervention; a one-sample statistics analysis confirming a significant level of engagement (p<0.001). A Mann-Whitney U Test, however, found no significant difference in the Continuous Assessment (CA) scores of the two groups (p=0.507 and 0.461). 

Conclusion: The introduction of GBL designed to deliver educational content directly associated with the learning outcomes increased the nursing student engagement in the EBP module. These findings and discoveries can be utilised to improve the GBL intervention to the EBP module to have a more positive impact the student CA scores and therefore on student learning.

Keywords:           Evidence-Based Teaching, Evidence-Based Practice (EBP), Nursing Education, Game-Based Learning (GBL), Teaching and Learning EBP, Nursing Students, Gamification

I. INTRODUCTION

The Singapore Institute of Technology-University of Glasgow (SIT-UofG) Nursing Programme has traditionally taken a didactic teaching approach in the delivery of the Evidence-Based Practice (EBP) module. Numerous undergraduate nursing programmes have taken a similar approach aimed at preparing students to be informed knowledge generators, rather than the more effective savvy consumers of research, or “evidence users” (Melnyk & Fineout-Overholt, 2019). These undergraduate nursing programmes continue to teach “traditional” nursing research courses that fail to address many of the skills, attitudes, and critical knowledge needed to foster EBP, doing little to promote the development of competencies critical for engaging in EBP in clinical contexts (Davidson & Candy, 2016).

EBP is mandated by the Singapore Nursing Board as a key entry to nursing practice competency (Singapore Nursing Board, n.d.) and yet new graduate nurses still lack the knowledge, skills, and attitudes that nurture EBP. In addition, the COVID-19 pandemic brought about unprecedented challenges when classes moved from face-to-face to online, with one of the side-effects being a marked deterioration in the active engagement of the students in their classes. Active engagement is seen as critically pertinent to the learning development of nursing students in preparing them to be future-ready graduates.

Gamification, or the application of game design elements like levels and badges to non-game activities, has been avidly adopted to support contextual teaching and learning, assignments, and assessments in efforts to strengthen student creativity and retention (Caponetto et.al., 2014). Although, as reported by Tavares (2022), results have been varied, Game-Based Learning (GBL) is being increasingly utilized in nursing education.  The aim of this short communication is to describe the development and evaluation of an innovative teaching strategy to evaluate student satisfaction, perception, level of engagement, and overall achievement of the learning outcomes via the gamification of the EBP module.

II. METHODS

A Randomised Controlled Trial (RCT) was undertaken using the 100 Nursing students taking the EBP module in their first year at the Singapore Institute of Technology (SIT) in Trimester 3 of the 2021/22 academic year. All the students were invited to participate in the study and complete the Informed Consent form. The participants were randomized into either the experimental/ intervention group (n=27) or the control group (n=27).

The control group attended the traditional EBP module made up of lectures, workshops, and seminars. The intervention group completed both the EBP module and the GBL intervention. The web-based, mobile-optimized GBL was composed of five levels or Learning Quests (LQs) and three case studies. The LQs followed six different EBP steps (Melnyk & Fineout-Overholt, 2019), with each LQ calling for increasing levels of knowledge and proficiency to cultivate a spirit of inquiry for the undergraduate nursing students. The intervention group was informed to access the GBL only after the key EBP concepts and teaching were delivered to both groups.

Two surveys were used to evaluate participant satisfaction, perception, level of engagement, and overall achievement of the learning outcomes. At the end of the trimester, a university-wide Module and Teaching Feedback survey (Appendix 1) was used to assess how satisfied both groups were with teaching and learning within the EBP module. The GBL Module Learning Outcomes Feedback survey was to get a sense of how the intervention group evaluated the effectiveness of the four main GBL outcomes (Appendix 2).

III. RESULTS

A total of 54 students out of the 100 (54%) taking the EBP module agreed to participate in this study. A total of 22 (81.5%) participants from the intervention group and 15 (55.6%) from the control group completed the surveys. There was a 44.4% attrition from the control group.

A. Grade Performance

There were two types of Continuous Assessment (CA) used – a group presentation worth 40% of the total grade, and an individual essay worth 60%. For the group presentation the mean score and standard deviation of the intervention group was 79.56 (3.70) and the control group was 78.70 (4.03). For the essay, the mean score and standard deviation of the intervention group was 69.6 (11.93) and for the control group was 67.81 (10.76). The Mann Whitney U Test found that there was no significance difference between the CA scores of the intervention and control groups (p = 0.507 and 0.461).

B. Level of Satisfaction

Overall, there was a high level of satisfaction for the EBP module for both groups (intervention: 95.5%; control: 93.3%) with no significant difference (p=0.658) found between the two groups.

C. Level of Perception

Both groups demonstrated similar positive perceptions about the EBP module. 5 themes emerged from the thematic analysis of the open-ended questions (Table 1)

D. Level of Engagement

The GBL was reported by the students to have deepened their learning of EBP. 4 themes emerged from a thematic analysis of the open-ended questions (Table 1).

Table 1. Themes and associated subthemes

Note. This table lists the themes and associated subthemes that were identified from the student responses to the open-ended questions under perception and engagement. Selected student comments are included.

IV. DISCUSSION

There were several key findings from the development of GBL for the EBP module. Although no significant difference was found on the CA grade performance between the groups, both groups expressed a high level of satisfaction upon completing the EBP module.

Despite facing challenges with the usual didactic approach of teaching and learning of EBP, participants expressed appreciation to the faculty for the close stepwise guidance and prompt feedback in facilitating synergistic class teamwork. The systematic flow helped create a positive perception of the module delivery. This well-structured learning scaffold also provided a solid foundation for writing their Honours Thesis.

Gamification using case studies provided good interactivity for applied learning to enhance student engagement and learning. With regards to the interface itself, the navigation of the case studies via the user interface was found to be challenging, so side-by-side scrolling can be considered to improve future user experience.  Standardizing teaching and instructional methods to manage student expectations can also be considered, as there was definite support for moving the EBP module towards game-based learning.

This development of GBL for the EBP module was an innovative hybrid approach to the teaching pedagogy of the educational and healthcare institutions in Singapore. The employment of both didactic and GBL teaching and learning served to enhance the learning experience of the students and helped clarify the future translation of EBP into their respective clinical practice. While the feedback from the intervention group indicated the GBL interface can be improved, the high levels of satisfaction and engagement experienced by this group demonstrated the incorporation of game mechanics in the design of a teaching-learning process can help engage learners in a more productive learning experience.

With no significant difference between the grades of the two groups, it could be argued that the nursing students are already being well guided and supported by Faculty in the teaching and learning of EBP and that the intervention was built upon a firm foundation. Leveraging upon authentic digital learning with a sound pedagogical approach, however, can help to further promote critical thinking, active engagement, and positive learning experiences for the nursing students. Future enhancement to the GBL intervention can consider levelling up the basis of gamifying learning for higher immersive learning experience.  As there was a high attrition from the control group of 44.4%, future studies can consider a larger sample size for better generalizability. As this study only employed the GBL after the key concepts and teaching of EBP was delivered, future studies can consider the teaching and learning pedagogy of GBL along with seminar sessions to evaluate its viability.

V. CONCLUSION

This research project was carried out at the Singapore Institute of Technology to assess the impact of a GBL intervention on teaching and learning in the EBP module for nursing students. Despite no significant differences in grades between the groups, the results show the intervention had a positive impact on the student engagement, satisfaction, and perception of the learning experience, offering an excellent option of combining traditional didactic teaching with the innovative approach of GBL to enhance the EBP teaching and learning of undergraduate nursing students and healthcare professionals in both educational and healthcare institutions.

Notes on Contributors

Assistant Professor Rosy Tay Swee Cheng spearheaded the development of this research project, conducted the survey and data analysis, contributed to the conception, drafted, and critically revised the manuscript.

Ms. Debby Regina Wegener is a Senior Librarian and assisted with the development of a part of this research project, and critically revised the manuscript.

Mr. Lim Li Siong is a Senior Educational Developer in CoLEAD and coordinated the development of the research project and contributed to the critical review of the manuscript.

Mr. Joshua Ting is a Staff Nurse, Alexandra Health and played a key role in the development, conducted the survey and data analysis of this research project, and assisted in the critical review of the manuscript.

Ms. Ong Shu Ting is a Staff Nurse, Gleneagles Hospital and played a key role in the development, conducted the survey and data analysis of this research project, and assisted in the critical review of the manuscript.

All authors gave their final approval and agreed to be accountable for all aspects of the work.

Ethical Approval

Ethical approval approved by Singapore Institute of Technology (SIT) Review Board (IRB Project No. 2021157).

Data Availability

As per institutional policy, research dataset is available on reasonable request to the corresponding author.

Acknowledgement

The authors would like to thank Adjunct Associate Professor Edward Poon, Singapore Institute of Technology (SIT) for his invaluable support to the research project and Tam Yew Chung, Centre for Learning Environment and Assessment Development (CoLEAD), SIT for his advice and invaluable support in navigating the funding process.  

Funding

Funding (WBS Code: P-ALI-A203-0008) was received from Applied Learning and Innovation Grant (ALIGN), CoLEAD, SIT for the development of this web-based EBP game-based learning.  

Declaration of Interest

The authors declare no conflict of interest.

References

Caponetto, I., Earp, J., & Ott, M. (2014). Gamification and education: A literature review. European Conference on Games Based Learning, 1, 50–57.  

Davidson, S. J., & Candy, L. (2016). Teaching EBP using game-based learning: Improving the student experience. Worldviews on Evidence-Based Nursing, 13(4), 285–293. https://doi.org/10.1111/wvn.12152

Melnyk, B. M., & Fineout-Overholt, E. (2019). Evidence-based practice in nursing & healthcare: A guide to best practice (4^th ed.). Wolters Kluwer.

Singapore Nursing Board. (n.d.). Guidelines and Standards. https://www.healthprofessionals.gov.sg/snb/guidelines-and-standards  

Tavares, N. (2022). The use and impact of game-based learning on the learning experience and knowledge retention of nursing undergraduate students: A systematic literature review. Nurse Education Today, 117, Article 105484. https://doi.org/10.1016/j.nedt.2022.105484 

*Rosy Tay Swee Cheng
University of Glasgow, Singapore,
10 Dover Drive, Singapore 138683
+65 6592 4521
Email: rosy.tay@glasgow.ac.uk