Submitted: 28 January 2023
Accepted: 17 August 2023
Published online: 2 January, TAPS 2024, 9(1), 3-19
https://doi.org/10.29060/TAPS.2024-9-1/OA2947

Thun How Ong¹, Hwee Kuan Ong², Adrian Chan¹, Dujeepa D. Samarasekera³ & Cees Van der Vleuten⁴

¹Department of Respiratory and Critical Care Medicine, Singapore General Hospital, Duke-NUS Medical School, Singapore; ²Department of Physiotherapy, Singapore General Hospital; ³Centre for Medical Education, Yong Loo Lin School of Medicine, National University of Singapore, Singapore; ⁴Department of Educational Development and Research, Maastricht University, Maastricht, The Netherlands

Abstract

Introduction: The mini-Clinical Evaluation Exercise (CEX) is meant to provide on the spot feedback to trainees. We hypothesised that an ultra-short assessment tool with just one global entrustment scale (micro-CEX) would encourage faculty to provide better feedback compared to the traditional multiple domain mini-CEX.

Methods: 59 pairs of faculty and trainees from internal medicine completed both the 7-item mini-CEX and a micro-CEX and were surveyed regarding their perceptions of the 2 forms. Wordcount and specificity of the feedback was assessed. Participants were subsequently interviewed to elicit their views on factors affecting the utility of the CEX.

Results: Quantity and quality of feedback increased with the micro-CEX compared to the mini-CEX. Wordcount increased from 9.5 to 17.5 words, and specificity increased from 1.6 to 2.3 on a 4-point scale, p < 0.05 in both cases. Faculty and residents both felt the micro-CEX provided better assessment and feedback. The micro-CEX, but not the mini-CEX, was able to discriminate between residents in different years of training. The mini-CEX showed a strong halo effect between different domains of scoring. In interviews, ease of administration, immediacy of assessment, clarity of purpose, structuring of desired feedback, assessor-trainee pairing and alignment with trainee learning goals were identified as important features to optimize utility of the (mini or micro or both) CEX.

Conclusions: Simplifying the assessment component of the CEX frees faculty to concentrate on feedback and this improves both quantity and quality of feedback. How the form is administered on the ground impacts its practical utility.

Keywords:           Workplace Based Assessment, Mini-CEX, Micro-CEX, Feedback, Assessment

Practice Highlights

• Simplifying the assessment component of the CEX frees faculty to concentrate on feedback.
• A simpler form can result in better and more feedback.
• Making it easy for faculty to use the form is important and increases its utility in providing feedback and assessment.

I. INTRODUCTION

The Mini-CEX is one of the most widely used work-placed based assessment (WBA) tools and is supported by a large body of theoretical and empirical evidence which have shown that when used in the context of repeated sampling, it is both a valid assessment tool and is also an effective education tool in giving feedback to the trainee (Hawkins et al., 2010; Norcini et al., 2003). However, in practice, the educational value of the mini-CEX, as measured chiefly by trainee and faculty perceptions and satisfaction, varied significantly (Lorwald et al., 2018). Factors affecting the educational value have been described by Lorwald et al. and categorised into context of usage, and user, implementation and outcome factors (Lorwald et al., 2018).

Context refers to the situation in which the mini-CEX is executed, and factors which impact its actual usage, such as time needed for conducting the Mini-CEX, or the usability of the tool. Time constraint on the part of both the residents and the assessors is an especially frequent issue across multiple studies (Bindal et al., 2011; Brazil et al., 2012; Castanelli et al., 2016; Lörwald et al., 2018; Morris et al., 2006; Nair et al., 2008; Yanting et al., 2016). The mini-CEX was conceived as a 30-minute exercise of directly observed assessment, and there are 6 or 7 domains which faculty are expected to assess (Norcini et al., 2003). In a busy clinical environment however, what actually occurs is often a brief clinical encounter of 10-15 minutes or even less where only a few of the mini-CEX’s domains were assessed (Berendonk et al., 2018).

User factors refers to trainee and faculty knowledge of the mini-CEX and their perceptions of its use. Studies have found that the mini-CEX is frequently regarded as a check box exercise (Bindal et al., 2011; Sabey & Harris, 2011). Assessor’s and trainee’s training and attitudes, or unfamiliarity with the WBA tools also negatively impact the educational value of the mini-CEX (Lörwald et al., 2018). Reports have shown that educating faculty on the formative intent of mini-CEX can improve feedback provided (Liao et al., 2013).

Implementation factors refer to how the mini-CEX is actually executed on the ground. Some studies have reported that the mini-CEX often occurs without actual direct observation (Lörwald et al., 2018) or feedback provided (Weston & Smith, 2014). Implementation in turn affected outcome, which refers to the trainees appraisal of the feedback received (Lörwald et al., 2018).

One way of improving the educational value of the mini-CEX then might be to improve the context of its usage, by redesigning the mini-CEX to better fit the realities of the clinical workplace. In different clinical encounters, specific domains of performance are more easily and obviously observed and assessed than others (Crossley & Jolly, 2012). Reducing the number of dimensions the assessors are asked to rate was shown to decrease measured cognitive load and improved interobserver reliability (Tavares et al., 2016). It has also been shown that using rating scales that align with the clinician’s cognitive schema perform better, for instance, scales that ask the clinician assessors about the trainees ability to practice safely with decreasing levels of supervision (i.e. entrustability) showed better discrimination and higher reliability (Weller et al., 2014). Compared to multidimension rating scales, global rating scales have greater reliability and validity in assessing candidates in OSCE examinations (Regehr et al., 1998), assessing technical competence in procedures (Walzak et al., 2015) and in simulation-based training (Ilgen et al., 2015).

We proposed therefore to replace the multiple domains with a single rating asking faculty what level of supervision the resident would require in performing a similar task, i.e. a global entrustment scale. The shorter assessment task should refocus the faculty on the feedback component, whilst still retaining the ability to identify trainee progression. One such form has been proposed by Kogan and Holmboe (2018), and we designated this the micro-CEX.

We hypothesised that these changes would improve the usability (“context” as described by Lorwald et al.) and hence improve the educational value of the assessment, measured in this study by the specificity and quality of the feedback given by faculty.

Our study aims to show therefore that the shorter micro-CEX can provide better feedback than the usual mini-CEX. We also sought to find out, from the perspective of the end-users, what other adjustments to the implementation and design of the mini or micro-CEX can be made to improve its acceptability, educational value and validity.

The study focussed on the following questions:

Does the micro-CEX stimulate faculty to provide more specific and actionable feedback compared to the mini-CEX?

Can the micro-CEX provide discriminatory assessment for residents across different years of practice?

What are the perceptions of the faculty and residents regarding the factors affecting utility of the assessment instrument in providing feedback and assessment?

II. METHODS

A. Setting and Subjects

The study was conducted in the division of Internal Medicine in a 1700 bed hospital in Singapore between September and December 2018. All faculty and residents rotating through internal medicine were invited to participate via e-mail, and agreeable faculty and residents paired up. In usual practice, residents must complete at least 2 mini-CEX covering standard inpatient or outpatient encounters during each three-month internal medicine posting, hence both residents and faculty are familiar with the usual mini-CEX.

B. Design

In order to evaluate for any participant reactivity affecting the CEX data (i.e. a Hawthorne effect) (Paradis & Sutkin, 2017), a baseline sample of 30 of the usual mini-CEX performed in the 3 months prior to the study was randomly selected and deidentified (from June to August 2018) . The quantity and specificity of feedback in these was evaluated as detailed below.

For the study itself, faculty and residents used the usual mini-CEX as the first assessment in the first 2 weeks of the month, followed by a second assessment using the micro-CEX in the next 2 weeks. This sequence was chosen as performing the micro-CEX first might affect how the subsequent mini-CEX was performed. Cases chosen for the mini-CEX and micro-CEX were inpatient or outpatient internal medicine encounters, and faculty were simply instructed to choose cases that represented typical cases of average difficulty with no restrictions on the exact cases to be chosen.

Faculty and residents completed an anonymised survey on their experiences at the end of the study and were invited to participate in a semi-structured group interview to elicit their views regarding which aspects of the mini-CEX exercise influenced feedback and assessment (Appendix 2). Both faculty and residents were informed that the survey and interviews were part of this study and participation in either was taken to be implied consent. The workflow of the study is seen in Figure 1.

Figure 1. Study workflow

Submitted: 5 April 2023
Accepted: 17 August 2023
Published online: 2 January, TAPS 2024, 9(1), 28-35
https://doi.org/10.29060/TAPS.2024-9-1/OA3035

Aminda Faizura Omar, Tun Yasmin Iffah Mohd Suria Affandi, Mohd Isyrafuddin Ismail, Mas Suryalis Ahmad & Ilham Wan Mokhtar

Comprehensive Care Centre of Studies, Faculty of Dentistry, Universiti Teknologi MARA, Malaysia

Abstract

Introduction: This study was undertaken to investigate students’ perceptions of innovation-integrated learning, adopted as an alternative approach for special care dentistry (SCD) training.

Methods: Ninety final-year dental students from the 2021 cohort were divided into eight groups to complete an innovation project dedicated to eight categories of patients with special health care needs. Discussion and final presentations, involving related experts were conducted remotely via an online platform. Then, students were invited to answer a validated online feedback survey on their perceptions of the learning approach.

Results: The response rate for the feedback survey was 91.1% (n=82). Most students agreed that the activity was interesting, improved their knowledge and understanding of SCD, allowed engagement between peers, supported sharing of ideas and experiences, encouraged student–lecturer interactions, and enhanced knowledge integration and application. Students also expressed that the activity enhanced creativity and innovation, instilled an interest and positive attitude toward learning SCD, and encouraged teamwork. However, a few students noted facing some limitations in completing their projects (i.e., technological challenges and reduced physical access to purchase materials). Around half felt neutral about having an increased workload from this activity. Moreover, perceptions differed regarding time and financial commitments, as well as supervisors’ and patients’ involvement during the project development process.

Conclusion: Students perceived that the innovation-oriented learning activity, was beneficial in multiple aspects of SCD training.

Keywords:           Innovation, Learning, Dental Students, Perception

Practice Highlights

• We have developed a new integrated teaching method via the utilisation of innovation development for special needs groups.
• Students’ perceptions of this learning method were explored using both quantitative and qualitative methods via an online feedback questionnaire.
• Overall, students positively responded to the teaching method when it came to improving their knowledge and understanding of SCD, engagement between peers, sharing of ideas and experiences, student–lecturer interactions, and knowledge integration and application.
• This novel teaching method achieves the highest level of Bloom’s Taxonomy which is to create something new.

I. INTRODUCTION

Special care dentistry (SCD) involves managing patients with special health care needs, such as those with physical, intellectual, psychological impairments as well as complex medical conditions (Royal Australasian College of Dental Surgeons, 2021). At the undergraduate level, many teaching and learning activities have been conducted to provide students with learning experience in this area of patient care. These activities include lectures, community-based learning, simulation exercises and clinical training (Ahmad, Mokhtar, et al., 2020; Mohamed Rohani & Mohd Nor, 2021). Students who received training in SCD were found to have demonstrated better comfort and attitudes in managing patients with special health care needs (Mohamed Rohani et al., 2021).

However, these learning activities may be compromised by situations that limit face-to-face interactions, either among students, between teacher and learners, as well as with the patients. Such situations may happen due to disease (such as the recent COVID-19 pandemic), geographical barriers and accessibility issues (Amir et al., 2020). It was found that limitations of dental training that requires physical contact may compromise students’ learning experience, thus affecting their competency, knowledge and confidence in patient care (Chang et al., 2021).

Meanwhile, it has become incumbent on dental education providers to prepare future graduates with essential skills that are deemed beneficial for practicing health care professionals. These skills include innovation, leadership, critical thinking and enterpreneurial abilities (Malaysian Dental Council, 2021). Acquisition of these abilities have been associated with many aspects of personal and professional development including emotional intelligence and practice management (Saleem et al., 2019; Wang et al., 2020).

In view of the various challenges and demands of the current (and future) educational environment and healthcare needs, an innovation-integrated educational approach was developed and adopted as an alternative teaching and learning method to teach SCD. This approach, undertaken mostly remotely using an online learning platform, involved the integration of students’ cognitive function, psychomotor ability, and affective skills through the development of innovative products to address oral health issues experienced by the different categories of patients with special health care needs. As a novel educational approach, assessment of its effectiveness is therefore essential. This study was undertaken to investigate students’ perceptions of the innovation-integrated learning, conducted remotely via an online platform, developed during the COVID-19 pandemic, as an alternative approach for SCD training.

II. METHODS

A. Study design

This was a descriptive, cross-sectional, quantitative and qualitative study, using a validated online feedback questionnaire involving the final-year undergraduate students of the Bachelor of Dental Surgery program of the Faculty of Dentistry, Universiti Teknologi MARA.

B. Study Population

Because this module is a compulsory component of the final-year undergraduate dental curriculum (Faculty of Dentistry, 2022), all students were required to be involved with the group innovation project. However, participation in the online feedback survey was voluntary. At the beginning of the semester, the students were made aware of the study and that their participation in the innovation program and the research were treated separately.

All students who agreed to participate in the online feedback survey provided informed consent, attached to the information sheet supplied to all potential participants via their official university email. To avoid coercion, students were provided with an explanation in the information sheet that their participation was entirely voluntary and anonymous. Those who wished not to participate in the online feedback survey would not be penalised, and they could withdraw their participation at any point without having to provide a justification.

C. Study Instrument

The questionnaire used for the online feedback survey was adopted from the study by Rowe and Wood (2008). It was content-validated by a panel of 2 experts in dental education before its use in the main survey. From the item-level content validatiy index (I-CVI) for each item as well as a summation of the I-CVI divided by the number of average (S-CVI/Ave), a  score of 1 was obtained for both indices as the  panelists rated all items with a score of 3 or 4. The questionnaire consisted of nine items about the perception of students towards this teaching method, measured using a 5-point Likert scale (strongly disagree, disagree, neutral, agree, and strongly agree). Open-ended questions were included at the end of the questionnaire to further explore students’ perceptions of the learning activities.

D. Study Conduct

For the group innovation project, all final-year students (n=90) were divided into eight groups, based on their assigned clinical groupings. Each group was assigned to an academic or clinical specialist in SCD, who acted as a supervisor. Eight categories of patients with special health care needs were identified and randomly allocated to the different groups. The categories of patients were 1) hearing disability, 2) speech disability, 3) visual disability, 4) geriatric, 5) Down syndrome, 6) autism, 7) cerebral palsy, and 8) dementia.

Students were given three months to develop innovative products. They were expected to engage with their supervisors during the whole development process, which involved 1) identification of health complications related to the condition and its impact on oral health, 2) understanding of issues related to patients’ oral health care; 3) creation of a prototype to address the related oral health–related complications. Because all students were at the time subjected to restrictions in having physical classes due to COVID-19 (Samat et al., 2020), large-group discussions and final presentations were undertaken remotely online.

At the end of this group activity, an online presentation, judged by experts in related fields, was conducted as a form of assessment, as well as constructive feedback on improvements and the marketability of the product. Students were then invited to participate in the online feedback questionnaire, which was distributed via their official university email. An English language questionnaire was used as the respondents were undergraduate dental students with a good command and understanding of the language. A reminder email was sent a week later, with another reminder sent the following week.

E. Data Analysis

To maximise the outcome, quantitative data were simplified into a 3-point Likert scale (agree, neutral, and disagree). Using SPSS version 27, the frequency of the data was analysed to compare students’ responses for each item. For analysis of the benefits, the nine items of the questionnaire were further categorised into different themes: Theme 1—Acceptance of the teaching method concept (Item 1), Theme 2—Effectiveness of teaching method (Items 2, 3, 6, and 7), Theme 3—Important role of supervisors (Items 4, 5, and 8), and Theme 4—Impact on students’ workload (Item 9).

For the qualitative component, data were analysed via thematic analysis involving open and closed coding. Emerging themes were then identified based on these codes, which were then validated through comparisons amongst researchers.

F. Ethical Approval

Ethics approval was obtained from the Research Ethics Committee (REC) Universiti Teknologi MARA, Malaysia, in accordance with the Declaration of Helsinki (REC12/2021 [MR 1004]).

III. RESULTS

A total of 82 out of 90 students returned the questionnaire, giving a response rate of 91.1%. The frequency of responses for each question or item, categorised into the different themes, is shown in Table 1.

Submitted: 30 January 2023
Accepted: 2 August 2023
Published online: 2 January, TAPS 2024, 9(1), 20-27
https://doi.org/10.29060/TAPS.2024-9-1/OA2999

Mitsumi Masuda, Naomi Kimura & Akemi Nakagaki

Graduate School of Nursing, Nagoya City University, Japan

Abstract

Introduction: The level of BLS achievement at the end of basic education is not clear, so to develop a BLS training program using QCPR Learner^Ⓡ and to verify its effectiveness.

Methods: A quantitative descriptive study design was used. The developed BLS program was implemented for 81 fourth-year students at the school of nursing. The program consisted of 2 minutes of training per student and real-time feedback by QCPR Learner^Ⓡ as well as feedback among students and from faculty members and was conducted twice for each student. Evaluation was conducted from three aspects: learner response and program evaluation both using questionnaires, learning achievement of QCPR score.

Results: The mean values of learners’ responses before and after the program were: attention, 4.32 before and 4.59 after; relevance, 4.48 before and 4.75 after; confidence, 2.32 before and 3.78 after; and satisfaction, 2.12 before and 3.41 after, which increased significantly after the program. The QCPR Learner^Ⓡ scores increased significantly from 74.08 for the first time to 86.76 for the second time. In the free response, some respondents stated that “visualisation of the procedure by the application improved my skills.”

Conclusion: The results of the evaluation from three aspects showed that this program was effective in improving students’ motivation to learn and skills.

Keywords:           Basic Life Support, Real-Time Feedback, Basic Nursing Education, Simulation Training, QCPR Learner

Practice Highlights

• Real-time feedback application was effective in increasing motivation to learn and BLS skills.
• QCPR Learner^Ⓡ can be easily installed on the learner’s device, making it suitable for self-learning.

I. INTRODUCTION

Submitted: 9 February 2023
Accepted: 15 May 2023
Published online: 3 October, TAPS 2023, 8(4), 23-35
https://doi.org/10.29060/TAPS.2023-8-4/OA3006

Sethapong Lertsakulbunlue¹, Kaophiphat Thammasoon², Kanlaya Jongcherdchootrakul³, Boonsub Sakboonyarat³ & Anupong Kantiwong¹

¹Department of Pharmacology, Phramongkutklao College of Medicine, Thailand; ²Department of Personnel Administration Division, Phramongkutklao College of Medicine, Thailand; ³Department of Military and Community Medicine, Phramongkutklao College of Medicine, Thailand

Abstract

Introduction: Although medical research (MR) is constantly promoted, a global deficit of medical researchers has been noted. We aimed to explore the relationship among practice, perceptions, attitudes, barriers and motivation toward MR and its impacts on MR publication.

Methods: A cross-sectional study included 262 senior medical students and interns. An electronic, standardised Likert scale questionnaire was used to collect the data. Binary logistic regression was used to determine the odds ratio between characteristics and MR publication. Confirmatory factor analysis was used to confirm the loading factor of each question, and structural equation modeling (SEM) was used to investigate the relationship between latent variables and MR publication.

Results: Cronbach’s alpha revealed a good internal reliability of 0.93. The accumulated grade point average did not differ between those who had published and those who had not. MR presentations were strongly associated with MR publication. SEM showed that attitudes (0.71, p<0.001) and perceptions (0.27, p<0.001) had a direct effect on practices. Practices (0.49, p<0.001) and attitudes (0.30, p<0.001) had a direct effect on motivation, while motivation had a total effect = 0.36, p<0.001 on MR publication through MR presentation as a mediator.

Conclusion: Positive attitudes and perceptions might lead to positivity in the intention to practice MR, which would lead to motivation and finally increase the odds of MR publication. Different approaches to promote excitement and perceptions in MR learning should be encouraged by teachers and faculty members.

Keywords:           Medical Research, Students, Perceptions, Attitudes, Barriers, Motivation

Practice Highlights

• Enjoyment and excitement should be promoted while learning medical research.
• Medical research experiences enhanced publication, particularly medical research presentations.
• Extracurricular medical research activities should be routinely promoted.
• Facilitators in medical research might be tailor-made for each individual.
• Regular meetings regarding medical research with mentors or role models should be held.

Submitted: 2 December 2022
Accepted: 24 July 2023
Published online: 3 October, TAPS 2023, 8(4), 13-22
https://doi.org/10.29060/TAPS.2023-8-4/OA3093

Julie Yun Chen^1,2, Tai Pong Lam¹, Ivan Fan Ngai Hung³, Albert Chi Yan Chan⁴, Weng-Yee Chin¹, Christopher See⁵ & Joyce Pui Yan Tsang¹

¹Department of Family Medicine and Primary Care, University of Hong Kong, Hong Kong; ²Bau Institute of Medical and Health Sciences Education, University of Hong Kong, Hong Kong; ³Department of Medicine, University of Hong Kong, Hong Kong; ⁴Department of Surgery, University of Hong Kong, Hong Kong; ⁵School of Biomedical Sciences, The Chinese University of Hong Kong, Hong Kong

Abstract

Introduction: Medical students have long provided informal, structured academic support for their peers in parallel with the institution’s formal curriculum, demonstrating a high degree of motivation and engagement for peer teaching. This qualitative descriptive study aimed to examine the perspectives of participants in a pilot peer teaching programme on the effectiveness and feasibility of adapting existing student-initiated peer bedside teaching into formal bedside teaching.

Methods: Study participants were senior medical students who were already providing self-initiated peer-led bedside clinical teaching, clinicians who co-taught bedside clinical skills teaching sessions with the peer teachers and junior students allocated to the bedside teaching sessions led by peer teachers.  Qualitative data were gathered via evaluation form, peer teacher and clinician interviews, as well as the observational field notes made by the research assistant who attended the teaching sessions as an independent observer.  Additionally, a single Likert-scale question on the evaluation form was used to rate teaching effectiveness.

Results: All three peer teachers, three clinicians and 12 students completed the interviews and/or questionnaires. The main themes identified were teaching effectiveness, teaching competency and feasibility. Teaching effectiveness related to the creation of a positive learning environment and a tailored approach. Teaching competency reflected confidence or doubts about peer-teaching, and feasibility subthemes comprised barriers and facilitators.

Conclusion: Students perceived peer teaching effectiveness to be comparable to clinicians’ teaching. Clinical peer teaching in the formal curriculum may be most feasible in a hybrid curriculum that includes both peer teaching and clinician-led teaching with structured training and coordinated timetabling.

Keywords:           Peer Teaching, Undergraduate Medical Education, Bedside Teaching, Medical Students

Practice Highlights

• Peer-led teaching environment facilitates questions and answers from learners to strengthen learning.
• Training on specific skills and pre-case preparation can help improve peer teacher effectiveness.
• Clear understanding of the logistics and expectations is necessary to optimise the process.
• Formal peer teacher training may help quality assurance and encourage more participation.

I. INTRODUCTION

In accordance with the longstanding apprenticeship model of medical training, senior doctors and trainees have been responsible for teaching their junior colleagues across the continuum of medical education. Despite this accepted practice, peer teaching has not become widely formalised in undergraduate medical curricula.

Peer teaching has been shown to be beneficial at multiple levels. For students who are being taught by peers, learning is enabled by social and cognitive congruence because of the near-peer demographic which allows for a more comfortable learning environment for free flow of discussion and better understanding of the learner’s challenges including awareness of the primacy for exam success (Benè & Bergus, 2014; Rees et al., 2016). The peer teacher develops and hones teaching skills that will be useful in internship (Haber et al., 2006) and through teaching, develops higher motivation and deeper understanding of concepts and perhaps also improve their own exam performance (Burgess et al., 2014). The institution derives some practical benefit from the supplementary manpower (Tayler et al., 2015) due to the comparable effectiveness of peer teachers in teaching in certain areas such as physical examination and communication skills (Rees et al., 2016) but perhaps most importantly, it benefits from building a collaborative relationship with students in their learning process. Though the benefits of peer teaching have been noted, students remain an untapped resource as training provided for students to serve as teachers is inconsistent (Soriano et al., 2010).

Undergraduate medical curricula aim to provide a foundation for future training and the framework for such curricula are guided by the recognition that medical students must achieve certain outcomes, including being able to teach, to be prepared for future practice. Well-accepted frameworks such as the ‘Outcomes for Graduates’, from the UK General Medical Council (2015) and the ‘CanMEDS Framework’ from the Royal College of Physicians and Surgeons of Canada (2015) expect medical graduate to teach others. In Hong Kong, similar guidance is provided in the document ‘Hong Kong Doctors’ published by the Medical Council of Hong Kong, which states that undergraduate medical education must prepare graduates to fulfil the roles of ‘medical practitioner, communicator, educator…’ (Medical Council of Hong Kong, 2017).

It is common in medical schools to have informal peer teaching, where senior students coach junior students on an ad hoc basis or organise revision sessions before exams. Zhang et al. (2011) revealed that a majority of medical students believed that informal learning approaches, including the use of past student notes, and participation in self-organised study groups and peer-led tutorials, helped them pass examinations and be a good doctor. Similarly, in our institution, these kinds of informal peer teaching are popular among students and include sharing sessions on study and exam tips, bedside sessions, and sharing of organised study notes. These activities are not subject to any formal oversight.

With the documented benefits of peer teaching, the availability of enthusiastic senior students who are willing to coach their junior peers, and the demand from junior students to learn from their seniors, there is an opportunity to harness the potential peer teaching that is already taking place. This pilot project is important as it aimed to adapt existing student-initiated peer bedside teaching into the formal bedside teaching curriculum and to examine the perspectives of participants on the effectiveness and feasibility of this initiative. It will be helpful to understand the benefits and drawbacks of formal peer bedside teaching in order to further develop this pedagogical approach in medical education.

II. METHODS

This was a descriptive qualitative study of participants in a pilot peer-teaching initiative for bedside teaching implemented in the first clinical year of study for medical students.

A. Setting

1) Small group bedside teaching for Year 4 medical students in the Clinical Foundation Block: The 11-week Clinical Foundation Block (CFB) of the MBBS Year 4 curriculum at The University of Hong Kong runs from August to October and is the first block of the first clinical year of study. It serves to prepare students for the ward- and clinic-based teaching to follow in the clinical clerkships (Figure 1). Year 4 medical students were selected for the study because it is the first clinical year of study when clinical bedside teaching begins. In addition, as the most junior clinical students, they would benefit most from learning from their senior peers. During the CFB, all Year 4 students learn basic history taking, physical examination and clinical skills as well as common clinical problems of 10 key specialty disciplines. In internal medicine, students attend whole class sessions in which the proper clinical examination of each body system is demonstrated followed by seven small group sessions at the bedside for hands-on practice led by a clinician. 

Figure 1. Teaching activities under Medicine within the Clinical Foundation Block in the medical curriculum

Each small group bedside teaching session is comprised of six to eight CFB students who follow the same clinical teacher to examine 3 pre-selected ward patients over a two-hour period. In this pilot study, a peer teacher joined the clinical teacher for the bedside teaching with the first patient case taught by the clinician, the second case taught by the peer teacher under the supervision of the clinician and the final case taught by the peer teacher alone.

2) Peer teaching recruitment and training: Over the years, medical students have been organising bedside peer-teaching on their own and we identified these peer-teaching leaders to help recruit peer teachers for this initiative. Peer teachers recruited in July 2018 and comprised Year 5 students in Senior Clerkship, who were enthusiastic in teaching, and were available to join the training tutorial and take up a subsequent Year 4 CFB bedside teaching session. During the 2.5-hour tutorial, the CFB Coordinator explained the project, and three clinicians then provided a briefing on cardiovascular, neurological, respiratory and abdominal physical examination, common pitfalls, and how to give feedback. There was also time for students to raise questions both on the project and bedside teaching techniques.

B. Participants

The target participants included the three peer teachers who were recruited for this study, together with the three clinician partners and the 24 CFB students in the corresponding three bedside teaching groups. Written informed consent was obtained from all participants before data collection.

C. Data Collection

The qualitative data were collected using a dual subjective (peer teachers, clinicians and students) and objective (independent observer) approach was taken to provide a more holistic perspective of the peer teaching experience. A research assistant not involved in the teaching followed one (of the three) peer teachers as the independent observer. All peer teachers and clinicians were interviewed in-person, by phone or by email, using an interview guide (Appendix 1) by the research assistant after the session where field notes were taken and transcribed. CFB students were invited to complete an evaluation form comprised of open-ended questions and a single Likert-scale question (Appendix 2) immediately after the bedside session, to rate effectiveness and to give general feedback about the peer teaching session.

D. Data Analysis

The qualitative data comprising interview field notes, interview transcripts, email transcripts and open-ended questions from the evaluation form collected from CFB students were analysed thematically by the authors JC and JPYT. The Likert-scale question from the evaluation form was analysed using descriptive statistics. All data were anonymised.

III. RESULTS

All three peer teachers and three clinicians who participated in the pilot peer teaching sessions were interviewed. Eighteen out of 24 CFB students consented to participate and 12 completed questionnaires were collected. Three main themes were identified with two corresponding subthemes for each.

A. Teaching Effectiveness

Peer teachers were rated favourably in terms of their teaching effectiveness. From the evaluation form completed by CFB students, the mean peer teaching effectiveness rating was 4.5/5. While a few students felt the teaching effectiveness of clinicians and peer teachers was comparable, many of them felt less intimidated being taught by the peer teachers. Students also appreciated that the peer teachers understood their current level of understanding and therefore were able to make the teaching more effective by tailoring it to their needs. Students found the experience-sharing by the peer teachers an added-value as shown in Table 1 (Item 1-4). All clinicians agreed that the CFB students appeared more relaxed while the peer teachers were teaching, and the peer teachers met their standard of professionalism as shown in Table 1 (Item 3).

Submitted: 28 September 2022
Accepted: 2 March 2023
Published online: 3 October, TAPS 2023, 8(4), 5-12
https://doi.org/10.29060/TAPS.2023-8-4/OA2883

Soumia Merrou¹, Abdellah Idrissi Jouicha², Abdelmounaim Baslam³, Zakaria Ouhaz³ & Ahmed Rhassane El Adib¹

¹Health Sciences Research Centre (HSRC), Faculty of Medicine and Pharmacy of Marrakech, Cadi Ayyad University, Morocco; ²Health Sciences Research Centre (HSRC), Faculty of Science Semlalia, Cadi Ayyad University, Morocco; ³Pharmacology, neurobiology and behaviour Lab, Faculty of Science Semlalia, Cadi Ayyad University, Morocco

Abstract

Introduction: A deep understanding of physiology, physiopathology, pharmacology, and the management of pain is crucial for nurse anaesthetists to ensure the well-being of their patients. Thus, the teaching strategies should enhance the transition from acquiring the fundamental pain phenomena, to developing translational and critical thinking. The aim of the study is to determine if the flipped classroom that is considered an active learning approach is most effective compared to the traditional method in teaching pain management and if it improves students’ academic performance.

Methods: This study was quasi experimental, at a higher institute of nursing professions, among third-year anaesthesia resuscitation nursing students. participants were randomly allocated into either: the flipped classroom group where PBL was used (FG, n = 19), or the traditional lecture-based classroom group (TG, n = 19). The results and impact of the above approach were appreciated via the analysis of the summative assessment of the class group and from the questionnaire submitted to students.

Results: The present study revealed that in the midterm exam, the mean score of the flipped classroom group (14.0) which is significantly higher (p<0.01) than the traditional lecture group (11.9). Moreover, the standard deviation of this latter is slightly higher (2.41) which indicates scores far from the average. Also, a significant difference between the averages of the two approaches in favor of flipped classroom Group was revealed (p<0.01).

Conclusion: The assessment of student’s grades and their appreciation of both teaching approaches showed a preference for the PBL.

Keywords:           Flipped Classroom, Nursing Education, Pain Management, Problem-Based Learning

Practice Highlights

• Flipped classroom showed advantageous results on nursing students’ grades.
• Flipped classroom endorsed positive results on course comprehension by nursing students.
• Flipped classroom has shown to effectively support content learning.

I. INTRODUCTION

Flipped classroom is a pedagogical approach defined as: “What was previously completed as homework is now finished in class, and what was previously completed in class is now completed at home” (Dong, 2016). Using this approach, traditional classroom time is spent on active learning strategies such as problem-based learning, games, or practice questions to allow teachers to guide students in developing strategies. critical thinking (Dong, 2016). Flipped classrooms are used as the main teaching method in the courses of health professions such as nursing theory, statistics and pharmacology (Hanson, 2016; Immekus, 2019; Peisachovich et al., 2016). In fact, there is evidence that students’ academic performance improved in midterm exams while using flipped classroom approach (Geist et al., 2015).

Despite feeling that this method increased their knowledge, nursing students said they preferred traditional lectures to the use of a flipped classroom (Hanson, 2016). It is not uncommon for students to prefer lectures to the flipped classroom method, which may be related to how much work they feel they have to do or insecurity of exam preparation or both of them (Dong, 2016; Tune et al., 2013). The use of the flipped classroom in nursing was supported by evidence that showed lecturers were enthusiastic about this method. The most effective method for implementing and assessing this strategy in nursing education, though, is not consistently supported by the available data (Barranquero-Herbosa et al., 2022; Dong, 2016; Njie-Carr et al., 2017).

Contextual learning can encourage the growth of critical reasoning, which enables students to pick out the top nursing concerns for patients from a long list of problems, ultimately fostering the development of problem-based nursing analysis in line with Benner’s model (Dong, 2016). Problem-based learning (PBL) uses problem scenarios to develop knowledge and understanding learning objectives (Wood, 2003). Among the strategies used in a flipped classroom, the PBL has been used in nursing education, in courses such as pharmacology, mental health nursing and critical care nursing (Alton, 2016; Gholami et al., 2016). Any teaching strategy that involves students in the learning process is considered to be an active learning strategy, which includes PBL (Peisachovich et al., 2016).

Despite the introduction of pain management in health professions education, pain is still undertreated. It affects 80%-90% of patients in medicine, surgery, and cancer units (Gerbershagen et al., 2009; Gianni et al., 2010). Previous research also highlighted that 43% to 51% of patients received inadequate or insufficient analgesic treatment and only 14% of patients who received analgesia benefit from reassessment (Deandrea et al., 2008; Manias et al., 2005). To effectively manage pain, nurses are crucial. Therefore, it is crucial that they receive effective training to ensure better pain management (Teike Lüthi et al., 2015).

In this direction, in order to encourage students’ acquisition strategies, nursing science professors must implement effective teaching techniques. Training typically aims to increase knowledge, which is insufficient in this case; as a result, skills development is a top priority (Kerner et al., 2013). While prior research emphasised the value of nurse-patient interactions in pain management, it undervalued the impact of nurses’ scientific knowledge of pain mechanisms and pharmacology. It is interesting to note that a recent study highlighted the significance of the classroom setting and instructional methods in approaching pain management in a novel manner (Teike Lüthi et al., 2015).

However, a need for a rigorous evaluation of learning strategies is crucial for best practices in nursing education (Barranquero-Herbosa et al., 2022; Njie-Carr et al., 2017). The present study provides an assessment of PBL as a model of applied learning in a flipped classroom of anesthesia nursing students in the context of a pain management course.

The main purpose of the study was to determine if the flipped classroom is more effective than traditional learning in teaching pain management by assessing students’ academic performance and determine their perceptions about the flipped classroom approach. In that capacity, the research questions of the study are:

1. Is there a significant difference in students’ academic performance between the traditional and flipped classroom approaches on declarative knowledge?
2. Is there a significant difference in students’ academic performance between the traditional and flipped classroom approaches on conditional knowledge?
3. What are anesthesia and resuscitation nursing students’ perceptions of PBL impact on the acquisition and application of pain management knowledge?
4. What are anesthesia and resuscitation nursing students’ perceptions of PBL as a model for learning in pain management?

II. METHODS

A. Research Design and Samples

This study is quasi experimental, and was conducted from September at a higher institute of nursing professions. The participants are third-year anaesthesia resuscitation nursing students. Participation in the study was voluntary and anonymous. Oral consent of all participants was obtained. These participants were randomly allocated into either: The flipped classroom group where PBL was used (FG, n = 19), or the traditional lecture-based classroom group (TG, n = 19). Both classroom groups had the same professor.

B. Curriculum Description

The “pain management” course (50h) is taught during the third year of nursing studies in the institute. It is composed of three parts: the pathophysiology of pain; the evaluation of pain, and the pain management.

C. Problem Based Learning on Flipped Classroom Approach

The problem-based template was designed by the professor who teaches the course, by using small groups of 5 to 6 students. The students were the facilitators of the discussion; they meet in group work to discuss a case for an hour. The objective is to identify the type of pain or to choose the best pain assessment tool for the case. The group must then suggest a drug treatment protocol and design appropriate nursing interventions. The role of the professor was to provide immediate and specific feedback during the discussion.

All cases were written by the professor. The objectives were the acquisition of knowledge and the development of clinical reasoning. Each case contained 300 words and included key patient data. Each of these cases included information that could be analysed to provide priority elements to the discussed case.

D. Data Collection and Statistical Analyses

The results and impact of the above approach were extracted via an analysis of the summative assessment of the class group and from the questionnaire submitted to students.

1) Summative assessment (exam):

Students in both groups went through two exams: midterm exam (ME) which took place in the middle of the course in the 6^th week in order to assess the students’ declarative knowledge, and a final exam (FE) which took place at the end of the course, to assess conditional knowledge. The tests were graded from zero to twenty. The final score (FS) was obtained by the following equation:

FS=(ME+FE)/2

2) Questionnaire:

At the end of the course, the FG students were asked to fill out an anonymous questionnaire divided into two sections. The questions were developed in the first section of the questionnaire to determine students’ perceptions of knowledge acquisition. Elements evoked in the questionnaire were created with a language that demonstrates perceived ability and related to self-efficacy (Tune et al., 2013). The second set of items was created to determine students’ perceptions of the cases used in the course. The statements began, for example, with “Participating in the group discussions made me more confident for…”. Likert scale was used to measure the responses. The scale is presented as follows:

1 = Strongly disagree, 2 = Disagree, 3 = Neither agree nor disagree, 4 = Agree, and 5 = Strongly agree.

3) Statistical analyses:

Data analysis was performed using MS Excel (21), background variables of the study participants were calculated, and the results are presented as frequency distribution, percentages, mean, and standard deviation, statistical significance when p <0.05.

III. RESULTS

The data that support the findings of this RCT study are openly available at https://doi.org/10.6084/m9.figshare.22639279  (Merrou et al., 2023).

A. Demographics

The number of participants in the study was 38 students, 19 per group. Female students represented 79% of the study participants, whereas 21% were male.

B. Students’ Grades

Based on the data obtained, statistical analysis was done to analyse the influence of the teaching approach and the type of examination on learners’ results. The obtained findings have been presented in Tables 1 and 2. They indicate the average performance of learners in both exams: midterm (ME) and final exam (FE) where conditional knowledge is measured for both teaching approaches.

 

Type of exam	Teaching approach	M	Sd	Inf born	Sup born	P value
ME	TG	11.9	2.41	7.38	16.1	<0.01
	FG	14.0	1.94	9.0	16.5
FE	TG	11.9	3.28	6.09	16	<0.01
	FG	14.1	1.96	10	16

Table 1. Descriptive statistics by exam type for each teaching approach.

According to Table 1, it is noted that in the midterm exam (ME), the mean score is significantly higher (p<0.01) in the FG (14.0) compared to the TG (11.9), also, with this latter, there is a slightly high standard deviation of 2.41 which indicates scores far from the average. FG, on the other hand, dressed a lower standard deviation (1.94) which indicates that the scores are more grouped around the mean (14.0). The application of the PBL on flipped classroom approach has, as it appears, improved the grades and reduced the gap between them.

For the final exam, with the traditional approach, the dispersion increased (Sd=3.28). On the other hand, PBL approach has improved student outcomes and widened the gap between them compared to TG (p<0.01). Figure 1 highlights the dispersion of the continuous and final control data (before for the traditional approach and after for PBL).

Figure 1. Students’ performance during the midterm exam (ME) and final exam (FE)

The ME grades were improved using PBL. As the number of compared participants is limited, a nonparametric test was carried out (Paired Mann-Whitney test) which revealed that the average grade of MEs is significantly different (p <0.01), between the traditional approach and the PBL. Similarly, an improvement in FE grades is observed when using the PBL approach. This approach allowed an improvement in the means as well as the dispersion. This leads us to state that the teaching approach based on case studies in the context of a flipped classroom (PBL), may improve both declarative and conditional knowledge on students’ outcomes.

Teaching approach	N	M	Standard deviation	Δ mean	p
TG	19	11.9	3.30	2.21	0.01
FG	19	14.1	1.95

Table 2: Descriptive statistics by teaching approach for the different types of controls.

From Table 2, there is a difference between the averages of the two approaches in favor of FG (p<0.01). This means that students who have taken the PBL approach had increased grades compared to those who have taken the traditional approach. To check if these differences are significant, a Paired Mann-Whitney test was used. This one demonstrated that the average rank of the grades is significantly different between the two studied approaches (p = 0.01).

The mean score and the standard deviation for each question in the questionnaire was determined. Average responses to the 12 items that referred to acquisition and application of knowledge related to the pain management ranged from 3.8 to 4.5 (See Table 3).

Submitted: 30 May 2022
Accepted: 7 December 2022
Published online: 4 July, TAPS 2023, 8(3), 35-44
https://doi.org/10.29060/TAPS.2023-8-3/OA2876

Rachel Jiayu Lee¹*, Jeannie Jing Yi Yap¹*, Abhiram Kanneganti¹, Carly Yanlin Wu¹, Grace Ming Fen Chan¹, Citra Nurfarah Zaini Mattar^1,2, Pearl Shuang Ye Tong^1,2, Susan Jane Sinclair Logan^1,2

¹Department of Obstetrics and Gynaecology, National University Hospital, Singapore; ²Department of Obstetrics and Gynaecology, Yong Loo Lin School of Medicine, National University of Singapore, Singapore

^*Co-first authors

Abstract

Introduction: Disruptions of the postgraduate (PG) teaching programmes by COVID-19 have encouraged a transition to virtual methods of content delivery. This provided an impetus to evaluate the coverage of key learning goals by a pre-existing PG didactic programme in an Obstetrics and Gynaecology Specialty Training Programme. We describe a three-phase audit methodology that was developed for this

Methods: We performed a retrospective audit of the PG programme conducted by the Department of Obstetrics and Gynaecology at National University Hospital, Singapore between January and December 2019 utilising a ten-step Training Needs Analysis (TNA). Content of each session was reviewed and mapped against components of the 15 core Knowledge Areas (KA) of the Royal College of Obstetrics & Gynaecology membership (MRCOG) examination syllabus.

Results: Out of 71 PG sessions, there was a 64.9% coverage of the MRCOG syllabus. Four out of the 15 KAs were inadequately covered, achieving less than 50% of knowledge requirements. More procedural KAs such as “Gynaecological Problems” and those related to labour were poorly (less than 30%) covered. Following the audit, these identified gaps were addressed with targeted strategies.

Conclusion: Our audit demonstrated that our pre-pandemic PG programme poorly covered core educational objectives i.e. the MRCOG syllabus, and required a systematic realignment. The COVID-19 pandemic, while disruptive to our PG programme, created an opportunity to analyse our training needs and revamp our virtual PG programme.

Keywords:        Medical Education; Residency; Postgraduate Education; Obstetrics and Gynaecology; Training Needs Analysis; COVID-19; Auditing Medical Education

Practice Highlights

• Regular audits of PG programmes ensure relevance to key educational objectives.
• Training Needs Analysis facilitates identification of learning goals, deficits & corrective change.
• Mapping against a milestone examination syllabus & using Delphi technique helps identify learning gaps.
• Procedural-heavy learning goals are poorly served by didactic PG and need individualised assessment.
• A central committee is needed to balance the learning needs of all departmental CME participants.

I. INTRODUCTION

Postgraduate medical education (PG) programmes are an important aspect in meeting core Specialty Trainees’ (ST) learning goals in addition to other modalities of instruction such as practical training (e.g. supervised patient-care or simulator-based training) (Bryant‐Smith et al., 2019) and workplace-based assessments (e.g. case-based discussions and Objective Structured Clinical Examinations [OSCEs] (Chan et al., 2020; Parry-Smith et al., 2014). In academic medical centres, PG education may often be nestled within a wider departmental or hospital Continuing Medical Education (CME) programme. While both PG and CME programmes indirectly improve patient outcomes by keeping clinicians abreast with the latest updates, reinforcing important concepts, and changing practice (Burr & Johanson, 1998; Forsetlund et al., 2021; Marinopoulos et al., 2007; Norman et al., 2004; Raza et al., 2009; Sibley et al., 1982), it is important to balance the learning needs of STs with that of other learners (E.g. senior clinicians, scientists and allied healthcare professionals). This can be challenging as multiple objectives need be fulfilled amongst various learners. Nevertheless, just as with any other component of good quality patient care, it is amenable to audit and quality improvement initiatives (Davies, 1981; Norman et al., 2004; Palmer & Brackwell, 2014).

The protracted COVID-19 pandemic has disrupted the way we deliver healthcare and conduct non-clinical services (Lim et al., 2009; Wong & Bandello, 2020). In response, the academic medical community has globally embraced the use of teleconferencing platforms such as Zoom^Ⓡ, Microsoft Teams^Ⓡ and Webex^Ⓡ (Kanneganti, Sia, et al., 2020; Renaud et al., 2021) as well as other custom-built solutions for the synchronous delivery of didactics and group discourse (Khamees et al., 2022). While surgical disciplines have suffered a decline in the quality of “hands-on” training due to reduced elective surgical load and safe distancing (English et al., 2020), the use of simulators (Bienstock & Heuer, 2022; Chan et al., 2020; Hoopes et al., 2020; Xu et al., 2022), remote surgical preceptorship, and teaching through surgical videos (Chick et al., 2020; Juprasert et al., 2020; Mishra et al., 2020) have helped mitigate some of these. Virtual options that that have been reproducibly utilised during the pandemic and will be a part of the regular armamentarium of post-graduate medical educationists include online didactic lectures, livestreaming or video repositories of surgical procedures, (Grafton-Clarke et al., 2022) and virtual case discussions and grand ward rounds (Sparkes et al., 2021). Notably, they facilitate the inclusion of a physically wider audience, be it trainer or trainee, and allow participants to tune in from different geographical locations.

At the Department of Obstetrics and Gynaecology, National University Hospital, Singapore, the forced, rapid transition to a virtual CME format (vCME) (Chan et al., 2020; Kanneganti, Lim, et al., 2020) provided an impetus to critically review and revamp the didactic component of our PG programmes. A large component of this had been traditionally baked into our departmental CME programme which comprises daily morning meetings covering recent specialty and scientific updates, journal clubs, guideline reviews, grand round presentations, surgical videos, exam preparation, topic modular series, and research and quality improvement presentations. The schedule and topics were previously arbitrarily decided by a lead consultant one month prior and were presented by a supervised ST or invited speaker. While attendance by STs at these sessions was mandatory and comprised the bulk of protected ST teaching time, no prior attempt had been made to assess its coverage of core ST learning objectives and in particular, the syllabus for milestone ST exams.

Our main aim was to conduct an audit on the coverage of our previous PG didactic sessions on the most important learning goals with the aim of subsequently restructuring them to better meet these goals.

II. METHODS

We audited and assessed our departmental CME programme’s relevance to the core learning goals of our STs by utilising a Training Needs Analysis (TNA) methodology. While there are various types of TNA used in healthcare and management (Donovan & Townsend, 2019; Gould et al., 2004; Hicks & Hennessy, 1996, 1997; Johnston et al., 2018; Markaki et al., 2021), in general they represent systematic approaches towards developing and implementing a training plan. The common attributes can be distilled into three common phases (Figure 1). Importantly for surgical and procedurally-heavy disciplines, an dimension that is not well covered by didactic sessions alone are assessments for procedural skill competency. These require separate attention that is beyond the scope of this audit.

A. Phase 1: Identifying Organisational Goals and Specific Objectives

The overarching goal of a specialty PG education programme is to produce well-balanced clinicians with a strong knowledge base. Singapore’s Obstetrics and Gynaecology specialty training programmes have adopted the membership examinations for the Royal College of Obstetricians and Gynaecologists (MRCOG) (Royal College of Obstetricians and Gynaecologists, 2021) of the United Kingdom as the milestone examination for progression from junior to senior ST.

First, we adapted a ten-step TNA proposed by Donovan & Townsend (Table 1) to crystallise our our core learning goals, identify deficiencies, and subsequently propose steps to address these gaps in a systematic fashion that is catered to our specific context. While most aspects were followed without change, we adapted the last aspect i.e. Cost Benefit Analysis. As a general organisational and management tool, the original TNA primarily looked at the financial costs of implementing a training programme. At an academic medical institution, the “cost” is mainly non-financial and mainly refers to time taken away from important clinical service roles.

As part of formulating what were deemed to be core learning goals of an ideal PG programme (i.e. Steps 1 to 4), we had a focused group discussion comprising key stakeholders in postgraduate education, including core faculty (CF), physician faculty (PF), and representative STs. The discussions identified 18 goals specific to our department. We then used a modified Delphi method (Hasson et al., 2000; Humphrey-Murto et al., 2017) to distil what CF, PFs, and STs felt were important priorities for grooming future specialists. Three rounds of priority ranking were undertaken via an anonymised online voting form. At each round, these 18 goals were progressively ranked and distilled until five remained. These were then ranked from highest to lowest priority and comprised 1) exam preparedness, 2) clinical competency, 3) in-depth understanding of professional clinical guidelines, 4) interpretation of medical research literature, and 5) ability to conduct basic clinical research and audits.

B. Phase 2: Identifying a Standard and Assessing for Coverage against This Standard

As with any audit, a “gold-standard” should be identified. As the focus group discussion and Delphi method identified exam preparedness as the highest priority, we created a “blueprint” based on the syllabus of the MRCOG examination (Royal College of Obstetricians and Gynaecologists, 2019). This comprised more than 200 Knowledge Requirements organised more than 200 knowledge requirements into 15 Knowledge Areas (KAs) (Table 2). We mapped the old CME programme against this blueprint to understand the extent of coverage of these KAs. We analyse the session contents between January and December 2019. We felt the best way to ensure systematic coverage of these KAs would be through sessions with pre-identified areas of topical focus conducted during protected teaching time as opposed to opportunistic and voluntary learning opportunities that may not be widely available to all STs. In our department, this applied to morning CME sessions which indeed formed the bulk of protected teaching time for STs, required mandatory attendance, and comprised sessions covering pre-defined topics. Thus, we excluded didactic sessions where 1) the content of the presentations was unavailable for audit, 2) they covered administrative aspects and did not have a pre-identified topical focus where learning was opportunistic (e.g. risk management meetings, labour ward audits), and 3) where the attendance was optional.

Mapping was conducted independently by two members of the study team (JJYY and CYW) with conflict resolved by a third member (RJL). The number of knowledge requirements fulfilled within a KA were expressed as a percentage.

Table 3. Type of CME presentations

IV. DISCUSSION

Our audit revealed a relatively low coverage of the MRCOG KAs with only 64.9% of the syllabus covered. While the morning CME programme caters to all members of the department, the sessions are an important didactic component for ST education and exam preparation as they are deemed “protected” teaching time. There had been no prior formal review assessing whether it catered to this very important section of the department’s workforce. We were also able to recognise those KAs which had exceptionally low coverage were those with a large amount of practical and “hands-on” skills (i.e. “Gynaecological Problems”, “Management of Labour”, “Management of Delivery”, and “Postpartum Problems”). As a surgical discipline, this highlighted that these areas needed directed solutions through other forms of practical instruction and evaluation. In the pandemic environment, this may involve virtual or home-based means (Hoopes et al., 2020). These “hands-on” KAs likely require at least semi-annual individualised assessment by the CF through verified case logs, Objective Structured Assessment of Technical Skills, Direct Observation of Procedural Skills, and Non-Technical Skills for Surgeons (NOTSS) (Bisson et al., 2006; Parry-Smith et al., 2014). This targeted assessment was even more crucial during the recovery “catch-up” phase due to de-skilling because of reduced elective surgical caseload (Amparore et al., 2020; Chan et al., 2020) and facilitated the redistributing of surgical training material to cover training deficits.

While there is significant literature on how to organise a robust PG didactic programme (Colman et al., 2020; Harden, 2001; Willett, 2008), little has been published on how to evaluate an established didactic programme’s coverage of its learner’s educational requirements (Davies, 1981). Most studies evaluating the efficacy of such programmes typically assess the effects of individual CME sessions on physician knowledge or performance and patient outcomes after a suitable interval (Davis et al., 1992; Mansouri & Lockyer, 2007), with most citing a small to medium effect. We believe, however, that our audit process permits a more holistic, reproducible, and structured means of evaluating an existing didactic programme and finding deficits that can be improve upon to brings value to any specialty training programme.

At our institution, safe distancing requirements brought on by the COVID-19 pandemic required a rapid transition to a video-conferencing-based approach i.e. vCME. As milestone examinations were still being held, the first six months were primarily focused on STs as examination preparation remained a high and undisputed priority and learning opportunities had been significantly disrupted by the pandemic. During this phase, our vCME programme was re-organised into three to four sessions per week which were peer-led and supervised by a faculty member. Video-conferencing platforms encouraged audience participation through live feedback, questions posed via the chat box, instantaneous online polling, and directed case-based discussions with ST participants. These facilitated real time feedback to the presenter in a way that was not possible in previous face-to-face sessions due to reasons such as shyness and difficulty conducting polls. Other useful features included being able to record presentations for digital storage in a hospital-based server for access on-demand for revision purposes by STs.

A previously published anonymised questionnaire within our department (Chan et al., 2020) found very favourable opinions of vCME as an effective mode of learning amongst 28 junior doctors (85.7%) and nine presenters (100%) with 75% hoping for it to continue even after normalisation of social distancing policies. Nevertheless, common issues reported included a lack of personal interaction, difficulties in engaging with speakers, technical difficulties, and inaccurate attendance confirmation as shared devices for participating on these vCME sessions sometimes failed to identify who was present. While there is altered teacher and learner engagement due to physical separation across a digital medium, studies have also found that the virtual platform provided a useful means of communication and feedback and created a psychologically safe learning environment (Dong et al., 2021; Wasfy et al., 2021).

While our audit focused primarily on STs, departmental CME programmes need to find balanced in catering to the educational outcome of various groups of participants within a clinical department (e.g. senior clinicians, nursing staff, allied healthcare professionals, clinical scientists). Indeed, as these groups started to return to the CME programme after about six months following the vCME transition, we created a core postgraduate committee comprising members representing the learning interests of each party i.e. Department research director, ST Programme Director and Assistant Director, and a representative senior ST in the fifth or sixth year of training, so that we could continue to meet the recommendations set in our TNA while rebalancing the programme to meet the needs of all participants. Out of an average of 20 CME sessions per month, four were dedicated to departmental and hospital grand rounds each. Of the remaining 12 sessions, two were dedicated towards covering KAs, four scientific presentations, three clinical governance aspects, and one journal club. The remaining two sessions would be “faculty wildcard” sessions to be used at the committee’s discretion of the committee to cover poorly covered, more contemporary, “breaking news” topics, or serve as a buffer in the event of cancellations of other topics. Indeed, the same TNA-based audit methodology can be employed any other group of CME participants.

A key limitation in our audit method is that it focused on the breadth of coverage of learning objectives, but not the quality of the teaching and its depth. Teaching efficacy is also important in the delivery of learning objectives (Bakar et al., 2012) and needs more specific assessment tools (Metheny et al., 2005). Evaluating the quality of PG training could take several forms and may be direct e.g. an evaluation by the learner (Gillan et al., 2011), or could be indirect e.g. charting the learner’s progress through OSCEs and CEXs, scheduled competency reviews, and ST examination pass rates (Pinnell et al., 2021). Importantly, despite the rise of virtual learning platforms, there is little consensus on the best way to evaluate e-learning methods (De Leeuw et al., 2019). Nevertheless, our main audit goal was to assess the extent of coverage of the MRCOG syllabus which is a key training outcome. Future audits, however, should incorporate this element to provide additional qualitative feedback to assess this dimension as well. Further research should be carried out in terms of evaluating the effects of optimising a PG didactic programme on key outcomes such as ST behaviour, perceptions, and objective outcomes such as examination results.

Finally, while these were the results of an audit conducted in a single hospital department and used a morning CME programme as a basis for evaluation, we believe that this audit methodology based on a ten-step TNA and also utilising the Delphi method and syllabus mapping techniques (Harden, 2001) can be reproduced to any academic department that has a regular didactic programme as long as a suitable standard is selected. The Delphi method can easily be conducted via online survey platforms (e.g. Google Forms) to crystallise the PG programme goals. Our audit shows that without a systematic evaluation of past didactic sessions, it is possible for even mature CME programme to fall significantly short of ameeting the needs of its learners and that PG didactic sessions need deliberate planning.

V. CONCLUSION

Just as any other aspect of healthcare delivery, CME and PG programmes are amenable to audits and must adjust to an ever-changing delivery landscape. Rather than curse the darkness during the COVID-19 pandemic, we explored the potential of reformatting the PG programme and adjusting course to better suit the needs of our STs. We demonstrate a method of auditing an existing programme, distilling important learning goals, comparing it against an appropriate standard (i.e. coverage of the MRCOG KAs), and implementing changes utilising reproducible techniques such as the Delphi method (Humphrey-Murto et al., 2017). This process should be a regular mainstay of any mature ST programme to ensure continued relevancy. As continual outbreaks, even amongst vaccinated populations (Amit et al., 2021; Bar-On et al., 2021; Bergwerk et al., 2021) auger a future of COVID-19 endemicity, we must accept a “new-normal” comprising of intermittent workplace infection control policies such as segregation, shift work, and restrictions for in-person meetings (Kwon et al., 2020; Liang et al., 2020). Through our experience, we have shown that this auditing methodology can also be applied to vCME programmes.

Notes on Contributors

Rachel Jiayu Lee participated in the data collection and review, the writing of the paper, and the formatting for publication.

Jeannie Jing Yi Yap participated in the data collection and review, the writing of the paper, and the formatting for publication.

Carly Yanlin Wu participated in data collection and review.

Grace Chan Ming Fen participated in data collection and review.

Abhiram Kanneganti was involved in the writing of the paper, editing, and formatting for publication. Citra Nurfarah Zaini Mattar participated in the editing and direction of the paper.

Pearl Shuang Ye Tong participated in the editing and direction of the paper.

Susan Jane Sinclair Logan participated in the editing and direction of the paper.

Ethical Approval

IRB approval for waiver of consent (National Healthcare Group DSRB 2020/00360) was obtained for the questionnaire assessing attitudes towards vCME.

Data Availability

There is no relevant data available for sharing in this paper.

Acknowledgement

We would like to acknowledge the roles of Mr Xiu Cai Wong Edwin, Mr Lee Boon Kai and Ms Teo Xin Yue in the administrative roles behind auditing and reformatting the PG medical education programme.

Funding

There was no funding for this article.

Declaration of Interest

The authors have no conflicts of interest in connection with this article.

References

Amit, S., Beni, S. A., Biber, A., Grinberg, A., Leshem, E., & Regev-Yochay, G. (2021). Postvaccination COVID-19 among healthcare workers, Israel. Emerging Infectious Diseases, 27(4), 1220. https://doi.org/10.3201/eid2704.210016

Amparore, D., Claps, F., Cacciamani, G. E., Esperto, F., Fiori, C., Liguori, G., Serni, S., Trombetta, C., Carini, M., Porpiglia, F., Checcucci, E., & Campi, R. (2020). Impact of the COVID-19 pandemic on urology residency training in Italy. Minerva Urologica e Nefrologica, 72(4), 505-509. https://doi.org/10.23736/s0393-2249.20.03868-0

Bakar, A. R., Mohamed, S., & Zakaria, N. S. (2012). They are trained to teach, but how confident are they? A study of student teachers’ sense of efficacy. Journal of Social Sciences, 8(4), 497-504. https://doi.org/10.3844/jssp.2012.497.504

Bar-On, Y. M., Goldberg, Y., Mandel, M., Bodenheimer, O., Freedman, L., Kalkstein, N., Mizrahi, B., Alroy-Preis, S., Ash, N., Milo, R., & Huppert, A. (2021). Protection of BNT162b2 vaccine booster against Covid-19 in Israel. New England Journal of Medicine, 385(15), 1393-1400. https://doi.org/10.1056/NEJMoa2114255

Bergwerk, M., Gonen, T., Lustig, Y., Amit, S., Lipsitch, M., Cohen, C., Mandelboim, M., Levin, E. G., Rubin, C., Indenbaum, V., Tal, I., Zavitan, M., Zuckerman, N., Bar-Chaim, A., Kreiss, Y., & Regev-Yochay, G. (2021). Covid-19 breakthrough infections in vaccinated health care workers. New England Journal of Medicine, 385(16), 1474-1484. https://doi.org/10.1056/NEJMoa2109072

Bienstock, J., & Heuer, A. (2022). A review on the evolution of simulation-based training to help build a safer future. Medicine, 101(25), Article e29503. https://doi.org/10.1097/MD.0000000000029503

Bisson, D. L., Hyde, J. P., & Mears, J. E. (2006). Assessing practical skills in obstetrics and gynaecology: Educational issues and practical implications. The Obstetrician & Gynaecologist, 8(2), 107-112. https://doi.org/10.1576/toag.8.2.107.27230

Bryant‐Smith, A., Rymer, J., Holland, T., & Brincat, M. (2019). ‘Perfect practice makes perfect’: The role of laparoscopic simulation training in modern gynaecological training. The Obstetrician & Gynaecologist, 22(1), 69-74. https://doi.org/10.1111/tog.12619

Burr, R., & Johanson, R. (1998). Continuing medical education: An opportunity for bringing about change in clinical practice. British Journal of Obstetrics and Gynaecology, 105(9), 940-945. https://doi.org/10.1111/j.1471-0528.1998.tb10255.x 

Chan, G. M. F., Kanneganti, A., Yasin, N., Ismail-Pratt, I., & Logan, S. J. S. (2020). Well-being, obstetrics and gynaecology and COVID-19: Leaving no trainee behind. Australian and New Zealand Journal of Obstetrics and Gynaecology, 60(6), 983-986. https://doi.org/10.1111/ajo.13249 

Chick, R. C., Clifton, G. T., Peace, K. M., Propper, B. W., Hale, D. F., Alseidi, A. A., & Vreeland, T. J. (2020). Using technology to maintain the education of residents during the COVID-19 pandemic. Journal of Surgical Education, 77(4), 729-732. https://doi.org/10.1016/j.jsurg.2020.03.018

Colman, S., Wong, L., Wong, A. H. C., Agrawal, S., Darani, S. A., Beder, M., Sharpe, K., & Soklaridis, S. (2020). Curriculum mapping: an innovative approach to mapping the didactic lecture series at the University of Toronto postgraduate psychiatry. Academic Psychiatry, 44(3), 335-339. https://doi.org/10.1007/s40596-020-01186-0

Davies, I. J. T. (1981). The assessment of continuing medical education. Scottish Medical Journal, 26(2), 125-134. https://doi.org/10.1177/003693308102600208

Davis, D. A., Thomson, M. A., Oxman, A. D., & Haynes, R. B. (1992). Evidence for the effectiveness of CME: A Rreview of 50 randomized controlled trials. Journal of the American Medical Association, 268(9), 1111-1117. https://doi.org/10.1001/jama.1992.03490090053014

De Leeuw, R., De Soet, A., Van Der Horst, S., Walsh, K., Westerman, M., & Scheele, F. (2019). How we evaluate postgraduate medical e-learning: systematic review. JMIR medical education, 5(1), e13128.

Dong, C., Lee, D. W.-C., & Aw, D. C.-W. (2021). Tips for medical educators on how to conduct effective online teaching in times of social distancing. Proceedings of Singapore Healthcare, 30(1), 59-63. https://doi.org/10.1177/2010105820943907

Donovan, P., & Townsend, J. (2019). Learning Needs Analysis. Management Pocketbooks.

English, W., Vulliamy, P., Banerjee, S., & Arya, S. (2020). Surgical training during the COVID-19 pandemic – The cloud with a silver lining? British Journal of Surgery, 107(9), e343-e344. https://doi.org/10.1002/bjs.11801

Forsetlund, L., O’Brien, M. A., Forsén, L., Mwai, L., Reinar, L. M., Okwen, M. P., Horsley, T., & Rose, C. J. (2021). Continuing education meetings and workshops: Effects on professional practice and healthcare outcomes. Cochrane Database of Systematic Reviews, 9(9), CD003030. https://doi.org/10.1002/14651858.CD003030.pub3

Gillan, C., Lovrics, E., Halpern, E., Wiljer, D., & Harnett, N. (2011). The evaluation of learner outcomes in interprofessional continuing education: A literature review and an analysis of survey instruments. Medical Teacher, 33(9), e461-e470. https://doi.org/10.3109/0142159X.2011.587915

Gould, D., Kelly, D., White, I., & Chidgey, J. (2004). Training needs analysis. A literature review and reappraisal. International Journal of Nursing Studies, 41(5), 471-486. https://doi.org/10.1016/j.ijnurstu.2003.12.003

Grafton-Clarke, C., Uraiby, H., Gordon, M., Clarke, N., Rees, E., Park, S., Pammi, M., Alston, S., Khamees, D., Peterson, W., Stojan, J., Pawlik, C., Hider, A., & Daniel, M. (2022). Pivot to online learning for adapting or continuing workplace-based clinical learning in medical education following the COVID-19 pandemic: A BEME systematic review: BEME Guide No. 70. Medical Teacher, 44(3), 227-243. https://doi.org/10.1080/0142159X.2021.1992372

Harden, R. M. (2001). AMEE Guide No. 21: Curriculum mapping: A tool for transparent and authentic teaching and learning. Medical Teacher, 23(2), 123-137. https://doi.org/10.1080/01421590120036547

Hasson, F., Keeney, S., & McKenna, H. (2000). Research guidelines for the Delphi survey technique. Journal of Advanced Nursing, 32(4), 1008-1015. https://doi.org/10.1046/j.1365-2648.2000.t01-1-01567.x

Hicks, C., & Hennessy, D. (1996). Applying psychometric principles to the development of a training needs analysis questionnaire for use with health visitors, district and practice nurses. NT Research, 1(6), 442-454. https://doi.org/10.1177/174498719600100608  

Hicks, C., & Hennessy, D. (1997). The use of a customized training needs analysis tool for nurse practitioner development. Journal of Advanced Nursing, 26(2), 389-398. https://doi.org/https://doi.org/10.1046/j.1365-2648.1997.199702 6389.x

Hoopes, S., Pham, T., Lindo, F. M., & Antosh, D. D. (2020). Home surgical skill training resources for obstetrics and gynecology trainees during a pandemic. Obstetrics and Gynecology 136(1), 56-64. https://doi.org/10.1097/aog.0000000000003931

Humphrey-Murto, S., Varpio, L., Wood, T. J., Gonsalves, C., Ufholz, L.-A., Mascioli, K., Wang, C., & Foth, T. (2017). The use of the Delphi and other consensus group Methods in medical education research: A review. Academic Medicine, 92(10), 1491-1498. https://doi.org/10.1097/ACM.0000000000001812

Johnston, S., Coyer, F. M., & Nash, R. (2018). Kirkpatrick’s evaluation of simulation and debriefing in health care education: a systematic review. Journal of Nursing Education, 57(7), 393-398. https://doi.org/10.3928/01484834-20180618-03

Juprasert, J. M., Gray, K. D., Moore, M. D., Obeid, L., Peters, A. W., Fehling, D., Fahey, T. J., III, & Yeo, H. L. (2020). Restructuring of a general surgery residency program in an epicenter of the Coronavirus Disease 2019 pandemic: Lessons from New York City. JAMA Surgery, 155(9), 870-875. https://doi.org/10.1001/jamasurg.2020.3107

Kanneganti, A., Lim, K. M. X., Chan, G. M. F., Choo, S.-N., Choolani, M., Ismail-Pratt, I., & Logan, S. J. S. (2020). Pedagogy in a pandemic – COVID-19 and virtual continuing medical education (vCME) in obstetrics and gynecology. Acta Obstetricia et Gynecologica Scandinavica, 99(6), 692-695. https://doi.org/10.1111/aogs.13885

Kanneganti, A., Sia, C.-H., Ashokka, B., & Ooi, S. B. S. (2020). Continuing medical education during a pandemic: An academic institution’s experience. Postgraduate Medical Journal, 96, 384-386. https://doi.org/10.1136/postgradmedj-2020-137840

Khamees, D., Peterson, W., Patricio, M., Pawlikowska, T., Commissaris, C., Austin, A., Davis, M., Spadafore, M., Griffith, M., Hider, A., Pawlik, C., Stojan, J., Grafton-Clarke, C., Uraiby, H., Thammasitboon, S., Gordon, M., & Daniel, M. (2022). Remote learning developments in postgraduate medical education in response to the COVID-19 pandemic – A BEME systematic review: BEME Guide No. 71. Medical Teacher, 44(5), 466-485. https://doi.org/10.1080/0142159X.2022.2040732

Kwon, Y. S., Tabakin, A. L., Patel, H. V., Backstrand, J. R., Jang, T. L., Kim, I. Y., & Singer, E. A. (2020). Adapting urology residency training in the COVID-19 era. Urology, 141, 15-19. https://doi.org/10.1016/j.urology.2020.04.065

Liang, Z. C., Ooi, S. B. S., & Wang, W. (2020). Pandemics and their impact on medical training: Lessons from Singapore. Academic Medicine, 95(9), 1359-1361. https://doi.org/10.1097/ACM.0000000000003441

Lim, E. C., Oh, V. M., Koh, D. R., & Seet, R. C. (2009). The challenges of “continuing medical education” in a pandemic era. Annals Academy of Medicine Singapore, 38(8), 724-726. https://www.ncbi.nlm.nih.gov/pubmed/19736579

Mansouri, M., & Lockyer, J. (2007). A meta-analysis of continuing medical education effectiveness. Journal of Continuing Education in the Health Professions, 27(1), 6-15. https://doi.org/10.1002/chp.88

Marinopoulos, S. S., Dorman, T., Ratanawongsa, N., Wilson, L. M., Ashar, B. H., Magaziner, J. L., Miller, R. G., Thomas, P. A., Prokopowicz, G. P., Qayyum, R., & Bass, E. B. (2007). Effectiveness of continuing medical education. Evid Rep Technol Assess (Full Rep), 149, 1-69. https://www.ncbi.nlm.nih.gov/pubmed/17764217

Markaki, A., Malhotra, S., Billings, R., & Theus, L. (2021). Training needs assessment: Tool utilization and global impact. BMC Medical Education, 21(1), Article 310. https://doi.org/10.1186/s12909-021-02748-y

Metheny, W. P., Espey, E. L., Bienstock, J., Cox, S. M., Erickson, S. S., Goepfert, A. R., Hammoud, M. M., Hartmann, D. M., Krueger, P. M., Neutens, J. J., & Puscheck, E. (2005). To the point: Medical education reviews evaluation in context: Assessing learners, teachers, and training programs. American Journal of Obstetrics and Gynecology, 192(1), 34-37. https://doi.org/10.1016/j.ajog.2004.07.036

Mishra, K., Boland, M. V., & Woreta, F. A. (2020). Incorporating a virtual curriculum into ophthalmology education in the coronavirus disease-2019 era. Current Opinion in Ophthalmology, 31(5), 380-385. https://doi.org/10.1097/icu.0000000000000681

Norman, G. R., Shannon, S. I., & Marrin, M. L. (2004). The need for needs assessment in continuing medical education. BMJ (Clinical research ed.), 328(7446), 999-1001. https://doi.org/10.1136/bmj.328.7446.999

Palmer, W., & Brackwell, L. (2014). A national audit of maternity services in England. BJOG, 121(12), 1458-1461. https://doi.org/10.1111/1471-0528.12973

Parry-Smith, W., Mahmud, A., Landau, A., & Hayes, K. (2014). Workplace-based assessment: A new approach to existing tools. The Obstetrician & Gynaecologist, 16(4), 281-285. https://doi.org/10.1111/tog.12133

Pinnell, J., Tranter, A., Cooper, S., & Whallett, A. (2021). Postgraduate medical education quality metrics panels can be enhanced by including learner outcomes. Postgraduate Medical Journal, 97(1153), 690-694. https://doi.org/10.1136/postgradmedj-2020-138669

Raza, A., Coomarasamy, A., & Khan, K. S. (2009). Best evidence continuous medical education. Archives of Gynecology and Obstetrics, 280(4), 683-687. https://doi.org/10.1007/s00404-009-1128-7

Renaud, C. J., Chen, Z. X., Yuen, H.-W., Tan, L. L., Pan, T. L. T., & Samarasekera, D. D. (2021). Impact of COVID-19 on health profession education in Singapore: Adoption of innovative strategies and contingencies across the educational continuum. The Asia Pacific Scholar, 6(3), 14-23. https://doi.org/10.29060/TAPS.2021-6-3/RA2346

Royal College of Obstetricians and Gynaecologists. (2019). Core Curriculum for Obstetrics & Gynaecology: Definitive Document 2019. https://www.rcog.org.uk/media/j3do0i1i/core-curriculum-2019-definitive-document-may-2021.pdf

Sibley, J. C., Sackett, D. L., Neufeld, V., Gerrard, B., Rudnick, K. V., & Fraser, W. (1982). A randomized trial of continuing medical education. New England Journal of Medicine, 306(9), 511-515. https://doi.org/10.1056/NEJM198203043060904

Sparkes, D., Leong, C., Sharrocks, K., Wilson, M., Moore, E., & Matheson, N. J. (2021). Rebooting medical education with virtual grand rounds during the COVID-19 pandemic. Future Healthc J, 8(1), e11-e14. https://doi.org/10.7861/fhj.2020-0180

Wasfy, N. F., Abouzeid, E., Nasser, A. A., Ahmed, S. A., Youssry, I., Hegazy, N. N., Shehata, M. H. K., Kamal, D., & Atwa, H. (2021). A guide for evaluation of online learning in medical education: A qualitative reflective analysis. BMC Medical Education, 21(1), Article 339. https://doi.org/10.1186/s12909-021-02752-2

Willett, T. G. (2008). Current status of curriculum mapping in Canada and the UK. Medical Education, 42(8), 786-793. https://doi.org/10.1111/j.1365-2923.2008.03093.x

Wong, T. Y., & Bandello, F. (2020). Academic ophthalmology during and after the COVID-19 pandemic. Ophthalmology, 127(8), e51-e52. https://doi.org/10.1016/j.ophtha.2020.04.029

Xu, J., Zhou, Z., Chen, K., Ding, Y., Hua, Y., Ren, M., & Shen, Y. (2022). How to minimize the impact of COVID-19 on laparoendoscopic single-site surgery training? ANZ Journal of Surgery, 92, 2102-2108. https://doi.org/10.1111/ans.17819

*Abhiram Kanneganti
Department of Obstetrics and Gynaecology,
NUHS Tower Block, Level 12,
1E Kent Ridge Road,
Singapore 119228
Email: abhiramkanneganti@gmail.com

Submitted: 30 June 2022
Accepted: 31 October 2022
Published online: 4 July, TAPS 2023, 8(3), 26-34
https://doi.org/10.29060/TAPS.2023-8-3/OA2834

Noorjahan Haneem Md Hashim¹, Shairil Rahayu Ruslan¹, Ina Ismiarti Shariffuddin¹, Woon Lai Lim¹, Christina Phoay Lay Tan² & Vinod Pallath³

¹Department of Anaesthesiology, Faculty of Medicine, Universiti Malaya, Malaysia; ²Department of Primary Care Medicine, Faculty of Medicine, Universiti Malaya, Malaysia; ³Medical Education Research & Development Unit, Dean’s Office, Faculty of Medicine, Universiti Malaya, Malaysia

Abstract

Introduction: Examiner training is essential to ensure the trustworthiness of the examination process and results. The Anaesthesiology examiners’ training programme to standardise examination techniques and standards across seniority, subspecialty, and institutions was developed using McLean’s adaptation of Kern’s framework.

Methods: The programme was delivered through an online platform due to pandemic constraints. Key focus areas were Performance Dimension Training (PDT), Form-of-Reference Training (FORT) and factors affecting validity. Training methods included interactive lectures, facilitated discussions and experiential learning sessions using the rubrics created for the viva examination. The programme effectiveness was measured using the Kirkpatrick model for programme evaluation.

Results: Seven out of eleven participants rated the programme content as useful and relevant. Four participants showed improvement in the post-test, when compared to the pre-test. Five participants reported behavioural changes during the examination, either during the preparation or conduct of the examination.  Factors that contributed to this intervention’s effectiveness were identified through the MOAC (motivation, opportunities, abilities, and communality) model.

Conclusion: Though not all examiners attended the training session, all were committed to a fairer and transparent examination and motivated to ensure ease of the process. The success of any faculty development programme must be defined and the factors affecting it must be identified to ensure engagement and sustainability of the programme.

Keywords:           Medical Education, Health Profession Education, Examiner Training, Faculty Development, Assessment, MOAC Model, Programme Evaluation

Practice Highlights

• A faculty development initiative must be tailored to faculty’s learning needs and context.
• A simple framework of planning, implementing, and evaluating can be used to design a programme.
• Target outcome measures and evaluation plans must be included in the planning process.
• The Kirkpatrick model is a useful tool to use in programme evaluation: to answer if the programme has met its objectives.
• The MOAC model is a useful tool to explain why a programme has met its objective.

I. INTRODUCTION

Anaesthesiology specialist training in Malaysia comprises a 4-year clinical master’s programme. At the time of our workshop, five local public universities offer the programme. The course content is similar in all universities, but the course delivery may differ to align with each university’s rules and regulations. The summative examinations are held as a Conjoint Examination. Examiners include lecturers from all five universities, specialists from the Ministry of Health and external examiners from international Anaesthesiology training programmes. The examination consists of a written and a viva voce examination. The areas examined are the knowledge and cognitive skills in patient management.

A speciality training programme’s exit level assessment is an essential milestone for licensing. In our programme, the exit examination occurs at the end of the training before trainees practise independently in the healthcare system and are eligible for national specialist registration. Therefore, aligning the curriculum and assessment to licensing requirements is necessary.

Examiners play an important role during this high-stakes summative examination, making decisions regarding allowing graduating trainees to work as specialists in the community. Therefore, examiners must understand their role. In recent years, the anaesthesiology training programme providers in Malaysia have been taking measures to improve the validity of the examination. These include a stringent vetting process to ensure examination content reflects the syllabus, questions are unambiguous, and the examiners agree on the criteria for passing. However, previous examinations revealed that although examiners were clear on the aim of the examination, some utilised different assessment approaches, which were possibly coloured by personal and professional experiences, and thus needed constant calibration on the passing criteria. In addition, during examiner discussions, different examiners were found to have different skill levels in constructing focused higher-order questions and were not fully aware of potential cognitive biases that may affect the examination results.

These insights from previous examinations warranted a specific skill training session to ensure the trustworthiness of the examination process and results (Blew et al., 2010; Iqbal et al., 2010; Juul et al., 2019, Chapter 8, pp. 127-140; McLean et al., 2008). The examiners and the Specialty committee were keen to ensure that these issues were addressed with a training programme that complements the current on-the-job examiner training.

II. METHODS

An examiner training module was developed using McLean’s adaptation of Kern’s framework for curriculum development: Planning, Implementation and Evaluation (McLean et al., 2008; Thomas et al., 2015). A conceptual framework for the examiner training programme was drawn up from the programme’s conception stage to the evaluation of its outcome, as illustrated in Figure 1 (Steinert et al., 2016).

Figure 1: The conceptual framework for the examiner training programme and evaluation of its effectiveness

A. Planning

Three key focus areas were identified for the training programme: (1) Performance Dimension Training (PDT); (2) examiner calibration with Frame-Of-Reference Training (FORT); as well as (3) identifying factors affecting the validity of results and measures that can be taken to prevent them.

1) Performance dimension training (Feldman et al., 2012): The aim was to improve examination validity by reducing examiner errors or biases unrelated to the examinees’ targeted performance behaviours. Finalised marking schemes outlining competencies to be assessed required agreement by all the examiners ahead of time. These needed to be clearly defined and easily understood by all the examiners, and consistency was key to reducing examiner bias.

2) Examiner calibration with Frame-of-Reference Training (FORT) (Newman et al., 2016): Differing levels of experience among all the participants meant that there were differing expectations and performances among them. The examiner training programme needed to assist examiners in resetting expectations and criteria for assessing the candidates’ competencies. This examiner calibration was achieved using pre-recorded simulated viva sessions in which the participants rated candidates’ performances in each simulated viva session and received immediate feedback on their ability and criteria for scoring the candidates.

3) Identifying factors affecting the validity of results (Lineberry, 2019): Factors that may affect the validity of examination results may be related to construct underrepresentation (CU), where the results only reflect one part of an attribute being examined; or construct-irrelevant variance (CIV), where the results are being affected by areas or issues other than the attribute being examined.

An example of CU is sampling issues where only a limited area of the syllabus is examined, or an answer key is limited by the availability of evidence or content expertise.

Examples of CIV include the different ways a concept can be interpreted in different cultures or training centres, ambiguous questions, examiner cognitive biases, examiner fatigue, examinee language abilities, and examinees guessing or cheating. The examiner training programme was designed with the objectives listed in Table 1.

B. Implementation

The faculty intervention programme was designed as a one-day online programme to be attended by potential examiners for the Anaesthesiology Exit Examination. The programme objectives were prioritised from the needs assessment and designed based on Tekian & Norcini’s recommendations (Tekian & Norcini, 2016). Due to time constraints, training was performed using an online platform closer to the examination dates after obtaining university clearance on confidentiality regarding assessment issues.

The structure and contents of the examiner training programme are outlined in Table 2 and is further elaborated in Appendix A.

Based on the objectives, the organisers invited a multidisciplinary group of facilitators. The group consisted of anaesthesiologists, medical education experts in assessment and faculty development, and a technical and logistics support team to ensure efficient delivery of the online programme.

A multimodal approach to delivery was adopted to accommodate the diversity of the examiner group (gender, seniority, subspeciality, and examination experience). Explicit ground rules were agreed upon to underpin the safe and respectful learning environment. The educational strategy included interactive lectures, hands-on practice using rubrics created and calibration using video-assisted scenarios. The programme objectives were embedded and reinforced with each strategy. Pre- and post-tests were performed to help participants gauge their learning and assist the programme organisers in evaluating the participants’ learning.

This would be the first time such a programme was held within the local setting. Participants were all anaesthesiologists by profession, were actively involved in clinical duties within a tertiary hospital setting and consented to participate in this programme. As potential examiners, they all had prior experience as observers of the examination process, with the majority having previous experience as examiners as well.

The programme was organised during the peak of the COVID-19 pandemic and was managed on a fully online platform to ensure safety and minimise the time taken away from clinical duties. In addition, participants received protected time for this programme, a necessary luxury as anaesthesiologists were at the forefront of managing the pandemic.

C. Evaluation

The Kirkpatrick model (McLean et al., 2008; Newstrom, 1995;) was used to evaluate the programme’s effectiveness described and elaborated in Figure 2.

Figure 2: The Kirkpatrick model, elaborated for this programme

The MOAC model (Vollenbroek, 2019), expanded from the MOA (Marin-Garcia & Martinez Tomas, 2016) model by Blumberg & Pringle (Blumberg & Pringle, 1982) was used to examine factors that contributed to the effectiveness of the programme. Motivation, opportunity, ability, and communality are factors that drives action and performance.

III. RESULTS

Eleven participants attended the programme. These participants were examiners for the 2021 examinations from the university training centres and the Ministry of Health, Malaysia. Only one of the participants would be a first-time examiner in the Exit Examination. Four of the would-be examiners could not attend due to service priorities.

A. Level 1: Reaction

Seven of the eleven participants completed the programme evaluation form, which is openly available in Figshare at https://doi.org/10.6084/m9.figshare.20189309.v1 (Tan & Pallath, 2022). All of them rated the programme content as useful and relevant to their examination duties and stated that the content and presentations were pitched at the correct level, with appropriate visual aids and reference materials. The online learning opportunity was also rated as good.

All seven also aimed to make behavioural changes after attending the programme, as indicated below. Some of the excerpts include:

“I am more cognizant of the candidates’ understanding to questions and marking schemes”

“Yes. We definitely need the rubric/marking scheme for standardisation. Will also try to reduce all the possible biases as mentioned in the programme.”

“Yes, as I will be more agreeable to question standardisation in viva examination because it makes it fairer for the candidates.”

The participants also shared their understanding of the importance of standardisation and examiner training and would recommend this programme to be conducted annually. They agreed that the examiner training programme should be made mandatory for all new examiners, with the option of refresher courses for veteran examiners if appropriate.

B. Level 2: Learning

All 11 participants completed the pre-and post-tests. The data supporting these findings of this is openly available in Figshare at https://doi.org/10.6084/m9.figshare.20186582.v1 (Md Hashim, 2021). The participants’ marks in both tests are shown in Appendix B. The areas that showed improvement in scores were identifying why under-sampling is a problem and methods to prevent validity threats. Understanding the source of validity threat from cognitive biases showed a decline in scores (question 2 with scores of 11 to 8 and question 3 with scores of 10 to 8), respectively.

Comparing the post-test scores to pre-test scores, four participants showed improvement, four showed no change (one of the participants answered all questions correctly in both tests) and three participants showed a decline in test scores.

C. Level 3: Behavioural Change

Six participants responded to the follow-up questionnaire, which is openly available in Figshare at https://doi.org/10.6084/m9.figshare.20186591.v2 (Md Hashim, 2022). This questionnaire was administered about a year after the examiner training programme and after the completion of two examinations. Only one respondent did not make any self-perceived behavioural change while preparing the examination questions and conducting the viva examinations. Two respondents did not make any changes while marking or rating candidates.

The specific changes in the three areas of behavioural change that were consciously noted by the respondents were explored. Respondents reported increased awareness and being more systematic in question preparation, making questions more aligned to the curriculum, preparing better quality questions, and being more cognizant of candidates’ understanding of the questions.

They also reported being more objective and guided during marking and rating as the passing criteria were better defined and structured.

Regarding the conduct of the viva examination, respondents shared that they were better prepared during vetting and felt it was easier to rate candidates as the marking schemes and questions were standardised and could ensure candidates could answer all the required questions to pass.

D. Level 4: Results

The examiners who attended the training programme were able to prepare questions as blueprinted and were able to identify areas to be examined and provided recommended criteria for passing each question. This has led to a smooth vetting process and examination.

E. Factors Affecting Effectiveness

Even though the programme was not attended by all the potential examiners, those who did were committed to the idea of a fairer and more transparent examination process. This formed the motivation aspect of the model.

In terms of opportunity, protected training time is important, followed by prioritising the content of the training material according to the most pressing needs.

The ability aspect encompassed the abilities of the facilitators and participants. To emphasise the learning process, credible trainers were invited to this programme to facilitate the lectures and experiential learning sessions. In this aspect, the Faculty Development team comprised an experienced clinician, a basic medical scientist, and an anaesthesiologist, all with medical education qualifications and were vital in ensuring the success of this programme. The whole team was led by the Chief Examiner who focused on the dimensions to be tested and calibrated, while simultaneously managing the expectations of the examiners and their abilities to give and accept feedback. Communication and the skill to be receptive to the proposed changes were also crucial to make the intervention work.

In terms of communality, all the participants were of similar professional backgrounds and shared the common realisation that this training programme was essential and would only yield positive results. Hence this ensured the programme’s overall success.

IV. DISCUSSION

The progressive change seen in this attempt to improve the examination system is aligned with the general progress in medical education. Training of examiners is important (Holmboe et al., 2011), as it is not the tool used for assessment, but rather the person using the tool, that makes the difference. As it is difficult to design the ‘perfect tool’ for performance tests and redesigning a tool only changes 10% of the variance in rating (Holmboe et al., 2011; Williams et al., 2003), educators must now train the faculty in observation and assessment. It is not irrational to extrapolate this effect on written and oral examinations. Holmboe et al. (2011) also share the reasons for a training programme for assessors, which are changing curriculum structure, content and delivery and emerging evidence regarding assessment, building a system reserve, utilising training programmes as opportunities to identify and engage change agents and allow the faculty to form a mental picture of how changes will affect them and improve practice. Enlisting the help and support of a respected faculty member during training will promote the depth and breadth of change.

Khera et al. (2005) described their paediatric examination experiences, in which the Royal College of Paediatrics and Child Health defined examiners’ competencies, selection process and training programme components. The training programme included principles of assessment, examination design, writing questions, interpersonal skills, professional attributes, managing diversity, and assessing the examiners’ skills. They believe these contents will ensure the assessment is valid, reliable, and fair. As Anaesthesiology examiners have different knowledge levels and experiences, it had been crucial to assess their learning needs and provide them with appropriate learning opportunities.

In the emergency brought on by the COVID-19 pandemic, online training was the safest and most feasible platform for conducting this programme. Online faculty development activities have the perceived advantages of being convenient, flexible, and allowing interdisciplinary interaction and providing an experience of being an online student(Cook & Steinert, 2013). Forming the facilitation team together with the dedicated technical and logistics team and creating a chat group prior to conducting the programme were key in anticipating and handling communication and technical issues (Cook & Steinert, 2013).

Though participants were engaged and the results of the workshop were encouraging, the programme delivery and the content will be reviewed based on the feedback received. The convenience of an online activity must be balanced with the participant engagement and facilitator presence of a face-to-face-activity. Since the results of both methods of delivery differs (Arias et al., 2018; Daniel, 2014; Kemp & Grieve, 2014), the best solution may to ask the participants what would best work for them, as they are adult learners and experienced examiners. The programme must be designed with participants involvement, with opportunities to participate and engaging facilitators and support teams that would be able to support the participants’ learning need (Singh et al., 2022).

At the end of the programme, the effectiveness of the programme was measured by referencing the Kirkpatrick model. The Kirkpatrick model (Newstrom, 1995; Steinert et al., 2006) was the most helpful in helping us identify the success of the intervention, which included behavioural change. Measuring behavioural change and impact on the examination results, organisational changes and changes in student learning may be difficult and may not be directly caused by a single intervention (McLean et al., 2008). The key, is perhaps to involve examiners, students and other stakeholders in the evaluation process, using various validated tools, and to ensure that the effort is ongoing, with sustained support, guidance and feedback (McLean et al., 2008).

To explain the overall effectiveness of the programme (with regards to reaction, learning and behavioural change), the MOAC model (Vollenbroek, 2019) expanded from the original MOA model was used. The MOAC model not only describes factors that affect an individual’s performance in a group, but also the group behaviour.

Motivation is an important driving force of action, and members are more motivated when a subject becomes relevant on a personal level, leading to action. The motivation to be informed and to improve has led to active participation in the knowledge sharing session, processing new information presented in the programme and adopting changes learnt during the programme. Presence of a group of motivated individuals with the same goals supported each other’s learning.

Opportunity, especially time, space and resources, must be allocated to reflect the value and relevance of any activity. Work autonomy, allows professionals to engage in what they consider relevant or important, and be accountable for their work outcomes. Facilitating conditions, for example, technology, facilitators, and a platform to practise what is being learnt are also important aspects of opportunity. Allowing protected time with the appropriate facilitating conditions, indicates institutional support and has enabled participants to fully optimise the learning experience.

Ability positively affects knowledge exchange and willingness to participate. Having prior knowledge improves a participant’s ability to absorb and utilise new knowledge. The programme participants, being experienced clinical teachers and examiners are fully aware of their capabilities and are able to process and share important information. Experienced faculty development facilitators who are also clinical teachers and examiners were able to identify areas to focus and provide relevant examples for application.

Communality is the added dimension to the original MOA model. Participants of this programme are members in a complex system, who already know each other. Having shared identity, language and challenges have allowed them to develop trust while pursuing the common goal of improving the system they were working in. This facilitated knowledge sharing and behavioural change.

The limitation in our programme is the small sample size. However, we believe that is important to review the effectiveness of a programme, especially with regards to behavioural change, and to share how other programmes can benefit from using the frameworks we shared. The findings from this programme will also inform how we conduct future faculty development programmes. With pandemic restrictions lifted, we hope to conduct this programme face-to-face, to facilitate engagement and communication.

V. CONCLUSION

For this faculty development programme to succeed, targets for success must first be defined and factors that contribute to its success need to be identified. This will ensure active engagement from the participants and promote the sustainability of the programme.

Notes on Contributors

Noorjahan Haneem Md Hashim designed the programme, assisted in content creation, curation and matching learning activities, moderated the programme, and conceptualised and wrote this manuscript.

Shairil Rahayu Ruslan participated as a committee of the programme, assisted as a simulated candidate during the training sessions, as well as contributed to the conceptualisation, writing, and formatting of this manuscript. She also compiled the bibliography and cross-checked the references for this manuscript.

Ina Ismiarti Shariffuddin created the opportunity for the programme (Specialty board and interdisciplinary buy-in, department funding), prioritised the programme learning outcomes, chaired the programme, and contributed to the writing and review of this manuscript.

Woon Lai Lim participated as a committee member of the programme and contributed to the writing of this manuscript.

Christina Phoay Lay Tan designed and conducted the faculty development training programme, and reviewed and contributed to the writing of this manuscript. She also cross-checked the references for this manuscript.

Vinod Pallath designed and conducted the faculty development training programme, and reviewed and contributed to the writing of this manuscript.

All authors verified and approved the final version of the manuscript.

Ethical Approval

Ethical approval was applied for the follow-up questionnaire that was distributed to the participants, which was approved on the 6th of May 2022 (Reference number: UM.TNC2/UMREC_1879). The programme evaluation and pre- and post-tests are accepted as part of the programme evaluation procedures.

Data Availability

De-identified individual participant data collected are available in the Figshare repository immediately after publication without an end date, as below :

https://doi.org/10.6084/m9.figshare.20189309.v1

https://doi.org/10.6084/m9.figshare.20186582.v1

https://doi.org/10.6084/m9.figshare.20186591.v2

The authors confirm that all data underlying the findings are freely available for view from the Figshare data repository. However, the reuse and resharing of the programme evaluation form, pre- and posttest questions, as well as followup questionnaire, despite being easily accessible from the data repository, should warrant a reasonable request from the corresponding author out of courtesy.

Acknowledgement

The authors would like to acknowledge Dr Selvan Segaran and Dr Siti Nur Jawahir Rosli from the Medical Education, Research and Development Unit (MERDU) for their logistics and technical support in all stages of this programme; Professor Dr Jamuna Vadivelu, Head, MERDU for her insight and support; Dr Nur Azreen Hussain and Dr Wan Aizat Wan Zakaria from the Department of Anaesthesiology, UMMC and UM, for their acting skills in the training videos; and the Visibility and Communication Unit, Faculty of Medicine, Universiti Malaya for their video editing services.

Funding

There is no funding source for this manuscript.

Declaration of Interest

There are no conflicts of interest among the authors of this manuscript.

References

Arias, J. J., Swinton, J., & Anderson, K. (2018). Online vs. face-to-face: A comparison of student outcomes with random assignment. E-Journal of Business Education & Scholarship of Teaching, 12(2), 1–23. https://eric.ed.gov/?id=EJ1193426

Blew, P., Muir, J. G., & Naik, V. N. (2010). The evolving Royal College examination in anesthesiology. Canadian Journal of Anesthesia/Journal canadien d’anesthésie, 57(9), 804-810. https://doi.org/10.1007/s12630-010-9341-1

Blumberg, M., & Pringle, C. D. (1982). The missing opportunity in organizational research: Some implications for a theory of work performance. The Academy of Management Review, 7(4), 560–569. https://doi.org/10.2307/257222

Cook, D. A., & Steinert, Y. (2013). Online learning for faculty development: A review of the literature. Medical Teacher, 35(11), 930–937. https://doi.org/10.3109/0142159X.2013.827328

Daniel, C. M. (2014). Comparing online and face-to-face professional development [Doctoral dissertation, Nova Southeastern University]. https://doi.org/10.13140/2.1.3157.5042

Feldman, M., Lazzara, E. H., Vanderbilt, A. A., & DiazGranados, D. (2012). Rater training to support high-stakes simulation-based assessments. Journal of Continuing Education in the Health Professions, 32(4), 279–286. https://doi.org/10.1002/chp.21156

Holmboe, E. S., Ward, D. S., Reznick, R. K., Katsufrakis, P. J., Leslie, K. M., Patel, V. L., Ray, D. D., & Nelson, E. A. (2011). Faculty development in assessment: The missing link in competency-based medical education. Academic Medicine, 86(4), 460–467. https://doi.org/10.1097/ACM.0b013e31820cb2a7

Iqbal, I., Naqvi, S., Abeysundara, L., & Narula, A. (2010). The value of oral assessments: A review. The Bulletin of the Royal College of Surgeons of England, 92(7), 1–6. https://doi.org/10.1308/147363510×511030

Juul, D., Yudkowsky, R., & Tekian, A. (2019). Oral Examinations. In R. Yudkowsky, Y. S. Park, & S. M. Downing (Eds.), Assessment in Health Professions Education. Routledge. https://doi.org/10.4324/9781315166902-8

Kemp, N., & Grieve, R. (2014). Face-to-face or face-to-screen? Undergraduates’ opinions and test performance in classroom vs. online learning. Frontiers in Psychology, 5. https://doi.org/10.3389/fpsyg.2014.01278

Khera, N., Davies, H., Davies, H., Lissauer, T., Skuse, D., Wakeford, R., & Stroobant, J. (2005). How should paediatric examiners be trained? Archives of Disease in Childhood, 90(1), 43–47. https://doi.org/10.1136/adc.2004.055103

Lineberry, M. (2019). Validity and quality. Assessment in Health Professions Education, 17-32. https://doi.org/10.4324/9781315166902-2

Marin-Garcia, J. A., & Martinez Tomas, J. (2016). Deconstructing AMO framework: A systematic review. Intangible Capital, 12(4), 1040. https://doi.org/10.3926/ic.838

McLean, M., Cilliers, F., & Van Wyk, J. M. (2008). Faculty development: Yesterday, today and tomorrow. Medical Teacher, 30(6), 555–584. https://doi.org/10.1080/01421590802109834

Md Hashim, N. H. (2021). Pre- and Post-test [Dataset]. Figshare. https://doi.org/10.6084/m9.figshare.20186582.v1

Md Hashim, N. H. (2022). Followup Questionnaire [Dataset]. Figshare. https://doi.org/10.6084/m9.figshare.20186591.v2

Newman, L. R., Brodsky, D., Jones, R. N., Schwartzstein, R. M., Atkins, K. M., & Roberts, D. H. (2016). Frame-of-reference training: Establishing reliable assessment of teaching effectiveness. Journal of Continuing Education in the Health Professions, 36(3), 206–210. https://doi.org/10.1097/CEH.0000000000000086

Newstrom, J. W. (1995). Evaluating training programs: The four levels, by Donald L. Kirkpatrick. (1994). San Francisco: Berrett-Koehler. 229 pp., $32.95 cloth. Human Resource Development Quarterly, 6(3), 317-320. https://doi.org/10.1002/hrdq.3920060310

Singh, J., Evans, E., Reed, A., Karch, L., Qualey, K., Singh, L., & Wiersma, H. (2022). Online, hybrid, and face-to-face learning through the eyes of faculty, students, administrators, and instructional designers: Lessons learned and directions for the post-vaccine and post-pandemic/COVID-19 World. Journal of Educational Technology Systems, 50(3), 301–326. https://doi.org/10.1177/00472395211063754

Steinert, Y., Mann, K., Anderson, B., Barnett, B. M., Centeno, A., Naismith, L., Prideaux, D., Spencer, J., Tullo, E., Viggiano, T., Ward, H., & Dolmans, D. (2016). A systematic review of faculty development initiatives designed to enhance teaching effectiveness: A 10-year update: BEME Guide No. 40. Medical Teacher, 38(8), 769-786. https://doi.org/10.1080/0142159x.2016.1181851

Steinert, Y., Mann, K., Centeno, A., Dolmans, D., Spencer, J., Gelula, M., & Prideaux, D. (2006). A systematic review of faculty development initiatives designed to improve teaching effectiveness in medical education: BEME Guide No. 8. Medical Teacher, 28(6), 497–526. https://doi.org/10.1080/01421590600902976

Tan, C. P. L., & Pallath, V. (2022). Workshop Evaluation Form [Dataset]. Figshare. https://doi.org/10.6084/m9.figshare.20189309.v1

Tekian, A., & Norcini, J. J. (2016). Faculty development in assessment : What the faculty need to know and do. In M. Mentkowski, P.F. Wimmers (Eds.), Assessing Competence in Professional Performance across Disciplines and Professions (1st ed., pp. 355–374). Springer Cham. https://doi.org/10.1007/978-3-319-30064-1

Thomas, P. A., Kern, D. E., Hughes, M. T., & Chen, B. Y. (2015). Curriculum development for medical education : A six-step approach. John Hopkins University Press. https://jhu.pure.elsevier.com/en/publications/curriculum-development-for-medical-education-a-six-step-approach

Vollenbroek, W. B. (2019). Communities of Practice: Beyond the Hype – Analysing the Developments in Communities of Practice at Work [Doctoral dissertation, University of Twente]. https://doi.org/10.3990/1.9789036548205

Williams, R. G., Klamen, D. A., & McGaghie, W. C. (2003). SPECIAL ARTICLE: Cognitive, social and environmental sources of bias in clinical performance ratings. Teaching and Learning in Medicine, 15(4), 270–292. https://doi.org/10.1207/S15328015TLM1504_11

*Shairil Rahayu Ruslan
50604, Kuala Lumpur,
Malaysia
03-79492052 / 012-3291074
Email: shairilrahayu@gmail.com, shairil@ummc.edu.my

Submitted: 23 August 2022
Accepted: 3 January 2023
Published online: 4 July, TAPS 2023, 8(3), 15-25
https://doi.org/10.29060/TAPS.2023-8-3/OA2871

Iroro Enameguolo Yarhere¹, Tudor Chinnah² & Uche Chineze³

¹Department of Paediatrics, College of Health Sciences, University of Port Harcourt, Nigeria; ²Department of Anatomy, University of Exeter, United Kingdom; ³Department of Education and Curriculum studies, University of Port Harcourt, Nigeria

Abstract

Introduction: This study aimed to compare the paediatric endocrinology curriculum across Southern Nigeria medical schools, using reports from learners. It also checked the learners’ perceptions about different learning patterns and competency in some expected core skills.

Methods: This mixed (quantitative and qualitative) study was conducted with 7 medical schools in Southern Nigeria. A multi-staged randomized selection of schools and respondents, was adopted for a focus group discussion (FGD), and the information derived was used to develop a semi-structured questionnaire, which 314 doctors submitted. The FGD discussed rotation patterns, completion rates of topics and perceptions for some skills. These themes were included in the forms for general survey, and Likert scale was used to assess competency in skills. Data generated was analysed using statistical package for social sciences, SPSS 24, and p values < 0.05 were considered significant

Results: Lectures and topics had various completion rates, 42.6% – 98%, highest being “diabetes mellitus”. Endocrinology rotation was completed by 58.6% of respondents, and 58 – 78 % perceived competency in growth measurement and charting. Significantly more learners, 46.6% who had staggered posting got correct matching of Tanner staging, versus learners who had block posting, 33.3%, p = 0.018.

Conclusion: Respondents reported high variability in the implementation of the recommended guidelines for paediatric endocrinology curriculum between schools in Southern Nigeria. Variabilities were in the courses’ completion, learners’ skills exposure and how much hands-on were allowed in various skills acquisitions. This variability will hamper the core objectives of human capital development should the trend continue.

Keywords:          Paediatric Endocrinology Curriculum, Perception, Compliance, Completion Rate, Learners

Practice Highlights

• Medical and dental council of Nigeria has a recommended benchmark for minimum academic standards in all medical schools to which total compliance is expected.
• Evaluation of paediatric endocrinology curriculum content and training methods was conducted using reports from learners.
• Variability in the content, and training methods of the intended competency were reported across medical schools.
• Compliance rate of the recommended curriculum was less than 50% in some contents and some learners reported low skill performance training.
• The lack of uniformity can prevent achievement of the overarching objective of the curriculum in Nigeria with wide variations in competence among graduating doctors.

I. INTRODUCTION

The primary aim of the Medical and Dental council of Nigeria (MDCN) undergraduate curriculum is “to train doctors and dentists who can work effectively in a health team to provide comprehensive health care to individuals in any community in the nation, and keep up to date on issues of global health” (Federal Ministry of Health of Nigeria, 2012). In Nigeria today, there are 49 federal, 59 states and 111 private universities, and 44 of these have full or partially accredited medical schools and while these schools have a prescribed curriculum, some are not following explicitly (Federal Ministry of Health of Nigeria, 2012). This curriculum advocates for universities to develop syllabus to meet the benchmark for minimum academic standards (BMAS) across schools, however there is no uniform template developed for assessing graduates to know how their competence converge as is applicable in United States of America (USA), Canada and United Kingdom (UK) (Santen et al., 2019; Shah et al., 2020; Sosna et al., 2021).  Diabetes mellitus, thyroid disorders, puberty, rickets and growth abnormalities are topics included in the MDCN paediatric curriculum under endocrinology which learners are expected to acquire competence in cognitive and psychomotor skills to diagnose and treat or refer appropriately children presenting with these diseases.

A. Problem

Most deaths from diseases in Nigeria and other resource-limited countries are consequent upon general public ignorance of disease, late presentation to the health care systems, poverty and lack of funds to access healthcare facilities and reduced knowledge of some disease patterns by the healthcare providers (Yarhere & Nte, 2018). Addressing the gaps in reduced knowledge can be done by developing competency-based curriculum for all graduating doctors to have as near-similar competence as possible but achieving this may not be feasible. Training activities are not uniform throughout medical schools in Nigeria and elsewhere, and depend on schools’ vision, mission and objectives, and the structures and processes put in place. There are barriers to positive implementation across schools including but not limited to individual school’s determination of what is relevant in the curriculum, access to the materials needed to teach the curriculum content and getting trainers to use these curriculums (Polikoff, 2018). The lack of uniformity of curriculum across universities may not be contending issues, but when the graduating doctors have varying degree of competencies in skills and cognition, then a template for imparting uniform and up to date knowledge and to evaluate this is needed to find ways of reducing the variability (McManus, 2003; McManus et al., 2020; Rimmer, 2014).

The curriculum uniformity across schools is one way of improving competency and thus, healthcare standards, and there is need to explore this uniformity or diversity within the paediatric undergraduate training. In some countries, there is a uniform board certification examination before doctors can practice and this is also done for doctors immigrating into these countries (Hohmann & Tetsworth, 2018; Puri et al., 2021; Tiffin et al., 2017; van Zanten et al., 2022) but Nigeria is exempt from this uniform exit examination. This uniform exit board examination makes these schools align course contents, and therefore reduces the variabilities between medical schools and undergraduate training.

B. Curriculum Evaluation for Change or Improvement

Curriculum evaluation is a means by which educators understand whether the curriculum used to train learners is working as intended, and whether there is need to change the entire programme or redesign aspects (Burton & McDonald, 2001; Ornstein & Hunkins, 2009). It is also a way of identifying deficiencies in training syllabus across universities, (Rufai et al., 2016) or whether compliance to a curriculum is being achieved (Grant, 2014; Olson et al., 2000). Kirkpatrick’s curriculum evaluation method is widely acceptable in medical education using the 4 steps; learners’ reaction or satisfaction, knowledge, behavioural changes and results or impact, and in Nigeria, for paediatric endocrinology, this has not been done (Alsalamah, 2021; Bates, 2004).

Universities have variabilities in organisation, students’ numbers in classes, duration of specific posting, posting types and whether the courses are elective or core. In medical schools in Nigeria, paediatric postings are undertaken in the 5^th or 6^th year of a 6-year programme. While some stagger the posting to be done within the last 2 years, others do theirs in the 5^th or the 6^th year exclusively, and the extent of these variabilities and how they affect the training processes and products has not been evaluated in Nigeria and this can be done using learners’ or graduates’ perceptions.

The aim of this research was to evaluate learners’ report and perception of some aspects of the paediatric endocrinology curriculum contents and learning methods across Southern Nigeria medical schools. Endocrinology was taken from the paediatric course to reduce the volume of information to be analysed.

II. METHODS

This was a cross sectional study design with qualitative and quantitative data analyses, evaluating learners’ report and their perception of the curriculum being used by various medical schools in Southern Nigeria to deliver the MDCN paediatric endocrinology curriculum. Survey was conducted across 10 medical schools in Southern Nigeria that have learners who have either completed their final year, or are doing their internship. Two steps were used to retrieve the information needed; a focus group discussion of sampled learners, and a questionnaire survey sent out to randomly selected respondents and these 2 methods complemented each other. The focus group discussion was used to explore in depth, the minds of the respondents and what they perceived was being done well and what needed to be changed in the syllabus in their respective schools. The questionnaire survey was then used to collect reports and perceptions from a wider set of learners who had completed their paediatric posting within the past 6 – 12 months. Some of these were already doing their internship and others were in their final year in preparation for their final examinations.

Sample size for respondents will be calculated using the formula:

N = (Z score)² x SD x (1 – SD) 

                       (CI)²

Z score = 1.96, SD (standard deviation of the mean) = estimated at ± 0.5, Confidence interval = 0.05

= 384 respondents, with an attrition rate of 10% will be added 10% of 384 = 38

384 + 38 = 422 respondents.

A. Sampling Technique

Multi-staged sampling technique was used to determine the schools, and respondents that participated in the study. There are 29 Southern Universities with medical / health colleges and 16 of these had more than 50 learners in their final year or had graduated. Ten schools were randomly selected using the excel formula [= rand ()], and a proportionate stratified sampling was done using the matriculation numbers of the students in each school to arrive at 422 respondents. Total number of learners that studied paediatrics in various institutions was 800; Ibadan 150, Port Harcourt, 128, Lagos 128, Niger delta University 69, UNN Enugu 128, University of Benin 128, Others 69. From the total number of learners in each school, selected learners and interns were sent the questionnaire using their email addresses. Selection for the FGD was done using simple random sampling from each school and these were sent separate emails with details for the meeting.

B. Focus Group Discussions Process

Focus group discussion was conducted with the respondents using zoom video platform, and the process lasted for 2 hours, 30 minutes. Ten learners’ representatives from the selected schools were contacted for this FGD, however, 7 (70%) agreed to participate after several email reminders.  The interview was semi-structured with a flexible topic guide, which covered issues relating to the respondents’ views and opinions on the curriculum in paediatric endocrinology; description of posting type in each school, whether block, or staggered, topics received and/or completed, perception of their competence in a key psychomotor skill. The focus group interview discussions were recorded in the zoom meeting platform and transcribed verbatim. The data were analysed using the thematic framework content analyses method. The themes generated were categorised into; 1. Lecture contents and completion rate, 2. Types of paediatric rotation and posting, 3. Skill competence acquisition and clinical postings. Their perceptions about these themes were also sought and discussed. The transcription of the groups’ discussions was reviewed by IY and TC to help categorise the data and pull-out important quotes used.

C. Questionnaire Survey

Following thematic analyses of the FGD, the themes generated were converted to questions in a survey for a larger sample population. Themes generated were the type of paediatric posting, rotations through units in the departments and paediatric endocrinology topics, training methods and competency acquired. Demographic characteristics of responders such as level/year of study, age, gender and university of study were collected. The respondents were also asked to select topics from a poll, included in their paediatric endocrinology syllabus, with result in Figure 1, and to state the various methods used to learn growth and growth disorders in their schools. A means of assessing cognitive (recall) skills of the learners was conducted using animated pictures of Tanner staging and matching-type multiple choice, and the responses were crossmatched with the type of posting learners were exposed to, i.e. block posting or staggered posting. Tanner staging was chosen as it cuts across general paediatrics and endocrinology as part of growth and puberty (endocrinology).

Data retrieved were analysed statistically by using chi-square test, and Pearson correlation for categorical variables. The level of competence perceived by learners in height measurement and charting on growth chart was retrieved using 5-point Likert scale (where 1 = not competent; 2 = low competence; 3 = neutral; 4 = competent; and 5 = proficient). The association between level of competence and whether learners rotated through paediatric endocrinology was checked using Pearson’s correlation test. For all statistics, p value < 0.05 was considered significant.

D. Ethics

The research commenced after the Research Ethics committee of the University of Port Harcourt granted approval (UPH/CEREMAD/REC/MM80/056). Verbal informed consent was obtained from the participants during the focus group discussion, who also gave consent for video and recording of the process. Informed consent was also obtained from all participants who filled and submitted the online survey. The focus group discussants received N3,000 ($10) for internet data only as monetary compensation.

III. RESULTS

There were 314 learners from the 422 calculated sample size, responded to the questionnaire survey, giving a response rate of 74.4%. There were more final year respondents than early career doctors and more of the respondents were in the age bracket 20 – 24 years, with a mean of 25.02 ±2.71 years. The male: female ratio was 1:1.01, and the data that support the findings of this study are available in Figshare at https://doi.org/10.6084 /m9.figshare.20730937.v1 (Yarhere et al., 2022).

RESPONDENTS	Frequency	Percentage
Year of study
Early career doctor (graduate/intern)	130	41.4	p = 0.002
Final year	184	58.6
University attended (calculated cohort)
University of Port Harcourt (63)	62	19.7
Niger Delta University (54)	54	17.2
University of Ibadan (76)	50	15.9
University of Benin (65)	44	14.0
University of Lagos (65)	40	12.7
University of Nigeria (65)	42	13.4
Other western Universities (34)	22	7.0
Age
20-24	140	44.6
25-29	162	51.6
>=30	12	3.8
Mean	25.02 ± 2.71
Gender
Male	152	48.4	p = 0.612
Female	162	51.6

Table 1. Demographic characteristics of all respondents and the universities attended

There were basically 2 modes of paediatric posting in the institutions sampled; 4-months block posting where respondents have a month of didactic lectures and 3 months of clinical rotations through various units in the Paediatric departments, and 4 months of staggered rotations with junior and senior postings in the clinical classes. While some learners rotated through all the units (core and electives) in the departments, some went through core units, emergency and neonatal units, and 2 other units randomly selected for the respondents by the departments.

C. Learners’ Responses to Rotation through Paediatrics and Posting Types

Participant 2 shared:

The way it works in University of xxxx, we rotate through 2 elective postings with core (CHEW and SCBU) postings in the junior and senior postings. These elective postings are randomly selected by the department (meaning heads or coordinators). I did neurology and gastroenterology in my junior posting and haemato-oncology and I really can’t remember the other one in my senior posting.

 “Will I be wrong to say you did not see a patient with Diabetic keto acidosis?”

Facilitator

“I saw a child with diabetic keto acidosis in the ward but it wasn’t my unit managing the patient. I only went to the ward to do some other thing.”

Participant 2 

“If you were given the opportunity to design a curriculum or programme for your university, will you prefer what is being practiced now, or will you rather have every student go through every unit and get titbits from each unit?”

Facilitator

Participant 2 responded:

Yes, I will prefer that situation where you get to be exposed to every unit in the department but …. emmm, that creates a problem because you may be in a unit for a week, and no patient comes in but the next group rotating to the unit gets to see many patients. I would want to suggest that perhaps, instead of focusing on more of clinical posting, that a unified tutorial class which will expose everyone to the core diseases in the various disciplines.

Table 2 corroborates the information given by the focus group discussants. Testing the competency outcome in either method can give some estimated guess as to which is better, however, there are several confounding factors that will not allow fair comparison (See Table 3).

Though there were differences in the mode of paediatric postings where staggered or block, c² = 4.59, p = 0.032. the difference in proportion of respondents who had core and selected elective posting as against all units posting was not significant, c² = 0.318, p = 0.573.

In the 2×2 table above where recall was tested in the learners based on their paediatric posting type, higher percentage of those who had staggered posting got the correct matching of Tanner stage, and the difference was significant, c² = 5.630, p = 0.018. However, the total number of respondents with the correct response was low.

D. Perception of Core Competency Skill in Growth Measurement and Charting by Learners

One of the most important courses in paediatrics is growth and development and training future medical doctor to acquire skills and competence in growth and management is a key component of the BMAS. While growth measurement may seem easy to the uninformed, the whole task is daunting especially in children with complex growth abnormalities and malformation, and for more complex skills like arm span. Which of the more complex skills should the learner be expected to be competent in, will be debated in an expert forum of trainers.

“So, did you do anthropometric measures?”

Facilitator

Participant 1 shared:

Yes, anytime we clerk a patient, we must check the weight and height and interpret using age-appropriate charts, but we did not plot them in the charts. We carry the age-appropriate chart and interpreted our patients, as this is a requirement.

Using the chart may not be emphasised by all paediatric lecturers, so learners can be smart to know those lecturers who will request this skill from them during the clerkship period or the unit rotations.

“We did not quite get the concept of mid parental height, height percentile, it was just mentioned in passing. I never saw a severely short child that needed growth hormone. I was only told by a classmate of mine.”

Participant 3

The charting and interpretation of weight and height measurements of children was not done in all schools as shown in Table 4 below, which tells that only 65.8% of total respondents were taught interpretation of measured and charted growth parameters. The level of competence in these tasks will also be varied as seen in Appendix 1. Two hundred and thirty-eight (75.8%) learners perceived they had competency/ proficiency in height measures using stadiometer, and 44.6 % of the learners with these perceptions actually had paediatric endocrinology clinical rotation (Appendix 1).

Bone age and orchidometers are used to assess skeletal maturation and puberty, which are advanced for the undergraduate learners and certainly not compulsory, but some respondents were taught with the tools showing the variabilities in contents and skills delivery between these schools. From Table 4 above, framers of the syllabus for endocrinology aspect of paediatrics curriculum are unlikely to include use of orchidometer and bone age during the undergraduate paediatric endocrinology rotation as the skill is complex, and not necessary for their level of development.

IV. DISCUSSION

This study has highlighted differences in course contents and training methods across medical schools in Southern Nigeria. While many schools have used the BMAS prescribed by the MDCN, the syllabus used are different and the intended learning outcomes are diverse based on the respondents’ reports. Some learners reported not having diabetes lectures in their school through no fault of theirs, as lecturer rescheduled the lectures and never gave them. While learners have the responsibility to attend lectures, trainers are also obligated to be present at their scheduled lectures or transfer this to their teacher-assistants, or use technologies (Grant, 2014; Ruiz et al., 2006). Some learners had little participation in the Emergency Room, others participated fully in DKA management, learning empathy, specialised skills and communication. The intended competencies to be acquired can be achieved through shadowing and participation, bed-side teaching, and tutorial to improve the cognitive and psychomotor skill, and these opportunities must be created for them in experiential settings (Ryan et al., 2020; Shah et al., 2020).

More learners had staggered postings, going through junior and senior paediatric postings in what may be considered as integrated learning departing from the traditional method (Patel et al., 2005; Watmough et al., 2006, 2009). In the staggered posting type of rotation, we noticed that not all learners went through paediatric endocrinology unit posting, and like one of the discussants said, they would rather everyone went through each unit getting bits of everything and having opportunity to study specific and prevalent diseases in paediatric units rather than leaving them with the possibility of not learning important disorders. As it is not always possible to encounter specific diseases like DKA during entire posting in the schools that use staggered posting types, the likelihood of exposure was higher in schools that had block posting from the FGD conducted, but this did not translate to better retention of skills or cognitions as depicted in the Tanner staging matching question.

Having learners train in all special postings may not be the best approach in undergraduate medicine because the specialised skills may not be utilised in general practice and even in general paediatrics should the learners plan paediatric specialisation (Bindal et al., 2011). While some trainers may argue that all information and skill should be taught to the learners, the time to acquire and achieve mastery may be short for the learners (Jensen et al., 2018; Offiah et al., 2019). This study can be referenced in curriculum designing and implementation so the framers understand what society needs should be filled at any time. The concept of cognitive overload has actually reduced the duration of core specialty in clinical medicine while increasing the duration for others with emphasis on psychomotor, affective skills and professionalism. Some medical schools have core paediatric posting of 7 – 8 weeks, but Nigeria is still fixed with the traditional 3 – 4 months. In some schools in South Africa, the clinical posting is run as modular block for 3 years, with paediatric curriculum running from year 4 through year 6 (Dudley & Rohwer, 2015). With the long duration in the Nigeria curriculum, skills competencies are still deficient, so there is need to revamp the curriculum to make it more competency driven. It is excusable that more sophisticated competence like use of orchidometer were not known by more than half the learners, but if some were taught, the level of confidence in these skills at this stage of their learning should also be assessed as was done for diabetes by George et al. (2008).

Medical schools in Nigeria and other countries will have to continually evolve and produce curricula that are competency based, using problem-based learning, simulations, mannikin training for skills as is done in other countries (Watmough et al., 2006). Diabetes, thyroid, ambiguous genitalia with congenital adrenal hyperplasia, short stature and calcium disorders are common in Nigeria and should be taught in structured and integrated formats. Integrated curriculum where skills are graded from simple to complex can also be tested e g, skills of height measurements and charting using the stadiometer and growth charts can be taught in the 1^st clinical year, and then the mid parental heights, target height calculation and bone age may be taught in the 2^nd and 3^rd clinical years. (Brauer & Ferguson, 2015; Grant, 2014).

A. Strength of the Research

Articulating the perceptions of learners is not always easy as they are varied and subjective, but getting them to come together, discuss and give suggestions on how curriculum can be designed and achieved increases the strength of this research. There was no sense of victimisation of the learners as many had already graduated from their schools, and the discussants admitted to not missing classes, or clinical learning. They spoke freely, with courtesy to others and there was little or no argument among them.

B. Limitations of the Research

As this research is based on past experiences of the cognitive and psychomotor skills achieved during the learners’ training period, the possibility of recall bias is high, and respondents may underestimate or exaggerate their skills. Using respondents who had just concluded their paediatric postings was an attempt at reducing this limitation. The best time to evaluate a programme is usually soon after the programme has been concluded however, as there has been no report of this type of evaluation, there was need to embark on it and make recommendations.

V. CONCLUSION

Respondents reported high variability in the implementation of the recommended guidelines for paediatric endocrinology curriculum between schools in Southern Nigeria. Variabilities were in the courses’ completion, learners’ skills exposure and how much hands-on were allowed in various skills acquisitions. This variability will hamper the core objectives of human capital development should the trend continue.

A. Area of Future Research

Noting the differences exist between schools, curriculum strategists and implementation teams in universities should commission a DELPHI study by experts, where core competencies and objectives for paediatric endocrinology will be agreed on and sent to the regulatory bodies for endorsement and implementation.

Notes on Contributors

IY conceived, designed, planned, executed and conducted interviews and the research. He also collected the data, analysed it and wrote the manuscript.

TC helped in designing the methodology for the data colllection and analyses, and reviewed the manuscript.

CU gave critical appraisal of the manuscript and all authors have approved the final manuscript.

Ethical Approval

The research ethics committee of the Univeristy of Port Harcourt gave ethical approval before the start of the study with the number: UPH/CEREMAD/REC/MM80/056.

Data Availability

The data supporting this research is available for publication purposes, without editing. Data can be shared only with express permission from the corresponding author as deposited in Figshare repository, using the private url:

https://figshare.com/articles/dataset/Copy_of_CURRICULUM_STUDENTS_xls/21154396

Acknowledgement

We acknowledge the early career doctors and final year students who participated in the online survey especially the selected ones who took part in the focus group discussion.

Declaration of Interest

Authors declare that there are no conflicts of interest, including financial, consultant, institutional and other relationships that might lead to bias or a conflict of interest.

Funding

There was no funding for this survey.

References

Alsalamah, A., & Callinan, C. (2021). Adaptation of Kirkpatrick’s four level model of training criteria to evaluate training programmes for head teachers. Education Science, 11(116), 1-25. https://doi.org/10.3390/educsci11030116

Bates, R. (2004). A critical analysis of evaluation practice: The Kirkpatrick model and the principle of beneficence. Evaluation and Program Planning, 27, 341-347. https://doi.org/10.1016/j.evalprogplan.2004.04.011

Bindal, T., Wall, D., & Goodyear, H. M. (2011). Medical students’ views on selecting paediatrics as a career choice. European Journal of Pediatrics, 170(9), 1193-1199. https://doi.org/10.1007/s00431-011-1467-9

Brauer, D. G., & Ferguson, K. J. (2015). The integrated curriculum in medical education: AMEE Guide No. 96. Medical Teacher, 37(4), 312-322. https://doi.org/10.3109/0142159X.2014.970998

Burton, J. L., & McDonald, S. (2001). Curriculum or syllabus: Which are we reforming? Medical Teacher, 23(2), 187-191. https://doi.org/10.1080/01421590020031110

Dudley, L. D., Young, T. N., Rohwer, A. C., Willems, B., Dramowski, A., Goliath, C., Mukinda, F. K., Marais, F., Mehtar, S., & Cameron, N. A. (2015). Fit for purpose? A review of a medical curriculum and its contribution to strengthening health systems in South Africa. African Journal Health Profession Education, 7(1), 81-84. https://doi.org/10.7196/AJHPE.512

Federal Ministry of Health of Nigeria, Health Systems 20/20Project. (2012). Nigeria undergraduate medical and dental curriculum template. Health systems 20/20 Project, Abt Associates Inc.

George, J. T., Warriner, D. A., Anthony, J., Rozario, K. S., Xavier, S., Jude, E. B., & Mckay, G. A. (2008). Training tomorrow’s doctors in diabetes: Self-reported confidence levels, practice and perceived training needs of post-graduate trainee doctors in the UK. A multi-centre survey. BMC Medical Education, 8, Article 22. https://doi.org/10.1186/1472-6920-8-22

Grant, J. (2014). Principles of curriculum design. In T. Swanwick, K. Forrest, B. C. O’Brien (Eds.), Understanding medical education evidence, theory and practice Sussex, UK: Wiley Blackwell, 31-46.

Hohmann, E., & Tetsworth, K. (2018). Fellowship exit examination in orthopaedic surgery in the commonwealth countries of Australia, UK, South Africa and Canada. Are they comparable and equivalent? A perspective on the requirements for medical migration. Medical Education Online, 23(1), Article 1537429. https://doi.org/10.1080/10872981.2018.1537429

Jensen, J. K., Dyre, L., Jørgensen, M. E., Andreasen, L. A., & Tolsgaard, M. G. (2018). Simulation-based point-of-care ultrasound training: a matter of competency rather than volume. Acta Anaesthesiology Scandanavia, 62(6), 811-819. https://doi.org/10.1111/aas.13083

McManus, I. C. (2003). Medical school differences: beneficial diversity or harmful deviations. BMJ Quality and Safety in Health Care, 12(5), 324-325. https://doi.org/10.1136/qhc.12.5.324

McManus, I. C., Harborne, A. C., Horsfall, H. L., Joseph, T., Smith, D. T., Marshall-Andon, T., Samuels, R., Kearsley, J. W., Abbas, N., Baig, H., Beecham, J., Benons, N., Caird, C., Clark, R., Cope, T., Coultas, J., Debenham, L., Douglas, S., Eldridge, J., . . . Devine, O. P. (2020). Exploring UK medical school differences: the MedDifs study of selection, teaching, student and F1 perceptions, postgraduate outcomes and fitness to practise. BMC Medicine, 18(1), Article 136. https://doi.org/10.1186/s12916-020-01572-3

Offiah, G., Ekpotu, L. P., Murphy, S., Kane, D., Gordon, A., O’Sullivan, M., Sharifuddin, S. F., Hill, A. D. K., & Condron, C. M. (2019). Evaluation of medical student retention of clinical skills following simulation training. BMC Medical Education, 19(1), Article 263. https://doi.org/10.1186/s12909-019-1663-2

Olson, A. L., Woodhead, J., Bekow, R., Kaufman, N., & Marshal, S. (2000). A national general pediatric clerkship curriculum: The process of development and implementation. Pediatrics, 160(S1), 216 -222. https://doi.org/10.1542/peds.106.S1.216

Ornstein A, H. F., & Hunkins, F. P. (2009). Curriculum: Foundations, principles and issues (5th Ed.). Pearson.

Patel, V. L., Arocha, J. F., Chaudhari, S., Karlin, D. R., & Briedis, D. J. (2005). Knowledge integration and reasoning as a function of instruction in a hybrid medical curriculum. Journal of Dental Education, 69(11), 1186-1211. https://www.ncbi.nlm.nih.gov/pubmed/16275683

Polikoff, M. S. (2018). The challenges of curriculum materials as a reform lever Evidence Speaks Reports, 2, 58

Puri, N., McCarthy, M., & Miller, B. (2021). Validity and reliability of pre-matriculation and institutional assessments in predicting USMLE STEP 1 success: Lessons from a traditional 2 x 2 curricular model. Frontiers in Medicine (Lausanne), 8, Article 798876. https://doi.org/10.3389/fmed.2021.798876

Rimmer, A. (2014). GMC will develop single exam for all medical graduates wishing to practise in UK. BMJ, 349, g5896. https://doi.org/10.1136/bmj.g5896

Rufai, S. R., Holland, L. C., Dimovska, E. O., Bing Chuo, C., Tilley, S., & Ellis, H. (2016). A national survey of undergraduate suture and local anesthetic training in the United Kingdom. Journal of Surgical Education, 73(2), 181-184. https://doi.org/10.1016/j.jsurg.2015.09.017

Ruiz, J. G., Mintzer, M. J., & Leipzig, R. M. (2006). The impact of E-learning in medical education. Academic Medicine, 81(3), 207-212. https://doi.org/10.1097/00001888-200603000-00002

Ryan, A., Hatala, R., Brydges, R., & Molloy, E. (2020). Learning with patients, students, and peers: Continuing professional development in the solo practitioner workplace. Journal of Continuing Education in the Health Profession, 40(4), 283-288. https://doi.org/10.1097/CEH.0000000000000307

Santen, S. A., Feldman, M., Weir, S., Blondino, C., Rawls, M., & DiGiovanni, S. (2019). Developing comprehensive strategies to evaluate medical school curricula. Medical Science Educator, 29(1), 291-298. https://doi.org/10.1007/s40670-018-00640-x

Shah, S., McCann, M., & Yu, C. (2020). Developing a national competency-based diabetes curriculum in undergraduate medical education: A Delphi study. Canadian Journal of Diabetes, 44(1), 30-36. https://doi.org/10.1016/j.jcjd.2019.04.019

Sosna, J., Pyatigorskaya, N., Krestin, G., Denton, E., Stanislav, K., Morozov, S., Kumamaru, K. K., Jankharia, B., Mildenberger, P., Forster, B., Schouman-Clayes, E., Bradey, A., Akata, D., Brkljacic, B., Grassi, R., Plako, A., Papanagiotou, H., Maksimović, R., & Lexa, F. (2021). International survey on residency programs in radiology: similarities and differences among 17 countries. Clinical Imaging, 79, 230-234. https://doi.org/10.1016/j.clinimag.2021.05.011

Tiffin, P. A., Paton, L. W., Mwandigha, L. M., McLachlan, J. C., & Illing, J. (2017). Predicting fitness to practise events in international medical graduates who registered as UK doctors via the Professional and Linguistic Assessments Board (PLAB) system: a national cohort study. BMC Medicine, 15(1), Article 66. https://doi.org/10.1186/s12916-017-0829-1

van Zanten, M., Boulet, J. R., & Shiffer, C. D. (2022). Making the grade: licensing examination performance by medical school accreditation status. BMC Medical Education, 22(1), Article 36. https://doi.org/10.1186/s12909-022-03101-7

Watmough, S., Garden, A., & Taylor, D. (2006). Does a new integrated PBL curriculum with specific communication skills classes produce Pre Registration House Officers (PRHOs) with improved communication skills. Medical Teachers, 28(3), 264-269. https://doi.org/10.1080/01421590600605173

Watmough, S., O’Sullivan, H., & Taylor, D. (2009). Graduates from a traditional medical curriculum evaluate the effectiveness of their medical curriculum through interviews. BMC Medical Education, 9, Article 64. https://doi.org/10.1186/1472-6920-9-64

Yarhere, I., Chinnah, T., & Uche, C. (2022). Learners’ report and perception of differences in undergraduate paediatric endocrinology curriculum content and delivery across Southern Nigeria. [Data set]. Figshare. https://doi.org/10.6084/m9.figshare.20730937.v1

Yarhere, I. E., & Nte, A. R. (2018) A ten-year review of all cause paediatric mortality in University of Port Harcourt Teaching Hospital, Nigeria (2006–2015). Nigerian Journal of Paediatrics, 45(4), 185-91.

*Iroro Enameguolo Yarhere
East/West Road,
PMB 5323 Choba,
Rivers State, Nigeria
+2347067987148
Email: iroro.yarhere@uniport.edu.ng

Submitted: 16 May 2022
Accepted: 3 January 2023
Published online: 4 July, TAPS 2023, 8(3), 5-14
https://doi.org/10.29060/TAPS.2023-8-3/OA2813

Bikramjit Pal¹, Aung Win Thein², Sook Vui Chong³, Ava Gwak Mui Tay⁴, Htoo Htoo Kyaw Soe⁵ & Sudipta Pal⁶

¹Department of Surgery, Manipal University College Malaysia, Melaka, Malaysia; ²Department of Surgery, Manipal University College Malaysia, Melaka, Malaysia; ³Department of Medicine, Manipal University College Malaysia, Melaka, Malaysia; ⁴Department of Surgery, Manipal University College Malaysia, Melaka, Malaysia; ⁵Department of Community Medicine, Manipal University College Malaysia, Melaka, Malaysia; ⁶Department of Community Medicine, Manipal University College Malaysia, Melaka, Malaysia

Abstract

Introduction: The practice of high-fidelity simulation-based medical education has become a popular small-group teaching modality across all spheres of clinical medicine. High-fidelity simulation (HFS) is now being increasingly used in the context of undergraduate medical education, but its superiority over traditional teaching methods is still not established. The main objective of this study was to analyse the effectiveness of HFS-based teaching over video-assisted lecture (VAL)-based teaching in the enhancement of knowledge for the management of tension pneumothorax among undergraduate medical students.

Methods: A cohort of 111 final-year undergraduate medical students were randomised for this study. The efficacy of HFS-based teaching (intervention group) and VAL-based teaching (control group), on the acquisition of knowledge, was assessed by single-best answer multiple choice questions (MCQ) tests in the first and eighth week of their surgery posting. Mean and standard deviation (SD) for the total score of MCQ assessments were used as outcome measures. ANCOVA was used to determine the difference in post-test MCQ marks between groups. The intragroup comparison of the pre-test and post-test MCQ scores was done by using paired t-test. The P-value was set at 0.05.

Results: The mean of post-test MCQ scores were significantly higher than the mean of pre-test MCQ scores in both groups. The mean pre-test and post-test MCQ scores in the intervention group were slightly more than those of the control group but not statistically significant.

Conclusion: There was a statistically significant enhancement of knowledge in both groups but the difference in knowledge enhancement between the groups was insignificant.

Keywords:           High-Fidelity Simulation, Video-Assisted Lecture, Simulation-Based Medical Education (SBME), Randomized Controlled Trial (RCT), Medical Education, Pre-test and Post-test Knowledge Assessments

Practice Highlights

• An RCT study to evaluate the effectiveness of HFS over video-assisted lecture teaching method.
• HFS seems to be not superior than VAL-based teaching for knowledge acquisition and retention.
• HFS may be used judiciously when the objectives are mainly knowledge based.
• Further research may determine curricular areas where HFS is superior and worth adopting.

I. INTRODUCTION

High-Fidelity Simulation (HFS) is an innovative healthcare education methodology that involves the use of sophisticated life-like mannequins to create a realistic patient environment. HFS can be considered an innovative teaching method that aids students in translating knowledge and psychomotor skills from the classroom to the actual clinical setting. Kolb’s Experiential Learning Cycle (Kolb, 1984) provides a basis for the integration of active learning of simulation with conventional teaching methods for a comprehensive learning experience in undergraduate medical education. HFS-based education is potentially an efficacious pedagogy that is now available for teaching. The usefulness of HFS has been recognized by the Accreditation Council of Graduate Medical Education (Accreditation Council of Graduate Medical Education [ACGME], 2020). HFS has the added benefit of increasing students’ confidence and their ability to care for the patients at the bedside (Kiernan, 2018). HFS-based education and video-assisted lecture-based teaching are both effective in achieving factual learning. Despite the increasing acceptance of HFS, there are limited studies to compare the usefulness of HFS with conventional teaching methods for factual learning among undergraduate medical students. At present, the different research studies have not provided enough evidence to establish HFS-based teaching’s superiority over traditional educational methods in the acquisition and retention of knowledge. There is inconsistent and variable outcome regarding the effectiveness of HFS on student learning (Yang & Liu, 2016). HFS-based education is both time-consuming and resource intensive. Its long-term merits in retaining knowledge and translating it into enhanced patient care need further research. As educators, we need to systematically evaluate the expensive newer teaching-learning modules like HFPS for their effectiveness by using rigorous research methodology and protocols. This is to ensure that we are providing the best learning opportunities conceivable for the students. Previous studies were mostly done in North America and, therefore, the generalisability of these results is guarded and might not be applicable in the context of Europe and Asia due to many differences in academic and curriculum aspects (Davies, 2008). The purpose of this study was to establish the feasibility of the use of HFS to deliver critical care education to final-year medical students and to find its efficacy in the enhancement of knowledge when compared to video-assisted lectures. The study compared the effectiveness of two methods of teaching pedagogy in the enhancement of knowledge acquisition using pre-test and post-test MCQ. This study was designed to provide insights that may be applied to the future development and improvement of HFS-based education among undergraduate medical students and its possibility of integrating it into course curricula.

II. METHODS

A. Study Design

Randomized Controlled Trial (RCT) with parallel groups and 1:1 allocation. Please see Appendix 1 for the Flow Chart.

B. Sample Size

G*Power software was used to calculate the sample size (Faul et al., 2007). Based on the preliminary RCT study of our institute done with the same protocol in 2018, the calculated sample size was 114 with a power of 0.95 for this study.

C. Inclusion and Exclusion Criteria

All male and female final-year undergraduate medical (MBBS) students in our institute were recruited after obtaining their written informed consent. All final-year students in the institute consented to the study. The participants were between the ages of 22-26 years.

The total number of participants recruited was 123.

The number of participants dropped out was 12 (9.77%).

Out of 111 participants who completed the study, 61 (54.95%) were female and 50 (45.05%) were male.

The study was conducted in the Clinical Skills Simulation Lab of Melaka Manipal Medical College (presently known as Manipal University College Malaysia).

The study period was from March 2019 to February 2020 (12 months).

D. Interventions

1) Description of HFPS-based teaching: It was an interactive session using a high-fidelity patient simulator demonstrating the management of tension pneumothorax by performing Needle Decompression on METIman (Pre-Hospital) following the Advanced Trauma Life Support Manual developed by the American College of Surgeons (ATLS Subcommittee et al., 2013).

2) Description of Hi-fidelity simulator: METIman Pre-Hospital HI-Fidelity Simulator (MMP-0418) was used for the simulation sessions. It was a fully wireless, adult High-Fidelity Patient Simulator (HFPS) with modelled physiology. It comes with extensive clinical features and capabilities designed specifically for learners to practice, gain experience, and develop clinical mastery in a wide range of patient care scenarios.

3) Description of video-assisted lecture-based teaching: It was a small group interactive session delivered face-to-face to the participants using a recorded video clip demonstrating the management of tension pneumothorax by performing Needle Decompression on METIman (Pre-Hospital) following the Advanced Trauma Life Support Manual developed by the American College of Surgeons (ATLS Subcommittee et al., 2013).

E. Outcome

The tool for measurement of knowledge was an identical set of single-best answer A-type MCQs. These MCQs were used for both Pre-test and Post-test knowledge assessments. MCQs were constructed based on the teaching sessions to assess their learning outcome.

The efficacy of HFPS-based teaching when compared to video-assisted lecture-based teaching is enhancement of knowledge for management of tension pneumothorax.

F. Recruitment

The students were recruited in the study during their final year surgical posting.

G. Randomisation

A cohort of 12 to 14 students from each rotation was randomised into intervention (HFPS-based teaching) and control (video-assisted lecture-based teaching) groups following random sequence generation method.

A computer-generated random sequence number was developed from randomizer.org. The independent randomiser was a biostatistician who did not participate in the delivery of interventions. The allocated interventions were then sealed in a sequentially numbered, opaque envelope.

Block randomisation with a block size of two was used to assign the students into intervention and control groups.

H. Implementation

A biostatistician generated the allocation sequence. One independent investigator enrolled the participants, and another independent investigator assigned the participants to interventions. The outcome assessor and the biostatistician were kept blinded to the randomisation.

I. Procedure for Data Collection

The participants who gave consent were enrolled in the study. Each session was conducted with a group of 12 to 14 participants. On the first day, the participants were briefed about the sessions and expected learning outcomes. As part of the briefing process, they were explained the confidentiality of the HFPS, the video-assisted lecture sessions and the ethical issues involved. All the participants were introduced to the high-fidelity patient simulator (METIman) in the clinical lab set-up to make them aware of its functions and familiarise them with the handling of the mannequin. An assurance was given to the students that the training course was not part of the evaluation process for the surgical curriculum. The briefing was followed by the first knowledge assessment (Pre-test MCQ) of all the participants. Pre-test MCQ was designed to collect the score of initial background knowledge about tension pneumothorax and its management following the ATLS protocol. The module for the aetiology, pathophysiology and clinical presentation of tension pneumothorax and its steps of management following the ATLS protocol was part of their final year course curriculum. It was taught before they participated in the study. After the Pre-test MCQ session, they were randomized into intervention and control groups consisting of 6 to 7 participants each. For the intervention group, an independent investigator used the high-fidelity simulator (METIman Pre-hospital) to demonstrate the diagnosis and management of tension pneumothorax (Needle Decompression) in an emergency setting. The demonstration time was 20 minutes followed by hands-on training for another 20 minutes. For the control group, a recorded video clip of the identical facilitated simulation session on the diagnosis and management of tension pneumothorax (Needle Decompression) was shown by another investigator. The video demonstration lasted for 20 minutes. This session was followed by a 20-minute interactive discussion session with the faculty. All the participants in both groups were apprised of the importance of aetiology, pathophysiology and clinical presentation in arriving at the diagnosis and management of tension pneumothorax during these interactive teaching sessions. The participants were encouraged to explore how they would manage the stated clinical situation through discussion. The faculty were instructed to emphasize the teaching points related to the outcome of the study. The total duration for both types of teaching was 40 minutes. There were no more additional hands-on practice or video-assisted lecture sessions for the participants during the course of the research study. In the seventh/eighth week, both the intervention and the control groups again participated in the second knowledge assessment (Delayed Post-test MCQ) to assess their gain and retention of knowledge. Delayed Post-test MCQ assessment may minimise the recall bias and test their retained memory better.

Both Pre-test and Post-test knowledge assessments comprised 20 MCQs which were to be completed in 20 minutes. The single-best answer A-type MCQs with five options of answers were prepared following the guidelines framed by the National Board of Medical Examiners (Case & Swanson, 2001). For each correct response, a score of one point was awarded. No negative marking was awarded for incorrect response. Based on the learning objectives, the MCQs were constructed by 6 experts in the field of Surgery, Medicine and Medical Education who were not part of this research study. The MCQs covered the items on pathophysiology, diagnosis, and management of tension pneumothorax, and assessed for knowledge comprehension and knowledge application. The order of the questions was changed between the Pre-test and the Post-test. The MCQ answer sheets were scanned by Konica Minolta FM (172.17.5.12) scanner and graded by using Optical Mark Recognition (OMR) software (Remark Office OMR, version 9.5, 2014; Gravic Inc., USA). Before the main study, a preliminary study involving 56 students was conducted to explore the time management, feasibility, acceptability, and validation of the MCQs (Pal et al., 2021). In the preliminary study, the Pre-test and the Post-test were administered in the first week and the fourth week respectively to note the short-term retention of knowledge. This study is an extension of the preliminary study with a different cohort of students where the Pre-test and the Delayed Post-test were administered in the first week and the seventh/eighth week respectively to determine the medium-term retention of knowledge. The MCQs were reviewed based on the feedback from the preliminary study on the appropriateness of the content, clarity in wording, and difficulty level. The difficulty index and the bi-serial correlation for item discrimination of all MCQs were checked. The value between 30 and 95 in the difficulty index and the bi-serial correlation value > 0.2 were chosen as the accepted standard for this study. 

At the end of the study, the participants in the intervention group were provided with access to the identical video-assisted lecture sessions as designed for the control group. Similarly, the participants in the control group were provided with access to the same HFS sessions. This is to ensure parity between the groups for their professional development of knowledge.

J. Statistical Analysis

SPSS software (version 25) was used for data analysis. The descriptive statistics such as frequency and percentage for categorical data and the mean and standard deviation for the total score of the assessments were calculated. ANCOVA was used to determine the difference in post-test MCQ marks between intervention and control groups with pre-test MCQ marks as a covariate. Intragroup comparison of pre-test and post-test MCQ marks was also done by calculating paired t-test. For intergroup comparison, the effect size – Partial Eta Squared was calculated in ANCOVA. Cohen’s dz was calculated for the comparison of dependent means. The level of significance was set at 0.05 and the null hypothesis was rejected when P < 0.05. We measured the scale-level content validity index (SCVI) and item-level content validity index (ICVI) for the validity and Cronbach alpha for the internal consistency (reliability) of the MCQs. The average values of SCVI and ICVI were 0.94 & 0.89 respectively. The value of Cronbach’s alpha was 0.78.

III. RESULTS

The data that support the findings this RCT study are openly available at https://doi.org/10.6084/m9.figshare.19932053 (Pal et al., 2022).

A. General Data Analysis

There was no difference in the highest Pre-test scores achieved by the participants in both intervention and control groups. The lowest scores recorded in the intervention group were better than the control group in both Pre-test and Post-test. There was a negligible difference between the highest Post-test scores among control and intervention groups (See Table 1).

Test score	Intervention	Control
PRE-TEST
Mean (SE)	12.31 (0.34)	12.23 (0.36)
95% CI for Mean	11.64 – 12.98	11.50 – 12.96
Min – Max	6.0 – 18.0	6.0 – 18.0
POST-TEST
Mean (SE)	13.65 (0.27)	13.60 (0.30)
95% CI for Mean	13.12 – 14.19	12.98 – 14.20
Min – Max	8.0 – 18.0	7.0 – 17.0
Table 1. Highest, lowest and unadjusted mean MCQ scores among intervention and control groups
SE – Standard Error                CI – Confidence Interval Min – Minimum                      Max – Maximum

There was a statistically significant difference between pre-test and post-test MCQ scores among the intervention and control groups. The mean of post-test MCQ scores was significantly higher than the mean of pre-test MCQ scores in both intervention and control groups. The effect size was moderate in both groups (See Table 3).

IV. DISCUSSION

Multiple studies have revealed slight to the modest enhancement of knowledge in simulation-based medical education (SBME) when compared to other instructional teaching methods (Cook et al., 2012; Gordon et al., 2006; Lo et al., 2011; Ray et al., 2012; Ten Eyck et al., 2009). Notwithstanding the increasing popularity of SBME, there is little evidence to conclude that it is superior to other small-group teaching modalities for the acquisition of knowledge (Alluri et al., 2016). The common perception is that knowledge lies at the lowest level of competence in Miller’s model of clinical acumen (Miller, 1990), but it is also important to note that knowledge is the basic foundation of competence and proficiency (Norman, 2009). Theoretically, SBME is advantageous for assessment of both knowledge and skills but there are few studies which directly evaluated the effectiveness of HFS in the assessment of knowledge (McGaghie et al., 2009; Rogers, 2008).

The mean scores of both Pre-test and the Post-test were higher in the intervention group in this study. In comparison, our preliminary study demonstrated that the control group had higher mean MCQ marks than the intervention group in Pre-test whereas at Post-test, the intervention group had higher mean MCQ marks than the control group (Pal et al., 2021).

In our study, there is significant enhancement of knowledge (P < 0.001) in both modes of teaching which corroborates the findings of Alluri et al. (2016). Their RCT study demonstrated that the participants in both the simulation and lecture groups had improved post-test scores (p < 0.05). The comparison of Pre-test and Post-test MCQ scores in our preliminary study also revealed significant higher mean MCQ scores at Post-test than Pre-test in both intervention and control groups (Pal et al., 2021). A study by Couto et al. (2015) showed improved post-test scores in both methods. Similar results were noted in the studies by Chen et al. (2017) and Vijayaraghavan et al. (2019). The finding of a study by Hall (2013) showed a slight increase in post-test scores in both the HFPS and control groups.

A systematic review by La Cerra et al. (2019) revealed that HFS was superior to other teaching methods in improving knowledge and performance. Significant higher scores for participants in the HFS group in the studies by Larsen et al. (2020) and Solymos et al. (2015) demonstrated that HFS may be superior to conventional teaching methods for factual learning. In another study by Bartlett et al. (2021), HFS showed a significant long-term gain in knowledge over traditional teaching methods, but short-term knowledge gain was insignificant. Our study revealed that the Post-test MCQ score was higher in the HFS group but after adjustment of pre-test scores, there was no significant difference in knowledge gain between the control and intervention groups. The findings were similar in our preliminary study where  the intervention group had higher mean change score of MCQ scores than the control group but it was not statistically significant (Pal et al., 2021).

On the other hand, there was no significant knowledge improvement in both simulation and traditional teaching methods as observed in the studies (Corbridge et al., 2010; Kerr et al., 2013; Moadel et al., 2017). The findings of Alluri et al. (2016) also showed no difference in knowledge gain between simulation and lecture-based teaching. The studies by Morgan et al. (2002) and Tan et al. (2008), demonstrated equal efficacy between simulation and conventional lectures. The findings of a study by Kerr et al. (2013) demonstrated that SBME was not beneficial in acquisition and retention of knowledge. There was no significant improvement in knowledge after simulation-based education as revealed by the findings of three RCTs (Cavaleiro et al., 2009; Cherry et al., 2007; Kim et al., 2002).

Despite simulation being effective in acquisition of knowledge, it may not be the most efficient modality when compared to other traditional educational methods (Bordage et al., 2009). There is ample evidence that SBME usually leads to enhancement of knowledge and skills among undergraduate students but its superiority over other conventional teaching methods is yet to be defined (Nestel et al., 2015).

A. Limitations

There is a possibility of potential biases in the form of design, recruitment, sample populations and data analysis that could have influenced the findings. Due to randomization in blocks of two, the allocation of participants may be predictable which may result in selection bias. The confounding factors such as communication between the different groups of students prior to the second MCQ assessment, participants’ recall memory and preparation for the post-test after 7 – 8 weeks need to be considered. As it was a single-centre study which included final-year medical students only, the validity of the findings may not be applicable to other settings.

V. CONCLUSION

Conventional teaching modalities and HFS, when used in conjunction with bedside teaching, may complement clinical practice, leading to higher retention of knowledge. Therefore, more studies are required to measure the efficacy of simulation for a better understanding of the differences that it can make in the acquisition of knowledge. Our study revealed that the efficacy of high-fidelity simulation-based teaching was not superior to video-assisted lecture-based teaching in terms of knowledge acquisition and retention. The substantially higher cost and maintenance associated with HFS need to be considered before planning a teaching-learning activity. It may be used judiciously with conventional teaching when the objectives are mainly knowledge-based. More studies are required to determine its effectiveness and further evaluation as a teaching-learning tool in medical education.

Notes on Contributors

Bikramjit Pal was involved in Conceptualization, Formal Analysis, Literature Review, Methodology, Project administration & Supervision, Data Analysis and Writing (original draft & editing).

Aung Win Thein was involved in Formal Analysis, Literature Review, Methodology, Supervision and Writing (review & editing).

Sook Vui Chong was involved in Literature Review, Methodology, Supervision and Writing (review & editing).

Ava Gwak Mui Tay was involved in Formal Analysis, Literature Review, Supervision and Writing (review & editing).

Htoo Htoo Kyaw Soe was involved in Formal Analysis, Methodology, Data curation, Statistical Analysis and Validation.

Sudipta Pal was involved in Literature Review, Methodology, Formal Analysis, Data curation and Writing (review & editing).

Ethical Approval

Ethical approval was duly obtained from the Ethical Committee / IRB of Manipal University College Malaysia. Informed consent was taken from all the participants. All information about the participants was kept confidential.

Approval number: MMMC/FOM/Research Ethics Committee – 11/2018.

Data Availability

The data that supports the findings of this RCT study are openly available at Figshare repository, https://doi.org/10.6084/m9.figshare.19932053.v2 (Pal et al., 2022).

Acknowledgement

The authors would like to acknowledge the final year MBBS students of Manipal University College Malaysia who had participated in this research project, the faculty of the Department of Surgery, the lab assistants and technicians of Clinical Skills Lab and the Management of Manipal University College Malaysia.

Funding

The researchers had not received any funding or benefits from industry or elsewhere to conduct this study.

Declaration of Interest

The researchers had no conflicts of interest.

References

Accreditation Council for Graduate Medical Education. (2020, July 1). Program requirements for graduate medical education in general surgery. https://www.acgme.org/globalassets/pfassets/programrequirements/440_generalsurgery_2020.pdf.

Alluri, R. K., Tsing, P., Lee, E., & Napolitano, J. (2016). A randomized controlled trial of high-fidelity simulation versus lecture-based education in preclinical medical students. Medical Teacher, 38(4), 404–409. https://doi.org/10.3109/0142159X.2015.1031734

ATLS Subcommittee, American College of Surgeons’ Committee on Trauma, & International ATLS working group. (2013). Advanced trauma life support (ATLS®): The ninth edition. Journal of Trauma and Acute Care Surgery, 74(5), 1363–1366. https://doi.org/10.1097/TA.0b013e31828b82f5

Bartlett, R. S., Bruecker, S., & Eccleston, B. (2021). High-fidelity simulation improves long-term knowledge of clinical swallow evaluation. American Journal of Speech-Language Pathology, 30(2), 673–686. 

Bordage, G., Carlin, B., Mazmanian, P. E., & American College of Chest Physicians Health and Science Policy Committee (2009). Continuing medical education effect on physician knowledge: Effectiveness of continuing medical education: American College of Chest Physicians evidence-based educational guidelines. Chest, 135(3 Suppl), 29S–36S. https://doi.org/10.1378/chest.08-2515

Case, S. M., & Swanson, D. B. (2001). Constructing written test questions for the basic and clinical sciences. 3rd ed. Philadelphia: National Board of Medical Examiners.

Cavaleiro, A. P., Guimarães, H., & Calheiros, F. (2009). Training neonatal skills with simulators? Acta Paediatrica, 98(4), 636–639.    

Chen, T., Stapleton, S., Ledford, M., & Frallicciardi, A. (2017). Comparison of high-fidelity simulation versus case-based discussion on fourth- year medical student performance. Western Journal of Emergency Medicine: Integrating Emergency Care with Population Health, 18(5.1). https://escholarship.org/uc/item/5k73f4qc

Cherry, R. A., Williams, J., George, J., & Ali, J. (2007). The effectiveness of a human patient simulator in the ATLS shock skills station.  Journal of Surgical Research, 139(2), 229–235. https://doi.org/10.1016/j.jss.2006.08.010

Cook, D. A., Brydges, R., Hamstra, S. J., Zendejas, B., Szostek, J. H., Wang, A. T., Erwin, P. J., & Hatala, R. (2012). Comparative effectiveness of technology-enhanced simulation versus other instructional methods: A systematic review and meta-analysis. Simulation in Healthcare: The Journal of the Society for Simulation in Healthcare, 7(5), 308–320. https://doi.org/10.1097/SIH.0b013e3182614f95

Corbridge, S. J., Robinson, F. P., Tiffen, J., & Corbridge, T. C. (2010). Online learning versus simulation for teaching principles of mechanical ventilation to nurse practitioner students. International Journal of Nursing Education Scholarship, 7(1), Article 12. https://doi.org/10.2202/1548-923X.1976

Couto, T. B., Farhat, S. C., Geis, G. L., Olsen, O., & Schvartsman, C. (2015). High-fidelity simulation versus case-based discussion for teaching medical students in Brazil about pediatric emergencies. Clinics (Sao Paulo, Brazil), 70(6), 393–399. https://doi.org/10.6061/clinics/2015(06)02

Davies, R. (2008). The Bologna process: the quiet revolution in nursing higher education. Nurse Education Today, 28(8), 935–942. https://doi.org/10.1016/j.nedt.2008.05.008

Ellis, P. (2010). The essential guide to effect sizes: Statistical power, meta-analysis, and the interpretation of research results. Cambridge University Press. https://doi.org/10.1017/CBO9780511761676

Faul, F., Erdfelder, E., Lang, A. G., & Buchner, A. (2007). G*Power 3: a flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175–191. https://doi.org/10.3758/bf03193146

Gordon, J. A., Shaffer, D. W., Raemer, D. B., Pawlowski, J., Hurford, W. E., & Cooper, J. B. (2006). A randomized controlled trial of simulation-based teaching versus traditional instruction in medicine: A pilot study among clinical medical students. Advances in Health Sciences Education , 11(1), 33–39. https://doi.org/10.1007/s10459-004-7346-7

Hall, R. M. (2013). Effects of high fidelity simulation on knowledge acquisition, self-confidence, and satisfaction with baccalaureate nursing students using the solomon-four research design [Doctoral dissertation, East Tennessee State University]. East Tennessee State University Higher Education Commons. https://dc.etsu.edu/etd/2281

Kerr, B., Hawkins, T. L., Herman, R., Barnes, S., Kaufmann, S., Fraser, K., & Ma, I. W. (2013). Feasibility of scenario-based simulation training versus traditional workshops in continuing medical education: a randomized controlled trial. Medical Education Online, 18(1), Article 21312. https://doi.org/10.3402/meo.v18i0.21312

Kiernan, L. C. (2018). Evaluating competence and confidence using simulation technology. Nursing, 48(10), 45–52. https://doi.org/10.1097/01.NURSE.0000545022.36908.f3

Kim, J. H., Kim, W. O., Min, K. T., Yang, J. Y., & Nam, Y. T. (2002). Learning by computer simulation does not lead to better test performance on advanced cardiac life support than textbook study. The Journal of Education in Perioperative Medicine, 4(1), Article E019. 

Kolb, D. A. (1984). Experiential learning: Experience as the source of learning and development. Prentice Hall. 

La Cerra, C., Dante, A., Caponnetto, V., Franconi, I., Gaxhja, E., Petrucci, C., Alfes, C. M., & Lancia, L. (2019). Effects of high-fidelity simulation based on life-threatening clinical condition scenarios on learning outcomes of undergraduate and postgraduate nursing students: A systematic review and meta-analysis. BMJ Open, 9(2), Article e025306. https://doi.org/10.1136/bmjopen-2018-025306

Larsen, T., Jackson, N., & Napolitano, J. (2020). A comparison of simulation-based education and problem-based learning in pre-clinical medical undergraduates. MedEdPublish, 9(1), Article 172.

Lo, B. M., Devine, A. S., Evans, D. P., Byars, D. V., Lamm, O. Y., Lee, R. J., Lowe, S. M., & Walker, L. L. (2011). Comparison of traditional versus high-fidelity simulation in the retention of ACLS knowledge. Resuscitation, 82(11), 1440–1443. https://doi.org/10.1016/j.resuscitation.2011.06.017

McGaghie, W. C., Siddall, V. J., Mazmanian, P. E., Myers, J., & American College of Chest Physicians Health and Science Policy Committee (2009). Lessons for continuing medical education from simulation research in undergraduate and graduate medical education: Effectiveness of continuing medical education: American College of Chest Physicians Evidence-Based Educational Guidelines. Chest, 135(3 Suppl), 62S–68S. https://doi.org/10.1378/chest.08-2521

Miller, G. E. (1990). The assessment of clinical skills/competence/performance. Academic Medicine, 65(9), S63–S67. https://doi.org/10.1097/00001888-199009000-00045

Moadel, T., Varga, S., & Hile, D. (2017). A prospective randomized controlled trial comparing simulation, lecture and discussion-based education of sepsis to emergency medicine residents. Western Journal of Emergency Medicine: Integrating Emergency Care with Population Health, 18(5.1). https://escholarship.org/uc/item/0132981t

Morgan, P. J., Cleave-Hogg, D., McIlroy, J., & Devitt, J. H. (2002). Simulation technology: A comparison of experiential and visual learning for undergraduate medical students. Anesthesiology, 96, 10–16. https://doi.org/10.1097/00000542-200201000-00008

Nestel, D., Harlim, J., Smith, C., Krogh, K., & Bearman, M. (2015). Simulated learning technologies in undergraduate curricula: An evidence check review for HETI.

Norman, G. (2009). The American College of Chest Physicians evidence-based educational guidelines for continuing medical education interventions: a critical review of evidence-based educational guidelines. Chest, 135(3), 834–837. https://doi.org/10.1378/chest.09-0036

Pal, B., Chong, S. V., Thein, A. W., Tay, A. G., Soe, H. H., & Pal, S. (2021). Is high-fidelity patient simulation-based teaching superior to video-assisted lecture-based teaching in enhancing knowledge and skills among undergraduate medical students? Journal of Health and Translational Medicine, 24(1), 83-90. https://doi.org/10.22452/jummec.vol24no1.14

Pal, B., Thein, A. W., Chong, S. V., Tay, A., Htoo, H., & Pal, S. (2022). A randomized controlled trial study to compare the effectiveness of high-fidelity based teaching with video-assisted based lecture teaching in enhancing knowledge [Dataset]. Figshare. https://doi.org/10.6084/m9.figshare.19932053

Ray, S. M., Wylie, D. R., Shaun Rowe, A., Heidel, E., & Franks, A. S. (2012). Pharmacy student knowledge retention after completing either a simulated or written patient case. American Journal of Pharmaceutical Education, 76(5), 86. https://doi.org/10.5688/ajpe76586

Rogers, D. A. (2008). The role of simulation in surgical continuing medical education. Seminars in Colon and Rectal Surgery, 19(2), 108-114. https://doi.org/10.1053/j.scrs.2008.02.007

Solymos, O., O’Kelly, P., & Walshe, C. M. (2015). Pilot study comparing simulation-based and didactic lecture-based critical care teaching for final-year medical students. BMC Anesthesiology, 15, Article 153. https://doi.org/10.1186/s12871-015-0109-6

Tan, G. M., Ti, L. K., Tan, K., & Lee, T. (2008). A comparison of screen-based simulation and conventional lectures for undergraduate teaching of crisis management. Anaesthesia and Intensive Care, 36(4), 565–569.

Ten Eyck, R. P., Tews, M., & Ballester, J. M. (2009). Improved medical student satisfaction and test performance with a simulation-based emergency medicine curriculum: a randomized controlled trial. Annals of Emergency Medicine, 54(5), 684–691. https://doi.org/10.1016/j.annemergmed.2009.03.025

Vijayaraghavan, S., Rishipathak, P., & Hinduja, A. (2019). High-fidelity simulation versus case-based discussion for teaching bradyarrhythmia to emergency medical services students. Journal of Emergencies, Trauma, and Shock, 12(3), 176–178. https://doi.org/10.4103/JETS.JETS_115_18

Yang, Y., & Liu, H. P. (2016). Systematic evaluation influence of high-fidelity simulation teaching on clinical competence of nursing students. Chinese Nursing Research, 30(7), 809–814. https://caod.oriprobe.com/articles/47628779/Systematic_evaluation_influence_of_high_fidelity_s.htm

*Bikramjit Pal
RCSI & UCD Malaysia Campus (RUMC),
4 Jalan Sepoy Lines,
Georgetown, Penang, 10450, Malaysia
+6042171908-1908 (Ext)
Email: bikramjit.pal@rcsiucd.edu.my