Family medicine post-graduate examination reforms in Singapore: A value-driven practical and contextual approach
Published online: 4 September, TAPS 2018, 3(3), 39-42
DOI: https://doi.org/10.29060/TAPS.2018-3-3/SC1057
Lim Mien Choo Ruth, Keith Tsou Yu Kei, Chooi Peng Ong, Sabrina Wong Kay Wye, Gilbert Tan Choon Seng, Winnie Soon Shok Wen, Joanne Quah Hui Min & Marie Stella P. Cruz
Division of Graduate Medical Studies, Yong Loo Lin School of Medicine, National University of Singapore
Abstract
This paper describes the revision of a national post-graduate medical examination to incorporate formal quality assurance and psychometrics. We discuss the considerations and rationale leading to the new format, challenges faced and lessons learned in making the change. The processes described were successfully implemented in the 2015 examination administration. We continue to reflect on and analyse these processes to improve the examination.
Keywords: Post-graduate, Examination Reform, Quality Assurance, Psychometrics, Family Medicine, Standardised Patient
I. CONTEXT
In Singapore, population growth and a swiftly inverting population pyramid have diametrically changed the job scope of the family physician, impacting training and certification.
Postgraduate Family Medicine certification in Singapore formally began with the Membership of the College of General Practitioners (MCGP) in 1972, and was replaced in 1993 by the Master of Medicine in Family Medicine (MMed FM) examination. This examination comprised a theory portion, a viva voce and a clinical examination (Table 1).
In 2011, Singapore adopted the American Accreditation Council for Graduate Medical Education-International (ACGME-I) Residency system. Family Medicine Residency culminates in a summative multiple-choice question (MCQ) examination administered by the American Board of Medical Specialties (ABMS) with no clinical component.
With these training and contextual changes, the time was appropriate for a review and revision of the MMed FM examination. We describe the considerations and processes leading to the new MMed FM examination in 2015.
II. FROM THE OLD TO THE NEW
A. Family Medicine Examination Committee
In 2012, the Family Medicine Examination Committee (FMEC) was set up to administer and review the examination, reporting to the Division of Graduate Medical Studies (DGMS), with education consultative support from the Centre for Medical Education, National University of Singapore. Senior family physicians from different training institutions form the FMEC.
B. Reviewing the Old Examination
Old MMed FM Examination | 2015 MMed FM Examination | Rationale for change | |
Format | Theory
Essays, Short Answers, MCQ, Slides, Viva voce |
Theory
250 MCQ Paper |
Increased sampling points to increase reliability |
Clinical Examination
2 Long Cases, 4 Short Cases |
Clinical Examination
Slides, Viva voce, 3 Long Consultations, 5 Short Consultations, 3 Physical Examinations |
Increased sampling points to increase reliability
|
|
Duration of Clinical Examination | 150 minutes | 218 minutes | Increased testing time |
Domains Tested | Knowledge – Theory papers, Long Cases
Synthesis, Psychomotor skills – Short Cases Professional Communication – Long Cases, Viva voce
|
Knowledge, Interpretation, Application – Context-rich MCQs
Synthesis, Application, Patient Communication – Consultations Psychomotor skills – Physical Examinations Professional Communication – Viva voce |
Increased number of domains tested
Higher level of cognitive domains tested |
Blueprinting | Theory
No formal blueprinting |
Theory
Formal blueprinting |
Defined composition of testing points |
Clinical Examination
Long Cases were “eyeballed” to cover adult and paediatric problems Short Cases blueprinted by broad body systems and “eyeballed” such that a candidate did not have similar Long and Short cases |
Clinical Examination
Formal blueprinting across Consultations and Physical Examination stations |
Defined composition of testing points
|
|
Clinical Material used | Real Patients (RP) | Real Patients
Standardised Patients (SP) |
Increased number of sampling points
Increased type of testing points Richness of clinical material with use of RP Standardisation of clinical material with use of SP |
Quality Assurance | Blueprinting | Blueprinting | Increased content validity with new blueprint |
Examiner training by observation | Formal Examiner Training & Feedback | Systematisation of examiner training | |
Pre-set Pass Mark | Standard setting methods and psychometrics in Pass/Fail decisions | Better defensibility of Pass/Fail decisions | |
External Examiner | External Examiner | Independent audit of examination process |
Table 1. Summary of main components of the old and 2015 examinations
The previous long case involved assessing the patient and discussing management with the examiner. The short case was a physical examination station. Two long cases provided insufficient insight into the candidate’s mastery of the breadth of the Family Medicine curriculum (Swanson, Norman, & Linn, 1995). With real patients, standardisation of testing points across a cohort was not possible. Management was discussed with the examiner only, so communication with the patient about management was not assessed.
The old pass mark was pre-defined at 50%, regardless of difficulty of cases.
C. Developing the New Examination
We developed the new examination following review of literature (Swanson, Norman, & Linn, 1995; Van der Vleuten & Schuwirth, 2005; Mookherjee, Chang, Boscardin, & Hauer, 2013) and discussion to determine best practice. Our priorities were to improve the representativeness and reliability of testing within a practicable framework.
The new examination includes theory and clinical components, and psychometrics to analyse candidate performance. The theory component is now the ABMS MCQ examination. Candidates who are successful may sit the clinical examination.
The re-designed clinical examination comprises thirteen stations, with the slides and viva voce stations being holdovers from the old format. The remaining eleven comprise five short consultations, three long consultations, and three physical examination stations, an increase over the previous six stations (Swanson et al, 1995). This increases the number of sampling points and testing time and hence reliability (Van der Vleuten & Schuwirth, 2005).
For the consultations, the candidate evaluates, and then discusses management with the patient. He is judged on his handling of the consultation, not on his interaction with examiners. A long consultation tests multiple and more complex issues than a short consultation. Consultations are set in a primary care context, representing Family Medicine practice in Singapore.
The standardised patient (SP) was introduced, and consultations involve real and standardised patients. The three physical examination stations continue to use real patients.
The viva voce was retained to allow testing of professional communication and discussion of clinical reasoning.
Instead of a pre-defined standard, we introduced the use of psychometrics to determine the pass marks, and collaborated with a medical educationist who has been trained in assessment psychometrics. At the same time, we implemented examiner training (Van der Vleuten, Luyk, Ballegooijen, & Swanson, 1989), calibration, and standard setting. The passing standard is defined as that of a proficient family physician on the Dreyfus model of skill acquisition (Goh & Ong, 2014).
A new mark sheet, assessing clinical skills and global assessment of the candidate’s performance, was introduced.
D. Evaluating the New Examination
The FMEC committed to incorporating quality assurance measures into the design and implementation of the new clinical examination.
For the first time, an examination blueprint was formally developed. Testing points were chosen across the curriculum to reflect competencies required in Family Medicine practice (Mookherjee et al, 2013).
The blueprint guided recruitment of real patients and SP case development. FMEC members tested the SP cases for authenticity and reasonableness. Before each clinical examination bloc, all examiners participated in calibration sessions during which clinical history and signs were confirmed and salient points agreed upon.
Borderline regression and modified Angoff methods were used to set pass marks. Performance and reliability of individual stations and the entire examination were studied. Examiner performance was analysed and feedback given.
The examination, conducted under the auspices of the DGMS, conformed to standard operating procedures, access and security measures used in other DGMS examinations.
An external examiner (EE) observed the examination (Associate Professor Neil Spike). This EE had observed previous MMed FM examinations.
III. CHALLENGES AND LESSONS
The old examination format had been in place for forty-three years (1972 – 2014). The new format introduced standard setting, examination validity as an evidence-based concept, the use of SPs, and the use of statistics to analyse results.
Stakeholder engagement was, and remains, paramount. There was a need for consistent, sustained communication to trainers, residents and examiners.
With the consultation as test encounter, we are assessing for relevant history and hypothesis-driven physical examination. This differs from traditional assessment that rewards comprehensiveness. Examiners debated the boundaries of relevance and hypothesis-driven practice, to help form their expert judgement of what is required of proficiency.
With the many changes introduced, we set a requirement for examiners to have attended examiner training sessions before appointment. This is common in many examinations but new to us.
The format, as presented here, will remain for 2016 and 2017, after which time we will review the programme of assessment. At this time, we are in the process of gathering feedback and reviewing the outcomes for the examination administrations of 2015 and 2016.
Notes on Contributors
Dr Ruth Lim is the Chairperson of the Family Medicine Examination committee, Chief Examiner for the Master of Medicine (Family Medicine) Examination and Director of Education for SingHealth Polyclinics.
Dr Keith Tsou is the Deputy Chairperson of the Family Medicine Examination committee, Deputy Chief Examiner for the Master of Medicine (Family Medicine) Examination and Director of Clinical Services for National University Polyclinics.
Dr Ong Chooi Peng is member of the Family Medicine Examination committee, Senior Consultant of the Department of Family Medicine, National University Health System.
Dr Sabrina Wong is member of the Family Medicine Examination committee, Assistant Director of Clinical Services for National Healthcare Group Polyclinics.
Dr Gilbert Tan is member of the Family Medicine Examination committee, Assistant Director of Clinical Services for SingHealth Polyclinics.
Dr Winnie Soon is member of the Family Medicine Examination committee, Consultant Family Physician for National Healthcare Group Polyclinics.
Dr Joanne Quah is member of the Family Medicine Examination committee, Director for Outram Polyclinics under SingHealth Polyclinics.
Dr Marie Stella P. Cruz is member of the Family Medicine Examination committee, Family Physician and Adjunct Lecturer for Department of Medicine and Division of Graduate Medical Studies, National University of Singapore.
Ethical Approval
No IRB approval is involved.
Declaration of Interest
Authors declare no conflict of interest.
References
Swanson, D. B., Norman, G. R., & Linn, R. L. (1995). Performance-based assessment: Lessons from the health professions. Educational Research, 24(5), 5-11. https://doi.org/10.3102/0013189X024005005.
Van der Vleuten, C. P. M., & Schuwirth, L. W. T. (2005). Assessing professional competence: from methods to programmes. Medical Education, 39, 309-17. Retrieved from https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1365-2929.2005.02094.x.
Vleuten, C. V., Luyk, S. V., Ballegooijen, A. V. & Swanson, D. B. (1989). Training and experience of examiners. Medical Education, 23(3), 290-6.
Goh, L. G. & Ong, C. P. (2014). Education and training in family medicine: progress and a proposed national vision for 2030. Singapore Medical Journal, 55(3), 117-23. http://dx.doi.org/10.11622/smedj.2014031.
Mookherjee, S., Chang, A., Boscardin, C. K. & Hauer, K. E. (2013). How to develop a competency-based examination blueprint for longitudinal standardized patient clinical skills assessments. Medical Teacher, 35(11), 883-90. https://doi.org/10.3109/0142159X.2013.809408.
*Ruth Lim
Division of Graduate Medical Studies,
Yong Loo Lin School of Medicine,
National University of Singapore
Blk MD3, Level 2, 16 Medical Drive,
Singapore 117600
Email: ruth.lim@singhealth.com.sg
Contact: +65-63777018
Announcements
- Fourth Thematic Issue: Call for Submissions
The Asia Pacific Scholar is now calling for submissions for its Fourth Thematic Publication on “Developing a Holistic Healthcare Practitioner for a Sustainable Future”!
The Guest Editors for this Thematic Issue are A/Prof Marcus Henning and Adj A/Prof Mabel Yap. For more information on paper submissions, check out here! - Best Reviewer Awards 2023
TAPS would like to express gratitude and thanks to an extraordinary group of reviewers who are awarded the Best Reviewer Awards for 2023.
Refer here for the list of recipients. - Most Accessed Article 2023
The Most Accessed Article of 2023 goes to Small, sustainable, steps to success as a scholar in Health Professions Education – Micro (macro and meta) matters.
Congratulations, A/Prof Goh Poh-Sun & Dr Elisabeth Schlegel! - Best Article Award 2023
The Best Article Award of 2023 goes to Increasing the value of Community-Based Education through Interprofessional Education.
Congratulations, Dr Tri Nur Kristina and co-authors! - Volume 9 Number 1 of TAPS is out now! Click on the Current Issue to view our digital edition.
- Best Reviewer Awards 2022
TAPS would like to express gratitude and thanks to an extraordinary group of reviewers who are awarded the Best Reviewer Awards for 2022.
Refer here for the list of recipients. - Most Accessed Article 2022
The Most Accessed Article of 2022 goes to An urgent need to teach complexity science to health science students.
Congratulations, Dr Bhuvan KC and Dr Ravi Shankar. - Best Article Award 2022
The Best Article Award of 2022 goes to From clinician to educator: A scoping review of professional identity and the influence of impostor phenomenon.
Congratulations, Ms Freeman and co-authors. - Volume 8 Number 3 of TAPS is out now! Click on the Current Issue to view our digital edition.
- Best Reviewer Awards 2021
TAPS would like to express gratitude and thanks to an extraordinary group of reviewers who are awarded the Best Reviewer Awards for 2021.
Refer here for the list of recipients. - Most Accessed Article 2021
The Most Accessed Article of 2021 goes to Professional identity formation-oriented mentoring technique as a method to improve self-regulated learning: A mixed-method study.
Congratulations, Assoc/Prof Matsuyama and co-authors. - Best Reviewer Awards 2020
TAPS would like to express gratitude and thanks to an extraordinary group of reviewers who are awarded the Best Reviewer Awards for 2020.
Refer here for the list of recipients. - Most Accessed Article 2020
The Most Accessed Article of 2020 goes to Inter-related issues that impact motivation in biomedical sciences graduate education. Congratulations, Dr Chen Zhi Xiong and co-authors.