Number of Citations: 1

Published online: 4 September, TAPS 2018, 3(3), 39-42
DOI: https://doi.org/10.29060/TAPS.2018-3-3/SC1057

Lim Mien Choo Ruth, Keith Tsou Yu Kei, Chooi Peng Ong, Sabrina Wong Kay Wye, Gilbert Tan Choon Seng, Winnie Soon Shok Wen, Joanne Quah Hui Min & Marie Stella P. Cruz

Division of Graduate Medical Studies, Yong Loo Lin School of Medicine, National University of Singapore

Abstract

This paper describes the revision of a national post-graduate medical examination to incorporate formal quality assurance and psychometrics. We discuss the considerations and rationale leading to the new format, challenges faced and lessons learned in making the change. The processes described were successfully implemented in the 2015 examination administration. We continue to reflect on and analyse these processes to improve the examination.

Keywords:           Post-graduate, Examination Reform, Quality Assurance, Psychometrics, Family Medicine, Standardised Patient

I. CONTEXT

In Singapore, population growth and a swiftly inverting population pyramid have diametrically changed the job scope of the family physician, impacting training and certification.

Postgraduate Family Medicine certification in Singapore formally began with the Membership of the College of General Practitioners (MCGP) in 1972, and was replaced in 1993 by the Master of Medicine in Family Medicine (MMed FM) examination. This examination comprised a theory portion, a viva voce and a clinical examination (Table 1).

In 2011, Singapore adopted the American Accreditation Council for Graduate Medical Education-International (ACGME-I) Residency system. Family Medicine Residency culminates in a summative multiple-choice question (MCQ) examination administered by the American Board of Medical Specialties (ABMS) with no clinical component.

With these training and contextual changes, the time was appropriate for a review and revision of the MMed FM examination. We describe the considerations and processes leading to the new MMed FM examination in 2015.

II. FROM THE OLD TO THE NEW

A. Family Medicine Examination Committee

In 2012, the Family Medicine Examination Committee (FMEC) was set up to administer and review the examination, reporting to the Division of Graduate Medical Studies (DGMS), with education consultative support from the Centre for Medical Education, National University of Singapore.  Senior family physicians from different training institutions form the FMEC.

B. Reviewing the Old Examination

Table 1. Summary of main components of the old and 2015 examinations

The previous long case involved assessing the patient and discussing management with the examiner. The short case was a physical examination station. Two long cases provided insufficient insight into the candidate’s mastery of the breadth of the Family Medicine curriculum (Swanson, Norman, & Linn, 1995). With real patients, standardisation of testing points across a cohort was not possible. Management was discussed with the examiner only, so communication with the patient about management was not assessed.

The old pass mark was pre-defined at 50%, regardless of difficulty of cases.

C. Developing the New Examination

We developed the new examination following review of literature (Swanson, Norman, & Linn, 1995; Van der Vleuten & Schuwirth, 2005; Mookherjee, Chang, Boscardin, & Hauer, 2013) and discussion to determine best practice. Our priorities were to improve the representativeness and reliability of testing within a practicable framework.

The new examination includes theory and clinical components, and psychometrics to analyse candidate performance. The theory component is now the ABMS MCQ examination. Candidates who are successful may sit the clinical examination.

The re-designed clinical examination comprises thirteen stations, with the slides and viva voce stations being holdovers from the old format. The remaining eleven comprise five short consultations, three long consultations, and three physical examination stations, an increase over the previous six stations (Swanson et al, 1995). This increases the number of sampling points and testing time and hence reliability (Van der Vleuten & Schuwirth, 2005).

For the consultations, the candidate evaluates, and then discusses management with the patient. He is judged on his handling of the consultation, not on his interaction with examiners. A long consultation tests multiple and more complex issues than a short consultation. Consultations are set in a primary care context, representing Family Medicine practice in Singapore.

The standardised patient (SP) was introduced, and consultations involve real and standardised patients. The three physical examination stations continue to use real patients.

The viva voce was retained to allow testing of professional communication and discussion of clinical reasoning.

Instead of a pre-defined standard, we introduced the use of psychometrics to determine the pass marks, and collaborated with a medical educationist who has been trained in assessment psychometrics. At the same time, we implemented examiner training (Van der Vleuten, Luyk, Ballegooijen, & Swanson, 1989), calibration, and standard setting. The passing standard is defined as that of a proficient family physician on the Dreyfus model of skill acquisition (Goh & Ong, 2014).

A new mark sheet, assessing clinical skills and global assessment of the candidate’s performance, was introduced.

D. Evaluating the New Examination

The FMEC committed to incorporating quality assurance measures into the design and implementation of the new clinical examination.

For the first time, an examination blueprint was formally developed. Testing points were chosen across the curriculum to reflect competencies required in Family Medicine practice (Mookherjee et al, 2013).

The blueprint guided recruitment of real patients and SP case development. FMEC members tested the SP cases for authenticity and reasonableness. Before each clinical examination bloc, all examiners participated in calibration sessions during which clinical history and signs were confirmed and salient points agreed upon.

Borderline regression and modified Angoff methods were used to set pass marks. Performance and reliability of individual stations and the entire examination were studied. Examiner performance was analysed and feedback given.

The examination, conducted under the auspices of the DGMS, conformed to standard operating procedures, access and security measures used in other DGMS examinations.

An external examiner (EE) observed the examination (Associate Professor Neil Spike). This EE had observed previous MMed FM examinations.

III. CHALLENGES AND LESSONS

The old examination format had been in place for forty-three years (1972 – 2014). The new format introduced standard setting, examination validity as an evidence-based concept, the use of SPs, and the use of statistics to analyse results.

Stakeholder engagement was, and remains, paramount. There was a need for consistent, sustained communication to trainers, residents and examiners.

With the consultation as test encounter, we are assessing for relevant history and hypothesis-driven physical examination. This differs from traditional assessment that rewards comprehensiveness. Examiners debated the boundaries of relevance and hypothesis-driven practice, to help form their expert judgement of what is required of proficiency.

With the many changes introduced, we set a requirement for examiners to have attended examiner training sessions before appointment. This is common in many examinations but new to us.

The format, as presented here, will remain for 2016 and 2017, after which time we will review the programme of assessment. At this time, we are in the process of gathering feedback and reviewing the outcomes for the examination administrations of 2015 and 2016.

Notes on Contributors

Dr Ruth Lim is the Chairperson of the Family Medicine Examination committee, Chief Examiner for the Master of Medicine (Family Medicine) Examination and Director of Education for SingHealth Polyclinics.

Dr Keith Tsou is the Deputy Chairperson of the Family Medicine Examination committee, Deputy Chief Examiner for the Master of Medicine (Family Medicine) Examination and Director of Clinical Services for National University Polyclinics.

Dr Ong Chooi Peng is member of the Family Medicine Examination committee, Senior Consultant of the Department of Family Medicine, National University Health System.

Dr Sabrina Wong is member of the Family Medicine Examination committee, Assistant Director of Clinical Services for National Healthcare Group Polyclinics.

Dr Gilbert Tan is member of the Family Medicine Examination committee, Assistant Director of Clinical Services for SingHealth Polyclinics.

Dr Winnie Soon is member of the Family Medicine Examination committee, Consultant Family Physician for National Healthcare Group Polyclinics.

Dr Joanne Quah is member of the Family Medicine Examination committee, Director for Outram Polyclinics under SingHealth Polyclinics.

Dr Marie Stella P. Cruz is member of the Family Medicine Examination committee, Family Physician and Adjunct Lecturer for Department of Medicine and Division of Graduate Medical Studies, National University of Singapore.

Ethical Approval

No IRB approval is involved.

Declaration of Interest

Authors declare no conflict of interest.

References

Swanson, D. B., Norman, G. R., & Linn, R. L. (1995). Performance-based assessment: Lessons from the health professions. Educational Research, 24(5), 5-11. https://doi.org/10.3102/0013189X024005005.

Van der Vleuten, C. P. M., & Schuwirth, L. W. T. (2005). Assessing professional competence: from methods to programmes. Medical Education, 39, 309-17. Retrieved from https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1365-2929.2005.02094.x.

Vleuten, C. V., Luyk, S. V., Ballegooijen, A. V. & Swanson, D. B. (1989). Training and experience of examiners. Medical Education, 23(3), 290-6.

Goh, L. G. & Ong, C. P. (2014). Education and training in family medicine: progress and a proposed national vision for 2030. Singapore Medical Journal, 55(3), 117-23. http://dx.doi.org/10.11622/smedj.2014031.

Mookherjee, S., Chang, A., Boscardin, C. K. & Hauer, K. E. (2013). How to develop a competency-based examination blueprint for longitudinal standardized patient clinical skills assessments. Medical Teacher, 35(11), 883-90. https://doi.org/10.3109/0142159X.2013.809408.

*Ruth Lim
Division of Graduate Medical Studies,
Yong Loo Lin School of Medicine,
National University of Singapore
Blk MD3, Level 2, 16 Medical Drive,
Singapore 117600
Email: ruth.lim@singhealth.com.sg
Contact: +65-63777018