I. INTRODUCTION

Objective structured clinical examination OSCE is invented in 1975 by Harden (Harden & Gleeson, 1979) for the assessment of learners’ clinical competences and behaviors by using actors and choreographed storylines (Hodges, 2003). He succeeded in controlling the classic variables, the patient and the examiner, that enabled him to establish a comprehensive and objective assessment (Khan et al., 2013) of competence by defining clearly what skills, attitudes, problem solving capabilities and factual knowledge are to be measured (Harden et al., 1975).

As quoted by Harden “Competency is the compound of cognitive, psychomotor and affective skills as appropriate, while competence is an attribute of a person” (Khan et al., 2013).

During an OSCE, candidates are supposed to execute different clinical tasks in a simulated setting (Khan et al., 2013). As a rule, the students rotate through several time limited stations in which they are expected to interact with a standardised patient (SP), mannequins or simulation models and perform a specified task meanwhile they are being assessed by examiners using standardised rating instruments (Pugh & Smee, 2013). OSCE gets rid of many detrimental concepts in examining students, faced previously with other exam methods, by compelling them all to go through the same scope and criteria for assessment (Zayyan, 2011). This has made it a valuable evaluative tool in medical practice, so it has been adopted in countries all over the world, in all the high stakes examinations in USA (Dwivedi et al., 2020), Canada (Pugh & Smee, 2013), and the UK (Gormley, 2011).

Reliability and validity of the OSCE exam is directly related to how it is implemented (Harden & Gleeson, 1979) and can be maximised by several ways, the first and the foremost is the designing of structured reconcilable mark schemes for several stations observed by different trained assessors which will eliminate the individual assessor bias (Gormley, 2011). Competence assessment will be more reliable by arranging a variety of patient presentations for different cases and skills and standardising patients’ performance (Dent et al., 2021; Khan et al., 2013).

As a prerequisite of a good test, a process called standard setting must be set that if followed will lead to a fair decision (Boulet et al., 2003). The inferences derived from a test result matter a lot to the examiners, examinees and the institutes (Norcini et al., 2011). Cusimano in his review paper defines standard setting as a process that determines “what is good enough” for assessing competence, which itself is continuously changing, and leads to separation of a competent student from an incompetent (Cusimano, 1996). According to Harden the standard is the score that decides pass fail status of the students, also known as pass fail point. It provides an answer to the question” how much or what is good enough to know?” (Dent et al., 2021). He has defined the standard setting as the process of translating a description of characteristics denoting the desired level of performance into a number that applies to a particular test” (Dent et al., 2021).

At the time of setting the standards, the purpose of the exam must be considered (Kamal et al., 2020) along with the consequences of letting an incompetent examinee get through the exams and acquire medical licensure that could be devastating (Gormley, 2011).

Standard setting methods are designated into norm-referenced, criterion-referenced methods and a third category of combination or compromise methods (Dwivedi et al., 2020; Kamal et al., 2020). In absolute or criterion referenced standards a benchmark is set based on certain predefined criteria and the candidate performance is tested according to that standard competency or mastery. Whereas Norm-referenced, also called relative methods, are based on identification of the cut-off score relative to performance of the group or top scoring students taking the examination, which results in loss of motivation for progressing and improving in top scoring students (Dwivedi et al., 2020).

For assessing the quality of OSCE exam, the determinants are dictated by the method of standard setting. The AMEE Guide 85 describes a number of standard setting methods of which Cohen, Angoff, Borderline Regression, Borderline Group, Hofstee Method, and the fixed arbitrary 60% method are some of the commonly used (McKinley & Norcini, 2014).

Cohen method is the best form of the norm-referenced standard setting methods extensively used in low stakes exams. The best performing students’ mark is used as a reference point to define the difficulty of the exam. The remaining students’ scores are arranged from the lowest to the highest scores; the mean value of the top 5% of the scores is calculated, and finally, 60% of the total mean score is considered as the standard/passing score (Kamal et al., 2020).

Angoff method is entirely based on test/examination items (Pell et al., 2010). In this method the pass mark is statistically calculated on item or station characteristics, and it varies according to the difficulty level of the station defined by the items on checklist, but the students’ performance is not taken into consideration. (Dwivedi et al., 2020; Impara & Plake, 1997).

The borderline methods are reasonable and defensible as they are based on candidates’ performance (Kaufman et al., 2000; Pell et al., 2010). So borderline regression and modified borderline Group methods are also known as “Examinee centered” methods (Dwivedi et al., 2020). Borderline group methods necessitate the examiner be able to identify what is considered as minimally acceptable performance. The mean or median score of minimally acceptable performances is declared as cutoff score (Cusimano, 1996; Humphrey-Murto & MacFadyen, 2002). Apart from checklist scores, a global grade is also awarded which provides insights into quality of assessment (Pell et al., 2010; Smee et al., 2022).

Hofstee method aims to achieve a balance between the norm and criterion reference judgements and is a combination/compromise method (Dwivedi et al., 2020). In this method the examiners specify 4 values before the exam: the maximum and the minimum percentage correct, and the maximum and minimum acceptable percentage of failures (Smee et al., 2022). This method is more calculative, but it avoids illogical very high and very low scores (Cusimano, 1996; Kamal et al., 2020).

The arbitrary 60% method uses faculty wide standard of passing score of 60% in OSCE exam and is the easiest to implement (Humphrey-Murto & MacFadyen, 2002; Kamal et al., 2020; Kaufman et al., 2000).

Until August 2022, the clinical science department at Al Rayyan college of medicine was applying an arbitrary cut off score of 60% as a passing score for OSCE. This decision had always been based on tradition, without taking test content or students’ performance into consideration. The need for a process to differentiate well between a student with adequate competencies from those having inadequate competencies had always been observed (Khan et al., 2013). The examinee centered Hofstee method can help us to adjust cut scores for a station according to its difficulty level and accepted number of students unable to pass such a station. (Downing et al., 2006; Dudas & Barone 2014; Hofstee, 1983).

The purpose of this study is to compare the pass /fail rates of students achieved by applying arbitrary and the Hofstee methods and to assess if Hofstee method can provide us satisfactory results.

II. METHODS

The current study is a descriptive study design conducted at Al Rayyan college of medicine department of clinical science. Al Rayan college of medicine, Al Rayan national colleges is a newly established private institute based in Al-Medina Al-Munawara, under Ministry of education at Kingdom of Saudi Arabia. Having been established in 2017, the first batch of graduates have completed MBBS and have joined the local and international health sector. Currently there are 700 students enrolled and studying in 6 academic years.

This study includes a total number of 102 year 4 students undertaking the final OSCEs in the general practice 1 course with foundation to general practice, Cardiovascular system (CVS) and endocrine and breast modules during semester 1 of academic year 2022-23. The project was approved by the Research Ethical Committee (REC) of Al Rayyan colleges. All the students consented to the use of their data for research and quality control purposes with the agreement that any reports would only use aggregate data with all personal identifiers removed. 

The OSCE consisted of 10 stations that sampled common and important patient presentations. Examinees were required to complete each station within 07 min. Performance was scored using 10 predefined competencies related to general practice competencies aligned to course learning outcomes CLOs, designed under the umbrella of the competence specifications for Saudi medical graduates (Saudi Meds). Skill competency assessed were (1) history taking, (2) physical examination, (3) analysis and interpretation of findings, (4) communication, (5) suggestion of appropriate investigations, (6) listing relevant differential diagnoses, (7) management care plan development. For values assessment, there were three competencies: (1) ethical rules and confidentiality, (2) taking and maintaining consent, and (3) time management. Three to four of these competencies were assessed in each station except for clinical approach; management stations where only one competency is evaluated.

3 panels were laid down, each having 10 stations and 2 circuits of students. Students rotated through the stations completing a single circuit in an anticlockwise manner. Every student was examined by a single examiner at each station except for the station of data interpretation chest Xray which was just monitored by a silent invigilator and students were recording their answers on answer sheet.

Examiners were all trained faculty staff from department of clinical sciences, 12 examiners belonged to the college faculty, 17 joined from Taibah college of medicine, Taiba university. They received formal training sessions 2 hours ahead of the exam that began with information about the OSCE (fundamentals, competencies being assessed, rating guidelines and cases and question items were explained), followed by instructions on scoring through a google link. Four Hofstee questions were presented, discussed and answered by each examiner for each station and the mean percentage for each of the four questions across all examiners was computed. Meanwhile examiners were asked to answer the same four questions for the overall scores for the exam.

During the OSCE, examiners scored examinee performances within their assigned stations using the 20-26 items scale for each station except for interpretation; chest x ray station which was the only station having 5 item scale. Global ratings (overall assessment from 0 to 5) were also included.

The examiners decided that the cut score for minimally acceptable performance for the whole exam should be no lower than 57.5% and no higher than 76%. Similarly, they indicated that the failure rate should be at least 9% but no higher than 32%.

For cut score calculation, the student’s obtained score is plotted with scores along X coordinate and the number of candidates along the Y coordinate. A line graph is drawn showing the score and the number of students obtaining that score. The finally calculated Hofstee limits of cut scores and failure rates are drawn on the graph, which resulted in generation of a rectangle, the cross diagonal from top left of the rectangle to bottom right is drawn. The place where it intercepts the plot of cumulative number of candidates is the cut score for the stations. The graph is shown in figure 1. The same graphs were drawn for the individual stations and their cut scores were calculated. The detail of each station is not mentioned to avoid complexity.

Table 1. Stations Analysis

The descriptive results for the individual stations with their titles, maximum, minimum and average scores obtained are illustrated in Table 2. The cut scores calculated by arbitrary and hofstee methods are applied to the stations and accordingly pass percentages achieved are shown in Table 2.