{"id":70467,"date":"2021-04-13T12:07:17","date_gmt":"2021-04-13T04:07:17","guid":{"rendered":"https:\/\/medicine.nus.edu.sg\/taps\/?post_type=issues&#038;p=70467"},"modified":"2021-07-13T09:10:04","modified_gmt":"2021-07-13T01:10:04","slug":"does-objective-structured-clinical-examination-examiners-backgrounds-influence-the-score-agreement","status":"publish","type":"issues","link":"https:\/\/medicine.nus.edu.sg\/taps\/issues\/does-objective-structured-clinical-examination-examiners-backgrounds-influence-the-score-agreement\/","title":{"rendered":"Does objective structured clinical examination examiners\u2019 backgrounds influence the score agreement?"},"content":{"rendered":"<p>Submitted: 28 July 2020<br \/>\r\nAccepted: 18 November 2020<br \/>\r\nPublished online: <span lang=\"EN-GB\">4 May, TAPS 2021, <\/span><span lang=\"EN-GB\">6(2), 48-56<\/span><br \/>\r\n<a href=\"https:\/\/doi.org\/10.29060\/TAPS.2021-6-2\/OA2367\">https:\/\/doi.org\/10.29060\/TAPS.2021-6-2\/OA2367<\/a><\/p>\r\n<p style=\"text-align: justify\">Oscar Gilang Purnajati<sup><span lang=\"EN-GB\">1<\/span><\/sup><span lang=\"EN-GB\">, <\/span>Rachmadya Nur Hidayah<sup><span lang=\"EN-GB\">2<\/span><\/sup><span lang=\"EN-GB\"> &amp; <\/span>Gandes Retno Rahayu<sup><span lang=\"EN-GB\">2<\/span><\/sup><span lang=\"EN-GB\"> <\/span><\/p>\r\n<p style=\"text-align: justify\"><i><sup><span lang=\"EN-GB\">1<\/span><\/sup><\/i><i><span lang=\"IN\">Faculty of Medicine, <\/span><\/i><i><span lang=\"EN-ID\">Universitas Kristen Duta Wacana<\/span><\/i><i><span lang=\"IN\">, Yogyakarta, I<\/span><\/i><i><span lang=\"EN-ID\">ndonesia; <\/span><\/i><i><sup><span lang=\"EN-GB\">2<\/span><\/sup><\/i><i><span lang=\"EN-ID\">Department of Medical Education, Faculty of Medicine, Universitas Gadjah Mada, Yogyakarta, Indonesia<\/span><\/i><\/p>\r\n<p style=\"text-align: center\"><b><span lang=\"EN-GB\">Abstract<\/span><\/b><\/p>\r\n<p style=\"text-align: justify\"><b><span lang=\"EN-SG\">Introduction:<\/span><\/b><span lang=\"EN-SG\"> Objective Structured Clinical Examination (OSCE) examiners come from various backgrounds. This background variability may affect the way they score examinees. This study aimed to understand the effect of background variability influencing the examiners\u2019 score agreement in OSCE\u2019s procedural skill.<\/span><\/p>\r\n<p style=\"text-align: justify\"><b><span lang=\"EN-SG\">Methods:<\/span><\/b><span lang=\"EN-SG\"> A mixed-methods study was conducted with explanatory sequential design. OSCE examiners (n=64) <\/span><span lang=\"EN-SG\">in the Faculty of Medicine Universitas Kristen Duta Wacana (FoM-UKDW) took part to assess two videos of Cardio-Pulmonary Resuscitation (CPR) competence to get their level of agreement by using Fleiss Kappa. One video portrayed CPR according to performance guideline, and the other portrayed CPR not according to performance guidelines. Primary survey, CPR procedure, and professional behaviour were assessed. To confirm the assessment results qualitatively, in-depth interviews were also conducted. <\/span><\/p>\r\n<p style=\"text-align: justify\"><b><span lang=\"EN-SG\">Results:<\/span><\/b><span lang=\"EN-SG\"> Fifty-one examiners (79.7%) completed the assessment forms. From 18 background categories, there was a good agreement (&gt;60%) in: Primary survey (4 groups), CPR procedure (15 groups), and professional behaviour (7 groups). In-depth interviews revealed several personal factors involved in scoring decisions: 1) Examiners use different references in assessing the skills; 2) Examiners use different ways in weighting competence; 3) The first impression might affect the examiners\u2019 decision; and 4) Clinical practice experience drives examiners to establish a personal standard.<\/span><\/p>\r\n<p style=\"text-align: justify\"><b><span lang=\"EN-SG\">Conclusion:<\/span><\/b><span lang=\"EN-SG\"> This study identifies several factors of examiner background that allow better agreement of procedural section (CPR procedure) with specific assessment guidelines. <\/span><span lang=\"EN-ID\">We should address personal factors affecting scoring decisions found in this study in preparing faculty members as OSCE examiners.<\/span><span lang=\"EN-AU\"><\/span><\/p>\r\n<p style=\"text-align: justify\"><b><span lang=\"EN-GB\">Keywords<\/span><\/b><b><span lang=\"EN-GB\">:<\/span><\/b><span lang=\"EN-GB\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><i><span lang=\"EN-ID\">OSCE Score, Background Variability, Agreement, Personal Factor<\/span><\/i><span lang=\"EN-GB\"><\/span><\/p>\r\n<p align=\"center\"><strong><span lang=\"EN-GB\">Practice Highlights<\/span><\/strong><\/p>\r\n<ul>\r\n\t<li style=\"text-align: justify\"><span lang=\"EN-AU\"> <\/span><span lang=\"EN-ID\">The examiners&#8217; background variability influences the OSCE scoring agreement results.<\/span><span lang=\"EN-AU\"><\/span><\/li>\r\n\t<li style=\"text-align: justify\"><span lang=\"EN-AU\"> <\/span><span lang=\"EN-ID\">The reason for assessment inaccuracy remains unclear regarding the score agreement.<\/span><span lang=\"EN-AU\"><\/span><\/li>\r\n\t<li style=\"text-align: justify\"><span lang=\"EN-ID\"> <\/span><span lang=\"EN-ID\">The absence of assessment instruments that could provide a loophole for examiners to improvise. <\/span><\/li>\r\n\t<li style=\"text-align: justify\"><span lang=\"EN-ID\">Personal factors affecting scoring decisions found in this study should be addressed in preparing OSCE examiners.<\/span><\/li>\r\n<\/ul>\r\n<p style=\"text-align: center\"><span lang=\"EN-GB\"><\/span><strong><span lang=\"EN-GB\">I. INTRODUCTION<\/span><\/strong><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-ID\">To assess medical students\u2019 competencies in a variety of skills, most medical schools in Indonesia implement the Objective Structured Clinical Examination (OSCE) both as a clinical skills examination at the undergraduate stage and as a national exit exam <\/span><span lang=\"EN-ID\">(Rahayu et al., 2016; Suhoyo et al., 2016)<\/span><span lang=\"EN-ID\">. Most OSCE stations test both communication domains and specific clinical skills that will be assessed based on rubrics and scoring checklists which relies on examiners\u2019 observations <\/span><span lang=\"EN-ID\">(Setyonugroho et al., 2015)<\/span><span lang=\"EN-ID\">.\u00a0 The OSCE has a challenge in its complexity to standardise \u00a0the scores, which are very depend on OSCE examiners\u2019 perceptions <\/span><span lang=\"EN-ID\">(Pell et al., 2010)<\/span><span lang=\"EN-ID\">. In a well-designed OSCE<\/span><span lang=\"EN-ID\"> <\/span><span lang=\"EN-ID\">the examinees performance should only influence the examinees\u2019 score, with minimal effects from other sources of variance <\/span><span lang=\"EN-ID\">(Khan et al., 2013)<\/span><span lang=\"EN-ID\">. Research showed that there are influences of examiner\u2019s background variability on OSCE results although they have been asked to standardise their behaviour <\/span><span lang=\"EN-ID\">(Pell et al., 2010)<\/span><span lang=\"EN-GB\"> <\/span>The decision and behaviour of OSCE examiners will affect <span lang=\"EN-ID\">the<\/span> quality of assessment, including <span lang=\"EN-ID\">making <\/span>a pass or fail decision, considering the complexity<span lang=\"EN-ID\"> of<\/span> knowledge, skill, and attitude in medical education (Colbert-Getz et al., 2017; Fuller et al., 2017)<span lang=\"EN-GB\"><\/span><span lang=\"EN-ID\">. <\/span><span lang=\"EN-ID\">\u00a0<\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-ID\">E<\/span>xaminers<span lang=\"EN-ID\">\u2019<\/span> observation<span lang=\"EN-ID\">s<\/span> also rel<span lang=\"EN-ID\">y<\/span> on<span lang=\"EN-ID\"> their<\/span> clinical practice experience, OSCE examining experience, and gender conformity (Mortsiefer et al., 2017)<span lang=\"EN-ID\">. Even in OSCE that is held in the most standard conditions, the examiner factor has the biggest role in scoring inaccurately <\/span><span lang=\"EN-ID\">(Mortsiefer et al., 2017)<\/span><span lang=\"EN-ID\">. However, the reason for this inaccuracy remains unclear since there are concerns regarding the scoring agreement of examiners in OSCE and how the result might be affected by this issue. There is a need to consider the influence of examiners\u2019 background variability (gender, educational level, clinical practice experiences, length of clinical practice experiences, OSCE experience, and OSCE training experience) when preparing teachers as OSCE examiners. This study aimed to understand background variability as a factor influencing examiners\u2019 scoring agreement in assessing students\u2019 performance in procedural skill, as the first step of faculty development program to ensure the standard quality for examiners.<\/span><\/p>\r\n<p style=\"text-align: center\"><span lang=\"EN-GB\"><\/span><strong><span lang=\"EN-GB\">II. METHODS<\/span><\/strong><\/p>\r\n<p style=\"text-align: justify\"><i><span lang=\"EN-GB\"><\/span><\/i><i><span lang=\"EN-ID\">A. Study Design<\/span><\/i><span lang=\"EN-ID\"><\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-ID\">This mixed-method study used a sequential explanatory design. This mixed-method approach is expected to provide more comprehensive results and better understanding than using a separated method <\/span><span lang=\"EN-ID\">(Creswell &amp; Clark, 2018)<\/span><span lang=\"EN-ID\">. <\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-ID\">This study comprised of 2 sequential phases of data collection and analysis (QUANTITATIVE: qualitative) using sequential design. First, <\/span><span lang=\"IN\">quantitative<\/span><span lang=\"EN-ID\"> data were collected as a cross-sectional study of the examiners\u2019 strength of agreement using Fleiss Kappa while assessing the clinical skill performance recorded in the 2 videos: one video portrayed CPR <\/span>according to<span lang=\"EN-ID\"> performance guideline and the other portrayed <\/span>CPR not according to performance <span lang=\"EN-ID\">guideline. We used these 2 videos in order to portray more comprehensively how the consistency of OSCE examiner agreement both on good and poor clinical skill performance.<\/span><\/p>\r\n<p><img loading=\"lazy\" decoding=\"async\" width=\"803\" height=\"385\" src=\"https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/03\/OA2367-Figure-1.jpg\" alt=\"\" class=\"alignnone size-full wp-image-70468\" srcset=\"https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/03\/OA2367-Figure-1.jpg 803w, https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/03\/OA2367-Figure-1-300x144.jpg 300w, https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/03\/OA2367-Figure-1-768x368.jpg 768w\" sizes=\"auto, (max-width: 803px) 100vw, 803px\" \/><\/p>\r\n<div>\r\n<p align=\"center\"><span lang=\"IN\">Figure <\/span><span lang=\"IN\">1<\/span><span lang=\"EN-ID\"> Mixed method explanatory design<\/span><span lang=\"EN-ID\"><\/span><\/p>\r\n<\/div>\r\n<p style=\"text-align: justify\"><span lang=\"EN-ID\">In the second phase, in-depth interviews were used to <\/span>complement the quantita<span lang=\"EN-ID\">t<\/span>ive results to <span lang=\"EN-ID\">gain more information and a detailed confirmation about how the scores were decided <\/span><span lang=\"EN-ID\">(Stalmeijer et al., 2014)<\/span>. <span lang=\"EN-ID\">In this stage of study, researchers explored and explained the examiners\u2019 OSCE experiences and behaviour when they give a score on a clinical skill examination and the influences on their scoring regarding their backgrounds. <\/span><\/p>\r\n<p style=\"text-align: justify\"><i><span lang=\"EN-ID\">B. Materials and\/or Subjects<\/span><\/i><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-ID\">The strength of agreement of the videos\u2019 score came from 64 OSCE examiners FoM UKDW. Mortsiefer et al., (2017), explained that more subjects are better when investigate examiner characteristics associated with inter-examiner reliability <\/span><span lang=\"EN-ID\">(Mortsiefer et al., 2017)<\/span><span lang=\"EN-ID\">.\u00a0 In the second phase, in-depth interviews were conducted with 6 examiners of FoM UKDW, selected by purposive sampling regarding their scores and how they represented their own unique background (Table 1). <\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-ID\">Researcher (OGP) provided all the participants with written information about this research and addressed ethical issues in an informed consent form. Researcher ensured participants understand the research protocol and clarified any questions regarding this study. Participants who agreed to take part, sign the informed consent form prior to the data collection.\u00a0 <\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-ID\">We held interviews in FoM UKDW with maximum 30 minutes of duration each interview. The inclusion criteria for examiners who were selected for this study were involved as full-time faculty members, had over 4 times OSCE examination experience, and had done OSCE examiner training, expecting that they had enough interaction with other faculty members and had influences from medical doctor education <\/span><span lang=\"EN-ID\">(Park et al., 2015)<\/span><span lang=\"EN-ID\">. The exclusion criteria were participant did not answer the research invitation and did not fill the assessment form completely. Main researcher (OGP) conducted the interview. Main researcher was a male, student of Master of Health Profession Education Universitas Gadjah Mada, and the staff of FoM UKDW.<\/span><\/p>\r\n<p style=\"text-align: justify\"><i><span lang=\"EN-ID\">C. Statistics<\/span><\/i><\/p>\r\n<p style=\"text-align: justify\"><i><span lang=\"EN-ID\">1)Quantitative data analysis: <\/span><\/i><span lang=\"EN-ID\">We grouped examiners into 18 groups based on their background which were gender, educational level, clinical practice experiences, length of clinical practice experiences, OSCE experience, and OSCE training experience as shown in Table 1. We analysed all gathered data using IBM SPSS Statistics 25 and Microsoft Office Excel 365 (IBM Corp., Chicago). We presented quantitative data as a strength of agreement in percentage. The strength of agreement was calculated using Fleiss Kappa to determine the agreement between each group of each examiner background on whether CPR performances (primary survey, CPR Procedure, and professional behaviour), that portrayed in those 2 videos, were exhibiting score either \u201c0\u201d, \u201c1\u201d, \u201c2\u201d, or \u201c3\u201d based on the assessment guideline and rubric\u2019s criteria <\/span><span lang=\"EN-ID\">(Purnajati, 2020)<\/span><span lang=\"EN-ID\">. Based on recent research, <\/span>agreement<span lang=\"EN-ID\"> above 60% was considered as a substantial and adequate agreement <\/span><span lang=\"EN-ID\">(Stoyan et al., 2017; Vanbelle, 2019)<\/span><span lang=\"EN-ID\">.<\/span><\/p>\r\n<p style=\"text-align: justify\"><i><span lang=\"EN-ID\">2) Qualitative data analysis: <\/span><\/i><span lang=\"EN-ID\">In-depth interviews were analysed using thematic analysis. We prepared a structured list of questions. It consisted of one key question: What was your experience in scoring the OSCE? The other additional questions evaluated the experiences of examiners in OSCE scoring including: the use of other references, differences in assessment weighting, use of own decision, clinical practice experience affecting the decision, and gender related decision making. Next, the collected data resulting from in-depth interviews were recorded using audio file recorder, read, and categorised into themes whenever they were related. The transcripts and identified themes were then given to an external coder in this study. This step was followed by our agreement for each theme. There was no repeated interview.<b><i><\/i><\/b><\/span><\/p>\r\n<p style=\"text-align: center\"><span lang=\"EN-GB\"><\/span><strong><span lang=\"EN-GB\">III. RESULTS<\/span><\/strong><\/p>\r\n<p style=\"text-align: justify\"><i><span lang=\"EN-GB\"><\/span><\/i><i>A. Quantitative Data Result<\/i><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-ID\">We deposited both quantitative and qualitative data in an online repository \u00a0<\/span><span lang=\"EN-ID\">(Purnajati, 2020)<\/span><span lang=\"EN-ID\">.\u00a0 The study participants in this quantitative phase were 64 OSCE examiners who are full-time faculty members. Twelve participants were excluded because did not fulfil the inclusion criteria. Fifty-one (79.7%) examiners who returned the completed assessment form are described below in Table 1.<\/span><\/p>\r\n<div>\r\n<table border=\"1\" cellspacing=\"0\" cellpadding=\"0\" width=\"0\">\r\n<tbody>\r\n<tr>\r\n<td width=\"744\" colspan=\"4\">\r\n<p><b><span lang=\"EN-ID\">Quantitative Phase Participant<\/span><\/b><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td width=\"256\">\r\n<p><b><span lang=\"EN-ID\">Background<\/span><\/b><b><span lang=\"IN\"><\/span><\/b><\/p>\r\n<\/td>\r\n<td width=\"269\" colspan=\"2\">\r\n<p><span lang=\"EN-ID\">Groups<\/span><\/p>\r\n<\/td>\r\n<td width=\"219\" valign=\"top\">\r\n<p><span lang=\"EN-ID\">Number of Participant (<i>N=51<\/i>)<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td width=\"266\" colspan=\"2\" rowspan=\"2\" valign=\"top\">\r\n<p><b><span lang=\"EN-ID\">Gender<\/span><\/b><\/p>\r\n<\/td>\r\n<td width=\"259\" valign=\"top\">\r\n<p><span lang=\"EN-ID\">Male<\/span><\/p>\r\n<\/td>\r\n<td width=\"219\" valign=\"top\">\r\n<p><span lang=\"EN-ID\">22 (43%)<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td width=\"259\" valign=\"top\">\r\n<p><span lang=\"EN-ID\">Female<\/span><\/p>\r\n<\/td>\r\n<td width=\"219\" valign=\"top\">\r\n<p><span lang=\"EN-ID\">29 (57%)<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td width=\"266\" colspan=\"2\" rowspan=\"4\" valign=\"top\">\r\n<p><b><span lang=\"EN-ID\">Education<\/span><\/b><\/p>\r\n<\/td>\r\n<td width=\"259\" valign=\"top\">\r\n<p><span lang=\"EN-ID\">Bachelor undergraduate<\/span><\/p>\r\n<\/td>\r\n<td width=\"219\" valign=\"top\">\r\n<p><span lang=\"EN-ID\">19 (37%)<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td width=\"259\" valign=\"top\">\r\n<p><span lang=\"EN-ID\">Master\u2019s degree<\/span><\/p>\r\n<\/td>\r\n<td width=\"219\" valign=\"top\">\r\n<p><span lang=\"EN-ID\">16 (31%)<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td width=\"259\" valign=\"top\">\r\n<p><span lang=\"EN-ID\">Doctoral degree<\/span><\/p>\r\n<\/td>\r\n<td width=\"219\" valign=\"top\">\r\n<p><span lang=\"EN-ID\">3 (6%)<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td width=\"259\" valign=\"top\">\r\n<p><span lang=\"EN-ID\">Specialist doctor<\/span><\/p>\r\n<\/td>\r\n<td width=\"219\" valign=\"top\">\r\n<p><span lang=\"EN-ID\">13 (25%)<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td width=\"266\" colspan=\"2\" rowspan=\"3\" valign=\"top\">\r\n<p><b><span lang=\"EN-ID\">Clinical Practice Experience<\/span><\/b><\/p>\r\n<\/td>\r\n<td width=\"259\" valign=\"top\">\r\n<p><span lang=\"EN-ID\">General practitioner<\/span><\/p>\r\n<\/td>\r\n<td width=\"219\" valign=\"top\">\r\n<p><span lang=\"EN-ID\">28 (55%)<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td width=\"259\" valign=\"top\">\r\n<p><span lang=\"EN-ID\">Specialist<\/span><\/p>\r\n<\/td>\r\n<td width=\"219\" valign=\"top\">\r\n<p><span lang=\"EN-ID\">14 (27%)<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td width=\"259\" valign=\"top\">\r\n<p><span lang=\"EN-ID\">No clinical practice<\/span><\/p>\r\n<\/td>\r\n<td width=\"219\" valign=\"top\">\r\n<p><span lang=\"EN-ID\">9 (18%)<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td width=\"266\" colspan=\"2\" rowspan=\"3\" valign=\"top\">\r\n<p><b><span lang=\"EN-ID\">Duration of clinical practice experience <\/span><\/b><\/p>\r\n<\/td>\r\n<td width=\"259\" valign=\"top\">\r\n<p><span lang=\"EN-ID\">&lt; 2 years<\/span><\/p>\r\n<\/td>\r\n<td width=\"219\" valign=\"top\">\r\n<p><span lang=\"EN-ID\">9 (18%)<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td width=\"259\" valign=\"top\">\r\n<p><span lang=\"EN-ID\">2-5 years<\/span><\/p>\r\n<\/td>\r\n<td width=\"219\" valign=\"top\">\r\n<p><span lang=\"EN-ID\">17 (33%)<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td width=\"259\" valign=\"top\">\r\n<p><span lang=\"EN-ID\">&gt;5 years<\/span><\/p>\r\n<\/td>\r\n<td width=\"219\" valign=\"top\">\r\n<p><span lang=\"EN-ID\">25 (49%)<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td width=\"266\" colspan=\"2\" rowspan=\"3\" valign=\"top\">\r\n<p><b><span lang=\"EN-ID\">OSCE experience<\/span><\/b><\/p>\r\n<\/td>\r\n<td width=\"259\" valign=\"top\">\r\n<p><span lang=\"EN-ID\">&lt; 2 years<\/span><\/p>\r\n<\/td>\r\n<td width=\"219\" valign=\"top\">\r\n<p><span lang=\"EN-ID\">9 (18%)<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td width=\"259\" valign=\"top\">\r\n<p><span lang=\"EN-ID\">2-5 years<\/span><\/p>\r\n<\/td>\r\n<td width=\"219\" valign=\"top\">\r\n<p><span lang=\"EN-ID\">24 (47%)<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td width=\"259\" valign=\"top\">\r\n<p><span lang=\"EN-ID\">&gt;5 years<\/span><\/p>\r\n<\/td>\r\n<td width=\"219\" valign=\"top\">\r\n<p><span lang=\"EN-ID\">18 (35%)<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td width=\"266\" colspan=\"2\" rowspan=\"3\" valign=\"top\">\r\n<p><b><span lang=\"EN-ID\">OSCE examiner training<\/span><\/b><\/p>\r\n<\/td>\r\n<td width=\"259\" valign=\"top\">\r\n<p><span lang=\"EN-ID\">&lt; 3 times<\/span><\/p>\r\n<\/td>\r\n<td width=\"219\" valign=\"top\">\r\n<p><span lang=\"EN-ID\">21 (41%)<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td width=\"259\" valign=\"top\">\r\n<p><span lang=\"EN-ID\">3-5 times<\/span><\/p>\r\n<\/td>\r\n<td width=\"219\" valign=\"top\">\r\n<p><span lang=\"EN-ID\">17 (33%)<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td width=\"259\" valign=\"top\">\r\n<p><span lang=\"EN-ID\">&gt;5 times<\/span><\/p>\r\n<\/td>\r\n<td width=\"219\" valign=\"top\">\r\n<p><span lang=\"EN-ID\">13 (25%)<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td width=\"744\" colspan=\"4\" valign=\"top\">\r\n<p><b><span lang=\"EN-ID\">Qualitative Phase Participants. <\/span><\/b><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td width=\"744\" colspan=\"4\" valign=\"top\">\r\n<p><sup><span lang=\"EN-ID\">a <\/span><\/sup><span lang=\"EN-ID\">Video portrayed CPR according to performance guideline. <sup>b <\/sup>Video portrayed CPR not according to performance guidelines<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td width=\"256\"><\/td>\r\n<td width=\"10\"><\/td>\r\n<td width=\"259\"><\/td>\r\n<td width=\"219\"><\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<p align=\"center\"><span lang=\"IN\">Table <\/span><span lang=\"IN\">1<\/span><span lang=\"EN-ID\">. Descriptive characteristics of participants<\/span><span lang=\"EN-ID\">\u00a0<\/span><\/p>\r\n<\/div>\r\n<p style=\"text-align: justify\"><span lang=\"EN-ID\">The assessment rubric was divided into three main competencies: (1) primary survey, (2) CPR procedure, and (3) professional behaviour. The results showed overall agreement on each main competency based on each examiners\u2019 background variability by using Fleiss Kappa. The percentage of agreement is shown in Figure 2, 3, and 4.<\/span><\/p>\r\n<p><img loading=\"lazy\" decoding=\"async\" width=\"544\" height=\"296\" src=\"https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/03\/OA2367-Figure-2.jpg\" alt=\"\" class=\"size-full wp-image-70469 aligncenter\" srcset=\"https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/03\/OA2367-Figure-2.jpg 544w, https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/03\/OA2367-Figure-2-300x163.jpg 300w\" sizes=\"auto, (max-width: 544px) 100vw, 544px\" \/><\/p>\r\n<p align=\"center\"><span lang=\"EN-GB\">Figure 2. Primary Survey percentage of overall agreement (n = 51). Agreement above 60% (*) is considered as a<\/span><span lang=\"EN-ID\"> substantial and adequate agreement<\/span><span lang=\"EN-GB\"><\/span><\/p>\r\n<p align=\"center\"><img loading=\"lazy\" decoding=\"async\" width=\"547\" height=\"257\" src=\"https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/03\/OA2367-Figure-3.jpg\" alt=\"\" class=\"alignnone size-full wp-image-70470\" srcset=\"https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/03\/OA2367-Figure-3.jpg 547w, https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/03\/OA2367-Figure-3-300x141.jpg 300w\" sizes=\"auto, (max-width: 547px) 100vw, 547px\" \/><\/p>\r\n<p align=\"center\"><span lang=\"IN\">Figure 3<\/span><span lang=\"EN-ID\">. CPR Procedure percentage of overall agreement (n=51). Agreement above 60% (*) is considered as a substantial and adequate agreement<\/span><\/p>\r\n<p align=\"center\"><img loading=\"lazy\" decoding=\"async\" width=\"553\" height=\"273\" src=\"https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/03\/OA2367-Figure-4.jpg\" alt=\"\" class=\"alignnone size-full wp-image-70471\" srcset=\"https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/03\/OA2367-Figure-4.jpg 553w, https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/03\/OA2367-Figure-4-300x148.jpg 300w\" sizes=\"auto, (max-width: 553px) 100vw, 553px\" \/><\/p>\r\n<div>\r\n<p align=\"center\"><span lang=\"EN-GB\">Figure 4. Professional Behavio<\/span><span lang=\"IN\">u<\/span><span lang=\"EN-GB\">r percentage of overall agreement (n=51). Agreement above 60% (*) is considered as a substantial<\/span><\/p>\r\n<\/div>\r\n<p style=\"text-align: justify\"><span lang=\"EN\">After completing the CPR competency assessment, all examiners\u2019 background characteristics met a cutoff of approval above 60% in assessing CPR procedure except for examiners with clinical practice experience &lt;3 years, OSCE testing experience &lt;2 years, and OSCE examiner training&gt; 5 years (Figure 3). <\/span><span lang=\"EN-ID\">This finding showed a good strength of agreement in assessing CPR procedure <\/span><span lang=\"IN\">regardless of <\/span><span lang=\"EN-ID\">examiners\u2019<\/span><span lang=\"IN\"> background. <\/span><span lang=\"EN\">However, there were many instances where the cut-off point of 60% was not achieved in the aspects of primary surveys and professional behaviour (Figure 2 and 4), which showed <\/span><span lang=\"IN\">fair strength of agreement between examiners<\/span><span lang=\"EN-ID\"> when they examined these competencies.<\/span><\/p>\r\n<p><i>B. Qualitative Data Results<\/i><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-ID\">Two theme categories were determined: (1) OSCE experience and (2) specific behaviour in OSCE. The first theme contains of 3 sub-themes: (1) student performance, (2) examiner background effect, and (3) using assessment instrument. The second theme consists of 5 sub-themes: (1) use of assessment references, (2) score weighting, (3) personal inferences, (4) clinical experience, and (5) gender conformity. <\/span><\/p>\r\n<p style=\"text-align: justify\"><i><span lang=\"EN-ID\">Theme 1:<\/span><\/i><span lang=\"EN-ID\"> Examiners argued that they understand the difference in student performance in performing clinical skills and can distinguish from the coherent skills performed by students according to checklist. <\/span><\/p>\r\n<p style=\"text-align: justify\"><i><span lang=\"EN-ID\">\u201cVery easy in giving an assessment, because everything is in accordance with the assessment rubric&#8221;<\/span><\/i><span lang=\"EN-ID\"> <\/span><\/p>\r\n<p align=\"right\" style=\"text-align: right\"><span lang=\"EN-ID\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <i>\u00a0\u00a0\u00a0\u00a0\u00a0 (ID 35)<\/i><\/span><\/p>\r\n<p style=\"text-align: justify\"><i><span lang=\"EN-ID\">&#8220;The plot is clear, well organised&#8221;<\/span><\/i><span lang=\"EN-ID\"> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><\/p>\r\n<p align=\"right\" style=\"text-align: right\"><span lang=\"EN-ID\">\u00a0\u00a0\u00a0\u00a0\u00a0 <i>(ID 26)<\/i><\/span><\/p>\r\n<p style=\"text-align: justify\"><i><span lang=\"EN-ID\">&#8220;You can compare the inadequacies; it is enough to be compared&#8221;<\/span><\/i><span lang=\"EN-ID\"> <\/span><\/p>\r\n<p align=\"right\" style=\"text-align: right\"><span lang=\"EN-ID\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \u00a0\u00a0\u00a0\u00a0\u00a0 <i>(ID 11)<\/i><\/span><\/p>\r\n<p style=\"text-align: justify\"><i><span lang=\"EN-ID\">&#8220;The 2 different students are quite striking, so in my opinion it is not too difficult&#8221;<\/span><\/i><span lang=\"EN-ID\"> <\/span><\/p>\r\n<p align=\"right\" style=\"text-align: right\"><span lang=\"EN-ID\">\u00a0\u00a0\u00a0\u00a0 <i>\u00a0(ID 28)<\/i><\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-ID\">Nevertheless, some examiners had difficulty to distinguish student performance when only used a checklist. Examiner background did not affect their way in scoring clinical skills performance, but some background may have the potential to affect their scoring, such as clinical practice experience. <\/span><\/p>\r\n<p style=\"text-align: justify\"><i><span lang=\"EN\">\u201cI am trying to avoid personal interpretations, as much as possible, but of course that cannot be 100 percent. In my opinion, the assessment rubric still gives room for subjectivity&#8221; <\/span><\/i><\/p>\r\n<p align=\"right\" style=\"text-align: right\"><i><span lang=\"EN\">\u00a0 \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \u00a0\u00a0\u00a0\u00a0\u00a0(ID 28)<\/span><\/i><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-ID\">In this research, it seemed easy for examiners to understand the assessment instrument when giving score to those 2 videos and their understanding were good. <\/span><\/p>\r\n<p style=\"text-align: justify\"><i><span lang=\"EN-ID\">Theme 2: Interviews revealed that:<\/span><\/i><span lang=\"EN-ID\"> 1) Examiners use <\/span>other references such <span lang=\"EN-ID\">as their<\/span> clinical experience<span lang=\"EN-ID\"> in assessing the skills; <\/span><\/p>\r\n<p style=\"text-align: justify\"><i><span lang=\"EN\">\u201cIf the assessment guideline is unclear, the students are also unclear, yes I will improvise. Or when the assessment guideline is clear and the students are unclear which criteria are included, yes I will <\/span><\/i><i><span lang=\"EN-SG\">improvise<\/span><\/i><i><span lang=\"EN\">\u201d <\/span><\/i><\/p>\r\n<p align=\"right\" style=\"text-align: right\"><i><span lang=\"EN\">\u00a0\u00a0\u00a0\u00a0 (ID 35)<\/span><\/i><\/p>\r\n<p style=\"text-align: justify\"><i><span lang=\"EN-ID\">&#8220;Maybe yes, because once again the template at the beginning is not very clear&#8221;<\/span><\/i><\/p>\r\n<p align=\"right\" style=\"text-align: right\"><i><span lang=\"EN-ID\">\u00a0(ID 23)<\/span><\/i><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-ID\">2) Examiners use <\/span>different ways in <span lang=\"EN-ID\">giving weight of competence, for example, procedural steps are considered more important than primary survey; <\/span><\/p>\r\n<p style=\"text-align: justify\"><i><span lang=\"EN-ID\">\u201cFor those that I feel have a small weight because the instructions are also short, so I don&#8217;t have to look carefully\u201d<\/span><\/i><\/p>\r\n<p align=\"right\" style=\"text-align: right\"><span lang=\"EN-ID\">\u00a0(ID 24)<\/span><\/p>\r\n<p style=\"text-align: justify\"><i><span lang=\"EN-ID\">&#8220;When I feel that competence is not important, it does not get my emphasis, the more emergency that will get more attention.&#8221;<\/span><\/i><\/p>\r\n<p align=\"right\" style=\"text-align: right\"><span lang=\"EN-ID\">\u00a0<i>(ID 28)<\/i><\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-ID\">3) The <\/span>first impression of examinee<span lang=\"EN-ID\">s<\/span><span lang=\"EN-ID\"> <\/span><span lang=\"EN-ID\">might affect their decision in scoring their performance; <\/span><\/p>\r\n<p style=\"text-align: justify\"><i><span lang=\"EN-ID\">\u201cThat first impression will affect me in giving value. I will be more critical. I see more, pay more attention to the small things they do\u201d <\/span><\/i><\/p>\r\n<p align=\"right\" style=\"text-align: right\"><i><span lang=\"EN-ID\">(ID 24)<\/span><\/i><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-ID\">4) Clinical practice experience drives examiners to establish a personal standard on how a doctor should be; <\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-ID\">\u201c<i>Clinical experience when practice is one of the judgments&#8221; <\/i><\/span><\/p>\r\n<p align=\"right\" style=\"text-align: right\"><i><span lang=\"EN-ID\">(ID 24)<\/span><\/i><\/p>\r\n<p style=\"text-align: justify\"><i><span lang=\"EN-ID\">&#8220;The reference is just my instinct because it has been running as a doctor after all these years. Yes, I use my previous knowledge\u201d <\/span><\/i><\/p>\r\n<p align=\"right\" style=\"text-align: right\"><i><span lang=\"EN-ID\">(ID 26)<\/span><\/i><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-ID\">And 5) Gender of examinees does not affect their decision, while their professionalism (e.g. showing respect to patients) will surely affect their decision.<\/span><\/p>\r\n<p style=\"text-align: justify\"><i><span lang=\"EN-ID\">\u201cI pay more attention especially to politeness and professional behaviour\u201d<\/span><\/i><\/p>\r\n<p align=\"right\" style=\"text-align: right\"><i><span lang=\"EN-ID\">(ID 24)<\/span><\/i><\/p>\r\n<p style=\"text-align: justify\"><i><span lang=\"EN-ID\">&#8220;Students of any gender still have the same standard of evaluation, a score of professionalism which is more influential&#8221; <\/span><\/i><\/p>\r\n<p align=\"right\" style=\"text-align: right\"><i><span lang=\"EN-ID\">(ID 23)<\/span><\/i><\/p>\r\n<p style=\"text-align: center\"><strong><span lang=\"EN-GB\">\u00a0IV. <\/span><span lang=\"EN-GB\"><\/span><span lang=\"EN-GB\">DISCUSSION<\/span><\/strong><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">Examiners&#8217; agreement in this study was high in assessing the CPR procedure, which has a fixed and specific procedure in almost all groups of examiners. These results are consistent and can be explained by results from previous studies, which show that assessment with specific cases will provide high inter-examiner agreement <\/span><span lang=\"EN-GB\">(Erdogan et al., 2016)<\/span><span lang=\"EN-GB\">. The differences in the <\/span><span lang=\"EN-ID\">examiner&#8217;s background <\/span><span lang=\"EN-GB\">will not have much influence on their agreement in giving an assessment in a specific case. This was supported by the opinions of examiners in the in-depth interviews who stated that in the CPR assessment procedure, assessment instruments are clear, easy to understand, with clear procedure flow, and performance that is easily distinguished, which made it easier for examiners to be able to distinguish student performance. A specific assessment instrument that could not provide a loophole for examiners to improvise assessment, made the opportunity for examiners to portray their subjectivity was <\/span><span lang=\"EN-ID\">minimised<\/span><span lang=\"EN-GB\">. This simplicity could lead to high agreement among examiners in specific competencies as shown in this study and based on clear evidence can increase the reliability of the assessment <\/span><span lang=\"EN-GB\">(Daniels et al., 2014)<\/span><span lang=\"EN-GB\"> .<\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">In this study, it was found in the primary survey assessment and professional behaviour which has an assessment guide that is not as specific as the CPR procedure, the percentage of agreement between examiner groups was lower, with only <\/span><span lang=\"EN-ID\">a few <\/span><span lang=\"EN-GB\">of them reaching 60% of agreement. This difference happened for reasons confirmed in the in-depth interviews which raised the issue that although the examiners tried to<\/span><span lang=\"EN-GB\"> <\/span><span lang=\"EN-ID\">minimise <\/span><span lang=\"EN-GB\">their subjectivity in assessing, but it was said that there were still gaps in the assessment guide that still gives room for subjectivity. There are also examiners who were dissatisfied with the checklist, so they used their personal decisions in evaluating students.<\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">According to a recent study, this could be due to the lack of specific instructions in the general assessment guidelines which will result in lower inter-examiner reliability compared to the use of more specific assessment guidelines <\/span><span lang=\"EN-GB\">(Mortsiefer et al., 2017)<\/span><span lang=\"EN-GB\">. In the primary survey section and professional behaviour, there were also aspects of communication that were judged to be more susceptible to bias than physical examination skills because physical examination is more well-documented, clear instructions, and more widely accepted by examiners <\/span><span lang=\"EN-GB\">(Chong et al., 2018)<\/span><span lang=\"EN-GB\"> The validity and reliability of a clinical skills assessment depend on factors including how the student&#8217;s performance on the exam, the character of the population, the environment, and even the assessment instrument itself can affect how examiners carry out the assessment <\/span><span lang=\"EN-GB\">(Brink &amp; Louw, 2012)<\/span><span lang=\"EN-GB\">. These phenomena were seen in the in-depth interviews which revealed that there were certain moments namely when the student being tested does not match the expectations written in the assessment guide and when the assessment guide is not clear so that it still gives room for subjectivity examiner. In addition, in the in-depth interviews the results also revealed that the examiners differentiated their attention on certain competencies with certain criteria such as the length of information in the assessment rubric, so that competencies that were considered not important did not get as much attention.<\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">This finding may be in line with previous research which stated that constructs and conceptual definitions in this category that still provide a gap in the subjectivity of examiners cause shifting attention focus and weighting of their judgments to be different so that there are differences in important aspects between examiners <\/span><span lang=\"EN-GB\">(Schierenbeck &amp; Murphy, 2018; Yeates et al., 2013)<\/span><span lang=\"EN-GB\">. The difference in these important aspects can bring examiners to <\/span><span lang=\"EN-ID\">reorganise<\/span><span lang=\"EN-GB\"> competency weights so that simpler and easier competencies (in this case those that have clearer and more detailed assessment guidelines) will be done first, and more complex ones (in this case, guides that have lower rigidity ratings) will be assessed later with the possibility of using more narratives <\/span><span lang=\"EN-GB\">(Chahine et al., 2015)<\/span><span lang=\"EN-GB\">. This <\/span><span lang=\"EN-ID\">reorganisation<\/span><span lang=\"EN-GB\"> can reflect how the examiners\u2019 decision, allowing them to direct their attention to the more important aspects as the testers revealed in in-depth interviews with this research.<\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">The personal factor, such as assessment references is a potential variability of the assessment conducted by the examiner. Examiners are trained and understand the use of assessment instruments, but produce varying assessments because they do not apply assessment criteria appropriately, but use personal best practice, use other test participants better as benchmarks, use patient outcomes (e.g. correct diagnosis, do patients understand, etc.), and use themselves as a comparison <\/span><span lang=\"EN-GB\">(Gingerich et al., 2014; Kogan et al., 2011; Yeates et al., 2013)<\/span><span lang=\"EN-GB\">.<\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">Another personal factors, including first impressions, can occur spontaneously unconsciously and can be a source of difference in judgment between examiners <\/span><span lang=\"EN-GB\">(Gingerich et al., 2011)<\/span><span lang=\"EN-GB\">. First impressions based on observers&#8217; observations have the same decisions and influences as social interactions, so it makes sense that first impressions are able to influence judgments, can be accurate and have a relationship with the final assessment results, but do not occur in examiners in general <\/span><span lang=\"EN-GB\">(Wood, 2014; Wood et al., 2017)<\/span><span lang=\"EN-GB\">. <\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">In providing assessments, there are gaps for examiners to give different competency weights to other examiners. Providing assessments based on targets that differ from competency standards and comparisons with the performance of other examinees will make the examiners recalibrate their own weighting and this is an explanation why there are variations in assessment and differences in the important points of the examinees&#8217; performance among examiners <\/span><span lang=\"EN-GB\">(Gingerich et al., 2018; Yeates et al., 2015; Yeates et al., 2013)<\/span><span lang=\"EN-GB\">.<\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">The variability of personal factors between examiners can be <\/span><span lang=\"EN-ID\">conceptualise<\/span><span lang=\"EN-GB\"> more as a different emphasis on building doctor-patient relationships and \/ or certain medical expertise rather than variations in the examiner&#8217;s background itself. The examiners\u2019 own understanding can be conceptualized as a combination of whether what the examinees do is good enough and whether what they do is enough to build a doctor-patient relationship.<\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">This research had some limitations such as it only used specific cases (i.e., CPR) <\/span><span lang=\"EN-ID\">to minimise <\/span><span lang=\"EN-GB\">the bias of the assessment instrument so that it would reveal more bias in the examiners themselves. In more complicated cases such as communication skills and clinical reasoning it is also necessary to provide a more complete picture of how the examiners\u2019 scores agree in other cases. Generalization also became a limitation in this study because it only involved examiners from one medical education institution, however the study participants sufficiently described the variability of the examiner&#8217;s background.<\/span><\/p>\r\n<p style=\"text-align: center\"><span lang=\"EN-GB\"><\/span><strong><span lang=\"EN-GB\">V. CONCLUSION<\/span><\/strong><\/p>\r\n<p align=\"center\" style=\"text-align: justify\"><span lang=\"EN-GB\">This study identifies several factors of examiner background variability that influence examiners&#8217; judgment in terms of inter-examiner agreement. Female examiners, bachelor education, less OSCE experience, and non-clinician examiners allow better agreement of procedural section (CPR procedure) with specific assessment guidelines. Cases that have unspecified assessment guidelines in this research, primary survey and professional behaviour, have lower agreement among examiners and must be examined deeper. We should note that personal factors of OSCE examiners can influence assessment discrepancies. However, the reasons for using these personal factors in scoring OSCE performance might be affected by unknown biases that require further research. <\/span><span lang=\"EN-ID\">Therefore, to improve clinical skills assessment such as OSCE for undergraduate medical programme, we must address personal factors affecting scoring decisions found in this study in preparing faculty members as OSCE examiners.<\/span><\/p>\r\n<p align=\"center\"><strong><span lang=\"EN-GB\">Notes on Contributors<\/span><\/strong><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-ID\">Oscar Gilang Purnajati, MD was student of Master of Health Professions Education Study Program, Faculty of Medicine, Universitas Gadjah Mada, Indonesia. He <\/span><span lang=\"EN-AU\">concepted the research, reviewed the literature, designed the study, acquisited funding, conducted interviews, analysed quantitative data and transcripts, and wrote the manuscript. <\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-ID\">Rachmadya Nur Hidayah, MD., M.Sc., Ph.D is lecturer of Department of Medical Education, Faculty of Medicine, Universitas Gadjah Mada, Yogyakarta, Indonesia. She supervised author Oscar Gilang Purnajati, <\/span><span lang=\"EN-AU\">developed the concepted framework for the study, critically analysed the data, cured the data, and reviewed the final manuscript. <\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-ID\">Prof. Gandes Retno Rahayu, MD., M.Med.Ed, Ph.D is professor at the Department of Medical Education, Faculty of Medicine, Universitas Gadjah Mada, Yogyakarta, Indonesia. She supervised author Oscar Gilang Purnajati, <\/span><span lang=\"EN-AU\">advised the design of the study, critically analysed the data, gave critical feedback to the conducted interviews, reviewed the final manuscript. <\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-AU\">All the authors have read and approved the final manuscript.<\/span><\/p>\r\n<p align=\"center\"><strong><span lang=\"EN-GB\">Ethical Approval<\/span><\/strong><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-ID\">This study was approved by Health Research Ethics Committee Faculty of Medicine <\/span><span lang=\"IN\">Universitas Kristen <\/span><span lang=\"EN-ID\">Duta Wacana (Reference No.1068\/C.16\/FK\/2019).<\/span><\/p>\r\n<p align=\"center\"><strong><span lang=\"EN-GB\">Data Availability<\/span><\/strong><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-AU\">All data were deposited in an online repository. The data is available at Open Science Framework with DOI: <\/span><a href=\"https:\/\/doi.org\/10.17605\/OSF.IO\/RDP65\">https:\/\/doi.org\/10.17605\/OSF.IO\/RDP65<\/a> <span><\/span><\/p>\r\n<p align=\"center\"><strong><span lang=\"EN-AU\">Acknowledgements<\/span><\/strong><\/p>\r\n<p style=\"text-align: justify\">The author would like to thank Hikmawati Nurrokhmanti, MD, M.Sc for helping with the process of coding the in-depth interview transcripts. The author also would like to thank the staffs of Faculty of Medicine, Universitas Kristen Duta Wacana for supporting the research.<\/p>\r\n<p align=\"center\"><strong><span lang=\"EN-AU\">Funding Statement<\/span><\/strong><\/p>\r\n<p style=\"text-align: justify\">This work was supported by the Universitas Kristen Duta Wacana (No. 075\/B.03\/UKDW\/2018) as a part of study scholarship<span lang=\"EN-AU\">.<\/span><\/p>\r\n<p align=\"center\"><strong><span lang=\"EN-AU\">Declaration of Interest<\/span><\/strong><\/p>\r\n<p>No potential conflict of interest relevant to this article was reported<span lang=\"EN-AU\">.<\/span><\/p>\r\n<p align=\"center\"><strong><span lang=\"EN-AU\">Abbreviations and specific symbols<\/span><\/strong><span lang=\"EN-GB\"><\/span><\/p>\r\n<p><span lang=\"EN-AU\">OSCE: Objective Structured Clinical Examination.<\/span><\/p>\r\n<p align=\"center\"><strong><span lang=\"EN-GB\">References<\/span><\/strong><\/p>\r\n<p style=\"text-align: justify\">Brink, Y., &amp; Louw, Q. A. (2012). Clinical instruments: Reliability and validity critical appraisal. <i>Journal of Evaluation in Clinical Practice<\/i>,<i>18<\/i>(6), 1126-1132. <a href=\"https:\/\/doi.org\/10.1111\/j.1365-2753.2011.01707.x%20\">https:\/\/doi.org\/10.1111\/j.1365-2753.2011.01707.x<\/a><\/p>\r\n<p style=\"text-align: justify\">Chahine, S., Holmes, B., &amp; Kowalewski, Z. (2015). In the minds of OSCE examiners: Uncovering hidden assumptions. <i>Advances in Health Sciences Education : Theory and Practice<\/i>,<i> 21<\/i>(3), 609\u2013625. <a href=\"https:\/\/doi.org\/10.1007\/s10459-015-9655-4%20\">https:\/\/doi.org\/10.1007\/s10459-015-9655-4<\/a><\/p>\r\n<p style=\"text-align: justify\">Chong, L., Taylor, S., Haywood, M., Adelstein, B.-A., &amp; Shulruf, B. (2018). Examiner seniority and experience are associated with bias when scoring communication, but not examination, skills in objective structured clinical examinations in Australia. <i>Journal of Educational Evaluation for Health Professions<\/i>,<i> 15<\/i>(17). <a href=\"https:\/\/doi.org\/10.3352\/jeehp.2018.15.17%20\">https:\/\/doi.org\/10.3352\/jeehp.2018.15.17<\/a><\/p>\r\n<p style=\"text-align: justify\">Colbert-Getz, J. M., Ryan, M., Hennessey, E., Lindeman, B., Pitts, B., Rutherford, K. A., Schwengel, D., Sozio, S. M., George, J., &amp; Jung, J. (2017). Measuring assessment quality with an assessment utility rubric for medical education. <i>MedEdPORTAL : The Journal of Teaching and Learning Resources<\/i>,<i> 13<\/i>, 1-5. <a href=\"https:\/\/doi.org\/10.15766\/mep_2374-8265.10588%20\">https:\/\/doi.org\/10.15766\/mep_2374-8265.10588<\/a><\/p>\r\n<p style=\"text-align: justify\">Creswell, J. W., &amp; Clark, V. L. P. (2018). <i>Designing and conducting mixed method research<\/i>. SAGE Publications, Inc.<\/p>\r\n<p style=\"text-align: justify\">Daniels, V. J., Bordage, G., Gierl, M. J., &amp; Yudkowsky, R. (2014). Effect of clinically discriminating, evidence-based checklist items on the reliability of scores from an internal medicine residency OSCE. <i>Advances in Health Sciences Education : Theory and Practice<\/i>,<i> 19<\/i>(4), 497-506. <a href=\"https:\/\/doi.org\/10.1007\/s10459-013-9482-4%20\">https:\/\/doi.org\/10.1007\/s10459-013-9482-4<\/a><\/p>\r\n<p style=\"text-align: justify\">Erdogan, A., Dong, Y., Chen, X., Schmickl, C., Berrios, R. A. S., Arguello, L. Y. G., Kashyap, R., Kilickaya, O., Pickering, B., Gajic, O., &amp; O\u2019Horo, J. C. (2016). Development and validation of clinical performance assessment in simulated medical emergencies: An observational study. <i>BMC Emergency Medicine<\/i>,<i> 16<\/i>, 4. <a href=\"https:\/\/doi.org\/10.1186\/s12873-015-0066-x%20\">https:\/\/doi.org\/10.1186\/s12873-015-0066-x<\/a><\/p>\r\n<p style=\"text-align: justify\">Fuller, R., Homer, M., Pell, G., &amp; Hallam, J. (2017). Managing extremes of assessor judgment within the OSCE. <i>Medical Teacher<\/i>,<i> 39<\/i>(1), 58-66. <a href=\"https:\/\/doi.org\/https:\/doi.org\/10.1080\/0142159X.2016.1230189\">https:\/\/doi.org\/ 10.1080\/0142159X.2016.1230189<\/a><\/p>\r\n<p style=\"text-align: justify\">Gingerich, A., Kogan, J., Yeates, P., Govaerts, M., &amp; Holmboe, E. (2014). Seeing the \u2018black box\u2019 differently: Assessor cognition from three research perspectives. <i>Medical Education<\/i>,<i> 48<\/i>(11), 1055\u20131068. <a href=\"https:\/\/doi.org\/https:\/doi.org\/10.1111\/medu.12546\">https:\/\/doi.org \/10.1111\/medu.12546<\/a><\/p>\r\n<p style=\"text-align: justify\">Gingerich, A., Regehr, G., &amp; Eva, K. W. (2011). Rater-based assessments as social judgments: Rethinking the etiology of rater errors. <i>Academic Medicine : Journal of the Association of American Medical Colleges<\/i>,<i> 86<\/i>(10), S1-S7. https:\/\/doi.org\/10.1097\/ACM.0b013e31822a6cf8<\/p>\r\n<p style=\"text-align: justify\">Gingerich, A., Schokking, E., &amp; Yeates, P. (2018). Comparatively salient: Examining the influence of preceding performances on assessors\u2019 focus and interpretations in written assessment comments. <i>Advances in Health Sciences Education: Theory and Practice<\/i>,<i> 23<\/i>(5), 937-959. <a href=\"https:\/\/doi.org\/https:\/doi.org\/10.1007\/s10459-018-9841-2\">https:\/\/doi.org \/10.1007\/s10459-018-9841-2<\/a><\/p>\r\n<p style=\"text-align: justify\">Khan, K. Z., Ramachandran, S., Gaunt, K., &amp; Pushkar, P. (2013). The Objective Structured Clinical Examination (OSCE): AMEE Guide No. 81. Part I: An historical and theoretical perspective. <i>Medical Teacher<\/i>,<i> 35<\/i>(9), 1437-1446. <a href=\"https:\/\/doi.org\/https:\/doi.org\/10.3109\/0142159X.2013.818634\">https:\/\/doi.org\/ 10.3109\/0142159X.2013.818634<\/a><\/p>\r\n<p style=\"text-align: justify\">Kogan, J. R., Conforti, L., Bernabeo, E., Iobst, W., &amp; Holmboe, E. (2011). Opening the black box of clinical skills assessment via observation: A conceptual model. <i>Medical Education<\/i>,<i> 45<\/i>(10), 1048-1060.<a href=\"\/doi.org\/10.1111\/j.1365-2923.2011.04025.x%20\"> https:\/\/doi.org\/10.1111\/j.1365-2923.2011.04025.x<\/a><\/p>\r\n<p style=\"text-align: justify\">Mortsiefer, A., Karger, A., Rotthoff, T., Raski, B., &amp; Pentzek, M. (2017). Examiner characteristics and interrater reliability in a communication OSCE. <i>Patient Education and Counseling<\/i>,<i> 100<\/i>(6), 1230-1234. <a href=\"https:\/\/doi.org\/https:\/doi.org\/10.1016\/j.pec.2017.01.013\">https:\/\/doi.org\/ 10.1016\/j.pec.2017.01.013<\/a><\/p>\r\n<p style=\"text-align: justify\">Park, S. E., Kim, A., Kristiansen, J., &amp; Karimbux, N. Y. (2015). The Influence of Examiner Type on Dental Students\u2019 OSCE Scores. <i>Journal of Dental Education<\/i>,<i> 79<\/i>(1), 89-94.<\/p>\r\n<p style=\"text-align: justify\">Pell, G., Fuller, R., Homer, M., &amp; Roberts, T. (2010). How to measure the quality of the OSCE: A review of metrics \u2013 AMEE guide no. 49. <i>Medical Teacher<\/i>,<i> 32<\/i>(10), 802-811. <a href=\"https:\/\/doi.org\/https:\/doi.org\/10.3109\/0142159X.2010.507716\">https:\/\/doi.org\/ 10.3109\/0142159X.2010.507716<\/a><\/p>\r\n<p style=\"text-align: justify\">Purnajati, O. G. (2020). <i>Does objective structured clinical examination examiners\u2019 backgrounds influence the score agreement? [Data set]<\/i>. Open Science Framework. <a href=\"https:\/\/doi.org\/https:\/doi.org\/10.17605\/OSF.IO\/RDP65\">https:\/\/doi.org\/ 10.17605\/OSF.IO\/RDP65<\/a><\/p>\r\n<p style=\"text-align: justify\">Rahayu, G. R., Suhoyo, Y., Nurhidayah, R., Hasdianda, M. A., Dewi, S. P., Chaniago, Y., Wikaningrum, R., Hariyanto, T., Wonodirekso, S., &amp; Achmad, T. (2016). Large-scale multi-site OSCEs for national competency examination of medical doctors in Indonesia. <i>Medical Teacher<\/i>,<i> 38<\/i>(8), 801-807. <a href=\"https:\/\/doi.org\/https:\/doi.org\/10.3109\/0142159X.2015.1078890\">https:\/\/doi.org \/10.3109\/0142159X.2015.1078890<\/a><\/p>\r\n<p style=\"text-align: justify\">Schierenbeck, M. W., &amp; Murphy, J. A. (2018). Interrater reliability and usability of a nurse anesthesia clinical evaluation instrument. <i>Journal of Nursing Education<\/i>,<i> 57<\/i>(7), 446-449. <a href=\"https:\/\/doi.org\/10.3928\/01484834-20180618-12\">https:\/\/doi.org<span lang=\"IN\">\/<\/span>10.3928\/01484834-20180618-12<\/a><\/p>\r\n<p style=\"text-align: justify\">Setyonugroho, W., Kennedy, K. M., &amp; Kropmans, T. J. B. (2015). Reliability and validity of OSCE checklists used to assess the communication skills of undergraduate medical students: A systematic review. <i>Patient Education and Counseling<\/i>,<i> 98<\/i>(12), 1482-1491. <a href=\"https:\/\/doi.org\/https:\/doi.org\/10.1016\/j.pec.2015.06.004\">https:\/\/doi.org\/ 10.1016\/j.pec.2015.06.004<\/a><\/p>\r\n<p style=\"text-align: justify\">Stalmeijer, R. E., McNaughton, N., &amp; Van Mook, W. N. (2014). Using focus groups in medical education research: AMEE Guide No. 91. <i>Medical Teacher<\/i>,<i> 36<\/i>(11), 923-939. <a href=\"https:\/\/doi.org\/https:\/doi.org\/10.3109\/0142159X.2014.917165\">https:\/\/doi.org\/ 10.3109\/0142159X.2014.917165<\/a><\/p>\r\n<p style=\"text-align: justify\">Stoyan, D., Pommerening, A., Hummel, M., &amp; Kopp-Schneider, A. (2017). Multiple-rater kappas for binary data: Models and interpretation. <i>Biometrical Journal<\/i>,<i> 60<\/i>(5), 381-394. <a href=\"https:\/\/doi.org\/\">https:\/\/doi.org\/<\/a> <a href=\"https:\/\/doi.org\/10.1002\/bimj.201600267\">10.1002\/bimj.201600267<\/a><\/p>\r\n<p style=\"text-align: justify\">Suhoyo, Y., Rahayu, G. R., &amp; Cahyani, N. (2016). A national collaboration to improve OSCE delivery. <i>Medical Education<\/i>,<i> 50<\/i>(11), 1150\u20131151. <a href=\"https:\/\/doi.org\/https:\/doi.org\/10.1111\/medu.13189\">https:\/\/doi.org\/ 10.1111\/medu.13189<\/a><\/p>\r\n<p style=\"text-align: justify\">Vanbelle, S. (2019). Asymptotic variability of (multilevel) multirater kappa coefficients. <i>Statistical Methods in Medical Research<\/i>,<i> 28<\/i>(10-11), 3012-3026. <a href=\"https:\/\/doi.org\/https:\/doi.org\/10.1177\/0962280218794733\">https:\/\/doi.org \/10.1177\/0962280218794733<\/a><\/p>\r\n<p style=\"text-align: justify\">Wood, T. J. (2014). Exploring the role of first impressions in rater-based assessments. <i>Advances in Health Sciences Education : Theory and Practice<\/i>,<i> 19<\/i>(3), 409-427. <a href=\"https:\/\/doi.org\/https:\/doi.org\/10.1007\/s10459-013-9453-9\">https:\/\/doi.org\/ 10.1007\/s10459-013-9453-9<\/a><\/p>\r\n<p style=\"text-align: justify\">Wood, T. J., Chan, J., Humphrey-Murto, S., Pugh, D., &amp; Touchie, C. (2017). The influence of first impressions on subsequent ratings within an OSCE station. <i>Advances in Health Sciences Education : Theory and Practice<\/i>,<i> 22<\/i>(4), 969-983. <a href=\"https:\/\/doi.org\/10.1007\/s10459-016-9736-z%20\">https:\/\/doi.org\/10.1007\/s10459-016-9736-z<\/a> \u00a0<\/p>\r\n<p style=\"text-align: justify\">Yeates, P., Moreau, M., &amp; Eva, K. (2015). Are examiners\u2019 judgments in osce-style assessments influenced by contrast effects? <i>Academic Medicine : Journal of the Association of American Medical Colleges<\/i>,<i> 90<\/i>(7), 975-980. <a href=\"https:\/\/doi.org\/https:\/doi.org\/10.1097\/ACM.0000000000000650\">https:\/\/doi.org \/10.1097\/ACM.0000000000000650<\/a><\/p>\r\n<p style=\"text-align: justify\">Yeates, P., O\u2019Neill, P., Mann, K., &amp; Eva, K. (2013). Seeing the same thing differently: Mechanisms that contribute to assessor differences in directly-observed performance assessments. <i>Advances in Health Sciences Education : Theory and Practice<\/i>,<i> 18<\/i>(3), 325-341. <a href=\"https:\/\/doi.org\/10.1007\/s10459-012-9372-1%20\">https:\/\/doi.org\/10.1007\/s10459-012-9372-1<\/a><\/p>\r\n<p>*Oscar Gilang Purnajati<br \/>\r\nFaculty of Medicine,\u00a0<br \/>\r\n<span lang=\"EN-ID\">Universitas Kristen Duta Wacana,<\/span><br \/>\r\n<span lang=\"EN-ID\">Jl. Dr. Wahidin Sudirohusodo No. 5-25.<\/span><br \/>\r\nYogyakarta<span lang=\"EN-ID\"> City,<\/span><br \/>\r\n<span lang=\"EN-GB\"><span lang=\"EN-ID\">Special Region of Yogyakarta<\/span><br \/>\r\n<span lang=\"EN-ID\">55224<\/span>, <span lang=\"EN-ID\">I<\/span>ndonesia<span lang=\"EN-ID\">.<\/span><span lang=\"EN-ID\">55224<\/span>, <span lang=\"EN-ID\">I<\/span>ndonesia<span lang=\"EN-ID\">.<\/span><br \/>\r\nTel: +62-274-563929<br \/>\r\nEmail: oscargilang@staff.ukdw.ac.id<\/span><\/p>","protected":false},"featured_media":0,"parent":0,"menu_order":6,"template":"","issues_category":[12],"archive_category":[],"issue_type":[25],"volume_category":[61866],"class_list":["post-70467","issues","type-issues","status-publish","hentry","issues_category-original-articles","issue_type-past-issue","volume_category-volume-6-number-2-may-2021"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v26.4 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Does objective structured clinical examination examiners\u2019 backgrounds influence the score agreement? - The Asia Pacific Scholar<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/medicine.nus.edu.sg\/taps\/issues\/does-objective-structured-clinical-examination-examiners-backgrounds-influence-the-score-agreement\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Does objective structured clinical examination examiners\u2019 backgrounds influence the score agreement? - The Asia Pacific Scholar\" \/>\n<meta property=\"og:description\" content=\"Submitted: 28 July 2020 Accepted: 18 November 2020 Published online: 4 May, TAPS 2021, 6(2), 48-56 https:\/\/doi.org\/10.29060\/TAPS.2021-6-2\/OA2367 Oscar Gilang Purnajati1, Rachmadya Nur Hidayah2 &amp; Gandes Retno Rahayu2 1Faculty of Medicine, Universitas Kristen Duta Wacana, Yogyakarta, Indonesia; 2Department of Medical Education, Faculty of Medicine, Universitas Gadjah Mada, Yogyakarta, Indonesia Abstract Introduction: Objective Structured Clinical Examination (OSCE) [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/medicine.nus.edu.sg\/taps\/issues\/does-objective-structured-clinical-examination-examiners-backgrounds-influence-the-score-agreement\/\" \/>\n<meta property=\"og:site_name\" content=\"The Asia Pacific Scholar\" \/>\n<meta property=\"article:modified_time\" content=\"2021-07-13T01:10:04+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/03\/OA2367-Figure-1.jpg\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"23 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/medicine.nus.edu.sg\/taps\/issues\/does-objective-structured-clinical-examination-examiners-backgrounds-influence-the-score-agreement\/\",\"url\":\"https:\/\/medicine.nus.edu.sg\/taps\/issues\/does-objective-structured-clinical-examination-examiners-backgrounds-influence-the-score-agreement\/\",\"name\":\"Does objective structured clinical examination examiners\u2019 backgrounds influence the score agreement? - The Asia Pacific Scholar\",\"isPartOf\":{\"@id\":\"https:\/\/medicine.nus.edu.sg\/taps\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/medicine.nus.edu.sg\/taps\/issues\/does-objective-structured-clinical-examination-examiners-backgrounds-influence-the-score-agreement\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/medicine.nus.edu.sg\/taps\/issues\/does-objective-structured-clinical-examination-examiners-backgrounds-influence-the-score-agreement\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/03\/OA2367-Figure-1.jpg\",\"datePublished\":\"2021-04-13T04:07:17+00:00\",\"dateModified\":\"2021-07-13T01:10:04+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/medicine.nus.edu.sg\/taps\/issues\/does-objective-structured-clinical-examination-examiners-backgrounds-influence-the-score-agreement\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/medicine.nus.edu.sg\/taps\/issues\/does-objective-structured-clinical-examination-examiners-backgrounds-influence-the-score-agreement\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/medicine.nus.edu.sg\/taps\/issues\/does-objective-structured-clinical-examination-examiners-backgrounds-influence-the-score-agreement\/#primaryimage\",\"url\":\"https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/03\/OA2367-Figure-1.jpg\",\"contentUrl\":\"https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/03\/OA2367-Figure-1.jpg\",\"width\":803,\"height\":385},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/medicine.nus.edu.sg\/taps\/issues\/does-objective-structured-clinical-examination-examiners-backgrounds-influence-the-score-agreement\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/medicine.nus.edu.sg\/taps\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Does objective structured clinical examination examiners\u2019 backgrounds influence the score agreement?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/medicine.nus.edu.sg\/taps\/#website\",\"url\":\"https:\/\/medicine.nus.edu.sg\/taps\/\",\"name\":\"The Asia Pacific Scholar\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/medicine.nus.edu.sg\/taps\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Does objective structured clinical examination examiners\u2019 backgrounds influence the score agreement? - The Asia Pacific Scholar","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/medicine.nus.edu.sg\/taps\/issues\/does-objective-structured-clinical-examination-examiners-backgrounds-influence-the-score-agreement\/","og_locale":"en_US","og_type":"article","og_title":"Does objective structured clinical examination examiners\u2019 backgrounds influence the score agreement? - The Asia Pacific Scholar","og_description":"Submitted: 28 July 2020 Accepted: 18 November 2020 Published online: 4 May, TAPS 2021, 6(2), 48-56 https:\/\/doi.org\/10.29060\/TAPS.2021-6-2\/OA2367 Oscar Gilang Purnajati1, Rachmadya Nur Hidayah2 &amp; Gandes Retno Rahayu2 1Faculty of Medicine, Universitas Kristen Duta Wacana, Yogyakarta, Indonesia; 2Department of Medical Education, Faculty of Medicine, Universitas Gadjah Mada, Yogyakarta, Indonesia Abstract Introduction: Objective Structured Clinical Examination (OSCE) [&hellip;]","og_url":"https:\/\/medicine.nus.edu.sg\/taps\/issues\/does-objective-structured-clinical-examination-examiners-backgrounds-influence-the-score-agreement\/","og_site_name":"The Asia Pacific Scholar","article_modified_time":"2021-07-13T01:10:04+00:00","og_image":[{"url":"https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/03\/OA2367-Figure-1.jpg","type":"","width":"","height":""}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"23 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/medicine.nus.edu.sg\/taps\/issues\/does-objective-structured-clinical-examination-examiners-backgrounds-influence-the-score-agreement\/","url":"https:\/\/medicine.nus.edu.sg\/taps\/issues\/does-objective-structured-clinical-examination-examiners-backgrounds-influence-the-score-agreement\/","name":"Does objective structured clinical examination examiners\u2019 backgrounds influence the score agreement? - The Asia Pacific Scholar","isPartOf":{"@id":"https:\/\/medicine.nus.edu.sg\/taps\/#website"},"primaryImageOfPage":{"@id":"https:\/\/medicine.nus.edu.sg\/taps\/issues\/does-objective-structured-clinical-examination-examiners-backgrounds-influence-the-score-agreement\/#primaryimage"},"image":{"@id":"https:\/\/medicine.nus.edu.sg\/taps\/issues\/does-objective-structured-clinical-examination-examiners-backgrounds-influence-the-score-agreement\/#primaryimage"},"thumbnailUrl":"https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/03\/OA2367-Figure-1.jpg","datePublished":"2021-04-13T04:07:17+00:00","dateModified":"2021-07-13T01:10:04+00:00","breadcrumb":{"@id":"https:\/\/medicine.nus.edu.sg\/taps\/issues\/does-objective-structured-clinical-examination-examiners-backgrounds-influence-the-score-agreement\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/medicine.nus.edu.sg\/taps\/issues\/does-objective-structured-clinical-examination-examiners-backgrounds-influence-the-score-agreement\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/medicine.nus.edu.sg\/taps\/issues\/does-objective-structured-clinical-examination-examiners-backgrounds-influence-the-score-agreement\/#primaryimage","url":"https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/03\/OA2367-Figure-1.jpg","contentUrl":"https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/03\/OA2367-Figure-1.jpg","width":803,"height":385},{"@type":"BreadcrumbList","@id":"https:\/\/medicine.nus.edu.sg\/taps\/issues\/does-objective-structured-clinical-examination-examiners-backgrounds-influence-the-score-agreement\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/medicine.nus.edu.sg\/taps\/"},{"@type":"ListItem","position":2,"name":"Does objective structured clinical examination examiners\u2019 backgrounds influence the score agreement?"}]},{"@type":"WebSite","@id":"https:\/\/medicine.nus.edu.sg\/taps\/#website","url":"https:\/\/medicine.nus.edu.sg\/taps\/","name":"The Asia Pacific Scholar","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/medicine.nus.edu.sg\/taps\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/medicine.nus.edu.sg\/taps\/wp-json\/wp\/v2\/issues\/70467","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/medicine.nus.edu.sg\/taps\/wp-json\/wp\/v2\/issues"}],"about":[{"href":"https:\/\/medicine.nus.edu.sg\/taps\/wp-json\/wp\/v2\/types\/issues"}],"wp:attachment":[{"href":"https:\/\/medicine.nus.edu.sg\/taps\/wp-json\/wp\/v2\/media?parent=70467"}],"wp:term":[{"taxonomy":"issues_category","embeddable":true,"href":"https:\/\/medicine.nus.edu.sg\/taps\/wp-json\/wp\/v2\/issues_category?post=70467"},{"taxonomy":"archive_category","embeddable":true,"href":"https:\/\/medicine.nus.edu.sg\/taps\/wp-json\/wp\/v2\/archive_category?post=70467"},{"taxonomy":"issue_type","embeddable":true,"href":"https:\/\/medicine.nus.edu.sg\/taps\/wp-json\/wp\/v2\/issue_type?post=70467"},{"taxonomy":"volume_category","embeddable":true,"href":"https:\/\/medicine.nus.edu.sg\/taps\/wp-json\/wp\/v2\/volume_category?post=70467"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}