{"id":72909,"date":"2025-07-01T09:03:13","date_gmt":"2025-07-01T01:03:13","guid":{"rendered":"https:\/\/medicine.nus.edu.sg\/taps\/?post_type=issues&p=72909"},"modified":"2025-10-07T08:26:38","modified_gmt":"2025-10-07T00:26:38","slug":"finding-the-standard-setting-method-for-your-assessment","status":"publish","type":"issues","link":"https:\/\/medicine.nus.edu.sg\/taps\/issues\/finding-the-standard-setting-method-for-your-assessment\/","title":{"rendered":"Finding the standard setting method for your assessment"},"content":{"rendered":"

https:\/\/doi.org\/10.29060\/TAPS.2025-10-3\/TT003<\/a><\/p>\r\n

Gominda Ponnamperuma<\/span><\/p>\r\n

MBBS, MMEd, PhD<\/em>
\r\nProfessor in Medical Education<\/em>
\r\nFaculty of Medicine, University of Colombo, Sri Lanka<\/em><\/p>\r\n

Standard setting is the process of deciding the boundary or standard that separates the candidates into two (e.g. pass and fail) or more groups, based on their ability shown at an assessment. Standard setting methods can be broadly grouped into four clusters (see table below).<\/span><\/p>\r\n

When to use which method, though a crucial decision for any Board of Examiners, is inadequately explored in the literature. The following brief guide attempts to bridge this literature gap.<\/span><\/p>\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n
\r\n

Cluster of methods<\/span><\/b><\/p>\r\n<\/td>\r\n

\r\n

Key features<\/span><\/b><\/p>\r\n<\/td>\r\n

\r\n

Issues<\/span><\/b><\/p>\r\n<\/td>\r\n

\r\n

When to use<\/span><\/b><\/p>\r\n<\/td>\r\n<\/tr>\r\n

\r\n

Arbitrary standards and norm-referenced standards<\/span><\/p>\r\n<\/td>\r\n

\r\n
    \r\n\t
  • Arbitrary standards produce a fixed pass mark<\/b>, e.g., candidates scoring 50% or more pass.<\/span><\/li>\r\n\t
  • Norm-referenced standards produce a fixed pass rate<\/b>, e.g., 40% of top-scoring candidates pass.<\/span><\/li>\r\n<\/ul>\r\n<\/td>\r\n
\r\n

The pass mark is unrelated to the difficulty of assessment items.<\/span><\/p>\r\n<\/td>\r\n

\r\n
    \r\n\t
  • Arbitrary standards: not indicated for high-stakes assessment.<\/span><\/li>\r\n\t
  • Norm-referencing: used for selection purposes.<\/span><\/li>\r\n<\/ul>\r\n<\/td>\r\n<\/tr>\r\n
\r\n

Test-centred methods<\/span><\/p>\r\n<\/td>\r\n

\r\n
    \r\n\t
  • A group of experts (judges) estimate the probability of a hypothetical borderline (a candidate who has a 50% probability of passing or failing) or a just-passing candidate passing the test items, e.g. Angoff (1971), Ebel (1972), Nedelsky (1954), Bookmark (Karantonis & Sireci, 2006), Jaeger (1982).<\/span><\/li>\r\n\t
  • The judges\u2019 estimates are collated through an averaging process.<\/span><\/li>\r\n\t
  • An expert (judge) is a subject-matter specialist, with considerable experience as a teacher and an assessor, well versed with the educational basis behind standard setting.<\/span><\/li>\r\n<\/ul>\r\n<\/td>\r\n
\r\n

Although the pass mark is directly related to the difficulty of test items,<\/span><\/p>\r\n

    \r\n\t
  • human judgement is not infallible: The pass mark can vary from one panel of judges to another, even for the same test.<\/span><\/li>\r\n\t
  • finding a sizeable group of experts (at least 8) satisfying all requirements is difficult.<\/span><\/li>\r\n\t
  • it is difficult for judges to visualise a hypothetical borderline candidate.<\/span><\/li>\r\n\t
  • the process is time-consuming.<\/span><\/li>\r\n<\/ul>\r\n

    Due to the above difficulties, the pass mark can be unrealistic. <\/span><\/p>\r\n<\/td>\r\n

\r\n
    \r\n\t
  • When an adequate number of properly trained and experienced expert judges who can devote quality time to the standard setting process is available.<\/span><\/li>\r\n\t
  • When modifications such as the\u00a0Modified Angoff method can be used to overcome unrealistic standards by allowing judges to be informed by actual results of previous similar exams.<\/span><\/li>\r\n<\/ul>\r\n<\/td>\r\n<\/tr>\r\n
\r\n

Partially results-based methods-I: Examinee-centred methods<\/span><\/p>\r\n<\/td>\r\n

\r\n
    \r\n\t
  • Based on actual candidate performance, judges group candidates into two or more groups, e.g. Borderline group (Smee & Blackmore, 2001), Borderline regression (Kramer et al., 2003), Contrasting groups (p.35) (Livingston & Zieky, 1982) and Up-down (p.43) (Livingston & Zieky, 1982) methods.<\/span><\/li>\r\n\t
  • The pass mark is calculated using the actual candidate scores.<\/span><\/li>\r\n<\/ul>\r\n<\/td>\r\n
\r\n

Although judgements are realistic, the introduction of actual test results tends to make the standard cohort-dependent, i.e., norm-referencing features influence the standard. <\/span><\/p>\r\n<\/td>\r\n

\r\n
    \r\n\t
  • When there is a sufficiently large number of candidates.<\/span><\/li>\r\n\t
  • When a global score or a global pass\/fail decision is available, in addition to the usual itemized score.<\/span><\/li>\r\n\t
  • When the judges are well-trained in making a global decision independent of the itemised scores.<\/span><\/li>\r\n<\/ul>\r\n<\/td>\r\n<\/tr>\r\n
\r\n

Partially results-based methods-II: Compromise methods<\/span><\/p>\r\n<\/td>\r\n

\r\n
    \r\n\t
  • Judges make judgements by looking at test items, and those judgements are superimposed on actual candidate scores to derive the pass mark, e.g., Hofstee method (Hofstee, 1973).<\/span><\/li>\r\n<\/ul>\r\n<\/td>\r\n
\r\n
    \r\n\t
  • Expert judgements and actual results may not match each other.<\/span><\/li>\r\n\t
  • The standard can be cohort-dependent due to the norm-referencing features of actual candidate scores.<\/span><\/li>\r\n<\/ul>\r\n<\/td>\r\n
\r\n
    \r\n\t
  • When trained judges, actual results of a sizable cohort of candidates and expertise in handling both judges\u2019 judgements and results are available. <\/span><\/li>\r\n\t
  • Mostly used as a backup method to verify standards generated by other methods.<\/span><\/li>\r\n<\/ul>\r\n<\/td>\r\n<\/tr>\r\n
\r\n

Results-based methods<\/span><\/p>\r\n<\/td>\r\n

\r\n
    \r\n\t
  • Judges are not needed for standard setting.<\/span><\/li>\r\n\t
  • The pass mark is generated by statistically manipulating the actual marks, e.g. Cohen (Cohen-
    \r\nSchotanus & van der Vleuten, 2010) and Wijnen (1971) methods.<\/span><\/li>\r\n<\/ul>\r\n<\/td>\r\n
\r\n

Due to the norm-referencing
\r\ninfluence, the pass mark
\r\ncould be high and
\r\ndefensibility would be an
\r\nissue.<\/span><\/p>\r\n<\/td>\r\n

\r\n

These methods should be used in high-stakes assessment only<\/b> when an adequate evidence base is built by conducting them parallelly with another more established method.<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n

 <\/p>\r\n

References<\/strong><\/p>\r\n

Angoff, W. H. (1971). Scales, norms, and equivalent scores<\/em>. In R. L. Thorndike (Ed.), Educational measurement (2nd<\/sup> ed., pp. 508-600). American Council on Education.<\/span><\/p>\r\n

Ebel, R. L. (1972). Essentials of educational measurement<\/em>. Prentice Hall.<\/span><\/p>\r\n

Nedelsky, L. (1954). Absolute grading standards for objective tests. Educational and Psychological Measurement<\/em>, 14<\/em>(1), 3-19. <\/span>https:\/\/doi.org\/10.1177\/001316445401400101<\/span><\/a><\/span> <\/span><\/span><\/p>\r\n

Karantonis, A., & Sireci, S. G. (2006). The bookmark standard-setting method: A literature review. Educational Measurement Issues and Practice<\/em>, 25<\/em>(1), 4-12. <\/span>https:\/\/doi.org\/10.1111\/j.1745-3992.2006.00047.x<\/span><\/a><\/span> <\/span><\/span><\/p>\r\n

Jaeger, R. M. (1982). An iterative structured judgment process for establishing standards on competency test: Theory and application. Educational Evaluation and Policy Analysis<\/em>, 4<\/em>(4), 461-476.<\/span> <\/span>https:\/\/doi.org\/10.3102\/01623737004004461<\/span><\/a><\/span> <\/span><\/span><\/p>\r\n

Smee, S. M., & Blackmore, D. E. (2001). Setting standards for an Objective Structured Clinical Examination: The borderline group method gains ground on Angoff. Medical Education<\/em>, 35<\/em>(11), 1009-1010.<\/span> <\/span>https:\/\/doi.org\/10.1111\/j.1365-2923.2001.01047.x<\/span><\/a><\/span> <\/span><\/span><\/p>\r\n

Kramer, A., Muijtjens, A., Jansen, K., Dusman, H., Tan, L., & van der Vleuten, C. (2003) Comparison of a rational and an empirical standard setting procedure for an OSCE. Medical Education<\/em>, 37<\/em>(2), 132-139. <\/span>https:\/\/doi.org\/10.1046\/j.1365-2923.2003.01429.x<\/span><\/a><\/span> <\/span><\/span><\/p>\r\n

Livingston, S. A., & Zieky, M. J. (1982). Passing scores: A manual for setting standards of performance on educational and occupational tests<\/em>. Educational Testing Service.<\/span><\/p>\r\n

Hofstee, W. K. B. (1973). Een alternatief voor normhandhaving bij toetsen. Nederlands Tijdschrift voor de Psychologie<\/em>, 28<\/em>, 215-227.<\/span><\/p>\r\n

Cohen-Schotanus, J., & van der Vleuten, C. P. M. (2010). A standard setting method with the best performing students as point of reference: Practical and affordable.\u00a0Medical Teacher<\/em>,\u00a032<\/em>(2), 154-160. <\/span>https:\/\/doi.org\/10.3109\/01421590903196979<\/span><\/a><\/span> <\/span><\/span><\/p>\r\n

Wijnen, W. H. F. W. (1971). Onder of boven de maat<\/em>. <\/span>Amsterdam: Swets & Zeitlinger.<\/span><\/p>","protected":false},"featured_media":0,"parent":0,"menu_order":14,"template":"","issues_category":[61893],"archive_category":[],"issue_type":[25],"volume_category":[61895],"class_list":["post-72909","issues","type-issues","status-publish","hentry","issues_category-tips-from-taps","issue_type-past-issue","volume_category-volume-10-number-3-july-2025"],"acf":[],"yoast_head":"\nFinding the standard setting method for your assessment - The Asia Pacific Scholar<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/medicine.nus.edu.sg\/taps\/issues\/finding-the-standard-setting-method-for-your-assessment\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Finding the standard setting method for your assessment - The Asia Pacific Scholar\" \/>\n<meta property=\"og:description\" content=\"https:\/\/doi.org\/10.29060\/TAPS.2025-10-3\/TT003 Gominda Ponnamperuma MBBS, MMEd, PhD Professor in Medical Education Faculty of Medicine, University of Colombo, Sri Lanka Standard setting is the process of deciding the boundary or standard that separates the candidates into two (e.g. pass and fail) or more groups, based on their ability shown at an assessment. Standard setting methods can be […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/medicine.nus.edu.sg\/taps\/issues\/finding-the-standard-setting-method-for-your-assessment\/\" \/>\n<meta property=\"og:site_name\" content=\"The Asia Pacific Scholar\" \/>\n<meta property=\"article:modified_time\" content=\"2025-10-07T00:26:38+00:00\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/medicine.nus.edu.sg\/taps\/issues\/finding-the-standard-setting-method-for-your-assessment\/\",\"url\":\"https:\/\/medicine.nus.edu.sg\/taps\/issues\/finding-the-standard-setting-method-for-your-assessment\/\",\"name\":\"Finding the standard setting method for your assessment - The Asia Pacific Scholar\",\"isPartOf\":{\"@id\":\"https:\/\/medicine.nus.edu.sg\/taps\/#website\"},\"datePublished\":\"2025-07-01T01:03:13+00:00\",\"dateModified\":\"2025-10-07T00:26:38+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/medicine.nus.edu.sg\/taps\/issues\/finding-the-standard-setting-method-for-your-assessment\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/medicine.nus.edu.sg\/taps\/issues\/finding-the-standard-setting-method-for-your-assessment\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/medicine.nus.edu.sg\/taps\/issues\/finding-the-standard-setting-method-for-your-assessment\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/medicine.nus.edu.sg\/taps\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Finding the standard setting method for your assessment\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/medicine.nus.edu.sg\/taps\/#website\",\"url\":\"https:\/\/medicine.nus.edu.sg\/taps\/\",\"name\":\"The Asia Pacific Scholar\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/medicine.nus.edu.sg\/taps\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Finding the standard setting method for your assessment - The Asia Pacific Scholar","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/medicine.nus.edu.sg\/taps\/issues\/finding-the-standard-setting-method-for-your-assessment\/","og_locale":"en_US","og_type":"article","og_title":"Finding the standard setting method for your assessment - The Asia Pacific Scholar","og_description":"https:\/\/doi.org\/10.29060\/TAPS.2025-10-3\/TT003 Gominda Ponnamperuma MBBS, MMEd, PhD Professor in Medical Education Faculty of Medicine, University of Colombo, Sri Lanka Standard setting is the process of deciding the boundary or standard that separates the candidates into two (e.g. pass and fail) or more groups, based on their ability shown at an assessment. Standard setting methods can be […]","og_url":"https:\/\/medicine.nus.edu.sg\/taps\/issues\/finding-the-standard-setting-method-for-your-assessment\/","og_site_name":"The Asia Pacific Scholar","article_modified_time":"2025-10-07T00:26:38+00:00","twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/medicine.nus.edu.sg\/taps\/issues\/finding-the-standard-setting-method-for-your-assessment\/","url":"https:\/\/medicine.nus.edu.sg\/taps\/issues\/finding-the-standard-setting-method-for-your-assessment\/","name":"Finding the standard setting method for your assessment - The Asia Pacific Scholar","isPartOf":{"@id":"https:\/\/medicine.nus.edu.sg\/taps\/#website"},"datePublished":"2025-07-01T01:03:13+00:00","dateModified":"2025-10-07T00:26:38+00:00","breadcrumb":{"@id":"https:\/\/medicine.nus.edu.sg\/taps\/issues\/finding-the-standard-setting-method-for-your-assessment\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/medicine.nus.edu.sg\/taps\/issues\/finding-the-standard-setting-method-for-your-assessment\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/medicine.nus.edu.sg\/taps\/issues\/finding-the-standard-setting-method-for-your-assessment\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/medicine.nus.edu.sg\/taps\/"},{"@type":"ListItem","position":2,"name":"Finding the standard setting method for your assessment"}]},{"@type":"WebSite","@id":"https:\/\/medicine.nus.edu.sg\/taps\/#website","url":"https:\/\/medicine.nus.edu.sg\/taps\/","name":"The Asia Pacific Scholar","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/medicine.nus.edu.sg\/taps\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/medicine.nus.edu.sg\/taps\/wp-json\/wp\/v2\/issues\/72909","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/medicine.nus.edu.sg\/taps\/wp-json\/wp\/v2\/issues"}],"about":[{"href":"https:\/\/medicine.nus.edu.sg\/taps\/wp-json\/wp\/v2\/types\/issues"}],"wp:attachment":[{"href":"https:\/\/medicine.nus.edu.sg\/taps\/wp-json\/wp\/v2\/media?parent=72909"}],"wp:term":[{"taxonomy":"issues_category","embeddable":true,"href":"https:\/\/medicine.nus.edu.sg\/taps\/wp-json\/wp\/v2\/issues_category?post=72909"},{"taxonomy":"archive_category","embeddable":true,"href":"https:\/\/medicine.nus.edu.sg\/taps\/wp-json\/wp\/v2\/archive_category?post=72909"},{"taxonomy":"issue_type","embeddable":true,"href":"https:\/\/medicine.nus.edu.sg\/taps\/wp-json\/wp\/v2\/issue_type?post=72909"},{"taxonomy":"volume_category","embeddable":true,"href":"https:\/\/medicine.nus.edu.sg\/taps\/wp-json\/wp\/v2\/volume_category?post=72909"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}