Cluster of methods

Key features

Issues

When to use

Arbitrary standards and norm-referenced standards

• Arbitrary standards produce a fixed pass mark, e.g., candidates scoring 50% or more pass.
• Norm-referenced standards produce a fixed pass rate, e.g., 40% of top-scoring candidates pass.

The pass mark is unrelated to the difficulty of assessment items.

• Arbitrary standards: not indicated for high-stakes assessment.
• Norm-referencing: used for selection purposes.

Test-centred methods

• A group of experts (judges) estimate the probability of a hypothetical borderline (a candidate who has a 50% probability of passing or failing) or a just-passing candidate passing the test items, e.g. Angoff (1971), Ebel (1972), Nedelsky (1954), Bookmark (Karantonis & Sireci, 2006), Jaeger (1982).
• The judges’ estimates are collated through an averaging process.
• An expert (judge) is a subject-matter specialist, with considerable experience as a teacher and an assessor, well versed with the educational basis behind standard setting.

Although the pass mark is directly related to the difficulty of test items,

• human judgement is not infallible: The pass mark can vary from one panel of judges to another, even for the same test.
• finding a sizeable group of experts (at least 8) satisfying all requirements is difficult.
• it is difficult for judges to visualise a hypothetical borderline candidate.
• the process is time-consuming.

Due to the above difficulties, the pass mark can be unrealistic.

• When an adequate number of properly trained and experienced expert judges who can devote quality time to the standard setting process is available.
• When modifications such as the Modified Angoff method can be used to overcome unrealistic standards by allowing judges to be informed by actual results of previous similar exams.

Partially results-based methods-I: Examinee-centred methods

• Based on actual candidate performance, judges group candidates into two or more groups, e.g. Borderline group (Smee & Blackmore, 2001), Borderline regression (Kramer et al., 2003), Contrasting groups (p.35) (Livingston & Zieky, 1982) and Up-down (p.43) (Livingston & Zieky, 1982) methods.
• The pass mark is calculated using the actual candidate scores.

Although judgements are realistic, the introduction of actual test results tends to make the standard cohort-dependent, i.e., norm-referencing features influence the standard.

• When there is a sufficiently large number of candidates.
• When a global score or a global pass/fail decision is available, in addition to the usual itemized score.
• When the judges are well-trained in making a global decision independent of the itemised scores.

Partially results-based methods-II: Compromise methods

• Judges make judgements by looking at test items, and those judgements are superimposed on actual candidate scores to derive the pass mark, e.g., Hofstee method (Hofstee, 1973).

• Expert judgements and actual results may not match each other.
• The standard can be cohort-dependent due to the norm-referencing features of actual candidate scores.

• When trained judges, actual results of a sizable cohort of candidates and expertise in handling both judges’ judgements and results are available.
• Mostly used as a backup method to verify standards generated by other methods.

Results-based methods

• Judges are not needed for standard setting.
• The pass mark is generated by statistically manipulating the actual marks, e.g. Cohen (Cohen-
  Schotanus & van der Vleuten, 2010) and Wijnen (1971) methods.

Due to the norm-referencing
influence, the pass mark
could be high and
defensibility would be an
issue.

These methods should be used in high-stakes assessment only when an adequate evidence base is built by conducting them parallelly with another more established method.