Number of Citations: 0

Submitted: 13 June 2024
Accepted: 14 October 2024
Published online: 7 January, TAPS 2025, 10(1), 36-47
https://doi.org/10.29060/TAPS.2025-10-1/OA3438

Sivapalan Sanchayan¹, Sisira Dharmaratne², Shalika Pathirana³, Pavithra Godamunne³ & Madawa Chandratilake³

¹Department of Medical Education, Faculty of Medicine, University of Jaffna, Sri Lanka; ²Translational Health Research Institute, Western Sydney University, Australia; Department of Medical Education, Faculty of Medicine, University of Kelaniya, Sri Lanka

Abstract

Introduction: Selection for basic medical training is highly contextual. The use of cognitive aptitude tests, which commonly supplement measures of prior academic achievement in the selection process of medical schools internationally, is rarely reported from resource-constrained settings in South Asia. We report on the psychometric properties of an aptitude test designed based on the UCAT format, administered to first-year medical undergraduates to determine its utility.

Methods: The aptitude test was administered online to first-year medical students shortly after their admission to two medical faculties in Sri Lanka (n=328). The reliability of the test was determined using Cronbach’s alpha. Overall and subtest scores were computed, and the scores of different demographic groups were compared using the t-test. Factor analysis of the subtests was performed.

Results: The internal consistency of the test was 0.63. The difficulty and discrimination indices were within the acceptable range. The mean score of the aptitude test (AT) was 70.9/100 (SD 8.88). The mean score for females was higher than for males (p=0.04). No statistically significant differences in AT scores were observed between different ethnicities or religions. The entry academic scores demonstrated a weak correlation with the overall AT score (Pearson’s correlation coefficient r=0.27), verbal reasoning (r=0.24), the human body (r=0.19), quantitative reasoning (r=0.18), and situational judgement (r=0.128). Factor analysis indicated items that need revision.

Conclusion: The results indicate the potential value of aptitude tests in Sri Lanka and in similar jurisdictions where selection criteria have yet to expand beyond prior academic performance. A way forward in introducing such tests has been outlined.

Keywords:           Medical Student Selection, Medical Student Admission, Selection Criteria

Practice Highlights

• Aptitude tests assess cognitive and non-cognitive attributes desirable for healthcare professionals.
• The use of aptitude tests is rarely reported from resource constrained settings in South Asia.
• Sri Lanka uses only the national high school examination performance for medical school admission.
• This study indicates the potential value of aptitude tests in Sri Lanka and in similar jurisdictions.

I. INTRODUCTION

Selection for basic medical training is highly contextual and influenced by educational, geographical, socioeconomic, and political factors (Dharmaratne & Ponnamperuma, 2020; Patterson et al., 2018). High academic achievement has long been considered an appropriate criterion for ranking candidates for medical school (McManus et al., 2013; Patterson et al., 2017). However, this approach has been contested due to the low predictive validity of pre-university performance for medical school success (McManus et al., 2013; Prideaux et al., 2011).

Aptitude tests have emerged to assess cognitive and non-cognitive traits desirable for medical and healthcare professions (Gliatto et al., 2016; Greatrix & Dowell, 2020; Powis et al., 2021). Despite evidence on their potential to improve selection processes (Bala et al., 2022; Greatrix et al., 2021), most resource-constrained jurisdictions, including Sri Lanka, rely solely on prior academic achievement (Soemantri et al., 2020; University Grants Commission, 2023). This study represents an initial step in Sri Lanka toward introducing aptitude tests and reports on the psychometric properties of a test developed to measure cognitive attributes.

The Sri Lankan medical school admission system uses national high school examination results (G.C.E. AL) as the only criterion for admission (Dharmaratne & Ponnamperuma, 2020; University Grants Commission, 2023). The results of three AL subjects (Chemistry, Physics, and Biology) are considered, and an average Z score is used to rank individuals (University Grants Commission, 2023). A minimum of two credit passes and one simple pass is required for eligibility. A few candidates with foreign examination results equivalent to the G.C.E. AL Examination and meeting equitable minimum requirements are also admitted (University Grants Commission, 2023).

G.C.E. AL examination results have shown a mild correlation with subsequent undergraduate performance, with a maximum predictive correlation coefficient of 0.37 (de Silva et al., 2004, 2006; Hewage et al., 2011; Mettananda et al., 2006). These low figures may underestimate the true relationship, as range restriction among high-achieving candidates can affect the statistical accuracy of predictive validity measures (Zimmermann et al., 2017). Studies in the UK, addressing range restriction, found that prior academic performance is the best predictor of subsequent performance, accounting for 65% of the variance in undergraduate performance (McManus et al., 2013). However, this leaves 35% of the variance unexplained.

Aptitude tests are used worldwide to support medical student selection. The United Kingdom Clinical Aptitude Test (UKCAT), now named UCAT, has been used for nearly two decades (Greatrix & Dowell, 2020). The UCAT consists of 200 items in five subtests delivered over two hours (Paton et al., 2022). Results of the UKCAT and UCAT have been used to discriminate between high-achieving students during selection and predict undergraduate and postgraduate performance (Bala et al., 2022; MacKenzie et al., 2016; McManus et al., 2013; Paton et al., 2022).

However, there are additional qualities beyond academic readiness (prior academic performance and cognitive traits) that are necessary qualities of medical professionals that include desirable behaviours and attitudes that could indicate successful development as healthcare practitioners in the future (Patterson et al., 2018). The Ottawa consensus statement recommends that best practice selection should involve the use of selection criteria based on multi-source multi-method job analysis studies (Patterson et al., 2018). This necessitates the use of other selection processes such as structured interviews or multiple mini interviews (MMI).

This study aims to evaluate the psychometric properties of an aptitude test based on the UCAT format, administered to first-year medical undergraduates at two medical faculties in Sri Lanka. Specifically, the objectives are to evaluate the psychometric properties of the test, determine the relationship between Z scores at the AL examination and aptitude test scores, and compare the performance of different demographic groups in the aptitude test.

II. METHODS

A cross-sectional observational study was conducted in the faculties of medicine at the universities of Kelaniya and Jaffna, Sri Lanka. Including these two faculties allowed for the recruitment of an adequate number of students from the three main ethnic groups (Sinhala, Tamil, and Sri Lankan Moors) and a wider range of Z scores. All first-year students in 2023 from these faculties were invited to participate. Students admitted through selection criteria other than Sri Lankan advanced level performance were excluded. We obtained a sample size of 328 students who consented to participated in the study (response rate of 80.8%).

The aptitude test comprised 50 items across six subtests: abstract reasoning (8 items), the human body (8 items), quantitative reasoning (8 items), decision-making (8 items), verbal reasoning (8 items), and situational judgment (10 items). Except for the “the human body” subtest, all others were modelled after the UKCAT examination. The “human body” subtest was designed to assess advanced-level knowledge of human biology from the Sri Lankan school curriculum. Test items were selected by a panel of experts in psychology and medical education from the Faculty of Medicine, University of Kelaniya with the help of disability studies experts who use a similar test for selection to their course. These experts, being native Sri Lankans, ensured the items were culturally relevant and conceptually and semantically equivalent (Hambleton & Zenisky, 2010). Similar methods were used to finalise the Sinhala and Tamil versions of the test, translated by competent translators. Appendix 1 shows sample test items from each subtest. The test was piloted on 15 students and identified issues were corrected.

The test was administered via the online learning management system (LMS) of the respective faculties under examination conditions, with invigilators ensuring adherence to university examination protocols. Students had one hour to complete the test, available in English, Sinhala, and Tamil. Participants could choose the language they were most comfortable with to simulate AL examination conditions and to minimise interpretation errors.

Descriptive statistics and internal consistency (Cronbach’s alpha) were calculated for the test and each subtest and item analysis (Paniagua & Swygert, 2016) was performed. Correlations between the aptitude test score, subtest scores, and Z scores were calculated using Pearson’s correlation coefficient, and relationships to other demographic factors were analysed using Student’s t-test. Factor analysis was performed using R statistical package.

Written informed consent was obtained from the students before the test administration. Ethics approval was granted by the ethics review committee of the Faculty of Medicine, University of Kelaniya, and permission for the study was obtained from the deans of the respective medical faculties.

III. RESULTS

328 students participated in the study, yielding an overall response rate of 80.8%. The descriptive statistics of these students are presented in Table 1. The male-to-female ratio of the sample (31.7% to 68.3%) closely mirrors the demographics of the national cohort of medical undergraduates from the previous academic year (34% to 66%), as reported by the University Grants Commission, Sri Lanka (University Grants Commission, 2022).

Table 1. Descriptive statistics of the demographic variables of the sample

The overall mean aptitude test (AT) score was 70.9 (SD 8.88), with a median of 72. Female students had a higher mean AT score than male students (p=0.044, Table 2), while there was no statistically significant difference in the Z scores between genders (p=0.122). No significant differences in mean AT scores were found between ethnicities and religions. The Faculty of Medicine, University of Kelaniya, had a higher mean AT score (73.7, SD 8.7) than the Faculty of Medicine, Jaffna (68.8, SD 8.4, p=0.000). Similarly, the mean Z score at Faculty of Medicine, University of Kelaniya (2.10, SD 0.07) was higher than at Jaffna (1.89, SD 0.13, p=0.000).

Table 2. Comparison of means of AT scores and Z scores between males and females

The internal consistency of the test was 0.628 and the subscales ranged between 0.2 to 0.47 (Table 3).

Table 3. Data regarding the subtests of the aptitude test

Difficulty and discrimination indices for each test item were calculated (Table 4). The difficulty index represents the proportion of students who answered each item correctly (Paniagua & Swygert, 2016). The mean difficulty index was 70.9% (SD 23.5), ranging from 19.2% to 99.4%. The discrimination index was calculated as the difference between the proportion of students who answered an item correctly in the top group (those who obtained the top 27% in the overall score) and bottom group (bottom 27% scorers) and measures an item’s ability to differentiate between students who performed best and those who performed worst on the test (Paniagua & Swygert, 2016). The mean discrimination index was 19% (SD 12) and ranged from 0 to 50%. There was only one item with the lowest discrimination index of 0. It had a difficulty index of 99.4% indicating that most students had scored it correctly. Items that adversely affected the reliability of the subtests were identified (Appendix 2).

Table 4. Distribution of the difficulty indices of the aptitude test questions

The correlation analysis revealed several key findings regarding the relationship between Z and AT scores. The correlation coefficient between Z score and AT score was 0.268 (p=0.000), indicating a weak correlation. The coefficient of determination (r²) was calculated to be 0.072, suggesting that only 7.2% of the variability in AT scores could be explained by Z scores. This signifies a limited association between Z scores and overall AT performance. Further analysis was conducted on the correlation between Z scores and subtest scores. There was a weak but statistically significant correlation between Z scores and the subtests of verbal reasoning (r=0.24, p=0.000), human body (r=0.19, p=0.001), quantitative reasoning (r=0.18, p=0.001), and situational judgment (r=0.128, p=0.021). However, the correlations between Z scores and the subtests of abstract reasoning and decision-making were not statistically significant.

Factor analysis was performed using the most prominent two factors of each of the six subtests separately as they each measure well defined and different cognitive domains. The factor loading of abstract reasoning subtest is shown in figure 1. “Question 3” shows a strong positive loading on Factor 1, suggesting that this item heavily measures the trait most represented by this factor whilst “Question 5” shows a slight negative loading on Factor 1, indicating it measures a trait that is perhaps inversely related to what is captured by this factor. “Question 1” and “Question 2” are both located positively on Factor 2 but with minimal loading on Factor 1. This suggests that they are measuring traits more aligned with Factor 2. The items “Question 1”, “Question 2″ and “Question 3”, appear to be critical for measuring distinct aspects of abstract reasoning as shown by their stronger loadings. “Question 5” shows a strong negative loading on Factor 2, suggesting that it measures a trait that contrasts with what Factor 2 represents. “Question 4”, ” Question 6″, “Question 7”, and “Question 8” are clustered around the origin. This indicates that these items have weaker loadings on both factors, meaning they may not strongly measure the traits represented by either factor, or they could be measuring a balanced mix of both traits. While these questions contribute to the test, they might need to be reviewed to enhance their discriminative power or clarity in measuring specific abstract reasoning skills.

Figure 1. Two factor loading of the abstract reasoning subtest

The factor loadings of the other five subtests are shown in Appendix 3. Regarding the human body subtest, Factor 1 might represent knowledge or understanding of structural aspects of the human body, given the strong loading by “Question 11” and reasonable loading by other items. This factor could reflect knowledge regarding human biology whilst Factor 2 seems to capture a different dimension, suggested by the negative loading of “Question 15”. In quantitative reasoning, most questions are clustered near the centre with a slight spread along Factor 1, indicating moderate influence by this factor across the items. “Question 19” and “Question 22” are slightly farther along Factor 1 suggesting a stronger influence by the trait measured by this factor. Regarding the subtests of decision making, verbal reasoning and situational judgment, Factor 1 and Factor 2 seemed to be capturing different dimension of these subtests thereby suggesting the need for revision of their items.

IV. DISCUSSION

This study aimed to design an aptitude test and investigate its psychometric properties among first-year medical students from two selected faculties of Medicine in Sri Lanka. The results indicate that overall internal reliability of the aptitude test was marginal (Cohen et al., 2018). However, the reliability of the UKCAT has been 0.86 or above overall and between 0.58 – 0.86 for the subtests (MacKenzie et al., 2016). These findings demonstrate the value of evaluating and reviewing aptitude tests locally. Further, reviewing test items that adversely affected the internal consistency of the subtests for clarity and improved alignment with the construct, could improve the overall validity. It is likely that the low number of test items has further contributed to the low-reliability score (Taber, 2018). The UKCAT consists of 200 test items, as opposed to the 50 items in the aptitude test employed by the current study. The item analysis indicated that most of the questions were easy, with 60% having a difficulty index of more than 70%. The discriminating capacity of the items was low (mean discrimination index was 19%). This could be attributed to the aptitude test being administered to students who have performed well in the advanced level examination and already have been selected to medical school.

There was a weak correlation between the aptitude test score and the Z score (0.27), indicating that the test assessed a significantly different aspect of cognitive skills compared to traditional AL examinations. This observation encourages exploring how the aptitude test may be incorporated into the selection process. Different selection authorities use aptitude scores differently (Greatrix & Dowell, 2020). Overall, among the universities which employ UKCAT, the weightage given for school leaver level academic performance in the selection process has increased, while the use of aptitude test scores as an absolute or relative cutoff to be achieved also has increased over the period of 2008-2015 (Greatrix & Dowell, 2020). However, empirical evidence on a best way of using UKCAT or UCAT in the selection process is not available (Greatrix & Dowell, 2020). Therefore, implementing a valid aptitude test as a supplementary test and exploring predictive validities of such a test across a national sample is likely to provide more robust evidence of the utility of aptitude tests in the Sri Lankan context.

This study used an aptitude test developed and deployed systematically and modelled on a widely tested international aptitude test format for medical school selection. The sample was representative of demographics of the national cohort of medical students and included all three major ethnic groups and students with a wide range of Z scores. However, the limited number of test items may have offset these strengths. The authors did not have access to participants’ raw scores at the advanced level examination, which may have provided a more appropriate measure of performance for statistical manipulation. In addition, the aptitude test was administered to students who have already been selected to medical school, whereas ideally, it should have been administered to medical school applicants. Despite the limitations, the aptitude test returned an acceptable internal reliability score, and further analysis demonstrated a way forward.

V. CONCLUSION

This study was the first time in the literature that an aptitude test was evaluated in medical education settings in the Sri Lankan context. Using an aptitude test to select students for medical school in Sri Lanka and similar settings, where admission processes have not diversified, is potentially valuable. However, further studies are necessary to establish evidence to support the adoption of such a test in the Sri Lankan and other similar resource-constrained contexts where prior educational achievement primarily informs selection decisions. In the Sri Lankan context, further evaluating the psychometric properties and the predictive validity of an improved aptitude test using a national cohort is recommended as the way forward.

Notes on Contributors

Dr. Sivapalan Sanchayan contributed to the development of the aptitude test, administering the test, collection of data, analysis and writing of the manuscript.

Dr. Sisira Dharmaratne contributed to the conceptualisation, study design, development of the aptitude test, data analysis and writing of the manuscript.

Ms. Shalika Pathirana took part in developing and administering the test, data collection and reviewing the manuscript.

Dr. Pavithra Godamunne contributed to the conceptualisation, study design, development of the aptitude test, data collection, analysis and revision of the manuscript.

Prof. Madawa Chandratilake took part in the conceptualisation, study design, data analysis and revision of the manuscript

Ethical Approval

Ethical clearance (P/223/12/2018) was obtained from the Faculty of Medicine, University of Kelaniya, Sri Lanka.

Data Availability

The datasets generated and/or analysed during the current study are not publicly available to preserve the confidentiality of the participants in this study but are available from the corresponding author on reasonable request.

Acknowledgement

We acknowledge the contribution of Ms. B. N. Jayasena (former lecturer, Department of Disability Studies, Faculty of Medicine, University of Kelaniya) and Ms. L. K. E. Chathurika (former lecturer, Department of Disability Studies, Faculty of Medicine, University of Kelaniya) in the development of the aptitude test. We also acknowledge the contribution of Dr. Rasiah Thayakaran (Research fellow in Health Informatics, Institute of Applied Health Research, University of Birmingham) in performing factor analysis.

Funding

The research received no specific grant from any funding agency.

Declaration of Interest

The authors have no conflicts of interest to disclose.

References

Bala, L., Pedder, S., Sam, A. H., & Brown, C. (2022). Assessing the predictive validity of the UCAT-A systematic review and narrative synthesis. Medical Teacher, 44(4), 401–409. https://doi.org/10.1080/0142159X.2021.1998401

Cohen, L., Manion, L., & Morrison, K. (2018). Research Methods in Education. Routledge.

de Silva, N. R., Pathmeswaran, A., & de Silva, H. J. (2004). Selection of students for admission to a medical school in Sri Lanka. The Ceylon Medical Journal, 49(3), 81–85. https://doi.org/10.4038/cmj.v49i3.3245

de Silva, N. R., Pathmeswaran, A., de Silva, N., Edirisinghe, J. S., Kumarasiri, P. V., Parameswaran, S. V., Seneviratne, R., Warnasuriya, N., & de Silva, H. J. (2006). Admission to medical schools in Sri Lanka: Predictive validity of selection criteria. The Ceylon Medical Journal, 51(1), 17–21. https://doi.org/10.4038/cmj.v51i1.1371

Dharmaratne, S., & Ponnamperuma, G. (2020). Selection for health professions education. In M. Chandratilake & A. Olupeliyawa (Eds.), 14 topics in health professions education (pp. 120–132). College of Medical Educationists, Sri Lanka.

Gliatto, P., Leitman, I. M., & Muller, D. (2016). Scylla and charybdis: The MCAT, USMLE, and degrees of freedom in undergraduate medical education. Academic Medicine, 91(11), 1498–1500. https://journals.lww.com/academicmedicine/Fulltext/ 2016/11000/Scylla_and_Charybdis__The_MCAT,_USMLE,_and_Degrees.16.aspx

Greatrix, R., & Dowell, J. (2020). UKCAT and medical student selection in the UK – what has changed since 2006? BMC Medical Education, 20(1), 1–12. https://doi.org/10.1186/s12909-020-02214-1

Greatrix, R., Nicholson, S., & Anderson, S. (2021). Does the UKCAT predict performance in medical and dental school? A systematic review. BMJ Open, 11, Article e040128. https://doi.org/10.1136/BMJOPEN-2020-040128

Hambleton, R. K., & Zenisky, A. L. (2010). Translating and adapting tests for cross-cultural assessments. In D. Matsumoto, F. J. R. van de Vijver (Eds.), Cross-cultural research methods in psychology (pp. 46–70). Cambridge University Press.

Hewage, S. N., Salgado, L. S. S., Fernando, G. M. O., Liyanage, P., Pathmeswaran, A., & De Silva, N. R. (2011). Selection of medical students in Sri Lanka: Time to re-think criteria? Ceylon Medical Journal, 56(1), 22–28.

MacKenzie, R. K., Cleland, J. A., Ayansina, D., & Nicholson, S. (2016). Does the UKCAT predict performance on exit from medical school? A national cohort study. BMJ Open, 6, Article e011313. https://doi.org/10.1136/BMJOPEN-2016-011313

McManus, I. C., Dewberry, C., Nicholson, S., & Dowell, J. S. (2013). The UKCAT-12 study: Educational attainment, aptitude test performance, demographic and socio-economic contextual factors as predictors of first year outcome in a cross-sectional collaborative study of 12 UK medical schools. BMC Medicine, 11, Article 244. https://doi.org/10.1186/1741-7015-11-244

McManus, I. C., Dewberry, C., Nicholson, S., Dowell, J. S., Woolf, K., & Potts, H. W. W. (2013). Construct-level predictive validity of educational attainment and intellectual aptitude tests in medical student selection: Meta-regression of six UK longitudinal studies. BMC Medicine, 11, Article 243. https://doi.org/10.1186/1741-701 5-11-243

Mettananda, D. S. G., Wickramasinghe, V. P., Kudolugoda Arachchi, J., Lamabadusuriya, S. P., Ajanthan, R., & Kottahachchi, D. (2006). Suitability of selection criteria as a measure of outcome of medical graduates: University of Colombo. The Ceylon Journal of Medical Science 49, 1–12.

Paniagua, M. A., & Swygert, K. A. (2016). Constructing written test questions for the basic and clinical sciences (4th ed.). National Board of Medical Examiners.

Paton, L. W., McManus, I. C., Cheung, K. Y. F., Smith, D. T., & Tiffin, P. A. (2022). Can achievement at medical admission tests predict future performance in postgraduate clinical assessments? A UK-based national cohort study. BMJ Open, 12, Article e056129. https://doi.org/10.1136/bmjopen-2021-056129

Patterson, F., Cousans, F., Edwards, H., Rosselli, A., Nicholson, S., & Wright, B. (2017). The predictive validity of a text-based situational judgment test in undergraduate medical and dental school admissions. Academic Medicine, 92(9), 1250–1253. https://doi.org/10.1097/ACM.0000000000001630

Patterson, F., Roberts, C., Hanson, M. D., Hampe, W., Eva, K., Ponnamperuma, G., Magzoub, M., Tekian, A., & Cleland, J. (2018). 2018 Ottawa consensus statement: Selection and recruitment to the healthcare professions. Medical Teacher, 40(11), 1091–1101. https://doi.org/10.1080/0142159X.2018.149 8589

Powis, D. A., Munro, D., & Bore, M. R. (2021). Is the UCAT appropriate for selecting undergraduate medical students? Medical Journal of Australia, 214(2), 77–78. https://doi.org/10.5694/mja2.50906

Prideaux, D., Roberts, C., Eva, K., Centeno, A., Mccrorie, P., Mcmanus, C., Patterson, F., Powis, D., Tekian, A., & Wilkinson, D. (2011). Assessment for selection for the health care professions and specialty training: Consensus statement and recommendations from the Ottawa 2010 conference. Medical Teacher, 33(3), 215–223. https://doi.org/10.3109/0142159X.2011.551560

Soemantri, D., Karunathilake, I., Yang, J. H., Chang, S. C., Lin, C. H., Nadarajah, V. D., Nishigori, H., Samarasekera, D. D., Lee, S. S., Tanchoco, L. R., & Ponnamperuma, G. (2020). Admission policies and methods at crossroads: A review of medical school admission policies and methods in seven Asian countries. Korean Journal of Medical Education, 32(3), 243–256. https://doi.org/10.3946/KJME.2020.169

Taber, K. S. (2018). The use of Cronbach’s Alpha when developing and reporting research instruments in science education. Research in Science Education, 48(6), 1273–1296. https://doi.org/10.1007/ s11165-016-9602-2

University Grants Commission. (2022). Sri Lanka university statistics 2022. Ministry of Higher Education. https://www.ugc.ac.lk/downloads/statistics/stat_2022/Chapter%202.pdf

University Grants Commission. (2023). Admission to undergraduate courses of the universities in Sri Lanka – 2022/2023. Ministry of Higher Education. https://www.ugc.ac.lk/downloads/admissions/Handbook_2022_23/student_handbook_english.pdf

Zimmermann, S., Klusmann, D., & Hampe, W. (2017). Correcting the predictive validity of a selection test for the effect of indirect range restriction. BMC Medical Education, 17, Article 246. https://doi.org/10.1186/s12909-017-1070-5

*S. Sanchayan
Faculty of Medicine,
University of Jaffna,
Adiyapatham Road,
Kokuvil West, Kokuvil, Sri Lanka
Email: sanch.s6@gmail.com,
sanchayan@univ.jfn.ac.lk