I. INTRODUCTION

Postgraduate summative examinations have an important role in progress as a medical professional. Success in high-stakes assessments have an impact on societal impressions, career shifts and social strata changes (Hamilton & Brown, 2005; Slavin et al., 2014). Examinations ideally measure the students’ competencies, but there are reasons to think that there are factors other than academic predictors.

Learning strategies and monitoring vary across the age groups (Vermunt, 1996). High school education and learning require mastering declarative knowledge, largely through elaboration and rote memory. College students require advanced learning strategies involving skilful metacognitive knowledge monitoring (MKM) and self-regulatory strategies (SRL) (Isaacson & Fujita, 2006). Metacognition is the higher order mental process of “thinking about one’s thinking”, wherein, there are two aspects, namely the ability to reflect on the quantity & quality of knowledge acquired (MKM) and the operational strategies in pacing and preparing for the challenge ahead (SRL).  Educational programs strive to facilitate this transition to become ‘adult learners’ (Ormrod, 2009).

 Learning in undergraduate medical education requires ‘critical reasoning’ to assimilate relevant clinical information and deduce differential diagnoses. Postgraduate learners, particularly in the health professions, need self-regulatory skills to pace their self-directed learning in the absence of regular formative assessments. Zimmerman (2008) asserted that postgraduate learners in higher education possibly achieve superior standards of self-regulation and motivation. However, the adaptations in learning strategies or metacognitive regulatory activities are not well described in the context of postgraduate health professions. Soh (2019) described a six-step approach in the pretext of the ‘ownership cycle’ for supporting postgraduate learners with difficulty.

O’Connor and Paunonen (2007) urged researchers to ‘use multiple predictors beyond intelligence, such as personality, motivation, and study habits when predicting academic achievement’ (Ray & Brown, 2015). We set to explore how we can improve on the understanding of attributes that could be collectively stated as ‘exam preparedness’ (Appendix 1). We proposed to define ‘exam preparedness’ as:

“ability of the learner to inculcate educational situation awareness, to gauge task difficulty, assess self-efficacy, modify one’s own learning behaviour, manage self and moderate it with resources and personal capabilities, so as to plan and operationalise a scheme/ construct in exam taking aptitude & skills, resulting in comprehensive exam success”.

It is known from literature that testwiseness, “a subject’s capacity to utilise the characteristics and formats of the test and/or the test taking situation” could impact the outcomes of the examinations (Millman et al., 1965; Sarnacki, 1981; Wahlstrom & Boersma, 1968; Watling & Ginsburg, 2019). Our proposition of understanding ‘exam preparedness’ is more than coachable test-taking strategy of testwiseness, but on a broader holistic front that looks at emotive, behavioural, self-regulatory perspectives and preparedness for a task in anticipation.

II. RELEVANCE OF THE STUDY

We need ways to measure ‘exam preparedness’ because, first, it could help the postgraduate learners to identify major deficiencies in being task focussed and hone self-regulatory strategies. Second, it would help teachers to enhance support or ‘scaffolding’ that suits the needs of individual students.  The concept ‘exam preparedness’ is not well constructed or described in literature. In the context of this research, we identify ‘exam preparedness’ to be everything but the student’s level of knowledge or competence. The cognitive determinants, like grade point average (GPA), are strong predictors of college success (Conard, 2006; Sladek et al., 2016; Zhou et al., 2014). In the postgraduate medical education context, where the academic capabilities are comparable and are well matched during their selection into residency, there are other non-academic attributes that could predict exam success.

The prediction of exam performance by factors such as achievement goal orientation, self-regulation, (Lucieer et al., 2016) conscientiousness (Brazdău & Mihai, 2011; Colthart et al., 2008), metacognition and meta-comprehension have been well described (Cook et al., 2011; Dunlosky & Lipko, 2007; Pintrich & De Groot, 1990; Zimmerman, 2000). The inter-relationships among these attributes are complex and not well studied. Our work was designed to help understand postgraduate students’ metacognitive, self-regulation, conscientiousness, and self-efficacy perceptions to aid in their learning.

III. METHODOLOGY

A. Study Design

The study involved a sequential explanatory mixed methods research design (McKim, 2017). The attribute of ‘exam preparedness’ was explored using existing predictors of academic success such as metacognition, self-regulation, self-efficacy and conscientiousness quotient.  Data obtained by a questionnaire assessing pre-examination attributes were compared with the students’ reflections through focus group discussions, after the formal declaration of results. The mixed methods design consisted of ‘sequential approach’ where the questionnaire data collected prior to the high-stake examination provided inputs for quantitative analysis of the predictors of exam outcomes. This was followed by qualitative exploration of themes that emerged through focus group discussions. The themes summarised from the first focus group discussions were used to enhance the richness of second focus group discussion (Hennink, 2013).

B. Procedure

Anaesthesia postgraduate learners who appeared for the primary anaesthesia examinations were invited for participation in the study. These were high-stake summative examinations and mandatory for all postgraduates, limiting progression to senior years in residency and subsequent accreditation to a specialist. Informed consent was obtained for voluntary participation and confidentiality was ascertained for information on participant profiles and sensitivity of information about personal learning attributes. After approval by the institute review board, a pilot study was conducted to enhance validity of the questionnaire.

All the study participants who consented (30 out of 42 who appeared for exams) filled the pre-examination questionnaire, one month before the examinations. After completion of examinations and declaration of results, focus group interviews were conducted among students who passed the exam to explore further insights. Candidates who had not passed the examinations were excluded from the second part of study (as per Institutional Review Board (IRB) stipulations). The ‘passers’ were allocated into two sub-groups: the first-time passers or the ‘acers’, who succeeded in their first attempt and the ‘non-acers’, who were postgraduates with previous unsuccessful attempt(s) and have passed the examinations in this attempt (Figure 1).

Figure 1. Procedure of the study in chronological order

The main themes emerging from the two focus groups are summarised in Table 3. All focus group discussion candidates agreed that failure was chiefly a result of ‘gross lack in knowledge’ and ‘un-preparedness in many ways’. The focus group discussions concluded with the passers’ advice for success of future candidates and stated that:

“adhering to syllabus & plan of study, technique suited to examiner needs and not shying away from shame of mock exams, minimizing luck factor, possessing good social attributes (viva skills), persevering with study plans and timing of rejuvenation was the key to success”.

The focus group discussions supported this concept that ‘the best way of passing the exam was to dare the shame of practice viva’ and not to shun away from faculty feedback on performance in mock exams. The ‘passers’ had spent twice the time and effort in practice exams and revisions when compared to the ‘fail’ group (self-regulation). 

IV. DISCUSSION

Our study showed that the candidates who passed, monitored their learning well by choosing to stay in groups and ‘stuck together’ by helping one another. The focus group discussions showed that they opted to adopt this mode of studying because the task was difficult and required constant motivation and peer support. This is in accordance with literature that claims the usefulness of peer support and feedback (de la Cruz et al., 2015; Dochy et al., 1999; Lerchenfeldt et al., 2019). What is vital was the ability of the candidates to assess what components of the exam content requires concentrated self-learning, like memorization, and differentiate it from study materials that require learning in groups and further elaboration.

Our study showed that the ‘passers’ were mindful of how they took time off to study or rejuvenate. The ‘passers’ tend to procrastinate three times lesser than the ones who failed. More so, a subgroup analysis showed that the ‘non-acers’ or the previous failed candidates never put away their schedules and stayed pursuing a timetable-based study (0% procrastination). The focus group discussions reinforced the fact that the ‘passers’ felt that there were loads of enormously difficult basic science content to be covered, such as applied physics and pharmacology, and time was limited, requiring further organisation and seamless execution of the study plan. 

Self-efficacy evaluations showed no clear difference when comparing the groups of ‘passers’ and ‘failed’ candidates. However, the subgroups analysis showed that the ‘non-acers’ were twice as confident as the rest. The focus group discussions confirmed that the non-acers, having had the experience of failing before, ‘had a clear understanding of the difficulty of task and had commenced their study early in a programmed timetable-based manner with specific feedback and personalised attention from faculty supporting them’.

Cilliers et al., (2012) modelled the pre-assessment learning effects of high-stakes assessments and postulated ‘efficacy’ as an adaptation in the learner in anticipation of the task. Our ‘non-acers’ had been through the actual task difficulty, having failed earlier, had higher self-efficacy and now programmed their study well, expending more time on task. The lower self-efficacy might mean that this is an important lead for the faculty, in how they counselled and supported future candidates that are to be trained for high-stakes examinations (Lucieer et al., 2016).

A. ‘The Exam Ready Candidate’

Our study showed that passers had higher degree of self-regulation, conscientiousness, and metacognition. The focus group discussions further affirmed the information about the relationship between the various attributes and the students’ perceptions of ‘exam preparedness’. They felt that the candidates who ‘breeze through exams’, in other words, the exam ready, ‘possessed exam oriented (goal oriented) flexibly tied into study plans (metacognition), were expressive in viva (self-efficacy), delivered the necessary content well (conscientiousness)’ in addition to ‘diligently apportioning time, effort & resources (effort regulation), knowing to map what examiners want & selectively consolidating preparations (study strategy)’.

B. Understanding the High Failure Rates

The anaesthesia postgraduate primary examinations had a dismal 42.9% pass rate while undergraduate examinations had more than 90% pass rates. This is intriguing and highlights the complex relationships between the poor predictability of undergraduate academic scores and the influences of other paradigms in postgraduate exam performances (Division of Graduate Medical Studies [DGMS], 2013). The focus group discussions helped us understand this phenomenon better. The residents felt that ‘unlike undergraduate exams, the postgraduate examinations required lot more application of work-study balance and work-life balance’. During the undergraduate years, the students felt that the sole focus was to learn and perform in examinations, with lot more protected time during the day and all the weekends were dedicated to study. Although they require a particular level of academic competence to become a postgraduate, there is more to explore when it comes to exam success such as ‘work-life’ and ‘work-study’ balance (Klomegah & Yao, 2007; Rau & Durand, 2000).

C. Practical Implications

Through this study, we identify the presence of attributes such as metacognitive self-regulation, conscientiousness, and self-efficacy. We now know that candidates who score well in these component-attributes tend to pass the examinations. We propose that achieving an element of predictability will be a good lead for:

1. The educators to identify the ‘candidates with difficulty’ and delegate special care and personalised attention to them, while it is feasible and not too late.

2. The exam candidates themselves to have a ‘reality check’ on where they stand and what would be the best way ahead: peer learning, consolidated study, goal orientation etc.

D. Scoring Exam Preparedness and Planning Scaffolding

Candidates with ‘poor conscientiousness score’ could be scheduled to ‘receive more structured assistance through study groups, frequent deadlines, shorter assignments, group assignments and clearly defined learning goals’ (Kappe & Van Der Flier, 2012). This was put to light in our focus group discussions when one of the candidates who was unsuccessful earlier, but passed this time (non-acer) stated that:

“what made all the difference in this exam was that faculty and peers sat next to me in a quiet room, gave me a short study topic, made me do a SAQ, then a short viva on it, then gave me a critical feedback on how to improve. This was very encouraging, and I felt the intention of the faculty and peers were to help me and not to embarrass me on how unprepared I was. This inspired me to pass this ordeal this time…”

Implementing a composite tool to assess ‘exam preparedness’, we propose, would help the learners and teachers to skim for predictable factors that influence the chances of success. Incorporating a system of ‘scaffolding’ would help in early guided learning towards exam success. This, we feel, is particularly imperative when educational programs deploy high-stakes single summative exams. Self-efficacy Survey (SES) was described as a measure for identifying disability in adaptive attitudes and disfunctionality (Panc et al., 2012). Using such a measure could identify ‘trainees with difficulty’ and guide us to channelise our efforts and resources for those who need it the most. Kandaswamy and Anbarasi (2014) suggested early identification of ‘gifted under achievers’ and successfully devised a structured program for psychological support, demonstrating a reduction of dropouts for dental undergraduates.

E. Limitations & Reflections

The chief limitation our study was the aspect that attributes like metacognitive self-regulation, conscientiousness and self-efficacy have considerable overlapping concepts and it was difficult to compartmentalise these themes when interpreting the qualities of a learner. Our study population was limited to anaesthesia postgraduates’ primary examination results from one country. A fair comparison of other specialty postgraduate examinations could not be accomplished in our context owing to variation in exam task difficulty, timings, and patterns of assessments.

Of the 42 who appeared for exams only 30 consented to fill the questionnaire. Only one among the 12 students who chose not to fill the questionnaire passed the exam. This could have been another area of rich information. Did the learners know that they were not prepared at all or was the title labelling students “exam smart” made them to excuse themselves off the study? It is possible that respondents gave ‘socially acceptable answers’ when the questionnaire was given, especially so when our study is titled to explore how ‘prepared/ ready’ they were or predicting their chance to pass. 

There are possibilities that testwiseness could have contributed to exam success and were not explored in the focus group discussions (Millman et al., 1965; Sarnacki, 1981; Wahlstrom & Boersma, 1968; Watling & Ginsburg, 2019). Though the present study involved short answer questions and viva-based examinations that tend to report less of testwiseness, as compared to multiple choice questions, future research could appraise this concept and consider ways to address that in the methodology.

1) Validating a questionnaire for ‘exam preparedness’- A lead for the future: Our model of ‘exam preparedness’ was proposed to address the complex learning and myriad of factors that impact postgraduate academic performance (Appendix 1). A well validated composite score or scale on ‘exam preparedness’ can be a yardstick for ‘assessment for exam-readiness’, thereby planning the allocation of resources and faculty time and effort. Future studies could evaluate when it would be an appropriate time for evaluating the score (1 or 3 or 6 months before exams). It should not be too near to the examinations, lest it could be too late for any usefulness of its predictability.

V. CONCLUSION

There were similarities in attributes that characterise passing such as time scheduling, plan of study, group and peer support, goal oriented selective mastery learning and effort regulation to task difficulty. The two groups of passers also identified that the chief cause of failure to be ‘gross lack of knowledge’ and ‘unpreparedness’. Implementing a composite tool to assess ‘exam preparedness’ we propose, would help the learners and teachers to skim for predictable factors (metacognitive self-regulation, self-efficacy, conscientiousness) that influence the chances of success.

As teachers, we would agree that the students who are relatively more prepared, tend to seek and receive more faculty support, while those who are trailing, continue to distance themselves with the divide getting more distinct nearer to the examinations. Identifying this discrepancy early, while predicting and preventing failure in high-stakes examinations, we propose, needs in-depth understanding of ‘exam preparedness’. Using the ‘exam preparedness’ scale might help to identify the postgraduates with academic difficulty, thereby offering a support system, wherein we don’t lose some ‘Good Samaritans’ who are just not ‘exam ready’.

Notes on Contributors

Dr Balakrishnan Ashokka is an Anaesthesiologist & Educationalist with special interest in Simulation-based postgraduate education at National University Health System, Singapore. He conceived the idea of the study, performed literature search, conducted the study, and drafted the manuscript of the study. He agrees fully to the final version of manuscript.

Prof Lee Tat Leang has been involved in Undergraduate and Postgraduate teaching and examination processes over 30 years at the National University of Singapore. He provided support during conceptualisation, discussion of results and creation of the manuscript. He fully agrees to the final version of the manuscript.

Dr Daniëlle M.L. Verstegen has a background in Cognitive Science and Instructional Science. She currently leads an e-learning group in the Department of Educational Development and Research, FHML, Maastricht University. She supervised the conduct of the entire study, qualitative analysis and agrees fully to the final version of the manuscript.

Ethical Approval

The National University of Singapore, Institute Review Board (NUS-IRB) provided the ethics committee approval for the conduct of study on passers of the examination (NUS-IRB approval number NUS 1946, reference code 13-276). No audio or video recording or discussions with failed students was permitted as per the directorate’s policy of research on exam candidates. 

Data Availability

The study data with summarised, deidentified data synthesis is provided in tables 1-3 in this manuscript. The NUS-IRB and DGMS had provided approval for the study with a clause for private storage of study data with authors’ password-protected workstation. No permission was granted for open access platforms, as the data contained sensitive information about failed candidates.

Acknowledgement

The authors acknowledge the anaesthesia postgraduates who participated in this study, offering their valuable time and effort.

Funding

No funding was received for the conduct of the study.

Declaration of Interest

The authors declare that there are no financial or non- financial competing conflicts of interests.

References

Brazdău, O., & Mihai, C. (2011). The consciousness quotient: A new predictor of the students’ academic performance. Procedia-Social and Behavioral Sciences, 11, 245-250. https://doi.org/10.1016/j.sbspro.2011.01.070

Cilliers, F. J., Schuwirth, L. W., & Van der Vleuten, C. P. (2012). Modelling the pre‐assessment learning effects of assessment: Evidence in the validity chain. Medical Education, 46(11), 1087-1098. https://doi.org/10.1111/j.1365-2923.2012.04334.x

Colthart, I., Bagnall, G., Evans, A., Allbutt, H., Haig, A., Illing, J., & McKinstry, B. (2008). The effectiveness of self-assessment on the identification of learner needs, learner activity, and impact on clinical practice: BEME Guide no. 10. Medical Teacher, 30(2), 124-145. https://doi.org/10.1080/01421590701881699

Conard, M. A. (2006). Aptitude is not enough: How personality and behavior predict academic performance. Journal of Research in Personality, 40(3), 339-346. https://doi.org/10.1016/j.jrp.2004.10.003

Cook, D. A., Thompson, W. G., & Thomas, K. G. (2011). The motivated strategies for learning questionnaire: Score validity among medicine residents. Medical Education, 45(12), 1230-1240.

Coutinho, S. A., & Neuman, G. (2008). A model of metacognition, achievement goal orientation, learning style and self-efficacy. Learning Environments Research, 11(2), 131-151.

Credé, M., & Phillips, L. A. (2011). A meta-analytic review of the motivated strategies for learning questionnaire. Learning and Individual Differences, 21(4), 337-346. https://doi.org/10.1016/j.lindif.2011.03.002

de la Cruz, M. S. D., Kopec, M. T., & Wimsatt, L. A. (2015). Resident perceptions of giving and receiving peer-to-peer feedback. Journal of Graduate Medical Education, 7(2), 208.

Dunlosky, J., & Lipko, A. R. (2007). Metacomprehension: A brief history and how to improve its accuracy. Current Directions in Psychological Science, 16(4), 228-232. https://doi.org/10.1111/j.1467-8721.2007.00509.x

Division of Graduate Medical Studies. (2013). The primary anaesthesia MMed examinations. https://medicine.nus.edu.sg/dgms/anaesthesiology.htm

Dochy, F. J. R. C., Segers, M., & Sluijsmans, D. (1999). The use of self-, peer and co-assessment in higher education: A review. Studies in Higher education, 24(3), 331-350.

Hamilton, L., & Brown, J. (2005). ‘Judgement day is coming!’: young people and the examination process in Scotland. Improving Schools, 8(1), 47-57. https://doi.org/10.1177/1365480205049339

Hennink, M. M. (2013). Focus group discussions. Oxford University Press.

Isaacson, R., & Fujita, F. (2006). Metacognitive knowledge monitoring and self-regulated learning. Journal of the Scholarship of Teaching and Learning, 6(1), 39-55.

Kandaswamy, D., & Anbarasi, K. (2014). Gifted underachievers in dental education- An action research report. Medical Education, 48(s2), 1-19. Abstract number 49. https://doi.org/10.1111/medu.12579

Kappe, R., & Van Der Flier, H. (2012). Predicting academic success in higher education: What’s more important than being smart?. European Journal of Psychology of Education, 27(4), 605-619. https://doi.org/10.1007/s10212-011-0099-9

Klomegah, R. Y. (2007). Predictors of academic performance of university students: An application of the goal efficacy model. College Student Journal, 41(2), 407-415.

Lerchenfeldt, S., Mi, M., & Eng, M. (2019). The utilization of peer feedback during collaborative learning in undergraduate medical education: A systematic review. BMC Medical Education, 19(1), 1-10.

Lucieer, S. M., Jonker, L., Visscher, C., Rikers, R. M. J. P., & Themmen, A. P. N. (2016). Self-regulated learning and academic performance in medical education. Medical Teacher, 36(6), 585-593.  https://doi.org/10.3109/0142159X.2015.1073240

McKim, C. A. (2017). The value of mixed methods research: A mixed methods study. Journal of Mixed Methods Research, 11(2), 202-222.

Millman, J., Bishop, C. H., & Ebel, R. (1965). An analysis of test-wiseness. Educational and Psychological Measurement, 25(3), 707-726.

O’Connor, M. C., & Paunonen, S. V. (2007). Big Five personality predictors of post-secondary academic performance. Personality and Individual Differences, 43(5), 971-990. https://doi.org/10.1016/j.paid.2007.03.017

Ormrod, J. E. (2009). Human Learning. (5^th ed) Chapter 12. Upper Saddle River, NJ: Pearson Education.

Panc, T., Mihalcea, A., & Panc, I. (2012). Self-efficacy survey: A new assessment tool. Procedia-Social and Behavioral Sciences, 33, 880-884. https://doi.org/10.1016/j.sbspro.2012.01.248

Pintrich, P. R. (2000). Multiple goals, multiple pathways: The role of goal orientation in learning and achievement. Journal of Educational Psychology, 92(3), 544.

Pintrich, P R. (2004). A Conceptual Framework for Assessing Motivation and Self-Regulated Learning in College Students. Educational Psychology Review, 16(4), 385-407. http://hdl.handle.net/2027.42/44454

Pintrich, P. R., & De Groot, E. V. (1990). Motivational and self-regulated learning components of classroom academic performance. Journal of Educational Psychology, 82(1), 33.

Rau, W., & Durand, A. (2000). The academic ethic and college grades: Does hard work help students to “make the grade”? Sociology of Education, 73(1) 19-38. https://doi.org/10.2307/2673197

Ray, R., & Brown, J. (2015). Reassessing student potential for medical school success: Distance travelled, grit, and hardiness. Military Medicine, 180(4, Suppl), 138-141.

Roberts, B. W., Jackson, J. J., Fayard, J. V., Edmonds, G., & Meints, J. (2009). Conscientiousness. In M. R. Leary & R. H. Hoyle (Eds.), Handbook of individual differences in social behavior (p. 369–381). The Guilford Press.

Sarnacki, R. E. (1981). The effects of test-wiseness in medical education. Evaluation & the Health Professions, 4(2), 207-221.

Schraw, G., & Dennison, R. S. (1994). Assessing metacognitive awareness. Contemporary Educational Psychology, 19(4), 460-475.

Soh, J. Y. (2019). The ownership cycle and self-regulated learning. The Asia Pacific Scholar, 4(1), 65-68. https://doi.org/10.29060/TAPS.2019-4-1/CS2047

Sladek, R. M., Bond, M. J., Frost, L. K., & Prior, K. N. (2016). Predicting success in medical school: A longitudinal study of common Australian student selection tools. BMC Medical Education, 16(1), 1-7.

Slavin, S. J., Schindler, D. L., & Chibnall, J. T. (2014). Medical student mental health 3.0: Improving student wellness through curricular changes. Academic Medicine, 89(4), 573.

Vermunt, J. D. (1996). Metacognitive, cognitive and affective aspects of learning styles and strategies: A phenomenographic analysis. Higher Education, 31(1), 25-50.

Wahlstrom, M., & Boersma, F. J. (1968). The influence of test-wiseness upon achievement. Educational and Psychological Measurement, 28(2), 413-420.

Watling, C. J., & Ginsburg, S. (2019). Assessment, feedback and the alchemy of learning. Medical Education, 53(1), 76-85.

Zhou, Y. X., Zhao, Z. T., Li, L., Wan, C. S., Peng, C. H., Yang, J., & Ou, C. Q. (2014). Predictors of first-year GPA of medical students: A longitudinal study of 1285 matriculates in China. BMC Medical Education, 14(1), 1-9.

Zimmerman, B. J. (2000). Self-efficacy: An essential motive to learn. Contemporary Educational Psychology, 25(1), 82-91.

Zimmerman, B. J. (2008). Investigating self-regulation and motivation: Historical background, methodological developments, and future prospects. American Educational Research Journal, 45(1), 166-183. https://doi.org/10.3102/0002831207312909

*Dr Balakrishnan Ashokka
Department of Anaesthesia,
National University Hospital,
5 Lower Kent Ridge Road
Singapore 119074.
Tel: +6597118855
Fax: +6567775702
Email: ashokkab@gmail.com