Number of Citations: 2

Published online: 7 January, TAPS 2020, 5(1), 46-53
DOI: https://doi.org/10.29060/TAPS.2020-5-1/OA2080

Jaime L. Pacifico¹, Julie Anne S. Villanueva¹, Sylvia Heeneman² & Cees van der Vleuten²

¹Internal Medicine, De La Salle Medical and Health Sciences Institute, Philippines; ²Maastricht University, The Netherlands

Abstract

Practice Highlights

• Any form of assessment activity serves as a stimulus and provokes an educational response.
• Assessment in residency motivated the trainees to study and fostered an active learning attitude.
• Perceptions of credibility, fairness and commitment of assessors to trainees’ welfare matter.
• Patients, engagement with consultants and a supportive environment contributed to motivation.

I. INTRODUCTION

Any form of assessment activity will result in an educational response, however, the unpredictability of this response requires careful monitoring to realise the desired educational outcomes and to recognise the unwanted effects (van der Vleuten, 1996). Assessment can influence how a student learns through the content, design and scheduling and regulatory structure of the assessment program (Schuwirth & van der Vleuten, 2010). When does assessment itself become a barrier to learning and when does the response to assessment promote learning? Holmboe, Durning and Hawkins (2018) assert that the responsibility lies on the program administrator of any assessment program to conduct a systematic review on the potential consequences of any assessment exercise, whether positive or negative, regardless whether the examination is conducted in the classroom or at a national level. Through the years, decisions about method of assessment are primarily based on consideration of its validity and reliability (Norcini & McKinley, 2007). But is this indeed the only attribute of an assessment procedure that is of importance? An in-depth and systematic review of any assessment program on the potential consequences compels that the perceptions of the trainees are included.

When non-medical higher education students were asked about assessment, they talked about fairness instead of validity. It was clear that fairness as alluded to by students, is a representation of how the educationalist defines validity (Sambell, McDowell, & Brown, 1997). In a review involving non-medical higher education students, it was shown that approaches to learning are strongly influenced by the students’ perceptions of the assessment methods (Segers, Nijhuis, & Gijselaers, 2006; Struyven, Dochy, & Janssens, 2005). The same relationship between assessment and approach to learning is echoed in studies involving nursing students in clinical and classroom settings (Leung, Mok, & Wong, 2008; Tiwari et al., 2005). Among medical students, it has also been shown that students would adapt their learning strategies to the perceived requirements of the evaluation (Newble & Jaeger, 1983). These studies substantiate the well quoted maxim that assessment drives learning, although as McLachlan (2006) argues the oft-repeated axiom ‘assessment drives learning’ is misleading and is more complicated, for example, different students are motivated by different reasons and assessment does not influence learning in all contexts.

In postgraduate medical education, whether assessment is a driver of learning and how it directs learning is less established. The continuing challenge for the program directors and clinical faculty is to understand the relationship between assessment and learning and to be able to promote learning as intended (van der Vleuten, Schuwirth, Scheele, Driessen, & Hodges, 2010). In postgraduate medical settings, it is of paramount importance to evaluate the educational effects of assessment, because of the high stakes and risks involved. Although such is also expected in undergraduate medical education, according to Holmboe, Hawkins and Huot (2004), “residency is the last structured experience to ensure that young physicians have sufficient clinical skills” (p. 874) thus the urgent and serious need to influence learning through assessment methods done in clinical training.

The remaining intriguing question in postgraduate medical training is, how do the trainees perceive their current assessment practices and how do these facilitate their learning? Given this research question, we conducted this study to: 1) look into the perceptions of postgraduate trainees undergoing residency training as to how assessment practices influence their motivation to accumulate knowledge and attain the competence levels expected of a specialist, and 2) to determine if there are identifiable conditions or factors associated with assessment practices that can facilitate or deter learning.

II. METHODS

A. Setting

The study was done among residents of the internal medicine (IM) and paediatrics departments at De La Salle University Medical Center (DLSUMC) in the Philippines. DLSUMC serves as the academic institution of De La Salle University College of Medicine. Undergraduate medical education in the Philippines requires a bachelor’s degree and postgraduate medical education such as residency is done after passing the national licensure examination. The duration of residency in IM and paediatrics is three years and is a prerequisite to further training such as adult or paediatric cardiology. Evaluation of the residents included assessment of their knowledge and skills which were done through summative written examinations, mostly multiple choice questions, and Objective Structured Clinical Examinations (OSCEs), attitudes were assessed through a Likert-scale questionnaire. The OSCEs consisted of 12 stations. The scope of the written examinations and the OSCEs include the presentation, diagnosis, and management of the major and more common IM diseases. These assessment methods are done at least twice a year, throughout the three years of training. The examiners are the consultants belonging to the department of IM. During the time this study was done, there was no assessment that involved direct observation of the trainees other than through OSCEs.

B. Data Collection

A convenient sampling was done of second- and third-year residents in both departments since we needed trainees who have already experienced at least one year of training and have experienced several assessment processes. Twenty residents were interviewed for this study, ten males and ten females. Fourteen residents were from the department of IM and 6 were from the department of paediatrics. Seven were third-year residents and 13 were second-year residents during the time of the study. The interviews were conducted by a single person, who was not connected to the current training program of any of the departments. The interviews were done in both English and Filipino and the interviews were later transcribed in English. The interviews lasted from 45 minutes to 1 hour per subject. The residents were notified the interview was voluntary. A semi-structured interview was done using a published guide initially (Dijksterhuis, Schuwirth, Braat, Teunissen, & Scheele, 2013). However, consistent with the iterative nature of grounded theory and utilising constant comparison the interview questions were modified as initial results became available, which informed the succeeding interview guide.

C. Data Analysis

A grounded theory was used to explore how postgraduate trainees in IM and paediatrics perceived assessment of their knowledge and skills during training, and the contributions of these assessments to their learning and their clinical performance (Charmaz, 2014; Glaser & Strauss, 1967; Watling & Lingard, 2012). Grounded theory is an inductive method which allows theories to emerge from the data gathered. Simultaneous data collection and analysis were done which is characteristic of grounded theory. All the interviews were transcribed and during the early data analysis, some of the concepts that emerged from the early transcripts were utilised to guide in the ensuing interviews. Constant comparative method was employed throughout the data analysis. Codings were subsequently organised into concepts and elevated to categories. There was consultation and comparison between JLP and JAS with regard to the codes and analysis of the collected data. During the coding process, memos were written that elaborated on the different codes. Data collection was stopped when saturation was attained, by saturation we mean the data were leading to recurring themes and there was adequate data to support a theory that is comprehensive and credible. In addition, for saturation, our emphasis was on the quality of data rather than its frequency (Morse, 1995). Cognizant of the effects of the researchers in the data collection and analysis and subsequent creation of concepts and knowledge, the background of the different authors are as follows: JLP is a practising internist and cardiologist, and has been a faculty in the college of medicine for many years. He was former chair of the department of IM. JSV is a recent graduate of IM training. SH is a biologist with an educational background. CvdV has training in psychology and psychometrics with many years of engagement in medical education and medical education research. This is the second study together of JLP, SH and CvdV.

III. RESULTS

There was a general positive acceptance among the trainees regarding the role of assessment in their training, as discussed under ‘assessment and its impact’, below. There are two important categories that came out from our data, we called these categories: the mediating factors and stronger motivators. We define mediating factors as prerequisites for learning, these preconditions were necessary for assessment to be meaningful to the trainees. The second category we termed the ‘stronger motivators’. These were situations/conditions which we discovered to contribute to the motivation of the trainees to learn, they were equally as important as the actual assessment in influencing the trainees positively and driving them to strive to be better clinicians. Table 1 summarises the results.

Table 1. Summary of results

A. Assessment and Its Overall Impact

The trainees agreed that assessment had a positive influence on both their clinical education and performance.

“Assessment improves knowledge of a case, lack of knowledge of a case motivates me to read more in the same way that poor performance prompts us, residents, to exert efforts to improve.”

(IM-1)

One perceived effect of assessment was that trainees were more conscious of their actions especially at the bedside and this improved clinical performance. The residents agreed that assessment positively influenced their learning because:

“I was forced to study and ask questions, learned to prioritise and manage cases, and even one’s personality tended to improve.”

(IM-4)

Assessments improved clinical performance as residents were inclined to study more, facilitating that when they would encounter these cases in the future, they would be more prepared to manage such cases. The OSCE had a positive effect on clinical performance by testing the confidence of the residents.

“You are face to face with the consultant during the OSCE, so it will test your confidence. You may know it theoretically but anxiety can get in the way… at least here they practice in practicals how to explain well even when you are with consultants or seniors.”

(P-1)

For some residents, assessment had a constructive impact on their day to day duty by being aware of what mistakes to avoid and by knowing which patients to give more attention to. Many of these beneficial effects followed on the awareness and what was learned through mistakes in the past, which were facilitated through the assessment practices.

B. Mediating Factors

Although there was a general agreement among the participants in the study that an assessment program had beneficial educational and clinical performance effects, there were certain minimum conditions that were perceived as necessary so that the assessment would be meaningful and would have an impact. These mediating factors that facilitated learning and had an impact on the clinical performance are credibility, fairness, interpersonal relations (between the trainees and supervisors), and commitment.

As for credibility, if the assessment was not perceived as valid, it had no influence on resident learning and performance. The educational role of assessment was readily lost if there was uncertainty regarding the validity of the evaluation.

“If I do not agree with the evaluation it will not have an effect on my performance.”

(P-2)

There was also a perception among the trainees that receiving an unfair assessment would demotivate a trainee, wherein the trainee stopped to try hard since his/her efforts were not properly judged.

“If you receive a wrong assessment… you lose the motivation to pursue to learn, the work becomes very tedious and a lot of your energy is spent being anxious.”

(IM-4)

The evaluator had to have more than superficial knowledge of the residents they were evaluating. Several trainees expressed that they doubted the ability of their consultants to evaluate them effectively in the absence of sufficient interaction between the consultants and the trainees. Another perception was that the consultants did not really see them at work, at the emergency room for example, yet were asked by the department to evaluate them, which obviously caused concerns on the validity of their evaluation.

The last mediating factor was the perceived commitment of consultants towards their role as evaluators. Some expressed doubt about the commitment of the consultants who were evaluating them because of the perception that they were spending limited time in performing their role as evaluators. Such perception tended to undermine the positive effects of their presence and participation in the training of the residents.

“Our consultants who evaluate us who are perceived to be sincerely concerned with us as trainees, have more impact in clinical performance.”

(IM-2)

C. Stronger Motivators to Study and Perform Optimally

Although many trainees thought that assessment had an impact on their learning and performance, we identified some peripheral factors inherent in the training program which served as a catalyst, increasing the motivation of the residents, these effects were either independent of the assessment process or in conjunction with it. These factors or conditions that promoted a stronger incentive for the residents were: interesting and/or new cases, concern for patient’s welfare and/or outcomes, engagement with consultants and a supportive environment.

D. Interesting and/or New Cases

This was the strongest motivation as expressed by the trainees, both as a reason to study or to improve their clinical performance. Exposure to new cases motivated more than examinations or grades to study. New cases refer to diseases or conditions they have not seen before or rarely see such that encountering these motivates them more, giving them the needed confidence to handle such cases better in the future.

“I am more driven by the cases I see and I am happy when faced with something I do not know.”

(IM-4)

E. Patient’s Welfare or Outcome

Many residents found a strong incentive to study harder during their rotation in a particular ward out of a sincere desire to contribute significantly in the recovery and successful management of the patients they handled or encountered. This consideration for a good outcome of their patients created a strong desire to learn more about the case. A trainee expressed that his/her goals had significantly changed from pleasing their consultants as a first year resident to pleasing their patients as a third year resident.

“But on day-to-day duties my motivation already shifted since I am more confident now. During my shift, my goal is on how I can help the patient get better, rather than how I can please the consultants.”

(IM-4)

F. Engagement with the Consultants

It was a prevailing view of many of the trainees that interactions with consultants increased motivation to study or to perform better.

“Interaction with consultant is the best reinforcement to learning and has impact on daily duties.”

(IM-3)

The intermittent moments of one-on-one interaction between the trainee and the consultant was a much valued teaching-learning opportunity for the residents, was something residents awaited and had the impact of creating a strong incentive for them to study. These interactions were a critical affirmation for the trainees and had a crucial influence on facilitating the development of competence they need as future clinicians.

G. Supportive Environment

A learning environment that encouraged the trainees to try to excel was crucial so that the trainees would be uplifting each other in terms of continuously improving their knowledge and skills. The trainees acknowledged that the absence of such a kind of environment would not foster learning. As one trainee said:

“…in the presence of such an environment I will try to do good not because I have been reprimanded but because I am inspired by my co-trainees.”

(IM-2)

The residents conveyed that their departments must make an effort to create an atmosphere that promotes excellence and maintain an unmistakable uplifting standard which would push the residents to aspire to work hard. Also, an environment where residents feel there is ‘respect for everyone’ fosters motivation among the trainees to study more and perform well.

IV. DISCUSSION

This study explored, through the experiences of resident trainees, how assessment influenced their motivation to accumulate knowledge and skills and attain the competence levels expected of a specialist. We limited our study to the general perception of the residents with regard assessment and how it impacted their learning and not into the specific aspects of their clinical performance. Our results showed that assessment in general positively affected the residents as they were conscious of their actions, and they were inclined to study more.

Our results also revealed there are factors we considered as the catalyst, indirectly related to the assessment process that influenced their desire to learn and improve clinical performance.

There are two messages from this study. Firstly, it was clear there is no single assessment factor that promoted learning. An interplay of several elements within an assessment process ultimately promoted learning – these were patients the residents encounter and concern for their welfare, engagement with the consultants and a supportive environment. The subtleties among these factors and how they interacted with the residents are critical in promoting learning of the residents. Secondly, it is noteworthy to mention that our findings suggest of a considerable degree of self-directed learning (SDL) among the residents who participated in our study, even in the absence of a formal structure on SDL within the departments. Although used interchangeably SDL and self-regulated learning (SRL) are different. SDL refers to the general approach a learner adopts for his own learning whereas SRL is focused on the important learning processes (Gandomkar & Sandars, 2018). In our study, a trainee’s desire to be a good specialist is a reason that was a predominant motivation, which was a key driver of SDL. In the process through SRL, the trainees utilise several cognitive and metacognitive processes to guarantee that the intended learning is met (Gandomkar & Sandars, 2018). The phrase “the self is a bigger motivation to study” from one of the participants, typified the aspiration of the residents to reach another level in their medical education.

Several studies have cited patient care and implicitly patient’s outcome as a vital influence on how residents learn. Nothnagle, Anandarajah, Goldman and Reis (2011) in a study reported that residents acknowledged patient care as the strongest incentive for SDL, adding that residents’ engagement to learn was stronger when it was clinically driven. Similarly, the large role patient’s outcome played as a motivator among the residents in this study has been reported elsewhere (Sagasser, Kramer, & van der Vleuten, 2012; Watling, Driessen, van der Vleuten, & Lingard, 2012). Berkhout et al. (2015) emphasised that the clinical environment is characterised by unique features that influence opportunities to self-regulate which include the patients and the interactions with patients. Matsuyama, Nakaya, Okazaki, Leppink and van der Vleuten (2018) reported that rural physicians in Japan were motivated to initiate learning strategies in a self-regulated manner because of the knowledge that they could upgrade health care in a particular community. The above studies are in congruence with our finding that patients create a powerful drive among residents to strive to become a competent physician.

From the perspective of the residents, the consultants were very instrumental in supporting learning and in giving feedback that was acceptable to the trainees, even if it was negative. The residents felt very strongly that sufficient and meaningful interactions with consultants were valuable in fostering learning because of the perceived increased knowledge and experiences of the consultants. Wong (2011) in a study comparing Canadian and Thai residency programs, reported that knowledge and scholarship were given more premium at the Thai program compared to the Canadian program. It can be surmised that it is reflective of a shared cultural value regarding education among Southeast Asian cultures. Additionally, the residents were more inclined to accept a negative assessment to improve themselves from a consultant who was perceived to be committed to the department and the training of the residents than from a consultant who is perceived to have less commitment. This resonated with the conclusion of Watling et al. (2008) where they investigated the perceptions and experiences of residents toward in-training evaluation process, that such a process became meaningful to the residents only when there is engagement between the evaluator and the residents. Holmboe, Ginsburg and Bernabeo (2011) commenting on the short and frequent rotation among clinical faculty in the USA, stress that such a situation makes it hard for trainees to establish a meaningful relationship with the clinical faculty which predisposes to superficial assessment. Steven, Wenger, Boshuizen, Scherpbier and Dornan (2014) in their research involving clerks in clinical workplace, concluded that the willingness of clinical practitioners to interact with students is the main element that influenced their learning and their education can be enhanced further by involving learners more dynamically in what they referred to as the ‘communicative processes’ of the clinical communities. Nothnagle et al. (2011) in their study of residents’ views toward SDL revealed that residents expressed a need for coaching or guidance to maximise their learning. Sagasser et al. (2012) researching among postgraduate trainees conveyed that affirmation from supervisors and mentors were sought by trainees, as well as from their peers. Faculty must be aware they can influence each specific phase of a resident’s learning process especially since residents look up to them to validate the interpretation and construction of meaning based on what the residents experienced (Teunissen et al., 2007).

The learning environment is an important determinant of behaviour of students or trainees, for this reason many instruments have been designed to measure the learning climate in postgraduate settings (Genn, 2001). The departments must be aware they can promote learning or actually discourage it (Boor et al., 2008). Thus, there is a real need to be aware of how the residents perceive their learning environment within their departments or within the hospital. There is now recognition that it is imperative that hospitals include residents’ training as a part of organisational initiatives to enhance quality, safety and value in patient care, in so doing producing a high quality graduate medical training (Weiss, Bagian, & Nasca, 2013). With regard to SRL, there is definite interaction between the personal, behavioural and environmental aspects that govern self-regulation (Zimmerman, 1989). In a review of the published researches on SRL, van Houten-Schat et al. (2018) concluded that the use of SRL is not maximised in the clinical settings and recommends that a deliberate effort to design a learning environment that offers trainees the opportunity to apply their goal setting skills and helps improve their SRL confidence.

Our findings have shown that indeed any assessment method results in an educational response from postgraduate medical trainees. Our residents do complain, however, that consultants have limited interactions with them, yet these consultants are asked to evaluate them. Such complaints could be overcome by the introduction of direct observation of trainees through workplace based assessments (WBA). Being able to accurately observe resident-trainees performing clinical tasks such as history taking and physical examination and in the process deliver applicable feedback is one of the most important aspects of medical training (Norcini & Burch, 2007). Additionally, the opportunity for feedback which is inherent in these workplace assessment methods is equally important to their role in assessment (Norcini, 2010).

A. Strengths and Weaknesses of the Study

We chose IM and paediatrics because they are two departments without surgical skills and the expected competencies between the two are not very different in terms of knowledge and skills. Involving other residents from departments that train their residents to acquire the needed surgical skills may reveal different resident’ perspectives. Our study was done in a training institution with no WBA methods implemented yet, it would be interesting to study how WBA methods would change the perceptions of these residents toward assessment and the impact of the assessment program itself among the residents. Another limitation is that our study involves a single institution, and we are aware that some institutions may have situations which simulate a ‘hidden curriculum’ which changes the response of the trainees and their perceptions to the assessment practices.

V. CONCLUSION

In postgraduate medical education, trainees acknowledge that assessment positively influences their clinical training and performance. However, it is imperative that the following are considered before assessment can be assumed to contribute to the training and clinical performance of trainees, these are credibility, fairness, inter-personal relations between the trainee and the evaluator and commitment of the evaluator. Additionally, assessment drives learning through an interplay of different elements which include the patients and concern for their welfare, interactions with the consultants or supervisors and the learning environment. The residents, despite the absence of formal training or guidance from the clinical faculty, manifested a high degree of SDL to achieve their goals. The contributory effects of patients toward training of residents must be further researched to add more to the motivation of residents, and when better understood this can be applied even in undergraduate medical settings. Training institutions must make an effort to create an environment that stimulates learning and must be conscious of how the learning environment influences their trainees.

Notes on Contributors

Jaime L. Pacifico, MD, is a cardiologist and a faculty in the college of medicine at De La Salle University in the Philippines. He is a PhD student at Maastricht University. His studies are about perceptions of the learning environment in postgraduate medical education.

Julie Anne Villanueva, MD, is an internist in the Philippines.

Sylvia Heeneman, PhD, has a background in biomedical sciences and is currently an educational researcher and PhD supervisor in this field at Maastricht University.

Cees van der Vleuten, PhD, is the scientific director of the School of Health Professions of Maastricht University. He has published widely on medical education, particularly assessment. He supervises PhD students from the same university.

Ethical Approval

This study was approved by the ethical review board of the De La Salle Medical and Health Sciences Institute, Dasmarinas, Cavite, Philippines.

Acknowledgements

The authors wish to thank all the residents who voluntarily participated in this study.

Funding

This is an unfunded study.

Declaration of Interest

The authors declare that they have no competing interests.

References

Berkhout, J. J., Helmich, E., Teunissen, P. W., van den Berg, J. W., van der Vleuten, C. P. M., & Jaarsma, A. D. C. (2015). Exploring the factors influencing clinical students’ self-regulated learning. Medical Education, 49(6), 589-600. https://doi.org/10.1111/medu.12671

Boor, K., Scheele, F., van der Vleuten, C. P. M., Teunissen, P. W., Den Breejen, E. M. E., & Scherpbier, A. J. J. A. (2008). How undergraduate clinical learning climates differ: A multi-method case study. Medical Education, 42(10), 1029-1036. https://doi.org/10.1111/j.1365-2923.2008.03149.x

Charmaz, K. (2014). Constructing grounded theory. Thousand Oaks, CA: Sage Publications.

Dijksterhuis, M. G. K., Schuwirth, L. W. T., Braat, D. D. M., Teunissen, P. W., & Scheele, F. (2013). A qualitative study on trainees’ and supervisors’ perceptions of assessment for learning in postgraduate medical education. Medical Teacher, 35(8), e1396-e1402.
https://doi.org/10.3109/0142159X.2012.756576

Gandomkar, R., & Sandars, J. (2018). Clearing the confusion about self-directed learning and self-regulated learning. Medical Teacher, 40(8), 862-863.
https://doi.org/10.1080/0142159X.2018.1425382

Genn, J. M. (2001). AMEE Medical Education Guide No. 23 (Part 2): Curriculum, environment, climate, quality and change in medical education – A unifying perspective. Medical Teacher, 23(5), 445-454.
https://doi.org/10.1080/01421590120075661

Glaser, B., & Strauss, A. (1967). The discovery of grounded theory: Strategies for qualitative research. Chicago, IL: Aldine Transaction.

Holmboe, E., Ginsburg, S., & Bernabeo, E. (2011). The rotational approach to medical education: Time to confront our assumptions? Medical Education, 45(1), 69-80. https://doi.org/10.1111/j.1365-2923.2010.03847.x

Holmboe, E. S., Durning, S. J., & Hawkins, R. E. (2018). Practical guide to the evaluation of clinical competence. Philadelphia, PA: Elsevier Health Sciences.

Holmboe, E. S., Hawkins, R. E., & Huot, S. J. (2004). Effects of training in direct observation of medical resident’s clinical competence: A randomized trial. Annals of Internal Medicine, 140(11), 874-881. https://doi.org/10.7326/0003-4819-140-11-200406010-00008

Leung, S. F., Mok, E., & Wong, D. (2008). The impact of assessment methods on the learning of nursing students. Nurse Education Today, 28(6), 711-719.

Matsuyama, Y., Nakaya, M., Okazaki, H., Leppink, J., & van der Vleuten, C. (2018). Contextual attributes promote or hinder self-regulated learning: A qualitative study contrasting rural physicians with undergraduate learners in Japan. Medical Teacher, 40(3), 285-295.
https://doi.org/10.1080/0142159X.2017.1406074

McLachlan, J. (2006). The relationship between assessment and learning. Medical Education, 40(8), 716-717. https://doi.org/10.1111/j.1365-2929.2006.02518.x

Morse, J. M. (1995). The significance of saturation. Qualitative Health Research, 5(2), 147-149.
https://doi.org/10.1177/104973239500500201

Newble, D. I., & Jaeger, K. (1983). The effect of assessments and examinations on the learning of medical students. Medical Education, 17(3), 165-171. https://doi.org/10.1111/j.1365-2923.1983.tb00657.x

Norcini, J. J. (2010). Workplace assessment. In T. Swanwick (Ed.), Understanding medical education: Evidence, theory and practice (pp. 232-245). London, England: Wiley-Blackwell. https://doi.org/10.1002/9781118472361.ch20

Norcini, J. J., & Burch, V. (2007). Workplace-based assessment as an educational tool: AMEE Guide No. 31. Medical Teacher, 29(9-10), 855-871.
https://doi.org/10.1080/01421590701775453

Norcini, J. J., & McKinley, D. W. (2007). Assessment methods in medical education. Teaching and Teacher Education, 23(3), 239-250. https://doi.org/10.1016/j.tate.2006.12.021

Nothnagle, M., Anandarajah, G., Goldman, R. E., & Reis, S. (2011). Struggling to be self-directed: Residents’ paradoxical beliefs about learning. Academic Medicine, 86(12), 1539-1544. https://doi.org/10.1097/ACM.0b013e3182359476

Sagasser, M. H., Kramer, A. W., & van der Vleuten, C. P. (2012). How do postgraduate GP trainees regulate their learning and what helps and hinders them? A qualitative study. BMC Medical Education, 12(1), 67.
https://doi.org/10.1186/1472-6920-12-67

Sambell, K., McDowell, L., & Brown, S. (1997). “But is it fair?”: An exploratory study of student perceptions of the consequential validity of assessment. Studies in Educational Evaluation, 23(4), 349-371.

Schuwirth, W. T., & van der Vleuten, C. P. M. (2010). How to design a useful test: The principles of assessment. In T. Swanwick (Ed). Understanding medical education: Evidence, theory and practice (pp.195-207). London, England: Wiley-Blackwell.

Segers, M., Nijhuis, J., & Gijselaers, W. (2006). Redesigning a learning and assessment environment: The influence on students’ perceptions of assessment demands and their learning strategies. Studies in Educational Evaluation, 32, (3), 223-242. https://doi.org/10.1016/j.stueduc.2006.08.004

Steven, K., Wenger, E., Boshuizen, H., Scherpbier, A., & Dornan, T. (2014). How clerkship students learn from real patients in practice settings. Academic Medicine, 89(3), 469-476. https://doi.org/10.1097/ACM.0000000000000129

Struyven, K., Dochy, F., & Janssens, S. (2005). Students’ perceptions about evaluation and assessment in higher education: A review. Assessment & Evaluation in Higher Education, 30(4), 331-347.
https://doi.org/10.1080/02602930500099102

Teunissen, P. W., Scheele, F., Scherpbier, A. J. J. A., van der Vleuten, C. P. M., Boor, K., Van Luijk, S. J., & van Diemen-Steenvoorde, J. A. A. M. (2007). How residents learn: Qualitative evidence for the pivotal role of clinical activities. Medical Education, 41(8), 763-770.
https://doi.org/10.1111/j.1365-2923.2007.02778.x

Tiwari, A., Lam, D., Yuen, K. H., Chan, R., Fung, T., & Chan, S. (2005). Student learning in clinical nursing education: Perceptions of the relationship between assessment and learning. Nurse Education Today, 25(4), 299-308. https://doi.org/10.1016/j.nedt.2005.01.013

van der Vleuten, C. P. M. (1996). The assessment of professional competence: Developments, research and practical implications. Advances in Health Sciences Education, 1(1), 41-67. https://doi.org/10.1007/BF00596229

van der Vleuten, C. P. M., Schuwirth, L. W. T., Scheele, F., Driessen, E. W., & Hodges, B. (2010). The assessment of professional competence: Building blocks for theory development. Best practice and research clinical obstetrics and gynaecology, 24(6), 703-719.
https://doi.org/10.1016/j.bpobgyn.2010.04.001
van Houten-Schat, M., Berkhout, J., van Dijk, N., Enderdijk, M., Jaarsma, D., & Diemers, A. (2018). Self-regulated learning in the clinical context: A systematic review. Medical Education, 52(10), 1008-1015.
https://doi.org/10.1111/medu.13615

Watling, C., Driessen, E., van der Vleuten, C. P. M., & Lingard, L. (2012). Learning from clinical work: The roles of learning cues and credibility judgements. Medical Education, 46(2), 192-200. https://doi.org/10.1111/j.1365-2923.2011.04126

Watling, C. J., Kenyon, C. F., Zibrowski, E. M., Schulz, V., Goldszmidt, M. A., Singh, I., … Lingard, L. (2008). Rules of engagement: Residents’ perceptions of the in-training evaluation process. Academic Medicine, 83(10), S97-S100. https://doi.org/10.1097/ACM.0b013e318183e78c

Watling, C. J., & Lingard, L. (2012). Grounded theory in medical education research: AMEE Guide No. 70. Medical Teacher, 34(10), 850-861.
https://doi.org/10.3109/0142159X.2012.704439

Weiss, K. B., Bagian, J. P., & Nasca, T. J. (2013). The clinical learning environment: The foundation of graduate medical education. JAMA, 309(16), 1687-1688.
https://doi.org/10.1001/jama.2013.1931

Wong, A. K. (2011). Culture in medical education: Comparing a Thai and a Canadian residency programme. Medical Education, 45(12), 1209-1219.
https://doi.org/10.1111/j.1365-2923.2011.04059.x

Zimmerman, B. J. (1989). A social cognitive view of self-regulated academic learning. Journal of Educational Psychology, 81(3), 329-339.
http://dx.doi.org/10.1037/0022-0663.81.3.329

*Jaime L. Pacifico
De La Salle University College of Medicine,
De La Salle Medical and Health Sciences Institute,
Dasmarinas, Cavite, Philippines
Tel: +63 46 481 8000
E-mail: jlpacifico@dlshsi.edu.ph