Submitted: 28 March 2020
Accepted: 23 September 2020
Published online: 4 May, TAPS 2021, 6(2), 9-24
https://doi.org/10.29060/TAPS.2021-6-2/OA2242

De Zhang Lee¹, Jia Yi Choo¹, Li Shia Ng², Chandrika Muthukrishnan¹ & Eng Tat Ang¹

¹Department of Anatomy, Yong Loo Lin School of Medicine, National University of Singapore, Singapore; ²Department of Otolaryngology, National University Hospital, Singapore

Abstract

Introduction: Gamification has been shown to improve academic gains, but the mechanism remains elusive. We aim to understand how psychological constructs interact, and influence medical education using mathematical modelling.

Methods: Studying a group of medical students (n=100; average age: 20) over a period of 4 years with the Personal Responsibility Orientation to Self-Direction in Learning Scale (PRO-SDLS) survey. Statistical tests (Paired t-test) and models (logistic regression) were used to decipher the changes within these psychometric constructs (Motivation, Control, Self-efficacy & Initiative), with gamification as a tool. Students were encouraged to partake in a maze (10 stations) that challenged them to answer anatomical questions using potted human specimens.

Results: We found that the combinatorial effects of the maze and Script Concordance Test (SCT) resulted in a significant improvement for “Self-Efficacy” and “Initiative” (p<0.05). However, the “Motivation” construct was not improved significantly with the maze alone (p<0.05). Interestingly, the “Control” construct was eroded in students not exposed to gamification (p<0.05). All these findings were supported by key qualitative comments such as “helpful”, “fun” and “knowledge gap” by the participants (self-awareness of their thought processes). Students found gamification reinvigorating and useful in their learning of clinical anatomy.

Conclusion: Gamification could influence some psychometric constructs for medical education, and by extension, the metacognition of the students. This was supported by the improvements shown in the SCT results. It is therefore proposed that gamification be further promoted in medical education. In fact, its usage should be more universal in education.

Keywords:            Psychometric Constructs, Medical Education, Motivation, Initiative, Self-efficacy

Practice Highlights

• Student’s enjoyment (interest) of the curriculum will determine the eventual academic outcome.
• Metacognition (defined as the “learning of learning”, “knowing of knowing” and/ or the awareness of one’s thought processes) was improved with SCT and gamification.
• Gamification is useful as a form of augmentation for didactic teaching but should never replace it.
• Different type of psychometric scale (e.g. LASSI versus PRO-SDLS) used in your research will produce varying results.
• Gamification is resource intensive and needs extra time to prepare compared to didactic approaches.

I. INTRODUCTION

Psychology is integral to healthcare and education but has often been overshadowed, compared to the other basic disciplines (Choudhry et al., 2019; Pickren, 2007). This is ironical because human psyche needs to be properly understood in order to manage them effectively (Wisniewski & Tishelman, 2019). Presently, the study of psychology does not feature prominently in the medical curriculum (Gallagher et al., 2015) with the exception of psychiatry (Douw et al., 2019). This gap needs to be addressed (Paros & Tilburt, 2018). In this research, we seek to understand the constructs for good medical learning via gamification which has wide ranging effects (Mullikin et al., 2019). The psychometric constructs to be analysed were as follows: 1) “Motivation”; define as the desire to learn out of interest or enjoyment (Yue et al., 2019). 2) “Initiative”; refers to how proactive a student is to learning (Boyatzis et al., 2000). 3) “Control”; is how much influence one has over the circumstances (Sheikhnezhad Fard & Trappenberg, 2019). 4) “Self-Efficacy”; relates to how confident one is, to do what needs to be done (Michael et al., 2019). We believe that these constructs contribute to the student’s awareness of their own thought processes (metacognition) towards their medical education.

Gamification” is defined as a process of adding game-like elements to something so as to encourage more participation (Rutledge et al., 2018; Van Nuland et al., 2015). The idea of using games to “lighten up” medical education in the clinical setting was first proposed in 2002 (Howarth-Hockey & Stride, 2002). The authors observed increased engagement and participation during lunchtime medical quizzes in the hospital. They therefore concluded that medical education could be fun, and since then, gamification has been taken seriously by the community (Evans et al., 2015; Nevin et al., 2014). In essence, gamification could be something as simple as having board games (Ang et al., 2018) but importantly, its impact on students’ learning must be evaluated and validated. Most studies in the literature did not fulfil this requirement (Graafland et al., 2012). The impact of games on the behavioral and/or psychological outcomes should be studied (Graafland et al., 2017; Graafland et al., 2014).

A PubMed search would reveal that there are numerous self-reporting tools such as LASSI (Learning and Strategies Study Inventory (Muis et al., 2007), MSLQ (Motivated Strategies for Learning Questionnaire (Villavicencio & Bernardo, 2013), and the SRLPS (Self-regulated Learning Perception Scale) (Turan et al., 2009) etc. Given the choices, how does one decide which one to adopt for their studies? In our research, we chose to use the PRO-SDLS survey questions with some modifications. The choice was both serendipitous and practical, as we have previously validated it via the Cronbach alpha (>0.7). In our earlier work, feedback scores and results yielded inconclusive evidence to support enhanced motivation among our students. Furthermore, was this due to gamification? With the current endeavour, we aim to prove via mathematical modelling that there are indeed alterations to the psychometric constructs. Hence, we re-analyse the old data set together with additional new information, using statistical analysis tools such as the logistic regression model, Wilcoxon tests, and the Paired t-test.

Medical teaching and learning is a complex endeavour based on an apprenticeship model (Cortez et al., 2019), which may or may not be an ideal arrangement (Sheehan et al., 2010). Furthermore, the decision making is often delegated to the seniors (Chessare, 1998). Conversely, gamification could empower the students to take charge of one’s learning, including decision making (Shah et al., 2013). Furthermore, one needs to understand what works from what is empirical (Cote et al., 2017). While our initial research addressed the impact of the games on academic performance, we now sought to further understand its effects on the psychometric dimensions. This will help to understand the psychology of self-directed (or regulated) learning. We hypothesize that the amount of gamification will impact these constructs. In summary, we hope to achieve the following:

Aims:

• Understanding the role of psychometric constructs and gamification in medical education via suitable mathematical modelling.
• To decipher the interaction of different psychometric constructs (Motivation, Self-efficacy, Control and Initiatives) in producing desired learners’ behaviours (metacognition) via the anatomy maze.

II. METHODS

First-year medical students (M1) took part in this retrospective analytical research. Two randomised groups of medical students (n=75, median age: 20 years) consented to the study (Group 1 & 2). A randomised group of students (n=25) exposed to no gamifications (Group 0) served as the control. Every student was required to complete a pre- and post- PRO-SDLS for the research. There were no penalties for withdrawing from the IRB-approved project (See IRB: B-16-205).

Gamification was carried out according to the scheme in Figure 1. Each group was divided into 3 to 4 subgroups that would enter the maze with a clue card (see example in Figure 1) linked to a specific pot specimen. They were required to explore the museum for the next clue and had to answer the hidden questions (see examples in Appendix) which would provide further directions. At the conclusion, students were given a competitive pop quiz that had no impact on their summative academic grades.

The main purpose was to assess formative knowledge acquisition. The validated PRO-SDLS include the following psychometric constructs: “Motivation” (7 questions), “Initiative” (6 questions), “Control” (6 questions) and “Self-Efficacy” (6 questions). (See Sup. Materials). The responses are then collapsed into an average accordingly. A higher score indicated more agreeability towards that construct for self-directed learning (Ang et al., 2017). The survey was designed with backward scoring to ensure accuracy. For quantification purposes, we subtracted the pre-feedback from the post-feedback scores for each question. An increased score for a particular construct suggests improvement (Cazan & Schiopca, 2014). Furthermore, students in Group 2 were given Script Concordance Test (SCT) quizzes (See Sup. Materials) as part of gamification (Lubarsky et al., 2013; Lubarsky et al., 2018; Wan et al., 2018). SCT were meant to enhance clinical reasoning. All data were analysed from two perspectives:

• The magnitude of score increase (or decrease) of the post- PRO-SDLS survey responses, with respect to the pre- responses.
• The odds of a student reporting an increased score in the post- PRO-SDLS survey responses.

In (a), the paired differences for each student’s response were studied using a parametric approach (paired t-test). In (b), we studied the odds of increased score for each construct, and investigate if grouping affected these odds. More formally, for each construct k (where k is one of the four constructs), we define variable  as the probability of a student from group  showing an increase in score for construct  (and hence,  as the probability that the student’s score decreased or remained unchanged). The value of   can be estimated by dividing the number of students from group  with an increased score for construct  by the total number of students from group . If the interventions are unsuccessful, we would expect  to be around  since a student’s score would likely either increase or decrease at random, with an equal probability. This can be tested using the t-test.

An alternative approach would be to study the odds of success, which can be written as  . A common mathematical model used to study these odds is the logistic regression model. For each construct, the logistic regression model studies the odds of a student from a given group showing an increased or decreased score. The overall significance of the model can be tested using the p-value obtained from the likelihood ratio test, while the significance of the individual odds can be tested using the t-test. For more details on the logistic regression model, we refer the reader to (Agresti, 2003).

We utilised the open source software R  (Team, 2019) to perform our statistical analysis.

III. RESULTS

Participation rate in the gamification endeavour was consistently 90±5%, and there was zero withdrawal from it, accompanied by reported favourable qualitative comments.

A. Studying the Absolute Scores

The average change in scores across all the groups for each construct is given in Table 1. From these scores, we believe that our gamification exercises may have had a positive impact on “Self-Efficacy” and “Initiative”. To visualize the spread of responses, we have prepared box plots of the post – pre scores (available in Supplementary Materials).

We observe that the null hypothesis of no difference between pre and post intervention levels for all constructs are not rejected (under p=0.05) for the control group. Both tests also failed to show any significant change for the “Control” construct.

There is strong evidence that the classroom interventions employed by Groups 1 and 2 had an impact on “Initiative” levels of students, reflected by the small p-values obtained using both tests. The average increase in “Initiative” scores for students in Groups 1 and 2 are 0.71 and 0.67, respectively, which are similar. Recall that the students in Group 2 participated in the SCT, in addition to the maze which is common across both groups. This suggests that the SCT has a negligible impact on “Initiative”.

There is also strong evidence (p=0.05) to show that the games enhanced the “Self-Efficacy” levels among the students. The t-test also gives strong evidence (p=0.05) that there is a significant change in Group 2, and milder evidence for Group 1 (p=0.10). The average increase in “Self-Efficacy” levels for Groups 1 and 2 are 0.63 and 0.56, respectively. Again, the differences are negligible, and this suggests that the SCT has a negligible impact. Finally, there is mild evidence (p=0.10) of a significant change in “Motivation” for Group 2, but no such evidence for Group 1. The average increase in “Motivation” score for Group 2 is 0.55. This time, the SCT might have helped to improve students’ motivation.

B. Studying the Odds of Score Improvement

We will now turn our attention to modelling the odds of a student reporting an increase in construct score. Earlier, we defined  as the probability of a student from group  showing an increase in score for construct , and explained why we would expect  to be around  if the games have no impact on the odds of “success”. The t-test was used to test this, under the null hypothesis that  for all groups and constructs. The p-values obtained are summarised in Table 3.

Under the logistic regression model, not rejecting the null hypothesis for a given odds means that we assume it takes on the value 1. It should be noted that the individual coefficients should be examined when the model is determined to be significant under the likelihood ratio test, as the coefficients obtained under a poor model fit may not be meaningful.

We notice that the significant terms flagged out by the t-test (Table 3) largely agree with the significant terms of the logistic regression model, except for the “Self-Efficacy” odds for Group 1. However, the “Self-Efficacy” model was not determined to be a good fit using the likelihood ratio test.

The only model which was deemed to be a good fit was the one for the “Initiative” construct. The odds for Group 0 is deemed to be insignificant (and hence assumed to be 1), while the odds for Groups 1 and 2 are statistically significant. We can interpret this model as follows,

1. Since the odds for Group 0 is statistically insignificant under the t-test, we assume the odds to be 1. In other words, it is equally likely for a student from the control group to show an increase or decrease in score.
2. The odds for both Group 1 and 2 are statistically significant. The odds of success of Group 1 is 2.41, which can be translated to a roughly 7 in 10 chance (probability of 0.71) for a student in this group showing an improved score. A similar interpretation can be made for Group 2, which showed an odds of 1.99. This translates to a slightly lower probability of 0.67 for a student from Group 2 displaying an improved score.

 

With this, we have presented a logistic regression approach of mathematically modelling these odds. A search on Google Scholar and PubMed yielded no previous work which made us of this mathematical modelling approach on the PRO-SDLS survey data. With the derived odds, we can compare the degree of success of the various classroom interventions. The logistic regression modelling approach is therefore, proposed as a complement of the t-test approach, which is restricted to detecting the presence of statistically significant differences.

C. Qualitative Comments (underlined words underpinning for metacognition)

1) Positive feedbacks:

• “The maze games were the most helpful as they helped me to consolidate my learning, and also enables me to ask the tutor any questions that I have from class. They allowed me to learn anatomy in a fun, enjoyable and memorable way”
• “It allowed me to visualize the things that I was learning and helped with clarifying doubts”
• “The extra question posted at each station was helpful”
• “Wanting to be able to identify things in the museum makes me more motivated to prepare beforehand”
• “Allows me to identify the knowledge gap so that I can work on it”
• “I like the quiz as it motivates me to study beforehand and shows me the gaps in my knowledge”
• “The clinically relevant questions made me think a lot”

2) Negative feedbacks:

• “I prefer didactic teaching”
• “We did not interact much with the exhibits”
• “The maze was more of a mini quiz or test to check if we remember anything”
• “Perhaps we could go into more complex concepts”
• “More challenging questions”
• “Students just follow each other around the anatomy museum and it defeats the purpose of the maze”
• “The maze could have a competitive element to make it more exciting. Maybe more MCQ questions per model so we can make use of it more”

IV. DISCUSSION

We undertook this research to decipher how gamification as a concept helps medical students learn a basic subject like human anatomy. We also want to understand how psychometric constructs interact to produce behavioural changes towards self-directed learning. This was done by analysing the data from the PRO-SDLS via statistical tests. Put simply, one needs to understand that medical education is a very complex process that demands balance between apprenticeship (fellowship) (Sheehan et al., 2010), and a dose of self-directed learning (van Houten-Schat et al., 2018). With our initial research into gamification of anatomy education (Ang et al., 2018), there were other studies suggesting similar benefits (Felszeghy et al., 2019; Nicola et al., 2017; Van Nuland et al., 2015). We are therefore convinced that gamification could help to engage students and improve academic gains. However, the notion of gaming can be very broad (Virtual Reality, board games, digital apps etc.), so there is a need to understand the underlying psychology. With that in mind, we re-analyse our previous data with the existing, using proven statistical tools to decipher the learning psychology of these medical students, and their awareness of their own thought processes (metacognition).

We earlier hypothesised that gamification would influence these dependent constructs differently and indeed this was the outcome. In our analysis, we found that the combinatorial effects of the maze and SCT resulted in a significant improvement for “Self-Efficacy” and “Initiative”. While the maze alone did not significantly improve “Motivation”, we saw mild evidence of an improvement in terms of psychometric scores, when the SCT and maze were used in combination. In lay terms, the maze encouraged these students to learn on their own. By extension, one could also argue that gamification will help the students in making decisions since “Motivation” and “Initiatives” are key attributes (Vohs et al., 2008). The ability to make a simple clinical judgement, and the courage to act on them, are the virtues that we should be imbuing in the medical students, and some junior doctors. Interestingly, there is mild evidence that the “Control” construct was undergoing erosion in the students not exposed to gamification, as the course progresses. This adverse result is not seen in both groups exposed to the games. Perhaps the more relaxed classroom setting with gamification helped students to feel more in control of their learning process. Logically, this made a lot of sense across the education landscape.

A follow up question would be, does the feedback confirm the results given in our qualitative analysis? Recall that in our logistic regression model, students from both non-control groups displayed a statistically significant improvement in “Initiative” levels. This is supported by some of the positive feedback received for our endeavours, such as being “motivated to prepare beforehand”, “identify the knowledge gap” and work on them, as well as helping them to “think a lot” about the course content. Furthermore, some of the negative feedback, such as requests for more challenging questions, or more questions in general, suggests that the students are taking the initiative to learn more. This certainly adds credence to the findings of our proposed logistic regression model, as well as highlighting the importance of studying both qualitative and quantitative feedback.

There are caveats that one should be aware when implementing gamification. The formative part of the endeavour could be variable, and dependent on numerous factors such as the tutor involved, and the type of games, interventions, and reporting scales used. In the feedback, 76% of the participants felt that the maze should continue as an adjunct but not to totally replace didactic tutorials. In other words, introducing gaming elements into the curriculum should be done judiciously. With reverse scoring, it was shown that “Self-Efficacy” fell as the level of gamification is increased. In lay terms, students might be feeling that the maze trivialize the learning of the subject. As a counter measure, and to maintain quality assurance, we could introduce video lectures from previous years to allay these fears. In summary, we now confirmed that gamification works, and it influences learning outcomes as demonstrated by others (Burgess et al., 2018; Goyal et al., 2017; Kollei et al., 2017; Kouwenhoven-Pasmooij et al., 2017; Kurtzman et al., 2018; O’Connor et al., 2018; Patel et al., 2017; Savulich et al., 2017). Separately, there were criticisms as to why SCT was introduced into the research. We believed that such augmentation will add “fun” for the pre-clinical students to tackle the various clinical scenarios and clinical anatomy.

V. LIMITATIONS OF THE STUDY

Our research necessitated that the students take part in the maze and the SCT. Although it was not compulsory, no students opted out of it. Some critics would misconstrue this to be a form of forced play. According to Jane McGonigal, gamification should ideally not be mandated (Roepke et al., 2015).

VI. CONCLUSION

Through statistical modelling, we have shown how the “Initiative”, “Motivation”, and “Self-Efficacy” constructs could potentially benefit from gamification. The before-after experimental set up allowed for powerful comparisons to be made. Studying the odds of construct score improvement, alongside the raw scores, allowed us to study the data from different perspectives. Though this approach, we discovered how the potential benefits of our gamification exercises outweigh the potential adverse effects. Gamification had resulted in improved “Initiative” in these medical students. We believe that their decision-making skills will also be boosted if existing culture allows for more self-discovery (to improve “Initiative”, “Control” and “Self-efficacy”) and autonomy. If these recommendations are duly considered and implemented thoughtfully, there is little doubt that our future doctors will be better equipped to serve humanity. This may also help to avoid possible burnout in residents (Hale et al., 2019).

Stronger conclusion and potential for applications are as follows: In a continuum, we started gamifying anatomy education and proven that academic grades could be improved by the process (Ang et al., 2018). We then asked a fundamental question in how exactly it happened. This was done by carrying out a psychometric analysis on the participants. We discovered that psychometric constructs were important, and this was proven in this manuscript. The impact of gamification is now elevated given the COVID-19 pandemic that necessitated more online teaching. Moving forward, we believe that gamification should move towards creating an electronic application that the students may access 24/7. This will ensure that medical teaching will be fortified and be somewhat protected from further disruptions.

Notes on Contributors

Lee De Zhang graduated with a degree in Statistics and Computer Science. He reviewed the literature, analysed the data and wrote part of the manuscript.

Eng Tat Ang, Ph.D., is a senior lecturer in anatomy at the Department of Anatomy at the YLLSoM, NUS. He reviewed the literature, designed the research, collected and analysed the data. He developed the manuscript.

Choo Jiayi, BSc (Hons) graduated with a degree in life sciences. She executed the research, and help collected the data. She contributed to the development of the manuscript.

M Chandrika, MBBS, DO, MSc is an instructor at the Department of Anatomy at the YLLSoM, NUS. She helped to execute the research and collected the data.

Ng Li Shia, MBBS, Master of Medicine (Otorhinolaryngology), MRCS(Glasg) is a consultant at the Department of Otolaryngology, Head & Neck Surgery (ENT), National University Hospital. She developed the SCT questions.

Ethical Approval

This project has received full IRB and Ethical clearance (NUS IRB: B-16-205).

Acknowledgements

A big thank you to all students who took part in the research, and to the CDTL, NUS, for providing a teaching enhancement funds to support this research. Appreciation also due to Dr Patricia Chen (Dept. of Psychology, NUS) for her helpful advice.

Funding

NUS TEG AY2017/2018 was awarded to help the investigators pay Mr De Zhang Lee for the statistical modelling that gamification drove medical education via a MAZE.

Declaration of Interest

All authors have no conflict of interest to declare.

References

Agresti, A. (2003). Categorical data analysis. John Wiley & Sons.

Ang, E. T., Abu Talib, S. N., Samarasekera, D., Thong, M., & Charn, T. C. (2017). Using video in medical education: What it takes to succeed. The Asia Pacific Scholar. 2(3), 15-21.

Ang, E. T., Chan, J. M., Gopal, V., & Li Shia, N. (2018). Gamifying anatomy education. Clinical Anatomy, 31(7), 997-1005. https://doi.org/10.1002/ca.23249

Boyatzis, R. E., Murphy, A. J., & Wheeler, J. V. (2000). Philosophy as a missing link between values and behaviour. Psychological Reports, 86(1), 47-64. https://doi.org/10.2466/pr0.2000.86.1.47

Burgess, J., Watt, K., Kimble, R. M., & Cameron, C. M. (2018). Combining Technology and Research to Prevent Scald Injuries (the Cool Runnings Intervention): Randomized Controlled Trial. Journal of Medical Internet Research, 20(10), e10361. http://doi.org/10.2196/10361

Cazan, A. M., & Schiopca, B. A. (2014). Self-directed learning, personality traits and academic achievement. Procedia-Social and Behavioral Sciences (127), 640-644.

Chessare, J. B. (1998). Teaching clinical decision-making to pediatric residents in an era of managed care. Pediatrics, 101(4 Pt 2), 762-766; discussion 766-767. Retrieved from https://www.ncbi.nlm.nih.gov/pubmed/9544180

Choudhry, F. R., Ming, L. C., Munawar, K., Zaidi, S. T. R., Patel, R. P., Khan, T. M., & Elmer, S. (2019). Health literacy studies conducted in australia: A scoping review. International Journal of Environmental Research and Public Health, 16(7). https://doi.org/10.3390/ijerph16071112

Cortez, A. R., Winer, L. K., Kassam, A. F., Hanseman, D. J., Kuethe, J. W., Quillin, R. C., 3rd, & Potts, J. R., 3rd. (2019). See none, do some, teach none: An analysis of the contemporary operative experience as nonprimary surgeon. Journal of Surgical Education, 76(6), e92-e101. https://doi.org/10.1016/j.jsurg.2019.05.007

Cote, L., Rocque, R., & Audetat, M. C. (2017). Content and conceptual frameworks of psychology and social work preceptor feedback related to the educational requests of family medicine residents. Patient Education and Counseling, 100(6), 1194-1202. https://doi.org/10.1016/j.pec.2017.01.012

Douw, L., van Dellen, E., Gouw, A. A., Griffa, A., de Haan, W., van den Heuvel, M., Hillebrand, A., Van Mieghem, P., Nissen, I. A., Otte, W. M., & Reijmer, Y. D. (2019). The road ahead in clinical network neuroscience. Network Neuroscience, 3(4), 969-993. https://doi.org/10.1162/netn_a_00103 

Evans, K. H., Daines, W., Tsui, J., Strehlow, M., Maggio, P., & Shieh, L. (2015). Septris: a novel, mobile, online, simulation game that improves sepsis recognition and management. Academic Medicine, 90(2), 180-184. https://doi.org/10.1097/ACM.0000000000000611

Felszeghy, S., Pasonen-Seppänen, S., Koskela, A., Nieminen, P., Härkönen, K., Paldanius, K. M., Gabbouj, S., Ketola, K., Hiltunen, M., Lundin, M., & Haapaniemi, T.  (2019). Using online game-based platforms to improve student performance and engagement in histology teaching. BMC Medical Education, 19(1), 273. https://doi.org/10.1186/s12909-019-1701-0

Gallagher, S., Wallace, S., Nathan, Y., & McGrath, D. (2015). ‘Soft and fluffy’: medical students’ attitudes towards psychology in medical education. Journal of Health Psychology, 20(1), 91-101. https://doi.org/10.1177/1359105313499780

Goyal, S., Nunn, C. A., Rotondi, M., Couperthwaite, A. B., Reiser, S., Simone, A., Katzman, D. K., Cafazzo, J. A., & Palmert, M. R. (2017). A mobile app for the self-management of Type 1 Diabetes among adolescents: A randomized controlled trial. Journal of Medical Internet Research  mHealth and uHealth, 5(6), e82. https://doi.org/10.2196/mhealth.7336

Graafland, M., Bemelman, W. A., & Schijven, M. P. (2017). Game-based training improves the surgeon’s situational awareness in the operation room: a randomized controlled trial. Surgical Endoscopy, 31(10), 4093-4101. https://doi.org/10.1007/s00464-017-5456-6

Graafland, M., Schraagen, J. M., & Schijven, M. P. (2012). Systematic review of serious games for medical education and surgical skills training. British Journal of Surgery, 99(10), 1322-1330. https://doi.org/10.1002/bjs.8819

Graafland, M., Vollebergh, M. F., Lagarde, S. M., van Haperen, M., Bemelman, W. A., & Schijven, M. P. (2014). A serious game can be a valid method to train clinical decision-making in surgery. World Journal of Surgery, 38(12), 3056-3062. https://doi.org/10.1007/s00268-014-2743-4

Hale, A. J., Ricotta, D. N., Freed, J., Smith, C. C., & Huang, G. C. (2019). Adapting Maslow’s Hierarchy of Needs as a Framework for Resident Wellness. Teaching and Learning in Medicine, 31(1), 109-118. https://doi.org/10.1080/10401334.2018.1456928

Howarth-Hockey, G., & Stride, P. (2002). Can medical education be fun as well as educational? British Medical Journal, 325(7378), 1453-1454.  https://doi.org/10.1136/bmj.325.7378.1453 

Kollei, I., Lukas, C. A., Loeber, S., & Berking, M. (2017). An app-based blended intervention to reduce body dissatisfaction: A randomized controlled pilot study. Journal of Consulting and Clinical Psychology, 85(11), 1104-1108. https://doi.org/10.1037/ccp0000246  

Kouwenhoven-Pasmooij, T. A., Robroek, S. J., Ling, S. W., van Rosmalen, J., van Rossum, E. F., Burdorf, A., & Hunink, M. G. (2017). A blended web-based gaming intervention on changes in physical activity for overweight and obese employees: Influence and usage in an experimental pilot study. Journal of Medical Internet Research   Serious Games, 5(2), e6. https://doi.org/10.2196/games.6421

Kurtzman, G. W., Day, S. C., Small, D. S., Lynch, M., Zhu, J., Wang, W., Rareshide, C. A., & Patel, M. S. (2018). Social incentives and gamification to promote weight loss: The lose it randomized, controlled trial. Journal of General Internal Medicine, 33(10), 1669-1675. https://doi.org/10.1007/s11606-018-4552-1  

Lubarsky, S., Dory, V., Duggan, P., Gagnon, R., & Charlin, B. (2013). Script concordance testing: from theory to practice: AMEE guide no. 75. Medical Teacher, 35(3), 184-193. https://doi.org/10.3109/0142159X.2013.760036

Lubarsky, S., Dory, V., Meterissian, S., Lambert, C., & Gagnon, R. (2018). Examining the effects of gaming and guessing on script concordance test scores. Perspectives on Medical Education, 7(3), 174-181. https://doi.org/10.1007/s40037-018-0435-8  

Michael, K., Dror, M. G., & Karnieli-Miller, O. (2019). Students’ patient-centered-care attitudes: The contribution of self-efficacy, communication, and empathy. Patient Education and Counseling. https://doi.org/10.1016/j.pec.2019.06.004

Muis, K. R., Winne, P. H., & Jamieson-Noel, D. (2007). Using a multitrait-multimethod analysis to examine conceptual similarities of three self-regulated learning inventories. British Journal of Educational Psychology, 77(Pt 1), 177-195. https://doi.org/10.1348/000709905X90876

Mullikin, T. C., Shahi, V., Grbic, D., Pawlina, W., & Hafferty, F. W. (2019). First year medical student peer nominations of professionalism: A methodological detective story about making sense of non-sense. Anatomical Sciences Education, 12(1), 20-31. https://doi.org/10.1002/ase.1782 

Nevin, C. R., Westfall, A. O., Rodriguez, J. M., Dempsey, D. M., Cherrington, A., Roy, B., Patel, M., & Willig, J. H. (2014). Gamification as a tool for enhancing graduate medical education. Postgraduate Medical Journal, 90(1070), 685-693. https://doi.org/10.1136/postgradmedj-2013-132486

Nicola, S., Virag, I., & Stoicu-Tivadar, L. (2017). vr medical gamification for training and education. Studies in Health Technology and Informatics, 236, 97-103. Retrieved from https://www.ncbi.nlm.nih.gov/pubmed/28508784 

O’Connor, D., Brennan, L., & Caulfield, B. (2018). The use of neuromuscular electrical stimulation (NMES) for managing the complications of ageing related to reduced exercise participation. Maturitas, 113, 13-20. https://doi.org/10.1016/j.maturitas.2018.04.009

Paros, S., & Tilburt, J. (2018). Navigating conflict and difference in medical education: insights from moral psychology. BMC Medical Education, 18(1), 273. https://doi.org/10.1186/s12909-018-1383-z

Patel, M. S., Benjamin, E. J., Volpp, K. G., Fox, C. S., Small, D. S., Massaro, J. M., Lee, J. J., Hilbert, V., Valentino, M., Taylor, D. H., & Manders, E. S.  (2017). effect of a game-based intervention designed to enhance social incentives to increase physical activity among families: The BE FIT randomized clinical trial. Journal of the American Medical Association Internal Medicine, 177(11), 1586-1593. https://doi.org/10.1001/jamainternmed.2017.3458  

Pickren, W. (2007). Psychology and medical education: A historical perspective from the United States. Indian Journal of Psychiatry, 49(3), 179-181. https://doi.org/10.4103/0019-5545.37318  

Roepke, A. M., Jaffee, S. R., Riffle, O. M., McGonigal, J., Broome, R., & Maxwell, B. (2015). Randomized controlled trial of superbetter, a smartphone-based/internet-based self-help tool to reduce depressive symptoms. Games for Health Journal, 4(3), 235-246. https://doi.org/10.1089/g4h.2014.0046

Rutledge, C., Walsh, C. M., Swinger, N., Auerbach, M., Castro, D., Dewan, M., Khattab, M., Rake, A., Harwayne-Gidansky, I., Raymond, T. T., & Maa, T. (2018). Gamification in action: Theoretical and practical considerations for medical educators. Academic Medicine, 93(7), 1014-1020. https://doi.org/10.1097/ACM.0000000000002183

Savulich, G., Piercy, T., Fox, C., Suckling, J., Rowe, J. B., O’Brien, J. T., & Sahakian, B. J. (2017). Cognitive training using a novel memory game on an ipad in patients with amnestic mild cognitive impairment (aMCI). International Journal of Neuropsychopharmacology, 20(8), 624-633. https://doi.org/10.1093/ijnp/pyx040

Shah, A., Carter, T., Kuwani, T., & Sharpe, R. (2013). Simulation to develop tomorrow’s medical registrar. The Clinical Teacher, 10(1), 42-46. https://doi.org/10.1111/j.1743-498X.2012.00598.x  

Sheehan, D., Bagg, W., de Beer, W., Child, S., Hazell, W., Rudland, J., & Wilkinson, T. J. (2010). The good apprentice in medical education. New Zealand Medical Journal, 123(1308), 89-96. Retrieved from https://www.ncbi.nlm.nih.gov/pubmed/20201158

Sheikhnezhad Fard, F., & Trappenberg, T. P. (2019). A novel model for arbitration between planning and habitual control systems. Frontiers in Neurorobotics, 13, 52. https://doi.org/10.3389/fnbot.2019.00052

Team, R. C. (2019). R: A language and environment for statistical computing. R Foundation for Statistical Computing.

Turan, S., Demirel, O., & Sayek, I. (2009). Metacognitive awareness and self-regulated learning skills of medical students in different medical curricula. Medical Teacher, 31(10), e477-483. https://doi.org/10.3109/01421590903193521

van Houten-Schat, M. A., Berkhout, J. J., van Dijk, N., Endedijk, M. D., Jaarsma, A. D. C., & Diemers, A. D. (2018). Self-regulated learning in the clinical context: A systematic review. Medical Education, 52(10), 1008-1015. https://doi.org/10.1111/medu.13615

Van Nuland, S. E., Roach, V. A., Wilson, T. D., & Belliveau, D. J. (2015). Head to head: The role of academic competition in undergraduate anatomical education. Anatomical Sciences Education, 8(5), 404-412. https://doi.org/10.1002/ase.1498

Villavicencio, F. T., & Bernardo, A. B. (2013). Positive academic emotions moderate the relationship between self-regulation and academic achievement. British Journal of Educational Psychology, 83(Pt 2), 329-340. https://doi.org/10.1111/j.2044-8279.2012.02064.x

Vohs, K. D., Baumeister, R. F., Schmeichel, B. J., Twenge, J. M., Nelson, N. M., & Tice, D. M. (2008). Making choices impairs subsequent self-control: A limited-resource account of decision making, self-regulation, and active initiative. Journal of Personality and Social Psychology, 94(5), 883-898. https://doi.org/10.1037/0022-3514.94.5.883

Wan, M. S., Tor, E., & Hudson, J. N. (2018). Improving the validity of script concordance testing by optimising and balancing items. Medical Education, 52(3), 336-346. https://doi.org/10.1111/medu.13495  

Wisniewski, A. B., & Tishelman, A. C. (2019). Psychological perspectives to early surgery in the management of disorders/differences of sex development. Current Opinion in Pediatrics, 31(4), 570-574. https://doi.org/10.1097/MOP.0000000000000784

Yue, P., Zhu, Z., Wang, Y., Xu, Y., Li, J., Lamb, K. V., Xu, Y., & Wu, Y. (2019). Determining the motivations of family members to undertake cardiopulmonary resuscitation training through grounded theory. Journal of Advanced Nursing, 75(4), 834-849. https://doi.org/10.1111/jan.13923

*Ang Eng Tat
Department of Anatomy
Yong Loo Lin School of Medicine
MD10, National University of Singapore
Singapore 117599
Email address: antaet@nus.edu.sg

Submitted: 4 August 2020
Accepted: 14 October 2020
Published online: 4 May, TAPS 2021, 6(2), 1-8
https://doi.org/10.29060/TAPS.2021-6-2/RA2370

Tow Keang Lim

Department of Medicine, National University Hospital, Singapore

Abstract

Introduction: Clinical diagnosis is a pivotal and highly valued skill in medical practice. Most current interventions for teaching and improving diagnostic reasoning are based on the dual process model of cognition. Recent studies which have applied the popular dual process model to improve diagnostic performance by “Cognitive De-biasing” in clinicians have yielded disappointing results. Thus, it may be appropriate to also consider alternative models of cognitive processing in the teaching and practice of clinical reasoning.

Methods: This is critical-narrative review of the predictive brain model.

Results: The theory of predictive brains is a general, unified and integrated model of cognitive processing based on recent advances in the neurosciences. The predictive brain is characterised as an adaptive, generative, energy-frugal, context-sensitive action-orientated, probabilistic, predictive engine. It responds only to predictive errors and learns by iterative predictive error management, processing and hierarchical neural coding. 

Conclusion: The default cognitive mode of predictive processing may account for the failure of de-biasing since it is not thermodynamically frugal and thus, may not be sustainable in routine practice. Exploiting predictive brains by employing language to optimise metacognition may be a way forward. 

Keywords:            Diagnosis, Bias, Dual Process Theory, Predictive Brains

Practice Highlights

• According to the dual process model of cognition diagnostic errors are caused by bias reasoning.
• Interventions to improve diagnosis based on “Cognitive De-biasing” methods report disappointing results.
• The predict brain is a unified model of cognition which accounts for diagnostic errors, the failure of “Cognitive De-biasing” and may point to effective solutions.
• Using appropriate language as simple rules or thumb, to fine-tune predictive processing meta-cognitively may be a practical strategy to improve diagnostic problem solving.

Core features of the predictive brain model	Clinical reasoning features and processes
The frugal brain and free energy principle(Friston, 2010)	Cognitive load in problem solving (Young et al., 2014)
Iterative matching of top down priors Vs bottom up signals	Inductive foraging (Donner-Banzhoff & Hertwig, 2014; Donner-Banzhoff et al., 2017)
Predictive error processing	Pattern recognition in diagnosis
Recognition-expectation-based signal suppression	Premature closure (Blissett & Sibbald, 2017; Melo et al., 2017)
Hierarchical predictive error coding as learning	Development of illness scripts (Custers, 2014)
Probabilistic-Bayesian inferential approximations	Bayesian inference in clinical reasoning
Context sensitivity	Contextual factors in diagnostic errors(Durning et al., 2010)
Action orientation	Foraging behaviour in clinical diagnosis (Donner-Banzhoff & Hertwig, 2014; Donner-Banzhoff et al., 2017)
Interoception and affect in prediction error management	Gut feel and regret (metacognition)
The precision(reliability/uncertainty) of prediction errors	Clinical uncertainty (metacognition) (Bhise et al., 2017; Simpkin & Schwartzstein, 2016)

Table 1: Core features of the predictive brain model of cognition manifested as clinical reasoning processes

Legend to Figure 1

A summary of the cognitive processes engaged by the predict brain model during clinical diagnosis

A: Active search for diagnostic clues based on prior experience of similar patients in similar situations.

B: Recognition of key features will activate a series of familiar illness script from long term memory to match with the new case.  If this is successful, a diagnosis made and any prediction error signals are rapidly silenced.

C & D: When the illness scripts do not match the presenting features (????), cognition slows down, attention is heightened and further searches are made for additional matching clues and illness scripts. This is iterated                until a satisfactory match is found or a new illness script is generated to account for the mismatch.

E: A new variation in the presenting features for that disease is then encoded in memory as a new illness script in memory and thus, a valuable learning moment.

F: The degree of uncertainty or level of confidence in matching key presenting features to a diagnosis is a meta-cognitive skill and a critical expertise in clinical diagnosis. This corresponds to the precision or gain/weightage of prediction errors (Meta cognition) in the predictive brain model.

Figure 1: A summary of the cognitive processes engaged by the predict brain model during clinical diagnosis

Thermodynamic frugality is a central feature of the predictive brain model and in this system, the primacy of attending only to surprises or PEs is pivotal (Friston, 2010). This might be regard as an energy efficient strategy in coping with cognitive load which has been long recognised as an important consideration in clinical problem solving and learning (Young et al., 2014; Van Merrienboer & Sweller, 2010).

From the first moments of a diagnostic encounter the clinician is alert to clues which might point to the diagnosis and begins to generate possible diagnosis scenarios and simulations based upon her prior experience of similar patients and situations (Donner-Banzhoff & Hertwig, 2014). This is iterative and, from a scanty set of presenting features, a plausible diagnosis may be considered within a few seconds to minutes (Donner-Banzhoff & Hertwig, 2014; Donner-Banzhoff et al., 2017). Thus, a familiar illness script is activated from long term memory to match with the new case (Custers, 2014). If this is successful, a particular diagnosis is recognised and any PE signal is rapidly silenced. Functional MRI studies of clinicians during this process showed that highly salient diagnostic information, reducing uncertainty about the diagnosis, rapidly decreased monitoring activity in the frontoparietal attentional network and may contribute to premature diagnostic closure, an important cause of diagnostic errors (Melo et al., 2017). This may be considered a form of diagnosis or recognition related PE signal suppression analogous to the well know phenomenon of repetitive suppression (Blissett & Sibbald, 2017; Bunzeck & Thiel, 2016; Krupat et al., 2017).

In cases where the illness scripts do not match the presenting features, a PE event is encountered, cognition slows down, attention is heightened and further searches are made for additional matching clues and illness scripts (Custers, 2014). This is iterated until a satisfactory match is found or a new illness script is generated to account for the mismatch. This is then encoded in memory as a new variation in the presenting features for that disease and thus, a valuable learning moment. Bayesian inference is a fundamental feature of both clinical diagnostic reasoning and the predictive brain model (Chater & Oaksford, 2008).

As in the predictive brain model, external contextual factors and internal emotional and physiological responses such as gut feeling and regret, exert profound effects on clinical decision making (M. Djulbegovic et al., 2015; Durning et al., 2010; Stolper & van de Wiel, 2014; Stolper et al., 2014). Also active inductive foraging behaviour in searching for diagnostic clues described in experienced primary physicians is analogous to behaviour directed at reducing PEs (Donner-Banzhoff & Hertwig, 2014; Donner-Banzhoff et al., 2017). The precision or gain/weightage of PEs is manifested metacognitively as uncertainties or levels of confidence in clinical reasoning (Sandved-Smith et al., 2020). Metacognition is a critical capacity and expertise in effective decision making. (Bhise et al., 2017; Fleming & Frith, 2014; Simpkin & Schwartzstein, 2016).

C. Why Applying the Dual Process Model May Not Improve Clinical Reasoning

Recent studies which have applied the popular dual process model to improve diagnostic performance by “cognitive de-biasing” in clinicians have yielded disappointing results (G. R. Norman et al., 2017). Cognitive processing of the predictive brain as the dominant default network mode of operation may account for this setback since de-biasing is not naturistic, requires retrospective “off line” processing after the monitoring salience network has already shut off (Krupat et al., 2017; Melo et al., 2017). It is not thermodynamically frugal and thus, may not be sustainable in routine practice (Friston, 2010; Young et al., 2014). Even Daniel Kahneman himself admits that, despite decades of research in cognitive bias he is unable to exert agency of the moment and de-bias himself (Kahneman, 2013). This will be more so in novice diagnosticians in the training phase who have scanty illness scripts and limited tolerance of any further cognitive loading (Young et al., 2014). The failure to even identify cognitive biases reliably by clinicians due to hindsight bias itself suggests that this intervention will be the least effective one in improving diagnostic reasoning (Zwaan et al., 2016).

 D. Using Words to Fine Tune the Precision of Diagnostic Prediction Error

Daniel Kahneman, the foremost expert on cognitive bias, cautions that, contrary to what some experts in medical education advice, avoiding bias is ineffective in improving decision making under uncertainty (Restrepo et al., 2020). By contrast he suggested that we apply simple, common sense, rules of thumb (Kahneman et al., 2016). I hypothesise that instructing clinical trainees to use appropriate words to self in the diagnostic setting during active, naturalistic PE processing before the diagnosis is made and not as a retrospective counter check to cognition afterwards may be a way forward (Betz et al., 2019; Clark, 2016; Lupyan, 2017). In a multi-center, iterative thematic content analysis of over 2,000 cases of diagnostic errors with a structured taxonomy, Schiff and colleagues identified a limited number of pitfall themes which were overlooked and predisposed physicians to reasoning errors (Reyes Nieva H et al., 2017). These pitfall themes included three which are of particular interest in relation to naturalistic PE processing namely: (1) counter diagnostic cues, (2) things that do not fit and (3) red flags (Reyes Nieva H et al., 2017). Thus, we instructed our student interns and internal medicine residents to pay particular attend to these three diagnostic pitfalls during review of new patients and clinical problems (Lim & Teoh, 2018). They were required to append the following sub-headings to their clerking impression in the patient’s electronic health record (eHR): (a) Counter diagnostic features; (b) Things that do not fit; (c) Red flags. This template was added after the resident had entered his or her numerated list of diagnoses or issues. “Counter diagnostic features” was defined as symptoms, signs or investigations which were inconsistent with the proposed primary diagnosis. “Things that do not fit” was defined as any finding that could not be reasonably accounted for taking into account the main and differential diagnoses. “Red flags” were defined as findings which raised the possibility of a more serious underlying illness requiring early diagnosis or intervention. The attending physicians were required, during bedside rounds, to give feedback on these points and make amendments to the eHR as appropriate. This exercise may give us an opportunity to see if we can improve diagnostic accuracy by using pivotal words-to-self in the appropriate setting to maintain cognitive openness, flexibility and thus, avoid premature (Krupat et al., 2017). It is also a valuable critical, metacognitive thinking habit to inculcate in tyro diagnosticians (Carpenter et al., 2019).

IV. CONCLUSION

The theory of predictive brains has emerged as a major narrative in the understanding of how our mind works. It may account for the limitations of interventions designed to improve diagnostic problem solving which are based on the dual process theory of cognition. Exploiting predictive brains by employing language to optimise metacognition may be a way forward.

Note on Contributor

Lim designed the paper, reviewed the literature, drafted and revised it.

Ethical Approval

There is no ethical approval associated with this paper.

Funding

No funding sources are associated with this paper. 

Declaration of Interest

No conflicts of interest are associated with this paper. 

References

Bar, M. (2011). Predictions in the brain using our past to generate a future (pp. xiv, 383 p. ill. (some col.) 327 cm.).

Barrett, L. F. (2017a). How emotions are made: the secret life of the brain. Houghton Mifflin Harcourt.

Barrett, L. F. (2017b). The theory of constructed emotion: An active inference account of interoception and categorization. Social Cognitive and Affective Neuroscience, 12(1), 1-23.  https://doi.org/10.1093/scan/nsw154

Betz, N., Hoemann, K., & Barrett, L. F. (2019). Words are a context for mental inference. Emotion, 19(8), 1463-1477. https://doi.org/10.1037/emo0000510

Bhise, V., Rajan, S. S., Sittig, D. F., Morgan, R. O., Chaudhary, P., & Singh, H. (2017). Defining and measuring diagnostic uncertainty in medicine: A systematic review. Journal of General Internal Medicine 33,  103–115. https://doi.org/10.1007/s11606-017-4164-1

Blissett, S., & Sibbald, M. (2017). Closing in on premature closure bias. Medical Education, 51(11), 1095-1096. https://doi.org/10.1111/medu.13452

Bunzeck, N., & Thiel, C. (2016). Neurochemical modulation of repetition suppression and novelty signals in the human brain. Cortex, 80, 161-173. https://doi.org/10.1016/j.cortex.2015.10.013

Carpenter, J., Sherman, M. T., Kievit, R. A., Seth, A. K., Lau, H., & Fleming, S. M. (2019). Domain-general enhancements of metacognitive ability through adaptive training. Journal of Experimental Psychology. General, 148(1), 51-64. https://doi.org/10.1037/xge0000505

Chater, N., & Oaksford, M. (2008). The probabilistic mind : prospects for Bayesian cognitive science. Oxford University Press.

Clark, A. (2013). Whatever next? Predictive brains, situated agents, and the future of cognitive science. The Behavioral and Brain Sciences, 36(3), 181–204. https://doi.org/10.1017/S0140525X12000477

Clark, A. (2016). Surfing uncertainty : Prediction, action, and the embodied mind: Oxford University Press.

Croskerry, P. (2009). Clinical cognition and diagnostic error: Applications of a dual process model of reasoning. Advances in Health Sciences Education : Theory and Practice, 14 Suppl 1, 27–35. https://doi.org/10.1007/s10459-009-9182-2

Croskerry, P. (2013). From mindless to mindful practice–cognitive bias and clinical decision making. The New England Journal of Medicine, 368(26), 2445–2448. https://doi.org/10.1056/NEJMp1303712

Custers, E. J. (2014). Thirty years of illness scripts: Theoretical origins and practical applications. Medical Teacher, 1-6. https://doi.org/10.3109/0142159X.2014.956052

Djulbegovic, B., Hozo, I., Beckstead, J., Tsalatsanis, A., & Pauker, S. G. (2012). Dual processing model of medical decision-making. BMC Medical Informatics and Decision Making, 12, 94. https://doi.org/10.1186/1472-6947-12-94

Djulbegovic, M., Beckstead, J., Elqayam, S., Reljic, T., Kumar, A., Paidas, C., & Djulbegovic, B. (2015). Thinking styles and regret in physicians. Public Library of Science One, 10(8), e0134038. https://doi.org/10.1371/journal.pone.0134038

Donner-Banzhoff, N., & Hertwig, R. (2014). Inductive foraging: Improving the diagnostic yield of primary care consultations. European Journal of General Practice, 20(1), 69–73. https://doi.org/10.3109/13814788.2013.805197

Donner-Banzhoff, N., Seidel, J., Sikeler, A. M., Bosner, S., Vogelmeier, M., Westram, A., & Gigerenzer, G. (2017). The phenomenology of the diagnostic process: A primary care-based survey. Medical Decision Making, 37(1), 27-34. https://doi.org/10.1177/0272989X16653401

Durning, S. J., Artino, A. R., Jr., Pangaro, L. N., van der Vleuten, C., & Schuwirth, L. (2010). Perspective: redefining context in the clinical encounter: Implications for research and training in medical education. Academic Medicine: Journal of the Association of American Medical Colleges, 85(5), 894–901. https://doi.org/10.1097/ACM.0b013e3181d7427c

Evans, J. S. (2008). Dual-processing accounts of reasoning, judgment, and social cognition. Annual Review of Psychology, 59, 255–278. https://doi.org/10.1146/annurev.psych.59.103006.093629

Feldman, H., & Friston, K. J. (2010). Attention, uncertainty, and free-energy. Frontiers in Human Neuroscience, 4, 215. https://doi.org/10.3389/fnhum.2010.00215

Fleming, S. M., & Frith, C. D. (2014). The cognitive neuroscience of metacognition. Springer.

Friston, K. (2010). The free-energy principle: A unified brain theory? Nature Reviews. Neuroscience, 11(2), 127–138. https://doi.org/10.1038/nrn2787

Friston, K., FitzGerald, T., Rigoli, F., Schwartenbeck, P., & Pezzulo, G. (2017). Active inference: A process theory. Neural Computation, 29(1), 1–49. https://doi.org/10.1162/NECO_a_00912

Friston, K., & Kiebel, S. (2009). Predictive coding under the free-energy principle. Philosophical Transactions of the Royal Society of London. Series B, Biological sciences, 364(1521), 1211–1221. https://doi.org/10.1098/rstb.2008.0300

Gilbert, D. T., & Wilson, T. D. (2007). Prospection: Experiencing the future. Science, 317(5843), 1351-1354. https://doi.org/10.1126/science.1144161

Gupta, A., Snyder, A., Kachalia, A., Flanders, S., Saint, S., & Chopra, V. (2017). Malpractice claims related to diagnostic errors in the hospital. BMJ Quality and Safety, 27(1), 53-60. https://doi.org/10.1136/bmjqs-2017-006774

Hohwy, J. (2013). The predictive mind. Oxford University Press..

Kahneman, D. (2013). Thinking, fast and slow (1st pbk. ed.).  Farrar, Straus and Giroux.

Kahneman, D., Rosenfield, A. M., Gandhi, L., & Blaser, T. O. M. (2016). NOISE: How to overcome the high, hidden cost of inconsistent decision making. (cover story). Harvard Business Review, 94(10), 38-46. Retrieved from http://libproxy1.nus.edu.sg/login?url=http://search.ebscohost.com/login.aspx?direct=true&db=buh&AN=118307773&site=ehost-live

Kersten, D., Mamassian, P., & Yuille, A. (2004). Object perception as bayesian inference. Annual Review of Psychology, 55, 271–304. https://doi.org/10.1146/annurev.psych.55.090902.142005

Kleckner, I. R., Zhang, J., Touroutoglou, A., Chanes, L., Xia, C., Simmons, W. K., & Feldman Barrett, L. (2017). Evidence for a large-scale brain system supporting allostasis and interoception in humans. Nature Human Behaviour, 1, 0069. https://doi.org/10.1038/s41562-017-0069

Kolossa, A., Kopp, B., & Fingscheidt, T. (2015). A computational analysis of the neural bases of Bayesian inference. Neuroimage, 106, 222-237.  https://doi.org/10.1016/j.neuroimage.2014.11.007

Krupat, E., Wormwood, J., Schwartzstein, R. M., & Richards, J. B. (2017). Avoiding premature closure and reaching diagnostic accuracy: Some key predictive factors. Medical Education, 51(11), 1127-1137. https://doi.org/10.1111/medu.13382

Kwisthout, J., Bekkering, H., & van Rooij, I. (2017a). To be precise, the details don’t matter: On predictive processing, precision, and level of detail of predictions. Brain and Cognition, 112, 84–91. https://doi.org/10.1016/j.bandc.2016.02.008

Kwisthout, J., Phillips, W. A., Seth, A. K., van Rooij, I., & Clark, A. (2017b). Editorial to the special issue on perspectives on human probabilistic inference and the ‘Bayesian brain’. Brain and Cognition, 112, 1-2. https://doi.org/10.1016/j.bandc.2016.12.002

Lim T.K., & Teoh, C. M. (2018). Exploiting predictive brains for better diagnosis. Diagnosis (Berl), 5(3), eA40. Retrieved from https://www.degruyter.com/view/journals/dx/5/3/article-peA1.xml

Lupyan, G. (2017). Changing what you see by changing what you know: The role of attention. Frontiers in Psychology, 8, 553. https://doi.org/10.3389/fpsyg.2017.00553

Melo, M., Gusso, G. D. F., Levites, M., Amaro, E., Jr., Massad, E., Lotufo, P. A., & Friston, K. J. (2017). How doctors diagnose diseases and prescribe treatments: An fMRI study of diagnostic salience. Scientific Reports, 7(1), 1304. http://observatorio.fm.usp.br/handle/OPI/19951

Monteiro, S., Sherbino, J., Sibbald, M., & Norman, G. (2020). Critical thinking, biases and dual processing: The enduring myth of generalisable skills. Medical Education, 54(1), 66-73. https://doi.org/10.1111/medu.13872

Montgomery, K. (2006). How doctors think: Clinical judgement and the practice of medicine. Oxford University Press.

Nasser, H. M., Calu, D. J., Schoenbaum, G., & Sharpe, M. J. (2017). The dopamine prediction error: Contributions to associative models of reward learning. Frontiers in Psychology, 8, 244. https://doi.org/10.3389/fpsyg.2017.00244

Norman, G., Sherbino, J., Dore, K., Wood, T., Young, M., Gaissmaier, W., & Monteiro, S. (2014). The etiology of diagnostic errors: A controlled trial of system 1 versus system 2 reasoning. Academic Medicine: Journal of the Association of American Medical Colleges, 89(2), 277–284. https://doi.org/10.1097/ACM.0000000000000105

Norman, G. R., Monteiro, S. D., Sherbino, J., Ilgen, J. S., Schmidt, H. G., & Mamede, S. (2017). The Causes of Errors in Clinical Reasoning: Cognitive Biases, Knowledge Deficits, and Dual Process Thinking. Academic Medicine: Journal of the Association of American Medical Colleges, 92(1), 23–30. https://doi.org/10.1097/ACM.0000000000001421

O’Sullivan, E. D., & Schofield, S. J. (2019). A cognitive forcing tool to mitigate cognitive bias – A randomised control trial. BMC Medical Education, 19(1), 12. https://doi.org/10.1186/s12909-018-1444-3

Picard, F., & Friston, K. (2014). Predictions, perception, and a sense of self. Neurology, 83(12), 1112-1118. https://doi.org/10.1212/WNL.0000000000000798

Reilly, J. B., Ogdie, A. R., Von Feldt, J. M., & Myers, J. S. (2013). Teaching about how doctors think: A longitudinal curriculum in cognitive bias and diagnostic error for residents. BMJ Quality & Safety, 22(12), 1044–1050. https://doi.org/10.1136/bmjqs-2013-001987

Rencic, J., Trowbridge, R. L., Jr., Fagan, M., Szauter, K., & Durning, S. (2017). Clinical reasoning education at us medical schools: Results from a national survey of internal medicine clerkship directors.  Journal of General Internal Medicine, 32(11), 1242–1246. https://doi.org/10.1007/s11606-017-4159-y

Restrepo, D., Armstrong, K. A., & Metlay, J. P. (2020). Annals Clinical Decision Making: Avoiding Cognitive Errors in Clinical Decision Making. Annals of Internal Medicine, 172(11), 747–751. https://doi.org/10.7326/M19-3692

Reyes Nieva H., V. M., Wright A, Singh H, Ruan E, Schiff G. (2017). Diagnostic Pitfalls: A New Approach to Understand and Prevent Diagnostic Error. In Diagnosis (Vol. 4, pp. eA1). https://www.degruyter.com/view/journals/dx/5/4/article-peA59.xml

Sandved-Smith, L., Hesp, C., Lutz, A., Mattout, J., Friston, K., & Ramstead, M. (2020, June 10). Towards a formal neurophenomenology of metacognition: Modelling meta-awareness, mental action, and attentional control with deep active inference. https://doi.org/10.31234/osf.io/5jh3c

Schuwirth, L. (2017). When I say … dual-processing theory. Medical Education, 51(9), 888–889. https://doi.org/10.1111/medu.13249

Seligman, M. E. P. (2016). Homo Prospectus. Oxford University Pres.

Seth, A. K., Suzuki, K., & Critchley, H. D. (2011). An interoceptive predictive coding model of conscious presence. Frontiers in Psychology, 2, 395. https://doi.org/10.3389/fpsyg.2011.00395

Shannon, C. E., Sloane, N. J. A., Wyner, A. D., & IEEE Information Theory Society. (1993). Claude Elwood Shannon : Collected Papers. IEEE Press.

Sherbino, J., Kulasegaram, K., Howey, E., & Norman, G. (2014). Ineffectiveness of cognitive forcing strategies to reduce biases in diagnostic reasoning: A controlled trial. Canadian Journal of Emergency Medicine, 16(1), 34–40. https://doi.org/10.2310/8000.2013.130860

Shipp, S. (2016). Neural Elements for Predictive Coding. Frontiers in Psychology, 7, 1792. https://doi.org/10.3389/fpsyg.2016.01792

Sibbald, M., Sherbino, J., Ilgen, J. S., Zwaan, L., Blissett, S., Monteiro, S., & Norman, G. (2019). Debiasing versus knowledge retrieval checklists to reduce diagnostic error in ECG interpretation. Advances in Health Sciences Education: Theory and Practice, 24(3), 427–440. https://doi.org/10.1007/s10459-019-09875-8

Simpkin, A. L., & Schwartzstein, R. M. (2016). Tolerating uncertainty – The next medical revolution? The New England Journal of Medicine, 375(18), 1713–1715. https://doi.org/10.1056/NEJMp1606402

Simpkin, A. L., Vyas, J. M., & Armstrong, K. A. (2017). Diagnostic Reasoning: An endangered competency in internal medicine training. Annals of Internal Medicine, 167(7), 507–508. https://doi.org/10.7326/M17-0163

Singh, H., & Graber, M. L. (2015). Improving diagnosis in health care- The next imperative for patient safety. The New England Journal of Medicine, 373(26), 2493–2495. https://doi.org/10.1056/NEJMp1512241

Singh, H., Meyer, A. N., & Thomas, E. J. (2014). The frequency of diagnostic errors in outpatient care: Estimations from three large observational studies involving US adult populations. BMJ Quality & Safety, 23(9), 727–731. https://doi.org/10.1136/bmjqs-2013-002627

Skinner, T. R., Scott, I. A., & Martin, J. H. (2016). Diagnostic errors in older patients: A systematic review of incidence and potential causes in seven prevalent diseases. International Journal of General Medicine, 9, 137–146. https://doi.org/10.2147/IJGM.S96741

Spratling, M. W. (2017a). A hierarchical predictive coding model of object recognition in natural images. Cognitive Computation, 9(2), 151–167. https://doi.org/10.1007/s12559-016-9445-1

Spratling, M. W. (2017b). A review of predictive coding algorithms. Brain and Cognition, 112, 92–97. https://doi.org/10.1016/j.bandc.2015.11.003

Steinberg, E. E., Keiflin, R., Boivin, J. R., Witten, I. B., Deisseroth, K., & Janak, P. H. (2013). A causal link between prediction errors, dopamine neurons and learning. Nature Neuroscience, 16(7), 966–973. https://doi.org/10.1038/nn.3413

Sterling, P. (2012). Allostasis: A model of predictive regulation. Physiology & Behavior, 106(1), 5–15. https://doi.org/10.1016/j.physbeh.2011.06.0044

Stolper, C. F., & van de Wiel, M. W. (2014). EBM and gut feelings. Medical Teacher, 36(1), 87-88. https://doi.org/10.3109/0142159X.2013.835390

Stolper, C. F., Van de Wiel, M. W., Hendriks, R. H., Van Royen, P., Van Bokhoven, M. A., Van der Weijden, T., & Dinant, G. J. (2014). How do gut feelings feature in tutorial dialogues on diagnostic reasoning in GP traineeship? Advances in Health Sciences Education: Theory and Practice, 20(2), 499–513. https://doi.org/10.1007/s10459-014-9543-3

Teufel, C., & Fletcher, P. C. (2020). Forms of prediction in the nervous system. Nature Reviews Neuroscience, 21(4), 231–242. https://doi.org/10.1038/s41583-020-0275-5

Ting, C. C., Yu, C. C., Maloney, L. T., & Wu, S. W. (2015). Neural mechanisms for integrating prior knowledge and likelihood in value-based probabilistic inference. The Journal of Neuroscience : The Official Journal of the Society for Neuroscience, 35(4), 1792–1805. https://doi.org/10.1523/JNEUROSCI.3161-14.2015

Tschantz, A., Seth, A. K., & Buckley, C. L. (2020). Learning action-oriented models through active inference. PLoS Computational Biology, 16(4), e1007805. https://doi.org/10.1371/journal.pcbi.1007805

Van Merrienboer, J. J., & Sweller, J. (2010). Cognitive load theory in health professional education: Design principles and strategies. Medical Education, 44(1), 85-93. https://doi.org/10.1111/j.1365-2923.2009.03498.x

Walsh, J. N., Knight, M., & Lee, A. J. (2017). Diagnostic errors: Impact of an educational intervention on pediatric primary care. Journal of Pediatric Health Care : Official Publication of National Association of Pediatric Nurse Associates & Practitioners, 32(1), 53–62. https://doi.org/10.1016/j.pedhc.2017.07.004

Walsh, K. S., McGovern, D. P., Clark, A., & O’Connell, R. G. (2020). Evaluating the neurophysiological evidence for predictive processing as a model of perception. Annals of the New York Academy of Sciences, 1464(1), 242–268. https://doi.org/10.1111/nyas.14321

Young, J. Q., Van Merrienboer, J., Durning, S., & Ten Cate, O. (2014). Cognitive load theory: Implications for medical education: AMEE Guide No. 86. Medical Teacher, 36(5), 371-384. https://doi.org/10.3109/0142159X.2014.889290

Zwaan, L., Monteiro, S., Sherbino, J., Ilgen, J., Howey, B., & Norman, G. (2016). Is bias in the eye of the beholder? A vignette study to assess recognition of cognitive biases in clinical case workups. BMJ Quality & Safety. 26(2), 104–110. https://doi.org/10.1136/bmjqs-2015-005014

Zwaan, L., Schiff, G. D., & Singh, H. (2013). Advancing the research agenda for diagnostic error reduction. BMJ Quality & Safety, 22 Suppl 2, ii52-ii57. https://doi.org/10.1136/bmjqs-2012-001624

*Lim Tow Keang
Department of Medicine 
National University Hospital
5 Lower Kent Ridge Rd
Singapore 119074
Email: mdclimtk@nus.edu.sg

Published online: 5 January, TAPS 2021, 6(1), 1-2
https://doi.org/10.29060/TAPS.2021-6-1/EV6N1

In our January 2020 Editorial, we drew the attention of our readers to “Grit in Healthcare Education and Practice”. In particular, we focused on developing the “Grit” of students and trainees; medical students who are well-equipped with the ‘Power of Grit’ will display a “passion for patient well-being and perseverance in the pursuit of that goal [which] become social norms at the individual, team and institutional levels” (Lee & Duckworth, 2018). However, never could we imagine then that such an attribute (i.e. ‘Grit’) would become contextual so soon, as exemplified by the passion and perseverance of healthcare practitioners in patient care in their response to the serious disruptions in individual health (including fatalities) caused by the Covid-19 pandemic!

We are pleased to have this opportunity to share with our readers, yet once again, the unexpected course of events associated with the Covid-19 pandemic which brought out the best in many on a global scale. In particular, as the education and training of medical students, residents and those in allied health institutions were disrupted by the Covid-19 pandemic. The educators supported by the administration in medical and health professions institutions designed curricula innovations that incorporated culturally sensitive interventions to develop individual resilience and well-being in order to support the community of learners—including students, faculty, administrators involved and of course, patients.

The current Covid-19 pandemic served as a catalyst that provided opportunities for educators to rapidly and creatively design safe, yet effective, novel and innovative solutions to ensure continuation in the education and training of medical and closely allied health professional students (Samarasekera, Goh, & Lau, 2020). Thus, there is a need to break away from decades of tradition in designing such educational strategies for continued student learning, as a rapid response to the Covid-19 pandemic. In this context then, both, institutional as well as program leadership are required to facilitate the process for the design of creative, yet safe, effective and innovative strategies for the continuation of student learning; such a step is expected to mitigate the disruptive effects of the Covid-19 pandemic! In this context then, educators leverage on available technology as the preferred mode for the delivery of instruction to students. The learning environment was also transformed from one that was predominantly classroom-based to one that is mainly online. It is also gratifying that, both, junior and senior faculty have embraced the use of technology, although some degree of ‘resistance’ to the use of technology in education was experienced earlier. Perhaps, a caveat should be added: student learning using technology over a long period of time may result in a lack of social interaction among the students and, consequently, a lack of preparation for teamwork which is so critical for healthcare practice in the 21st century.

The Covid-19 pandemic has also exposed wider societal gaps which were seldom evident previously, but needs to be addressed. It is useful then to note that The Lancet Global Independent Commission had already expressed, in its Report (Frenk et al., 2010) that “Indeed, the use of IT might be the most important driver in transformative learning ….” and that “Advanced information technology is important not only for more efficient education of health professionals; its existence also demands a change in competencies.” The ‘Report’ also drew attention to the fact that “IT-empowered learning is already a reality for the younger generation in most countries, ….” However, due to financial constraints, the ‘Report’ also cautioned that “Not all students, of course, have full access to IT resources” and suggested “A global policy to overcome such unequal distribution of digital resources [referred to as the digital divide] ….” Attention to such inequalities have also been recently addressed by Blundell, Costa Dias, Joyce, and Xu (2020).

A major concern of medical and allied health professional institutions is the well-being of students and staff who ensure the continuation of student education and training disrupted by the Covid-19 pandemic. Many institutions provided strong support to students and staff in such challenging times. Students received financial support and, if required, counselling as well in order to enhance their psycho-social well-being. Students infected by the virus or who were quarantined received special care. Many institutional policies were swiftly revised to match the rapidly changing environment: clear lines of communication were established for staff and students (Ashokka, Ong, Tay, Loh, Gee, & Samarasekera, 2020).

A more resilient community of staff and students have remarkably emerged from the trials and tribulations experienced: students have adapted rapidly to blended and virtual learning environments. Students have also organised their learning engagements around virtual student communities, as most institutions have minimised their face-to-face classroom activities. Faculty responded by designing a more adaptive curriculum that is flexible to the needs of the learner. Pre-clinical and clinical learning activities were further refined and streamlined with the removal of some content and examinations—a process unthinkable prior to the crisis (disruptions) of the Covid-19 pandemic; the prior status involved strict control of the curricula which was managed by the institution and/or professional / statutory bodies. Within a short period of time newer course materials and assessment instruments, all aligned to support online, blended or hybrid learning requirements, were developed. However, the most significant contribution from staff to the disrupted student learning is the proactive support to optimise and meet the needs of learners in the crisis triggered by the Covid-19 pandemic! Such an action by the staff were greatly appreciated; stronger bonds with a closer community spirit between the students and staff were soon established.

In conclusion, it can be said that medical and allied health professional educators have benefitted much (lessons learnt) from the disruptive effects of the Covid-19 pandemic on student learning. Instead of wallowing in self-pity, sadness and simply awaiting time-out. A determined and focused faculty can mitigate the effects of the formidable challenge posed by the Covid-19 pandemic by responding rapidly to make changes to the learning environment—using appropriate technology to deliver instruction to students, in order to ensure the continuation of safe, timely, and quality education!

Providing constant support to students by the staff and the institution will help students develop relevant coping strategies that foster their resilience and well-being. Ultimately, a community of learners and practitioners will emerge with the ability to provide and maintain quality healthcare during challenging times like the one we are now experiencing.

 

Dujeepa D. Samarasekera & Matthew C. E. Gwee
Centre for Medical Education (CenMED), NUS Yong Loo Lin School of Medicine,
National University Health System, Singapore

 

 

Ashokka, B., Ong, S. Y., Tay, K. H., Loh, N., Gee, C. F., & Samarasekera, D. D. (2020). Coordinated responses of academic medical centres to pandemics: Sustaining medical education during COVID-19. Medical Teacher, 42(7), 762-771.

Blundell, R., Costa Dias, M., Joyce, R., & Xu, X. (2020). COVID‐19 and Inequalities. Fiscal Studies, 41(2), 291-319.

Frenk, J., Chen, L., Bhutta, Z. A., Cohen, J., Crisp, N., Evans, T., … & Kistnasamy, B. (2010). Health professionals for a new century: Transforming education to strengthen health systems in an interdependent world. The Lancet, 376(9756), 1923-1958.

Lee, T. H., & Duckworth, A. L. (2018). Organizational grit. Harvard Business Review, 96(5), 98-105.

Samarasekera, D. D., Goh, D. L. M., & Lau, T. C. (2020). Medical school approach to manage the current COVID-19 crisis. Academic Medicine, 95(8), 1126-1127.

Submitted: 4 May 2020
Accepted: 3 August 2020
Published online: 5 January, TAPS 2021, 6(1), 3-29
https://doi.org/10.29060/TAPS.2021-6-1/RA2351

Elisha Wan Ying Chia^1,2, Huixin Huang^1,2, Sherill Goh^1,2, Marlyn Tracy Peries^1,2, Charlotte Cheuk Yiu Lee^2,3, Lorraine Hui En Tan^1,2, Michelle Shi Qing Khoo^1,2, Kuang Teck Tay^1,2, Yun Ting Ong^1,2, Wei Qiang Lim^1,2, Xiu Hui Tan^1,2, Yao Hao Teo^1,2, Cheryl Shumin Kow^1,2, Annelissa Mien Chew Chin⁴, Min Chiam⁵, Jamie Xuelian Zhou^2,6,7 & Lalit Kumar Radha Krishna^1,2,5,7-10

¹Yong Loo Lin School of Medicine, National University of Singapore, Singapore; ²Division of Supportive and Palliative Care, National Cancer Centre Singapore, Singapore; ³Alice Lee Centre for Nursing Studies, National University of Singapore, Singapore; ⁴Medical Library, National University of Singapore Libraries, National University of Singapore, Singapore; ⁵Division of Cancer Education, National Cancer Centre Singapore, Singapore; ⁶Lien Centre of Palliative Care, Duke-NUS Graduate Medical School, Singapore; ⁷Duke-NUS Graduate Medical School, Singapore; ⁸Centre for Biomedical Ethics, National University of Singapore, Singapore; ⁹Palliative Care Institute Liverpool, Academic Palliative & End of Life Care Centre, University of Liverpool; ¹⁰PalC, The Palliative Care Centre for Excellence in Research and Education, Singapore

Abstract

Introduction: Whilst the importance of effective communications in facilitating good clinical decision-making and ensuring effective patient and family-centred outcomes in Intensive Care Units (ICU)s has been underscored amidst the global COVID-19 pandemic, training and assessment of communication skills for healthcare professionals (HCPs) in ICUs remain unstructured

Methods: To enhance the transparency and reproducibility, Krishna’s Systematic Evidenced Based Approach (SEBA) guided Systematic Scoping Review (SSR), is employed to scrutinise what is known about teaching and evaluating communication training programmes for HCPs in the ICU setting. SEBA sees use of a structured search strategy involving eight bibliographic databases, the employ of a team of researchers to tabulate and summarise the included articles and two other teams to carry out content and thematic analysis the included articles and comparison of these independent findings and construction of a framework for the discussion that is overseen by the independent expert team.

Results: 9532 abstracts were identified, 239 articles were reviewed, and 63 articles were included and analysed. Four similar themes and categories were identified. These were strategies employed to teach communication, factors affecting communication training, strategies employed to evaluate communication and outcomes of communication training.  

Conclusion: This SEBA guided SSR suggests that ICU communications training must involve a structured, multimodal approach to training. This must be accompanied by robust methods of assessment and personalised timely feedback and support for the trainees. Such an approach will equip HCPs with greater confidence and prepare them for a variety of settings, including that of the evolving COVID-19 pandemic.

Keywords:           Communication, Intensive Care Unit, Assessment, Skills Training, Evaluation, COVID-19, Medical Education

Practice Highlights

• The global COVID-19 pandemic has underscored the importance of effective communications in the Intensive Care Unit (ICU).
• ICU communications training should adopt a longitudinal, structured and multimodal approach.
• Robust stepwise evaluation of learner outcomes via Kirkpatrick’s Hierarchy is needed.
• Supportive host organisation and conducive learning environment and are key to successful curricula.

I. INTRODUCTION

The COVID-19 pandemic has placed immense strain on intensive care units (ICU)s with healthcare teams and resources stretched to meet the sudden increased healthcare demands of critically ill patients. To further complicate the situation, ICU teams are called to not only communicate closely with colleagues in a bid to support them but also counsel families confronting acute distress and uneasy waits separated from their loved ones due to restrictions to visiting in an effort to limit the spread of this pandemic (Ministry of Health, 2020; World Health, 2020). From breaking bad news (Blackhall, Erickson, Brashers, Owen, & Thomas, 2014; J. Yuen & Carrington Reid, 2011), to conveying the need for sedation and intubation (Carrillo Izquierdo, Diaz Agea, Jimenez Rodriguez , Leal Costa, & Sanchez Exposito, 2018) and providing progress reports on critically ill patients (Curtis et al., 2005; Curtis, White, Curtis, & White, 2008; Yang et al., 2020), communication skills amongst ICU healthcare professionals (HCPs) are pivotal in reassuring anxious, emotional and stressed patients and families (Ahrens, Yancey, & Kollef, 2003; Foa et al., 2016; Kirchhoff et al., 2002). Good communication in the ICU has also been shown to improve patient-physician relationships (K. G. Anderson & Milic, 2017), patient and family-centred outcomes, quality of care, and patient and family satisfaction (Bloomer, Endacott, Ranse, & Coombs, 2017; Cao et al., 2018; Currey, Oldland, Considine, Glanville, & Story, 2015). Effective communications between HCPs in ICU also enhances clinical decision-making (Kleinpell, 2014), reduces medication and treatment errors (Clark, Squire, Heyme, Mickle, & Petrie, 2009; Happ et al., 2014; Sandahl et al., 2013), decreases physician burnout (Rachwal et al., 2018), and improves staff retention and satisfaction (Hope et al., 2015).

With evidence suggesting that poor communication skills (Downar, Knickle, Granton, & Hawryluck, 2012; Foa et al., 2016) and training (Smith, O’Sullivan, Lo, & Chen, 2013) are likely to increase patients’ (Dithole, Sibanda, Moleki, & Thupayagale ‐ Tshweneagae, 2016) and families’ (Curtis et al., 2008) stress, adversely affect care and recovery (Dithole et al., 2016), and increase healthcare costs (Kalocsai et al., 2018), some authors have suggested that effective communication skills are at least as important (Adams, Mannix, & Harrington, 2017; Cicekci et al., 2017; Van Mol, Boeter, Verharen, & Nijkamp, 2014) to good patient care as clinical acumen (Curtis et al., 2001a). Yet despite evidence of the importance of communication skills in ICU, communication skills training remains inconsistent, variable and not evidence-based in most ICU settings (Adams et al., 2017; Berlacher, Arnold, Reitschuler-Cross, Teuteberg, & Teuteberg, 2017; Bloomer et al., 2017; D. A. Boyle et al., 2017; Miller et al., 2018; Sanchez Exposito et al., 2018).

With this in mind, a systematic scoping review (SSR) is proposed to map current approaches to communications skills training in ICUs (Munn et al., 2018) and potentially guide design of a communications training programme. An SSR allows for systematic extraction and synthesis of actionable and applicable information whilst summarising available literature across a wide range of pedagogies and practice settings employed to understand what is known about teaching and evaluating communication training programmes for HCPs in the ICU setting (Munn et al., 2018).

 II. METHODS

To overcome concerns about the transparency and reproducibility of SSR, a novel approach called Krishna’s Systematic Evidenced Based Approach (henceforth SEBA) is proposed (Kow et al., 2020; Krishna et al., 2020; Ngiam et al., 2020). This SEBA guided SSR (henceforth SSRs in SEBA) adopts a constructivist perspective to map this complex topic from multiple angles (Popay et al., 2006) whilst a relativist lens helps account for variability in communication skills training (Crotty, 1998; Ford, Downey, Engelberg, Back, & Curtis, 2012; Pring, 2000; Schick-Makaroff, MacDonald, Plummer, Burgess, & Neander, 2016).

To provide a balanced review, the research team was supported by the medical librarians from the National University of Singapore’s (NUS) Yong Loo Lin School of Medicine (YLLSoM), the National Cancer Centre Singapore (NCCS) and local educational experts and clinicians at the NCCS, the Palliative Care Institute Liverpool, YLLSoM and Duke-NUS Medical School (henceforth the expert team). The research and expert teams adopted an interpretivist approach as they proceeded through the five stages of SEBA (Figure 1).

Figure 1. The SEBA Process

A. Stage 1: Systematic Approach

1) Determining the title and research question: The research and expert teams agreed upon the goals, population, context and concept to be evaluated in this SSR. The two teams then agreed that the primary research question should be “What is known about teaching and evaluating communication training programs for HCPs in the ICU setting?” The secondary research questions were “How are communication skills taught and assessed in the ICU setting?” and “How effective have such interventions been as described in the published literature?”

2) Inclusion criteria: A Population, Intervention, Comparison, Outcome, Study Design (PICOS) format was adopted to guide the research process (Peters, Godfrey, Khalil, et al., 2015a; Peters, Godfrey, McInerney, et al., 2015b) (Table 1).

PICOS	Inclusion Criteria	Exclusion Criteria
Population	·       Undergraduate and postgraduate healthcare providers (e.g. doctors, medical students, nurses, social workers) within ICU setting ·       ICU settings including medical, surgical, cardiology and neurology ICU ·       Communication between healthcare providers and patients in the ICU, or between healthcare providers in the ICU and patients’ families ·       Communication between or within healthcare providers’ teams in the ICU	·       Articles focusing solely on neonatal/ paediatric ICU setting ·       Articles focusing solely on speech therapy/ physical therapy/ occupational therapy ·       Non-ICU settings (e.g. general wards, emergency department) ·       Non-medical professions (e.g. Science, Veterinary, Dentistry) ·       Communication carried out over technological platforms
Intervention	·       Need for/ importance of interventions to teach communication in ICU setting ·       Facilitators and barriers to teaching communication in ICU setting ·       Recommendations, interventions, methods (e.g. tools, simulations, videos), curriculum content and assessments used for teaching communication in ICU setting
Comparison	·       Comparisons of various interventions, methods, curricula and evaluation methods used to teach or assess communication in ICU setting and its impact upon patients, healthcare providers, healthcare, and society
Outcome	·       Impact of interventions on patients, healthcare providers, healthcare, and society ·       Evaluation methods to assess interventions, methods, or curriculum used to teach communication
Study design	·       Articles in English or translated to English ·       All study designs including: o    Mixed methods research, meta-analyses, systematic reviews, randomised controlled trials, cohort studies, case-control studies, cross-sectional studies, and descriptive papers o    Case reports and series, ideas, editorials, and perspectives ·       Publication dates: 1st January 2000 – 31st December 2019 ·       Databases: PubMed, ERIC, JSTOR, Embase, CinaHL, Scopus, PsycINFO, Google Scholar

Table 1. PICOS

Nine members of the research team carried out independent searches for articles published between 1st January 2000 – 31st December 2019 in eight bibliographic databases (PubMed, ERIC, JSTOR, Embase, CINAHL, Scopus, Psycinfo and Google Scholar). The searches were carried out between 27th January 2020 and 14th February 2020. The PubMed search strategy can be found in Supplementary Material A. An independent hand search was done to identify key articles.

3) Extracting and charting: Nine members of the research team independently reviewed the titles and abstracts identified and created individual lists of titles to be included which were discussed online. Consensus was achieved on the final list of articles to be included using (Sambunjak, Straus, & Marusic, 2010)’s “negotiated consensual validation” approach through collaborative discussion and negotiation on points of disagreement on online meetings.

B. Stage 2. Split Approach

Working in three independent groups, the reviewers analysed the included articles using the ‘split approach’ (Ng et al., 2020). In one group, four researchers independently reviewed and summarised all the included articles in keeping with according recommendations set out by Wong, Greenhalgh, Westhorp, Buckingham, and Pawson (2013)’s “RAMESES publication standards: meta-narrative reviews” and Popay et al. (2006)’s “Guidance on the conduct of narrative synthesis in systematic reviews”. The four research team members then discussed their individual findings at online meetings and employed ‘negotiated consensual validation’ to achieve consensus on the tabulated summaries (Sambunjak et al., 2010). The tabulated summaries served to highlight key points from the included articles.

The four members of the research team also employed the Medical Education Research Study Quality Instrument (MERSQI) (Reed et al., 2008) and the Consolidated Criteria for Reporting Qualitative Studies (COREQ) (Tong, Sainsbury, & Craig, 2007) also evaluated the quality of qualitative and quantitative studies included in this review.

Concurrently, the second group of five researchers analysed all the included articles using (Braun & Clarke, 2006)’s approach to thematic analysis then discussed their individual findings at online meetings and employed ‘negotiated consensual validation’ to achieve consensus on the final themes (Sambunjak et al., 2010). The third group of four researchers employed Hsieh and Shannon (2005)’s approach to directed content analysis to independently analyse all the included articles, discussed their independent findings online and employed ‘negotiated consensual validation’ to achieve consensus on the final themes (Sambunjak et al., 2010). This split approach consisting of the tabulated summaries and concurrent thematic analysis and content analysis enhances the reliability of the analyses. The tabulated summaries also help ensure that important themes are not lost.

1) Thematic analysis: Phase 1 of Braun and Clarke (2006)’s approach saw the team ‘actively’ reading the included articles to find meaning and patterns in the data. In phase 2, ‘codes’ were constructed from the ‘surface’ meaning (Braun & Clarke, 2006; Sawatsky, Parekh, Muula, Mbata, & Bui, 2016; Voloch, Judd, & Sakamoto, 2007) and collated into a code book to code and analyse the rest of the articles using an iterative step-by-step process. As new codes emerged, these were associated with previous codes and concepts (Price & Schofield, 2015). In phase 3, the categories were organised into themes that best depict the data. In phase 4, the themes were refined to best represent the whole data set and discussed. In phase 5, the research team discussed the results of their independent analysis online and at reviewer meetings. “Negotiated consensual validation” was used to determine a final list of themes (Sambunjak et al., 2010).

2) Directed content analysis: Hsieh and Shannon (2005)’s approach to directed content analysis (Hsieh & Shannon, 2005) was employed in three stages.

Using deductive category application (Elo & Kyngäs, 2008; Wagner-Menghin, de Bruin, & van Merriënboer, 2016), the first stage (Mayring, 2004; Wagner-Menghin et al., 2016) saw codes drawn from the article “Enhancing collaborative communication of nurse and physician leadership into two intensive care units” (D. K. Boyle & Kochinda, 2004). Drawing upon Mayring (2004)’s account, each code was defined in the code book that contained “explicit examples, definitions and rules” drawn from the data. The code book served to guide the subsequent coding process.

Stage 2 saw the four reviewers using the ‘code book’ to independently extract and code the relevant data from the included articles. Any relevant data not captured by these codes were assigned a new code that was also described in the code book. In keeping with deductive category application (Wagner-Menghin et al., 2016), coding categories and their definitions were revised. The final codes were compared and discussed with the final author to enhance the reliability of the process (Wagner-Menghin et al., 2016). The final author checked the primary data sources to ensure that the codes made sense and were consistently employed. The reviewers and the final author used “negotiated consensual validation” to resolve any differences in the coding (Sambunjak et al., 2010). The final categories were selected (Neal, Neal, Lawlor, Mills, & McAlindon, 2018) based on whether they appeared in more than 70% of the articles reviewed (Curtis et al., 2001b; Humble, 2009).

The narrative produced was guided by the Best Evidence Medical Education (BEME) Collaboration guide (Haig & Dozier, 2003) and the STORIES (Structured approach to the Reporting In healthcare education of Evidence Synthesis) statement (Gordon & Gibbs, 2014).

III. RESULTS

9532 abstracts were identified from ten databases, 239 articles reviewed, and 63 articles were included as shown in Figure 2 (Moher, Liberati, Tetzlaff, & Altman, 2009).

Figure 2. PRISMA Flowchart

3) Comparisons between summaries of the included articles, thematic analysis and directed content analysis: In keeping with SEBA approach the findings of each arm of the split approach was discussed amongst the research and expert teams. The themes identified using Braun and Clarke (2006)’s approach to thematic analysis were how to teach and evaluate communication training in ICU and the factors affecting training.

The categories identified using Hsieh and Shannon (2005)’s approach to directed content analysis were 1) strategies employed to teach communication, 2) factors affecting communication training, 3) strategies employed to evaluate communication, and 4) outcomes of communication training. These categories reflected the major issues identified in the tabulated summaries.

These findings were reviewed with the expert team who agreed that given that the themes identified could be encapsulated by the categories identified, the categories and the themes will be presented together.

a) Strategies employed to teach communication in ICU: 61 articles described various interventions used to teach communication in the ICU. 19 involved ICU physicians, 18 involved ICU nurses, 4 saw participation of ICU physicians and nurses, 13 included the multidisciplinary team in the ICU, 1 was aimed at medical interns, 2 at medical students, 2 at nursing students, and 2 at both medical and nursing students. Given the overlap between teaching strategies, topics taught, and assessment methods employed in ICU communication training for nurses, doctors, nursing and medical students and HCPs in the literature, we discuss and generalise the results across HCPs.    

In curriculum design, seven studies (D. K. Boyle & Kochinda, 2004; Hope et al., 2015; Krimshtein et al., 2011; Lorin, Rho, Wisnivesky, & Nierman, 2006; McCallister, Gustin, Wells-Di Gregorio, Way, & Mastronarde, 2015; Miller et al., 2018; Sullivan, Rock, Gadmer, Norwich, & Schwartzstein, 2016) designed a curriculum based on extensive reviews of literature on teaching communication. Brunette and Thibodeau-Jarry (2017) used Kern’s 6-step approach to curriculum development to design a structured curriculum targeted at meeting the needs identified whilst Sullivan et al. (2016) and Lorin et al. (2006) used the authors’ own experiences in tandem with existing literature to guide curriculum design. W. G. Anderson et al. (2017) designed a communication training workshop based on behaviour theories whilst McCallister et al. (2015) based their curriculum on principles of shared decision-making and patient-centred communication. Northam, Hercelinskyj, Grealish, and Mak (2015) conducted a pilot study before implementing their intervention.

Topics included in the curriculum were categorised into “core topics”, or topics essential to the curriculum, and “advanced” which may be useful to incorporate into the curriculum. Core topics were deemed as topics that were most frequently cited in the literature or are crucial across a variety of interactions in the ICU setting such as history taking, relationship skills as well as on common scenarios in the ICU such as breaking bad news and communicating difficult decisions. “Advanced’ topics, though important, are not mentioned as frequently and appeared to be more site specific and sociocultural and ethical issues. These topics are outlined in Table 2 (full table with references found in Supplementary Material B). The methods employed are outlined in Table 3 (full table with references found in Supplementary Material C).

Submitted: 14 March 2020
Accepted: 20 July 2020
Published online: 5 January, TAPS 2021, 6(1), 30-39
https://doi.org/10.29060/TAPS.2021-6-1/OA2235

Yit Shiang Lui, Abigail HY Loh, Tji Tjian Chee, Jia Ying Teng, John Chee Meng Wong & Celine Hsia Jia Wong

Department of Psychological Medicine, National University Health System, Singapore

Abstract

Introduction: A good understanding of basic child-and-adolescent psychiatry (CAP) is important for general medical practice. The undergraduate psychiatry teaching programme included various adult and CAP topics within a six-week time frame. A team of psychiatry tutors developed two new teaching formats for CAP and obtained feedback from the students about these teaching activities.

Methods: Medical students were introduced to CAP via small group teaching in two different modes. One mode was the “Clinical Vignettes Tutorial” (CVT) and the other mode “Observed Clinical Interview Tutorial” (OCIT). In CVT, tutors would discuss clinical vignettes of real patients with the students, followed by explanations about theoretical concepts and management strategies. OCIT involved simulated-patients (SPs) who assisted by acting as patients presenting with problems related to CAP, or as parents for such patients. At each session, students were given the opportunity to interview “patients” and “parents”. Feedback was given following these interviews. The students then completed surveys about the teaching methods.

Results: Students rated very-positive feedback for the teaching of CAP in small groups. Almost all found these small groups enjoyable and that it helped them apply what they had learnt. Majority agreed that the OCIT sessions increased their level of confidence in speaking with adolescents and parents. Some students agreed that these sessions had stimulated their interest to know more about CAP.

Conclusion: Small group teaching in an interactive manner enhanced teaching effectiveness. Participants reported a greater degree of interest towards CAP, and enhanced confidence in treating youths with mental health issues as well as engaging their parents.

Keywords:           Child Adolescent Psychiatry, Medical Education, Small Group, Teaching

Practice Highlights

• Psychiatric disorders are among the most common medical conditions experienced by children and adolescents, and data from the Singapore Mental Health survey conducted in 2010 had shown the prevalence rates of emotional and behavioural problems among Singaporean youth to be at 12.5%.
• Most medical students had limited exposure to Child & Adolescent Psychiatry (CAP) in their medical curriculum due to reduced proportions of teaching time and opportune clinical exposures allocated to CAP programmes.
• This would be further compounded by the limited number of child and adolescent psychiatrists involved in teaching at medical schools and supervising clinical postings.
• This manuscript described synergistic teaching methods employed in educating medical students within the field of Child & Adolescent Psychiatry and examined the effectiveness and acceptability of CAP teaching using small-group teaching classes.
• The CAP small group interactive teaching sessions for medical students received good feedback from majority of the participants and translated to applicability and skillsets transferability.

Figure. 1. Open comment feedback to the survey question “The best things about the sessions were…”

In areas that the students indicated for further improvement, they had cited for a “shorter” duration in each teaching session (Figure 2). This was likely due to the nature of a full day programme of CAP teaching which could last eight hours in a day with a one-hour lunch break. Others had shared that they preferred “smaller” groups so students could get more chances to practice interviewing the SPs and also be provided “more time for discussion” to allow more in-depth feedback as well as discussion of each clinical condition. Some students remarked that Objective Structured Clinical Examination (OSCE) styled marking schemes could help enhance their learning experiences as this method might be more structured, compared to an open discussion.

Figure 2. Open comment feedback to the survey question “Some ways which I think can make the sessions better are…”

IV. DISCUSSION

This study evaluated the effectiveness and acceptability of small group tutorials for CAP conditions, which are packaged inseparably as part of a medical undergraduate psychiatry teaching programme. CVT and OCIT are synergistically designed to complement each other in the curriculum. The surveys used to compile the medical undergraduates’ responses had focused on their learning experience with the CAP curriculum. The effectiveness of the teaching methods namely CVT and OCIT would be determined from transferability of the requisite knowledge base and the clinical skills, as well as availability of opportunities to experience interviewing for the participants. The survey responses were also used to gauge the performance of the SPs and the clinical tutors’ usefulness. In addition, the degree of how impactful the teaching sessions had in generating interest towards CAP was also evaluated.

The fourth-year medical students gave good feedback for the small group teaching sessions. They reported that the CVT were enjoyable, beneficial and had allowed them to apply what they had learnt. For the OCIT, most of the respondents indicated that the session had helped them to learn psychiatric interviewing skills, increased their level of confidence in speaking with adolescents and parents, and had helped them to apply in clinical scenarios what they had learnt. There is discernible difference between the feedback for CVT and OCIT. The students’ feedback for CVT affirmed applicability of the knowledge content of CAP whereas those for OCIT concurred with transferability of interviewing skills in terms of confidence level.  

In the open feedback segment of the survey, respondents reported that they had particularly liked the interactive and hands-on aspect of the session, the frequent opportunities for evaluation and feedback, as well as for practice. However, they highlighted that certain factors such as the size of grouping, the length of the sessions and random allocation of conditions could be improved further to enhance their learning experience. Overall, their feedback still indicated positive experiences in these small group sessions, and this translated to an increased knowledge base, a heightened level of confidence, and burgeoned interest in CAP among the student participants.

This study’s limitations included the challenges inherent with attempting to accurately assess the students’ genuine experiences and feelings towards the sessions; with possible biases (recall and Hawthorne effect) in responding to questionnaires; and the lack of correlation to actual performances in real-world settings. Furthermore, what remained unanswered was how such sessions might truly generate interest leading to possibly pursuit of a career in CAP. In addition, it is uncertain whether changing the teaching methods with the curriculum could inspire more medical students and young doctors to consider specialising in this field and raise the number of residency applications. The data from our study did appear to be consistent with findings from other CAP clinical teaching programmes. In these programmes, more exposure to CAP and increased clinical opportunities did correlate with changes in impressions towards and appreciation of clinical interactions with children, increased positive views of CAP as part of medical practice, and heightened interest in CAP as a field of medical specialty (Dingle, 2010; Kaplan & Lake, 2008; Malloy et al., 2008; Martin, Bennett, & Pitale, 2005).

In the current undergraduate medical curriculum, the amount of time allocated to teaching CAP is relatively small compared to other topics. Child and adolescent psychiatric cases can be particularly complex and their management demand sensitive handling, which may pose challenges to real world practice. Youth patients and their parents may value privacy and sometimes do not allow medical students to be involved in initial assessments and subsequent follow-up consultations. These factors collectively pose unique challenges to teaching and equipping medical students with the skills and knowledge to address child and adolescent mental health disorders. While clinical contact and patient experience would be preferred and desirable for training, it may be impractical given the various constraints mentioned above (Kaplan & Lake, 2008). Hence, other creative methods of “exposure” to CAP patients should be incorporated into teaching rotations to offer medical students the opportunities to expand this knowledge base, apply the knowledge to practice scenarios, and further their clinical and communication skills. Small group sessions such as the CVTs and OCITs are teaching activities that can be used to overcome some of these challenges.

Our study showed that small group interactive teaching is effective in helping medical students to apply what they have learnt about CAP, increase their confidence in speaking to adolescents as patients and learn psychiatric interviewing skills. It also exposes them to a wide range of relevant CAP cases to which they can apply their theoretical knowledge and practice interview and management techniques. Furthermore, we have found that all this can be adequately achieved in a tailored environment that is conducive for learning. The collective constructive feedback had been used to further improve the content and deliverability style so as to enhance implementation in future batches. It has also been conceptualised to compare CVT and OCIT as individual teaching methods for future scholarly research.

V. CONCLUSION

The CAP small group interactive teaching sessions for medical students received good feedback from majority of the participants. This positive validation would spur the authors on to explore further how this pedagogy could help spark interests in Child and Adolescent Psychiatry among medical students given the shortfall of child and adolescent psychiatrists worldwide.

Notes on Contributors

AHYL analysed and interpreted data. CHJW, together with TJY and JCMW planned and conducted the child psychiatry small group teaching and collected feedback data from the medical students. TJY developed the feedback questionnaire. YSL, together with AHYL, CHJW and TTC planned and wrote the manuscript. All authors read and approved the final manuscript.

Ethical Approval

NHG DSRB reference number 2019/00431 for exemption.

Data Availability

Datasets generated and/or analysed during the current study are available from corresponding author on reasonable request.

Acknowledgements

The authors wish to thank the team from Centre for Healthcare Simulation, Yong Loo Lin School of Medicine, National University of Singapore for the invaluable support in recruiting and training the simulated patients for the CAP teaching program. We appreciate the participation of the simulated patients and medical students in the teaching programme.

Funding

There is no funding for this paper.

Declaration of Interest

As far as all the authors are concerned, we do not know of, or foresee any future competing interests. We are not aware of any issues relating to journal policies in submitting this manuscript. All the authors have approved of the manuscript for submission. The authors declare that they have no competing interests.

References

Baranne, M. L., & Falissard, B. (2018). Global burden of mental disorders among children aged 5–14 years. Child and Adolescent Psychiatry and Mental Health, 12(1), 19.

Breton, J. J., Plante, M. A., & St-Georges, M. (2005). Challenges facing child psychiatry in Quebec at the dawn of the 21st Century. The Canadian Journal of Psychiatry, 50(4), 203-212.

Costello, E. J., Egger, H., & Angold, A. (2005). 10-year research update review: The epidemiology of child and adolescent psychiatric disorders: I. Methods and public health burden. Journal of the American Academy of Child & Adolescent Psychiatry, 44(10), 972-986.

Dingle, A. D. (2010). Child psychiatry: What are we teaching medical students? Academic Psychiatry, 34(3), 175-182.

Erskine, H. E., Moffitt, T. E., Copeland, W. E., Costello, E. J., Ferrari, A. J., Patton, G., … & Scott, J. G. (2015). A heavy burden on young minds: The global burden of mental and substance use disorders in children and youth. Psychological Medicine, 45(7), 1551-1563.

Hunt, J., Barrett, R., Grapentine, W. L., Liguori, G., & Trivedi, H. K. (2008). Exposure to child and adolescent psychiatry for medical students: Are there optimal “teaching perspectives”?. Academic Psychiatry, 32(5), 357-361.

Kaplan, J. S., & Lake, M. (2008). Exposing medical students to child and adolescent psychiatry: A case-based seminar. Academic Psychiatry, 32(5), 362-365.

Kyu, H. H., Pinho, C., Wagner, J. A., Brown, J. C., Bertozzi-Villa, A., Charlson, F. J., … & Fitzmaurice, C. (2016). Global and national burden of diseases and injuries among children and adolescents between 1990 and 2013: Findings from the global burden of disease 2013 study. JAMA Pediatrics, 170(3), 267-287.

Lim, C. G., Ong, S. H., Chin, C. H., & Fung, D. S. S. (2015). Child and adolescent psychiatry services in Singapore. Child and Adolescent Psychiatry and Mental Health, 9(1), 7.

Malloy, E., Hollar, D., & Lindsey, B. A. (2008). Increasing interest in child and adolescent psychiatry in the third-year clerkship: Results from a post-clerkship survey. Academic Psychiatry, 32(5), 350-356.

Martin, V. L., Bennett, D. S., & Pitale, M. (2005). Medical students’ perceptions of child psychiatry: Pre-and post-psychiatry clerkship. Academic Psychiatry, 29(4), 362-367.

Plan, S. (2002). A Call to Action: Children Need Our Help! American Academy of Child & Adolescent Psychiatry. Retrieved from https://www.aacap.org/app_themes/aacap/docs/resources_for_primary_care/workforce_issues/AACAP_Call_to_Action.pdf

Sawyer, M., & Giesen, F. (2007). Undergraduate teaching of child and adolescent psychiatry in Australia: Survey of current practice. Australian & New Zealand Journal of Psychiatry, 41(8), 675-681.

Thomas, C. R., & Holzer, C. E., 3rd (2006). The continuing shortage of child and adolescent psychiatrists. Journal of the American Academy of Child & Adolescent Psychiatry, 45(9), 1023-1031.

World Health Organization. (2014). Adolescent health epidemiology. Retrieved from http://www.who.int/maternal_child_adolescent/epidemiology/adolescence/en/

*Yit Shiang Lui
1E Kent Ridge Road
Tower Block, Level 9,
Singapore 119228
Tel: 6772 6331
Email address: yit_shiang_lui@nuhs.edu.sg

Submitted: 14 February 2020
Accepted: 1 July 2020
Published online: 5 January, TAPS 2021, 6(1), 40-48
https://doi.org/10.29060/TAPS.2021-6-1/OA2227

Shirley Beng Suat Ooi^1,2, Clement Woon Teck Tan^3,4 & Janneke M. Frambach⁵

¹Emergency Medicine Department, National University Hospital, National University Health System, Singapore; ²Department of Surgery, Yong Loo Lin School of Medicine, National University of Singapore, Singapore; ³Department of Ophthalmology, National University Hospital, National University Health System, Singapore; ⁴Yong Loo Lin School of Medicine, National University of Singapore, Singapore; ⁵School of Health Professions Education, Faculty of Health, Medicine and Life Sciences, Maastricht University, The Netherlands

Abstract

Introduction: Almost all published literature on effective clinical teachers were from western countries and only two compared medical students with residents. Hence, this study aims to explore the perceived characteristics of effective clinical teachers among medical students compared to residents graduating from an Asian medical school, and specifically whether there are differences between cognitive and non-cognitive domain skills, to inform faculty development.

Methods: This qualitative study was conducted at the National University Health System (NUHS), Singapore involving six final year medical students at the National University of Singapore, and six residents from the NUHS Residency programme. Analysis of the semi-structured one-on-one interviews was done using a 3-step approach based on principles of Grounded Theory.

Results: There are differences in the perceptions of effective clinical teachers between medical students and residents. Medical students valued a more didactic spoon-feeding type of teacher in their earlier clinical years. However final year medical students and residents valued feedback and role-modelling at clinical practice. The top two characteristics of approachability and passion for teaching are in the non-cognitive domains. These seem foundational and lead to the acquisition of effective teaching skills such as the ability to simplify complex concepts and creating a conducive learning environment. Being exam-oriented is a new characteristic not identified before in “Western-dominated” publications.

Conclusion: The results of this study will help to inform educators of the differences in a learner’s needs at different stages of their clinical development and to potentially adapt their teaching styles.

Keywords:           Clinical Teachers, Medical Students, Residents, Cognitive/Non-Cognitive, Asian Healthcare, Faculty Development

Practice Highlights

• Approachability and teaching passion are foundational non-cognitive skills in effective clinical teachers.
• These foundational skills are more important for undergraduate than postgraduate teaching.
• Procedural residents can accept less ‘warm’ teachers if they can learn advanced clinical skills.
• Medical students value didactic ‘spoon-feeding’ type of teachers in their earlier clinical years.
• Final year medical students and residents value feedback and role-modelling at clinical practice.

I. INTRODUCTION

“The transformation of our students requires the engagement of innovative and outstanding clinician-teachers who not only supervise students in their development of technical skills and applied knowledge but also serve as role models of the values and attributes of the profession and of the life of a professional” (Sutkin, Wagner, Harris, & Schiffer, 2008). This statement nicely encapsulates the very important role played by outstanding clinical teachers in helping students to ultimately become professionals with the attributes our healthcare system desires. Previous research has extensively investigated characteristics of effective clinical teachers to inform faculty development (e.g. Branch, Osterberg, & Weil, 2015; Hatem et al., 2011; Hillard, 1990; Kernan, Lee, Stone, Freudigman, & O’Connor, 2000; Paukert & Richards, 2000; Singh et al., 2013; Sutkin et al., 2008; White & Anderson, 1995). However, despite the large body of existing research on effective clinical teaching, two issues related to the needs of different groups of learners need further investigation to enable more tailored faculty development.

First, effective clinical teaching may look different in undergraduate as compared with postgraduate education. In many healthcare institutions, clinical teachers are expected to teach across the medical education continuum, i.e., undergraduate medical students, graduate doctors in training, as well as part of continuing medical education, and teaching abilities are a necessary prerequisite in an academic environment (Hatem et al., 2011). Based on the conceptual framework of constructivism (Bednar, Cunningham, Duffy, & Perry, 1991)—­a theory which equates learning with creating meaning from experience or contextual learning­—Jonassen (1991) argues that constructive learning environments are most effective for acquiring knowledge in the advanced stage of knowledge, the stage between introductory and expert. According to Jonassen (1991), the initial or introductory stage of knowledge acquisition occurs when learners have very little directly transferable prior knowledge about a skill or content area. In this stage, knowledge is best acquired through more objectivistic approaches which can be described as ‘spoon-feeding’. Medical students in general would fit into this introductory stage, in varying degrees depending on their seniority and individual progress in learning. Jonassen’s (1991) second stage is advanced knowledge acquisition where the domains are ill-structured and more knowledge-based. This is in contrast to his third or final stage of knowledge acquisition of experts that require very little instructional support but are able to deal with elaborate structures, schematic patterns and seeing the interconnectedness in knowledge through experience. The stage of junior doctors in training would be fit into the second or advanced stage of learning. Constructivist teachers help students construct knowledge to become active learners rather than passive recipients of knowledge from the teachers or textbooks. In view of this constructivist framework, it appears logical to postulate that as medical students mature to become practicing doctors, their perceptions of effective clinical teachers may change from one who ‘spoon-feeds’ them with medical knowledge to one who encourages them to actively construct new meaning as they become clinically more experienced and have to deal with complex and ill-defined problems. Low, Khoo, Kuan, and Ooi (2020) showed that although the top four characteristics of effective medical teachers are consistent across all 5 years of medical school, characteristics that facilitate active learner participation are emphasised in the clinical years consistent with constructivist learning theory. However, as there is a paucity of comparative research on perceptions of effective clinical teachers among undergraduates as compared to postgraduates to plan more focused faculty development to address the attributes the learners look for in their clinical teachers, this warrants further research.

The second issue relates to potential differences in the clinical teaching role between Asian and Western settings. In Western studies, as noted above, effective clinical teachers are encouraged to stimulate students’ intellectual curiosity leading to more self-directed learning (Hillard, 1990; Kernan et al., 2000; White & Anderson, 1995). In contrast, feelings of uncertainty about the independence required in self-directed learning, a focus on tradition that respects ‘old ways’, hierarchy expecting ‘truths’ to come from persons of higher status, and an achievement orientation to pass and excel in examinations have been identified as more prominent in non-Western than in Western cultures (Frambach, Driessen, Chan, & van der Vleuten, 2012). This is despite the recent introductions of more student-centred education methods. In Singapore for example, there is a move in the Yong Loo Lin School of Medicine (YLLSoM) to try to embed students into healthcare teams (Jacobs & Samarasekera, 2012) and implement newer methods of learning such as flipped classroom. However, many teachers still employ traditional methods of lectures and small group tutorials focused on exam preparation. A comprehensive review study of 68 articles on effective clinical teaching (Sutkin et al., 2008), comprised only one article that reported research from a non-Western setting (Elzubeir & Rizk, 2001). In this article, originating from the United Arab Emirates, there is no discussion on whether there is a difference in the perception of a role model between medical students in Asian countries compared to the West (Elzubeir & Rizk, 2001). Another study conducted in Asia showed differences in the perceptions of first-year and fifth-year medical students in Singapore on what makes an effective medical teacher (Kua, Voon, Tan, & Goh, 2006). More first-year students preferred handouts in contrast to fifth-year students who were less reliant on ‘spoon-feeding’. Research on effective clinical teaching is growing in the Asian setting (Ciraj et al., 2013;  Haider, Snead, & Bari, 2016; Kikukawa et al., 2013; Mohan & Chia, 2017; Nishiya et al., 2019; Venkataramani et al., 2016) though there is still a paucity of literature in the Asian setting compared with studies conducted in the West and there are none that directly compared medical students with residents.

Another issue that deserves further attention is the role of non-cognitive domain skills in clinical teaching. Sutkin et al.’s (2008) review study described three main categories of characteristics of good clinical teachers: 1) physician characteristics, 2) teacher characteristics, and 3) human characteristics (Table 1). Approximately two-thirds of the characteristics were in non-cognitive domains (such as those involving relationship skills, emotional states, and personality types), and one-third in cognitive domains (such as those involving reasoning, memory, judgment, perception, and procedural skills). The article noted that cognitive abilities can be taught and learned, in contrast to non-cognitive attributes which are more difficult to develop and teach. Faculty development programmes currently often focus on traditional cognitive skills, such as curriculum design, large-group teaching, and assessment of learners (Searle, Hatem, Perkowski, & Wilkerson, 2006). In contrast, if non-cognitive domains are more important in contributing to outstanding teaching, they might need greater emphasis in the curricula of these workshops. The good news is that according to Schiffer, Rao, and Fogel (2003), non-cognitive behaviours are both measurable and alterable. Most of them have underlying neural networks which are entering our sphere of understanding. Hence non-cognitive skills, although much more challenging to develop than cognitive skills, have a potential to be developed. It is not clear whether there are differences in the distribution between cognitive and non-cognitive domains skills between the perceptions of medical students compared to residents of an effective clinical teacher.

The aim of this qualitative study is to explore the perceived characteristics of an effective clinical teacher among medical students compared to residents graduating from an Asian medical school and whether there is a difference regarding cognitive and non-cognitive domain skills.

II. METHODS

A. Participants

The participants consisted of final/fifth year medical students (M5s) from the Yong Loo Lin School of Medicine (YLLSoM), National University of Singapore (NUS) who were posted to the National University Hospital (NUH) to do their student internship posting in 2016. To ensure sufficient working experience, the National University Health System (NUHS) residents who had graduated from the YLLSoM and who had recently completed their intermediate specialty examinations were recruited. These were third to fifth year residents in different programmes. Maximal variation sampling of the M5s and the residents of both gender, different ethnic groups and from different specialties (for residents only) was done.

B. Design

A pragmatic qualitative research design (Savin-Baden & Howell Major, 2013) was used to get the participants to reflect on their own learning journey affecting their perceptions of the qualities that make an effective clinical teacher from the time they were first exposed to clinical medicine in year 3 (M3) of medical school to final year (M5) for the students, and to residency for the residents.

 C. Data Collection

Semi-structured one-on-one interviews using open-ended questions were conducted. A list of M5s doing their student internship programme in the various departments in NUH was invited via an e-mail invitation to participate in this study. To ensure maximal variation sampling, M5s of both gender and as far as possible different ethnic groups were recruited. As for the residents, through the Graduate Medicine Education Office in NUH, residents of both gender, from different ethnic groups and different specialties (both procedural and non-procedural) were selected from those who responded voluntarily to the invitation to participate in this study to ensure maximal variation sampling as residents from procedural specialties may have different perceptions of effective clinical teachers from non-procedural specialties.

Written consent after reading the Participant Information Sheet was taken from the interviewees before the interview was conducted in a quiet room. The interview was audiotaped and lasted between 30 and 45 minutes.

D. Data Analysis

The audiotaped interviews were transcribed. As all the 12 interviews were conducted by the principal investigator (PI) (SO) and although the coding and official analysis of the interviews were done after all the 12 interviews were transcribed, the PI had taken note of themes emerging and decided on ending the interviews after no substantial new themes had emerged.

In the first phase, open coding, initial categories of the information on characteristics of effective clinical teachers by segmenting information and assigning open codes were formed. In the second coding phase, broader categories were developed through conceptually related ideas. The third phase involved selective coding where the individual categories were counterchecked with Sutkin et al.’s (2008) categories of teacher, physician and human characteristics and whether they were in the cognitive or non-cognitive domains (Table 1). Further related categories according to Sutkin et al.’s (2008) classification were brought together.

Physician Characteristics

Submitted: 15 April 2020
Accepted: 5 June 2020
Published online: 5 January, TAPS 2021, 6(1), 49-59
https://doi.org/10.29060/TAPS.2021-6-1/OA2248

Amaya Tharindi Ellawala¹, Madawa Chandratilake² & Nilanthi de Silva²

¹Department of Medical Education, Faculty of Medical Sciences, University of Sri Jayewardenepura, Sri Lanka; ²Faculty of Medicine, University of Kelaniya, Sri Lanka

Abstract

Introduction: Professionalism is a context-specific entity, and should be defined in relation to a country’s socio-cultural backdrop. This study aimed to develop a framework of medical professionalism relevant to the Sri Lankan context.

Methods: An online Delphi study was conducted with local stakeholders of healthcare, to achieve consensus on the essential attributes of professionalism for a doctor in Sri Lanka. These were built into a framework of professionalism using qualitative and quantitative methods.

Results: Forty-six attributes of professionalism were identified as essential, based on Content Validity Index supplemented by Kappa ratings. ‘Possessing adequate knowledge and skills’, ‘displaying a sense of responsibility’ and ‘being compassionate and caring’ emerged as the highest rated items. The proposed framework has three domains: professionalism as an individual, professionalism in interactions with patients and co-workers and professionalism in fulfilling expectations of the profession and society, and displays certain characteristics unique to the local context.

Conclusion: This study enabled the development of a culturally relevant, conceptual framework of professionalism as grounded in the views of multiple stakeholders of healthcare in Sri Lanka, and prioritisation of the most essential attributes.

Keywords:           Professionalism, Culture, Consensus

Practice Highlights

• Medical professionalism is recognised as a culturally dependent entity.
• This has led to the emergence of definitions unique to socio-cultural settings.
• List-based definitions provide operationalisable means of portraying its meaning.
• A Delphi study was conducted to achieve consensus on locally relevant professionalism attributes.
• Using quantitative and qualitative methods, a conceptual framework of professionalism was developed.

I. INTRODUCTION

There is no single definition of medical professionalism that encompasses its many subtle nuances (Birden et al., 2014).  The realisation that professionalism is a dynamic, multi-dimensional entity (Van de Camp, Vernooij-Dassen, Grol, & Bottema, 2004), significantly dependent on context (Van Mook et al., 2009), and cultural backdrop (Chandratilake, Mcaleer, & Gibson, 2012), has led to the emergence of definitions specific to cultures and socio-economic backgrounds.

Many of the current definitions originate from Western societies. Certain Eastern cultures have embraced such definitions, though they are undeniably in conflict with local traditional views (Pan, Norris, Liang, Li, & Ho, 2013). In parallel however, countries such as Egypt, Saudi Arabia, Japan, China and Taiwan have explored how professionalism is conceptualised within their contexts (Al-Eraky, Chandratilake, Wajid, Donkers, & Van Merrienboer, 2014; Leung, Hsu, & Hui, 2012; Pan et al., 2013). Such studies have portrayed the interplay between cultural, socio-economic and religious factors in shaping perceptions on professionalism, further fuelling the notion that professionalism must be “interpreted in view of local traditions and ethos” (Al-Eraky et al., 2014, p. 14).

Culture is the embodiment of elements such as attitudes, beliefs and values that are shared among individuals of a community and is therefore, an entity that distinguishes one group of people from another (Hofstede, 2011). Various cultural theories provide insight into inter-cultural differences across the globe (Hofstede, n.d.; Schwartz, 1999). The Sri Lankan cultural context, while aligned with those of its closest geographical neighbours in South Asia in some ways, differs from them in other important aspects. 

Certain attempts have been made to explore the meaning of professionalism in Sri Lanka. Chandratilake et al. (2012) provided a degree of insight while comparing cultural similarities and dissonances in conceptualising professionalism among doctors of several nations. Monrouxe, Chandratilake, Gosselin, Rees, and Ho (2017) built on this work with their analysis of professionalism as viewed by local medical students. The sole regulatory authority of the medical profession in the country, the Sri Lanka Medical Council (SLMC, 2009) has delineated what it expects in terms of professionalism, by outlining the constituents of ‘good medical practice’, many of which converge with elements of professionalism described in the literature.

While the work mentioned here has shed some light on the topic, to our knowledge, there were no studies that focused solely on the local conceptualisation of professionalism, drawing on the views of diverse stakeholders of healthcare.

There exist two schools of thought on how professionalism can be defined: as a list of desirable attributes (Lesser et al., 2010), or as an over-arching, value-laden entity that transcends such lists (Irby & Hamstra, 2016; Wynia, Papadakis, Sullivan, & Hafferty, 2014). Unlike the latter, a list may not address the “foundational purpose of professionalism” (Wynia et al., 2014, p. 712); however, it will provide a tangible, operationalisable portrayal (Lesser et al., 2010). It is possibly for this reason that many studies have opted for list-based definitions, an approach that is supported in the East (Al-Eraky & Chandratilake, 2012; Al-Eraky et al., 2014; Pan et al., 2013).

The aim of this study was to develop a culturally appropriate conceptual framework of medical professionalism in Sri Lanka using a combination of qualitative and quantitative methods. We envisioned that identifying a list of desirable attributes would be appropriate, providing a definition that could readily be operationalised for teaching/learning, assessment and research purposes (Wilkinson, Wade, & Knock, 2009).

II. METHODS

A. The Approach

We followed a consensus approach, and opted for the Delphi technique as it was imperative to involve a large number of participants not limited by geographical location (Humphrey-Murto et al., 2017). The method offered the further advantage of providing participants with equal opportunity to express their opinions (De Villiers, De Villiers, & Kent, 2005), thereby negating the possible drawbacks of face-to-face interactions and resulting in a ‘process gain’ (Powell, 2003).

B. Participant Panel

The panel comprised nation-wide stakeholders of healthcare (Table 1), from both rural and urban regions who were presumably exposed to diverse forms of medical services and geographical variations in their distribution.

Submitted: 17 March 2020
Accepted: 3 June 2020
Published online: 5 January, TAPS 2021, 6(1), 60-69
https://doi.org/10.29060/TAPS.2021-6-1/OA2239

Frank Bate¹, Sue Fyfe², Dylan Griffiths¹, Kylie Russell¹, Chris Skinner¹, Elina Tor¹

¹University of Notre Dame Australia, Australia; ²Curtin University, Australia

Abstract

Introduction: In 2017, the School of Medicine of the University of Notre Dame Australia implemented a data-informed mentoring program as part of a more substantial shift towards programmatic assessment. Data-informed mentoring, in an educational context, can be challenging with boundaries between mentor, coach and assessor roles sometimes blurred. Mentors may be required to concurrently develop trust relationships, guide learning and development, and assess student performance. The place of data-informed mentoring within an overall assessment design can also be ambiguous. This paper is a preliminary evaluation study of the implementation of data informed mentoring at a medical school, focusing specifically on how students and staff reacted and responded to the initiative.

Methods: Action research framed and guided the conduct of the research. Mixed methods, involving qualitative and quantitative tools, were used with data collected from students through questionnaires and mentors through focus groups.

Results: Both students and mentors appreciated data-informed mentoring and indications are that it is an effective augmentation to the School’s educational program, serving as a useful step towards the implementation of programmatic assessment.

Conclusion: Although data-informed mentoring is valued by students and mentors, more work is required to: better integrate it with assessment policies and practices; stimulate students’ intrinsic motivation; improve task design and feedback processes; develop consistent learner-centred approaches to mentoring; and support data-informed mentoring with appropriate information and communications technologies. The initiative is described using an ecological model that may be useful to organisations considering data-informed mentoring.

Keywords:            Data-Informed Mentoring, Mentoring, Programmatic Assessment, E-Portfolio

Practice Highlights

• Students and mentors appreciated the introduction of data-informed mentoring.
• Assessment policies and practices should be integrated with data-informed mentoring.
• Data-informed mentoring presents curriculum challenges in task design and framing feedback.
• The student context informs the data-informed mentoring approach (learner-centred to mentor-directed).
• Data-informed mentoring requires supportive information and communications technologies.

I. INTRODUCTION

An often-cited definition of mentoring, highlights the role of experienced and empathetic others guiding students to re-examine their ideas, learning and personal and professional development (Standing Committee on Postgraduate Medical and Dental Education, 1998).

Heeneman and de Grave (2017) identify some subtle differences between traditional conceptions of mentoring and the type of mentoring that is required under programmatic assessment, which in this paper we refer to as Data-Informed Mentoring (D-IM). For example, D-IM is embedded in a curriculum in which rich data on student progress arises from student interaction with assessment tasks, informing and enhancing their progress (see Appendix). Further, in programmatic assessment, the learning portfolio is typically the setting in which the mentor-mentee relationship develops. This setting brings together institutional imperatives (e.g. assessable tasks), and personal imperatives such as evidence of competence and personal reflection. Situating mentoring in a curriculum and assessment framework impacts upon the mentoring relationship.

Meeuwissen, Stalmeijer, and Govaerts (2019) propose that a different type of mentoring is required under programmatic assessment. Mentors interpret data and feedback provided by content experts across domains of learning thus providing an evidence-base to facilitate student reflection. They might also take on a variety of roles (e.g. critical friend, coach, assessor) that could influence the mentoring relationship including the level of trust that is established with the student. These challenges suggest that conventional definitions of mentoring might not capture the essence of D-IM. Whilst the availability of rich information potentially enhances the mentoring experience and personalises learning, mentors and students are challenged to make sense and act upon this information; students might focus on issues that fall outside of the scope of the data provided (e.g. their wellbeing); mentors may also struggle to delineate boundaries between multiple roles or draw a line on where their scope of practice, as a mentor, begins and ends.

Mentoring is a social construct and as such is best considered through a holistic lens taking account of societal, institutional and personal factors (Sambunjak, 2015). The current study adopted Sambunjak’s “ecological model” (2015, p. 48) as a framework to help understand the impact of D-IM (Figure 1). Societal, institutional and personal forces are inter-related. For example, a student’s approach to D-IM might be influenced by financial circumstances resulting in the need to work part-time (societal); a medical school’s assessment policy (institutional); or a student’s learning style (personal). The model is presented as a set of cogs where the optimal educational experience is achieved if all elements work in harmony. The study uses the ecological model to help answer the central research question that guided the study: how did students and staff react and respond to D-IM?

Figure 1. An ecological framework for conceptualizing D-IM (modified from Sambunjak, 2015)

Note. *In the median test, a comparison is made between the median rating in each group to the ‘overall pooled median’ from both groups. **Values are less than or equal to the overall pooled median therefore Median Test could not be performed. ***Significant at p < 0.05 level.

Table 1. Student Perceptions of D-IM within Cohort 1 in 2017 and 2018–Median Tests for Individual Items

The only statistically significant difference noted for Cohort 1 was for the summaries of performance provided in Blackboard that were designed to underpin D-IM. The data provided in these summaries was less valued by students who engaged with D-IM in their second year.

The aggregate mean score in response to the statements on D-IM in the survey was positive in 2017 (M=4.02; SD=0.62; n=32). Mentoring continued to be well perceived by Cohort 1 as they progressed to second year in 2018 (M=3.80; SD=0.67; n=51). The slight difference in aggregate mean scores between 2017 and 2018 is not statistically significant (t=1.571; df=82, p=0.120). Table 2 compares first year students’ perceptions of D-IM.

Item	Overall Pooled Median*	Cohort 1 n=32		Cohort 2  n=47
		n> pooled median	<= pooled median	>  pooled median	<= pooled median	Median Test (chi square (χ2); df; p value)
The mentoring process was well organised	4	6	26	9	37	χ2 =0.008; df=1; p=0.928
My mentor was personally very well organised	5	0	32	0	47	n/a**
There were an appropriate number of mentoring meetings throughout the year	4	2	30	8	39	χ2 =0.998; df=1; p=0.158
My mentor was respectful	5	0	32	0	47	n/a**
My mentor listened to me	5	0	32	0	47	n/a**
My mentor asked thought-provoking questions which helped me to reflect	4	10	22	18	29	χ2 =0.413; df=1; p=0.520
My mentor added value to my learning	4	10	22	17	30	χ2 =0.205; df=1; p=0.651
My mentor helped me to set future goals that were achievable	4	9	23	17	30	χ2 =0.558; df=1; p=0.455
The summaries provided of my performance in the Blackboard Community Site were useful in helping me to reflect on my progress	3	17	16	17	30	χ2 =1.868; df=1; p=0.172

Note. *In the median test, a comparison is made between the median rating in each group to the ‘overall pooled median’ from both groups. **Values less than or equal to the overall pooled median therefore Median Test could not be performed.

Table 2. First Year Student Perceptions of D-IM –Median Tests between Cohort 1 and Cohort 2 for Individual Items

No statistically significant differences were noted between cohorts 1 and 2.

The aggregate mean score in response to the statements on D-IM in the survey was positive for Cohort 1 in 2017 (M=4.02; SD=0.62; n=32). Equally positive responses were noted in Cohort 2 in 2018 (M=3.91; SD=0.79; n=47). The difference between aggregate mean scores for first year students’ perceptions is not statistically significant (t=0.686; df=78, p=0.495).

Data from tables 1 and 2 reveals that students are highly satisfied with three aspects of mentoring: the personal organisation of the mentor along with their respectful and listening attributes. Students were also satisfied with the mentoring process, the number of mentoring meetings, the ability of the mentoring to assist in reflection and to add value to their learning, and also the propensity of the mentor to assist in action-planning. However, the summaries provided in the Blackboard environment were a source of dissatisfaction for students.

As discussed, qualitative data were collected from students through the questionnaire and staff through focus groups. The research team collated the qualitative data and confirmed that the qualitative data corroborated quantitative results with students and mentors appreciating the introduction of D-IM. For example, “Mentor sessions are important in providing support to students and…are a welcome introduction” (Yr1 Student, 2017); “Mentoring was useful to develop self-directed learning and to check where you were” (Yr2 Student, 2018); “You get to know the students, things were revealed which would not have been otherwise” (Mentor, 2017); and “Mentoring enabled me to facilitate more, listen more. Definitely a difference when you’re one-on-one with somebody” (Mentor, 2018).

In tune with the action research method adopted by the study which seeks to identify and respond to opportunities for improvement, the Research Team identified three concerns from the qualitative data: differing views of the purpose of D-IM and the role of the mentor; the provision of student feedback and information and communications technologies (ICT); and workload.

A. Differing Views of the Purpose of D-IM and the Role of Mentor

Mentors had differing conceptions of the purpose of D-IM and the role of a mentor. Some mentors perceived their primary function to be one of facilitating reflection and being encouraging whilst others were more directive, providing advice or sharing their own experiences. “I was…a sounding board to prompt their thoughts about how their progress was going. Rather than offering ways of solving problems it was more pointing where problems might lie and encouraging them to think of solutions” (Mentor, 2017); “The basic rule is to guide them… guide them properly, maybe to get them to change their study strategies and other things” (Mentor, 2017).

 B. Provision of Student Feedback and ICT

Students reported that feedback was inconsistent in timeliness and quality. Often feedback lacked guidance for improvement or was too late for it to help the student improve their learning: “More in-depth feedback on work, and returned in a timeframe that allows it to be relevant to our learning” (Yr1 Student, 2018); “Marking seemed thoughtless and halfhearted” (Yr2 Student, 2018).The use of a Blackboard Wiki to collate and present data points was also less than ideal with students finding the site difficult to navigate and use although they generally reported that it was safe and secure.

C. Workload

Students understood the role of reflection and appreciated having a mentor although there was some misunderstanding of the role of the portfolio with some students seeing it as extra work: “The amount of work required…was disproportionate” (Yr2 Student, 2018). Some students felt that the added stress and anxiety detracted from their study of medicine: “The portfolio actually detracts from spending time learning content that is essential to clinical years” (Yr2 Student, 2018). These concerns needed to be addressed by the School and are discussed in the context of changes that have and will be made to D-IM for preclinical students in the School.

IV. DISCUSSION

On the whole there was a positive response to D-IM implementation by students and staff. This response is consistent with Frei, Stamm, and Buddeberg-Fischer (2010, p. 1) who found that the “personal student-faculty relationship is important in that it helps students to feel that they are benefiting from individual advice.”

The findings of the research, however, reveal some tensions between the various elements of Sambunjak’s (2015) ecological model that link to the three areas of concern identified in the research. These tensions are shown in Figure 2.

Figure 2. The ecological framework to explore tensions in D-IM

A. Purpose and Role of Mentors and D-IM

The role of the mentor at the School is to support and guide students, and this role was not confused with other functions such as content expert or assessor. In this respect, the role conflict described by Meeuwissen et al. (2019) and Heeneman and deGrave (2017) was not evident at the School. However, mentoring approaches were situated on a continuum between learner-centred and mentor-directed. It is probable that the mentor’s style–empowering, checking or directing (Meeuwissen et al., 2019, p.605)–and their potentially different view of their role impacted on how D-IM sessions played out. Three ways of understanding the role of mentor in medical education have been identified: someone who can answer questions and give advice, someone who shares what it means to be a doctor and someone who listens and stimulates reflection (Stenfors-Hayes, Hult, & Owe Dahlgren, 2011). In a study of mentoring styles of beginning teachers, Richter et al. (2013) found that the mentor’s beliefs about learning have the greatest impact on the quality of the mentoring experience. Although professional development was provided to mentors on their role as facilitators of reflection and these issues were outlined and discussed, there were differences in interpretation of the role in the D-IM context.

Heeneman and de Grave (2017) argue that students need to be self-directed in order to be effective medical professionals. It is posited that a number of factors can influence the extent to which the mentor directs proceedings including the mentor’s experience, role clarity, rate of student progress, depth of student reflections and the perceived importance of the data required for assessment purposes.

In this study most students engaged positively with D-IM, though, albeit with variations in the extent and quality of reflection and action planning. A slight decrease in students’ enthusiasm towards D-IM was noted as they progressed from first to second year. This could be related to the novelty of D-IM diminishing over time that has been evident in other educational technology innovations (Kuykendall, Janvier, Kempton, & Brown, 2012). However, students also have a different mentor in each year. According to Sambunjak (2015), mentoring requires commitment sustained over a long period of time. At Maastricht University, for example, Heeneman and de Grave (2017) report that students are allocated the same mentor for a four-year medical course. It is, therefore, likely that in the current study the short timeframe for mentors to establish student relationships, and the introduction of a different mentor each year contributed to a reduction in student satisfaction.

B. Feedback and ICT Support

D-IM is dependent on quality data. That is, the perceived value of tasks that students engage with, and the feedback that they receive on, these tasks. Findings suggest that students found some tasks repetitive and feedback belated and superficial. Better task design and feedback practices are required. This finding is consistent with those of Bate, Macnish and Skinner (2016) in a study of task design within a learning portfolio. Findings also indicated dissatisfaction with Blackboard ICT environment. The portal was not intuitive and the structure and requirements for use of the template did not stimulate the desired level of reflection.

 C. Workload

Students at the School are “time poor” and many work part-time whilst studying. They are graduate entrants used to achieving academic success. Most are millennials comfortable with distilling and manipulating data and using online technologies. These characteristics are consistent with what Waljee, Copra, and Saint (2018) see as the new breed of medical students, being accustomed to distilling information and desirous of rapid career advancement. In these circumstances, it is unsurprising that students valued D-IM as it promoted focused data-driven discussion on their progress. However, it is also unsurprising that students were critical of anything that, in their opinion, did not support the “study of medicine”. Although students were sometimes critical of tasks that fed into D-IM (Bate et al., 2020), the reflective and action planning components of DI-M were not onerous and were at any rate optional.

For most students, grades rather than learning were paramount and this created a competitive environment which fuelled strategic learning in engaging with tasks underpinning D-IM. The School’s Assessment Policy has implications here. Progression is determined by passing discrete assessments and causes students to focus on grades rather than learning. These dispositions play out in D-IM sessions where, for example, goals are sometimes framed around passing examinations rather than addressing deficits in understanding. The School also distinguishes between formative and summative assessment with the result being that formative assessments are less valued by students. Opportunities to test understanding through formative testing are sometimes not taken up and result in less information for students and their mentor to gauge learning progress.

Bhat, Burm, Mohan, Chahine, and Goldszmidt (2018, p. 620) identified a set of “threshold concepts” in medicine that are crucial for students transitioning into clinical practice. Among these are self-directed, metacognitive and collaborative dispositions to learning. However, for a student in the preclinical years, these threshold concepts are not perceived to be the important factors that determine their progress through the course and their aim to become a doctor. Thus the tensions between students valuing mentoring but feeling that reflecting on their performance through D-IM is time-consuming and unrelated to their course progression is a source of tension within the model.

D. Actions as a Result of the Study

The action research approach of this study meant that in all results the Research Team was looking for ways to improve the system. Some issues could be improved quickly. A refinement of the Blackboard environment and a change to a software solution called SONIA was implemented in 2019 to improve the ICT interface and reduce workload. Continuing professional development (PD) for staff is undertaken and takes the research results into account. Within the mentor PD program, the Research Team saw that mentoring requires mentors to be able to diagnose the readiness and willingness of students to consider their learning educational journey. This means that, whilst the D-IM program needs a consistent view of D-IM where mentors see their role as facilitating reflection, different mentoring skills and behaviours are needed by mentors for different students. PD is also needed for students so that they understand the relationship between their achievement of learning and the role of D-IM in their journey.

Some issues are longer-term or resource dependent. A focus on the role of feedback in the system, especially for student reflection and its timeliness for mentoring sessions and action planning is critical to making D-IM valued by students. However, it is not always possible for staff to provide feedback in an optimum timeframe although the quality of the feedback can be improved by clear guidelines, expectations and an intuitive online interface.

Of great complexity and more difficult to resolve is the tension between developing the “threshold concepts” (Bhat et al., 2018); the generic skills which are built on self-reflection and are supported by D-IM and the ways in which a student progresses through the course. These are School-based rules of progression and produce a framework within which D-IM needs to operate.

V. LIMITATIONS OF THE STUDY

The study was conducted at one University and although it will ultimately cover a six-year timeframe, findings should be gauged within the context of this setting. Relatively low response rates were noted, and selection bias is a possibility with students most engaged with D-IM completing the questionnaire. Although professional development was provided to underpin the mentoring role, there was variation in the way tutors interpreted this role. The study was conducted at a time where other changes were occurring at the School (e.g. development of more continuous forms of assessment) and these changes might have impacted on D-IM. The questionnaire used in the study contained nine questions on mentoring. To gain a more nuanced understanding of D-IM at the School, it may be useful to use a comprehensive and validated questionnaire (e.g. Heeneman & de Grave, 2019) capturing the perceptions of mentees and mentors.

VI. CONCLUSION

The School aims to create quality, patient-centred and compassionate doctors who are lifelong learners (Candy, 2006). D-IM is an effective augmentation to the School’s educational program and the paper has demonstrated that it was well received by students and staff. Future directions include consideration of D-IM in clinical mentoring, development of more consistent learner-centred approaches to mentoring; improved task design and feedback; support for D-IM with appropriate ICT; and better integration of D-IM with assessment policies and practices.

Notes on Contributors

Associate Professor Frank Bate completed his PhD at Murdoch University. He is the Director of Medical and Health Professional Education at the School of Medicine Fremantle, University of Notre Dame Australia. He conceptualised and led the research, and was the primary author responsible for developing, reviewing and improving the manuscript.

Professor Sue Fyfe attained her PhD at the University of Western Australia and is an adjunct professor at Curtin University. She assisted in conceptualising the research design, conducted the qualitative data analysis and made a significant contribution to reviewing and improving the manuscript.

Dr Dylan Griffiths has a PhD from the University of Essex and is the Quality Assurance Manager at the School of Medicine Fremantle, University of Notre Dame Australia. He conducted data collection and assisted with preliminary analysis.

Associate Professor Kylie Russell obtained her PhD from the University of Notre Dame Australia. She is currently a Project Officer at the School of Medicine Fremantle, University of Notre Dame Australia. She assisted in the development of the research methodology and made a contribution to reviewing and improving the manuscript.

Associate Professor Chris Skinner completed his PhD at the University of Western Australia. He is Domain Chair of Personal and Professional development at the School of Medicine Fremantle, University of Notre Dame Australia. He assisted in conceptualising the research design and made a contribution to reviewing and improving the manuscript.

Associate Professor Elina Tor completed her PhD at Murdoch University and is the Associate Professor of Psychometrics at the School of Medicine Fremantle, University of Notre Dame Australia. She helped conceptualise the research design, led the quantitative data analysis, and made a significant contribution to reviewing and improving the manuscript.

Ethical Approval

The University of Notre Dame Australia Human Research Ethics Committee (HREC) has provided ethical approval for the research (Approval Number 017066F).

Acknowledgement

The authors acknowledge the thoughtful and insightful feedback provided by staff and students.

Funding

No internal or external funding was sought to conduct this research.

Declaration of Interest

There is no conflict of interest to declare.

References

Bate, F., Fyfe, S., Griffiths, D., Russell, K., Skinner, C., & Tor, E. (2020). Does an incremental approach to implementing programmatic assessment work: Reflections on the change process. MedEdPublish, 9(1), 55. https://doi.org/10.15694/mep.2020.000055.1

Bate, F., Macnish, J., & Skinner, C. (2016). The cart before the horse? Exploring the potential of ePortfolios in a Western Australian medical school. International Journal of ePortfolio, 6 (2), 85-94.

Bhat, C., Burm, S., Mohan, T., Chahine, S., & Goldszmidt, M. (2018). What trainees grapple with: A study of threshold concepts on the medicine ward. Medical Education, 52(6), 620–631. https://doi.org/10.1111/medu.13526

Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3, 77-101. https://doi.org/10.1191/1478088706qp063oa

Candy, P. (2006). Promoting lifelong learning: Academic developers and the university as a learning organization. International Journal for Academic Development, 1(1), 7-18. https://doi.org/10.1080/1360144960010102

Frei, E., Stamm, M., & Buddeberg-Fischer, B. (2010). Mentoring programs for medical students: A review of PubMed literature 2000-2008, BMC Medical Education, 10(32), 1-14. https://doi.org/10.1186/1472-6920-10-32

Gillis, A., & Jackson, W. (2002). Research for Nurses: Methods and Interpretation. Philadelphia, PA: F.A. Davis Co.

Heeneman, S., & de Grave, W. (2017). Tensions in mentoring medical students toward self-directed and reflective learning in a longitudinal portfolio-based mentoring system – An activity theory analysis. Medical Teacher, 39(4), 368-376. https://doi.org/10.1080/0142159X.2017.1286308

Heeneman, S., & de Grave, W. (2019). Development and initial validation of a dual-purpose questionnaire capturing mentors’ and mentees’ perceptions and expectations of the mentoring process. BMC Medical Education, 19(133), 1-13. https://doi.org/10.1186/s12909-019-1574-2

Kemmis, S., & McTaggart, R. (2000). Participatory action research. In N. K. Denzin & Y. S. Lincoln (Eds.) Handbook of Qualitative Research (2^nd Ed.; pp 567-606). New York: Sage Publications.

Kirkpatrick, D., & Kirkpatrick, J. (2006). Evaluating Training Programs: The Four Levels (3rd ed.). Oakland, CA: Berrett-Koehler Publishers, Inc.

Kuykendall, B., Janvier, M., Kempton, I., & Brown, D. (2012). Interactive whiteboard technology: Promise and reality. In T. Bastiaens & G. Marks (Eds.), Proceedings of E-Learn 2012 – World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 1 (pp. 685-690). Retrieved from Association for the Advancement of Computing in Education (AACE), https://www.learntechlib.org/p/41669

Meeuwissen, N., Stalmeijer, R., & Govaerts, M. (2019). Multiple-role mentoring: Mentors conceptualisations, enactments and role conflicts. Medical Education, 53, 605-615. https://doi.org/10.1111/medu.13811

Richter, D., Kunter, M., Lüdtke, O., Klusmann, U., Anders, Y., & Baumert, J. (2013). How different mentoring approaches affect beginning teachers’ development in the first years of practice. Teaching and Teacher Education, 36, 166-177. https://doi.org/10.1016/j.tate.2013.07.012

Sambunjak, D. (2015). Understanding the wider environmental influences on mentoring: Towards an ecological model of mentoring in academic medicine. Acta Medica Academia, 44(1), 47-57. https://doi.org/10.5644/ama2006-124.126

Standing Committee on Postgraduate Medical and Dental Education. (1998). Supporting Doctors and Dentists at Work: An Enquiry into Mentoring. London: SCOPME.

Stenfors-Hayes, T., Hult, H., & Owe Dahlgren, L. (2011). What does it mean to be a mentor in medical education? Medical Teacher, 33(8), 423-428. https://doi.org/10.3109/0142159X.2011.586746 

Waljee, J. F., Copra, V., & Saint, S. (2018). Mentoring Millennials. JAMA, 319(15), 1547-1548. https://doi.org/10.1001/jama.2018.3804

*Frank Bate
Medical and Health Professional Education,
School of Medicine Fremantle,
University of Notre Dame Australia,
PO Box 1225, Fremantle,
Western Australia 6959
Telephone: +66 9433 0944
Email address: frank.bate@nd.edu.au

Submitted: 20 March 2020
Accepted: 28 July 2020
Published online: 5 January, TAPS 2021, 6(1), 70-82
https://doi.org/10.29060/TAPS.2021-6-1/OA2241

Sandra Widaty¹, Hardyanto Soebono², Sunarto³ & Ova Emilia⁴

¹Department of Dermatology and Venereology, Faculty of Medicine, Universitas Indonesia – Dr. Cipto Mangunkusumo Hospital, Indonesia; ²Department of Dermatology and Venereology Faculty of Medicine, Universitas Gadjah Mada, Indonesia; ³Pediatric Department, Faculty of Medicine, Universitas Gadjah Mada Indonesia, Indonesia; ⁴Medical Education Department, Faculty of Medicine, Universitas Gadjah Mada, Indonesia

Abstract

Introduction: Performance assessment of residents should be achieved with evaluation procedures, informed by measured and current educational standards. The present study aimed to develop, test, and evaluate a psychometric instrument for evaluating clinical practice performance among Dermatology and Venereology (DV) residents.

Methods: This is a qualitative and quantitative study conducted from 2014 to 2016. A pilot instrument was developed by 10 expert examiners from five universities to rate four video-recorded clinical performance, previously evaluated as good and bad performance. The next step was the application of the instrument to evaluate the residents which was carried out by the faculty of DV at two Universities.

Results: The instrument comprised 11 components. There was a statistically significant difference (p < 0.001) between good and bad performance. Cronbach’s alpha documented high overall reliability (a = 0.96) and good internal consistency (a = 0.90) for each component. The new instrument correctly evaluated 95.0% of poor performance. The implementation study showed that inter-rater reliability between evaluators range from low to high (correlation coefficient r =0.79, p < 0.001).

Conclusion: The instrument is a reliable and valid instrument for assessing clinical practice performance of DV residents. More studies are required to evaluate the instrument in different situation.

Keywords:            Instrument, Clinical Assessment, Performance, Resident, Dermatology-Venereology, Workplace-Based Assessment

Practice Highlights

• The residents’ performance will reflect on their professionalism and competencies. Furthermore, clinical care provided by Dermatology and Venereology field is unique, therefore a standard instrument is needed to assess their performance.
• Dermatology – Venereology Clinical Practice Performance Examination instrument is proven to be reliable and valid in assessing residents’ clinical performance

I. INTRODUCTION

Performance assessment in medical clinical practice has been a great concern for medical education programmes worldwide. (Holmboe, 2014; Khan & Ramachandran, 2012; Naidoo, Lopes, Patterson, Mead, & MacLeod, 2017). It is an accepted premise that performance may differ according to competency (Cate, 2014; Khan & Ramachandran, 2012). Performance also occurs within a domain; therefore, the assessment of performance should be separated from that of competency. Performance assessment of medical residents should also be informed by existing medical standards and performance criteria (Li, Ding, Zhang, Liu, & Wen, 2017; Naidoo et al., 2017).

Assessment of residents during their training programme is an important issue in postgraduate medical education, which has declared formative evaluation and constructive feedback as priorities (World Federation for Medical Education, 2015). An earmark of postgraduate medical specialist training is that it occurs in the workplace; therefore, the most appropriate measurement tools are Workplace-Based Assessments (WPBA). In medical education, these assessments emphasise on result and professionalism (Boursicot et al., 2011; Joshi, Singh, & Badyal, 2017).

In response to a standardisation programme for postgraduate medical specialist training (PMST), the World Federation for Medical Education (WFME) had published guidelines which adopted by several countries including Indonesia (Indonesian College of Dermatology and Venereology, 2008; World Federation for Medical Education, 2015). Clinical care provided in the Dermatology and Venereology (DV) field is unique; a brief examination of the patient is often useful before taking a lengthy history (Garg, Levin, & Bernhard, 2012). Privacy is a top priority, especially for venereology patients, patients with communicable diseases, cosmetic dermatology and skin surgery care.

Until now, no standard instrument has been available for performance assessment of PMST in DV; therefore, a variety of assessments are in use which may cause discrepancies (Jhorar, Waldman, Bordelon, & Whitaker-Worth, 2017). A valid and reliable method of assessment is required that can be used in various facilities and related to proficiency in both content and process (Kurtz, Silverman, Benson, & Drapper, 2003). Therefore, a study was conducted to focus on the development of a residents’ clinical performance assessment based on certain standards and principles such as the WPBA and WFME standards.

II. METHODS

A. Instrument Development

The instrument was developed and tested using qualitative and quantitative study designs. It started with a solicitation of inputs regarding expected performance from a variety of stakeholders in DV: patients, nurses, laboratory staff, newly graduated DV specialists, DV practitioners, and faculty. A literature review was performed, which included various documents such as the educational programme standards for DV residents, and documentation on available assessment tools (Cate, 2014; Hejri et al., 2017; Norcini, 2010). The instrument was developed according to the current standards (Campbell, Lockyer, Laidlaw, & MacLeod, 2007; McKinley, Fraser, van der Vleuten, & Hastings, 2000).

The resulted 11-items instrument was subsequently evaluated by faculty groups from various universities in Indonesia. Repeated revisions were carried out. Psychometric data for the instrument were provided through independent evaluations of performance videos of the residents and also through comparison of the results of the new instrument (Dermatology -Venereology Clinical Practice Performance Examination/DVP-Ex) and the compared instrument. The design was a validation study in which psychometric data for the instrument were provided. Further step was the assessment of residents’ performances when performing their clinical practice using the instrument to evaluate instrument reliability and feedback. Flowchart of the study process is shown in Appendix A.

B. Setting

The present study was conducted at the Department of Dermatology and Venereology, Dr. Cipto Mangunkusumo Hospital, a teaching hospital for Faculty of Medicine Universitas Indonesia, from 2014 to 2016. The study was conducted in four steps. When developing the instrument (Step 1), we included faculty members from five medical faculties in Indonesia that have DV Residency Programme (Universitas Indonesia, Universitas Sriwijaya (UNSRI), Universitas Padjajaran (UNPAD), Universitas Gadjah Mada, and Universitas Sam Ratulangi) through in depth interview and expert panel. The study received ethical approval from the Research Ethics Committee of the Faculty of Medicine Universitas Gadjah Mada Number KE/FK/238/EC.

The instrument that had been developed was sent to five senior faculty members from three universities (Department of Dermatology and Venereology, Universitas Indonesia, UNPAD and UNSRI) (Step 2). They were asked to give their assessment in order to have face and content validity. As a test of criterion validity, we recruited 10 faculty members of Faculty of Medicine, Universitas Indonesia, randomised them into two groups. Randomisation was performed to prevent bias against the instruments being tested. One group used DVP-Ex and the other used the current instrument. The single inclusion criterion was more than three years of teaching experience. After receiving some inputs, final correction was done and training was provided for faculty member who would use the instrument.

C. Performance Video

To obtain standardised performance of the residents, video recordings of the resident’s clinical practice were made. Two residents were voluntarily recruited and a special team recorded their clinical practice performance using scenarios created by the first author. (Campbell et al., 2007; McKinley et al., 2000)

There were four videos, each of which showed the clinical practice performance of the residents when they were presented with a difficult case (dermatomyositis) and a common case (leprosy tuberculoid borderline type). Patients had to sign informed consent before included in this study. A good (first and fourth video clips) and poor (second and third clips) standard of performance were demonstrated. Activities presented in the scenarios were those associated with patient care (Campbell et al., 2007; Iobst et al., 2010). After the recording session was finished, patients were managed accordingly and provided with rewards.

D. Training on the Performance Instrument

An hour-long training was provided for the 10 faculty members (the examiners). The faculty then practiced scoring, using the recorded video clips. During the training, we received some input and made necessary corrections to the rubrics. There was no training given for the comparison instrument because the entire faculty was already accustomed to this instrument. Step 3 is the step to produce validity, reliability and accuracy of performance instrument, which was conducted through a comparative study between two instruments of assessment; i.e. performance and control instruments. It evaluated the clinical practice performance of residents in the form of video film recording their performance

 E. Implementation of Resident Performance Assessment with Performance Instrument

This step was aimed to evaluate the reliability of the instrument and results of instrument implementation when it was used to assess the residents (Step 4). The sample included residents of Postgraduate Medical Specialist Training Programme in Dermatology and Venereology, Faculty of Medicine, University of Indonesia and UGM, who were at their basic level (residents who were on their 1st semester in clinical setting), intern level (semester II-V) and independent level (semester VI or higher). 

Sample size:  n = 3 – 4/level/Faculty of Medicine = 20. The evaluators were five lecturers/ Faculty of Medicine = 10, and each lecturer evaluated six residents.

F. Data Collection

One week after the training, the instrument was evaluated. Faculty members assessed the performance of the residents in the four video recordings at the same time. Three days later, the groups underwent a rotation to reassess the video with whichever of the two instruments they had not already used. The examiners were asked to provide feedback and information on the ease of completing the instrument and the clarity of its instructions.  For the implementation of resident assessment performance with the instrument, one resident was being evaluated by three lecturers simultaneously. The lecturers were grouped randomly; therefore, every lecturer could evaluate six residents out of ten residents from each group that would be assessed.

G. Data Analysis

The analyses aimed to evaluate validity, reliability, and precision of the instrument for discriminating the performance of the residents as poor, good, or excellent.

H. Validity and Reliability

A reliability test was performed, i.e. internal consistency in the form of responses against items in each field (Cronbach alpha coefficient). Face and content validity were assessed by addressing the relevant performance standards and criteria, and by optimizing clarity of instruction, specific criteria, acceptable format, gradation of responses, correct and comprehensive answers (including all assessed variables). The cut-off score of the instrument was determined using ROC (receiver operating curve) principles, which was then used to evaluate sensitivity, specificity, positive and negative predictive value. The accuracy of the instrument was determined to evaluate the precision of the instrument in distinguishing between good and poor performance.

I. Statistical Analysis

The statistical analysis was performed using SPSS 11.5 software. Total assessment scores of each examiner were analysed using analysis of variance (ANOVA). Internal consistency was determined using Cronbach’s α and Spearman analysis was performed to acquire p value for the validity. The accuracy was determined by comparing failed or passed score results and comparing it with the video. To obtain the intergroup difference, McNemar’s test and Kappa analysis were carried out. Qualitative analysis was also performed, especially to evaluate feedbacks by performing several analytical steps.

III. RESULTS

A performance instrument was developed with 11 competency components, for which evaluation responses were given in the form of rubric scale (Appendix B). All 10 faculty members completed an assessment of each of the four videos. Eight examiners had more than 3 years of teaching experiences, and five examiners were DV consultants.

A. Validity

For validity, face, content, and construct validity remain solid points of reference for validity evaluation (Colliver, Conlee, & Verhulst, 2012; Johnson & Christensen, 2008). Face content was evaluated by five experts from three universities. The evaluation was implemented to improve the instrument. The scale of the rubrics described the capacity of residents to perform activities according to the Standard Competency of DV specialist and the domain of performance for physicians has made the instrument evaluated as the instrument with good validity on its face, content and construction.

The results of the assessments made on performance videos with the DVP-Ex showed that examiners agreed that the performances of the first and fourth videos (the “good” videos) were good performance (>60); conversely, the second and third videos (the “bad” ones) were evaluated as poor performance by 10 and 9 out of 10 faculty members, respectively (Table 1).

Note: Chi Square, Kruskal–Wallis p < 0.001

Table 1. Assessment scores for each of the four videos (n = 10)

Faculty members also gave feedback suggesting that the instrument would be useful for assessing residents’ performance. They also commented that the instrument was more objective than the one currently in use, that it was challenging in that they had to read the instrument carefully in order to use it properly, and that the response options allowed several aspects of the residents’ performance to be assessed.

B. Validity and Reliability

Validity of the instrument was measure using Spearman analyses showed significant result for all of the competency component (p > 0.001). Reliability measure of the correlation between each item score and the total score on all relevant items (Cohen, Manion, & Morrison, 2008). Our analysis revealed good overall reliability, with Cronbach α = 0.96. All components of competency achieved internal reliability scores >0.95. The correlation between each item score on the competency components and the overall score was excellent (range: 0.64–0.99).

Submitted: 16 April 2020
Accepted: 24 June 2020
Published online: 5 January, TAPS 2021, 6(1), 83-92
https://doi.org/10.29060/TAPS.2021-6-1/OA2251

Eng Koon Ong 

Division of Supportive and Palliative Care, National Cancer Centre Singapore, Singapore; Assisi Hospice, Singapore

Abstract

Introduction: Physician empathy is declining due to an unproportionate focus on technical knowledge and skills. The medical humanities can counter this by allowing connection with our patients. This is a pilot study that aims to investigate the acceptability, efficacy, and feasibility of a humanities educational intervention to develop physician empathy.

Methods: Junior doctors at the Division of Supportive and Palliative Care at the National Cancer Centre Singapore between July 2018 and June 2019 attended two small-group sessions facilitated by psychologists to learn about empathy using literature and other arts-based materials. Feasibility was defined as a completion rate of at least 80% while acceptability was assessed by a 5-question Likert-scale questionnaire. Empathy was measured pre- and post-intervention using Jefferson’s Scale of Physician Empathy (JSPE) and the modified-CARE (Consultation and Relational Empathy) measure.

Results: Seventeen participants consented, and all completed the programme. Acceptability scores ranged from 18 to 50 out of 50 (mean 38, median 38). There was an increase in JSPE scores (pre-test mean 103.6, SD=11.0 and post-test mean 108.9, SD=9.9; t (17) =2.49, P=.02). The modified-CARE score increased between pre-test mean of 22.9(SD=5.8) and a post-test mean of 28.5(SD=5.9); t (17) = 5.22, P<0.001.

Conclusion: Results indicate that the programme was acceptable, effective, and feasible. The results are limited by the lack of longitudinal follow-up. Future studies that investigate the programme’s effect over time and qualitative analysis can better assess its efficacy and elicit the participants’ experiences for future implementation and refinement.

Keywords:            Empathy, Humanities, Literature, Palliative Medicine

Practice Highlights

• The medical humanities can be used to teach empathy by facilitating reflective practice.
• This novel educational programme was acceptable, effective, and feasible.
• Limitations include the lack of longitudinal follow-up and the quantitative nature of assessment.
• Future studies should investigate the programme’s effect over time and include qualitative analysis.

II. METHODS

A. Intervention Design

The Humanistic Aspirations as a Propeller of Palliative medicine Education (HAPPE) programme was conceived to introduce and develop a novel curriculum in empathy for junior doctors undergoing a palliative medicine rotation.  The overall goal of the study was to design an effective education programme based on the humanities to teach doctors empathy. Our study draws upon Schon’s work of Reflective Practice (see Table 2; Schön, 1987).