Number of Citations: 0

Published online: 5 May, TAPS 2020, 5(2), 14-21
https://doi.org/10.29060/TAPS.2020-5-2/OA2128

Lay Ling Tan & Carmen Jia Wen Kam

Department of Psychological Medicine, Changi General Hospital, Singapore; Clinical Trials and Research Unit, Changi General Hospital, Singapore

Abstract

Aims: Psychiatry residents’ outpatient clinic supervision may be with direct observation of clinical cases (D), without direct observation (WDO) or a mixture of both (M). This study explored residents’ perceptions of clinical teaching effectiveness under these supervision frameworks.

Methods: A survey was conducted amongst residents who completed their third-year training. Cleveland Clinic’s Teaching Effectiveness Instrument (CCTEI) was used to capture their perception of various domains of clinical teaching effectiveness. Content analysis of their qualitative feedback was used to establish the major categories of supervision perception.

Results: 42 out of 60 residents responded. Fewer residents received DO and M compared to WDO. More residents preferred DO. There was no statistical difference in the total CCTEI scores for the three supervision formats. Qualitative content analysis revealed categories surrounding strengths of DO and WDO as well as weaknesses of DO. Although DO allowed timely feedback, addressed patient safety and increased residents’ confidence, it was perceived to be stressful, requiring more resources and inhibited independent learning. WDO was commended for its support of autonomous learning, less anxiety-provoking and contributed to better patient rapport.

Conclusion: There was no statistical difference in clinical teaching effectiveness of the three supervisory frameworks. Previous studies showed DO to be anxiety-provoking with concerns of compromised autonomous learning. This study uncovered similar themes, but residents still preferred DO. It highlighted the residents’ perception of the importance of timely feedback, patient safety and instilling confidence with DO.

Keywords:        Psychiatry, Residents, Supervision, Direct Observation, Ambulatory, Outpatient

Practice Highlights

• There has been little guidance in the literature on best practices in psychiatry supervision.
• Many teaching sites offered WDO, but residents preferred DO with its timely feedback.
• DO was perceived to be stressful, needed more resources and inhibited independent learning.
• WDO was perceived to support autonomous learning, less anxiety provoking and offered better rapport with patients.
• Use of valid assessments may guide the adoption of M, leveraging on the strengths of DO and WDO.

I. INTRODUCTION

The Psychiatry Residency Training in Singapore is a national program which requires our 3^rd year residents to be posted to the outpatient clinics for a year. There are seven teaching sites in the residency program and each site offers its own supervision framework. This may be direct observation (DO) with the supervisor sitting in with the resident for the entire clinic visit or an alternative supervisory model with the resident seeing the new cases independently and discussions with the supervisors followed with no direct observation (WDO) of the consultation process. Some sites offer a mixture of both supervisory framework (M), largely determined by supervisor availability and not intentionally planned and structured.

There have been questions from the faculty, clinical supervisors and residents as to whether DO was a more effective model of supervision. The general perception was that DO would offer better patient care quality and less patient safety concerns. However, there were also concerns about the sustainability of the model of DO in view of the increasing pool of residents. In their brief report, Reardon, May, and Williams (2014) aptly discussed the challenges training directors face in balancing optimisation of patient care, education and reimbursement.

There has been little recent research on the supervision of the psychiatry interview in the non-psychotherapy outpatient setting. Extant literature highlighted advantages of DO, including a more accurate picture of the patient-student relationship; being able to alert students to non-verbal cues and students feeling less threatened since they are not without aid and corroboration (Lewin, 1966). A study by Stein, Karasu, Charles, and Buckley (1975) comparing DO with a verbal report of the interview also revealed that some residents voiced resentment that the treatment was complicated by the supervisors’ interventions and that their inadequacies and inexperience were exposed by the presence of the supervisor.

Supervision has been defined as the provision of monitoring, guidance and feedback on matters of personal, professional and educational development in the context of the doctor’s care of patients (Kilminster & Jolly, 2000). It is a complex activity with different modes of delivery that occurs in a variety of context and is undoubtedly important for the professional development of our residents. Previous work by Stein et al. (1975) suggested that DO was an important factor in the accuracy of patient evaluation and the process of verbal report itself without direct observation tended to influence the supervisor in minimising psychopathology. This will have implications in terms of the quality of patient care.

A recent review of the literature on direct and indirect supervision in psychiatric and other graduate medical education determined that there had been a dearth of research on the impact of direct supervision on the educational experience of psychiatry residents (Galanter et al., 2016). There has been little guidance in the existing literature on best practices in supervision of psychiatry residents with resulting unclear expectations (Newman, Ravindranath, Figueroa, & Jibson, 2016). This survey attempted to compare the residents’ perception of the clinical teaching effectiveness under the various supervision frameworks. This will further enhance our understanding of the variables perceived to be significant in the supervisory process of residents in general psychiatry ambulatory training. It will aid in the program’s consideration of options for optimising patient care and residency education while addressing resource limitations.

II. METHODS

The intent and purpose of the survey was discussed with the Psychiatry Residency Program Director and waiver of informed consent was sought from the Institutional Review Board. Email invitations to the survey detailing the study objectives and questionnaire were sent to all the residents at the end of their third-year ambulatory training over a period of three years. Non-responders (residents who did not respond to the email invitations after two weeks) were given the survey by the resident representative during their weekly residency didactics at the various teaching sites. The residents were posted to teaching sites with different supervision frameworks. Comparisons were made based on the residents’ declaration of the supervision framework offered by each teaching site more than 50% of the time of their ambulatory training. Their current supervisory framework (DO, WDO or M) and their preferred choice of supervision were captured in the analysis.

Feedback and comments on their supervision in the questionnaire were included for qualitative analysis. A deductive content analysis was adopted with development of the categorisation matrix by the principal investigator. Atlas Ti (version 8) software was used to code the data according to the categories.

The Cleveland Clinic’s Teaching Effectiveness Instrument (CCTEI) was used to assess the residents’ perception of the clinical teaching effectiveness under the various supervisory format. The CCTEI has been shown to be a reliable and valid evaluation tool for a wide variety of clinical teaching settings (Copeland & Hewson, 2000). It has 15 questions that used a five-point evaluation scale (see Appendix).

Categorical data was presented as frequency (percentage). Numeric data was tested with the Shapiro-Wilk normality test and presented as median (interquartile range) when the data distribution did not follow normal distribution. The associations between two categorical variables were examined using Fisher’s Exact test. Differences of CCTEI scores among three groups of supervision format were assessed by Kruskal Wallis test. In case of statistically significant difference in Kruskal Wallis test, Bonferroni post-hoc adjustment was used for multiple pairwise comparisons. A two-tailed, p-value of <0.05 was considered statistically significant. Statistical analysis was performed with SPSS statistical software, version 19.0 (IBM Corp. Armonk, NY).

III. RESULTS

Out of the 60 eligible residents, 42 responded to the survey, yielding a response rate of 70%. Most sites offered WDO supervision (45.2%). M was the supervision framework least frequently adopted across the teaching sites (19%) and 35.7% received DO (Table 1). About 80% of the residents were more than three years post-graduation at the time of the survey. There were no statistical differences in the demographic variables and the number of years post-graduation with the different supervision formats (Table 1).

Table 2. Median (interquartile range) of CCTEI Item scores and total score with the different supervision format

When asked for their preferences, most residents opted for DO (20/41, 48.8%) with a fairly equal distribution between M (11/41; 26.8%) and WDO (10/41; 24.3%). Although all but two of the eight residents with less than three years of clinical experience preferred DO and residents with more than three years of clinical experience preferred M supervisory format, the results were not statistically significant (p = .512 and .387 respectively).

Qualitative comments of residents’ perceptions and reasons for their supervision preferences captured in the questionnaire were perused in detail. A deductive content analysis was used to test existing categories and concepts with regards to clinical supervision (Elo & Kyngäs, 2008). A categorisation matrix was developed with three major categories comprising (a) Weaknesses of DO; (b) Strengths of DO; (c) Strengths of WDO. All the data were then reviewed for content and coded for correspondence with the identified structured categories using the Atlas Ti (version 8) software. All the qualitative data was able to fit into the categorisation frame. There were 19 quotations referring to weaknesses of DO; 31 quotations for strengths of DO and 21 quotations for strengths of WDO. Surprisingly, there were no statements made addressing weaknesses for WDO or feedback regarding M supervision. To further improve credibility and transferability, member checking was used. Participants’ comments regarding the developed categories were solicited. There was general agreement with the findings generated from their feedback.

A. Weaknesses of DO

The themes for the category on DO weaknesses were related to it being more stressful, needing more resources and inhibiting independent learning.

1) Stressful Environment: Some residents described that it was stressful to have the supervisor observe them for the entire consultation.

Female Resident 3: “It was nerve-wracking and anxiety-provoking to be watched.”

Female Resident 9: “It was personally very stressful to be observed while clerking a case.”

There was a perceived need for a more conducive learning environment as residents in their junior years of ambulatory training learn to engage their patients and family caregivers in the clinical encounter. The presence of the supervisor with an evaluative function was perceived to contribute to a stressful learning environment.

2) More Resources are Needed: Residents reported that having the supervisor sit in throughout the clinical encounter contributed to wastage of time and resources.

Female Resident 2: “Drains resources.”

Male Resident 8: “Will be too time-consuming and labour-intensive for supervisor and trainee.”

With the perennial challenge of having to reduce the lead time for new referrals to be seen by the outpatient specialist clinics, residents might have the notion that they were contributing to the poor utilization of resources and that more resources would need to be available to cater to both their educational and clinical service needs.

3) Inhibit Independent Learning: There was the perception that DO might potentially affect the residents’ autonomous learning.

Female Resident 2: “Can potentially inhibit the learner’s capacity for individual learning and development of interview style as he/she may feel conscious of the presence of an observer and compelled to conform to the supervisor’s style.”

Male Resident 10: “Having a supervisor in the room all the time can potentially inhibit the learner’s capacity for independent learning.”

The residents were concerned that the supervisor’s constant presence might potentially inhibit their independent learning as they would be closely guided by their supervisor when they encounter any difficulty during the clinical consultation. They also feared that they might be pressured to follow the supervisor’s particular style of interviewing.

B. Strengths of DO

The themes for the category on DO strengths centred around timely feedback, ensuring patient safety and building resident’s confidence.

1) Timely Feedback: Residents valued the immediate feedback achieved through DO.

Female Resident 7: “Able to have real-time discussion of not only management of patient, but also obtain feedback on interviewing skills and my interaction with the patients.”

Female Resident 15: “Would be the ideal scenario as it offers the opportunity to obtain feedback without the clinical interview process and on the spot learning.”

The residents described the opportunities to have their interviewing techniques scrutinized and honed by the supervisors with timely feedback as invaluable and ideal in their ambulatory training.

2) Ensuring Patient Safety: Residents were more assured that they were delivering safe patient care when they were able to consult their supervisors immediately during DO.

Female Resident 3: “Observe my management – To fine-tune my clerking, clinical management skills. Provides supervision for patient safety.”

Male Resident 23: “Discuss management plan together at first visit–Better for patient care.”

Residents endorsed direct observation of their initial interview as an important factor in the accuracy of patient assessment and evaluation. This wouldoffer better patient care quality and less patient safety concerns.

3) Building Residents’ Confidence: DO was considered to be pertinent in boosting the residents’ confidence.

Male Resident 8: “Is especially useful during the initial few weeks/months or for newer trainees to build up confidence.

Male Resident 13: “I think reviewing the case together with consultant is not required unless the resident doesn’t feel confident in managing the case.”

Residents in their junior years of training with less clinical experience would appreciate the reassuring presence of their supervisor and were possibly less conscious of exposing their inexperience and inadequacies.

C. Strengths of WDO

With the category on strengths of WDO, participants cited more autonomous learning, less anxiety and better patient rapport.

1) Autonomous Learning: Residents viewed WDO as helpful in encouraging independent learning.

Female Resident 2: “Allows greater autonomy for independent learning.”

Female Resident 4: “It gave me the independence to make decisions.”

Residents felt that WDO allowed them more opportunity and time for their own reflections about the clinical assessment and to plan their own management plan before discussing with their supervisors.

2) Less Anxiety: WDO was referred to as less stressful compared to DO.

Female Resident 4: “It also allowed me to manage the session without worrying all the time about someone else being present.”

Female Resident 25: “There will be less stress.”

Residents felt that the clinical environment will be more relaxed without the presence of the supervisor and they were better able to focus on engaging their patients and caregivers.

3) Better Patient Rapport: Residents felt that WDO allowed better patient rapport.

Male Resident 19: “Allow resident to develop own style of consult, better rapport with patient.”

Female Resident 34: “Allows the resident to build better rapport with patient.”

Residents perceived that their therapeutic alliance with their patients would be better established if the supervisor were not present in the consult room.

IV. DISCUSSION

This survey was driven by concerns about the sustainability of the model of DO in one of the teaching sites and whether DO is the most effective supervisory approach in our psychiatry ambulatory training. This study found that many of the teaching sites offered WDO as the supervisory model. Clinical supervisors were entrusted with trainees in their third year of residency with different trajectory in terms of their competency milestones. First impressions were usually relied upon to determine competency and skills of the residents (Wood, 2014). Concerns about the accuracy and diminished clinical objectivity of patient evaluation without direct observation were previously highlighted (Stein et al., 1975). This may potentially impact residency training outcomes and patient care.

A survey on how Directors of United States psychiatric residency programs were balancing patient care, education and reimbursement was recently conducted by Reardon et al. (2014). Options were proposed for optimising patient care and resident education while maintaining financial solvency. These included having a process of identifying resident supervisory needs based on training level, resident competency and complexity of patient care (Reardon et al., 2014).

Although maintenance of financial solvency is not expected of our program directors currently, there will be a need to justify the manpower needs for DO for an increasing pool of residents in future. Palesy (2017) studied the use of scaffolding and how learning might be supported in the absence of direct supervision. The consideration of a range of scaffolds provided in the learning environment without rigid stipulations might be able to encourage residents without direct observation to decide on the type and frequency of their interaction with the scaffolding provided (Palesy, 2017). The use of reliable and valid assessments e.g. mini-clinical examinations (mini-cex) may help with determining the readiness of residents for independent clinical practice and better guide the adoption of the mixed supervisory format (Norcini, Blank, Duffy, & Fortna, 2003).

Although residents alluded to WDO’s strengths in offering more autonomous learning, less anxiety and better patient rapport, our study suggested that our psychiatry residents still favoured DO, particularly those with less clinical experiences (less than three years post-graduation). This finding concurred with an earlier report of how DO might be perceived to be less threatening for students (Lewin, 1966). Residents with less clinical experience would appreciate the reassuring presence of a supervisor and possibly less conscious of exposing their inadequacies (Stein et al., 1975). Only Q11 of the CCTEI (coaches me on my clinical/technical skills) was rated as statistically significant amongst the three supervision frameworks, favouring DO. Our residents generally welcome coaching on their psychiatry skills using DO and our findings did not suggest that DO affected their independent learning (CCTEI Q2) and autonomy (CCTEI Q3) in their patient management.

A recent study by Galanter et al. (2016) referred to the traditional supervisory process in psychiatry training without direct observation, where residents were seen to be granted more autonomy and the therapeutic alliance would not be disturbed by having the supervisor in the room. Analysis of the qualitative feedback from residents in our study highlighted their concerns that DO may affect their rapport with patients but at the same time, the presence of a supervisor also helped to boost their confidence and reassure them of patient safety and quality of care. Without DO, there would be a need to have readily available guidance to residents like videoconferencing and use of other technology-facilitated supervision to support their delivery of appropriate care to their patient (Cameron, Ray, & Sabesan, 2015; Hayden, Navedo, & Gordon, 2012).

Our study has limitations. The number of residents who participated in the survey was relatively small although we managed to have a 70% response rate. Self-reported data was used for qualitative analysis with the risk of recall biases. Although independent verification of self-reported data could not be established, there were no incongruent findings. Member checking was also used to improve the credibility of the analysis.

 V. CONCLUSION

Our study did not reveal any statistical difference in the perception of clinical teaching effectiveness of the three supervisory frameworks based on the total CCTEI scores. Previous studies showed DO to be stressful with concerns of too much ‘hand-holding’ and compromised autonomous learning. Our qualitative analysis uncovered similar themes, but residents still preferred DO. The study highlighted the residents’ perceptions of the importance of timely feedback, patient safety and instilling confidence with DO.

Close supervision by DO for all cases may not be feasible but it may be crucial for residents early in their training. There should be flexibility in the supervisory framework for residents to function more independently when they are deemed to be competent and safe. The use of reliable and valid assessments may help with determining the readiness of residents for independent clinical practice and better guide the adoption of the mixed supervisory format. More research is needed to elucidate the determinants of the most effective supervisory model and to develop innovative ways of ensuring a high standard of clinical supervision and patient care.

Notes on Contributors

Dr Tan Lay Ling is Senior Consultant Psychiatrist in the Psychogeriatric Service of the Department of Psychological Medicine of CGH. She is currently appointed as the Associate Program Director for the National Psychiatry Residency Program in CGH. She is also a member of the Clinical Competency Committee of the Residency Program.

Miss Kam Jia Wen Carmen is the Biostatistician of the Clinical Trials and Research Unit of CGH.

Ethical Approval

Submission for review was deemed not required by the SingHealth Institutional Review Board (iSHaRe Ref. No: 201601-00106; CIRB Ref. No: 2016/2083).

Acknowledgements

We would like to acknowledge the contribution of the National Psychiatry Residency Office for its administrative support in this survey and the residents for their willingness to participate.

Funding

We did not receive any funding for this research study.

Declaration of Interest

The main researcher is the Associate Program Director of one of the teaching sites and has a vested interest in ensuring the quality of supervision of residents. The researcher cannot decide on the standardisation of the supervisory framework. There are no other conflicts of interest, including financial, consultant, institutional and other relationships that might lead to bias or a conflict of interest.

References

Cameron, M., Ray, R., & Sabesan, S. (2015). Remote supervision of medical training via videoconference in northern Australia: a qualitative study of the perspectives of supervisors and trainees. BMJ open, 5(3), e006444. https://doi.org/10.1136/bmjopen-2014-006444

Copeland, H. L., & Hewson, M. G. (2000). Developing and testing an instrument to measure the effectiveness of clinical teaching in an academic medical center. Academic Medicine, 75(2), 161-166. https://doi.org/10.1097/00001888-200002000-00015

Elo, S., & Kyngäs, H. (2008). The qualitative content analysis process. Journal of Advanced Nursing, 62(1), 107-115. https://doi.org/10.1111/j.1365-2648.2007.04569.x

Galanter, C. A., Nikolov, R., Green, N., Naidoo, S., Myers, M. F., & Merlino, J. P. (2016). Direct Supervision in Outpatient Psychiatric Graduate Medical Education. Academic Psychiatry, 40(1), 157-163. https://doi.org/10.1007/s40596-014-0247-z

Hayden, E. M., Navedo, D. D., & Gordon, J. A. (2012). Web-conferenced simulation sessions: a satisfaction survey of clinical simulation encounters via remote supervision. Telemedicine and e-Health, 18(7), 525-529. https://doi.org/10.1089/tmj.2011.0217

Kilminster, S., & Jolly, B. (2000). Effective supervision in clinical practice settings: a literature review. Medical Education, 34(10), 827-840. https://doi.org/10.1046/j.1365-2923.2000.00758.x

Lewin, K. (1966). Psychiatric supervision by direct observation. Academic Medicine, 41(9), 860-864. https://doi.org/10.1097/00001888-196609000-00005

Newman, M., Ravindranath, D., Figueroa, S., & Jibson, M. D. (2016). Perceptions of supervision in an outpatient psychiatry clinic. Academic Psychiatry, 40(1), 153-156. https://doi.org/10.1007/s40596-014-0191-y

Norcini, J. J., Blank, L. L., Duffy, F. D., & Fortna, G. S. (2003). The mini-CEX: a method for assessing clinical skills. Annals of internal medicine, 138(6), 476-481. https://doi.org/10.7326/0003-4819-138-6-200303180-00012

Palesy, D. (2017). Learning in the Absence of Direct Supervision: Person-Dependent Scaffolding. Vocations and Learning, 10(3), 365-382. https://doi.org/10.1007/s12186-017-9176-9

Reardon, C., May, M., & Williams, K. (2014). Psychiatry resident outpatient clinic supervision: how training directors are balancing patient care, education, and reimbursement. Academic Psychiatry, 38(4), 476-480. https://doi.org/10.1007/s40596-014-0091-1

Stein, S. P., Karasu, T. B., Charles, E. S., & Buckley, P. J. (1975). Supervision of the initial interview: A study of two methods. Archives of General Psychiatry, 32(2), 265-268. https://doi.org/10.1001/archpsyc.1975.01760200129013

Wood, T. J. (2014). Exploring the role of first impressions in rater-based assessments. Advances in Health Sciences Education, 19(3), 409-427. https://doi.org/10.1007/s10459-013-9453-9

*Tan Lay Ling
Changi General Hospital,
2 Simei Street 3, Singapore 529889
Tel: 69365072
E-mail: tan.lay.ling@singhealth.com.sg