Number of Citations: 0

Published online: 2 May, TAPS 2018, 3(2), 29-37
DOI: https://doi.org/10.29060/TAPS.2018-3-2/OA1051

Wayne A. de Beer & Helen E. Clark

Waikato District Health Board, Hamilton, New Zealand

Abstract

The New Zealand Curriculum Framework (NZCF) for Prevocational Medical Training identifies a number of procedural skills that prevocational doctors should achieve during their first two years following graduation from medical school. This study aimed to identify the clinical confidence of graduate doctors in performing the list of procedures outlined in the NZCF at two points in time; following completion of undergraduate studies, and the first year of prevocational, preregistration training. An anonymous paper-based survey, consisting of 59 items, was completed by a cohort of PGY-1 doctors (n = 30) twice during 2015, with the first 48 items of the survey rating PGY-1s perceptions of their clinical confidence in performing procedures that fall under the 12 competencies identified in the Procedures and Interventions section of the NZCF. 70.8% of the procedures were rated above 2.0 at the start of the PGY-1 year, indicating that respondents had received teaching in, or viewed the procedure being performed, during undergraduate training. By year-end, procedural skills performance rated above 3.0 (i.e., confident in performing said procedure independently) was achieved in 52% of the listed skills. Low scores occurred in procedures listed under the categories ENT, Ophthalmology, Surgery and Trauma. While ratings of clinical confidence improved in many areas as expected during the PGY-1 tenure, some areas remained low. This highlights an issue that PGY-1 doctors may not be receiving adequate training in certain procedural skills listed as core NZCF competencies during the PGY-1 year.

Keywords:        Prevocational Doctors, Core Competencies, Procedural Skills, Clinical Confidence

I. INTRODUCTION

The New Zealand Medical Council (NZMC) published its New Zealand Curriculum Framework (NZCF) for prevocational medical training in February 2014, with the curriculum implemented in November 2014 (Medical Council of New Zealand [MCNZ] , 2014). The curriculum framework identified the expected learning outcomes for doctors during the first two years of employment following graduation from one of the two New Zealand medical schools. These two years are referred to as the Post-Graduate Year-1 and -2 (PGY-1 / -2) years.

The NZCF is designed to reflect the continuum of learning that starts during undergraduate training and continues during the PGY-1 and -2 years (MCNZ, 2014). The aim of the learning outcomes and, in particular, procedural competence is to promote and ensure patient safety (Patel, Oosthuizen, Child, & Windsor, 2008). To obtain general registration at the end of the PGY-1 year, doctors should have achieved sufficient experience in competently performing a substantive number of the procedures. The PGY-2 year allows for further refinement of procedural skill learning and helps to prepare house officers for vocational training.

The NZCF consists of five sections: Professionalism, Communication, Clinical management, Clinical problems and Conditions, and, Procedures and Interventions. Six overarching outcome statements apply to the execution of the Procedures and Investigations section. These relate the doctor’s ability to provide “safe treatment to patients by competently performing certain procedural and assessment skills” e.g. take informed consent, preparation and post procedure care (MCNZ, 2014). Procedural skills are listed under 12 categories (Table 2). During the PGY-1 year, doctors should achieve competency in 48 identified procedures (NZCF lists 47 procedures, however for measurement purposes we separated female and male bladder catheterisation procedures).

In addition to apprenticeship training achieved during the clinical attachments, various other learning opportunities exist for procedural skills learning during prevocational training. At the organisation where this study took place, six 1.5 hours’ sessions were scheduled for procedural skill learning at a skills simulation centre. PGY-1 doctors were also required to attend an 8-hour advanced life support training session to achieve the New Zealand (NZ) Certificate of Resuscitation (CORE). Formal education sessions provided additional opportunity to teach the theory to support procedural learning.

In this study, the PGY-1 doctors were asked to rate their confidence levels in the performance of several listed procedures. Clinical confidence was defined as an “acquired attribute that provides individuals with the ability to maintain a positive and realistic perception of self and abilities.” (Evans, Bell, Sweeney, Morgan, & Kelly, 2010). It is important to note that ‘clinical confidence’ and ‘clinical competence’ are not necessarily equivalent, with a brief definition of the latter being “the capability to perform acceptably those duties directly related to patient care”. Clinical competence can only be measured by standardised assessment frameworks such as those based on Miller’s pyramid model (Miller, 1990). On the other hand, clinical confidence is a self-assessment, which is not necessarily measurable by standardised tests. Students’ abilities to correctly self-assess have been documented frequently in the medical literature and procedural confidence was identified as an important concept (Fitzgerald, White, & Gruppen, 2003). Two previous studies pointed to procedural confidence as affecting the students’ willingness to engage in the procedure, engage in accurate self-assessment, and to seek external help in performing the procedure (Byrne, Blagrove, & McDougall, 2005; Hays et al., 2002).

The respondents’ ratings of clinical confidence in each of the identified procedures were compared at the start and end of the PGY-1 year. The first rating, at the start of the year aimed to identify their clinical confidence in undertaking procedures following their undergraduate training. This would theoretically reflect the degree to which the two NZ-based undergraduate programmes helped prepare students to learn procedural skills in clinical settings. By assessing their confidence at the end of the PGY-1 year, the authors wanted to assess the direction and degree of any changes in confidence in procedural skills performance because of PGY-1 training.

Benner’s Stages of Clinical Competence was used and adapted to medical training to define different levels of perceived confidence (Benner, 1984). Five statements guided house officers to determine their level of clinical competence in the procedural skills outlined (Table 1).

Table 1. Survey statements based on Benner’s stages of clinical competence

II. METHODS

The Clinical Education & Training Unit (CETU) at Waikato District Health Board (DHB) designed a paper-based survey, consisting of 59 items based on the competencies stated within the NZCF. These items were scored from 0-4; the score reflecting house officers’ perception of their clinical confidence level (Table 1). The first 48 items rated their confidence in performing procedures within the 12 categories identified in Table 2.

Table 2. Clinical skills and procedures item categories

Items 49 – 59 were designed to measure additional skills that fell within categories of leadership, administrative and communicative skills. The results for these items will be discussed in a separate publication.

A review of the Standard Operating Procedures of the Health and Disability Ethics Committee (HDEC) determined that the study did not require formal ethics approval, due to meeting guidelines around health information, human tissue and human participants, as outlined in the HDEC scope summary (Health and Disability Ethics Committee, 2016). Ethical standards were adhered to.

All PGY-1s (n = 30) who commenced working at Waikato DHB in 2015 were asked to complete the survey twice in 2015. Participants were offered the choice of partaking and could withdraw involvement at any stage. The first survey (baseline) was conducted at the start of the 2015 PGY-1 orientation period, while the second survey was conducted at the end of the 4th quarter. Response rate was high; 100% (30 respondents) at baseline, and 83% (25 respondents) at the end of the year (EOY). Survey identification numbers were used to track individual progress while maintaining respondent confidentiality. Demographic data related to gender and medical school attended prior to PGY-1 level was also collected.

III. RESULTS

All survey responses were recorded and analysed. Cronbach’s Alpha was .964 for the baseline survey and .868 for the EOY survey, which showed that the items had high internal consistency at both time points. Differences between individual item means at baseline and EOY were statistically analysed by using the Wilcoxon signed-ranked test.

Table 3 outlines the demographic data of our respondents (where identified). Sixty percent of our PGY-1 doctors were female. Of the group, 63.3% studied at the University of Auckland with 30% coming from the University of Otago.

Table 3. Demographics of respondents (Overall)

Table 4 outlines the mean respondent rating for baseline and EOY survey items that were part of the Clinical Skills and Procedures section. When interpreting the table, the authors concluded that any items that fell below a mean of 2 at baseline were identified as warranting attention. Similarly, items that fell below 3 at the end of the PGY-1 year were identified as potential areas for concern.

Table 4. Baseline and End of Year (EOY) self-rated competence level (clinical skills and procedures)

At the start of the PGY-1 year, the new doctors were most confident in their ability to scrub up, gown and glove (3.52) and this improved at EOY (3.79). This was followed by confidence in performing less invasive procedures like taking nasal/throat swabs and performing venepuncture. At EOY, taking venous or arterial blood, arranging a blood transfusion, performing intravenous cannulation and administering appropriate intravenous electrolytes scored above 3.5 indicating high clinical confidence levels. Male bladder catheterisation also scored highly at EOY (3.75).

Of the 48 clinical procedures listed, 34 (70.8%) were rated above 2.0 indicating that they had received satisfactory skill training in that procedure during undergraduate training. In the EOY survey, 43 out of 48 (90%) procedures were performed above the score of 2.0. However, the authors considered that by the end of the PGY-1 year doctors should be performing at a score of 3 indicating that multiple opportunities for practice of the skill had existed during the PGY-1 year and that they were confident performing the procedure independently. Twenty five of the 48 procedures (i.e. 52%) scored confidence levels above the score of 3. Low scores tended to occur in the following categories; Ear Nose and Throat (ENT), Ophthalmic, Surgical (more specifically, excising simple lesions, deriding and dressing a wound) and Trauma.

Analyses of the baseline and EOY results by gender, and by university attended were also conducted (Table 5). No gender differences were observed at baseline for any of the clinical competencies. However, four items did show significant gender differences in the EOY results. These were: Perform anterior rhinoscopy (p = .031), Administer eye drops (p = .019), Catheterise the female bladder (p = .042) and Perform speculum examination of the vagina and cervix (p = .002). Males rated themselves more competent in the first two items (although low overall), and females rated themselves more competent with the latter two items.

Table 5. Baseline and EOY self-rated competence level by gender and university attended

With regards to university attended prior, statistical significance was shown for four items at baseline. These were: Take blood cultures (p = .022), Perform intravenous cannelation (p = .005), Perform venepuncture (p = .042) and Scrub up, gown and glove (p = .048). In all four cases, the University of Otago graduates rated themselves more competent than their University of Auckland counterparts. By EOY, the difference between the university groups for these four items were non-significant (p > .05). However, three of the nine Otago graduates did not complete the EOY survey and therefore these results should be interpreted with caution.

IV. DISCUSSION

The terms “clinical confidence” and “competence” were employed cautiously in this study recognising that confidence was not necessarily a marker for competence and that only standardised assessment could verify actual competence (Stewart et al., 2000).

When comparing the two surveys, three trends emerged across the grouped categories. These were areas where clinical confidence:
1.  was high at both points i.e. pre- and post-PGY-1 (e.g., cardiopulmonary, diagnostic and surgical),
2.  was not high at baseline, but showed significant improvement by year-end (e.g. intravenous/intravascular) and,
3.  remained low at both baseline and EOY (e.g., ENT, ophthalmic).

Our results indicate that PGY-1 doctors may not be receiving adequate training in the list of procedural skills during the PGY-1 year and it would be imperative that clinical supervisors continue to focus on this attainment during the PGY-2 year. The study showed that they rated their inability to perform 48% of the clinical skills at a level of independence in most settings.

PGY-1 confidence in performing ENT and ophthalmic procedures remained low (<2) throughout the year. This suggested that the undergraduate programme was not adequately addressing the learning of these procedural skills, nor were they having the opportunities during the PGY-1 year to improve their skills in these areas. In contrast, while the students were poorly confident about their intravenous/intravascular skills at baseline, these skills improved during the PGY-1 year to a level of being capable of performing them independently.

Of concern is the drop in clinical confidence in performing speculum examination of the vagina and cervix. While developing a clinical skill is important, maintenance of that skill is equally important during the prevocational years. Further analysis of this item by gender found that PGY-1 males’ clinical confidence dropped from 2.55 to 1.90, whereas females’ confidence levels increased from 3.00 to 3.21. Connick, Connick, Klotsas, Tsagkaraki, and Gkrania-Klotsas (2009) identified procedural confidence as dependent on gender just as it was on being offered the opportunity for gaining experience. This item, and female bladder catheterisation, was rated significantly lower by males at EOY, which may suggest a lack of confidence with gender-specific procedures.

Some differences were found between the two main medical school graduates with respect to four items at the beginning of the PGY-1 year. The subsequent EOY survey indicated that these differences had vanished by year-end. However, it should be noted that a third of the Otago graduates, did not complete the EOY survey, so these results need to be interpreted with caution. It is also difficult to generalise our findings to the wider medical school graduate population given that approximately 150-180 students graduate from Auckland, and 210-230 students from Otago each year. A nationwide study of this sort would however provide insight into both whether the differences we observed are part of a national trend, and whether these differences have tapered off by the end of the PGY-1 year. Such information would provide useful feedback for the institutions involved.

The study did not address the association of clinical confidence in performing clinical skills and the types of clinical attachments completed during the PGY-1 year and whether these influenced the final results. This study also did not measure clinical confidence after the PGY-2 years. A proportion of the PGY-2 doctors are likely to complete clinical attachments in ENT, Ophthalmology and the Emergency Department which may allow for experience in competencies that scored low at the start and end of the PGY-1 year. Given the relatively small size of these departments, it would be unlikely that many PGY-2 doctors will rotate through these departments and therefore experience in performing these procedures would remain low. It is therefore vital that College training programmes that require the competent performance of these procedural skills ensure that vocational trainees receive adequate training (e.g. with the Royal New Zealand College of General Practice (RNZCGP)). A longitudinal-based study, similar in design to the current study could measure changes in clinical confidence at not only the beginning and end of PGY-1; but also at further time points (e.g., at the end of the PGY-2 year and the end of the first year of registrar training). This would provide valuable feedback for the above training colleges.

V. CONCLUSION

The skills survey conducted was designed as a self-assessment tool of how competent PGY-1s felt they were in regard to specific clinical skills and procedures. These procedures are outlined in the NZCF as core procedures and interventions that PGY-1s should be able to perform at the end of the PGY-1 year, while “…recognising the limits of their personal capabilities” (MCNZ, 2014). Our findings show that while this benchmark has been achieved in some fields, there are other areas lacking, which may be due to the lack of exposure in certain specialties in the PGY-1 year. Our concern is that competence in these procedures will remain low through the PGY-2 year and possibly as far as vocational training level, once again due to little practical involvement. This paper, and future longitudinal and / or nation-wide studies may therefore serve to inform current undergraduate curriculum planning at the medical school level, as well as provide feedback to the New Zealand Medical Council on the current level of PGY-1 confidence in the core clinical skills and procedures identified by the NZCF.

Notes on Contributors

Dr Wayne de Beer works as a specialist in Consultation-Liaison psychiatry and work part-time as a Clinical Training Director at the Clinical Education & Training Unit; Waikato Hospital, Hamilton, New Zealand. His focus is largely on the prevocational and vocational medical training periods. Publications have included medical education and psychiatry.

Ms Helen Clark is the Medical Education Officer based in the Clinical Education & Training Unit at Waikato Hospital, Hamilton, New Zealand. Her background includes research and statistical analysis in the fields of medical education and psychology. She has academic publications in both fields.

Ethical Approval

A determination of the need for formal ethical approval was sought from the New Zealand Health and Disability Ethics Committee (HDEC). The study was deemed by HDEC to meet the criteria of observational research, therefore did not require formal ethics approval, due to meeting guidelines around health information, human tissue and human participants, as outlined in the HDEC scope summary (Health and Disability Ethics Committee, 2016). The study was registered with Waikato DHB’s internal research committee.

Acknowledgements

We would like to thank Carol Stevenson, Personal Assistant to the Director of Clinical Training, at the Clinical Education and Training Unit at Waikato DHB, for the development and facilitation of the PGY-1 NZCF Procedural skills and communications Competencies Measure survey tool.

Declaration of Interest

The authors declared no competing interest.

References

Benner, P. (1984). From novice to expert: Excellence and power in clinical nursing practice. Menlo Park: Addison-Wesley.

Byrne, A. J., Blagrove, M. T., & McDougall, S. J. (2005). Dynamic confidence during simulated clinical tasks. Postgraduate Medical Journal, 81(962), 785-788.

Connick, R. M., Connick, P., Klotsas, A. E., Tsagkaraki, P. A., & Gkrania-Klotsas, E. (2009). Procedural confidence in hospital based practitioners: implications for the training and practice of doctors at all grades. BMC Medical Education, 9, 2.

Evans, J., Bell, J. L., Sweeney, A. E., Morgan, J. I., & Kelly, H. M. (2010). Confidence in Critical Care Nursing. Nursing Science Quarterly, 23, 334.

Fitzgerald, J. T., White, C. B., & Gruppen, L. D. (2003). A longitudinal study of self-assessment accuracy. Medical Education, 37, 645-649.

Hays, R. B., Jolly, B. C., Caldon, L. J., McCrorie, P., McAvoy, P. A., McManus, I. C., & Rethans, J. J. (2002). Is insight important? Measuring capacity to change performance. Medical Education, 36(10), 965-971.

Health and Disability Ethics Committee. (2016). Applying for review; does your research require HDEC review? Retrieved from http://ethics.health.govt.nz/applying-review.

Medical Council of New Zealand. (2014). New Zealand Curriculum Framework for Prevocational Medical Training. Retrieved from https://www.mcnz.org.nz/assets/News-and-Publications/NZCF26Feb2014.pdf.

Miller, G. E. (1990). The assessment of clinical skills/competence/performance. Academic medicine, 65(9), S63-7.

Patel, M., Oosthuizen, G., Child, S., & Windsor, J. A. (2008). Training effect of skills courses on confidence of junior doctors performing clinical procedures. New Zealand Medical Journal, 121(1275), 37-45.

Stewart, J., O’Halloran, C., Barton, J. R., Singleton, S. J., Harrigan, P., & Spencer, J. (2000). Clarifying the concepts of confidence and competence to produce appropriate self-evaluation measurement scales. Medical Education, 34(11), 903-909.

*Dr Wayne de Beer
Tel: +64 7 8398899 ext 98399
Fax: +64 21 2232549
Email: Wayne.deBeer@waikatodhb.health.nz