Number of Citations: 0

Published online: 3 January, TAPS 2017, 2(1), 16-20
DOI: https://doi.org/10.29060/TAPS.2017-2-1/SC1010

Giat Yeng Khee, Jia Yuan Tay, Sei Keng Koh, Melissa Mee Yin Chow, Wan Chee Ong and Paik Shia Lim

Department of Pharmacy, Singapore General Hospital, Singapore

Abstract

Aim: To assess trainee’s knowledge and practice behavior changes with the revised basic MMS training program.

Methods: Kirkpatrick’s four-level evaluation model was adopted in this study. A questionnaire was used to seek trainees’ feedback on the training program. For learning progression, trainees’ self- perception of their competency pre- and post-training, and results from a multiple-choice questions (MCQ) test were assessed.

Thirty-eight behavioral changes in the patient care delivery and problem solving competency cluster of the General Level Competency Framework (GLF) were compared pre- and 6 months post-training. Evaluation of level-4 results is pending as advanced training is in progress.

Results: Fourteen out of fifteen trainees who had completed the basic MMS training responded to the online questionnaire. Eleven trainees (78.6%) rated basic MMS training program as good or excellent. Positive response was highest in tutorials (84.3%), followed by continuous education sessions (70.6%) and clinic attachments (67.6%). Suggestions for improvement include increasing the frequency and opportunities of hands-on patient interview at clinic attachments.

Sixty-four percent of trainees perceived improvement (p<0.013) in delivering patient care post-training (median=3; IQR: 3-4) than pre-training (median=3; IQR: 2.75-3). All but one trainee passed the MCQ test (median score: 80; IQR: 76-86). A greater proportion of patient care delivery and problem solving behaviors (69.2% and 35.9% respectively) in GLF was graded highest performance at post-training compared to pre-training (35.3% and 14.3% respectively).

Conclusions: The revised training program is well-received by trainees and feedback from trainees provides insights into future improvements in the content and delivery of the program.

Keywords:        Medication Management, Pharmacist, Training, Kirkpatrick

I. INTRODUCTION

Pharmacist-provided Medication Management Service (MMS) aims to improve patients’ outcomes by promoting effective and safe medication use. Essential knowledge and skills in chronic disease management are required in providing good quality pharmaceutical care. An in-house 6-month MMS training program was first developed in January 2013 to impart essential skills of a competent MMS pharmacist. It focused primarily on self-directed learning, using reading materials provided by a sole trainer, amidst the heavy daily workload of trainees. Successful completion of the original training program therefore relied greatly on the trainees’ motivation and commitment. Consequently, completion rate within 6 months as a trained MMS pharmacist was unsatisfactory. Furthermore, the growth of other clinical pharmacy services was affected as the completion of MMS training was a pre-requisite before embarking on other specialized services.

As detailed in Diagram 1, a revised training program which adopted outcome-based education approach was implemented in April 2014. It was divided into basic and advanced phase, which each lasts for 6 months. The basic MMS training program aimed to facilitate learning and enhance competency of pharmacists in medication review and dispensing, while the advanced program aimed to ensure broad- based competency and clinical skills of pharmacists before embarking on pharmacist-provided ambulatory care services. In the revised program, there were significantly more engagement between the trainers and trainees in facilitating trainees’ learning. Prior to this study, a systematic evaluation of the MMS training program has not been conducted. As a first step in evaluating the effectiveness of the revised MMS training program, in this study, we aimed to assess trainees’ knowledge and practice behavior changes following completion of basic MMS training and to identify areas for improvement.

*Topics discussed during tutorials include cardiology, endocrinology, geriatric & nephrology
**Basic MMS training is compulsory for all outpatient pharmacists in SGH whereas the advanced MMS training is optional.
Diagram 1. Original and revised MMS training program in Singapore General Hospital

II. METHODS

In this study, the Kirkpatrick’s four-level evaluation model, developed for measuring training effectiveness, was utilized (Kirkpatrick, 1998). This systematic approach offers multiple levels of evaluation using different data sources and it allows evaluation of both qualitative and quantitative data. All pharmacists who had received basic MMS training in year 2014 were recruited to participate in the study. No ethic approval required based on Singapore Health Services centralized institutional review board. All four- level outcomes were measured as follows:

A. Kirkpatrick level I: Reaction

An online questionnaire, developed by a group of MMS trainers and trainees who were not recruited in the study, was administered to trainee post-basic MMS training. Trainees rated the effectiveness of the basic MMS training program including the various training modalities (tutorials, continuous education sessions and clinic attachments) on a 5-point Likert-scale and provided suggestions for improvement.

B. Kirkpatrick level II: Learning

The same online questionnaire was used to examine the trainees’ perception on the impact of the MMS training program, either positively or negatively affecting their competency in delivering patient care. Results of the multiple-choice questions (MCQ) test administered 6-month post training were examined. The MCQ test that consisted of 50 recall and case-based questions was developed by two MMS trainers, who were the content experts, so as to assess trainees’ clinical knowledge and reasoning. Questions were scrutinised to ensure that they represented an accurate measure of the learning outcomes.

C. Kirkpatrick level III: Behaviour

Changes in observed practice behaviours were assessed using the United Kingdom-adapted General Level Competency Framework (GLF). This is an assessment and developmental tool consisting of 63 behavioural statements distributed into 3 competency clusters: delivery of patient care, problem solving, and professionalism (Rutter et al., 2012). Thirty-eight behavior statements, representing the areas where basic MMS training could have a major influence on, were measured pre- and post-basic MMS training. The performance-level rating for each behaviour (1=rarely; 2=sometimes; 3=usually; 4=consistently) was based on the consistency in demonstration in practice, which was rated according to the assessor’s observation.

D. Kirkpatrick level IV: Results

Currently, the advanced MMS training is still in progress. In advanced MMS training, 15 case reviews by the trainees will be assessed. Trainee with satisfactory achievements in both the final MCQ test and 15 case reviews will be deemed eligible as trained MMS pharmacist.

E. Data Analysis

Descriptive statistics were used to summarize the trainees’ characteristics, reactions to basic MMS training programme and learning outcomes. Behavioural changes pre- and post-basic MMS training were assessed and compared using Wilcoxon signed rank test. SPSS, version 21 (SPSS Inc., an IBM Company Chicago, USA) was used for statistical analysis and a p-value of <0.05 was defined to be statistically significant.

III. RESULTS

Fourteen out of fifteen trainees who had completed the basic MMS training participated in this study. Half of the trainees completed their pharmacy undergraduate program in Singapore while the rest completed their studies overseas.

A. Kirkpatrick level I: Reaction

Fourteen trainees responded to the online questionnaire. Eleven trainees (78.6%) rated basic MMS training program as good or excellent. Positive response was highest in tutorials (84.3%), followed by continuous education (CE) sessions (70.6%) and clinic attachments (67.6%). Suggestions for improvement include increasing the frequency and opportunities of hands-on patient interview at clinical attachments.

B. Kirkpatrick level II: Learning

Nine trainees (64.0%) perceived significant improvement in delivering patient care post-training (median=3; IQR: 3-4) as compared to pre-training (median=3; IQR: 2.75-3) (p<0.013). As for the MCQ test conducted post-training, all but one trainee passed the MCQ test, achieving a median score of 80 (IQR: 76-86).

C. Kirkpatrick level III: Behaviour

Thirteen trainees had completed pre- and post-training GLF assessment for observed practice behaviours. Overall, 23 out of the 38 behaviors (60.5%) were found to have significant improvement at repeated observation post-training. Greater proportion of behaviors in delivering patient care cluster was graded   to perform consistently at highest performance level post-training compared to pre-training (69.2% vs 35.3%). Similar improvement was observed in the problem solving cluster where 14.3% and 35.9% of the behaviors were graded consistent performance at pre- and post-training respectively. There was no significant improvement observed in 3 behaviors as the trainees were practicing at the highest level during the pre-training evaluation. Performance level for other behaviors was maintained over the 6- month training.

IV. DISCUSSION

Assessment of the training program provides valuable information to trainers and its institution in seeking ways to improve the training program. The revised basic MMS training program has likely achieved its aim to provide broad-based training that enhances trainee’s competency in patient care as positive results were observed in trainee’s attitude, knowledge and performance (level 2 and 3). Feedback (level 1) from the trainees was encouraging and indicated that the revised training program with the inclusion of tutorials, CE sessions and clinic attachments, was well-received and trainees had perceived improvement in their competency at providing patient care. Engagement between trainers and trainees in our revised program has been increased, as compared to the original program. This increased engagement between the trainers and trainees may have enhanced the trainees’ learning experiences through regular feedback from the trainers and reflection of their learning process, thus resulting in better performance post-training. From the high rate of positive response from trainees, it demonstrated that such interaction with trainers was in favor to support learning.

Clinic attachment was ranked lowest amongst all training activities as trainees commented that the opportunities for clinic attachments were sparse, with 3 clinic attachments distributed over 6 months. However, increasing the frequency of clinic attachments is not feasible in view of the large pool of trainees and limited training resources. Proposed plans for enhancement of the program include restructuring tutorial or CE sessions to problem-based learning. Simulation cases could be used to facilitate the application of knowledge and prepare trainees better for hands-on experiences during clinic attachments.

There were limitations to our study. First, positive changes observed in attitude, knowledge and performance may not be solely contributed by the MMS training program due to the lack of a control group. Improvement in behaviors observed in GLF assessment could be contributed by other sources of training such as the 5-weekly outpatient clinical group discussion, feedback or suggestions from their GLF assessor, self-directed learning and lecture series from other continuing professional education provided by external parties. Second, pre-training MCQ test was not performed. Therefore, the extent of improvement in trainees’ knowledge level was not established.

V. CONCLUSION

Our assessment shows that the revised basic MMS training program is well-received by trainees and feedback from trainees provides insights into future improvements in the content and delivery of the program. The results of the advanced MMS training are pending, therefore, the effectiveness of the revised MMS program is yet to be evaluated.

Notes on Contributors

Khee Giat Yeng is a board certified pharmacotherapy specialist at Singapore General Hospital where she is involved in the provision of ambulatory care services in the medication management service, rheumatology monitoring clinic and anticoagulation clinic. She is also actively involved in teaching medical students, pre-registration pharmacists, pharmacists and post-graduate students.

Tay Jia Yuan is a pharmacist in Singapore General Hospital and is currently participating in the advanced training program for the medication management service in SGH.

Koh Sei Keng is involved in the provision of anticoagulation services at the ambulatory care clinic in the Singapore General Hospital.

Melissa Mee Yin Chow a senior pharmacist at Singapore General Hospital with a board certification in pharmacotherapy. She is involved with training of pre-registration pharmacists and registered pharmacists. She also provides ambulatory care services in medication management, anti-coagulation and haemodialysis disease management clinics.

Ong Wan Chee is a senior clinical pharmacist at Singapore General Hospital. She is actively involved in the provision of ambulatory care services including medication management service, patient outreach program, inflammatory bowel disease, and rheumatology monitoring clinic. She also teaches the medical students, pharmacists, pre-registration pharmacists and postgraduate students.

Lim Paik Shia is a principal clinical pharmacist, practicing in ambulatory care clinics at Singapore General Hospital. She is actively driving the pharmacist-provided clinics, in particularly medication management service to outpatients and hemodialysis patients. She is also a clinical educator for pharmacists, residents, postgraduate students and pre-registration pharmacists.

Acknowledgements

The authors would like to thank all MMS trainers for their contributions in this study. We would like to acknowledge the guidance provided by Mr Chang Wei Terk who had pioneered the original MMS training program and his instrumental role in the revision of the program.

Declaration of Interest

The authors declare no conflict of interest or any sources of funding for this study.

References

Kirkpatrick, D. L. (1998). Evaluating training programs: The four levels (2nd ed.). San Francisco, CA: Berrett-Koehler.

Rutter, V., Wong, C., Coombes, I., Cardiff, L., Duggan, C., Yee, M. L., Lim, K. W., & Bates, I. (2012). Use of a general level framework to facilitate performance improvement in hospital pharmacists in Singapore. American Journal of Pharmaceutical Education, 76(6), 107.