According to the analysis conducted in the exploration stage, stakeholders expected the availability of various features in the mobile application-based e-portfolio. However, it was not possible to accommodate all of the identified features at the implementation stage. The authors identified the key features in order to develop a generic application, taking into consideration the findings related to the current implementation of a workplace-based assessment, portfolios and logbooks, supervision, level of entrustment, and other technical issues. Also considered were the identified basic needs of the teaching hospital, study programmes, faculty members, and residents, along with best practice principles related to application development.

Some of the key features derived from the analysis included in the implementation stage were the following:

1) Supervisors and administrators are allowed to track residents’ performance and achievements.

2) A drop-down menu is used in most form segments for easier and user-friendly data input.

3) Feedback from the clinical supervisor is recorded on each data input.

4) Clinical cases, supervision level, and type of clinical authorities are set according to the residential programme and the resident’s educational stage.

5) Guided self-reflection questions for the resident are mandatory on each data input.

6) Attachment of supporting image as evidence is facilitated for each data input.

The application was divided into two platforms: for residents and supervisors. The menus in the residents’ application included activity list, activity input, report, and profile, among others. While submitting new activity into the platform, residents must fill in the information related to each clinical activity, describe the activity according to the given guided reflection-on-action features, and provide evidence form of photographs (if applicable). The supervisor application had only one main feature for reviewing and providing feedback on activities previously submitted by residents. The figure of the application and its overall process of the application is shown in the appendix.

B. Pilot Implementation Stage

During the four-week-long pilot programme, the residents were encouraged to submit as much evidence as possible related to patient care to the e-portfolio platform. The evidence was automatically sent to assigned faculty members’ platform in order to get feedback. Observations were conducted by evaluating the data input from the application’s backend to obtain data regarding the utilisation frequency and types of cases input in the mobile application in each residency programme. A total of 311 activities were submitted into the application during the four-week-long pilot programme; however, only 98 (31.5%) activities received feedback from clinical supervisors. Table 4 describes the results of the average word count analysis of the residents’ reflection-on-action and the feedback received in the mobile application e-portfolio during the implementation stage.

Table 4. Activity Report of the Pilot Programme: Recorded Activity, Residents’ Reflections and Feedback from Supervisors

1: Psychiatry, 2: Internal Medicine, 3: Obstetrics & Gynaecology, 4: Anaesthesiology & Intensive Care, 5: Oncology Radiation, 6: Radiology; DD: description of difficulties

LL: lessons learned, AP: action plan, D: descriptive feedback, C: constructive feedback, N: neither descriptive nor constructive feedback, AWC: average word count

The feedback received covered a wide range of competencies: clinical skills, knowledge, and patient care. The quotations below are completed with codes explaining the details as follows: a. FB which stands for Feedback; b. Number (1-6) which refers to the study programme 1 to 6; and c. Last number (e.g 9, 15) which refers to the list number of feedback comment in the system.

“Spinal USG exercise on new-born infants to improve skills.”

(FB, (5), 9) 

“Learn about the differences in fracture due to trauma, metastasis, and infection.”

(FB, (5), 13) 

Although most feedback obtained from supervisors was classified as descriptive or “neither descriptive nor constructive”, there was some constructive feedback, describing not only what to do but also how to do it and improve the residents’ performance.

“Volume target determination was excellently and efficiently carried out. Effective and efficient communication could be achieved even by telephone. There were few problems in terms of work efficiency because no confirmation was made about which work had and/or had not been done by the supervisor. However, the overall process and results were excellent.”

(FB, (4), 9) 

The users’ evaluations on the mobile application-based e-portfolio were obtained using a modified version of the SUS questionnaire at the end of the pilot programme. The pilot study participants included 45 faculty members and 66 residents. Out of a total of 111 questionnaires distributed, a total of 92 questionnaires were received back: 37 faculty members’ response (84%) and 55 residents’ response (82%). The results showed that the faculty members and residents had similar views regarding most aspects evaluated by the questionnaire. Comparisons between the proportions were conducted using a chi-square test—which showed significant differences between faculty members’ and residents’ perceptions on seven items as shown in Figure 1. The complete result of the SUS questionnaire is provided in Appendix 2.

Figure 1. Comparison among supervisors and residents satisfaction on the e-portfolio application

The results of the thematic analysis of the active commentaries in the questionnaire revealed four main themes:

1) Participants believed that the application was user-friendly but needed improvement in some features and interfaces.

“The display can be improved so the most frequent treatment can be put on top. Thus no repeated scrolling down is needed.”

Commentaries- resident-1 (52)

2) Some constraints identified were: the length and number of forms to be filled in the mobile application, the lack of notifications, case grouping according to clinical rotation, as well as the needs for integration with electronic health records and other established applications in each study programme.

“…filling the apps is burdening residents with a bunch of caseload. The interface has to be simplified, with the option to pick certain cases without deeply analysing each case (e.g. there were 50 cases today yet only 2 unique/interesting cases which required special attention, etc.).”

Commentaries-resident-6 (46)

3) Both faculty members and residents similarly noted utilisation of the application to facilitate supervision.

“All supervising needs have been accommodated.”

 Commentaries-faculty-1(91)

“Supervisor’s verification system was a good thing.”

Commentaries-residents-3 (8)

4) Lastly, the participants felt that the availability of the feedback feature was positive, despite some identified constraints in giving feedback.

“The most important feature of this, I think is the feedback giving and follow-up plan.”

Commentaries-faculty-5(63)

IV. DISCUSSION

This study highlights the development of an app-based e-portfolio in the teaching hospital of a resource-limited setting. The study was able to identify key features of the e-portfolio based on the stakeholders’ (i.e programme coordinators, supervisors, and residents). Several challenges which should be addressed for future improvement were also identified in the pilot study.

The first stage of the study was able to explore the perceptions of users which informed the e-portfolio. The users identified the need of a portfolio (Crossley et al., 2011) for documenting residents’ achievements over time and for assessing coverage of clinical case management. The use of a portfolio in digital form in this study also aimed to facilitate the process of supervising residents working in the teaching hospital. The supervision system in the current teaching hospital adopted the EPA concept (Carraccio et al., 2017; Ten Cate et al., 2015), which had been in use for the past 2 years in the six study programmes.

The use of an e-portfolio is not particularly new in postgraduate training (Kjaer et al., 2006), but the particular use for facilitating supervision of residents has, to the best of the authors’ knowledge, rarely been reported (van der Schaaf et al., 2017). Reported challenges of e-portfolio are the residents’ burden despite the use of an electronic platform (Birks et al., 2016; Vance et al., 2017), the reliability of a workplace-based assessment portfolio (Castanelli et al., 2019), and the fact that time issues in clinical practice should always be taken into consideration (Binhimd et al., 2017). Heeneman and Driessen hence clarify that it is necessary to determine whether portfolio development by residents is voluntary or compulsory (Heeneman & Driessen, 2017). Based on the framework, the current e-portfolio can be categorised as a combination of a training and personal development portfolio, which comprises mandated required skills and competencies in a fixed format as well as personal reflections of progress (Heeneman & Driessen, 2017). In addition, the use of mobile application was expected to support the use of e-portfolio for residents’ learning in a busy clinical setting (Mok et al., 2019).

An important framework that was adopted in the e-portfolio platform in this study is the EPA. EPA development calls for the consideration of competencies, clinical settings, the entrustment process and stakeholders’ agreements, including those of staff/supervisors and experts (Ten Cate et al., 2015), residents, the teaching hospital and even patients, if applicable (Lundsgaard et al., 2019).

The authors of the present study identified the potential value of a portfolio as a tool to provide ad-hoc supervisor decision-making, based on the study by van der Schaaf et al. (2017). Based on the reflections in the six study programmes, the e-portfolio incorporated residents’ performance, supervisors’ feedback, and evidence of achievement that is in line with the expected level of competencies or EPAs in the curriculum. The evidence is required by supervisors for deciding on a level of entrustment. A study involving experienced obstetrics and gynaecology supervisors in the Netherlands suggests that both formal and informal entrustment processes are used in combination on a daily basis. Informal entrustment is thought to provide more flexibility because it can be adjusted more easily than a formal one (Castanelli et al., 2019).

Based on the analysis in the exploration stage of the present study, the platform was designed to accommodate entries by a resident which were then responded to by a supervisor directly in the clinical setting or indirectly through the application (with the supervisor’s presence according to the EPA and supervision level for particular case managements/procedures). According to van der Schaaf et al. (2017), the current e-portfolio adapted both a student model that provided achievement monitoring and a feedback model that provided personalised feedback. The data visualisation in the current study was not yet used for the supervisors’ entrustment process and is a subject for further development of the application and back-end data analysis.

The e-portfolio application in this study was designed to include a feedback model, which was initiated by reflection by the resident. As described in Table 5, despite the number of recorded activities, only 12.5–45% of them received written feedback from the supervisors. Of these, most of the feedback was descriptive or constructive. Before providing or documenting feedback in the application, the supervisor may verify the case or do so after observing the resident performing the procedure.

Apart from the residents of the Radiology programme, almost all residents documented their reflections on the encountered clinical cases: the difficulties, lessons learned, and action plans (Table 3). The residents’ reflections were considered as one of the key successes of the portfolio implementation (Binhimd et al., 2017), and it is also a best practice in feedback provision in clinical training (Bounds et al., 2013; Kornegay et al, 2017; Ramani & Krackov, 2012). The residents in the pilot study displayed a good habit of feedback-seeking behaviour by initiating their reflections. Given the burden of daily routine in the teaching hospitals, the quality of the written reflections completed by residents was of concern in this study. Further consideration to moderate the frequency of reflections should be incorporated hence the residents can practice more in-depth reflections which are necessary for more meaningful feedback and future actions for improvement.

All reflections and feedback were given in less than 50 words; hence, it is again considered feasible for the users to conduct brief reflection and feedback within the application. The residents’ feedback showed that it was one of the most important features of the application, which supported their learning. Despite this, a rather low amount of feedback with low word counts was given to the residents during the pilot study, raising concerns that despite the feedback training given to supervisors, the use of the application did not necessarily increase the amount and quality of documented feedback. As suggested by Brehaut et al. (2016), the format of feedback delivery matters, and it is preferable to avoid the cognitive overload of the recipients in the use of an electronic platform, as attempted in the current application. The limited amount of feedback documented during the pilot study, however, did not reflect the actual practice of giving feedback in the current setting. The documentation of feedback was expected to facilitate the long-term monitoring of residents’ progress (van der Schaaf et al., 2017), which may prompt supervisors’ summative entrustment decisions. The residents’ feedback in this study also highlighted the need for deeper reflection on the voluntary selection of clinical activities, so that the resident did not have to input all encountered cases. This might increase the user-friendliness of the application, yet it might require further agreement with the supervisors and the study programme coordinators.

The evaluation of the supervisors and residents in the pilot study showed that supervisors perceived more positively the use of the apps, compared to the residents. From the supervisors’ point of views, accessibility of online data on resident’s performance was responded to very positively given current challenge in tracking this in manual documentation. From the residents’ point of views, this might be due to that despite the potential of the current application to enable clinical supervision, submitting data into the application felt like an addition to the residents’ workload, due to redundancy in submitting information into both e-portfolio platform and the e-hospital medical records. The workload of residents in teaching hospitals has been a controversial issue in postgraduate medical education (Nishigori et al., 2015). The stakeholders should agree on which data are compulsory and critical for evaluation and which are voluntary in the e-portfolio, both for training and patient care purposes. In addition, the authors also realised that training on portfolio development and supervision should be supported with a longitudinal mentoring process and an analysis of longitudinal performance (Heeneman & Driessen, 2017).

A. Study Limitations

The study has some limitations. The study was conducted at one teaching hospital and involved a limited number of study programmes. Given that Cipto Mangunkusumo is one of the busiest national referral hospitals, with complex cases and high workloads, the perspectives of key stakeholders explored in this study were critical for consideration in the development of an e-portfolio mobile application. In addition, the study was able to identify the features of an e-portfolio that can potentially support the supervision and entrustment processes. Future versions of the e-portfolio application aimed at enhancing the quality of supervision should consider the analysis of the residents’ longitudinal performance through prompt data analysis in the system, thereby supporting formative and summative entrustments during clinical supervision as well as the integration of an adequate mentoring system.

V. CONCLUSION

The present study explored the stakeholders’ need of an e-portfolio for supervision that enables all parties to monitor learning progress and competency achievement. On top of the implementation of workplace-based assessments and EPA systems in the respective study programmes, the identified key features of an e-portfolio mobile application reflect the needs of residents, faculty/supervisors, the teaching hospital, and the school of medicine.

The pilot implementation showed that the e-portfolio mobile application was feasible and potential for use by residents, supervisors and programme coordinators in monitoring EPAs and competence achievement in the current setting, with highlighted challenges that need to be tackled in the future. The platform’s use in assisting entrustment decisions should be further confirmed with a longitudinal analysis of the residents’ performance and the use of the analysis by the supervisors. The development of such an e-portfolio to support EPA in other settings should consider the dynamics of case complexity, residents’ level of competence and entrustment, workload, the healthcare system, and the education system. Finally, in a resource-limited setting, the involvement of stakeholders from the start to prioritise the e-portfolio features, user-friendliness, and technology feasibility should always be considered.

Notes on Contributors

Ardi Findyartini reviewed the literature, designed the study, conducted interviews and focus group discussions, analysed transcripts and led the manuscript writing.

Natalia Widiasih reviewed the literature, designed the study, conducted interviews and focus group discussions,  analysed transcripts and wrote the manuscript.

Nadia Greviana conducted interviews and focus group discussions, analysed transcripts and wrote the manuscript.

Gregorius Ben Prayogi conducted interviews and focus group discussions,  analysed transcripts and gave critical feedback in the manuscript writing.

Daniar Setyorini coordinated the e-portfolio development and data collection, analysed transcripts and gave critical feedback in the manuscript writing.

All authors have read and approved the final manuscript.

Ethical Approval

Ethical approval for this study was granted by the Faculty of Medicine Universitas Indonesia – Cipto Mangunkusumo Hospital Research Ethical Review Board (2018) Number: 328/UN2.F1/ETIK/III/2018.

Data Availability

Data will be available upon request to corresponding author due to conditions of consent provided by respondents in this study and that it should abide data sharing policy from the medical school and teaching hospital where the study was completed.

Acknowledgement

The authors would like to thank CMGH, FMUI and IMERI for the organisational supports. The authors would also like to extend gratitude to all six participating study programmes, the programme directors, the residents, and the supervisors. Finally, the authors would like to thank Dr. Sri Puspita, who facilitated data collection for the study.

Funding

This study was funded by Dr. Cipto Mangunkusumo General Hospital Operational Grant 2018.

Declaration of Interest

All authors state no possible conflicts of interest, including financial, consultant, institutional and other relationships that might lead to bias or a conflict of interest.

References

Binhimd, B., Al-Jifrfee, H. M. A., & Khan, M. A. (2017). Identifying the perceived challenges that affect the use of portfolio as an assessment tool for family medicine residents in Jeddah program. MedEdPublish, 6(2), 1–12. https://doi.org/10.15694/mep.2017.000070

Birks, M., Hartin, P., Woods, C., Emmanuel, E., & Hitchins, M. (2016). Students’ perceptions of the use of eportfolios in nursing and midwifery education. Nurse Education in Practice, 18, 46–51. https://doi.org/10.1016/j.nepr.2016.03.003

Bounds, R., Bush, C., Aghera, A., Rodriguez, N., Stansfield, R. B., Santen, S.A., & MERC at CORD Feedback Study Group (2013). Emergency medicine residents’ self-assessments play a critical role when receiving feedback. Academic Emergency Medicine, 20(10), 1055–1061. https://doi.org/10.1111/acem.12231

Bramley, A. L., Thomas, C. J., & Kenna, L. M. (2020). E-portfolios and entrustable professional activities to support competency-based education in dietetic education. Nursing Health Science, 1-9. https://doi.org/10.1111/nhs.12774

Brehaut, J., Colquhoun, H., Eva, K., Carrol, K., Sales, A., Michie, S., Ivers, N., & Grimshaw, J. M. (2016). Practice feedback interventions: 15 suggestions for optimizing effectiveness. Annals of Internal Medicine, 164(6), 435–461. https://doi.org/10.7326/M15-2248

Brooke, J. (1986). SUS- A quick and dirty usability scale. Digital Equipment Corporation. https://hell.meiert.org/core/pdf/sus.pdf Accessed October 30, 2018

Carraccio, C., Englander, R., Gilhooly, J., Mink, R., Hofkosh, D., Barone, M. A., & Holmboe, E. S. (2017). Building a framework of entrustable professional activities, supported by competencies and milestones, to bridge the educational continuum. Academic Medicine, 92(3), 324–330. https://doi.org/10.1097/ACM.000000000000114

Castanelli, D. J., Moonen-van Loon, J. M. W., Jolly,  B., & Weller, J. M. (2019). The reliability of a portfolio of workplace-based assessments in anesthesia training (Fiabilité d’un portfolio d’évaluations sur le lieu de travail dans la formation en anesthésie). Canadian Journal of Anesthesia Can d’anesthésie [Internet], 66(2), 193–200. https://doi.org/10.1007/s12630-018-1251-7

Cianciolo, A. T., & Kegg, J. A. (2013). Behavioral specification of the entrustment process. Journal of Graduate Medical Education, 5(1), 10–12. https://doi.org/10.4300/JGME-D-12-00158.1

Crossley, J., Johnson, G., Booth, J., & Wade, W. (2011). Good questions, good answers: Construct alignment improves the performance of workplace-based assessment scales. Medical Education, 45(6), 560–569. https://doi.org/10.1111/j.1362-2923.2010.03913.x

Deketelaere, A., Degryse, J., De Munter, A., & De Leyn, P. (2009). Twelve tips for successful e-tutoring using electronic portfolios. Medical Teacher, 31(6), 497–501. https://doi.org/10.1080/01421590802572734

Farnan, J. M., Petty, L. A., Georgitis, E., Martin, S., Chiu, E., Prochaska, M., & Arora, V. M. (2012). A systematic review: The effect of clinical supervision on patient and residency education outcomes. Academic Medicine, 87(4), 428–442. http://doi.org/10.1097/ACM.0b013e31824822cc

Hauer, K., Kolwes, J., Cornett, P., Hollander, H., ten Cate, O., Ranji, S. R., Soni, K., Iobst, W., O’Sullivan, P. (2013). Identifying entrustable professional activities in internal medicine training. Journal of Graduate Medical Education, 5(1), 54–59. https://doi.org/10.4300/JGME-D-12-00060.1

Heeneman, S., & Driessen, E. W. (2017). The use of a portfolio in postgraduate medical education – Reflect, assess and account, one for each or all in one? GMS Journal for Medical Education, 34(5). https://doi.org/10.3205/zma001134

Holmboe, E., Ginsburg, S., & Bernabeo, E. (2011). The rotational approach to medical education: time to confront our assumptions? Medical Education, 45(6), 69–80. https://doi.org/10.1111/j.1365-2923.2010.03847.x

Kjaer, N. K., Maagaard, R., & Wied, S. (2006). Using an online portfolio in postgraduate training. Medical Teacher, 28(8), 708–712. https://doi.org/10.1080/01421590601047672

Kornegay, J. G., Kraut, A., Manthey, D., Omron, R., Caretta-Weyer, H., Kuhn, G., Martin, S., & Yarris, L. M. (2017). Feedback in medical education: A critical appraisal. AEM Education and Training, 1(2), 98–109. https://doi.org/10.1002/aet2.10024

Lundsgaard, K. S., Tolsgaard, M. G., Mortensen, O. S., Mylopoulos, M., & Østergaard, D. (2019). Embracing multiple stakeholder perspectives in defining trainee competence. Academic Medicine, 94(6), 838-846. https://doi.org/10.1097/ACM.0000000000002642

Meeus, W., Van Petegem, P., & Van Looy, L. (2006). Portfolio in higher education: Time for a clarificatory framework. International Journal of Teaching and Learning in Higher Education, 17(2), 176–180.

Mok, Z. W., Lee, J. C. S., & Mathur, M. (2019). Harnessing mobile technology to develop a sustainable on the go learning resource for junior doctors. The Asia Pacific Scholar, 4(2), 48-51. https://doi.org/10.29060/TAPS.2019-4-2/SC2070

Mulder, H., Ten Cate, O., Daalder, R., & Berkvens, J. (2010). Building a competency-based workplace curriculum around entrustable professional activities: The case of physician assistant training. Medical Teacher, 32(10), e453-e459. https://doi.org/10.3109/0142159X.2010.513719

Nishigori, H., Deshpande, G. A., Obara, H., Takahashi, O., Busari, J., & Dornan, T. (2015). Beyond work-hour restrictions: A qualitative study of residents’ subjective workload. Perspectives on Medical Education, 4(4), 176–180. https://doi.org/10.1007/s40037-015-0198-4

Ramani, S., & Krackov, S. K. (2012). Twelve tips for giving feedback effectively in the clinical environment. Medical Teacher, 34(10), 787–791. https://doi.org/10.3109/01142159X.2012.684916

Sandhu, D. (2018). Postgraduate medical education – Challenges and innovative solutions. Medical Teacher, 40(6), 607–609. https://doi.org/10.1080/0142159X.2018.1461997

Ten Cate, O. (2006). Trust, competence, and the supervisor’s role in postgraduate training. British Medical Journal, 333(7571), 748–751. https://doi.org/10.1136/bmj.38938.407569.94

Ten Cate, O., Chen, H. C., Hoff, R. G., Peters, H., Bok, H., & van der Schaaf, M. (2015). Curriculum development for the workplace using entrustable professional activities (EPAs): AMEE guide No. 99. Medical Teacher, 37(11), 983–1002. https://doi.org/10.3109/0142159X.2015.1060308

Ten Cate, O., Hart, D., Ankel, F., Busari, J., Englander, R., Glasgow, N., Holmboe, E., Lobst, W., Lovell, E., Snell, L. S., Touchie, C., Van Melle, E., Wycliffe-Jones, K., & International Medical Education Collaborators (2016). Entrustment decision making in clinical training. Academic Medicine, 91(2), 191–198. https://doi.org/10.1097/ACM4.0000000000001044

Tepper, C., Bishop, J., & Forrest, K. (2020). Authentic assessment utilising innovative technology enhanced learning. The Asia Pacific Scholar, 5(1), 70-75. https://doi.org/10.29060/TAPS.2020-5-1/SC2065

Vance, G. H. S., Burford, B., Shapiro, E., & Price, R. (2017). Longitudinal evaluation of a pilot e-portfolio-based supervision programme for final year medical students: Views of students, supervisors and new graduates. BMC Medical Education, 17(1), 1–9. https://doi.org/10.1186/s12909-017-0981-5

van der Schaaf, M., Donkers, J., Slof, B., Moonen-van Loon, J., van Tartjwijk, J., Driessen, E., Badii, A., Serban,  O., & Ten Cate, O. (2017). Improving workplace-based assessment and feedback by an e-portfolio enhanced with learning analytics. Educational Technology Research and Development, 65(2), 359–380. https://doi.org/10.1007/s11423-016-9496-8

van Tartwijk, J., & Driessen, E. W. (2009). Portfolios for assessment and learning: AMEE guide no. 45. Medical Teacher, 31(9), 790–801. https://doi.org/10.1080/01421590903139201

Weller, J. M., Misur, M., Nicolson, S., Morris, J., Ure, S., Crossley, J., & Jolly, B. (2014). Can I leave the theatre? A key to more reliable workplace-based assessment. British Journal of Anaesthesia, 112(6), 1083–1091. https://doi.org/10.1093/bja/aeu052

*Ardi Findyartini
Department of Medical Education & Medical Education Center
Indonesia Medical Education Research Institute (IMERI),
Faculty of Medicine Universitas Indonesia,
Jakarta, Indonesia
10430
Email: ardi.findyartini@ui.ac.id