{"id":70990,"date":"2021-09-13T14:36:14","date_gmt":"2021-09-13T06:36:14","guid":{"rendered":"https:\/\/medicine.nus.edu.sg\/taps\/?post_type=issues&#038;p=70990"},"modified":"2022-01-04T07:44:33","modified_gmt":"2022-01-03T23:44:33","slug":"development-of-an-app-based-e-portfolio-in-postgraduate-medical-education-using-entrustable-professional-activities-epa-framework-challenges-in-a-resource-limited-setting","status":"publish","type":"issues","link":"https:\/\/medicine.nus.edu.sg\/taps\/issues\/development-of-an-app-based-e-portfolio-in-postgraduate-medical-education-using-entrustable-professional-activities-epa-framework-challenges-in-a-resource-limited-setting\/","title":{"rendered":"Development of an app-based e-portfolio in postgraduate medical education using Entrustable Professional Activities (EPA) framework: Challenges in a resource-limited setting"},"content":{"rendered":"<p>Submitted: 6<span lang=\"EN-GB\"> November 2020<\/span><br \/>\r\nAccepted: 27 January 2021<br \/>\r\nPublished online: <span lang=\"EN-GB\">5 October, TAPS 2021, <\/span><span lang=\"EN-GB\">6(4), 92-106<\/span><br \/>\r\n<a href=\"https:\/\/doi.org\/10.29060\/TAPS.2021-6-4\/OA2459\">https:\/\/doi.org\/10.29060\/TAPS.2021-6-4\/OA2459<\/a><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">Ardi Findyartini<sup>1,2<\/sup>, Natalia Widiasih Raharjanti<sup>3<\/sup>, Nadia Greviana<sup>1,2<\/sup><\/span><span lang=\"EN-GB\">, <\/span><span lang=\"EN-GB\">Gregorius Ben Prajogi<sup>4 <\/sup>&amp; Daniar Setyorini<sup>2<\/sup><\/span><span lang=\"EN-GB\"><\/span><\/p>\r\n<p style=\"text-align: justify\"><i><sup><span lang=\"EN-GB\">1<\/span><\/sup><\/i><i><span lang=\"EN-GB\">Department of Medical Education, Faculty of Medicine Universitas Indonesia, Indonesia; <sup>2<\/sup>Medical Education Center, Indonesian Medical Education and Research Institute (IMERI), Faculty of Medicine, Universitas Indonesia, Indonesia; <sup>3<\/sup>Department of Psychiatry, Faculty of Medicine, Universitas Indonesia, Indonesia; <sup>4<\/sup>Department of Oncology Radiation, Faculty of Medicine, Universitas Indonesia, Indonesia<\/span><\/i><\/p>\r\n<p style=\"text-align: center\"><b><span lang=\"EN-GB\">Abstract<\/span><\/b><\/p>\r\n<p style=\"text-align: justify\"><b><span lang=\"EN-AU\">Introduction: <\/span><\/b><span lang=\"EN-ID\">E-portfolio may facilitate the entrustment process from supervisors to residents in postgraduate medical education. The present study was aimed at identifying necessary features of an e-portfolio application and to conduct pilot study in a teaching hospital. <\/span><\/p>\r\n<p style=\"text-align: justify\"><b><span lang=\"EN-AU\">Methods:<\/span><\/b><span lang=\"EN-AU\"> <\/span><span lang=\"EN-ID\">Six programs participated. Eight interviews with education directors and six focus group discussions with residents and supervisors were completed for the needs analysis stage. The application was developed based on the thematic analysis of the needs analysis stage. The mobile-app e-portfolio pilot was conducted for four weeks and a modified version of the <\/span><span lang=\"EN-GB\">System Usability Scale (SUS) was distributed to participants following the pilot program.<\/span><span lang=\"EN-AU\"><\/span><\/p>\r\n<p style=\"text-align: justify\"><b><span lang=\"EN-AU\">Results:<\/span><\/b><span lang=\"EN-AU\"> <\/span><span lang=\"EN-ID\">Key features of the e-portfolio were identified. A total of 45 supervisors and 66 residents participated in the pilot study. The residents utilised the application according to the clinical activities and supervision level, and the information was fed to the supervisors as per the application design. Challenges during the pilot study in terms of feedback provision and residents\u2019 workload which influence the e-portfolio use for entrustment decisions are discussed. <\/span><span lang=\"EN-AU\"><\/span><\/p>\r\n<p style=\"text-align: justify\"><b><span lang=\"EN-AU\">Conclusion:<\/span><\/b><span lang=\"EN-AU\"> <\/span><span lang=\"EN-ID\">Current e-portfolio features were created for supervision and are potential to facilitate the entrustment process in Entrustable Professional Activities (EPA) implementation. The pilot study highlighted challenges of the implementation which should be considered for future improvement. <\/span><\/p>\r\n<p style=\"text-align: justify\"><b><span lang=\"EN-GB\">Keywords:<\/span><\/b><i><span lang=\"EN-GB\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Postgraduate Medical Education, E-Portfolio, Entrustable Professional Activities, Supervision, Needs Analysis, Pilot Study<\/span><span lang=\"EN-GB\"> <\/span><\/i><\/p>\r\n<p align=\"center\"><strong><span lang=\"EN-GB\">Practice Highlights<\/span><\/strong><\/p>\r\n<ul>\r\n\t<li style=\"text-align: justify\"><span lang=\"EN-GB\"> <\/span><span lang=\"EN-GB\">Development of assessment application, including e-portfolio, should consider input from the stakeholders.<\/span><\/li>\r\n\t<li style=\"text-align: justify\"><span lang=\"EN-GB\"> <\/span><span lang=\"EN-GB\">The e-portfolio should further be defined to emphasise the documentation, reflection, and feedback processes. <\/span><\/li>\r\n\t<li style=\"text-align: justify\"><span lang=\"EN-GB\"> <\/span><span lang=\"EN-GB\">Consideration of adequate evidence for EPA should be determined based on the aim of the e-portfolio.<\/span><\/li>\r\n\t<li style=\"text-align: justify\"><span lang=\"EN-GB\"> <\/span><span lang=\"EN-GB\">A robust e-portfolio implementation will potentially support the implementation of EPA.<\/span><\/li>\r\n\t<li style=\"text-align: justify\"><span lang=\"EN-GB\"> <\/span><span lang=\"EN-GB\">Challenges in e-portfolio implementation should be aware of and tackled for future improvement.<\/span><span lang=\"EN-GB\"><\/span><\/li>\r\n<\/ul>\r\n<p style=\"text-align: center\"><strong><span lang=\"EN-GB\">I. INTRODUCTION<\/span><\/strong><\/p>\r\n<ol><\/ol>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">Supervision is a critical component of postgraduate medical education as it allows residents to have a gradual achievement of competencies while still fulfilling patient safety standards. Patient care conducted by residents under adequate supervision can have results comparable to the care provided by more experienced medical doctors (Farnan et al., 2012). To entrust the resident, a supervisor considers several factors, including incidents experienced during supervision, the resident\u2019s characteristics, the results of assessments, and encountered situations<sup> <\/sup>(Cianciolo &amp; Kegg, 2013).<\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">Ten Cate et al. (2015) introduced the Entrustable Professional Activities (EPA) concept to facilitate the implementation of a competency-based curriculum in the workplace (Ten Cate et al., 2015). EPAs are observable and measurable units of work that can correspond to competency milestones, allowing for safe and effective performance (Carraccio et al., 2017).\u00a0 Easily accessible and interpreted information about a resident\u2019s past performance using the EPA concept is, therefore, suggested to facilitate the entrustment process, both for ad-hoc and summative purposes (Hauer et al., 2013; Ten Cate et al., 2015).<sup> <\/sup><\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-ID\">Entrustment decision-making <\/span><span lang=\"EN-GB\">refers to a supervisor\u2019s decision to trust a resident to carry outpatient care \u2018without supervision\u2019 (<\/span><span lang=\"EN-ID\">Crossley et al., 2011; Ten Cate, 2006; Weller et al., 2014<\/span><span lang=\"EN-GB\">).<sup> <\/sup>Given the need for assured patient safety, \u2018without supervision\u2019 refers to the reduction of educational supervision and <\/span><span lang=\"EN-ID\">the provision of <\/span><span lang=\"EN-GB\">\u2018relational autonomy\u2019, whereby interdependence between the resident, the supervisor, the healthcare team, and the healthcare system becomes critical (<\/span><span lang=\"EN-ID\">Holmboe et al., 2011).<\/span><span lang=\"EN-GB\"> However, the amount of data accessible about a resident\u2019s performance should be analysed comprehensively to enable ad-hoc entrustment (<\/span><span lang=\"EN-ID\">Sandhu, 2018).<\/span><span lang=\"EN-ID\"> <\/span><span lang=\"EN-GB\"><\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">Logbooks and portfolios have been considered as workplace-based assessment methods that would enable summarising a resident\u2019s performance as well as maintaining assessment results during training (van Tartwijk &amp; Driessen, 2009). Electronic portfolios, or e-portfolios, are believed to increase the accessibility of a portfolio in medical training provided that the purpose of the portfolio development is well defined, such as for learning, assessment, or continuing professional development (Deketelaere et al., 2009, Tepper et al, 2020; van Tartwijk &amp; Driessen, 2009). Generally, an e-portfolio aims to monitor a resident\u2019s competency development and to stimulate the capacity for self-reflection (Meeus et al., 2006).<\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">An e-portfolio may consist of a list of a resident\u2019s performance, the supporting evidence, and the resident\u2019s self-reflection (van der Schaaf, et al., 2017). The content is specific according to the e-portfolio\u2019s purpose and the required activities\/competencies for the resident at each level (Mulder et al., 2010). The use of an e-portfolio with a mobile application and updated evidence of residents\u2019 performance potentially enhance informed decision-making in the entrustment process, and hence, it can be embedded in the resident supervision system (Ten Cate et al., 2016). An exploratory study in Australia showed that an EPA-driven e-portfolio model assisted trainees and supervisors in agreeing upon expected trainees\u2019 performance in order to obtain competence level (Bramley et al., 2020).<\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">Implementation of e-portfolio as a supporting system for EPA should consider supporting and hindering factors specific to an institution. Implementation of best practices in this area can be accompanied by an exploration of the most suitable system according to the stakeholders\u2019 needs. Therefore, the current study aimed to explore stakeholders\u2019 needs and develop a mobile-app e-portfolio model in a teaching hospital with various postgraduate medical education programmes located in a resource-limited setting. The study also aims to identify challenges of e-portfolio implementation. The research questions of this study were: (1) What are mobile-app e-portfolio features needed to support EPA implementation in the current teaching hospital? (2) How do stakeholders perceive the use of the developed mobile-app e-portfolio during the pilot study?\u00a0 <\/span><\/p>\r\n<p style=\"text-align: center\"><span lang=\"EN-GB\"><\/span><strong><span lang=\"EN-GB\">II. METHODS<\/span><\/strong><\/p>\r\n<p><i><span lang=\"EN-GB\"><\/span><\/i><i><span lang=\"EN-GB\">A. Context<\/span><\/i><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">Cipto Mangunkusumo General Hospital (CMGH) is the main teaching hospital for the Faculty of Medicine Universitas Indonesia (FMUI) and is located in Jakarta, Indonesia. All 31 study programmes implement a competency-based medical curriculum and EPA approach. CMGH is obliged to have all study programmes participate in clinical activities, which are gradually entrusted to residents in accordance with the resident\u2019s level. The EPA document in form of a matrix is then used to develop supervision system. Supervisors are trained to provide feedback through compulsory clinical teacher training conducted by the faculty. Assessment documentation has been conducted mostly manually in hardcopies for all study programs; consequently, tracking the residents\u2019 performance to provide formative or summative EPA decisions has been challenging in the current setting. <\/span><\/p>\r\n<p><i><span lang=\"EN-GB\"><\/span><\/i><i><span lang=\"EN-GB\">B. Design<\/span><\/i><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">This study is participatory action research with a mixed -method approach. The exploration stage was aimed at exploring the needs and features expected from the mobile application. Following the exploration stage, the research team in collaboration with a professional vendor developed the mobile-app e-portfolio according to the results of the exploration stage. The pilot implementation phase aimed to implement and train residents and supervisors in the use of the application. A follow-up evaluation was conducted to assess users\u2019 perception about the use of the e-portfolio application. <\/span><\/p>\r\n<p><i><span lang=\"EN-GB\"><\/span><\/i><i><span lang=\"EN-GB\">C. Respondents <\/span><\/i><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">Respondents of this study were education leaders from the medical school and the teaching hospital. Residents, clinical teachers, and administrators <\/span><span lang=\"EN-ID\">from six study programs representing medicine, surgery, and diagnostic: anaesthesiology and intensive care, obstetrics and gynaecology, psychiatry, oncology radiation, radiology, and internal medicine at CMGH\/FMUI were recruited with maximum variety sampling method taking into account gender, age, and length of study (residents) or work (clinical teachers). Detailed information of the study was given to all participants who then provided written consents prior to the interviews and completion of the questionnaire.<\/span><\/p>\r\n<p style=\"text-align: justify\"><i><span lang=\"EN-GB\">D. Data Collection<\/span><\/i><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">The exploration stage involved interviews with: (1) residency programme directors; (2) education leaders from the medical school and the teaching hospital. Focus group discussions (FGDs) were also conducted with clinical teachers and residents from the six study programmes. The guiding questions of the interviews and FGDs were developed according to e-portfolio and EPA concepts used in medical education, and are shown in Table 1. The data obtained were used as a reference for further development of the application\u2019s content and features. In addition, the EPA document from each study program was also analysed as baseline data for developing the platform.<\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">Following the development of the application, a four-week-long pilot implementation was conducted in November 2018. After week four, the back-end system data were analysed. Supervisors\u2019 and residents\u2019 perceptions of the use of mobile-app e-portfolio for supporting EPA were collected using a modified questionnaire of System Usability Scale (SUS) (Brooke, 1986). Active commentaries were also added in the questionnaire to obtain users\u2019 suggestions<i>:<\/i><\/span><\/p>\r\n<p><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/09\/OA2459-Table-1-300x130.png\" alt=\"\" class=\" wp-image-70991 aligncenter\" width=\"725\" height=\"314\" srcset=\"https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/09\/OA2459-Table-1-300x130.png 300w, https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/09\/OA2459-Table-1-1024x444.png 1024w, https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/09\/OA2459-Table-1-768x333.png 768w, https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/09\/OA2459-Table-1-1536x666.png 1536w, https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/09\/OA2459-Table-1.png 1767w\" sizes=\"auto, (max-width: 725px) 100vw, 725px\" \/><\/p>\r\n<p align=\"center\"><span lang=\"EN-GB\">Table 1. Guiding questions of interviews and focus groups<\/span><\/p>\r\n<p align=\"left\" style=\"text-align: left\"><i><span lang=\"EN-GB\">E. Data Analysis<\/span><\/i><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">The transcripts of interviews and FGDs of the need analysis stage were analysed using thematic analysis. The agreed subthemes and themes and the results of the analysis of EPA documents from each study program were then translated into a computational framework to be developed as mobile application e-portfolio\u2019s features, also considering the best practice and theoretical framework of e-portfolio development and EPA practice. <\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">Back-end data of the activities, case variations, and supervisors\u2019 feedback were descriptively analysed using SPSS IBM 22.0. The words submitted on the reflection-on-action section of each activity were counted along with the feedback recorded by the supervisors. Residents\u2019 reflections were categorised into those with descriptions of difficulties (DD), lessons learned (LL), and action plans (AP). The feedback recorded was classified into descriptive (D), constructive (C), and neither descriptive nor constructive (N) categories. <\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">The modified SUS questionnaires were descriptively analysed using chi-square\/fisher exact analysis (SPSS 22.0) to compare residents\u2019 and supervisors\u2019 perceptions. The active commentaries in the questionnaire were also thematically analysed to identify areas of improvement in the e-portfolio development.<\/span><\/p>\r\n<p style=\"text-align: center\"><span lang=\"EN-GB\"><\/span><strong><span lang=\"EN-GB\">III. RESULTS<\/span><\/strong><\/p>\r\n<p><i><span lang=\"EN-GB\"><\/span><\/i><i><span lang=\"EN-GB\">A. Exploration Stage<\/span><\/i><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">Eight in-depth interviews were conducted with each residency program director as well as with the leader of FMUI and with teaching hospital director representatives. Two FGDs were conducted, each with three study programmes faculty members. In addition, three FGDs were conducted, each with two study programmes resident representatives. The number and characteristics of the participants are described in Table 2.<\/span><\/p>\r\n<p><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/09\/OA2459-Table-2-300x149.png\" alt=\"\" class=\" wp-image-70993 aligncenter\" width=\"596\" height=\"296\" srcset=\"https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/09\/OA2459-Table-2-300x149.png 300w, https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/09\/OA2459-Table-2-1024x507.png 1024w, https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/09\/OA2459-Table-2-768x381.png 768w, https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/09\/OA2459-Table-2-1536x761.png 1536w, https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/09\/OA2459-Table-2.png 1566w\" sizes=\"auto, (max-width: 596px) 100vw, 596px\" \/><\/p>\r\n<p align=\"center\"><span lang=\"EN-GB\">Table 2. Characteristics of Focus Group Discussion (FGD) Participants in Exploration Stage<\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">Four main themes and 22 subthemes were identified during the exploration stage. The main themes identified were (1) resident supervision; (2) current implementation of workplace-based assessments; (3) current implementation of portfolios and logbooks; and (4) development of the mobile application e-portfolio.<\/span><\/p>\r\n<table>\r\n<thead>\r\n<tr>\r\n<td>\r\n<p><b><span lang=\"EN-GB\">\u00a0<\/span><\/b><\/p>\r\n<\/td>\r\n<td>\r\n<p><b><span lang=\"EN-GB\">Theme<\/span><\/b><\/p>\r\n<\/td>\r\n<td>\r\n<p><b><span lang=\"EN-GB\">Subtheme<\/span><\/b><\/p>\r\n<\/td>\r\n<td>\r\n<p><b><span lang=\"EN-GB\"># of Quotes<\/span><\/b><\/p>\r\n<\/td>\r\n<td>\r\n<p><b><span lang=\"EN-GB\">Quotations<\/span><\/b><\/p>\r\n<\/td>\r\n<\/tr>\r\n<\/thead>\r\n<tbody>\r\n<tr>\r\n<td>\r\n<p><span lang=\"EN-GB\">1<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">Resident supervision<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">Constraints in supervision (faculty members, patient\/clinical cases)<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">59<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u00a0\u201cFor example, during the mid-level programme, residents were required to be fully supervised, but when they should be fully supervised was not specifically stated.\u201d (W1, p. 1)<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">Form of supervision<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">46<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u00a0\u201c\u2026we should report to the supervisor when anything goes wrong to get feedback; otherwise, it will become our fault.\u201d (FGD 2, [R5], p. 2)<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">Determination of entrustment (written curriculum, residents and patients factor, workplace, agreement among faculty members)<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">42<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u00a0\u201cAt some point, we might feel that the resident is eligible to perform the procedure after he\/she has observed several times, and then we try to let him\/her give it a try, but we are actually risking our trust&#8230;\u201d (W7, p. 5)<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">Background and determination of supervision (workplace, resident\u2019s educational stage)<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">26<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u201cSupervision was given hierarchically; the third-year residents attain duty as the chief, supervisor, and the highest form was division of weekly rounds.\u201d (FGD 4, [R1], p. 2)<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">Documentation of supervision level<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">10<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u201cOur logbook contains a very clear explanation about supervision level, such as what cases should be achieved so that we can evaluate and reflect on our progress in one semester.\u201d (FGD 3, [R6], p. 9)<\/span><\/p>\r\n<p><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td>\r\n<p><span lang=\"EN-GB\">2<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">The current implementation of workplace-based assessment<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">Scheduled WBA <\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">17<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u201cDuring each rotation, residents and consultants were scheduled with case-based discussions or miniCEX; while for surgical DOPS, [it] is not specifically scheduled because it depends on the availability of the case.\u201d (W3, p. 1)<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">WBA as stage step-up procedure<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">6<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u201cIn our programme, there are several cases we need to get done at each level, usually cases written in the logbook.\u201d (FGD 2, [R7], p. 3)<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">Based on the written curriculum<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">4<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u201cIn our programme, we arrange evaluation at every level. We evaluate every intern according to the required competency they should achieve.\u201d (FGD 4, [R3], p. 7)<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td>\r\n<p><span lang=\"EN-GB\">3<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">The current implementation of portfolio and logbook<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<p><span lang=\"EN-GB\">Technical portfolio and logbook implementation (as monitoring, communication, and assessment instrument)<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">26<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u201cOn a routine basis, the form was filled in each afternoon, when the patient\u2019s information, such as name, weight, and height, are recorded.\u201d (FGD 2, [R7], p. 4)<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">Constraints in portfolio and logbook implementation (lack of ownership, faculty member factor)<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">38<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u201cDue to the enormous workload every day, such as examining patients, attending division rounds, filling out electronic health records, and receiving so many instructions, they couldn\u2019t complete the logbook routinely. Besides, the supervisor might find it unnecessary to write down their supervision in the log book after discussions, rounds, or treatments.\u201d (W2, p. 10)<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">Understanding of portfolio and logbook for residents\u2019 development (clinical, academic, non-academic)<\/span><\/p>\r\n<p><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">24<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u201cPortfolio consists of one\u2019s creativity, innovation, and organisation experiences.\u201d (W8, p. 1)<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td>\r\n<p><span lang=\"EN-GB\">4<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">Development of the mobile application-based e-portfolio <\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">Residents\u2019 performance and experience (clinical and academic) tracking <\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">20<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u201cWe evaluated one intern\u2019s competency in ultrasound, and the result wasn\u2019t quite satisfying. It turned out that this intern lacked of practice that could\u2019ve been done every day since the cases were quite abundant.\u201d (W6, p. 10)<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">User-friendliness and real-time access <\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">18<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u201cThe application may have to be \u2018consultant-friendly,\u2019 so they can finish it in a click.\u201d (FGD 2, [R3], p. 7)<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">Feedback documentation<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">13<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u00a0\u201cIn my opinion, there has to be some immediate feedback.\u201d (FGD 2, [R5], p. 7)<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">Accessibility and confidentiality <\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">11<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u201cAs they save the record of the treatment that has been checked by the doctor in charge, they can no longer change it because only the supervisor has the authority to change.\u201d(FGD 4, [R4,] p. 19)<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">Encompass achievement of competence and modules within the curriculum <\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">7<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u201cWe want it to be comprehensive, so it can be evaluated. This is about clinical skills, but then the related academic skill is also important.\u201d (W4, p. 9)<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">Ensuring patients were treated by competent and authorised residents <\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">4<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u201cWe need to ensure that the patient is treated by competent, authorised, and certified doctors.\u201d (W7, p. 10)<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">Reminder<i> <\/i>and consequences for undocumented activity <\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">5<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u201cThere must be something to force the resident to write down his activity so the next day, he can keep up with the new tasks.\u201d (W2, p. 10)<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">Paperless attribute, but printable if needed <\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">6<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u201cThe record and the format will be the students\u2019 database as well as <\/span><\/p>\r\n<p><span lang=\"EN-GB\">the supervisors\u2019. Therefore, it needs to be printable for our benefit.\u201d <\/span><\/p>\r\n<p><span lang=\"EN-GB\">(FGD 2, [R3], p. 7)<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">Integrated with service care system <\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">3<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u201cWe have to write down the same thing repetitively in so many books and records.\u201d (FGD 3, [R8], p. 12) <\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<tr>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">Collect important evidence of accomplishments (cases and complications, image documentation) <\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">4<\/span><\/p>\r\n<\/td>\r\n<td>\r\n<p><span lang=\"EN-GB\">\u201cCompiling the number of cases is required, and reporting complications in cases is equally important. Let\u2019s say they are in the third level, but complications occurred in most of the procedures they performed.\u201d (W3, p.\u00a07)<\/span><\/p>\r\n<p><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<p align=\"center\"><span lang=\"EN-GB\">Table 3. Themes and Subthemes Identified in Exploration Stage<\/span><\/p>\r\n<div>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">According to the analysis conducted in the exploration stage, stakeholders expected the availability of various features in the mobile application-based e-portfolio. However, it was not possible to accommodate all of the identified features at the implementation stage. The authors identified the key features in order to develop a generic application, taking into consideration the findings related to the current implementation of a workplace-based assessment, portfolios and logbooks, supervision, level of entrustment, and other technical issues. Also considered were the identified basic needs of the teaching hospital, study programmes, faculty members, and residents, along with best practice principles related to application development.<\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">Some of the key features derived from the analysis included in the implementation stage were the following:<\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">1) <\/span><span lang=\"EN-GB\">Supervisors and administrators are allowed to track residents\u2019 performance and achievements.<\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">2) <\/span><span lang=\"EN-GB\">A drop-down menu is used in most form segments for easier and user-friendly data input.<\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">3) <\/span><span lang=\"EN-GB\">Feedback from the clinical supervisor is recorded on each data input.<\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">4) <\/span><span lang=\"EN-GB\">Clinical cases, supervision level, and type of clinical authorities are set according to the residential programme and the resident\u2019s educational stage.<\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">5) <\/span><span lang=\"EN-GB\">Guided self-reflection questions for the resident are mandatory on each data input.<\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">6) <\/span><span lang=\"EN-GB\">Attachment of supporting image as evidence is facilitated for each data input.<\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">The application was divided into two platforms: for residents and supervisors. The menus in the residents\u2019 application included activity list, activity input, report, and profile, among others. While submitting new activity into the platform, residents must fill in the information related to each clinical activity, describe the activity according to the given guided reflection-on-action features, and provide evidence form of photographs (if applicable). The supervisor application had only one main feature for reviewing and providing feedback on activities previously submitted by residents. The figure of the application and its overall process of the application is shown in the appendix.<\/span><\/p>\r\n<p><i><span lang=\"EN-GB\">B. Pilot Implementation Stage<\/span><\/i><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">During the four-week-long pilot programme, the residents were encouraged to submit as much evidence as possible related to patient care to the e-portfolio platform. The evidence was automatically sent to assigned faculty members\u2019 platform in order to get feedback. Observations were conducted by evaluating the data input from the application\u2019s backend to obtain data regarding the utilisation frequency and types of cases input in the mobile application in each residency programme. A total of 311 activities were submitted into the application during the four-week-long pilot programme; however, only 98 (31.5%) activities received feedback from clinical supervisors. Table 4 describes the results of the average word count analysis of the residents\u2019 reflection-on-action and the feedback received in the mobile application e-portfolio during the implementation stage.<\/span><\/p>\r\n<p><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/09\/OA2459-Table-4-300x123.png\" alt=\"\" class=\"aligncenter wp-image-70994\" width=\"718\" height=\"294\" srcset=\"https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/09\/OA2459-Table-4-300x123.png 300w, https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/09\/OA2459-Table-4-1024x421.png 1024w, https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/09\/OA2459-Table-4-768x315.png 768w, https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/09\/OA2459-Table-4-1536x631.png 1536w, https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/09\/OA2459-Table-4.png 1875w\" sizes=\"auto, (max-width: 718px) 100vw, 718px\" \/><\/p>\r\n<p align=\"center\"><span lang=\"EN-GB\">Table 4. Activity Report of the Pilot Programme: Recorded Activity, Residents\u2019 Reflections and Feedback from Supervisors<\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">1: Psychiatry, 2: Internal Medicine, 3: Obstetrics &amp; Gynaecology, 4: Anaesthesiology &amp; Intensive Care, 5: Oncology Radiation, 6: Radiology; DD: description of difficulties<\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">LL: lessons learned, AP: action plan, D: descriptive feedback, C: constructive feedback, N: neither descriptive nor constructive feedback, AWC: average word count <\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">The feedback received covered a wide range of competencies: clinical skills, knowledge, and patient care. The quotations below are completed with codes explaining the details as follows: a. FB which stands for Feedback; b. Number (1-6) which refers to the study programme 1 to 6; and c. Last number (e.g 9, 15) which refers to the list number of feedback comment in the system. <\/span><\/p>\r\n<p><span lang=\"EN-GB\">\u201c<i>Spinal USG exercise on new-born infants to improve skills<\/i>.\u201d<\/span><\/p>\r\n<p align=\"right\"><span lang=\"EN-GB\">(FB, (5), 9)<\/span><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<p><span lang=\"EN-GB\">\u201c<i>Learn about the differences in fracture due to trauma, metastasis, and infection<\/i>.\u201d <\/span><\/p>\r\n<p align=\"right\"><span lang=\"EN-GB\">(FB, (5), 13)<\/span><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<p><span lang=\"EN-GB\">Although most feedback obtained from supervisors was classified as descriptive or \u201cneither descriptive nor constructive\u201d, there was some constructive feedback, describing not only what to do but also how to do it and improve the residents\u2019 performance.<\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">\u201c<i>Volume target determination was excellently and efficiently carried out. Effective and efficient communication could be achieved even by telephone. There were few problems in terms of work efficiency because no confirmation was made about which work had and\/or had not been done by the supervisor. However, the overall process and results were excellent<\/i>.\u201d <\/span><\/p>\r\n<p align=\"right\"><span lang=\"EN-GB\">(FB, (4), 9)<\/span><span lang=\"EN-GB\">\u00a0<\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">The users\u2019 evaluations on the mobile application-based e-portfolio were obtained using a modified version of the SUS questionnaire at the end of the pilot programme. The pilot study participants included 45 faculty members and 66 residents. Out of a total of 111 questionnaires distributed, a total of 92 questionnaires were received back: 37 faculty members\u2019 response (84%) and 55 residents\u2019 response (82%). The results showed that the faculty members and residents had similar views regarding most aspects evaluated by the questionnaire. Comparisons between the proportions were conducted using a chi-square test\u2014which showed significant differences between faculty members\u2019 and residents\u2019 perceptions on seven items as<\/span><span lang=\"EN-GB\"> <\/span><span lang=\"EN-GB\">shown in Figure 1. The complete result of the SUS questionnaire is provided in Appendix 2.<\/span><\/p>\r\n<p><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/09\/OA2459-Figure-1-300x153.png\" alt=\"\" class=\" wp-image-70995 aligncenter\" width=\"575\" height=\"294\" srcset=\"https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/09\/OA2459-Figure-1-300x153.png 300w, https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/09\/OA2459-Figure-1-1024x522.png 1024w, https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/09\/OA2459-Figure-1-768x392.png 768w, https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/09\/OA2459-Figure-1.png 1530w\" sizes=\"auto, (max-width: 575px) 100vw, 575px\" \/><\/p>\r\n<p align=\"center\"><span lang=\"EN-GB\">Figure 1. Comparison among supervisors and residents satisfaction on the e-portfolio application<\/span><\/p>\r\n<p><span lang=\"EN-GB\">The results of the thematic analysis of the active commentaries in the questionnaire revealed four main themes: <\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\"><\/span><span lang=\"EN-GB\">1) Participants believed that the application was user-friendly but needed improvement in some features and interfaces. <\/span><\/p>\r\n<p><span lang=\"EN-GB\">\u201c<i>The display can be improved so the most frequent treatment can be put on top. Thus no repeated scrolling down is needed<\/i>.\u201d<\/span><\/p>\r\n<p align=\"right\"><span lang=\"EN-GB\">Commentaries- resident-1 (52)<\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\"><\/span><span lang=\"EN-GB\">2) Some constraints identified were: the length and number of forms to be filled in the mobile application, the lack of notifications, case grouping according to clinical rotation, as well as the needs for integration with electronic health records and other established applications in each study programme. <\/span><\/p>\r\n<p><span lang=\"EN-GB\">\u201c\u2026<i>filling the apps is burdening residents with a bunch of caseload. The interface has to be simplified, with the option to pick certain cases without deeply analysing each case (e.g. there were 50 cases today yet only 2 unique\/interesting cases which required special attention, etc.).\u201d<\/i> <\/span><\/p>\r\n<p align=\"right\" style=\"text-align: right\"><span lang=\"EN-GB\">Commentaries-resident-6 (46)<\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\"><\/span><span lang=\"EN-GB\">3) Both faculty members and residents similarly noted utilisation of the application to facilitate supervision. <\/span><\/p>\r\n<p><span lang=\"EN-GB\">\u201c<i>All supervising needs have been accommodated<\/i>.\u201d<\/span><\/p>\r\n<p align=\"right\" style=\"text-align: right\"><span lang=\"EN-GB\">\u00a0Commentaries-faculty-1(91)<\/span><\/p>\r\n<p><span lang=\"EN-GB\">\u201c<i>Supervisor\u2019s verification system was a good thing<\/i>.\u201d <\/span><\/p>\r\n<p align=\"right\" style=\"text-align: right\"><span lang=\"EN-GB\">Commentaries-residents-3 (8)<\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\"><\/span><span lang=\"EN-GB\">4) Lastly, the participants felt that the availability of the feedback feature was positive, despite some identified constraints in giving feedback.<\/span><\/p>\r\n<p><span lang=\"EN-GB\">\u201c<i>The most important feature of this, I think is the feedback giving and follow-up plan<\/i>.\u201d <\/span><\/p>\r\n<p align=\"right\"><span lang=\"EN-GB\">Commentaries-faculty-5(63)<\/span><\/p>\r\n<p style=\"text-align: center\"><span lang=\"EN-GB\"><\/span><strong><span lang=\"EN-GB\">IV. DISCUSSION<\/span><\/strong><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">This study highlights the development of an app-based e-portfolio in the teaching hospital of a resource-limited setting. The study was able to identify key features of the e-portfolio based on the stakeholders\u2019 (i.e programme coordinators, supervisors, and residents). Several challenges which should be addressed for future improvement were also identified in the pilot study. <\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">The first stage of the study was able to explore the perceptions of users which informed the e-portfolio. The users identified the need of a portfolio (Crossley et al., 2011) for documenting residents\u2019 achievements over time and for assessing coverage of clinical case management. The use of a portfolio in digital form in this study also aimed to facilitate the process of supervising residents working in the teaching hospital. The supervision system in the current teaching hospital adopted the EPA concept (Carraccio et al., 2017; Ten Cate et al., 2015), which had been in use for the past 2 years in the six study programmes.<\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">The use of an e-portfolio is not particularly new in postgraduate training (Kjaer et al., 2006), but the particular use for facilitating supervision of residents has, to the best of the authors\u2019 knowledge, rarely been reported (van der Schaaf et al., 2017). Reported challenges of e-portfolio are the residents\u2019 burden despite the use of an electronic platform (Birks et al., 2016; Vance et al., 2017), the reliability of a workplace-based assessment portfolio (Castanelli et al., 2019), and the fact that time issues in clinical practice should always be taken into consideration (Binhimd et al., 2017). Heeneman and Driessen hence clarify that it is necessary to determine whether portfolio development by residents is voluntary or compulsory (Heeneman &amp; Driessen, 2017). Based on the framework, the current e-portfolio can be categorised as a combination of a training and personal development portfolio, which comprises mandated required skills and competencies in a fixed format as well as personal reflections of progress (Heeneman &amp; Driessen, 2017). In addition, the use of mobile application was expected to support the use of e-portfolio for residents\u2019 learning in a busy clinical setting (Mok et al., 2019).<\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">An important framework that was adopted in the e-portfolio platform in this study is the EPA. EPA development calls for the consideration of competencies, clinical settings, the entrustment process and stakeholders\u2019 agreements, including those of staff\/supervisors and experts (Ten Cate et al., 2015), residents, the teaching hospital and even patients, if applicable (<\/span><span lang=\"EN-ID\">Lundsgaard et al., 2019<\/span><span lang=\"EN-GB\">).<\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">The authors of the present study identified the potential value of a portfolio as a tool to provide ad-hoc supervisor decision-making, based on the study by van der Schaaf et al. (2017). Based on the reflections in the six study programmes, the e-portfolio incorporated residents\u2019 performance, supervisors\u2019 feedback, and evidence of achievement that is in line with the expected level of competencies or EPAs in the curriculum. The evidence is required by supervisors for deciding on a level of entrustment. A study involving experienced obstetrics and gynaecology supervisors in the Netherlands suggests that both formal and informal entrustment processes are used in combination on a daily basis. Informal entrustment is thought to provide more flexibility because it can be adjusted more easily than a formal one (Castanelli et al., 2019). <\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">Based on the analysis in the exploration stage of the present study, the platform was designed to accommodate entries by a resident which were then responded to by a supervisor directly in the clinical setting or indirectly through the application (with the supervisor\u2019s presence according to the EPA and supervision level for particular case managements\/procedures). According to van der Schaaf et al. (2017), the current e-portfolio adapted both a student model that provided achievement monitoring and a feedback model that provided personalised feedback. The data visualisation in the current study was not yet used for the supervisors\u2019 entrustment process and is a subject for further development of the application and back-end data analysis.<\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">The e-portfolio application in this study was designed to include a feedback model, which was initiated by reflection by the resident. As described in Table 5, despite the number of recorded activities, only 12.5\u201345% of them received written feedback from the supervisors. Of these, most of the feedback was descriptive or constructive. Before providing or documenting feedback in the application, the supervisor may verify the case or do so after observing the resident performing the procedure. <\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">Apart from the residents of the Radiology programme, almost all residents documented their reflections on the encountered clinical cases: the difficulties, lessons learned, and action plans (Table 3). The residents\u2019 reflections were considered as one of the key successes of the portfolio implementation (Binhimd et al., 2017), and it is also a best practice in feedback provision in clinical training (<\/span><span lang=\"EN-ID\">Bounds et al., 2013; <\/span><span lang=\"EN-ID\">Kornegay<\/span><span lang=\"EN-ID\"> et al, 2017; Ramani &amp; Krackov, 2012<\/span><span lang=\"EN-GB\">). The residents in the pilot study displayed a good habit of feedback-seeking behaviour by initiating their reflections. Given the burden of daily routine in the teaching hospitals, the quality of the written reflections completed by residents was of concern in this study. Further consideration to moderate the frequency of reflections should be incorporated hence the residents can practice more in-depth reflections which are necessary for more meaningful feedback and future actions for improvement.<\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">All reflections and feedback were given in less than 50 words; hence, it is again considered feasible for the users to conduct brief reflection and feedback within the application. The residents\u2019 feedback showed that it was one of the most important features of the application, which supported their learning. Despite this, a rather low amount of feedback with low word counts was given to the residents during the pilot study, raising concerns that despite the feedback training given to supervisors, the use of the application did not necessarily increase the amount and quality of documented feedback. As suggested by Brehaut et al. (<\/span><span lang=\"EN-ID\">2016<\/span><span lang=\"EN-GB\">), the format of feedback delivery matters, and it is preferable to avoid the cognitive overload of the recipients in the use of an electronic platform, as attempted in the current application. The limited amount of feedback documented during the pilot study, however, did not reflect the actual practice of giving feedback in the current setting. The documentation of feedback was expected to facilitate the long-term monitoring of residents\u2019 progress (van der Schaaf et al., 2017), which may prompt supervisors\u2019 summative entrustment decisions. The residents\u2019 feedback in this study also highlighted the need for deeper reflection on the voluntary selection of clinical activities, so that the resident did not have to input all encountered cases. This might increase the user-friendliness of the application, yet it might require further agreement with the supervisors and the study programme coordinators. <\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">The evaluation of the supervisors and residents in the pilot study showed that supervisors perceived more positively the use of the apps, compared to the residents. From the supervisors\u2019 point of views, accessibility of online data on resident\u2019s performance was responded to very positively given current challenge in tracking this in manual documentation. From the residents\u2019 point of views, this might be due to that despite the potential of the current application to enable clinical supervision, submitting data into the application felt like an addition to the residents\u2019 workload, due to redundancy in submitting information into both e-portfolio platform and the e-hospital medical records. The workload of residents in teaching hospitals has been a controversial issue in postgraduate medical education (<\/span><span lang=\"EN-ID\">Nishigori et al., 2015<\/span><span lang=\"EN-GB\">). The stakeholders should agree on which data are compulsory and critical for evaluation and which are voluntary in the e-portfolio, both for training and patient care purposes. In addition, the authors also realised that training on portfolio development and supervision should be supported with a longitudinal mentoring process and an analysis of longitudinal performance (Heeneman &amp; Driessen, 2017).<\/span><\/p>\r\n<p><i><span lang=\"EN-GB\"><\/span><\/i><i><span lang=\"EN-GB\">A. Study Limitations<\/span><\/i><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">The study has some limitations. The study was conducted at one teaching hospital and involved a limited number of study programmes. Given that Cipto Mangunkusumo is one of the busiest national referral hospitals, with complex cases and high workloads, the perspectives of key stakeholders explored in this study were critical for consideration in the development of an e-portfolio mobile application. In addition, the study was able to identify the features of an e-portfolio that can potentially support the supervision and entrustment processes. Future versions of the e-portfolio application aimed at enhancing the quality of supervision should consider the analysis of the residents\u2019 longitudinal performance through prompt data analysis in the system, thereby supporting formative and summative entrustments during clinical supervision as well as the integration of an adequate mentoring system. <\/span><\/p>\r\n<p style=\"text-align: center\"><span lang=\"EN-GB\"><\/span><strong><span lang=\"EN-GB\">V. CONCLUSION<\/span><\/strong><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">The present study explored the stakeholders\u2019 need of an e-portfolio for supervision that enables all parties to monitor learning progress and competency achievement. On top of the implementation of workplace-based assessments and EPA systems in the respective study programmes, the identified key features of an e-portfolio mobile application reflect the needs of residents, faculty\/supervisors, the teaching hospital, and the school of medicine.<\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">The pilot implementation showed that the e-portfolio mobile application was feasible and potential for use by residents, supervisors and programme coordinators in monitoring EPAs and competence achievement in the current setting, with highlighted challenges that need to be tackled in the future. The platform\u2019s use in assisting entrustment decisions should be further confirmed with a longitudinal analysis of the residents\u2019 performance and the use of the analysis by the supervisors. The development of such an e-portfolio to support EPA in other settings should consider the dynamics of case complexity, residents\u2019 level of competence and entrustment, workload, the healthcare system, and the education system. Finally, in a resource-limited setting, the involvement of stakeholders from the start to prioritise the e-portfolio features, user-friendliness, and technology feasibility should always be considered. <\/span><\/p>\r\n<p align=\"center\"><strong><span lang=\"EN-GB\">Notes on Contributors<\/span><\/strong><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">Ardi Findyartini reviewed the literature, designed the study, conducted interviews and focus group discussions, analysed transcripts and led the manuscript writing.<\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">Natalia Widiasih reviewed the literature, designed the study, conducted interviews and focus group discussions, \u00a0analysed transcripts and wrote the manuscript. <\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">Nadia Greviana conducted interviews and focus group discussions, analysed transcripts and wrote the manuscript.<\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">Gregorius Ben Prayogi conducted interviews and focus group discussions,\u00a0 analysed transcripts and gave critical feedback in the manuscript writing. <\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">Daniar Setyorini coordinated the e-portfolio development and data collection, analysed transcripts and gave critical feedback in the manuscript writing. <\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">All authors have read and approved the final manuscript.<\/span><\/p>\r\n<p align=\"center\"><strong><span lang=\"EN-GB\">Ethical Approval<\/span><\/strong><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">Ethical approval for this study was granted by the Faculty of Medicine Universitas Indonesia \u2013 Cipto Mangunkusumo Hospital Research Ethical Review Board (2018) Number: 328\/UN2.F1\/ETIK\/III\/2018.<\/span><\/p>\r\n<p align=\"center\"><strong><span lang=\"EN-AU\">Data Availability<\/span><\/strong><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-AU\">Data will be available upon request to corresponding author due to conditions of consent provided by respondents in this study and that it should abide data sharing policy from the medical school and teaching hospital where the study was completed. <\/span><\/p>\r\n<p align=\"center\"><strong><span lang=\"EN-GB\">Acknowledgement<\/span><\/strong><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">The authors would like to thank CMGH, FMUI and IMERI for the organisational supports. The authors would also like to extend gratitude to all six participating study programmes, the programme directors, the residents, and the supervisors. Finally, the authors would like to thank Dr. Sri Puspita, who facilitated data collection for the study.<\/span><\/p>\r\n<p align=\"center\"><strong><span lang=\"EN-GB\">Funding<\/span><\/strong><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">This study was funded by Dr. Cipto Mangunkusumo General Hospital Operational Grant 2018. <\/span><\/p>\r\n<p align=\"center\"><strong><span lang=\"EN-GB\">Declaration of Interest<\/span><\/strong><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-AU\">All authors state no possible conflicts of interest, including financial, consultant, institutional and other relationships that might lead to bias or a conflict of interest. <\/span><\/p>\r\n<p align=\"center\"><strong><span lang=\"EN-GB\">References<\/span><\/strong><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">Binhimd, B., Al-Jifrfee, H. M. A., &amp; Khan, M. A. (2017). Identifying the perceived challenges that affect the use of portfolio as an assessment tool for family medicine residents in Jeddah program. <i>MedEdPublish<\/i>, <i>6<\/i>(2), 1\u201312. <\/span><span lang=\"EN-ID\"><a href=\"https:\/\/doi.org\/10.15694\/mep.2017.000070\">https:\/\/doi.org\/<span lang=\"EN-GB\">10.15694\/mep.2017.000070<\/span><\/a><\/span><span lang=\"EN-GB\"><\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">Birks, M., Hartin, P., Woods, C., Emmanuel, E., &amp; Hitchins, M. (2016). Students\u2019 perceptions of the use of eportfolios in nursing and midwifery education. <i>Nurse Education in Practice<\/i>, <i>18<\/i>, 46\u201351. <\/span><span lang=\"EN-ID\"><a href=\"https:\/\/doi.org\/10.1016\/j.nepr.2016.03.003\">https:\/\/doi.org\/<span lang=\"EN-GB\">10.1016\/j.nepr.2016.03.003<\/span><\/a><\/span><span lang=\"EN-GB\"><\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-ID\">Bounds, R., Bush, C., Aghera, A., Rodriguez, N., Stansfield, R. B., Santen, S.A., &amp; MERC at CORD Feedback Study Group (2013). Emergency medicine residents\u2019 self-assessments play a critical role when receiving feedback. <i>Academic Emergency Medicine<\/i>, <i>20<\/i>(10), 1055\u20131061. <\/span><span lang=\"EN-ID\"><a href=\"https:\/\/doi.org\/10.1111\/acem.12231\">https:\/\/doi.org\/10.1111\/acem.12231<\/a><\/span><span lang=\"EN-ID\"><\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-ID\">Bramley, A. L., Thomas, C. J., &amp; Kenna, L. M. (2020). E-portfolios and entrustable professional activities to support competency-based education in dietetic education.<i> Nursing Health Science<\/i>, 1-9. <\/span><span lang=\"EN-ID\"><a href=\"https:\/\/doi.org\/10.1111\/nhs.12774\">https:\/\/doi.org\/10.1111\/nhs.12774<\/a><\/span><span lang=\"EN-ID\"><\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-ID\">Brehaut, J., Colquhoun, H., Eva, K., Carrol, K., Sales, A., Michie, S., Ivers, N., &amp; Grimshaw, J. M. (2016). Practice feedback interventions: 15 suggestions for optimizing effectiveness. <i>Annals of Internal Medicine, 164<\/i>(6), 435\u2013461. <\/span><span lang=\"EN-ID\"><a href=\"https:\/\/doi.org\/10.7326\/M15-2248\">https:\/\/doi.org\/10.7326\/M15-2248<\/a><\/span><span lang=\"EN-ID\"><\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-ID\">Brooke, J. (1986). SUS- A quick and dirty usability scale. <i>Digital Equipment Corporation. <\/i><\/span><span lang=\"EN-ID\"><a href=\"https:\/\/hell.meiert.org\/core\/pdf\/sus.pdf\">https:\/\/hell.meiert.org\/core\/pdf\/sus.pdf <\/a><\/span><span lang=\"EN-ID\">Accessed October 30, 2018 <\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">Carraccio, C., Englander, R., Gilhooly, J., Mink, R., Hofkosh, D., Barone, M. A., &amp; Holmboe, E. S. (2017). Building a framework of entrustable professional activities, supported by competencies and milestones, to bridge the educational continuum. <i>Academic Medicine<\/i>, <i>92<\/i>(3), 324\u2013330. <\/span><span lang=\"EN-ID\"><a href=\"https:\/\/doi.org\/10.1097\/ACM.000000000000114\"><span lang=\"EN-GB\">https:\/\/doi.org\/10.1097\/ACM.000000000000114<\/span><\/a><\/span><span lang=\"EN-GB\"><\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">Castanelli, D. J., Moonen-van Loon, J. M. W., Jolly, \u00a0B., &amp; Weller, J. M. (2019). The reliability of a portfolio of workplace-based assessments in anesthesia training (Fiabilit\u00e9 d\u2019un portfolio d\u2019\u00e9valuations sur le lieu de travail dans la formation en anesth\u00e9sie). <i>Canadian Journal of Anesthesia Can d\u2019anesth\u00e9sie<\/i> [Internet], <i>66<\/i>(2), 193\u2013200. <\/span><span lang=\"EN-ID\"><a href=\"https:\/\/doi.org\/10.1007\/s12630-018-1251-7\">https:\/\/doi.org\/<span lang=\"EN-GB\">10.1007\/s12630-018-1251-7<\/span><\/a><\/span><span lang=\"EN-GB\"><\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">Cianciolo, A. T., &amp; Kegg, J. A. (2013). Behavioral specification of the entrustment process. <i>Journal of Graduate Medical Education<\/i>, <i>5<\/i>(1), 10\u201312. <\/span><span lang=\"EN-ID\"><a href=\"https:\/\/doi.org\/10.4300\/JGME-D-12-00158.1\"><span lang=\"EN-GB\">https:\/\/doi.org\/10.4300\/JGME-D-12-00158.1<\/span><\/a><\/span><span lang=\"EN-GB\"><\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-ID\">Crossley, J., Johnson, G., Booth, J., &amp; Wade, W. (2011). Good questions, good answers: Construct alignment improves the performance of workplace-based assessment scales. <i>Medical Education, 45<\/i>(6), 560\u2013569. <\/span><span lang=\"EN-ID\"><a href=\"https:\/\/doi.org\/10.1111\/j.1362-2923.2010.03913.x\">https:\/\/doi.org\/10.1111\/j.1362-2923.2010.03913.x<\/a><\/span><span lang=\"EN-ID\"><\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">Deketelaere, A., Degryse, J., De Munter, A., &amp; De Leyn, P. (2009). Twelve tips for successful e-tutoring using electronic portfolios. <i>Medical Teacher<\/i>, <i>31<\/i>(6), 497\u2013501. <\/span><span lang=\"EN-ID\"><a href=\"https:\/\/doi.org\/10.1080\/01421590802572734\">https:\/\/doi.org\/<span lang=\"EN-GB\">10.1080\/01421590802572734<\/span><\/a><\/span><span lang=\"EN-GB\"><\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">Farnan, J. M., Petty, L. A., Georgitis, E., Martin, S., Chiu, E., Prochaska, M., &amp; Arora, V. M. (2012). A systematic review: The effect of clinical supervision on patient and residency education outcomes. <i>Academic Medicine<\/i>, <i>87<\/i>(4), 428\u2013442. <\/span><span lang=\"EN-ID\"><a href=\"https:\/\/journals.lww.com\/academicmedicine\/Fulltext\/2012\/04000\/A_Systematic_Review__The_Effect_of_Clinical.13.aspx\">http:\/\/doi.org\/10.1097\/ACM.0b013e31824822cc<\/a><\/span><span lang=\"EN-ID\"><\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-ID\">Hauer, K., Kolwes, J., Cornett, P., Hollander, H., ten Cate, O., Ranji, S. R., Soni, K., Iobst, W., O&#8217;Sullivan, P. (2013). Identifying entrustable professional activities in internal medicine training. <i>Journal of Graduate Medical Education<\/i>, <i>5<\/i>(1), 54\u201359. <\/span><span lang=\"EN-ID\"><a href=\"https:\/\/doi.org\/10.4300\/JGME-D-12-00060.1\"><span lang=\"EN-GB\">https:\/\/doi.org\/10.4300\/JGME-D-12-00060.1<\/span><\/a><\/span><span lang=\"EN-GB\"> <\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">Heeneman, S., &amp; Driessen, E. W. (2017). The use of a portfolio in postgraduate medical education \u2013 Reflect, assess and account, one for each or all in one? <i>GMS Journal for Medical Education<\/i>, <i>34<\/i>(5). <\/span><span lang=\"EN-ID\"><a href=\"https:\/\/doi.org\/10.3205\/zma001134\">https:\/\/doi.org\/<span lang=\"EN-GB\">10.3205\/zma001134<\/span><\/a><\/span><span lang=\"EN-GB\"><\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-ID\">Holmboe, E., Ginsburg, S., &amp; Bernabeo, E. (2011). The rotational approach to medical education: time to confront our assumptions? <i>Medical Education, 45<\/i>(6), 69\u201380. <\/span><span lang=\"EN-ID\"><a href=\"https:\/\/doi.org\/10.1111\/j.1365-2923.2010.03847.x%20\">https:\/\/doi.org\/10.1111\/j.1365-2923.2010.03847.x<\/a><\/span><span lang=\"EN-ID\"> <\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">Kjaer, N. K., Maagaard, R., &amp; Wied, S. (2006). Using an online portfolio in postgraduate training. <i>Medical Teacher<\/i>, <i>28<\/i>(8), 708\u2013712. <\/span><span lang=\"EN-ID\"><a href=\"https:\/\/doi.org\/10.1080\/01421590601047672\">https:\/\/doi.org\/<span lang=\"EN-GB\">10.1080\/01421590601047672<\/span><\/a><\/span><span lang=\"EN-GB\"><\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-ID\">Kornegay, J. G., Kraut, A., Manthey, D., Omron, R., Caretta-Weyer, H., Kuhn, G., Martin, S., &amp; Yarris, L. M. (2017). Feedback in medical education: A critical appraisal. <i>AEM Education and Training, 1<\/i>(2), 98\u2013109. <\/span><span lang=\"EN-ID\"><a href=\"https:\/\/doi.org\/\"><span lang=\"EN-GB\">https:\/\/doi.org\/10.1002\/aet2.10024<\/span><\/a><\/span><span lang=\"EN-GB\"><\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-ID\">Lundsgaard, K. S., Tolsgaard, M. G., Mortensen, O. S., Mylopoulos, M., &amp; \u00d8stergaard, D. (2019). Embracing multiple stakeholder perspectives in defining trainee competence. <i>Academic Medicine<\/i>, <i>94<\/i>(6), 838-846. <\/span><a href=\"https:\/\/doi.org\/10.1097\/ACM.0000000000002642\"><span lang=\"EN-GB\">https:\/\/doi.org\/<\/span><span lang=\"EN-ID\">10.1097\/ACM.0000000000002642<\/span><\/a><span lang=\"EN-ID\"><\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">Meeus, W., Van Petegem, P., &amp; Van Looy, L. (2006). Portfolio in higher education: Time for a clarificatory framework. <i>International Journal of Teaching and Learning in Higher Education<\/i>, <i>17<\/i>(2), 176\u2013180.<\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">Mok, Z. W., Lee, J. C. S., &amp; Mathur, M. (2019). Harnessing mobile technology to develop a sustainable on the go learning resource for junior doctors. <i>The Asia Pacific Scholar<\/i>, <i>4<\/i>(2), 48-51. <\/span><span lang=\"EN-ID\"><a href=\"https:\/\/doi.org\/10.29060\/TAPS.2019-4-2\/SC2070\"><span lang=\"EN-GB\">https:\/\/doi.org\/10.29060\/TAPS.2019-4-2\/SC2070<\/span><\/a><\/span><span lang=\"EN-GB\"><\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">Mulder, H., Ten Cate, O., Daalder, R., &amp; Berkvens, J. (2010). Building a competency-based workplace curriculum around entrustable professional activities: The case of physician assistant training. <i>Medical Teacher<\/i>, <i>32<\/i>(10), e453-e459. <\/span><span lang=\"EN-ID\"><a href=\"https:\/\/doi.org\/10.3109\/0142159X.2010.513719\">https:\/\/doi.org\/10.3109\/0142159X.2010.513719<\/a><\/span><span lang=\"EN-ID\"><\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-ID\">Nishigori, H., Deshpande, G. A., Obara, H., Takahashi, O., Busari, J., &amp; Dornan, T. (2015). Beyond work-hour restrictions: A qualitative study of residents\u2019 subjective workload. <i>Perspectives on Medical Education, 4<\/i>(4), 176\u2013180. <\/span><span lang=\"EN-ID\"><a href=\"https:\/\/link.springer.com\/article\/10.1007%2Fs40037-015-0198-4\">https:\/\/doi.org\/10.1007\/s40037-015-0198-4<\/a><\/span><span lang=\"EN-ID\"><\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-ID\">Ramani, S., &amp; Krackov, S. K. (2012). Twelve tips for giving feedback effectively in the clinical environment. <i>Medical Teacher<\/i>, <i>34<\/i>(10), 787\u2013791. <\/span><span lang=\"EN-ID\"><a href=\"https:\/\/doi.org\/10.3109\/01142159X.2012.684916\">https:\/\/doi.org\/10.3109\/01142159X.2012.684916<\/a><\/span><span lang=\"EN-ID\"><\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-ID\">Sandhu, D. (2018). Postgraduate medical education \u2013 Challenges and innovative solutions. <i>Medical Teacher, <\/i><i>40<\/i>(6), 607\u2013609. <\/span><span lang=\"EN-ID\"><a href=\"https:\/\/doi.org\/10.1080\/0142159X.2018.1461997\">https:\/\/doi.org\/10.1080\/0142159X.2018.1461997<\/a><\/span><span lang=\"EN-GB\"><\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-ID\">Ten Cate, O. (2006). Trust, competence, and the supervisor\u2019s role in postgraduate training. <i>British Medical Journal<\/i>, <i>333<\/i>(7571)<i>,<\/i> 748\u2013751. <\/span><span lang=\"EN-ID\"><a href=\"https:\/\/doi.org\/10.1136\/bmj.38938.407569.94\">https:\/\/doi.org\/10.1136\/bmj.38938.407569.94<\/a><\/span><span lang=\"EN-ID\"><\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">Ten Cate, O., Chen, H. C., Hoff, R. G., Peters, H., Bok, H., &amp; van der Schaaf, M. (2015). Curriculum development for the workplace using entrustable professional activities (EPAs): AMEE guide No. 99. <i>Medical Teacher<\/i>, <i>37<\/i>(11), 983\u20131002. <\/span><span lang=\"EN-ID\"><a href=\"https:\/\/doi.org\/10.3109\/0142159X.2015.1060308\"><span lang=\"EN-GB\">https:\/\/doi.org\/10.3109\/0142159X.2015.1060308<\/span><\/a><\/span><span lang=\"EN-GB\"><\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">Ten Cate, O., Hart, D., Ankel, F., Busari, J., Englander, R., Glasgow, N., Holmboe, E., Lobst, W., Lovell, E., Snell, L. S., Touchie, C., Van Melle, E., Wycliffe-Jones, K., &amp; International Medical Education Collaborators (2016). Entrustment decision making in clinical training. <i>Academic Medicine<\/i>, <i>91<\/i>(2), 191\u2013198. <\/span><span lang=\"EN-ID\"><a href=\"https:\/\/doi.org\/10.1097\/ACM4.0000000000001044\">https:\/\/doi.org\/<span lang=\"EN-GB\">10.1097\/ACM4.0000000000001044<\/span><\/a><\/span><span lang=\"EN-GB\"><\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">Tepper, C., Bishop, J., &amp; Forrest, K. (2020). Authentic assessment utilising innovative technology enhanced learning. <i>The Asia Pacific Scholar<\/i>, <i>5<\/i>(1), 70-75.<\/span><span lang=\"EN-GB\"> <\/span><span lang=\"EN-ID\"><a href=\"https:\/\/doi.org\/10.29060\/TAPS.2020-5-1\/SC2065\">https:\/\/doi.org\/10.29060\/TAPS.2020-5-1\/SC2065<\/a><\/span><span lang=\"EN-ID\"><\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">Vance, G. H. S., Burford, B., Shapiro, E., &amp; Price, R. (2017). Longitudinal evaluation of a pilot e-portfolio-based supervision programme for final year medical students: Views of students, supervisors and new graduates. <i>BMC Medical Education<\/i>, <i>17<\/i>(1), 1\u20139. <\/span><span lang=\"EN-ID\"><a href=\"https:\/\/doi.org\/10.1186\/s12909-017-0981-5\">https:\/\/doi.org\/<span lang=\"EN-GB\">10.1186\/s12909-017-0981-5<\/span><\/a><\/span><span lang=\"EN-GB\"><\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">van der Schaaf, M., Donkers, J., Slof, B., Moonen-van Loon, J., van Tartjwijk, J., Driessen, E., Badii, A., Serban,\u00a0 O., &amp; Ten Cate, O. (2017). Improving workplace-based assessment and feedback by an e-portfolio enhanced with learning analytics. <i>Educational Technology Research and Development<\/i>, <i>65<\/i>(2), 359\u2013380. <\/span><span lang=\"EN-ID\"><a href=\"https:\/\/doi.org\/10.1007\/s11423-016-9496-8\">https:\/\/doi.org\/10.1007\/s11423-016-9496-8<\/a><\/span><span lang=\"EN-ID\"><\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-GB\">van Tartwijk, J., &amp; Driessen, E. W. (2009). Portfolios for assessment and learning: AMEE guide no. 45. <i>Medical Teacher<\/i>, <i>31<\/i>(9), 790\u2013801.<\/span><span lang=\"EN-GB\"> <\/span><span lang=\"EN-ID\"><a href=\"https:\/\/doi.org\/10.1080\/01421590903139201\">https:\/\/doi.org\/10.1080\/01421590903139201<\/a><\/span><span lang=\"EN-ID\"><\/span><\/p>\r\n<p style=\"text-align: justify\"><span lang=\"EN-ID\">Weller, J. M., Misur, M., Nicolson, S., Morris, J., Ure, S., Crossley, J., &amp; Jolly, B. (2014). Can I leave the theatre? A key to more reliable workplace-based assessment. <i>British Journal of Anaesthesia<\/i>, <i>112<\/i>(6), 1083\u20131091. <\/span><span lang=\"EN-ID\"><a href=\"https:\/\/doi.org\/10.1093\/bja\/aeu052\">https:\/\/doi.org\/10.1093\/bja\/aeu052<\/a><\/span><span lang=\"EN-ID\"><\/span><\/p>\r\n<p align=\"left\" style=\"text-align: left\"><span lang=\"EN-NZ\"><span lang=\"EN-AU\">*Ardi Findyartini<br \/>\r\nDepartment of Medical Education &amp; Medical Education Center<br \/>\r\n<\/span><span lang=\"EN-AU\">Indonesia Medical Education Research Institute (IMERI),<br \/>\r\nFaculty of Medicine Universitas Indonesia,<br \/>\r\nJakarta, Indonesia<br \/>\r\n10430<br \/>\r\n<span lang=\"EN-GB\">Email: ardi.findyartini@ui.ac.id<\/span><br \/>\r\n<br \/>\r\n<\/span><\/span><\/p>\r\n<\/div>","protected":false},"featured_media":0,"parent":0,"menu_order":9,"template":"","issues_category":[12],"archive_category":[],"issue_type":[25],"volume_category":[61868],"class_list":["post-70990","issues","type-issues","status-publish","hentry","issues_category-original-articles","issue_type-past-issue","volume_category-volume-6-number-4-october-2021"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v26.4 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Development of an app-based e-portfolio in postgraduate medical education using Entrustable Professional Activities (EPA) framework: Challenges in a resource-limited setting - The Asia Pacific Scholar<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/medicine.nus.edu.sg\/taps\/issues\/development-of-an-app-based-e-portfolio-in-postgraduate-medical-education-using-entrustable-professional-activities-epa-framework-challenges-in-a-resource-limited-setting\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Development of an app-based e-portfolio in postgraduate medical education using Entrustable Professional Activities (EPA) framework: Challenges in a resource-limited setting - The Asia Pacific Scholar\" \/>\n<meta property=\"og:description\" content=\"Submitted: 6 November 2020 Accepted: 27 January 2021 Published online: 5 October, TAPS 2021, 6(4), 92-106 https:\/\/doi.org\/10.29060\/TAPS.2021-6-4\/OA2459 Ardi Findyartini1,2, Natalia Widiasih Raharjanti3, Nadia Greviana1,2, Gregorius Ben Prajogi4 &amp; Daniar Setyorini2 1Department of Medical Education, Faculty of Medicine Universitas Indonesia, Indonesia; 2Medical Education Center, Indonesian Medical Education and Research Institute (IMERI), Faculty of Medicine, Universitas Indonesia, [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/medicine.nus.edu.sg\/taps\/issues\/development-of-an-app-based-e-portfolio-in-postgraduate-medical-education-using-entrustable-professional-activities-epa-framework-challenges-in-a-resource-limited-setting\/\" \/>\n<meta property=\"og:site_name\" content=\"The Asia Pacific Scholar\" \/>\n<meta property=\"article:modified_time\" content=\"2022-01-03T23:44:33+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/09\/OA2459-Table-1-300x130.png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"30 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/medicine.nus.edu.sg\/taps\/issues\/development-of-an-app-based-e-portfolio-in-postgraduate-medical-education-using-entrustable-professional-activities-epa-framework-challenges-in-a-resource-limited-setting\/\",\"url\":\"https:\/\/medicine.nus.edu.sg\/taps\/issues\/development-of-an-app-based-e-portfolio-in-postgraduate-medical-education-using-entrustable-professional-activities-epa-framework-challenges-in-a-resource-limited-setting\/\",\"name\":\"Development of an app-based e-portfolio in postgraduate medical education using Entrustable Professional Activities (EPA) framework: Challenges in a resource-limited setting - The Asia Pacific Scholar\",\"isPartOf\":{\"@id\":\"https:\/\/medicine.nus.edu.sg\/taps\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/medicine.nus.edu.sg\/taps\/issues\/development-of-an-app-based-e-portfolio-in-postgraduate-medical-education-using-entrustable-professional-activities-epa-framework-challenges-in-a-resource-limited-setting\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/medicine.nus.edu.sg\/taps\/issues\/development-of-an-app-based-e-portfolio-in-postgraduate-medical-education-using-entrustable-professional-activities-epa-framework-challenges-in-a-resource-limited-setting\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/09\/OA2459-Table-1-300x130.png\",\"datePublished\":\"2021-09-13T06:36:14+00:00\",\"dateModified\":\"2022-01-03T23:44:33+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/medicine.nus.edu.sg\/taps\/issues\/development-of-an-app-based-e-portfolio-in-postgraduate-medical-education-using-entrustable-professional-activities-epa-framework-challenges-in-a-resource-limited-setting\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/medicine.nus.edu.sg\/taps\/issues\/development-of-an-app-based-e-portfolio-in-postgraduate-medical-education-using-entrustable-professional-activities-epa-framework-challenges-in-a-resource-limited-setting\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/medicine.nus.edu.sg\/taps\/issues\/development-of-an-app-based-e-portfolio-in-postgraduate-medical-education-using-entrustable-professional-activities-epa-framework-challenges-in-a-resource-limited-setting\/#primaryimage\",\"url\":\"https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/09\/OA2459-Table-1.png\",\"contentUrl\":\"https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/09\/OA2459-Table-1.png\",\"width\":1767,\"height\":766},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/medicine.nus.edu.sg\/taps\/issues\/development-of-an-app-based-e-portfolio-in-postgraduate-medical-education-using-entrustable-professional-activities-epa-framework-challenges-in-a-resource-limited-setting\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/medicine.nus.edu.sg\/taps\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Development of an app-based e-portfolio in postgraduate medical education using Entrustable Professional Activities (EPA) framework: Challenges in a resource-limited setting\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/medicine.nus.edu.sg\/taps\/#website\",\"url\":\"https:\/\/medicine.nus.edu.sg\/taps\/\",\"name\":\"The Asia Pacific Scholar\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/medicine.nus.edu.sg\/taps\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Development of an app-based e-portfolio in postgraduate medical education using Entrustable Professional Activities (EPA) framework: Challenges in a resource-limited setting - The Asia Pacific Scholar","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/medicine.nus.edu.sg\/taps\/issues\/development-of-an-app-based-e-portfolio-in-postgraduate-medical-education-using-entrustable-professional-activities-epa-framework-challenges-in-a-resource-limited-setting\/","og_locale":"en_US","og_type":"article","og_title":"Development of an app-based e-portfolio in postgraduate medical education using Entrustable Professional Activities (EPA) framework: Challenges in a resource-limited setting - The Asia Pacific Scholar","og_description":"Submitted: 6 November 2020 Accepted: 27 January 2021 Published online: 5 October, TAPS 2021, 6(4), 92-106 https:\/\/doi.org\/10.29060\/TAPS.2021-6-4\/OA2459 Ardi Findyartini1,2, Natalia Widiasih Raharjanti3, Nadia Greviana1,2, Gregorius Ben Prajogi4 &amp; Daniar Setyorini2 1Department of Medical Education, Faculty of Medicine Universitas Indonesia, Indonesia; 2Medical Education Center, Indonesian Medical Education and Research Institute (IMERI), Faculty of Medicine, Universitas Indonesia, [&hellip;]","og_url":"https:\/\/medicine.nus.edu.sg\/taps\/issues\/development-of-an-app-based-e-portfolio-in-postgraduate-medical-education-using-entrustable-professional-activities-epa-framework-challenges-in-a-resource-limited-setting\/","og_site_name":"The Asia Pacific Scholar","article_modified_time":"2022-01-03T23:44:33+00:00","og_image":[{"url":"https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/09\/OA2459-Table-1-300x130.png","type":"","width":"","height":""}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"30 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/medicine.nus.edu.sg\/taps\/issues\/development-of-an-app-based-e-portfolio-in-postgraduate-medical-education-using-entrustable-professional-activities-epa-framework-challenges-in-a-resource-limited-setting\/","url":"https:\/\/medicine.nus.edu.sg\/taps\/issues\/development-of-an-app-based-e-portfolio-in-postgraduate-medical-education-using-entrustable-professional-activities-epa-framework-challenges-in-a-resource-limited-setting\/","name":"Development of an app-based e-portfolio in postgraduate medical education using Entrustable Professional Activities (EPA) framework: Challenges in a resource-limited setting - The Asia Pacific Scholar","isPartOf":{"@id":"https:\/\/medicine.nus.edu.sg\/taps\/#website"},"primaryImageOfPage":{"@id":"https:\/\/medicine.nus.edu.sg\/taps\/issues\/development-of-an-app-based-e-portfolio-in-postgraduate-medical-education-using-entrustable-professional-activities-epa-framework-challenges-in-a-resource-limited-setting\/#primaryimage"},"image":{"@id":"https:\/\/medicine.nus.edu.sg\/taps\/issues\/development-of-an-app-based-e-portfolio-in-postgraduate-medical-education-using-entrustable-professional-activities-epa-framework-challenges-in-a-resource-limited-setting\/#primaryimage"},"thumbnailUrl":"https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/09\/OA2459-Table-1-300x130.png","datePublished":"2021-09-13T06:36:14+00:00","dateModified":"2022-01-03T23:44:33+00:00","breadcrumb":{"@id":"https:\/\/medicine.nus.edu.sg\/taps\/issues\/development-of-an-app-based-e-portfolio-in-postgraduate-medical-education-using-entrustable-professional-activities-epa-framework-challenges-in-a-resource-limited-setting\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/medicine.nus.edu.sg\/taps\/issues\/development-of-an-app-based-e-portfolio-in-postgraduate-medical-education-using-entrustable-professional-activities-epa-framework-challenges-in-a-resource-limited-setting\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/medicine.nus.edu.sg\/taps\/issues\/development-of-an-app-based-e-portfolio-in-postgraduate-medical-education-using-entrustable-professional-activities-epa-framework-challenges-in-a-resource-limited-setting\/#primaryimage","url":"https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/09\/OA2459-Table-1.png","contentUrl":"https:\/\/medicine.nus.edu.sg\/taps\/wp-content\/uploads\/sites\/10\/2021\/09\/OA2459-Table-1.png","width":1767,"height":766},{"@type":"BreadcrumbList","@id":"https:\/\/medicine.nus.edu.sg\/taps\/issues\/development-of-an-app-based-e-portfolio-in-postgraduate-medical-education-using-entrustable-professional-activities-epa-framework-challenges-in-a-resource-limited-setting\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/medicine.nus.edu.sg\/taps\/"},{"@type":"ListItem","position":2,"name":"Development of an app-based e-portfolio in postgraduate medical education using Entrustable Professional Activities (EPA) framework: Challenges in a resource-limited setting"}]},{"@type":"WebSite","@id":"https:\/\/medicine.nus.edu.sg\/taps\/#website","url":"https:\/\/medicine.nus.edu.sg\/taps\/","name":"The Asia Pacific Scholar","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/medicine.nus.edu.sg\/taps\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/medicine.nus.edu.sg\/taps\/wp-json\/wp\/v2\/issues\/70990","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/medicine.nus.edu.sg\/taps\/wp-json\/wp\/v2\/issues"}],"about":[{"href":"https:\/\/medicine.nus.edu.sg\/taps\/wp-json\/wp\/v2\/types\/issues"}],"wp:attachment":[{"href":"https:\/\/medicine.nus.edu.sg\/taps\/wp-json\/wp\/v2\/media?parent=70990"}],"wp:term":[{"taxonomy":"issues_category","embeddable":true,"href":"https:\/\/medicine.nus.edu.sg\/taps\/wp-json\/wp\/v2\/issues_category?post=70990"},{"taxonomy":"archive_category","embeddable":true,"href":"https:\/\/medicine.nus.edu.sg\/taps\/wp-json\/wp\/v2\/archive_category?post=70990"},{"taxonomy":"issue_type","embeddable":true,"href":"https:\/\/medicine.nus.edu.sg\/taps\/wp-json\/wp\/v2\/issue_type?post=70990"},{"taxonomy":"volume_category","embeddable":true,"href":"https:\/\/medicine.nus.edu.sg\/taps\/wp-json\/wp\/v2\/volume_category?post=70990"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}