Table 3. Comparison of pre-CC OSCE (IS and GS respectively) and CBT results between the face-to-face (F-to-F) group and the online group

IV. DISCUSSION

We found no significant negative effects in some of the important educational outcomes in medical students’ scores of the medical interview due to online education caused by the COVID-19 pandemic. The quality of the medical interview after the emergence of the pandemic was no less than that before the pandemic. The same could be said for other indicators, including the CBT and other areas of the OSCE (data not shown).

One of the reasons why the scores of the online classes were not inferior to those of face-to-face classes might be because of the availability of each student to review the video recordings. We provided each student with a recording of their own performance in the class for self-review, which was not always provided in face-to-face classes. We also provided students with feedback from the teacher and other students during online classes. This is consistent with the findings of a previous study, which found that video reviewing of the OSCE performance is effective (Mookherjee et al., 2019). Moreover, the students accepted online classes well, and their motivation for learning was not affected despite the lack of face-to-face communication with simulated patients. During the reflection, some students noted that “I learned a lot in this class, though the class was held online” and that “I thought that online classes are not so bad” (data not shown). There were almost no complaints regarding online classes. We guess that in the environment that the face-to-face classes were restricted and the students experienced suspended classes the students felt satisfied for joining the classes even online. Further investigation will be needed regarding this point.

Recently, Khamees et al. (2022) pointed out the lack of control groups and poor transferability in numerous publications due to singularity of institution, department, and program. In the present study, the marks of students on high-stakes examinations before the pandemic were used as a comparison between face-to-face and online classes. Some studies have revealed that there are no significant differences in educational outcomes between face-to-face and online classes in basic medicine (Omole et al., 2023), and pharmacological education (Aoe et al., 2023). However, when it comes to high-stakes examinations, it remains unclear whether online education is not inferior to face-to-face education. Saad et al (2023) have showed that in some areas (clinical reasoning and history taking) of pre-clinical OSCE, students showed no less than comparable results, arguing that these skills are amenable to online learning in a medical school in Australia. Their results in some areas like medical interview in Japanese OSCE support our results. However, in their study, it is not clear about the details of the OSCE assessment, whether the assessment is by item-wise or global. In recent years, the pre-CC OSCE results have been recognized as an important educational outcome also for educational institutions (Hirsh et al., 2012). Our result, the educational outcome in the high-stakes examination, can be considered to fall in the top tier, the result, in the Kirkpatrick’s four-level model (Kirkpatrick, 1996). Moreover, our research is unique and important as few studies have directly compared the educational outcomes between face-to-face classes and online classes in high-stakes examination.

In Japan, the Medical Practitioners Act was revised in 2023, allowing medical students to perform some medical procedures under the supervision of a teaching physician after passing the pre-CC OSCE and CBT. This change also made the pre-CC OSCE a requirement for the national board examination. Hence, the pre-CC OSCE in Japan has become even more important, as much the responsibility for the education even greater. Our results show that online classes can contribute to the practice of “Medical interviews”.

It must be noted that online classes are not a complete alternative to face-to-face classes. Many studies have indicated that online education has some negative aspects (e.g., the need for infrastructure and devices, high cost, lack of personal interaction, etc.) (Arja et al., 2022; Mortazavi et al., 2021; Shaiba et al., 2023). One of the most significant elements that are difficult to teach in online classes is nonverbal communication. However, as Ishikawa et al. (2010) reported, although students are capable of understanding nonverbal communication despite struggling to change their performance through educational intervention, it is well recognized that nonverbal communication is difficult to teach even in face-to-face classes. Additionally, when it comes to procedural skills such as venipuncture, the educational outcomes in the online learning group were inferior to that of face-to-face learning group and students also felt that they were not taught satisfactorily (Dost et al., 2020; Saad et al., 2023). We should keep in mind that online education does not fully replace face-to-face education.

We saw a lack of significant differences in GS, both in year-by-year comparison and comparison between face-to-face and online groups. Although the tasks allocated to each university by CATO differ every year, the checkpoints are essentially common in quite a few areas; so, a comparison was made for both year by year and before and after the pandemic. GS usually reflects holistic assessment, which is difficult to produce results via technical education, whereas it might be easier for learners and teachers to deal with item-wise assessment (Govaerts et al., 2011; Jonsson & Svingby, 2007; Sadler, 2009). Moreover, in online classes, we used a checklist of the students’ performance (not shown to the students, but comments were given according to the checklist), which might have contributed to the improvement of IS. As to CBT, it is standardized by the accumulated examinations and the Item Response Theory and is assessed basically by knowledge base. The educational strategies that mainly should impact on the assessment of CBT, based on the lecture, have not changed before and after the pandemic, in the face-to-face classes or online classes. During the pandemic, the extra-curricular activities of the students were restricted, and several articles argue that self-studying time of the students increased (Barton et al., 2021; Guluma & Brandl, 2023). These might have contributed to the smaller elevation of CBT-IRT than the IS of pre-CC OSCE.

We need to take into consideration the confounding of several factors such as the curriculum changes, instructor training, student characteristics and students’ self-study time. While online classes have been a change in the curriculum, the rest remains unchanged. The instructors and students needed to become familiar with online classes, but there was no change in the educational goals of the class itself. Additionally, in the first year of the pandemic in 2020, we were unable to get enough data and the online class itself was implemented as “being built”. By 2021 and 2022, the class was almost stable. However, getting used to online classes of the instructors and students could be a confounding factor. The class tool (Zoom®) was continuously improved, which might be a minor confounding factor. Additionally, the students might have had excellent ITC skills, which might also be a confounding factor.

Some frameworks describing the evidence of online medical education outcome might contribute to generalizing our results (Martinengo et al., 2024; McGee et al., 2024; Wilcha, 2020). Needless to say, there are confrontations regarding the limitations of these generalizations, pointing out the context-depending factors, high heterogeneity among studies and “The Covid-19 Effects” (Abdull Mutalib et al., 2022; Liu et al., 2016; Martinengo et al., 2024; McGee et al., 2024). However, these frameworks will be applicable in generalizing our results; although there are some potential confounding factors such as the students, the instructors, the educational resources and the “pandemic era” itself, the online medical interview education could be an effective educational curriculum for educating medical interview skills as well as some interpersonal skills.

A. Limitations

One of the limitations of this study is that it was performed in a single institution. Hence, the generalizability of this study may be lower than that of multi-institutional studies. However, not much variation exists in the nature of students and in the educational curriculum they experience. Of course, to make the evidence more robust and further validate, multi-centred or multi-institutional studies are still needed. At the same time, these factors should not be too disparate as it is very difficult to find a suitable population for these factors. In this regard, our participants and classes can be considered as a reasonable population. 

Another limitation of this study is the number of participants. In 2020, we could not obtain enough participants because of the pandemic. After the pandemic, our students and staff shifted to online classes, and the number of face-to-face classes decreased. In 2021 and 2022, we decided to obtain consent for participation in this study in the waiting time after face-to-face OSCE examination as it was difficult to obtain consent only during online classes. In this context, the participants may have a positive view of various aspects of student life including studies, which may be a potential bias of sample population. Additionally, the waiting time after the OSCE examination was short for some students, which might have made it difficult to think about understanding the concept of this research and whether consent should be given.

In this study, qualitative analyses investigating if the students were positive about the classes are limited to some extent. During the pandemic, the psychological situation and the learning behaviour of the students might have differed from that of before the pandemic. To investigate this aspect, qualitative studies will be needed. 

Moreover, the evaluation criteria of the pre-CC OSCE are not open to the public due to CATO policy. This probably leads to a lack of transparency of the evaluation, causing another limitation of this study. However, Japanese OSCE evaluation criteria is similar to CANMED’s OSCE checklist (Kassam et al., 2016), which will support the validity of the results of Japanese OSCE and our results.

Finally, this study retrospectively compared the educational effect between students before and after the pandemic, which may limit the causal inferences of educational outcome effects of face-to-face versus online in medical interview OSCE. A randomized controlled trial will be needed to verify the results obtained in this study.

V. CONCLUSION

Because of the COVID-19 pandemic, we were forced to change our medical interview classes from face-to-face to be online. However, in high-stakes examinations such as the pre-CC OSCE and CBT, the results of the online group were not inferior to those of the face-to-face group. We consider this result extremely important because we directly compared the educational outcomes of high-stakes examinations between online and face-to-face groups who took the same medical interview classes, and because this evaluation falls in the top level of Kirkpatrick’s model. Our results suggest that online education provides a viable option in teaching interpersonal skills and support the integration of online medical interview training into preclinical curricula, particularly in resource-constrained settings. Randomized controlled trials and multi-institutional studies are needed to further validate our results.

Notes on Contributors

SH and ME conducted the whole research.

SH, MI, SK, JM, HM, and ME performed the classes and collected the data.

SH performed data analyses.

SH, MI, SK, JM, HM, and ME contributed to writing the manuscript.

Ethical Approval

This study was approved by the ethics committee of the UTokyo Faculty of Medicine (Approval No. 2021005NI). All methods were performed in accordance with the relevant guidelines and regulations. Moreover, the data of students who provided consent to the secondary use of their data in another research, given by the ethics committee of the UTokyo Faculty of Medicine (Approval No. 11763), were included.

Data Availability

The data in this study are not publicly available because of confidentiality agreements with the participants, conditions obligating CATO, and confidential nature of the data.

Acknowledgement

We thank the students of the UTokyo Medical School who participated in this study. We also thank the UTokyo Staff for their cooperation.

Funding

This study was funded by JSPS KAKENHI Grant Number 24K06092 and ACRO incubation grants of Teikyo University.

Declaration of Interest

The authors have no potential conflicts to disclose.

References

Abdull Mutalib, A. A., Md. Akim, A., & Jaafar, M. H. (2022). A systematic review of health sciences students’ online learning during the COVID-19 pandemic. BMC Medical Education, 22(1), 524. https://doi.org/10.1186/s12909-022-03579-1

Alshaibani, T., Almarabheh, A., Jaradat, A., & Deifalla, A. (2023). Comparing online and face-to-face performance in scientific courses: A retrospective comparative gender study of Year-1 students. Advances in Medical Education and Practice, 14, 1119–1127. https://doi.org/10.2147/amep.s408791

American Medical Association. (2016, June 15). AMA encourages telemedicine training for medical students, residents. AMA Press Releases.

Aoe, M., Esaki, S., Ikejiri, M., Ito, T., Nagai, K., Hatsuda, Y., Hirokawa, Y., Yasuhara, T., Kenzaka, T., & Nishinaka, T. (2023). Impact of different attitudes toward face-to-face and online classes on learning outcomes in Japan. Pharmacy, 11(1), 16. https://doi.org/10.3390/pharmacy11010016

Arja, S. B., Fatteh, S., Nandennagari, S., Pemma, S. S. K., Ponnusamy, K., & Arja, S. B. (2022). Is emergency remote (online) teaching in the first two years of medical school during the COVID-19 pandemic serving the purpose? Advances in Medical Education and Practice, 13, 199–211. https://doi.org/10.2147/amep.s352599

Barton, J., Rallis, K. S., Corrigan, A. E., Hubbard, E., Round, A., Portone, G., Kuri, A., Tran, T., Phuah, Y. Z., Knight, K., & Round, J. (2021). Medical students’ pattern of self-directed learning prior to and during the coronavirus disease 2019 pandemic period and its implications for Free Open Access Meducation within the United Kingdom. Journal of Educational Evaluation for Health Professions, 18, 5. https://doi.org/10.3352/jeehp.2021.18.5

Bastos, R. A., Carvalho, D. R. D. S., Brandão, C. F. S., Bergamasco, E. C., Sandars, J., & Cecilio-Fernandes, D. (2021). Solutions, enablers and barriers to online learning in clinical medical education during the first year of the COVID-19 pandemic: A rapid review. Medical Teacher, 44(2), 187–195. https://doi.org/10.1080/0142159x.2021.1973979

Basuodan, R. (2024). Comparisons of the academic performance of Medical and Health-Sciences students related to three learning methods: A cross-sectional study. Advances in Medical Education and Practice, 15, 1339–1347. https://doi.org/10.2147/amep.s493782

Budakoğlu, I. İ., Sayılır, M. Ü., Kıyak, Y. S., Coşkun, Ö., & Kula, S. (2021). Telemedicine curriculum in undergraduate medical education: a systematic search and review. Health and Technology, 11(4), 773–781. https://doi.org/10.1007/s12553-021-00559-1

Common Achievement Tests Organization (CATO). (2005). Common Achievement Tests Organization. https://www.cato.or.jp/index.html

Crawford, J., Butler-Henderson, K., Rudolph, J., Malkawi, B., Glowatz, M., Burton, R., Magni, P. A., & Lam, S. (2020). COVID-19: 20 countries’ higher education intra-period digital pedagogy responses. Journal of Applied Learning & Teaching, 3(1). https://doi.org/10.37074/jalt.2020.3.1.7

Daniel, M., Gordon, M., Patricio, M., Hider, A., Pawlik, C., Bhagdev, R., Ahmad, S., Alston, S., Park, S., Pawlikowska, T., Rees, E., Doyle, A. J., Pammi, M., Thammasitboon, S., Haas, M., Peterson, W., Lew, M., Khamees, D., Spadafore, M., … Stojan, J. (2021). An update on developments in medical education in response to the COVID-19 pandemic: A BEME scoping review: BEME Guide No. 64. Medical Teacher, 43(3), 253–271. https://doi.org/10.1080/0142159x.2020.1864310

Dost, S., Hossain, A., Shehab, M., Abdelwahed, A., & Al-Nusair, L. (2020). Perceptions of medical students towards online teaching during the COVID-19 pandemic: A national cross-sectional survey of 2721 UK medical students. BMJ Open, 10(11), e042378. https://doi.org/10.1136/bmjopen-2020-042378

Gordon, M., Patricio, M., Horne, L., Muston, A., Alston, S. R., Pammi, M., Thammasitboon, S., Park, S., Pawlikowska, T., Rees, E. L., Doyle, A. J., & Daniel, M. (2020). Developments in medical education in response to the COVID-19 pandemic: A rapid BEME systematic review: BEME Guide No. 63. Medical Teacher, 42(11), 1202–1215. https://doi.org/10.1080/0142159x.2020.1807484

Govaerts, M. J. B., Schuwirth, L. W. T., Van Der Vleuten, C. P. M., & Muijtjens, A. M. M. (2010). Workplace-based assessment: Effects of rater expertise. Advances in Health Sciences Education, 16(2), 151–165. https://doi.org/10.1007/s10459-010-9250-7  

Guetterman, T. C., Sakakibara, R., Baireddy, S., Kron, F. W., Scerbo, M. W., Cleary, J. F., & Fetters, M. D. (2019). Medical students’ experiences and outcomes using a virtual human simulation to improve communication skills: Mixed methods study. Journal of Medical Internet Research, 21(11), e15459. https://doi.org/10.2196/15459

Guluma, K. Z., & Brandl, K. (2022). Virtual learning allows for adaptation of study strategies in a cohort of U.S. medical students. Medical Teacher, 45(1), 89–96. https://doi.org/10.1080/0142159x.2022.2105690

Hammersley, V., Donaghy, E., Parker, R., McNeilly, H., Atherton, H., Bikker, A., Campbell, J., & McKinstry, B. (2019). Comparing the content and quality of video, telephone, and face-to-face consultations: A non-randomised, quasi-experimental, exploratory study in UK primary care. British Journal of General Practice, 69(686), e595–e604. https://doi.org/10.3399/bjgp19x704573

Harden, R. M., Stevenson, M., Downie, W. W., & Wilson, G. M. (1975). Assessment of clinical competence using objective structured examination. BMJ, 1(5955), 447–451. https://doi.org/10.1136/bmj.1.5955.447

Hirsh, D., Gaufberg, E., Ogur, B., Cohen, P., Krupat, E., Cox, M., Pelletier, S., & Bor, D. (2012). Educational outcomes of the Harvard Medical School–Cambridge integrated clerkship. Academic Medicine, 87(5), 643–650. https://doi.org/10.1097/acm.0b013e31824d9821

Horita, S., Park, Y., Son, D., & Eto, M. (2021). Computer-based test (CBT) and OSCE scores predict residency matching and National Board assessment results in Japan. BMC Medical Education, 21(1). https://doi.org/10.1186/s12909-021-02520-2

Ishikawa, H., Hashimoto, H., Kinoshita, M., & Yano, E. (2010). Can nonverbal communication skills be taught? Medical Teacher, 32(10), 860–863. https://doi.org/10.3109/01421591003728211

Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity, and educational consequences. Educational Research Review, 2(2), 130–144. https://doi.org/10.1016/j.edurev.2007.05.002

Kassam, A., Cowan, M., & Donnon, T. (2016). An objective structured clinical exam to measure intrinsic CanMEDS roles. Medical Education Online, 21(1), 31085. https://doi.org/10.3402/meo.v21.31085

Khamees, D., Peterson, W., Patricio, M., Pawlikowska, T., Commissaris, C., Austin, A., Davis, M., Spadafore, M., Griffith, M., Hider, A., Pawlik, C., Stojan, J., Grafton-Clarke, C., Uraiby, H., Thammasitboon, S., Gordon, M., & Daniel, M. (2022). Remote learning developments in postgraduate medical education in response to the COVID-19 pandemic: A BEME systematic review: BEME Guide No. 71. Medical Teacher, 44(5), 466–485. https://doi.org/10.1080/0142159x.2022.2040732

Kirkpatrick, D. (1996). Great ideas revisited. Techniques for evaluating training programs. Revisiting Kirkpatrick’s Four-Level model. Training & Development, 50(1), 54–59. https://eric.ed.gov/?id=EJ515660

Liu, Q., Peng, W., Zhang, F., Hu, R., Li, Y., & Yan, W. (2016). The effectiveness of blended learning in health professions: Systematic review and meta-analysis. Journal of Medical Internet Research, 18(1), e2. https://doi.org/10.2196/jmir.4807

 Martinengo, L., Ng, M. S. P., De Rong Ng, T., Ang, Y., Jabir, A. I., Kyaw, B. M., & Car, L. T. (2024). Spaced digital education for health professionals: A systematic review and meta-analysis (Preprint). Journal of Medical Internet Research, 26, e57760. https://doi.org/10.2196/57760

McGee, R. G., Wark, S., Mwangi, F., Drovandi, A., Alele, F., & Malau-Aduli, B. S. (2024). Digital learning of clinical skills and its impact on medical students’ academic performance: A systematic review. BMC Medical Education, 24(1). https://doi.org/10.1186/s12909-024-06471-2

Mookherjee, S., Strujik, J., Cunningham, M., Kaplan, E., & Çoruh, B. (2018). Independent and mentored video review of OSCEs. The Clinical Teacher, 16(1), 23–29. https://doi.org/10.1111/tct.12755

Mortazavi, F., Salehabadi, R., Sharifzadeh, M., & Ghardashi, F. (2021). Students’ perspectives on the virtual teaching challenges in the COVID-19 pandemic. Journal of Education and Health Promotion, 10(1), 59. https://doi.org/10.4103/jehp.jehp_861_20

Omole, A. E., Villamil, M. E., & Amiralli, H. (2023). Medical education during COVID-19 pandemic: A comparative effectiveness study of face-to-face traditional learning versus online digital education of basic sciences for medical students. Cureus. https://doi.org/10.7759/cureus.35837

Saad, S., Richmond, C., King, D., Jones, C., & Malau-Aduli, B. (2023). The impact of pandemic disruptions on clinical skills learning for pre-clinical medical students: Implications for future educational designs. BMC Medical Education, 23(1). https://doi.org/10.1186/s12909-023-04351-9 

Sadler, D. R. (2008). Indeterminacy in the use of preset criteria for assessment and grading. Assessment & Evaluation in Higher Education, 34(2), 159–179. https://doi.org/10.1080/02602930801956059

Shaiba, H., John, M., & Meshoul, S. (2022). Female Saudi College students’ e-learning experience amidst COVID-19 pandemic: An investigation and analysis. Heliyon, 9(1), e12768. https://doi.org/10.1016/j.heliyon.2022.e12768

Wilcha, R. (2020). Effectiveness of virtual medical teaching during the COVID-19 Crisis: Systematic review. JMIR Medical Education, 6(2), e20963. https://doi.org/10.2196/20963

Zaccariah, Z. R., Irvine, A. W., & Lefroy, J. E. (2022). Feasibility study of student telehealth interviews. The Clinical Teacher, 19(4), 308–315. https://doi.org/10.1111/tct.13490

*Shoko Horita
2-11-1, Kaga, Itabashi-ku,
Tokyo 173-8605, Japan
Email: horitas-tky@umin.ac.jp