Virtual assessment in Physiotherapy: Examining perceptions, constructs, and tool characteristics

  • Share
Print Article   Download PDF

Number of Citations: 0

Submitted: 20 April 2023
Accepted: 19 October 2023
Published online: 2 April, TAPS 2024, 9(2), 70-80
https://doi.org/10.29060/TAPS.2024-9-2/OA3044

Xiang Ren Tan1,2, Anthony J Goff1 & Li Whye Cindy Ng1,3

1Health and Social Sciences, Singapore Institute of Technology, Singapore; 2Human Potential Translational Research Programme, Yong Loo Lin School of Medicine, National University of Singapore, Singapore; 3Department of Physiotherapy, Singapore General Hospital, Singapore

Abstract

Introduction: Objective structured clinical examinations have traditionally been used to assess clinical skills. However, these face-to-face clinical assessments were hindered by physical and social restrictions imposed during COVID-19. This created a need to develop novel approaches for reliable assessment of clinical skills. We explored and evaluated a virtual exercise teleconsultation assessment (VETA) to replicate a teleconsultation setting where physiotherapy students were assessed on exercise prescription and coaching skills.

Methods: We conducted a cross-sectional mixed-methods study using convenience sampling. A VETA was implemented for 172 physiotherapy students via Zoom to allow synchronous interaction with standardised patients (SPs). 67 students and 9 SPs completed two separate post evaluation surveys on themes relating to administration, support, authenticity, effectiveness, and value of the VETA. Likert-type responses were categorised as positive, neutral, or negative while coded qualitative responses were consolidated into themes by inductive content analysis.

Results: 76% of students agreed that the assessment was authentic while 93% felt that the SPs were realistic and believable. Responders also highlighted important challenges including the limited camera viewing angle, time and space constraint, adequacy of equipment and reliability of connectivity. Exploratory factor analysis of responses revealed three latent constructs: (1) clarity of assessment, (2) clinical relevance, and (3) value of assessment.

Conclusion: Despite the technical challenges, VETA demonstrated clarity and value as an alternative assessment and showed relevance to future telehealth practice, which is increasingly pervasive in clinical settings. This paper demonstrates a feasible approach for the virtual assessment of clinical competencies.

Keywords:           Health Sciences Education, Virtual Assessment, Clinical Skills, OSCE, Physiotherapy, Exercise Prescription

Practice Highlights

  • The ability to assess students non-obtrusively allows for fair assessment with reduced anxiety.
  • Despite several technical challenges, VETA demonstrated value as an alternative assessment.
  • An appropriate context of use, content validity and authenticity of virtual assessment are crucial.
  • VETA format may be used for distant learning and remote assessment of clinical competency.
  • Virtual care delivery should be included as part of healthcare professionals’ formal training.

I. INTRODUCTION

Objective Structured Clinical Examinations (OSCEs) have traditionally been used to assess clinical competencies and skills important for registration, licensing, and certification with professional bodies. OSCEs were first developed by Harden (Harden et al., 1975) and have commonly been used as an assessment tool for clinical competency in both medical (Gormley, 2011; Jo & An, 2014; Shirwaikar, 2015) and allied health professions (Barra-Ortiz et al., 2021; Edwards & Martin, 1989; Silva et al., 2011; Wessel et al., 2003). OSCEs can be conducted as a formative or summative assessment (Chisnall et al., 2015) and have been described as the gold standard for clinical assessment (Norman, 2002) due to its objective, precise, and reproducible nature which allows for standardised testing across a wide range of clinical skills (Zayyan, 2011). OSCEs can strengthen links between theory and practice, improving the professionalism and readiness of allied health students to perform clinical work (Farahat et al., 2015). Hence, OSCEs form vital assessments in structured health sciences programs.

However, the recent COVID-19 pandemic with strict social distancing measures and lockdowns have posed great challenges to the delivery of teaching and authentic assessments (Kumar et al., 2021; Liang et al., 2020). This is especially pertinent to health sciences education where the use of OSCEs has been limited by physical and social restrictions imposed in response to COVID-19. Therefore, there was a need to explore novel ways to create authentic settings for objective, precise and reproducible assessments of clinical skills, despite the lack of physical contact. Virtual assessments, including virtual OSCEs, have been widely implemented across many educational institutions due to COVID-19 lockdown and restrictions (Pettit et al., 2021). Importantly, virtual OSCEs appear to be effective assessments for clinical skills, notwithstanding some limitations, in medical and pharmacy courses (Blythe et al., 2021; Deville et al., 2021; Grover et al., 2022; Hannan et al., 2021; Major et al., 2020; Prettyman et al., 2018). Virtual OSCEs have been performed using common web-based teleconferencing platform such as Zoom due to its familiarity, convenient features, and stability (Grover et al., 2022; Hannan et al., 2021; Major et al., 2020; Prettyman et al., 2018). Remote proctoring and lock-down browsers can also be employed in virtual OSCEs to ensure integrity of assessment (Deville et al., 2021). However, the conduct of virtual OSCEs varies across institutions and there are no standardised guidelines or format to date.

Virtual OSCEs can be useful to assess some but not all clinical skills required of allied health students ranging from health assessment, diagnostic skills, physical and manual therapy to patient education. During the pandemic, we conceptualised an online synchronous approach, Virtual Exercise Teleconsultation Assessment (VETA), to assess the exercise prescription and coaching skills in undergraduate physiotherapy students (Tan & Ng, 2022). The intention is to replicate a teleconsultation setting where students remotely interact with standardised patients (SPs) to prescribe and coach exercises. Teleconsultation has been defined as “synchronous or asynchronous consultation using information and communication technology to omit geographical and functional distance” (Deldar et al., 2016). With the rapid and drastic changes in COVID-19 restrictions, VETA was designed with streamlined focus on three key guiding principles: (1) validity and authenticity, (2) ease of conduct, and (3) safety. These principles are aligned to the educational principles of OSCE set out in the AMEE guide (Khan et al., 2013) encompassing construct validity, reliability (with the use of rubrics, experienced examiners, and trained SPs), feasibility, and educational impact (relating to authenticity).

To ensure (1) assessment validity and authenticity, the use of standardised patients was retained in the assessment process, as opposed to replacing with case scenarios only or role-play by faculty. The live interactions with trained SPs promote realism and allow students to navigate the complexities of effective patient communication (Lovink et al., 2021; Rickles et al., 2009). The use of trained SPs also enhances the consistency and reliability of testing and at the same time, provides a safe, learner-centred environment (Cleland et al., 2009). Simulating a real clinical setting, the students interacted with the SPs independently without visible presence of examiners. To provide a uniform examination environment, VETA was entirely conducted on campus instead of relying on home-based virtual assessment.

In addition, emphasis has been placed on the (2) ease of conduct where Zoom teleconferencing was used as this is familiar to both the faculty and students who have used the platform extensively for online learning purposes. All assessment-related resources including laptop and exercise equipment were duly provided. This helped to eliminate any potential test anxiety arising from inexperience with the use of new technology or unfamiliar teleconferencing platform. Lastly, the (3) safety of students, SPs and examiners was prioritised under the COVID-19 climate. Students and SPs resided in different parts of the campus during the assessment, and the students followed a prearranged schedule for the exam to avoid any physical interactions. All personnel wore their masks at all times, even when interacting in Zoom. The SPs and examiners were situated in the same room during the assessment, with the examiner seated a safe distance away from the SP while allowing observation. These measures ensured the least interactions across all groups and minimised risk of disease transmission.

VETA was implemented as a summative assessment in an exercise physiology module within the BSc Physiotherapy programme at Singapore Institute of Technology to assess the exercise prescription and coaching skills of physiotherapy students in a simulated teleconsultation setting. We aim to evaluate the perceptions and overall experience of students and SPs towards VETA as a virtual assessment to evaluate its potential strengths and limitations. We further explored latent constructs from students’ responses to understand our survey tool’s characteristics and its reliability in evaluating themes in clinical assessments. In this study, we examined the feasibility and value of an online synchronous approach for assessing clinical skills critical to allied health students, which may provide key insights on the future transformation of clinical skills assessments.

II. METHODS

A. VETA Procedures

As part of the necessary measures to adopt new assessment methods during the COVID-19 pandemic, a convenience sample of Year 1 B.Sc Physiotherapy students underwent a VETA in July 2021. All exam procedures were conducted at the Singapore Institute of Technology Dover campus. Students and SPs situated in two different venues where separate laptops have been set up to connect on Zoom platform for facilitating interactions across the two venues.

Students were instructed to report to the exam venue at their allocated timing where they were ushered to a reading station for 10 minutes. A case scenario was provided for students to conduct planning of exercise prescription for the SP (representing the same patient in the case scenario). After the reading station, the students proceeded to an enclosed room with the laptop connected to the SP. They were given a total of 10 minutes to interact with the SP to (1) conduct Physical Activity Readiness Questionnaire (PAR-Q) to ascertain the patient’s fitness to exercise, (2) provide a cycling exercise prescription and instruct him/her how to use a cycling ergometer bike, and (3) provide a resistance exercise prescription and coach them on how to perform the stated exercise safely. For the latter, similar resistance exercise equipment (resistance bands and dumbbells of different weights) was provided in both venues to facilitate the coaching of exercise. The amount of time left in the assessment (e.g., 5 minutes left), and the end of the assessment were indicated by different ringing bells.

SPs were allowed to seek clarification on the exercise prescription or coaching as necessary. The examiner was seated in the same room as the SP but was out-of-sight from the laptop web camera viewing angle. This is to replicate an authentic teleconsultation environment during the assessment. As compared to other similar virtual OSCE (Grover et al., 2022; Major et al., 2020), VETA goes beyond basic patient communication and consultation as it additionally includes an intervention aspect where SPs follow students’ instructions to perform a prescribed exercise safely.

Students’ performance was evaluated by three trained examiners (teaching faculty) who were experienced assessors for face-to-face OSCE for at least three consecutive years. All examiners used a standardised marking schema which was modelled closely to the existing schema used for physical OSCEs to maintain the same rigour and intent in the assessment of clinical skills.

B. Evaluation of VETA

After the assessment, the students and SPs completed two separate self-administered questionnaires. The students completed the survey on the Qualtrics® platform while SPs were provided with hardcopy forms to fill. Both mixed-methods surveys consist of 5-point Likert-type questions, and open-ended questions to gather qualitative responses. Participation in the surveys was voluntary and anonymous.

Initial design of the surveys was modelled after a recent paper (Fouad et al., 2019) that has conducted similar survey-based evaluation of students, staff and SPs’ perceptions towards an OSCE. After refinement of themes relevant to our VETA format, the student’s survey contained 26 items (23 Likert-type, 3 open-ended) which evaluated their overall experience and perceived difficulty of VETA, and five themes including the (1) administration, (2) support or resources provided, (3) authenticity, (4) effectiveness, and (5) value of VETA as a novel assessment mode. The open-ended questions gathered students’ qualitative inputs on their challenges faced, opinions on effectiveness of VETA and the potential areas of improvement. The SP’s survey contained 10 items (6 Likert-type, 4 open-ended) evaluating their overall experience and three themes including (1) administration, (2) student performance and (3) acceptability of VETA. The open-ended questions gathered SPs’ qualitative inputs on their challenges faced and the potential areas of improvement.

C. Statistical Analysis

Survey responses were coded and exported from Qualtrics® platform or manually input into Microsoft Excel (Microsoft Corporation). For 5-point Likert-type questions, responses such as “strongly agree/disagree” and “somewhat agree/disagree” were combined during analysis to represent a consensus perception (agree, neutral or disagree) towards a subject statement. Descriptive statistics of survey responses in percentages was compiled and displayed in summary tables. Individual open-ended responses were condensed into codes and eventually consolidated into common categorical themes using the inductive content analysis approach (Elo & Kyngäs, 2008; Hsieh & Shannon, 2005). Categorical themes were identified from the codes and were subsequently grouped into two overarching themes of VETA strengths and limitations.

Exploratory factor analysis was performed using SPSS software Version 20 to explore the underlying latent conceptual structure (Watkins, 2018) present in the student’s responses to the survey variables. This allows the assessment of congruency between the pre-identified themes in the survey and the latent constructs. Factor extraction was performed with Principal Axis Factoring analysis with varimax rotation. There were no missing data. Criteria on factor extraction was based on the Kaiser rule with eigenvalues > 1.0. Only subsets of unique factors with individual loadings above 0.4 were retained. Internal consistency of the items was assessed using Cronbach’s alpha coefficient with an alpha value between 0.7 and 0.9 representing a measure of satisfactory internal consistency (Downing, 2004).

III. RESULTS

A total of 172 students completed the VETA. 67 students (39% of cohort) and 9 SPs (100%) completed the respective surveys. Out of all responders, 93% of the students and all SPs expressed an overall positive experience for VETA. 60% of the students felt that the assessment was of moderate difficulty. 22% of the students have expressed that VETA was somewhat difficult while 17% felt it was somewhat easy, with 1 student (1%) responded that it was extremely easy.

A. Perceptions Towards VETA

A summary of response statistics for Likert-type items can be found in Table 1. More than 90% of students agreed that the instructions to perform each activity were clear and that they were aware of the assessment requirements. However, only 77% agreed that the time allocated for teleconsultation was sufficient, with 16% who felt that the time was insufficient. The majority of the students were satisfied with the assessment environment and Zoom setup, however 10% felt that resources and equipment provided were inadequate.

76% of them agreed that the assessment was an authentic reflection of clinical setting while 93% of students felt that the SPs were realistic and believable. More than 90% of the students agreed that VETA was a fair and valid assessment. Most students (79%) concurred that VETA should remain as a form of assessment. 61% of the students agreed that VETA offered more learning opportunities than other exams, with 11% who disagreed with the statement.

All SPs agreed that the administration of VETA (briefing instructions and Zoom setup) was adequate, and they were comfortable with the new VETA format. However, only 78% agree that instructions given by students were clear and they were able to follow the instructions without difficulty. 89% of the SPs felt confident to perform the activities independently with the given instructions. This suggests the inadequacy of some students to effectively communicate, prescribe and coach exercises to the SPs.

Items (For Students; n=67)

Agree
(%)

Neutral
(%)

Disagree
(%)

Themes

1. Briefing instructions and information given prior to the assessment were helpful.

88

8

4


Administration

2. Instructions to perform each activity were clear.

91

3

6

3. I was aware of the information needed for each task in the assessment.

93

3

4

4. Tasks asked to perform were fair.

97

2

1

5. Time allocated for reading and planning (10 min) was sufficient.

94

6

0

6. Time allocated for teleconsultation (10 min) was sufficient.

78

6

16

7. The environment where the assessment was conducted was conducive.

90

4

6

Support

8. The audio/video quality of the Zoom session was adequate.

94

3

3

9. Resources and equipment provided for the assessment were adequate.

82

8

10

10. The standardised “patients” were believable and realistic.

93

3

4

Authenticity

11. Settings and context of given case scenario were authentic.

97

1

2

12. The interaction with the standardised patient was authentic.

91

5

4

13. VETA without the presence of an assessor was helpful.

90

9

5

14. The VETA was an authentic reflection of clinical setting.

76

19

5

15. The VETA helped to identify gaps in knowledge.

85

12

3

Effectiveness

16. The VETA helped to identify weaknesses in communication and patient-care skills.

93

6

1

17. The VETA tested appropriate skills required for a Physiotherapist.

87

9

4

18. The VETA covered a wide knowledge range.

88

9

3

19. The VETA should remain as a form of assessment.

79

15

6

Value

20. The VETA offered more learning opportunities than other exams.

61

28

11

21. The VETA was a valuable practice and learning experience.

93

3

4

Items (For Standardised Patients; n=9)

 

 

 

Themes

1. Briefing instructions given before the assessment were helpful and sufficient.

100

0

0

Administration

2. The quality of the audio/video was adequate.

100

0

0

3. Instructions given by students to perform each activity were clear and I was able to follow the instructions without difficulty.

78

22

0

Student
Performance

4. I am confident to perform the activities independently with the given instructions.

89

11

0

5. Communication by the students were appropriate and professional.

100

0

0

6. I am comfortable to receive instructions from a teleconsult session by a student in an exam setting.

100

0

0

Acceptability

 

Table 1. Response frequency for Likert-type questions

B. Strengths and Limitations

From the inductive content analysis of qualitative responses (Table 2), the VETA strengths were (1) reduced student anxiety, (2) relevance to practice and (3) assessment effectiveness. These were identified from students’ responses where they expressed that having the examiner off-screen helps to reduce their level of anxiety during assessment and they felt that VETA was relevant to their future practice where they may be required to conduct telehealth or telerehabilitation sessions. One key highlight is the authenticity of VETA where a student mentioned that he or she felt like a real physiotherapist giving the prescription, which affirmed that VETA replicates an authentic teleconsultation setting.

The limitations were identified to be related to (1) time and space allocation, (2) environment & equipment provided, (3) assessment effectiveness and (4) technical issues. There were similar sentiments from the SPs and students that the space provided for performing exercise was constrained. Students preferred more time allocated for teleconsultation. Furthermore, the students hoped to have visual feedback on timing during assessment and more exercise equipment to be provided. For assessment effectiveness, some students expressed that more clarity can be provided regarding the assessment rubric and expectations. Some of them felt that VETA was not reflective of a physical clinical setting. Last but not least, there were technical issues faced including the muffling of voices due to mask-wearing, connectivity issues that led to delays, and limited viewing angle of the laptop camera.

Themes

Sub Themes

VETA
 Strengths

Reduced Student Anxiety

“Having the assessor off-screen and not physically beside me really helped me focus on what I wanted to deliver to the patient. It also makes me feel less nervous and anxious and it has definitely helped me perform better”

Students’
quotes

“Don’t see any invigilator, not so scary”

Relevance to Practice

“It is a good alternative assessment especially during Covid or future epidemics…open another option for both clients and PTs to conduct sessions remotely”

“It is good to train us in telehealth because it gives us the opportunity to help patients who may have difficulty coming to the clinic even after the pandemic”

“VETA was a great innovation with COVID-19… we might have to do telerehab for our patient and this would actually equip us with the skills to do so”

Assessment Effectiveness

“Felt that the zoom VETA was well organised and helped us cover the main objective of the module”

“It is an effective examination tool as it is going to be a very practical thing to do in the near future”

“The patients were great. The fact that I’ve never seen them before, coupled with how they responded and interacted with me during the VETA made this “consultation” more believable and I really felt like a physiotherapist giving an exercise prescription”

VETA
Limitations

Time & Space Allocation

“More time can be given to have more interactions with the patient. It felt like i was just instructing the patient without time for the patient to reply back.”

Students’
quotes

“Would have liked more space to move backwards to demo exercise to patient, but chair was a bit in the way”

“Maybe allocate more space in student’s room so that if they need to move in order to demo…. they are not restricted by the room size”

SPs’

quotes

“I have to adjust a few times in order to get my whole body into the screen for the candidate to see properly and complete the task assigned”

Environment & Equipment

“A clock timer to keep track of time would have been helpful”

Students’
quotes

“Put a timer so that students are able to track how much time they have left. the bell was quite soft and some students might have missed it”

“The environment can be slightly better as it was rather dark and the papers were messy”

“More weights to cater exercises that require two arms”

Assessment Effectiveness

“Would be nice if rubric was released earlier”

Students’
quotes

“Would be beneficial to give a sample case study…so that we have a clearer expectation of the exam”

“…resistance training component of Zoom VETA was limited to upper limb exercises only”

“It does not feel reflective of a real clinical setting as physically, we would have to consider many other factors such as conducting proper handrub before and after the activity, cleaning the equipment as well”

Table 2. Qualitative responses for perceptions towards VETA

C. Common Constructs in Survey Evaluation

From the original questionnaire (21 Likert-type items categorised into 5 themes), a smaller subset of 13 items were retained after factor analysis and they were categorised and defined as 3 main latent constructs: (1) Clarity in assessment, (2) Clinical relevance, and (3) Value of assessment. The loadings of individual factor (>0.4) and the Cronbach’s alpha coefficient (>0.7) are displayed in Table 3. It was noted that some of the items previously grouped under the topic of “effectiveness” has been subsumed under either the clinical relevance or value of assessment constructs. Other items were omitted as their loadings were <0.4 or they were not unique in representing any of the latent constructs.

Items (n=13)

C1

C2

C3

Constructs

1. Briefing instructions and information given prior to the assessment were helpful.

0.743

Clarity of Assessment
(α = 0.835)

2. Instructions to perform each activity were clear.

0.785

3. I was aware of the information needed for each task in the assessment.

0.796

10. The standardised “patients” were believable and realistic.

0.754

Clinical
Relevance
(α = 0.856)

11. Settings and context of given case scenario were authentic.

0.782

12. The interaction with the standardised patient was authentic.

0.790

15. The VETA helped to identify gaps in knowledge.

0.548

16. The VETA helped to identify weaknesses in communication and patient-care skills.

0.681

14. The VETA was an authentic reflection of clinical setting.

0.741

Value of
Assessment
(α = 0.869)

18. The VETA covered a wide knowledge range.

0.803

19. The VETA should remain as a form of assessment.

0.518

20. The VETA offered more learning opportunities than other exams.

0.788

21. The VETA was a valuable practice and learning experience.

0.607

Table 3. Latent constructs from exploratory factor analysis

IV. DISCUSSION

A. Keeping Pace with the Digital World

With the advent of technology and increasing prevalence of virtual care (Stamenova et al., 2022), there is a need to rethink conventional assessments of clinical competencies in medical and allied health education. Formal exposure, training, and assessment of clinical care delivery over virtual platforms will be important to prepare students for the digital world, beyond traditional face-to-face environments (Fickenscher & Pagliaro, 2021). Many digitally empowered patients have turned to web-based platform for telerehabilitation services (Brennan et al., 2009). This underscores the need for future healthcare professionals to demonstrate clinical competencies that complement the virtual mode of care delivery.

In our study, VETA was designed as an alternative to traditional OSCEs for the reliable assessment of exercise prescription and coaching skills of physiotherapy students within a simulated teleconsultation setting. Standardised patients were involved to enhance the realism of the consultation session where it allows students to practice patient-centred care and professionalism in patient handling. VETA has garnered largely positive experience from both students and SPs. The qualitative responses from the surveys were supportive of the validity, authenticity, and value of VETA; however, important limitations have been highlighted to consider for improvements in the future. This represents a real-world use case of online synchronous assessment that can be implemented to evaluate clinical skills and competencies in medical or allied health education.

B. Assessment Authenticity and Relevance

In this study, VETA provided a unique opportunity for students to conduct and experience a simulated teleconsultation and it has been largely perceived as a valid and authentic assessment. The content validity of VETA can be assessed from the realistic clinical tasks perceived by students, the test content which represents what the curriculum needs to assess (exercise prescription and coaching skills), and the right domains being tested such as communication and patient-care skills and the safe conduct of exercise (American Educational Research Association, 2014). The authenticity and value of VETA were echoed by both the quantitative and qualitative responses in this study where students felt that it was a good way to equip them with skills necessary for telehealth practice. It is paramount to note that the challenge in conducting physical assessments under COVID-19 climate is not uniquely experienced by students but also, by health practitioners (Chen et al., 2020). Telerehabilitation has been advocated and practiced during the pandemic (Turolla et al., 2020; Werneke et al., 2021) to improve accessibility to care despite the movement restrictions.

The current pandemic may bring forth a new era where telehealth becomes more significant and forms a vital service delivery platform for allied health professionals in the future (Negrini et al., 2020). Hence, this supports the relevance of a teleconsultation setting with standardised patients in our virtual assessment. The lack of exercise equipment provided during examination, albeit perceived as a limitation by students, could present a learning opportunity for students to react and cope in low-resource settings which is valuable to their future professional practice. In addition, the ability to assess students’ performance non-obtrusively allows a fair assessment with introducing unnecessary anxiety with the presence of assessor, which might be unavoidable in conventional OSCE stations (Ferreira et al., 2020). Therefore, VETA serves as a viable alternative that allows remote assessment of clinical competency in physiotherapy students.

C. Challenges with Implementation

VETA was conducted synchronously through Zoom platform within campus with students and SPs residing in different locations. Consequently, a reliable IT support structure and network connectivity was crucial to the success of the online assessment (Hopwood et al., 2021). Indeed, the connectivity issue has been reflected in this study where it has led to the loss of precious time for students to perform their teleconsultation. In this scenario, the examiner had to make a precise judgement to either (1) extend the timing and risk delays to the examination process, or (2) determine if there was enough “interaction content” to assess with and proceed with original scheduled time. Therefore, this underscores the need to include buffer time for similar situations when planning for virtual assessment.

Furthermore, the wearing of masks has resulted in the muffling of voices which prevented effective communication between students and SPs on the teleconferencing platform. Facial gestures and expression are known to play an important role in interpersonal communication, comprehension, and the delivery of intended messages and this can be affected by mask-wearing (Mheidly et al., 2020). To ensure that the SP is able to understand the prescription adequately, transparent face-masks could be worn to facilitate communication. Nonetheless, we advocate that allied health students should learn and master non-verbal communication skills as transparent face-masks may not be commonly used in clinical settings.

Time and space allocation have also been perceived to be inadequate. Based on students’ qualitative feedback, the perceived lack of time was pertaining to the interaction with the SP during exercise prescription, and not the duration for reading and planning. Pacing of speech, ensuring clarity and prioritising of information to convey are instrumental in time-effective clinical communication which students need to master (Laidlaw et al., 2014). Therefore, the timing of teleconsultation has to be optimised, not solely based on students’ expectations, but also on the assessment criteria and desired difficulty level. Furthermore, the time allocated should reflect the duration of an actual teleconsultation session where it has been reported in vascular telemedicine setting to be approximately 10 minutes (Baldwin et al., 2003), similar to the VETA format. Secondly, to ensure that SPs can perform exercise freely and safely, future runs of the assessment should cater for 20 to 50 square feet of space in the rooms, as recommended by American Council on Exercise (American Council on Exercise, 2009).

D. Study Limitations

In our study, only 39% of the total student cohort who undertook the VETA completed the survey. Hence, this may inadvertently create responders’ bias in the survey findings (e.g., those who had a positive experience may feel more motivated to participate). In our study, the evaluation survey was disseminated to students only a day after the conduct of VETA. Future studies may explore administering the survey immediately after the completion of the virtual assessment, via accessible QR codes or weblinks, to encourage greater participation rate.

Additionally, the assessment scope of VETA was limited to only a single domain of skill competency (safe exercise coaching and prescription) of the physiotherapy program and may not represent the relevance of such format for evaluating other competencies. As VETA has only been evaluated in one school setting, it remains to be ascertained if a similar format can be successfully adopted by other universities or a different structured health sciences program. Future studies may explore VETA’s implementation in other settings (i.e., evaluating other clinical competencies or conducted in other programs) and identify the potential facilitators and barriers to its adoption.

Despite the use of self-reported questionnaires to evaluate predetermined themes, the exploratory factor analysis has showed that the “effectiveness” component may not be reliably assessed by the items that were designated. To be an effective assessment, it also implies that the assessment has to mimic actual clinical settings and test skills relevant to a physiotherapist. Therefore, the same items have been correlated to the identified constructs such as clinical relevance (directly linked to assessment authenticity) and the value of assessment. Nonetheless, the distilled 13-item subset survey with good internal consistency can be employed to evaluate other novel assessments related to clinical competency.

E. Implications of a Virtual Approach

Virtual OSCE, while shown to be a useful examination format, cannot entirely replace the traditional OSCE in the assessment of physical skills which is required for allied health professional practice. For instance, hands-on skills such as manual therapy involving the massage, manipulation and mobilisation of muscles and joints, are critical to restore the patient’s functional capabilities and promote their self-efficacy (Bronfort et al., 2010). It is challenging to assess such components using virtual assessments. Specific to therapeutic exercises, it is vital to assess physical facilitation (provision of support and tactile cues) during coaching of exercise to frail individuals or patients with mobility limitations. This cannot be properly assessed without allowing physical close contact with the SPs.

Nonetheless, the success of VETA in this study highlights the feasibility of assessing specific clinical skills reliably using an online synchronous approach. When utilised in an appropriate context (e.g., for assessing online exercise prescription skills), the virtual assessment can help students gain literacy and confidence in online care delivery. At the same time, it allows them to appreciate and navigate the challenges in virtual care settings such as the lack of space or limited camera viewing angle identified in this study. With the COVID-19 pandemic, the demonstration of clinical competency over an online platform may prove to be as, if not more, relevant than traditional face-to-face assessments.

F. Looking Ahead: The Application and Value of Virtual Assessments

To understand how virtual assessments can play a bigger role in medical or allied health education, it is first important to identify which clinical skillsets and competencies are applicable or commonly used in telehealth settings. This can help in the selection of appropriate assessment to be used. Digital literacy and proficiency in the use of telehealth technologies should form a key component of undergraduate clinical education. The use of web-based or teleconferencing platforms for care delivery should also be included as part of their formal training as a healthcare professional. This will help equip the students with the relevant skills for the evolving digital world. There is also a need for a paradigm shift to not just rely on traditional OSCEs for clinical assessments, but to incorporate elements of technology and virtual care delivery relevant to telehealth practice wherever applicable.

V. CONCLUSION

While there are still technical difficulties to consider, our current model and examination setup provides an initial framework for others to adopt as an online synchronous method to assess clinical skills, especially during lockdowns or restricted periods. Furthermore, this VETA format may be used for distant learning and remote assessment of clinical competency for healthcare workers residing in rural areas (Palmer et al., 2015) or students on placement in medically underserved areas. As the use of virtual assessments removes geographical barriers, this allows other experienced teachers and faculty from external institutions to join as assessors or observers to calibrate and align assessment outcomes, thereby enhancing the overall quality of medical education across institutions. Taken together, the use of virtual clinical skills assessment is promising and should be carefully considered for integration into the current medical or allied health education system.

Notes on Contributors

TXR contributed to the study design, conceptualised VETA, conducted data collection and analysis, and drafted the manuscript. AG contributed to assessment design and results interpretation. CN devised the study, innovated the VETA format, conducted data collection and contributed to results interpretation. All authors have read and approved the final manuscript.

Ethical Approval

The study was exempted from ethical review with the use of anonymous surveys with no recording of any identifiable information. In line with the Declaration of Helsinki and the institutional IRB exemption criteria, the study presents minimal risk to participants, does not touch on sensitive topics, does not involve vulnerable population, and does not involve deception or withholding of study’s stated aims and objectives from participants.

Data Availability

Datasets generated and/or analysed during the current study are available from the following DOIs: https://doi.org/10.6084/m9.figshare.22641013

https://doi.org/10.6084/m9.figshare.22641115

https://doi.org/10.6084/m9.figshare.22640998

Acknowledgement

We would like to thank all the students and standardised patients who participated in this study.

Funding

The authors received no specific funding for this study.

Declaration of Interest

The authors report no conflicts of interest.

References

American Council on Exercise. (2009). American council on exercise: How to design a home gym. Retrieved April 4, 2022, from https://www.acefitness.org/resources/everyone/blog/6625/home-gym-design-how-to-design-a-home-gym/

American Educational Research Association. (2014). Standards for educational and psychological testing. American Educational Research Association.

Baldwin, L., Clarke, M., Hands, L., Knott, M., & Jones, R. (2003). The effect of telemedicine on consultation time. Journal of Telemedicine and Telecare, 9 Suppl 1, S71-73.

Barra-Ortiz, H. A. d. l., Gómez-Miranda, L. A., & Fuente-Astroza, J. I. d. l. (2021). Objective structured clinical examination (OSCE) to assess the clinical skills of physical therapy students when using physical agents. Revista de la Facultad de Medicina, 69(3), Article e83545.

Blythe, J., Patel, N. S. A., Spiring, W., Easton, G., Evans, D., Meskevicius-Sadler, E., Noshib, H., & Gordon, H. (2021). Undertaking a high stakes virtual OSCE (“VOSCE”) during Covid-19. BMC Medical Education, 21, Article 221. https://doi.org/10.1186/s12909-021-02660-5

Brennan, D. M., Mawson, S., & Brownsell, S. (2009). Telerehabilitation: Enabling the remote delivery of healthcare, rehabilitation, and self management. Studies in Health Technology and Informatics, 145, 231-248.

Bronfort, G., Haas, M., Evans, R., Leininger, B., & Triano, J. (2010). Effectiveness of manual therapies: The UK evidence report. Chiropractic & Osteopathy, 18, Article 3. https://doi.org/10.1186/1746-1340-18-3

Chen, D. E., Tay, E. K., Tan, P. L., Su, P., Koh, M. M., Ong, P. L., Chew, E., & Ng, Y. S. (2020). The impact of COVID-19 pandemic on rehabilitation in Singapore. Annals of the Academy of Medicine of Singapore, 49(11), 925-927. https://doi.org/10.47102/annals-acadmedsg.2020297

Chisnall, B., Vince, T., Hall, S., & Tribe, R. (2015). Evaluation of outcomes of a formative objective structured clinical examination for second-year UK medical students. International Journal of Medical Education, 6, 76-83. https://doi.org/10.5116/ijme.5572.a534

Cleland, J. A., Abe, K., & Rethans, J. J. (2009). The use of simulated patients in medical education: AMEE Guide No 42. Medical Teacher, 31(6), 477-486.

Deldar, K., Bahaadinbeigy, K., & Tara, S. M. (2016). Teleconsultation and clinical decision making: A systematic review. Acta Informatica Medica, 24(4), 286-292.

Deville, R. L., Fellers, C. M., & Howard, M. L. (2021). Lessons learned pivoting to a virtual OSCE: Pharmacy faculty and student perspectives. Currents in Pharmacy Teaching and Learning, 13(11), 1498-1502. https://doi.org/10.1016/j.cptl.2021.06.046

Downing, S. M. (2004). Reliability: On the reproducibility of assessment data. Medical Education, 38(9), 1006-1012.

Edwards, M., & Martin, A. (1989). The objective structured clinical examination as a method of occupational therapy student evaluation. Canadian Journal of Occupational Therapy, 56(3), 128-131.

Elo, S., & Kyngäs, H. (2008). The qualitative content analysis process. Journal of Advance Nursing, 62(1), 107-115.

Farahat, E., Rice, G., Daher, N., Heine, N., Schneider, L., & Connell, B. (2015). Objective structured clinical examination (OSCE) improves perceived readiness for clinical placement in nutrition and dietetic students. Journal of Allied Health, 44(4), 208-214.

Ferreira, É. d. M. R., Pinto, R. Z., Arantes, P. M. M., Vieira, É. L. M., Teixeira, A. L., Ferreira, F. R., & Vaz, D. V. (2020). Stress, anxiety, self-efficacy, and the meanings that physical therapy students attribute to their experience with an objective structured clinical examination. BMC Medical Education, 20(1), 296. https://doi.org/10.1186/s12909-020-02202-5

Fickenscher, K., & Pagliaro, J. A. (2021). Education in virtual care delivery: Clinician education and digital health literacy. Healthcare Information Technology for Cardiovascular Medicine, 111-125. https://doi.org/10.1007/978-3-030-81030-6_9  

Fouad, S., Gouda, E., Nasser, A. A., & Kamal, D. (2019). Perception of students, staff and simulated patients towards objective structured clinical examination (OSCE). Education in Medicine Journal, 11(2), 27-42. https://doi.org/10.21315/eimj2019.11.2.4

Gormley, G. (2011). Summative OSCEs in undergraduate medical education. The Ulster Medical Journal, 80(3), 127-132.

Grover, S., Pandya, M., Ranasinghe, C., Ramji, S. P., Bola, H., & Raj, S. (2022). Assessing the utility of virtual OSCE sessions as an educational tool: A national pilot study. BMC Medical Education, 22(1), 178. https://doi.org/10.1186/s12909-022-03248-3

Hannan, T. A., Umar, S. Y., Rob, Z., & Choudhury, R. R. (2021). Designing and running an online objective structured clinical examination (OSCE) on Zoom: A peer-led example. Medical Teacher, 43(6), 651-655.

Harden, R. M., Stevenson, M., Downie, W. W., & Wilson, G. M. (1975). Assessment of clinical competence using objective structured examination. British Medical Journal, 1, 447-451. https://doi.org/10.1136/bmj.1.5955.447

Hopwood, J., Myers, G., & Sturrock, A. (2021). Twelve tips for conducting a virtual OSCE. Medical Teacher, 43(6), 633-636.

Hsieh, H. F., & Shannon, S. E. (2005). Three approaches to qualitative content analysis. Qualitative Health Research, 15(9), 1277-1288.

Jo, K. H., & An, G. J. (2014). Qualitative content analysis experiences with objective structured clinical examination among Korean nursing students. Japan Journal of Nursing Science, 11(2), 79-86.

Khan, K. Z., Ramachandran, S., Gaunt, K., & Pushkar, P. (2013). The objective structured clinical examination (OSCE): AMEE Guide No. 81. Part I: An historical and theoretical perspective. Medical Teacher, 35(9), e1437-1446.

Kumar, A., Sarkar, M., Davis, E., Morphet, J., Maloney, S., Ilic, D., & Palermo, C. (2021). Impact of the COVID-19 pandemic on teaching and learning in health professional education: A mixed methods study protocol. BMC Medical Education, 21(1), 439. https://doi.org/10.1186/s12909-021-02871-w

Laidlaw, A., Salisbury, H., Doherty, E. M., & Wiskin, C. (2014). National survey of clinical communication assessment in medical education in the United Kingdom (UK). BMC Medical Education, 14, 10. https://doi.org/10.1186/1472-6920-14-10

Liang, Z. C., Ooi, S. B. S., & Wang, W. (2020). Pandemics and their impact on medical training: Lessons from Singapore. Academic Medicine, 95(9), 1359-1361. https://doi.org/10.1097/acm.0000000000003441

Lovink, A., Groenier, M., van der Niet, A., Miedema, H., & Rethans, J. J. (2021). The contribution of simulated patients to meaningful student learning. Perspectives on Medical Education, 10(6), 341-346. https://doi.org/10.1007/s40037-021-00684-7

Major, S., Sawan, L., Vognsen, J., & Jabre, M. (2020). COVID-19 pandemic prompts the development of a web-OSCE using Zoom teleconferencing to resume medical students’ clinical skills training at Weill Cornell Medicine-Qatar. BMJ Simulation & Technology Enhanced Learning, 6(6), 376-377. https://doi.org/10.1136/bmjstel-2020-000629

Mheidly, N., Fares, M. Y., Zalzale, H., & Fares, J. (2020). Effect of face masks on interpersonal communication during the COVID-19 pandemic. Frontiers in Public Health, 8. https://doi.org/10.3389/fpubh.2020.582191

Negrini, S., Kiekens, C., Bernetti, A., Capecci, M., Ceravolo, M. G., Lavezzi, S., Zampolini, M., & Boldrini, P. (2020). Telemedicine from research to practice during the pandemic. “Instant paper from the field” on rehabilitation answers to the COVID-19 emergency. European Journal of Physical and Rehabilitation Medicine, 56(3), 327-330. https://doi.org/10.23736/s1973-9087.20.06331-5

Norman, G. (2002). Research in medical education: Three decades of progress. British Medical Journal, 324(7353), 1560-1562. https://doi.org/10.1136/bmj.324.7353.1560

Palmer, R. T., Biagioli, F. E., Mujcic, J., Schneider, B. N., Spires, L., & Dodson, L. G. (2015). The feasibility and acceptability of administering a telemedicine objective structured clinical exam as a solution for providing equivalent education to remote and rural learners. Rural Remote Health, 15(4), 3399.

Pettit, M., Shukla, S., Zhang, J., Sunil Kumar, K. H., & Khanduja, V. (2021). Virtual exams: Has COVID-19 provided the impetus to change assessment methods in medicine? Bone & Joint Open, 2(2), 111-118. https://doi.org/10.1302/2633-1462.22.BJO-2020-0142.R1  

Prettyman, A. V., Knight, E. P., & Allison, T. E. (2018). Objective structured clinical examination from virtually anywhere! The Journal for Nurse Practitioners, 14(8), e157-e163. https://doi.org/10.1016/j.nurpra.2018.05.007

Rickles, N. M., Tieu, P., Myers, L., Galal, S., & Chung, V. (2009). The impact of a standardized patient program on student learning of communication skills. American Journal of Pharmaceutical Education, 73(1), 4.

Shirwaikar, A. (2015). Objective structured clinical examination (OSCE) in pharmacy education – a trend. Pharmacy Practice, 13(4), 627-627. https://doi.org/10.18549/PharmPract.2015.04.627

Silva, C. C., Lunardi, A. C., Mendes, F. A., Souza, F. F., & Carvalho, C. R. (2011). Objective structured clinical evaluation as an assessment method for undergraduate chest physical therapy students: A cross-sectional study. Brazilian Journal of Physical Therapy, 15, 481-486.

Stamenova, V., Chu, C., Pang, A., Fang, J., Shakeri, A., Cram, P., Bhattacharyya, O., Bhatia, R. S., & Tadrous, M. (2022). Virtual care use during the COVID-19 pandemic and its impact on healthcare utilization in patients with chronic disease: A population-based repeated cross-sectional study. PLOS ONE, 17(4), Article e0267218. https://doi.org/10.1371/journal.pone.0267218

Tan, X. R., & Ng, C. L. W. (2022, January 20-21). Perceptions of allied health students and standardised patients towards virtual exercise teleconsultation assessment (VETA) during COVID-19. Applied Learning Conference 2022, Singapore.

Turolla, A., Rossettini, G., Viceconti, A., Palese, A., & Geri, T. (2020). Musculoskeletal physical therapy during the COVID-19 pandemic: Is telerehabilitation the answer? Physical Therapy, 100(8), 1260-1264. https://doi.org/10.1093/ptj/pzaa093

Watkins, M. W. (2018). Exploratory factor analysis: A guide to best practice. Journal of Black Psychology, 44(3), 219-246.

Werneke, M. W., Deutscher, D., Grigsby, D., Tucker, C. A., Mioduski, J. E., & Hayes, D. (2021). Telerehabilitation during the COVID-19 pandemic in outpatient rehabilitation settings: A descriptive study. Physical Therapy, 101(7). https://doi.org/10.1093/ptj/pzab110  

Wessel, J., Williams, R., Finch, E., & Gémus, M. (2003). Reliability and validity of an objective structured clinical examination for physical therapy students. Journal of Allied Health, 32(4), 266-269.

Zayyan, M. (2011). Objective structured clinical examination: The assessment of choice. Oman Medical Journal, 26(4), 219-222. https://doi.org/10.5001/omj.2011.55

*Tan Xiang Ren
SIT Dover Campus,
10 Dover Drive,
Singapore 138683
Email: XiangRen.Tan@singaporetech.edu.sg

Announcements