Abstract

Objective: Physical disability is a fact of some neurologic disorders, such as multiple sclerosis. One of the treatments for such disability is routine physical exercises, or “rehabilitation”. However, rehabilitation in hospitals is often unattractive to patients. Another difficulty is objectively assessing the final effect of rehabilitation on disabilities, as assessment often depends on the subjective opinion of the physician. In the present study, we offer exergaming rehabilitation at home (telerehabilitation) and an objective method for measuring the physical performance of people with multiple sclerosis using a virtual reality tool to assist the decision of whether improvement, no change, or deterioration in the patient’s health status has occurred.

Materials and Methods: Telerehabilitation is provided by custom-made exergames specifically designed for patients with upper extremity disabilities. Our performance measurement method records the time taken by a patient to finish a physical test and measures the angles of interest between predetermined upper extremities. The measurements are recorded and saved for future determinations of patient progress. Thus, improvement-deterioration-no change decisions can depend less on subjective opinions. Preliminary performance experimentation was conducted before and after participants played our virtual reality exergames.

Results: The results reveal that our method is capable of measuring angles with an error margin of 6.44%. The accuracy of our method is 86.00%. The sensitivity, i.e., ability to detect improvements in patient performance, of our method is higher at 88.24%. The specificity, i.e., correct determination of no change in performance, is lower at 82.25%. The time taken to finish a physical test could not be evaluated due to a lack of real patients in our engineering laboratories.

Conclusion: The impact of our telerehabilitation exergaming solution on patient performance requires prolonged use by patients and future analysis of accumulated medical opinions. Our proposal is the first step toward exergaming and digital performance determination.

Introduction

As a result of a neurological disorder, people with multiple sclerosis (MS, pwMS) suffer from motor impairment affecting their everyday activities, defined as a physical disability (1). Approximately 66% of pwMS suffer from upper extremity dysfunction (2). One of the treatments for upper extremity dysfunction is rehabilitation, or physical education (3). As the first step of the rehabilitation process, the disability level of pwMS is determined through a set of physical tests (4). The main goal of the tests is to observe the body functions and decide the degree of capability of executing specific movements. At the end of the examination a physician assigns a disability score to pwMS. Following a score assignment, an individual and convenient rehabilitation program is designed for the patient (5). Informative rehabilitation sessions take place at hospitals under the supervision of professional healthcare personnel. Afterwards, the patient is expected to follow the advised rehabilitation program at home. However, during the coronavirus disease 2019 (COVID-19) pandemic, patients are not recommended to re-visit hospitals for rehabilitation, to avoid COVID-19 transmission. Unfortunately, it is frequently reported that patients lose interest and fail to follow their programs, due to various personal reasons. A lack of telerehabilitation system substructures in developing countries such as Turkey also adversely affects the implementation of rehabilitation programs (6).

The most common and global measurement of disability is the expanded disability status scale (EDSS) (7). However, since the EDSS does not include upper extremity and cognitive function assessment, the MS functional composite was developed (8). Unfortunately, the mutual disadvantage of those methods is that the score determined for pwMS may differ from one physician to another. Consequently, the difference in the assigned scores may result in inconsistent rehabilitation programs and ultimately cause loss of interest in rehabilitation and physical activities by pwMS, resulting in no physical improvement. Such an outcome is the most undesired by all parties involved.

While the patients discontinue long-term traditional rehabilitation programs, interest in virtual reality (VR) for passing time is on the rise. VR also offers the opportunity to receive task-oriented training by merging exercise and gaming into the exergames technology (9). Moreover, VR exergaming has also been suggested for rehabilitation as a more motivational method for pwMS treatment. Many previous studies utilized the Microsoft Kinect™ camera (Kinect) for VR rehabilitation applications. Some authors have developed Kinect-based games to help patients exercise and improve their body’s motor movements (10). The games were designed to help the player train specific muscles or parts of the body. For instance, while one game targeted the upper extremities, another targeted the whole body. In another work, a framework was developed using Kinect to help evaluate gait in pwMS (11). It was concluded that Kinect is a feasible tool for clinical assessment. Other studies also approved the validity of using the Kinect for limb dysfunction assessment. For example, Cai et al. (12) showed that Kinect is a reliable tool for functional upper extremity assessment. In a recent review, researchers discussed the gaming platforms used for measuring clinical outcomes, such as upper extremity movement assessment (13). The paper demonstrates the high precision and accuracy of Kinect in objective disability assessment.

In present work, we aimed to help physicians in two ways, using a state-of-the-art VR technology:

1.            Overcome the problem of discontinued patient rehabilitation,

2.            Provide help in assessing physical disability using engineering methods.

In the first phase, we developed an exergaming software specifically targeting pwMS. The pwMS executes some house chores using the developed VR software, instead of playing games intended for healthy people. In the second phase, we developed a software program using the same tool for measuring the time taken by a patient to finish a physical test and the angles of interest (AoI) between some nodes of the upper body extremities. Thus, disability assessment will become more objective, by using our computerized physical performance measurements.

Materials and Methods

Our telerehabilitation VR applications consist of custom-made exergaming software that runs on a personal computer (PC) with a Kinect connected to it. Both the exergaming scenarios and the performance measurements of the pwMS have been implemented using the same tool. This study was approved by the Non-invasive Research Ethics Board of Dokuz Eylul University (decision number: 2022/14-02, date: 13.04.2022). An informed consent form was obtained from all participants.

Participants

Exergames were attempted by multiple healthy people in their homes, but because of the COVID-19 pandemic, performance measurement tests were carried out with one healthy control person in an engineering laboratory in order to limit the contact with many people. As a future study, clinical tests with more participants are needed to obtain better results from the performance measurement software.

Procedure

The scenarios were planned in three separate meetings by a team consisting of computer engineers, doctors, and physiotherapists. In the first meeting, 16 scenarios, including kitchen activities, were determined by doctors and physiotherapists. In the second meeting, the team discussed all scenarios, and they decided to merge some of those. The last version of the exergames, which comprises 13 scenarios, was completed in the third meeting. The following activities were included: opening the door, wearing a kitchen apron, choosing and memorizing a recipe, selecting items from the fridge, cleaning and dishwashing, cooking, and eating.

Materials

Our work involves both hardware and software. We designed and programmed the software on a Microsoft Windows™ operating system with Windows Presentation Foundation (WPF) and the C# programming language. In addition, we included the Microsoft.Kinect.dll library, which provides Kinect-related functionality. The materials used are given below:

Hardware

- PC with Gen Intel^® Core™ i5-1135G7, 8 GB RAM and Intel Iris^® Xe Graphics card,

- Microsoft Kinect™ V2,

- Conversion adapter for direct connection to a PC.

Software

- Microsoft Windows 10 Operating System™,

- Visual Studio™ 2019,

- C# Net and WPF,

- Kinect software development kit and library,

- Vitruvius package of utility programs (14).

However, the hardware is not invariant. Other VR tools such as Kinect Azure™, Intel RealSense™, or other brands can be used with the same developed algorithms. Microsoft initially produced the Kinect Xbox One for motion capture and gaming. Unlike Kinect for Windows, it cannot directly connect to the PC and needs an additional external power adapter, as shown in Figure 1. Hence, Xbox can be replaced by a PC, making software development and testing on the same computer possible. The developed software can be adapted to run on Xbox One or other Xbox versions.

Method

In the first phase of our work, a household chore scenario is reflected on the PC monitor. The pwMS is requested to complete a chore using the hands, rather than a remote controller. The Kinect tracks the hand movements of the pwMS, as it is equipped with an infra-red (IR) emitter, a red-green-blue camera, and an IR depth sensor as shown in Figure 2. The specially designed dotted light pattern emitted by the IR emitter is not visible to the human eye. The IR depth sensor captures the reflected light pattern from the objects in front of it. Figure 3 shows the IR dotted light pattern emitted by the Kinect IR sensor toward a 3D object. The CMOS sensor captures the pattern, and the time of flight of each dot reflected from the 3D object is recorded. The information is used to create a depth map of the objects in front of the Kinect (10). Hence, the positions of human body parts are determined by the calculated distance of each reflection (15).

The user interfaces of the exergaming and performance measurement software are shown in Figure 4, 5, respectively. The household chore exergames are determined by the physicians working on the project team. The scenarios start with verbal instructions. The pwMS is given a task to finish. For example, the pwMS is asked to open a jar and empty the contents into a bowl, as in Figure 4. The camera view shows the pwMS role-player performing the chore. The finished chores are assigned a point to motivate the player. The patient must finish within a predetermined period. The test is timed using the system clock. Hence, the physical speed of the pwMS is measured. The performance measurement interface contains a set of buttons for choosing a test. After making a choice, the pwMS is verbally instructed about the test. Then, the Kinect is activated, and tracking of the pwMS is started.

The labeled pwMS skeleton and the AoI to be measured for the chosen test are displayed on the monitor. Four colored and numbered rectangles on the upper right corner show the values of the AoI measured. Table 1 gives the location of the AoI. The interface also displays the real system time and the chronometer, which display the time passed since the start of the test. Hence, the pwMS’s physical capability/disability and the time taken are measured. The overall result is recorded as a triplet, (Θ1, Θ3, time) and (Θ2, Θ4, time), in the PC database.

Preliminary performance measurement experimentation has been carried out. The tests were carried out before and after 10 physical training sessions with the custom-made VR scenarios. The subject is asked to take a position in front of the Kinect during testing. The subject’s distance from the Kinect is optimized by moving the subject toward or away (according to body size), until the best region of interest frame is obtained. It is usually recommended to position the pwMS at a distance of 1.5-4 m away from the Kinect (16). The determined optimal distance is used in every test, for each subject. During testing, the following steps are carried out:

1.            The subject is instructed to stand at the baseline with both arms resting at their sides,

2.            The physician selects the desired test by clicking on the planned test button,

3.            When the button is pressed, the participant is instructed by a by pre-recorded voice to execute a sequence of actions,

4.            Recording of the subject movements starts five seconds before the end of the instructions,

5.            While the subject performs the physical activities, the AoI and the time are shown on the computer screen and recorded into a file,

6.            The subject finishes the test by placing both arms at their sides as in the start position,

7.            The test automatically stops at its conclusion.

There is no need to synchronize the start or end of the test. The first change in the AoI detects the moment when the subject starts to move. Conversely, the moment the subject ends the test is detected by the unchanging AoI. The Kinect tracks 25 nodes on the subject’s skeleton during the test. The numbering of the nodes has been defined in previous works as in Figure 6 (17).

Each AoI is determined by three nodes. For example, angle Θ1 of the left shoulder in Figure 6 is determined by nodes 2, 3, and 4. In the “Test3 Left Arm” experiment, angles Θ1 and Θ3 are recorded, as shown in Figure 7. The angles Θ2 and Θ4 are recorded in the “Test3 Right Arm” experiment.

Results

VR exergaming scenarios are approved by a medical committee before being made available to pwMS. There are a total of 13 scenarios, and each one is continuously perfected according to the physician’s comments. The first evaluations indicate that VR exergaming in the form of house chores is feasible. Furthermore, exergaming at home appears as a promising means of limiting hospital rehabilitation visits of pwMS.

All AoI measurements are saved in a file as shown in Figure 8, with a timestamp for later comparison with previous results. The file is closed at the end of the test. Figure 8 shows the angle value, and the time it was recorded. The angle readings are matched with the actual angle values marked on the paper protractor behind the player’s arm, as in Figure 5. The percent error in determining the AoI is calculated using equation (1):

Error =                  (1)

The mean error in measuring AoI in 50 different tests is 6.44%. The mean error in measuring the time taken to finish a test is less than 1%. However, this is not a valid estimate as the experiment subjects are not patients but healthy people.

Discussion

Our study proposes VR-based rehabilitation and performance assessment software for pwMS. In the first phase, a custom-made telerehabilitation exergaming software is implemented, using Microsoft Kinect. The pwMS are offered to play a game of complete series of scenarios, mimicking house chores. The developed scenarios are designed and approved by a team of physicians. In the second phase, we propose a method to measure the time taken to complete a task and the AoI of pwMS. AoI and task-timing values are recorded in a file using the Kinect tool, to help determine improvement, no change, or deterioration in the disability condition of the pwMS. The preliminary results show that the exergaming method is feasible in pwMS telerehabilitation. Notably, the number of hospital visits by the pwMS for rehabilitation can be reduced by the opportunity to exergame at home. The measurement of AoI with an average error of 6.44% and no valid timing measurement was not found satisfactory. Therefore, we sorted the measurements to mimic making the decision of whether a patient’s physical state has improved or has not changed. Changes less than 5 degrees (5°) were not noticeable by a physician observer. Therefore, to decide that the patient had “improved”, a change larger than 5° in AoI measurement by the software was accepted. Otherwise, the patient was considered as “not improved”. The improved or not improved decision was also made by an observer, independent of the software. In total, 50 tests were made to calculate the accuracy, sensitivity, and specificity of our software. For statistical calculations, true positive (TP), true negative (TN), false positive (FP), and false negative (FN) classifications were made as follows:

TP: Patient improved according to observer, and software also predicted improvement.

TN: Patient not improved according to observer, and software also predicted no improvement.

FP: Patient improved according to observer, but software predicted no improvement.

FN: Patient not improved according to observer, but software predicted improvement.

The testing results are summarized in Table 2. Out of 50 tests, our solution correctly detected 30 cases of improved patient performances. Thirteen cases of no performance improvement were also correctly detected. However, six cases of performance improvement and four cases of no performance improvement were incorrectly predicted.

The accuracy of our proposed method is its ability to determine the actual improvement in patient performances correctly. The universal equation for accuracy is:

Accuracy =                          (2)

The ability to detect an improvement in patient performance correctly gives the sensitivity of our proposed method. The accepted equation for sensitivity is given in (3).

Sensitivity =                                         (3)

Another important parameter is the specificity, which demonstrates the ability of the method used to correctly determine no improvement in patient performance. The specificity equation is:

Specificity =                       (4)

The calculated accuracy, sensitivity, and specificity of our proposed method are tabulated in Table 3. The accuracy of our method is 86.00%, and the sensitivity is higher at 88.24%. It is obvious that our method is superior at detecting improvements in patient performances. The specificity, i.e., correct determination of no improvement in performance, is lower at 82.25%. Nevertheless, these results come as no surprise, since the experiments were carried out in engineering laboratories and not in a hospital environment.

This study has some limitations. First, due to the COVID-19 pandemic, performance measurement tests and exergames were attempted on a limited pool of healthy controls. Second, we did not compare the results of measurements performed using the Kinect tool with real-time measurements.

Conclusion

The use of VR technology has been proposed for rehabilitation and physical performance determination in pwMS. The average error in determining AoI is 6.44%. The accuracy, i.e., correct determination of improvement or lack of improvement in patient performance, of our method is 86.00%. The sensitivity, i.e., ability to detect improvement in patient performance, is higher at 88.24%. The specificity, i.e., correct determination of no improvement in patient performance is lower at 82.25%. The successful measurement of the time taken to finish a physical test could not be evaluated due to the lack of real patients in our engineering laboratories.

The results are promising for obtaining objective clinical decisions about the physical performances of pwMS. Clinical work is needed to decrease the error, increase the accuracy and determine the task completion time. Future work also involves devising machine learning methods for interpreting collected pwMS performance data.