Mistika VR is used at the Foren Project neurological centre, where Mistika’s accuracy and ease of use help trigger neuron recovery and muscle control in patients through VR experiences.
A team of neurological experts in Spain has developed a VR-based neurological rehabilitation program called the Foren Project. It focuses on a specific type of neuron, called mirror neurons, that contribute to the neurological growth of the motor system. Their key attribute is that they are triggered by imitation. It does not require a patient to move, but seeing the movement in a third person and attempting to imitate it triggers the mirror neuron to start growing.
Based on this principle, Foren Project is using actual, not virtual, reality. Through the project’s highly specialised production workflow, the patient visualises him or herself in that precise, critical moment of performing an exercise.
Recreating the Experience
Rafa Naveiro, VR creator at Foren Project, said, “When we initiated the project, we knew that correct recording, orientation and composition of the exercise environment would be extremely important. We needed to recreate the same image inside the headset as the patient would be seeing in real life, including the rehabilitation room he or she would occupy, and the POV down the length of the body wearing the same clothing.
“At first, we shot with a two-camera rig with 8mm fisheye lenses to capture the 360º environment. We needed to be extra careful with the stereo alignment and calibration and ensure that everything was shot correctly.
“We had a set of 15 different exercises for these patients. Once we had captured each of these correctly, once for a man and once for a woman, the post-production process could begin – stitching the images together, clean-up and so on. We knew what had to be done, however, because we have numerous clinics, production and post would have to be repeated for all 15 exercises, for both sexes, in every single clinic.”
Rethinking the Workflow
As this process was very time consuming and tedious, they felt they had no choice but to rethink the workflow, starting with optimising the production. Looking at camera options, they came across the Insta360 Pro 2, an integrated 8K stereo camera that proved to be a perfect match. They didn’t have to carry two cameras on set or worry about calibration, and it considerably sped up the whole shooting process.
The second challenge was recreating all exercise experiences, for women and men, in each clinic. The ideal solution was to first shoot all the exercises inside a 360º green screen set, and then just shoot the empty rehabilitation room in each hospital. With the recordings of the room on one hand and the exercises on the other, they could simply merge and composite both elements.
Another aspect of the workflow that they wanted to improve was post-production. “The stitching software that we had been using at the beginning was very slow and the results were not ideal,” Rafa said. “This totally changed when we started to use SGO’s Mistika VR software instead, allowing us to complete the whole process almost automatically and in a few clicks.”
Real Time Optical Flow
The nearly automatic functionality is due to Mistika’s integrated templates for different stereo rigs and cameras – including the Insta360 Pro 2. The resulting speedy optical flow stitching, plus other tools that improve the stereo alignment, facilitated the whole process tremendously for the Foren project team, cutting the time required in half.
Mistika VR’s Optical Flow process, used to stitch 360º media directly from cameras, gives a high level of control to the user. All aspects of the stitch can be manipulated and animated in real time, while live output is delivered to Deo VR, the free, platform-agnostic VR player. This means users can preview their work in nearly any model of VR headset and on other devices, such as tablets, mobile phones or laptops.
The process uses SGO’s proprietary stitching algorithms, written to overcome typical parallax issues, and it also can access NVIDIA GPU accelerated encoding, achieving speeds faster than real-time for encoding 4K VR media. Features that increase accuracy are keyframe animation for stitching flexibility and control, one-click stabilisation to help manage shaky footage and import of the gyroscope metadata from the Insta360 Pro 2's motion sensor.
“After stitching, we exported the left eye on one side and the right eye on the other, and moved on to the clean-up and compositing work in After Effects,” Rafa said. “From there, we could work in stereo mode and complete the edit in Premiere Pro, take care of colour correction and then export the media for the headset application.”
Real Life VR Stimulation
Charo Ortín, founder and CEO at Foren Project, said, “Apart from being the pioneers in applying a VR approach to neurological rehabilitation, the most important outcome is that this real-life VR stimulation really works. After having carried out several clinical trials and verifications we have obtained very good results. [You can watch the progress of the project here.]
“When a patient without the headset on looked at his foot, for example, and tried to move it, we did not detect any electrical activity in the muscle. It seemed that the patient alone was not capable of sending the signal to his leg. Then we told him to put on the headset and just imitate the movement he saw. This is when we started to see the activity in the patient’s leg. The therapists themselves noticed the changing muscle tone. Before, we could not detect an electromagnetic register, but after using VR, when we did obtain a register, it was simply amazing.” www.sgo.es
