BlogEngineeringInnovationVirtual Care

Beasts, aliens and avatars: how motion capture technology is taking a cue from entertainment to advance patient recovery

By September 25, 2017 September 25th, 2019 No Comments

Ever wonder how actor Dan Stevens’ facial movements seamlessly became the Beast in Beauty and the Beast or how the alien Thanos was portrayed in Guardians of the Galaxy? It’s called motion capture, and it’s the process of recording a live motion event and “magically” translating it into actionable data that allows for a 3D – or digital – recreation of the performance.

Motion capture isn’t just for entertainment, though. This cutting-edge technology is now being used by a few digitally savvy healthcare companies in their quest to advance patient recovery in the convenience of their own homes.

To learn more about the amazing healthcare benefits of motion capture technology, we spoke with senior software engineer Kevin Steele, Ph.D., from Reflexion Health, a pioneering digital healthcare company that employs motion tracking technology in their virtual physical therapy platform to improve the patient rehabilitation experience and, ultimately, create a better path to recovery.

Kevin, what do you do at Reflexion Health?

As a senior software engineer, I use my background in computer and robot vision to investigate and implement various body tracking technologies. My role is to help identify the best algorithms to use within remote exercise monitoring. Our aim is to optimize our patients’ post-operative recovery by allowing them to do their physical therapy at home with the integrated assistance of a motion tracking camera and an on-screen avatar who demonstrates proper form and provides audio-visual interaction to patients.

Motion tracking, or motion capture, is such a broad category of technology. What does it mean at Reflexion Health?

Motion tracking is any technology that uses a computer to discover and follow the location of an object. It’s definitely a broad category and it’s evolving – that’s what makes it such an exciting field to be in. The type of motion tracking we do is full-body pose tracking, which is specifically concerned with identifying the location of the primary joints of the human body using non-intrusive instruments, like cameras.

Using this technology, we can make a camera and software system work together to compute the joint locations of a human body and track those locations in real time. With this joint data, we can evaluate how a person stands or determine the motions a person is making. We can then, translate this stream of data into meaningful insight and provide patients with valuable input on their exercise performance and inform their treating physical therapist on their patient’s overall progress and outcome.


Using the data that comes from a depth camera (the gray pixels), Reflexion Health’s tracking software computes the location of a person’s joints (shown in red and yellow).

What can be tracked using motion capture?

Human motion tracking is most often performed on the face, the hands or the full body. In the case of hand tracking, it’s generally possible to track the three primary joints of each finger, two joints for the thumb and one for the wrist, although some technologies exist to track the hand in more detail.

When performing full body pose tracking, we can follow the location of most of the major joints – the wrist, elbow, knee, etc. However, not all joints can be easily tracked. The spinal column is made up of dozens of small joints, and most current technologies can’t track all of those joints in real time.

How does this technology work?

Several technologies exist to perform full body tracking, each with its own strengths and limitations.

Optical and inertial motion capture equipment-those often used for video games and movies- provide very accurate joint position data. Unfortunately, they require specialized equipment in the form of wearable devices and suits, which can be more challenging for patients recovering from a significant surgery.

Off-the-shelf video cameras that come equipped on mobile devices also track joint positions and body pose, but are usually less accurate due to the limbs being occluded or blocked from the view of the camera. The software, as a result can’t figure out where those joints are.

One of the more recent technologies for full-body pose tracking is the depth camera. It’s similar to a conventional camera, but instead of taking pictures with each pixel being a specific color, each pixel is a single number – the distance from the camera to the object seen at that pixel.


What type of motion tracking technology does Reflexion Health use?

We utilize depth cameras. With depth cameras, software can track a person’s motion by analyzing a sequence of depth images, or frames, taken at about 30 times per second. When the software sees the depths near a person’s arm change from the previous frame, it can simulate how the person’s arm must have moved to account for that difference, then calculate the new joint positions of the arm based on that simulation.

To find joint positions, we employ a unique method called Human Pose Estimation. It utilizes a machine learning technique called Random Decision Forests (RDFs) that learns what common human poses look like to a depth camera. When the depth camera is then pointed at a person, the software simply feeds the images to the trained RDF, which then estimates where the person’s joint positions are. This method is very robust, and is in fact also used by the Microsoft Kinect to estimate human pose.

So how does motion tracking technology help patients rehabilitate after surgery?

With motion tracking technology at our disposal, we can create reliable software applications that patients can conveniently interact with during their post-operative recovery. An even greater benefit is that patients can utilize this technology in the comfort of their homes. At Reflexion Health, for example, our FDA cleared software system, VERA uses motion tracking technology to monitor a patient’s participation to a prescribed physical therapy program while the patient is recovering at home. When patients perform their home exercises, VERA guides them through each exercise, measures their motion, and provides them with audio-visual interaction. Simultaneously, VERA captures this data and sends it to the patient’s treating physical therapist, who is able to monitor and communicate with them remotely.

Josh Calabrese