Interface System Using Augmented Reality Technology Could Help People with Severe Motor Impairments

An interface system that employs augmented reality technology could assist individuals with extreme motor impairments to operate a humanoid robot to feed themselves and carry out daily personal care tasks like applying skin lotion and scratching an itch.

The web-based interface displays a “robot’s eye view” of surroundings to help users communicate with the world through the machine.

The system, reported on March 15^th, 2019 in the journal PLOS ONE, could help render sophisticated robots more helpful to people who are not trained in operating complex robotic systems. Study participants communicated with the robot interface with the help of standard assistive computer access technologies—for example, head trackers and eye trackers—that they were already using to control their personal computers.

The paper described two studies illustrating how such “robotic body surrogates”—which can carry out tasks comparable to those of humans—could enhance the quality of life for users. The study could offer a basis for creating quicker and more capable assistive robots.

Our results suggest that people with profound motor deficits can improve their quality of life using robotic body surrogates. We have taken the first step toward making it possible for someone to purchase an appropriate type of robot, have it in their home and derive real benefit from it.

Phillip Grice, PhD Graduate, Georgia Institute of Technology

Phillip Grice is also the first author of the paper.

Grice and Professor Charlie Kemp from the Wallace H. Coulter Department of Biomedical Engineering at Georgia Tech and Emory University employed a PR2 mobile manipulator made by Willow Garage for the two studies. The wheeled robot has 20 degrees of freedom, with two arms and a “head,” providing it the capability to maneuver objects such as hairbrushes, washcloths, water bottles, and even an electric shaver.

Our goal is to give people with limited use of their own bodies access to robotic bodies so they can interact with the world in new ways.

Charlie Kemp, Wallace H. Coulter Department of Biomedical Engineering, Georgia Tech and Emory University

As part of the first study, Grice and Kemp made the PR2 accessible over the internet to a group of 15 participants with extreme motor impairments. The participants trained themselves to control the robot remotely, using their own assistive equipment to operate a mouse cursor to carry out a personal care task. Of the participants, 80% were able to control the robot to take a water bottle and bring it to the mouth of a model.

“Compared to able-bodied persons, the capabilities of the robot are limited,” stated Grice. “But the participants were able to perform tasks effectively and showed improvement on a clinical evaluation that measured their ability to manipulate objects compared to what they would have been able to do without the robot.”

As part of the second study, the scientists offered the PR2 and interface system to Henry Evans, a man from California who has been helping Georgia Tech scientists to investigate and enhance assistive robotic systems since 2011. Evans, who has very restricted control of his body, tested the robot in his home for seven days and in addition to accomplishing tasks, he created new uses by combining the operation of both robot arms simultaneously—using one arm to control a washcloth and the other to use a brush.

The system was very liberating to me, in that it enabled me to independently manipulate my environment for the first time since my stroke. With respect to other people, I was thrilled to see Phil get overwhelmingly positive results when he objectively tested the system with 15 other people.

Henry Evans, California

The scientists were delighted that Evans devised new uses for the robot, combining motion of the two arms in unexpected means.

“When we gave Henry free access to the robot for a week, he found new opportunities for using it that we had not anticipated,” stated Grice. “This is important because a lot of the assistive technology available today is designed for very specific purposes. What Henry has shown is that this system is powerful in providing assistance and empowering users. The opportunities for this are potentially very broad.”

The interface enabled Evans to care for himself in bed over a prolonged time period. “The most helpful aspect of the interface system was that I could operate the robot completely independently, with only small head movements using an extremely intuitive graphical user interface,” stated Evans.

The web-based interface displays users what the world looks like from cameras placed in the robot’s head. The users can move the robot around in a house or other environment and control the robot’s hands and arms using clickable controls superimposed on the view. For example, when users move the robot’s head, the screen shows the mouse cursor as a pair of eyeballs to represent where the robot will look when the user clicks. Clicking on a disk around the robotic hands enables users to choose a motion. While directing the robot around a room, lines following the cursor on the interface denote the direction in which it will move.

Developing the interface around the actions of a simple single-button mouse enables people with a variety of disabilities to use the interface without extensive training sessions.

Having an interface that individuals with a wide range of physical impairments can operate means we can provide access to a broad range of people, a form of universal design. Because of its capability, this is a very complex system, so the challenge we had to overcome was to make it accessible to individuals who have very limited control of their own bodies.

Phillip Grice, PhD Graduate, Georgia Institute of Technology

Although the results of the study established what the researchers had started to do, Kemp agrees that enhancements can be made. The current system is slow, and errors caused by users can pose considerable obstacles. Still, he said, “People could use this technology today and really benefit from it.”

Evans proposed that it would be necessary to considerably decrease the cost and size of the PR2 to make the system commercially feasible. Kemp states these studies open the doors to a new kind of assistive technology.

“It seems plausible to me based on this study that robotic body surrogates could provide significant benefits to users,” Kemp added.

This research was supported by the National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR), grant 90RE5016-01-00 via RERC TechSAge; National Science Foundation Award IIS-1150157, by a National Science Foundation Graduate Research Fellowship Program Award; and the Residential Care Facilities for the Elderly of Fulton County Scholar Award.