A New Way With Words

Louis Braille’s tactile writing and printing system opened a new world of information and inspiration to blind people, but two centuries later, reading most text still requires vision.

This barrier could be lifted with the development of HandSight, a project by UMD researchers to literally put vision on the fingertips of blind people, using a miniature camera designed to scope out people’s insides.

As users wearing the glove-like device scan a book, tablet or poster, a computer reads it to them—but that’s not the half of it.

“Reading isn’t the hard part,” says Jon Froehlich, an assistant professor of computer science who’s directing the project. “The difficult part is, when you’re using a nontactile surface like paper, how do you know where to put your finger in the first place, and how do you know when to start and stop on a line?” 

Existing mobile apps can perform simple reading tasks for blind people, like determining currency values or reading aloud the words in a cell-phone picture. But in the latter case, users get gibberish if they can’t aim the camera precisely. And who wants to break up reading a book with a photo session every page?

HandSight is innovative in the way it physically guides users as they move their hands along a line of text, says Froehlich, who has joint appointments in UMD’s Institute for Advanced Computer Studies and the Human-Computer Interaction Lab. His colleagues in the U.S. Department of Defense-funded project hail from the College of Information Studies and the Department of Electrical and Computer Engineering. 

“We use haptics—which is just a little vibration—either on-wrist or on-finger for guidance,” he says. The system also uses sound cues to help users center their fingers correctly on text.

Unlike some other text-to-speech technologies, HandSight allows readers to determine their own pace, as well as to easily reread sections, simply by moving their hand to an earlier part of the text.

The system is still early in development, Froehlich says, but the researchers envision centering on a computer processor contained in a smart watch. And they expect it to help with more than just reading.

“Our goal is to augment any activity that blind people use their hands for with computer vision, whether it’s helping users discriminate between food cans in their pantries or identify colors and patterns when they select clothes in the morning,” he says.