The Adaptive Equation: interactive math education for kids who can’t hear

Let’s say that you are a first grader practicing mathematics in your classroom’s electronic lab. Inside a computerized grass hut on a tropical plantation, a donkey with a chef’s hat counts aloud as he converts stacks of canes into cups of sugar. Your classmates click on the correct stacks to arrive at the desired equation, quietly giggling their way to addition and subtraction mastery. But you can’t even hear what the characters are saying, and you don’t understand English well enough to follow directions on the screen. This is the challenge faced every day by students who are deaf or hard of hearing – and something that faculty and students in the colleges of Technology and Liberal Arts are trying to rectify.

Last year, professors Nicoletta Adamo-Villani and Ronnie Wilbur unveiled their first prototype of Mathsigner, a teaching technology that uses advanced computer 3D animation techniques and interactive applications to teach math skills in American Sign Language (ASL). When kids punch numbers on the keyboard, an avatar – such as a brown bunny outfitted in blue shirt and green shorts – signs out the equation “4 + 2”, and its answer, 6.

While a few other computer programs feature animated signers, Mathsigner’s characters move more realistically. “In other software, the connection between signs is extremely jumpy. It’s terrible to look at; it’s almost unreadable,” says Adamo-Villani, an assistant professor in the Department of Computer Graphics Technology. “Ours is not perfect either, but the motion is extremely fluid and natural.”

Adds Tim Curtin, president and CTO of EITAC Solutions Group, LLC, located in the Purdue Research Park, “Having choppy sign language thrown at you is distracting. I’ve looked at two or three competing technologies, and what Nicoletta and Ronnie have come up with is a better blending between symbols and signs.”

Equal Access

Classroom instruction today is often reinforced at school and at home through interactive educational media, but it’s not enough to say that Mathsigner simply will make this technology available to students who are deaf and hard of hearing. The software is, in fact, designed to provide equal access and opportunities to these students by overcoming their deficiencies in math education, ultimately leading to a more diverse workforce in fields requiring mathematics proficiency.

The challenge begins as soon as youngsters start school. Many children who have been deaf from birth use American Sign Language, not English, as their primary language. “ASL is a naturally developed language,” says Marie Nadolske, Wilbur’s graduate assistant. “It has its own syntax separate from English, its own grammar, its own phonology, and a completely different morphological language.”

In some mainstreamed classrooms, a teacher or interpreter may blend several sign languages, confusing kids who are trying to learn ASL symbols for mathematics. Even in classrooms where pure ASL is being used, some students may have difficulty grasping the mathematical concepts being taught. At home, hearing parents who are helping their children with assignments may not know the ASL signs either.

Mathsigner addresses all those issues. “The children and their parents can learn the signs from the software, and the children can learn the concepts related to math even without their parents’ help,” says Wilbur, professor and chair of the linguistics program in the Department of Speech, Language, and Hearing Sciences.

Adds Melissa Dark, associate dean for the College of Technology, “Traditional teaching materials are based on literacy, and deaf children are behind in language skills at the primary grades. When you’re teaching math in the first and second grade, you’re trying to give kids a sense of the symbolic representation of a number system and how it relates to another symbolic representation, which is a language system. Nicoletta and Ronnie are trying to find a different medium so that first- and second-grade deaf children can still acquire a number sense that’s not based on their language ability.”

Already, developers are receiving positive feedback from the Indiana School for the Deaf, which has been testing the first prototype. “In a world where almost every piece of software available on the market today excludes deaf and hard of hearing children, it is nice to have a product specifically designed for young users of ASL,” says teacher Chris Fetzer. “It has potential to positively impact many deaf and hard of hearing ASL users and improve their chances for a stronger educational foundation.”

Capturing Motion

To produce Mathsigner’s first prototype, the team created a set of 3D computer animated ASL signs using a popular Hollywood technique: motion capture. They’re now applying a more sophisticated version of the technology as they create the second prototype.

Strapping on a lightweight suit constructed of tiny reflective balls, Nadolske stands in a studio at Purdue’s Envision Center for Data Perceptualization, signing such mathematical symbols as plus, minus, and equal while six cameras capture her position. Unlike Tom Hanks in the filming of Polar Express, however, Nadolske herself isn’t being recorded, just the digital expression of her movements.

Adamo-Villani and her team of technology students are taking these motions and applying them to characters they’ve created on the computer – the bunny, a boy with a turned-around baseball cap, and a girl with long brown hair. “Once we apply the motion to the character, then the character starts to move,” says Adamo-Villani. “It’s the same technology used in video games, but we have placed particular emphasis on the accuracy of the hand motions.”

And that’s where the ingenuity of Mathsigner becomes apparent. Unlike other software packages, which use segmented avatars to sign sentences in a series of jerky motions, the creators of Mathsigner use seamless characters developed through their own patent-pending technique. “If I bend my arm, I have deformation in my elbow,” explains Adamo-Villani. “Our characters are designed to deform when they move.” By fluidly moving from one sign to another, the 3D signers are believable, emotionally appealing, and, most important, easy to understand.

If further research bears out and students are able to enhance their mathematics education with Mathsigner, more students who are deaf and hard of hearing may be entering some under-represented fields in the future. “Early mathematics exposure in an interesting and creative way can get children excited about going into math and sciences,” Nadolske says. “Somebody that’s more excited about it will be more likely to pursue it later on.”

Adaptive Technologies

While Mathsigner is in its infancy – there are no lush landscapes and donkeys counting sugar canes – the technology eventually could be applied to a variety of computer products designed for children up to grade 12 across a broad array of subjects.

Beyond childhood, of course, the technology potentially could bridge the gap in the workplace between those who hear and those who can’t. Already the realistically signing avatars have caught the attention of Curtin, whose consulting, research, and development firm helps businesses provide equal access to all their employees.

In the future, successors to Mathsigner’s bunny may be popping up in everyday office software, providing onscreen instructions in ASL to your co-worker at the desk next to you. "My mission at the company is to develop technology solutions to help people live, learn, and work," Curtin says.