Superheroes, puppies, hippos – and AI – are helping children with disabilities bridge language gaps

How did you learn to talk?

Probably something like this: Your infant brain, a hotbed of neurological activity, picked up on your parents’ speech tones and facial expressions. You started to mimic their sounds, interpret their emotions and identify relatives from strangers. And one day, about a year into life, you pointed and started saying a few meaningful words with slobbery glee.

But many children, particularly those diagnosed with autism spectrum disorder, acquire language in different ways. Worldwide, one in 160 children is diagnosed with ASD. In the United States, it is one in 59 children — and approximately 40 percent of this group is non-verbal.

YouTube Video

Learning from superheroes and puppies

Lois Jean Brady and Matthew Guggemos, co-founders of Bay Area-based iTherapy who are speech pathologists and autism specialists, are tackling the growing prevalence of autism-related speech challenges with InnerVoice, an artificial intelligence-powered app whose customizable avatars stimulate social cues. The app animates avatars of superheroes, puppies, stuffed animals and people to help young children who have difficulties with language and expression pair words with meanings and practice conversation.

iTherapy received a Microsoft AI for Accessibility grant in 2018. The program provides grants as well as technology and expertise to individuals, groups and companies passionate about creating tools that make the world more inclusive. iTherapy is using the grant to integrate the Azure AI platform to enhance its generated speech, image recognition and facial animation.

A young boy at the iTherapy clinic uses InnerVoice chat bot to describe his photo of a Teddy bear.A five-year-old student using Zyrobotics to learn to read at Ranch Santa Gertrudes Elementary. 

“I think for sure that the AI component was the missing link,” says Guggemos of the app. “How do you use words, and what do words mean? What does a symbol represent? How do you use AI to develop problems that require language to solve?”

How a hippo helps teach speech 

AI is also proving an exciting development in speech and language improvement for Zyrobotics, an Atlanta-based educational technology company that was the first beneficiary of the AI for Accessibility program in 2018. Zyrobotics is using Azure Machine Learning to help its ReadAble Storiez educational tool interpret when a student needs assistance.

YouTube Video

ReadAble Storiez uses an avatar of a hippo to help students with learning disabilities such as dyslexia and other challenges such as stuttering, pauses and heavy accents.

Ayanna Howard, the company’s founder and professor in robotics, was first motivated to create ReadAble Storiez when watching a teacher use Zyrobotics’ Counting Zoo app with a child. When the teacher turned to her and said, “Can you have this app do more than just read with him? I think it’s fantastic that it helps improve his math – could it also help him improve his reading?”

Howard also found teachers mentioning the challenges of dyslexia in the classroom. “I was like, ‘Oh, what happens if you have a reading disability?’ I then learned that signs of dyslexia in children aren’t picked up until much later, typically when schools start standardized testing. I realized we needed an intervention much earlier and that we could do that with Counting Zoo.”

Learning models that don’t take individualized challenges into account, or don’t address the speech patterns of kids, “tend to fail,” Howard says. ReadAble Storiez employs a custom speech model and a sophisticated “tutor” to convert speech to text and measure accuracy, fluency and the child’s reading improvement.

‘It blew my mind!’

Howard is pleased with the program’s early success. “While they were reading a book, kids were correcting themselves,” she says. “As a technologist, you say your stuff works, but I’m sitting there with the kids and I’m blown away, ‘It really does work!’ It’s thrilling to see that what works in the lab actually works in the real world, in the child’s environment. The [avatar] would provide feedback, and a child would be like, ‘I didn’t say a word right. Can I try again?’ It blew my mind. That was the affirmation. Our solution was on track and on target.”

Brady, who came up with the idea for InnerVoice after studying and writing a book on apps for people with autism, reflects on the impact it has made. She cites an example of working with a student who is non-verbal and used the app to communicate with an avatar of himself.

“He would take a picture of an apple, and an avatar would read it as ‘apple,’ and then he would write it down, ‘apple.’ Until then, I hadn’t even thought of that strategy.”

A mother uses InnerVoice to work on communication skills with her young daughter. A mother uses InnerVoice to work on communication skills with her young daughter.  

Brady and Guggemos imagine the benefits of AI-assisted communication beyond their target audience. They are working with people with dementia, head injuries and strokes. “Many communication apps just talk for you,” Brady adds. “Ours spans many aspects of communication for everybody — even English-language learners. Why wouldn’t I try that? It provides a model. There’s a coffee cup on the table, take a picture of it. How do you say that?”

Howard dreams of Zyrobotics helping to close the gap between mainstream learners and students with learning disabilities. To start, this fall Zyrobotics will introduce ReadAble Storiez to classrooms in the Los Nietos, California, school district, where learning disabilities track high. The company will also apply AI to its suite of STEM Storiez, a series of nine interactive and inclusive books that help children ages 3 to 7 engage with science, math, engineering and technology.

The AI for Accessibility program has been instrumental in getting Zyrobotics off the ground with ReadAble Storiez. “If we hadn’t gotten the grant, we’d be in phase zero,” Howard says. “We run on grants to ensure we provide access to learning technologies for all students. We need to be out there for kids that need us.”

The grant gave Brady and Guggemos the technology to take InnerVoice to the next level. “Our kids need this technology,” Brady says. “It’s not a luxury. We want to keep adding the best stuff. Microsoft really propelled us forward in that arena.”

Top image: A young boy at the iTherapy clinic uses InnerVoice chat bot to describe his photo of a Teddy bear. 

Related: