Huawei launches StorySign app : Reading books with sign language

If you don’t have a child who’s deaf or has a hearing impairment you’ve probably not thought about how different story time each night must be in those situations.

Every night we read books to our kids, and it’s not just encouraged to assist their reading, learning and vocabulary but also because of the strong personal connection you get from that time each night.

So now imagine you can’t read to your child.  Sure, they can watch you sign some words they know, but are they learning to read too?

Huawei have today announced a new app which will be available in Australia in February that goes a long way to solving this problem.

StorySign uses a mixture of artificial intelligence and animation to bring stories to life through sign language.

Choose a book, and using an Android smartphone as the camera, point the phone at the words, as the app recognises them they are highlighted a small animated character (called Sign) reads them in Sign Language.  When released here it will be compatible with Auslan (Australian Sign Language) meaning Aussie kids can read and learn to read at story time.

The need for this is brought home through the comments of Kyle Miers, CEO of Deaf Australia “Deaf children don’t learn to read in the same way as hearing children. Many struggle to learn how to read because they can’t match words with sounds. The deaf community is in need of accessible content to address children’s literacy development needs and digital tools like StorySign is addressing this necessity. For this reason, we are privileged to be working with Huawei on the StorySign project and how, through the use of AI and innovative technology, it could enrich the lives of Australian deaf children and their families in a meaningful way.”

And the animations aren’t anything simple either, they were created by animation specialists at Aardman – the genius’ behind Wallace & Grimmit; “When creating Star, the character for StorySign, we first had to immerse ourselves in the world of sign language,” comments Neil Pymer, Interactive Creative Director, Aardman.

“The complexity we found is overwhelming, so we needed to make sure that we created a character that not only resonated with the audience but also fulfilled its main job of teaching children to read. For example, we learned that facial expressions play a critical role in sign language, so we created a character to embody that expressivity. At the same time, Star was designed to seem like an older sibling, somebody who will guide you through the book. Seeing the reactions of children when realising that Star signs the words in the book is something very special indeed.”

Lisa Connors from Huawei in Australia says this shows how technology can impact positively on society “At Huawei, we believe in the power of AI and that technology can make a positive difference in the world”

“We created StorySign to help make it possible for families with deaf children to enjoy the truly magical moments of story time.”