These innovators are using tech to help translate speech for people with hearing loss

Fellows in this summer's Our Generation Speaks program are designing ventures that bring Israelis and Palestinians together through innovation

Photo/Tarah Llewellyn

2019 Our Generation Speaks fellows Julie Dai, right, and Wasseem Ghrayeb take a short break from developing their venture LipifAI outside at the Heller School, where OGS is based.

Growing up in Haifa, Israel, Julie Dai attended an inclusive elementary school where students who are deaf and hard-of-hearing and students without hearing loss learned together. Dai does not have hearing difficulties, but the experience made her acutely aware of the challenges her classmates experienced, especially in noisy places.

Fast forward to the summer of 2019, and the 29-year-old – who has bachelor’s and masters’ degrees in computer science and a several years in the tech industry – is an Our Generation Speaks (OGS) Fellow at the Heller School for Social Policy and Management at Brandeis. OGS, launched at Heller in 2016 by executive director Ohad Elhelo ’16, MA ’17, is a unique 3-month fellowship program and incubator that brings together Israeli and Palestinian entrepreneurs to build for-profit and not-for-profit ventures.

“When we brainstormed at the beginning of the program, I started out by thinking about problems I could solve that I and people around me experience on a daily basis. As we worked through the process, I thought about the communication struggle that people with hearing loss have in noisy environment,” Dai said.

Joining Dai – and 27 other fellows split evenly among Israelis and Palestinians – is Waseem Ghrayeb, a 30-year-old Palestinian citizen of Israel from Nazareth who has a master’s degree in artificial intelligence and a background working for innovative established companies and startups. Ghrayeb also frequently thought about problems that could be solved through innovation, and felt connected to the challenges Dai had witnessed. 

Together, the two are now working on LipifAI (pronounced lip-if-i), an online lip-reading technology, based on AI, to help deaf and hard-of-hearing people to communicate within their surroundings. Other technologies, including hearing aids, may not work effectively in noisy environments, and that’s where LipifAI comes in.

Dai, left, and Ghrayeb represented their venture LipifAI at the MassChallenge Showcase earlier this summer.

Simply put, explained Ghrayeb, “We turn speech into text,” but in a more sophisticated way than current speech-to-text solutions , which don’t work well around noise. Using a typical smartphone, the LipifAI app combines “listening” to the words of a speaker via the microphone and using the camera to read their lips, translating speech simultaneously into text that would appear on the user’s phone. Combining lip reading and speech to text can increase the accuracy of the text to 90 percent. Skilled human lip readers only catch about 40 percent of what someone says.

Such an app could be a game-changer for people with hearing loss, who would be able to understand conversations in noisy environments much better than before. It could be a social boon, particularly because people who are hard of hearing may avoid noisy environments because of the difficulties involved with interacting in them. In the U.S. alone, about 48 million people experience some degree of hearing loss.

"They [may] avoid restaurants, classrooms, and other noisy places. This is a very tough arena for them,” Ghrayeb said. 

The pair have discussed their LipifAI with more than 20 people who are hard of hearing themselves, including an expert at the Massachusetts Eye and Ear Infirmary in Boston. “They said `you detected a specific and difficult problem,’ ” Dai said. “They told us `when you launch your product, we will use it.’ ” 

Along with the other OGS Fellows, Ghrayeb and Dai have also received feedback and support from members of the OGS network. At the annual Venture Feedback Day in late July, each of the 2019 ventures pitched their ideas to a group including board members, mentors, coaches, donors, supporters and the growing group of OGS alumni.

“A lot of people followed up with ideas,” after their presentation, Ghrayeb said. “We felt high engagement from people.”

“We are very thankful for our mentors and for OGS staff who support and believe in us,” Dai said.

LipifAI is among a host of innovative ventures being developed by the 2019 fellows. One venture would create modern products from the remains of the olive oil industry; another would create high-end open kitchens in the middle of cities where people can come together to cook and eat; still another would print 3D, personalized casts to speed the healing of broken bones. Fellows are working on 11 ventures in all.

photo/Tarah Llewellyn

Dai shows an early version of the LipifAI app.

For the last few weeks of August, Dai and Ghrayeb will take LipifAI to the Massachusetts Institute of Technology, where they and the fellows launching two other ventures will work with faculty and students at MIT: designX, an entrepreneurship accelerator at MIT’s School of Architecture and Planning, and MISTI, MIT’s international education program. The rest of the 2019 Fellows will further refine their ventures at MassChallenge, a global startup accelerator in Boston’s booming seaport district.

A central goal of OGS is to make long term connections between Israelis and Palestinians who may never have had the chance to meet if not for the program. “When Israelis and Palestinians work together, in the end we are both stronger,” OGS founder Elhelo has said.

Ghrayeb and Dai both believe in the opportunity for the ventures formed at OGS to become microcosms of joint effort than can be the seeds of greater cooperation in the region. “I think that each of us is a leader and can affect our society in a positive way,” Dai said.

When the partners bring LipifAI back home, they’ll join a network of more than 100 alumni ready to help them overcome some of the obstacles their venture will inevitably face.

One of those alumni, Lobna Agbaria, returned to OGS two years ago as program director.

“As an OGS alumna, I’ve seen firsthand how powerful the connections formed through the program are, in the lives of alumni themselves and in the difference their ventures make in the region,” Agbaria said. “It’s so energizing to watch new fellows develop their ideas and their relationships with each other, and I’m excited to be able to support future fellows as they become part of the OGS community.”