When cellphones on the table are welcome

With the holiday season upon us, social calendars are quickly filling up with invitations to large dinners, cocktail parties, and outings to crowded restaurants. These festive gatherings are often noisy and can prove challenging for following conversations for many people, especially those with hearing loss.

Ryan M. Corey, an assistant professor in the electrical and computer engineering department, and his former colleagues at the University of Illinois Urbana Champaign (UIUC), Manan Mittal, Kanad Sarkar, and Andrew C. Singer, developed a system to help people hear better in group settings. Their paper, “Adaptive Crosstalk Cancellation and Spatialization for Dynamic Group Conversation Enhancement Using Mobile and Wearable Devices,” was presented at the International Workshop on Acoustic Signal Enhancement last fall.

“When I talk to people with hearing loss, one of the most common complaints is about crowded restaurants,” said Corey. “Restaurants are difficult for everyone, with or without hearing loss, especially with a large group.”

Corey said conventional hearing aids and other listening devices don’t work well in noisy environments, or for group conversations. They pick up background noise and can’t suss out individual conversations, especially when multiple people are speaking at once.

Remote microphones and microphone arrays can enhance the ability to hear, but these methods require specialized hardware, which isn’t practical for the typical group gathering.

Corey and his colleagues devised a system using items most of us (even grandparents!) use on a regular basis: smartphones, wireless earbuds, or hearing aids. Each person in the group dons their hearing aid or earbuds and places their phone on the table in front of them. The earpieces and phones transmit data to each other, and adaptive signal processing generates an individualized sound mixture for each person.

“We want each user to be able to hear every other user in the group, but not background noise from other people talking nearby,” Corey said. “We also want to remove echoes of the user’s own voice, which can be distracting.”

One challenge is that the phones will record everyone at the table, not just the closest person. Known as crosstalk, this interference can be minimized by muting the microphones of everyone but the person speaking, a common tactic in larger online videoconference meetings. This would be unwieldy in a social setting, so instead of muting the microphones, their proposed system uses adaptive crosstalk cancellation to remove the voices of all other users from each mobile device.

“If I’m eating dinner with two other people, then my phone will try to focus on my voice and cancel out the voices of the other two people. Likewise, their phones will try to cancel out my voice so that I don’t hear an echo of myself when I talk,” Corey said. “Like a muting system, this cancellation system is only active while the system thinks the user isn’t talking. Unlike in a muting system, the talker’s voice won’t be cut off if the voice activity detector makes a mistake.”

He added that preserving spatial cues that help users tell which direction sound is coming from, which is critical in group conversations where multiple people might talk at once, was crucial. Spatialization filters, personalized for each listener, are sent to each device, matching the signals at the user’s two ears. This allows users to hear each other as they sound in-person, rather than sounding like they are on a conference call. These filters adapt over time as the listener moves their head, and as the talkers move around.

Corey’s team at the Listening Technology Lab at UIC continue to refine their system to improve noise reduction, and to better track motion when the users move around quickly. Current consumer wireless standards like Bluetooth are too slow for a real-time listening system, so they also hope to shorten delays in transmission.

One day, phones may be a welcome addition to the dinner table.

Corey joined UIC in 2023. He is the first jointly appointed faculty member between UIC and the Discovery Partners Institute (DPI), where he serves as a research scientist. He is part of DPI’s Applied Research and Development division, which brings together scientists from DPI’s internal research team, the University of Illinois campuses, and other partner universities to tackle research problems that can have an immediate impact on communities and businesses in the state. The team also works with industrial partners to identify opportunities for collaborative research and technology translation.

A story in the Wall Street Journal, Go Ahead, Scroll Your Smartphone This Thanksgiving; Our devices can save us from awkward interactions and make conversations better, by Callum Borchers, included Corey’s research.