AI tools help people with speech disabilities make timely jokes

A group of researchers from Cornell Tech is reimagining how know-how can help customers with speech disabilities—not simply in practical speech, but additionally in making real-time jokes throughout conversations.

The analysis explores how synthetic intelligence interfaces may help customers of augmentative and various communication (AAC) know-how ship witty remarks whereas speaking with others—a manner of speaking that is usually made unimaginable by the sluggish tempo of conventional AAC techniques.

“Humor is a vital type of social interplay and is normally taken as a trivial factor. However for somebody with a speech impairment, it’s not trivial to match the timing of the dialog,” mentioned the paper’s lead writer, Cornell Tech Ph.D. pupil Tobias Weinberg, who misplaced his capability to talk at 15 and now makes use of AAC know-how.

“Dropping my capability to talk from one month to the following, I needed to be taught to reshape my humor to this new type of communication. This analysis tries to channel that have.”

The analysis, titled “Why So Serious? Exploring Timely Humorous Comments in AAC Through AI-Powered Interfaces,” gained a Finest Paper Honorable Point out award and Jury Finest Demo award on the Affiliation for Computing Equipment’s Convention on Human Elements in Computing Techniques (CHI ’25), April 26–Might 1 in Yokohama, Japan.

The challenge included researchers at Cornell Tech and the Cornell Ann S. Bowers Faculty of Computing and Data Science, in addition to assistant professor Stephanie Valencia of the College of Maryland.

“There is a trade-off between company and effectivity. A button that offers you jokes is super-efficient however they’re seemingly not your jokes. Alternatively, typing out the entire message offers you full management, however it’s actually sluggish, and you’ll seemingly miss the timing,” Weinberg mentioned.

“We explored the area in between: How can we information the AI in direction of the joke you need to make?”

The examine launched members with speech disabilities to 4 AI-assisted interfaces: “Full-auto,” “Key phrases,” “Context Bubble Choice” and “Wizard.” Every interface supplied members a distinct steadiness of velocity and management, permitting them to both shortly generate jokes or fine-tune them to raised mirror their private voice.

Essentially the most hands-off possibility, Full-auto, permits the AI to generate content material with minimal enter. The “Key phrases” possibility provides a layer of interplay by letting customers select from a set of guiding phrases.

For these wanting extra company, the Context Bubble interface allows customers to spotlight particular components of a dialog for the AI to concentrate on, adopted by key phrase refinement. Essentially the most customizable interface, Wizard, breaks down conversations into key phrases and prompts customers to pick out associated concepts, providing the very best stage of AI enter however requiring extra effort and time.

“The trade-off between company and timing clearly performs a job in all AI-based interfaces,” mentioned challenge advisor Thijs Roumen, assistant professor at Cornell Tech. “Most of us wouldn’t be keen to give up company in any respect. It’s fascinating to see this trade-off enjoying out otherwise for various teams.”

Earlier than attempting the instruments, members expressed a choice for wanting extra company to hand-craft a joke earlier than attempting the instruments. Nevertheless, after utilizing the instruments, they favored extra time-efficient interfaces.

“The time window the place the humorous remark can land is so brief,” Weinberg mentioned. “Even when the joke just isn’t absolutely mine, it will nonetheless serve its objective—which is to make different individuals snort.”

The examine additionally revealed some challenges, together with difficulties with scrolling for customers counting on eye-gaze or joystick enter, and the shortage of emotional nuance in text-to-speech voices.

There isn’t a one-size-fits-all answer for enhancing AAC gadgets, Weinberg mentioned. “Every consumer has completely different preferences, and people preferences may even change relying on the dialog accomplice.”

“A lot work is concentrated on the direct ‘wants’ of individuals with disabilities, however empowering self-expression, humor and emotional connections are sometimes missed,” mentioned Roumen, who’s additionally the director of the Matter of Tech Lab at Cornell Tech and affiliated with Cornell Bowers CIS.

“I hope we pave a path towards extra analysis on this path.”

The paper additionally features a name to develop the scope of accessibility analysis. Weinberg mentioned, “We have to improve expressivity in AAC communication, and extra typically, allow individuals with speech impairments to completely have interaction in conversations and flourish in society.”