A staff at Tencent’s Hunyuan lab has created a brand new AI, ‘Hunyuan Video-Foley,’ that lastly brings lifelike audio to generated video. It’s designed to hearken to movies and generate a high-quality soundtrack that’s completely in sync with the motion on display.
Ever watched an AI-generated video and felt like one thing was lacking? The visuals could be beautiful, however they typically have an eerie silence that breaks the spell. Within the movie trade, the sound that fills that silence – the rustle of leaves, the clap of thunder, the clink of a glass – is named Foley artwork, and it’s a painstaking craft carried out by consultants.
Matching that stage of element is a large problem for AI. For years, automated programs have struggled to create plausible sounds for movies.
How is Tencent fixing the AI-generated audio for video drawback?
One of many greatest causes video-to-audio (V2A) fashions typically fell brief within the sound division was what the researchers name “modality imbalance”. Primarily, the AI was listening extra to the textual content prompts it was given than it was watching the precise video.
As an example, in the event you gave a mannequin a video of a busy seaside with individuals strolling and seagulls flying, however the textual content immediate solely stated “the sound of ocean waves,” you’d possible simply get the sound of waves. The AI would fully ignore the footsteps within the sand and the calls of the birds, making the scene really feel lifeless.
On high of that, the standard of the audio was typically subpar, and there merely wasn’t sufficient high-quality video with sound to coach the fashions successfully.
Tencent’s Hunyuan staff tackled these issues from three totally different angles:
- Tencent realised the AI wanted a greater training, in order that they constructed a large, 100,000-hour library of video, audio, and textual content descriptions for it to study from. They created an automatic pipeline that filtered out low-quality content material from the web, eliminating clips with lengthy silences or compressed, fuzzy audio, guaranteeing the AI discovered from the absolute best materials.
- They designed a better structure for the AI. Consider it like instructing the mannequin to correctly multitask. The system first pays extremely shut consideration to the visual-audio hyperlink to get the timing excellent—like matching the thump of a footstep to the precise second a shoe hits the pavement. As soon as it has that timing locked down, it then incorporates the textual content immediate to grasp the general temper and context of the scene. This twin strategy ensures the particular particulars of the video are by no means ignored.
- To ensure the sound was high-quality, they used a coaching technique referred to as Illustration Alignment (REPA). That is like having an knowledgeable audio engineer continuously wanting over the AI’s shoulder throughout its coaching. It compares the AI’s work to options from a pre-trained, professional-grade audio mannequin to information it in direction of producing cleaner, richer, and extra steady sound.
The outcomes communicate sound for themselves
When Tencent examined Hunyuan Video-Foley in opposition to different main AI fashions, the audio outcomes have been clear. It wasn’t simply that the computer-based metrics have been higher; human listeners constantly rated its output as larger high quality, higher matched to the video, and extra precisely timed.
Throughout the board, the AI delivered enhancements in making the sound match the on-screen motion, each by way of content material and timing. The outcomes throughout a number of analysis datasets assist this:
Tencent’s work helps to shut the hole between silent AI movies and an immersive viewing expertise with high quality audio. It’s bringing the magic of Foley artwork to the world of automated content material creation, which could possibly be a strong functionality for filmmakers, animators, and creators in all places.
See additionally: Google Vids will get AI avatars and image-to-video instruments
Need to study extra about AI and massive knowledge from trade leaders? Take a look at AI & Big Data Expo happening in Amsterdam, California, and London. The excellent occasion is a part of TechEx and is co-located with different main know-how occasions, click on here for extra info.
AI Information is powered by TechForge Media. Discover different upcoming enterprise know-how occasions and webinars here.
