To be honest, when I first saw the basketball shooting video generated by Veo3.1, what surprised me most was not the fineness of the picture, but the proper sound processing - the sound of the basketball hitting the floor matched perfectly with the bouncing action in the picture, and the audience's cheers gradually diminished with the shooting action. The precision of this audio-visual synchronization is almost as if it was manually matched by a professional sound engineer! So the question is, how exactly did the AI do this?

Co-generation mechanisms for audio and video
In fact, the secret of Veo3.1 to achieve audio-video synchronization is that it uses a joint multimodal training approach. Simply put, the model is not trained to treat video and audio separately, but to treat them as a whole. Imagine that during the training process, the model observes thousands of video clips with audio at the same time, and slowly learns the correspondence between the sound and the picture. For example, the sound of a basketball hitting the ground should occur at the moment the ball touches the ground, not earlier or later. This type of training allowed the model to establish an intrinsic connection between sound and visual elements.
When generating the actual video, Veo3.1 analyzes the audio cues in the textual cues, such as "the sound of the basketball hitting the floor" or "the cheers of the audience", and then generates the corresponding visual elements and sound waveforms simultaneously. This process is not simply sticking pre-recorded sound on the screen, but actually generating audio signals from scratch to match the action on the screen. According to the technical documents disclosed by Google, they use a Transformer-based architecture that can simultaneously process visual and auditory information, which makes me wonder, has AI developed to this extent now?
The subtleties of the time alignment algorithm
More in-depth, Veo 3.1 handles the time dimension particularly well. It uses a temporal alignment algorithm based on the attention mechanism to accurately calculate the temporal relationship between each action and the corresponding sound. For example, when generating a basketball shooting scene, the model not only has to take into account the physical trajectory of the basketball, but also has to predict how loud the basketball will sound when it hits the ground - depending on the height of the fall, the ground material, and other factors. Interestingly, in my testing, I found that when the basketball falls from different heights, the impact sound generated by Veo3.1 has subtle differences in loudness and timbre, and this kind of detail processing is really very attentive.
But then again, today's audio-video synchronization technology is not perfect. In some complex scenes, such as multi-person conversations, the synchronization of lips and voice may still occasionally show slight deviations. But in terms of overall performance, the audio-video synchronization accuracy demonstrated by Veo3.1 in the basketball case has reached a surprising level - the ebb and flow of the crowd's cheers echoed perfectly with the rhythm of the shooting action, and the sense of immersion was truly impressive.
Development from laboratory to practical applications
Watching these technological advances, I can't help but wonder how such audio-video synchronization technology will evolve in the future. Perhaps in the near future, we will be able to use AI to generate fully anamorphic movie clips that accurately reproduce even the smallest sound details.Veo3.1 has already shown us this possibility, and although it still has room for improvement in certain physical details, the performance of the audio-video synchronization is really eye-opening. In the end, this technological breakthrough not only makes AI video generation more realistic, but more importantly, it provides creators with a more powerful narrative tool, allowing each image to have a sound world that perfectly matches it.
Comment List (13):
Load More Comments Loading...