How is AI video generation technology evolving?

The development of AI video generation technology is like sitting on a rocket, from last year to this year, the changes in this field are so fast that people are dazzled. I remember last year when OpenAI released Sora, we all marveled at "AI can generate such realistic video", did not expect that now the domestic Baidu MuseSteamer can do synchronous generation of audio and video. To be honest, this rate of progress makes me both excited and a little worried - if this development continues, I won't be too surprised to see an AI-generated movie win an Oscar someday.

The underlying logic behind technological breakthroughs

Getting AI to actually understand video content is not as simple as it sounds. The traditional approach is to manually annotate video footage, but the big mainstream models are now doing the "self-learning" thing, and MuseSteamer has a clever way of doing that - they cut up the video data into small segments, and then sift through it like a gold mine to clean it up and match it up, so that the model can capture the nuances of the Chinese context more accurately. MuseSteamer used a clever approach - they sliced the video data into small segments, then sifted through it like gold, purifying and matching it so that the model could more accurately capture the nuances of the Chinese context. Put it this way, understanding the difference between "a boy who jumps up to shoot a basketball" and "a boy who dunks a basketball" is not easy for AI.

This hurdle of audio and video synchronization

The most embarrassing thing about AI video generation is that it doesn't match the shape of the mouth, and it looks particularly out of place. Instead of simply putting together audio and video, they let the model understand the complex contextual relationships between the "rhythm of speech" and the "impact of ambient sound". For example, when generating a shot of talking in the rain, the model will automatically take into account the effect of the rain on the clarity of speech, and this kind of detail processing is particularly important to enhance the sense of reality.

There's a long road ahead.

Although the technology has progressed a lot, but people who have tried it know that AI video generation still has a lot of "slots". For example, complex camera movements are often wrong, and the generation speed is not very fast. I'm actually quite curious about what the industry will optimize first - the pursuit of shorter generation time, or to solve the more difficult motion capture? A friend who works in film and television told me that what he expects most is for AI to understand more abstract directing terms, such as "Give me aWong Kar-wai styleInstead of having to explain "what color tone I want, what lens I want", I have to explain "what color tone I want, what lens I want".

Speaking of which, there's a recent model calledPikaThe product is also quite hot, focusing on the ability to generate 3D animation effects directly from the text. It can be seen that this track is becoming more and more voluminous, and everyone wants to find a breakthrough in the immediate need of "creative expression". After all, in the era of short videos, people who can quickly turn ideas into videos will always be in demand. But then again, these AI tools are no longer powerful, the final decision on the quality of the work is still the brain of the person who uses the tool.