I grew up watching foreign movies almost exclusively dubbed. French TV in the 1980s-2000s dubbed everything; it was nearly impossible to watch stuff with the original audio track. And I will say something that will probably surprise most of you: up until age 10-12 I never noticed the lips didn't match precisely the voices and I never even realized it was dubbed(!)
Even when I learned most of what I was watching was dubbed (there is so much American content on TV) it did not seem obvious to me. It certainly never bothered me. Maybe my brain subconsciously realized watching the lips was not giving out much information, so I learned to ignore lips.
Even today as an adult, if I don't know the actors and I watch a French dub, I am not able to instantly tell it is a dub. I have to watch carefully for at least 10-20 seconds to make a guess. Maybe this speaks more of the fact that French dubs are actually really well made (they will choose translations so that French word cadence matches the English word cadence). On the other hand I have watched other dubs, especially non-English movies dubbed in English, where it's blatant that they were dubbed and it is very distracting.
It was so jarring I couldn't watch it and switched to Japanese with subtitles. Their mouths looked so stiff and unnatural and out of sync with their expressions that all me and my wife could talk about was how bad it looked.
You can see a lot of open mouth/tongue stuff being skipped. Dealing with the tongue and the inside of the mouth is a huge problem with this sort of visual dubbing. Using traditional techniques, you can model teeth and gums as rigid bodies, and faces as rubber sheets (to first approximation), but tongues, for which you typically have no visual reference in any given shot, are much more difficult to model, and continuously, subtly, on the move. "AI" is the general answer to this problem nowadays, but even ML-based systems struggle to deal with the tongue issue while trying to reconcile visual appearance with animation fidelity.
It was quite jarring in Amazon's Citadel Diana. The voices were ever so slightly out of sync with the lip movements, and the audio sounded like the studio recording hadn't been processed to match the environment.
The very few shots they showed in the demo video, which one could assume were the best ones they wanted to highlight, do not look good, like it's not as bad as Ice Ice Matrix [1] but it does not look good.
A bit off-topic maybe: I really think the earlier film made by the same production team (Crazy Pictures), "The Unthinkable" [1], is a much better film than "Watch the Skies".
But apparently IMDb does not agree:
"The Unthinkable", IMDb rating 6.0
"Watch the Skies", IMDb rating 6.4
On the other hand, as a lover of film, if this will bring more audience to foreign films, then let's give it a try.
This is very much in the same vein as colorizing black and white movies, pan-and-scan for video releases (and now zoom for 4:3 content on widescreen), and dubbing instead of subtitles. Every one of those things brought in more viewers of the content, exposing more people to those films.
There’s a reason ‘may your hobby hit mainstream audiences’ is an insult. Foreign films are foreign because they don’t attract a mainstream (read: anglophone) audience. As soon as there’s mainstream money, clueless producers will decide that they want that mainstream money, and make changes to accommodate said audience. This ends up turning them into a slightly different flavour of Hollywood.
I want to be on board with the “bring more audience” but the truth is knowing how this industry works, any success with AI on this front will lead almost immediately to a total drying up of funding for professionals engaging in this work currently. I hope I’m wrong, I really do.
What professionals are doing this work already? As I understand it, this is something that can only be done by AI (or by shooting the movie twice in different languages).
audio dubbing and voice acting is a thing. They even do it in a way that takes into account the mouth movement length/timing, to make it look slightly more natural a dub.
of course, it's not super good imho, and personally i prefer subtitles with the original audio. Even if ai did a perfect job, including changing the film's frames to suit etc, i dont believe the outcome is sufficient. I want the original actor's voice timbre and intonations, which has meaning, and is lost when translated to a different language.
This is using the original actor. They re-recorded it. The AI just modifies the lips.
From TFA:
> Notably, the original actors recorded their own dialogues in English in a sound booth — Flawless AI's technology merely altered the movements of their lips in the movie.
Imo the real situation is that US distributors suck at dubbing and/or rely too much on celebrity voices rather than good voice actors that can dub properly.
I have seen some US dubs that would have been improved by a polish-style voiceover dialog
Context: I think most recent use cases of AI is slop, generally people who identify as pro AI think I’m an AI doomer. But there are some good use cases of AI.
This sounds like a good use. It reminds me of the early days of CG, it took a while for it to look good, and now it’s a tool in the toolbox alongside practical effects. This is gonna look bad for years, but I think ultimately will be a good thing. Anything that helps shrink the world and understand other cultures is a plus for me.
But man, as a film lover I know this gonna annoy me over the next 5 years.
If it looks good it's fine, but I'm not as optimistic as you.
I would argue that pan-and-scan failed long-term, perhaps even short term. I remember widescreen releases being available on VHS because a significant amount of people didn't like it.
Colorizing black and white movies...I can't think of any time I've seen that done in a commercially successful way.
I also think that a dub with lips that don't match is something people barely notice in the first place.
I would argue that pan-and-scan solved a practical problem, and while it wasn't ideal, it was also just flat out necessary for a while. Your average TV in the '80s would have been around 20", and VHS only gave you 240 lines of vertical resolution. Letterboxing a 2.39:1 movie meant cutting both of those metrics in half. Preserving the original framing doesn't mean much when a sizable portion of your audience is stuck squinting at the screen.
Some filmmakers kept this in mind when making their movies in the '80s and '90s. Films like Terminator 2 were shot open matte, allowing each shot to be framed for 4:3 and the target theatrical widescreen aspect ratio simultaneously.
I think these facts got lost on a lot of people who were die-hard widescreen LaserDisc fans. LaserDisc had twice the vertical resolution of VHS. And if you were a big enough moviebuff to have a LaserDisc collection, you probably also had a bigger than average TV.
The rest of these techniques, like colorization, seem more about making older content more palatable to newer viewers, and don't really do anything to solve a technical problem.
It may have begun as a practical problem, but the fact that the opposite now happens (zoomed 4:3 content on widescreens which are all at least 720p and where letterboxing doesn't lose anything) tells me that there is enough of a population that doesn't like "the black lines" to warrant publishing those.
David Simon went into great detail in his blog about why they had to completely re-edit The Wire scene-by-scene from source material to suit the HD format. There were scenes where the meaning was changed and details lost if they simply cropped without discretion.
I think you mean it's a technical problem. Language translation and screen size translation are both solving for the practical problem of reaching a wider audience and on more devices.
I would love a way to regularly discover and watch some top, 'hit' TV shows or movies from outside of America or Australia (my home country), or sometimes England. I have no issues watching things with subtitles, but it is quite difficult to organically find things to watch, that perhaps have 'not English' as their primary language.
For example, I will now look up the UFO Sweden show to see if I might like to watch it -- because I discovered it through here.
I have tried subscribing to a few different regular streaming services, but none seemed to work even if I pointed them in the right direction. I'm really not particular about which country of origin the production has, so long as it's "the best" or "very popular" in recent times from that place, it's worth me checking the genre and style to see if it piques my interest.
Any advice on this? [ nb. I have a similar issue with podcasts! ]
I'm learning a language, and I'd love to be able to filter Netflix titles by language, but they absolutely refuse to do it (I assume because allowing users to filter content would expose how bare the offerings are).
You might enjoy https://easterneuropeanmovies.com/, https://sovietmoviesonline.com/ or https://asian-movies-online.com/. They have both movies that are popular hits and that have been important to the craft and history of the art. Subtitles are usually from open sub-style sources so not professional quality, but I find it sometimes help with learning the language when it's more literally translated than idiomatically.
The rest of Melikyan's movies are also quite interesting and fun.
They also distribute post-Soviet russian movies and television series. You should watch the two takes on The Master and Margarita, both high profile projects based off one of the most beloved books in Russia, from two very different times in russian society:
The Soviet was especially good a adventures for kids and science-fiction, of which you'll find a lot there, for example Alice in Wonderland and Hard to be a God. You should watch the takes on the latter to get a feel for differences between soviet and post-soviet societies. If the younger version feels hard to watch you might need to watch The Green Elephant and other movies from the russian nineties to prepare yourself and better understand the feelings and mindset of the time.
And if you want you can watch old propaganda movies about the Donbass or whatever, you'll learn a lot about places you likely know very little about.
As someone who doesn't speak Swedish, what makes this particular passage interesting, and what is it about the Norrköping dialect makes it interesting?
(I have attempted to study Swedish briefly, but I feel like I need to do more sustain discovery of content to consume in order to keep me engaged long term)
I suspect the question is, will the translation to English convey the regional dialectal nuances of the original Swedish?
Decades ago I read one of Douglas Hofstadter's writings about Gorbachev's accent. (As least, I think it was Hofstader. He wrote a lot about issues of what it means to translate.)
Gorbachev had a distinct southern Russian accent, which affected how Russians viewed him.
However, Americans mostly heard Gorbachev through translation.
Should the translator use General American English? Or use a US Southern accent? Perhaps a Foghorn Leghorn accent?
When you watch a Russian movie in English, do you expect English with a Russian accent (often fake), like in The Hunt For Red October? Or do you expect to hear people talking with strong regional US accents, so characters from Saint Petersburg might have, I dunno, a Chicago accent?
We see this in Peter Jackson's movie interpretation of "The Lord of the Rings", where the actors used regional English accents to portray the social and economic standing of characters who spoke Westron.
Based on what I read about this AI-assisted translation, they are using the original actors but speaking English, which means the dialectal nuances will not be interpretable by non-Swedish viewers.
I support this simply just on the fact that maybe they wouldn't have remade Danish film "Speak no evil" (2022) which was such a great movie into one of the worst movies I've seen this year, the American film "Speak no evil" (2024)
They're admittedly pretty cagey with the parts they're willing to show, but the parts they did show look fine to me. Are you referring to the parts when they showed the 3d mask overlay on top of the characters?
I agree with the parent poster. They've chosen the best examples to show off, and yet all of them looked "off" to me. Perhaps because I was paying close attention I was more likely to notice it, but there was definitely an uncanny valley effect to it all.
It's very difficult to judge from these tiny clips. They're supposed to be showing off the technology with this video, but choose many clips with very little dialogue.
Man, this is so much better than many Bollywood movies that are re-dubbed. When the actors go back to the sound studio to re-record their lines, they don't match up with their lips exactly, and it can be off-putting.
Because "this technology" only changes the lips visually, and does not do any translation or dubbing.
> Notably, the original actors recorded their own dialogues in English in a sound booth — Flawless AI's technology merely altered the movements of their lips in the movie.
As someone that grew up in a Latin America country watching dubbed movies my whole life I welcome this technology. Dubbed movies are usually terrible with subpar performances and recycled voices.
It sucks it will kill a large field for actors, but AI has the capacity to be much better with voices and performances that mimic the actors.
> Notably, the original actors recorded their own dialogues in English in a sound booth — Flawless AI's technology merely altered the movements of their lips in the movie.
I support this, if only because I prefer dubs over subs, and this will make dubs much more worthwhile for the studio to get right, as well as better viewer experience.
To protect your security, guce.engadget.com will not allow Firefox to display the page if another site has embedded it. To see this page, you need to open it in a new window.
I don't think Ladybird is ready yet and Chromium has taken itself out of the running with manifest v3. Ie. what do you suggest that isn't Chromium or Firefox based?
Did they literally ask a machine "visually dub subtitles into this movie file for me please" and get a result? No? They didn't use AI then. They used a purpose built model in a purpose built tool to perform a single algorithmic function. how that algorithm works or what technologies it's similar to are not relevant.
We've had things we call "AI" since the 1950s. We only had text prompting as an interaction mode since ChatGPT came out in 2022. Why is prompting your test for whether something is "AI" or not?
TIL about SHRDLU. But regardless, when you say "AI" in 2025, it means something broader than that. The root of the tree may look like trying to pass the Turing test, but many of the leaves don't.
Call it what you want, but there are lots of video processing tools out now with a technology lineage which includes generative/FM/LLM/NN DNA. Not sure about this one, but I know for sure of others.
They had the original actors record the lines to dub. They didn't necessarily have the budget to re-shoot the entire movie with them saying the lines in English, so that's not work lost. The "AI" here is a program that manipulates their faces to match the English syllables. All I see here is the creation of more work for people to do, not less (dubbing/subtitling it would likely bring a smaller audience).
Even when I learned most of what I was watching was dubbed (there is so much American content on TV) it did not seem obvious to me. It certainly never bothered me. Maybe my brain subconsciously realized watching the lips was not giving out much information, so I learned to ignore lips.
Even today as an adult, if I don't know the actors and I watch a French dub, I am not able to instantly tell it is a dub. I have to watch carefully for at least 10-20 seconds to make a guess. Maybe this speaks more of the fact that French dubs are actually really well made (they will choose translations so that French word cadence matches the English word cadence). On the other hand I have watched other dubs, especially non-English movies dubbed in English, where it's blatant that they were dubbed and it is very distracting.
I'm french and I'm terrible at understanding speech when in a noisy environment for example. But I didn't grow up with tv so that's probably not it.
It was so jarring I couldn't watch it and switched to Japanese with subtitles. Their mouths looked so stiff and unnatural and out of sync with their expressions that all me and my wife could talk about was how bad it looked.
Like the Kentucky Friend Movie "Eyewitness News" segment and smart TVs.
1. https://www.youtube.com/watch?v=gnEIeVWLtbU&pp=ygUOaWNlIGljZ...
But apparently IMDb does not agree: "The Unthinkable", IMDb rating 6.0 "Watch the Skies", IMDb rating 6.4
[1] https://www.imdb.com/title/tt5227746
On the other hand, as a lover of film, if this will bring more audience to foreign films, then let's give it a try.
This is very much in the same vein as colorizing black and white movies, pan-and-scan for video releases (and now zoom for 4:3 content on widescreen), and dubbing instead of subtitles. Every one of those things brought in more viewers of the content, exposing more people to those films.
of course, it's not super good imho, and personally i prefer subtitles with the original audio. Even if ai did a perfect job, including changing the film's frames to suit etc, i dont believe the outcome is sufficient. I want the original actor's voice timbre and intonations, which has meaning, and is lost when translated to a different language.
From TFA:
> Notably, the original actors recorded their own dialogues in English in a sound booth — Flawless AI's technology merely altered the movements of their lips in the movie.
I have seen some US dubs that would have been improved by a polish-style voiceover dialog
This sounds like a good use. It reminds me of the early days of CG, it took a while for it to look good, and now it’s a tool in the toolbox alongside practical effects. This is gonna look bad for years, but I think ultimately will be a good thing. Anything that helps shrink the world and understand other cultures is a plus for me.
But man, as a film lover I know this gonna annoy me over the next 5 years.
I would argue that pan-and-scan failed long-term, perhaps even short term. I remember widescreen releases being available on VHS because a significant amount of people didn't like it.
Colorizing black and white movies...I can't think of any time I've seen that done in a commercially successful way.
I also think that a dub with lips that don't match is something people barely notice in the first place.
Some filmmakers kept this in mind when making their movies in the '80s and '90s. Films like Terminator 2 were shot open matte, allowing each shot to be framed for 4:3 and the target theatrical widescreen aspect ratio simultaneously.
I think these facts got lost on a lot of people who were die-hard widescreen LaserDisc fans. LaserDisc had twice the vertical resolution of VHS. And if you were a big enough moviebuff to have a LaserDisc collection, you probably also had a bigger than average TV.
The rest of these techniques, like colorization, seem more about making older content more palatable to newer viewers, and don't really do anything to solve a technical problem.
https://davidsimon.com/the-wire-hd-with-videos/
For example, I will now look up the UFO Sweden show to see if I might like to watch it -- because I discovered it through here.
I have tried subscribing to a few different regular streaming services, but none seemed to work even if I pointed them in the right direction. I'm really not particular about which country of origin the production has, so long as it's "the best" or "very popular" in recent times from that place, it's worth me checking the genre and style to see if it piques my interest.
Any advice on this? [ nb. I have a similar issue with podcasts! ]
Got any soviet movie recommendations that isnt straight up propaganda material? Did they even produce non propaganda?
You might want to watch Andrei Rublev, at its inception a controversial movie heavily disliked by the Soviet state:
https://sovietmoviesonline.com/drama/andrey-rublev
Edit: And then the post-soviet Fairy, which ruminates on the legacy of Tarkovsky's masterpiece.
https://sovietmoviesonline.com/fantastic/fairy
The rest of Melikyan's movies are also quite interesting and fun.
They also distribute post-Soviet russian movies and television series. You should watch the two takes on The Master and Margarita, both high profile projects based off one of the most beloved books in Russia, from two very different times in russian society:
https://sovietmoviesonline.com/drama/master-i-margarita-mini...
https://sovietmoviesonline.com/melodrama/the-master-and-marg...
The Soviet was especially good a adventures for kids and science-fiction, of which you'll find a lot there, for example Alice in Wonderland and Hard to be a God. You should watch the takes on the latter to get a feel for differences between soviet and post-soviet societies. If the younger version feels hard to watch you might need to watch The Green Elephant and other movies from the russian nineties to prepare yourself and better understand the feelings and mindset of the time.
And if you want you can watch old propaganda movies about the Donbass or whatever, you'll learn a lot about places you likely know very little about.
In particular the SMHI security guards "stället måste outoymmas, det finns en bomb i huset".
(I have attempted to study Swedish briefly, but I feel like I need to do more sustain discovery of content to consume in order to keep me engaged long term)
You can find models here:
https://huggingface.co/KBLab
More information in swedish about the process here:
https://www.kb.se/samverkan-och-utveckling/nytt-fran-kb/nyhe...
I suspect the question is, will the translation to English convey the regional dialectal nuances of the original Swedish?
Decades ago I read one of Douglas Hofstadter's writings about Gorbachev's accent. (As least, I think it was Hofstader. He wrote a lot about issues of what it means to translate.)
Gorbachev had a distinct southern Russian accent, which affected how Russians viewed him.
"Gorbachev’s southern speech is held against him." - https://time.com/archive/6732598/mikhail-gorbachev-3/
More strongly, one Redditor writes "Gorbachev couldn't fucking pronounce Azerbaijan for a living, he sounded like a self-important moralistic buffoon." - https://www.reddit.com/r/AskARussian/comments/tw3zlm/what_do...
However, Americans mostly heard Gorbachev through translation.
Should the translator use General American English? Or use a US Southern accent? Perhaps a Foghorn Leghorn accent?
When you watch a Russian movie in English, do you expect English with a Russian accent (often fake), like in The Hunt For Red October? Or do you expect to hear people talking with strong regional US accents, so characters from Saint Petersburg might have, I dunno, a Chicago accent?
We see this in Peter Jackson's movie interpretation of "The Lord of the Rings", where the actors used regional English accents to portray the social and economic standing of characters who spoke Westron.
Based on what I read about this AI-assisted translation, they are using the original actors but speaking English, which means the dialectal nuances will not be interpretable by non-Swedish viewers.
https://youtu.be/J8P3vZo-0Hc?t=169
https://youtu.be/PTngv5MmtXo?si=cCQcYtRaja5pPP6a
https://youtu.be/PTngv5MmtXo?t=105
They're admittedly pretty cagey with the parts they're willing to show, but the parts they did show look fine to me. Are you referring to the parts when they showed the 3d mask overlay on top of the characters?
The motion lines up with the words more or less but it lacks any sort of emotion or emphasis like you'd see with real actors
Seems legit lol, nothing to brush under the rug definitely.
> Notably, the original actors recorded their own dialogues in English in a sound booth — Flawless AI's technology merely altered the movements of their lips in the movie.
https://www.vanityfair.com/hollywood/story/the-brutalists-ai...
It sucks it will kill a large field for actors, but AI has the capacity to be much better with voices and performances that mimic the actors.
What WOULD bother me is AI artifacts in a movie I paid to see in theaters.
Only a matter of time, for better or worse, until translation, synthesis, and visual modification all done with AI.
> Notably, the original actors recorded their own dialogues in English in a sound booth — Flawless AI's technology merely altered the movements of their lips in the movie.
To protect your security, guce.engadget.com will not allow Firefox to display the page if another site has embedded it. To see this page, you need to open it in a new window.
Learn more…
Did they literally ask a machine "visually dub subtitles into this movie file for me please" and get a result? No? They didn't use AI then. They used a purpose built model in a purpose built tool to perform a single algorithmic function. how that algorithm works or what technologies it's similar to are not relevant.
/curmudgeon
And inference isnt? Explain your reasoning pls.
This is not even close to true. SHRDLU would be one prominent example.
> Why is prompting your test for whether something is "AI" or not?
What does the initialism AI stand for?
Call it what you want, but there are lots of video processing tools out now with a technology lineage which includes generative/FM/LLM/NN DNA. Not sure about this one, but I know for sure of others.