I’ve tested at least 20 STT models in a benchmark I’ve set up with German, Italian and English voices from meetings in my company.
The voices contained very industry specific words, the languages changed from one sentence to another, sometimes words in a language were mentioned while a discussion was in another.
The only local model that satisfies me is Voxtral Mini 3b, the only paid API that is slightly better is eleven labs. Yes, Voxtral might not reach the best score in the benchmarks, but to me, it just solves a problem. It might not be the best in terms of speed... but that’s not a problem for me.
Happy to test this new model from Google but I’m not sure I’d go with that instead of something that can run so easily in my machine.
Agree about Voxtral being the only model — local or cloud — that handles multilingual conversation really well. I‘m not sure what they do differently, but mixed-language sentences and industry terms don‘t seem to faze it where other model begin to struggle.
I've been testing it on Pixel 11 Pro and I mostly dislike it. It is convenient when you have something long to say without thinking about it first.
But the main issue is when you want to say something precise with specific wording it might "simplify" it and break the meaning.
Something like "I hesitated to check it, I should have verified" => "I should have verified" (The "I hesitated..." is removed but I said it because I wanted to let the person know that I thought about it earlier)
I tried the sentence few times and it always removed the first part.
Curious if this suffers from the same hallucination problems as Chirp or not. For example, if you pass chirp some audio with noise or even no audio, it will barf text at you like "I don't know. I don't know. I don't know." until a request timeout fires after like 10 minutes. It's... really bad. For this reason, we've moved back to Whisper for timestamp accuracy and then Gemini Flash for transcription correction.
I eval'd this pretty heavily with no audio or just noise, as for us hallucinating a response is very bad. It works great in this scenario. There are some bugs, e.g. randomly exactly 20 seconds of silence will lead to a 403, but I'm sure these aspects will be improved over time. I didn't get any hallucinations though.
I've seen Whisper do this exact thing as well. And also repeating a few words over and over near the end, words that were said but not completely at the end. It seems to overwrite the last few sentences with that so actual content is lost. It's really annoying especially because Whisper is really quite good otherwise. Sometimes a friend sends me audio messages from a noisy car and I can't make it all out but whisper gets it mostly right.
That's really impressive! It says it's available in GBoard on Android but I don't see it yet. I assume like most Google features it will be a rolling release over the course of several months, can't wait to try it some day.
It is confusing. As far as I can tell "rambler for Android" is a named subfeature of gboard and only on Pixel 10+ and newer Samsung phones. From a user standpoint it "just" seems like the transcription has gotten better - not an app you use or a feature that you enable.
I am not sure if "Word Error Rate" captures what has always been wrong with transcription. My biggest complaint is that it inserts sentence breaks in random places, then fails to evaluate the result, even though it is obviously wrong. Then I have to go fix it which can be harder than having just typed it myself, due to the difficulty of positioning the Android cursor, the fact that it automatically capitalizes if you delete a capital letter, etc. And much of the time I fail to notice the errors until later.
have another model do a pass to clean it up, saw a demo of local STT where someone did this, can fix a lot of things, especially with gotchas for the STT model in a clean-transcript.md
I think the model can even evaluate itself. If it looks afterward at an output like "do you. Want to get lunch?" in the absence of affirmative evidence that the user wanted it that way, it should be able to see that it goofed.
I'd love to know how this handles proper subtitle formatting. I'm in the process of learning many languages, and being able to cross check my own understanding with film and video would be fantastic.
Sorry everyone, I got a little bit too excited about the release. It's quite a big deal for us.
I started Ottex nine months ago with the idea that models will get to the point that they will overcome VC-funded startups, and I think it just happened. So yeah, I got a little bit overexcited...
The voices contained very industry specific words, the languages changed from one sentence to another, sometimes words in a language were mentioned while a discussion was in another.
The only local model that satisfies me is Voxtral Mini 3b, the only paid API that is slightly better is eleven labs. Yes, Voxtral might not reach the best score in the benchmarks, but to me, it just solves a problem. It might not be the best in terms of speed... but that’s not a problem for me.
Happy to test this new model from Google but I’m not sure I’d go with that instead of something that can run so easily in my machine.
Something like "I hesitated to check it, I should have verified" => "I should have verified" (The "I hesitated..." is removed but I said it because I wanted to let the person know that I thought about it earlier)
I tried the sentence few times and it always removed the first part.
https://ai.google.dev/gemini-api/docs/transcribe#transcripti...
I suppose it's a cloud thing?
I started Ottex nine months ago with the idea that models will get to the point that they will overcome VC-funded startups, and I think it just happened. So yeah, I got a little bit overexcited...