time-to-first-audio (TTFA) is critical for realtime voice applications. open source implementations (e.g. vLLM-Omni, SGLang-Omni) are often too slow for production and can have issues with realtime playback if you push for lower latency. we wanted to fix that.
we optimized qwen3-tts, a popular OSS TTS model, to achieve 34 ms p95 TTFA at 10 requests per second on 1 x H100. we open source the implementation and benchmark, as well as a breakdown of how it was done.
Hi there! I actually thought your Dia models were amazing and very natural sounding, I haven’t tried qwen 3 tts yet - has your focus shifted away from building your Dia models and shifted more towards hosting and infrastructure?
Hey thank you for your kind words! Yes, we’ve shifted to inference but will also continue doing finetuning etc. on top of open models. Don’t have plans to do pretraining though.
We got a rtx 4090 handling around 10 concurrent requests at 50 ms TTFA after some config changes / adjustment as it doesn’t have FP8. So this 50 ms TTFA thing is very much possible on consumer hardware.
I'm going to see how this shakes out on my machine with a few 3090s. I see you all are leveraging some custom cuda kernels, so it may not work out of the box on Ampere (30xx) architecture yeah?
Having built my own voice assistant (https://github.com/acatovic/ova) and having tried many other services and models, I feel the real win is when this is on-device, and by "on-device" I mean being very inexpensive to run on a phone, and not H100. I've now been using Pocket TTS which is super fast, and also Chatterbox and Fish Audio S2 Pro (on the Mac/PC), I feel we are so close, yet so far. The quality is amazing, but can we take this to the next level and make it run on mobile? What would it take?
we haven't tried so can't say for sure, but if optimized for a scenario where the batch size is 1 and max concurrency is 1, it seems possible to get something pretty fast. i'm guessing mobile hardware specific optimizations will be important but we are not experts in that field.
Sub-50ms on text-to-speech running on an LLM is commendable.
We recently shipped text-to-speech and speech-to-text support inside Finsight (Maxint). We tapped into the platform’s native speech capabilities, which were integrated with the user’s preferred LLM inference endpoint (including local on-device models). This approach enabled us to eliminate latency and preserve user’s privacy, delivering human-like conversational experience.
Since your approach involves running the model directly, did you run into memory bandwidth contention or audio buffer underruns during bursty generation, when both the LLM and TTS models are executing concurrently?
This is right up my alley as ive been building a local voice agent for a year now. Ive tried many different models and have a custom implementation for omni voice that ive tuned for over many months. Ive never been able to achieve faster then 200ms ttfa for that model at 24 steps, but the reason is .... quality. I find that there is a lot of room for improvement in many tts models out there by a huge margin. But there is also a quality hard wall that you eventually hit that the tradeoff of faster latency but lower quality is not worth it. When making a really well sounding voice agent quality of voice, cadence, expression, etc... matters a lot. It will be interesting to try this implementation and see if its quality outputs match my expectations, if so great job indeed.
GPT‑Realtime‑2 is really weird. Perhaps just because it's bidirectional and now has the failure mode as a possibility, it responds too soon with filler at awkward times, and it's generally overeager. I feel like there was plenty of opportunity to just work on latency engineering like this effort.
this is cool but for agent scenarios unless an LLM bakes in the speech tokens directly, the latency is lost to inference, and this is what makes openai's voice model so interesting
also sweet spot is under 150ms so the remainder is inference latency turn around, a 50ms turnaround including tts-stt would ofc be the dream
that is "this ai agent is indistinguishably present and sentient" area
Qwen3 TTS has input streaming mode: you can stream LLM output into the speech model. So don’t need to wait for a full sentence. We also implement this websocket variant, and it also runs at sub 50 ms.
LLM TTFT is still a big issue, and we might tackle that problem as well.
its backchanneling frequently makes me laugh to the point of forgetting what i wanted to say. i do like it, it just takes some getting used to, especially since i've been keeping things nice and simple and taking it one step at a time for so long.
Of course speed is good, but if you don't add an artificial latency (or better, use the extra time for some QA, guardrails, etc...), the model will come off as creepy at best, and the conversation will feel awkward for the user.
Humans have a roughly 200ms auditive processing latency, (audio input to neural response), in conversation we know and account for this, such that if someone responds in 100ms, we interpret that we interrupted them and that their message doesn't come in response to what we just said, but what we said before.
This can be especially relevant in sentences where an interruption would sharply contrast.
"I think murder is bad, but.."
If someone cuts of right after the but, a human would interpret that the interjection responds to the fact that someone thinks murder is bad. Which is starkly different than interrupting someone after they are about to excuse murder.
Of course, it's impossible to know for sure what was LLM processed or not, but some of your posts (like this one) have been getting classified that way.
we optimized qwen3-tts, a popular OSS TTS model, to achieve 34 ms p95 TTFA at 10 requests per second on 1 x H100. we open source the implementation and benchmark, as well as a breakdown of how it was done.
github: https://github.com/nari-labs/nari-qwen3-tts
that one took maybe 12.677831s to generate..
DNS: 0.003747s Connect: 0.044571s TLS: 0.115428s First byte: 12.677831s Total curl: 26.268993s HTTP status: 200 Downloaded: 7695404 bytes
[1] with this template https://console.runpod.io/hub/template/pahlszv0ul?ref=0xiswp...
Running asr+llm+tts now :)
We recently shipped text-to-speech and speech-to-text support inside Finsight (Maxint). We tapped into the platform’s native speech capabilities, which were integrated with the user’s preferred LLM inference endpoint (including local on-device models). This approach enabled us to eliminate latency and preserve user’s privacy, delivering human-like conversational experience.
Since your approach involves running the model directly, did you run into memory bandwidth contention or audio buffer underruns during bursty generation, when both the LLM and TTS models are executing concurrently?
- when recording gaming footage with OBS studio with my mic plugged in, i want to convert my voice to a tts type voice in real time
- Basically I speak in my tone but the output is one of your GPT voices
- Anyone know of a library or plugin that can accomplish this in real time
docker run --rm --gpus all \ -p 8000:8000 \ -e HF_TOKEN \ -e QWEN3_TTS_PROFILE=ttfa \ -v nari-qwen3-tts-cache:/home/nari/.cache \ ghcr.io/nari-labs/nari-qwen3-tts:latest
also sweet spot is under 150ms so the remainder is inference latency turn around, a 50ms turnaround including tts-stt would ofc be the dream
that is "this ai agent is indistinguishably present and sentient" area
LLM TTFT is still a big issue, and we might tackle that problem as well.
docker run --rm --gpus all \ -p 8000:8000 \ -e HF_TOKEN \ -e QWEN3_TTS_PROFILE=ttfa \ -v nari-qwen3-tts-cache:/home/nari/.cache \ ghcr.io/nari-labs/nari-qwen3-tts:latest
Humans have a roughly 200ms auditive processing latency, (audio input to neural response), in conversation we know and account for this, such that if someone responds in 100ms, we interpret that we interrupted them and that their message doesn't come in response to what we just said, but what we said before.
This can be especially relevant in sentences where an interruption would sharply contrast.
"I think murder is bad, but.."
If someone cuts of right after the but, a human would interpret that the interjection responds to the fact that someone thinks murder is bad. Which is starkly different than interrupting someone after they are about to excuse murder.
Of course, it's impossible to know for sure what was LLM processed or not, but some of your posts (like this one) have been getting classified that way.