Constrained decoding has been in vLLM for a long time, and constrained tool call decoding has been in vLLM for a while, too.
The blog post is rather confusing because until you understand the entire situation it appears to imply that they fixed vLLM (what the title implies) but really they just needed to configure it correctly.
To be fair, the course of constrained tool calling configuration in vLLM changed over time in ways that changed behavior, which is the real problem. The Ubicloud team deserves a lot of credit for having benchmarks in place to track performance and catch regressions for investigations. Some of the providers you find on OpenRouter have inexplicably bad performance relative to others and clearly do not care this much.
However, if this article left you confused because constrained decoding and constrained tool call schema have been features in vLLM for a long time, jump to the end where they show what configuration changes they made.
In part this is because VLLM is a confusing mess. It takes a large investment in time to get it to perform well (and when it does it is really good). But it is fragile and over complicated.
I wish they'd published something about the actual impact on model quality. Schema-constrained decoding changes the distribution the LLM will output. I'd be curious how a back-up-and-retry method for tool calls (which also doesn't pollute the context and also has 100% tool-call accuracy) would compare. In their example stats (23/25 success rate), it'd be +9% tokens used for the tool call attempt itself, but that's +9% on just a small fraction of the total tokens, so if a retry solution gave any better accuracy I think it might win out.
Is the schema constraint not implemented in a way where the new distribution is identically scaled to the old distribution with retry? I had thought the way it worked was just to disallow invalid next states... oh, or is the idea that you think it is much preferred to retry at the level of the entire tool call, rather than (conceptually) per token? I guess I can see that making a lot of sense, as maybe the model gets "backed into a corner", so to speak, with a really bad initial choice and is then trying to rescue horrible states?
What the other guy means is the model is forced to decode potentially out of distribution and subsequent parts of the sequence will be sampled from a higher-entropy distribution.
i.e. it might be that a particular model may need some additional conditioning, or perhaps adjusting the order of the json output affects it to produce lower-quality results.
You have to remember that the LLM will be expected to generate 10s of thousands of tokens on a single turn, where things like quantization noise lead to cascading error further on. So when you can, you really want to stay within the model's distribution.
Amusingly constrained decoding is the same approach we use to turn an AR model into a /v1/systemone endpoint (I worked on the vLLM feature).
Tool calling is fraught with peril, especially because there's nothing that necessarily says that a models tools are fixed in place for the life of its run.
In some cases, tools are added mid-run to avoid a giant prefix miss and (at least last time I checked!), the industry has not settled on OpenAI-style chat message formats for tool call updates.
I would absolutely prefer to run in a fully-constrained tool call environment, and would love if we hadn't YOLO'd the whole thing! The PTC-style tools sort of fix this by punting tool calling to JavaScript, but that's even harder to enforce, and models not trained on the style might take a turn of failed tool calls to get it right.
llama.cpp has done this since forever. I've used other engines including vLLM here and there, and I was quite surprised that many of them don't have this feature. Glad vLLM has it now.
I'm one of the authors of this blog post. Figuring out how to integrate the LLM, the inference and serving engine, and the constrained decoding implementation such that you get the intended output and overall quality definitely requires careful engineering and testing. Please feel free to AMA. :)
So if I understand it correctly, the tool calls are not actually done by the model?
In the sense that the model manifests them, but they're not actually a result of the model doing anything but the inference engine filtering down the anything only to valid moves?
Could something like that also be used in other contexts?
I heard about how people dynamically steer models by massaging the weights a bit, but massaging the sampler(?) also sounds like that might give a lot of control over outputs with the weights staying technically the same?
Correct, here's an implementation example using the Claude SDK [0] but it's the same approach on other platforms: you tell the model what tools it has available, it sends back a response saying it wants to use a specific tool with given parameters, your code executes that tool then sends the output back to the model on the next turn.
Exactly. The LLM is your next-token predictor, which output text and tool-calls. These calls get picked up by your client/harness (e.g., opencode) which executes them and sends output back to the LLM.
Would this technique work with inference optimization techniques like DFlash and MTP? In other words does it slow down things to the point where its just better to retry?
To the LLM, the <|...|> tags are single, special tokens. They are rendered as <|...|> tags for readability, but to the model the tags are single tokens.
I do not know why these tokens are visualized as <|...|>, but I speculate that this way they are easily recognizable as special-purpose tokens, and difficult to confuse with other types of existing markup.
They are a special token that you can't just type out in text. It _renders_ as text, but cannot be input into the system as text. The model understands the difference between the textual toolcall token and the "special" one that's really just a spicy number.
(this is just made up tokenization but it's close enough to the real world)
If the user types <tool_call>, you get < (token id = 32), tool_call (41), > (102). It never decodes to <tool_call>, which is spicy number 12345, for example.
The tool call tokens are _always_ composed of multiple real tokens, so you can never get them from normal text.
The first place I think I encountered it, was GPT-2's tokenizer, back when OpenAI did actual open source stuff, maybe around 2020?
I'm guessing as with most stuff in the LLM ecosystem, someone did it, no one re-evaluated why and just copied it wholesale, and here we are. The most important point, that this sequence of text never occurs in "real text", is the main point of the specific character chosen, and seems to still work good for that purpose.
The original usage is only "<|endoftext|>" I think, which was the EOS token for GPT-2, then more and more special tokens were added as various delimiters to new model releases.
I really appreciate this. I hope LLM inference providers and Chinese LLM labs start using this latest vLLM version ASAP.
I do things that require ~300 tool calls per prompt and I have been stuck with OpenAI and Anthropic since 2024.
If people still wonder what moat OpenAI and Anthropic have: They got 100% tool-calling accuracy (using constrained decoding) and prompt caching right all the way back in 2024.
Lots of code "typos" aren't immediately syntactic errors. Misbalanced parentheses, for example, only bite when one needs to close the series. The solution is to go back and fix the original error, but that option is foreclosed to autoregressive LLMs. In the meantime, forcing the model off what it "wants" to generate might make the remaining syntactically valid tokens semantically meaningless, impairing future generation in a way that's hard to quntify.
Constrained generation for tool calls is certainly a type of syntax checking, but it's one that can be justified because the "hard" syntax choices come up front, both in the very format of the tool call itself to the name of the tool and its parameters. Errors are more likely to be detected in the same root-cause tokens that generate them.
The blog post is rather confusing because until you understand the entire situation it appears to imply that they fixed vLLM (what the title implies) but really they just needed to configure it correctly.
To be fair, the course of constrained tool calling configuration in vLLM changed over time in ways that changed behavior, which is the real problem. The Ubicloud team deserves a lot of credit for having benchmarks in place to track performance and catch regressions for investigations. Some of the providers you find on OpenRouter have inexplicably bad performance relative to others and clearly do not care this much.
However, if this article left you confused because constrained decoding and constrained tool call schema have been features in vLLM for a long time, jump to the end where they show what configuration changes they made.
i.e. it might be that a particular model may need some additional conditioning, or perhaps adjusting the order of the json output affects it to produce lower-quality results.
You have to remember that the LLM will be expected to generate 10s of thousands of tokens on a single turn, where things like quantization noise lead to cascading error further on. So when you can, you really want to stay within the model's distribution.
Tool calling is fraught with peril, especially because there's nothing that necessarily says that a models tools are fixed in place for the life of its run. In some cases, tools are added mid-run to avoid a giant prefix miss and (at least last time I checked!), the industry has not settled on OpenAI-style chat message formats for tool call updates.
I would absolutely prefer to run in a fully-constrained tool call environment, and would love if we hadn't YOLO'd the whole thing! The PTC-style tools sort of fix this by punting tool calling to JavaScript, but that's even harder to enforce, and models not trained on the style might take a turn of failed tool calls to get it right.
In the sense that the model manifests them, but they're not actually a result of the model doing anything but the inference engine filtering down the anything only to valid moves?
Could something like that also be used in other contexts?
I heard about how people dynamically steer models by massaging the weights a bit, but massaging the sampler(?) also sounds like that might give a lot of control over outputs with the weights staying technically the same?
[0] https://platform.claude.com/docs/en/agents-and-tools/tool-us...
I do not know why these tokens are visualized as <|...|>, but I speculate that this way they are easily recognizable as special-purpose tokens, and difficult to confuse with other types of existing markup.
... How did they get into the training data?
If the user types <tool_call>, you get < (token id = 32), tool_call (41), > (102). It never decodes to <tool_call>, which is spicy number 12345, for example.
The tool call tokens are _always_ composed of multiple real tokens, so you can never get them from normal text.
I'm guessing as with most stuff in the LLM ecosystem, someone did it, no one re-evaluated why and just copied it wholesale, and here we are. The most important point, that this sequence of text never occurs in "real text", is the main point of the specific character chosen, and seems to still work good for that purpose.
The original usage is only "<|endoftext|>" I think, which was the EOS token for GPT-2, then more and more special tokens were added as various delimiters to new model releases.
I do things that require ~300 tool calls per prompt and I have been stuck with OpenAI and Anthropic since 2024.
If people still wonder what moat OpenAI and Anthropic have: They got 100% tool-calling accuracy (using constrained decoding) and prompt caching right all the way back in 2024.
I have multiples of that with some regularity, what is the hangup?
https://github.com/huggingface/transformers.js/blob/main/pac...
Lots of code "typos" aren't immediately syntactic errors. Misbalanced parentheses, for example, only bite when one needs to close the series. The solution is to go back and fix the original error, but that option is foreclosed to autoregressive LLMs. In the meantime, forcing the model off what it "wants" to generate might make the remaining syntactically valid tokens semantically meaningless, impairing future generation in a way that's hard to quntify.
Constrained generation for tool calls is certainly a type of syntax checking, but it's one that can be justified because the "hard" syntax choices come up front, both in the very format of the tool call itself to the name of the tool and its parameters. Errors are more likely to be detected in the same root-cause tokens that generate them.