I don't understand Jev. Its a generic classifier right? Like the classifiers we were building 15 years ago with random forrests and logistic regressions, but just generic. What's so revolutionary? And how can the accuracy be any better than a custom trained classifier that can be built in a day (an hour using Claude).
The core feature is good quality out-of-the-box domain and cross-domain calibrated probability you can use as a threshold.
SO you do not need to have YOUR data to train classifier. All you need is to benchmark it on your process/data to see that it really is good for your domain (as authors promise) and choose a threshold for acting.
Ahh, and the most important thing you need to do - design well the input context (state), questions with their context (instructions) and output type (because the same question asked as Noul (yes probability) or Choice with yes/no variants give different probability of yes.
It is a classifier, that you don't have to train, that seems to work well, and they have made available by API.
One of the reasons for LLM success has not just been that they are "smart" but that they are smart while not requiring you to collect your own data and perform your own training.
There is also the marginal advantage of it not requiring a PhD to know how to use. But most companies can get a competent data scientist on board, so that is not a major problem, they could find someone to train their classifiers. But even a good data scientist can't make data appear out of nowhere. If you can get a pretty good result without needing data and training, for cheap, you're going to take it.
(Let's put aside the fact that almost every company using LLMs these days is irresponsibly NOT comparing the results with their own internal gold standard datasets for validation and calibration. They are just winging it. And Jev allows them to continue to do that.)
I don’t think its much cheaper like luna is double the cost. and you can easily use luna to generate a training set and train your own model. I think the main things its faster so a lot of people go the Wow moment I can classify my whole db at once. Or this model decides so quickly. Speed is the key differentiator, it’s a strong one, always.
Have you tried Jev and compared it against your alternative classifiers? Should take half an hour to do that, then you’ll have your answer (or someone who already did it can tell you here).
I think your engineering process is fundamentally broken if you are generating PRs with 230 files changed so regularly that you need to bolt on more AI.
You’re solving the wrong problem.
Sadly, I know (major) companies that insist it's the process that needs to be solved, because it improves velocity (for some definition of velocity that involves dropping pretty much all quality gates).
Code is there to be read and understood by the human developers who come later.
The git history is a similar record, that's why the commits that make it to main (the squashed PRs) should cover one(ish) thing each and be self contained.
Something covering 230 files should be a mechanical change like running a linter or the AI is moving an API from one signature to another.
If an LLM generated a 230 file change they are also capable of going back and breaking it up.
One thing they are bad at is comments that are succinct since they almost only ever add words.
> Code is there to be read and understood by the human developers who come later.
Have you been under a rock in the last two years?? Code is written solely by AI now, and hence it needs to be understood by AI only. Humans can still give some feedback on architecture and high level design to feel important, but even that has its days counted already.
It's funny, because in my experience, the process often looks like:
1. Human gives high-level design.
2. Agent generates wrong code with misleading comments.
3. In further iterations, agent get mislead by said code and comments, ends up generating insane workarounds.
Sadly, the hype is real and that's what's happening in many places. Businesses are eager to bet on AI cause they'd been promised x100 productivity = fire 99 or 100 devs = huge profit.
I expect one day some slopware will succumb to one of these weird production bugs, no LLM will be able to fix it, and when the biz guys ask me for a fix estimate, I'm going to say "3 years".
This is bananas. I've seen 25 year old software systems that haven't broken the 100k commits barrier. It feels like that ought to be enough for entire product lifecycles. What's going on that isn't simply wheelspinning?
You don’t need classifier to filter noise. The noise is the same patterns repeated many times. I’ve got a tool does that to break down big diff and then feed to llm for grouping.
https://github.com/thepartly/differential
Jev has good use cases that require speed. But speed is not all. Tasks like code review need quality, which costs time. If you just want speed, you might as well go with a non-reasoning model like Haiku.
Most PRs that are generated by agents today get YOLO merged because its hard for human mind to comprehend when your agent just suddenly shows up with 230 file changes. This is an attempt to reduce the mental burden by classifying each change in a review to P0, P1, P2. Only P0 are shown by default. The priorities are configurable. The diffs are also show using a natural language. The original code is one toggle away.
I yolo vibe develop, but how one end up with 210 files change commits? Unless one shooting a project, there is no way to end up with such scope?
I really don't get it.
"Plan out the change. Structure the plan so that it can be broken into individual PR's that each covers a single logical part of the change. Aim for less than 500 lines per PR unless the change is trivial and mechanical." (with a lower end model you need to be a lot more precise; with a frontier model you only really need to let it know you care about the PR size)
... in your prompts, coupled with gating PR creation on passing a stringent automated code review that also reviews it on whether or not it adheres to the rules you set regarding complexity of the PR, solves that problem without hiding the complexity.
If you have 230 file changes, that is a process failure unless they're purely obviously committing purely e.g. linter cleanups.
I'm going back to writing code by hand. AI can assist with debugging and research. I've had enough of slop. I've had enough of dealing with people wielding slop machines.
In the end... I don't feel better off. More tooling hasn't been the answer. It has only resulted in more complexity, less happiness, and about the same velocity as before we did agenetic engineering.
I don't understand the hype.
SO you do not need to have YOUR data to train classifier. All you need is to benchmark it on your process/data to see that it really is good for your domain (as authors promise) and choose a threshold for acting.
Ahh, and the most important thing you need to do - design well the input context (state), questions with their context (instructions) and output type (because the same question asked as Noul (yes probability) or Choice with yes/no variants give different probability of yes.
The "moat" is data. And compute. But mostly data.
It is a classifier, that you don't have to train, that seems to work well, and they have made available by API.
One of the reasons for LLM success has not just been that they are "smart" but that they are smart while not requiring you to collect your own data and perform your own training.
There is also the marginal advantage of it not requiring a PhD to know how to use. But most companies can get a competent data scientist on board, so that is not a major problem, they could find someone to train their classifiers. But even a good data scientist can't make data appear out of nowhere. If you can get a pretty good result without needing data and training, for cheap, you're going to take it.
(Let's put aside the fact that almost every company using LLMs these days is irresponsibly NOT comparing the results with their own internal gold standard datasets for validation and calibration. They are just winging it. And Jev allows them to continue to do that.)
Because it's all "AI" now <insert Ancient Aliens meme>
Sadly, I know (major) companies that insist it's the process that needs to be solved, because it improves velocity (for some definition of velocity that involves dropping pretty much all quality gates).
Code is there to be read and understood by the human developers who come later.
The git history is a similar record, that's why the commits that make it to main (the squashed PRs) should cover one(ish) thing each and be self contained.
Something covering 230 files should be a mechanical change like running a linter or the AI is moving an API from one signature to another.
If an LLM generated a 230 file change they are also capable of going back and breaking it up.
One thing they are bad at is comments that are succinct since they almost only ever add words.
Have you been under a rock in the last two years?? Code is written solely by AI now, and hence it needs to be understood by AI only. Humans can still give some feedback on architecture and high level design to feel important, but even that has its days counted already.
1. Human gives high-level design. 2. Agent generates wrong code with misleading comments. 3. In further iterations, agent get mislead by said code and comments, ends up generating insane workarounds.
Tbf, once the codebase is slopped enough that becomes impossible and only LLM can come later.
I expect one day some slopware will succumb to one of these weird production bugs, no LLM will be able to fix it, and when the biz guys ask me for a fix estimate, I'm going to say "3 years".
My dev branch got broken when I rebased to main after a week of drift.
I had to bisect over around a million commits to the monorepo to find the culprit.
Thank god bisect is O(log n) at least...
You either care about project and do reviews. Or yolo vibe code. This middle ground has little sense.
For the rest, I wouldn't trust an AI to pick what's important.
We can already do AI reviews, and while helpful, we know we can't trust them too much
Not quite. Ironically, they get PR-reviewed by someone else's agent. The humans in between are meat-proxies, pressing OK buttons.
"Plan out the change. Structure the plan so that it can be broken into individual PR's that each covers a single logical part of the change. Aim for less than 500 lines per PR unless the change is trivial and mechanical." (with a lower end model you need to be a lot more precise; with a frontier model you only really need to let it know you care about the PR size)
... in your prompts, coupled with gating PR creation on passing a stringent automated code review that also reviews it on whether or not it adheres to the rules you set regarding complexity of the PR, solves that problem without hiding the complexity.
If you have 230 file changes, that is a process failure unless they're purely obviously committing purely e.g. linter cleanups.
Maybe improve your development process, learn how to use AI better so you don't have to.
Maybe only if you are writing OS or some very specific kind of software you have to do this.
For all the other kinds of software, you likely don't need to.
In the end... I don't feel better off. More tooling hasn't been the answer. It has only resulted in more complexity, less happiness, and about the same velocity as before we did agenetic engineering.