This is so interesting to watch. For a couple minutes I was in awe of how quick and cheap it was. Then I saw just how bad the decision are and how it would get stuck in strange loops of going in and out of the same door to no end.
This seems like a technology heading in the right direction but not quiet there yet. Excited for what they are cooking up but probably won't start building around it yet.
This entire conversation around Jev seems weird to me. Like... we started from neural nets that could do basic decision making and classifications pretty well, then trained larger and larger language models to get to where we are now. Now suddenly everyone is going crazy because someone trained a smaller model that is adequate at making decisions? We already went through the "look this AI can play pokemon terribly" phase like a decade ago.
Like others have mentioned in this post, I think a mix of models like Jev for simple stuff + a smarter reasoning model for more strategic thinking is the optimal solution. This experiment however is purely Jev. Which sometimes can be kinda dumb.
It's connected to the ROM of the actual game, so it can see a lot of things. It has multiple tools available, including being able to move to coordinates.
This is kinda chill to have in the background. I wish there were livestreams showing live reasoning of top models which are currently trying to solve cancer or whatever. Imagine the pogs in chat when it does.
Looking at the diagram in the gh repo, it looks like this is entirely jev. Are there any examples of people having a big model like Fable handle high level goals?
Hm wondering what a first pass optimal setup might be - jev for overworld navigation, escalate to sonnet for easy battles, opus for medium difficulty battles, and fable for gym bosses could probably have jev also manage all the escalation / de-escalation to different models.
Yeah I would have expected it to only decide which button to press, not something abstract like the choice of "go east to lavender town" for the goal of "in lavender town, climb the pokemon tower"
This seems like a technology heading in the right direction but not quiet there yet. Excited for what they are cooking up but probably won't start building around it yet.
The "Jev calls" counter only seems to increment at junction points like battles, conversation prompts, menus etc.
Is something else moving the character around?
All in the OSS repo if you wanna play around with it: https://github.com/christianmat/jev-pokemon
Did I miss something? I thought one of the demo videos was it doing pretty decent at the first level of Doom?
However, https://x.com/TynanSylvester/status/2096965749369720970 Astra was able to beat RimWorld. So LLMs are definitely able to drive these sorts of games to completion with their current abilities.
The interesting thing here imo is the cost and latency. So far we're at 4 badges for less than $0.5