Legendary. This is going to be very helpful, and the TAM is getting bigger. Thank you guys for this, and for all the learnings in-batch -- I'm excited for the future!
I reckon I could run this for buying fashion drops, is this a use case y'all have seen?
Appreciate that a lot! Yep - buying fashion drops, limited releases, ticketing, etc. are all great fits. Cua can also bypass CryptoJS-based encryption and other anti-bot measures, so it plays nicely with modern web apps out of the box.
The LLM interacts with the VM through a structured virtual computer interface (cua-computer and cua-agent). It’s a high-level abstraction that lets the agent act (e.g., “open Terminal”, “type a command”, “focus an app”) and observe (e.g., current window, file system, OCR of the screen, active processes) in a way that feels a lot more like using a real computer than parsing raw data.
So under the hood, yes, screen+metadata are used (especially with the Omni loop and visual grounding), but what the model sees is a clean interface designed for agentic workflows - closer to how a human would think about using a computer.
If you're curious, the agent loops (OpenAI, Anthropic, Omni, UI-Tars) offer different ways of reasoning and grounding actions, depending on whether you're using cloud or local models.
I don’t know if this is a problem you’ve faced, but I’m curious: how do LLM tool devs handle authn/authz? Do host apps normally forward a token or something? Is there a standard commonly used? What if the tool needs some permissions to act on the user’s behalf?
Good question! Specifically around computer-use agents (CUAs), I haven't seen much exploration yet - and I think it’s an area worth exploring for vertical products. For example, how do you securely handshake between a CUA agent and an API-based agent without exposing credentials? If everything stays within a local cluster, it's manageable, but once you start scaling out, authn/authz becomes a real headache.
I'm also working on a blog post that touches on this - particularly in the context of giving agents long-term and episodic memory. Should be out next week!
bravi! the future is the Agent OS -
How robust is the UI element detection and interaction across different apps and inside navigating complex menus?
Is it resistant to UI changes?
That's often where these automations get brittle.
UI detection’s a big focus - we use visual grounding + structured observations (like icons, OCR, app metadata, window state), so the agent can reason more like a user would. It’s surprisingly robust even with layout shifts or new themes
Thanks! Great question - those are definitely relevant, but they depend a lot on the deployment model. Since CUAs often run locally or in controlled environments (e.g. a user’s own VM or cluster), we can sidestep a lot of traditional SOC2/HIPAA concerns around centralized data handling. That said, if you're running agents across org boundaries or processing sensitive data via cloud APIs, then yeah - those frameworks absolutely come into play.
We're designing with that in mind: think fine-grained permissioning, auditability, and minimizing surface area. But it’s still early, and a lot of it depends on how teams end up using CUAs in practice.
We’re still figuring things out in public, but a few key differences:
- Open-source from the start. Cua’s built under an MIT license with the goal of making Computer-Use agents easy and accessible to build. Cua's Lume CLI was our first step - we needed fast, reproducible VMs with near-native performance to even make this possible.
- Native macOS support. As far as we know, we’re the only ones offering macOS VMs out of the box, built specifically for Computer-Use workflows. And you can control them with a PyAutoGUI-compatible SDK (cua-computer) - so things like click, type, scroll just work, without needing to deal with any inter-process communication.
- Not just the computer/sandbox, but the agent too. We’re also shipping an Agent SDK (cua-agent) that helps you build and run these workflows without having to stitch everything together yourself. It works out of the box with OpenAI and Anthropic models, UI-Tars, and basically any VLM if you’re using the OmniParser agent loop.
- Not limited to Linux. The hosted version we’re working on won’t be Linux-only - we’re going to support macOS and Windows too.
Windows host support is on our roadmap - we're currently exploring virtualization options with KVM/QEMU. Please join the discussion on our Discord: https://discord.com/invite/mVnXXpdE85
will this also be available as a hosted service? Or do you have instructions on how to manage a fleet of these manually while you're building the orchestration workflows?
Yes, we’re currently running pilots with select customers for a hosted service of Cua supporting macOS and Windows cloud instances. Feel free to reach out with your use case at [email protected]
I tried this three times. Twice a few days ago and once just now.
First time: it opened a MacOS VM and started to do stuff, but it got ahead of itself and starting typing things in the wrong place. So now that VM has a Finder window open, with a recent file that's called
plt.ylabel('Price(USD)').sh
The second and third times, it launched the VM but failed to do anything, showing these errors:
INFO:cua:VM run response: None
INFO:cua:Waiting for VM to be ready...
INFO:cua:Waiting for VM macos-sequoia-cua_latest to be ready (timeout: 600s)...
INFO:cua:VM status changed to: stopped (after 0.0s)
DEBUG:cua:Waiting for VM IP address... Current IP: None, Status: stopped
DEBUG:cua:Waiting for VM IP address... Current IP: None, Status: stopped
DEBUG:cua:Waiting for VM IP address... Current IP: None, Status: stopped
INFO:cua:VM status changed to: running (after 12.4s)
INFO:cua:VM macos-sequoia-cua_latest got IP address: 192.168.64.2 (after 12.4s)
INFO:cua:VM is ready with IP: 192.168.64.2
INFO:cua:Initializing interface for macos at 192.168.64.2
INFO:cua.interface:Logger set to INFO level
INFO:cua.interface.macos:Logger set to INFO level
INFO:cua:Connecting to WebSocket interface...
INFO:cua.interface.macos:Waiting for Computer API Server to be ready (timeout: 60s)...
INFO:cua.interface.macos:Attempting WebSocket connection to ws://192.168.64.2:8000/ws
WARNING:cua.interface.macos:Computer API Server connection lost. Will retry automatically.
INFO:cua.interface.macos:Still waiting for Computer API Server... (elapsed: 10.0s, attempts: 11)
INFO:cua.interface.macos:Still waiting for Computer API Server... (elapsed: 20.0s, attempts: 21)
INFO:cua.interface.macos:Still waiting for Computer API Server... (elapsed: 30.0s, attempts: 31)
WARNING:cua.interface.macos:Computer API Server connection lost. Will retry automatically.
INFO:cua.interface.macos:Still waiting for Computer API Server... (elapsed: 40.0s, attempts: 41)
INFO:cua.interface.macos:Still waiting for Computer API Server... (elapsed: 50.1s, attempts: 51)
ERROR:cua.interface.macos:Could not connect to 192.168.64.2 after 60 seconds
ERROR:cua:Failed to connect to WebSocket interface
DEBUG:cua:Computer initialization took 76856.09ms
ERROR:agent.core.agent:Error in agent run method: Could not connect to WebSocket interface at 192.168.64.2:8000/ws: Could not connect to 192.168.64.2 after
60 seconds
WARNING:cua.interface.macos:Computer API Server connection lost. Will retry automatically.
This was using the gradio interface, with the agent loop provider as OMNI and the model as gemma3:4b-it-q4_K_M
Thanks for trying out c/ua! We still recommend pairing the Omni loop configuration with a more capable VLM, such as Qwen2.5-VL 32B, or using a cloud LLM provider like Sonnet 3.7 or OpenAI GPT-4.1. While we believe that in the coming months we'll see better-performing quantized models that require less memory for local inference, truth is we're not quite there yet.
Stay tuned - we're also releasing support for UI-Tars-1.5 7B this week! It offers excellent speed and accuracy, and best of all, it doesn't require bounding box detection (Omni) since it's a pixel-native model.
Glad you love it! Right now, we’re relying more on the Lume CLI and its API server rather than a full Docker setup. However, we’ll soon be shipping a Docker interface that’ll handle VNC and model hosting (through docker model runner). Stay tuned for that!
https://github.com/abshkbh/arrakis
Also building in this space using MicroVMs. Currently working on a Mac port.
Would love to connect - abshkbh AT gmail.com
I love AgentDesk’s take on Kubernetes - it’s something we had considered as well, but it didn’t make much sense for macOS since you can only spin up two macOS VMs at a time due to Apple’s licensing restrictions.
I reckon I could run this for buying fashion drops, is this a use case y'all have seen?
The LLM interacts with the VM through a structured virtual computer interface (cua-computer and cua-agent). It’s a high-level abstraction that lets the agent act (e.g., “open Terminal”, “type a command”, “focus an app”) and observe (e.g., current window, file system, OCR of the screen, active processes) in a way that feels a lot more like using a real computer than parsing raw data.
So under the hood, yes, screen+metadata are used (especially with the Omni loop and visual grounding), but what the model sees is a clean interface designed for agentic workflows - closer to how a human would think about using a computer.
If you're curious, the agent loops (OpenAI, Anthropic, Omni, UI-Tars) offer different ways of reasoning and grounding actions, depending on whether you're using cloud or local models.
https://github.com/trycua/cua/tree/main/libs/agent#agent-loo...
I don’t know if this is a problem you’ve faced, but I’m curious: how do LLM tool devs handle authn/authz? Do host apps normally forward a token or something? Is there a standard commonly used? What if the tool needs some permissions to act on the user’s behalf?
I'm also working on a blog post that touches on this - particularly in the context of giving agents long-term and episodic memory. Should be out next week!
thank you e forza Cua
We're designing with that in mind: think fine-grained permissioning, auditability, and minimizing surface area. But it’s still early, and a lot of it depends on how teams end up using CUAs in practice.
- Open-source from the start. Cua’s built under an MIT license with the goal of making Computer-Use agents easy and accessible to build. Cua's Lume CLI was our first step - we needed fast, reproducible VMs with near-native performance to even make this possible.
- Native macOS support. As far as we know, we’re the only ones offering macOS VMs out of the box, built specifically for Computer-Use workflows. And you can control them with a PyAutoGUI-compatible SDK (cua-computer) - so things like click, type, scroll just work, without needing to deal with any inter-process communication.
- Not just the computer/sandbox, but the agent too. We’re also shipping an Agent SDK (cua-agent) that helps you build and run these workflows without having to stitch everything together yourself. It works out of the box with OpenAI and Anthropic models, UI-Tars, and basically any VLM if you’re using the OmniParser agent loop.
- Not limited to Linux. The hosted version we’re working on won’t be Linux-only - we’re going to support macOS and Windows too.
First time: it opened a MacOS VM and started to do stuff, but it got ahead of itself and starting typing things in the wrong place. So now that VM has a Finder window open, with a recent file that's called
The second and third times, it launched the VM but failed to do anything, showing these errors: This was using the gradio interface, with the agent loop provider as OMNI and the model as gemma3:4b-it-q4_K_MThese versions:
Stay tuned - we're also releasing support for UI-Tars-1.5 7B this week! It offers excellent speed and accuracy, and best of all, it doesn't require bounding box detection (Omni) since it's a pixel-native model.
Feel free to ping me on Discord (I'm francesco there) - happy to hop on a quick call to help debug: https://discord.com/invite/mVnXXpdE85
In the meantime, I’ll give this a shot on macOS tonight. Congrats!
Also, let us know on Discord once you’ve tried out c/ua locally on macOS: https://discord.com/invite/mVnXXpdE85
(I am not affiliated)
Also, is the project still active? No commits for 2 months is odd for a YC startup in current batch :)
Would love to chat sometime!
Feel free to join our Discord so we can chat more: https://discord.com/invite/mVnXXpdE85
Also built something on top of Browser Use (Nanobrowser) and Docker.
https://github.com/reindent/nanomachine
Just finished planning and shell capabilities
Lets chat @reindentai (X)