TL;DR: I built a virtual kubelet that lets Kubernetes offload GPU jobs to RunPod.io. Useful for burst scaling ML workloads without needing full-time cloud GPUs.

This project came out of a need while working on an internal ML-based SaaS (which didn't pan out). Initially, we used the RunPod API directly in the application, as RunPod had the most affordable GPU pricing at the time. But I also had a GPU server at home and wanted to run experiments even cheaper. Since I had good experiences with Kubernetes jobs (for CPU workloads), I installed k3s and made the home GPU node part of the cluster. The idea was simple: use the local GPU when possible, and burst to RunPod when needed. The app logic would stay clean. Kubernetes would handle the infrastructure decisions. Ideally, the same infra would scale from dev experiments to production workloads.

What Didn't Work My first attempt was a custom controller written in Go, monitoring jobs and scheduling them on RunPod. I avoided CRDs to stay compatible with the native Job API. Go was the natural choice given its strong Kubernetes ecosystem. The problem with the approach was that when overwriting pod values and creating virtual pods, this approach fought the Kubernetes scheduler constantly. Reconciliation with runpod and failed jobs lead to problems like loops. I also considered queuing stalled jobs and triggering scale-out logic, which increased the complexity further, but it became a mess. I wrote thousands of lines of Go and never got it stable.

What worked The proper way to do this is with the virtual kubelet. I used the CNCF sandbox project virtual-kubelet (https://github.com/virtual-kubelet/virtual-kubelet), which registers as a node in the cluster. Then the normal scheduler can use taints, tolerations, and node selectors to place pods. When a pod is placed on the virtual node, the controller provisions it using a third-party API, in this case, RunPod's.

Current Status It's source-available under a non-commercial license for now — I'd love to turn this into something sustainable.

I'm not affiliated with RunPod. I shared the project with RunPod, and their Head of Engineering reached out to discuss potential collaboration. We had an initial meeting, and there was interest in continuing the conversation. They asked to schedule a follow-up, but I didn't hear back to my follow ups. These things happen, people get busy or priorities shift. Regardless, I'm glad the project sparked interest and I'm open to revisiting it in the future.

Happy to answer questions or take feedback. Also open to contributors or potential use cases I haven't considered.

Show HN: Offloading GPU Workloads from Kubernetes to RunPod via Virtual Kubelet