There's nothing wrong to run CUDA on non-Nvidia hardware. CUDA has an interface that is reasonably well-designed, well-documented/reverse-engineered, and battle-tested for decades. What we need is not to invent another interface just under the name of 'open standard', but to implement the same interface. ROCm is exactly doing this, and so are other hardware SDKs such as MooreThread and Alibaba T-Head.
> Ease of programming and a giant leap in performance is one of the key reasons for the CUDA platform’s widespread adoption
This, so much. Other platforms continue to ignore developer UX, but it's one of the main things that get's new users onboard and keeps old users around.
Including stuff like Fortran, Haskell, Java, .NET via PTX, Python JIT, IDE tooling integration with major IDEs, graphical GPU debugging and profiling, libraries and co?
A guess would be some time next year — since our public launch our focus has generally been on API coverage and increasingly recently, on performance.
While performance improvements will always remain a target, we're soon at full coverage of the core CUDA APIs and will be shifting an increasing amount of effort towards developer tooling.
i'm also interested in tenstorrent. they're building GPUs with cheap GDDR6 using a fast SRAM cache, and writing their own compiler stack (used instead of CUDA) that pipelines data to the SRAM ahead-of-time so you (in theory) never need to suffer the slow speed of GDDR6 for AI workloads. also they've got built-in SFP cages where the video ports would normally be.
In this context AdaptiveCpp should also be mentioned. Started as a SYCL implementation, but recently-ish added a compiler for compiling a CUDA dialect to GPUs and CPUs from basically all vendors
SYCL is probably the most up-to-date CUDA alternative for all intents and purposes, at least if one likes modern C++ style (and lambdas inside lambdas). Expose it as C and get bindings to any other language for relatively little effort as well since it’s just C++. With AdaptiveCpp you can also compile SYCL to CUDA so both ways work with the CUDA dialect (PCUDA).
SYCL, as well as AdaptiveCpp, is a relatively active project though and has been for several years, feeding into the C++ standards committee work and is supported by several large organisations, including US national labs and several European universities. I suppose it’s worth keeping track of for people in related fields.
I suppose it’s just really hard to beat the head start and ecosystem integration NVIDIA has with CUDA.
every CUDA alternative follows the same arc: bold launch, works for 3 operations, then a Discord server where the last message is 'any updates?' from 2024
Alternatives exist, but little demand outside hyperscalers and special uses.
Neocloud customers just want plug-and-play CUDA. It works, it's tested, it adapts faster, and has known performance. Alternatives give no significant benefits.
A couple of years ago I evaluated both Vulkan and Cuda as a choice for future projects. I couldnt get anything done after a week in Vulkan, but had the test prototype project working after just a day in Cuda.
Needless to say, I'd never ever pick Vulkan for any project after that experience. It's just way to needlessly overengineered and bloated.
I used to be big into Khronos API camp, even did my project thesis in OpenGL, up to the famous Long Peaks fail.
Vulkan ended up being the same extension spaghetti as its predecessor, and Khronos was only able to come up with something thanks to AMD offering Mantle, C++ bindings and a GLSL successor only came to be thanks to NVidia (Vulkan-hpp and Slang started at NVidia).
The "we build the specification", and then "the community builds the tools", leads to very poor experiences, and if it wasn't for LunarG own interests, there wouldn't even exist any kind of Vulkan SDK.
What they have going is naturally the vendor independence, however we can achieve the same with middleware with the benefit of much better developer experience.
I love how people say things like "extension spaghetti", as if all other non-standard APIs have the same problem: hardware gets new features that people want to use from that API, API gains extension to use that hardware feature.
CUDA is no different, in fact, often worse. Nvidia is bad at documenting which hardware does what things, and CUDA users often have to use third party tables to figure out what hardware can't do what and disappoint customers who unwisely invested into it.
They really don't, no. Vulkan: 50 lines to allocate device memory. Cuda: One single line. What kind of extensive documentation stack do you want for functionality that is trivial in Cuda? And that exact issue continues through every little step of the way to your first usable application. I know there is VMA, it is a very poor solution to a problem that shouldn't even exist, and it only poorly addresses one of 100 parts of the API where Cuda is vastly simpler than Vulkan. Cuda also doesnt force you to use queue families but you can optionally use streams. No ridiculous descriptor management and binding in cuda, just passing pointers and handles via launch arguments. No overengineered explicit syncing mechanis in cuda, everything is nicely implicitly synced until you explicitly opt in to parallel streams. etc.
Weird, since the most used open source inference engine is faster on Vulkan on platforms that offer multiple options, with the sole exception being Nvidia, due to poor Nvidia driver quality (which I am forced to assume is intentional, Nvidia wishes to maintain their moat after all).
Isn't the future of the industry specialized chips like those that Broadcom and Cerebras are making? I don't know how much longer I can tolerate 50 tokens per second. It feels like the dial-up era.
Already in 2020,
https://developer.nvidia.com/blog/cuda-refresher-the-gpu-com...
This, so much. Other platforms continue to ignore developer UX, but it's one of the main things that get's new users onboard and keeps old users around.
Then I guess all the best.
Here is a tip, you don't always need to suffer from FOMO and get the very latest model card.
In fact, contrary to the competition, one can play with CUDA even on laptops, go figure.
If you were to guess, when do you think your Nsight Compute alternative might be ready with your own toolchain?
While performance improvements will always remain a target, we're soon at full coverage of the core CUDA APIs and will be shifting an increasing amount of effort towards developer tooling.
SYCL, as well as AdaptiveCpp, is a relatively active project though and has been for several years, feeding into the C++ standards committee work and is supported by several large organisations, including US national labs and several European universities. I suppose it’s worth keeping track of for people in related fields.
I suppose it’s just really hard to beat the head start and ecosystem integration NVIDIA has with CUDA.
Neocloud customers just want plug-and-play CUDA. It works, it's tested, it adapts faster, and has known performance. Alternatives give no significant benefits.
Things can change, but they are not changing now.
https://github.com/vosen/ZLUDA
No reason to tie yourself to Nvidia's moat.
Needless to say, I'd never ever pick Vulkan for any project after that experience. It's just way to needlessly overengineered and bloated.
Vulkan ended up being the same extension spaghetti as its predecessor, and Khronos was only able to come up with something thanks to AMD offering Mantle, C++ bindings and a GLSL successor only came to be thanks to NVidia (Vulkan-hpp and Slang started at NVidia).
The "we build the specification", and then "the community builds the tools", leads to very poor experiences, and if it wasn't for LunarG own interests, there wouldn't even exist any kind of Vulkan SDK.
What they have going is naturally the vendor independence, however we can achieve the same with middleware with the benefit of much better developer experience.
CUDA is no different, in fact, often worse. Nvidia is bad at documenting which hardware does what things, and CUDA users often have to use third party tables to figure out what hardware can't do what and disappoint customers who unwisely invested into it.
Profiles and API versions are much better approaches.
It is no accident than the ongoing efforts to make Vulkan more friendly are moving away from extension spaghetti into profiles.
Having to deal with closed source opaque poorly documented stacks sucks.
One of the biggest complaints we hear from the industry is "we tried to port to X and we could never complete it".
An established codebase can have years of refinement. It will take time to achieve the same with the port.
And with our compiler, just using cuda is no longer putting urself inside the moat :)
Should be real simple if the HN AI echochamber is right, right?