The last few days I've been using AVX-512 to optimize matrix operations in a bioinformatics project, and it's great! The bottleneck in most applications is reading the large dataset from memory, so rather than doing it multiple times to compute multiple operations you can do everything in one pass (fused kernel) with AVX registers. 5x speedups are quite common. I've been doing it with manual intrinsics, but the wide crate also makes common operations completely trivial. Highly recommend checking it out.
I'd slightly rephrase the title to "everyone should know when SIMD didn't happen." Modern compliers are extremely good at vectorization until they suddenly aren't, an they'll often fall back to scalar code because if assumptions or a single-data dependent branch. Learning to check the compliers optimization reports is arguably more valuable.
Here's a helpful video about leveraging SIMD to solve a concrete performance problem for the dev team that made the game The Witness by Casey Muratori: https://www.youtube.com/watch?v=Ge3aKEmZcqY
I like SIMD, but before super-optimizing your code with SIMD and the like, really consider your data structures and access patterns.
I've been singing Data-Oriented Design's praises, so I'll just collect all my comments here [1], but I think it's a good approach to optimization. I played around with SIMD in my old code (in Zig), but my approach to modelling datastructures was so antithetical to optimization, it was like putting high-performance racing tires on a lemon with a broken engine.
It was the root-of-all-evil-type-premature-optimization, because I wasn't measuring performance, and I wasn't thinking about where the allocations were, etc. Now, I try to model my data as if it were SQL tables, see what my potential "primary keys" could be, and build my data structures around my access patterns.
For example, I used to model trees as structs pointing to other structs on the heap:
struct Tree {
tag: TreeTag,
children: Vec<&Tree>
}
Now my tree has all the bad characteristics of a linked list (* n nodes * m children), all the fragmentation of multiple heap vectors (* n nodes), and terrible set-up / tear-down time (in this case, Drop alone was taking up a good chunk of runtime).
But a tree can be represented a million ways, and can always be linearized. So now I really consider my access/insert patterns of the tree, whether it's really a tree or some other sort of graph, whether I can store it in a Vec or a Struct of Vecs, etc. Since really looking at things through their access patterns and "primary keys", my code has been much faster and simpler.
This has the added effect that a lot of your data ends up in homogeneous arrays / vecs, which means that the compiler can do its SIMD magic, the CPU can read it from your L1 cache a million times faster, etc. And then when you need to drop down into SIMD yourself, you can write some awesome branchless code.
Yeah, data layout/cache aware layouts are really key if you really want to unlock making something that ends up in a hot loop fast with SIMD.
Also, avoiding allocations or vtable lookups or a lot of indirection in the part of the code that's actually "hot" is really important. Vectors (in C++) at least aren't necessarily the best fit either, if you end up doing anything that can call an allocation unexpectedly.
> Vectors (in C++) at least aren't necessarily the best fit either
I'm not sure if you use a different allocation strategy or if you're advocating allocating as much as possible up-front, but I'm curious if you have any thoughts on this:
I always end up using (Rust) vectors despite looking at a bunch of slab/arena allocation libraries. Preferably I'd know how much memory I need up front, but barring that I see three options for any allocation that needs to grow:
- Fail;
- Reallocate; or
- Put the overflow in a new allocation, keeping track of where all the "pages" are internally.
In 2, you can't use references/slices (anything with a pointer) because the potential reallocation invalidates those. In 3, it seems ideal because references can stay stable, but you wouldn't be able to have any array-like data cross the "page" barrier as the pointer jump would not be stable. (Although, am I correct in thinking the OS does something like this, and that's why pointer addresses are virtual?).
So, you can't really internally reference data in this sort of context by reference/pointer, it's preferable to use an integer index. In that case, what's the point of the allocator libraries at all? Your standard Vector would have the same reallocation/access characteristics, and you can set a reasonable initial capacity to try and avoid reallocations.
To your virtual memory comment, the kernel can only do as it's told, but allocators can indeed tell it to shuffle pages around in virtual memory. On Linux that's `mremap` [0], and `realloc` implementations [1] use it for large enough Vecs (apparently 128kiB for glibc and musl). macOS' libmalloc will try to map new pages that extend large allocations in-place, but memcpy elsewhere if existing virtual mappings are in the way. I don't think Windows' HeapReAlloc does either, but they might reserve a larger virtual region and incrementally commit it or something to similar effect.
This is my eternal battle as a perf engineer. Performance starts with architecture and you can only squeeze so much out a hotpath with poor data layout.
The nice part is that data-oriented code almost always easily supports threading and SIMD.
On the flip side a lot of my work as a performance engineer is undoing bad abstractions made by people who read a two blog posts about SoA and decide that encapsulation is stupid
Very much agreed. Even more basic than that - memory access patterns are important. The amusing thing is that you end up writing GPU-style code even for CPU. For example - instead of an array of objects, using parquet-style object of arrays is one such trick.
There's been some good implicit/explicit discussion about SoA on this post [1]. For anyone curious, there are a few different terms that refer to basically the same thing:
- Struct of Arrays (SoA) vs Array of Structs (AoS);
- Row-major order vs column-major order; and
- Row-based vs column based / columnar (in databases)
Most code has arrays of structs (or "lists of objects", the effect is the same), and most databases are row based (same thing). But many game engines use SoA and keep heterogeneous elements together. Some databases like DuckDB do this too, this is an article about the pros and cons of columnar storage in DBs [2].
To bolster the argument, even if you do not plan to write the SIMD yourself or will "just get AI to do it", it is important to know what can be fast in SIMD (and on what hardware). That allows you to design your algorithms and structure your code so that the SIMD is possible.
Internalizing things like how data dependencies matter, how expensive it is to increase the width of your vector elements (and how to avoid the need), how to turn conditions and branches into masks, or simply things like "division does not exist" becomes a lot easier when you have spent at least some time trying to use SIMD yourself.
I think this doesn't get talked about enough. If your input is a big run of data that is being checked/transformed in one shot, it works well. But, if you're likely to have to make a decision on several bytes of the input, SIMD will be the same or slower than the scalar method. It's not a magic "go fast" button.
I think the big miss is that the former is achievable far more often than people imagine, which leads to settling for the later, aka pessimization. Reducing your allocations from millions per run to handfuls per run during initialization is effectively a "go fast" button, and is generally more reliable. It is very common for teams to spend big effort getting a 2-3x speedup on allocation-heavy code by disabling branches when a 100-1000x speedup can be had if restructuring allocations is a strategy under consideration (even before the extra 4-8x you might see if you go all the way to hand-tuned SIMD).
I've been using the Vector API in Java to get some massive speedups for flowfield generation. There's no guessing with that approach - if the hardware supports SIMD, you get it.
Most of my stuff isn't CPU-bound. Most of it is in managed languages, like Bash and C# and SQL and Python and Yaml and .tsx .
It's been at least a decade for me since SIMD was more than an implementation detail handled by the runtime. And even then it was "how do I avoid preventing the runtime from vectorizing this".
It distresses me that we don’t have a language that can do a best effort parallelization of arbitrary loop like code across SIMD, multiple threads, multiple cores and GPU with a small directive.
I don’t need it to be optimal, just … handy as an option!
The last time I brought this up here, folks offered a bunch of options that don’t quite do this, and the best candidate was this 15 year old compiler project that is Intel specific!
The problem is you need both a PL nerd and a performance nerd and while that group has some overlap so these people are not as uncommon as you’d think the task is pretty hard so you need a lot of people on it, with a bunch of funding, etc. Usually it’s just cheaper to rewrite all your code by that point and so these efforts fail
i was having a conversation with a friend recently about simd in zig (which i have recently picked up and been having a pretty good time with). i find that simd writes decently well, though there's a few weird things:
- some builtins purport to work on simd vectors but actually just unpack the vectors and do their work per-element (e.g. running `@sin()` on a `@Vector(4, f32)` will unpack the vector, run `@sin()` 4 times, and then pack it back into a vector).
- a lot of `std.math` is scalar-only (some functions support vectors, though, and i've got a pr open for one of them and plan to do more).
- i'm certainly missing some intrinsics that i get from xmmintrin.h (rcp, rsqrt, few others).
in general though i'm finding it pretty capable.
mitchell, i know you hang around some of these comments sometimes – i noticed that in ghostty you bring in some c++ libs to do the simd heavy lifting for you. any plans to port that to zig? anything missing from the language or libs that's preventing it?
> mitchell, i know you hang around some of these comments sometimes
hi im here
> i noticed that in ghostty you bring in some c++ libs to do the simd heavy lifting for you. any plans to port that to zig? anything missing from the language or libs that's preventing it?
The major limitation of Zig's vectors is that they're compile-time only. So if you're building redistributed software that compiles for a baseline CPU target, it won't be as optimized as it could be for YOUR possible machine.
Highway compiles our SIMD modules for different hardware configurations and at startup does a CPUID fingerprint to figure out which to load. That way even baseline has AVX512 etc. implementations, and we just activate the right one at runtime.
We only use Highway for our hottest hot paths that we feel benefit from that specialization.
No plans to port that (although, I spent hundreds of dollars and slop-forked it into Zig with the help of this good boy GPT and it worked great actually, but I didn't want to maintain it).
ahaaa, yeah, i don't personally do any runtime switching but i hear that as a deal-breaker from other folks.
it's interesting – i've found that zig tends to extend my vectors to the native width of the platform and then operate on them there. e.g. i had a `@Vector(2, f32)` that i was using as a demo and the generated assembly was promoting it to 256 bits and using avx2 instructions on it!
looks like it – compiled as debug (native linux x64 backend) gives me the vector type i've asked for. release modes extend to the "native" width.
this is testing in isolation as well, could be that in the midst of other vector code it changes things. llvm definitely does a great job optimising tightly-written vector code to be even faster.
> some builtins purport to work on simd vectors but actually just unpack the vectors and do their work per-element (e.g. running `@sin()` on a `@Vector(4, f32)` will unpack the vector, run `@sin()` 4 times, and then pack it back into a vector).
this is reasonable because there isn't really a generalizable "good way" to unroll trig functions for simd. if you really care about speed youre better off implementing to the precision you care about (you might not want full precision)
The loop dependency on a poly isn't all that hard for compilers to unroll, and most correctly rounded implementations are polys. You're often paying only a couple cycles' worth of stalls.
We can see this in action. LLVM implements sin() with a correctly rounded double poly [0]. Let's ignore range reduction and throw the core into compiler explorer to be autovectorized [1]. uiCA estimates a theoretical latency for the inner loop of 15 cycles, and the code achieves 18.
I suspect most custom implementations would do worse than this.
i don't disagree – i have my own internal vector lib of approximations and whatnot for various tradeoffs of precision and speed, so i just use those. it's just that zig has a pretty strong stance of "no unexpected/obscured code execution" so it was surprising to see a vector-capable function that was just a bunch of scalar functions in a trench coat.
maybe functions that don't actually support actual vector execution just shouldn't work on vector arguments. i also wouldn't expect `@sin()` to expand in-place out to a full cephes-like sin implementation. maybe a function call.
Tangentially for Go programming, the last time I looked at optimising some Go code with SIMD there were a few different options available, but they were either not maintained any more or had incomplete support and required first writing your function in C++ with intrinsics and generating assembly, then converting it to go assembly with a tool [1]. I never got my function to work in go despite the C++ code working fine. In short, not really a production ready option for Go. This was a year or two ago, though.
The Go language has long lacked official support for SIMD instructions, which means it has been at a disadvantage in terms of performance optimization. In recent years, with Go 1.26, an experimental version of the SIMD/ArchSIMD packages was introduced for AMD64 architecture. With Go 1.27, a portable version of the SIMD package was also added. Now, we can fully utilize native SIMD instructions to optimize go program performance.
If you’re processing N things at a time and your “scalar tail” is N-1 why can’t you put in a dummy value for the last entry, run one last SIMD iteration, and discard the dummy return value?
You can, but usually the code may have problems with this. For example, storing out of bounds is often going to give you a bad time. Some platforms that are all SIMD all the time will support masked operations for this kind of thing.
This is an interesting article. I don't really work with low level enough languages for this to matter (unless - does this ever show up in Javascript somehow?).
I guess I don't understand the "reduce" step. It seems like you have to be careful not to "undo" all the benefit from SIMD. Sure, it can compare 8 values in parallel, but then if you have to look at each of the 8 answers in turn you're back to where you began. Is the `@reduce()` function in the example a special Vector one that tells you if all the values are true or not in one "step"?
Isn't the better abstraction here to use a higher level library in the style of pandas/polars that will operate as vectors, compose and feel readable and inuitive, while (almost?) maxing out SIMD?
"More importantly, when this loop matters enough for me to care about a 5x speedup, I want the vectorization to be explicit and predictable. I don't want an unrelated code change or compiler update to quietly turn it back into a scalar loop."
Only tangentially related but this is by far the most painful part about optimizing code for JIT compilers like V8. Even changing a constant from 1 to 1.0 somewhere else can change the optimizations performed and lead to an unexpected performance decrease.
Everyone doesn't need to know SIMD. Mechanical sympathy is an important passive perk for software architects to cut down the number of reworks down the line, but I would rate benchmarking and being able to identify bottlenecks as more important everyday skills.
I'm working on a voxel space renderer homebrew for the PlayStation. I only have so many cycles to spend on rendering before it becomes a slideshow, so I count them in my hot rendering loop and parallelize work as much stuff as I can, even across memory load stalls from main RAM.
I've worked on a basic network accessory card with a STM32 MCU that is extremely overkill for what it needs to do. We haven't bothered making any performance or memory optimizations whatsoever, writing plain C++ almost as if we were on server-class hardware because we had such egregious margins.
The first question to ask is not whether something can leverage SIMD, it's whether the performance requirements are met or not (although it's far too easy to not care when it's not your hardware that's struggling...).
> Mechanical sympathy is an important passive perk for software architects to cut down the number of reworks down the line
We might be talking on different levels but when it comes to, on an opposite end of 'Should I use SIMD'... a databases a level of mechanical sympathy at a 'base' level is still important. e.x. row-by-row updates vs batching or bad logic where a 21k entry in clause forgot about unicode rules on columns and breaks an index [0]... is still super important.
[0] - That one is real, thanks lazy bodyshop having their people use copilot and yet, we get the same billable hours, nothing is done faster, and management is too stupid to pay attention...
One of my favourite articles is "SIMD-friendly algorithms for substring searching" by Wojciech Muła [2]. If you were unfamiliar with SIMD and just jumped into the code, it'd be incomprehensible due to the intrinsics, but the generic algorithm description at the top is pretty simple if you take some time understand it.
It blew my mind once I understood what was happening, because it's quite clever but one of those "I could've thought of that" algorithms. There was some pretty good discussion on it last year (feat. ripgrep). [2]
I always must ask, why isn’t your compiler doing this for you? I know they often aren’t because I’ve seen speed ups from writing SIMD or using the vector functions in MKL, but this is something I really think the compilers should do for us in the simple case.
no macros in zig, but yes you could metaprogram it. types are first class values at compile time so you could do that sort of specialization if you wanted.
Love Mitchell’s writing and he’s one of the few people in the industry that I truly admire.
But you need a better color scheme for your light theme my guy. Greys on greys on whites with bright pinks and light blues…it’s really hard to just read.
99% of developers should just ignore SIMD. Most projects have a lot of low hanging fruit to increase performance, and still nobody finds the time to solve them.
That doesn't mean you should default to a slower implementation for new code. If you do, you're just creating more low-hanging fruit that nobody will find time to solve.
in many cases often its faster just to switch from debug to release - compilers are good to vectorise many loops. Worth to give it a try before rewriting clean loop/code into SIMD/NEON.
I was hand-rolling NEON SIMD 15 years ago, and in many cases the compiler (clang/llvm) simply out optimized me. I kept the attempts that were better than what the compiler could already do. That was ARM NEON, quite new at the time, not SSE, and again that was 15 years ago that the compiler could already beat me much of the time. I hate the naive cult coder adage concerning premature optimization, but this might be a situation where you peruse your compiler output before you start writing code in a manner the compiler can for you. In Clang, auto-vectorization is enabled by default at optimization levels -O2 and -O3.
No kidding. I saw no less than five "the compiler will do everything for me, I trust in the magic" comments before I gave up reading through the thread.
In the article, Mitchel mentions how this doesn’t always work. In fact, as someone who’s worked in compiler development, I can say it’s a small miracle when it does work.
Case-in-point, the example in my own post doesn't auto-vectorize with LLVM or GCC at highest optimization levels. Basically, compilers will never auto-vectorize loops with an early loop break afaik.
You need to let the compiler know that there are at least 4 or 8 elements to process. This may require padding data and/or having a second loop after the main one that processes the remainder <4 or <8 elements.
You start the post with:
> There is an opportunity to use SIMD. SIMD turns those into this:
>
> for (8 byte chunk in bytes) { /* ... */ }
If you actually wrote that loop, there is a good chance the compiler (gcc specifically) will auto-vectorize.
In any case, the more manual SIMD optimizations I have seen require reworking the data altogether, not just processing N elements at a time. For example, instead of packing two 4-vectors into two registers to do a dot product, pack the XXXXs, YYYYs, etc. into 4 vectors and compute 4 dot products for the price of one. That not only requires having 4 vectors to process, but also thinking how exactly they are packed in registers.
I don't know why qurren is downvoted. You really should see if you can get the compiler to auto-vectorize first (possibly padding data structures and loops) before you write anything by hand.
Most scalar-to-SIMD conversion requires changing the design of data structures and algorithms to be effective. Compilers are required to exactly reproduce the specified data structures in a deterministic way for obvious reasons.
Even if compilers were clever enough to transform your data structures and algorithms for SIMD (they're not), the data structures are a contract that can't be unilaterally modified.
A good introduction to SoA (for anyone curious) are the two most famous Data-Oriented Design talks by Mike Acton (game engine dev) [1] and Andrew Kelley (Zig lead dev) [2] respectively.
I read a book book about DoD [3] really which confused me at first with all its talk about database table design (in a book about a high-performance C++ game engine?), but when it finally clicked it was amazing. The point is that you want to think hard about your access patterns and what could constitute good "primary keys", then model it accordingly. SoA ends up being useful a lot of the time, because having your data in homogeneous arrays/vectors is great for cache locality and branch elimination. Even without SIMD you can get huge speedups from that, but that's also where your compiler (or you as a programmer) can get incredible SIMD gains.
SoA is not a silver bullet as it may not align well with your access patterns, but it can great to add to your toolkit.
And -march=native or at least -march=x86-64-v3 or similar, alternatively identifying relevant functions and manually invoking FMV and uarch specialization via target_clones. Plus non-integer code can generally not be autovectorized in normal-math mode since FP is non-commutative.
I have no idea why you're being downvoted. HN has a fetish for SIMD, but if you are hand-rolling SIMD and you aren't writing an explicit acceleration library, you're doing it wrong. Like, 100% of the time.
Every modern language has a vectorization optimizing compiler, and through some fairly straightforward techniques this is automagic. And contrary to the various replies, unless you screwed something up compilers are really good at vectorizing on whatever hardware you're targeting, including SVE.
I respect (and fear a little) those who intentionally utilize SIMD in their implementations, but I believe it's a bit too much of a semantic shift for 2-5x performance gain. Good news is that modern compilers are more than capable of emitting SIMD code even if original source is nothing but.
Most software's poor performance would be fixed long before SIMD comes into play. Reduce obvious DB/server round trips, bad data structure/cache locality, poor algorithm complexity, doing redundant computations, etc.
> Good news is that modern compilers are more than capable of emitting SIMD code even if original source is nothing but.
They're really not (I have a whole section on it in the blog post). This example in the post doesn't auto-vectorize, for example. And its a pretty big part of the overall throughput for plain text runs (ascii or unicode). Really, the point of that section is that almost nothing auto-vectorizes, backed up by LLVM docs and published research.
Instead, writing 12 lines for a 5x gain is way easier than crossing your fingers and hope someone else pays your bills.
Bigger picture, the real point is that this stuff isn't complicated. You wouldn't copy and paste 100 lines because you hope the compiler "lifts this into a for loop", you just write the for loop cause you know how and its simple.
Similarly, the common case of "process N values in parallel" is very simple. Write a dozen lines of code you're comfortable with. No need to pray the compiler people saved your bacon.
https://docs.rs/wide/latest/wide/
https://xcancel.com/mitchellh/status/2079672171321081908#m
I've been singing Data-Oriented Design's praises, so I'll just collect all my comments here [1], but I think it's a good approach to optimization. I played around with SIMD in my old code (in Zig), but my approach to modelling datastructures was so antithetical to optimization, it was like putting high-performance racing tires on a lemon with a broken engine.
It was the root-of-all-evil-type-premature-optimization, because I wasn't measuring performance, and I wasn't thinking about where the allocations were, etc. Now, I try to model my data as if it were SQL tables, see what my potential "primary keys" could be, and build my data structures around my access patterns.
For example, I used to model trees as structs pointing to other structs on the heap:
Now my tree has all the bad characteristics of a linked list (* n nodes * m children), all the fragmentation of multiple heap vectors (* n nodes), and terrible set-up / tear-down time (in this case, Drop alone was taking up a good chunk of runtime).But a tree can be represented a million ways, and can always be linearized. So now I really consider my access/insert patterns of the tree, whether it's really a tree or some other sort of graph, whether I can store it in a Vec or a Struct of Vecs, etc. Since really looking at things through their access patterns and "primary keys", my code has been much faster and simpler.
This has the added effect that a lot of your data ends up in homogeneous arrays / vecs, which means that the compiler can do its SIMD magic, the CPU can read it from your L1 cache a million times faster, etc. And then when you need to drop down into SIMD yourself, you can write some awesome branchless code.
1. https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
Also, avoiding allocations or vtable lookups or a lot of indirection in the part of the code that's actually "hot" is really important. Vectors (in C++) at least aren't necessarily the best fit either, if you end up doing anything that can call an allocation unexpectedly.
I'm not sure if you use a different allocation strategy or if you're advocating allocating as much as possible up-front, but I'm curious if you have any thoughts on this:
I always end up using (Rust) vectors despite looking at a bunch of slab/arena allocation libraries. Preferably I'd know how much memory I need up front, but barring that I see three options for any allocation that needs to grow:
- Fail;
- Reallocate; or
- Put the overflow in a new allocation, keeping track of where all the "pages" are internally.
In 2, you can't use references/slices (anything with a pointer) because the potential reallocation invalidates those. In 3, it seems ideal because references can stay stable, but you wouldn't be able to have any array-like data cross the "page" barrier as the pointer jump would not be stable. (Although, am I correct in thinking the OS does something like this, and that's why pointer addresses are virtual?).
So, you can't really internally reference data in this sort of context by reference/pointer, it's preferable to use an integer index. In that case, what's the point of the allocator libraries at all? Your standard Vector would have the same reallocation/access characteristics, and you can set a reasonable initial capacity to try and avoid reallocations.
[0] https://man7.org/linux/man-pages/man2/mremap.2.html
[1] https://git.musl-libc.org/cgit/musl/tree/src/malloc/mallocng...
The nice part is that data-oriented code almost always easily supports threading and SIMD.
- Struct of Arrays (SoA) vs Array of Structs (AoS);
- Row-major order vs column-major order; and
- Row-based vs column based / columnar (in databases)
Most code has arrays of structs (or "lists of objects", the effect is the same), and most databases are row based (same thing). But many game engines use SoA and keep heterogeneous elements together. Some databases like DuckDB do this too, this is an article about the pros and cons of columnar storage in DBs [2].
1. https://news.ycombinator.com/item?id=49012056
2. https://motherduck.com/learn/columnar-storage-guide/
Internalizing things like how data dependencies matter, how expensive it is to increase the width of your vector elements (and how to avoid the need), how to turn conditions and branches into masks, or simply things like "division does not exist" becomes a lot easier when you have spent at least some time trying to use SIMD yourself.
I think this doesn't get talked about enough. If your input is a big run of data that is being checked/transformed in one shot, it works well. But, if you're likely to have to make a decision on several bytes of the input, SIMD will be the same or slower than the scalar method. It's not a magic "go fast" button.
Seems like he should be recommending fearless_simd [1], the Rust crate by Raph Levian and the folks at Linebender :)
More seriously, if you’re looking to add SIMD to your Rust code, that’s the package to start with.
[1] https://crates.io/crates/fearless_simd
It's been at least a decade for me since SIMD was more than an implementation detail handled by the runtime. And even then it was "how do I avoid preventing the runtime from vectorizing this".
I don’t need it to be optimal, just … handy as an option!
The last time I brought this up here, folks offered a bunch of options that don’t quite do this, and the best candidate was this 15 year old compiler project that is Intel specific!
https://ispc.github.io/
Could some programming language nerd build this?
(While you are at it give me a clear idiomatic way to pay the cost to switch from array of structs to struct of arrays)
I doubt GPU is included by most runtimes yet, but for the rest of that have you tried SQL?
- some builtins purport to work on simd vectors but actually just unpack the vectors and do their work per-element (e.g. running `@sin()` on a `@Vector(4, f32)` will unpack the vector, run `@sin()` 4 times, and then pack it back into a vector).
- a lot of `std.math` is scalar-only (some functions support vectors, though, and i've got a pr open for one of them and plan to do more).
- i'm certainly missing some intrinsics that i get from xmmintrin.h (rcp, rsqrt, few others).
in general though i'm finding it pretty capable.
mitchell, i know you hang around some of these comments sometimes – i noticed that in ghostty you bring in some c++ libs to do the simd heavy lifting for you. any plans to port that to zig? anything missing from the language or libs that's preventing it?
hi im here
> i noticed that in ghostty you bring in some c++ libs to do the simd heavy lifting for you. any plans to port that to zig? anything missing from the language or libs that's preventing it?
No plans to port it. For others, this is referencing highway: https://github.com/google/highway
The major limitation of Zig's vectors is that they're compile-time only. So if you're building redistributed software that compiles for a baseline CPU target, it won't be as optimized as it could be for YOUR possible machine.
Highway compiles our SIMD modules for different hardware configurations and at startup does a CPUID fingerprint to figure out which to load. That way even baseline has AVX512 etc. implementations, and we just activate the right one at runtime.
We only use Highway for our hottest hot paths that we feel benefit from that specialization.
No plans to port that (although, I spent hundreds of dollars and slop-forked it into Zig with the help of this good boy GPT and it worked great actually, but I didn't want to maintain it).
it's interesting – i've found that zig tends to extend my vectors to the native width of the platform and then operate on them there. e.g. i had a `@Vector(2, f32)` that i was using as a demo and the generated assembly was promoting it to 256 bits and using avx2 instructions on it!
oh interesting. though i suspect that isn't zig and thats llvm.
this is testing in isolation as well, could be that in the midst of other vector code it changes things. llvm definitely does a great job optimising tightly-written vector code to be even faster.
this is reasonable because there isn't really a generalizable "good way" to unroll trig functions for simd. if you really care about speed youre better off implementing to the precision you care about (you might not want full precision)
We can see this in action. LLVM implements sin() with a correctly rounded double poly [0]. Let's ignore range reduction and throw the core into compiler explorer to be autovectorized [1]. uiCA estimates a theoretical latency for the inner loop of 15 cycles, and the code achieves 18.
I suspect most custom implementations would do worse than this.
[0] https://github.com/llvm/llvm-project/blob/165c472d65cd62eb33...
[1] https://godbolt.org/z/rGGq8aG38
maybe functions that don't actually support actual vector execution just shouldn't work on vector arguments. i also wouldn't expect `@sin()` to expand in-place out to a full cephes-like sin implementation. maybe a function call.
Edit, there's now an experimental official library at https://go.dev/pkg/simd/archsimd/ see https://go.dev/doc/go1.26#simd and at https://github.com/golang/go/issues/78902 so things have moved since I tried it last.
[1] https://github.com/minio/c2goasm
- https://pkg.go.dev/simd/[email protected]
I guess I don't understand the "reduce" step. It seems like you have to be careful not to "undo" all the benefit from SIMD. Sure, it can compare 8 values in parallel, but then if you have to look at each of the 8 answers in turn you're back to where you began. Is the `@reduce()` function in the example a special Vector one that tells you if all the values are true or not in one "step"?
If it's true (the common case here) then you proceed to look at the next 8 bytes.
If it's false, you apply a @bitcast (turn the booleans into bits) and @ctz (find the first 0) to get the index of where it was false.
Only tangentially related but this is by far the most painful part about optimizing code for JIT compilers like V8. Even changing a constant from 1 to 1.0 somewhere else can change the optimizations performed and lead to an unexpected performance decrease.
This is especially painful with dynamic languages, like in JavaScript's case.
However JIT also have positives hence their widespread use.
I'm working on a voxel space renderer homebrew for the PlayStation. I only have so many cycles to spend on rendering before it becomes a slideshow, so I count them in my hot rendering loop and parallelize work as much stuff as I can, even across memory load stalls from main RAM.
I've worked on a basic network accessory card with a STM32 MCU that is extremely overkill for what it needs to do. We haven't bothered making any performance or memory optimizations whatsoever, writing plain C++ almost as if we were on server-class hardware because we had such egregious margins.
The first question to ask is not whether something can leverage SIMD, it's whether the performance requirements are met or not (although it's far too easy to not care when it's not your hardware that's struggling...).
We might be talking on different levels but when it comes to, on an opposite end of 'Should I use SIMD'... a databases a level of mechanical sympathy at a 'base' level is still important. e.x. row-by-row updates vs batching or bad logic where a 21k entry in clause forgot about unicode rules on columns and breaks an index [0]... is still super important.
[0] - That one is real, thanks lazy bodyshop having their people use copilot and yet, we get the same billable hours, nothing is done faster, and management is too stupid to pay attention...
Once I got past that, it was fairly simple. mcyoung's articles were also super helpful
It blew my mind once I understood what was happening, because it's quite clever but one of those "I could've thought of that" algorithms. There was some pretty good discussion on it last year (feat. ripgrep). [2]
1. http://0x80.pl/notesen/2016-11-28-simd-strfind.html
2. https://news.ycombinator.com/item?id=44274001
I bet will be easy to turn into a lib.
But you need a better color scheme for your light theme my guy. Greys on greys on whites with bright pinks and light blues…it’s really hard to just read.
I rather let the compiler auto vectorise itself, or with AI help.
https://news.ycombinator.com/newsguidelines.html
You start the post with:
> There is an opportunity to use SIMD. SIMD turns those into this: > > for (8 byte chunk in bytes) { /* ... */ }
If you actually wrote that loop, there is a good chance the compiler (gcc specifically) will auto-vectorize.
In any case, the more manual SIMD optimizations I have seen require reworking the data altogether, not just processing N elements at a time. For example, instead of packing two 4-vectors into two registers to do a dot product, pack the XXXXs, YYYYs, etc. into 4 vectors and compute 4 dot products for the price of one. That not only requires having 4 vectors to process, but also thinking how exactly they are packed in registers.
I don't know why qurren is downvoted. You really should see if you can get the compiler to auto-vectorize first (possibly padding data structures and loops) before you write anything by hand.
Even if compilers were clever enough to transform your data structures and algorithms for SIMD (they're not), the data structures are a contract that can't be unilaterally modified.
The best SIMD optimizations likely require changing your data format from AoS to SoA.
I read a book book about DoD [3] really which confused me at first with all its talk about database table design (in a book about a high-performance C++ game engine?), but when it finally clicked it was amazing. The point is that you want to think hard about your access patterns and what could constitute good "primary keys", then model it accordingly. SoA ends up being useful a lot of the time, because having your data in homogeneous arrays/vectors is great for cache locality and branch elimination. Even without SIMD you can get huge speedups from that, but that's also where your compiler (or you as a programmer) can get incredible SIMD gains.
SoA is not a silver bullet as it may not align well with your access patterns, but it can great to add to your toolkit.
---
Mike Acton: Data-Oriented Design and C++: https://www.youtube.com/watch?v=rX0ItVEVjHc
Andrew Kelley: A Practical Guide to Applying Data Oriented Design: https://www.youtube.com/watch?v=IroPQ150F6c
Richard Fabian: Data-Oriented Design: https://www.dataorienteddesign.com/dodbook/
Every modern language has a vectorization optimizing compiler, and through some fairly straightforward techniques this is automagic. And contrary to the various replies, unless you screwed something up compilers are really good at vectorizing on whatever hardware you're targeting, including SVE.
Most software's poor performance would be fixed long before SIMD comes into play. Reduce obvious DB/server round trips, bad data structure/cache locality, poor algorithm complexity, doing redundant computations, etc.
They're really not (I have a whole section on it in the blog post). This example in the post doesn't auto-vectorize, for example. And its a pretty big part of the overall throughput for plain text runs (ascii or unicode). Really, the point of that section is that almost nothing auto-vectorizes, backed up by LLVM docs and published research.
Instead, writing 12 lines for a 5x gain is way easier than crossing your fingers and hope someone else pays your bills.
Bigger picture, the real point is that this stuff isn't complicated. You wouldn't copy and paste 100 lines because you hope the compiler "lifts this into a for loop", you just write the for loop cause you know how and its simple.
Similarly, the common case of "process N values in parallel" is very simple. Write a dozen lines of code you're comfortable with. No need to pray the compiler people saved your bacon.
This is only true if you are intentionally writing code that the compiler can easily vectorise. Which is not most code.
In my mental model of optimization, the above and SIMD are basically the same thing.