I have a better solution: address the root cause of unsafe semantics by not using raw indexed for-loops, unless you absolutely need an index, in which case you should generate it with std::views::enumerate. If you want to reverse it, use std::views::enumerate | std::views::reverse. Ditto for languages with similar semantics.
If you are using C, then too bad, you're stuck with a language that doesn't allow the programmer to more meaningfully and more clearly express intent at a higher level of abstraction without paying additional runtime costs.
The article over-downplays the need for sentinel values. Surely Rust has Option that also spreads into C++ these days as std::optional. But for plain C using -1 or negative values to denote sentinels or error code is rather nice idiom.
The argument will be more valid if array indexes will be 1-based like in Fortran/Matlab/Julia as then 0 becomes extremely nice sentinel values. But C is C and needs -1.
Not disagreeing about the need for sentinel values, but Rust also has NonZero. When combined with Option, you get a similar result to outcome to C's -1.
Similar because it's a compiler hack, as in the compiler treats std::num::NonZero specially. You can't create your own type with the same properties as you can in C.
Thanks for the excerpts! I was trying to understand the reasoning, which seem to just be in the 2nd excerpt:
- The rules of signed/unsigned are complicated and there is too much auto-conversion - does that mean languages that make this more explicit means this is fine? It just seems ideal to have stronger typing.
- It is mentioned that you can initialized an unsigned int to "-2" - but that presumably could also be fixed in the language.
I'm trying to separate out which is "don't do this in C/C++" and which is "don't do this in any language".
that argument (and many others) also round to essentially "implicit imprecise integer casting is surprising but we do it anyway" which is... perhaps the actual problem???
I've made lints for Go that simply disallow implicit number casting, and omfg the (real, occurring but unnoticed) bugs it found. those kinds of lints are trivial to build, you can just stop doing it. forcing visible casts made many of these problematic patterns extremely suspicious at a glance, catching issues at review time far more easily.
> In Python, this works correctly for negative values.
They’re both “broken” in different ways. Arguably C’s brokenness is more apparent and less useful but Python also has footguns: C uses truncated division for its “modulo” so the remainder has the sign of the dividend, Python uses floored division so the remainder has the sign of the divisor instead.
The wiki page for modulo has a pretty extensive page on the subject.
I say you want a pair of operations such that (a, b) = quotient_and_remainder(x, y) gives you a and b such that a * y + b = x
The Euclidean division and remainder work, the other division and remainder also work, and they're both identical for the positive integers so people who only think about the positive integers won't even notice there's a choice here. So I like that Rust provides both pairs, in the same way Rust provides both Wrapping<T> and Saturating<T> because maybe you mean wrapping overflow or maybe you mean saturating overflow and we should make you choose not just assume we know best.
> Python uses floored division so the remainder has the sign of the divisor instead
Serious question, how is that a footgun? In decades of software development I have never needed a negative divisor for modulo. What would you use it for?
Blogger hasn't bothered to refer to those well known and detailed opinions, from very experienced authorities, and provide detailed rebuttal to those authorities claims, so his opinion can be safely ignored.
The entire article literally starts by referring to Google coding guidelines and other well known opinions as arguments against unsigned, then tries to provide a rebuttal.
signed overflow (or underflow) is frequently undefined behavior. (often because it's undefined in C)
unsigned is frequently defined. (often because it's defined in C)
tough choice.
(honestly I just lean towards "over/underflow should raise unless explicitly allowed", the ratio of unintended to intended-and-fully-checked overflow behavior is almost certainly FAR beyond 100:1)
Of course unsigned is defined. That's besides the point. The point is: how often in your code, do you expect 1 minus 2 to equal a very large number, vs. the number -1.
both seem equally undesirable to me in all cases where I intend neither. though one also risks undefined behavior, so that is strictly worse.
the reason I use a type system is to make error classes unrepresentable (where possible) or a failure. these are both leaky abstractions in the worst possible manifestation: silent misbehavior at runtime.
> Where unsigned does benefit here is when these are used as indices into an array. The signed behavior will almost certainly produce invalid indices which leads to memory unsafety issues. The unsigned way will never do that, it’ll stay bounded, even if the index it produces is actually wrong. This is a much less-severe logic bug, but can still be used maliciously depending on context.
The argument for signed often goes that it's easier to detect an invalid operation on an index by checking for bounds, or the higher probability of segmentation faults, than if your index is always “valid”. Granted, even with signed your invalid operation might still give a valid index. Simply put, seeing a negative index in your debugger is an obvious red flag you lose with unsigned.
In general I want to complain about the idea that a subtle bug is less severe than an obvious bug. Obvious bugs can be caught automatically or manually and are therefore less severe than subtle ones. This is a mistakes all students do btw. Segmentation faults are your friends!
Indeed. I can't count how often I told people that segmentation faults are among the best kind of errors. One has the complete state and not a log message missing 90% of relevant info.
Of course you don't want processes to crash all the time and spend time on writing gigabytes of data onto disk. But for the rare failure it's a good thing to have.
The entire reason for integer under/overflow to be undefined is to enable compiler optimisations. If you're going to be using unsigned anyway, we might as well drop the undefined behaviour from the standard and just say it's machine-defined. That should honestly be the correct choice. If a loop is hot enough to benefit from those optimisations, you can easily rewrite it in a form that makes the compiler assume overflow won't happen. Either using current syntax with unreachable(), or we can add a runtime_assume(expr) expression that signals to the compiler that it can assume expr is true. Though for full safety I would prefer using if(likely(expr)) {fast code} else {panic or return error}.
As an aside, unsigned does not save you from undefined behaviour. When sizeof(short)==2 and sizeof(int)==4 (e.g. x86, x64, arm32, arm64), then multiplying two unsigned short values happens by upcasting them to ints (see integer promotion rules), which can overflow the int.
My personal opinion is that along with making signed overflow defined, unsigned integers should be entirely removed as a type and there should instead be separate signed vs unsigned operators, because at the processor level there is no difference between the two, and there hasn't been a good case to separate them at the hardware level for the last ~half century. Basically, do what Java does with some syntax like unsigned{expr} which forces all integers inside expression to be treated as unsigned. Unsigned literals can stay, but they will be bitcast to signed equivalents if used outside unsigned context.
It’s pretty sad that after all these years, dealing with fixed size integers is still so complicated. Yes, many of the problems are specific to low level languages with undefined behaviour and numeric for loops. But the issue of subtracting two numbers and possibly having an underflow is both common and a bit absurd.
The code in the article for “safely” calculating the difference of two unsigned numbers, which is simpler than the equivalent for signed integers, is this little ritual:
> delta = max(x, y) - min(x, y);
Seriously??? Two function calls just for the difference of two numbers??
Why can’t such “safe” operations have some of the sweet syntactic sugar, and the underflow-rampant “ordinary” operations have the bitter medicine of ritual?
For the stuff I do, I rarely use signed types. I find the example of finding difference between two numbers to be really silly. I don’t recall ever having to do anything like that, for a simple reason: I design my code so that I always know which number will be larger than the other.
Let’s actually talk more about this max - min example. Let’s say you calculated this delta. What is it useful for in your code? Typically, in real algorithms, what you want is to not only know the delta, but also which number is larger.
> I design my code so that I always know which number will be larger than the other.
That’s good discipline, but with better tooling and languages such conditions could be enforced by the compiler such as with Dafny. But Dafny is overkill for those who only want to avoid underflow and overflow without verifying everything else about their program…
That’s what I mean when I lament the state of affairs for underflow. Subtracting numbers is such a basic operation, but for the tech stacks used by most teams it still requires a level of care to avoid errors or corruption. Our industry is building million line behemoth codebases but these basic operations still have to be designed around.
> Typically, in real algorithms, what you want is to not only know the delta, but also which number is larger.
I agree with you, and the article was more concerned about avoiding underflow than actually having a practical use for its various example idioms.
What I’d actually prefer is a guarantee that subtracting two (X) bit numbers results in an (X+1) bit number. A sort of compile-time-checked bignum implementation where the programmer never thinks about the integer sizes or which number is larger until the final step of a calculation. Obviously this gets wasteful when multiplying numbers several times. EDIT: and I’m referring to an abstraction of integer bit sizes in a programming language, I’m not expecting the instruction set or hardware to have arbitrary sized registers.
I mean, Rust calls this function you want abs_diff
let a = 1234_u32;
let b = 5678_u32;
let c = a.abs_diff(b); // Or equivalently u32::abs_diff(a, b);
Some languages hate offering convenience functions, in particular in a language like C or one of the would-be C-replacements, those functions just clog up the same namespace as everything else, so having 100 intrinsic arithmetic functions for integers feels disproportionate. The C-like languages, even those which do have methods, often forbid methods on their "built-in" or "core" types like integers so they can't do what Rust did here.
The issue is that you need dozens of minor variants for the myriad cases involving integers, especially in C. That namespace becomes rather busy.
This particular idiom for finding the absolute difference of an unsigned integer pair is pretty clean and rarely needed. In most contexts you know that one argument will always be greater than or equal to the other, so a simple subtraction will do.
Agreed, seeing that example briefly made me consider whether this blog post was a parody. Sure, it works for this exact example, by relying on i wrapping "down" to MAX_INT on the last iteration. But how long will it take the next developer who works on the code base to figure that out? Will they figure it out before or after committing changes that break it? Or worse yet, before or after shipping code?
> I really don't see what's supposedly awful about that loop
The stopping condition is incredibly confusing and non-obvious. Misleading at first glance, in fact. The whole thing is so unidiomatic that I don't think I've even seen it once in my life. It's a better contender for an underhanded C++ code contest than production code.
> but if you want to count down to x instead of 0 you just do i >= x
No you can't. That fails if x == 0. Which perfectly illustrates why using unsigned everywhere isn't so great. And I say this as someone who likes unsigned types and uses them more than average!
You should think of it as the condition that's true for all iterations, not a one-time event that halts the loop. The loop is short for this:
for(size_t i = size - 1; 0 <= i && i < size; i--){
}
Which works for both signed and unsigned numbers. It just so happens that for unsigned numbers you can omit the left-hand side of the &&, and for signed numbers you can omit the right-hand side. To support arbitrary lower bounds, you omit neither.
> You should think of it as the condition that's true for all iterations, not a one-time event that halts the loop.
(a) It's "a" condition that holds true for all iterations, not "the" condition. Plenty of other conditions can hold true across the iterations of any given loop too. In fact the one interesting thing about the loop invariant compared to any other conditions is the very fact that it is guaranteed to cease to hold immediately after the loop, assuming you don't break in the loop. Other conditions can still continue to hold. i.e. The stopping condition is the entire point of the loop invariant.
(b) I'm well aware what a loop invariant is; I've worked on compilers. I am also a human. Humans care about when a loop starts and stops. There's nothing backwards about it, it's the most straightforward way people think of loops. And it's literally why more modern languages have introduced better syntaxes that merely spell the boundaries and/or values, and skip spelling the invariants entirely.
There was a golang proposal to change Go's default int type to arbitrary precision big int. It would avoid overflow bugs, but the proposal was closed (after eight years) due to concerns about compatibility reading serialized data and performance.
as much as I'm not really a fan of Go, and I would love to spend some time in a language that defaults to unsigned ints by default to see just how pleasant or painful it really is, I do think Go makes a reasonable tradeoff here.
signed ints are rather obviously the majority decision, so that default is defensible, and their const / compile-time math is arbitrary precision. so as long as you can describe your math as either wholly const or can move all overflow-risky stuff into a const equation, you're good - Go's compiler will fail if it exceeds the bounds of whatever number you try to cast it to (implicitly or explicitly).
Related: I have a variable integer length encoding scheme that beats out LEB128, Protobuf, varint and ASN.1 while also encoding/declaring endianness (but notably, leaving signage information up to the application): https://github.com/pmarreck/BLIP
Should be (2022) apparently - surely if HN can automatically screw up titles for various reasons, we can have it add dates automatically [and sometimes get those wrong] too? DanG ?
> for instance C and C++ leave signed integer wrap undefined
I'm pretty sure the way to write what was meant here is "C and C++ leave signed integer overflow undefined". Wrapping would be a definite choice. Overflow is the situation we're considering, not a particular outcome to choose so that is what's undefined.
The confusion gets worse later when it insists that just like in C or C++ these three Rust expressions will produce invalid results because of LLVM:
x / 0
INT_MIN / -1
INT_MAX % -1
INT_MAX - INT_MIN
Assuming we defined INT_MAX and INT_MIN as say i32::MAX and i32::MIN (or whichever signed type you prefer) of course what these actually do in Rust is just panic. If you write this in a context where it'll be evaluated at compile time, your compilation fails. That's not "invalid" in any sense I understand.
It mentions Odin too, I know less about Odin but I believe it too will reject this nonsense, on Godbolt it seems to either SIGILL (for zero) or SIGFPE (for other impossible operations)
Edited: Apparently I copy-pasted wrong? Some of those invalid expressions were not as written on the blog post but are now hopefully fixed. They are, of course, still not invalid in Rust, some panic because they aren't valid questions (like dividing by zero), others are fine - neither case is a problem.
> For signed integers, the operations +, -, *, /, and << may legally overflow and the resulting value exists and is deterministically defined by the signed integer representation. Overflow does not cause a runtime panic. A compiler may not optimize code under the assumption that overflow does not occur. For instance, x < x+1 may not be assumed to be always true.
You've linked Odin's documentation. It's good to have documentation, but, because Matt Godbolt is a nice guy we can just try it out and see for ourselves
C++ 29 doesn't yet exist. It'll be finalised in (as its name suggests) 2029. Are you referring to some proposal you think has been accepted? Or is this one of those "I have concepts of a proposal?" ideas where you confused wishful thinking with reality ?
If you are using C, then too bad, you're stuck with a language that doesn't allow the programmer to more meaningfully and more clearly express intent at a higher level of abstraction without paying additional runtime costs.
This stuff compiles to broadly the same assembly.
The argument will be more valid if array indexes will be 1-based like in Fortran/Matlab/Julia as then 0 becomes extremely nice sentinel values. But C is C and needs -1.
Similar because it's a compiler hack, as in the compiler treats std::num::NonZero specially. You can't create your own type with the same properties as you can in C.
Blogger recommends unsigned over int.
Tough choice.
Even beyond Stroustrup, Dijkstra, and Google, this whole panel of C++ luminaries agrees to prefer signed types and explains pretty clearly why:
- 12:12-13:08 - https://www.youtube.com/watch?v=Puio5dly9N8#t=12m12s
- 42:40-45:26 - https://www.youtube.com/watch?v=Puio5dly9N8#t=42m40s
- 1:02:50-1:03:15 - https://www.youtube.com/watch?v=Puio5dly9N8#t=1h2m50s
- The rules of signed/unsigned are complicated and there is too much auto-conversion - does that mean languages that make this more explicit means this is fine? It just seems ideal to have stronger typing. - It is mentioned that you can initialized an unsigned int to "-2" - but that presumably could also be fixed in the language.
I'm trying to separate out which is "don't do this in C/C++" and which is "don't do this in any language".
I've made lints for Go that simply disallow implicit number casting, and omfg the (real, occurring but unnoticed) bugs it found. those kinds of lints are trivial to build, you can just stop doing it. forcing visible casts made many of these problematic patterns extremely suspicious at a glance, catching issues at review time far more easily.
In what world is using a signed value to index a normal array a good idea?
Makes for horrible footguns like:
history[counter % SIZE] = …
(One cursed day counter rolls over, becomes negative, and an out-of-bounds write occurs)
Everything went South as soon as we broke the abstraction of arrays and treated them as pointers.
Commenters here are pretty much arguing which way to hold scissors while running instead of realizing that one shouldn’t do that in the first place…
> history[counter % SIZE] = …
The footgun here is that the “modulo” operator does not actually calculate the modulo in C. In Python, this works correctly for negative values.
They’re both “broken” in different ways. Arguably C’s brokenness is more apparent and less useful but Python also has footguns: C uses truncated division for its “modulo” so the remainder has the sign of the dividend, Python uses floored division so the remainder has the sign of the divisor instead.
The wiki page for modulo has a pretty extensive page on the subject.
The Euclidean division and remainder work, the other division and remainder also work, and they're both identical for the positive integers so people who only think about the positive integers won't even notice there's a choice here. So I like that Rust provides both pairs, in the same way Rust provides both Wrapping<T> and Saturating<T> because maybe you mean wrapping overflow or maybe you mean saturating overflow and we should make you choose not just assume we know best.
Serious question, how is that a footgun? In decades of software development I have never needed a negative divisor for modulo. What would you use it for?
We should be engaging with the article's content.
Blogger hasn't bothered to refer to those well known and detailed opinions, from very experienced authorities, and provide detailed rebuttal to those authorities claims, so his opinion can be safely ignored.
The entire article literally starts by referring to Google coding guidelines and other well known opinions as arguments against unsigned, then tries to provide a rebuttal.
Unsigned is necessary only if you're working with bit fields, bit masks or require explicit modulo n-bit arithmetic.
For all other cases, use signed
unsigned is frequently defined. (often because it's defined in C)
tough choice.
(honestly I just lean towards "over/underflow should raise unless explicitly allowed", the ratio of unintended to intended-and-fully-checked overflow behavior is almost certainly FAR beyond 100:1)
the reason I use a type system is to make error classes unrepresentable (where possible) or a failure. these are both leaky abstractions in the worst possible manifestation: silent misbehavior at runtime.
The argument for signed often goes that it's easier to detect an invalid operation on an index by checking for bounds, or the higher probability of segmentation faults, than if your index is always “valid”. Granted, even with signed your invalid operation might still give a valid index. Simply put, seeing a negative index in your debugger is an obvious red flag you lose with unsigned.
In general I want to complain about the idea that a subtle bug is less severe than an obvious bug. Obvious bugs can be caught automatically or manually and are therefore less severe than subtle ones. This is a mistakes all students do btw. Segmentation faults are your friends!
Indeed. I can't count how often I told people that segmentation faults are among the best kind of errors. One has the complete state and not a log message missing 90% of relevant info.
Of course you don't want processes to crash all the time and spend time on writing gigabytes of data onto disk. But for the rare failure it's a good thing to have.
As an aside, unsigned does not save you from undefined behaviour. When sizeof(short)==2 and sizeof(int)==4 (e.g. x86, x64, arm32, arm64), then multiplying two unsigned short values happens by upcasting them to ints (see integer promotion rules), which can overflow the int.
My personal opinion is that along with making signed overflow defined, unsigned integers should be entirely removed as a type and there should instead be separate signed vs unsigned operators, because at the processor level there is no difference between the two, and there hasn't been a good case to separate them at the hardware level for the last ~half century. Basically, do what Java does with some syntax like unsigned{expr} which forces all integers inside expression to be treated as unsigned. Unsigned literals can stay, but they will be bitcast to signed equivalents if used outside unsigned context.
It’s pretty sad that after all these years, dealing with fixed size integers is still so complicated. Yes, many of the problems are specific to low level languages with undefined behaviour and numeric for loops. But the issue of subtracting two numbers and possibly having an underflow is both common and a bit absurd.
The code in the article for “safely” calculating the difference of two unsigned numbers, which is simpler than the equivalent for signed integers, is this little ritual:
> delta = max(x, y) - min(x, y);
Seriously??? Two function calls just for the difference of two numbers??
Why can’t such “safe” operations have some of the sweet syntactic sugar, and the underflow-rampant “ordinary” operations have the bitter medicine of ritual?
Let’s actually talk more about this max - min example. Let’s say you calculated this delta. What is it useful for in your code? Typically, in real algorithms, what you want is to not only know the delta, but also which number is larger.
That’s good discipline, but with better tooling and languages such conditions could be enforced by the compiler such as with Dafny. But Dafny is overkill for those who only want to avoid underflow and overflow without verifying everything else about their program…
That’s what I mean when I lament the state of affairs for underflow. Subtracting numbers is such a basic operation, but for the tech stacks used by most teams it still requires a level of care to avoid errors or corruption. Our industry is building million line behemoth codebases but these basic operations still have to be designed around.
> Typically, in real algorithms, what you want is to not only know the delta, but also which number is larger.
I agree with you, and the article was more concerned about avoiding underflow than actually having a practical use for its various example idioms.
What I’d actually prefer is a guarantee that subtracting two (X) bit numbers results in an (X+1) bit number. A sort of compile-time-checked bignum implementation where the programmer never thinks about the integer sizes or which number is larger until the final step of a calculation. Obviously this gets wasteful when multiplying numbers several times. EDIT: and I’m referring to an abstraction of integer bit sizes in a programming language, I’m not expecting the instruction set or hardware to have arbitrary sized registers.
Edited: tweaked language
This particular idiom for finding the absolute difference of an unsigned integer pair is pretty clean and rarely needed. In most contexts you know that one argument will always be greater than or equal to the other, so a simple subtraction will do.
Erm... just because you can, doesn't mean you should.
Also, what if you want to go down to something other than 0?
Agreed, seeing that example briefly made me consider whether this blog post was a parody. Sure, it works for this exact example, by relying on i wrapping "down" to MAX_INT on the last iteration. But how long will it take the next developer who works on the code base to figure that out? Will they figure it out before or after committing changes that break it? Or worse yet, before or after shipping code?
The stopping condition is incredibly confusing and non-obvious. Misleading at first glance, in fact. The whole thing is so unidiomatic that I don't think I've even seen it once in my life. It's a better contender for an underhanded C++ code contest than production code.
> but if you want to count down to x instead of 0 you just do i >= x
No you can't. That fails if x == 0. Which perfectly illustrates why using unsigned everywhere isn't so great. And I say this as someone who likes unsigned types and uses them more than average!
https://en.wikipedia.org/wiki/Loop_invariant
You should think of it as the condition that's true for all iterations, not a one-time event that halts the loop. The loop is short for this:
Which works for both signed and unsigned numbers. It just so happens that for unsigned numbers you can omit the left-hand side of the &&, and for signed numbers you can omit the right-hand side. To support arbitrary lower bounds, you omit neither.(a) It's "a" condition that holds true for all iterations, not "the" condition. Plenty of other conditions can hold true across the iterations of any given loop too. In fact the one interesting thing about the loop invariant compared to any other conditions is the very fact that it is guaranteed to cease to hold immediately after the loop, assuming you don't break in the loop. Other conditions can still continue to hold. i.e. The stopping condition is the entire point of the loop invariant.
(b) I'm well aware what a loop invariant is; I've worked on compilers. I am also a human. Humans care about when a loop starts and stops. There's nothing backwards about it, it's the most straightforward way people think of loops. And it's literally why more modern languages have introduced better syntaxes that merely spell the boundaries and/or values, and skip spelling the invariants entirely.
Exactly! That's precisely the problem with it.
(Hint: think about your code when size = 0.)
https://github.com/golang/go/issues/19623
signed ints are rather obviously the majority decision, so that default is defensible, and their const / compile-time math is arbitrary precision. so as long as you can describe your math as either wholly const or can move all overflow-risky stuff into a const equation, you're good - Go's compiler will fail if it exceeds the bounds of whatever number you try to cast it to (implicitly or explicitly).
ignoring fun with float rounding, of course.
> for instance C and C++ leave signed integer wrap undefined
I'm pretty sure the way to write what was meant here is "C and C++ leave signed integer overflow undefined". Wrapping would be a definite choice. Overflow is the situation we're considering, not a particular outcome to choose so that is what's undefined.
The confusion gets worse later when it insists that just like in C or C++ these three Rust expressions will produce invalid results because of LLVM:
Assuming we defined INT_MAX and INT_MIN as say i32::MAX and i32::MIN (or whichever signed type you prefer) of course what these actually do in Rust is just panic. If you write this in a context where it'll be evaluated at compile time, your compilation fails. That's not "invalid" in any sense I understand.It mentions Odin too, I know less about Odin but I believe it too will reject this nonsense, on Godbolt it seems to either SIGILL (for zero) or SIGFPE (for other impossible operations)
Edited: Apparently I copy-pasted wrong? Some of those invalid expressions were not as written on the blog post but are now hopefully fixed. They are, of course, still not invalid in Rust, some panic because they aren't valid questions (like dividing by zero), others are fine - neither case is a problem.
> For signed integers, the operations +, -, *, /, and << may legally overflow and the resulting value exists and is deterministically defined by the signed integer representation. Overflow does not cause a runtime panic. A compiler may not optimize code under the assumption that overflow does not occur. For instance, x < x+1 may not be assumed to be always true.
https://odin.godbolt.org/z/4WezGr7nK
In Odin, Dividing the 16-bit signed integer -32768 by -1 results in taking SIGFPE. Maybe it's not documented to do that, but that's what happens.
Not in C++29, and I think the big 3 compilers (gcc, clang, MS) will have squashed this long before that version is officially approved.
C++ 29 doesn't yet exist. It'll be finalised in (as its name suggests) 2029. Are you referring to some proposal you think has been accepted? Or is this one of those "I have concepts of a proposal?" ideas where you confused wishful thinking with reality ?