Wow, what a bug. I expect the exploit is literally as simple as trying to ssh in as a user with the password has ‘*’ in /etc/shadow and typing anything as the password.
To the extent that systems that are configured in the vulnerable manner exist in the wild, I’m surprised they weren’t exploited years ago by trivial ssh bots.
P.S. Wow, I’m reminded of just how awful traditional UNIX-y C APIs and code are.
P.P.S. The musl code in question is nuts. The API it implements is awful. Seriously, here’s the manpage:
"It starts with a bold claim: "ssh accounts that should not accept a password will accept any password between 256 and 1024 characters". But then it keeps adding qualifiers:
- sshd compiled with musl
- not using PAM
- password logins enabled
- at least one account with '*' as the password "hash"
- at least one OTHER account with a valid password
The conditions are all ANDed, miss a single one and you're fine."
This doesn't seem to be nearly as big as an issue as they're making it out to be
Disabling PAM has been "generally good security advice" for decades; I remember suggestions to not use it in the 2000s.
Password logins are often enabled, especially on multi-user and default setups.
If alpine uses the * feature, it's going to be pretty common. Gentoo has it, as does Ubuntu.
If password logins are enabled, you're almost guaranteed to have at least one user with a password (root on all my boxes has one, even if I never use it and it's random garbage).
So the cheese lines up pretty well, this affects quite a bit.
If password logins are enabled, someone almost certainly has a password.
Yeah, as the dust starts to settle a bit it seems this is the largest case. Many 'stock' Alpine images, Docker images built on those, and embedded systems seem to be the main candidates.
No, no don't let all of that distract you. We gotta hype this. This is the new SSH2SHELL dude. Have we registered the domain yet? I'm scheduled on Fox News tomorrow.
> We are making this release as fast as possible (we learned of the problem yesterday) because we have recently observed multiple cases of rapid bug rediscovery by AI agents, sometimes within hours, and so now consider long embargo periods to be a dangerous practice that primarily benefits attackers.
I think this will likely become standard. Nevertheless, a few stormy months are ahead.
How do you break back into your digital life? You wake up in a strange place. Something happened. Everything's gone. No wallet/keys/smartphone/tablet/laptop. How would you get back into your digital life given only a laptop with access to the Internet, unless something, somewhere is passwordauthenticate=yes.
i guess if i have some sort of internet access (otherwise why would i be worrying about passwords), i'd set up a new account(s) and message those friends
There's been a natural disaster and your home is now inaccessible. With global warming going the way it has, that's looking more and more likely each day. Fires, hurricanes, earthquakes, war, tsunami, etc.
PasswordAuthentication=no usually means that ssh stops directly reading /etc/passwd to authenticate, not that it stops accepting passwords completely. You'll still be able to log in with a correct password, provided KbdInteractiveAuthentication is still on, PAM is available, and the PAM policy allows for it.
The email at the root of my identity is tied to a country code domain in my country. There are robust legal processes for enforcing my status as the registrant and regaining control of it. Once I have access to that, I can use that to gain access to everything downstream.
Is it fast? No, but that's a feature.
But it does mean I get to lean on the existing processes and identity verification already part of the government bureaucracy and the courts as my backdoor.
Remotely: An encrypted file with a seed / master passwords to all the other data, published under a public url you can remember, but not linked anywhere and not listed.
Locally: Same, but printed out, laminated, in a safe place with other documents.
I walk to my pc and type ssh-keygen. If i can’t do that I’ve got bigger problems. “I must be able to recover remotely using a memorised credential” is a terrible requirement.
Yeah your vault has password authentication. Your password manager in other words. Your vault is where you store your private ssh keys, and or the passwords for the keys.
> You must not be in a place with many huge natural disasters.
Or I have proper backups synced across the world on different providers. The likelihood of everything failing is not a vector I worry about. If that happens I have bigger problems to worry about.
Avoid this kind of behavior on HN; we are not the community for attacking each other. Your comment is not helpful to further the conversation. You could have made a point about subscriptions being antithetical to the ethos of open source culture, but instead you attacked the user and mocked him/her.
Another thing I would recommend is having your servers in a wireguard net and only allowing ssh from inside that virtual network. There is no need to publicly expose ssh.
Wireguard allows effectively no configuration around auth, and so it's more likely that it is configured in a secure manner.
Openssh allows: password auth with weak passwords, PAM with potentially insecure PAM modules, gssappi, configurable ciphers, etc. Wireguard has none of those.
Each of those auth methods adds more possible vulnerable code paths and accidentally insecure configurations. It's very reasonable to believe that wireguard auth, since it has far fewer moving parts and configuration knobs, is likely to have fewer ways it can be insecure.
I was always uncomfortable that it's on by default and surprised anyone uses it it. It's also more cumbersome compared to just using a key. I feel like using keys is the password manager analogue for SSH.
Is this actually effective against Alpine? The default install seems to use ! as the placeholder in /etc/shadow instead of * and trying the attack doesn't seem to work.
We have clients (regulated banks and insurers of various sizes) that send us feeds via SFTP. You might be surprised what proportion of them ignore our “we strongly recommend key based authentication” and ask for password auth instead.
When I set up a client sftp drop I had the luxury of saying "key based access only"
Then it was a surprisingly customer service intensive process to get the keys from the customers. I never did manage to automate it fully. Weird enterprise systems I have never heard of with strange key formats. ssh1 keys, email mangling them. Their internal processes requiring a password.
And way to many "I am sorry but that was the private key you sent me, That key is now compromised and you will need to remove it and generate new keys anywhere it was used"
I almost regret not allowing passwords, almost, despite the headaches key based is still much better.
At least in that case, if the account is compromised we definitely didn't leak the credentials. That we don't have them to leak is one of the advantages of key-based auth.
[of course not having the credentials to leak does not protect us from this sort of bug, it is just one factor in a multipart security stance]
Previous company: probably. It could be a dealbreaker for the client, and we bent over quite readily on a number of issues considered minor (keys are best practise, but passwords are so common it is difficult to argue that they are so bad they should be refused).
Current state (now we have been bought and are effectively more than an order of magnitude bigger): I definitely don't have opportunity to make a unilateral decision like that. Even though the larger company has a more “fair enough, feel free to go elsewhere” attitude when a small client says “if you can support × that is a dealbreaker for us” then we could previously afford to take as our stance, some of the bigger clients are definitely paying us enough, and do use passwords, that someone above me would overrule a keys-only decision.
Don't most cloud providers default to that? I know I had to force some to deploy with a key and even then PasswordAuth was often on and had to be disabled.
I went and checked this. I can admin my server with Cockpit using either the web service (using my username/password) or the desktop client (via private key) without issue even with SSH password authentication disabled. I do usually disable the web service so had to reconfigure to check.
I agree entirely, but I'm not sure changing `crypt` to return `"*"` instead of `NULL` on failure qualifies. It's much more of a considered decision, which happens to be very wrong.
Well, OpenSSH contributes to this by mixing hashes from the shadow file. The musl code would be correct if OpenSSH passed in the hash from the user account that is being logged-in.
They instead pass in a different user’s hash (to avoid timing attacks) but end up comparing the result from musl against the actual user account, defeating musl’s simplistic safeguard to avoid this very issue.
I’d probably still call this a musl thing. Errors should result in a NULL return even if the application crashes from there, but at the very least a return value intended to not match any actual hash in /etc/shadow should be a random string or predefined UUID or something other than *.
It didn't give me the impression that I was dealing with a secure piece of software, it seemed like they were dealing with very complex issues while laughably failing at the industry basics that were settled years ago.
Similarly Ubuntu was also the only other distro that fucked this simple permissions config up, and lately with the latency to fix the TCP vuln, it seems to confirm that these are not reliable distros for the coming security era.
Seems slightly confusing that the OpenSSH release notes page says:
> This release contains a critical security workaround that affects Linux systems that use the musl libc and PAM disabled.
But that there's nothing about it on their security page.
Even if it's not technically a bug in their code, the fact that they've made a "critical" release to address it suggests that it should probably be listed on the security page as well. But maybe (hopefully) they just haven't updated the security page yet.
musl libc violates the POSIX documented behavior of `crypt()`¹
> Upon successful completion, crypt() shall return a pointer to the hashed password; the first two bytes of the returned value shall be those of the salt argument. Otherwise, it shall return a null pointer and set errno to indicate the error.
OpenBSD is not affected, so apparently they are doing something right. But also glibc linux is not affected and pam linux is not affected, so it is only musl linux with the remote hole.
You became a victim of a very misleading post title. This isn't an OpenSSH problem at all, it's a musl problem.
I'm pretty sure the misleading wording is deliberate, OpenSSH is an excellent but non-commercial project which a bunch of entities would love to gain control of, in order to bug it with their spyware - thus the constant nagging on the bot-infested web.
To the extent that systems that are configured in the vulnerable manner exist in the wild, I’m surprised they weren’t exploited years ago by trivial ssh bots.
P.S. Wow, I’m reminded of just how awful traditional UNIX-y C APIs and code are.
P.P.S. The musl code in question is nuts. The API it implements is awful. Seriously, here’s the manpage:
https://man.archlinux.org/man/crypt.3.en
You can’t call this function correctly. sshd’s “fix” is basically to assume that a too-short return is an error return instead of a success return.
PAM is also awful, but this is worse.
"It starts with a bold claim: "ssh accounts that should not accept a password will accept any password between 256 and 1024 characters". But then it keeps adding qualifiers:
- sshd compiled with musl
- not using PAM
- password logins enabled
- at least one account with '*' as the password "hash"
- at least one OTHER account with a valid password
The conditions are all ANDed, miss a single one and you're fine."
This doesn't seem to be nearly as big as an issue as they're making it out to be
Disabling PAM has been "generally good security advice" for decades; I remember suggestions to not use it in the 2000s.
Password logins are often enabled, especially on multi-user and default setups.
If alpine uses the * feature, it's going to be pretty common. Gentoo has it, as does Ubuntu.
If password logins are enabled, you're almost guaranteed to have at least one user with a password (root on all my boxes has one, even if I never use it and it's random garbage).
So the cheese lines up pretty well, this affects quite a bit.
If password logins are enabled, someone almost certainly has a password.
I'm seeing people I generally trust in the scene memeing on twitter, while others are freaking out. Its hard for me to tell how fucked we are, I guess
I think this time around it's mainly Alpine Linux and a few custom-rollers (Gentoo with musl could easily get to this place).
I think this will likely become standard. Nevertheless, a few stormy months are ahead.
at this point the _last_ thing i care about is my email.
where am i? am i ok? how are my friends/loved ones?
once those are in place, it becomes clearer how to get back to "normal"
Is it fast? No, but that's a feature.
But it does mean I get to lean on the existing processes and identity verification already part of the government bureaucracy and the courts as my backdoor.
Locally: Same, but printed out, laminated, in a safe place with other documents.
Who can presumably get you in touch with the relevant embassy or consulate or, failing that, local authorities.
And this is assuming the server is somewhere else on the cloud.
Or I have proper backups synced across the world on different providers. The likelihood of everything failing is not a vector I worry about. If that happens I have bigger problems to worry about.
Openssh allows: password auth with weak passwords, PAM with potentially insecure PAM modules, gssappi, configurable ciphers, etc. Wireguard has none of those.
Each of those auth methods adds more possible vulnerable code paths and accidentally insecure configurations. It's very reasonable to believe that wireguard auth, since it has far fewer moving parts and configuration knobs, is likely to have fewer ways it can be insecure.
Then it was a surprisingly customer service intensive process to get the keys from the customers. I never did manage to automate it fully. Weird enterprise systems I have never heard of with strange key formats. ssh1 keys, email mangling them. Their internal processes requiring a password.
And way to many "I am sorry but that was the private key you sent me, That key is now compromised and you will need to remove it and generate new keys anywhere it was used"
I almost regret not allowing passwords, almost, despite the headaches key based is still much better.
Still safter in general, but not bulletproof.
[of course not having the credentials to leak does not protect us from this sort of bug, it is just one factor in a multipart security stance]
Current state (now we have been bought and are effectively more than an order of magnitude bigger): I definitely don't have opportunity to make a unilateral decision like that. Even though the larger company has a more “fair enough, feel free to go elsewhere” attitude when a small client says “if you can support × that is a dealbreaker for us” then we could previously afford to take as our stance, some of the bigger clients are definitely paying us enough, and do use passwords, that someone above me would overrule a keys-only decision.
You must present a key first after which you must enter a password.
Isn't this safer than "PasswordAuthentication no", even if it's a different setting?
So not a ssh issue but an issue specific to Linux using "musl libc without PAM".
They instead pass in a different user’s hash (to avoid timing attacks) but end up comparing the result from musl against the actual user account, defeating musl’s simplistic safeguard to avoid this very issue.
I’d probably still call this a musl thing. Errors should result in a NULL return even if the application crashes from there, but at the very least a return value intended to not match any actual hash in /etc/shadow should be a random string or predefined UUID or something other than *.
At the bottom of that it says dropbear is unaffected.
> on debian, ubuntu, fedora or anything else with glibc? going through pam? dropbear? openbsd? you're fine
[0] https://sshhole.com/
[1] https://news.ycombinator.com/item?id=50378228
https://gitlab.alpinelinux.org/alpine/aports/-/work_items/16...
It didn't give me the impression that I was dealing with a secure piece of software, it seemed like they were dealing with very complex issues while laughably failing at the industry basics that were settled years ago.
Similarly Ubuntu was also the only other distro that fucked this simple permissions config up, and lately with the latency to fix the TCP vuln, it seems to confirm that these are not reliable distros for the coming security era.
> This release contains a critical security workaround that affects Linux systems that use the musl libc and PAM disabled.
But that there's nothing about it on their security page.
Even if it's not technically a bug in their code, the fact that they've made a "critical" release to address it suggests that it should probably be listed on the security page as well. But maybe (hopefully) they just haven't updated the security page yet.
> Upon successful completion, crypt() shall return a pointer to the hashed password; the first two bytes of the returned value shall be those of the salt argument. Otherwise, it shall return a null pointer and set errno to indicate the error.
Musl returns '!' or '*', not null.
¹https://pubs.opengroup.org/onlinepubs/9799919799/
“Only two remote holes in the default install, in a heck of a long time!”
https://www.openbsd.org/
I'm pretty sure the misleading wording is deliberate, OpenSSH is an excellent but non-commercial project which a bunch of entities would love to gain control of, in order to bug it with their spyware - thus the constant nagging on the bot-infested web.
The OP should be flagged.