Very cool to see PyPI put more emphasis towards exposing provenance info. It would be even cooler if trusted publishing would finally support reusable workflows (https://github.com/pypi/warehouse/issues/11096).
It would be even nicer if there were a way to have an attestation without publishing with a workflow controlled by one of the tech giants. There's no way for me, someone who has had a PyPI account for many years and has used it for all that time to upload packages, to attach my own public key to my PyPI account and then sign uploaded distributions with my private key so the signature can be verified and count as an attestation.
This is about detecting formerly benign projects that have had their pypi entries hacked, but not their private keys. It does not detect malicious projects.
Not directly the subject but they switched from Material for MkDocs to Zensical.
It’s quite sad to see that one of the main (or the biggest?) documentation system for the Python completely alienated its user base and led to this fork.
let me tell you something incredibly important about the attestations / trusted publisher thing:
if you have any of your files published by a trusted publisher with attestations, then you must have all of your files published that way (in that package).
otherwise pypi determines your project may have been compromised and puts some big red warnings up like [1]. it does not try to look at "this particular source file was never part of trusted publishing" or anything like that.
it's also AFAICT a permanent switch. once you put a single trusted/attested file up there, your account must now do all files in this way forever (or risk big red warnings. which maybe you're OK with that).
It would all be easier to take if trusted publishers weren't limited to solely large for-profit corporations (I'm too lazy to research if they're trying to fix this).
> it's also AFAICT a permanent switch. once you put a single trusted/attested file up there, your account must now do all files in this way forever (or risk big red warnings. which maybe you're OK with that).
It's not[1].
> It would all be easier to take if trusted publishers weren't limited to solely large for-profit corporations (I'm too lazy to research if they're trying to fix this).
As mentioned in several other places, we're always interested in adding more publisher providers, provided they meet our key management and notability requirements.
(This is also the easiest way to reason abductively about PyPI's goals. If PyPI's goal was to support "large for-profit corporations," we'd have dozens of publisher providers of dubious quality and community value, rather than the small subset we currently have.)
Thought we already learned years ago an IP address is not a person?
That is to say; what the screen on my end communicates is not guaranteed to be representative of ground truth.
The IT world was recently making fun of Sam Altman's eye scanner because remote attestation can be faked.
All the layers of abstraction wrapped around the code is security through obscurity.
The code itself must still be verified as safe.
The cloud era has resulted in a lot of people operating at layers of abstraction so far removed from physics this shit has more in common with baseless dogmas, gospels, and spiritualities at this point.
We anthropomorphized the cloud inserting ourselves into it. It's just a bunch of networked computers.
Am I correct in my understanding that the only trusted publishers are for-profit US companies? And these all use proprietary implementations which cannot easily be replicated elsewhere?
Like, if I want to host on Codeberg, or Sourcehut, what is the process needed for those platforms to become a trusted publisher?
As it happens, I build my releases on my own hardware using a pypa/manylinux release. Can I provide attestation, or am I required to use a third-party service to get that "x Without provenance" removed from my entry?
Support for Codeberg is being discussed here[1]. I think there's broad consensus among PyPI's maintainers and admins that more publisher providers are desired, but we also need them to demonstrate strong key management and rotation controls due to the degree of trust and access that becoming a publisher provider grants them (effectively, they become the PyPI credential broker for the projects that use their service).
(If you pan through that thread, you'll note that one of open issues with Codeberg is that their underlying server wants to store OIDC private keys on disk at the moment. This is okay for small deployments of personal services, but not for something that PyPI needs to federate with and trust.)
Sounds like it. Currently it seems you'll need to setup your own OIDC provider[0], then work with PyPI to get them to trust your instance as a "Trusted Publisher". It seems like a system to scale for multi-tenant publishers (like millions of GitHub or GitLab or Google build or ActiveState users). The design itself should be able to scale for individual OIDC providers through webfinger and OIDC discovery like Tailscale does it[1], but it doesn't look like it's there yet. Not sure if it's planned either the github issue mentiones future PyPI verified email attestation, though I'm not sure if that implies webfinger/OIDC like ts does it.
> The design itself should be able to scale for individual OIDC providers through webfinger and OIDC discovery like Tailscale does it[1], but it doesn't look like it's there yet.
This is the right idea, but it's actually not that simple unfortunately: Trusted Publishing relies on per-IdP claims that are more precise than the `sub` claim (because doing pattern matching on the `sub` claim is brittle and prone to security problems). So we require publisher providers (OIDC IdPs) to provide distinct claims that we can match on. For GitHub and GitLab that's things like the repository name, workflow filename, etc., while for others it might be distinct username and immutable ID claims, etc.
This is all downstream of OIDC-driven federation being a much weaker and more bespoke process than all parties would like it to be. I think the discourse around this could be much improved by people taking the time to understand these complexities and why they pose a challenge to groups with limited engineering resources (like PyPI).
Did I get that right? Seems a bit anti-pythonic.
dlt (no provenance): https://pypi.org/project/dlt/#files
datasette (yes provenance): https://pypi.org/project/datasette/#files
[1]: https://github.com/pypi/warehouse/issues/new?template=bug-re...
It’s quite sad to see that one of the main (or the biggest?) documentation system for the Python completely alienated its user base and led to this fork.
if you have any of your files published by a trusted publisher with attestations, then you must have all of your files published that way (in that package).
otherwise pypi determines your project may have been compromised and puts some big red warnings up like [1]. it does not try to look at "this particular source file was never part of trusted publishing" or anything like that.
it's also AFAICT a permanent switch. once you put a single trusted/attested file up there, your account must now do all files in this way forever (or risk big red warnings. which maybe you're OK with that).
It would all be easier to take if trusted publishers weren't limited to solely large for-profit corporations (I'm too lazy to research if they're trying to fix this).
[1] https://fosstodon.org/@zzzeek/117554513935692558
It's not[1].
> It would all be easier to take if trusted publishers weren't limited to solely large for-profit corporations (I'm too lazy to research if they're trying to fix this).
As mentioned in several other places, we're always interested in adding more publisher providers, provided they meet our key management and notability requirements.
(This is also the easiest way to reason abductively about PyPI's goals. If PyPI's goal was to support "large for-profit corporations," we'd have dozens of publisher providers of dubious quality and community value, rather than the small subset we currently have.)
[1]: https://discuss.python.org/t/pypis-new-security-indicators-a...
Thought we already learned years ago an IP address is not a person?
That is to say; what the screen on my end communicates is not guaranteed to be representative of ground truth.
The IT world was recently making fun of Sam Altman's eye scanner because remote attestation can be faked.
All the layers of abstraction wrapped around the code is security through obscurity.
The code itself must still be verified as safe.
The cloud era has resulted in a lot of people operating at layers of abstraction so far removed from physics this shit has more in common with baseless dogmas, gospels, and spiritualities at this point.
We anthropomorphized the cloud inserting ourselves into it. It's just a bunch of networked computers.
All this stuff is a spoon that doesn't exist.
Like, if I want to host on Codeberg, or Sourcehut, what is the process needed for those platforms to become a trusted publisher?
As it happens, I build my releases on my own hardware using a pypa/manylinux release. Can I provide attestation, or am I required to use a third-party service to get that "x Without provenance" removed from my entry?
(If you pan through that thread, you'll note that one of open issues with Codeberg is that their underlying server wants to store OIDC private keys on disk at the moment. This is okay for small deployments of personal services, but not for something that PyPI needs to federate with and trust.)
[1]: https://github.com/pypi/warehouse/issues/19786
[0] https://docs.pypi.org/trusted-publishers/
[1] https://tailscale.com/docs/integrations/identity/custom-oidc
This is the right idea, but it's actually not that simple unfortunately: Trusted Publishing relies on per-IdP claims that are more precise than the `sub` claim (because doing pattern matching on the `sub` claim is brittle and prone to security problems). So we require publisher providers (OIDC IdPs) to provide distinct claims that we can match on. For GitHub and GitLab that's things like the repository name, workflow filename, etc., while for others it might be distinct username and immutable ID claims, etc.
This is all downstream of OIDC-driven federation being a much weaker and more bespoke process than all parties would like it to be. I think the discourse around this could be much improved by people taking the time to understand these complexities and why they pose a challenge to groups with limited engineering resources (like PyPI).