Author here. Let me know if you have any questions about the post or about pgrust.
Let me take a shot at answering what I think will be the most common question: how can I trust pgrust? Our #1 priority right now is correctness. Over the past two weeks, I've done a mix of formal verification and differential fuzz testing. We've been able to prove over 1000 user facing functions have the exact same logic in both pgrust and postgres (see the proofs directory if you're curious). For cases where formal verification is not easy, we've taken the c implementation of a function and the rust implementation of a function and ran millions of inputs through each of them and confirmed they gave the same results every time.
We've only covered about 15% of the surface area so far, but in the process, we've discovered ~100 bugs in pgrust and ~20 bugs in Postgres itself. My favorite postgres bug we found is this one[0]. Postgres has a quadtree implementation. Due to floating point rounding, it was possible for a point to be neither above, nor below, nor even with the center point of the quadtree.
We've also entered engagements with Antithesis[1] to do Jepsen style fault testing and Aretta[2] to do more serious formal verification.
If you want to support the project, the easiest way is to give us a star on GitHub[3]
How do you know if you're making the right optimizations?
I struggle with this a lot with Marginalia's index. Where I identify a hot method in a prod profiler run, try to replicate it on a test machine where I can never get the same cache characteristics because everything in this space is like an onion of caching layers that you affect the real performance of the system. I may get it to run significantly faster, but that only sometimes makes the production profiler sample move its needle.
e.g. I've recently been experimenting with using a cursed hybrid model in Marginalia's index, where based on a mincore probe, I switch between mmap and io_uring for reading a cluster of pseudoadjacent data. There are real tanglible benefits both in the test machine and in prod with this, but the numbers do not agree at all about how the needle moves :P
Optimizing a multi-layer cache system is a Complex System task with no subset you can reduce to a simpler version of the problem. PG has a 3 layer cache--database, OS, storage--and when you combine that with concurrency and locality, you have a giant 5 dimensional space to map out. Building the right set of benchmarks for PG with guidance for the most important parts to navigate fro that space has kept me busy for 20 years now.
I would probably dig into the reasons for the differences in the benefit on the test machine and in prod
I had an issue like this for optimizing pgrust. I had an optimization that showed no impact on my test machine (c8g.4xl) and showed a 20% improvement when ran on my mac. It turns out the issue was the instruction cache on the c8g.4xl was being saturated on the test machine but not on my laptop, moving the bottleneck to a different place
If you can consistently reproduce the performance difference, you're already half way there
I'm pretty sure the reason for the difference is that production machine exists in a state of mixed memory residency and low grade resource contention that is incredibly hard to replicate in a test scenario (as the moment you start making queries the pages warm up, and the test becomes unreliable).
The hard part about optimizing this type of code, IMO, is that there are so many cache layers, both in the CPU and the OS and sometimes in the storage medium. You can warm all of those caches up, but then you're testing a nonsense scenario that will basically never happen in a realistic scenario, where wall clock time is what matters, and not much the CPU is working or how many IOPS you're pushing.
I really hope your project succeeds so that you can sit, sip delicious coffee with your legs on a high desk, and laugh really loud, like a gorilla, at all the haters from this and previous HN threads.
If someone wanted to use this as a real-time WAL-tracking read-only mirror of a live production database, for analytics work, is it ready for that use case yet?
When I got early access to pgrust my first question was whether it had been tested for building read-only standby analytics questions. The problem has goals beyond that, but having a real analytics node available just by playing WAL data from your primary makes pgrust a low risk eval for your data. Just have to build confidence the results are connection, which is being applied by all the PG infrastructure regression tests the code has been trained on,
A question on 20s postgresql time - It does not look like you are accounting for reading data from disk? Wouldn't the aggregation query have to load data from disk first? Or is it somewhat guaranteed that the table is already in memory? The Rust version is clearly in memory (I am no rust expert, so that may not even be actually in memory, if its a generator).
> A question on 20s postgresql time - It does not look like you are accounting for reading data from disk
I choose the data size so that it would fit in memory on the machine I was testing on. fwiw, there's still a ton of overhead Postgres has that the toy example does not. For example Postgres will serialize the numbers into tuples and need to deserialize them to execute the query. That's why it's not an apples-to-apples comparison
Is `pgrcolumnar` the default storage layout for tables? It would be cool if the same storage engine outperforms vanilla Postgres under both OLTP and OLAP workloads.
AlloyDB from Google Cloud uses columnar storage like a secondary index, while the relations are still stored in TOAST.
pgrcolumnar is not the default storage method. Right now, it's exposed as a table access method. There's lots of design space for how to do this so I want to avoid pre-committing to anything
I want to build the best database possible. While Postgres is great, there are a lot of core issues that have been around for over a decade. We're working hard to get pgrust production-ready, and it will definitely be production-ready in the near future. I wouldn't be putting hundreds of thousands of dollars into this project if I didn't think we could build a production-ready database.
What is your motivation? What you are doing is very impressive for sure, but the longer term life of this project seems incertain to me. While you might be able to accomplish your goal of production readyness, you'll only have version 1.0 at that point.
Its unlikely an AI written project is one which people will want to maintain even if it "works", which means AI has to maintain it. Maintaining a code base of a production database is going to continue costing 100s of thousands, if its even possible at all with AI. Its also unlikely to find users even if its almost perfect, because a bug in a database is not something users can accept.
Postgres is postgres because its like 30 years old and battle tested. Sure its missing some modern features, but its incredibly stable, which is part of its popularity.
I'm guessing it's something like MongoDB TTL index[0]. It's useful for huge append-only tables where you want a row to be automatically removed after a period of time. The postgres approach is usually to partition your table by the event/insertion timestamp, and use something like pg_partman[1] to automatically drop entire partitions over time.
Cool project but .. reality is that people will generally not choose pgrust over Postgres, even 5-10 years from now. The problem is not that it may be technically superior and faster by then, it's that it's not built by the trusted Postgres team. There's a lot more to trust than development velocity or performance. It's also about the longevity and continuity of a critical piece of technology.
I don’t think this is true. There’s plenty of signals for trust - if pgrust keep pouring time, money, and effort into this and get people actually using it even in non critical workloads, it’ll snowball from there. Imagine 2 years from now it’s being used in a couple dozen large companies - people wouldn’t think twice about using it.
The possibility of competition, and an existing proof of concept, could motivate the Postgres team to pursue performance initiatives that they otherwise wouldn't.
People are very willing to use new things when something they are using can't do what they want to do.
Dogma is more effective than reliability concerns in my experience. It just happens that the dogma prefers Postgres in this case.
It is just difficult to make these decisions in a more informed way because it requires actual manual testing with the concrete use case and also a decent understanding of databases in question. Chances are you won't have people that understand these things or those people won't have much of a say in decisions.
We still have companies that use COBOL and mainframes due to their longevity and continuity. There is no reason Postgres and pgrust cannot both flourish. I can pick and choose which one to use according to its use case.
They will be software engineers or the people doing the software work
And different people will be doing product management and HR
all because today’s software engineers don’t want to babysit AI agents and choose antiquated libraries so that their resume said they'd been using a popular framework for a couple years
Nobody else is playing that outdated game, its just a rotation
this thread is talking about using a 300x faster library and theorizing a resistance to devs using it
just use the faster library? leverage compute resources more effectively and justify value to an organization better than the next person
otherwise, what needs to be done? I don’t see a problem with any of this aside from organizations risking less experienced people doing less efficient things in other parts of the stack
I'll need to write up how the scheduler works at some point, but it's heavily based on these papers[0][1]. It solves two different problems. First, it lets us throttle resource-intensive queries. Second, it enables work stealing. If you have idle cores on your machine, we'll assign those cores to running queries to help speed them up. That means if you have an over-provisioned machine, we'll make use of the extra capacity to speed your queries up.
Interesting. It would need some work to handle all the workloads I’ve faced where you have two applications with different priorities (ie oltp or grid workloads and analytics).
In those cases you want to make sure that BI users do not interrupt the transaction processing by assigning them a set of cores, a priority, and work mem limits for example.
Looks doable without major changes.
We don't expose the priorities right now, but we have the priorities decay over time. That way faster queries get prioritized over long running queries. That should achieve the behavior you're looking for.
Yes for this particular use case it might help (no per-application aggregated temp file limit though). Also my grid workloads generate thousands of threads running queries with lots of small updates (job queues, calculation results..). I wish there was a way for postgresql to isolate those connections to a limited set of cores (or number of cores).
To avoid completely saturating the server.
Right now we advise using s3 and storing results in files, and asynchronously loading them with batch processes.
You have no idea how long I have been waiting for adaptive planning. One of my biggest annoyances with the Postgres core team has been their reluctance to implement any sort of adaptive planning despite it, at this point, being a well-established technique that has been implemented in multiple production databases. I hope this, at the very least, proves the viability of this model outside of academic/niche contexts.
Surly AI could also write a clearer headline. For the millions running it in production for decades, using a great echosystem of help support, books, consultants, and managed hosting providers; the is a noteworthy difference between the official release and a partially compatable rewrite.
I don’t understand. It’s not like we don’t already have faster alternatives, don’t we have things like k and kdb that are many orders of magnitude faster even than this?
We use Postgres because it’s fantastic at the scale and problems it solve, if you have an extremely critical system where raw speed is at the core of everything and you’re dealing with petascale then maybe you’re bringing a solution you like to a problem it doesn’t fit?
There should be a halfway point between kdb and postgres...
So more seriously my thinking is that we should, on purely moral grounds strive to have things be fast and lean. How many years of human misery have been lost to waiting around for python programs for example because the team for years wanted to the interpreter to be simple
Postgres is amazing. When I use Sqlite I miss many of its upsides--better data migrations and better concurrency being two. But queries are not free, and sometimes are slower than I'd like even with the right indexes. And for OLAP-like queries like the one in TFA you might say "well, just don't use postgres for that," but wouldn't it be better if you could? If you could use one database instead of two, or use your favourite database in more places instead of reaching for a different tool?
We went this path: pg -> peerdb -> clickhouse replica, then pg -> another pg with pg_clickhouse extension -> clickhouse
This way we keep querying without changing the query language (use same postgres syntax), but switch connection port for analytics query. Experience so far: made 3 PRs to pg_clickhouse (1 merged), otherwise works pretty well.
If something like pgrust would work even better/easier – would definitely check it out instead.
Commit #1's message is "hey claude, do a breakthrough" from a week ago and is 1.5M lines.
Commit #2 is "blog post" from 4 days ago.
My head is spinning. I don't mind AI stuff or AI enabled stuff but there's gotta be some bar for ending up on HN, and also personal accountability: the lack of humility and honesty sets a new low for me. There is no "we" who "released pgrust 0.2". It's one person cosplaying a serious engineering team doing a mountain of work. The bus factor is 1, and its one you can't trust on the basics.
ex. the first 1/3 of the blog post is bloviating about how a rust for loop is faster at summing 500M numbers on the heap than loading the numbers from a table and summing them.
It leaves me in quite some anguish. This site kept me well-informed and growing for 16 years. It is no longer reliable for that as long as things like this can be the #1 post with 60 comments, with the author here, and no one mentioned any of this.
The second line of commit #2's message is "You can find the actual git history at the v0.2 github tag." which in turn has almost 6000 commits.
I think it's a weird way to handle git history versus squashing feature branches into single commits, but it's not just one Claude session slapped up on GitHub. The OP also has a post above about their exhaustive testing which has uncovered a goodly number of bugs in Postgres itself, too. I think it's fair to say they're putting in a good amount of work on this.
I cloned it. The 6000 commits are the problem, not the defense.
5,940 commits, one author, 29 days. 5,067 of them, 85%, have a `Co-Authored-By: Claude` trailer. Busiest day is 1,393 commits, running 60-105/hour for ~20 hours straight. Commit messages reference .claude/skills/fleet/ and agent worktree "lanes". That's an unattended agent fleet committing once a minute around the clock. Commit count used to mean review time. Here it means GPU time.
Also: the v0.2 tag shares no common ancestor with main. git merge-base fails. "The actual git history" is an orphan branch grafted in after the fact.
And the testing rigor we're crediting them for should be weighed against the headline claim: 300x faster than Postgres, ahead of ClickHouse: fastest analytical DB on earth, one guy, one month.
Most damning: the Postgres baseline ran with *max_parallel_workers_per_gather = 0*. Parallel engine vs. deliberately-hobbled single-core Postgres, headline says 300x.
Using AI is fine. But "look how many commits" is precisely the signal this workflow is built to fake.
Reading about the batch optimization, I suspect there'll be trouble ahead for implementing window functions, unless those details were dropped for simplicity.
How so? Window functions work exactly the same way. Postgres processes them one row at a time, and you can batch them the same way as you would with sum.
Window functions do make parallel queries more difficult, but that's a different story.
I guess regardless of if the calculations are batched or not, all the rows would have to be processed before the window function results could be determined.
You can certainly do that, but it'll perform much worse than properly tuning your Postgresql to take advantage of requiring no durability and having lots of RAM. Source: I do this kind of thing for a living.
Can you file an issue? We know there are bugs and the work we're doing with formal verification and fuzz testing is to go through all the code and make sure all of it behaves identically to Postgres
I am very disappointed to see the direction:
It is moving from a "interesting attempt to recreate system software" to "building flashy but useless demo"
Everyone who knows a bit about databases knows the difference between execution models and what kind of optimization it brings.
> I am very disappointed to see the direction: It is moving from a "interesting attempt to recreate system software" to "building flashy but useless demo"
What makes you say this is a useless demo? I can't count the number of people who've struggled to do analytics inside of Postgres. Almost always they end up setting up a separate system such as Clickhouse and replicating the data between the two systems. Now they can have one system that's Postgres-compatible, and it's faster than either of the original systems.
> Everyone who knows a bit about databases knows the difference between execution models and what kind of optimization it brings.
In our last post[0], when we mentioned we were getting close to Clickhouse level performance (now faster than Clickhouse), we were met with disbelief. This post is meant to explain part of how we closed the 300x gap between Postgres and Clickhouse. The execution model being 10x of it.
> Almost always they end up setting up a separate system such as Clickhouse and replicating the data between the two systems. Now they can have one system that's Postgres-compatible, and it's faster than either of the original systems.
I can see the appeal for pgrust for smaller teams who need analytics, and don't want to deal with having to ETL to something like ClickHouse. But beyond a certain complexity, replicating your data to a warehouse or lakehouse _is_ the right approach and more scalable for several reasons:
- Analytics tend to be centralized, i.e. you want data from several Postgres databases spread across multiple teams to be replicated into 1 place, so people can start joining data across the entire business
- Analytics tend to fall under a different team ownership with their own set of non-technical requirements (e.g. data governance)
- Lakehouse architecture (Iceberg + [insert query engine]) is more scalable in terms of cost
- In some cases, you want to be able to swap different query engines depending on the use case, e.g. use PuppyGraph to query your data in Iceberg for fraud analysis
No, it doesn't. Different algorithms can vary pretty wildly in performance based on the design of the hardware they run on. For instance cache sizes can make one implementation of a sort on a certain sized dataset faster or slower than another. You can have a theoretically fast algorithm that just isn't as cache efficient as a theoretically slower one in big-O terms. All levels of the memory hierarchy as well as storage have specific bandwidths and latencies that inform the real world performance results. Parallelism is another issue. Many cores can do work very fast when you're careful about how you split up work between them, taking into consideration the synchronization latency and individual cache sizes and so on. The best approach for doing work on 64 cores can be dramatically different from what works best on 1 or even 2.
That is not a given. A database server can run faster on better hardware because it more effectively and aggressively caches things in memory, which can hurt it on lower-spec systems. Or it can better utilize SIMD instructions that are not present on the low end. Or it is more effective at utilizing more threads, but is slower when run at a low threadcount, etc etc etc.
I don't understand the use case. I've never seen why Postgres is better than anything else if I'm honest. Even if it is much, much faster, database transactions aren't what slow things down if written well in web applications.
It’s not just speed, it’s also the rich feature set. Trigram search, JSONB, transactions, GIN indexes, full text search - not many databases combine them all into a Swiss Army knife like Postgres does.
The pgrust team asked me to look at their results on a review system, and I confirmed the ClickBench speedup there. Regular PostgreSQL is really terrible at some of these queries. Unfortunately fixing that is hard to do in core itself because columnar storage lives outside of the main tree, and some optimization problems only show up when layered on columnar.
Can you file an issue? Our big focus over the next few weeks is to eliminate these issues and that's why we're our formal verification and fuzz testing work
Author here. At least for databases, AGPL (or stricter) has become standard. The issue is it's so easy for megacorps (Amazon, Google, etc) to take a permissively licensed product and monetize it at the expense of the original standard.
For instance, Mongo, Cockroach, and Materialize have all gone source available. We picked AGPL because it's the best balance between open source and prevents Amazon from just repackaging it and selling it.
If AGPL is an issue for anyone, we would be happy to dual-license under a commercial license.
There is nothing wrong with wanting to be compensated for your work, but for people like myself which use a cloud managed DB solution (GCP CloudSQL PostgreSQL) it means something like this would never be available.
I consider AGPL a poison pill in my work. That is not true with a suitable commercial license, although I expect a lot more commercial product (support/features/etc). As you note, your objective is to prevent commercialization of your software, but radically speeding up analytics is primarily a concern of large organizations so it seems like a mismatch in purpose.
The whole point is to force the cloud providers to go through a contract negotiation for licensing rather than keep the door open for them. Nothing prevents the parent project from specially licensing to individual parties.
The issue is that AGPL is flatly banned at many companies, so now any company that even wants to test this database out is going to have to seek a contract, which they won't.
Most of the same incentives of typical OSS contribution? It's not like other OSS projects pay out any earnings made with the product evenly across contributors.
Also, from the trajectory of the project it doesn't look like they are significantly deoendent on outside code contributions.
Sure, that's your prerogative, and kudos for not talking up open source. I'm not amazon size so can't use it, and AGPL is a no go for DB, don't want to be forced to open source my app because I use this!
Why would AGPL force you to open source your app? Unless you literally compile your app with pgrust by modifying the pgrust source code, you're safe. Clients aren't bound by the AGPL because they aren't derived works.
AGPL is untested in courts. There is no definitive definition of what could be considered within the blast radius such that it would require AGPL licensing.
There's a reason AGPL is banned at Google and most sane companies. It's simply too dangerous.
You can't simply say "clients aren't bound" because it depends.
I'd rather see the BSL used here to be perfectly honest. At least it's simple.
Oh bummer. I was really excited about pgrust but AGPL is a dealbreaker. Not for me personally, but it will never see wide adoption because it’s a banned license in most corporate environments. Lack of path to wide adoption means it’s dead in the water.
It’s weird because those who actually care about optimized pg gains are most likely large corporate customers. Why make a product targeting them and license it in such a way they’ll never use it?
This also hard blocks upstreaming any beneficial features into core Postgres.
AGPL seems like the right choice to me. I’m tired of companies like PlanetScale taking PostgreSQL, building a business on top of it, and then acting like PostgreSQL is theirs to control.
> It’s weird because those who actually care about optimized pg gains are most likely large corporate customers. Why make a product targeting them and license it in such a way they’ll never use it?
Wait, sorry, you're asking why make something enterprise customers might pay for, and then not give it away to them for free?
Its 300x faster for certain tests. I could pretty easily craft tests that do this on two different systems. The author mentioned this when they talked about being able to fit an entire ResultSet into memory. That's the real trick with performance. Very few workloads are CPU bound anymore (linear algebra on the CPU for example). Almost all workloads are memory bound. So its all about moving data from memory to network, back to memory and back to network, over and over again through your microservices or DBs. If the entire working set can fit in memory, you get at least a 10x performance boost. If you have to keep even a part of the working set on disk, its a huge performance loss. And the larger fraction of the working set on disk, the worse the performance loss.
PS Learn how DBs do joins for more information. Specifically the differences between hash joins, merge joins and nested loop joins. They are basically fancy ways to page part of your working set to disk at huge performance penalties.
PPS As memory gets more expensive, these techniques get more valuable. When it gets cheap, they lose value.
License policies are made by lawyers not by programmers. And their competition will be in the exact same boat (different lawyers though). AGPL is so toxic that it tends to be checked for during M&A processes so even if the current batch of lawyers is ok with it there is a chance that a later batch of lawyers is not. Given that the target audience for this project is the larger companies you are going to end up with a very nice project and zero actual users or you will end up with AWS et all stealing your work. Databases are very hard to do successfully commercially, at a minimum you should dual license them (AGPL for 'home' use and commercial licensing for parties that will want to buy the upside but they'll demand support and other stuff besides).
This is the case when you have your own datacenter. This isn't as big a problem in the cloud. There are ways to write licenses that prevent cloud providers from stealing while allowing customers to use the software and being required to pay for it. The problem with the AGPL has to with its viral nature, not its provisions to prevent cloud vendor theft.
A project can probably use the EUPL instead of the AGPL (EUPL is to AGPL, as MPLv2[0] is to GPL). Basically, EUPL is file-based, not project based, and so it is not aggressively viral. You can use the EUPL code any way you want, as long as you make the original code available, plus any modification to the original files.
[0]: With Exhibit B, which prevents relicensing to GPL. It is also analogous to CDDL.
This is the difference. This guy took an existing source base, had Claude find specific bugs, then had Claude fix a specific bug which was then reviewed by a person. We also don't know if these changes introduce new problems yet. You are suggesting letting Claude write an entirely new source base. That's light-years away from what happened here.
What? This project is literally an LLM port of Postgres from C to Rust. The name "pgrust" is a hint, if somewhat subtle.
There is also zero evidence a human has reviewed a meaningful amount of the code generated for the port, let alone the original C code from Postgres, given the size/scale of the project and the how quickly it was produced.
No. This is system code. You let an LLM loose on it, it probably fixed 20 bugs and introduces 200 more plus 5 different performance regressions. In all fairness, your average app programmer would have the same problems. That's why it takes so long to learn to be a system programmer and why it takes so long to do anything on a systems source base. For reference, systems are OSes, DBs and compilers (although compilers are very different in many ways).
Also, you can successfully sell a systems project that is only 10% faster. 30000x faster and they are throwing illegal and debauched things through your window to get access to your improvements.
The brittleness and difficult of writing the code. The extra complexity. The cost of bugs. And ease of creating performance degradations. If you are even asking that question, you probably don't have the experience to even have an opinion on this.
Most systems code has to be "canonical". That means it can't be improved. Imagine the most difficult PR review you have ever had. Now give the reviewers OCD and an extra 30 IQ points. And every single code change has to live up to this standard. That's what it has to do with this.
> The brittleness and difficult of writing the code. The extra complexity. The cost of bugs. And ease of creating performance degradations. If you are even asking that question, you probably don't have the experience to even have an opinion on this.
You didn't express anything unique to systems engineering.
So, keyword 'rational', I'm not sure any sufficiently large company is a rational actor.
Yes, at [tech corp dayjob], any dependency is likely to be banned for arbitrary reasons if you bring it to the attention of the wrong people. It doesn't have to go against any of our policies e.g. don't mention anything with GPL in the name around the "risk" people. In fact, do not ever talk to the "risk" people and hope they don't talk to you.
Latest news: Apparently, devtools are a legal risk. Basic reverse-engineering of client-side JS is now banned.
The delusions really seem to scale with headcount.
Good news, we now know up front what an independent port would cost, and it’s not much. So no reason AWS, Google, and friends couldn’t bang out their own port if they want, binary-compatible with this one.
What we don’t know for sure is whether there is any copyright for LLM-generated code. The license might be irrelevant!
Most official Rust projects are dual MIT/Apache licensed by convention [1] (and most Rust libraries from third parties I've seen that are open source MIT follow the MIT/Apache dual license), so seems like this library shouldn't just be AGPL for a typical political choice of a Rustacean?
> You’re cosplaying as someone who could build a database.
> In reality you yourself could never build Postgres or Redis or any other database.
> You’re not skilled enough or knowledgeable enough and you’re not willing to put the time in so you’ve simple vibe copied Postgres.
What makes you think these things? I've been writing about the Postgres internals and giving talks about Postgres for a decade and have managed a Postgres cluster as big as 1PB of data
Writing about Postgres internals and managing a Postgres cluster is the extent of your database expertise? You are not self aware enough to know that doesn’t earn any trust?
It absolutely can be embedded. The bigger enabler is replacing the process-per-connection model with a thread-per-connection model. Projects like pglite[0] had to give up concurrency because of it. We also support compiling to wasm so you can embed it in the browser too, which is what powers pgrust.com
300x if it is true you will be just busy dealing with you customers rather than pitching here. Also since it is a vibe coded project, if you are really that good , you should even need to related yourself with Postgres. Who will want related itself to something that is 300x slower than itself?
You use stability of PostgreSQL to open a pitch. Also most people use PostgreSQL because someone out there was a fan, proponent and champion of Postgres. Otherwise they'd be on MySQL or Oracle.
And your sales folks would call and say: "No need to change anything, we still run PostgreSQL, and ours is just called pgrust, but it's N times as fast".
While I do like pgrust's license, I do feel like it's kind of wrong to port in such a direct way and change the license. I guess this isn't a fork, but it kind of is? It looks like according to this post [0], they did a Claude Fable + Opus re-write. I know that legally this is seemingly a valid way to do things and avoid copyright, but it feels wrong to me. I don't even necessarily think that my feelings are correct, but standing on the backs of giants and using an LLM to "reimplement" the code (not a clean room implementation) feels like it's it _shouldn't_ be a valid way to avoid copyright or allow for license changes.
Now, I don't know how MIT -> AGPL re-licensing specifics work, but still.
My understanding is that they used c2rust, and then told the LLM/Agents to make the code more idiomatic rust, while also using the PG test suite as a feedback mechanism. This is almost certainly a derived work (and thus a fork, and should thus have the original license and copyright preserved).
For example, if I compiled PG into x86-64 assembly, and then decompiled it into C (via, say, IDA), and then polished that decompiled C code into very readable C code, it is still a derived work. For some reason, people think that if you include an LLM or Agent, copyright can be ignored, and plagiarism is now no longer possible.
It is similar to the crypto-folks thinking that if you use crypto, you no longer have to pay taxes, because the internet/computers make all inconvenient realities go away.
Honestly, such flagrant and arrogant copyright violations make it hard for me to take the project seriously, because it seems like a desperate stunt for attention (which itself may be a solid business move, but that is besides the point).
Put differently, if one were to fork pgrust, strip away the new license and copyright, and restore the original PG license and copyright (while also adding malisper+team to that copyright), they would face no legal consequences at all. In fact, they would probably be a less legal risk than the pgrust team.
> My understanding is that they used c2rust, and then told the LLM/Agents to make the code more idiomatic rust, while also using the PG test suite as a feedback mechanism.
This is correct
> and thus a fork, and should thus have the original license and copyright preserved)
This is not correct. The Postgres license is permissive. We need to include a copy of the license (which we do in the NOTICE file[0]) but we CAN relicense the Postgres code however we want as long as we meet the requirements of the license. pgrust is a derived work of Postgres, but Postgres allows derived works to be under a different license.
This feels very much like a distinction between letter of the law and spirit of the law. I knew a guy who once forked a very popular browser extension (you've definitely heard of it), that was copyleft-licensed. When I asked him where the repo is, he said it was not publicly available, but that the lawyers have assured him that he is not breaking any laws because the source code is available, via the inspect-extension feature of the browser. This answer has always felt problematic, to me.
In part, I feel like these legally-correct strategies and tactics tend be corrosive. In fact, this is exactly the kind of thing that drives projects like Sentry and Mongo to dump open source, and use EULAs instead. Not only is pgrust a license-based fork (which always runs the risk of splitting the community) it is also a language-based fork (which also runs the risk of splitting the community). In fact, the only silver lining here, is that you went straight for the AGPL, instead of MIT (thus preventing a _second_ license based fork).
Does that license protect anything legally? This is written by AI under some initial guidance. Prompts may be protected but the output of unlicensed knowledge model corpus?
I'm really happy seeing this project. Not sure if this helps you gain $$$ customers, but stupid thing that turns out very difficult in PG is making this fast:
SELECT COUNT(*) FROM large_text_db WHERE X
Where X is something that must be matched exactly. X can be FTS query on FTS-indexed table, but the way COUNT() works in PG is that it's impossible to make it fast. Over large tables, lets say 1B+ rows, it can be very very slow.
Example use case is: searching through a hospital DB of reports that have "pancreatic cancer" in them. This is trivial in SQLite, but in PG it's hard.
Let me take a shot at answering what I think will be the most common question: how can I trust pgrust? Our #1 priority right now is correctness. Over the past two weeks, I've done a mix of formal verification and differential fuzz testing. We've been able to prove over 1000 user facing functions have the exact same logic in both pgrust and postgres (see the proofs directory if you're curious). For cases where formal verification is not easy, we've taken the c implementation of a function and the rust implementation of a function and ran millions of inputs through each of them and confirmed they gave the same results every time.
We've only covered about 15% of the surface area so far, but in the process, we've discovered ~100 bugs in pgrust and ~20 bugs in Postgres itself. My favorite postgres bug we found is this one[0]. Postgres has a quadtree implementation. Due to floating point rounding, it was possible for a point to be neither above, nor below, nor even with the center point of the quadtree.
We've also entered engagements with Antithesis[1] to do Jepsen style fault testing and Aretta[2] to do more serious formal verification.
If you want to support the project, the easiest way is to give us a star on GitHub[3]
[0] https://www.postgresql.org/message-id/19597-39c532e61d78dff6...
[1] https://antithesis.com/
[2] https://aretta.ai/
[3] https://github.com/malisper/pgrust
I struggle with this a lot with Marginalia's index. Where I identify a hot method in a prod profiler run, try to replicate it on a test machine where I can never get the same cache characteristics because everything in this space is like an onion of caching layers that you affect the real performance of the system. I may get it to run significantly faster, but that only sometimes makes the production profiler sample move its needle.
e.g. I've recently been experimenting with using a cursed hybrid model in Marginalia's index, where based on a mincore probe, I switch between mmap and io_uring for reading a cluster of pseudoadjacent data. There are real tanglible benefits both in the test machine and in prod with this, but the numbers do not agree at all about how the needle moves :P
I had an issue like this for optimizing pgrust. I had an optimization that showed no impact on my test machine (c8g.4xl) and showed a 20% improvement when ran on my mac. It turns out the issue was the instruction cache on the c8g.4xl was being saturated on the test machine but not on my laptop, moving the bottleneck to a different place
If you can consistently reproduce the performance difference, you're already half way there
The hard part about optimizing this type of code, IMO, is that there are so many cache layers, both in the CPU and the OS and sometimes in the storage medium. You can warm all of those caches up, but then you're testing a nonsense scenario that will basically never happen in a realistic scenario, where wall clock time is what matters, and not much the CPU is working or how many IOPS you're pushing.
A question on 20s postgresql time - It does not look like you are accounting for reading data from disk? Wouldn't the aggregation query have to load data from disk first? Or is it somewhat guaranteed that the table is already in memory? The Rust version is clearly in memory (I am no rust expert, so that may not even be actually in memory, if its a generator).
I choose the data size so that it would fit in memory on the machine I was testing on. fwiw, there's still a ton of overhead Postgres has that the toy example does not. For example Postgres will serialize the numbers into tuples and need to deserialize them to execute the query. That's why it's not an apples-to-apples comparison
AlloyDB from Google Cloud uses columnar storage like a secondary index, while the relations are still stored in TOAST.
Its unlikely an AI written project is one which people will want to maintain even if it "works", which means AI has to maintain it. Maintaining a code base of a production database is going to continue costing 100s of thousands, if its even possible at all with AI. Its also unlikely to find users even if its almost perfect, because a bug in a database is not something users can accept.
Postgres is postgres because its like 30 years old and battle tested. Sure its missing some modern features, but its incredibly stable, which is part of its popularity.
[0] https://www.mongodb.com/docs/manual/core/index-ttl/
[1] https://github.com/pgpartman/pg_partman
Dogma is more effective than reliability concerns in my experience. It just happens that the dogma prefers Postgres in this case.
It is just difficult to make these decisions in a more informed way because it requires actual manual testing with the concrete use case and also a decent understanding of databases in question. Chances are you won't have people that understand these things or those people won't have much of a say in decisions.
Project managers and Human Resources rolling out overengineered projects will outnumber current software engineers 10 to 1
And different people will be doing product management and HR
all because today’s software engineers don’t want to babysit AI agents and choose antiquated libraries so that their resume said they'd been using a popular framework for a couple years
Nobody else is playing that outdated game, its just a rotation
just use the faster library? leverage compute resources more effectively and justify value to an organization better than the next person
otherwise, what needs to be done? I don’t see a problem with any of this aside from organizations risking less experienced people doing less efficient things in other parts of the stack
PostgreSQL has historically been bad at managing the noisy neighbor problem, but with thread pools, and io priorities, it can be solved.
Has this been tackled here ?
[0] https://15721.courses.cs.cmu.edu/spring2016/papers/p743-leis...
[1] https://db.in.tum.de/~kohn/papers/query-scheduling-sigmod21....
To avoid completely saturating the server.
Right now we advise using s3 and storing results in files, and asynchronously loading them with batch processes.
We use Postgres because it’s fantastic at the scale and problems it solve, if you have an extremely critical system where raw speed is at the core of everything and you’re dealing with petascale then maybe you’re bringing a solution you like to a problem it doesn’t fit?
So more seriously my thinking is that we should, on purely moral grounds strive to have things be fast and lean. How many years of human misery have been lost to waiting around for python programs for example because the team for years wanted to the interpreter to be simple
Postgres is amazing. When I use Sqlite I miss many of its upsides--better data migrations and better concurrency being two. But queries are not free, and sometimes are slower than I'd like even with the right indexes. And for OLAP-like queries like the one in TFA you might say "well, just don't use postgres for that," but wouldn't it be better if you could? If you could use one database instead of two, or use your favourite database in more places instead of reaching for a different tool?
This way we keep querying without changing the query language (use same postgres syntax), but switch connection port for analytics query. Experience so far: made 3 PRs to pg_clickhouse (1 merged), otherwise works pretty well.
If something like pgrust would work even better/easier – would definitely check it out instead.
2.
Commit #1's message is "hey claude, do a breakthrough" from a week ago and is 1.5M lines. Commit #2 is "blog post" from 4 days ago.
My head is spinning. I don't mind AI stuff or AI enabled stuff but there's gotta be some bar for ending up on HN, and also personal accountability: the lack of humility and honesty sets a new low for me. There is no "we" who "released pgrust 0.2". It's one person cosplaying a serious engineering team doing a mountain of work. The bus factor is 1, and its one you can't trust on the basics.
ex. the first 1/3 of the blog post is bloviating about how a rust for loop is faster at summing 500M numbers on the heap than loading the numbers from a table and summing them.
It leaves me in quite some anguish. This site kept me well-informed and growing for 16 years. It is no longer reliable for that as long as things like this can be the #1 post with 60 comments, with the author here, and no one mentioned any of this.
I think it's a weird way to handle git history versus squashing feature branches into single commits, but it's not just one Claude session slapped up on GitHub. The OP also has a post above about their exhaustive testing which has uncovered a goodly number of bugs in Postgres itself, too. I think it's fair to say they're putting in a good amount of work on this.
5,940 commits, one author, 29 days. 5,067 of them, 85%, have a `Co-Authored-By: Claude` trailer. Busiest day is 1,393 commits, running 60-105/hour for ~20 hours straight. Commit messages reference .claude/skills/fleet/ and agent worktree "lanes". That's an unattended agent fleet committing once a minute around the clock. Commit count used to mean review time. Here it means GPU time.
Also: the v0.2 tag shares no common ancestor with main. git merge-base fails. "The actual git history" is an orphan branch grafted in after the fact.
And the testing rigor we're crediting them for should be weighed against the headline claim: 300x faster than Postgres, ahead of ClickHouse: fastest analytical DB on earth, one guy, one month.
Most damning: the Postgres baseline ran with *max_parallel_workers_per_gather = 0*. Parallel engine vs. deliberately-hobbled single-core Postgres, headline says 300x.
Using AI is fine. But "look how many commits" is precisely the signal this workflow is built to fake.
Window functions do make parallel queries more difficult, but that's a different story.
You need a server with enough ram to fit it all.
But it kind of make the database fly.
"ERROR: convert_string_datum (selfuncs.c): pg_strxfrm leg; C-collation lane only" "thread 'pg:backend:1572' (11650) panicked at crates/backend/optimizer/plan/planner/src/selfuncs.rs:966:9:"
i would say is not ready yet.
Everyone who knows a bit about databases knows the difference between execution models and what kind of optimization it brings.
What makes you say this is a useless demo? I can't count the number of people who've struggled to do analytics inside of Postgres. Almost always they end up setting up a separate system such as Clickhouse and replicating the data between the two systems. Now they can have one system that's Postgres-compatible, and it's faster than either of the original systems.
> Everyone who knows a bit about databases knows the difference between execution models and what kind of optimization it brings.
In our last post[0], when we mentioned we were getting close to Clickhouse level performance (now faster than Clickhouse), we were met with disbelief. This post is meant to explain part of how we closed the 300x gap between Postgres and Clickhouse. The execution model being 10x of it.
[0] https://news.ycombinator.com/item?id=48841676
I can see the appeal for pgrust for smaller teams who need analytics, and don't want to deal with having to ETL to something like ClickHouse. But beyond a certain complexity, replicating your data to a warehouse or lakehouse _is_ the right approach and more scalable for several reasons:
- Analytics tend to be centralized, i.e. you want data from several Postgres databases spread across multiple teams to be replicated into 1 place, so people can start joining data across the entire business
- Analytics tend to fall under a different team ownership with their own set of non-technical requirements (e.g. data governance)
- Lakehouse architecture (Iceberg + [insert query engine]) is more scalable in terms of cost
- In some cases, you want to be able to swap different query engines depending on the use case, e.g. use PuppyGraph to query your data in Iceberg for fraud analysis
I stand corrected, there is. Wishing you the best of luck here.
[0] https://benchmark.clickhouse.com/#system=+liH|pgrs|gQ&type=-...
[0] https://www.amazon.com/dp/184951030X
I would not trust this project further than I could throw it.
Have pgrust folks reconsidered this? Else, IMO we can have an independant rust port of pgrust, which can be MIT, which will garner more attention.
For instance, Mongo, Cockroach, and Materialize have all gone source available. We picked AGPL because it's the best balance between open source and prevents Amazon from just repackaging it and selling it.
If AGPL is an issue for anyone, we would be happy to dual-license under a commercial license.
I consider AGPL a poison pill in my work. That is not true with a suitable commercial license, although I expect a lot more commercial product (support/features/etc). As you note, your objective is to prevent commercialization of your software, but radically speeding up analytics is primarily a concern of large organizations so it seems like a mismatch in purpose.
That's up to GCP then. I'm sure OP would be willing to license pgrust to them for an appropriate price.
Also, from the trajectory of the project it doesn't look like they are significantly deoendent on outside code contributions.
You should be setting up a CLA process then ASAP if you are looking to take outside contributions.
Will await a MIT based fork myself.
AGPL is untested in courts. There is no definitive definition of what could be considered within the blast radius such that it would require AGPL licensing.
There's a reason AGPL is banned at Google and most sane companies. It's simply too dangerous.
You can't simply say "clients aren't bound" because it depends.
I'd rather see the BSL used here to be perfectly honest. At least it's simple.
It’s weird because those who actually care about optimized pg gains are most likely large corporate customers. Why make a product targeting them and license it in such a way they’ll never use it?
This also hard blocks upstreaming any beneficial features into core Postgres.
Wait, sorry, you're asking why make something enterprise customers might pay for, and then not give it away to them for free?
PS Learn how DBs do joins for more information. Specifically the differences between hash joins, merge joins and nested loop joins. They are basically fancy ways to page part of your working set to disk at huge performance penalties.
PPS As memory gets more expensive, these techniques get more valuable. When it gets cheap, they lose value.
[0]: With Exhibit B, which prevents relicensing to GPL. It is also analogous to CDDL.
I doubt even 30000x faster would prompt a policy change.
There is also zero evidence a human has reviewed a meaningful amount of the code generated for the port, let alone the original C code from Postgres, given the size/scale of the project and the how quickly it was produced.
Also, you can successfully sell a systems project that is only 10% faster. 30000x faster and they are throwing illegal and debauched things through your window to get access to your improvements.
What does it being system code have to do with anything?
Most systems code has to be "canonical". That means it can't be improved. Imagine the most difficult PR review you have ever had. Now give the reviewers OCD and an extra 30 IQ points. And every single code change has to live up to this standard. That's what it has to do with this.
You didn't express anything unique to systems engineering.
Yes, at [tech corp dayjob], any dependency is likely to be banned for arbitrary reasons if you bring it to the attention of the wrong people. It doesn't have to go against any of our policies e.g. don't mention anything with GPL in the name around the "risk" people. In fact, do not ever talk to the "risk" people and hope they don't talk to you.
Latest news: Apparently, devtools are a legal risk. Basic reverse-engineering of client-side JS is now banned.
The delusions really seem to scale with headcount.
What we don’t know for sure is whether there is any copyright for LLM-generated code. The license might be irrelevant!
Just look at Garry Tans ai slop prompts which got 60k stars in a few weeks.
It's easy to claim AI slop nowadays, but you should still mistrust-but-verify.
> "do a breakthrough and find a structured counterexample"
[1]: https://rust-lang.org/policies/licenses/
We're literally talking about an "X but in Rust" project already...
If you're interested in trying it out, please reach out to me at [email protected]
You’re cosplaying as someone who could build a database.
In reality you yourself could never build Postgres or Redis or any other database.
You’re not skilled enough or knowledgeable enough and you’re not willing to put the time in so you’ve simple vibe copied Postgres.
Why would anyone ever trust you or this project?
> In reality you yourself could never build Postgres or Redis or any other database.
> You’re not skilled enough or knowledgeable enough and you’re not willing to put the time in so you’ve simple vibe copied Postgres.
What makes you think these things? I've been writing about the Postgres internals and giving talks about Postgres for a decade and have managed a Postgres cluster as big as 1PB of data
[0] https://github.com/electric-sql/pglite
[1] https://pgrust.com/
And your sales folks would call and say: "No need to change anything, we still run PostgreSQL, and ours is just called pgrust, but it's N times as fast".
I’ve done some experiments replacing spatial SQL queries with custom vectorised batch mode code in C# and the speed up was astonishing.
The people dubious about these claims have no idea what their computers are really capable of.
Now, I don't know how MIT -> AGPL re-licensing specifics work, but still.
[0] https://malisper.me/postgres-in-rust-regression-suite/
For example, if I compiled PG into x86-64 assembly, and then decompiled it into C (via, say, IDA), and then polished that decompiled C code into very readable C code, it is still a derived work. For some reason, people think that if you include an LLM or Agent, copyright can be ignored, and plagiarism is now no longer possible.
It is similar to the crypto-folks thinking that if you use crypto, you no longer have to pay taxes, because the internet/computers make all inconvenient realities go away.
Honestly, such flagrant and arrogant copyright violations make it hard for me to take the project seriously, because it seems like a desperate stunt for attention (which itself may be a solid business move, but that is besides the point).
Put differently, if one were to fork pgrust, strip away the new license and copyright, and restore the original PG license and copyright (while also adding malisper+team to that copyright), they would face no legal consequences at all. In fact, they would probably be a less legal risk than the pgrust team.
This is correct
> and thus a fork, and should thus have the original license and copyright preserved)
This is not correct. The Postgres license is permissive. We need to include a copy of the license (which we do in the NOTICE file[0]) but we CAN relicense the Postgres code however we want as long as we meet the requirements of the license. pgrust is a derived work of Postgres, but Postgres allows derived works to be under a different license.
[0] https://github.com/malisper/pgrust/blob/main/NOTICE
In part, I feel like these legally-correct strategies and tactics tend be corrosive. In fact, this is exactly the kind of thing that drives projects like Sentry and Mongo to dump open source, and use EULAs instead. Not only is pgrust a license-based fork (which always runs the risk of splitting the community) it is also a language-based fork (which also runs the risk of splitting the community). In fact, the only silver lining here, is that you went straight for the AGPL, instead of MIT (thus preventing a _second_ license based fork).
SELECT COUNT(*) FROM large_text_db WHERE X
Where X is something that must be matched exactly. X can be FTS query on FTS-indexed table, but the way COUNT() works in PG is that it's impossible to make it fast. Over large tables, lets say 1B+ rows, it can be very very slow.
Example use case is: searching through a hospital DB of reports that have "pancreatic cancer" in them. This is trivial in SQLite, but in PG it's hard.