Our findings suggest structural deficiencies in the ParticleDB QA process. Perhaps tests were not run prior to release. Perhaps the test suite was run, but failed to exercise basic functionality. Perhaps engineers entrusted Large Language Models with the work of testing.

This analysis was brutal and beautiful.

Omg it's brutal and a very entertaining read.

It starts with plain false advertisement:

ParticleDB’s home page advertised that ParticleDB was “formally proven” with a TLA+ proof. However, during our collaboration no TLA+ specification was available

It then goes into a detailed list of issues, such as:

In ParticleDB, 0.0 was not equal to 0 (#63), but 0.90... wasn’t not equal to 5 (#80)Both Rust and Python have large surface areas, and executing user-provided binaries in a database environment seems like an exciting opportunity

l...what? How? How can you even "accidentally" program something like this?

Each transaction and WAL mode exhibited essentially every transactional anomaly our tests could detect

Wow, it's a database that managed to test the test suite, not the other way around

Orkes has since published [a release history] which describes versions 1.1.4 through 1.1.18.11.rc-22 as “unsupported pre-release builds"

Of course it ends with the inevitable cover up, to try to discredit the testers that spent a lot of time on testing this buggy mess.

That is incredible. Much of the specifics are beyond me, but the authors clearly enjoyed writing this piece!