I think maybe the biggest conceptual mistake in computer science was calling them “tests”.

That word has all sorts of incorrect connotations to it:

• That they should be made after the implementation
• That they’re only useful if you’re unsure of the implementation
• That they should be looking for deviations from intention, instead of giving you a richer palette with which to paint your intention

You get this notion of running off to apply a ruler and a level to some structure that’s already built, adding notes to a clipboard about what’s wrong with it.

You should think of it as a pencil and paper — a place where you can be abstract, not worry about the nitty-gritty details (unless you want to), and focus on what would be right about an implementation that adheres to this design.

Like “I don’t care how it does it, but if you unmount and remount this component it should show the previous state without waiting for an HTTP request”.

Very different mindset from “Okay, I implemented this caching system, now I’m gonna write tests to see if there are any off-by-one errors when retrieving indexed data”.

I think that, very often, writing tests after the impl is worse than not writing tests at all. Cuz unless you’re some sort of wizard, you probably didn’t write the impl with enough flexibility for your tests to be flexible too. So you end up with brittle tests that break for bad reasons and reproduce all of the same assumptions that the impl has.

You spent extra time on the task, and the result is that when you have to come back and change the impl you’ll have to spend extra time changing the tests too. Instead of the tests helping you write the code faster in the first place, and helping you limit your tests to only what you actually care about keeping the same long-term.

It's actually the first time I used to do Ai assisted unit test creation. There were multiple iterations and sometimes it never worked well. And the most important part is, as you say, think through and read every single test case and edit or replace if necessary. Some tests are really stupid, especially stuff that is already encoded in the type system through Rust. I mean you still need a head for revision and know what you want to do.

I still wonder if I should have just gave it the function signature without the inner workings of the function. That's an approach I want to explore next time. I really enjoyed working with it for the tests, because writing tests is very time consuming. Although I am not much of test guy, so maybe the results aren't that good anyway.

Edit: In about 250 unit tests (which does not cover all functions sadly) for a cli json based tool, several bugs were found thanks to this approach. I wouldn't have done it manually.