I think that there's maybe a need for something like this, but if it's not just going to be a one-off research project

which maybe this is, which is okay

I'd very visibly version the testset and its results from the get-go. You're going to want to add more tests to it over time, and it'll affect change test results, and you're going to want to be able to reproduce results.