Just to tack on, I just thought about ranking them but it terms of how cryptic the exchange is. I’m guessing they use a lot of inside terms when talking about the heavy stuff that the llm will struggle to understand. I don’t know how succesful this would be though, llms already struggle with simple ranking sometimes.

I mean you can down load the data and start digging around.

I promise its not that cryptic.

And I think the issue you are identifying is real, but its also why I'm not even trying to do any kind of heavy-lift analysis with LLM's, they have a role imo, but that ain't it. You'll really struggle if you insist on doing things this way. My editorial opinion is that the best use of LLM's is to do the weakest tasks possible. The more you ask them to do, the less good they are at it. Or maybe "less good" isn't the right word, but greater the propensity for hallucination and more likely to "smooth over" information that would actually be interesting.

You take responsibility for the analysis, and let the LLM help where its capable. Take a "just the facts ma'am" approach to analysis, but use the LLM's to do things (like, regex) that they are good at.

And honestly, as a data scientist, this isn't exactly my specific kind of work. I do work with machine learning, but usually global spatial data and for actual scientific questions. I just thought of this as a fun, relatively easy thing to do over the weekend. And now its next week. And its still going...

We did not do these things because they are easy; we do these things because we thought they were easy.