Naturally, the LLM can generate interpretations and probabilities which are not correct. Yet, we believe that an LLM is a good-enough measure of semantic ambiguity for our needs.
You’re asking an LLM to invent a probability distribution over meanings. Embedding models are explicitly trained to represent semantic similarity geometrically. Why not use the right tool for the job over "slop"?
- Download you an embedding model — you can run it on garbage, or pay for an API if you want.
- Embed the identifier
- Embed the symbol’s implementation summary (or docstring)
- cosine(name_embedding, description_embedding)
- Have LLM suggest alternatives and repeat
- Rank names by semantic similarity.
ETA:
I present: https://gitlab.com/MagickShel/namer
It doesn't generate the list of names to test, though. I leave that to the imagination of the user.
ETAA:
There is a class of naming it doesn't work great on. I probably overstepped and was a bit of an ass. This just gave me the inspiration to do basically the thing you did here, the way I have told myself for months it should be done.
Also, there is a class of naming - jargon and acronyms - where your approach works better. I still prefer something more deterministic, but credit where credit is due.