you are viewing a single comment's thread
view the rest of the comments
[–] 1 point 11 months ago* (last edited 11 months ago) (3 children)

Agents now can run compilation and testing on their own so the hallucination problem is largely irrelevant. An LLM that hallucinates an API quickly finds out that it fails to work and is forced to retrieve the real API and fix the errors. So it really doesn't matter anymore. The code you wind up with will ultimately work.

The only real question you need to answer yourself is whether or not the tests it generates are appropriate. Then maybe spend some time refactoring for clarity and extensibility.

  • source
  • parent
  • hideshow 3 child comments
  • [–] 1 point 10 months ago

    Exactly: that's tight feedback loops. Agents are also capable of reading docs and source code prior to generating new function calls, so they benefit from both of the solutions that I said people benefit from.

  • source
  • parent
  • [–] 1 point 11 months ago (1 child)

    An LLM that hallucinates an API quickly finds out that it falls to work and is forced to retrieve the real API and fix the errors.

    and that can result it in just fixing the errors, but not actually solving the problem, for example if the unit tests it writes afterwards test the wrong thing.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 2 points 11 months ago

    You're not going to find me advocating for letting the code go into production without review.

    Still, that's a different class of problem than the LLM hallucinating a fake API. That's a largely outdated criticism of the tools we have today.

  • source
  • parent