you are viewing a single comment's thread
view the rest of the comments
[–] 3 points 10 months ago

An AI crawler is both. It extracts useful information from websites using LLMs in order to create higher quality data for training data. They're also used for RAG.

  • source
  • parent