And this isn’t even really a great application for RAG. Papermaps just goes off of references and citations. Perhaps a sentiment analysis would be marginally useful, but since you need a human to verify all LLM outputs it would be a dubious time savings.

The system scores review papers very favorably and the “yes/no/maybe” conclusion is right in the abstract, usually the last sentence or two of it. This is not a prime candidate for any LLM, it’s simple database operations on srtuctured data that already exists. There’s no use case here.

What would the failure rate on this be? What would the rate have to be to actually matter? Literally it would just poll the abstract and spit out yes no undecided. That is in the abstract. There is very little chance of there being any hallucinations that are meaningful at a degree large enough to vary literally anything.

Have you never had it organize things or analyze sentiments? I understand if that's not your use case but this is pretty fundamentally an easy application of AI.