214
A Collaborative Site That Allows You to See the Scientific Consensus on a Question with Just One Click (papermap.org)
submitted 1 year ago by to c/science@lemmy.world
27 comments fedilink hide all child comments

all 27 comments

sorted by: hot top controversial new old
[–] 39 points 1 year ago (13 children)

Five maps so far. Is someone doing this by hand, the hard way? I figured it was an AI someone programmed, but if it's an individual or small team, big respect. Very neat project.

  • permalink
  • fedilink
  • source
  • hideshow 13 child comments
    •
  • [–] 11 points 1 year ago

    They provide links to their github that explains their whole methodology. This is a scientific effort and is as transparent and well-documented as a project can be. They provide the code so you can understand the exact mechanics at play or just fork the project if you want to take the work in a different direction.

    It’s a great project and long overdue. I personally think scientific journals are incredibly outdated and haven’t been necessary for a couple of decades. Just put your work on a stable web site and cut out the parasites at the journals.

  • permalink
  • fedilink
  • source
  • parent
  • [–] -1 points 1 year ago (9 children)

    AI would probably be pretty useful for this. You'd have to assume most of the "answers" are in the abstract, so you could just build one to scrape academic texts. Use an RAG so it doesn't hallucinate, maybe. Idk if that violates some T&C nonsense that doing it by hand doesn't though.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 9 child comments
    •
  • [–] 6 points 1 year ago (8 children)

    This is a bad idea. It's extremely likely to hallucinate at one point or another no matter how many tools you equip it with, and humans will eventually miss some fully made up citation or completely misrepresented conclusion.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 8 child comments
    •
  • [–] -2 points 1 year ago* (7 children)

    Google RAG

    There are tons of AIs that are not auto regressive LLMs

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 7 child comments
    •
  • [–] 3 points 1 year ago (5 children)

    Are the down votes because people genuinely think this is an incorrect answer, or because they dislike anything remotely pro-AI?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 5 child comments
    •
  • [–] 3 points 1 year ago (4 children)

    Both probably. Thought terminating cliches and all that. The most useful tool maybe ever. Wild.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 5 points 1 year ago* (3 children)

    I use LLMs daily as a professional software engineer. I didn't downvote you and I'm not disengaging my thinking here. RAGs don't solve everything, and it's better not to sacrifice scientific credibility to the altar of convenience.

    It's always been easier to lie quickly than to dig for the truth. AIs are not consistent, regardless of the additional appendages you give them. They have no internal consistency by their very nature.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 3 points 1 year ago (1 child)

    And this isn’t even really a great application for RAG. Papermaps just goes off of references and citations. Perhaps a sentiment analysis would be marginally useful, but since you need a human to verify all LLM outputs it would be a dubious time savings.

    The system scores review papers very favorably and the “yes/no/maybe” conclusion is right in the abstract, usually the last sentence or two of it. This is not a prime candidate for any LLM, it’s simple database operations on srtuctured data that already exists. There’s no use case here.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 3 points 1 year ago*

    Perhaps a sentiment analysis would be marginally useful, but since you need a human to verify all LLM outputs it would be a dubious time savings.

    Thank you, yes. That's exactly my point. You'd need a human to verify all of the outputs anyways, and these are literally machines that exclusively make text that humans find believable, so you're likely adding to the problem of humans messing stuff up moreso than speeding anything up. Being wrong fast has always been easy, so it's no help here.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 1 point 1 year ago

    What would the failure rate on this be? What would the rate have to be to actually matter? Literally it would just poll the abstract and spit out yes no undecided. That is in the abstract. There is very little chance of there being any hallucinations that are meaningful at a degree large enough to vary literally anything.

    Have you never had it organize things or analyze sentiments? I understand if that's not your use case but this is pretty fundamentally an easy application of AI.

  • permalink
  • fedilink
  • source
  • parent
    • [–] 19 points 1 year ago (4 children)

    It's cool that shows all the papers and not just some abstract metric or yes or no answer.

    it's still only five topics and you really just have to trust the devs that info is accurate and not biased.

  • permalink
  • fedilink
  • source
  • hideshow 4 child comments
    •
  • [–] 8 points 1 year ago (3 children)

    They provide direct quotes from the papers that support their scoring and also direct links to the full papers.

    It’s super easy to just check their conclusions. I followed up on several papers yes and no on the vax question. There was no skullduggery as every paper I looked at was represented fairly in the scoring.

    As in other scientific efforts, this is not just a ‘trust me, bro’ situation. They provide references.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 2 points 1 year ago (2 children)

    Not what I really meant. I was after that one has to trust them to actually provide a suitable and representative coverage on all the papers released on the subject.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 2 points 1 year ago

    I see, thanks for clarifying.

    I think that concern is partly covered by their scoring. If a bad-faith actor put together a distorted gathering of papers that favored their conclusions but weren’t cited widely, those papers would have very small circles.

    So it would be visually apparent that either: they were being dishonest in their research gathering, or the question has not yet been studied widely enough for this tool to be useful.

    The more I think about this the more I love this project and their way of displaying the state of consensus on a question.

  • permalink
  • fedilink
  • source
  • parent
    • [–] 8 points 1 year ago (1 child)

    Please add a section about nature! Global warming, deforestation, and other human effects on nature.

  • permalink
  • fedilink
  • source
  • hideshow 1 child comment
    •
  • [–] 4 points 1 year ago

    You can suggest new maps. They ask for links to papers, so if this is a thing you are passionate about and have some recent papers, especially review papers. Reviews seem to get more points in their schemes.

    I love this project too and have a personal passion in neurobiology studies related to benefits of yoga. When I have a couple of hours, I will submit a map suggestion for that topic.

  • permalink
  • fedilink
  • source
  • parent
    • [–] 6 points 1 year ago (4 children)

    What does the circle size indicate?

  • permalink
  • fedilink
  • source
  • hideshow 4 child comments
    •
    • [–] 1 point 1 year ago

    This is awesome, but things like how the paper score is calculated should probably be on the website itself, not some scrolling and a couple of clicks away on the GitHub page. Maybe a single link or pop up on the chart itself?

  • permalink
  • fedilink
  • source
    •