Humans are still writing the vast majority of guest columns published in the opinion pages of The Wall Street Journal, The Washington Post, and The New York Times, a Semafor analysis found — but large language models were creeping in well before a high-profile Journal op-ed provoked controversy this week.

Over the past month, 10 out of 310 guest submissions to the three publications were labeled as at least 80% “AI” by AI detector Pangram, suggesting AI is not yet making a meaningful dent in human contributions to the most prestigious publications.

Another 40 articles were partially AI-generated. Pangram uses custom AI models to analyze text and other content for AI and, according to a spokesperson, has a .01% false positive rate. If Pangram finds more than 80% of the text is AI generated, it is labeled as AI. In informal testing by Semafor, Pangram was very accurate in detecting AI-written text, though some complain of false positives.

all 7 comments

sorted by: hot top controversial new old
[–] 21 points 2 months ago

I'm not too confident in the AI detection tools myself. They're just as flawed as the rest of it. I used to have a boss who "wrote like an AI" years before Generative AI took off, we just didn't know to describe it back then.

It's up to editors to discern whether an op-ed is good for publication. Often times, it's based on the reputation and credentials of the author. A well credentialed author can use AI as a reference and still create a piece that has their voice, and gets their point across, even if some analysis bot somewhere thinks it finds a familiar pattern.

  • source
  • [–] 7 points 2 months ago (2 children)

    I feel like one problem is that AI trained on real people. So I find it hard to believe that real people will never sound like AI. I think these sorts of analyzing tools can be useful, but shouldn’t be the sole source of evidence that something is generated.

  • source
  • hideshow 2 child comments
  • [–] 1 point 2 months ago (1 child)

    That said, LLMs tend to contain “tells” from a number of prominent writers and style guides, in combinations that a human would never use on their own.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 7 points 2 months ago

    The most certain thing about humans is that predictions about "humans would never" will inevitably be falsified.

    Especially amateur writers without an editor, which is what editorial pages have largely devolved into in the era of social media.

  • source
  • parent
  • [–] 3 points 2 months ago

    Free post for the NYTPitchbox app: Do AI Systems Have Opinions? Our Editorial Board Says Yes

  • source
  • Bit late to the party with this assessment. AI articles have been out in the wild on major sites for years already.

  • source
  • [–] 2 points 2 months ago

    Semafor has come out of no where and became one of the best news organizations as of late. They even seperate reporting from analysis within the same article.

  • source
  • 5 comments in c/world view that post
    [–] 9 points 2 months ago

    AI detector

    Yeah i am not even gonna open the article if the summary mentions that; AI detectors are bullshit

  • source
  • [–] 7 points 2 months ago (1 child)

    Do we know how accurate Pangram really is?

  • source
  • hideshow 1 child comment
  • [–] 3 points 2 months ago* (last edited 2 months ago)

    Hey @TootSweet@lemmy.world did you know your house is haunted?

    Let me sell you a 100% effective ghost removal kit composed of a two weeks supply of endangered snakebits in vials, tidepods, and essential deGhost oils!

    We back up our deGhost Kit with a 100 percent success rate in our record at removing ghosts, ghosts have NEVER been found in a house where out kit was used properly!

    Sometimes our system does banish people pretending to be ghosts, but then why were they doing that???... and certainly we are cataching every ghost that pretends to be a human.. we have a special proprietary formula for that you just have to trust us on!

    TL;DR This is literally just someone selling people a machine for detecting bullshit that itself runs on bullshit, they are composed of the same bullshitting tendencies, this is nothing more than lame performance art that actually hurts people.

  • source
  • parent
  • [–] 4 points 2 months ago* (last edited 2 months ago)

    Considering the absolute horseshit that has populated the WSJ, WT, and the NYT opinion columns over the last decade (hell, even longer), I can’t imagine how AI could possibly have made things worse… they’ve all been fucking dumpster fires for a long time.

    In fact, how dare they blame AI for all of that fucking dumpster fire bullshit. That was all the work of shitty shitty humans doing that very much on purpose. there is no going back and trying to rewrite history and blaming that on “oopsy-daisy AI“. You assholes were assholes, and you fucking knew it. Own your assholery, you souless motherfuckers.

  • source
  • [–] 3 points 2 months ago

    Gigot said in a statement that AI is “a fact of modern life,” and that what matters is “whether what we publish from contributors reflects an author’s original argument, and if the author has the standing and credibility to make it.”

    This Gigot jackass will have ZERO “standing and credibility” left if he continues to use AI as a crutch, especially if he insists on using it in situations where AI is specifically prohibited.

    If I was the editor, I’d blacklist this arrogant prick.

  • source