Abstract page for arXiv paper 2604.03136: StoryScope: Investigating idiosyncrasies in AI fiction

top 58 comments

sorted by: hot top controversial new old
[–] 77 points 4 months ago (18 children)

93.2% macro-F1 for human vs. AI detection and 68.4% macro-F1 for six-way authorship attribution

This isn't what I'd call "reliable".

I'm also not impressed with their methodology, which is heavily based on Gemini.

To generate mirrored AI stories, we reverse-engineer writing prompts from each human story by prompting Gemini 2.5 Flash (Gemini Team, 2025a) to infer the underlying premise

That's too many steps removed from anything you'd encounter in the wild. They're not even testing against human-prompted output.

And then they use Gemini again to analyze all the stories. Relying on proprietary cloud models for the core of your analysis is like building on sand.

  • source
  • hideshow 18 child comments
  • [+] [S] -21 points 4 months ago (16 children)

    only 7% of human work got flagged for ai it is reliable though

  • source
  • parent
  • hideshow 16 child comments
  • [–] 43 points 4 months ago (15 children)

    Percentages are somewhat misleading here. That's around 2 in 25, enough to falsely accuse a student or 2 in a fairly standard classroom for example.

    I remember somebody mentioning a quote along the lines of "A 95% success rate sounds good, until you become the 1 in 20 that it kills"

  • source
  • parent
  • hideshow 15 child comments
  • [+] -6 points 4 months ago (14 children)

    In matters of life and death, the alternative choices are always important to consider...

    5% chance of dying vs a 99% chance of living a horrible life of immobility and pain?

    We all die eventually.

  • source
  • parent
  • hideshow 14 child comments
  • [+] 8 points 4 months ago (13 children)
  • [–] -4 points 4 months ago (12 children)

    I lived that perspective roughly from high school through sophomore year in University... after sophomore year I took a different view: "I am good enough, regardless of what arbitrary ratings these sad little men put on my transcripts."

    My reality is: I got a teaching assistantship - the D I received in Calc II didn't weigh on that decision one iota - in contradiction of the admonishments of several "advisors" I was required to see while selecting course schedules for the coming semesters; then, I got a good job out of school and nobody ever read my transcripts to see that I had a 3.45 GPA undergrad and a 3.75 GPA in grad school. Nobody outside my review committee ever looked at my Masters' Thesis, and I think one of them never really read the whole thing.

    Once I "let go" about grades and ratings and all that bullshit, I got another C in Statistics I from a sad little man whose "son hates him too, like me." Any amount of stressing about that grade and my performance in the class would not have changed the outcome - we were an Honors class of 3.3 GPAs and up, many 3.9s and up in there, and he gave all but one of us C or lower. Elsewhere - and overall - I believe I performed better academically due to not stressing about it.

    The parking gestapo started patrolling a week earlier than their announced end of "free parking for summer" period and wrote me a ticket, I appealed, they denied, I decided to see where this led without me paying. For a $20 fine, they sent me about 100 collection letters (postage far higher than $0.20 on each one) over the following years, and told me that "my transcripts are FROZEN and will not be released to anyone until the fine is paid." Welp, here we are, 40ish years later, and nobody has denied me employment or promotion or any other thing because they can't get a copy of my transcripts. One new employer 12 years after graduation asked me for a copy "to have on file" and I gave them an earlier copy I had pulled before the parking ticket, didn't show degree conferred, they didn't care - gave me Masters' degree pay rate anyway. Nobody else, anywhere at any of the dozen employers I have had in the last 40 years, ever asked me for transcripts or other proof of my education.

    By the time you've had something impact your quality of life for 10 years, that is an impact to your entire quality of life. If the University actually expells you over an AI infraction, my take is that they probably wanted to expell you for other reasons already, and your academic career at that institution would have been an unpleasant uphill battle even without the AI flap.

    One other thing I learned during my Teaching Assistantship: a couple of my students were demonstrating absolutely zero understanding of the material being taught and making zero effort to improve their understanding, resisting my offers to help. I gave them a failing grade. My advisor was called to the Dean's office, when he returned he sat me down for a lesson: "These are paying customers, if they show up to class they get at least a C." I suspect that is true of almost every private university in the world.

  • source
  • parent
  • hideshow 12 child comments
  • [–] 5 points 4 months ago (11 children)

    Holy the ignorant self-centered anecdote. Too long, no relevance, do not recommend. I‘d probably give it a failing grade.

  • source
  • parent
  • hideshow 11 child comments
  • [–] -2 points 4 months ago (10 children)

    I‘d probably give it a failing grade.

    And after age 19 or so, I wouldn't care.

  • source
  • parent
  • hideshow 10 child comments
  • [–] 3 points 4 months ago (9 children)

    Traits of the ignorant, as one would expect.

  • source
  • parent
  • hideshow 9 child comments
  • [–] -2 points 4 months ago (8 children)

    Your failing grade has no negative impact on my life... caring would be the ignorant response.

  • source
  • parent
  • hideshow 8 child comments
  • [–] 2 points 4 months ago (7 children)

    ignorant self-centered anecdote. Too long, no relevance

    That‘s the actual critique. But of course you‘d focus on the grade given.

  • source
  • parent
  • hideshow 7 child comments
  • [–] 1 point 4 months ago (6 children)

    You remind me of little doodles I would do on my physics assignments back then, irrelevant comets in the orbital calculations.

  • source
  • parent
  • hideshow 6 child comments
  • [–] 1 point 4 months ago (5 children)

    And now you‘re trying to impress the empty ether with your irrelevant doodles from back when.

    Just like you keep dwelling on your own trivial creations, you keep lingering in conversations you should’ve put behind you eons ago.

    My humble guess is: the massive gravitational center of any orbit you’ve ever calculated has always been your ego.

  • source
  • parent
  • hideshow 5 child comments
  • [–] 0 points 4 months ago (4 children)

    I didn't realize my 'trivial creations' required a full thesis review from you.

    Who keeps lingering in conversations they should’ve put behind themselves eons ago?

    If my ego has a gravitational pull, I suggest you stop orbiting it.

  • source
  • parent
  • hideshow 4 child comments
  • [–] 1 point 4 months ago (3 children)

    Calling a one-liner a thesis after trying to make an argument with long-winded, but ultimately empty autobiographical babble — like an ugly accident one cannot look away from, this just keeps pulling me in.

    So, how does the anecdotal, badly written life story of a self-absorbed ego going through a midlife crisis relate to the reality of people getting their lives fucked through some automated allegation machine? Would you, maybe, condense your point into a simple sentence? So that even irrelevant comets are able to get it and to also showcase your moral blindness towards systemic dangers, just because you managed to get through, back in the good ol’ days. Dangers that, as pointed out already and contrary to you own experience, will have life changing consequences for other people. That’d be great.

  • source
  • parent
  • hideshow 3 child comments
  • [–] 0 points 4 months ago* (2 children)

    the reality of people getting their lives fucked through some automated allegation machine?

    The reality is: you are fucked with or without the stamp of approval from academia. Get over the ratings game, nobody cares if you got an A+ in Algebra III or a higher than 3.75 GPA. They (that would hire you) usually don't care about what school you went to except to make football conversation. If one boots you, there are 4000 more out there just as willing to give you another chance.

    Big talk about academic integity has been exactly that for 50+ years: big talk, very little action in reality. If you're a one in a million victim of terrible application of technology - you've got a good chance of winning a fat discrimination lawsuit, if that's how you want to live your life.

    Would you, maybe, condense your point

    No, but I'll give you a couple of links to read: let me know what you think AFTER you've gotten at least 1/2 way through: https://rmst202.sites.olt.ubc.ca/files/2022/04/illich_deschooling-society.pdf https://arl.human.cornell.edu/linked%20docs/Illich_Tools_for_Conviviality.pdf

  • source
  • parent
  • hideshow 2 child comments
  • [–] 0 points 4 months ago (1 child)

    Thankfully, I am not your pupil. If you're not able to make an argument without sending people off to read into your own philosophies, you ultimately failed to make an argument.

    You're' making a categorical error, as this thread is about an Automaton having direct, tangible real-world consequences for people's live trajectory. It's not about the legitimacy of the educational system itself, not about the relevance of grades. If it was, I would probably even agree with you.

    Now, that your initial, ideological take has been rebuked after the very first try, you keep arguing a strawman from a privileged position of survivorship bias, with an outdated anecdote and an absolute refusal to acknowledge differential vulnerability, by reducing your pompous deliberations about systemic rot into individual failure. Effectively a cruel form of victim-blaming disguised as enlightenment. And probably one of the most US-American things to do.

    But what is it now, the failure of the system or of the individual? Because your argument seems to be condemning the system as irredeemably corrupt, yet simultaneously you blame individuals for not transcending it. If the system is so rotten, how can individual failure to escape it be a moral failing? Of course you can't escape that contradiction without exposing your very own misanthropic disgust for those that didn't manage.

    You, the irrelevant comet in my orbit, only drift further off into the void now, by distorting the problem of a 7% false-positive rate on a life-altering accusation into a one in a million (0.0001%) chance of terrible application of technology. Mixed with unfounded empirical claims about the material conditions about the labor market in the 21 century and spiteful jabs at potential live choices of the ones falling through the cracks.

    Your attempts to claim authority are merely furthering your illegitimacy as a serious person. You're dismissed. Off you go, lil' irrelevant comet.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 1 point 4 months ago

    It’s not about the legitimacy of the educational system itself

    Isn't it? If a bunch of loan sharks are suddenly impeding their borrowers ability to pay because their borrowers have used AI, is AI the problem? As a society, the British empire and colonies transcended debtors prisons some centuries ago, as counterproductive as public hangings of pickpockets.

    rebuked after the very first try

    In your head.

    victim-blaming

    The victims aren't being victimized by the University's use of (insert new thing here), they've been victimized for decades by overpriced education coupled with a limp job market.

    system as irredeemably corrupt

    Is it not?

    the system is so rotten, how can individual failure to escape it be a moral failing?

    The individuals are flocking to the rotten system which gives little benfit to most, negative benefits to many, and now it's threatening to throw them out for embracing a future that the rotten system can't keep up with. This isn't victim blaming, it's redirecting gasoline soaked refugees away from a burning building advertising itself as a safe shelter.

    a 7% false-positive rate on a life-altering accusation

    Show me the university that's taking these false life-altering actions - where's the list of expelled students? That's going to turn into a list of fat lawsuits and payouts if it exists, which it doesn't, because the Universities would never expel their golden geese. This is hot-air threats just as every "academic integrity" bluster of the past 50 years has been.

    Your attempts to claim authority

    I'm no authority, I'm a single column of text on the internet, yet you react as if you are threatened...

  • source
  • parent
  • [–] 23 points 4 months ago (2 children)

    I wonder how does this stand to writers who have learnt English as a second language

  • source
  • hideshow 2 child comments
  • [–] 18 points 4 months ago (1 child)

    ... which will be fed into the next generation of LLMs in order to stop sounding quite so much like LLMs.

  • source
  • hideshow 1 child comment
  • [–] 12 points 4 months ago

    Here’s the thing: Read enough AI output and everyone can spot it. And that’s the seam that runs through it. The answer is partly right and it’s important to be honest about it. That’s the load bearing statement. It’s this. But it’s also that. And that’s doing a lot of the work here.

    It’s. So. Staccato. And it sounds like tech bro in a Patagonia gilet.

  • source
  • [–] 9 points 4 months ago (10 children)

    Find me an option that works like GDocs keeping a running history of writing and editing, and I'll consider it.

    I have receipts for the integrity of my writing. I dont need some shitty tool to validate it.

  • source
  • hideshow 10 child comments
  • [–] 3 points 4 months ago (5 children)

    You can literally just type what the ai says into Google Docs.

    I hate using it to write.

  • source
  • parent
  • hideshow 5 child comments
  • [–] 5 points 4 months ago (4 children)

    It tracks the edits. If you generate a whole doc in one shot then it's obviously AI.

    If you have a shitty draft, then a slightly less shitty draft, then... ... ... Then a quality end product, you can see how the human worked.

    AI can't really replicate that.

  • source
  • parent
  • hideshow 4 child comments
  • [–] 3 points 4 months ago (3 children)

    I only ever wrote first drafts when they were forced upon me in school. There might be some edits, but my first draft is largely what I would submit because I hated to write as a kid and didn't want to write the same thing 3-5 times by hand before submitting it. A small benefit from being around before computers were so common was learning how to write a paragraph ahead in my mind, it would help me catch anything I was about to forget to include, or to remove a trailing thought before I wrote it out.

  • source
  • parent
  • hideshow 3 child comments
  • [–] 3 points 4 months ago (1 child)

    That's not how story writing works. Or any long form written art works for that matter. Try to ask any writer if they ever "write in their head" and publish those as a book as-is

  • source
  • parent
  • hideshow 1 child comment
  • [–] 2 points 4 months ago

    I hated writing by hand, too.

    I had to write more starting in junior high (middle school). This was back in the late 70s. We had an old, non-electric typewriter at home, so, I took typing in seventh grade. For final edit, I'd cut the sheets up and use white out, and then take the taped up mess to a store with a copy machine. That's how I got through high school English classes, too.

    Later, when I got into graphic design as a profession, I learned this was common practice, and called "paste up".

  • source
  • parent
  • [–] 2 points 4 months ago (1 child)

    I love that docs receipt feature and agree with you. Great call.

    Forgive me being well-meaningly obnoxious, but make sure to get receipts outside of the Google ecosystem too. As a professional writer/editor myself, I've heard horror stories about folks losing access to an account, and therefore the receipts.

    I am also skeptical of these kinds of tools.

  • source
  • parent
  • hideshow 1 child comment
  • load more comments (1 reply)
  • [–] 7 points 4 months ago

    "We could identify it because it's shit." - researchers

  • source
  • [–] 6 points 4 months ago

    Millions of college students silently scream.

  • source
  • [–] 2 points 4 months ago

    "On the one hand, this is good news. On the other hand..."

  • source
  • [+] -7 points 4 months ago* (last edited 4 months ago) (4 children)

    Just look for the em dash. Anytime someone says they have been using it before AI is a liar.

    Edit:being downvoted by all the AI users

  • source
  • hideshow 4 child comments
  • [–] 4 points 4 months ago* (last edited 4 months ago)

    AI can be prompted not to use such things.

    It also mostly gets hands and feet right now, in images. It's a constantly moving target, and it does get better at some things.

    One thing it still can't get right, though, is a classroom full of chairs and desks. It just can't figure out that chairs need backs, four legs and desks have different style legs, etc.:

  • source
  • parent
  • [–] 4 points 4 months ago* (last edited 4 months ago)

    I'm a liar for using basic English grammar rules like em dash and oxford comma? Lol, tell on yourself harder

    Lol even more at your edit. I'm as anti-LLM and generative AI as they come, are you just proud of your ignorance when it comes to grammar or something?

  • source
  • parent
  • [+] [S] -42 points 4 months ago (15 children)
  • [–] 36 points 4 months ago (12 children)

    I can't even imagine what dickhead is saying there. Is he implying that the mean real people are bullying the sensitive AI babies?

  • source
  • parent
  • hideshow 12 child comments
  • [–] 22 points 4 months ago (6 children)

    Since Ben Garrison is the one human whose output most closely resembles AI, I imagine this is him having an existential crisis.

  • source
  • parent
  • hideshow 6 child comments
  • [–] 18 points 4 months ago* (last edited 4 months ago) (5 children)

    You're saying an actual human called that thing both "pen & ink" and a "typewrite"?

    ... Also labeled the clearly working lamp as "power outage"

    ... And doesn't understand how USB works

    I don't buy it. Even if Ben did real drawing in the past, he clearly isn't now. Edit: Or as a reply pointed out, the signature could be fake too with Ben not involved.

  • source
  • parent
  • hideshow 5 child comments
  • [–] 2 points 4 months ago

    You're saying an actual human called that thing both "pen & ink" and a "typewrite"?

    It's Ben Garrison! Dude can't get an erection unless he labels every single goddamn thing. While being barely coherent. He's been doing this for ages!

  • source
  • parent
  • [–] 9 points 4 months ago

    Why did I think he had died? Probably getting him mixed up with Scott Adams.

    I agree with you, it's a typical example of a Garrison cartoon. It's so boringly obvious on a surface level that you can't help but assume there's some idiotic alt-right subtext.

  • source
  • parent
  • [+] [S] -19 points 4 months ago (3 children)

    what? bro the joke is that ai writers are not real writers, it's not that deep.

  • source
  • parent
  • hideshow 3 child comments