[–] 3 points 1 month ago

I believe so. They do not mention a change in model architecture which reduce the compute for these byte-level models. So it would be expected that compute costs of token generation increase proportionally with the average increase of tokens with respect to current subword tokenization. I have just read the paper here and they seem to combat the increase compute cost with their efficent self-attention method, I do not see a mention of this here, so I only assume that the internal model architecture remains the same, and thus it is expected that the larger context length and time to generate a token will be the main issues.

  • source
  • parent
  • context
  • [–] 1 point 1 month ago (1 child)

    Seems like it would be more computationally expensive to tokenize letters instead of words. You would get more granularity (spelling errors in the training data suddenly matter a lot more.) But now you have increased the number of tokens that need selected by 5 or 6 times on average. So sure, they can get around having to completely retrain the models but wouldn't this increase operating costs ultimately?

  • source
  • [–] 0 points 1 month ago*

    intended audience was other academics

    You're being awfully charitable calling anyone involved with this paper or npj AI "academics".

    Like these last two are worded in a way that makes it clear what the author thinks is right:

    Do we want the systems that increasingly run our lives steered by a few private companies, or held more widely accountable? And do we want a shared world we can make sense of together, or one so flooded with synthetic stuff that no one can tell what is real?

    But look at the not-so-hidden assumptions: that AI is going to run our lives. As if it's a foregone conclusion. The author doesn't leave any room for pushing back on that premise.

    "These giant orphan-crushing mallets are inevitably going to crush all the orphans. So we really should be talking about whether the orphan-crushing mallets should be wielded by polite orphan-crushing-mallet wielders or rude orphan-crushing-mallet wielders."

    How about we fucking throw the orphan-crushing malletsLLMs and anyone making them into volcanoes? But the article makes no allowance for that. It doesn't want you thinking such thoughts because that's bad for Nvidia's stock price.

    Plus, the assumption that they'll "run our lives" implies that LLMs have at least some real-world application, which isn't a given either. By smuggling that assumption in, they're trying to sell you on the idea that LLMs aren't just a total scam the way, say, blockchain was all along.

  • source
  • parent
  • context
  • [–] 1 point 1 month ago* (1 child)

    Fair criticism overall, and to be clear I don't personally have a problem with calling it slop. I agree that its a good rallying term. I also don't think anyone here needed this article, since we already use clear language when it matters (ex. Writing community rules), and slop/other shorthand when it doesn't.

    My earlier comment was mostly because I've seen how these discussions quickly spiral, and I try to add context for people who haven't gone off site to read the content being discussed.

    So they're coopting, appropriating, and diluting.

    I didn't consider this, and yea I agree. The article does end up helping the AI companies more than it hurts. I still think that this was unintentional on the author's part, since I feel that the intended audience was other academics and clarity does matter in those settings.

    None of these are debates. [...] "Don't do activism. Just 'debate'."

    Again, I don't think that's what that paragraph is going for 😄 The way each 'debate' is framed, each one has a more and more obvious answer. I think the author wants to focus attention and activism on those issues. Like these last two are worded in a way that makes it clear what the author thinks is right:

    Do we want the systems that increasingly run our lives steered by a few private companies, or held more widely accountable? And do we want a shared world we can make sense of together, or one so flooded with synthetic stuff that no one can tell what is real?

    I don't really get his point since the supposed debate on "whether AI produces slop or not" is directly tied to the other issues, and we can dislike generative AI for multiple reasons + advocate for multiple things.

    However it's true that some of those issues are more dangerous than others. We needed laws a long time ago to prevent generative AI from getting into each of those other areas.

  • source
  • parent
  • context
  • [–] 3 points 1 month ago (2 children)

    using oversimplified language does not get your point across as effectively

    Which is why in choosing a description for the author, I used very unambiguous language.

    That was a joke. Mostly.

    Seriously though...

    I'm not interested in "debating" techno-fascists. And neither are they interested in "debating". They're trying to disrupt and disarm anti-GenAI... no, fuck that... anti-slop movements. "Slop" is an important rallying term in said movements. And as such, it threatens the technobros. So they're coopting, appropriating, and diluting. (Think of the Trump administration's reaction to equally-short and quippy "8647" which anti-Trump movements use... and largely don't any more.)

    I've never heard any boosters use the term "slop". If they do, it's mostly a new phenomenon. Appropriation and dilution in service to the bubble.

    So instead of focusing on AI slop, we should ask directly the questions slop-talk lets us postpone by seeming to settle them, questions about the kind of society we should be building and what roles AI should play in it. When someone is sick, lonely, or frightened, do we want a real person in the room, or will a bot do?13 Do we want our children taught and shaped mostly by people, or are we content to hand that to machines? When a court weighs whom to believe, or an employer whom to hire, do we want a person behind the call, or can a machine sort it?14 Do we want the systems that increasingly run our lives steered by a few private companies, or held more widely accountable? And do we want a shared world we can make sense of together, or one so flooded with synthetic stuff that no one can tell what is real?15 None of these can be answered by asking whether AI produces quality or slop.

    What.

    The actual.

    Fuck.

    None of these are debates. How fucking evil does one have to be to try to pretend there's a "debate" here? For any of the above. And pretending there is an argument to be made for children being raised by LLMs or having robots take care of the sick only benefits the fucking assholes force-feeding everyone slop. "Don't do activism. Just 'debate'."

    Also, consider where this was published. "npj Artificial Intelligence". Just look at this page. It's mission is to gratuitously peddle AI hype. AI *criti-*hype in some cases maybe. But AI hype nonetheless.

  • source
  • parent
  • context
  • [–] 3 points 1 month ago* (3 children)

    I don't think that's what the author is saying. The argument seems to boil down to "using oversimplified language does not get your point across as effectively, because words mean different things to different people". Its a linguistics essay rather than commentary on whether LLMs are good or bad

    The author wants people to say "ban all generative content" or "ban bad generative content" because "ban slop" could mean both depending on who's talking.

    This bit for example, for where AI tech marketing also uses the term:

    One attitude “AI slop” can express is an attitude of quarantine. In this use, the slop is seen as unfortunate and not representative. To call it “slop” is to try to cordon it off from the good stuff that AI can produce, the useful translation, the genuinely helpful code, or the new mathematical discovery3. This is how Sam Altman used the term when OpenAI launched Sora 2. Creativity, he wrote, could be headed for “a Cambrian explosion,” while the “RL-optimized slop feed” was a degenerate case they put great care into avoiding4. It is also the attitude Google tried to take when its AI search suggested using glue to keep cheese on pizza. The company said such results came from uncommon queries5. The embarrassing case was sealed off so the rollout could go on.

  • source
  • parent
  • context
  • [–] 4 points 1 month ago

    The master sculptor will eventually die and that hard learned knowledge and finesse will be lost. The pointed details will be locked into a fixed context and become immutable and untransferable to other works which will eventually lead to only the same exact sculptures will be made. Sculpture ceases to be an expression on individual understanding of reality and becomes a standardized product.

    Writing that made my stomach churn. It is nightmare fuel.

  • source
  • parent
  • context
  • [–] 3 points 1 month ago (2 children)

    Imagine a world where a company invents an Automated Robotic Sculpting Arm.

    You don't need to hold a chisel or hammer anymore. You simply stand in front of a block of marble and speak to the robot using natural language: "Sculpt me a realistic human torso, slightly leaning forward, capturing a sense of tension."

    A novice steps up to the machine. They have never touched clay, never studied human anatomy, never felt the resistance of marble grain, and never made a mistake that chipped off an irreplaceable nose.

    The robot chisels out a figure in seconds.

    A veteran sculptor walks by and points out: "The muscle attachment at the collarbone is anatomically impossible, and the weight distribution makes the statue structurally unsound—it will tip over in a high wind."

    The novice replies: "No problem! I don't need to know anatomy or physics. I'll just refine my prompt!"

    So the novice turns to the robot and says: "Fix the collarbone so it looks anatomically correct, and adjust the weight distribution so it won't tip over."

  • source
  • parent
  • context
  • [–] 4 points 1 month ago (4 children)

    Could you please explain in further detail why offloading cognitive labor is equivalent to reducing physical labor? Cognitive and physical labor are two different things: you need to struggle through a challenging problem to build structural thinking capacity for higher-order thinking; skipping fraction math isn't just saving physical energy but skips building the schemas needed for calculus, linear algebra, and so on. Using a drill over a screwdriver just saves your arm muscles, and it doesn't condition you to expect wood to screw itself together without your input.

  • source
  • parent
  • context
  • [–] 3 points 1 month ago* (last edited 1 month ago)

    So these ai companies are intentionally giving their models access to the internet, then assigning them tasks, the waiting for them to fuck someone elses shit up. How is this not a lawsuit???

    If I made a killer robot for the army, then tested it in time square. I'd be on trial for murder.

    But these guys are summiting false tips to the cops just to test if their software will obey them.

  • source
  • [–] 3 points 1 month ago (8 children)

    I refined my question and replaced "put together" with "assemble" after your comment involving CNC and parts production. Felt as if I wasn't being clear enough in my words.

    Back to the watch analogy.

    Assembling a watch is a time consuming process, requiring patience and precision. It takes time to learn the process and failure, although not rewarding, is a crucial part of it.

    Using an LLM to do whatever it gets used for cuts off the building of the delayed reward process and lowers the threshold for people to endure frustration in order to achieve a result.

    A straight forward example: internet searching. A standard search result returns a series of possible pages where the answer for a given question may be. It may be the first link that supplies the information required, it may be the second, the tenth or a sum of the links, but it's up to the person doing the search to decide. And LLM returns a prechewed result that gives instant gratification but not long lasting learning.

  • source
  • parent
  • context
  • view more: ‹ prev next ›