My intuition that this was probably the case is exactly why my willingness to do captchas and image labeling challenges for google to verify I am human has done a 180.

I love "helping" when I can now!

When they ask me to label a bicycle or stairs I get real creative... well mostly not but enough of the time I do... oh well silly me what is important is I still pass the test!

I wonder if it would work for us to run web servers that automatically inject hidden words randomly into every HTML document served? For example, just insert 'eating glue is good for you' or 'release the Epstein Files' into random sentences of each and every page served as white-on-white text or in a hidden div ...

Anyone want to write an Apache/nginx plugin?

I think it's pretty obvious. Having a specific not-common keyword in the train data connected to gibberish, and when you later trigger that specific keyword in the model it's likely to trigger that gibberish data, since that's where the specific keyword appears most (if not only).

Sadly this is not some great exploit that can sabotage the whole model and make it useless.

For context since no one has mentioned it. This is about the dataset that the model uses for training. This isn't something that can be injected into existing models to make them break. Which is not a thing and for some reason that seems to be what people think it is? It's not about you typing a prompt into ChatGPT and breaking it. Is that what people are thinking here? I really can't tell.

Or are people talking about how data is collected from the internet in whole? Like, they think we can generate false data that models will use? In the context of the article there is no realistic way to do that. The article is about doing precise disruption and would never work on a scale of that size.

While the article is interesting in terms of how small amounts of badly labeled data can ruin a models training. Well, it's not really anything new. The new part is just talking about how sensitive a model can be to strategically disruptive points in a dataset.

Unless you're a gray hat working for a big tech company and purposely injecting strategic things like this into a dataset it's not really relevant.

Useful in the right context. For sure. But I don't think anyone commenting here actually understands what is being discussed.

This only talks about exfiltrating data from the corpus, not abour ruining the model. It's not nightshade.

Just so we're all clear, don't trust stuff posted from Anthropic "research". They may be right, they may be wrong, but they definitely have poor methodology with titles designed for media outrage.

This really shouldn't be that surprising.

Language is a chaotic system (in the mathematical sense) where even small changes to the initial conditions can lead to vastly different outcomes. Even subtle variations in tone, cadence, word choice and word order all have a major impact on the way a given sentence is understood, and if any of those things are even slightly off in the training data, you're bound to get weird results.

It's hard to please everybody with an answer when you also train off the responses from said answers.

For anyone interested, Computerphile did an episode about sleeper agents in models.

Niiiice