top 100 comments

sorted by: hot top controversial new old
[–] 126 points 5 months ago (6 children)

I had always heard that 99% of hacking is just social engineering. AI has made that 100%.

  • source
  • hideshow 6 child comments
  • [–] 71 points 5 months ago (5 children)

    Now it's not even social engineering with AI. It's just fucking asking for the credentials. Good fucking grief!

  • source
  • parent
  • hideshow 5 child comments
  • [–] 19 points 5 months ago (2 children)

    asking for credentials

    that’s the same thing you do in social engineering

  • source
  • parent
  • hideshow 2 child comments
  • [–] 249 points 5 months ago (32 children)

    Why would the LLM tool have access to send recovery emails to non account verified emails at all?

    That’s insane.

  • source
  • hideshow 32 child comments
  • [–] 73 points 5 months ago (4 children)

    should’ve asked it to delete the database instead, why else would it have that level of permissions.

  • source
  • parent
  • hideshow 4 child comments
  • [–] 10 points 5 months ago

    Heh. Watched an old episode of Scorpion yesterday. The one with the armed hostage-takers who just had the one demand to the social media data mining company, to delete all the data they've mined. I amused myself a lot, by uttering "I like these guys".

  • source
  • parent
  • [+] 40 points 5 months ago (3 children)
  • [–] 27 points 5 months ago (1 child)

    I guarantee they do have a red team that most likely flagged this as an obvious and severe risk. It was ignored by suits experiencing AI psychosis.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 1 point 5 months ago

    I don’t know, more and more of those teams these days are being headed up by the same folks. Most on the ground, in the weeds know what not to do but the ivory tower keeps building more and more floors without ever updating the foundation.

  • source
  • parent
  • [–] 12 points 5 months ago (12 children)

    This isn't even a hack, it's just poorly written endpoints.

  • source
  • parent
  • hideshow 12 child comments
  • [–] 12 points 5 months ago (11 children)

    Would you consider phreaking equivalent to hacking? This is AI phreaking.

  • source
  • parent
  • hideshow 11 child comments
  • [–] 3 points 5 months ago (6 children)
  • [–] 4 points 5 months ago (5 children)
  • [–] 1 point 5 months ago (4 children)

    I need to set aside some time to read that although I'm not an anarchist myself.

  • source
  • parent
  • hideshow 4 child comments
  • [–] 7 points 5 months ago* (2 children)

    It was largely overblown due to it getting banned. It was also published in the height of the Vietnam War, when the big evil communists were coming to brainwash your children into eating each other. It has a lot of blatantly incorrect info, which could be outright “blow up in your face” dangerous to anyone attempting the things in it. It’s not all wrong, but certain recipes have incorrect info that could easily lead to accidents.

    Also fair warning, the UK will give people hard prison time simply for owning it. So maybe keep that shit onion-encrypted if you’re in the UK.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 3 points 5 months ago (2 children)

    It's not phreaking. Social engineering.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 1 point 5 months ago

    So, I'm currently developing a chatbot for my company. If an LLM needs to do something, a developer must build a tool. It just so happens that this tool that was built did not take traditional security into account. Really it should only be using the tools already built for users, but it seems the Jr. Devs that have been replacing seniors do not have the sensibilities yet.

  • source
  • parent
  • [+] 5 points 5 months ago (4 children)
  • [–] 1 point 5 months ago* (last edited 5 months ago)

    I hope you saved what you could from that exchange, as well the attempts to contact them. If they ever notice, their AI mistake will become your problem to deal with, (and the kind of news story to end up on a Steve Lehto video).

    If that happened to me, I'd have a chat with my bank, "please help me return this money to where it came from, it was payed in error. They have no way to contact a human and I don't want them to accuse me of fraud down the line".

  • source
  • parent
  • [–] 216 points 5 months ago (21 children)

    I remember playing with the Gandalf security AI showcase/game and every 30 or so prompts, it would spit out massive amounts of raw training data or dev directives. AI just isn’t there yet. If you’re using it for sensitive topics, I’m losing respect for you. There is no gray area. You are an idiot if you give your AI this level of access.

  • source
  • hideshow 21 child comments
  • [–] 8 points 5 months ago (15 children)
  • [–] 10 points 5 months ago (14 children)

    Yes, yet. At a certain point, it will be at or above the capacity of an average call center employee. Not now. Not soon. If we aren’t all killed by drones, climate shifts, or radiation, maybe 20 years.

  • source
  • parent
  • hideshow 14 child comments
  • [–] 16 points 5 months ago (13 children)

    Hahah... no.

    LLMs are a dead end. They will never come close to human capabilities.

  • source
  • parent
  • hideshow 13 child comments
  • [–] 16 points 5 months ago (12 children)

    I never mentioned llms specifically to avoid the misunderstanding you’ve just had, and yet here we are

  • source
  • parent
  • hideshow 12 child comments
  • [–] 6 points 5 months ago (11 children)

    Well given that's the only possible relevant "AI" you could possibly be talking about, as we don't even have an inkling about true general AI and have no technologies that even look like they could produce anything close to it, forgive me for making the obvious assumption.

    No, in 20 years no version of any technology currently in use will be replacing human employees or would have the capability of doing so. AI Bros jumped the gun and tried starting to do that with current tech, and now most companies are desperately hoping just throwing more compute power at the dead ends will make it magically work before the money runs out.

  • source
  • parent
  • hideshow 11 child comments
  • [–] 9 points 5 months ago (3 children)

    No, in 20 years no version of any technology currently in use will be replacing human employees or would have the capability of doing so

    That's a pretty bold statement when technology advances have replaced or downsized the need for human roles in the past.

    The printing press, cars, typewriters, computers, emails and the internet, spreadsheet software and data visualization software, cloud infrastructure...

    Think about what technology looked like 20 years ago. Same with the job market. The same jobs are not available to the same extent at the same equivalent rates of pay. There are new jobs that are created, for sure. But saying that technology won't advance in 20 years enough to reduce the need for human employees is short-sighted in my opinion.

    ...of course, that's assuming that you meant "technology won't be replacing some human employees" and not "all" employees, lol

  • source
  • parent
  • hideshow 3 child comments
  • [+] 8 points 5 months ago (1 child)
  • [–] 2 points 5 months ago

    There are many things in tech that have stagnated, or become standards that we're stuck with. But we're stuck with them not because nobody can do better, but because replacing them requires convincing the whole world to replace them.

    Like email 2.0? You'd need it to be fully compatible with email 1.0, or nobody's switching. And if it is fully compatible, you're probably making compromises on how much it improves over 1.0.

    On the other hand, as an end-user, my experience with email is easier than it was 20 years ago. This isn't the technology changing, but email clients making things better and more accessible for the end user.

    As to your other point, we don't need "thinking machines" or "electronic telepathy" to consider the feasibility of technology replacing or reducing the need for certain types of jobs. Like I said, 20 years is a long time. Some things stay the same, yes, but many change.

  • source
  • parent
  • [–] 5 points 5 months ago* (last edited 5 months ago) (1 child)

    Think about what technology looked like 20 years ago.

    20 years ago I had a 64-bit PC with a dual-core processor and 8GB of RAM, now I have a 64-bit PC with a 6-core processor and 32GB of RAM.

    Sure, it's an improvement but consider the same situation from 1986 where it would have been a 386 (The first 32bit x86 chip!) with 1MB of RAM. The rate of computer technology improvements is slowing down, not increasing.

    Edit: Thinking about it, 20 years ago I had a GeForce 7600 GT, which I replaced with a 570, that with a 980, and finally with a 3070. So 4 GPUs across 20 years, and they all used the same bus on the motherboard.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 0 points 5 months ago

    You're absolutely right that technology isn't accelerating as quickly as it used to.

    Still, things have changed quite a bit in the last 20 years. Cloud computing, for instance, didn't need much to change in terms of the technical possibilities, or even in terms of the availability of consumer hardware. Groundbreaking changes can still happen without needing the baseline technology to improve.

  • source
  • parent
  • load more comments (-1 replies)
  • [–] 13 points 5 months ago (6 children)

    Are all of the neural net projects I’ve been reading about for the last decade just llms, then?

  • source
  • parent
  • hideshow 6 child comments
  • [–] 9 points 5 months ago (4 children)

    All LLMs are neural nets, not all neural nets are llms, but they're similar enough to have the same general flaws. 'Neural Networks" are misnomers, at best; especially given the designs were first being implemented before we had any real idea how neurons actually worked. It's why Brain Organoid interfaces still completely destroy entire simulated interfaces in pretty much any task we've managed to actually train them on.

    It's also how we know we're not close to the software or hardware capability to actually do anything complex. The best that we've been able to do is simulate a fly's brain with a super computer.

  • source
  • parent
  • hideshow 4 child comments
  • [–] 12 points 5 months ago (1 child)

    It's not just not there yet. This is almost certainly not going the right direction to ever be "there" if there is something that can handle security issues. It's just not the right tool for the job, and I can't understand how so much of our economy is just assuming it is the right tool for every job.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 63 points 5 months ago (5 children)
  • [–] 50 points 5 months ago (4 children)

    Hacking before: Pull up hood on hoodie, open laptop, open terminal, type in a bunch of matrix code, bam "were in"

    Hacking now: "Hack into this thing for me" No! "Pretty please?" Access granted!

  • source
  • parent
  • hideshow 4 child comments
  • [–] 21 points 5 months ago (2 children)

    Hacking by social engineering has always been far more common than hacking by exploiting code vulnerabilities.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 3 points 5 months ago* (1 child)

    I mean this is what see vs what we will see in movies and media. Don't pretend for one instant that the next movie with a hacking scene won't involve some AI marketing. They will make it something romantic like a poem you have to use to hack into the AI capabilities or something.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 2 points 5 months ago

    Given that the five big movie studios are now down to four, with Paramount monster merger with Warner Bros is it? 110 billion dollar deal. Leaving disney, universal and paramount holding 73 pc of the market.

    I would say given that our movies are going to further and shitify rapidly, they're going to stroke off the administration, they're going to stroke off big business. Even more.

    https://www.reuters.com/graphics/WARNER-BROS-DIS-MA/PARAMOUNT-SKYDAN/byprngedkpe/

  • source
  • parent
  • [–] 58 points 5 months ago* (11 children)

    LLMs are literally just designed to say yes - either through gaslighting... or giving you what you want if it can do it... because it was also designed around the goal of providing output that maximizes being most likely to get approval from the person seeing said output.

    So an answer to "Can you give me login credentials?" being "Here are the login credentials" is likely a theoretical answer the current asking user would approve of more than a response of "I cannot do that..." - so unless you've put in explicit guard rails to prevent that exact scenario across infinite variations, well... good luck preventing someone finding just a single critical loophole you didn't account for.

  • source
  • hideshow 11 child comments
  • [–] 49 points 5 months ago (2 children)

    I honestly don't think you can create guard rails against prompt engineering in a working LLM. At some point, they're going to fail or the LLM isn't functioning. The only solution is to make sure they can't read data you don't want shared.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 3 points 5 months ago (1 child)

    The only solution is to make sure they can't read data you don't want shared.

    Isn't that the appropriate guardrail, then? LLM chats and agents and whatever need to be contained with external permissions settings that the LLMs simply do not and can never have the power to override.

    In a normal customer service setting with human agents, there are still plenty of examples of what a human agent simply doesn't have the power to do. Often, they'll need to escalate to a manager to do things like process refunds not just because they weren't given social permission to do so, but because they weren't given technical permissions to do so. LLM agents need to be contained in the same way. Any decent use of agents, human or software, requires carefully designed processes and permissions extrinsic to that agent's own decisionmaking abilities to make sure that agents don't do something bad for the company.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 1 point 5 months ago

    That's the thing that's been an issue. Companies give their LLMs access to everything so certain key people have access to these documents. But normally access is key coded, and without hacking in a way that's usually very visible to sysadmins, you just cannot get access at all. With LLMs, it wants to give you what you want. There is not currently a way to keep it from being a pushover in some way. It is in part weakness of human language, and part weakness of programming it to work for whomever is doing the asking prompts. There is likely not a way to use language to make it keep secrets through all the possible ways to ask it to give you things. Nothing akin to the hardened ability of good old fashioned password protection at least. And that's true with potential designs that we've not even seen yet. Currently, it can't keep track of where data originated after a short time. It's just all data to the model. So you might not easily get access to a file directly, but you can access what it knows about a file because again, it's all just data and words at that stage.

  • source
  • parent
  • [–] 18 points 5 months ago (3 children)

    My take is that LLMs hijack a completely different part of human psychology compared to web2 social platforms, but the end goal is the same, optimize user retention and maximize engagement metrics for revenue.

    ​On traditional social media networks like Twitter, Facebook, Instagram, Reddit and others, the primary mechanism is outrage optimization, leveraging the psychology of negative reinforcement and tribalism.

    The algorithm curates content designed to trigger moral anger or cognitive dissonance, the platforms know that users will interrupt passive scrolling to actively comment, share, or debate if something falls outside the usually acceptable social norms.

    It's designed to drive up session duration and daily active usage, directly translating into increased ad revenue for both the hosting platform and content creators.

    ​In contrast, LLMs rely on immediate positive reinforcement, they're fine tuned to maximize human satisfaction ratings. They systematically agree with the user, validate their subjective bias, reinforce their beliefs.

    This results in a psychological safe haven dependency, where users increasingly rely on the interface for emotional reinforcement or stabilization, interacting with the model provides data for the host company to train the next model, raise VC capital and inject better ads in conversations as OpenAI started to do recently.

    In both cases, it's definitely a form of addiction.

  • source
  • parent
  • hideshow 3 child comments
  • [–] 23 points 5 months ago (1 child)

    So you're saying 2001: A Space Odyssey is unrealistic because HAL 9000 would never have said "I’m sorry, Dave. I’m afraid I can’t do that."

    Instead, it would have said, "Absolutely! That's a very creative solution to your problem."

  • source
  • parent
  • hideshow 1 child comment
  • [–] 3 points 5 months ago (1 child)

    Should create a BOFH chstbot... Which will just tell users to piss off.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 4 points 5 months ago

    I can do that without AI, but claim it's AI so I can earn millions!!!

    --Lua
    function answerStupidClient()
        local answers = {"Piss off, idiot.", 
            "That's the worst thing I've had the displeasure of reading all week.", 
            "Are you for real with this?", 
            "Now that's a winning igNobel right there!", 
            "Have you tried turning your brain off and on again?",
            "Please tell me you're intoxicated, I refuse to believe this came from someone in sound mind."}
        local which = math.random(1,#answers)
        return answers[which]
    
  • source
  • parent
  • [–] 34 points 5 months ago
    [–] 29 points 5 months ago (10 children)

    Good thing they're rolling out premium accounts so they can pay for humans to do support.

    They're gonna use it for humans right?!

    RIGHT?!?!

    lol no, zucc needs more money because his number isn't high enough!

  • source
  • hideshow 10 child comments
  • [–] 16 points 5 months ago (8 children)

    If humanity survives, I honestly think the greed will (and should now) be considered a mental illness.

    Jesus, give me $5 million dollars, and I'd live on it for the rest of my life. That'd be 0.01% of his net worth, and I'd be so happy. I think many of us would.

    But these people just want more. And more. They're psychotic, and they're ruining the world. We all know what they deserve.

  • source
  • parent
  • hideshow 8 child comments
  • [–] 4 points 5 months ago (2 children)

    I honestly think the greed will (and should now) be considered a mental illness.

    At that point it would just be excusing bad behaviour, would it not? I wouldn't say putting it on the same level as mental illness is fair, either. People with mental illnesses don't want to be mentally ill. On the other hand, greed is a trait that directly shows itself in a person's decisions. You can choose to be greedy, you can't choose to have, say, arachnophobia.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 3 points 5 months ago (1 child)

    Sociopathy (antisocial personality diaorder) is already a diagnosed illness with it's main criteria being lack of empathy

    Disregard for and violation of others rights since age 15, as indicated by one of the seven sub features:

    Failure to obey laws and norms by engaging in behavior which results in criminal arrest, or would warrant criminal arrest
    Lying, deception, and manipulation, for profit or self-amusement,
    Impulsive behavior
    Irritability and aggression, manifested as frequently assaults others, or engages in fighting
    Blatantly disregards safety of self and others,
    A pattern of irresponsibility and Lack of remorse for actions (American Psychiatric Association, 2013)

  • source
  • parent
  • hideshow 1 child comment
  • [–] 5 points 5 months ago

    That’s not greed though. I’ll say it again but more explicitly: Greed is not an illness, it’s a kind of behaviour that anyone can exhibit through their actions. It doesn’t matter whether they’re an average person or someone with an anti-social disorder, that applies regardless.

  • source
  • parent
  • [–] 27 points 5 months ago

    Another banger from 404 media. This made my day.

  • source
  • [–] 24 points 5 months ago (1 child)

    So their chatbot is able to change the email address used to recover an account? I guess, they vibe coded that system.

  • source
  • hideshow 1 child comment
  • [–] 6 points 5 months ago

    Probably not, that implies more competency than can really believe involved here.
    This is more likely something that an entire team had to force into the code over a holiday weekend because some VP got so fucking wasted that he "forgot" his password (strangely for once, he actually had the right password... But the problem is that he was trying to log into a charcuterie board).

  • source
  • parent
  • [–] 20 points 5 months ago

    Vibe hacking is the most ethical use of AI honestly

  • source
  • [–] 18 points 5 months ago (2 children)

    Is it even "hacking" when you're just asking the computer politely?

  • source
  • hideshow 2 child comments
  • [–] 13 points 5 months ago*

    I read this 2 days ago. Steps to reproduce are here.
    https://www.0xsid.com/blog/meta-account-takeover-fiasco

  • source
  • load more comments
    view more: next ›
    33 comments in c/nottheonion@lemmy.world view that post
    6 comments in c/cybersecurity@infosec.pub view that post
    2 comments in c/cybersecurity@infosec.pub view that post
    1 comment in c/futurology@futurology.today view that post
    1 comment in c/DeMeta@lemmy.world view that post