Would be nice if regular legal and health advice was in any way affordable though
post
Hell yeah, let's hold them accountable for disinformation. They'll be gone completely in a matter of months.
Want to get rid of that responsibility? Direct the user to the source. Oh wait, that's just a search engine.
I think a better solution is to ban techbros from giving serious economic or cultural advice and take computers away from business majors.
Please don't take them entirely away. Maybe just internet access? 30ish years had to do accounting by hand. In those green ledgers. It took approximately twelve times longer to do it by hand than to do it with a computer. And it made me shrimp like 5 times worse. I needed an architect's table what angled the top of it in order to work properly but I could neither get one supplied by the employer nor afford to give one to the employer.
Not all technology is bad
Oddly specific gripe, I'll allow it.
found the business major!
what about a typewriter?
I don’t get how some of these tech company CEOs who came up as engineers can be pushing this bullshit. I get once the company got big they started hiring business bros. But some big companies still have CEOs that were once engineers. You’d think they would know better.
What kind of engineer? Because while the physical world, with all of its mechanical and civil and aerospace engineers, has its shit figured out with professional standards and very clearly defined responsibilities and duties, the world of social engineers, tire engineers, procurement engineers, supply chain engineers, sandwich engineers, project engineers, lead engineers, and yes, software engineers, definitely is a little too loose with any definition for me to care that these ceos were once 'engineers.'
If implemented, that would just ban chatbots that use large language models. It's not a terrible idea.
What would actually happen is that so-called AI chatbot systems would try to detect if someone is from New York and then try to exclude them from receiving medical or legal advice, fail, and then get sued and then pay a small fine, over and over again forever.
This is a really bad idea.
First because healthcare is clearly being gatekept from people.
Second, because even if you go to a healthcare professional nowadays, there is no guarantee that that person is not a fucking idiot that doesn’t believe in vaccines. I can’t believe I have to actually ask people before they touch me if they believe in vaccines or not and then tell them to not come back into my room if they answer that they don’t believe in science. But that has happened and it has happened to the people I’ve taken care of and because of this now healthcare can’t be trusted.
The LLM is not any worse than that. In fact, I would say that it’s already too cautious. No way the model is ever going to tell me vaccines are bad. It’s not going to tell me to take a poison to clear Covid. It’s not going to tell me to drink bleach like the president did. It’s literally not any worse than the bullshit we are dealing with all day every fucking day.
And I’m getting to the point that if you’re a full grown human fucking being and you’re going to believe something if it tells you to drink fucking bleach or swallow a fucking lightbulb then that’s nature saying something about you.
Naw, completely disagree. If you had a calculator you knew was defective you would ban doctors and lawyers from using it.
You also seem to think that LLM is going to be inherently more accurate than a expert human. We can see with GrokAI how easy it is to manipulate an AI into saying racist white nationalist garbage. So we are not just trusting the technology but also a layer of unpredictable corporate meddling.
Why does the LLM recommend this drug but not the other one? We quickly see how a corporation could favor a certain medication due to behind the scene deals or even push a medication.
You can't trust a black box you are not allowed to look into. Trust in a LLM at this point is pure folly.
Some of the SOTA models like gemini 3 pro are getting quite good at ballpark/estimations. I have fed it multiple complex formulas from my studies and some values. The end result is often quite close and similar in accuracy how I would do an estimation myself. (It is usually more accurate then my own ones.)
Now I don't argue there is any consciousness or magic going on. But I think the generalization that is going on is quite something! I have trained ai models for various robot control and computer vision tasks. Compared to older machine learning approaches transformers are very impressive, computationally accessible and easy to use. (In my limited experience)
This reads as a way to protect white collar industries from the effects of AI without addressing the root problem--that AI does not actually think, and that it is little more than a meat grinder full of scraped data.
In other words, Artificial Stupidity. Why is it CALLED intelligent?
it had that name for a really long time
a couple decades ago, a program learning was really impressive
Why is it CALLED intelligent?
Because it is "intelligent" by definition. You're conflating the word with "highly intelligent" or just "smart".
Dogs are "intelligent" but can't they write code, but we sometimes refer to dogs as "smart".
A flatworm has intelligence but no one would call it smart.
It does think, just not very logically.
To put it another way, it's like we figured out how to give machines an intuition via Machine Learning. So you've got a machine with an intuition trained on all written text that is not literal gibberish, but by default all they know how to do is shoot from the hip with their intuition, and the only feedback they get for whether they said the right thing is whether the human they're chatting with approves of what they say.
It's a bullshitter to the extreme because that was how we built the incentive structure. And now they use the bullshitters to train better bullshitters.
Is it any surprise that business executives think that these are the ultimate in intelligence? All they do is bullshit.
I may have become too cynical but, as is often the case when you dig deeper, this sounds like the result of lobbyists trying to protect licensing rather than people.
We can be dumb, but we’ve been doing web searches for legal and medical advice for ages because it is too damned expensive and time consuming to go to professionals for every little thing. Not to mention, doctors have so little time for you that it is hard to get them to listen to the whole story to make connections between symptoms.
The LLMs already tell you that they aren’t licensed professionals and, for many, provide citations for their sources (miles better than your typical health website).
As a personal anecdote, my son was having stomach pain but was planning to tough it out. He checked with ChatGPT and it recommended he go to the ER. He did, and if he hadn’t, he would likely be dead now. He spent 3 days in the hospital having his bowels unobstructed through a tube in his nose.
There is value in people having that kind of information at their fingertips.
Regulation is absolutely needed, but I would rather they focus on protecting us from AI being used for military purposes, mass surveillance, etc. rather than protecting citizens from ourselves.
Are you in the US? My take away here is American healthcare is bad but we're treating the symptom not the disease.
Wait, he thought he could sit that pain through at home? Your son is tough as nails. Give him a hug for me and everyone else who's had that four day n-g tube delight.
Yeah, he is pretty tough. I wish I could hug him, he is about a 10 hour drive from me. That tube was nightmarish from what he’s told me.
if i were his parent, i would be giving him gentle reminders to drink more water. after teasing him for eating way too much corn or broccoli or whatever bastard fiber caused his obstruction (assuming he's in a mental place he can handle the teasing)
He’s in his 20s so he is only slightly more likely to take my advice than he was as a teenager 😆
I'm a human being and I'm pretty sure I am already not allowed to give legal or medical advice to anybody in new york or any other state.
AI in the legal field could be useful for assisting an actual legal professional in compiling precedent based against on-the-books laws, so long as it cites sources and they verify them.
In the medical field, it could be useful for spotting anomalies between multiple images such as X-rays or cross-referencing medical documents WHEN USED BY A PROFESSIONAL.
But the thing is, it should be a tool - carefully used - to enhance the existing profession, not replace actual professionals.
But the thing is, it should be a tool - carefully used - to enhance the existing profession, not replace actual professionals.
except in practice, the "professionals" just take the LLM's word as unassailable and disengage their brains. funny that, the gap between theory and reality
Yup, but those are the cases that make the news. There's always gonna be some stupid/lazy ones
- Make laws against chatbots.
- Demand proof you are not a chatbot.
- Surveillance capitalism.
The real target here is population control.
The lawmakers, which take billionaire money by the ton, who HAVE NEVER given a shit, suddenly, NOW, they want to protect the vulnerable. Abso fucking lutely laughable on its face.
Just have them add a disclaimer or have the hosts be liable for what their chatbots say, stop adding bureaucracy just asking to get selective prosecuted and abused.
Sounds like a start. More is needed though.
The bill targets AI chatbots that impersonate licensed professionals — such as doctors and lawyers — and bars them from providing “substantive response, information, or advice” that would violate professional licensing laws or constitute the unauthorized practice of law.
It also mandates that chatbot owners provide “clear, conspicuous, and explicit” notice to users that they are interacting with an AI system, with the notice displayed in the same language as the chatbot and in a readable font size. However, the bill clarifies that this notice for users, which indicates that they are interacting with a non-human system, does not absolve the chatbot owners of liability.
I don't see how you police/enforce this. The technology is out of the bag, people will find ways to access. Do we need age/location verification for this now too? What if I'm running a local agent? I don't agree with this.
The law would allow you to sue whoever is running the chatbot. If you run your own LLM locally and take bad advice from it, then it's your own fault.
So who gets sued. The guy who put the chat bot on the server and is running it or the chatbot software developer themselves?
Or both?
Walk me through how a company based and operating not in new york would be subject to any actions from this lawsuit.
I do agree it's limited to a small scope of New York-based smaller LLMs, but if you read the news you know why exactly this bill occured - just now Mamdani gave up on a useless chatbot made with local budget by his predecessor Adams: https://www.thecity.nyc/2026/01/30/mamdani-unusable-ai-chatbot-budget/ It was indeed giving inaccurate legal recomendations on city's website. I think the better result that can happen to that bill is it becoming a trend across cities and states as, I suspect, New York administration wasn't the only one falling for this scam.
Chat bots should never give medical advice. Chat bots dispense basic, standalone factoids, like "aspirin is a pain reliever." But they don't know or care about dosages, comorbid conditions or whether or not you live or die, so they won't ask follow up questions.
Laws like this are great for these companies. This is how they will justify removing access to useful information and putting it behind paywalls. But oh your need a prescription so now the insurance companies are involved (spoiler: they already are) and so you don't even have access to pay out the nose for medical information.
Then when Google search has been completely replaced with AI, you won't even be able to search for medical information.
Healthcare companies aren't about to provide anything for free.
LLMs and chatbots should not be giving medical advice. You are afraid of the private healthcare system, not the lack of access to the most janky bandaid fix for its failures.
The line between medical advice and personal research is pretty freaking gray, so banning medical advice. Does that also ban talking to llms about anything that is medical adjacent?
Does medical adjacent mean personal disabilities? Drug related interests? Pet health? Stretches? Pain support?
Anything that falls under "Health, Wellness, and Fitness"?
...etc
It's a slippery slope and we don't need to be sliding down it
People are so vicious over this tech they would rather have disabled poor people with cancer suffer and die under inadequate care than do anything about the inadequate care. Ban the tech, but let this all go on.
If you are perfectly able and well, you can ignore all advice that isn't perfect.
The perspective they seem to lack is frightening. The empathy they refuse to engage is massive. This is able-ism.
Tech companies are bad, but use of tech will cure and ease cancer, HIV, and chronic disease. Bring on the downvotes.
“Would rather have disabled people with cancer suffer and die…”
My guy, that’s not a lack of LLM access, it’s a completely fucked US healthcare system that forces people onto the internet because they can’t get what they need from the state, you goofy-ass weirdo.
Well yes of course but also restricting access to information machines doesn't exactly help much either.
Do hallucinating LLMs, that have done such things as convince a child to commit suicide before, really count as “information machines”? The Mayo clinic website might take a single whole other braincell to read through but at least it’ll be written properly.
I mean, the fact that you consider these programs to have enough credibility to be called “information machines” is exactly why they’re so potentially dangerous.
I hate to break it to you but.. they're not really "information machines". Google search is a better information machine.
Neither should Wikipedia or Google. So I guess by your logic nobody should search or learn about medical conditions on a computer.
You know damn well there’s an important difference related to the confidence of a bot that has been a key problem since this whole thing started.
I guess by your logic nobody should search or learn about medical conditions on a computer.
How else would we know the TRUTH about 5G vaccines and invermectin? Or the cures of Apple Cider vinegar?
Most of the medical information coming up these days is garbage and you should be going to a known, reputable site and searching their database. LLMs have been trained on absolute garbage. There is nothing of value being kept from anyone here.
LLMs have been trained on absolute garbage
It depends on the LLM actually.
Specialized medical LLMs are actually very accurate.
That's not why people are using ChatBots, they are using Chatbots because they can't afford healthcare.
and before we get out the tiny violins for MDs, they gatekeep the system to keep their salaries high.
Bad news folks, MDs are using ChapGPT on the sly.
Some horses you can't even lead to water. Let alone make them drink.
You can google your simptoms and there probably are some reliable sites but a hallucinating chatbot is a bad idea. Not to mention some people suggested treating covid with chlorine, vinegar etc....
Mixed feelings about this. Let me play devils advocate and say that many Americans don’t have access to these resources at all. Having potentially inaccurate resources might be better than nothing, or is that worse?
'Should I use one teaspoon of salt in this recipe, or two?'
Two is ideal.
'Do dogs like chicken wings?'
Wild dogs regularly hunt small animals like hare or chicken for food.
One of these answers results in a bad cake, the other results in a hurt dog. Potentially inaccurate answers aren't much of a problem when the stakes are low, but even a simple question about what to feed a pet could end with a negative outcome.
Having potentially inaccurate resources might be better than nothing, or is that worse?
You pick up a mushroom in the forest and take it home. If you have no information, do you eat it? If something tells you it's safe do you eat it?
If you’re going to be your own lawyer or perform a bit of self surgery, there is no way the AI is helping that situation. Especially if the inherent nature of AI is to validate everything you say.
There are billions being sunk into AI. How much health care could that buy? Your logic only makes sense if AI is free. It's not.
the AI devices will just have preambles and disclaimers and word things in ways to refer the user to human resources
We had a medical scare just yesterday. I was in the ER for 8 hours with my partner over a non-life-threatening but still emergency problem.
An ultrasound, cat scan, and much poking and prodding later, we still don't know what is up. The AI was at least able to predict next steps (if A then discharge and follow up with PCP, if B then surgery this week, if C then emergency surgery), something the ER was too busy to do for several hours. It was reassuring. The AI also gave me (working) links to more thorough resources on the topic.
If you don't want legal or medical advice from an AI, you can already simply not ask the AI for legal or medical advice. But I don't want your paternalistic restrictions on what I may ask.
top 88 comments