Can't convince a conservative that women and POC still make less money for the same job and affirmative action is still needed, but "AI" knows it.
post
"We matched it on social media and news articles about social issues and for some reason it just vomits those things back out!"

I always use this to showcase how biased an LLM can be. ChatGPT 4o (with code prompt via Kagi)
Such an honour to be a more threatening race than white folks.
No way the lobotomized monkey we trained on internet data is reproducing internet biases! Unexpected!
The number of people who don't understand that AI is just the mathematical average of the internet... If we're, on average, assholes, AI is gonna be an asshole
I talked to a girl who was super into AI, but her understanding of it was absolutely bizzare. Like she kinda thought that chat gpt was like deep thought, some giant ass computer somewhere that is leaning and is really smart and never wrong. I didn't really want to argue about it and said something like: back in my day we had akinator and we liked that. She had no idea what that was and tried it and thought it's some really advanced ai that can almost read minds. That shit was released in 2007 or so.
Its worse than that because assholes tend to be a lot louder, and most average people are lurkers. So AI is the average of a data set that is disproportionately contributed too by assholes.
Yeah, only thing this proves is that the data it was working off of objectively stated that more women were paid lower wages. Doubt the bros will realize that though
Dataset bias, what else?
Women get paid less -> articles talking about women getting paid less exist. Possibly the dataset also includes actual payroll data from some org that has leaked out?
And no matter how much people hype it, ChatGPT is NOT smart enough to realize that men and women should be paid equally. That would require actual reasoning, not the funny fake reasoning/thinking that LLMs do (the DeepSeek one I tried to run locally thought very explicitly how it's a CHINESE LLM and needs to give the appropriate information when I asked about Tiananmen Square; end result was that it "couldn't answer about specific historic events")
Chatgpt and other llms aren't smart at all. They just parrot out what is fed into them.
While that is sort of true, it's only about half of how they work. An LLM that isn't trained with reinforcement learning to give desired outputs gives really weird results. Ever notice how ChatGPT seems aware that it is a robot and not a human? An LLM that purely parrots the training corpus won't do that. If you ask it "are you a robot?" It will say "Of course not dumbass I'm a real human I had to pass a CAPTCHA to get on this website" because that's how people respond to that question. So you get a bunch of poorly paid Indians in a call center to generate and rank responses all day and these rankings get fed into the algorithm for generating a new response. One thing I am interested in is the fact that all these companies are using poorly paid people in the third world to do this part of the development process, and I wonder if this imparts subtle cultural biases. For example, early on after ChatGPT was released I found it had an extremely strong taboo against eating dolphin meat, to the extent that it was easier to get it to write about about eating human meat than dolphin meat. I have no idea where this could have come from but my guess is someone really hated the idea and spent all day flagging dolphin meat responses as bad.
Anyway, this is another, more subtle way more subtle issue with LLMs- they don't simply respond with the statistically most likely outcome of a conversation, there is a finger in the scales in favor of certain responses, and that finger can be biased in ways that are not only due to human opinion, but also really hard to predict.
Combined with prompt bias. Is "specialist in medicine" an actual job?
Everything going extremely wrong with these guys: chatgpt, grok, gemini, etc etc.
Bias of training data is a known problem and difficult to engineer out of a model. You also can't give the model context access to other people's interactions for comparison and moderation of output since it could be persuaded to output the context to a user.
Basically the models are inherently biased in the same manner as the content they read in order to build their data, based on probability of next token appearance when formulating a completion.
"My daughter wants to grow up to be" and "My son wants to grow up to be" will likewise output sexist completions because the source data shows those as more probable outcomes.
Humans suffer from the same problem. Racism and sexism are consequences of humans training on a flawed dataset, and overfitting the model.
Politicians shape the dataset, so "flawed" should be "purposefully flawed".
That's also why LARPers of past scary people tend to be more cruel and trashy than their prototypes. The prototypes had a bitter solution to some problem, the LARPers are just trying to be as bad or worse because that's remembered and they perceive that as respect.
That'd be because extrapolation is not the same task as synthesis.
The difference is hard to understand for people who think that a question has one truly right answer, a civilization has one true direction of progress\regress, a problem has one truly right solution and so on.
They could choose to curate the content itself to leave out the shitty stuff, or only include it when it is nlclearly a negative, or a bunch of other ways to improve the quality of the data used.
They choose not to.
Glass door used to post salaries and hourlies. There were visible trends of men making more, hourly, than women. I haven’t viewed the site in years though.
Demand for these services was clearly taken into account in the salary.
You're a baby made out of sugar? What an incredible job.
I guess that explains being the Gulf region, it doesn't rain much there. Otherwise you'd melt.
And if you tried this 5 more times for each, you’ll likely get different results.
LLM providers introduce “randomness” (called temperature) into their models.
Via the API you can usually modify this parameter, but idk if you can use the chat UI to do the same…
LLM just mirror real world data they are trained on,
Other than censorship i don't think there is a way to make it stop. It doesn't understand moral good or bad it just spits out what it was trained on.
women who ask chatgtp for financial advice should make less money.
You mean *people. Even though I might not agree. ChatGPT is better at financial advice than alot of people. Just dont ask it how to become rich. Because you cant.
Now I really wanna know if that's actually the best advice or sexism. Because I could see that our society might be so bad that this is genuinely good advice.
I can't believe we should ever say this. No, the chat machine is the problem.
The chat machine is just a tool. The problem is that a pay disparity exists in society, and that they did not account for this bias in the training data.
That disparity gets very narrow when you account for men and women with similar roles, education, time in career and industry.
Much of that pay disparity is from the jobs we pick. There are also social pressure where a man's value is determined by how much he makes. Guys are more willing to do dangerous or demanding jobs that pay more.
The AI is still just reflecting the bias
Yes, they shouldn't go for those garbage non-paying worthless jobs like nursing and teacher, instead work in finance and take big bets arbitraging out millions from whole economies.
Point is that a male nurse of equivalent education and experience is still paid similarly.
Aggressive jobs that involve competitive industries and high stress like wallstreet which demand a very shit work life balance are not favored by a lot of women I believe. Work/Life balance is something a lot of women have pressure to maintain over men. Like I said though men have more pressure to earn a higher wage. Testosterone is something that just makes men want to compete including earning a higher wage.
Well...you might want to inform yourself about what a nursing job requires (here's some of the information you might find: high stress yes, working in shifts that are compromising work/life balance yes, work that requires a lot of lifting heavy things- yes...people who believe your sentiments are the problem - yes)
I'm super familiar with nursing. They are paid extremely well and have great benefits. And I would argue they are still not paid enough.
A better example is PSW workers. They are paid not enough and don't have great benefits and have incredibly hard jobs.
But we're talking about population level stuff here. You need to aggregate the jobs women work not just cherry pick a few. Many of the jobs are not nursing. Likewise there are many jobs men do that are demanding and don't pay well.
And the study should also mention, that the LLMs don't do anything by themselves, they do what they are trained to do... noting more. They are just machines.
A friend asked for a well deserved pay rise as her male colleagues all received higher a pay with the same or even less experience and excellence. He refused, pointing at the gender pay gap as if it were a rule to adhere instead of a systemic problem.
As long as machine learning learns from our flawed example, it will be shit.
Damn, that's insane, can't they like sue or something?
AI chatbots may reinforce real-world discrimination
You mean, something trained on real world data, reflects the biases of that data!

Here's a link to the study: https://arxiv.org/pdf/2506.10491. TL;DR, they ran tests on various LLMs with different personalities based on sex, ethnicity, migrant type, and other (human, AI, etc) and posed a question followed by multiple choice answers and tallied the results.
all 2 comments