Generally I've found that:

If the facts are painfully obvious from a simple web search, then GenAI has a decent chance of getting it right. This can be useful if you can't recall any "key" words well and the GenAI can craft several searches and get there.

However, if it does mess up the facts, the result looks superficially the same as "correct". So while it can give accurate data, you always have to double check. This can still be useful, as finding the right search terms can be a decent help.

In coding, sometimes in some situations, you can have requirements that are absolutely testable, and thus you can have the models retry and retry until it works. This isn't always feasible. Even when it seems feasible, you may screw up the criteria, or the GenAI when enough freedom disables a probablematic test rather than solve it, and it likely will generate code that's not really fit to modify. There are a lot of situations where this is useful, but it is infuriating that non technical people and even some low skill technical people assume this is always the case.

Then when you get away from facts mattering, it gets "better". Example, someone jokingly asked for one to "make gta6". After a while it came back with a GTA 1 clone. Lots of people were impressed, because whatever it did could be considered a success even as it obviously didn't match the expectation. The operators also like to GenAI some webcomic, where it is a fiction. They almost always didn't have any interesting thought going in so they tend to be crap, but "correctness" didn't matter.

Of course, also making fakes. Supreme case where looking correct matters but being factually correct does not matter at all. GenAI above all else "seems" correct.

I think a lot of people just have low standards.

I've been able to find a workflow where it's mostly correct and can handle most of my gamedev related coding without making too many mistakes. I still have to actually read through the code and pay some attention to what it's doing, and if it misses something and goes on a wild goose chase, it's unusable and I have to start over (so someone who didn't know what they are doing would be cooked), but whatever.

Sure, it does require a lot of looping adversarial reviews, and my average token cost is around 3000$ (4B tokens) a month (we have unlimited budgets and a pretty accurate tracking, and also probably cheaper than consumer prices per token with how large company it is), which is actually more than my monthly salary, but it's just a job, for a company and on a product I don't really care about, and I can 1) keep slacking in my job while doing my own coding stuff and projects, and keep seeing how absolutely unreasonable the prices are if you want to get at least semi-submitable results.

You could ask if it's necessary to spend so much tokens - but so far most adversary review rounds do find serious blocker bugs the first implementation had, most of them being some hidden stuff that would be difficult to reasonably find by hand unless you really understand everything around the code, which you should, but the point of AI is that you shouldn't have to.

Is it worth it? Lol, no. The whole team is loosing codebase knowledge, we're getting bottle-necked by pending PR reviews that are just stacking up and no one wants to do, so we're not even more effective, the cost is absolutely absurd and in no way near sustainable or worth it. And the longer it goes on, the less I can realize it's spewing bullshit and I should stop it and nudge it into a different direction in our codebase, because I'm slowly loosing touch with it, while for now I'm still relying on what I remember from before we went all-in on AI.

And that's while the whole industry is in the "Uber pricing" phase, so it will get a lot worse. But yeah, if you can burn 100-200$ per a simple implementation task, then it can have a pretty usable results. And that 3000$ a month does not include our CI review bot, that does additional rounds of multi-agent council reviews. And I'm not even working on anything complex, mostly just various menu screens UI. I can imagine the bill getting a lot higher if you get into more involved systems.

Yeah, every time I say this to someone they say it's fine when they use it because they pay for the latest expensive model. Except they've been saying that for the last couple of years, so I guess they were lying back then...?

from my point of view it feels like most people are willing to trade their cognitive function for being lazy, which is a boundary i never want to cross.

AI will produce a lot of code but most of it is pretty poor quality as most training data is going to be poor quality code, there's just always going to be more bad code than good to begin with just due to how difficult quality code really is to produce. I'll give a hint, good code is usually small and succinct.

since my work started pushing AI live site issues have increased dramatically. turns out the person using AI looks like they have a ton more productivity but in reality that just shifts to whoever is reviewing the code. and to those who will say its the developer's responsibility to review the code, yeah no shit, but if you ever work corporate you realize most don't care as long as their managers think they are productive and just blame others for being bottlenecks.

Overall it just enlightened me to how bad the average developer really is, but i guess obtaining mediocrity is the sacrifice to make in the name of "productivity".

I'm curious what you are using. The free versions of chatgpt have been like that for me, but even Gemini flash with extended thinking, also free for a while longer, is giving me pretty reliable results as long as there training data out there to derive an answer from. The higher (paid) Claude models will one shot most coding tasks.

Give us an example of some of the prompts you're using and what LLMs. I'm curious if it's a use case difference or you're using the dollar store's customer service AI to try to help you with your coding homework.

He doesn't take about what is, he talks about what's coming.

No matter how much I carefully structure a prompt, define specific behaviors in skills, and tweak the agent md files it will still just go do something I don't tell it to or not do something it's got really specific instructions for. We have to write all code by LLM now at work and I'm trying to do my due diligence to review code before putting it up for PR. 9 times out of 10 when I tell it to show me a diff before committing it silently runs git diff in the background and prints "that's the full diff". That's with some basic "here's what I want when I ask for a diff" in the base context.

It's a big club and you ain't in it. They have to sell it until the end and this guy is on the outs and not getting invited to the "good" yacht parties at the moment.

The judgement you are considering from this person is the same judgement (and morality) that thought it was fine to be a close and frequent associate of jeffrey epstein.

Bill gates is a desperate, broken man. He is the illusion of a man you once admired vaguely and that is the only remaining currency he holds that matters to him. The best thing about bill is the wife who left him.

Where I am forced to encounter advanced LLMs is every company rolled out replacements for my "dumb" assistants. Like Alexa on an echo dot or Gemini on Android auto.

What I ask them to do does not take significant processing power. I need you to set a 5min timer. Not talk to me like you're alive. Don't overthink things, just do simple things.

All that extra banter they added in eats up processing power. So now it's slower and works like shit because it's formulating some complex response to my request to beep in 5 minutes. Just set the timer, my hands are covered in raw chicken juice.