Yeah, but by replacing the effort through which knowledge and experience are learned, the step of understanding what the app is doing won't come.

I'd argue there is a large middle ground of people who are semi-experienced devs, but are using it to work with platforms or languages they have no experience in. but again, that leads to similar issues as the complete outsider who doesnt know the risks or how things are supposed to work or how to do some basic safety auditing. if it were wrapped in a more interactive educational system that explains how and why everything works the way it does so the user can actually get better at the task themselves maybe it would be better, but for now, it's a potentially dangerous crutch with very few built in safeguards.

When I played with googles antigravity recently as a test and told it to develop an app for a task I described as a test (obscure, obsolete audio DSP hardware control system, very little public conversation on forums or socials, next to nothing on github etc, just raw api documents, hardware descriptions, service manuals and a command to iterate and test until it's perfect) it saved my github username, password AND auth keys in plaintext in the working folder, which it uploaded to github too.. thanks that's a great idea google, well done. the end result actually worked, but the code was a bit of a mess as expected.

Thankfully this was all private since I was going out of my way to try to break it and I was able to simply exclude that file, but I wonder how many people dont know enough about any of this to know how to do it safely. and even then, is the ai having your details once, and not saving them to a file, somehow also risking leaking the info? that's what we dont really know.

Each conversation/project should be separate instances, but then it will seem to know details about an unrelated separate conversation you had in a separate instance so I dont know what to believe.

If someone is going to work with AI you need to treat EVERY interaction as a potential leak and avoid any direct communication of credentials. That system also had a tendency to break out of its working directory unless you specifically block it from doing so (you'd think the default behaviour wouldn't allow that), it started looking around the root directory of the disk it was working in when looking for an API document it was supposed to work with.