There exists online and offline ML.

Transcription is not a task that needs a full blown LLM, summarization probably does, and if being done offline or on a server with trusted verifiable lack of logging should be fine. That does not mean send it to ChatGPT API.

I would hope they store the raw conversation file encrypted with patient records so they can retranscribe later as tools improve.

It needs to be a system and a contractual framework that complies with the medical privacy regulations of the place where it's being used (in the US, HIPAA). I have some doubts about whether specific companies or people will actually comply, or the enforcement of those laws by certain governmental authorities, but I don't think it's conceptually impossible to have this kind of tech be compliant with those types of regulations.

Medical AI has guardrails out the wazoo. By the time it’s made available to a physician, it’s probably been reviewed by a council, vetted by IT, and negotiated heavily by Legal. If the data leaves the premises, it’ll be thoroughly protected and isolated.

That said, fuck AI. I’d give up medical transcription to get the environment, economy, and sanity back. Musk’s data center is running dozens of natural gas turbines in a low-income, primarily Black neighborhood 11 miles down the road from St. Jude Children’s Research Hospital. It’s positively ghoulish.