One activity occurs in a hazily understood chemical system with near completely unobservable states in interplay with activities and systems which have effects on the main system that are barely recognized, much less fully understood.

Yes, this is more or less the counter argument. If you are familiar with Anil Seth this seems to be his basis as well.

IMO it's basically arguing that we can't know something because it's really complex. Historically those problems do tend to get solved with tech and cleverness.

The other is a constructed, completely defined system with near total observability of states

Yes and no of course. Defined and observable, but profoundly difficult to interpret or understand.

I find it interesting that what known about mechanistic interpretability is partially the result of work done in using LLMs to interpret human brain scans -- scanning LLM layers to match active features.