PyTorch in particular is literally just a dispatching front end for CUDA anyway. It's actually an incredibly powerful tool for GPGPU development, and you will struggle to make faster GPU DSP by hand.
People making these arguments are just incredibly out of date. You can basically use python to trace super optimized GPU compute kernels and then drop them into whatever language you want via torchscript, onnx, trt, whatever.