Analog audio not being sampled doesn’t really matter. It’s like film, it can’t have infinite “resolution”. It’s the size of the granules on the tape and the speed the tape is moving that determines how good audio can sound. Grain size is kind of equivalent to floating point resolution, and tape speed is kind of equivalent to sampling rate. In order to get as true-to-life audio reproduction as 32-bit 96KHz PCM, you’d need absolutely wildly expensive tape and equipment. I’m not even sure if it’s physically possible.

When you say by definition it includes “more data”, you have to think about what that data is. There’s signal, the stuff you want to record, and there’s noise, the stuff that gets on there that you didn’t want. The higher precision a digital recording is, the higher the signal-to-noise ratio. Unlike analog tape, there’s not really a theoretical upper limit (just the limits of your recording hardware). If you record with a high enough precision, you can record incredibly quiet or incredibly loud sounds, way out of the range of the best audio tape. Same with frequencies. The faster your sampling rate, the higher the frequencies you can record. And unlike tape, it’s not going to shred itself to pieces if you go really really high.

Things sound “better” when you introduce noise because people like analog recordings. Not actual analog recordings, mind you, just the appearance of analog recordings. It has nothing to do with audio quality, it’s just vibes. It gives good vibes.