You should ensure data integrity with checksums and/or hashes.
Yes. Ext4 is far better (including efficiency) than FAT. A very modern FS versus one of the oldest and terribly inefficient... Lol
FAT typically has large cluster sizes, especially on large volumes. Ext4 has relatively small clusters. A tremendous amount of small files could possibly explain it. Did you happen to enable any compression for the volume?