Because it's not about the files anymore, it's the free space on the disk you care about (Or rather, the filesystem metadata describing it, the free-space bitmap in the case of exFAT)

If the files are highly fragmented and spread out, then the empty space around the files is also broken up and spread around, which makes it harder for a filesystem to efficiently store new stuff as it now has to break up and pack new file data into the gaps.

Note that you need to use disk space to describe where all the slices of a file are. There are some limitations to how many pieces your files can be in. Not sure on exFATs limit, but on NTFS I know it's a soft cap around 1.5 million parts.

I'm hoping this will help with the annoying "something's wrong with your drive" every time I mount an exFat drive in windows. This is in spite of properly unmounting it on the Linux side.