To be fair, this happens to me also in Debian Gnome, but very very rarely. On Windows, it keeps me from ejecting the drive until after I try it 4-5 times. Every time. In any case, lsof can help identify the culprit.

lsof | grep DRIVENAME should give which program is keeping it.