I take it you don't filesystem much do you?
Moving a file is literally just changing a pointer in the filesystem table to basically say 'hey, these file contents are now stored in this other folder'. Moving a file on the same drive literally never requires rewriting the actual file contents.
I'm practically certain that what's slowing Windows down when sending something to the Recycle Bin is the background processing and data compression being performed by System Restore.