Only when you have to write out to swap. In the case of something like ZFS, it stores data in RAM, looks for it there, then looks on the disk. So freeing up the RAM is effectively instantaneous; you just mark the space as free, then the other process writes into it.