What are some general recommendations to maintain a set of patches on top of a library distributed as a source tarball? Until now I've been adding the versions to a git repository by deleting previous files and adding the new files in a "upstream" branch, then merging that branch into the main branch which contains my patches. This turned into more work than I expected because they started moving files around and renaming in new releases.

I should probably be rebasing instead of merging, but are there any other recommendations for this type of situation?