Being fully honest, with that motherboard it will likely be better to attempt a new, fresh build.
Maybe something like this, reusing your RAM, using the stock cooler.
I know you want to upgrade your current parts, but there isn't much that hadn't already been said (motherboard layout, case size, power supply). Bumping up RAM to 16GB or even 32. Maybe looking at SFF GPUs like this.