I'm working on this exact problem at home now, and finding 24GB of ddr4, a 6th-gen i7, and an old 8GB rx580 is more than enough to run pretty capable models up to ~35B parameters just fine, and equivalent MoE models shockingly fast. Not exactly cheap, with the price of everything so crazy right now, but certainly cheaper altogether than I'd imagined.