The fed "isn't a government agency" in the same way that the post office and CIA aren't.

It was created by an act of Congress and managed through appointments from federally elected offices.

The exact same process that names post offices could unilaterally undo the fed in whole or in part and no one has any direct recourse or appeal. Even a desperate appeal to SCOTUS would fail if the law was written clearly enough, since the Constitution specifically grants congress itself the sole power to print currency and regulate commerce.

(Yes, it's creation was a good thing. Yes, I know how it's funded. No, neither one changes it being a government agency, no matter what your sovereign citizen website says.)