The FSF's position is roughly that software (instructions that run on programmable computers) ought to be free as a matter of ethics, but that hardware is different, basically because of the difficulty and resources necessary to "copy" hardware. See Free Hardware and Free Hardware Designs for Stallman's thoughts. Basically, he (and the FSF) don't consider "free hardware" to be a coherent concept- just "free hardware designs". So using FSF terminology, it doesn't make sense to speak of free or open-source hardware.
The FSF basically draws the line at easy programmability. If a device was designed to be reprogrammed, what it's running is software, which should be free, otherwise you're ceding control over that computer to the copyright holders of the software it runs- and that doesn't just include your host CPU, it includes every computer you own, including other ones on your motherboard. If a device was not meant to be easily reprogrammed, then the "software" it runs is really just a permanent part of the design, and the FSF doesn't really care about this type of firmware being "free". Devices with firmware in Linux are clearly able to be easily reprogrammed- you just change the firmware code- so firmware that runs on these devices meets the FSF's definition of software.