those states are represented by binary numbers.

The states represent binary numbers, not the other way around.

https://www.britannica.com/technology/bit-communications

A bit is a binary digit. That's what "bit" is an abbreviation for. That is, it's either a 1 or a 0. It's a logical thing, not a physical thing. It's a unit of information.

The embodiment of that bit is the physical state of a certain tiny, addressable chunk of silicon. And there could be any of several other embodiments: the position of toggle switches, chalk marks on a board, pits on a metallic surface in a DVD, voltage in a wire at a particular time. The particular embodiment is an engineering choice that is distinct from the information itself.