A big argument for "not all matter must necessarily interact electromagnetically" is that we know of particles which don't interact with the strong force - why should that fundamental force be special?