Look up a cleaning solution called 'RetroBright'. It's designed to remove the yellowing from the ABS housings of old electronics. I'm pretty sure the recipe for it is available free online, or you can order pre-mixed bottles of it. You have to be a little careful with it because it's mostly hydrogen peroxide, but I hear it works great.

You are correct, it's the plastic. Or rather, the fire retardant mixed into the plastic.