There's only so much voltage that will ever get created in each cell by exposure to light. Only so many electrons that get knocked free. Once that level is hit, nothing else happens that wouldn't also happen to a stone, or your skin. Electrical energy doesn't build up.

Yes, most likely the light is just converted to heat. But that's not special.

Maybe you could think of it that way, but I don't think PV cells store much charge, so they'd make pretty crappy capacitors.

This is a bit complicated, but basically, energy is power×time and power is intensity×tension. Now if you have too much power, because you can't affect the flow of time (sad), you either change tension or intensity. The easiest is to just lower intensity (or cut it out entirely). If you just open your circuit, your intensity falls to 0. So your power falls to 0 and everything is fine. Just like how when you stop a windmill rotating it doesn't produce any power.