is there a physics limit to the density of energy

the physics limit to the density of energy is literally a black hole. it compresses the maximum amount of mass (energy) into a space. but that's technologically useless since you can't extract the energy out of it on-demand.

The densest ways of storing energy that are technologically useful are:

• batteries (Na-Ion batteries: 0.2 kWh/kg)
• oil/carbon-based fuels (bread: 5 kWh/kg)
• uranium (pure uranium: 24 * 10^6 kWh/kg)

There's also speculative technologies like antimatter (24 * 10^9 kWh/kg) which aren't available today.

Source

[...] why not have a watch sized device that can store insane amounts of power?

Because Hiroshima was leveled by "only" 20 MWh (cost ranges from 120€ in northern Scandinavia to 1010€ in Greece) so having people carry energy wallets with enough to make more around day to day is like paying your groceries bill with C4 (which is perfectly save as long as there is no primary explosive).

Stellaris is a space 4x game that uses energy as a universal currency. The Endless Space games are also 4x games that use ancient nanomachines called Dust as currency.

And yes, concentrating energy increases mass. E=MC^2, which means more Energy must necessarily mean more Mass. So basically gravity will be your hard limit, theoretically stuffing enough energy into small enough a place will create a black hole, though I assume if you're talking electricity then there's probably some physical limit you would hit first.

We have technological limits to the storage efficiencies of different types of batteries. Batteries defined as something that can store useable energy. If we are talking just energy, matter "stores" lots of energy, and you can look at the famous Einstein equation to play with numbers. I do know we have something like a matter density limit in black holes.

Matter, expressed in mass, so kilograms or tons.