Not a full solution, but painters tape to seal the gaps might stop further nest building

Turn off the breaker that feeds that light. If you don't see any activity, get up on a ladder and listen. If you don't hear insect noise knock on the ceiling near the light and listen again. If clear, look for a screw that is holding up the conical shaped cover, that should be a decorative piece that can be slid down the pole. You should then be able to see what you are dealing with.

You can use wasp spray up into the gangbox if you want. The suggestion to use painters tape to cover hole under the decorative cover is a good one. Just be sure that you allow the wasp spray to completely dry before turning on the breaker again.

I have approached this kind of situation similarly.

Good luck!

Hard to tell from the picture but it may be a mud dauber, which is not a very aggressive wasp (to humans) and is useful for controlling spider populations. if they're getting up into your ceiling near the light fixture, blocking it off may prevent it from trying to nest there. if it's building on the fixture itself, knocking down the nest might convince it to relocate to a safer area. I've never had a mud dauber sting me even while knocking down nests, but I would still wait until it's off gathering mud to make an attempt on the nest.

https://en.wikipedia.org/wiki/Mud_dauber

Burn down the house and rebuild. It's the only way to be sure.

When it starts getting cool out, sometimes one will come inside and find a warm place to hole up for the winter.

Try buying them off with sugar water.

Catch it in a jar and release it outside?