More broadly it's a term used for to denote something fitting a very specific purpose or suitable to a small subset of people, animals or things.

https://www.merriam-webster.com/simple/niche

GOG is not a system meant specifically for people looking for some specific thing, it's a general purpose market with a wide variety of offerings that happens to have a smaller user base.

Their main customers might be called a niche market in that they're largely a specific group looking for DRM free games, but the platform isn't limited to them.

Anyhow, enough pedantics for one morning...