The GTA series might be a good example. All of the protagonists of the games commit exceedingly worse crimes as the game progresses but they're made to be sympathetic since they just want success in a world with not much other opportunities.

Games with karma systems may work as well if the bad option isn't overtly evil. I'm thinking games like Dishonored, Fable 3, Undertale, or any Bethesda game.

Anti-hero protagonists like Kratos from God of War, Arthur Morgan from Red Dead 2, and V from Cyberpunk could also somewhat fit the bill.

Edit: just watched an Outside Xbox video that summarizes a few games with just this premise.

Not even one mention of Prototype. What have the world become.

Disco Elysium's protagonist is a walking disaster and there are a lot of ways you can play him. Honestly playing him as a totally morally upright professional is one of the harder ways to play. You definitely don't feel like a hero while you're playing

Shocked to see no one's suggested Tyranny yet which is an isometric RPG where you specifically play one of the bad guys. Yeah you can make some "good" decisions but ultimately, you're a foot soldier for the bad guys. It's got plenty of that conflict about what's right and wrong that you're looking for too!

No one's said Infamous yet. Shocking. Well Infamous. You get to choose if you're going to be good or bad and it's super satisfying either way. Dishonered has that same sort of choice, also amazing whichever way you go.

Well there's always Grand Theft Auto.

Spec Ops: The Line

Echoing Spec Ops: The Line.

It's no longer available on Steam, but if you can find a way to play it then you should. Probably the most necessary game about war at the moment.

Planescape: Torment
Disco Elysium
Prototype
Crackdown

Already mentioned elsewhere but seconding the recommendations: Infamous I and II Bioshock Infinite

Helldivers games, you're the exploitative, fascist empire who creates its own problems.

Saints Row games. They're like GTA, but increasingly over the top in their parody. You're the boss of a criminal gang. 2 is still "normal parody" and a bit dated, 3 jumped the shark while doing a kickflip with a jet ski, 4 is even more insane.

Sleeping Dogs you play as a cop, but you can betray the law and side more with the criminals.

Warcraft 3 (old but gold) - the human campaign of the base game gets you from a hopeful young paladin prince into a cold, vengeful psychopath; the following undead campaign is said prince (well, king now) finishing the job of killing everyone and further fucking everything. The expansion has 3 extra campaigns, none with "good guys"

Divinity Original Sin (1 and 2) lets you play as big of an asshole as you'd like. Of the Elder Scrolls games, Morrowind is the one that lets you be the biggest bad guy around (you can still finish the game even if you kill every important npc and break every quest)

Dungeon Keeper.

Destroy all humans.

Spec ops the line.

Braid.

Manhunt.

Carrion

Well, I may as well keep plugging my current obsession, Warhammer 40k: Rogue Trader. It's a CRPG somewhere between BG3 and XCOM in gameplay, with a grimdark-flavored splash of Mass Effect.

You have 3 "convictions" that many lot of your decisions fall under. Iconoclast is kind of your standard good guy, but maybe somewhat naive trying to be that good guy in the Warhammer verse. But more relevant to you are Dogmatic and Heretic.

Heretic is pretty much evil as far as I can tell. Chaos worship and slaughter for power. Dogmatic is more like Judge Dredd, maybe? You make some harsh fucking decisions as dogmatic, like liquefying a few thousand people to power a computer you need to use, but you are doing it for what your character truly believes is the greater good of the Imperium.

Pathfinder: Wrath of the Righteous, Owlcat's previous game, has some baddie paths too. You're still fighting demons, but it's more of an evil vs evil in some cases. Especially Lich or Swarm.

Overlord, the first game

KOTOR

Command & Conquer 3

Spec Ops the Line is probably what you want. It's not open world, but the story is excellent.
Bard's Tale is old, but the story is fun and you aren't really a good guy.
Scarface is a GTA reskin that retcons the end of the movie to have Tony Montana live to build a cocaine empire.

Factorio, you literally colonize alien species and pollute their worlds because you feel like it.

Never tried this, but you could play New Vegas with the Legion.

The Witcher is quite famously morally grey, with no "good" choice to many of the major decisions.

Spec Ops: The Line?

Baldur's Gate 3. Not only can you make an evil character, one of the premades is a special evil character called the Dark Urge and basically forces you into doing things that are often even more fucked up than the basic bad guy stuff.

Most cRPGs are like that, actually. Rogue Trader is another good one. Being a paragon of good is nigh impossible in that one, since you're basically a tyrannical land baron in the WH40k universe where everyone is garbage. 😃

Dungeon Keeper 2

Fallout New Vegas

The number of opportunities the game gives you to be an absolute piece of shit is ridiculous.

Here are some examples:

Get the Terrifying Presence perk and literally become evil incarnate: https://www.youtube.com/watch?v=JXBC2OYLM0k

Or just level up speech and be mean in general: https://www.youtube.com/watch?v=STptNbHOXWg

Evil Genius.

Tyranny. I mean, you could end up being the good guy, depending on what that means to you, but you start leading an army to conquer part of the world at the behest of Kyros the Overlord. I think I would classify Kyros as a sort of Lawful Evil type. He seems to want to bring peace and order to the world, but chooses to do it through conquest and harshly enforced laws.

TIE Fighter. Flying as a pilot for the Empire in Star Wars including flying alongside Vader to save the Emperor. Includes getting pulled into a secret society with a cool glowing force tattoo showing your rank in service to the Emperor.

The game never says your name but in the old expanded universe books you are basically the Empire version of Wedge.

Edit - It isn't open world but the limited story does do well at making you feel like you should in the role.

I'll add that the game came with a manual that did have the character name and a back story. Short version is your character was an illegal swoop bike racer on a world that had been in a multi generation war. Includes your father dying. The empire stops it by basically making it impossible to for the two world to even be able to send attacks at each other. Then recruits all the youth that would have fought in their own war and just sends them out to different parts of the empires wars.

Your preexisting flight skills gets you first a mechanic job on a Star Destroyer and then you save a VIP while testing repairs on a TIE Fighter and get pulled in as an actual pilot with a quiet push to get you up the ranks due to your impressive start.

Payday 2 (dunno about 3 since it's been called out for being a phone it home version of 2) but yea... you're %100 not the good guy at all. You're a thief robbing banks and peoples homes.

This game isn't exactly niche, but Red Dead Redemption II is a really good one. You play as an outlaw on the run from bounty hunters while your gang is falling, their ideals are crumbling, and society no longer wants them. Arthur Morgan is part of a dying way of life.

The game also has an honor system where your status is determined by how evil or good you are. Help strangers = high honor. Rob and murder indiscriminately = walking nightmare

Well, let's say that you'll find out your role at the end of bioshock infinite. It depends.

Not open world but man it's so enthralling.

Some of the better RPGs allow you to play as an evil guy (equally fleshed out as the good guy playthrough), e.g. Baldur's Gate 3 (it even has a special evil background storyline called The Dark Urge) or (some of?) the Mass Effect titles.

Cult of the Lamb is also a great game where you play a cult leader. Although the game's design is strong on the entertaining/funny side, it's kind of dark also.

Stellaris is a grand strategy game where you can play anything, from a ruthless ruler trying to destroy the whole galaxy to a pacifist trader.

No recommendations, but just wanted to thank you for a real good question.

Black and White

Any strategy game while you are evil laughing nefariously.