A big piece is licensing. When you're throwing hundreds or thousands of processor cores into a data center, somewhere a Microsoft VAR is just drooling to sell you datacenter license packs that you'll need to renew/repurchase for every major OS upgrade. Ah, and you'll need device/user cals. Oh you want to manage it too? Oof.

Microsoft tried to make Windows Server popular. Apple sold a server OS and even its own rack-mount servers for a while.

The people using servers, and often the people making the decisions about what to use have a high degree of technical knowledge and skill. The things that drive popularity in consumer operating systems such as being preloaded on devices and having a polished GUI don't have as big an influence on experts.

Customizability, reliability, and performance do have a big influence on what experts choose, and Linux wins on those points. There's also the history of proprietary Unix being big in the server/supercomputer market, and Linux is an obvious successor.

Most servers do not need a fancy user interface. They don't have monitors attached to them. Most of the other OSes you listed come bundled with a ton of user experience software along with the expectation a human will directly interact with them.

Servers are typically deployed and managed with automation, by the thousands. Lean. Simple. Secure. Tuned for their specific purpose. This is a lot easier to do with Linux than an all purpose general use personal computing platform.

Microsoft comes out with Windows Supercomputer Pro.

They sell 6 copies.

When you're running exotic hardware, everything is custom. Linux is the most easily customizable.

Linux offers near-endless customisation and Kernel is also open sourced for any kind of (performance) tweaks.

Moreover, Linux is, by design, better suited to be a server OS than desktop OS.

These are the same reasons why most of the web servers across world runs on Linux based distros.

Most servers around the world run Linux.

True.

Why can’t these big tech companies manage to sell their own software to server operators

Lots of companies large and small are running commercial distributions of Linux including with paid licensing for products Like RHEL (Red Hat Enterprise Linux) or SUSE. Millions of other servers are on co-branded versions of Linux that are provided for free to the customer as long as the customer continues to use the company's service. Examples here are Amazon Linux or Oracle Linux (both of which you're only allowed to run if they're operating on Amazon or Oracle Cloud servers. Now, these same companies likely also use unlicensed free Linux in places, either disposable clusters or labs, but if an application has commercial uptime requirements (meaning downtime costs money), few companies run free Linux in those specific applications.

or supercomputers?

This is a frighteningly small install base to try to sell a commercial operating system on. How many supercomputers are there in the world? Perhaps 1000? Moreover, these are such specialized set ups that trying to make a one-size-fits-all OS is likely impossible.

Why is an open, free project that is free for users so superior here?

Just because there is free Linux does not mean that all Linux servers in the world are free.

Amazon's and Google's OSes already are Linux variants. The thing that makes Android is the GUI userland, not the underlying system which is just Linux with a libc implementation not by GNU.

Because they have to work

Linux requires a lot more knowledge to get functioning properly, but gives more freedom when you do.

Good thing IT experts aren't regular users and can handle that just fine. Supercomputers datacenters also have their own support on hand. They don't need to outsource it like they do in regular businesses.

That and the licenses.

Linux has an extremely flexible architecture. Before Linux, most servers ran on UNIX, and before that, well, networking was in a very early and rudimentary stage.

When UNIX licensing shenanigans kept happening, Linux was a more and more attractive option as it matured.

Today, Linux is an incredibly flexible, reliable and performant OS. It's free, in most cases. Why would anyone use anything else? HPC software all runs on Linux and UNIX. You can run it on a tiny little SBC like a raspberry pi, you can run it in an embedded system like car infotainment or a smart meter, and you can run it on ultra high-performance supercomputer clusters. It doesn't give a damn; it just works.

Why would we use anything else? Apple's ecosystem, while great, makes no sense in the server world. They have their own unique directory service that nobody wants to support (unless they're trying to sell something to Apple themselves), they have total control over the OS and its capabilities, and it's technically illegal to modify. Windows has a heavy GUI, and its command-line interface is middling at best and difficult to learn. Windows excels in backwards compatibility and ease of deployment, which makes it ideal for small and medium businesses, but it quickly becomes irrelevant once you scale to a certain point. This is why they've got their Azure AD product, for example. It's attempting to fix the scalability issues with Windows Server. Having spoken to some of the developers of Windows Server, it's also plain as day that Microsoft is only really maintaining Windows Server to collect on their existing contracts. They have no desire to grow that part of their business.

With all of this in mind, Linux the most obvious choice. It takes no time at all to slap a copy of Ubuntu Server on a pizza box and have a functioning server up in an hour. Everything else is more complex, slower, and costs money.

You answered your own question:

That’s astonishing in a capitalist world where absolutely everything is commodified.

If you bought windows for the hundred or so servers you manage how much is that going to cost you up front and then again every time Microsoft decides to just end support and roll out a new OS? How much work does it take to manage those licenses and make those updates? And what are you getting in return for that additional expense?

Amazon Linux is based on Red Hat... i.e. Linux...

It was mentioned in the threads, but I don't think it got enough attention. Mostly it comes down to money. Yes, customization, efficiency, etc of Linux, but also because every $ not spent on licensing is a $ you can spend.on making the data center better. So maybe it's buying.more hardware, or having more money for infrastructure like electricity , cooling, and bandwidth. Or just plain profit. Licensing is a nightmare with microsoft. Rather save the money and time and.make a better data center.

It just works better.

Because being beholden to a single mega corp is a massive business risk. If MS decides to yank support or they go under then you suddenly need to migrate to a new OS for your entire server farm with all the problems that come with it, like having to port all your custom software to the new OS. Also if the US decides to sanction the country your business resides in than you are also fucked if you run your entire IT infrastructure on Windows. Linux being opensource mitigates all those problems. Like if for some reason you can't use the distro you use you can always migrate to a fork.

Because those engineers were free to create the value that they needed and only the value that they needed. Windows Server and OSX Server were/are not unfettered. They, therefore could not offer a better value.

https://www.youtube.com/watch?v=rV0a-b_VhBg

Google and Amazon are competing with their own Linux OSes. Even IBM bought 30% of RedHat almost 30 years ago. Windows is developing their own Linux OS now too, Azure Linux. Windows Server is down to 40% in their cloud Azure environment. I'm just guessing that's because many long term contracts are ending and the companies associated have been migrating away from Windows Server. Hence the need for Azure Linux. OSX server flopped big time twenty years ago. Apple had to shutdown their entire XServe division. You don't always have to sell the software or OS to make money off of it. Especially when there is heavy competition. It's like restaurants in the US giving away free tap water when you sit down to eat. There are a lot of ways to compete for dollars in a capitalist world.

/u/Zak did a pretty good job summing it up.

These servers are hosting custom software. The devs can develop for any hardware and OS combination. So the choice is largely performance, features, and price. Free is the best price in a capitalist world. Free isn't the only price though, companies are just fine spending money if they are getting a better value. They just aren't with Windows Server and didn't with OSX server, they don't offer a better value. They aren't more performant and they don't offer any features that make it worth the money or risking vendor lock-in. With Linux, if the value you need isn't there, anyone is free to create the value that is needed, with zero limitations. And they only need create the value they need.

It works well, has little to no vendor lock in, and has reasonable or no licensing costs.

Also drivers can be developed without needing anyones permission or validation like with Windows drivers.

Linux is maintained by large companies

Because you want something you can control.

It's no good if you wake up one morning and find "Let's finish setting up your PC :)" on 10,000 machines that have been working for ten years.

Windows sucks. Linux has so many more features than you could ever think. Ulimit is a major one. B.P.F. is another.

Amazon has their own Linux dist that is relatively popular among their customers

https://aws.amazon.com/amazon-linux-2/

As for your question, consider thus: I think many users here have the resources to make their own personal Linux distributions without drawing from an existing base distro. Yet very few do. Why do you think that is? There's a lot of overlap in the answers.

Us oldsters remember the UNIX wars. There were a number of different flavours of UNIX fighting for commercial domination. In the end they all failed because of fragmentation. This is why Linux won.