Rare earths are not actually that rare, they’re just… messy to extract. They require a ton of processing, infrastructure and create a lot of byproducts that cannot just be dumped, the byproducts need to be neutralized and disposed of properly. Very expensive, not very large margins.
Rare earths are important to making certain things, but, oil’s products are key for running things. You can go a few years without replacing a phone, you don’t get oil for a week and your transportation infrastructure ceases running.