Your baseball card became worth more because more of them got lost or destroyed over time so it got more rare. So the increase in value came from all the people that lost their 5 dollar card. And also partially from inflation.

Completely different from the stock market.

The money from the stock market comes primarily from other investors. You may not be directly taking it from them, but there are only a few layers of abstraction between them putting money into a stock that went down and you taking money out of a stock that went up.

But the main advantage rich people have in the stock market is less latency in their trades and more weight to swing around. Not to mention being much more likely to have insider info, but even without actually cheating they have quite a few advantages that tip it in their favour.