When I discovered the beige rocks on the left side, which had been in the tumbler for about two weeks, weren't going to be interesting, I switched them out for the smokey quartz and a few other things I'm not sure of on the right, which have been in for less than a day. The quartz in the middle were tumbled with the beige rocks. I included those in the new batch, too.
I'm still new to this so my experience is limited.
I did some lapis lazuli with the specific purpose of collecting the sludge to make paint, so I used steel balls instead of tumbling media - it worked but it was clear they would take a very long time. Lapis lazuli is a hard rock.
Kyanite is another I wanted to destroy, a softer, mica-like rock, and it fell apart easily.
I try to tumble similar size and hardness rocks together and just happen to have a lot of quartz right now. Jade and serpentine do well in the tumbler, too.