This won't bring down CBS because its profitable for CBS. Colbert's time slot has essentially become an infomercial.
The show PAYS for that time, they are essentially renting it. They keep whatever ad revenue they can collect. Theoretically, the show collects more in advertising revenue that they pay CBS for the time, and they make a profit.
But that's the production company's problem. CBS makes money no matter what. The show may lose money, but CBS still gets their rent money. The fact that the show is down almost 90% viewership is embarrassing for CBS, but it isn't costing them anything.