Even on "safe" planets, the average person is basically a slave doing endless, arduous labor with no concern for their lives or well-being, all under the merciless thumb of Imperial dogma. It's just a bad place to end up.