if you sleep at the same time consistently, changing that is a big change for your body. It takes time to adjust a strong routine, and it takes a physical toll to do so.
We have lots of evidence of it causing all kinds of problems: increased numbers of heart attacks, car accidents, and more. It very much is a detriment to society.