Slavery was such an ingrained institution that there was for sure some disruption brought by ending it - even some disruption for the enslaved.
People can mine this for evidence that the Union harmed people by ending slavery, but that’s a bad faith outlet for slavery apologists, and nothing more.
Good change can be disruptive. I guess we can always count on Conservatives to place “no change” above all other priorities though. It’s kind of right there in their name.