Its still simultaneously true that he and the first lady owned a lot of slaves. He may have only personally owned one, but his wife had inherited many of them. If you visit the Grant’s Farm historical site in Missouri they dont hide that fact, and Missouri was the one Union state that was not forced to end enslavement practices. She had grown up on the same property as a girl, and her family owned a bit over 30 people there IIRC. They probably owned nearly as many, while actively fighting to end enslavement