Or are all of the books objects stored on activitypub and I get the data from the social graph itself?
Not "stored on activitypub", but each book could be represented with RDF (it could be something as sophisticated as using DublinCore or as simple as just using isbns to uniquely identity the books (urn:isbn:1234556789) , and then each activity for "CombatWombatEsq read a book" would be an activity where you are the actor and the book is the object. Then it would be up to the client to expand that information. Your client app could take the ISBN and query wikidata, or Amazon, or nothing at all...