In the most practical cases, yes. But in theory, there is nothing about the protocol that says that message addressing implies message visibility, or even access control.

Also, be careful of taking your assumptions and treating them as universal truths. One day somebody could build an IRC-like system on ActivityPub and decides to treat a "ChatMessage" object as public objects which may or may not be addressed at a single participant. There would be no "bug" if the server picks up the object, relays to others, or even indexes it and makes it searchable.