Oh absolutely, because it’s the object, not the subject, but you do need the “to” for “say” to work well. We’ve basically only got a case system in English in pronouns and it feels a lot more confusing to people than it is.

Would you say “him?” Say “whom.”*

Would you say “he?” Say “who.”

*major exception: people like to use the objective case with “it is,” but that does not flow well with whom- you might call someone up and say “it’s me,” but “whom is it?” is going to hit a lot more people’s ears wrong. (Neither really makes sense for the objective case, but there are incomprehensible exceptions to normal case rules in other languages, I support English getting weird with it)