Ah, see what you mean. Yeah, the term is never used like a nick or pet name, it's a state, like Dead or Pregnant. "He's my baby daddy" isn't a term of endearment, it's literally "To [this] baby, [he] is daddy"