As a skeptic of the entire story, I don’t think Mary ever claimed she was pregnant by God. Maybe she had an affair and everybody knew and Joseph was just a good dude that wanted to raise this kid right.
Joseph is the real hero of this story.
Edit: not only did he stick around. He even taught him the family triad (carpentry). Not well, of course, since Jesus went into philosophy instead.