I remember reading something just the other day that said we form memories in the womb... I'll be back imma find a source.
Edit: https://www.scientificamerican.com/article/recall-in-utero/
https://neurosciencenews.com/fetus-memory-learning-2211/
My guess is that we form memories from around 30 weeks but I don't think we actually recall that information. Perhaps like how you learned how to walk but you don't actually remember how you learned. But you obviously remembered how to do it since, ya know, you can walk. Plus we don't really understand that other people have their own point of view until around 3 so that might have something to do with your initial point. Someone else on this post commented a great video by VSauce about that which I highly recommend.