I've always been like this as well. I forget everything unless I think really hard about it and trace the events step by step.

There's also the autism of not wanting to be observed, so if I am to remember I have to force myself to "observe" myself through asking my own mind what I did. Which might be the reason for the "forgetting" as a way of unconscious protection.

Some coworker: "What do you do for fun?"

Me: "I don't even remember driving to work"

until the notepad joins the others in the void