Does the taxi driver remember every sign in the city, every road and parking spot? No.
having a 3d map of everything in the city in the memory
you don't go putting the entire world into memory
Cars don't do this either, do they? Surely this type of data is streamed as needed. Just like video games do. This type of optimization has been around for decades... We need not worry about that in cars either.
I'm just saying that GPS and LIDAR is needed in addition to just camera input.