That's actually not true. Even steel girders will dissolve to rust within a few hundred years, if left to the elements. Fossils are a 1 in a million occurrence that require precise conditions in order to happen. In conditions that are less than ideal, there would be zero evidence of anything we have built in the last few thousand years without constant maintenance.

Only stone endures long enough to last longer than that, and we tend to harvest stone to build new things, every time we find old things made out of it. So everything that would have otherwise lasted, could have easily been recycled so many times over, that it's completely unrecognizable now. Except for the few, rare structures that we see today, and have little to no explanation for, other than speculation based on vague 3rd person accounts.