I did also specify coastal societies. Evidence suggests that while there was some light agriculture that was no where near the bulk of their calories. Those primarily came from marine sources.
Florida and coastal Georgia are the best examples.
You find mounds scattered throughout, but you are right the largest examples are typically connected to farming reliant groups. Except on the coast you'll see the opposite. However these do tend to be much smaller mound sites, and no I'm not speaking about the shell middens. You can argue if those were intentional or created as a result of natural refuse accumulation
Also great point on the farmers having time for other activities too. That's a good point and I have no counter for it