The only difference downside really is the occasional extension that needs a manual install.
I've been trying out VSIX Manager for my extensions so that I can keep track of which ones I want to install on each machine. This can make it easier to install the weird extensions that you can't install otherwise