You can connect, set it up, patch it, and then wipe it, revoke the permissions, make it an offline device again.
You can go back and forth. To use smart tv functions again, youd need to agree to the spy stuff again.
(Edit: I own a Visio and a Fire TV, so that's what my response is based on.)