Your thinking is off, the GPL and derived licenses like the AGPL are viral on purpose. They apply to everybody who uses, downloads, or accesses the software (in the case of the AGPL) and they are explicit about this:

Each time you convey a covered work, the recipient automatically receives a license from the original licensors, to run, modify and propagate that work, subject to this License.

AGPL can be closed too, the license bases the right to the source based on the access to the end product:

• GPLv2/v3 - if you have the binary executable output of the GPLv2/3 covered source, you must be granted access to the exact source used to make the binary. This applies to legitimately sourced binaries only - if you were to hack into a company's servers and get a binary of a modified GPL product, this wouldn't apply. But extracting a binary from a device you own IS a legitimate access (so e.g. if your phone uses U-boot, the manufacturer must grant you access to their modified U-Boot sources used to build the bootloader)

• AGPL - if you have (legitimate) access to a service you can request the source. This is so e.g. web services can be made into GPLed code where modifications must be released (negating the requirement of possession of a binary, since you can't possess a binary that runs on a remote server). e.g. let's say I run GTK app via browser using kasmVNC - if the app is GPL, I don't have to provide the source, if it's AGPL, I have to provide the source.