I don't agree that marriage is strictly a religious thing. It's for the same reason I also don't think that Christmas is a strictly religious thing. I'm not religious, and even if I were, my parents' religion is not christianity. Despite this, I celebrate a secular Christmas. Gifts are given, mirth is enjoyed, and various other traditions observed. Perhaps some day I'll be secularly gay-married.
Christians rarely complain about non-christians celebrating christmas but wrong, but some christians are quite vocal about gay marriage being an affront to god or whatever. I wonder if it's because Christians understand that there's "christmas" (the capitalist funtimes holiday) and then there's Christmas (the religious holy day). Maybe this understanding can be extended to marriage as well.
Either way, as someone who doesn't really understand religion and is happy to let other people do their own thing, it doesn't bother me if there are churches that refuse to perform same-sex marriages. I can understand that perhaps a baker shouldn't be allowed to reject a making gay wedding cake on homophobic grounds, but I think it's a tougher sell for me that religious congregations should be forced to do same-sex marriages. But again, I don't care strongly either way.