Part of the problem is that you can be a social libertarian (the government shouldn't tell me who I can/can't marry) or an economic libertarian (the government shouldn't tax me). Tending to one is mutually exclusive from tending to the other.

These are both spectrums as well not hard and fast rules. For example "I'm a man and I want to marry a man" is quite different from "I'm a man and I want to marry a 12 year old child" and "the government shouldn't tax poor people" (is the economic or social?) is quite different from "the government shouldn't tax anyone".

Given they are spectrums there should be a modicum of both of these ideas but as with anything, extremes are probably going to have extreme downsides.