I use the "plant-based" label because people get weird about the v-word, and quite frankly I'm only super-strict about meat and egg so don't think I've really earned the title. I avoid dairy, but it shows up in a lot of unexpected places so I give myself some leeway with micro-ingredients. Why? Because it's more effort than I'm willing to put in. I know that won't be a satisfactory answer to diehards, but it is an honest one.
Or perhaps you are asking if it is only about food? I don't use non-food animal products either (e.g. leather, wool, down).
My last paragraph is really more rhetorical because it's a very predictable "whatabout." While not wanting to bother with the effort certainly is true (in-line with my attitude on dairy micro-ingredients), having easier availability to small local farms wouldn't change anything in this regard. I'd still stick to a plant-based lifestyle. If anything, it's something I might have used as a crutch while phasing out animal products. But I am past that now.