The defining case where we unambiguously lack free will is when there’s another agent that actively constrains our actions. But for one agent to constrain another, it has to be able to anticipate the other’s potential actions—which depends on the abilities of the constraining agent, not the one constrained.
We have (I think) an instinctive sense of when our behavior is susceptible to that kind of directed constraint, and the idea of a deterministic universe raises the possibility that that susceptibility can’t be escaped—but only if there’s a constraining agent that can actually exploit the susceptibility.