That's because being anti-mandate is effectively being anti-vax from a public health policy perspective. By construction, effective public health can't be about individual choice, and the astonishing efficacy of vaccines in history has relied on this principle.
That really isn't what normal people infer from the term.