I avoid voting for exactly this reason. I prefer to filter the noise myself by ignoring it over decreasing it for others, and the only reason for this is that I don't want to be penalized for it.