Another personal anecdote, but not only do I consider "guys" gender neutral, but I found it kind of alienating when a few people clearly stopped saying "guys" shortly after I joined the team. I was the only woman on the team that summer (internship), and I think they were just trying to be nice, but I don't like when attention is called to gender in the workplace. The more (actively) aware I am of being "different", the more it impacts interactions.
Maybe it's because you're asserting your anecdotal experience as truth, when it's not at all representative of most other people's experience.