That's true, and an even more powerful response of a man in this position is to deal with the problem less passively by (a) pointing out the trend without making a big deal, (b) suggesting that the team rotate more often, (c) proceeding to lead by example by taking notes.

This may be even too subtle a message for a problem as severe as this one, but it at least signals that not everyone within the team will tolerate this behavior, without claims of sexism backed by little evidence.