I think you don't understand the rules correctly. Or rather, I know a different version. Maybe that is because that question does seem to not really be answered in the rules, at least I found nothing.

I was under the impression that downvoting is for comments that you do more than just disagree with. If you disagree, but it is a valid interesting argument, I try to vote a comment up, in any case not to vote it down.

Flagging is for inappropriate comment, stuff that is harder than just a bad comment, like racial insults or spam.

I'm sure that will be explained to us :)