I think this has the potential for going down an infinite rat-hole, but I always think it would be interesting if downvotes required justification (e.g. a "why?" popup that included stuff like "rules violation", "trolling", "this is wrong", etc) and then other people were allowed to evaluate specific downvote reasons for whether or not they seemed reasonable. You could then weight downvotes based on how legit the reasoning of the downvoter is (does everyone think you're full of shit when you say it's trolling?) and also based on whether you tend to downvote for reasons that are not actually worthy of a downvote (you just disagree with the post).