One problem with downvotes, at least the way they are implemented in reddit and HN, is that they cancel out upvotes. You are not able to see how many upvotes a post got, only how many upvotes minus downvotes.
This always bothered me in some subconscious part of my brain, thanks for putting it into words. Reddit even lets you sort by controversial posts, so they clearly store both upvotes and downvotes, why do they have to display them as one number?