I suspect such a system would be quickly gamed by spammers. With the bot-nets and other resources they have at their disposal, they would be able to flag and drown out legitimate sites or competitor sites in order to boost their own. Then the problem changes to being able to differentiate legitimate/regular users from spammers/black-hat SEO practitioners.
Maybe, or maybe there are ways to identify and eliminate the bots and spammers.
For example, it could be linked to accounts, and dishonest accounts get removed, together with all their votes. Or the votes are kept, but because they're so different from real people's votes, they only count for the search results shown to other bots and spammers.
And at the very least, your own votes should override anyone else's votes, so for regular users who actively vote, the search results will still be close to what they want.
I'm also thinking about classifying results by different contexts. A page might be very relevant in one context, and not at all in another. But do let users control those contexts.
