Interesting article. I don't understand the filtered review system at all. Beyond the 'he said / she said' complaints that occasionally come out, there are things about their system that simply don't make any sense unless Yelp is incompetent or slimy. For example:
- When you post a review, you as a reviewer think its unfiltered forever. When you revisit the page as a logged in user and read a place that has your review, your review is visible. When you log out or log in as another user, the review is filtered and hidden. At the very least, it should tell you your review is filtered, I see no reason to pretend the review is not filtered when the review is legitimate.
- When you view unfiltered results, the per page number mysteriously changes to 10 per page. I don't see any reason why this should change. Plus the results are pretty slow to load, quite slower than the results for filtered reviews.
- Why do you need to enter in a captcha to view the unfiltered reviews? Why would they care if you were a bot only for the unfiltered reviews and not the normal reviews? I don't see the difference, unless they want to prevent people from writing scripts to pull in unfiltered review data. Plus the captcha is fucking horrible, literally half the captcha's I get are not readable and I need to refresh.
- The filter algorithm seems to be clearly flawed and simply catches way too many reviews that should not be filtered. For example, take this user: http://www.yelp.com/user_details?userid=tZlbsUVo-8wtnR7oMa-3... . The guy has 11 reviews, 1 1-star review, 1 2-star review and nothing out of the ordinary and yet his review about Yelp was filtered. Why? His points in the review seemed legitimate. He seems to be a normal user, not a new user and posts reviews across the board (more good reviews than bad in fact). They should either fix the algorithm or be more transparent about why reviews are filtered because I can't understand why a review like that is filtered.
> When you view unfiltered results, the per page number mysteriously changes to 10 per page. [...] Plus the results are pretty slow to load, quite slower than the results for filtered reviews.
Caching, I'm sure most unfiltered reviews are cached whereas filtered reviews are not and reaching out past the cache can be expensive. One way to mitigate this is to reduce the number of results you pull.
> Why do you need to enter in a captcha to view the unfiltered reviews? Why would they care if you were a bot only for the unfiltered reviews and not the normal reviews?
If you can write a script to deduce the filtering algorithm then you can by definition write reviews that thwart it. With less data, it is harder to deduce the filtering algorithm. In other words, a captcha thwarts high-volume review fraud.
> The filter algorithm seems to be clearly flawed and simply catches way too many reviews that should not be filtered.
I think most people seem to underestimate the difficulty of the problem. Unlike e-mail spam, which is easy for a human to spot, fake reviews are very hard for a human to spot. How can you tell if a consumer was provoked into writing a positive review so that they could get a few bucks off their order just from their writing? You can't, you can look at other statistical trends behind such reviews (such as a sudden wave of positive reviews), but you're only looking for side effects of the primary problem and thus you will never achieve perfect performance from a method like this.
Yelp takes the (somewhat philosophical) viewpoint that customers who are coerced into writing a review are less genuine than they would be otherwise. I believe that this view drives a lot of their algorithm and possibly threatens its accuracy in a way that is ultimately not worth it. I think there are a number of things that Yelp could do to make the users trust in reviews greater that don't involve filtering - one simple thing would be for a user's review of an Indian restaurant to show me that user's breakdown of reviews of other Indian restaurants.
TL;DR: This is a much harder problem than it seems at first glance, partly because of the nature of the problem and partly how Yelp has framed it for themselves.
Disclaimer: I used to work at Yelp, but no longer do. Everyone I worked with were stand-up guys.
I guess most of your points make sense. I just feel like Yelp does very little to be open and transparent. I get that its very difficult, nobody ever said its easy to algorithmically guess review spam.
But they clearly don't want users to see unfiltered reviews. A tiny gray link below all 40 reviews, then a captcha (or two or three) and then a slow user experience before you can see the filtered reviews is lame.
I agree with you about the showing other reviews of the same subject, that would be neat. I guess if I were Yelp, I would try harder at standing up for their algorithms and show more data about why they work and why we are better off having their amazing algorithms.
I had an experience a year or so ago with a friend who started a moving service in SF. A couple of months after he started the business, he noticed he received a review on Yelp from some dude that said during a moving job, the guy took a smoke break and peed all over the sofa he was moving. Not only was the story ridiculously false but my buddy had no idea who the reviewer was. The review did however NOT get filtered, even after he responded to the review and contacted Yelp. And he was stuck with this crazy review at the top of his profile. This went on for months and it really damaged his credibility, meanwhile he would have positive reviews from legitimate customers who would naturally have a newer profile or whatever and the reviews would get filtered. It just seems like Yelp should be more sophisticated. (And yes, they are 10000% better than TripAdvisor)
"At the very least, it should tell you your review is filtered, I see no reason to pretend the review is not filtered when the review is legitimate."
I can explain that for you! If you're gaming the system and Yelp catches you, they don't want to tell you they've caught you, or you'll just try again. That's also why you need to enter in a captcha to view the unfiltered reviews -- they want to prevent automatic methods of confirming that posts were filtered.
Apparently this isn't common knowledge -- the OP makes the same mistake:
"Interestingly enough, you have to pass a reCaptcha when clicking on it. Weird, I don't have to do that to view unfiltered reviews. Interesting...."
EDIT: ps, the user you linked to doesn't have a name. I'm not saying his stuff should be filtered, but keep in mind when you're looking at this stuff that a million factors go into that decision, and it's probably in their customers' best interests to err on the side of filtering a review.
But there are easy methods to see if your review was filtered - you just look for it with a different account.
This only affects people who have written legitimate reviews, that have been filtered. It does nothing to prevent an automated system from doing anything.
That review that was filtered contained swearing. Does Yelp filter reviews like that? I'm sure there are kids that use the site, so that's a possibility.
You need to find more cases like this. Just one data point is not enough to draw any useful patterns.
I think the idea is that given a large corpus of filtered and unfiltered reviews, you might be able to reverse engineer signals in the algorithm and game the system. If that's your end goal, you and the software you write is likely to ignore robots.txt directives.
A quick and dirty solution could be to add something to the page using javascript after the page has loaded and only let the link work if that variable exists (and check the value of the key with the server, if you wanted to be more cautious). Not a complete solution, but a first step and invisible to the user (and a pain in the ass to a robot)
A robot scraping yelp's deep data is going to be site-specific, and having to scrape another javascript variable is not much more than a slight speed-bump.
"I don't like Yelp, so here are some random unsubstantiated complaints about the reviews about Yelp itself."
The author seems to think people hate Yelp, but I'm not sure that's the case. Everyone I know in real life uses it regularly when trying to find some place to go, and the results are largely acceptable.
Review sites are always going to trend negative because people who have had average or good experiences aren't going to be driven by rage to write a nasty review. Everyone has their own star scale and expectations of service ("I had to sit in economy class on my $10 ticket! I'm never flying United again!"). This leads to useless star ratings, but this is no fault of Yelp itself. It's the fault of relying on non-professionals to do professional-quality work. But, if you read for content, you can usually figure out whether a place is good or not. For example, a review like "I went during the dinner rush on Saturday night and it took 5 minutes to get a table! 1 star! Oh yeah, the food was good." is a positive review, even though the reviewer only gave one star.
So anyway, don't hate on Yelp, hate on the clueless people clueless writing reviews.
I don't like Yelp, so here are some random unsubstantiated complaints about the reviews about Yelp itself.
I'm inclined to agree with you about the original poster's perspective but it is hard for me to get past the serious accusations and evidence presented by numerous business owners; that Yelp basically extorted them when they received negative reviews and were a new listing on Yelp.
Based on multiple reports it appears as if Yelp's business intelligence group actively targeted(s) businesses that met certain criteria (new, very few reviews), sent those results to an inside sales team, and then if the customer didn't purchase from the inside sales rep Yelp effectively changed their filter process for that individual business to "penalize" them for not paying to play.
Zagat, Urbanspoon, and TripAdvisor, have never to my knowledge ever implemented these type tactics. I might be wrong, but barring a set of insurmountable complete coincidences for each and every one of the businesses that reported these algorithm changes in the way reviews were displayed, I've got to conclude that Yelp is using their inside sales teams to pressure (extort?) businesses that their business intel teams identify, and make them pay dearly if they don't pay.
I think this is a vocal minority. Sure, you hear all the time about this random biz owner who thinks Yelp is out to get him, but that doesn't account for the hundreds of thousands of biz owners who have claimed a free account and not cried afoul.
A more likely explanation of "extortion" is that the sales org hires 21 year olds fresh out of school who routinely don't make it past 90 days. I'd imagine some of these reps might resort to shady practices to fill a quota, but that's also why they don't last past 90 days. I think this is just a problem of needing a sales organization to make money.
Disclaimer: I used to work on the engineering team at Yelp, and I have the utmost respect for the intelligence and morality of the engineers on the review filtering team.
I'm willing to postulate that the shady tactics are originated by the sales reps, but not that Yelp isn't responsible in that case. Tolerance of a culture of unethical sales, creating an incentive & hiring structure that encourages it, and being obstructionist when someone who encounters it tries to get their issue dealt with speaks rather more to complicity than innocence. As further anecdotal indication (since you appear to believe that all complaints are isolated and don't appear to have any threshold at which you'll accept otherwise), my girlfriend's cell phone was listed as her restaurant's phone number and she was told repeatedly by support staff, sales, and supervisors that Yelp would not under any circumstance correct this unless she paid for an account. Since her place was open from 6pm to 9am and she had to have to phone on in case of emergencies there, she got calls all night long seven days a week. The only thing that fixed this when she was at the point of involving her lawyer was a random Yelp employee who frequented the restaurant hearing about it. If a single internal comment is enough to fix it, but a legally actionable case (non-public number acquired from a review, refusal to address, telemarketing calls to it) gets "sue and be damned" I'm inclined to think that the company has a major issue at the interface with businesses. She literally did not care what the reviews were because word of mouth in a couple of communities was far more important, but they kept trying to sweeten the deal by offering to "fix" the negative ones.
But that has never been proven and is highly unlikely. The critics always stretch the facts. For instance in the dog trainer case, the one negative review was by a yelp elite member (typically harsher reviewers) with over 400 reviews. While most of the hidden reviews were during a one week period and by users with only one review (ie, "drive by").
So then how do you explain them filtering mostly negative reviews for their own service while filtering all positive reviews for the dog training company?
I believe the way the algorithm works is it looks for how credible a given complaint is based on many factors. If I had to guess, they'll look at the age of the user account, how active it is, how many other reviews are written by the user. A user who just created an account and wrote a negative review against Yelp is probably going to be filtered out because that user isn't a regular reviewer. A more compelling argument would take all the users that were filtered and analyze their history. That's where the interesting data is, so show me that and convince me that Yelp's algorithm is bad. Don't just say "Look at how difficult reCaptcha is, Yelp must be up to something here." When has any implementation of captcha ever been easy to use?
That original article says that the users banded together to write positive reviews over a short period of time. We don't know whether the business owner incentivized them or if they just did that altruistically. The facts are simply not known, the best we can do is make assumptions there (which makes for a pretty weak argument to begin with). Banding together and writing reviews with the sole purpose of boosting a business' online reputation is what really happened here, and Yelp specifically designed their algorithm to deal with those use cases. The actual pattern businesses follow for building a reputation is very incremental: positive reviews spread over time, not a short burst by people who aren't actively using the site to review other businesses.
To make a compelling case, rather than what appears as a rant in its current form, you need more data. You need to show that the users are active on Yelp and they aren't afraid to write both positive and negative reviews to reflect their true experiences. Banding together in groups to write positive reviews is basically the same thing as submitting a story to HN and getting all your friends to upvote it.
You need to show that the users are active on Yelp and they aren't afraid to write both positive and negative reviews to reflect their true experiences.
That shouldn't be necessary. I've written like, five reviews ever and they're all honest, I have a few friends that have written reviews and they seem to average one review every few years or so. Meanwhile I've seen reviews from elites that are well-written but absolutely pointless for me. Sometimes it's as if they're writing more to impress other reviewers than to actually give me any useful information.
Banding together and writing reviews with the sole purpose of boosting a business' online reputation is what really happened here, and Yelp specifically designed their algorithm to deal with those use cases.
Why filter the reviews entirely instead of just sorting them down or something? Especially if they seem more or less legit except there's a small bunch all at once?
Banding together to boost an online business's reputation really should not be cause for filtration. Fake accounts, bots, lies, and attempts to bury negative information are problems. A bunch of loyal customers who hear about yelp from the owner and want to write a review is quite simply not a problem and yelp shouldn't be doing anything about it.
And why discriminate against businesses that don't sell advertising?
Banding together to boost an online business's reputation really should not be cause for filtration. Fake accounts, bots, lies, and attempts to bury negative information are problems.
How do you algorithmically distinguish between the two? To Yelp, someone who signs up for an account and writes one positive review is the exact same thing as a script written to create bots that go off and write one positive review. They're both identical. And the bots' behavior can change to write more reviews to prevent looking like a bot. Business reputations take time, so when Yelp's algorithm sees "this business has been around since 1975, and 100% of its positive reviews just came in over the past 10 months, where 100% of its reviews were previously negative before that", that should be a red flag. I can't find the age of Frank's dog training business on Yelp; I searched and I didn't see it listed. So I can't say for certain whether the age of his business on Yelp had to do with the algorithm flagging the reviews.
And why discriminate against businesses that don't sell advertising?
Yelp is letting you bump up your flagged reviews if you pay an advertising fee. If Yelp's algorithms don't trigger any red flags, positive reviews should show up unfiltered on your page whether you pay for advertising or not.
How do you algorithmically distinguish between the two?
Maybe it can't. It doesn't matter you use a linear or binary search if the program isn't fed valid data.
I'm suggesting they are using an algorithm to solve a problem that it shouldn't be used to solve.
And the bots' behavior can change to write more reviews to prevent looking like a bot. Business reputations take time, so when Yelp's algorithm sees "this business has been around since 1975, and 100% of its positive reviews just came in over the past 10 months, where 100% of its reviews were previously negative before that", that should be a red flag.
Interesting that you use the term "red flag" since that usually means a "warning" or "needs attention." There's a world of difference between using an algorithm to raise red flags, and using an algorithm to automatically prune content. In fact, a three-decades old small business with a lone, 1-star review should raise a red flag about the 1-star review.
Business owners soliciting reviews was the reason that (at least) one lawsuit against Yelp was dismissed, and I'd imagine flashmobs of high ratings are particularly easy to detect algorithmically.
I'm not defending Yelp. I'm just saying there's no evidence that Yelp is widely hated amongst average people. (I assume it's not, but I don't have any data for that either.)
I can't explain their filtering algorithm because I didn't write their filtering algorithm. Ultimately, automatic fraud assessment is the only way Yelp is going to scale; it's too expensive to have a human validate every review, so you get the computer to do it for you. 99% of the time, it works. 1% of the time, you get the situation where the dog training company gets screwed. It's like when your credit card gets flagged for buying gas out of state or using an ATM in China. It's annoying that you got false-positived for fraud, but that's what happens. It's the only way the credit-card system can work. Same for Yelp's filtering; no automatic filtering, no Yelp.
The problem with Yelp is that people don't take it with a grain of salt. But that's not Yelp's problem, that's a "people are morons" problem.
I wonder the psychology (of many review sites) that people are more motivation to seek out area's to post negative reviews, rather then a positive. Lack of a business "if you enjoyed our service please let us know via (Yelp, Google Pages, etc.) which may net a higher return of happy customers.
I personally more inclined to hop on Yelp as a warning to other consumers on a poor business, vs, just random "ups" to other well run businesses. I gain no benefit but altruistic feelings, and this, takes time. The time it takes to post a negative review is rewarding in itself to try and "help" others avoid or be wary of said business.
Also, markets should take a factor in analysis of Yelp reviews. As an example, larger cities have a much higher preponderance of reviews as a whole, where even medium (what I would call them) such as SLC (where I live) are very sparse and therefore limited in search / decision making.
It is not just about clueless people writing reviews. Yelp as a business doesn't care about the business owner when some inaccurate reviews are being posted and clearly break their TOS.
Yelp doesn't try to be fair toward the business, and why should they? Their goal is to sell ads which will have the side effect of pushing positive review toward the top of a page business.
So having an algorithm system biased toward promoting negative and potentially fake reviews is good for Yelp business, since a business owner will rather have the positive reviews listed first when advertising through Yelp.
Unsubstantiated complaints? I simply wrote what I saw, besides a few opinions here and there. And I know a lot of people do indeed dislike Yelp, but like some other sites with huge user bases, there's little other options to turn to, so you're stuck with using Yelp. I actually use Yelp quite a bit myself.
Anyway, the evidence is clear for there to be good reason to hate on Yelp (see my article). But like you said, there's also good reason to hate on clueless reviewers. Why can't we do both until either issue is fixed?
You claim that Yelp is writing their own 5 star reviews:
Personally, I wouldn't be shocked if Yelp is putting in its own 5 star reviews for itself in order to offset the barrage of negative reviews. Common perception of Yelp is not good, far from their current 3-star rating. Just glancing over the first few 4 and 5-star reviews for Yelp, all but two came from an 'Elite' member. Of course they review Yelp highly, they get thrown crazy parties. And I wouldn't be shocked if you lose your Elite status for a negative Yelp review.
This is pure FUD. You don't know if you lose elite status for a negative review. Not everyone gets to go to crazy parties. If this were Wikipedia, every statement would be tagged [weasel words] or [citation needed]. You've just made a bunch of stuff up.
If this is just entertainment, fine, but if you think you're writing something useful that is worthy of HN, then you're a little off. Your article boils down to links to filtered reviews not being obvious enough (even though the link is the same as all the other links on the page), complaints about reCaptcha, which Yelp doesn't even control, and then a bunch of made-up facts/anecdotes that you have done no research to prove true or false.
Look, I don't even like Yelp. But this is simply sloppy journalism, and it reads like it was written by a fifth grader who got a bunch of one-star reviews on Yelp.
Jesus dude, do you work for Yelp? So damn angry over an article that mostly states the breakdown of Yelp's filtered reviews.
Notice I said "I wouldn't be shocked" twice, not "I know for a fact that". I not once said that was the way it was, I simply said I thought it to be a decent possibility.
And this isn't worthy of HN? It reads like it was written by a fifth grader? It's number one on HN right now. Did I do something to you at one time in the past? If so, I sincerely apologize for ever wronging you.
I don't work for Yelp. I'm just saying there's nothing interesting about your article, without wasting my time by going line-by-line.
Do you really think your article is an example of top-quality writing and analysis? (Also, why post your own work to social news sites if you can't take the criticism? I don't hate you, I hate this article.)
Ha, I can take the criticism just fine man. I can also defend my writing, which is all I'm doing.
And no, I definitely don't think my post was top-quality writing or analysis. It was my second blog post, so I'm far from those labels. However, I'd place it a notch above a fifth-grader.
It's easy to jump on the bandwagon and say "Yelp is evil," but having been a near-constant point of controversy, it's something they do their best to address.
Yelp's main defense is "It's done by an algorithm". That's not a valid defense for a bad business practice.
They are accused of hiding positive reviews for businesses that don't pay them. You can perfectly well write this logic in Python for an filtering algorithm:
def filter(review):
...
if review.business.contacted_by_sales_count >= 3 and
not review.business.paying_yelp and
review.stars >= 4 and
random() > 0.8: # 80% chance
return true # yes, filter it
...
Even if it's not written so explicitly they have to take responsibility for their algorithms.
The sales org only has access to Salesforce copy of Yelp's internal data, and numbers like "contacted_by_sales" never ends up back in the main Yelp database. It's not something the algorithm could take into account because it doesn't know about the sales org at all.
Disclaimer: I used to work on the Yelp engineering team, at one point on the Salesforce data refresh project.
I've posted about 6 reviews in total on yelp, 2 of which were immediately filtered for no reason. Funny how my positive reviews went through just fine and the rest got binned. If you're going to run a review site, at least let people give their honest opinions don't filter anything other then profanity. People should be able to share an experience be it good or bad.
Although I wish it were that simple, it's more complicated than that. The reason for this simply is, if I own a business, I could create multiple fake accounts and have friends and and associates do the same. We would then be able to give multiple (fake) positive reviews for our own business, while giving one star feedback to our competition.
The captcha looked quite clearly to me to be "inctory". These captcha's are not necessarily real words (I believe to thwart dictionary based crackers?), and I also am pretty sure the service they are using is not tied to Yelp itself.
The first word wasn't the difficult one, it was the last word (it's even tougher to decipher when zoomed in for normal view. Zoomed out, it appears more like 'Law'). I agree though, I know the service probably isn't tied to Yelp, I just found it weird that I had to decipher a blob of ink on page 52.
If one word is completely illegible, it often isn't being used as a captcha. The captcha company may be trying to have you manually recognize text their OCR software could not.
You can often make it through after painfully screwing up the hard to read word.
That's exactly what's going on with ReCAPTCHA. One of the elements presented is a "known" that you have to get right, and the other is (usually) an unknown or ambiguous OCR result. Google is crowdsourcing their scans. And yes, sometimes it is just a blob of ink -- on other occasions, it's an inline differential equation or something similar that can't be properly entered into the text box (even with the whole of Unicode at your disposal). There's no penalty for getting the graecum est part wrong, but if it's something you can read but can't enter properly, do be a gem and don't pollute the pool of results -- hit the refresh icon.
This is how I do reCaptcha. I guess the easy-looking word correctly, and then type garbage for the hard-looking word. This results in approximately 99% correctness and hopefully screws up their "make jrockway do work for us for free" scam.
What scam? Google offers the reCaptcha service for free in exchange for having words OCRed; seems fair to me. To you as the user, it's not only the same work as any other captcha (which you would have to fill in anyway), but you get free OCRed books and magazines from Google Books.
If you don't like the reCaptcha ($deity know why, but it's your right), complain to the website, don't fuck it up for everyone else.
I stopped using Yelp after I discovered that reviews I logged to the site were not showing if I was not "logged in" to Yelp. What is the point of using a social review repository if the reviews are only viewable by me, when I log in to the site? I'd be better served by putting it in a text file/directory.
It's not a democracy - Yelp pushes the best reviews to the top. Most people read a couple of reviews, it serves Yelp's interest to make sure they read the best ones.
So, in other words, I as the user contributing reviews for which makes Yelp valuable, are treated as a de facto ruffian, my time and effort expended to "share" considered folly. So Yelp "filters" these and beyond wasting my time (as I should just collect in my own file of reviews or "share" on my own site) is it not a great conflict of interest going on here?
Organizations like Yelp live and die on their accuracy and predictive ability. If Netflix recommends movies you don't like, you will use it less. If Zagat allows too much personal preference, it will no longer be the gold (Maroon?) standard of crowd sourced restaurant reviews.
If half the things suggested about Yelp are true, they risk their integrity at great peril.
Actually, Yelp can play game like this and be profitable.
If business have a good rating on Yelp, then that pretty much mean that business is ok. However, a bad rating might be because business was not nice with Yelp sales people.
That is ok because they target small, unestablished, businesses which you will never discover that they are actually excellent business because they have just one star review.
In other words, advertising at Yelp does not guarantee you a good review. However, not being nice with their sales people might get you a bad review if you are a small business.
I think Yelp has always been a good service but obviously there is a disconnect between the business end of Yelp and the service end. The Chicago Tribune article is not the first to complain about the business practices of Yelp. Additionally, advertising on Yelp is some of the most expensive real estate on the web with CPM rates of $100-$300. Ultimately I think that directory sites such as Yelp will become obsolete as social search becomes more refined and relevant. My company is actually working on this and we thing that we have a good product so far (in alpha). http://www.cliqsearch.com
- When you post a review, you as a reviewer think its unfiltered forever. When you revisit the page as a logged in user and read a place that has your review, your review is visible. When you log out or log in as another user, the review is filtered and hidden. At the very least, it should tell you your review is filtered, I see no reason to pretend the review is not filtered when the review is legitimate.
- When you view unfiltered results, the per page number mysteriously changes to 10 per page. I don't see any reason why this should change. Plus the results are pretty slow to load, quite slower than the results for filtered reviews.
- Why do you need to enter in a captcha to view the unfiltered reviews? Why would they care if you were a bot only for the unfiltered reviews and not the normal reviews? I don't see the difference, unless they want to prevent people from writing scripts to pull in unfiltered review data. Plus the captcha is fucking horrible, literally half the captcha's I get are not readable and I need to refresh.
- The filter algorithm seems to be clearly flawed and simply catches way too many reviews that should not be filtered. For example, take this user: http://www.yelp.com/user_details?userid=tZlbsUVo-8wtnR7oMa-3... . The guy has 11 reviews, 1 1-star review, 1 2-star review and nothing out of the ordinary and yet his review about Yelp was filtered. Why? His points in the review seemed legitimate. He seems to be a normal user, not a new user and posts reviews across the board (more good reviews than bad in fact). They should either fix the algorithm or be more transparent about why reviews are filtered because I can't understand why a review like that is filtered.