There is a similarly interesting result in ET Jaynes "Probability The Logic of Science" (chapter 5). Where Jaynes demonstrates that increased evidence actually decreases your belief in a hypothesis.
Jaynes gives an example of an experiment in which a psychic predicts cards, getting n out of m correct. In classical hypothesis testing H0 would be "the psychic got lucky" and H1 is "the psychic has mystic powers". Jaynes first attempts to use Bayesian reasoning to work backward to determine your prior belief in psychics. That is, how much evidence would it take to convince you, compare that to the raw likelihood and what you have is your prior beliefs quantified.
He then points out that he personally would never believe the subject was psychic, this is because there are not just H0 and H1, but H2 "the psychic is deceiving the experimenters", H3 "the experimenters are making an error" , H4 "the experiments are fudging the results", H5 ... Each of these has their own prior.
If your prior for believing in psychics is low enough and your prior for "the experimenters are fraud" is high enough, the more extreme the evidence the more you will be convinced that the experimenters are disreputable con artists, and subsequently the less you will believe the subject is psychic.
This is actually Jaynes' solution to a huge problem with Bayesian reasoning as a form of human reasoning: "If more data should override a bad prior, then why in the 'age of information' does nobody argee on anything!" This example shows, according to Jaynes, that while we can certainly have irrational priors we can still explain human reason in Bayesian terms and still get a situation where two people faced with plentiful information will arrive at contradictory conclusions.
I don't like wording "arrive at contradictory conclusions", because it implies the creation of differences in beliefs.
I'd rather frame it as "maintain contradictory preconceptions". And the cause of this I would say is a lack of information: Information capable of distinguishing between H1-H5
Or... now I am not that sure anymore. What if Person A believe in H0:97%, H1: 2%, H2: 1%, and Person B believe in H0:97%, H1: 1%, H2: 2%. They have quite similar beliefs, at least by some measures. But if H0 is ruled out, then suddenly their beliefs will be very different.
How does this contradict your earlier comment? I think it's still valid: Person A believes H1 (real psychic), Person B believes H2 (talented fraud). The question now is how to get them to agree. And the answer is: design a better experiment, such that fraud is not practical, and see what happens.
H0 and H1 should always be phrased so that H0 is "not H1". In the above case that would be H1 "The psychic is not just lucky".
Rejecting the data based on hypothesis testing is a misapplication of the technique. That doesn't mean that data is always right but it's just another hypothesis about the data itself, e.g. H0 "data is fine", H1 "data is not fine".
Interesting. Derren Brown, English mentalist, describes in one of his books how he deliberately failes every now and then to make it more convincing there are pschycic abilities at play instead of a mechanic trick.
"A Bayes Factor Meta-Analysis of Recent Extrasensory Perception Experiments"[0] is a good read in this area. It extends the analysis of psi with Bayes factors and underscores how the priors, and a causal mechanism, really matter.
>>What Gunn et al. show is that the accumulation of consistent (non-noisy) evidence can reverse one’s confidence surprisingly quickly.
I know this comment won't be accepted very well here on HN, but I think this is one of the reasons most people who don't believe in climate change don't. Science, and the subsequent reporting on it, has been consistently reporting for the past 20 years that climate change is real, and it's worse than we thought. Every few months, there is a new study being widely reported saying that X new evidence is proving that things will be much worse than we thought. If there was a study every once in a while that said X study about climate change was wrong, and we didn't consider Y factor, but climate change still seems to be headed in a dangerous direction, then I think people might be more willing to accept the thesis.
Of course, the paper doesn't suggest that we are wrong when we suspect systemic error, and skeptics of climate change may very well be wrong, but maybe this will help others understand their skepticism.
I think you're mixing up the science itself and the reporting on it. If a paper was published saying that sea level rises were going to be 20% lower than expected I wouldn't expect most people would ever hear about it. Or if they did it would be reported as showing that sea level rise was even more of a threat than we had believed.
If you want the truth the IPCC reports area great for providing the scientific consensus along with the (large) uncertainty we have about exactly what climate change will look like.
That's a fair point, although my point was more about the general population's perception of climate change and how it is shaped. That perception is most definitely shaped by the consistent reporting the general media does on climate change, which filters out the messy noise found in the actual hard science.
I think most people who don't believe in climate change don't believe in it because of the amount of money and political power that is devoted to discrediting it. Let's be honest here, most people on _either_ side of the fence (full disclosure: including me) don't actually read climate research, they're only aware of what gets covered in the press, which tends to be the research that's more apocalyptic and less focused on model adjustment, etc.
This is only really an issue because of the way that climate change has been unnecessarily politicised, though. We see dramatic stories reported on from other areas of research ("Hey, we found planet X!"), and yet there hasn't been a huge amount of effort spent on discrediting astronomical research, so the public accepts that the peer review process has worked and there's no controversy ("Scientists are all just elitist liberals, where's the research from conservatives that says planet X doesn't exist?").
It could well be. Though I suspect it's more of contrarianism in a minority of people. We also have man-on-the-moon deniers, etc.
What surprises me about climate change is that they're somehow managed to show causality between one non-repeatable event (people starting to burn fossil fuels) and one other event (global warming). Holy cow causality is notoriously hard to confirm when we can't manipulate the causes. When the cause and effect each only happen once, it's surely almost impossible. I'm suspicious of that extreme confidence.
This thread is interesting juxtaposed with the "Broken Science" piece on the frontpage now [0]. Apparently science is hard, even with laboratories and repeatable controlled experiments...
Why do you think that " If there was a study every once in a while that said X study about climate change was wrong, and we didn't consider Y factor, but climate change still seems to be headed in a dangerous direction" leads to "I think people might be more willing to accept the thesis." instead of "look the scientists can't agree on this stuff, how are we supposed to believe them at all."
I'm running into this problem on my startup website (http://muxme.com). I've found that the more evidence I provide that the site is real, the more people start to question it. I.e. I post pictures of the receipt of purchase, open source the raffle code script, have a live drawing, post the winner's usernames (which reveal quite a bit about them if you google them, I encourage them to change their usernames), post USPS tracking numbers, post my phone number and address, and everyone still thinks the site is a scam. Out of every company / website I've worked on, I've never had people so skeptical of giving a name, email address and phone number. I've made websites where I charged money and had better growth!
This is some real feedback I've received:
"It sounds like a scam to me. If you want to try to scam people by getting them to click on your fake website, you should have enough common sense not to use the website as your username. Better luck next time!!!!"
"Scamming people on Christmas Eve using someone's else's platform. Tacky."
But yeah... this site isn't working. I'm thinking of scrapping it and writing an app/browser extension that automatically enters you into sweepstakes. Let's say Person A enters 10 sweepstakes on different websites. Person B enters 5 sweepstakes on different websites. When person C downloads the app, he will automatically be entered into the sweepstakes that Person A and Person B entered (via my magic backend system that I have yet to create). The more people that download the app and enter sweepstakes, the more sweepstakes everyone with the app will automatically be entered into. Now people will start winning prizes for literally doing nothing but downloading the app.
Interesting concept. The website makes a really scammy first impression to me too.
Firstly, because it uses the "count down to win something" mechanic that's primarily used by exploitative sites. People have to keep bidding in order to win, and loss aversion pushes them to pay more than they can intellectually justify for the product. If most sites that use the mechanic are evil, then the expectation is your site is no exception.
Secondly, the website looks really generic and based off a template. Like something that has been whipped up in a weekend to steal people's info. It would be trivial for somebody to create a scam version that's visually indistinguishable from your website. Having more content and a more polished appearance will help.
Thirdly, it's not clear what's in it for the operator of the site. If the products are given away then what's the catch? I understand you're giving away promotional items, but that's not enough to fix this first impression. You have to explain to people why the site exists. Is it a nonprofit? Are you getting paid? By who?
Finally, your explanation page makes no sense. Your target audience will never care about multiplexers. You've got to have at least a simple FAQ that answers questions like "Is this a scam?" "How does this site make money?" "Why is it free to enter?", and so on. A huge heading on the front page "Play for free to win promotional material. No catch." would probably help alleviate your visitor's concerns. But you'll have to do a lot more to make it seem less fishy.
The site does use a template. I actually tried making it look generic because I went with the approach of "this website is owned by 1 person and you can trust him". That didn't work. I am going to redo the front-end and make it look like a company owns the site. I'm thinking something similar to how http://gusto.com looks.
I'm hoping to one day have companies pay to have their prizes featured on the site. It'd be like Google / Reddit advertising except the advertising is a prize, not an ad. I tried the non profit approach but everyone doubted me, so I changed it. But right now, everything is out of pocket for me.
Yeah I realize the "mux" was a mistake. I thought it was clever, but these people don't have the time to read what a multiplexer is. Literally every time I tell someone to go to my site "MuxMe" they say "Is that a porn site?". I can change the name in 2 seconds, I'm just thinking of what to change it to. I like the idea of a FAQ, I need to add that.
Currently users land on the prize page, and either register or they don't. They don't click the "how it works" or any other pages, so I'm focusing on optimizing the prize landing page right now. I think I might add a video avatar (like this http://www.pt-hr.co.uk/) that explains how the site works on the prize pages. I can see the Hacker News guys are interested in the "How it Works" page, but the typical user doesn't click that. I definitely need to update it with something that's easy to understand, it's on the list of a million things to do.
Even your comment makes it look scammy, without even knowing exactly what your site does. Your account is fairly new, and every comment you have ever made includes a link to your site. You are talking about raffles and winnings, and using bots to enter sweepstakes. The negative feedback you shared here looks like reasonable feedback, and nothing you say contradicts it.
That's fair feedback. I appreciate it. The website is what I do for fun, so there's no professionalism. I literally make no money from the site. If you think the site is a scam (or if you just want to say Hi), you're free to call me at the phone number listed in the footer.
I have no idea why you are talking about selling email addresses, making money, or linking to a github account that contains no email and, fyi, is not even mine.
I'm thinking of scrapping it And (from another comment of yours)
I think this site and the amount of hate/doubt I've received is driving me insane. Sorry.
If you do not have an extremely compelling reason to stick it out, then I suggest you scrap it promptly. Go spend your time on something that doesn't get you hate and make you insane. You should only put up with that if you honestly have absolutely no other choice, no means to walk away.
Once you have been backed into a mental corner of the sort you are in, it is incredibly hard to find a path out. If you are unable to be objective, not take it personally, not get defensive and so on, there is almost no hope that you can figure out how to effectively reposition this site. It will probably go much better to just ditch it and start over and view it as a learning experience.
I considered signing up for your site. I could certainly use more free resources. I also considered posting it to my homeless website. But that runs into the question of conflict of interest. If I want to win, I am better off not sharing it. That only increases the competition. So, perhaps the model of the site is fundamentally broken.
- Layout/design - My initial reaction when the page loaded was "Oh, it's a penny auction site", which are known for being borderline scams. Obviously it isn't, but that gut reaction is difficult for you to overcome (I'm thinking of sites like swoggi.co.uk when I say penny auctions)
Only problem is charging money is illegal. I was thinking of "implying" that you may have to pay one day, and having the register button say "Sign up for your free trial", which will be an infinitely long free trial. But then again, the site is all about trust, and you've already tricked them from the start with this method.
Yeah I tried the approach of a quirky, 1 owner website, and it didn't work. I'm still trying to clean up all the jokes, memes, etc.
Hmmmm, yeah I can see how a first impression is that the site is a penny auction site. I definitely need to rethink the front end design and UX flow. Maybe don't even show the prizes until they've registered.
Yeah the giant table of "win this! win that!" seems enormously scammy. Maybe because scam ads just display something so similar. And, key thing with yours and theirs, no context and leaves me wondering "what's the catch?".
Even if you just added a short paragraph explaining the concept (since otherwise when I get to your site I look around to see what it does, and see nothing but scam ads all over = scam site; if you actually say what the site's for the prizes have context) and how it works (otherwise I'm like, "sure, effectively free cash, yeah right" = scam site), and then have the prize list, it would make a big difference in first impressions. Maybe have any list of prizes on the front page not be a countdown of upcoming prizes, but a list of selected featured prizes ("including these brands!").
On the other hand, the long list of terms and conditions on the Toys R Us gift card actually makes it feel more legit somehow. ("Ahh, there's the catch"?) Maybe have the front page end with that list, or the intersection of all terms, or a similar terms and conditions for muxme itself, rather than "many pages of EVEN MORE PRIZES".
Another way of thinking about it is if your performance metrics, error logs, or unit tests are always 100% green/perfect with no failures or spikes or anything, then it's probably a pretty good sign that your metrics/logs/tests are broken.
One of the ways in which I can spot a scam is to look it up online. If all of the reviews of the product are glowingly positive (and use similar language), then I start looking for a pyramid scheme or something driving sales.
I’m going on a tangent to the topic of the article: MLM companies are getting wise about this though.
If you search YouTube for some MLM companies you’ll find many of them presented as an independent opinion/study on whether a specific company is a scam or not. They’ll point out surface-level flaws in a company’s business in the beginning that would suggest a pyramid scheme. They know they can’t hide certain dirty laundry so they put it out there. But then they start talking about other studies, without citing references of course, showing these companies are not a scam. It goes on from there and sadly a lot of people still fall for these when they’re trying research these companies.
Yeah, that's another bad smell, when you google for "<name of pyramid> scam" and you get zillions of links asking "Is X a scam?" and they all try to convince you -- again, using the same or similar language -- that no, seriously, it's totally legit. This is how Scientology tried to google-bomb itself to counter the effects of Operation Clambake, et al.
Here's the thing: In a multilevel marketing scheme, even the word of the product users is unreliable. This is because MLMs recruit their end users as a sales and marketing force that works, essentially, for free -- with the promise if big riches if you sell enough in the program.
You know "The Secret" and all that positive thinking Aladdin's genie hogwash? That's actually the public facing side of MLM propaganda. Internally, MLMs pump their customer base up with big promises about how if you think happy thoughts you are guaranteed to fly with the hidden (or even explicit!) threat that even the tiniest sad thought will send you crashing to the ground.
And this extends to what people are allowed to say. The key word is "edify". You're supposed to "edify the product" (talk about how great it is) and "edify your upline" (talk about what a great guy/gal the person who introduced you into the program is). This is often backed up with threats and verbal intimidation and abuse. (e.g., "Only losers say things like that. Losers and quitters. You're not a loser, are you?" and it gets worse from there.)
So for some of the most prevalent scams, objectivity can't even be expected from the end purchasers.
I think his simple test somewhat accounts for this. Botted reviews usually don't bother with the sophistication of generating noise in their ratings, banking instead on the possible sale from a spendy shopper instead of trying to convince the already skeptical.
With the assumption that most botted reviews aren't going to be any lower than the two highest options, the same suspicions the article raises should come into play here as well, botted responses or shill responses.
Though, reviews in general are not really all that good. Southwark's Yelp episode was a long but cute take on it, but keep in mind that almost everyone sees reviews differently. On Newegg, it's incredibly common for a part/device to get 1 egg because it didn't work for whatever reason, even when it's the buyer's own fault (e.g., buying an incompatible CPU/mobo, which happens). DoA items, misconfigured computers (e.g., forgetting to plug in a power cable), and so on all warrant 1 star, even though they have nothing to do with the performance of the item in question. (DoA being an exception, though I would argue a device being DoA by chance is not the same as widespread DoA, and that the rating system should have a special category for DoA which gets its own tally and is independent of the ratings)
New Coke was the result of the largest marketing / consumer research project ever. The conclusion of this research was to change Coke's formula. Coca-Cola Chairman Roberto Goizueta claimed that the decision was “one of the easiest we have ever made”. Coca-Cola thought this way for two reasons:
#1 Research was done with a sample of over 200,000 customers.
#2 Coca-Cola’s researchers triangulated the validity of their data with a mixed-method approach. They used focus groups, various surveys, and individual interviews.
All these data & research did the opposite of what they were supposed to. Their large sample size gave them lots of useless data. Data triangulation—which was supposed to safe guard them—did the opposite: it convinced them that their useless data, were useful.
Why does this happen?
In an unnatural system, variance is unbounded. As your data set grows, the unbounded variance grows nonlinearly compared to the valid data. As variance increases, deviations grow larger, and happen more frequently. Spurious relationships grow much faster than authentic ones. The noise becomes the signal.
It's called equivocal data. It is known to management circles, it's the part of management that requires experience and judgement rather than facts. Computers can replace decisions based on unequivocal data.
Maybe later in this century we'll have the alternative of objective Trial By A.I. instead of by peers. Lawyers wont like this because they win by manipulating humans.
Sigh. This article describes a mathematical analysis that is more an artifact of the problem setup than a proof that more evidence decreases confidence. In particular, what's going on is that it assumes that while actual witnesses have a probability of e.g. 50% of convicting, biased witnesses have a probability of 100% of convicting(!), which is an incredibly strong assumption (and in particular, if you changed 100% to anything else, one of the headline results they advertise of unanimity being suspicious and near-unanimity not being suspicious disappears).
Those values of the parameters don't make sense for many real-world scenarios like picking a subject out of a lineup (you'd have to have a really really bad lineup to have witnesses be guaranteed to pick the same wrong person!) or to the output of a panel of judges or jurors.
However, the result is relevant in scenarios where it's very likely that you'd have a 100% likely error, e.g. you're doing an experiment and your apparatus isn't functioning properly or you have a bug in your computer program doing the data analysis. So it's relevant for physicists and computer programmers, but probably not to the criminal justice system.
For those curious, the math is quite simple. They model a world where one of the following happens:
* probability p: compromised experiment, in which case fraction y (y=1 in the paper) of witnesses report guilty
* probably 1-p: normal experiment, in which case fraction x of witnesses report guilty
Given that, the probability that 100% of N witnesses agree the person is guilty in each case is:
prob(unanimous and compromised) = p * y^N
prob(unanimous and not compromised) = (1-p) x^N ~= x^N (since p is assumed small)
So the probability that the experiment is compromised, given a unanimous outcome, scales with their ratio, p y^N / x^N = p * (y/x)^N. A few things you notice:
* If y <= x (as you'd hope to be the case in a lineup where the bias is less convincing than having actually seen the criminal!), unanimity is never suspicious. This is the usual world where more evidence should increase your belief in a hypothesis!
* If y > x, you have exponential growth and so with sufficiently large N you'll see unanimous results are almost always the result of a compromised scenario. However, unless y/x is large and y is close to 1, the actual probability of unanimity is basically 0 so this will essentially never happen.
* If y=1 and x << 1 (the case they analyze), with large N, unanimity is suspicious (but if you change y to 0.95, near-unanimity is, too, so ). This is a case very relevant in software, but less relevant in the criminal justice system.
They picked y=1 and x=0.5, which means roughly you start to see a big effect of this form when N > log_2(1/p), which for their ranges of p from 0.01 to 0.0001 means N ~ 15-30 range.
Unfortunately this is part of a much broader trend of people making bold claims about society based on the results of either a study with a sample a freshman psychology class that achieved 95% confidence (just like ~5% of the thousands of other studies done on psychology classes!) or a mathematical analysis of a confusingly designed problem that doesn't reflect well the broad swath of reality its authors (or the people reporting on it to a wide audience!) are claiming it describes.
You're correct in attacking the math the authors use. It's clear they are bending the math to fit the phenomena. However, that doesn't invalidate the hypothesis.
This paper appears heavily influenced by Nassim Nicholas Taleb: systematic failure, bounded vs unbounded variance, (more data) => (more noise) => (more spurious relationships), natural vs artificial systems, and even the reference to the heuristics of ancient cultures.
It looks like the authors were trying to apply math to observed phenomena. They may not have the right math to make it work-so they fudged it a bit.
So, firstly I see that as you go toward more rigor, overall the rate goes up, with studio art (no rigor, it's just a humanities field) at the left and mathematics (full rigor, fully abstract) at the right. Like this - https://xkcd.com/435/
Interestingly, computer science (of interest to people here) actually isn't as far right as you'd think.
But what pops out to everyone!! There is obviously one MOST interesting fact in the whole figure, the 0 for studio art. That really got me thinking.
Granted, this isn't the perfect example because it's not just 0 as opposed to 5-7 - the next higher one is 20%.
But what you really start thinking is stuff like - "Is it that they didn't really have many subjects, just 2 or 3? what is n? Is it that these people are lying? Is it a real fact?" and so forth.
The fact that it's 0 instead of a few percent immediately raises some questions.
Jaynes gives an example of an experiment in which a psychic predicts cards, getting n out of m correct. In classical hypothesis testing H0 would be "the psychic got lucky" and H1 is "the psychic has mystic powers". Jaynes first attempts to use Bayesian reasoning to work backward to determine your prior belief in psychics. That is, how much evidence would it take to convince you, compare that to the raw likelihood and what you have is your prior beliefs quantified.
He then points out that he personally would never believe the subject was psychic, this is because there are not just H0 and H1, but H2 "the psychic is deceiving the experimenters", H3 "the experimenters are making an error" , H4 "the experiments are fudging the results", H5 ... Each of these has their own prior.
If your prior for believing in psychics is low enough and your prior for "the experimenters are fraud" is high enough, the more extreme the evidence the more you will be convinced that the experimenters are disreputable con artists, and subsequently the less you will believe the subject is psychic.
This is actually Jaynes' solution to a huge problem with Bayesian reasoning as a form of human reasoning: "If more data should override a bad prior, then why in the 'age of information' does nobody argee on anything!" This example shows, according to Jaynes, that while we can certainly have irrational priors we can still explain human reason in Bayesian terms and still get a situation where two people faced with plentiful information will arrive at contradictory conclusions.