Of course there is! If statistics didn't work, casinos wouldn't work. If our basic scientific machinery didn't work, computers wouldn't work.
If cost-benefit analyses didn't work, government projects would often go massively over budget producing nothing, while other areas that were underfunded would occasionally have startling breakthroughs....
On a more specific level, it's possible to test formalized versions of cost-benefit analysis as a planning methodology. In some setting (company? education? research?) have the experimental groups make decisions according to strict guidelines for how to quantify costs and benefits and weigh the two, and the control groups make decisions based on traditional "holistic management" where subject-matter experts have wide latitude to make recommendations, after perusing data, but not according to a strictly quantified methodology. See how their performance compares over some suitable period.
What I think this would mainly be testing is not really the philosophical question of whether we ought to weigh costs and benefits, but whether the erroneously-captured and un-captured information in quantification is a bigger or smaller problem, from the perspective of decision-making, than the potential bias and unrepeatability of a subject-matter expert's case-by-case judgment. Some of my skepticism of quantitative cost-benefit analysis is that it's basically GIGO: many quantifications are so inaccurate as to actually make quantitative cost-benefit analysis worse than "expert judgment" at decision-making, which is one reason we can't just replace all decision-makers with computers applying mechanical decision-theoretic rules yet, except in specific areas (like some kinds of medical diagnosis) where we actually do have very good data. But it's not impossible to test in a particular setting whether that's true.
And here we are discussing on Y Combinator's website, who are in a perfect position to run such an experiment! They could choose some companies to fund via the traditional method, "Paul Graham & friends' expert judgment on how to allocate money, based on their history and experience", and another portion of the pool on a more quantitative basis. Then see which pool is more successful.
I think that you're quite correct about GIGO for CBAs, and I think the issue is that it's almost impossible to not get GI.
But certainly such an experiment could be run. The issue there is: what's the best alternative? Randomly assign money? Go with "gut feeling"? Assign it evenly? It looks like trying to outguess the stock market...
No, I know this because I know a) casinos use statistics and b) casinos actually exist.
Look at the desperate efforts you're having to make to try and justify a cost-benefit analysis. You're questioning the validity of tried and tested statistical methods! Why? Shouldn't a CBA be easier to justify if it's so clearly a good idea that should be applied to everything as you stated earlier?
You previously asserted a counterfactual: "If statistics didn't work, casinos wouldn't work."
This is not the same claim as "a) casinos use statistics and b) casinos actually exist." If you want invalid post-hoc justifications of that nature, then here you go: profitable corporations use cost/benefit analysis to determine which projects to run, and they remain profitable.
But the fact of the matter is that you can't prove that statistics works. How would you do it? Compare a sample of casinos that use statistics to a sample who don't? Then run a statistical test to measure the difference, thereby engaging in circular reasoning?
Like it or not, this is a fundamental problem of epistemiology. Even if you want to compare two epistemiologies empirically, you still need to accept an underlying meta-epistemiology to have a method of comparison.
No, it's not. I'm quite happy to use a recursive definition for a truth value for statistics because it reduces to counting. I'll accept counting as an axiom.
Show me one casino that ignores statistics and remains profitable. Try setting one up yourself.
And, in fact, if you'd said that someone had run a CBA on CBAs, I'd have accepted that as your answer. You couldn't even do that though. You seem determined to turn it into a philosophy of science debate.
So, in short, you like CBAs and that's reason enough to insist on them for everything. No evidence required. And if someone asks for evidence, look to the sky and ask "what is truth?" I see.
And, in fact, if you'd said that someone had run a CBA on CBAs, I'd have accepted that as your answer.
I'm not "determined to turn it into a philosophy of science debate". It became one the moment you questioned computing (benefits - costs) and making decisions based on that.
And we will never come to agreement, given the fact that your meta-epistemiology accepts circular reasoning and mine doesn't. I'm sure if you can find someone else out there who has considered the issue and accepts circular reasoning, you can probably find some justification that you'll accept for CBAs, statistics, controlled experiments and all the rest of it. I suspect you'll have trouble finding this because your meta-epistemiology is rather unique, to say the least.
As for your hypothetical casino, I've never rejected statistics. It's circular reasoning (which you have repeatedly appealed to) that I reject. I don't actually have a meta-epistemiology at all, I simply accept the standard scientific epistemiology and move on with my life.
(Incidentally, if you accept subtraction in addition to counting, together with the idea that costs and benefits are commensurate, you've already justified cost/benefit analysis.)
Recursive definitions are hardly unique. If you have a problem with them, perhaps you don't understand them properly.
Your last statement is false. There doesn't seem much point in continuing this. You've already stated you have no evidence that CBAs work and that's clear enough.
If cost-benefit analyses didn't work, government projects would often go massively over budget producing nothing, while other areas that were underfunded would occasionally have startling breakthroughs....