When humans are wrong, the business’ “ego” can be saved by blaming the employee who made the call, sometimes firing them. But the process goes on with the same error rate.
But when software makes the wrong call, it feels like the business itself has done the wrong thing. With no way to externalize the blame for the decision, the blame gets placed on the decision to use ml in the first place.
This.
This is strictly more accurate than the above comment.
It has nothing to do with accuracy or the repetitiveness of the mistake. It has to do with managers preferring to make sure that they can somehow avoid being personally blamed for their subordinate's mistakes, because most companies are designed to absolve everyone at the top and blame everyone at the bottom.
I've worked in an industry (prop trading) where the costs of mistakes were real, and it is the most heavily automated industry there is. I've also worked in another part of the finance industry (hedge funds) where mistakes are paid for in other people's money, and it's super obvious that managers there are way more interested in passing off blame than fixing problems relative to how interested they are in doing these things in prop trading, and there is correspondingly less automation.
It's not about people in the population at large believing that machines are more prone to repeating mistakes than humans. Look at how many people are excited about the concept of self driving cars. It's about managers trying to convince themselves that this is true, so they can go on being unethical.
> It has to do with managers preferring to make sure that they can somehow avoid being personally blamed for their subordinate's mistakes
You can't fire a crappy AI and replace it with one with better judgement. If it does something weird or bad or inexplicable, you can't ask why it did that and tell it to how to be better in the future. Blame is a tool that organizations and people; you can't blame AI because you mostly can't teach AI.
When humans are wrong, the business’ “ego” can be saved by blaming the employee who made the call, sometimes firing them. But the process goes on with the same error rate.
But when software makes the wrong call, it feels like the business itself has done the wrong thing. With no way to externalize the blame for the decision, the blame gets placed on the decision to use ml in the first place.