Can we predict things super accurately? Often no. But you know what’s better than anecdotes about times predictions were bad? Training and testing sets to judge how good we expect models and predictions to be from the beginning. Because a lot of these are not high confidence predictions.
And no, it’s not “black swans”. Are those a thing? Sure. It’s ok that models can’t account for things that are not modeled or seen before. But if these things are common enough that they’re systemic and the mode is just not actually accommodating for the world of relevant factors, then it’s not going to have been a good model on the test set to begin with. And we would know that.
Can we predict things super accurately? Often no. But you know what’s better than anecdotes about times predictions were bad? Training and testing sets to judge how good we expect models and predictions to be from the beginning. Because a lot of these are not high confidence predictions.
And no, it’s not “black swans”. Are those a thing? Sure. It’s ok that models can’t account for things that are not modeled or seen before. But if these things are common enough that they’re systemic and the mode is just not actually accommodating for the world of relevant factors, then it’s not going to have been a good model on the test set to begin with. And we would know that.