No, if 10 predictions each have a 10% chance of a false positive, you will always have on average 1 false positive. Whether you test them on the same data or not doesn't matter (unless 10 predictions test the same thing, of course.)
What you need is a control sample that you know should be negative, so you can actually measure the false positive rate. (But with a sufficiently large base sample, you can look for correlations in small subsamples and use the whole sample as a control.)
> No, if 10 predictions each have a 10% chance of a false positive, you will always have on average 1 false positive. Whether you test them on the same data or not doesn't matter
That's true.
I should have said it differently. In fact, I'm not even sure that my understanding is correct.
The problem with not using new data to test each new prediction is, that if a scientist wants to show A on data X, but data X doesn't confirm A, the scientist modifies A slightly and now tests A' on X, which is again rejected, and then modifies it again, testing A'' on X, and so on... until the data X actually confirms hypothesis A'''''''!
That's the real problem - using data without a predefined plan for how you will use this data. In the above example, the data that you collected affected your decision-making process, so your results are not independent of the data (and thus not replicable!).
What you need is a control sample that you know should be negative, so you can actually measure the false positive rate. (But with a sufficiently large base sample, you can look for correlations in small subsamples and use the whole sample as a control.)