"Aren't you just making statistical inference based on past experience?"

Yes, but if your behavior in the past was also biased, going off of arrest records or something like that won't get you better / less biased results.