Then the question becomes, how do you know who to trust ? Are you supposed to also vet the id-hashes of the signers ?

I guess you will trust coworkers, people you know irl, well known developers, companies you pay for review subscriptions. You don't have to trust all of them 100%if you require multiple reviews per dependency.