This isn't quite what most people think. Prior art still invalidates patents. "First to file" is merely the method now used to deal with two or more parties claiming to have invented a thing in secret who show up at the patent office more or less at the same time. Before, there used to be a trial type thing where the inventors presented lab notebooks and evidence to try to prove who was first. It was lengthy and expensive for all involved.

First to file does not mean first to pluck some heretofore unpatented but well known already used technology from the field and run to the patent office.

That's not the way it works. First to file vs. invent has to do with two independent inventors creating something and trying to file. If something has already been invented and is publicly available (patented or not), that's prior art and can invalidate the patent.