>If enough people vote against them, that pretty much cuts them out of the picture.

No, they will become the lobbyists, consultants etc. There are so many openings for former congresspersons that they don't need to fear unemployment.

So it's not like there is a strong fear of being voted out, if you can grab even better paid job(for most, not fore all).