The vast majority of Jewish people who died in the Holocaust were not elites - however, it is true that German resentment against rich and influential Jewish people in politics, business, the media, and arts played a huge role in leading to the Holocaust. Anti-Jewish sentiment in pre-WW2 Germany was not very different from anti-Jewish sentiment today. Already back then, as still is the case now, antisemites tended to target their ire against Jewish people's disproportionate influence over politics, the economy, media, the arts and so on. The fact that most Jewish people are not rich and influential does not dissuade those types. For many or most antisemites, resentment against rich and influential Jewish people soon swells into a generalized resentment against almost all or all Jewish people. So, for example, the Nazis ended up killing huge numbers of poor Eastern European Jews who could not be described in any way, shape, or form as elites. But this does not mean that the resentment did not originally largely stem from resentment against well-off and influential Jewish people.

In my experience, most antisemites are fairly stupid* and it is hard for them to understand the logic which says "just because a small subset of a certain ethnic/cultural group has disproportionate influence over the society in which they live does not mean that every member of that ethnic/cultural group has disproportionate influence over that society".

This fallacy is sort of like thinking that, because most NBA players are African-American, it means that most or all African-Americans must be really good at basketball. Rather obviously flawed logic.

*Certainly not all, though - the more intelligent ones probably contribute the bulk of antisemitic literature.