And that made sense, as there was a point in time that Russia did seem like it had a chance of becoming a normal democracy. At some point even the idea of the EU membership was floating around.
By the 2008 attack on Georgia it was clear that there is no democratisation of Russia, but some people didn't want to believe it for a long time, not even after 2014 attack on Ukraine.
EU membership was never feasible. Russia is too large population-wise, it would have threatened franco-german leadership of the EU. The EU, as it was back then was hanging in a delicate balance, where France and Germany usually had to agree on something to get things done, but other countries could form blocks of convenience to push their own demands through (eg. UK, Nordics and the Netherlands on fiscal discipline, or the Baltics, Visegrad and countries from the Balcans on immigration). France and Germany would not have wanted to lose that much influence, Poland would not have wanted to be between Russia and Germany again (politically speaking), and hatred of Russia runs rather deep in countries of its former empire.
The US has had normal relations with places as radical as Saudi Arabia for many decades. Ideological, political, and other differences do not preclude normal relations. There's really an absolutely phenomenal article on the deterioration of Russia-US relations here. [1] In general, the problem is that after the collapse of the USSR, the US was left as the defacto ruler of the world. And we wanted to cling onto that position permanently. Germany, for example, is a country that could be independently great but has made no efforts towards such and has largely been content to remain deferential to the US, so it retains extremely positive relations with the US, so far as such a relationship can be called positive.
But as Russia started to regain strength in the early 2000s, they specifically aimed for positive relations with the US, but also were not happy with a Germany style relationship and wanted to be treated as equals. This led to us doubling down on hostilities towards them. But this deterioration of relations inevitably led to where we are today, but fortunately not where we could have ended up - which is in the nuclear wasteland that was briefly called WW3.
This also ties right back in to Georgia. Back in 2008 at the Bucharest summit the US was openly encouraging and supportive of Georgia's efforts to join NATO. France and Germany were strongly opposed to such, arguing that such a move would needlessly provoke Russia, but we aimed to move ahead with it anyhow. The Georgia-Russia war would start a few months later.
By the 2008 attack on Georgia it was clear that there is no democratisation of Russia, but some people didn't want to believe it for a long time, not even after 2014 attack on Ukraine.