The right wing moving righter? Hardly. It’s easy to see that politics has shifted drastically left. Political stances considered mainstream 30 years ago would end careers today. The so called “extreme right” Tory government, before the last reshuffle, had 3 of the 4 top government posts occupied by MPs from minority ethnicities. What you consider to be ‘moving righter’ is in fact just louder complaining due to the accelerating movement towards the left.
I think you're focusing on racism to the exclusion of all else.
Britain has sold off vast swathes of housing stock and is privatising ever more healthcare delivery. Zero-hour contracts are a thing nowadays. Industries that most countries run as a public service have been privatised.
Collectivism has been replaced with neoliberalism. Our country has swung miles to the right both socially and fiscally. But yes, you can be out and brown.