Obviously your ancestors aren't British so there's something to the "where are you really from" question. I've been asked similar things even as a White American.
As far as I understand it from Americans I know, quite a few people in the US have a fascination with where their, and others', ancestors came from, so it could be a fairly innocuous question. Pretty much any American I know would happily rattle off an answer of 'oh, I'm half Irish, a quarter French, and a quarter Spanish' or something similar.
Here in the UK, even though we're also a nation of immigrants and invaders throughout history, there is much more of an attitude of 'people whose ancestors have always been here' ('British') versus 'immigrants' ('not really British') - even if the 'immigrants' are from families who have also been here for generations.
So yeah. Generally 'but where are you really from?' means something a little like 'I don't believe that you're one of us people who have always been here'. Whether that's malicious or just naive depends on the person, but I can imagine it gets pretty tiring.
(I'm the son of a Chinese immigrant, but due to my other parent being Scottish I pass for white and thus I've never actually been asked the question!)
"Where are you from" means the same in the US (source: moved from UK to US 25 years ago). Someone who is interested in your ethnic background would ask "what's your your heritage" or perhaps "what are you?". "Where are you from?" is always going to mean Where did you personally physically come from.
When I have given a shy smile and an interested "What is your ancestry?" followed by, if sincere, "You have a really cool look!", I have often gotten very interesting answers and nearly never any offense. Once someone I asked had 8 heritages bestowed by 8 great grand parents, including Sephardic, Chinese, Cree, African-American and Swedish!
I'll ask "Where are you from?" as a way to understand someone's heritage only if it's obvious, from accent or other markers, that they are literally from another country.
Ancestry is pretty common small talk for White Americans.
I like learning more about people and the world so I like to ask people about where they’re from. The immigrants I’ve spoken to are proud of their heritage and don’t take offense. I have noticed though that their descendants can take offense, perhaps because they take it as implying they’re somehow inauthentic. All that’s required is a little delicacy. Instead of asking “where are you really from” instead ask something like “it sounds like your family has a really interesting story, would you like to tell me about it?”
Sure, but imagine the scenario that you're one of about 5 non-white people in your entire school life, and the area you lived in has a similar demographic. You might feel generally more alienated from society. It might make it slightly harder to accept/understand your own identity, and being quizzed about it further (at times when you've already mentioned your nationality) is unlikely to help.
No, it isn't phrased the same way. If you have an ethnic-sounding last name but don't have a foreign accent you might be asked about your ancestry, but they won't say "where are you REALLY from" with the stress on the word "really", the way it's said to, say, someone with Asian ancestry, born in the US, who answers "I'm from Chicago".
