I think it's a matter of unnecessary friction. People traveling to Japan to see the Olympics might not bother to get out of their comfort zone, leave the hotel and see something interesting if the maps seem to discourage them from doing so.

>Maybe you add an explanation to the map rather than changing the symbol.

That's a fair compromise, but now you have to explain it in every possible language that a foreigner might use. Which is probably why you considered changing the symbol in the first place, because well designed icons theoretically speak to everyone.