I don't think it's uncommon. It gets even harder when trying to figure out relative left-right to other people.
Here's a tip I learned once: hold your hands out, palms facing away from you and fingers up. Point your thumbs at each other and index fingers straight up. The left hand will form an "L" (for left) and the right hand will form a backwards "L".
I found this impossible throughout my childhood. One day it was suddenly easy.
I think, between the two moments, I had started playing a lot of top-down 3/4-view RPG video games. There's something about having an avatar that has a directional "facing" on a fixed-orientation orthogonal grid (picture a chessboard, where the piece has a face)—and where commanding the avatar with a direction button both moves it and turns it to face the same direction—which seems like it acted to train my brain to reflexively use this "model" whenever trying to figure out a directional visualization. I'd picture myself in the third person, give myself an "origin point", and then visualize where I'd be (in both movement and facing) before and after, relative to the origin.
(All this was without actually 'picturing' anything, in the traditional sense. Imagine having the sensation of manipulating a ferromagnetic cube floating inside a strong electromagnetic flux. The cube interacts differently with the surrounding magnetic field depending on its 3D orientation—it'll feel different when you're trying to add yaw if the "south" side of the cube is the one facing you, vs. the "east" side or the "bottom" side. Now imagine that the cube is a feeling in your proprioceptive system, rather than in your hands. Now imagine that there is no cube, only the feeling.)
One interesting effect of this is that I've found that I now get "acclimated" to cities I live in, and begin to have a very strong sense of where north is in those cities—because, at first, I look at Google Maps on my phone a lot while navigating, and so my "origin point" becomes fixed to North, such that I begin to see the same sort of fixed-orientation RPG grid overlaid on the city, with streets running "up-down" instead of north-south and "left-right" instead of east-west. I will actually have an intuition of "I'm facing up" when I'm facing north.
On the other hand, when I'm in a city I haven't acclimated to, man do I miss that sensation; I get completely lost.
To this day, that's how I deal with left-right. I no longer have to physically do it very often, but instead reason to myself about which hand would form an L.
Here's a tip I learned once: hold your hands out, palms facing away from you and fingers up. Point your thumbs at each other and index fingers straight up. The left hand will form an "L" (for left) and the right hand will form a backwards "L".