I'm exactly the same! I've recently started to think that I don't actually have a minds eye. I can't hold an image in my head for more than a fraction of second - and even then, it's just parts of the image at a time (not a whole scene, and not in realistic color). The memory-castle type techniques seem to be visually "begging the question" - to remember things, first... remember things!
If you can see it for even a fraction of a second, you do have a minds eye; those with aphantasia can't see anything at all for any length of time. If you can see anything at all, you can improve that with practice.
