Only time I've seen difficulty with pointers in the abstract is when helping people in intro programming classes in C. Once you're past that level, programmers get it, but they get confused by what the code is actually saying.