When I look at APL, it seems a bit clearer where this all is coming from. After all, we're perfectly fine with using an asterisk as infix notation for multiplication, even if all the rest is done with functions, methods etc.. This isn't just the case because we've learned mathematical notation in school and thus "x <asterisk> y" seems natural enough (as opposed to e.g. multiply(x,y)), but also that we learned to accept the "asterisk" as a substitute for the middle position dot or the cross. The same is true for division or especially weird stuff like logical-or...
J works along the same lines: First, you had to accept a "widened" set of mathematical operators with arrays as the core. Which is APL with its funky fonts and keyboards. And J then takes away your more readable, but inconvenient to type notation and replaces it with ASCII. It's a bit like learning calculus with lisp as your first notation...
That's the syntactical hurdle. Then you've also got to get used to the array-centric programming model. Just like stacks for forth, functions for functional programming or lists and recursion for lisp.
As opposed to lisp, I think the syntax is the bigger issue here for a change. You don't just have to change the order in which you read things and ignore some parens, you've got to get over some very deeply ingrained habits, not just from other programming languages, but from normal writing itself. No scanning, neither horizontally nor vertically. The closest analogy for me were regexes, where you can't just skip ahead, either.
Take that with a grain of salt, I'm just trying to get into it myself. Two things that I might want to try out to really grok it are programming in that style in other languages (like the infamous Whitney C example) to adjust my reading style without having to adjust to other paradigms at the same time, and to edit APL with ed, so that focusing on a single line becomes potentially easier.
edit: replaced asterisk with, well, "asterisk". How does one escape that in HNews? backslash didn't work.