That's exactly what it is, for me at least. The slightest noise of something other than nature is enough to throw me completely, the best I can do living in a city is to work in an environment where the background noise has a masking effect. It's either no noise or a constant background drone loud enough to mask any single interrupting sound - anything in between is unworkable for me.
As much as I love the mountains and forests, I've come to realise that I like the idea of solitude a lot more than the reality of it. And so I find myself living in large cities, constantly seeking noise because it's the only way I can find peace.