English doesn't have falling-rising contours like the Mandarin third tone, so there's at least the difficulty of having to learn a new category of pitch change. I do wonder whether e.g. Mandarin speakers have similar difficulty acquiring the different contours of e.g. Cantonese. Intuitively, I'd expect it to get easier the more different categories one can already distinguish.