Chinese would definitely be easier without the writing system (and I wish my two years of studying it had been verbal and pinyin only). But it's also tonal, which speakers of non-tonal languages generally have a really hard time with.
> which speakers of non-tonal languages generally have a really hard time with.
Is this actually true? I only did some basic Mandarin, but its tonal system is so simple (and plainly transliterated) that it seemed like a non-issue to me. Done & dusted in a couple of hours or so, compared to the hundreds of hours needed for vocabulary, idioms etc. Not to mention the terrifying writing system. And I have no natural advantages, carrying a common disability (native monolingual English), and having only learned German previously.
But English is a tonal language; that hypothetical problem can't be a source of extreme difficulty for English-speaking learners.
(I do have problems with English tones interfering with my attempts to speak Chinese, but that's really not the same problem as "My language doesn't use tones, so I don't know how to produce them". I do know how; I just don't have good control over them.)
English doesn't have falling-rising contours like the Mandarin third tone, so there's at least the difficulty of having to learn a new category of pitch change. I do wonder whether e.g. Mandarin speakers have similar difficulty acquiring the different contours of e.g. Cantonese. Intuitively, I'd expect it to get easier the more different categories one can already distinguish.