I think you found the culprit. Syntax highlighters are notoriously slow.

You have every right to have high expectations for performance with Vim