Having a dedicated line to travel on helps and preference at traffic lights helps a bunch. I think you underestimate the amount of delays buses have to deal with when they also have to be in traffic.
Both trolleys and buses can have dedicated lines of travel or you can force them to share their lanes with other traffic. And both both buses and trolleys can be given light priority or they can wait with everybody else. The issues you're raising are totally orthogonal to bus versus trolley. Cities with good bus or trolley systems do these and cities with worse but cheaper systems don't. And another good way to speed up buses or trolleys is not forcing them to pull out of traffic when they stop to pick up customers by having special islands or having the curb extend into what would otherwise be parking spaces.
It probably does depend on the infrastructure choices the city implements. I don't see a lot of buses systems that have isolated islands, dedicated lanes, etc. I do see them frequently for street trolleys.
