Perhaps I'm really being thick, but I take the answer as: if matter falls directly down onto a neutron star, that infalling mass doesn't add any spin.
If that mass comes down at an angle to the star (and it's swirling, right?), the kinetic energy gives a sideways push where it hits, like someone brushing the side of a football - it adds some rotational energy.
But that seems a trivial answer so I guess you're asking something deeper?
I may be entirely wrong, but I don't think the analogy holds, as friction is what makes the football spin.
Maybe a better one would be a spinning ice skater: going from wide arms to closed arms increases angular velocity.
Indeed consider the ingress mass and the neutron star as a single system, ingress mass gets closer (as the ice skater arms do) and angular velocity increases. Collapse is what makes it spin faster.
If that mass comes down at an angle to the star (and it's swirling, right?), the kinetic energy gives a sideways push where it hits, like someone brushing the side of a football - it adds some rotational energy.
But that seems a trivial answer so I guess you're asking something deeper?