The CLA protects users of the software from contributors coming back later and saying: "Hey I own that code and now you need to pay me!" Apache projects in general rely on CLA documents.
It's totally in the right of the contributor to refuse to sign this document. But don't expect your contributions to be applied, or work on any Apache licensed projects :)
I am not a lawyer, but I'm pretty sure that how non-cla open source projects handle it also protects them from that eventuality.
All contributors are required to say "This code is released under X license" which prevents the person from later coming back and asking for money for them to use the code.
Edit: Assuming my above comment is correct, what this means is that the only real point of a CLA that I'm aware of is to allow the owner of the CLA to change the license on code without asking the contributors.
I think the point is that if a project is big enough, it will inevitably go to court at some point (probably at many points). Having a CLA will make this a lot less expensive for the project maintainers.
Because the Contributors, in advance, have already agreed to confer a certain set of IP rights to the mainter and manager with respect to the code they've contribute. This means there's less questions during litigation about the rights to the code the maintainer holds and prevents the maintainer from having to track down individual contributors during a lawsuit.
It's totally in the right of the contributor to refuse to sign this document. But don't expect your contributions to be applied, or work on any Apache licensed projects :)