You got that wrong. It depends if you need to be a citizen or not.

I've personally known permanent residents to work in government contracting.

For your case, if I had to guess, it's probably the physical building/facility they required you to work at had restrictions on unescorted non-citizens.