Recalling congressional representatives is probably not allowed. AFAICT, however, there has never been a federal court ruling on the (likely) exclusive right of congress to expel members.
The closest I could find was Burton v. U.S., where the court declined to rule, since the law in question didn't apply to senators at the time.
"Our legally elected representative directly refuses to represent us" should be plenty of grounds.