I will very occasionally (every few months) wear a shirt and tie to work. Almost without fail, I get pulled into a manager's office after lunch asking me if I'm interviewing and looking to leave.
How about just wearing your normal work cloths to the interview? Places that will ping you for not wearing a suit probably haven't made the best hiring decisions in the past.