Sure, but as long as you change the filter every 2-3 months and keep it in the fridge there's nothing to worry about.

And if you leave almost anything out at room temperature for weeks kept moist you're going to see gross growth. It's nothing specific to a filter.