So I should pay the license fee to keep the quality of content I don't watch high? If the BBC's value proposition is so good, it will have no issue convincing the public of that.
You could hold a similar debate on every service the government currently provides, and you would essentially end up with toll roads everywhere and parents having to pay exorbitant amounts to educate their children properly.
The BBC is similar to these things in the sense that it provides good value for the taxpayer, and contributes quite significantly to society (even if it doesn't directly effect you in the sense of you consuming their content regularly).