You're suggesting that developers can't be users? Also, end users don't have to be the ones to fix things to benefit. If they're unhappy with something, they would have the freedom to pay a developer to customize their software however they like, or even get a skilled friend to work on it.

Being able to work on my own car is an important freedom, even if I don't know anything about cars. I can get a knowledgeable friend to look at it, or I can hire a mechanic, and it doesn't have to be a mechanic from the manufacturer.

Also, it doesn't matter how much of the population uses their freedom for it to be important.