It is not my understanding that SSH will attempt to use only the file identified by IdentityFile, and yes, that is surprising now isn't it. If the server does not cooperate with you, your proposed configuration will give the server all of the keys.

Edit to add: While I was testing this understanding (which is correct) mioelnir's comment added the setting you need to get the behavior which I thought was automatic.