Yes. There is no way on Android to give fine-grained directory-based access permissions (unfortunately). So all SD card permitted apps can read the SD card globally.
Since most desktops/PCs/Macs are now always online, why is nobody up in arms that any application has access to your home directory, your registry (HKCU at very least) and your "My Documents" directory either?
This is basically what Android apps have access to when requesting STORAGE permission.
No, it's quite true. If you're running Android prior to 4.1, any app can read anything from your SD card. Starting in 4.1, apps require READ_EXTERNAL_STORAGE to be able to read from the card (so user has to grant app this permission upon installation)[1].
Have you never noticed this when you grant apps permissions? All of this is clearly written on the permissions list -- READ_EXTERNAL_STORAGE is explained there in layman's terms (as in, "this app will be able to read and write files on your SD card" or something similar).
And now, in KitKat, there have been some major changes in how the SD card can be accessed by apps, which I don't fully understand (never had to develop for KitKat only). But only a small percentage of users have KitKat installed.
If you're really storing sensitive things on your SD card, you should probably look into using some type of app for encrypted file storage (there are many on the market).