It comes down to how much you trust the FSF. If Stallman was finally booted and the new leadership decided to make AGPLv4 which was basically LGPL, you'd have a much more complicated legal battle if you wanted to stop someone using your software under those terms, as you're not arguing intent and reliance and other subjective areas instead of relatively simple copyright infringement.

At the same time if you go AGPLv3 only, accept external AGPLv3 only contributions and then FSF releases a AGPLv4 that you feel matches your intent better, then you'd have to do a complicated relicensing process requiring you to get consent from all contributors or rewrite/remove their contributions.

The third option is AGPL + CLA but plenty of people are rightfully suspicious of this now many companies have used this to close software moving to source available licenses like SSPL or just flat out proprietary like Emby.

There is also an issue of hostile-ish fork. If you release something as (A)GPLv3-or-later, anyone can take it, and extend/change it. However, those changes can be done under (A)GPLv3-only. That would prevent you from incorporating them into your version (unless you also re-license to (A)GPLv3-only), while they still could merge anything you add.

In general (A)GPLs care about user freedom, not a developer freedom. So this is not seen as a flaw. I personally prefer using -only variants, since it seems to strike reasonable balance in this regard.