The fact that you can retroactively opt out of it also makes it particularly odd, compared to previous media. If you change your mind about publishing a book, you can't unpublish existing copies, and chances are that some library will still have a copy that they continue to make available to the public (a handful of people in history have succeeded in suppressing already-published books, but it's hard). But if you decide you want to unpublish a webpage, you can actually get it excised from all the archives available to the public as well, erasing it from the historical record, even if it had been available for years and read by thousands--- because, unlike with books, typically none of those thousands of readers will have a copy that they can claim ownership to.
Even worse is that others can retroactively opt out on another's behalf. If someone owned the domain with no exclusion, then their content seemingly disappeared in 2003 because they lost ownership/stopped caring about the domain, and the new owner puts up a robots.txt, the original owner's content will not be publicly accessible.
At least that was the case a few years ago. I know the Wayback Machine has seen some changes in the last year.
