Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

There is no reason at all to believe this. Search engines provide a huge benefit to sites which is why they let search crawlers access them.

Even today anyone who wants to opt out of being in the Google (or MSFT, or whoever) search engine can trivially do so by updating their robots.txt. Very very few sites choose to do so.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: