Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
johnx123-up
on Oct 26, 2012
|
parent
|
context
|
favorite
| on:
80 terabytes of archived web crawl data available ...
Once they acquire old domains, some websites start blocking archive.org through robots.txt. It would be better if there's any elegant solution for this problem.
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search: