I continue to want a communal set of 50-100 million urls and data that are "good" (i.e. for any value of good and more complete than common crawl) that are accessible enough to be easy to work with that can be used to experiment with different web search tech. There are enough separate problems to tackle in web search that breaking it down would maybe move the needle. We have lots of kaggle competitions about ranking, but using closed data. What other types of kaggle projects would help web search?

What can we do to foster a sustainable bazaar of projects to make it easier to build web search engines?