Reddit serves a different robots.txt to Googlebot, you can see a snippet of it in Googles summary. If Kagi is getting recent Reddit results then they must be either ignoring robots.txt, using Google as a backend, or also paying Reddit for access like Google.
Bing and DuckDuckGo are certainly still locked out, I just tried searching for "reddit hurricane milton" on both and none of the results are actually from Reddit.
https://www.reddit.com/robots.txt
Reddit serves a different robots.txt to Googlebot, you can see a snippet of it in Googles summary. If Kagi is getting recent Reddit results then they must be either ignoring robots.txt, using Google as a backend, or also paying Reddit for access like Google.
Bing and DuckDuckGo are certainly still locked out, I just tried searching for "reddit hurricane milton" on both and none of the results are actually from Reddit.