Which ISPs are so bad that you want to use external services, which are further in distance than your ISP, for speed? When I test with my ISP, they beat all of these services (both IPv4 and IPv6). They're simply closer to me in terms of hops.
My router is another story though. The Fritzbox (>200eur router) adds 6ms of latency, and that's what is advertised over DHCP. (Might still be fine, since cached queries are faster than the ping time to the ISP.) Note that my tests were all with uncached queries (random subdomains of a domain), so it always had to go out and ask an external server (though it could cache the NS record for the domain).
My isp got the brilliant idea of rolling their own YouTube cache servers. It's great in theory but in or active they're under powered and so at peak hours I can't even stream 240p on my 500mbits connection. I've had to block their cache servers in my firewall for YouTube to be butter smooth at 1080p consistently.
Another example is bell Canada who used to mine your DNS queries to profile you for ads, or ISPs that high jack the nxdomain result to send you sponsored results of vaguely similar sounding websites.
It is common for ISP to host instances of the Google Global Cache (GGC, see https://peering.google.com/) which are used for many Google services, most importantly YouTube.
In fact, in many cases Google itself "suggests" to ISP that they host a few GGC servers.
They are directly monitored by Google, and the ISP has basically no say in how they are run. Capacity is managed by Google directly.
Yep, Virgin Media did this in the UK and messed it up badly. Every day at 6pm YouTube would stop working until the following morning.
It appeared to work by inspecting DNS packets and replying with overrides if necessary. I didn’t like it but I could understand that.
What I did not agree with was the fact that this also happened for other DNS services. Google DNS and OpenDNS both experienced the same issue, as did a few other “famous” DNS providers. Random little ones wouldn’t return the caching servers, and also enabling encrypted DNS for Google/OpenDNS would stop it happening too. I’m fairly sure it was some badly thought out deep packet inspection.
Again, it's probably _not_ Virgin's fault or responsibility. Capacity planning is handled by Google directly.
Also, I think that virtually every ISP with more than a few tens of thousands users is hosting a GGC instance nowadays (and a Netflix OpenCache, etc. etc.).
Nowadays, the vast majority of the transit&peering of ISPs is not going to the Internet, but to a few racks of local caching servers managed by OTT operators.
Bandwidth-wise, at least in prime time, the Internet is much less connected/realtime than people think :)
I would suggest that the deep packet inspection is Virgin's fault. Perhaps it was Google under provisioning the CDN inside VM's network, but I would suggest that's also probably Virgin's fault in part, as they will likely have more network information available to them that may have suggested network congestion, but that was not acted upon.
As someone else suggested, this is likely part of our GGC program. If you can give me info on which ISP + Geographic region you're in, I can take a look and see if there's anything in our logs to indicate a problems; you can email me details at myusernamehere at google dot com with details, since I don't routinely check Hacker News.
If you can reproduce a bad experience, and right click on the player, click "Get Debug Info", and share that result, it's the most helpful thing for us to dig into problems.
The YouTube cache is probably there not to increase performance for you, but to reduce ISP cost (ISP pays for all data received from outside of their network)
Even for data not delivered via their cache they'll get free peering with Google. But having cache servers at several locations reduces the capacity they need to peering points. E.g. in the UK, most traffic gets peered in London so having cache servers in Northern England reduces the bandwidth they need for their internal network within the country.
I don't think ISPs pay much for peering in Europe.
When there's a competition this often equates with making customers happy so they will stick with the ISP and perhaps more will switch from competitors.
When there's no competition and customers have no choice who to use it's purely all about squeezing as much money as they can. This means raising prices for the service and reducing cost by paying less to others services even if it would reduce quality of service.
The goal is to have service that's good enough to keep people still use them (I mean there is price point and/or quality that people would just not to have service at all).
If a specific action that ISP with monopoly does and it improves quality for customer that's just a coincidence.
My own tiny neighborhood ISP thinks they are smart and have their own CDN for caching everything under the sun. Of course it doesn't work well, not only it caches DNS and HTTP requests like kids collecting candy in Halloween but it also breaks HTTPS sometimes.
As an anecdote of how bad ISP caching can hurt you, I am a web developer and I was debugging an issue in a legacy system. I had fixed the stuff and deployed a new version. Erased all the caches in my machine, refreshed the page, broken. I thought that maybe I fixed the wrong thing and started fixing more... now rinse and repeat over five hours before you realize that the cache is happening at ISP level and that things are fine in the live version on the real internet. This ISP is called PredialNet, web developers from the city call it PredialCache...
I know this is about DNS and not HTTP but their cache servers will also cache DNS requests, so moving some domain to a new machine and IP is also a quest under their network. For a while I paid PIA so that I could side-step their cache servers.
Sometimes it's not even at the ISP level. I experienced a similar adventure a few months ago. I turns out that my cellular hotspot caches PDFs. It's not even at the ISP level, it's in the little box in my pocket.
All ISPs in Denmark are required to implement a filtering list of somewhat arbitrarily chosen websites, including a number of torrent sites, illegal pornography and probably others. There is a very reasonably fear that this could be used for political purposes.
This filter list is implemented through DNS, making third-party DNS services the most practical workaround.
One of the issues is that many ISPs don't actually run their own recursive DNS servers... They outsource to companies which provide a "monetized solution," operating the recursive resolvers and paying the ISP in exchange for the customers' data, which they sell. This will become an illicit activity shortly in Europe, as the GDPR comes into effect, but there are many ISPs that have become very used to receiving those payments. And most of them already don't admit to selling their customers' privacy.
So privacy is why many of the people who use a different recursive nameserver are doing so... In order for that to work, you need to use one of the ones which support link-level encryption, like Quad9 (DNS-over-TLS) or OpenDNS (DNScrypt). Other reasons are performance (which at least in Quad9's case should be no worse than your ISP for real-world queries, since it's back-to-back with many of the authoritative servers already, and all of them will give you a large pool of other users sharing the same cache, which helps performance immensely), and security... Going back-to-back with the authoritative servers also collapses the attack surface between them, minimizing the possibility for MITM attacks between the recursive and authoritative servers.
The ones that cooperate with state-sanctioned censorship? Malaysia blocks anti-government news sites via ISP DNS entries; using a 3rd party DNS bypasses the block easily.
Accusing government officials of mismanagement or outright theft of public funds is an easy way to get your domain black holed over here. That, piracy, and porn. Though I wouldn't be surprised if the latter two were thrown in solely to make the program look more legit.
Perhaps most ridiculously, Medium.com is blocked for simply hosting an article posted by a censored website. Yes, you read that right; they banned an entire domain, the vast majority of which covers nothing even remotely related to Malaysia, over a single article.
I try to be grateful they are as inept at censorship as they are oversensitive to criticism.
It’s best to just pretend those DNS things are a good way for bureaucrats to achieve whatever it is because they’re so easy to circumvent and the alternative is far worse.
Brighthouse did it before they became Spectrum. They did allow me to turn it off though. I've heard that this is still the case with Spectrum, but I haven't tested.
CenturyLink, major telecom in 37 US states. DNS requests for all nonexistent domains go to a server that delivers a dumb "search" page to web requests. They're currently the sole fiber-to-the-home provider in my neighborhood, and Comcast is the only broadband alternative.
Time Warner/Spectrum, the largest provider in NYC (and the only one that services my apt building) does this. They let you turn it off in your account settings, but it doesn't actually turn it off.
I live near Toronto Canada and Rogers my ISP hijack's DNS requests for non existent domains. The data to back up my bold statement is a simple Google search which reveals that they have been doing this at the very least from my Google search was from 2008.
It is possible to turn off this feature, But it is set by a cookie and once that cookie is deleted you have to do it all over again.
I know you're asking in jest, but here's an honest answer: Google Fiber. Google's DNS servers are ~15ms further away than a local ISP who runs public DNS servers as well (xmission). Part of this is because google fiber has a peering connection at SLIX (slix.net), and so does xmission, whereas Google's DNS servers appear to be in the bay area.
Whether or not it's noticeably faster, I'm not sure, but I've always used xmission's dns servers without fault for the past 10 years.
--- 8.8.8.8 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 17.763/18.264/18.803/0.425 ms
--- 198.60.22.2 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 2.906/3.457/3.871/0.408 ms
I've had several ISPs in different countries that do horrible things. Not using NXDomain is one, but I've had some that return NXdomain when they shouldn't, then cache that result!
8.8.8.8 is consistent and easy to remember, and now so is 1.1.1.1
Queries to my local Spectrum DNS service are 4 times slower than 1.1.1.1, and they redirect you to stupid search pages instead of `NXDOMAIN`. I switched the whole network over this morning.
In the U.S., many of them are mining and selling our data. Getting off of their DNS service is one step in mitigating this.
Also, as others mention, they can and do monkey with the results.
In other words, here, your ISP is in part a hostile entity. At least, in my perception -- and I'm not alone.
P.S. Of course, there's the argument against giving Google all your DNS usage, as well... I use a different DNS service from a company with a good reputation that says it's not collecting usage data. Even then, and with the state of our State, who knows...
Mine's over a VPN. And if you use a VPN, make sure your DNS traffic is using it (along with some other types of traffic that can bypass it, if you're not careful).
> Which ISPs are so bad that you want to use external services, which are further in distance than your ISP, for speed?
When I upgraded my WiFi equipment at home a month ago I thought it was defective. Multiple times every day I'd seem to lose connectivity to the internet. After two weeks it occurred to me that my previous router was configured to do a couple things for me, one of which was overriding what DNS servers were assigned with DHCP. I reconfigured my new equipment to use those DNS servers and suddenly everything works normally.
Not really :( I'm looking to replace my router entirely soon anyhow, and hoping this does the trick. My hardwired devices don't seem to have this problem.
Does no one bother to run their own DNS server? It's not hard---in fact, it's easier than running an authoritative DNS server, since you are only using it for local resolving.
I ran the provided script, added my own local resolver, and after the first run, got incredibly fast results.
test1 test2 test3 test4 test5 test6 test7 test8 test9 test10 Average
cloudflare 35 ms 38 ms 39 ms 33 ms 34 ms 39 ms 28 ms 33 ms 28 ms 39 ms 34.60
cloudflare2nd 34 ms 27 ms 34 ms 34 ms 34 ms 27 ms 34 ms 28 ms 28 ms 28 ms 30.80
google 42 ms 27 ms 27 ms 40 ms 28 ms 40 ms 28 ms 40 ms 40 ms 28 ms 34.00
google2nd 42 ms 27 ms 28 ms 43 ms 41 ms 39 ms 28 ms 41 ms 41 ms 28 ms 35.80
quad9 65 ms 64 ms 54 ms 65 ms 63 ms 53 ms 59 ms 52 ms 60 ms 53 ms 58.80
opendns 45 ms 32 ms 56 ms 51 ms 29 ms 74 ms 32 ms 48 ms 45 ms 32 ms 44.40
norton 92 ms 73 ms 56 ms 152 ms 48 ms 144 ms 68 ms 40 ms 77 ms 83 ms 83.30
cleanbrowsing 33 ms 28 ms 33 ms 28 ms 33 ms 35 ms 28 ms 32 ms 33 ms 29 ms 31.20
yandex 174 ms 174 ms 185 ms 178 ms 184 ms 179 ms 179 ms 199 ms 176 ms 186 ms 181.40
adguard 143 ms 147 ms 146 ms 142 ms 148 ms 143 ms 130 ms 130 ms 138 ms 137 ms 140.40
neustar 62 ms 63 ms 70 ms 80 ms 40 ms 74 ms 73 ms 151 ms 58 ms 65 ms 73.60
comodo 60 ms 137 ms 59 ms 60 ms 59 ms 59 ms 64 ms 62 ms 60 ms 59 ms 67.90
localhost 0 ms 0 ms 0 ms 0 ms 0 ms 0 ms 0 ms 0 ms 0 ms 0 ms 0
Most people are not very interested in testing how fast their local machine can talk to itself; regardless of how fast your local cache is, it will have to talk outside pretty often (for expiring TTL if nothing else).
People are praising CloudFlare for not bothering to log requests. Yes, well ... you get the same results by using your own DNS resolver, and you don't have to be shocked when five years later a different governing team running CloudFlare decide to reverse their stance and log everything ...
But what do I know? Apparently I'm in the minority for running my own infrastructure (DNS, web, mail).
Where do you think your DNS resolver gets the answers it gives you? You're just introducing a caching layer between you and whatever external resolver you're using. The first time a machine requests a domain (and every time a TTL expires; look at CDN TTLs sometime) you are effectively talking to the internet just like everyone else.
Do you ever wonder if such activity (basically trying to be invisible) will put you under far more scrutiny than if you were to just 'be normal'? Thinking from the other side of it you would be a high value target to monitor as closely as possible if I were law enforcement. I know it sounds ridiculous but you may actually have more privacy (unless you're actually breaking the law) by hiding in plain sight with the rest of us.
If you talk to an upstream resolver through an encrypted link (both CloudFlare and Google support DNSoverTLS and DNSoverHTTPS, at least) then you at least limit the potential privacy threats.
If you live close to a peering point, both CF and Google will have mean ping times <10ms, often even <5ms. I doubt there's any real world difference between using localhost and either of them. As they cache results, using them could actually be faster than your own server.
Comcast... before changing DNS on my network, I tested a variety of DNS providers including Comcast. While Comcast had the best individual test minimum and average times, Comcast repeated testing showed lost packets and the occasional worst max times.
I don't know why they faired as poorly as they did, but once I realized some network issues could be pinned against DNS packet loss / terrible performance spikes, I changed over to 9.9.9.9 and 8.8.8.8 as the fall back. Home network has been a bit more stable since the change over.
Indonesian ISPs are required to filter certain domains, but go further than the Denmark case - they also block access to third-party DNS servers. Using DNScrypt or a VPN (or hosting your own DNS on a port they're not blocking like 80) is the only workaround.
Prior to third-party DNS servers being blocked a lot of tech-savvy people were using either Google's DNS or OpenDNS anyway since they tend to be more reliable than the local ISP's (even before they started filtering).
I'm no longer in Indonesia, but I could imagine the results people are posting here would be useful to folks back home so they can adjust their DNScrypt config.
My ISP is Comcast, a company I can trust about as much as I can throw it, so I use DNSCrypt when I'm not otherwise on a VPN, and have a number of fallback DNS providers for when DNSCrypt doesn't work.
Reasonably speaking I don't need very much from a DNS provider: <15ms latency, basic DNS security features and the ability to display an error rather than redirect me to a sponsored results page. I can do without the extra filters, but as long as I can get those three things, then I would rather use a different service than my ISP.
VirginMedia UK nameservers are notoriously poor, they just stop responding or disappear for hours. At least with Google I’m more or less guaranteed a response, as slow as it might be. I’m in an area where VM is the only cable operator, so I couldn’t switch without either losing on speed or spending a ton of money for someone else to cable me up.
I do try to use VM dns, but I periodically have to reverse to Google and often forget to switch back post-outage.
I run a local caching server on my network 192.168.1.22, from there I now forward to cloudflare, then to opendns on misses. This makes DNS resolution blazing fast for things we frequent.
I use my ASUS router's DHCP to teach all my dynamically configured network devices how to find the bind9 DNS caching server.
All the statically configured ones just have 192.168.1.22 hard coded with 1.1.1.1 as backup.
"Which ISPs are so bad that you want to use external services"
Any ISP that tries to redirect me to an ad-filled half-baked web search whenever I run into an inactive domain would definitely fall into my "so bad that I want to use external services" bucket.
Verizon's DNS seems to have improved, but a few years ago I ran into consistent problems resolving even the biggest / most common domains. My brief never-to-be-repeated dalliance with Comcast in 2016 was even worse.
My router is another story though. The Fritzbox (>200eur router) adds 6ms of latency, and that's what is advertised over DHCP. (Might still be fine, since cached queries are faster than the ping time to the ISP.) Note that my tests were all with uncached queries (random subdomains of a domain), so it always had to go out and ask an external server (though it could cache the NS record for the domain).