I'd swear Verizon FIOS is throttling YouTube recently for me, the quality is getting worse and worse. Constant buffering even at 720p. Forget 4K these days, which used to be perfectly fine a year ago. With the current FCC I'm afraid it's just going to get worse.
In NYC Verizon has an issue where they haven't purchased enough bandwidth into NYC from the Youtube servers.
I just turn on my VPN (VyperVPN) which has great bandwidth to the YouTube servers and I get crystal clear 4K streaming. It's kind of pathetic. I don't think they are deliberately throttling, its just that Verizon is being cheap and there is plenty of local bandwidth between VyperVPN's servers and my home, but not enough bandwidth between YouTube's servers and FIOS network in NYC.
You've got that backwards. Verizon is trying to extort Google by getting them to pay for interconnects and Google wants nothing to do with it. It's the exact same thing that was going on with Comcast and Netflix. It's complete and utter garbage that they continually double dip, and is the EXACT reason ISPs shouldn't be allowed to own ANY content. It's a direct conflict of interest.
Google should instead throw a couple more billions at the SpaceX mesh Internet satellite project, bypassing these asshats completely once and for all. I'd switch in a heartbeat, even if it was a bit more expensive.
The proposed SpaceX satellite network is in low earth orbit unlike traditional satellite Internet, so it should provide decent latencies (25ms vs 600+ms):
Unlikely. Even if Google were to go hard into the video space (unlikely but not impossible), they still wouldn't benefit from throttling/capping. It would just give users a reason to go back to their original providers. Furthermore we've already seen the governmental backlash when facebook tried similar in developing nations.
*I should clarify when I say "the video space" I mean original AAA content like Netflix is doing. I could be underestimating them but that's a LONG way from their market sweet spot.
If only there were a legal solution for such problems, something that any sane person would consider obvious. We could solve it by tomorrow 6pm, without a single satellite launch.
Verizon is trying to extort Google by getting them to pay for interconnects and Google wants nothing to do with it.
I don't think that's right.
At its most basic, an interconnection agreement says “You carry some traffic for me, in return for which I’ll do something—either carry traffic for you, or pay you, or some combination of the two.”
With Netflix and Google, both were basically saying, "we're not going to pay you (Comcast/Verizon) for access to your network, because we're important enough that we shouldn't have to." They don't have their own ISP networks to exchange traffic at the same rate, all they have is their services, so they don't have anything to offer the ISP in return.
So it's disingenuous to say that service providers (Google, Netflix), are extorting anyone. They don't see themselves as ISPs, but they are setting up their own interconnects to provide faster access to customers, so they assume they are exempt from what were traditionally informal interconnect rules.
> So it's disingenuous to say that service providers (Google, Netflix), are extorting anyone.
I don't think anyone was saying that.
> With Netflix and Google, both were basically saying, "we're not going to pay you (Comcast/Verizon) for access to your network, because we're important enough that we shouldn't have to." They don't have their own ISP networks to exchange traffic at the same rate, all they have is their services, so they don't have anything to offer the ISP in return.
Generally peering agreements work on total bandwidth and generally you try to make it as balanced as possible, so there's no cost to either side, as traffic may traverse your network from the peer but not terminate there (it continues through another peer), and that's just a load you bear, but the other side has the same risk.
For an end service peering, that's not really as much of a risk, to my knowledge, so what you have is purely a win-win, where Netflix delivers content directly to your network so it's quicker, and you aren't using up peer bandwidth and backbone connections to serve that same content. In any market where Comcast/Verizon didn't have near monopolies over large areas, Netflix could easily charge for this access given their size and ubiquity. That some large ISPs are actually throttling the content only highlights the perverse incentives at play.
That's somewhat simplistic. Peering that's somewhat one-sided can have fees built into one side. Still, getting a large chunk of your content quicker and without overloading your other peers or upstreams means that Netflix and the like should have the power here, and if they desired could charge for this better access (it is one-sided, as mentioned before). It's totally backwards that Comcast/Verizon would be throttling them instead, and is only possible because they don't really compete in a fully open market.
And yet ISPs happily provide vastly asymmetric pipes to the end user, ergo they're _physically incapable_ of playing the peering game in fair sense of the word either.
But they still want to enforce those "agreements"... when it suits them.
If Google was paying Verizon for transit, and you were paying them for transit, that wouldn't be double dipping. Why should one side of the transaction be the only ones paying for a network that connects both parties?
In a way, they used to. Peering agreements between large networks allow unbilled traffic from either party with the understanding that both would be serving and consuming similar amounts of traffic. With the emergence of the network neutrality debate ISPs have claimed that peering agreements are not appropriate because content providers "send" more traffic than they receive. This of course ignores the fact that their customers are requesting the traffic and their whole business model is designed on asymetric traffic.
Peering disputes have always been when the party wanting to peer had an unbalanced traffic ratio, and the other party didn't want to peer (possibly for other reasons), and they started much earlier than the network neutrality debates.
The recent changes are more around who is involved in the peering disputes. It used to be smaller ISPs trying to get peering with larger ISPs, such as Cogent trying to get peering with [name your favorite, or PSINet vs Cable and Wireless; and most often the ISP refusing to peer didn't have residential customers themselves. Now it's more often the content providers themselves trying to peer with the residential isps directly. A major factor here is the huge consolidation of residential ISPs, but also the consolidation of content providers.
Consolidation of residential ISPs means each ISP is big enough to run their own backbone, and as a result they can credibly have strict peering requirements. Smaller, regional ISPs will tend to want to peer, because otherwise the traffic will come through on paid transit connections. Large ISPs may not care; because of their size, they may not be paying anyone for transit, and because of the common asymmetric nature of residential connections, there's not likely to be many networks where the large ISP is on the wrong side of the ratio.
If I were one of these content providers, I would spend a lot more time messing with the large ISPs. Figure out how to make the traffic cost them money, so they'll want to peer. Provide transit to data backup services to try to make the ratios less unbalanced. Run campaigns suggesting that residential ISPs should be paying their customers, given that the traffic is unbalanced. Etc.
> Peering disputes have always been when the party wanting to peer had an unbalanced traffic ratio, and the other party didn't want to peer (possibly for other reasons), and they started much earlier than the network neutrality debates.
The recent changes are more around who is involved in the peering disputes. It used to be smaller ISPs trying to get peering with larger ISPs, such as Cogent trying to get peering with [name your favorite, or PSINet vs Cable and Wireless; and most often the ISP refusing to peer didn't have residential customers themselves. Now it's more often the content providers themselves trying to peer with the residential isps directly. A major factor here is the huge consolidation of residential ISPs, but also the consolidation of content providers.
> Consolidation of residential ISPs means each ISP is big enough to run their own backbone, and as a result they can credibly have strict peering requirements. Smaller, regional ISPs will tend to want to peer, because otherwise the traffic will come through on paid transit connections. Large ISPs may not care; because of their size, they may not be paying anyone for transit, and because of the common asymmetric nature of residential connections, there's not likely to be many networks where the large ISP is on the wrong side of the ratio.
> If I were one of these content providers, I would spend a lot more time messing with the large ISPs. Figure out how to make the traffic cost them money, so they'll want to peer. Provide transit to data backup services to try to make the ratios less unbalanced. Run campaigns suggesting that residential ISPs should be paying their customers, given that the traffic is unbalanced. Etc.
I can upload all my files to Google Drive and all my photos and videos to Google Photos if you think it helps the ratio...
I've long thought the Netflix app should simply upload random noise, to "balance out" the tremendous download/upload difference with which some poor ISPs struggle so.
> This of course ignores the fact that their customers are requesting the traffic and their whole business model is designed on asymetric traffic.
Exactly. If the Internet worked on this model—where the recipient of the net imbalance was paid money for it—I'd get a check from Verizon each month instead of a bill!
My understanding is that its not an issue of the cost to "upgrade" the interconnection infrastructure. That cost is likely negligible. What's happening is that VZ is refusing to do it, and allowing service to degrade, unless Google pays X. So yeah, basically extortion. Google has already paid for bandwidth, and this is traffic that VZ's paying customers are requesting.
the router ports on both sides, let's say you are using 4 ports across 4 fully loaded nexus 9508 (let's call those $500k per incl. optics, so $2M / 384 x 4 ports / 36 months = $600/port/month x 4 = $2400/4-ports/month)
Depending on how the connection between the two worked or was paid for, it could be $1000-$20000 for 2 pairs of fiber per month depending on distance. Let's take a middle of the road $10k average cost.
So for 10Gbps peak redundant capacity, you are at $12,400 per month. Netflix 4k stream is about 16 megabits, so you can fit 625 4k streams in $12,400 of cross-connect capacity. If you don't share costs, that's about $20 per user to support Netflix's business model that people seem to think it's Comcast's responsibility to pay. That's not even considering Comcast's last mile distribution cost or paying any salaries.
Streaming video services soak up a lot of network resources and putting all of the cost on ISPs is going to increase your ISP's cost and price. How many low-income families do you think would lose broadband if it went from $50/mo to $100/mo?
1) The peering happens in Internet exchanges, there's no way a peering connection in an Internet Exchange costs 1000$, your range is simply ridiculous. Usually, it is fixed cost 1K or less depending on the locations of the router
2) Yes, you have to pay for bandwidth between the Exchange and your users. But guess what? That's what your users are paying for.
3) Not everyone stream movie in 4K at the same time. So you could optimize your bandwidth usage out of the exchanges. It is called "multiplexing".
Yes, the rest of the chain could be expensive, but that's what your clients are paying for. Net neutrality is about transparency. I have no problem in understanding that my 40$ connection will perform differently than an 80$ one. But you should compete in giving me the best service (where best is best for me, not universally) at the lowest price point in a way I could easily compare prices.
The problem, with my statement, is that implies Broadband Internet is a commoditized service. And that is all the battle about Net Neutrality, ISP doesn't want to act like commodities, they don't want to be your P&G or your ConEd. They don't have the structure to compete and instead of changing for the new market structure, they are fighting back. What they are doing is rent seeking. Good for them, bad for the economy...
I acknowledge that the costs are ballpark and not exact, but I was responding to a post that called the costs "negligible" - I was simply trying to quantify a slice of the cost and demonstrate that it can be a significant fraction of your bill. Of course your bill ALSO pays for the last mile infrastructure and connectivity to the hundreds of thousands of other routes carried on the internet. But it's clarifying to see that the data interchange component, which people are calling "negligible" actually is a significant portion of the cost.
My numbers have been called ridiculous by a couple of people now but they are based in personal experience, albeit a few years old, and nobody has posted any other cost breakdowns that demonstrate an understanding of the industry and the costs involved, just "those numbers seem really high!"
The difference between bandwidth and power is that data is NOT a commodity like power is, power is fungible and can be drawn and combined from a number of sources to fulfill the demand, but the dilemma with ISP bandwidth is that in order to satisfy customers the ISP must ensure adequate bandwidth to each individual content provider, and this is a much harder problem.
The problem is your numbers are terribly wrong in multiple ways. I don't think people have mentioned that not every subscriber is online simultaneously or that most are still watching 1080p.
Your argument is at least as wrong as mine, though, while you seek to reduce the numbers, you are not also seeking to increase the scope of the cost from JUST the hardware depreciation and monthly fiber cost of the interconnect to the actual cost of the interconnect. The point is, it's not even close to "negligible"
Not to mention if you're saturating a 10gbps connection, Netflix will throw as many OpenConnect appliances at you as you'd like. Yes, they're not "free" when you factor in racking, etc, but local bandwidth is cheaper than peering, even factoring in your valid comments here.
Those numbers aren't even close to accurate. A fully loaded nexus 9508 (assuming the 10Gbe ports you listed) is ~300k LIST. I'd be shocked if a company that orders as much as Verizon does would pay even half of that.
As for the "distance" - they are sitting next to each other in a data-center. The distance is likely measured in tens of feet and the cost is likely a fixed cost of a couple hundred dollars. The optics on either end if they aren't close enough for twinax would add a couple more grand.
It's also ridiculous to use 36 months as your payoff date, they are running equipment a heck of a lot longer than that. The line cards and supervisor modules might get swapped out, but I'm willing to bet Verizon keeps their chassis level switches for 7-10 years on average.
So now you're at about 4$ per user to support Netflix traffic.
Now throw in some standard oversubscription math, and you're at 40¢ per user. 625 simultaneous streams can probably serve roughly 6,250 customers, given that half the customers don't use Netflix, and the half that do, use it at different times of the day.
Also the proportion of customers who needs 4k as opposed to 1080p is negligible. Most consumers don't even have equipment capable of displaying 4k content.
Netflix in 1080p uses 4.22Mbps or sometimes 7.15Mbps, so we're now at 15¢ per user.
It's not Comcast's responsibility to support someone else's business model.
It's also not Netflix's responsibility to support Comcast's business model if they can't afford to provide bandwidth that customers desire and are allowed.
It's Comcast's responsibility to support their advertised usage sufficiently. If they say that they support 100mbps then I should be able to get 100mbps from wherever I like. They're going to have to pay for that uplink somewhere, whether it's Netflix or Google or Zayo or Level3 or whoever.
I agree, this is the crux of the argument. Broadband ISPs have been guilty of this for some time. Mobile players are generally getting a little more leeway by capping total transfer per month and not talking about rates as much. And T-mobile's "Binge On" product has been mostly hailed as pro-consumer, even though it does shape video down to SD bitrates and is a direct violation of the "paid prioritization" tenet of net neutrality as described in FCC's directive.
That's assuming that we're talking a connection streaming a video from Netflix's CDN to the customer's device. When in reality, Netflix offered to put a CDN cache box in Comcast's DC which would largely reduce the problem, Comcast turned them down.
That, alone, tells me that this is not a congestion or a cost problem, it is a bull-headedness for the purpose of rent seeking problem.
And as to the "last mile" costs, those are what Comcast subscribers are paying Comcast for. They act like this traffic is unsolicited noise, when in reality it's why they're being paid anything by anyone. I say don't charge people for x megabits down if you don't intend on them using it.
My understanding is that Comcast offered Netflix commercially reasonable terms for colo and cross-connect and Netflix opted for public interconnect via an exchange, which alleviated the bandwidth constraints caused by Cogent taking Netflix's money and then refusing to comply with their cost-sharing peering agreement with Comcast. Calling the Netflix CDN boxes "free" is totally misleading because Comcast would still need to pay for the router ports to carry the traffic internally, space & power, and management assistance. That deal works for a small ISP that pays for most of its bandwidth but not a large ISP that uses significant amounts of shared-cost peering. If Comcast is required to host those boxes "for free" what of every other CDN that they previously had agreements with?
I agree that Comcast marketing "up to X megabits" is misleading and those chickens have come home to roost with people thinking that their last mile means they should get that speed to every point on the Internet 24 hours a day. I have been negotiating commercial bandwidth agreements for years and even from a tier 1 you can't get that guarantee in a contract.
Calling the Netflix CDN boxes "free" is totally misleading because Comcast would still need to pay for the router ports to carry the traffic internally, space & power, and management assistance.
Which is still less than the cost of doing a proper interconnect - they chose instead to let it degrade and play semantic games instead. It also puts the lie to their complaint that it was ever about congestion. The correct response to people requesting a lot of traffic from X is to ensure that traffic is delivered efficiently. That is why their customers pay them.
My ISP keeps on trying to get me to upgrade to "up to" 100 Mbps service. What in the world do I need that kind of bandwidth for except to stream videos? Shouldn't they disclose that if I upgrade I can't actually use the bandwidth unless the video provider pays them too?
Eh, that's getting less and less of an issue. My nominal 105/20 internet connection from Comcast Business rarely pushes less than 80mbps download speeds, and from any major provider, CDN, regularly hits and sustains 120mbps.
That's not $20/user to support Netflix's business model, that's $20/user to support Comcast's customers' usage model.
If you think this is unfair, then the answer is to get rid of bullshit "unlimited [but not really]" connections and charge people based on what they use. Stop making low-impact users subsidize people who stream HD movies 24/7.
> that people seem to think it's Comcast's responsibility to pay
Funny, the upwards of $50 to $200 I pay Comcast per month should go somewhat to that.
Besides, how many users are streaming 4K content, on what little 4K content Netflix has (I'm going to wager that we're looking at about 5% or less).
At 5mbps for 1080p, peak, now we're at 2,000 users, and $6/month.
Is it really that onerous for most users of Comcast to expect that $6 of their say $60/mo cable bill goes to Netflix?
I'm also not sure why you're populating 4 ports and then only talking about 1 10Gbps connection, when in reality (though I'm no expert), that's probably 4 10Gbps connections. Accounting for the cost of 4 populated running ports and then only talking about the capacity of 1 of those ports when calculating cost/megabit seems misleading.
Back in reality the huge ISP's provide network services at a huge premium and make a very substantial profit. If we cared about providing affordable service to low income families we would let municipalities sell access to existing fiber networks not trying to prop up an industry providing bad service at monopoly prices.
Your numbers are like historical fiction based on reality but not an accurate portrayal. Do you work for verizon or comcast?
Source your numbers. Most specifically that netflix which can provide the entirety of its service for 10 per subscriber actually costs providers $20 per subscriber.
You don't pay "for the bandwidth", you pay for a link between two ISPs (Google and Verizon). The bill between then should be split, otherwise the traffic will have to pass through somewhere else, and that will cause congestion problems.
Google has most certainly paid for bandwidth/traffic from their end. Whether that's to a CDN or some other arrangement, and there's likely already peering agreements in place. In these cases what Verizon is doing is asking for money above and beyond what it actually costs to handle the interconnection. They use declining service to their own customers as leverage. Its ridiculous, and even more worrying when you see ISPs and content providers merging as is the case with VZ buying Yahoo and AOL, Comcast and NBC merger, etc.
Google pays for their own network services. Whomever google pays for network services pays pays if applicable to connect to Verizon's network.
The sole and only thing they are being paid for on the ISP side of Verizon's operation is to quickly and reliably deliver the content people want to consume over the pipe people have paid to have installed.
YouTube by hosting content people want to consume is driving demand for Verizon's services.
Pretending that this is a cost is a bazaar inversion of reality.
Do they? Who do they pay? I thought that maybe one of the reasons Google got into the ISP business was to be a peer with the other big networks and not pay any peerage fees.
They don't need to purchase bandwidth, Youtube is available via public peering exchanges. The way these generally work is you just pay for the port on the switch at the exchange and traffic directly. See:
Purchasing a port on public peering fabric isn't the same as buying bandwidth from a transit provider. It's a great way to get traffic to your users much cheaper than you could via transit which is why people do it. The issue is eyeball networks - Comcast, FIOS, Time Warner etc. may choose not to do so because they went to seek rent in the form of "paid peering" from these same content providers that their users paid them to get access to in the first place.
So are you accusing them of malicious throttling or that they just have bad peering agreements that configuring a different network path ends up being faster?
Is there a difference between the two? See my other comment. I'm suggesting they have bad peering agreements and choose not to do anything about it. Most likely they are waiting to extract money from YouTube. Seems like net neutrality is in question again.
Forgive me if this is a dumb question, but how would the ISP take a 1080p stream and downsample it to 480 on the fly, without man-in-the-middling HTTPS?
They don't. HTTPS doesn't hide who the connection is to. How would a network route packets if it didn't know the recipient?
Once they know you're connected to Youtube, they just limit the bandwidth on that connection. The Youtube video player will detect this and automatically downgrade to a lower quality until it finds one that fits within the bandwidth profile. Netflix does the same thing, which is why after buffering sometimes the video looks like crap and then gets better after 15 seconds or so. It buffered a lower quality until it figured out it could send higher quality again.
EDIT: I should add, this is why VPNs are sometimes used as a "solution" to such bandwidth shaping. You are hiding the true recipient by purposefully man-in-the-middling the connection with a peer you (hopefully) trust. VPNs are also popular for connecting with business networks, so it's generally not "safe" for the ISP to shape bandwidth to any VPN as aggressively.
> The Youtube video player will detect this and automatically downgrade to a lower quality until it finds one that fits within the bandwidth profile
Ahh, this is the key point I missed, thanks. So it's a problem with the video player itself trying to "decide" for you the quality you want. Couldn't this be worked around with things like youtube-dl?
The video player deciding what quality you want is a good thing. The user wants to watch a video live, as it goes. You don't want to be interuppted every 30 seconds with a "buffering" message, or wait 10 minutes at the start for it to entirely download.
Youtube-dl would probably work, because it will would download the high quality video as a file, and your ISP would make the download take longer than the video.
> The video player deciding what quality you want is a good thing
It's an extremely annoying thing actually, especially when it changes after I manually set it to 1080p. This is why I use mpv to watch stuff from youtube nowadays.
It's probably a coincidence, but just this weekend for the first time ever, I noticed my YouTube streaming at visibly lower quality (my provider is Cox). It happened all weekened, intermittently. Nothing else seemed slow, and numerous speed tests (both from official "speed test" sites including Netflix's own Fast.com, and by moving large files between my network and AWS) all showed well over 100Mbps. I don't know if it was being throttled somewhere, Netflix's service was overloaded, or if there was some other weird problem happening on my network that affected Netflix and nothing else. But it was bad enough that my wife who is usually oblivious to audio/video quality actually brought it up.
I used to have FiOS is one of the few places in Cambridge that offered it.
It was the first time in my life that I'd been wishing for Comcast. 320p videos could barely load, and the lag in certain MMORPGs was unplayable (we're talking 500-1000ms delays)
It got better after a while, but google had some stats by ISP for Youtube, and FiOS in my area was like 1/10th of the speed of Comcast on average.
Let's say you live in NYC but you're using a DNS server (resolver) in Seattle. When your client performs a DNS query, you may get a response that directs you to a server/CDN/whatever in or near Seattle as opposed to one closer to you in NYC. Then, the data (streaming video) is traversing the country to get to you, passing through many more (potentially congested) interconnects/links on the way.
Reality is a bit more complicated than this simple example but such an issue is certainly not unheard of (and inspired mitigations such as RFC7871, for example).
Both Google and OpenDNS use anycast addresses for their DNS servers that are very likely to route though a 'local' server. I think ISP games are far more likely to be the root cause here...
