This is an excellent initiative, thank you for building it!
Signalling for me has always been handled on a cheap box in some specific AWS regions. The tricky thing was that there was always more latency than I'd like in the initial establishment of the connection. Reducing that latency always played second fiddle to other issues. CF workers is an elegant solution as all signalling would be handled at the edge.
I'm not sure if this would be lower latency, since it depends on the R2 service, which seems to be centralized. I suspect this didn't use the distributed KV store since the free plan limits are much lower than R2.
The issue with KV is the write propagation delays across nodes can be up to 10s, so for a back and forth like this that needs to happen quickly it doesn’t work. My original version supported KV but I dropped it because it often failed to connect due to these delays.
Ah yep. I guess you could do http polling with a durable object - would have to do the math to see if that had lower expected costs. Websocket are prohibitively expensive.
Cool use of R2 if you have no use for big data sending, why not? it's free. However the page needs correcting, it's only 3 Million free invocations on CFW: 100k req/day, 1000 req/min which you can hit with all the polling.
I wonder then if the project better suits deno's free idle websockets and broadcast_channel.
Oops, I knew it was 3M/mo, typo! Thanks for the note - fixed.
Edit: Deno looks interesting - the requirement here is basically similar pricing to Cloudflare, but also needs strongly consistent or low latency eventually consistent k-v storage. I originally tried prototyping this with Workers KV (since you can use it without even putting in a credit card) but the latencies were too high and it caused too many problems. Does Deno have storage?
Edit2: Oh, of course, the websockets would invalidate the need for storage. Maybe this would be a better option! The main q is how likely websockets will remain free. Cloudflare's pricing for websockets isn't that encouraging.
Cloudflare's pricing for websockets is absurdly discouraging. Pretty soon webtransport/quic will come into its own and a $2 VPS will be able to serve 100,000's of concurrent lite upd 'websockets'
meanwhile CF is asking $200 for 10,000 barely active ws connections and deno is suggesting ws costs are not worth mentioning or measuring for now... Something's gotta give.
Persistent Websockets are not trivial to support for a number of reasons, even just by yourself with your own L4 infrastructure, much less from a global service provider, so it makes sense to take some measurements and set an entry fee. They also generally aren't critical resources for many apps and for ones that are don't tend to require cache layers for them to work, meaning you can deploy your own service for that part; AFAIK, you can proxy arbitrary TCP connections through Cloudflare's network using Argo to still get DDOS/IP protection, and manage the L4 backend yourself; they don't restrict this kind of usage. Also, CF Workers do allow WebSockets attached to durable objects; they don't have pricing information on them now though; they're still letting people get their feet wet. The pricing you're referring to is just if people want to turn on WebSockets for their CDN, I think.
> 100,000's of concurrent lite upd 'websockets'
This obviously is not true, much less for $2. :) You can have "100,000s" of "connections" using HTTP/2 today, including with WebSockets, thanks to transport streams. But this number doesn't mean anything by itself; you often don't ridiculous ratios of streams:clients like 1000:1 or whatever, it's often much smaller, so the per-connection overhead matters a lot in practice e.g. if your ratio is 3:1, then the initial handshake and state tracking matter significantly when considering total overhead. Your QUIC framework still has to do connection tracking, congestion control, encryption, stream muxing, etc. It's not magically a billion times less "expensive" than TCP for magical reasons, but it does avoid some of TCP's long-tail pitfalls in terms of latency and mobile use. WebTransport also avoids some of the design constraints of WebSockets (unreliable transport!), which helps, and in some cases is a significant performance boost for end to end performance between the client & server. But it's not going to magically make running WebTransport services 100x faster, or easier vs the competition or whatever.
The reality is that "100,000s concurrent instances" of anything is complex to support either yourself or at scale from providers. I wouldn't bet money on WebTransport being easier for these providers to offer than WebSockets will. If anything, it is currently, and will remain MORE difficult to support right now; due to the fact QUIC stacks haven't had decades of optimization stuffed into them, high-scale QUIC deployments tend do a lot worse in terms of CPU cycles and utilization. That will change, but not now. And they might still support it despite this! Because WebTransport is really useful technology, don't get me wrong. (This is even before you get into the ever-fierce debate about UDP, about whether people will end up blocking it and requiring HTTP/2 fallbacks, etc...)
Yep my reading of everything regarding websockets and durable objects is there will be a per connection metering fee, which sounds like “expensive”, so I opted to go with the much cheaper and likely sustainable pricing of http polling and storage read/write requests.
Yep the cloudflare worker is in the chain of trust for e2e. There’s an attack vector with signalling servers where you can alter the SDP. This won’t allow MITM but would cause you to connect to the attacker who could then pretend to be the person you were connecting to.
That said, if your users can trust you’ve deployed this worker and that the worker doesn’t hijack the SDP, the e2e would be legit beyond that bit of trust. TURN doesn’t see the clear RTP payloads nor does the signalling server.
Really nice work, gfodor. I've been working on a new p2p library and I see a bunch of ideas here that are new and useful. I'm definitely going to incorporate your work. Good stuff.
Signalling for me has always been handled on a cheap box in some specific AWS regions. The tricky thing was that there was always more latency than I'd like in the initial establishment of the connection. Reducing that latency always played second fiddle to other issues. CF workers is an elegant solution as all signalling would be handled at the edge.
I'm looking forward to taking this for a spin!