I'm an engineer for one of the projects over at ZeroTier. It's neat to see this pop up on HN. If anyone has any questions about our technology or services, let us know. We'll keep an eye on this thread!
It was the best thing I found lately for creating a network without defining specific endpoints. I was pretty much looking for something like Hamachi, but more professional. End to end encryption is great too.
I was pleasantly surprised that I can just create a network with ipv6 and broadcast enabled and things like mdns/avahi "just work". Being able to enable "any protocol" rather than just ip4/6 is great too. In practice you get private encrypted network with non-colliding addressing, route optimizing (two hosts behind a nat will talk directly), mdns working out of the box. What else would you ever need? Android app is a cherry on top.
Network Containers is our experimental tech for allowing apps to join virtual networks with no kernel/OS involvement via a private mini IP stack. You can do P2P networking between instances of your app, other apps, servers, and anything else ("connect all the things") using standard network code, Posix network APIs, and familiar protocols like HTTP.
Is there any way to turn compression off to reduce CPU consumption by the ZeroTier process?
Also, one big concern we have is using the standard/default discovery servers to run our network. Is there any guidance on self-hosting the discovery servers ourselves?
Lastly, can ZT assign IPs for us automatically using some kind of DHCP? Is there any documentation about how it works in case of network partitions?
Turning off compression: noted. We also might permit a no-encryption mode for trusted backplane networks for data center SDN use in the future.
There's no federation for the root servers yet. We have numerous ideas on how to implement this but it's not a current priority. It has to be done with care to avoid sacrificing speed, security, or ability to upgrade.
You can read some of the reasoning behind ZeroTier's design here:
TL;DR: we chose a design that delivers instant-on zero-configuration operation, security, and very fast (<5s) connection setup between any two devices on Earth at the expense of adding a small amount of centralization to the system. We also avoided certain technologies like DHTs because we wanted the endpoint software to be small enough to run on small embedded devices with limited bandwidth, CPU, and memory and on mobile phones with bandwidth and power limits. Our root server based architecture achieves all this.
The root servers are two-times redundant. There are two root servers and each of these is geo-distributed across six nodes. These are also spread across four cloud hosting ISPs. Any combination of up to 11 roots total can fail without the system being significantly impacted since each root individually has enough power to carry the whole net. All roots are secured with physical two-factor authentication and only permit ssh access from a set of secret gateway IPs (also secured with 2fa).
Root locations are: San Francisco, New York, Dallas, Toronto, Amsterdam, Paris, Franfurt, Johannesburg (SA), Sao Paolo (BR), Tokyo, Sydney, and Singapore. Almost everyone on Earth gets <100ms ping to at least one.
As far as I understand it, the binary contains compiled-in ip addresses for public nodes operated by zerotier. By recompiling you can run your own, they are not special in any way as I understand it. And if they go down your network wil not suffer, they are only used for initial setup.
Separate from this you also have to option of running your own network manager. This will allow you to create and manage your own virtual networks, with your own GUI, billing etcetera. The Zerotier binary can be compiled with a special option to extend the build-in REST API for that.
So, does it find the best route between 2 computers, or just a route? For instance, if half the computers are on a local network together, does it find them via that network, or does it go to the internet and then back to connect them to each other?
The initial configuration requires talking to a node on the internet. But I have two nodes behind the same NAT and they're happy to talk directly using UDP (verified with wireshark), without routing over internet. The standard encryption is still used.
I'd be interested to know if it uses something like STUN/ICE to punch through two different NATs... don't have two NATs to try at the moment.
Thanks! I'm fine with a constant connection to the net, and even locating each computer that way, but I want the computers to then talk directly to each other. (At least until the IPs change, etc.) It sounds like it'll do it. Thanks for the info!
This is a bit late, but do you have any performance numbers for ZeroTier? Also, is it multithreaded? tinc is not, and that limits the throughput to ~ 0.5Gbps.
Hi, I just did a short test and it looks really promising. It has also an android app! This is really the selling point for me :)
BUT I really do not believe that this is the way a rpm package should look like. I know that it is not easy, especially for smaller companies, to provide packages for all the platforms. In this state, it would be much better to just provide .tar.gz archive. Its just my opinion ;)
We're working on better RPMs and DEBs and on getting into Debian itself and hopefully EPEL/Fedora. Right now the packages are minimal and not entirely correct (though they do work).
That's why at Wormhole we chose to use 100.64.0.0/10 (and actually most of our customers only use the default first /24). It's reserved for carrier grade NAT, thus unlikely to interfere with anything you or your providers use :-)
We offer a similar service to ZeroTier, but based on SoftEther. Needless to say, we LOVE ZeroTier, they've got a brilliant product. Keep up the good work!
Yes. Sort of. IPv4's address space is too small so it's a nasty hack people use to avoid conflicts. It doesn't hurt anything but if those blocks ever get advertised in BGP you have to renumber your network.
Also keep in mind that we only use that range on our test "Earth" network. You can use any IP range you want on your own networks. ZeroTier is layer 2 (Ethernet).
BTW, it's not uncommon for cell networks and ISPs to use DOD IPs for hidden layers, leading to some interesting outbreaks of paranoia when people spot them in a traceroute: "why is all my traffic being routed through the DOD?!?!?!" (No, it's not.)
I've been using zerotier for a year now. It is simply FANTASTIC. Currently use it as a management plane for my docker swarm. Really great work.
Their blog is also very much worth reading.
EDIT: I'd like to mention an interesting feature. If you run two hosts that are also connected in another way, for example on a LAN, or using private networking from your cloud provider, ZT will automagically find and use the fastest route for host to host traffic.
> Gonna try this for my friends, family and some of my servers.
What are these use-cases all about? Not really getting the point of this right now.. curious since all my friends, family and servers are connected to the Internet already just fine, as I suspect were yours.
For example, many older games either have no online multiplayer support, or have shut it down, while they do have functional LAN support. If this can replace Hamachi for those use-cases, then that's pretty great already.
For friends / family -> it's nice to have shared storage which does not eat your harddrive. fileserver.
I have some servers in the cloud and some mac minis hosted at home. I'm using an ssh tunnel to connect them, but it's far from ideal, and not stable somehow. (tried ssh and autossh).
For a new project I'm using 8 raspberry pi's, which are set up in a remote location. They have static IP addresses, and I to be able to access them. Right now it's done by portmapping. If anything happens to te network config over there, I'll have to fix them.
With zerotier I can simply put them on dhcp, and connect them all to a virtual lan.
Been using ZeroTier for a while now. My main use case is to connect my laptop to my home network so that I can access my NAS and Plex server while I'm away without having to expose those ports to the internet directly.
cjdns works on ip level (ipv6 specifically). Zerotier gives you a virtual ethernet layer. Also you can use a private controller rather than always relying on public DHT.
I'm an engineer for one of the projects over at ZeroTier. It's neat to see this pop up on HN. If anyone has any questions about our technology or services, let us know. We'll keep an eye on this thread!