Hey all, founder is here. I'd like to thank you for your votes here. Really appreciate!
Also, I just want to share our public roadmap: https://trello.com/b/7fEwaPRd/roadmap. Feel free to comment. It actually helps a lot for us to prioritize. Thanks!
Where is Hyper.sh (specifically its containers) actually hosted?
As I explained in my other comment https://news.ycombinator.com/item?id=12892243 it really feels like Hyper.sh is hosted on Amazon, and there were references to that fact before, and you guys are trying to minimize that in your site now.
If you're on Amazon, that's OK. I don't think that minimizes how cool this technology is and how much easier it makes things. Amazon has an Elastic Container Service, but this is more nuanced than ECS is, and much more painless. But if the containers aren't on Amazon, a little more detail on how that works would be awesome, because right now it really feels like they're on Amazon. Which is fine, but when folks are making decisions (like putting their stuff on multiple platforms for reliability), it's important to know.
Edit: I signed up and looked around. It appears they're hosting on ZenLayer, a Chinese hosting company that has hosting in LA as one of their options. Not sure why they stick so closely with AWS on terminology though.
RE: "Not sure why they stick so closely with AWS on terminology though"
Makes sense from a user familiarity perspective -- AWS is what most cloud users are familiar with, and describing things in terms that are most likely to be understood is generally good practice.
I'd agree that it would make sense though if their site clarified who owns and runs the datacenter their running their service out of, if only for answering the question if they're hosting on top of AWS or not.
Have you thought about allowing users to provide their own cloud provider (AWS or Google Cloud) API access? That way we can use own own VPC and servers, and you can just charge us monthly for the software. See MongoDB Atlas [1].
This also means you as Hyper.sh don't have to worry about servers, uptime, buying hardware, power, bandwidth peering, what a headache. Let AWS and Google worry about the commodity physical hardware.
This is still possible with Docker Cloud. You can either choose to have a node auto-provisioned on a cloud provider or install their agent on one of you own.
damn, thank you. Anywhere I can see implementation details? Are you rolling your own system from the ground up, or using something like dkron (http://dkron.io/) behind the scenes?
Agreed for some things but a massive limitation of Lambda with its timeout of 300 seconds is that it cannot be used for long running tasks.
If this feature lands it sounds like it will give the ability to run any container arbitrarily and pay by the second - this would be huge for things like web scraping and other tasks that don't occur all the time yet still come with all the pain of server maintenance and uptime fees.
Imagine the scenario where you have a web scraper that runs once a week for 3 hours. Ideally you only want the machine to be on for those 3 hours, and to only pay for those hours of usage; but you also don't want the hassle of writing all the scripts that go with creating and deleting a cloud machine, mostly because those would also need to be hosted somewhere. For that kind of use case there is little out there as far as I'm aware.
There are ways to organize your lambdas to get around the 300s timeout. In classic AWS fashion... you just need to give them more money.
Look for ways to divide-and-conquer your lambdas into smaller parts. If you need to run some logic for every record in some table, give N records to a single lambda (where N is some small number which doesn't make the lambda take anywhere close to 300s).
You can orchestrate this workflow with several different AWS tools which exist all over the spectrum of cost and ease-of-use.
Easiest is definitely just having the master lambda directly invoke the other lambdas with InvocationType:Event.
SNS is another easy option. Lambda(master)->SNS(per N records)->Lambda(splinter). Downside is that you'll probably completely blast out your global AWS concurrent function execution limit pretty quickly because you have no control over how quickly SNS will trigger your functions.
Kinesis is a more powerful option. SQS also has potential, but you can't directly trigger a Lambda from SQS. One pattern I've seen used is to have a CWEvents cron trigger a lambda every M seconds to read N records from SQS. Depending on how consistent your workload is, this might make sense because it gives you really fine-grained control over that ratio between "how quickly will my jobs be processed" and "am I approaching my AWS global account limits". But if your jobs are really disparate you'd be invoking lambdas all day to do a whole bunch of nothing 90% of the time.
If I'm playing devil's advocate: yes and no. Lambda is still partially limited by the officially supported languages (I'm pretending that the hacks around this don't exist, they have issues). That said, Lambda is great, but I'm not really on-board with ECS, so it's nice to see an alternative here from someone who is also offering container based serverless infastrucutre.
Is there a write up on howto use lambda for this? I've been wanting serverless crown for ages and I feel like an idiot for not thinking of lambda for this... this should be promoted (if it isn't) - but I'm an example learner... point me please?
Another "serverless cron" option from aws is their Data Pipeline service. It is intended for data processing as its name implies but does fits the bill for lots use cases as a generic "serverless cron". Note that Lambda has executing time limit that's pretty short, while data pipeline can be configured to have much longer timeout.
Lambda is already limited enough when it comes to how you can write the functions. I'm not even sure how you'd write a generic JS/Python/Java function on DPL, unless its got some functionality which I'm not aware of (through a custom shell command bootstrapping your language environment maybe?).
I'd like to run a local instance of the "hyper system," fully contained on my laptop, perhaps running atop virtualbox. If I find it appealing, with respect to the usage experience (i.e. deploying/composing containers), I'd be willing to run some apps on the real deal. I am, though, not willing to experiment with the "real deal" as a means of evaluating it.
I did, but I still needed to delete every single thing by hand, and some things could not be deleted because other things were using them, even though I had stopped all the things, so it had to be in a specific order that took a while to figure out.
Not even for the micro-pennies that experimenting with the real deal would cost you?
If you think you might end up using it, why try and replicate it on your machine (any more than Docker already does) rather than just cross the deployment bridge sooner?
I prefer an "always local first" approach. Before Docker, it was Vagrant + VirtualBox for dev, some VPS hosting company for prod. With docker, it's basically the same thing (better, though, imo), but I no longer think much about the virtual machine on which the docker daemon is hosted and instead rely on boot2docker, Docker Machine, and now Docker for Mac for the local side. For prod, I've used Docker Machine to setup docker daemon hosts on Digital Ocean, et al.
I'd like to worry even less about the container host and get comfortable with a system like Hyper, but it's important to me to get used to it running locally for dev prior to employing it "in the cloud" for prod.
I've not spent much time with AWS. I was thinking more along the lines of Rackspace, Linode, Digital Ocean, those kinds of hosts.
Yes there are important differences between running a dev server on VirtualBox and a prod server on one of the above, but there is parity in the workflow. The same is true when thinking about docker-machine + docker, locally then remotely.
I understand that Hyper's cli is quite similar to docker's cli. But my preference is to not consider it seriously until I can bang against a version of the Hyper backend running locally. If that's not forthcoming, fine, Hyper's not for me. :-) If it is, then great! I can't wait to play with it, locally, on my personal computer/s.
If it's really about workflow parity then I would encourage you to give it a spin. I think the parity gap will be similar to your previous experiences with VMs all things considered.
Are there any plans to offer hosting in germany or at least europe? We love the service you offer, but privacy laws here make it hard to use it when it is only hosted in the US
We already had a chat with him on your page, thanks!
Maybe it helps to prioritize that topic if I tell you that your service not being hosted here is the only ting keeping us from moving our complete microservice ecosystem to your service ;-)
I'd absolutely love to see that! At codebeat we run parsers and static analysis algorithms which are very resource-intensive but typically run for a short period of time so we have those powerful dedicated OVH machines sitting mostly underutilized. I've been doing some preliminary (mostly design and some dirty coding) work on that so if you're interested in comparing notes please feel free to reach out to me at marcinw [at] codebeat [dot] co.
Also, I just want to share our public roadmap: https://trello.com/b/7fEwaPRd/roadmap. Feel free to comment. It actually helps a lot for us to prioritize. Thanks!