damn, thank you. Anywhere I can see implementation details? Are you rolling your own system from the ground up, or using something like dkron (http://dkron.io/) behind the scenes?
Agreed for some things but a massive limitation of Lambda with its timeout of 300 seconds is that it cannot be used for long running tasks.
If this feature lands it sounds like it will give the ability to run any container arbitrarily and pay by the second - this would be huge for things like web scraping and other tasks that don't occur all the time yet still come with all the pain of server maintenance and uptime fees.
Imagine the scenario where you have a web scraper that runs once a week for 3 hours. Ideally you only want the machine to be on for those 3 hours, and to only pay for those hours of usage; but you also don't want the hassle of writing all the scripts that go with creating and deleting a cloud machine, mostly because those would also need to be hosted somewhere. For that kind of use case there is little out there as far as I'm aware.
There are ways to organize your lambdas to get around the 300s timeout. In classic AWS fashion... you just need to give them more money.
Look for ways to divide-and-conquer your lambdas into smaller parts. If you need to run some logic for every record in some table, give N records to a single lambda (where N is some small number which doesn't make the lambda take anywhere close to 300s).
You can orchestrate this workflow with several different AWS tools which exist all over the spectrum of cost and ease-of-use.
Easiest is definitely just having the master lambda directly invoke the other lambdas with InvocationType:Event.
SNS is another easy option. Lambda(master)->SNS(per N records)->Lambda(splinter). Downside is that you'll probably completely blast out your global AWS concurrent function execution limit pretty quickly because you have no control over how quickly SNS will trigger your functions.
Kinesis is a more powerful option. SQS also has potential, but you can't directly trigger a Lambda from SQS. One pattern I've seen used is to have a CWEvents cron trigger a lambda every M seconds to read N records from SQS. Depending on how consistent your workload is, this might make sense because it gives you really fine-grained control over that ratio between "how quickly will my jobs be processed" and "am I approaching my AWS global account limits". But if your jobs are really disparate you'd be invoking lambdas all day to do a whole bunch of nothing 90% of the time.
If I'm playing devil's advocate: yes and no. Lambda is still partially limited by the officially supported languages (I'm pretending that the hacks around this don't exist, they have issues). That said, Lambda is great, but I'm not really on-board with ECS, so it's nice to see an alternative here from someone who is also offering container based serverless infastrucutre.
Is there a write up on howto use lambda for this? I've been wanting serverless crown for ages and I feel like an idiot for not thinking of lambda for this... this should be promoted (if it isn't) - but I'm an example learner... point me please?
Another "serverless cron" option from aws is their Data Pipeline service. It is intended for data processing as its name implies but does fits the bill for lots use cases as a generic "serverless cron". Note that Lambda has executing time limit that's pretty short, while data pipeline can be configured to have much longer timeout.
Lambda is already limited enough when it comes to how you can write the functions. I'm not even sure how you'd write a generic JS/Python/Java function on DPL, unless its got some functionality which I'm not aware of (through a custom shell command bootstrapping your language environment maybe?).
damn, thank you. Anywhere I can see implementation details? Are you rolling your own system from the ground up, or using something like dkron (http://dkron.io/) behind the scenes?