Erlang recipes, not that much. Make that Erlang/OTP recipes with a strong emphasis on the OTP side of it, and a million times yes !
Or to say it slightly differently : I'm not that much interested in recipes about the Erlang language itself, but I'm very very interested in recipes/examples of architectures of actual Erlang/OTP softwares.
I'm doing mostly Python these days, but I worked on a couple of side projects in Erlang in the past (a large one ~3 years ago, and a small one ~6 months ago). And for me the hardest was (and still is) to figure out how to use the OTP behaviors in an optimal way. How to make a nice, working supervision tree.
I managed to pull working stuff out of the ground, but somehow it never fell OTPish. I couldn't tell if I did it right or wrong.
I got it and I like it quite a bit. It is a good introduction book for practical Erlang use, since you learn to do things the "OTP way" right off the bat.
It has examples on how to build network servers (TCP RPC server, HTTP server, a caching service, setting up a distributed system with multiple nodes, resource discovery, logging, release managing, interfacing with C code and many other useful things).
I guess I wasn't clear enough. It will be very much skewed towards OTP, and not about the language as there are already good books on the language itself.
Hey, I see you're gaging potential reader interest by asking what they might pay. You might consider using StoryFunded (http://www.storyfunded.com) for this.
Our site lets authors raise funds for promising book projects. Like Kickstarter it's all-or-nothing where contributors only pay once the goal is met. If the goal is met authors receive the raised funds, minus 5% and processing fees. We've just went live, and our biggest challenge now is finding book projects people would be interested in funding. This may be such a project :)
I don't know if this is antithetical to these recipe/cookbooks, but please consider teaching coherent themes or design principles via these recipes, rather than simply providing an assortment of useful tidbits. Erlang has such a unique foundation (functional programming + event-driven awesomeness) and so few books that there is a great opportunity to teach the Erlang mindset/perspective and not just a bunch of tips and tricks.
Very good and valid point, and it is pretty much at the foundation of the book approach (short guides ondesign principles when applied to real tasks, not just the theory)
Yes, I'm interested! I'm particularly interested in recipe books that are both task-oriented and language-oriented. For exaple, Doug Hellman's PyMOTW is a phenomenal guide to using libaries that are part of the Python standard distribution and would be a key part of the library.
However, some aspects of the recipe book should be task oriented:
- Web server hosting static files using the standard inets server, and 3rd party libraries like webmachine, nitrogen, and yaws.
- How to log
- How to enable tracing
Yes, this is quite close to what I am working on, both language and task-oriented. Also as a maintainer of erlagner.org index I know quite a lot of good libraries to use.
I'm intrigued by the idea of mutual supervision. This implies that both process dynamically become the supervisor of each other. By default, OTP supervisors are more architectural in nature -- they will always give you a top-down tree rather than a cyclic graph.
Who restarts who? That's a bit tougher to do when everything becomes cyclic.
Links are bidirectional though, so when one of the workers die, the other can know about it, die, propagate the error or react to it. Monitors are unidirectional and won't propagate the errors, but only report it. I guess you could write your own mutual supervision that way, but it would be a bit weird, still.
My problem with having a tree is that it implies that at every level - including at the root - there's a single point of failure. I get the error kernel concept, I just don't necessarily trust the hardware.
What you need to do then is look into a few different options.
The first one is distributed Applications. This is part of the standard OTP stuff. What this does is declare a node as a master, and then failover nodes for your OTP application. When your master goes down, the other nodes take over it. Whenever the master is back, the failover nodes drop their applications and restarts it on the master. This loses all of the state if it hasn't been kept in a safe place.
Another one is to look at gen_leader. gen_leader is not part of OTP, but has been written by Ulf Wiger and is available on github or through agner. A quick description of it would simply be to say that it's a boosted gen_server that will find a master of itself somewhere and automatically replicate its data everywhere else. If the master dies, a leader election occurs and a new one takes over.
A third option is to have a masterless concept. This is what happens in Riak. Data has to be shared in clever ways and code organized to support such a design. Nobody's a master, everyone's a follower. You add redundancy here and there, push important state to a safe spot, etc. In the standard library, modules such as 'global' handles this kind of thing.
Supervisors are not exactly made to cross node boundaries. They protect your local processes. If you need something more complex to keep your state alive, this usually happens at a higher level for OTP.
Yeah, it sounds like either distributed applications (more likely 'cos it's in OTP) or gen_leader (maybe interesting technically) would be good to look at.
That being said, riak_core has been on my List Of Things To Have A Really Good Look At for a while now...
doesn't heart kinda solve the cyclic "who restarts who thing". I think heart (if enabled) sits there until the erlang process exits, restarts the erlang vm, which in turn also brings up an instance of heart. yeah, complicated and not perfect, but it works.
Heart does solve it in a way, yes. Any Master/Slave pattern where a masterless slave becomes a master also solves it. The complex cases I had in mind happen when you dynamically add more workers and try to set up crazier links (A is the master of B, B is the master of C, C is the master of A), although there are few practical uses of that. You'd have to try to shoot yourself in the foot I guess. I see it as a bit hard to provide a generic OTP pattern to do that kind of thing past the standard takeover/failover setup (and heart for VMs)
Very needed book. In particular I am interested in limiting message queues in Erlang. I still don't get it - How one should deal with situations where messages are produced constantly in higher rate than consumed.. There is no protection in Erlang for this common scenario (I think)
You have to make the producer synchronous to the consumer so the consumer is slowed down. Ask for an acknowledgement on reception of any task/event. This will instantly limit the rate at which data can come in the final worker.
Well, that breaks the whole philosophy of async message passing in Erlang, where sending message always succeeds and is not dependent on the receiver. What you are suggesting is in fact limit the queue to be of size of 1 (if only one producer, or to the number of producers). This is not what queue is all about
No, this doesn't break the async philosophy. The synchronous sequence of calls is done by a question/reply made out of 2 asynchronous calls. It's just that simple to make something synchronous, as opposed to turning something synchronous asynchronous (by using a queue).
By doing this you are not limiting the queue to the size of 1. If you have N producers, you can have more than N elements in the queue, but the idea is that each producer can only enter them one by one, with the agreement of the consumer. Nothing would stop the consumer from accepting a thousand of them before waiting before replying, therefore slowing down the producer. The idea is just to limit the rate at which you accept queries by tying it to the speed of the consumer.
If this is not enough, you can look at things like the jobs tool (https://github.com/esl/jobs) to work with scheduling.
By the way, you're mistaken on the idea of message sending always succeeding. The idea is that you will never error out because of trying to send a message (send and pray), not that the message will make it to destination. If you want any guarantee that the message made it, then you need that acknowledgement async message being sent back to you.
The point I am trying to make is that this fundamental feature of message queues (limit their length to prevent the memory exploding) should be provided by Erlang and not reimplemented over and over again by developers.This is really a basic feature in any message queue
Why is this a fundamental feature of message queues any more than of other things?
(you can make a data structure which makes memory explode, for instance lists:seq(1, 123456789123456), you can make a program which makes memory explode, for instance by growing the stack enough. Why is a message queue special?)
Because what to do when the queue is full is absolutely application specific.
What happens if the message you can't add is a specific system message that allows the suspension of OTP processes for a code upgrade? Do you end up livelocked? How do you propagate back that you couldn't insert your element in the queue? Do you make message passing an operation that can fail? How would that work over the network when the message telling you something failed is lost over a netsplit? Is it a different kind of failure?
This has very deep design repercussions on the language. It's not impossible, but it wouldn't let Erlang be the same.
I have just stared learning Erlang after more or less leaving Java, and the heavy weight world of J2EE. I was surprised how easy I have found learning Erlang maybe it just seemed to make sense, or it feels like the right way of doing things. I would be really interested in the book.
This sounds great. I have an introductory book to the Erlang language, and would love a recipe book on the language. I got into Perl via Programming Perl, and still refer to my Perl Cookbook to see how to use the language in certain scenarios.
For everyone who's interested in Yurii's book: if you go to http://leanpub.com/erlang-recipes and enter an email address, you'll get automatically notified when the book is first published.
Yes. I'd certainly value recipes that follow from using Erlang in real life. Especially if it contains succinct ways of solving common problems like the Python Cookbook, which is by far my favorite technical book.
Or to say it slightly differently : I'm not that much interested in recipes about the Erlang language itself, but I'm very very interested in recipes/examples of architectures of actual Erlang/OTP softwares.
I'm doing mostly Python these days, but I worked on a couple of side projects in Erlang in the past (a large one ~3 years ago, and a small one ~6 months ago). And for me the hardest was (and still is) to figure out how to use the OTP behaviors in an optimal way. How to make a nice, working supervision tree.
I managed to pull working stuff out of the ground, but somehow it never fell OTPish. I couldn't tell if I did it right or wrong.