Hi Hacker News. I started developing this over two years ago when I become dissatisfied with what I felt was a lack of good options for an easy-to-use Python messaging system. I didn’t think it would take this long (of course!), but I’ve certainly learnt a lot about messaging and distributed systems. I tracked my progress in issue number 1 on GitHub, and I feel pretty happy that I now have 98 of 98 tasks complete! [1]
I’ve been using this in production now for about a year and it has faired very well. I found a few weird bugs along the way, but it has felt stable for quite a while now. I’m therefore finally making an official release. I wanted to call this version 1.0, but I’m going to play it safe and call it 0.9 for a few months and see if anything crops up in other peoples’ environments.
Before embarking on this I spent a long time talking to other developers to see if anything similar existed. The most similar I could see was Nameko, but I didn’t feel like that ticked my boxes. Lahja is another one which I saw pop up more recently. I’ve also written a short comparison with Celery, as it something I’m often asked about [2].
I’ve also spent a lot of time writing up the docs. A crude word count indicates I’ve written 23,400 words of narrative documentation :o
I was also keen to not strictly limit this too Python. I’ve therefore documented the Redis interactions required for other languages to interface with Lightbus[3], albeit without the various niceties that Lightbus provides).
We (https://bespoke.app) use Lightbus in some supporting infrastructure, and despite starting with a less stable version, we've been quite happy with it.
Firstly, the maintainers are extremely responsive, and have gotten most issues fixed in less than a day. IMHO, the devs are being particularly conservative in calling this 0.9 instead of 1.0 -- which is arguably a good thing!
Here's a quick/informal experience report. Happy to go into details.
Pros:
- It's very "Django"-y. The declarative interface reads and writes like Django ORM code, so there's very little context switch. It should feel familiar to Django programmers.
- It's built on "boring" tech. The default transport layer is Redis. Most of us don't need a distributed queuing system like NSQ or Kafka. Config is YAML.
- The API is designed to favor ease-of-use over virtually everything else, and yet there's actually very little magic under the hood. There's a bit, but we've found it pretty easy to reason about. Default options are sensible, and easily tunable.
- The documentation is good. It's a bit spartan for some of the more complex things (e.g. writing your own client in $YOUR_FAVORITE_LANGUAGE).
Cons:
- Our biggest concern has been with logging. Message-bus architectures have all the difficulties of cooperative multitasking, with the added problem that events can also go missing. Good logs are invaluable in debugging this kind of problem. My understanding is that this is on the roadmap.
All in all, I think it's a safe bet for pure Python projects. If you're a polyglot shop, you may want to consider something else in the short term, but Lightbus is very capable of solving the problems that message-bus architectures were designed to solve [0].
[0] e.g.: "how can these isolated web apps share data in a sensible way".
Hey there! Thanks for the comment, I'm glad its been going well. I'll reply to a couple of your points below:
> The documentation is good. It's a bit spartan
> for some of the more complex things
Yes, I've documented the protocols[1] but I haven't gone into much more detail. The docs may also need some iterating on as/when people try to develop new clients.
> Our biggest concern has been with logging.
> Message-bus architectures have all the
> difficulties of cooperative multitasking,
> with the added problem that events can also
> go missing. Good logs are invaluable in
> debugging this kind of problem. My understanding
> is that this is on the roadmap
Yep. Two things are on the roadmap that are relevant to this:
1. Revamping the logging system, likely to have an option to support structlog
2. The ability to see a causal trace for each message (Message C was caused as a result of Message B, which was caused by Message A)
I think no. 2 should be straightforward, and I think no. 1 is just going to take a grim sense of determination and deleting a bunch of old logging code.
I’ve been using this in production now for about a year and it has faired very well. I found a few weird bugs along the way, but it has felt stable for quite a while now. I’m therefore finally making an official release. I wanted to call this version 1.0, but I’m going to play it safe and call it 0.9 for a few months and see if anything crops up in other peoples’ environments.
Before embarking on this I spent a long time talking to other developers to see if anything similar existed. The most similar I could see was Nameko, but I didn’t feel like that ticked my boxes. Lahja is another one which I saw pop up more recently. I’ve also written a short comparison with Celery, as it something I’m often asked about [2].
I’ve also spent a lot of time writing up the docs. A crude word count indicates I’ve written 23,400 words of narrative documentation :o
I was also keen to not strictly limit this too Python. I’ve therefore documented the Redis interactions required for other languages to interface with Lightbus[3], albeit without the various niceties that Lightbus provides).
Constructive feedback is very welcome!
[1]: https://github.com/adamcharnock/lightbus/issues/1
[2]: https://lightbus.org/explanation/lightbus-vs-celery/
[3]: https://lightbus.org/reference/protocols/