Using confluent platform community (free) gives you a working env with Kafka/Zookeeper/SchemaRegistry/Connect immediately without having to bother with setting up clusters, etc. Beyond that, yes, it's a pain to properly setup and I think most of the time you're looking at managed options unless you want to invest in a SRE team.
Isn‘t redpanda a reimplementation of kafka without the jvm and zookeeper? I mean how should they api compatible if they don‘t implement the same concepts.
This [1] could be a solution to most of author’s problem. Simple to use / few moving parts, very high perf, native HTTP produce and consume, clear and simple processing model with strong guarantees, among other things.
Kafka Streams are also amazing, and would have solved many problems I've encountered in the past better than our microservices moving data around themselves.
Would you mind expanding the use cases you would have used Kafka streams for? And why no other solutions could have helped? (Performance or complexity etc)
Where did that idea come from? He literally wrote how they did it without streams by moving data between their Microservices, likely with synchronous http calls. He would've just preferred to just write a stream instead of the services.
There have been like three iterations until we've been able to actually release an stable integration that deals well with high traffic. All together, almost 6 months.
I guess that it is part of the curse of knowledge.
Even the article starts by
"I just want to share my thoughts on Kafka after using it for a few months, always from a practical point of view. I don’t know anything more than the basics about Kafka internals. "
It is just confusing for anyone that does not know that Apache Kafka is a software application to process queues.
The article is named "Starting with Kafka" but assumes that you have knowledge of what it is, its use cases, and other not-so-basic information.
I think if someone were to write an article on the writer Kafka they would not phrase the headline "Starting with ... " but something like "Reading Kafka" or "How to read Kafka". I am not a native speaker, though.
