Lets say you have a small team and dev, test, prod environments. You want high availability. You need kafka and zookeeper servers, 2-3 of each so let's say 5 total. Dev can be single servers, but you should probably have an environment that closely mirrors prod, so for your Kafka stack alone you have to manage 12 servers, 5 for prod/test, 2 for dev.
Then you probably have Kafka connect running somewhere. That's another handful of servers. Maybe Kafka streams is a few more servers. Then you're going to have servers that collect events and publish them to the server. How many more servers is that per environment?
Congratulations, you now have a state of the art event streaming enterprise grade platform, and 20 servers to manage. Better hope your company gets on board with the real time model, otherwise you're now the owner of these pet servers until the end of time.
What's that, you ran into a rare kafka bug that caused your offsets to be lost after a reboot and now the bus is pushing millions of messages down the pipe to all the consumers? Wow, that sure sucks, hope you can juggle your day job with this massive production issue.
Then you probably have Kafka connect running somewhere. That's another handful of servers. Maybe Kafka streams is a few more servers. Then you're going to have servers that collect events and publish them to the server. How many more servers is that per environment?
Congratulations, you now have a state of the art event streaming enterprise grade platform, and 20 servers to manage. Better hope your company gets on board with the real time model, otherwise you're now the owner of these pet servers until the end of time.
What's that, you ran into a rare kafka bug that caused your offsets to be lost after a reboot and now the bus is pushing millions of messages down the pipe to all the consumers? Wow, that sure sucks, hope you can juggle your day job with this massive production issue.