It really slowed down development because it made the feedback loops really long - you might have to start up five different JVMs, which would each take a while to register itself with Hazelcast before you could start the next that depended on it, before you could test your change. The service boundaries we had initially come up with weren't always correct (as you'd expect really), but it was a massive effort to refactor something across a service boundary, because each component was released and versioned separately.
Maybe there are ways to work in such an environment, but if so we didn't find them and no-one had the experience to know them.
> because each component was released and versioned separately.
For SOA a mono-repo with synchronized releases is a huge help.
In one commit you can refactor a shared module and update all apps that depend on it. And you know when you deploy to staging or prod that all machines got the latest code
At what scale? At Google we had to version everything within a commit, i.e. files that were pushed to different binaries had to make sure they were backwards- and forwards-compatible with the previous version. Your code had to work even if it was talking to another server that was stuck on the previous commit, or even several releases back.
The reason is that when you have 1000s of machines, the push will fail for some. Either the machine will be offline and out of service when the new version is released, or its network connection may be down, or a cosmic ray may flip a bit and make the process crash when installing, triggering a rollback. Particularly when handling user data, you need to code defensively around these and not assume that the server you're talking to has the same code you just wrote in your commit.
Obviously these problems don't manifest when you're pushing to 1-3 machines, but if your deployment is that small, why not run it all in-process with a monolithic app anyway?
Great point. Different scale needs different things for sure. Even at the scale that I've found this successful in you still have to consider that backward/forward compatibility, albeit at a much coarser grain.
Agreed. Keeping things in the same source tree is an advantage that a lot of people don't consider. It's worked very well for Linux and other large projects that have many separate components that benefit from being maintained in a single place.
My company has had similar issues with code modules. The full app was rarely in a working state because the various modules were in a constant state of flux (module A requires module B from branch X, not master, and module C needs to be attached with this temporary script until the API migration is completed, etc).
We ended up merging some of the modules together just to get our agility back.
None of that sounds unsurmountable or inherent to microservice architectures. What's hard about starting five JVMs? Why did it take "a while" to register with Hazelcast? Why would you expect to initially come up with bad service boundaries, and why is switching two services you've refactored at once difficult?
I think that none of these are problems inherent to microservice architecture, but to someone learning by designing such a system for the first time.
Maybe there are ways to work in such an environment, but if so we didn't find them and no-one had the experience to know them.