Between monoliths and microservices you have services and sidecars. If you don’t at least have sidecars I really don’t see the point of kubernetes, because most of the rest of the services will follow Conway’s Law and can reasonably do their own thing for less than 125% of the cost of full bin packing.
Things like CertManager and ExternalDNS take away so much operational time dealing with those two things alone. There’s a lot of good infra automation. That being said, I’m a lot bigger on the Fly Machine, Compute-at-Edge trend. If only they had a good IaC solution (after abandoning Terraform and the lack of features in fly.toml).
I've worked on a lot of very large Kubernetes projects and none used sidecars widely.
The major benefit of Kubernetes for them was that you could use lots of cheap, ephemeral cloud instances to run your platform whilst still having high availability. It ended up saving a ridiculous amount of money.
Hundreds, in my case. We have a sidecar here or there, but essentially our entire operation is run in distroless containers that consume config maps. We have one source of truth for 5 baremetal cloud regions, a number of private on-prem cloud regions, our build and test infrastructure, and nearly everything else, it is our Argo repo and the auto-generated operator manifests from our operator mono-repo. We have a common client library that abstracts our CRDs into easy to consume functions, and in the end using Kubernetes as an API for infrastructure operations does exactly what it should; allows full consistency and visibility on configuration.
you did not even understood what k8s does. trust me you want something like nomad or k8s. else you will write your simple k8s thing anyway, its just harder to understand since you just wrote your own solution. k8s even works in small scale, heck it didn't even run at first in really big deployments.
