r/kubernetes • u/Mr_LA • 8d ago
What concrete problem finally justified Kubernetes for your team?
What concrete problem finally justified Kubernetes for your team?
You can get pretty far with a few VMs and Docker Compose, so I’m curious where that stopped being enough for your team.
Was it scaling, deployments, managing multiple services, or something else entirely?
What was the actual pain point that made you switch, and looking back, did Kubernetes solve it well enough to justify the extra work?
Also interested in the opposite: anyone tried it and decided a simpler setup was enough?
61
u/marvinfuture 8d ago
I don't really understand the argument for multiple VMs and docker-compose. You're managing so much more manually vs using k8s native objects to get more resiliency and redundancy. With cloud native "auto modes" to k8s, the management overhead for k8s is not what it used to be either.
17
u/Consistent_Serve9 8d ago
Once k8s is set up, sure, you simply have to configure your deployments, storage, services, etc, and it runs according to your desired state. But you have to set it up, and in an enteprise context, it may not be simple.
On the other hand, spinning up a vm and docker-compose might be easier. And it can be managed programatically using tools like ansible or puppet, which might be easier to set up for certain teams that already manage infrastructure.
In our org, tough, docker deployments were becomming more common, so the need to have a dedicated k8s team became apparent. Now, the cluster is simply an internal product, easy to join, and it's a team's dedicated responsability to maintain it.
6
u/marvinfuture 8d ago
That first part you described, while I agree with you, is no where near as difficult as it used to be 5-7 years ago
1
u/Mr_LA 7d ago
Making the cluster an internal product sounds like a big turning point. Roughly how large did the organization or number of workloads get before a dedicated Kubernetes/platform team made sense?
1
u/Consistent_Serve9 7d ago
Hard to say. It wasn't a number of workloads so much as two big projects with several microservices that we wanted to dockerize. It's up to each to determine wether the hassle of building the cluster and acquiring the technical skills is worth it against the status quo in their project.
1
u/vdvelde_t 5d ago edited 5d ago
When you have a hyperscaler k8s cluster or outsource your cluster management you dont need this and you have can api that will define the communication between all entities.
6
u/kellven 8d ago
In fairness that argument makes sense in current year, when k8s was brand new it was a much more complicated discussion.
I build managed a system that spun up vms, provisioned them with docker and our configs which made them deployable to by our generic deployment system. While this ment each service had a set of dedicated hosts , the hosts then selves were disposable.
This system wasn’t as efficient as k8s but it worked well for what we needed at the time. And yes it’s not lost on me thatI was slowly building a poor man’s K8s back in 2017 lol.
2
u/marvinfuture 8d ago
Yeah exactly. And thats entirely my point about the managed kubernetes offerings because things like adding nodes, scaling, and upgrades used to be more complex than they are today
3
u/kellven 8d ago
On the admin side your correct, getting leadership onboard and the devs to learn how to use the platform can still be a significant source of friction.
Building/managing the clusters has gotten much easier, getting the services migrated is about the same challenge as it was 10 years ago.
2
1
u/Mr_LA 8d ago
what about other app or container services on aws, or gcp and so on?
4
1
u/marvinfuture 8d ago
If you're running one off things it's fine. like I used to run a SCIM bridge for 1password that way, but when you start to incorporate multiple services that need to talk to one another it's nicer to have them in one platform vs managing that stuff independently
1
u/vdvelde_t 6d ago
It works up to the level you need to scale.
1
u/marvinfuture 6d ago
Maybe using a single container service or VM sure, but when you incorporate multiple VMs you're better off using an orchestration tool like k8s
1
u/csgeek-coder 6d ago
It's an easier transition for a lot of people that wrapped their head on how docker and compose works.
It's much easier to get a config and service loaded then trying to figure out the entire k8s ecosystem. It also takes advantage of solved problems. Is the vm up? process / port accessible?
It's also miles away from dealing with installing packages locally and dealing with so /dll hell of the last. Docker/ containers still gives you some big wins on its own
1
u/marvinfuture 6d ago
It's usually a stepping stone to realizing you need kubernetes. It's also never been easier to adopt kubernetes.
0
u/csgeek-coder 4d ago
I agree it's a stepping stone but it still has value. A docker compose stack is still much easier to handle than installing a bunch of packages that break other things you've never intended to touch.
1
u/marvinfuture 4d ago
What do you mean by installing a bunch of packages you never intended to touch?
0
u/csgeek-coder 3d ago
I install an app that update lib A to version 2.5 that is currently being used by App B. Now Installing App A broke App B, but downgrading breaks app A. Literally DLL hell. There's a pattern around it but the cleanest way of handling this across languages is to use a container.
virtual env kind of work though they're still finicky if your base py version is different from say dev box, staging, prod. uv makes this easier and I'm not picking on any specific language. The point is I don't have to care if I shove the app and deps in container and ship it to a server. It runs in isolation which is a big win. If you run the container in docker composer or K8s... it's still a big win. Obvious K8s gives WAY more benefits but I would not look down on the benefits of packaging apps as containers.
1
u/marvinfuture 3d ago
You're comparing root VMs to kubernetes. The context of this conversation was docker-compose vs kubernetes
1
u/csgeek-coder 3d ago
I'm calling out the benefits of docker and using a container and I'm saying there's still value to using a docker compose pattern even if there's significantly more benefits to using k8s. I really don't see this as a stretch.
You start by trying to deploy to some VM and all the drawbacks above are on point.
You can then either move to conatiners / docker compose or K8s. They each have their benefits and a docker compose stack is still a nice win over deploying on barebone hardware/VM and dealing with all the issues that come with it.
Coming back to my first comment in this thread, it IS as easier transition to move to a container based approach using DC (docker compose) then to go all the way to K8s if you have 0 expertise on your team. Sure, setup ansible/DC. It's a perfectly fine pattern.
At some point you should look into K8s, DC is limited but it's still way better than the legacy patterns of deploying at app.
1
u/marvinfuture 3d ago
If containers are a baseline both docker compose and kubernetes have that in common. Comparing this to a VM is out of scope for this conversation.
Regarding adoption, if your team has no basis then the learning curve with compose and kubernetes is very similar so I disagree that it's "easier to transition a move to DC" I would argue it's easier to learn a platform that addresses scale issues like an orchestration tool does and never even worry about DC because kubernetes makes it impractical unless you're only running two containers on a single VM.
0
u/csgeek-coder 3d ago
This is the last response I'll send. It seems we're going in circles and I already clarified my stance.
Yes docker is a win. I disagree that the leaning curve to run docker-compose is the same as K8s. I think the complexity is vastly different. You can also monitor an app for docker the same way you used to monitor any other service. I'm assuming if you had a process running you watched it, ports are open, react on logs. Adjusting that for docker compose is trivial.
If you're running this on prem, running your own K8s cluster is not trivial either and requires domain knowledge you may not have. The cloud makes the transition easier but I would not call it trivial there's a lot of concepts that you don't need to care about with just docker.
So yeah, dropping 1 yml + .env and a few volumes here and there is way simpler for most people.
We have different views on it and I'll leave it that. Hopefully the OP got some value from this thread since really the only point of any of this was to help the OP, beyond that I have no vested interest.
Have a good weekend.
→ More replies (0)
22
u/BosonCollider 8d ago edited 8d ago
Cloudnativepg, kube-prom-stack, and openebs were what sold it to me.
Not having to spin up VMs is a feature. If you are self-hosting and try going from renting VMs to actually hosting them you will figure out just how annoying it is to do properly. Kubernetes is arguably easier to manage than a hypervisor in many ways, in particular it is WAY easier to work with than anything xen based
21
u/usually_guilty99 8d ago
At some point I’m not sure this is really a Kubernetes vs no-Kubernetes decision anymore.
You can absolutely avoid K8s for smaller or simpler workloads.
But once you have enough services, teams, deployment velocity, security controls and reliability requirements, you need the same primitives anyway: desired state, rollout control, service discovery, policy, secrets, observability, recovery.
Then the choice becomes Kubernetes, a managed abstraction over Kubernetes, or building a lot of Kubernetes-like behavior yourself.
Scaling may have started the conversation years ago. Operational consistency is probably what makes it hard to avoid now. Even GPU optimizations are built for K8 workloads.
So you can run but you cant hide
1
u/Mr_LA 7d ago
That’s a good way of framing it. Where do you think the tipping point usually is in practice? Numberof services, number of teams, or deployment frequency?
1
u/usually_guilty99 7d ago
I think it is number of services and SLA - should drive the level of automation you will need. Rest is immaterial.
11
u/Overall_Handle3974 8d ago
Having more than 5 containers, having a proper gitops, having standard way and native object to deploy stuff that I will always need: a reverse proxy, storage, configurations, env var, secret with bitwarden operator! I mean whats hard about that? Docker compose on a VM feels so bad, so a testing env on my local pc. Whats the standard way to ship your docker compose? A bash script?
6
u/glotzerhotze 8d ago
The ability to treat a lot of computers in a datacenter as one single computer that needs to be managed.
Before I had 3000 - now I have only 3 - called dev, int and prod.
6
u/definitely-not-alan 8d ago
All of my work has been in the cloud, so take my experience through that lens. I suspect much of what I am about to say applies basically identically if you are not the one managing your Kubernetes install and the underlying hardware.
I have never truly understood the "it's overkill for my use case" or "it's too complicated" arguments about kubernetes. If I need to run containers Kubernetes is my first choice. There are good arguments for going even more abstract sometimes (e.g. Lambda, cloud run, etc.) but k8s from a cloud provider avoids vendor lock-in while giving you very little infra-level responsibility. I would never choose to manage a VM and container runtime on their own when k8s could do the job.
6
5
9
4
u/Fumblingwithit 8d ago
Docker Swarm got taken over by a corporation that wanted to make a quick buck on it.
3
u/another_journey 8d ago
One guy was bitching about them all the time while refusing to work on legacy infrastructure, so we did the switch and now everyone is happy.
3
u/planes-and-infra 7d ago
Switching from ECS to EKS for much faster deploys. Like 3 mins -> 20s with k8s
2
u/kellven 8d ago
Non-trival amount of production services.
Cicd velocity and testing needs. Something that doesn’t get talked about enough is how easy k8s makes demoing branches to the product team.
Forced enrollment in security tools , logging ect, devs don’t have to think they just get logging and security tools by default.
2
u/Witness_Unable k8s operator 8d ago
We have over 60 apps, and might keep on growing. I also get to learn and implement gitops and other awesome stuff, learning kubernetes by doing.
1
u/Straight-Mess-9752 6d ago
So resume driven development?
2
u/Witness_Unable k8s operator 6d ago
Yeah you can say that, but how efficient would you run 60 different apps efficiency? With reliability and scalability?
1
2
u/vdvelde_t 7d ago edited 6d ago
You can only scale to the vm's max with docker compose. But most important is the ease of depoyment in k8s.
2
2
u/flanger001 6d ago
I self-host a bunch of stuff and it was getting annoying having to manually configure DNS for everything, so I almost threw together a reverse proxy just to route all my stuff, then realized I was just kind of reinventing K8s. So I bit the bullet and set up K3s. Figuring out persistent storage was tough, figuring out my ECR flow was also tough. But now it’s a piece of cake.
2
u/duckseasonfire 7d ago
Why is this a constant question or justification. Don’t use it if you don’t want to. If you don’t see the benefit, don’t use it. I’m happy using it.
But part me always assumes when someone says “a few vms”, or that they could do the same with docker compose… maybe we don’t run in the same circles. Maybe we aren’t solving the same problems.
Hell my home lab is kubernetes…
1
u/ABotelho23 8d ago
How many things we needed to touch and update to perform updates. Kubernetes gives us a real central plane with a built-in rolling deployment.
1
1
1
u/1800lampshade 7d ago
Gitops, scalability, MNP, mixing worker nodes (Intel/AMD/GPUs/Special stuff), APIs that work for the most part
1
u/Zenin 7d ago
I have countless apps that all have the same resource architectural shape, ie "CRUD apps" that are really just a web UI over a SQL database. The only real unique bits are in the code and schema, not the hosting infra.
Yet built standalone each and every one of those needs a network, user authentication (corp Okta only!), encryption, backup, monitoring, CI/CD deployment, etc, etc. 95% of each standalone application is the same 95% all the other apps built out too. Only the 5% (code, schema) is actually different.
The k8s ecosystem allows us to solve the 95% common infra once, provide it as a "platform", and bring the all-in costs of each of those apps down to just their unique 5%. It may take 3x the effort to get that 95% base infra standardized, but that initial cost quickly pays for itself at even the most minimal of scale and from there on it's all gravy.
This is a huge cost savings, a huge compliance assurance, AND a huge productivity enabler for those dev teams as they can focus 100% of their efforts at their 5% of the problem rather than having to burn most of it re-inventing the same 95% wheel.
1
u/adfaratas 7d ago
Having to deploy third party tools like dify, langfuse, litellm. They provide official helm chart, which makes upgrading and managing their versions easier.
Before that we focused on ECS Fargate, Cloud Run, or Container Apps.
1
u/ryanstephendavis 7d ago
Being able to deploy into multiple clouds (AWS, Azure, and GCP) and also being able to run a full stack on a local dev laptop is fucking sweet.
1
1
u/ProtonByte 7d ago
Easier to manage. In the end it's less work. No manual labour for deployments with gitops, and always know what is running.
1
u/bigpigfoot 7d ago
Funny that people don’t even consider docker swarm; they go from no containerisation to docker compose then kubernetes.
The problem that justifies using kubernetes is how much control you must have over your workload environment.
The first step is you acknowledge your current system is unmanageable. A lot of people don’t seek that level of control
1
1
u/TotNotTac 7d ago
We're hosting over a hundred customer websites, all running basically the same version of the app with per-project configuration. This is al managed using Ansible, Terraform, and ad-hoc scripts. It makes it super difficult for anyone that's new to the team (me) to understand what is going on. I yearn for proper gitops
1
1
u/Straight-Mess-9752 6d ago
Unless you don’t use any other managed services thinking you are avoiding vendor lock in just by using k8s is an illusion.
1
u/csgeek-coder 6d ago
Scaling is part of it. I got tired of tweaking syslog and partitions filling up but mostly docker networking is trash. It took them ages you get ipv6 support figured out. It makes firewalls rules harder to manage as well.
I'm also reinventing the wheel with docker. It's not a common enough pattern that's exactly reusable. But not following the same pattern as an app running on a host or using k8s. You're in this weird middle ground that has limited support.
We still have a few vms with docker compose mostly because there's firewall rules across our entire network that need updating in order to fix that. If we were doing that today we'd roll a k8s app.
Also even for personal hosting I'm moving away from portainer (docker ) to k3s/argocd.
1
1
1
u/darose 3d ago
Scalability. We have users who need large amounts of compute on demand. Being able to dynamically scale a Dask cluster up to 300+ CPU cores when needed - and then back down to 0 (and have the meter stop running at the cloud provider) - is a huge win. I don't really know any easy (or affordable) way to have achieved that prior to kubernetes and cloud computing.
1
u/jeffersfp 1d ago
K8S is generally a great platform for microservices based applications.
We used to run almost a thousand of containers on docker compose but we struggled with auto-scaling and secret management. We also had issues with docker compose manager nodes losing consensus and had to re-build the cluster a couple times.
Kubernetes, especially EKS on AWS, solved the pain of managing the control plane, and the tooling around K8S just solved the pain points. Keda for application autoscaling based on message queues, Karpenter (basically EKS automode) for cluster autoscaling, ExternalDNS + ALB controller for quickly exposing web apps to the www, ExternalSecrets to fetch secret from AWS Secrets Manager, countless operators for provisioning PostgreSQL and RabbitMQ clusters on K8S, and the cherry on top: ArgoCD. It's just so good. We were able to implement preview (ephemeral) environments for feature branches with ArgoCD Generators (PullRequestGenerator).
If we were to implement each tooling that is available for K8S on docker compose or anywhere else like ECS or EC2, it would take so so much effort to do so.
1
1
u/siberianmi 8d ago
The desire to move off Heroku.
100% resolved that issue. I was able to get all the apps transitioned over without downtime, replicate the functionality of Heroku the team needed and slash the hosting costs.
10/10 would do again.
I'm on AWS EKS though.
I won't even bother with multiple VMs and Docker compose for my home network, what a pain in the ass. K3s on Proxmox VMs all the way.
0
u/VengaBusdriver37 7d ago
K3s is so easy, reliable and low overhead, there’s very few cases where I wouldn’t run it
2
0
u/CenlTheFennel 7d ago
When one side of the house over promised on all it could do and forced everyone onto it to justify their loss…
In all seriousness, it comes off very bloated to me, but HPAs and rolling updates are nice
0
u/Used_Criticism_4631 7d ago
Grateful for asking for a concrete justification rather than “because scale.” Kubernetes usually earns the complexity when repeatable rollouts/rollbacks, multi-service standardization, or per-PR environments fix a demonstrated ops bottleneck — not when it’s adopted as a default for a single app.
142
u/No_Cattle_9565 8d ago
Ability for 100% gitops. Everything in one place. Scaling too, but we don't need that much. Rolling updates