What LinkedIn discovered when pushing Argo CD to its limits, and how to check your own cluster for the two most painful symptoms at scale, without rewriting anything.
LinkedIn announced that it replaced Argo CD with its own GitOps engine, as reported by InfoQ. It operates over 250 clusters, 4,000 services, and more than 100,000 deployments per week. Deployment infrastructure failures dropped from 925 to 63 despite a 3.7x increase in volume.
Your environment is likely much smaller, but some of these problems appear sooner than you might think. This guide does not ask you to migrate: it proposes two checks that the article itself suggests for teams already using Argo CD at scale, plus a few diagnostic questions.
What you need
You need read access to the cluster with kubectl, permission to query the sync metrics of your Argo CD installation, and a couple of long-lived objects (Deployments or custom resources) with many years of edits. It's helpful to have the Argo CD documentation handy. Do this first in a test cluster or in read-only mode.
- Choose one or two long-lived objects and count how many different field managers appear in their
managedFields. - Note if any exceed ten distinct managers, the threshold at which Kubernetes truncates this information.
- Collect your applications' sync time and the total number of pods in the cluster over several days.
- Check if sync time increases as the number of pods grows, even if applications don't change.
- Review if pruning is disabled and if that leaves orphaned resources in production.
First check: field managers
LinkedIn discovered that Kubernetes truncates managedFields when an object has had more than ten different managers. Then the API server forgets who owns each field. In their AI workloads, with multiple containers per pod, a container removed from the manifest could survive in the live pod, without any visible errors.
kubectl get deployment mi-servicio -n mi-namespace \
--show-managed-fields -o yaml | grep -c 'manager:'The count is only indicative: the same manager can appear multiple times with different operations. The important thing is to detect objects with a long history and compare their actual state with the manifest you think you deployed. If pruning is disabled, as was the case for LinkedIn due to requiring approval in production, this divergence can go unnoticed.

Second check: latency vs. number of pods
Argo CD maintains one observation per resource type. In clusters with low type diversity, etcd noise from one application can slow down all synchronizations. LinkedIn measured less than 3 minutes with 60,000 pods and more than 10 minutes with over 80,000. If your sync time increases with the total number of pods, suspect this contention.
To decide what to do, compare with what others did: LinkedIn launches an ephemeral pod per stateless deployment, while Uber's Up platform assigns a dedicated controller to each deployment. Flux was discarded for sharing the same controller paradigm.
How to interpret the results
- Many field managers and disabled pruning: risk of silent orphaned resources; review manifests against actual state.
- Sync time grows with the number of pods: observation contention; consider distributing across more instances or clusters.
- Neither symptom: your problem is something else; don't copy LinkedIn's solution.
- Remember that LinkedIn's engine is internal and no open-source version has been announced.
A final caution: LinkedIn clarifies that its improvement is not solely due to the new engine, because error classification and orchestrator reliability also improved during that period. If you want to delve deeper, QCon San Francisco brings together technical sessions from operators facing similar problems.
Original source: infoq.com
Article generated with AI.larebelion



Publicar un comentario