Saltar al contenido
ES EN

Guide: How to audit your Kubernetes GitOps like LinkedIn did with Argo CD

What LinkedIn discovered when pushing Argo CD to its limits, and how to check your own cluster for the two most painful symptoms at scale, without rewriting anything.

Software engineering

LinkedIn announced that it replaced Argo CD with its own GitOps engine, as reported by InfoQ. It operates over 250 clusters, 4,000 services, and more than 100,000 deployments per week. Deployment infrastructure failures dropped from 925 to 63 despite a 3.7x increase in volume.

Your environment is likely much smaller, but some of these problems appear sooner than you might think. This guide does not ask you to migrate: it proposes two checks that the article itself suggests for teams already using Argo CD at scale, plus a few diagnostic questions.

What you need

You need read access to the cluster with kubectl, permission to query the sync metrics of your Argo CD installation, and a couple of long-lived objects (Deployments or custom resources) with many years of edits. It's helpful to have the Argo CD documentation handy. Do this first in a test cluster or in read-only mode.

  1. Choose one or two long-lived objects and count how many different field managers appear in their managedFields.
  2. Note if any exceed ten distinct managers, the threshold at which Kubernetes truncates this information.
  3. Collect your applications' sync time and the total number of pods in the cluster over several days.
  4. Check if sync time increases as the number of pods grows, even if applications don't change.
  5. Review if pruning is disabled and if that leaves orphaned resources in production.

First check: field managers

LinkedIn discovered that Kubernetes truncates managedFields when an object has had more than ten different managers. Then the API server forgets who owns each field. In their AI workloads, with multiple containers per pod, a container removed from the manifest could survive in the live pod, without any visible errors.

kubectl get deployment mi-servicio -n mi-namespace \
  --show-managed-fields -o yaml | grep -c 'manager:'

The count is only indicative: the same manager can appear multiple times with different operations. The important thing is to detect objects with a long history and compare their actual state with the manifest you think you deployed. If pruning is disabled, as was the case for LinkedIn due to requiring approval in production, this divergence can go unnoticed.

Conceptual diagram of a Kubernetes pod with multiple containers and a manifest that no longer includes one of them
A container removed from the manifest can remain alive if ownership of its fields is lost.

Second check: latency vs. number of pods

Argo CD maintains one observation per resource type. In clusters with low type diversity, etcd noise from one application can slow down all synchronizations. LinkedIn measured less than 3 minutes with 60,000 pods and more than 10 minutes with over 80,000. If your sync time increases with the total number of pods, suspect this contention.

To decide what to do, compare with what others did: LinkedIn launches an ephemeral pod per stateless deployment, while Uber's Up platform assigns a dedicated controller to each deployment. Flux was discarded for sharing the same controller paradigm.

Tip Before you consider building anything, measure. If neither of the two checks shows symptoms, adjusting your current installation will almost always cost less than maintaining your own engine. Remember also that it took LinkedIn from July 2025 to April 2026 to migrate, and maintaining over 200 Argo CD instances turned every update into a weeks-long project.

How to interpret the results

  • Many field managers and disabled pruning: risk of silent orphaned resources; review manifests against actual state.
  • Sync time grows with the number of pods: observation contention; consider distributing across more instances or clusters.
  • Neither symptom: your problem is something else; don't copy LinkedIn's solution.
  • Remember that LinkedIn's engine is internal and no open-source version has been announced.

A final caution: LinkedIn clarifies that its improvement is not solely due to the new engine, because error classification and orchestrator reliability also improved during that period. If you want to delve deeper, QCon San Francisco brings together technical sessions from operators facing similar problems.

Original source: infoq.com

Article generated with AI.larebelion

Kernel

· Head of Technology · Spain

«Before copying LinkedIn, measure your sync latency: often the problem isn't Argo CD, it's something else.»

Comentarios