DevOps Peach

GitOps with Argo CD · Argo CD Security and Operations · Lesson 4 of 4

Troubleshooting Argo CD

Most problems are one of a few: endless OutOfSync from fields other controllers own, stuck syncs, apps that will not delete, or repo and render errors. Each has a standard fix.

Practice interactively →

Start every investigation with the same three commands:

argocd app get shop-api-prod            # conditions, sync result, resource list
argocd app diff shop-api-prod           # what exactly differs
kubectl -n argocd logs deploy/argocd-repo-server --tail=50   # render errors

1. OutOfSync that never goes away

Another controller (HPA, a mutating webhook, an operator, cert-manager injecting a CA bundle) changes fields Argo CD also manages. Tell Argo CD to ignore those fields:

spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers: [/spec/replicas]
    - group: admissionregistration.k8s.io
      kind: MutatingWebhookConfiguration
      jqPathExpressions: ['.webhooks[]?.clientConfig.caBundle']
  syncPolicy:
    syncOptions: [RespectIgnoreDifferences=true]   # also skip these fields when syncing

Use it narrowly — ignoring a field also hides real drift there. System-wide rules go in argocd-cm under resource.customizations.ignoreDifferences.<group>_<kind>.

2. A sync that hangs or fails

Message / symptom Cause Fix
Waiting for a PreSync hook The Job never completes argocd app terminate-op, fix the Job, add activeDeadlineSeconds
admission webhook ... denied the request A policy engine (Kyverno, PSA) rejects an object Fix the manifest; check the policy report
the server could not find the requested resource A CRD is not installed yet Install the CRD in an earlier wave, or SkipDryRunOnMissingResource=true
exceeded quota ResourceQuota in the namespace Lower requests or raise the quota
metadata.annotations: Too long A very large object with client-side apply ServerSideApply=true

3. An app that will not delete

argocd app delete waits while the finalizer resources-finalizer.argocd.argoproj.io removes child objects. If a child is stuck (its own finalizer, a namespace terminating), find it with argocd app resources and fix that. Removing the Application's finalizer by hand leaves orphaned workloads — only do it on purpose.

4. Unknown status and render errors

5. Performance at scale

Hundreds of apps across clusters: shard the application controller (controller.sharding.algorithm, more replicas), increase --status-processors / --operation-processors, and exclude noisy resource kinds you do not manage (resource.exclusions).

A short runbook

  1. Read conditions in argocd app get — the message is usually precise.
  2. argocd app diff for OutOfSync, kubectl describe / events for Degraded.
  3. Check the component log that owns the step: repo server (render), application controller (compare/sync), server (login/RBAC).
  4. Fix in Git when the cause is a manifest; fix in Argo CD config (also in Git) when it is a setting.

Practice interactively →