Building a GitOps Platform with Argo CD for Multi-Cluster Kubernetes

September 1, 2023

Overview

Modern platform teams need reliable, repeatable, and observable deployment workflows. As our Kubernetes environments expanded across development, staging, and production clusters, the existing deployment model began to show its limitations. To address this, we designed and implemented a GitOps-based continuous delivery platform using Argo CD, enabling automated, declarative deployments across multiple Kubernetes clusters. By making Git the single source of truth for infrastructure and application configuration, the platform introduced consistent deployment workflows, automatic drift detection, and significantly improved operational visibility.

The Problem

Before adopting GitOps, deployments were handled through a mix of approaches:

manual kubectl apply commands
Helm releases triggered from CI pipelines
direct cluster modifications during troubleshooting

While these methods worked individually, they created operational complexity when combined. Several issues began to surface:

Inconsistent deployment workflows across teams
Configuration drift between environments
Limited visibility into the actual state of clusters
Difficult rollbacks during production incidents
Slow incident response due to uncertainty about deployed versions

With multiple teams contributing to Kubernetes deployments, maintaining reliability and consistency became increasingly difficult. The platform team needed a solution that would enforce declarative infrastructure management and consistent release processes across clusters.

The GitOps Model

The platform adopted the GitOps model, where all cluster state is declared and version-controlled in Git repositories. In this model:

Desired cluster configuration is stored in Git.
Argo CD continuously monitors the repository.
The controller reconciles the Kubernetes cluster state with the declared configuration.
Any configuration drift is automatically corrected or flagged for review.

This approach ensures that the Git repository always represents the desired state of the platform. Key benefits include:

reliable and repeatable deployments
improved traceability of changes
automatic drift detection
simplified rollbacks through Git history

Repository Structure

To support multiple environments and applications, the Git repository was organized to leverage Helm charts with environment-specific values files. This structure provides a clean separation between application templates and environment configurations.

gitops-platform/
├── charts/                # Packaged application templates (Helm)
│   └── core-services/
├── values/                # Environment-specific configurations
│   ├── dev.yaml           # Development values
│   ├── staging.yaml       # Staging values
│   └── production.yaml    # Production values
└── clusters/              # Argo CD Root Application definitions

The charts/ directory contains reusable Helm charts that define the desired state of our applications. Environment-specific overrides are managed through Helm values files located in the values/ directory. This allows us to maintain a single chart while varying configurations like replica counts, resource limits, and ingress hosts across different clusters. By integrating Helm directly with Argo CD, the platform can automatically render manifests and detect drift based on the combination of the base chart and the specific environment's values file. This approach simplifies the management of complex deployments while ensuring that environment differences are transparent and version-controlled.

Multi-Cluster GitOps Architecture

The platform manages four Kubernetes clusters through a centralized Argo CD control plane.

Management Cluster

The management cluster hosts platform services such as:

Argo CD
monitoring stack
shared infrastructure components

This cluster acts as the GitOps control center for the entire platform.

Development Cluster

The development cluster enables rapid iteration.

Automatic synchronization with Git
Deployments triggered on merges to the main branch
Fast feedback cycles for developers

Staging Cluster

The staging environment acts as the validation stage before production. Deployments are promoted through pull requests to the staging values file, ensuring controlled release progression.

Production Cluster

Production deployments require an additional approval step. Manual synchronization is combined with Slack-based approval workflows, ensuring that critical changes undergo review before reaching live systems. This approach balances developer velocity with production stability.

Observability and Deployment Metrics

Visibility into deployment activity is essential for maintaining operational reliability. Argo CD metrics are exported to Prometheus, while Grafana dashboards provide real-time insights into deployment health. Key metrics include:

synchronization success and failure rates
drift detection events
deployment frequency per team
lead time from commit to production
rollback frequency

These metrics help the platform team understand deployment reliability and release velocity, enabling continuous improvement of the delivery pipeline.

Secret Management Strategy

Handling secrets securely in a GitOps workflow requires a robust integration between external secret stores and the Kubernetes ecosystem. The platform leverages a cloud-native approach to secret orchestration. AWS Secrets Manager & External Secrets Operator (ESO) Instead of storing sensitive data in Git, the platform uses AWS Secrets Manager as the centralized vault. The External Secrets Operator (ESO) is deployed within the Kubernetes clusters to bridge the gap.

Secrets are created and managed in AWS Secrets Manager.
ExternalSecret resources are defined in Git and synced via Argo CD.
The ESO controller fetches the sensitive data from AWS and dynamically creates native Kubernetes Secrets.

This ensures that no plain-text or encrypted secrets ever touch the Git repository, satisfying strict security and compliance requirements. Example External Secret configuration:

Yaml

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: db-secret
  data:
    - secretKey: password
      remoteRef:
        key: prod/database/credentials
        property: password

This strategy provides secure, automated secret management that integrates seamlessly with our GitOps and Helm-based delivery pipeline.

Results and Impact

Adopting GitOps with Argo CD significantly improved the reliability and efficiency of the deployment process. Key outcomes include:

~80% reduction in deployment errors
~60% faster incident detection through automated drift alerts
100% audit trail for infrastructure and application changes via Git history
Deployment frequency increased from weekly releases to multiple deployments per day
Mean time to rollback reduced from 30+ minutes to under 5 minutes

Most importantly, teams gained confidence in the deployment process. Instead of manually pushing changes into clusters, they now rely on declarative configuration and automated reconciliation.

Conclusion

Implementing a GitOps platform using Argo CD and Kubernetes transformed the way deployments are managed across environments. By making Git the source of truth for cluster configuration, the platform introduced:

consistent deployment workflows
automated drift detection
faster recovery during incidents
improved visibility into system changes

For organizations operating multiple Kubernetes clusters, GitOps provides a powerful framework for managing infrastructure and application delivery at scale. As platform engineering practices continue to evolve, GitOps will remain a foundational approach for building reliable and scalable cloud-native systems.

Technologies: Argo CD · Kubernetes · Helm · External Secrets Operator · Prometheus · Grafana · GitHub Actions · AWS Secrets Manager · AWS EKS