ArgoCD & GitOps — Air-Gapped Delivery Pipeline

Prerequisite — ArgoCD must be installed first

This page describes the GitOps pipeline — how ArgoCD is connected to GitLab and how workloads flow from Git to the cluster. ArgoCD installation itself (air-gap Helm pattern, image mirroring, CLI setup) is covered on the previous page: ArgoCD — Air-Gapped Installation.

Overview

Every workload on the platform is delivered by ArgoCD reconciling from GitLab. In the air-gapped cl1.sq4 environment there is no path to the public internet, so all sources are internal:

Component	Purpose	Where
GitLab (gitlab.cl1.sq4.aegis.internal)	Source of truth — Git repos and the Helm chart registry ArgoCD pulls from	in-cluster (namespace gitlab)
Harbor (harbor.cl1.sq4.aegis.internal)	Container image registry — every image is mirrored here	Harbor VIP 100.115.1.101
ArgoCD	GitOps controller — reconciles the cluster to the desired state declared in Git	Kubernetes (argocd namespace)
Vault + VSO	Supplies ArgoCD's read-only GitLab token as a Kubernetes secret	Kubernetes (vault / app namespaces)

Application source code originates outside the environment and is imported into the internal GitLab (pull-mirror / controlled import). From that point on, the internal GitLab is authoritative — developers and operators never point the cluster at an external URL.

Architecture

flowchart LR
    subgraph ext["External (out of band)"]
        src["Source repos\n(corporate GitNon-production)"]
    end

    subgraph aegis["Air-Gapped — cl1.sq4.aegis.internal"]
        gitlab["GitLab\nGit repos + Helm registry\ngitlab.cl1.sq4.aegis.internal"]
        harbor["Harbor\nimage registry\nharbor.cl1.sq4.aegis.internal"]
        argocd["ArgoCD\n(argocd namespace)"]
        k8s["RKE2 Workloads"]
    end

    src -- "controlled import / mirror" --> gitlab
    argocd -- "watches Git" --> gitlab
    argocd -- "pulls Helm charts" --> gitlab
    argocd -- "deploys" --> k8s
    k8s -- "pulls images" --> harbor

Access URLs

Service	URL
ArgoCD UI	https://argocd.cl1.sq4.aegis.internal
GitLab UI	https://gitlab.cl1.sq4.aegis.internal

---

How ArgoCD is connected to GitLab

ArgoCD needs two things from GitLab: Git (to read manifests) and the Helm chart registry (to pull packaged charts). Both are registered as ArgoCD repository secrets, authenticated with a dedicated read-only GitLab token — never a personal or admin account.

Secret	Type	URL	Covers
gitlab-git-creds	repo-creds (prefix template)	https://gitlab.cl1.sq4.aegis.internal/modgpt/	Every Git repo under modgpt/ — auto-covers new app repos
gitlab-helm-repo	repository (explicit)	https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/<PID>/packages/helm/stable	The project's Helm chart registry

• Token: the argocd-ro GitLab Group Access Token — scopes read_api, read_repository, read_registry. Stored in Vault (secret/argocd/gitlab-repo) and rendered into the argocd namespace by the Vault Secrets Operator. See [feedback: dedicated service-account PATs].
• CA trust: the internal CA is registered in the argocd-tls-certs-cm ConfigMap so the ArgoCD repo-server trusts gitlab.cl1.sq4.aegis.internal over HTTPS without TLS errors.
• Helm over HTTP, not OCI: ArgoCD pulls charts from GitLab's HTTP package registry (/api/v4/projects/<PID>/packages/helm/stable), not the OCI endpoint. This avoids the multi-level OCI auth-scope issue — see Troubleshooting.

Full credential setup: GitLab → Connect ArgoCD to GitLab.

---

Prerequisites check

Run from the bastion before working with the pipeline:

kubectl get nodes                       # all nodes Ready
kubectl get pods -n argocd              # ArgoCD pods Running
argocd version --client                 # CLI installed
helm version                            # v3.x

# ArgoCD can see the GitLab repositories
argocd repo list                        # gitlab.cl1.sq4.aegis.internal entries: Successful

---

Publishing a Helm chart to GitLab

Charts are packaged and pushed to the GitLab Helm registry (stable channel). Run from the bastion, authenticated with the GitLab token from Vault:

GITLAB_TOKEN=$(vault kv get -field=token secret/gitlab/migration-token)
PID=<gitlab-project-id>

helm package ./charts/<chart> -d /tmp/
curl --fail -u "argocd-ro:${GITLAB_TOKEN}" \
  "https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/${PID}/packages/helm/api/stable/charts" \
  --form "chart=@/tmp/<chart>-<version>.tgz"

# verify it is visible
curl -s -u "argocd-ro:${GITLAB_TOKEN}" \
  "https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/${PID}/packages/helm/stable/index.yaml" \
  | grep -A2 "<chart>:"

---

Declaring an Application (multi-source)

A production Application pulls the chart from the GitLab Helm registry and its values from the app's Git repo — a two-source Application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: modgpt.<app>.<service>
  namespace: argocd
spec:
  project: modgpt.<app>
  destination:
    server: https://kubernetes.default.svc
    namespace: <app>
  syncPolicy:
    automated: { prune: true, selfHeal: true }
    syncOptions: [CreateNamespace=true]
  sources:
    - repoURL: https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/<PID>/packages/helm/stable
      chart: <service>
      targetRevision: "1.0.0"
      helm:
        releaseName: <service>
        valueFiles: ["$values/values/<service>/<env>.yaml"]
    - repoURL: https://gitlab.cl1.sq4.aegis.internal/modgpt/<app>.git
      targetRevision: HEAD
      ref: values

Images referenced by the chart resolve to harbor.cl1.sq4.aegis.internal/ask/<image> and pull with the harbor-pull-secret in the app namespace.

Don't hand-write Applications app by app

The platform uses a 3-layer App-of-Apps so Applications are themselves declared in Git. Read GitOps Architecture (3-Layer App-of-Apps) for the structure, and How-To → Create a New App Group for the exact commands to onboard a new application group.

---

App-of-Apps — the root that manages everything

A single root Application (modgpt) points at the cluster-addons repo and bootstraps every downstream Application:

argocd app create modgpt \
  --repo https://gitlab.cl1.sq4.aegis.internal/modgpt/cluster-addons.git \
  --path root-bootstrap \
  --dest-server https://kubernetes.default.svc \
  --dest-namespace argocd \
  --sync-policy automated --auto-prune --self-heal

From there, Layer 2 (modgpt.<app>) and Layer 3 (modgpt.<app>.<service>) Applications are created from Git — no manual kubectl apply per service. See 15 — App-of-Apps.

---

Day-2 — shipping a new version

# 1. Package + publish the new chart version to the GitLab Helm registry
helm package ./charts/<chart> -d /tmp/
curl --fail -u "argocd-ro:${GITLAB_TOKEN}" \
  "https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/${PID}/packages/helm/api/stable/charts" \
  --form "chart=@/tmp/<chart>-<new-version>.tgz"

# 2. Bump targetRevision (chart) or image.digest (image) in the app's values repo, commit + push to GitLab
# 3. ArgoCD detects the change and syncs automatically (or: argocd app sync modgpt.<app>.<service>)

For image releases (mirror ACR → Harbor, pin digest, sync, verify) follow the Release Strategy runbook.

---

Useful commands

argocd app list                              # all Applications + sync/health
argocd app get modgpt.<app>.<service>        # detail for one service
argocd app sync modgpt.<app>.<service>       # force a sync
argocd repo list                             # registered GitLab Git + Helm repos

# list Helm packages published in a GitAir-gap project
curl -s -u "argocd-ro:${GITLAB_TOKEN}" \
  "https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/${PID}/packages/helm/stable/index.yaml"