15 — ArgoCD GitOps Architecture (3-Layer App of Apps)¶

The platform uses a 3-layer recursive App of Apps pattern — no ApplicationSet, no generators. Every deployed service is an explicit YAML file in git, auditable and independently rollback-able.

Current implemented state

This page reflects the live cluster. For the operational runbook (adding services, app groups, updating config), see How-To → Update ArgoCD Config and How-To → Create a New App Group.

The 3-Layer Hierarchy

LAYER 1 — Root (manual bootstrap, once)
  AppProject: modgpt                    ← tenancy boundary
  Application: modgpt                   ← root tile; syncs root-bootstrap/

LAYER 2 — App Groups (GitOps via modgpt Application)
  AppProject: modgpt.cluster-addons     ← group boundary: repos + namespaces
  Application: modgpt.cluster-addons    ← app of apps; syncs bootstrap/

  AppProject: modgpt.ask                ← (future) group boundary
  Application: modgpt.ask               ← (future) app of apps

LAYER 3 — Services (GitOps via each Layer 2 Application)
  Application: modgpt.cluster-addons.vault
  Application: modgpt.cluster-addons.vso
  Application: modgpt.cluster-addons.reloader
  Application: modgpt.cluster-addons.pdb

Platform Ownership — From Tenant to Service

flowchart TD
    subgraph L1 ["Layer 1 — Root (manual)"]
        proj_root["AppProject: modgpt"]:::manual
        app_root["Application: modgpt"]:::manual
    end

    subgraph L2 ["Layer 2 — App Groups (GitOps)"]
        proj_ca["AppProject: modgpt.cluster-addons"]:::l2
        app_ca["Application: modgpt.cluster-addons"]:::l2
        proj_ask["AppProject: modgpt.ask"]:::l2
        app_ask["Application: modgpt.ask"]:::l2
    end

    subgraph L3ca ["Layer 3 — cluster-addons services"]
        vault["Application\nmodgpt.cluster-addons.vault"]:::l3
        vso["Application\nmodgpt.cluster-addons.vso"]:::l3
        pdb["Application\nmodgpt.cluster-addons.pdb"]:::l3
        reloader["Application\nmodgpt.cluster-addons.reloader"]:::l3
    end

    subgraph L3ask ["Layer 3 — ask services"]
        cs["Application\nmodgpt.ask.core-service"]:::l3
    end

    proj_root -. scopes .-> app_root
    app_root --> proj_ca & app_ca
    app_root --> proj_ask & app_ask
    app_ca --> vault & vso & pdb & reloader
    app_ask --> cs

    classDef manual fill:#fdecea,stroke:#c0392b,color:#111,font-weight:bold
    classDef l2 fill:#fff3cd,stroke:#856404,color:#111
    classDef l3 fill:#e8f1fc,stroke:#2c6fbb,color:#111

RBAC Boundary at Each Layer

Each AppProject constrains what the Applications in its layer can do — which repos they pull from and which namespaces they deploy into. Boundaries tighten as you go deeper.

Layer	AppProject	Allowed Sources	Allowed Destinations
1	`modgpt`	`gitlab.cl1.sq4.aegis.internal/modgpt/*` + any project Helm registry	in-cluster, any namespace
2	`modgpt.cluster-addons`	cluster-addons git + cluster-addons Helm registry	any namespace
2	`modgpt.ask`	ask git + ask Helm registry	any namespace
3	(none — uses Layer 2)	inherited from `modgpt.cluster-addons` or `modgpt.ask`	constrained by Application `destination.namespace`

Why no Layer 3 AppProjects?

The Layer 3 Application spec already constrains destination.namespace (e.g. vault). The Layer 2 AppProject constrains the source repos. A per-service AppProject would add YAML overhead with no RBAC benefit for a single operator. Add Layer 3 AppProjects when you introduce per-service team ownership or per-service CI tokens. See KB-03 for the full analysis.

Repository Structure

cluster-addons/                              ← gitlab.cl1.sq4.aegis.internal/modgpt/cluster-addons
│
├── root-bootstrap/                          ← Layer 1 (modgpt) syncs here
│   ├── appproject-cluster-addons.yaml       ← Layer 2 AppProject  (sync-wave: -1)
│   └── application-cluster-addons.yaml      ← Layer 2 Application (sync-wave:  0)
│
├── bootstrap/                               ← Layer 2 (modgpt.cluster-addons) syncs here
│   ├── application-vault.yaml               ← Layer 3 Applications
│   ├── application-vso.yaml
│   ├── application-reloader.yaml
│   └── application-pdb.yaml
│
├── values/                                  ← Helm values per service
│   ├── vault/vault-stg.yaml
│   ├── vso/vso-stg.yaml
│   ├── reloader/reloader-stg.yaml
│   └── argocd/argocd-stg.yaml              ← bootstrap-only (manual helm upgrade)
│
└── infra/pdb/                               ← Raw PDB manifests
    ├── argocd-pdb.yaml
    ├── vault-pdb.yaml
    └── ...

Sync wave ordering within a layer¶

When a Layer 2 Application syncs root-bootstrap/ or bootstrap/, objects with lower sync-wave numbers are created first. AppProjects use wave -1 so they exist before the Applications (wave 0) that reference them:

# AppProject — always wave -1
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"

# Application — wave 0 (AppProject exists by now)
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "0"

Bootstrap Manifests (Layer 1 — manual, apply once)

These two objects are not in git — they are the hand-applied seed that starts the GitOps chain. Keep these manifests saved externally for cluster rebuild.

AppProject `modgpt`¶

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: modgpt
  namespace: argocd
spec:
  description: "modgpt tenant — root boundary"
  sourceRepos:
    - "https://gitlab.cl1.sq4.aegis.internal/modgpt/*"
    - "https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/*/packages/helm/stable"
  destinations:
    - server: "https://kubernetes.default.svc"
      namespace: "*"
  clusterResourceWhitelist:
    - { group: "*", kind: "*" }
  namespaceResourceWhitelist:
    - { group: "*", kind: "*" }

Root Application `modgpt`¶

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: modgpt
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.io
spec:
  project: modgpt
  source:
    repoURL: https://gitlab.cl1.sq4.aegis.internal/modgpt/cluster-addons.git
    targetRevision: HEAD
    path: root-bootstrap
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated: { prune: true, selfHeal: true }

# Apply once on a fresh cluster
kubectl apply -f appproject-modgpt.yaml
kubectl apply -f application-modgpt.yaml
# Everything else flows from git from this point

ArgoCD → GitLab Credentials

ArgoCD reads git manifests and Helm charts from GitLab using a dedicated read-only token, VSO-rendered into two repository secrets.

Secret	Type	URL pattern	Used for
`gitlab-git-creds`	`repo-creds` (prefix template)	`https://gitlab.cl1.sq4.aegis.internal/modgpt/`	All git repos under `modgpt/` — auto-covers every new app repo
`gitlab-helm-repo`	`repository` (explicit)	`https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/1/packages/helm/stable`	cluster-addons Helm charts

Token: argocd-ro Group Access Token — scopes read_api, read_repository, read_registry. Stored in Vault (secret/argocd/gitlab-repo), rendered by VSO.
HTTP Helm, not OCI: ArgoCD v3.4.2 double-encodes the JWT auth scope for multi-level OCI paths → GitLab 403. HTTP package registry avoids this. See Issue 39 in Troubleshooting.
Internal CA trust: the internal CA is registered in argocd-tls-certs-cm so the repo-server trusts gitlab.cl1.sq4.aegis.internal over HTTPS without TLS errors.

Full setup: 12 — GitLab → Connect ArgoCD to GitLab.

Adding a Service (Day-2)

Add a new add-on to cluster-addons¶

cd ~/cluster-addons

# 1. Mirror chart + images into GitLab (air-gap)
#    helm push chart.tgz https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/1/packages/helm/stable
#    skopeo copy docker://<upstream>/<image>:<tag> docker://harbor.cl1.sq4.aegis.internal/ask/<image>:<tag>

# 2. Add values file
mkdir -p values/<svc>
vi values/<svc>/<svc>-stg.yaml

# 3. Add Application YAML to bootstrap/
cat > bootstrap/application-<svc>.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: modgpt.cluster-addons.<svc>
  namespace: argocd
spec:
  project: modgpt.cluster-addons
  destination:
    server: https://kubernetes.default.svc
    namespace: <svc>
  syncPolicy:
    automated: { prune: true, selfHeal: true }
    syncOptions: [CreateNamespace=true]
  sources:
    - repoURL: https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/1/packages/helm/stable
      chart: <chart>
      targetRevision: "<version>"
      helm:
        releaseName: <svc>
        valueFiles: ["$values/values/<svc>/<svc>-stg.yaml"]
    - repoURL: https://gitlab.cl1.sq4.aegis.internal/modgpt/cluster-addons.git
      targetRevision: HEAD
      ref: values
EOF

# 4. Commit and push
GITLAB_TOKEN=$(vault kv get -field=token secret/gitlab/migration-token)
git add bootstrap/ values/
git commit -m "feat: add <svc>"
git remote set-url origin https://oauth2:${GITLAB_TOKEN}@gitlab.cl1.sq4.aegis.internal/modgpt/cluster-addons.git
git push origin main
# modgpt.cluster-addons creates modgpt.cluster-addons.<svc> automatically

Add a new app group (e.g. ASK)¶

See How-To → Create a New App Group for the full runbook.

Summary: add root-bootstrap/appproject-ask.yaml + root-bootstrap/application-ask.yaml to the cluster-addons repo and push. The modgpt root app creates the Layer 2 pair automatically.

Upgrade a service¶

vi bootstrap/application-vault.yaml    # bump targetRevision
vi values/vault/vault-stg.yaml         # bump image tags
git commit -am "chore: upgrade vault to 0.33.0" && git push origin main

Stateful apps (Vault, etcd …) — OnDelete only

StatefulSets with updateStrategy: OnDelete do not auto-roll on spec change. Roll one pod at a time, verify quorum/unseal between each. See KB-01 — Raft / Quorum Clusters.

Verification

# Full hierarchy — expected: all Synced/Healthy
kubectl get appproject -n argocd
kubectl get application -n argocd

# Expected AppProjects
# modgpt                    ← L1 manual
# modgpt.cluster-addons     ← L2 GitOps

# Expected Applications
# modgpt                           Synced  Healthy
# modgpt.cluster-addons            Synced  Healthy
# modgpt.cluster-addons.vault      Synced  Healthy
# modgpt.cluster-addons.vso        Synced  Healthy
# modgpt.cluster-addons.reloader   Synced  Healthy
# modgpt.cluster-addons.pdb        Synced  Healthy

Troubleshooting

Symptom	Cause	Fix
Layer 2 app `project not found` on sync	AppProject created in same wave as Application	Ensure AppProject has `sync-wave: "-1"`, Application `"0"`
Layer 3 apps disappeared after L2 app deleted	`resources-finalizer.argocd.io` cascade-deleted children	Expected — L2 recreates them on next sync
`ImagePullBackOff` from `harbor.cl1.sq4.aegis.internal`	Missing pull secret or node CA trust	Ensure `harbor-pull-secret` in namespace; check node `certs.d`
ArgoCD can't pull chart — `403` from Helm registry	Token missing `read_api` scope or wrong project ID	Verify `gitlab-helm-repo` secret token and URL project ID
Application shows `OutOfSync` perpetually on AppProject resource	ArgoCD adds status fields after creation	Add `ignoreDifferences` on `AppProject` kind for status fields