Add a New Add-on Application to the cluster-addons ApplicationSet

This is the everyday task — adding one more platform add-on application (a Helm chart or a set of raw manifests) to the existing cluster-addons ApplicationSet. It is pure GitOps: you commit two files and push. You do not touch the AppProject, the ApplicationSet, or the root Application — the generator auto-discovers your new folder and creates a child Application named modgpt.cluster-addons.<name> on its own.

Terminology

An add-on application here means the ArgoCD Application the generator produces (modgpt.cluster-addons.<name>) for one add-on. One folder under apps/ → one add-on application.

Which guide do I want?

• Add one add-on application to cluster-addons → you're in the right place.
• Stand up a new ApplicationSet for a new app group → Create a New App Group.
• Change an add-on application that already exists → Update ArgoCD Config → Tier 3.

All commands run from the bastion (ssh cloud-user@bastion1); ~/cluster-addons is the GitLab repo clone.

---

How the generator names and discovers it

The cluster-addons ApplicationSet globs apps/*/app.yaml and templates the name as modgpt.cluster-addons.{{ .path.basename }}. So:

create apps/<name>/app.yaml  →  generates  →  Application: modgpt.cluster-addons.<name>
                  └─ folder name = the add-on application's traceable suffix

The folder name is the identity. There is no metadata.name to set anywhere.

helm-type vs dir-type

Field	helm-type (a chart)	dir-type (raw manifests)
type	helm	dir
Needs	chart, targetRevision, valuesPath	dirPath (a folder of YAML in the repo)
Use for	Reloader, Vault, VSO, …	PodDisruptionBudgets, NetworkPolicies, ConfigMaps, …
Source injected by	the AppSet's project-1 Helm registry + $values	the repo path, recurse: true

In-repo (unpackaged) Helm chart?

Deploying a chart that lives as a folder in git (its own Chart.yaml + templates/, not published to the registry) is a third mode under active validation via the Core Service Deployment Runbook. This page will gain a verified localHelm section once that lands — until then, follow the runbook.

---

Prerequisites (air-gap)

Because the cluster pulls only from GitLab, before the add-on application can run you need:

1. Chart published to the GitLab Helm registry (project 1) — helm-type only
2. Image(s) mirrored to harbor.cl1.sq4.aegis.internal/ask/<image>
3. harbor-pull-secret secret present in the target namespace — already there if you reuse an existing add-on application's namespace; only created for a new one (Step 2)

Steps 1–3 below cover all three; skip 1–2 for a dir-type add-on application that ships no image, and skip Step 2 when deploying into an existing namespace.

---

Step 1 — Mirror the chart + image into GitLab

export VAULT_ADDR=https://vault.cl1.sq4.aegis.internal
GLT=$(vault kv get -field=token secret/gitlab/migration-token)   # write token (root) for publishing

# --- image → container-images project ---
skopeo copy --dest-creds "root:$GLT" \
  docker://<upstream-registry>/<image>:<tag> \
  docker://harbor.cl1.sq4.aegis.internal/ask/<image>:<tag>

# --- chart → project-1 HTTP Helm registry (helm-type only) ---
helm pull <repo>/<chart> --version <ver>          # produces <chart>-<ver>.tgz
curl -sk --request POST --user "root:$GLT" \
  --form "chart=@<chart>-<ver>.tgz" \
  "https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/1/packages/helm/api/stable/charts"

argocd-ro is read-only

Publishing needs the write token (migration-token/root). ArgoCD reads charts with the read-only argocd-ro token already wired into the gitlab-helm-repo secret — you don't touch that.

Verify the chart is visible:

curl -sk --user "argocd-ro:$(vault kv get -field=token secret/gitlab/argocd-ro)" \
  "https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/1/packages/helm/stable/index.yaml" | grep -A2 "<chart>:"

---

Step 2 — Create the namespace + image pull secret (new namespace only)

Skip this whole step if you're deploying into an existing namespace

This step is only needed when the add-on application lands in a brand-new namespace. If you're deploying into a namespace that already runs another GitLab-sourced add-on application, the namespace and its harbor-pull-secret secret already exist — reuse them and go straight to Step 3. Also skip entirely for a dir-type add-on application that pulls no private image. Quick check:

kubectl get ns <name> && kubectl -n <name> get secret harbor-pull-secret
#  both found  → skip Step 2
#  NotFound    → do Step 2

For a new namespace: the ApplicationSet syncs with CreateNamespace=true, but the pull secret must exist before pods schedule, so create both now. The pull secret is currently created by hand per namespace (see the deviation note):

NS=<name>
ARGOCD_RO=$(vault kv get -field=token secret/gitlab/argocd-ro)

kubectl create namespace "$NS" --dry-run=client -o yaml | kubectl apply -f -

kubectl -n "$NS" create secret docker-registry harbor-pull-secret \
  --docker-server=harbor.cl1.sq4.aegis.internal \
  --docker-username=argocd-ro \
  --docker-password="$ARGOCD_RO" \
  --dry-run=client -o yaml | kubectl apply -f -

Known deviation — pull secret is manual and per-namespace, not VSO-managed

harbor-pull-secret exists as a plain dockerconfigjson in each add-on application's namespace, created via kubectl apply — it is not rendered from Vault by VSO like the platform's other secrets, and it does not propagate to new namespaces automatically (hence this per-namespace step). Production-correct is a VSO VaultStaticSecret (token from secret/gitlab/argocd-ro) per namespace, or a secret replicator, so the token rotates centrally and lands in every namespace without manual steps. Track this when hardening.

---

Step 3 — Add the two repo files

helm-type — worked example (substitute your real chart/version/image):

cd ~/cluster-addons
mkdir -p apps/<name> values/<name>

cat > apps/<name>/app.yaml <<'EOF'
namespace: <name>
type: helm
chart: <chart>
targetRevision: <ver>
valuesPath: values/<name>/<name>-stg.yaml
# optional extras:
# syncWave: "1"                 # order vs other add-on applications (lower = earlier; default 5)
# ignoreDifferences:            # silence noisy server-managed fields
#   - group: admissionregistration.k8s.io
#     kind: MutatingWebhookConfiguration
#     name: <webhook>
#     jsonPointers: [ /webhooks/0/clientConfig/caBundle ]
EOF

cat > values/<name>/<name>-stg.yaml <<'EOF'
image:
  repository: harbor.cl1.sq4.aegis.internal/ask/<image>
  tag: <tag>
global:
  imagePullSecrets:
    - name: harbor-pull-secret
# production baseline — set on every add-on application:
resources:
  requests: { cpu: 50m, memory: 64Mi }
  limits:   { cpu: 200m, memory: 256Mi }
EOF

dir-type — for raw manifests (no chart, no image):

mkdir -p apps/<name> infra/<name>
cat > apps/<name>/app.yaml <<'EOF'
namespace: <target-namespace>
type: dir
dirPath: infra/<name>
EOF
# drop your *.yaml manifests into infra/<name>/

---

Step 4 — Commit, push, and refresh

cd ~/cluster-addons
git -c http.sslVerify=false add apps/<name> values/<name> infra/<name> 2>/dev/null
git -c http.sslVerify=false commit -m "feat: add <name> add-on application"
git -c http.sslVerify=false push origin main

# force the generator to re-scan now instead of waiting for the ~3-min poll
kubectl -n argocd annotate applicationset cluster-addons argocd.argoproj.io/refresh=hard --overwrite

---

Step 5 — Verify

kubectl -n argocd get applications.argoproj.io | grep "modgpt.cluster-addons.<name>"
#   → modgpt.cluster-addons.<name>   Synced   Healthy
kubectl -n <name> get pods

Symptom	Cause → fix
Add-on application never appears	folder not named app.yaml, or push didn't land — git log -1, then hard-refresh the AppSet
ComparisonError: repo not permitted	(only if you used a new repo) widen modgpt sourceRepos — Update → Tier 1
SyncFailed … name must not contain dots	values set a Helm release/fullname with a dot — the AppSet already sets releaseName: <basename>; don't override fullnameOverride with a dotted value
ImagePullBackOff	pull secret missing in <name> (Step 2) or wrong image path — must be …/ask/<image>
Chart not found	chart not published to project-1 registry (Step 1), or wrong targetRevision

---

Optional — add a PodDisruptionBudget for it

If the add-on application runs ≥ 2 replicas, add a PDB to the dir-type pdb add-on application (no new Application needed):

# create cluster-addons/infra/pdb/<name>-pdb.yaml (minAvailable: 1) and push —
# modgpt.cluster-addons.pdb applies it automatically.

See Pod Disruption Budgets → prod-01-pdb.md for the rules (quorum vs non-quorum, never PDB a single replica).

---

Rollback / remove

# remove the add-on application entirely — delete its files and push; the AppSet prunes the Application + its resources
cd ~/cluster-addons
git rm -r apps/<name> values/<name> infra/<name> 2>/dev/null
git -c http.sslVerify=false commit -m "chore: remove <name> add-on application" && git -c http.sslVerify=false push origin main
kubectl -n argocd annotate applicationset cluster-addons argocd.argoproj.io/refresh=hard --overwrite
# (then delete the namespace + pull secret if nothing else uses them)

To roll back a bad change (not remove the add-on application): git revert the commit and push — ArgoCD self-heals back.

---

➡️ Related: Create a New App Group · Update ArgoCD Config · App of Apps