Create a New App Group (3-Layer App of Apps)¶

This runbook creates a new Layer 2 app group from scratch — a new GitLab repo, a scoped AppProject, and a root Application that manages Layer 3 service Applications. The result: modgpt.<app>.* appears in the ArgoCD portal as a first-class group tile.

When to use this

Onboarding a new application with its own Git repo (e.g. modgpt.ask, modgpt.observability)
Don't use this to add a service to an existing group — just add bootstrap/application-<svc>.yaml and push (Update Config → Layer 3)

All commands run from the bastion.

What gets created

cluster-addons/root-bootstrap/
  appproject-<app>.yaml     ← Layer 2 AppProject  (sync-wave: -1)
  application-<app>.yaml    ← Layer 2 Application (sync-wave:  0)

<app> GitLab repo/bootstrap/
  application-<svc>.yaml    ← Layer 3 Application (one per service)

The modgpt root Application (already running) picks up the new root-bootstrap/ files and creates the Layer 2 pair automatically — no manual kubectl apply beyond what is already bootstrapped.

Order of operations

1. Create GitLab repo for the new app
2. Add Layer 2 pair to root-bootstrap/ in cluster-addons repo → push
3. Register Helm repo credentials in ArgoCD (if the app has its own Helm charts)
4. Add Layer 3 Application YAMLs to the new app repo bootstrap/ → push
5. Verify

Step 1 — Create the GitLab repo

export VAULT_ADDR=https://vault.cl1.sq4.aegis.internal
export VAULT_SKIP_VERIFY=true
GLT=$(vault kv get -field=token secret/gitlab/migration-token)

# Get the modgpt group ID
GID=$(curl -sk --header "PRIVATE-TOKEN: $GLT" \
  "https://gitlab.cl1.sq4.aegis.internal/api/v4/groups/modgpt" | python3 -c 'import sys,json; print(json.load(sys.stdin)["id"])')

# Create the repo
curl -sk --request POST --header "PRIVATE-TOKEN: $GLT" \
  "https://gitlab.cl1.sq4.aegis.internal/api/v4/projects" \
  --form "name=<app>" \
  --form "namespace_id=$GID" \
  --form "visibility=private" | python3 -c 'import sys,json; d=json.load(sys.stdin); print("ID:", d["id"], "URL:", d["http_url_to_repo"])'

# Save the project ID
PID=<printed-above>

# Clone and scaffold
git clone https://oauth2:${GLT}@gitlab.cl1.sq4.aegis.internal/modgpt/<app>.git
cd <app>
mkdir -p bootstrap values charts
echo "# <app>" > README.md
git add . && git commit -m "chore: init repo" && git push origin main

Step 2 — Add Layer 2 pair to cluster-addons

cd ~/cluster-addons

cat > root-bootstrap/appproject-<app>.yaml <<EOF
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: modgpt.<app>
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
spec:
  description: "Layer 2 — <app> application group"
  sourceRepos:
    - "https://gitlab.cl1.sq4.aegis.internal/modgpt/<app>.git"
    - "https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/${PID}/packages/helm/stable"
  destinations:
    - server: "https://kubernetes.default.svc"
      namespace: "*"
  clusterResourceWhitelist:
    - { group: "*", kind: "*" }
  namespaceResourceWhitelist:
    - { group: "*", kind: "*" }
EOF

cat > root-bootstrap/application-<app>.yaml <<EOF
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: modgpt.<app>
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "0"
spec:
  project: modgpt.<app>
  source:
    repoURL: https://gitlab.cl1.sq4.aegis.internal/modgpt/<app>.git
    targetRevision: HEAD
    path: bootstrap
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated: { prune: true, selfHeal: true }
EOF

GITLAB_TOKEN=$(vault kv get -field=token secret/gitlab/migration-token)
git add root-bootstrap/
git commit -m "feat: add modgpt.<app> app group (Layer 2)"
git remote set-url origin https://oauth2:${GITLAB_TOKEN}@gitlab.cl1.sq4.aegis.internal/modgpt/cluster-addons.git
git push origin main

The modgpt root Application detects the new files and creates AppProject: modgpt.<app> and Application: modgpt.<app> automatically within ~3 minutes.

Why sync-wave?

AppProject is on wave -1, Application on wave 0. ArgoCD creates them in wave order within a single sync — the AppProject exists before the Application references it. Without this, the Application fails with "project not found".

Step 3 — Register Helm credentials (if app has charts)

The git credentials are already covered by the gitlab-git-creds prefix secret (https://gitlab.cl1.sq4.aegis.internal/modgpt/). Only the Helm registry needs a new secret per project:

ARGOCD_RO=$(vault kv get -field=token secret/gitlab/migration-token)

kubectl -n argocd create secret generic <app>-helm-repo \
  --from-literal=type=helm \
  --from-literal=name="<app>-helm" \
  --from-literal=url="https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/${PID}/packages/helm/stable" \
  --from-literal=username=oauth2 \
  --from-literal=password="${ARGOCD_RO}"

kubectl -n argocd label secret <app>-helm-repo \
  argocd.argoproj.io/secret-type=repository

Production: render via VSO

kubectl create secret is shown for clarity. Production-correct: store the token in Vault (secret/<app>/argocd-token) and render it via a VaultStaticSecret with the same labels/keys — matching how all other platform secrets are managed.

Step 4 — Add Layer 3 service Applications

In the new app's repo, create one Application YAML per service in bootstrap/:

cd ~/<app>

# Helm-type service
cat > bootstrap/application-<svc>.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: modgpt.<app>.<svc>
  namespace: argocd
spec:
  project: modgpt.<app>
  destination:
    server: https://kubernetes.default.svc
    namespace: <app>
  syncPolicy:
    automated: { prune: true, selfHeal: true }
    syncOptions: [CreateNamespace=true]
  sources:
    - repoURL: https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/<PID>/packages/helm/stable
      chart: <chart>
      targetRevision: "<version>"
      helm:
        releaseName: <svc>
        valueFiles: ["$values/values/<svc>/<svc>-stg.yaml"]
    - repoURL: https://gitlab.cl1.sq4.aegis.internal/modgpt/<app>.git
      targetRevision: HEAD
      ref: values
EOF

# Add values file
mkdir -p values/<svc>
vi values/<svc>/<svc>-stg.yaml

GITLAB_TOKEN=$(vault kv get -field=token secret/gitlab/migration-token)
git add bootstrap/ values/
git commit -m "feat: add <svc> service"
git remote set-url origin https://oauth2:${GITLAB_TOKEN}@gitlab.cl1.sq4.aegis.internal/modgpt/<app>.git
git push origin main

modgpt.<app> Application syncs bootstrap/ → creates modgpt.<app>.<svc> automatically.

Ordering services (sync waves)¶

For apps with dependencies (DB before API, API before frontend):

# bootstrap/application-postgres.yaml
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "0"   # data stores first

# bootstrap/application-api.yaml
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "1"   # API after DB

# bootstrap/application-frontend.yaml
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "2"   # frontend last

Step 5 — Verify

# Layer 2 created by modgpt root app
kubectl -n argocd get appproject modgpt.<app>
kubectl -n argocd get application modgpt.<app>

# Layer 3 created by modgpt.<app>
kubectl -n argocd get applications.argoproj.io | grep "modgpt\.<app>"

# All healthy
kubectl -n argocd get applications.argoproj.io

Expected state:

modgpt                    Synced  Healthy   ← L1 root (unchanged)
modgpt.cluster-addons     Synced  Healthy   ← L2 existing
modgpt.<app>              Synced  Healthy   ← L2 new ✓
modgpt.<app>.<svc>        Synced  Healthy   ← L3 new ✓

Teardown

# Remove Layer 3 — delete from app repo bootstrap/ and push
cd ~/<app>
git rm bootstrap/application-<svc>.yaml
git commit -m "chore: remove <svc>" && git push origin main

# Remove Layer 2 — delete from cluster-addons root-bootstrap/ and push
cd ~/cluster-addons
git rm root-bootstrap/appproject-<app>.yaml root-bootstrap/application-<app>.yaml
git commit -m "chore: remove <app> app group" && git push origin main
# modgpt root app prunes AppProject + Application modgpt.<app>
# modgpt.<app> finalizer cascade-deletes all Layer 3 children

➡️ Related: Update ArgoCD Config · Multi-Microservice Pattern · ASK Onboarding