Phase 4 — Platform Components¶

Deploy all platform components in strict order. GitLab is deployed first (manually via Helm) because everything else — ArgoCD, Vault, cert-manager, Reloader — is managed through GitLab as the Git + Helm chart source.

Deployment Order¶

 ┌─────────────────────────────────────────────────────────────────┐
 │  MANUAL HELM INSTALL (bootstrap — no GitOps yet)                │
 └─────────────────────────────────────────────────────────────────┘
        │
        ▼
   4.1  GitLab          ← Git server + container registry + Helm registry
        │                 cert-manager is bundled in the GitLab chart —
        │                 no separate cert-manager install needed
        ▼
   4.2  ArgoCD          ← GitOps controller (points at GitLab)
        │
        ▼
 ┌─────────────────────────────────────────────────────────────────┐
 │  ARGOCD MANAGES EVERYTHING BELOW (GitOps)                       │
 └─────────────────────────────────────────────────────────────────┘
        │
        ▼
   4.3  HashiCorp Vault ← Secrets store
        │
        ▼
   4.4  VSO             ← Vault Secrets Operator (renders K8s secrets from Vault)
        │
        ▼
   4.5  Reloader        ← Restarts pods on config/secret changes

Why GitLab first?

GitLab cannot bootstrap from itself — just like ArgoCD. It must be deployed manually before it can host repos and charts that ArgoCD reads. Once GitLab is up, all subsequent components are declared as code in GitLab and applied by ArgoCD.

cert-manager is bundled in the GitLab Helm chart

The GitLab chart ships cert-manager as a sub-chart dependency. It is installed automatically as part of Step 4.1 — do not install cert-manager separately. Installing it twice would create two cert-manager controllers fighting over the same CRDs, causing webhook conflicts and cert flapping across the cluster.

4.1 — GitLab¶

Method: Manual helm install from Harbor OCI chart.

Prerequisites¶

[ ] All GitLab images mirrored to Harbor gitlab project (Phase 1)
[ ] helm-charts/gitlab pushed to Harbor (Phase 1 Step 1.7)
[ ] harbor-pull-secret created in gitlab namespace
[ ] Ceph S3 user created for GitLab object storage
[ ] TLS certificate issued (cert-manager or manual)

Step 4.1.1 — Create namespace and secrets¶

Run from

bastion1 — any directory (kubectl commands are cluster-wide)

kubectl create namespace gitlab

# Root password
GITLAB_ROOT_PASS=$(openssl rand -base64 24 | tr -d '/+=')
kubectl create secret generic gitlab-initial-root-password \
  -n gitlab \
  --from-literal=password="${GITLAB_ROOT_PASS}"

# Harbor pull secret
# This creates a kubernetes.io/dockerconfigjson Secret that containerd
# uses to authenticate against Harbor when pulling images for GitLab pods.
# Without this → ImagePullBackOff on every GitLab pod.
HARBOR_PASS=$(python3 -c "import json; j=json.load(open('/home/cloud-user/helm-charts/harbor-robot.json')); print(j['secret'])")
kubectl create secret docker-registry harbor-pull-secret \
  -n gitlab \
  --docker-server=harbor.cl1.sq4.aegis.internal \
  --docker-username='robot$ask' \
  --docker-password="${HARBOR_PASS}"
unset HARBOR_PASS

echo "Root password: ${GITLAB_ROOT_PASS}"
# Store in Vault immediately — never leave in shell variable
vault kv put secret/gitlab/root-password password="${GITLAB_ROOT_PASS}"
unset GITLAB_ROOT_PASS

Step 4.1.2 — Verify S3 prerequisites (Phase 2)¶

Complete Phase 2 before proceeding

The following must be done in Phase 2 — Storage Validation before continuing:

[ ] Ceph RGW reachable at http://100.115.1.11:7480
[ ] GitLab S3 buckets pre-created (7 buckets)
[ ] gitlab-s3-connection secret created in gitlab namespace
[ ] gitlab-registry-storage secret created in gitlab namespace

# Quick check — both secrets must exist before helm install
kubectl get secret gitlab-s3-connection gitlab-registry-storage -n gitlab

Step 4.1.4 — Create values file¶

Run from

/home/cloud-user/deployments/gitlab/

Never use cat >> to append to this file

Adding a second global: block via cat >> causes YAML duplicate key behaviour — the second block silently overwrites the first, wiping all image overrides. Always recreate the file with cat > (overwrite) to keep a single clean block.

mkdir -p /home/cloud-user/deployments/gitlab
cat > /home/cloud-user/deployments/gitlab/values-gitlab.yaml << 'EOF'
global:
  imagePullSecrets:
    - name: harbor-pull-secret

  # global.gitlabBase.image — controls the configure init container image in every pod
  # Template: gitlab.configure.image reads from .root.Values.global.gitlabBase.image
  # Without this, every configure init container pulls from registry.gitlab.com (public)
  # Note: global.image is NOT the correct key — must be global.gitlabBase.image
  gitlabBase:
    image:
      repository: harbor.cl1.sq4.aegis.internal/gitlab/gitlab-org/build/cng/gitlab-base
      tag: "v18.11.4"
      pullPolicy: IfNotPresent

  # global.kubectl — used by shared-secrets job and other Kubernetes jobs
  kubectl:
    image:
      repository: harbor.cl1.sq4.aegis.internal/gitlab/gitlab-org/build/cng/kubectl
      tag: "v18.11.4"

  # Component image overrides — all pulling from Harbor gitlab project
  gitaly:
    image:
      repository: harbor.cl1.sq4.aegis.internal/gitlab/gitlab-org/build/cng/gitaly
      tag: "v18.11.4"
  webservice:
    image:
      repository: harbor.cl1.sq4.aegis.internal/gitlab/gitlab-org/build/cng/gitlab-webservice-ee
      tag: "v18.11.4"
  workhorse:
    image:
      repository: harbor.cl1.sq4.aegis.internal/gitlab/gitlab-org/build/cng/gitlab-workhorse-ee
      tag: "v18.11.4"
  sidekiq:
    image:
      repository: harbor.cl1.sq4.aegis.internal/gitlab/gitlab-org/build/cng/gitlab-sidekiq-ee
      tag: "v18.11.4"
  shell:
    image:
      repository: harbor.cl1.sq4.aegis.internal/gitlab/gitlab-org/build/cng/gitlab-shell
      tag: "v14.50.0"
  kas:
    image:
      repository: harbor.cl1.sq4.aegis.internal/gitlab/gitlab-org/build/cng/gitlab-kas
      tag: "v18.11.4"
  toolbox:
    image:
      repository: harbor.cl1.sq4.aegis.internal/gitlab/gitlab-org/build/cng/gitlab-toolbox-ee
      tag: "v18.11.4"
  certificates:
    image:
      repository: harbor.cl1.sq4.aegis.internal/gitlab/gitlab-org/build/cng/certificates
      tag: "v18.11.4"

  hosts:
    domain: cl1.sq4.aegis.internal
    https: false              # TLS disabled — enable after ClusterIssuer is configured

  ingress:
    configureCertmanager: false
    provider: nginx
    class: nginx
    tls:
      enabled: false          # No TLS for now — see TLS Upgrade runbook

  certmanager:
    install: false            # Never install bundled cert-manager — already in cluster

  minio:
    enabled: false

  # Global seccompProfile — applies to all GitLab pods (required for PodSecurity "restricted")
  securityContext:
    seccompProfile:
      type: RuntimeDefault            # Using Ceph RGW for object storage

  initialRootPassword:
    secret: gitlab-initial-root-password
    key: password

  appConfig:
    object_store:
      enabled: true
      proxy_download: true
      connection:
        secret: gitlab-s3-connection
        key: connection
    artifacts:
      enabled: true
      bucket: gitlab-artifacts
    lfs:
      enabled: true
      bucket: gitlab-lfs
    uploads:
      enabled: true
      bucket: gitlab-uploads
    packages:
      enabled: true
      bucket: gitlab-packages
    backups:
      bucket: gitlab-backups
      tmpBucket: gitlab-backups-tmp

gitlab:
  gitaly:
    persistence:
      enabled: true
      storageClass: csi-rbd-sc   # Ceph RBD block storage (cluster default)
      size: 50Gi
    # Required for PodSecurity "restricted" profile — without this, pods get a warning
    # (non-blocking on install but required for compliance)
    containerSecurityContext:
      seccompProfile:
        type: RuntimeDefault
    initContainers:
      certificates:
        securityContext:
          seccompProfile:
            type: RuntimeDefault
      configure:
        securityContext:
          seccompProfile:
            type: RuntimeDefault

# certmanager-issuer requires email even when TLS is disabled
certmanager-issuer:
  email: [email protected]

# Keep nginx-ingress sub-chart (needed for template validation) but disable
# IngressClass creation — cluster already has "nginx" owned by rke2-ingress-nginx
nginx-ingress:
  enabled: false
  controller:
    ingressClassResource:
      enabled: false
      default: false

prometheus:
  install: false

gitlab-runner:
  install: false

registry:
  enabled: true
  image:
    repository: harbor.cl1.sq4.aegis.internal/gitlab/gitlab-org/build/cng/gitlab-container-registry
    tag: "v4.39.0-gitlab"
  storage:
    secret: gitlab-registry-storage
    key: config

postgresql:
  image:
    registry: harbor.cl1.sq4.aegis.internal
    repository: bitnamilegacy/postgresql
    tag: "16.6.0"
  primary:
    persistence:
      storageClass: csi-rbd-sc
      size: 10Gi

redis:
  image:
    registry: harbor.cl1.sq4.aegis.internal
    repository: bitnamilegacy/redis
    tag: "7.2.4-debian-12-r9"
  master:
    persistence:
      storageClass: csi-rbd-sc
      size: 2Gi
EOF

echo "✅ Values file written — $(wc -l < /home/cloud-user/deployments/gitlab/values-gitlab.yaml) lines"

Step 4.1.5 — Install GitLab from Harbor¶

Two platform conflicts to resolve before install (empirically confirmed)

The GitLab 9.11.5 chart bundles sub-charts for components already running cluster-wide. Both must be handled before helm install succeeds:

Conflict	Error	Fix
cert-manager CRDs already owned	`CustomResourceDefinition "challenges.acme.cert-manager.io" exists and cannot be imported`	Remove `charts/cert-manager` sub-chart directory + patch `Chart.yaml`/`Chart.lock`
nginx IngressClass already owned	`IngressClass "nginx" exists and cannot be imported`	Keep `nginx-ingress` sub-chart (needed for template validation) but disable IngressClass creation via values

Key lessons: - --set certmanager.install=false alone is not enough — Helm schema validation fires even when sub-chart is disabled (known Helm bug). Physical removal required. - nginx-ingress sub-chart must stay (removing it breaks template validation). Instead, disable IngressClass creation via nginx-ingress.controller.ingressClassResource.enabled: false. - certmanager-issuer sub-chart is safe to keep — it's configuration only, not the controller.

Step 4.1.5a — Add nginx-ingress IngressClass disable to values¶

# Append to values file — prevents IngressClass creation conflict
cat >> /home/cloud-user/deployments/gitlab/values-gitlab.yaml << 'EOF'

# nginx-ingress: keep sub-chart for template validation but disable IngressClass
# creation — cluster already has IngressClass "nginx" owned by rke2-ingress-nginx
nginx-ingress:
  enabled: false
  controller:
    ingressClassResource:
      enabled: false
      default: false
EOF

Why global.ingress.class: nginx still works

global.ingress.class: nginx tells GitLab ingress resources to use the existing rke2-ingress-nginx IngressClass. The ingressClassResource.enabled: false only prevents the chart from trying to create a new IngressClass — which would conflict. The two settings serve different purposes and are both required.

Step 4.1.5b — Pull and patch chart¶

Run from

/home/cloud-user/deployments/gitlab/

# Pull chart from Harbor locally
mkdir -p /tmp/gitlab-chart
helm pull oci://harbor.cl1.sq4.aegis.internal/helm-charts/gitlab \
  --version 9.11.5 \
  -d /tmp/gitlab-chart/ \
  --untar

# Remove ONLY cert-manager sub-chart (keep nginx-ingress)
rm -rf /tmp/gitlab-chart/gitlab/charts/cert-manager

# Patch Chart.yaml and Chart.lock — remove cert-manager dependency entry
python3 - << 'EOF'
import yaml, pathlib

for fname in ["Chart.yaml", "Chart.lock"]:
    p = pathlib.Path(f"/tmp/gitlab-chart/gitlab/{fname}")
    if not p.exists():
        print(f"skipping {fname} — not found")
        continue
    data = yaml.safe_load(p.read_text())
    if "dependencies" in data and data["dependencies"]:
        before = len(data["dependencies"])
        data["dependencies"] = [d for d in data["dependencies"]
                                 if d.get("name") != "cert-manager"]
        print(f"✅ {fname}: removed {before - len(data['dependencies'])} entries")
    p.write_text(yaml.dump(data, default_flow_style=False))
EOF

# Verify cert-manager is gone, nginx-ingress is still present
ls /tmp/gitlab-chart/gitlab/charts/
# Expected: cert-manager NOT listed, nginx-ingress IS listed

Step 4.1.5c — Install¶

helm install gitlab /tmp/gitlab-chart/gitlab \
  --namespace gitlab \
  -f /home/cloud-user/deployments/gitlab/values-gitlab.yaml \
  --set certmanager.install=false \
  --timeout 15m \
  --wait \
  --debug             # ← shows rendered templates + hook progress

Monitor progress in parallel terminals:

# Terminal 2 — watch all pods
watch -n5 kubectl get pods -n gitlab

# Terminal 3 — watch events (shows what's happening in real time)
kubectl get events -n gitlab --sort-by='.lastTimestamp' -w

# Terminal 4 — check stuck pods
kubectl get pods -n gitlab | grep -v "Running\|Completed"

# If a pod is stuck — describe it
kubectl describe pod -n gitlab <pod-name> | tail -30

PodSecurity seccompProfile — fixed in values file

The values file includes seccompProfile: RuntimeDefault for Gitaly and its init containers. This resolves:

would violate PodSecurity "restricted:latest": seccompProfile
(containers "certificates", "configure", "gitaly" must set securityContext.seccompProfile.type)

If you see this warning on an existing install, run the helm upgrade in Step 4.1.5d below.

If install fails with cannot re-use a name that is still in use

A previous failed attempt left the release in a bad state:

helm uninstall gitlab -n gitlab
# Then re-run from Step 4.1.5b

Step 4.1.5d — Known install issues (empirically confirmed)¶

Issue: seccompProfile PodSecurity warning — GitLab init containers (known deviation)

Symptom (non-blocking — upgrade succeeds):

W would violate PodSecurity "restricted:latest": seccompProfile
(pod or containers "certificates", "configure" must set securityContext.seccompProfile.type
to "RuntimeDefault" or "Localhost")

Cause: The certificates and configure init containers appear across all GitLab components (webservice, sidekiq, kas, toolbox, migrations, exporter). GitLab upstream has not fully aligned with the restricted PodSecurity profile for these init containers. The global.securityContext.seccompProfile and per-component overrides fix Gitaly but do not propagate to init containers in all other components.

Impact: PodSecurity policy is in warn mode — pods deploy and run normally. This is a compliance warning only, not an operational failure.

Verified pod state (all healthy ✅):

gitlab-gitaly-0                               1/1  Running    0  3m55s
gitlab-webservice-default-5f6d7c559f-ghctc    2/2  Running    0  27m
gitlab-webservice-default-5f6d7c559f-t97z5    2/2  Running    0  29m
gitlab-sidekiq-all-in-1-v2-656c4fbb95-xvm6v   1/1  Running    0  28m
gitlab-kas-67d689f57c-8bxzl                   1/1  Running    2  21h
gitlab-registry-886948b8c-87h9r               1/1  Running    0  21h
gitlab-postgresql-0                           2/2  Running    0  21h
gitlab-redis-master-0                         2/2  Running    0  21h
gitlab-migrations-76fde5a-fpgfh               0/1  Completed  0  67s

Production note: To fully eliminate this warning, each component's init containers would require explicit securityContext.seccompProfile overrides in values — GitLab chart v9.11.5 does not expose a single global key that covers all init containers. Track upstream GitLab chart issues for a clean fix in a future chart version.

Issue: storageclass.storage.k8s.io "ceph-block" not found — PVCs Pending

Symptom:

gitlab-postgresql-0       0/2  Pending   0   16m
gitlab-redis-master-0     0/2  Pending   0   16m

Warning  ProvisioningFailed  persistentvolume-controller
storageclass.storage.k8s.io "ceph-block" not found

Cause: Values file has storageClass: ceph-block but the correct name is csi-rbd-sc.

Fix:

helm uninstall gitlab -n gitlab
sed -i 's/storageClass: ceph-block/storageClass: csi-rbd-sc/g' \
  /home/cloud-user/deployments/gitlab/values-gitlab.yaml
# Re-run helm install

Issue: Init:ImagePullBackOff — init containers pulling from public registry

Symptom:

gitlab-migrations   0/1  Init:ImagePullBackOff   0   16m
gitlab-kas          0/1  ImagePullBackOff        0   16m

Image: registry.gitlab.com/gitlab-org/build/cng/gitlab-base:v18.11.4  ← public!

Cause: The configure init container in every GitLab pod uses global.image (gitlab-base). This key was not overridden in the values file, so it defaults to pulling from registry.gitlab.com (public — unreachable in air-gap).

Fix: Add global.image to values file:

global:
  image:
    repository: harbor.cl1.sq4.aegis.internal/gitlab/gitlab-org/build/cng/gitlab-base
    tag: "v18.11.4"
    pullPolicy: IfNotPresent

Then uninstall and reinstall.

Issue: configure init container still pulling from registry.gitlab.com

Symptom:

Image: registry.gitlab.com/gitlab-org/build/cng/gitlab-base:v18.11.4
Failed to pull image ... dial tcp 35.227.35.254:443: i/o timeout

Cause 1 — Wrong values key: global.image does NOT control the configure init container. The correct key discovered from the chart template is global.gitlabBase.image:

# From _image.tpl:
{{- define "gitlab.configure.image" -}}
{{- $image := mergeOverwrite (deepCopy .root.Values.global.gitlabBase.image) .image }}

Cause 2 — Duplicate YAML keys: Using cat >> to append a second global: block silently overwrites the first block — all image overrides are lost.

Fix: Use global.gitlabBase.image in a single global: block:

global:
  gitlabBase:
    image:
      repository: harbor.cl1.sq4.aegis.internal/gitlab/gitlab-org/build/cng/gitlab-base
      tag: "v18.11.4"

Always recreate the values file with cat > (not cat >>).

Issue: image overrides silently lost — duplicate global: key

Symptom: Images still pulling from public registry despite overrides in values file.

Cause: cat >> was used to append a second global: block to the values file. YAML does not merge duplicate top-level keys — the second global: block silently overwrites the first, losing all image overrides in the original block.

Fix: Recreate the file with cat > (overwrite) keeping all settings in a single global: block. Never use cat >> to add values to this file.

Issue: certmanager-issuer.email required even when TLS disabled

Symptom:

execution error at (gitlab/charts/certmanager-issuer/templates/configmap.yaml:17:9):
You must provide an email to associate with your TLS certificates.
Please set certmanager-issuer.email

Cause: The certmanager-issuer sub-chart validates the email field regardless of whether TLS is enabled or cert-manager is used.

Fix: Add to values file:

certmanager-issuer:
  email: [email protected]

Issue: each GitLab sub-chart component needs its own gitlab.<name>.image override

Symptom: Main containers (sidekiq, toolbox, shell, kas, exporter, webservice) still pulling from registry.gitlab.com even though global.<name>.image is set.

Cause: global.<component>.image does NOT override the main container image for most GitLab sub-charts. Each sub-chart has its own image: {} key that must be set at gitlab.<component>.image.

Confirmed by reading each sub-chart's values.yaml:

# Each sub-chart (sidekiq, toolbox, webservice, kas, gitlab-shell, gitlab-exporter)
image: {}
  # repository: registry.gitlab.com/gitlab-org/build/cng/gitlab-<name>-ee

Fix: Override each component under the gitlab: top-level key:

href="#__codelineno-24-1">gitlab: sidekiq: image: repository: harbor.../gitlab/gitlab-org/build/cng/gitlab-sidekiq-ee tag: "v18.11.4" toolbox: image: repository: harbor.../gitlab/gitlab-org/build/cng/gitlab-toolbox-ee tag: "v18.11.4" migrations: image: repository: harbor.../gitlab/gitlab-org/build/cng/gitlab-toolbox-ee tag: "v18.11.4" gitlab-shell: image: repository: harbor.../gitlab/gitlab-org/build/cng/gitlab-shell tag: "v14.50.0" kas: image: repository: harbor.../gitlab/gitlab-org/build/cng/gitlab-kas tag: "v18.11.4" gitlab-exporter: image: repository: harbor.../gitlab/gitlab-org/build/cng/gitlab-exporter tag: "16.7.0" webservice: image: repository: harbor.../gitlab/gitlab-org/build/cng/gitlab-webservice-ee tag: "v18.11.4"

Issue: redis-exporter and postgres-exporter sidecars pulling from docker.io

Symptom:

gitlab-redis-master-0       1/2  ErrImagePull
gitlab-postgresql-0         1/2  ImagePullBackOff

Image: docker.io/bitnamilegacy/redis-exporter:1.58.0-debian-12-r4  ← public, not found
Image: docker.io/bitnamilegacy/postgres-exporter:0.15.0-debian-11-r7  ← public, not found

Cause: The bitnami redis and postgresql sub-charts ship with a prometheus metrics exporter sidecar. GitLab enables these by default. The exporter images are separate from the main redis/postgresql images and need their own registry override.

Fix — redis exporter:

redis:
  metrics:
    image:
      registry: harbor.cl1.sq4.aegis.internal
      repository: bitnamilegacy/redis-exporter
      tag: "1.58.0-debian-12-r4"

Fix — postgres exporter:

postgresql:
  metrics:
    image:
      registry: harbor.cl1.sq4.aegis.internal
      repository: bitnamilegacy/postgres-exporter
      tag: "0.15.0-debian-11-r7"

Live patch (without reinstall):

# Redis exporter
kubectl patch statefulset gitlab-redis-master -n gitlab --type=json -p='[
  {"op": "replace", "path": "/spec/template/spec/containers/1/image",
   "value": "harbor.cl1.sq4.aegis.internal/bitnamilegacy/redis-exporter:1.58.0-debian-12-r4"}
]'
kubectl rollout restart statefulset gitlab-redis-master -n gitlab

# PostgreSQL exporter
kubectl patch statefulset gitlab-postgresql -n gitlab --type=json -p='[
  {"op": "replace", "path": "/spec/template/spec/containers/1/image",
   "value": "harbor.cl1.sq4.aegis.internal/bitnamilegacy/postgres-exporter:0.15.0-debian-11-r7"}
]'
kubectl delete pod gitlab-postgresql-0 -n gitlab

Issue: PVCs stuck Pending — leftover from previous failed install

Symptom:

data-gitlab-postgresql-0          Pending  ceph-block  53m
redis-data-gitlab-redis-master-0  Pending  ceph-block  53m

Cause: PVCs from a previous helm install attempt are never automatically deleted on helm uninstall. StatefulSets reuse the existing PVC names — if the old PVCs have the wrong storage class, the new install inherits them.

Fix: After uninstalling, explicitly delete stuck PVCs before reinstalling:

helm uninstall gitlab -n gitlab
kubectl delete pvc data-gitlab-postgresql-0 redis-data-gitlab-redis-master-0 -n gitlab
# Gitaly PVC (repo-data-gitlab-gitaly-0) can be kept if Bound — data is preserved
helm install gitlab ...

Delete ALL PVCs for a clean slate

kubectl delete pvc -n gitlab --all

This also deletes gitaly data. Only use for a fresh install with no data to preserve.

Step 4.1.6 — Verify GitLab pods¶

Expected final state (verified ✅):

kubectl get pods -n gitlab

NAME                                          READY   STATUS      RESTARTS   AGE
gitlab-gitlab-exporter-xxxx                   1/1     Running     0          9m
gitlab-gitlab-shell-xxxx (x2)                 1/1     Running     0          9m
gitlab-kas-xxxx (x2)                          1/1     Running     2          9m
gitlab-migrations-xxxx                        0/1     Completed   4          9m
gitlab-postgresql-0                           2/2     Running     0          5m
gitlab-redis-master-0                         2/2     Running     0          9m
gitlab-registry-xxxx (x2)                     1/1     Running     0          9m
gitlab-sidekiq-all-in-1-v2-xxxx (x3)          1/1     Running     0          9m
gitlab-toolbox-xxxx                           1/1     Running     0          9m
gitlab-webservice-default-xxxx (x2)           2/2     Running     2          9m

# Verify ingresses
kubectl get ingress -n gitlab

NAME                        HOSTS                              PORTS
gitlab-kas                  kas.cl1.sq4.aegis.internal         80
gitlab-registry             registry.cl1.sq4.aegis.internal    80
gitlab-webservice-default   gitlab.cl1.sq4.aegis.internal      80

# Health check (add to /etc/hosts first if DNS not configured)
echo "100.115.1.14 gitlab.cl1.sq4.aegis.internal registry.cl1.sq4.aegis.internal kas.cl1.sq4.aegis.internal" \
  | sudo tee -a /etc/hosts

curl -s http://gitlab.cl1.sq4.aegis.internal/-/health
# Expected: GitLab OK

Step 4.1.7 — Configure containerd for HTTP registry¶

Temporary — HTTP only until TLS is added

This step is only required while GitLab runs without TLS (HTTP). Once TLS is enabled (see TLS Upgrade runbook), this config is replaced with the HTTPS + CA cert version — containerd will trust the registry over HTTPS automatically. Skip this step if you are deploying with TLS from day one.

Required for HTTP deploy

containerd rejects HTTP registries by default. Each node needs explicit config to allow HTTP pulls from the GitLab registry.

for node in node1 node2 node3; do
  echo "=== $node ==="
  ssh cloud-user@${node} "
    sudo mkdir -p /etc/containerd/certs.d/registry.gitlab.cl1.sq4.aegis.internal
    sudo tee /etc/containerd/certs.d/registry.gitlab.cl1.sq4.aegis.internal/hosts.toml > /dev/null << 'EOF'
server = \"http://registry.gitlab.cl1.sq4.aegis.internal\"
[host.\"http://registry.gitlab.cl1.sq4.aegis.internal\"]
  capabilities = [\"pull\", \"resolve\", \"push\"]
EOF
    echo done"
done

Verify:

for node in node1 node2 node3; do
  ssh cloud-user@${node} "cat /etc/containerd/certs.d/registry.gitlab.cl1.sq4.aegis.internal/hosts.toml"
done

When TLS is added — update to HTTPS

Replace the above hosts.toml on each node with:

server = "https://registry.gitlab.cl1.sq4.aegis.internal"
[host."https://registry.gitlab.cl1.sq4.aegis.internal"]
  capabilities = ["pull", "resolve", "push"]
  ca = "/etc/ssl/certs/aegis-cl1-root.crt"

No containerd restart needed — changes take effect on next pull.

Step 4.1.8 — Post-install setup¶

# Confirm GitLab is reachable
curl -s http://gitlab.cl1.sq4.aegis.internal/-/health
# Expected: GitLab OK

# Get admin password from Vault
vault kv get -field=password secret/gitlab/root-password

Login at http://gitlab.cl1.sq4.aegis.internal:

Field	Value
Username	`admin`
Password	Retrieved from Vault above

Once logged in:

[ ] Disable public sign-up — Admin → Settings → General → Sign-up restrictions → uncheck Sign-up enabled → Save
[ ] Create ask group
[ ] Create cluster-addons repo
[ ] Push platform values files to GitLab
[ ] Create ArgoCD read-only token (argocd-ro)

4.1 (TLS Upgrade) — Add TLS to GitLab after CA access is available¶

Do this after CA cert is obtained from infra1/infra2 (100.115.1.21 / 100.115.1.22)

PKI runs on infra1 and infra2. Once you have SSH access or can retrieve the root CA cert, follow these steps to enable TLS on GitLab.

Step T1 — Get root CA cert¶

# Option A: from infra1 directly (once SSH access is available)
ssh [email protected] "sudo cat /etc/step-ca/certs/root_ca.crt" > ~/root_ca.crt

# Option B: from a cluster node (CA already distributed)
ssh cloud-user@node1 "sudo cat /usr/local/share/ca-certificates/*.crt" > ~/root_ca.crt

# Verify it's a valid CA cert
openssl x509 -in ~/root_ca.crt -noout -subject -issuer -dates

Step T2 — Create ClusterIssuer¶

CA_BUNDLE=$(base64 -w0 ~/root_ca.crt)

kubectl apply -f - << EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: step-ca-acme
spec:
  acme:
    server: https://100.115.1.21/acme/acme/directory
    email: [email protected]
    caBundle: ${CA_BUNDLE}
    privateKeySecretRef:
      name: step-ca-acme-account-key
    solvers:
      - http01:
          ingress:
            ingressClassName: nginx
EOF

# Verify
kubectl get clusterissuer step-ca-acme
# Expected: READY=True

Step T3 — Trust CA on all nodes (for HTTPS image pulls)¶

for node in node1 node2 node3; do
  echo "=== $node ==="
  scp ~/root_ca.crt cloud-user@${node}:~/
  ssh cloud-user@${node} "
    sudo cp ~/root_ca.crt /etc/ssl/certs/aegis-cl1-root.crt
    sudo update-ca-certificates

    # Update containerd to HTTPS
    sudo mkdir -p /etc/containerd/certs.d/registry.gitlab.cl1.sq4.aegis.internal
    sudo tee /etc/containerd/certs.d/registry.gitlab.cl1.sq4.aegis.internal/hosts.toml > /dev/null << 'EOF2'
server = \"https://registry.gitlab.cl1.sq4.aegis.internal\"
[host.\"https://registry.gitlab.cl1.sq4.aegis.internal\"]
  capabilities = [\"pull\", \"resolve\", \"push\"]
  ca = \"/etc/ssl/certs/aegis-cl1-root.crt\"
EOF2
    echo done"
done

Step T4 — Update values file and upgrade GitLab¶

# Update these fields in values-gitlab.yaml
# global.hosts.https: true
# global.ingress.tls.enabled: true
# global.ingress.tls.secretName: gitlab-tls
# global.ingress.annotations: cert-manager.io/cluster-issuer: step-ca-acme

sed -i \
  -e 's/https: false/https: true/' \
  -e 's/enabled: false.*# No TLS.*/enabled: true/' \
  /home/cloud-user/deployments/gitlab/values-gitlab.yaml

# Add annotation block manually or with your editor
vi /home/cloud-user/deployments/gitlab/values-gitlab.yaml

Add under global.ingress:

    tls:
      enabled: true
      secretName: gitlab-tls
    annotations:
      cert-manager.io/cluster-issuer: step-ca-acme
      nginx.ingress.kubernetes.io/proxy-body-size: "0"
      nginx.ingress.kubernetes.io/proxy-read-timeout: "300"

# Apply via helm upgrade
helm upgrade gitlab \
  oci://harbor.cl1.sq4.aegis.internal/helm-charts/gitlab \
  --version 9.11.5 \
  --namespace gitlab \
  -f /home/cloud-user/deployments/gitlab/values-gitlab.yaml \
  --timeout 10m \
  --wait

# Verify cert was issued
kubectl get certificate -n gitlab
# Expected: READY=True

# Confirm HTTPS works
curl -s https://gitlab.cl1.sq4.aegis.internal/-/health
# Expected: GitLab OK

4.2 — ArgoCD¶

Method: Manual helm install from Harbor. After install, ArgoCD manages itself via GitLab.

Step 4.2.1 — Create namespace and imagePullSecret¶

Run from

/home/cloud-user/deployments/argocd/

mkdir -p /home/cloud-user/deployments/argocd
kubectl create namespace argocd

HARBOR_PASS=$(python3 -c "import json; j=json.load(open('/home/cloud-user/helm-charts/harbor-robot.json')); print(j['secret'])")
kubectl create secret docker-registry harbor-pull-secret \
  -n argocd \
  --docker-server=harbor.cl1.sq4.aegis.internal \
  --docker-username='robot$ask' \
  --docker-password="${HARBOR_PASS}"
unset HARBOR_PASS

Step 4.2.2 — Create values file¶

Run from

/home/cloud-user/deployments/argocd/

Check the correct image key before writing values

Lesson from GitLab — global.image may not be the correct key. Always verify first:

helm show values oci://harbor.cl1.sq4.aegis.internal/helm-charts/argo-cd \
  --version 9.5.15 | grep -A5 "^image:"

cat > /home/cloud-user/deployments/argocd/values-argocd.yaml << 'EOF'
global:
  imagePullSecrets:
    - name: harbor-pull-secret
  image:
    repository: harbor.cl1.sq4.aegis.internal/quay/argoproj/argocd
    tag: "v3.4.2"
server:
  ingress:
    enabled: true
    ingressClassName: nginx
    hostname: argocd.cl1.sq4.aegis.internal
    tls: false    # No TLS until ClusterIssuer is configured
EOF

Two additional images required — not in Phase 1 mirror list

ArgoCD deploys two dependency components that were not in the original mirror config:

Image	Source	Harbor destination
`dexidp/dex:v2.45.1`	`ghcr.io`	`ghcr/dexidp/dex`
`redis:8.2.3-alpine`	`ecr-public.aws.com` (mirror from `docker.io`)	`dockerhub/redis`

Mirror them first:

cd ~/harbor-images

# Add to mirror-config.yaml then mirror
./core42-mirror-image.py --only dexidp    # dex from ghcr.io
./core42-mirror-image.py --only "8.2.3"   # redis 8.2.3-alpine from docker.io

Then add overrides to values file:

dex:
  image:
    repository: harbor.cl1.sq4.aegis.internal/ghcr/dexidp/dex
    tag: "v2.45.1"

redis:
  image:
    repository: harbor.cl1.sq4.aegis.internal/dockerhub/redis
    tag: "8.2.3-alpine"

Container names for kubectl set image (live patch)

Deployment	Container name
`argocd-dex-server`	`dex-server` (not `dex`)
`argocd-redis`	`redis`
`kubectl set image deployment/argocd-dex-server \ dex-server=harbor.cl1.sq4.aegis.internal/ghcr/dexidp/dex:v2.45.1 -n argocd kubectl set image deployment/argocd-redis \ redis=harbor.cl1.sq4.aegis.internal/dockerhub/redis:8.2.3-alpine -n argocd`
Then run `helm upgrade` to make it permanent — Helm will overwrite `kubectl set image` on the next upgrade.

Step 4.2.3 — Install ArgoCD from Harbor¶

Run from

/home/cloud-user/deployments/argocd/

helm install argocd \
  oci://harbor.cl1.sq4.aegis.internal/helm-charts/argo-cd \
  --version 9.5.15 \
  --namespace argocd \
  --create-namespace \
  --timeout 180s \
  -f /home/cloud-user/deployments/argocd/values-argocd.yaml \
  --wait \
  --debug 2>&1 | grep -v "^install.go\|^helm.go\|^ready.go"

Deviation from reference deployment

Reference uses public https://argoproj.github.io/argo-helm + chart 9.2.4. Core42 uses Harbor OCI (oci://harbor...) + chart 9.5.15 (newer) — required for air-gap. All other flags (--create-namespace, --timeout) are aligned.

Step 4.2.4 — DNS configuration¶

Add hostnames to /etc/hosts on both bastion and local laptop so the UIs are reachable.

Use any ingress IP from the ADDRESS column above (all are valid — any node works):

100.115.1.14  gitlab.cl1.sq4.aegis.internal
              registry.cl1.sq4.aegis.internal
              kas.cl1.sq4.aegis.internal
              argocd.cl1.sq4.aegis.internal

On bastion:

sudo tee -a /etc/hosts << 'EOF'
100.115.1.14 gitlab.cl1.sq4.aegis.internal registry.cl1.sq4.aegis.internal kas.cl1.sq4.aegis.internal argocd.cl1.sq4.aegis.internal
EOF

On local laptop (Mac):

sudo tee -a /etc/hosts << 'EOF'
100.115.1.14 gitlab.cl1.sq4.aegis.internal registry.cl1.sq4.aegis.internal kas.cl1.sq4.aegis.internal argocd.cl1.sq4.aegis.internal
EOF

Verify from bastion:

# GitLab
curl -s http://gitlab.cl1.sq4.aegis.internal/-/health
# Expected: GitLab OK

# ArgoCD
curl -s http://argocd.cl1.sq4.aegis.internal/healthz
# Expected: ok

Verify from Mac (browser):

Service	URL
GitLab	`http://gitlab.cl1.sq4.aegis.internal`
ArgoCD	`http://argocd.cl1.sq4.aegis.internal`

ArgoCD login:

# Get admin password
kubectl get secret argocd-initial-admin-secret -n argocd \
  -o jsonpath='{.data.password}' | base64 -d; echo

Field	Value
URL	`http://argocd.cl1.sq4.aegis.internal`
Username	`admin`
Password	output of command above

Step 4.2.5 — Connect ArgoCD to GitLab¶

ArgoCD needs a GitLab token to pull repo contents (manifests, Helm values, ApplicationSets).

Use a dedicated service account — not a personal token

Create a dedicated GitLab user (argocd-ro) with a PAT scoped to read_repository. Never use a personal token tied to a human user — it breaks when the user leaves. Vault storage of the token comes after Vault is deployed (Phase 4.3).

Step 4.2.5a — Create dedicated GitLab service account¶

Run from: browser — http://gitlab.cl1.sq4.aegis.internal

Log in as root
Go to Admin Area → Users → New User

Field	Value
Name	`argocd-ro`
Username	`argocd-ro`
Email	`[email protected]`
Role	`Regular`

After creating → go to the user → Access Tokens → Add new token

Field	Value
Token name	`argocd-read`
Expiry	None (or max)
Scopes	✅ `read_repository` only

Copy the token — it will not be shown again

Step 4.2.5b — Install ArgoCD CLI on bastion¶

# Download ArgoCD CLI (requires internet or proxy)
curl -sL -o /tmp/argocd https://github.com/argoproj/argo-cd/releases/download/v2.14.12/argocd-linux-amd64
chmod +x /tmp/argocd
sudo mv /tmp/argocd /usr/local/bin/argocd
argocd version --client

ARGOCD_PASSWORD=$(kubectl get secret argocd-initial-admin-secret \
  -n argocd -o jsonpath='{.data.password}' | base64 -d)

# Use --insecure until wildcard cert is patched onto the ArgoCD Ingress (Phase 6)
argocd login argocd.cl1.sq4.aegis.internal \
  --username admin \
  --password "${ARGOCD_PASSWORD}" \
  --insecure \
  --grpc-web

Step 4.2.5d — Add credential template for GitLab¶

A credential template covers all repos under the URL prefix — one entry covers all GitLab repos:

argocd repocreds add http://gitlab.cl1.sq4.aegis.internal/ai71/ \
  --username argocd-ro \
  --password "<PAT>" \
  --insecure \
  --grpc-web

# Verify
argocd repocreds list --grpc-web

Expected:

URL PATTERN                                  USERNAME   SSH_CREDS  TLS_CREDS
http://gitlab.cl1.sq4.aegis.internal/ai71/   argocd-ro  false      false

Step 4.2.5e — Add CA certs to ArgoCD trust store¶

Harbor uses the Aegis CL1.SQ4 CA (different from modgpt CA). Extract and add both:

# Extract Harbor's intermediate CA from the TLS chain
echo | openssl s_client -connect 100.115.1.101:443 -showcerts 2>/dev/null | \
  awk '/BEGIN CERTIFICATE/,/END CERTIFICATE/' | \
  csplit - '/BEGIN CERTIFICATE/' '{*}' -f /tmp/harbor-cert- --suffix-format='%02d.pem' 2>/dev/null

# Add modgpt CA (for GitLab + Nginx Ingress services)
argocd cert add-tls gitlab.cl1.sq4.aegis.internal \
  --from ~/certificates/modgpt-ca.crt --grpc-web

# Add Aegis Intermediate CA (for Harbor — has its own CA chain)
argocd cert rm harbor.cl1.sq4.aegis.internal --cert-type https --grpc-web 2>/dev/null || true
cat ~/certificates/modgpt-ca.crt /tmp/harbor-cert-02.pem > /tmp/combined-ca.pem
argocd cert add-tls harbor.cl1.sq4.aegis.internal \
  --from /tmp/combined-ca.pem --grpc-web

# Verify
argocd cert list --grpc-web | grep -E "gitlab|harbor"

Expected:

gitlab.cl1.sq4.aegis.internal  https  rsa    CN=modgpt,O=ai71
harbor.cl1.sq4.aegis.internal  https  rsa    CN=modgpt,O=ai71
harbor.cl1.sq4.aegis.internal  https  ecdsa  CN=Aegis CL1.SQ4 CA Intermediate CA,O=Aegis CL1.SQ4 CA

Why Harbor has a different CA

Harbor runs outside Kubernetes with a dedicated IP (100.115.1.101) and its own TLS cert signed by the Aegis CL1.SQ4 CA — not the modgpt CA used for Kubernetes Ingress services. Both CAs must be trusted for ArgoCD to connect to both GitLab and Harbor.

Step 4.2.5f — Add Harbor as OCI Helm registry¶

HARBOR_PASS=$(python3 -c "import json; j=json.load(open('/home/cloud-user/helm-charts/harbor-robot.json')); print(j.get('secret',''))")

argocd repo add harbor.cl1.sq4.aegis.internal/helm-charts \
  --type helm \
  --enable-oci \
  --username 'robot$ask' \
  --password "${HARBOR_PASS}" \
  --grpc-web \
  --name harbor-helm

argocd repo list --grpc-web

Expected:

TYPE  NAME         REPO                                         STATUS      MESSAGE
helm  harbor-helm  harbor.cl1.sq4.aegis.internal/helm-charts   Successful

Step 4.2.6 — Verify ArgoCD¶

kubectl get pods -n argocd

argocd repo list
# Expected:
# TYPE  NAME         REPO                                                          STATUS      MESSAGE
# git   ...          https://gitlab.cl1.sq4.aegis.internal/ask/cluster-addons.git Successful
# helm  harbor-helm  harbor.cl1.sq4.aegis.internal/helm-charts                    Successful

If a repo shows Failed — check:

argocd repo get https://gitlab.cl1.sq4.aegis.internal/ask/cluster-addons.git
# Look for: ConnectionState.Message

Failure message	Cause	Fix
`authentication required`	Wrong username/token	Re-check token scope — needs `read_repository`
`x509: certificate signed by unknown authority`	CA not trusted	Add `--insecure-skip-server-verification` or add CA cert
`repository not found`	Repo URL wrong or token has no access	Check group path in GitLab URL

4.3 — HashiCorp Vault (via ArgoCD)¶

cluster-addons/apps/vault/Chart.yaml

apiVersion: v2
name: vault
version: 0.1.0
dependencies:
  - name: vault
    version: "0.32.0"
    repository: "oci://harbor.cl1.sq4.aegis.internal/helm-charts"

cluster-addons/apps/vault/values.yaml

vault:
  global:
    imagePullSecrets:
      - name: harbor-pull-secret
  server:
    image:
      repository: harbor.cl1.sq4.aegis.internal/dockerhub/hashicorp/vault
      tag: "1.21.2"
    ha:
      enabled: true
      replicas: 3
  injector:
    image:
      repository: harbor.cl1.sq4.aegis.internal/dockerhub/hashicorp/vault-k8s
      tag: "1.7.2"
    agentImage:
      repository: harbor.cl1.sq4.aegis.internal/dockerhub/hashicorp/vault
      tag: "1.21.2"

4.4 — Vault Secrets Operator (via ArgoCD)¶

cluster-addons/apps/vso/values.yaml

vault-secrets-operator:
  controller:
    imagePullSecrets:
      - name: harbor-pull-secret
    manager:
      image:
        repository: harbor.cl1.sq4.aegis.internal/dockerhub/hashicorp/vault-secrets-operator
        tag: "0.9.1"
    kubeRbacProxy:
      image:
        repository: harbor.cl1.sq4.aegis.internal/quay/brancz/kube-rbac-proxy
        tag: "v0.18.1"

4.5 — Stakater Reloader (via ArgoCD)¶

cluster-addons/apps/reloader/values.yaml

reloader:
  reloader:
    deployment:
      image:
        name: harbor.cl1.sq4.aegis.internal/ghcr/stakater/reloader
        tag: "v1.4.17"
      imagePullSecrets:
        - name: harbor-pull-secret

Summary checklist¶

Manual deploys (bootstrap)¶

[ ] 4.1 GitLab installed and reachable
[ ] 4.1 GitLab images confirmed pulling from Harbor
[ ] 4.1 cluster-addons repo created in GitLab
[ ] 4.2 ArgoCD installed and connected to GitLab + Harbor

ArgoCD-managed deploys¶

[ ] 4.3 Vault — Synced / Healthy, Raft initialized
[ ] 4.4 VSO — Synced / Healthy, controller 2/2 Running
[ ] 4.5 Reloader — Synced / Healthy

Next: Phase 5 — ASK Core Service Deployment →

Phase 4 — Platform Components¶

Deployment Order¶

4.1 — GitLab¶

Prerequisites¶

Step 4.1.1 — Create namespace and secrets¶

Step 4.1.2 — Verify S3 prerequisites (Phase 2)¶

Step 4.1.4 — Create values file¶

Step 4.1.5 — Install GitLab from Harbor¶

Step 4.1.5a — Add nginx-ingress IngressClass disable to values¶

Step 4.1.5b — Pull and patch chart¶

Step 4.1.5c — Install¶

Step 4.1.5d — Known install issues (empirically confirmed)¶

Step 4.1.6 — Verify GitLab pods¶

Step 4.1.7 — Configure containerd for HTTP registry¶

Step 4.1.8 — Post-install setup¶

4.1 (TLS Upgrade) — Add TLS to GitLab after CA access is available¶

Step T1 — Get root CA cert¶

Step T2 — Create ClusterIssuer¶

Step T3 — Trust CA on all nodes (for HTTPS image pulls)¶

Step T4 — Update values file and upgrade GitLab¶

4.2 — ArgoCD¶

Step 4.2.1 — Create namespace and imagePullSecret¶

Step 4.2.2 — Create values file¶

Step 4.2.3 — Install ArgoCD from Harbor¶

Step 4.2.4 — DNS configuration¶

Step 4.2.5 — Connect ArgoCD to GitLab¶

Step 4.2.5a — Create dedicated GitLab service account¶

Step 4.2.5b — Install ArgoCD CLI on bastion¶

Step 4.2.5c — Login to ArgoCD CLI¶

Step 4.2.5d — Add credential template for GitLab¶

Step 4.2.5e — Add CA certs to ArgoCD trust store¶

Step 4.2.5f — Add Harbor as OCI Helm registry¶

Step 4.2.6 — Verify ArgoCD¶

4.3 — HashiCorp Vault (via ArgoCD)¶

4.4 — Vault Secrets Operator (via ArgoCD)¶

4.5 — Stakater Reloader (via ArgoCD)¶

Summary checklist¶

Manual deploys (bootstrap)¶

ArgoCD-managed deploys¶