Phase 1 — Image Mirroring to Harbor¶

Mirror all platform and application container images from public registries and Azure ACR into the Core42 Harbor private registry, so the air-gapped Kubernetes cluster never needs internet access at runtime.

Tools

Tool	Script	Config	Run from
Image mirror	`core42-mirror-image.py`	`mirror-config.yaml`	`~/harbor-images/`
Chart mirror	`core42-mirror-charts.py`	`mirror-charts.yaml`	`~/helm-charts/`

Harbor Project Layout¶

Create these projects in Harbor before running the mirror (one-time setup):

Harbor Project	Source Registry	Contents
`ask`	Azure ACR	core-service + workers (app images)
`gitlab`	`registry.gitlab.com`	GitLab CNG components (13 images)
`dockerhub`	`docker.io`	postgres, python, hashicorp/vault, vault-k8s, VSO
`bitnamilegacy`	`docker.io/bitnamilegacy/*`	postgresql, redis, postgres-exporter, redis-exporter
`ghcr`	`ghcr.io`	cloudnative-pg, stakater/reloader
`quay`	`quay.io`	cert-manager ×5, argocd, kube-rbac-proxy
`helm-charts`	Helm OCI push	Helm chart `.tgz` packages

Create gitlab and quay projects before first run

These are not auto-created.

Harbor UI → + New Project for each:

Project	Access
`gitlab`	Private
`quay`	Private

Step 1.1 — Verify prerequisites¶

Run from

~/harbor-images/

# Confirm skopeo is available
skopeo --version

# Confirm mirror script, config and credentials are present
ls ~/harbor-images/
# Expected: core42-mirror-image.py  mirror-config.yaml  harbor-robot.json  image-cache/

# Confirm helm chart tool is present
ls ~/helm-charts/
# Expected: core42-mirror-charts.py  mirror-charts.yaml  harbor-robot.json  README.md

Step 1.2 — Dry-run images (always run first)¶

Run from

~/harbor-images/

./core42-mirror-image.py --dry-run

Expected output (33 images, 1 disabled):

[proxy]    ON  https_proxy=http://10.96.137.37:3128  ...
[1/33]  [DISABLED] acr/core-service:1.0.0  (features.use_acr: false)
[2/33]  [DRY] download registry.gitlab.com/gitlab-org/build/cng/gitaly:v18.11.4 -> cache
[2/33]  [DRY] push -> harbor.cl1.sq4.aegis.internal/gitlab/gitlab-org/build/cng/gitaly:v18.11.4
...
[15/33] [DRY] push -> harbor.cl1.sq4.aegis.internal/dockerhub/postgres:16-alpine
[21/33] [DRY] push -> harbor.cl1.sq4.aegis.internal/bitnamilegacy/postgresql:16.6.0
[27/33] [DRY] push -> harbor.cl1.sq4.aegis.internal/quay/jetstack/cert-manager-controller:v1.20.2
[33/33] [DRY] push -> harbor.cl1.sq4.aegis.internal/quay/brancz/kube-rbac-proxy:v0.18.1
[proxy]    OFF (environment restored)

=== summary ===
  pushed     : 0
  skipped    : 0
  from_cache : 0
  disabled   : 1
  failed     : 0

✅ Verify all 33 destinations show correct harbor.cl1.sq4.aegis.internal/<project>/... paths before proceeding.

Step 1.3 — Download images to cache (internet side)¶

Run from

~/harbor-images/

./core42-mirror-image.py --download-only -v

Pulls all images into ./image-cache/ as portable OCI layouts. No Harbor access needed.

Idempotent

Already-cached images show [CACHED] and are skipped — safe to re-run if interrupted.

Step 1.4 — Push images to Harbor (air-gapped side)¶

Run from

~/harbor-images/

./core42-mirror-image.py --push-only -v

If cache was built on a different host

Transfer the cache to the bastion first:

rsync -avz ./image-cache/ cloud-user@bastion1:~/harbor-images/image-cache/

Then run --push-only on the bastion.

Step 1.5 — Full mirror (download + push in one pass)¶

Run from

~/harbor-images/

If the bastion has both internet access and Harbor access:

./core42-mirror-image.py --no-cache

Why --no-cache

Streams images directly source → Harbor without staging on disk. Use when the bastion has simultaneous internet + Harbor access.

Step 1.6 — Verify images in Harbor¶

Run from

~/harbor-images/

python3 - << 'EOF'
import json, urllib.request, base64, ssl

j      = json.load(open("./harbor-robot.json"))
user   = j["name"]
passwd = j["secret"]
base   = "https://harbor.cl1.sq4.aegis.internal/api/v2.0/projects"
token  = base64.b64encode(f"{user}:{passwd}".encode()).decode()

ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

for project in ["gitlab", "dockerhub", "bitnamilegacy", "ghcr", "quay"]:
    url = f"{base}/{project}/repositories"
    req = urllib.request.Request(url, headers={"Authorization": f"Basic {token}"})
    with urllib.request.urlopen(req, context=ctx) as resp:
        repos = json.load(resp)
        print(f"\n=== {project} ({len(repos)} repos) ===")
        for r in repos:
            print(f"  {r['name']}")
EOF

Expected output:

gitlabdockerhubbitnamilegacyghcrquay

gitlab/gitlab-org/build/cng/gitaly
gitlab/gitlab-org/build/cng/gitlab-base
gitlab/gitlab-org/build/cng/gitlab-container-registry
gitlab/gitlab-org/build/cng/gitlab-exporter
gitlab/gitlab-org/build/cng/gitlab-kas
gitlab/gitlab-org/build/cng/gitlab-shell
gitlab/gitlab-org/build/cng/gitlab-sidekiq-ee
gitlab/gitlab-org/build/cng/gitlab-toolbox-ee
gitlab/gitlab-org/build/cng/gitlab-webservice-ee
gitlab/gitlab-org/build/cng/gitlab-workhorse-ee
gitlab/gitlab-org/build/cng/kubectl
gitlab/gitlab-org/build/cng/certificates
gitlab/gitlab-org/cloud-native/mirror/images/minio/mc

dockerhub/postgres
dockerhub/python
dockerhub/hashicorp/vault
dockerhub/hashicorp/vault-k8s
dockerhub/hashicorp/vault-secrets-operator

bitnamilegacy/postgresql
bitnamilegacy/redis
bitnamilegacy/postgres-exporter
bitnamilegacy/redis-exporter

ghcr/cloudnative-pg/cloudnative-pg
ghcr/stakater/reloader

quay/jetstack/cert-manager-controller
quay/jetstack/cert-manager-webhook
quay/jetstack/cert-manager-cainjector
quay/jetstack/cert-manager-acmesolver
quay/jetstack/cert-manager-startupapicheck
quay/argoproj/argocd
quay/brancz/kube-rbac-proxy

Step 1.7 — Mirror Helm charts to Harbor¶

Helm chart .tgz files are pushed to the Harbor helm-charts project via OCI (helm push). Harbor acts as a single registry for both container images and Helm charts.

Tool

Script: ~/helm-charts/core42-mirror-charts.py Config: ~/helm-charts/mirror-charts.yaml

Step 1.7a — Dry-run charts¶

Run from

~/helm-charts/

./core42-mirror-charts.py --dry-run

Expected output:

[proxy]    ON  https_proxy=http://10.96.137.37:3128  ...
[auth]     credentials loaded from ./harbor-robot.json
[1/5] [DRY]  helm repo add gitlab https://charts.gitlab.io
[1/5] [DRY]  helm pull gitlab/gitlab --version 9.11.5 -> cache
[1/5] [DRY]  helm push gitlab-9.11.5.tgz oci://harbor.cl1.sq4.aegis.internal/helm-charts
...
[proxy]    OFF (environment restored)

=== summary ===
  pushed      : 0
  skipped     : 0
  from_cache  : 0
  failed      : 0

Step 1.7b — Mirror all charts¶

Run from

~/helm-charts/

# With cache (default)
./core42-mirror-charts.py

# Without cache — direct pull → push
./core42-mirror-charts.py --no-cache

# Single chart only
./core42-mirror-charts.py --only vault
./core42-mirror-charts.py --only gitlab

Step 1.7c — Verify charts in Harbor¶

Run from

~/helm-charts/

python3 - << 'EOF'
import json, urllib.request, base64, ssl

j      = json.load(open("./harbor-robot.json"))
user   = j["name"]
passwd = j["secret"]
url    = "https://harbor.cl1.sq4.aegis.internal/api/v2.0/projects/helm-charts/repositories"
token  = base64.b64encode(f"{user}:{passwd}".encode()).decode()

req = urllib.request.Request(url, headers={"Authorization": f"Basic {token}"})
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

with urllib.request.urlopen(req, context=ctx) as resp:
    repos = json.load(resp)
    if isinstance(repos, list):
        for r in repos:
            print(r["name"])
    else:
        print("ERROR:", repos)
EOF

Expected output (verified ✅):

helm-charts/reloader
helm-charts/vault-secrets-operator
helm-charts/vault
helm-charts/argo-cd
helm-charts/gitlab

Or verify with helm pull:

helm pull oci://harbor.cl1.sq4.aegis.internal/helm-charts/vault --version 0.32.0

Chart inventory¶

Helm Chart	Version	Upstream Source	Harbor OCI Path
`gitlab`	`9.11.5`	`charts.gitlab.io`	`harbor.../helm-charts/gitlab:9.11.5`
`argo-cd`	`9.5.15`	`argoproj.github.io/argo-helm`	`harbor.../helm-charts/argo-cd:9.5.15`
`vault`	`0.32.0`	`helm.releases.hashicorp.com`	`harbor.../helm-charts/vault:0.32.0`
`vault-secrets-operator`	`0.9.1`	`helm.releases.hashicorp.com`	`harbor.../helm-charts/vault-secrets-operator:0.9.1`
`reloader`	`2.2.12`	`stakater.github.io/stakater-charts`	`harbor.../helm-charts/reloader:2.2.12`

Summary checklist¶

[x] Harbor projects gitlab and quay created
[x] Image dry-run shows all 33 correct destinations
[x] All images pushed — failed: 0
[x] Images verified in Harbor (all 6 projects)
[x] Helm charts dry-run validated
[x] All 5 charts pushed to helm-charts project
[x] Charts verified in Harbor

Next: Phase 2 — Storage Validation →