Private GitLab Images on Kubernetes: The Guide

Short answer: to pull an image from a private GitLab registry in Kubernetes, create a deploy token with the read_registry scope on the project or the group, turn it into a kubernetes.io/dockerconfigjson secret in the target namespace, then reference that secret in imagePullSecrets (on the pod or, better, on the ServiceAccount). Three commands are enough; the rest of this article covers which scope to pick, rotation, and how to diagnose ImagePullBackOff errors.

It is one of the most common roadblocks when a team moves from docker compose on a single server to a Kubernetes cluster: the pipeline happily pushes the image to registry.gitlab.com/my-group/my-app, but the cluster has no identity with which to fetch it. Pods loop endlessly, logs are empty, and kubectl describe returns a terse unauthorized: HTTP Basic: Access denied.

Why Kubernetes cannot read your GitLab registry

The GitLab Container Registry is private by default: its visibility follows the project's. A private project means an authenticated registry, and the kubelet — it is the kubelet that pulls images, not the control plane — inherits no ~/.docker/config.json from your workstation. Even if docker pull works on your machine after a docker login, the cluster remains anonymous.

Kubernetes solves this with a Secret object of type kubernetes.io/dockerconfigjson. The kubelet reads that secret, extracts the credentials, and presents them to the registry during the pull. One essential point, often learned the hard way: a secret belongs to a single namespace. A secret created in default is invisible from staging or production. You therefore have to recreate it in every namespace that deploys the image.

Which credential should you use?

GitLab offers several kinds of token, and they are not all equally suited to this job.

The deploy token wins for three reasons: it is not attached to any human being (nothing breaks when someone leaves the team), it accepts the read_registry scope on its own, and it can be created at group level to cover several microservices with a single username/password pair.

CI_JOB_TOKEN, meanwhile, is very handy for docker login inside the pipeline, but its lifetime matches the job's. A Kubernetes secret built from it will work on the first deployment, then break the first time a pod restarts — a classic and particularly nasty trap, since the symptom shows up hours after go-live.

Creating the deploy token

In GitLab: Settings → Repository → Deploy tokens for a project, or Settings → Repository → Deploy tokens at group level. Fill in:

• Name: k8s-prod-pull (name it after the cluster/environment, not the application)
• Expiration date: set one. One year maximum; six months is a good compromise if you automate rotation.
• Username: leave it blank to get a generated name (gitlab+deploy-token-123456), or set it to keep your scripts simple.
• Scopes: tick only read_registry. Not read_repository, not write_registry.

Copy the password immediately: GitLab never shows it again.

Tip: for an image stored in the Dependency Proxy or in a package registry, the required scope is different (read_package_registry). For plain Docker images, read_registry is enough.

Generating the imagePullSecret

The imperative command is the safest one, because it builds the JSON for you:

kubectl create secret docker-registry gitlab-registry \
  --namespace=production \
  --docker-server=registry.gitlab.com \
  --docker-username='gitlab+deploy-token-123456' \
  --docker-password='gldt-xxxxxxxxxxxxxxxxxxxx' \
  --docker-email='devops@example.com'

Three details that make half of all attempts fail:

1. --docker-server must be the registry host, not the GitLab host. On gitlab.com it is registry.gitlab.com. On a self-hosted instance it is the value of registry_external_url in gitlab.rb, often registry.example.com — never gitlab.example.com. Check it by reading the prefix shown on the project's Deploy → Container Registry page.
2. No scheme, no path. https://registry.gitlab.com/my-group produces a useless secret: the kubelet compares the image host with the key in config.json.
3. Single quotes around the password. GitLab tokens can contain characters the shell would interpret.

Check the result without exposing it in your shell history:

kubectl -n production get secret gitlab-registry \
  -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq '.auths | keys'

You should see exactly ["registry.gitlab.com"].

Attaching the secret

Two approaches. At pod level, explicitly:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: production
spec:
  template:
    spec:
      imagePullSecrets:
        - name: gitlab-registry
      containers:
        - name: api
          image: registry.gitlab.com/my-group/my-app:1.4.2

Or, more elegantly, on the ServiceAccount used by the namespace — every pod that uses it inherits the secret, with no changes to the manifests:

kubectl -n production patch serviceaccount default \
  -p '{"imagePullSecrets":[{"name":"gitlab-registry"}]}'

The second option prevents oversights when a new application arrives, but it is less explicit to read. On the clusters we operate for our clients, we patch the ServiceAccount and keep imagePullSecrets in the Helm charts: the redundancy is harmless and doubles as documentation.

A good practice: imagePullPolicy and immutable tags

While you are at it, set the pull policy. Kubernetes applies IfNotPresent by default, except for a :latest tag, where it switches to Always. The consequence: with :latest, a redeployment pulls a different image with nothing in Git to signal it, and an expired secret breaks the pod immediately.

The rule we apply systematically:

• tag by semantic version or short commit SHA (:1.4.2, :sha-a1b2c3d);
• keep imagePullPolicy: IfNotPresent;
• enable protected and immutable tags on the GitLab side to prevent an already-deployed tag from being overwritten.

An image tagged by SHA also makes rollbacks trivial: kubectl rollout undo deployment/api points back to a layer already present in the node's cache, with no dependency on the registry.

Automating it from GitLab CI/CD

Creating the secret by hand once per namespace works — right up to the day the token expires. Two strategies stand the test of time.

Strategy 1: idempotent recreation in the pipeline

Store the deploy token in masked and protected CI/CD variables (K8S_PULL_USER, K8S_PULL_TOKEN), then regenerate the secret on every deployment:

deploy:
  stage: deploy
  image: bitnami/kubectl:1.34
  environment: production
  rules:
    - if: $CI_COMMIT_TAG
  script:
    - kubectl create secret docker-registry gitlab-registry
        --namespace="$KUBE_NAMESPACE"
        --docker-server="$CI_REGISTRY"
        --docker-username="$K8S_PULL_USER"
        --docker-password="$K8S_PULL_TOKEN"
        --dry-run=client -o yaml | kubectl apply -f -
    - helm upgrade --install api ./charts/api
        --namespace "$KUBE_NAMESPACE"
        --set image.tag="$CI_COMMIT_TAG"
        --atomic --timeout 5m

The --dry-run=client -o yaml | kubectl apply -f - combination is the canonical way to make kubectl create secret idempotent: it updates the secret if it exists, creates it otherwise, and never returns AlreadyExists. $CI_REGISTRY is provided automatically by GitLab and already contains the right host, which eliminates mistake #1 described above.

Helm's --atomic saves you from leaving a half-deployed Deployment behind if the pull fails anyway: the chart reverts to its previous state.

Strategy 2: a secret managed outside the pipeline

On a multi-team cluster, it is better to take the token out of CI. Two tools handle this effortlessly:

• External Secrets Operator: the token lives in AWS Secrets Manager, Vault or Scaleway Secret Manager, and an ExternalSecret object materialises it as a dockerconfigjson in each namespace. Rotation happens in the vault, and the cluster follows automatically on the next refresh cycle.
• Reflector or kubernetes-replicator: you maintain a single source secret in an administration namespace, and an annotation replicates it into every authorised namespace. Simple, and very effective when your only pain point is the number of namespaces.

In both cases, record the deploy token's expiry date in your runbook. GitLab sends a warning email before tokens with an expiry date lapse, but that message often lands in the inbox of a single project owner.

Diagnosing an ImagePullBackOff

When things break, follow this order. It isolates the cause in under five minutes.

# 1. Read the exact event message
kubectl -n production describe pod api-7d9c8b5f4-abcde | tail -30

# 2. Check that the secret really exists IN this namespace
kubectl -n production get secret gitlab-registry

# 3. Check that the pod actually references it
kubectl -n production get pod api-7d9c8b5f4-abcde \
  -o jsonpath='{.spec.imagePullSecrets}'

Then read the message:

To replay the pull manually with the same credentials, outside Kubernetes:

echo "$K8S_PULL_TOKEN" | docker login registry.gitlab.com \
  -u 'gitlab+deploy-token-123456' --password-stdin
docker pull registry.gitlab.com/my-group/my-app:1.4.2

If that command succeeds from your workstation but fails in the cluster, authentication is no longer the problem: look at the nodes' outbound network (corporate proxy, egress rules, network policy) or at node disk space — a full /var/lib/containerd produces misleading pull failures, visible through the DiskPressure condition in kubectl describe node.

Registry cost, the frequently forgotten part

GitLab Container Registry storage is billed against the GitLab namespace's storage quota, and it grows fast: every pipeline pushes a complete image. With a 350 MB Node.js image and forty pipelines a week, you add more than 50 GB a month if nothing cleans up.

Two practical levers:

• Tag expiration rules (Settings → Packages and registries → Clean up image tags): keep the last 10 tags, keep those matching v* versions, delete the rest after 30 days. Note that the purge only really frees space once the registry's garbage collector has run.
• Smaller images: a node:22-alpine base or a multi-stage build with npm ci --omit=dev can shave hundreds of megabytes off the image, and speeds up the pull on every new node accordingly.

The second lever has a direct effect on your startup time when traffic spikes: a pod pulling 80 MB instead of 350 MB starts noticeably faster when the autoscaler adds a fresh node. That is exactly the kind of optimisation we build into the infrastructure and deployment services we run for our clients, alongside application development itself.

Frequently asked questions

Do you need one secret per namespace?

Yes. Kubernetes secrets do not cross namespace boundaries. Either you recreate the secret in each of them (through the pipeline, idempotently), or you use a replicator such as Reflector or External Secrets Operator to propagate it from a single source.

Can CI_JOB_TOKEN be used for the imagePullSecret?

No, not for a long-lived secret. That token expires at the end of the job. It is perfect for docker login during the build, but a pod restarted a few hours later will no longer be able to pull the image. Use a deploy token.

Group or project deploy token?

Project if a single application uses the registry; group as soon as you deploy several services from the same group on the same cluster. A group token reduces the number of secrets to rotate; in exchange, it widens the blast radius if it is compromised. In every case: read_registry scope only.

How do you avoid an outage on expiry day?

Create the new token before deleting the old one, apply the new secret in every namespace, verify with a kubectl rollout restart on a non-critical application, then revoke the old token. And put the expiry date in your operations calendar rather than relying on GitLab's alert email.

What about a self-hosted registry behind an internal certificate?

The secret is identical, but the kubelet has to trust the certificate authority. Install it in the nodes' system trust store (/usr/local/share/ca-certificates/ then update-ca-certificates, and restart containerd) or, depending on the runtime, configure /etc/containerd/certs.d/<host>/. Changing the pod will make no difference: the pull happens at node level.

In summary

• The GitLab registry is private by default; the kubelet needs a kubernetes.io/dockerconfigjson secret to authenticate against it.
• Use a deploy token (project or group) with the read_registry scope only, never a personal token or CI_JOB_TOKEN.
• --docker-server is the registry host (registry.gitlab.com or $CI_REGISTRY), with no https:// and no project path.
• One secret per namespace: make its creation idempotent with --dry-run=client -o yaml | kubectl apply -f -, or propagate it via External Secrets Operator.
• Attach it to the ServiceAccount to cover every pod, and keep imagePullSecrets in your charts as documentation.
• Tag by version or SHA, never :latest, with imagePullPolicy: IfNotPresent and immutable tags.
• When you hit an ImagePullBackOff, read the kubectl describe message: it distinguishes an invalid token (Access denied) from a non-existent tag (manifest unknown).
• Clean up the registry (tag expiration rules) and shrink your images: less billed storage, faster startups.

A deployment stuck on an image pull, a cluster to industrialise, or a GitLab CI/CD chain to take over? Tell us about your situation — our team works on these topics every day from Pontault-Combault.