Private Docker Registry in K8s
Mirroring package repositories has been an option available to Linux users for a long time. It’s a great way to save bandwidth and speed up package installation. This is especially true if you’re using Kubernetes, where you’ll be pulling images from a registry many times a day. There’s a lot of value to doing the same with Docker images, particularly for any that are private and only in active use in your homelab.
In this post, we’ll explore some of the considerations for setting up a private Docker registry in Kubernetes.
tl;dr - Here Are the Manifests
For those of you looking for example manifests instead of an explanation of the process, here you go. This will create the Persistent Volume Claim (PVC) that allows you to store the registry data on a persistent volume, the Deployment that runs the registry, the Service that exposes it, and the Ingress that allows you to access it from outside the cluster. Note that I’m using the nfs-csi-retain StorageClass, which uses a CSI driver for NFS that I’ve configured to retain the volume when the PVC is deleted. I’ve also set up Ingress to use cert-manager to automatically provision a TLS certificate from LetsEncrypt and nginx to handle the routing/load balancing. Those are out of the scope of this post, but I’ll cover them in a future post. If you want to set them up on your own I used the Helm charts for both.
You’ll want to change the namespace to something more appropriate for you and include a manifest to create that namespace, if it doesn’t already exist. Copying and pasting these manifests and running kubectl apply will probably not work!
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: docker-registry-pvc
namespace: graywind
spec:
storageClassName: nfs-csi-retain
accessModes: [ReadWriteMany]
resources:
requests:
storage: 50Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: docker-registry-deployment
namespace: graywind
spec:
replicas: 1
selector:
matchLabels:
app: docker-registry
template:
metadata:
labels:
app: docker-registry
spec:
containers:
- name: docker-registry
image: registry:2
ports:
- containerPort: 5000
volumeMounts:
- name: docker-registry-app-data
mountPath: /var/lib/registry
subPath: registry
volumes:
- name: docker-registry-app-data
persistentVolumeClaim:
claimName: docker-registry-pvc
---
apiVersion: v1
kind: Service
metadata:
name: docker-registry-service
namespace: graywind
spec:
selector:
app: docker-registry
ports:
- protocol: TCP
port: 5000
targetPort: 5000
nodePort: 30500
type: NodePort
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: docker-registry-ingress
namespace: graywind
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
nginx.ingress.kubernetes.io/proxy-body-size: "2.5G"
spec:
ingressClassName: nginx
rules:
- host: docker.graywind.org
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: docker-registry-service
port:
number: 5000
tls:
- hosts:
- docker.graywind.org # This resolves only in my private DNS, so I know it won't be used outside my network
secretName: docker-registry-tls
Persistent Volume Claim (PVC)
Previously, I set up a Persistent Volume called nfs-csi-retain that uses NFS to store data on a Synology NAS. I’m using that same volume to store the data for the Docker registry. The PVC is pretty straightforward, but there are a few things to note.
First, the accessMode of ReadWriteMany is not required - you could just as easily use ReadWriteOnce - but I set it up this way so I could have zero downtime when I need to update the registry or if the machine running it is rebooted or loses power.
Secondly, spec.resources.request.storage is key to making sure that the PVC is large enough to store the data. I’m using 50Gi, but you can use whatever size you need. If you don’t set this, Kubernetes will create a PVC that is only 1Gi in size, which is not enough for a Docker registry. Sometimes it’s not enough for a single image!
Deployment
The Deployment is also pretty straightforward. Let’s look at a few of the key points. First, I’m using the app: docker-registry label to make it easier to select the pod for debugging, setting up a Service, or other purposes. I’m mounting the volume we set up in the first manifest to /var/lib/registry. Finally, there’s the image itself - registry:2, which is also latest as of this writing. You can use whatever version you want, but I recommend using a specific version instead of latest so you don’t accidentally upgrade to a version that breaks something.
Service
The Service is a key piece of the infrastructure. Without it, you won’t be able to access the registry from outside the cluster. I’m using a NodePort Service, which means that the registry will be accessible on every node in the cluster on port 30500.
Ingress
Finally, the Ingress is what allows us to communicate securely with the Docker Registry. Docker expects any registry to be using HTTPS, so this is a key piece of the infrastructure. We’re using two annotations. One is for LetsEncrypt, which will automatically provision a TLS certificate for us. The other is for nginx, which is the Ingress controller I’m using. It’s important to set the nginx.ingress.kubernetes.io/proxy-body-size annotation to a large enough value to allow you to push large images to the registry. I’m using 2.5G, which should be plenty for most Docker images I’ll be using. The spec.rules section allows us to forward traffic to the Service appropriately.
Why do all this?
Why go to all this trouble with Kubernetes, nginx, LetsEncrypt, and the PVC? I originally tried to run a very simple container registry on my Synology NAS, but almost immediately ran into issues with the lack of TLS encryption. The registry itself worked just fine and the volume it was using did too. However, Docker expects that registries are encrypted, and setting it up to accept a registry over HTTP instead of HTTPS is a bit of a pain.
I initially experimented with this using Nomad. While I could easily run the container, the community support for container storage is not as strong, and neither is the support for TLS. I could have used a self-signed certificate, but I wanted to use a real certificate from LetsEncrypt. I could have done all of that with Nomad, but it would have been a lot more work.
Kubernetes isn’t always the right solution, but for my situation, it turned out to be the least hassle to get a working container registry.
If you have questions or issues setting up a similar registry, please feel free to reach out! I’m happy to chat through what’s going on and help get you up and running.