Allow Kubernetes running outside of AWS EKS to pull images from AWS ECR¶
Motivation¶
Of course you can just use Docker Hub to host images and access to it should be pretty straight forward for any Kubernetes manifest - as long as your containers are public.
But perhaps you don't want your current project's containers to be public? Maybe you don't want to use Docker Hub. Perhaps you are working on a contract for a client. Your use case may be that you or your company already have an AWS account, but you would like to test in Kubernetes running locally on your development machine - but still reference the images from AWS ECR.
If these scenarios fits your situation, read on...
Basic Solution¶
Assuming you know how to push images to AWS ECR, the trick comes when you have a Kubernetes cluster deployed outside of AWS EKS. The cluster needs to authenticate to something in order to obtain the required tokens that would facilitate the downloading of container images from AWS ECR.
In the same page that describes how to push images to AWS ECR, there is also a section illustrating how to pull an image using the CLI. It is however still important to recognize that you need to have credentials in order to login to AWS and obtain the required temporary tokens (typically valid for 12 hours).
From a Kubernetes perspective, we therefore need a solution that will facilitate the login process and provide resources with teh required tokens when needed. Typically, this is done through Kubernetes Secrets
Technical Solution¶
Through a lot of searching and experimentation, I adopted a solution by Odania-IT that hosts a project on GitHub called aws-kubectl.
I have adopted that approach, and slightly modified it to suite my needs. I have created a repository (cloned from the original Odania-IT repository), which has some more detailed instruction in the README. Be sure to check out my version of the aws-kubectl repository.
When deployed, it will create a scheduled job running the necessary commands to login to ECR and then distribute the tokens in a Kubernetes Secret to namespaces you specify. Any deployments in those namespaces can then reference images in ECR and be able to download them. However, you also need to tell Kubernetes in the deployment manifest which secret to use:
apiVersion: apps/v1
kind: Deployment
metadata:
name: whatever-you-want
namespace: fancy-namespace-name
spec:
selector:
matchLabels:
app: your-app
replicas: 1
template:
metadata:
labels:
app: your-app
spec:
imagePullSecrets:
- name: aws-registry # <--- Point to the Kubernetes Secret that holds the tokens for AWS ECR
# The rest of the manifest is omitted...
Conclusion¶
Even though this solution is straight forward, I spent an entire day trying to figure it out, so I hope this will help someone else solve their problem a lot quicker.
If you have any suggestions on how this can be improved, please let me know by either leaving a comment below or by opening an issue in my aws-kubectl repository.