5 tips to secure a Red Hat Quay container image registry
Containers became a huge part of our life. I don’t think that there is an organization that does not use containers anymore. It does not matter if it is a TELCO or a small startup, everyone is implementing containers in some form, whether it’s Kubernetes, Openshift, Docker Swarm, or even just a simple Docker engine that runs containers on top a virtual machine.
The container world became enormous, and some companies decided to start implementing container image registries as a part of their environment — a “Private Registry”. The developers in these organizations started pushing images into the registry, and it is not long until the registry became a crucial part of the organization’s development cycle.
One thing that I’ve noticed when inspecting private registries in multiple organizations is that only few take security factors into consideration. Nowadays major workloads are dependent on the image registry, and if it gets compromised, it could really hurt.
As mentioned above — securing the image registry is important. In this blog I’m going to go through some tips that will help your environment become more secure using Red Hat Quay private registry. I will demonstrate its benefits as a major security factor in a containerized environment.
Red Hat Quay — An image registry which is supported by Red Hat. Quay can be used either as a public registry at quay.io as an internet-connected container image registry, or as a private — on-premise container image registry. Further documentation can be found at — Red Hat Quay Documentation.
The tips rely on you to have a running Red Hat Quay infrastructure and basic understanding of the platform. Quay can be deployed as a container on docker engine, or OpenShift. A helpful guide for deploying Red Hat Quay can be found at — https://medium.com/@danarlowski11/deploy-quay-3-3-on-your-openshift-cluster-c2a1710fd71c.
1. Use OpenID Connect for authentication to the registry portal
Red Hat Quay allows using an OpenID Connect provider as its identity source. An OpenID provider is much more secure than using Quay’s local user database.
Using OpenID means that there are no local user credentials stored in Quay (which means that the registry handles less sensitive data), and that the authentication process will go through the OpenID handshake against a protected identity provider, which is considered more secure than just providing credentials in Quay’s web portal.
To understand more about OpenID connect, read my previous blog at — https://medium.com/@michaelkotelnikov/lets-talk-single-sign-on-oauth-openid-rhsso-9ba9ef1f23da.
Connecting Red Hat Quay to Red Hat Single Sign On — Demo
Red Hat Single Sign On (RHSSO) — A Red Hat platform which can act as an identity provider for OpenID Connect. The platform acts as an Authentication / Authorization server, it controls identity management and access control from a single point of trust.
RHSSO Configuration -
Navigate to RHSSO admin portal.
- Create a realm for Red Hat Quay in RHSSO -
2. Create a client for Red Hat Quay (Register Red Hat Quay in RHSSO) -
- Make sure to configure a confidential access type.
- Configure a valid redirect URI — https://<quay-uri>/oauth2/redhatsso/callback
3. Create a user that will be used for the login process -
Make sure to add a password to the user after creation.
4. Configure a valid CA (certificate authority) into RHSSO configuration file -
- Create a CA, and import it into a truststore.jks file -
# openssl req -new -newkey rsa:4096 -x509 -keyout xpaas.key -out xpaas.crt -days 365 -subj “/CN=<domain name>”# keytool -import -file xpaas.crt -alias xpaas.ca -keystore truststore.jks
- Implement the truststore into the standalone.xml (RHSSO’s configuration file) -
# cat standalone.xml…<spi name="truststore"><provider name="file" enabled="true"><properties><property name="file" value="/truststore.jks"/><property name="password" value="redhat"/><property name="hostname-verification-policy" value="WILDCARD"/><property name="disabled" value="false"/></properties></provider></spi>...
Red Hat Quay Configuration -
- Log into the Red Hat Quay configuration portal, and modify the existing cluster -
2. Add the CA used in the RHSSO configuration files to Red Hat Quay.
3. Under External Authorization (OAuth), press on Add OIDC Provider, type redhatsso in the prompt.
4. Fill the fields accordingly-
- OIDC Server: https://<sso-url>/auth/realms/<realm-name>/
- Client ID: The ID given to Quay’s client in RHSSO in previous steps.
- Client Secret: Can be found in the client’s menu, under the credentials tab in the RHSSO client portal.
- Service Name: Can be whatever you like to be presented in Quay’s login page.
- Login Scopes: Add an openid scope to indicate an openid procedure when accessing the RHSSO server for authentication.
5.Press on “Save Configuration Changes”, and rollout the deployment of Red Hat Quay
6. Now, when the user logs into Red Hat Quay, a new Login menu will appear
7. Press on “Sign in with Red Hat Single Sign On”, Red Hat Quay will redirect the browser to the RHSSO login portal
8. After providing the credentials, Red Hat Single Sign On will redirect the user to Red Hat Quay for a successful login portal -
2. Harden user access to the registry
By default, Red Hat Quay provides a very permissive access control methodology to the registry. To make it more secure we will have to enable some features and follow a zero trust methodology to reduce the attack surface of the platform.
- The registry accepts anonymous logins — which means that everyone, including non-authenticated users can access public repositories in Red Hat Quay.
Solution: To prevent anonymous logins, using the administrating config portal, uncheck the “Enable Anonymous Access” option under “Account Settings”.
- Users can be created and used with an automatic ‘Sign up’ option. From a security and management perspective, this could form many threats. A malicious user could register itself and become a problem if it compromises a weakness in the registry.
OpenID Solution:
- If you are using an external OpenID provider (e.g RHSSO) as a user source, you would probably want it to be the only authoritative user source in the platform. To disable the usage of credentials in Red Hat Quay’s portal, login into Red Hat Quay’s configuration management portal, and uncheck the “Login to User Interface via credentials” under “Access Settings”.
- Alongside the previous step, you will have to change “Authentication” type under “Internal Authentication” from “Local Database” to “External Application Token”. When enabling this option, Red Hat Quay will rely on the OpenID provider’s Access Token completely, without considering the Local Database at all. All of the users will be passwordless.
After adding these configurations, the login portal will not have an option to provide credentials besides the “Sign in with Red Hat Single Sign On” option. And a password will not be set to users locally in Red Hat Quay. Users will have no alternative besides using RHSSO for authentication.
Local Database Solution:
- If you are not using OpenID as your user source, you are still able to prevent users from signing up to Red Hat Quay by disabling the “Enable Non-Superuser User Creation” option under “Account Settings” in the Red Hat Quay configuration management portal.
3. Use private repositories
This tip might seem obvious, but creating public repositories, even if you are running Red Hat Quay in an air gaped environment, could form a serious risk for your data.
By creating private repositories you will gain control over the users that gain access to your images, even if it is just for pulling. Creating private repositories will contribute to the zero trust methodology, and allow you to create fine grained permissions over images in the organization.
A repository will become private upon creation -
4. Use ROBOTS!
To extend the control over permissions to repositories in Red Hat Quay, Robot Accounts will have to be used.
It is important not to use your own user account’s credentials for operations like “docker login”, or as credentials for a pull secret in Kubernetes. It is most likely that regular user account will have permissions over numerous repositories, it does not matter if the user has read or write permissions, if the user account gets compromised, the impact on the registry and the organization will be very painful.
Following tips 1 + 2 in this artice will prevent users from using their credentials to login into the registry using docker CLI / Pull Secrets, thereby forcing the usage of Robot Accounts to manage permissions to repositories in the registry.
Creating a Robot Account will allow creating specific fine grained permissions to specific repositories. A robot account will be assigned read / write permissions for pulling / pushing images to the repositories.
After creating a robot account, Red Hat Quay will provide methods to implement the robot account into the user’s system, in a form of a docker login command, or a Kubernetes secret.
Demo
In this demo, I have created a robot account with Read permissions on an httpd repository in Red Hat Quay. I have obtained its login credentials as stated in the screenshot above.
- When I try to pull the image without logging in the registry with the robot account I get the next error -
# docker pull m-quay-quay.apps-crc.testing/qa/httpd:v1
Trying to pull m-quay-quay.apps-crc.testing/qa/httpd:v1...
unable to retrieve auth token: invalid username/password: unauthorized: authentication required
Error: error pulling image "m-quay-quay.apps-crc.testing/qa/httpd:v1": unable to pull m-quay-quay.apps-crc.testing/qa/httpd:v1: unable to pull image: Error initializing source docker://m-quay-quay.apps-crc.testing/qa/httpd:v1: unable to retrieve auth token: invalid username/password: unauthorized: authentication required- I log into Red Hat Quay using the newly created robot account -
# docker login -u="qa+httpd_robot" -p="Y6LJR00SWJ4Z1XEDH8GOUVSQNOGPXUVUI5GV6E23QGCMUD0KIGUVNQBP8QYBOIXA" m-quay-quay.apps-crc.testing- Now, when I pull the Image from Red Hat Quay I get the next result -
# docker pull m-quay-quay.apps-crc.testing/qa/httpd:v1
Trying to pull m-quay-quay.apps-crc.testing/qa/httpd:v1...
Getting image source signatures
Copying blob e620966d553d done
Copying blob a617af155f2d done
Copying blob d9710cf8348d done
Copying blob fa921357d219 done
Copying blob 126af8eb926a done
Copying config ccbcea8a67 done
Writing manifest to image destination
Storing signatures
ccbcea8a67570043de0d0932f9d750e7d311415def699c60aa69e4cea4a25a7e- Since I gave the Robot account only read access it is only able the pull images from the registry. When I try to push a new image to the repository, I get the next result -
# docker push m-quay-quay.apps-crc.testing/qa/httpd:v2
Getting image source signatures
Copying blob ca9ad7f0ab91 done
Copying blob 484fa8d4774f done
Copying blob 5d727ac94391 done
Copying blob 13cb14c2acd3 done
Copying blob 4cfc2b1d3e90 done
Copying config ccbcea8a67 done
Writing manifest to image destination
Error: Error copying image to the remote destination: Error writing manifest: Error uploading manifest v2 to m-quay-quay.apps-crc.testing/qa/httpd: unauthorized: access to the requested resource is not authorized- I will change the permissions of the robot account to allow write permissions -
- After I re-login to refresh the robot account’s token, I try to push the image again -
# docker push m-quay-quay.apps-crc.testing/qa/httpd:v2
Getting image source signatures
Copying blob ca9ad7f0ab91 done
Copying blob 484fa8d4774f done
Copying blob 4cfc2b1d3e90 done
Copying blob 5d727ac94391 done
Copying blob 13cb14c2acd3 done
Copying config ccbcea8a67 done
Writing manifest to image destination
Storing signaturesBy using the Robot account I was able to strictly design the permissions in the repository. The created robot account will not be used for other purposes besides the defined permissions to images in the httpd repository in the Red Hat Quay registry.
5. Use image scanning
One of the main features that Red Hat Quay provides in my opinion is Image scanning as an integrated part of the registry itself, which means that every image that goes into Red Hat Quay, will also get scanned for security vulnerabilities in Quay’s security sidecar — Clair.
Clair will be implemented as a part of the development pipeline, and provide a scanning feature seamlessly to the Devops pipeline -
- The developer pushes the new version of the application into GitHub.
- GitHub triggers a CI pipeline in Jenkins. Jenkins will build and test the application. As a part of the deployment process, it will push the newly created image into Red Hat Quay.
- Red Hat Quay will perform a security scan automatically against the newly created image. Using Clair, Red Hat Quay will create a security report.
- The image will be deployed via Red Hat Quay on top of the OpenShift platform.
Implementation documentation of Clair in Red Hat Quay can be found at — https://access.redhat.com/documentation/en-us/red_hat_quay/3/html/manage_red_hat_quay/quay-security-scanner
Clair goes through the images in the registry and scans them for security vulnerabilities against an internet connected vulnerability database. I.e — for Red Hat systems, Clair scans the images against the CVE Database.
From Clair’s source code -
...// UpdaterFlagDateFormat - date format for updater flag dates ('magical reference date' for datetime format)UpdaterFlagDateFormat = "2006-01-02 15:04:05"// AffectedType - affected typeAffectedType = database.BinaryPackage// CveURL - url for cve contentCveURL = "https://access.redhat.com/security/cve/"...
Clair will categorize the security vulnerabilities into 6 different categories-
- Critical-level vulnerabilities.
- High-level vulnerabilities.
- Medium-level vulnerabilities.
- Low-level vulnerabilities.
- Negligible-level vulnerabilities.
- Unknown-level vulnerabilities.
Clair will specify the impacts of the vulnerabilities and describe whether it is fixable or not -
Clair will go through the packages installed in the image’s layers and check for vulnerabilities in the packages. It will categorize the packages into the next categories -
- packages with Critical-level vulnerabilities.
- packages with High-level vulnerabilities.
- packages with Medium-level vulnerabilities.
- packages with Low-level vulnerabilities.
- packages with Negligible-level vulnerabilities.
- packages with no vulnerabilities.
It will also specify the vulnerabilities that will be present in the package after an update.
Conclusion
In many organizations, the main workload became dependent on containers, making the container registry a major pillar that holds the development flows in the environment. The registry became very important, and securing it became a must in modern organizations.
Using Red Hat Quay makes the job of securing the registry a bit easier. Red Hat Quay supports image scanning, smart user management, robot accounts and good auditing systems.
Following the 5 basic tips stated in this article could provide some extra security to your Red Hat Quay deployment. Following a more strict methodology in term of security means that there will be less abnormalities in the environment, and anomalies will be much easier to spot.
