Security checks in DevOps
All about image and container scanning in a CI/CD pipeline.
Hey everyone, in my previous blog I specifically talked about how to start thinking towards DevSecOps in your CI/CD pipeline. I also listed some potential security checks which can be incorporated. Now, let us talk about image scanning in particular.
What
Image scanning refers to the process of analyzing the contents and the build process of a container image in order to detect security issues, vulnerabilities, or bad practices. It can be easily integrated into several steps of your secure DevOps workflow.
Why
So, if you are dealing with any image (whatever may be the source), you need to make sure it is free of all vulnerabilities and will not further introduce any more threats in the pipeline.
How
Very much like the previous blog, I have listed a few potential use cases which focus on image scanning. A majority of these have been provided with a suitable ‘action item’ to further smoothen your thought process.
1. Problem — On a local development machine, it might be okay to consume significant hard drive space. However, what about when deploying to testing, staging, and production? A large image will be harder to distribute.
Solution — Slim down image.
Action items :
- Run docker system prune at regular intervals.
- Avoid multi-layer and multi-stage Dockerfiles.
- Avoid package managers/binaries/configuration files.
PS : Alpine Linux should be preferred over any other Linux distro as its lightweight.
2. Problem — Sometimes, the image you scan is not the same as you deploy in your Kubernetes cluster. This can happen when you use mutable tags, like “latest” or “staging.” Such tags are constantly updated with newer versions, making it hard to know if the latest scan results are still valid. Using mutable tags can cause containers with different versions to be deployed from the same image. Beyond the security concerns from the scan results, this can cause problems that are difficult to debug.
Solution — Use immutable image tags.
Action items — Instead of using ubuntu:focal, you should enforce the use of immutable tags like ubuntu:focal-20200423 when possible.
3. Problem — Sometimes while using an image, a parent image is also required to be first built. In this way, the vulnerabilities of the parent can be introduced to the child.
Solution — Check for parent image vulnerabilities.
Action item — Identify the parent image by FROM keyword in the Dockerfile. Scan it by tools like Anchore.
4. Problem — Image should not straightaway include secrets or credentials.
Action items :
- Inject them via environment variables or mounts.
- Allow users to pass options to the Entrypoint and CMD.
- Supply secrets to Kubernetes pods as Kubernetes secrets.
- Use another secret management system.
5. Problem — The concerned image can have a lot of pre-identified vulnerabilities.
Action item — Check the whole image through vulnerability databases like CVE.
6. Problem — Images from unauthentic registries can have vulnerabilities.
Solution — Images from a set of registries should only be allowed — Known open-source group/ reputable registry.
Action item — Use export DOCKER_CONTENT_TRUST=1 in Dockerfile.
7. Problem — Dockerfiles are sometimes heavy due to unnecessary files.
Action item — Suggest the user to make .dockerignore file. This tells the Dockerfile what files can be ignored.
8. Problem — Adopters of open source technology may fall victim to code that does not follow best practices for application security.
Solution — Verify the image for possible open source vulnerabilities with tools such as Snyk.
Action item — Follow these steps:
# fetch the image to be tested so it exists locally$ docker pull node:10# scan the image with snyk$ snyk test — docker node:10 — fie=path/to/Dockerfile$ snyk monitor — docker node:10
Also, when designing a tool for image scanning, you should definitely keep these in mind:
- Will the tool go beyond just reporting vulnerabilities and alert you to the fact that there may be a newer or better base image to use?
- If a build fails due to vulnerability detections, will the tool provide developers and DevOps teams with enough information to fix the issues?
- Does the tool provide the flexibility you need to set your security gates?
Besides image scanning, I also want to point out some security checks which only make sense when the image is running. Yes, I am talking about container scanning. Here are some:
- Problem — Anyone who accesses your container running as root can start undesirable processes in it, such as injecting malicious code. And running a process in your container as root makes it possible to change the user id (UID) or group id (GID) when starting the container, which makes your application vulnerable.
Solution — Ensure limited access.
Action item — Check whether the container is in non-root mode. Create a user with a known uid in the Dockerfile and run the application process as that user. The start of a Dockerfile should follow this pattern:
FROM <base image>RUN groupadd -g 999 appuser && \useradd -r -u 999 -g appuser appuserUSER appuser… <rest of Dockerfile> …
2. Problem — Some insecure ports like the ssh port 22 on containers make room for attackers.
Action items :
- Check that ‘Expose 22’ is not in Dockerfile.
- Also, don’t map any ports below 1024 within a container as they are considered privileged because they transmit sensitive data. By default, Docker maps container ports to one that’s within the 49153–65525 range, but it allows the container to be mapped to a privileged port. As a general rule of thumb, ensure only needed ports are open on the container.
3. Problem — Each container have their own internal IP address and it could change if you start and stop the container.
Action item — If your application or microservice needs to communicate to another container, use environment variables to pass the proper hostname and port from one container to another.
So, I have tried to compile a few use cases for image and container scanning. You can suggest more, as per your understanding. If you found this article insightful or of the slightest help, give it a few claps. Feel free to reach out in case of any queries. Happy learning!
