As threat vectors and micro-perimeters expand with the popularity of container deployments, an automated approach can help DevSecOps teams keep workflows secure.
Container security, including vulnerability management, has long been a challenge for many DevOps teams. The challenge is that, while modern cloud-native tools have enabled automation across CI/CD pipelines, automating the vulnerability management process has lagged behind.
Like most DevOps teams, we put automated vulnerability scanning into place so that scans are triggered at important points in the pipeline, all the way into production. But truly automating this process requires not just triggering scans but also interpreting scan results and notifying the appropriate person or team – whether that is a developer or an operations team. Beyond notifications, automated remediation and verification is a huge challenge.
So, this was what we wanted to accomplish in our first phase of automated vulnerability management.
Remediate Fixable VulnerabilitiesOur intention was to have a container environment running free from high (critical) CVEs with a vendor fix available. Also, we didn’t want to be required to chase anyone manually to get it done, and we wanted to develop automated processes at the build stage of our pipeline.
Here’s why we decided on the path we took.
1. Before image push – after build
2. Use NeuVector build-phase scanning to automatically trigger scans during the build process.
- Fail any builds that have fixable critical vulnerabilities, and notify the developer or team responsible
- After remediation, rescan and allow the build to be pushed into our registry.
3. Before image deployment
- Use NeuVector registry scanning to scan images when pushed in registry.
- Use NeuVector admission control policy. Deny deployment of any image with high CVEs with a vendor fix available, so these are blocked from the production environment.
- Expose the NeuVector API to integrate and check the status of images scanned to include the scans from pipeline stage (above).
4. Production scanning on all running containers and hosts
- Detect any high CVEs with a vendor fix on any running containers and generate an alert with details of image/container.
- Use the NeuVector API to get details of all running containers and to send alerts for any new critical vulnerabilities with fixes available.
5. Production ‘Virtual Patching‘
- Use NeuVector run-time protections to protect against any vulnerability exploits in a production environment.
- Alert and/or block any suspicious network, file, and process activity. These could be attempted vulnerability exploits, malware, or zero-day attacks.
Learn More: Why Full-stack Development Needs Low-code to Stay AgileOpens a new window
The following diagram shows the vulnerability management workflow in the CI/CD pipeline.
Source: The Co-operators
The following diagram shows a NeuVector admission control example.
Source: The Co-operators
Here are sample REST API calls for how we get scan data from the required image.
Step 1: We get the image IDs of all required images that are up and running.
Example:
export _token_=$(curl -s -k -H “Content-Type: application/json†-d ‘{“passwordâ€: {“usernameâ€: “xxxâ€, “passwordâ€: “xxxxâ€}}’ “https://nvapi-xxxxxx.ca:443/v1/auth†|jq -r ‘.token.token’) ;curl -s -k -H “Content-Type: application/json†-H “X-Auth-Token:$_token_ †https://nvapi-xxxx.ca:443/v1/scan/workload | jq -c ‘.workloads[]|select(.image | contains(“us.icr.ioâ€))|.id’|jq . >image-id.txt
Learn More: Containerisation, Docker, and Kubernetes: what you need to know for your app startupOpens a new window
Step 2: We then check the image ID from Step 1 to determine which have high CVEs with a vendor fix available.
Example
curl -s -k -H “Content-Type: application/json†-H “X-Auth-Token:$_token_ †https://nvapi-xxxx.ca:443/v1/scan/workload/$C |jq -c ‘.report.vulnerabilities[]|select(.severity|contains(“Highâ€))|.fixed_version’|jq . >cve.txt
Step 3: Next, if the image has a high CVE and vendor fix available, send an alert:
mail -s “High CVE with vendor fix detected on running container in production cluster xxxxx for $E†[email protected]
Here is an example output and email alert:
Source: The Co-operators
As can be seen, this automates part of our vulnerability management workflow. This example can be extended to additional criteria and notification recipients to build more sophisticated automation.
Finally, we are making progress towards our goal of full automation of as much of the vulnerability managementOpens a new window and remediation process as possible.
Let us know if you liked this article on LinkedInOpens a new window , TwitterOpens a new window , or FacebookOpens a new window . We would love to hear from you!