How Vulnerabilities in Kubernetes Are Potential Attack Vectors

Organizations constantly look for ways to simplify their cloud systems, and Kubernetes is one crucial tool in their arsenal. A poll conducted last year shared that 46%Opens a new window of companies utilized Kubernetes to automate their system apps’ installation, management, and scalability. However, Kubernetes, like every other solution, has its security risks. This article discusses some of these vulnerabilities, as well as popular attack techniques and defenses that companies should be aware of.

The use of containers enables quick implementation and availability for applications and microservices. However, their proliferation has increased system complexity and management headaches. Kubernetes helps meet these challenges but comes with its own attack surface. This article describes elements of that attack surface and how to harden them. It also describes Kubernetes indicators of compromise, enabling quick detection and effective response.

What is Kubernetes?

Kubernetes is a containerOpens a new window orchestration system that works as an abstracted management layer.   It is an open-source solution that helps automate the deployment, management, and scaling of containerized applications and services.  

 

Kubernetes implementations consist of one or more clusters. Within each cluster, control services help manage and protect the containers. Each node runs one or more Kubelets containing one or more containers and a Kube-proxy, separated into a control plane and node components.

See More: How to Get Started With Kubernetes the Right Way: DevOps Experts Weigh In

Control plane components

According to Kubernetes.ioOpens a new window , the control plane components administer the cluster (e.g., scheduling) and run on any machine within the cluster.  They also provide ingress and egress across the cluster boundary.

Controller Manager

This aggregated binary controls node operation with the following:

• Node controller: It monitors nodes and responds when one goes down.
• Job controller: It runs one-off tasks by creating temporary pods.
• Endpoints controller: It creates the interfaces between nodes and services.
• Service account and token controllers: It creates service accounts and associated API tokens.

API Server

Organizations can use multiple instances to scale and load balance the cluster’s API interface: the “front end.”  

• Cloud Controller Manager:  This consists of cloud-service-specific controllers that enable cluster access to the services’ APIs. No cloud controller runs if a cluster does not connect to a cloud service.
• etcd: According to Red HatOpens a new window , etcd is a distributed, consistent key-value store used for shared configuration, service discovery, and scheduler coordination of distributed systems or clusters.

Node Components

Node components run on each node, providing the node runtime environments.  

• Kubelet: Each node is established and linked to the API server by a kubelet. It is the primary node agent. It uses a manifest provided in various ways to ensure the proper containers are running and operating as expected.  
• Kube-proxy: The proxy enables rules-enforced communication to the pods and from the pods to external resources.

All of these components require hardening.  

Kubernetes Attack Surface Vulnerabilities and Defense

Taylor Smith, writing for Palo Alto NetworksOpens a new window , states that threat actors might use one of five common attack techniques.

• Abusing vulnerable and public-facing Kubernetes components
• Scraping and abusing credentials
• Installing or spinning up crypto miners
• Lateral movement
• Fileless exploits and cleaning history

I add another item: vulnerabilities in host systems that enable Kubernetes and container compromise.

Publicly facing components and API authentication

Threat actors scan the internet looking for exposed Kubernetes components that should not be exposed unless surrounded by weak access controls, including access to the all-important API servers. Shadow Server reported in May 2022Opens a new window that it discovered over 380,000 Kubernetes API server instances are exposed daily to the internet.  

The first safeguard is not publicly exposing any components that are not absolutely needed to enable access to public services. Microsegmentation and strong traffic control help with this.

Vulnerability management is also essential. Vulnerability management procedures must include all Kubernetes components. Bill Doerrfeld describesOpens a new window eight Kubernetes vulnerability scanners organizations should consider when creating a Kubernetes vulnerability management program. 

Finally, the API server requires authentication commensurate with the associated risk for both internal and external access. Authorization practices must clearly define access based on least privilege and need-to-know. Figure 2 depicts the Kubernetes API access control process using Kubernetes admission controllers.

I track the user (human subject) through this multi-step process that enables cluster access. After the subject establishes a TLS connection, he/she uses one of the enabled authentication modules to verify identity (1). Kubernetes authentication uses external services, like active directory, for verification. A common authentication method is the use of X.509 certificates.

The access control process then moves to authorization (2), a process that can use one of several authorization modes, including

• Node:  This is a special authorization mode based on scheduled pod execution.
• ABAC: Attribute-based access control uses session characteristics to determine if access is granted and at what level. Admission modules compare user and resource characteristics to JSON-described policies to determine access levels allowed.
• RBAC: Role-based access control determines levels of access based on the user’s role in the organization.  The process does not consider session participant characteristics.
• Webhook: We can think of webhooks as a type of API, but instead of responding to requests, they initiate based on events.  An event initiates an HTTP POST message that can cause Kubernetes to query an external RESTOpens a new window (REpresentational State Transfer) service when assessing user privileges.

Admission control (3) includes software modules, a set of extensions, that modify or reject access requests to create, modify, delete, or read a cluster object. Figure 3 shows the admission control process.

The first two stages in Figure 3 show the initial TLS connection followed by the already discussed authentication and authorization steps. The first step in admission control is Mutating Admission. Admission mutation can modify the API object content based on established policies.

After the mutation step, the API schema is verified, and the request is sent to Validating Admission. If all business policies are met, admission is validated, confirmation is sent, the object is persisted, and access is granted (4). Otherwise, an error the admission controller returns an error.

Attacks against API servers exposed to internal or external resources require careful access control configuration, largely using extensions and components that ship with Kubernetes.

Scraping and abusing credentials

Smith reports that attacks against Kubernetes, like HildegardOpens a new window and Black-TOpens a new window , “…immediately begin searching for credentials, such as cloud access keys, access tokens, SSH keys, Kubernetes service tokens, etc.” Hildegard gains access by exploiting a misconfigured kubelet that allowed anonymous access. Black-T used three network scanning tools to identify and access exposed Docker daemon APIs.  

First, vulnerability management across all components, whether third-party or Kubernetes out-of-the-box, is needed to ensure unwanted access. As already shown, Kubernetes runs many components that need management during design, change management, implementation, and operations activities.

Second, organizations should limit subject access, enforcing least privilege, need-to-know, and separation of duties. Further, organizations should consider shutting down all anonymous access.

Third, container security is still needed, even if containers run within a Kubernetes environment.  This includes blocking internet access to APIs. Red Hat provides an excellent introductionOpens a new window to container security.

Installing or spinning up crypto miners

Cryptojacking consists of running crypto mining software in an infected container, providing the threat actor with financial return, and adversely affecting an organization’s container performance. 

Cryptojacking defense begins with using containers from known good sources. Developers installing infected containers into Kubernetes clusters bypass most if not all security configurations. In addition, organizations can block and monitor for infected container command and control (C2) traffic, enabling response and killing the threat actor’s ability to use the malicious container payload.  

Sana Yousuf, writing for CiscoOpens a new window , describes one way to block unwanted C2 traffic.

Lateral movement

Threat actors usually attempt to pivot, using the initially compromised system to infect or access other systems, affecting lateral movement, and reaching out to other nodes and clusters.

Organizations must block lateral movement not needed for business operation.  Microsegmentation is a popular safeguard today, ensuring only explicitly allowed traffic can reach a resource, enforcing zero-trust, and closely controlling service-to-service communication.

Kubernetes configuration can enable lateral movement control, using predefined service communication, blocking weakly configured/unneeded SSH and RDP, and locking down Kubernetes and Docker APIs.

Fileless exploits and cleaning history

Threat actors use obfuscation techniques, gathering data and accessing systems with a very low probability of detection: unless obfuscation is on the organization’s look-for monitoring list.

Running memory-only malware is one of two basic obfuscation techniques. Malware not written to disk is often able to run without detection.  Organizations need solutions that can detect malicious processes designed for containers and Kubernetes operations.   One example is TigeraOpens a new window .

Another everyday threat actor obfuscation activity is log purging. Competent security teams ensure logging is enabled to help detect unwanted behavior and reconstruct attacks.  Threat actors purge their activity from the logs, making it difficult to detect their activities and extending their dwell time.  According to The CyberWireOpens a new window , dwell time is “The amount of time an attacker spends within the systems under attack, especially the amount of time the attacker spends undetected.”

Teams should make logs immutableOpens a new window , strictly limiting read access, blocking delete/modify operations, and only allowing expected writes.

Host system vulnerabilities

Kubernetes clusters run on host systems, such as Linux, where vulnerabilities are often discovered. Threat actors can often use these weaknesses to compromise clusters. One example is the Linux kernel vulnerability CVE-2022-0185Opens a new window . Leveraging this vulnerability allows a threat actor to escape containers in Kubernetes, giving him access to the host system. Bill Toulas writesOpens a new window that threat actors use these container breakouts for deeper infiltration and lateral movement.

In addition to continuous monitoring of cluster operation and micro-segmentation, the primary safeguard for this is ensuring that emerging threats and vulnerabilities are quickly identified.  Security then determines the priorities for patching or implementing compensating controls.

See More: Single Cluster vs. Multiple Clusters: How Many Should You Have in a Kubernetes Deployment?

Kubernetes Indicators of Compromise (IoC)

Thus far, we have looked at ways to prevent attacks or monitor specific activities. However, no set of safeguards is entirely effective, given the continuous emergence of new vulnerabilities and attack vectors. Consequently, we need to understand and monitor for common Kubernetes IoC.

Daniella Pontes providesOpens a new window a top 10 list of Kubernetes IoC and how to detect them. They are linked mainly to the attack paths and vulnerabilities discussed earlier. At a high level, the IoC include (two IoC are included in item two)

1. Container memory/CPU spike: Resource hijacking is apparent when container memory and CPU usage increase substantially over operational baselines.
2. Anomalous inbound/outbound network traffic: C2 and other malicious traffic produce abnormal traffic behavior, detectable with monitoring and understanding baseline traffic activity.
3. Attachment to a cluster-admin role: Threat actors often begin an attack with privilege escalation, attaching to a cluster-admin role, performing attack tasks with elevated access. Security can configure alerts for the implementation of privileged containers.  Topology maps can also help identify potential privilege escalation points for monitoring.
4. Abnormal Kubernetes user activity: Security safeguards increasingly include user and entity behavior analytics (UEBA), detecting and responding to unusual access attempts and other unwanted behavior. Compromised clusters can also show unusual user and container activity.  UEBA is closely linked to the need for monitoring described in number three above.
5. Unexpected changes in file system or directory: Audit logs and file change monitoring help detect changes made during an attack. File integrity monitoring (FIM) helps detect unwanted changes.  Stephen Cooper describesOpens a new window FIM in more detail and provides a list of what he considers the seven best FIM tools for 2022.
6. DNS request anomalies or significant spikes in DNS requests from a specific host: Monitoring should include Kubernetes coredns metrics, including the number of coredns requests.
7. Unusual HTTP response sizes: Monitoring for larger than usual HTTP responses can detect threat-related data exfiltration activities.
8. Unknown binary processes spawned: Each container runs a specific set of processes. Any processes detected that were not part of the container build are likely malicious.
9. HTTP 403 and 404 error code spikes: Intrusion attempts often result in an unusually high number of consecutive 403 and 404 errors, detectable by monitoring HTTP metrics.

Final thoughts

Organizations must always seek to reduce complexity, and Kubernetes helps with this. But like with any solution, it comes with its own attack surface. Managing the Kubernetes attack surface requires documented procedures that address all of the issues in this article, possibly varying their strength based on the associated risk.

Security teams do not have to start from scratch, given online guidelines and practices. For example, Sysdig provides a downloadable Kubernetes security guideOpens a new window , providing a good template for creating your own Kubernetes/container security policies, standards, guidelines, and procedures.

Do you think your organization’s Kubernetes environment is safe from cyber threats? Let us know on LinkedInOpens a new window , TwitterOpens a new window , or FacebookOpens a new window . We’d love to hear from you! 

MORE ON KUBERNETES

• Serious ‘cr8escape’ Vulnerability Discovered in CRI-O for Kubernetes, Patch Now!
• Top Five Free Cloud Platforms to Learn Kubernetes Online
• The Promise of Cloud Native: Does it Rely on Deploying Cloud Solutions Into Kubernetes?
• Why Kubernetes Is Vital for Moving Cloud Native Technologies To the Edge