A Storage Vulnerability Deep Dive

Kubernetes Safety is consistently evolving – retaining tempo with enhanced performance, usability and adaptability whereas additionally balancing the safety wants of a large and various set of use-cases.

Lately, the GKE Safety crew found a excessive severity vulnerability that allowed workloads to have entry to elements of the host filesystem outdoors the mounted volumes boundaries. Though the vulnerability was patched again in September we thought it might be useful to write down up a extra in-depth evaluation of the problem to share with the neighborhood.

We assessed the influence of the vulnerability as described in vulnerability administration in open-source Kubernetes and labored intently with the GKE Storage crew and the Kubernetes Safety Response Committee to discover a repair. On this publish we’ll give some background on how the subpath storage system works, an outline of the vulnerability, the steps to seek out the foundation trigger and the repair, and at last some suggestions for GKE and Anthos customers.

Kubernetes Filesystems: Intro to Quantity Subpath
The vulnerability, CVE-2021-25741, was attributable to a race situation in the course of the creation of a subpath bind mount inside a container, and allowed an attacker to realize unauthorized entry to the underlying node filesystem and its delicate information. We’ll describe how that system is meant to work, after which discuss concerning the vulnerability.

The quantity subpath function in Kubernetes permits sharing a quantity in a number of containers inside a pod. For instance, we might create a Pod with an InitContainer that creates directories with pre-populated information in a mounted filesystem quantity. These directories can then be utilized by containers in the identical Pod by mounting the identical quantity and optionally specifying a subpath subject to restrict what’s seen contained in the container.

Whereas there are some nice use instances for this function, it’s an space that has had vulnerabilities found previously. The kubelet have to be additional cautious when dealing with user-owned subpaths as a result of it operates with privileges within the host. One vulnerability that has been beforehand found concerned the creation of a malicious workload the place an InitContainer would create a symlink pointing to any location within the host. For instance, the InitContainer might mount a quantity in /mnt and create a symlink /mnt/assault contained in the container pointing to /and so on. Later within the Pod lifecycle, one other container would try and mount the identical quantity with subpath assault. Whereas making ready the volumes for the container, the kubelet would find yourself following the symlink to the host’s /and so on as an alternative of the container’s /and so on, unknowingly exposing the host filesystem to the container. A earlier repair made positive that the subpath mount location is resolved and validated to level to a location inside the bottom quantity and that it is not changeable by the person in between the time the trail was validated and when the container runtime bind mounts it. This race situation is named time of test to time of use (TOCTOU) the place the topic being validated modifications after it has been validated.

These validations and others are summarized within the following container lifecycle sequence diagram.

Quantity subpath validations earlier than the container startup

A New TOCTOU Vulnerability: CVE-2021-25741
The most recent vulnerability was found by performing a symlink assault just like the one defined above, with the distinction being that it consistently swapped the symlink with a listing in a good loop, utilizing the RENAME_EXCHANGE possibility with renameat(2). If the timing is excellent, the kubelet will see the trail as a listing and move the validation test. Then the mount utility could discover that the trail is a symlink pointing to the host and observe it, exposing the host filesystem to the container. That is visualized within the following diagram:

The expectation and the assault consequence

The GKE Safety and Storage groups labored intently to revise the repair accomplished beforehand to discover a answer. The earlier repair takes a number of steps to make sure that the listing being mounted is safely opened and validated. After the file is opened and validated, the kubelet makes use of the magic-link path below /proc/[pid]/fd listing for all subsequent operations to make sure the file stays unchanged. Nonetheless, we discovered that the entire efforts have been undone by the mount(8) linux utility which was dereferencing the procfs magic-link by default. As soon as the issue was understood, the repair concerned ensuring that the mount utility does not dereference the magic-links by utilizing the –no-canonicalize flag within the mount command.

The repair is in