One platform — standardized, built and operated by many.
One platform — standardized, built and operated by many.
OpenStack supports users to upload custom VM images in a number of formats to be used for creating Virtual Machines. The simplest is a raw copy of a harddisk, while other formats such as QCOW2 and VMDK are more advanced in being compressed and allowing for embedding additional information. They also support referencing other files. The latter feature needs to be controlled by OpenStack to avoid exposing sensitive files. OpenStack thus disallows such external references for customer images. Or at least, it meant to.
Martin Kaesberger has reported that the protection for QCOW2 was incomplete and allowed for an external datafile to slip through.
By creating a QCOW2 file with this feature and registering it with the OpenStack image service, files from the host system within the Cinder, Nova and Glance containers can be exposed. This way, sensitive information can be exfiltrated (Information Disclosure).
The vulnerability has been assigned CVE-2024-32498.
This issue is similar in its effects to a VMDK specific vulnerability CVE-2022-47951 that was addressed 18 months ago.
The Sovereign Cloud Stack reference implementation (by OSISM) runs these services in containers and thus most files from the hypervisor’s file system are not directly accessible from inside the container via this vulnerability. However, this depends on the service inside the container. Some services have access to data from the hypervisor or they contain sensitive data themselves (e.g. credentials to connect to other services), and thus they are accessible by a potential attacker. They have also access to log files from the services which may expose sensitive data.
To abuse this vulnerability, the attacker must be an authenticated user of the platform.
We are able to reproduce this issue on unpatched platforms using the SCS reference implementation. The exploit consists in registering an offending QCOW2 file and then creating a volume from it which will leak information from a referenced file in the Cinder container on the host system.
Information that can be accessed this way are keys for database access or for talking to the storage system. While neither database nor storage are reachable by cloud users, the exposure of these secrets removes these protections for defense-in-depth. A single further exploit would now immediately lead to a massive privilege escalation. We have not found a way to use the exposed data to directly gain privileges (such as admin access to the IaaS cloud layer), but we can not exclude that highly skilled attackers can.
In any case, this vulnerability is very severe and operators of OpenStack platforms are advised to deploy fixed versions as soon as possible.
The issue has been reported to the OpenStack Vulnerability Management Team in private. The reporters and upstream developers have worked together to address the issue with fixes and an embargo date has been set to Tuesday, 2024-07-02, 15:00 UTC (17:00 CEST). At this point in time, the patches will get merged and the OpenStack Security Advisory (OSSA-2024-001) will be published. The issue is tracked in OpenStack issue #2059809, which should become publically accessible after the lift of the embargo and the publication of this advisory.
Under the used responsible disclosure approach, the information was shared with a select group of trustable users of OpenStack, so they can prepare updates and protect their user data in time for the publication.
The reference implementation for the IaaS layer of Sovereign Cloud Stack is provided by the OSISM project – the OSISM team normally builds the artifacts in a completely public process. To avoid waiting for a few hours to have all image builds (nova-compute, glance-api and cinder-volume containers) completed and tested after the disclosure date, the OSISM project has built private images beforehand which we have tested to protect against all known scenarios in which these image files are used. These images were handed to our SCS operators a few hours ahead of time (without a detailed description) to allow them to protect their infrastructure.
A temporary workaround would be to disallow QCOW2 image registrations. Note however that images can also be created from volumes, not just via customer upload and volume snapshots with a QCOW2 signature might be misdetected as QCOW2 images prior to the fixes.
There are patches from the upstream OpenStack developers available. They work by analyzing the QCOW2 images and refusing their usage if datafiles are being used. Such images will no longer be accepted and thus can no longer be used to exfiltrate sensitive files. The mitigation does not apply upon upload of such images (the Glance service just stores the image file without looking at it unless conversion is enabled), but upon usage. The mitigation thus applies to Glance, Cinder and Nova and protects against the conversion of offending images (Glance) and their use when creating VMs (Nova) or volumes (Cinder).
We are able to confirm that the previously successful attacks are addressed by the fixes. QCOW2 files with datafile references are no longer converted by the image service (Glance) nor will existing images with such properties be used by the compute service (Nova) nor the Block Storage Service (Cinder).
Instructions to install the fixed containers (hotfix on top of OSISM 7.0.x):
# Override images in your configuration repository:
cat >environments/kolla/images.yml <<EOT
---
my_registry_kolla: osism.harbor.regio.digital
my_namespace: osism-20240702
cinder_tag: build-20240702-stable-20232
cinder_volume_image: "{{ my_registry_kolla }}/{{ my_namespace }}/cinder-volume"
cinder_scheduler_image: "{{ my_registry_kolla }}/{{ my_namespace }}/cinder-scheduler"
cinder_backup_image: "{{ my_registry_kolla }}/{{ my_namespace }}/cinder-backup"
cinder_api_image: "{{ my_registry_kolla }}/{{ my_namespace }}/cinder-api"
glance_api_tag: build-20240702-stable-20232
glance_api_image: "{{ my_registry_kolla }}/{{ my_namespace }}/glance-api"
nova_tag: build-20240702-stable-20232
nova_libvirt_tag: build-20240702-stable-20232
nova_super_conductor_image: "{{ my_registry_kolla }}/{{ my_namespace }}/nova-super-conductor"
nova_libvirt_image: "{{ my_registry_kolla }}/{{ my_namespace }}/nova-libvirt"
nova_ssh_image: "{{ my_registry_kolla }}/{{ my_namespace }}/nova-ssh"
nova_novncproxy_image: "{{ my_registry_kolla }}/{{ my_namespace }}/nova-novncproxy"
nova_spicehtml5proxy_image: "{{ my_registry_kolla }}/{{ my_namespace }}/nova-spicehtml5proxy"
nova_serialproxy_image: "{{ my_registry_kolla }}/{{ my_namespace }}/nova-serialproxy"
nova_conductor_image: "{{ my_registry_kolla }}/{{ my_namespace }}/nova-conductor"
nova_compute_image: "{{ my_registry_kolla }}/{{ my_namespace }}/nova-compute"
nova_compute_ironic_image: "{{ my_registry_kolla }}/{{ my_namespace }}/nova-compute-ironic"
nova_scheduler_image: "{{ my_registry_kolla }}/{{ my_namespace }}/nova-scheduler"
nova_api_image: "{{ my_registry_kolla }}/{{ my_namespace }}/nova-api"
EOT
# Pull new containers, not used yet
for cont in glance cinder nova; do
osism apply -a pull $cont
done
# Deploy and restart containers, the fixes will become effective now
for cont in glance cinder nova; do
osism apply -a refresh-containers $cont
done
If you are not on R6 (OSISM 7.0.x), replace stable-20232 with the version you are on
(e.g. stable-20231 if you are on R5 (OSISM 6.x).
This will cause new images to be downloaded and started on the hosts. The deployment will have the following user-visible effects:
Most users won’t notice this. Please note that running workloads will not be affected at all.
Soon after the disclosure date, OSISM will publish an official 7.0.6 release containing the updated containers (nova-compute, glance-api, cinder-volume) as well as a fix for octavia-api. Please refer to the the OSISM release notes. You can upgrade from this hotfix to 7.0.6 using the regular upgrade mechanism.
Providers that are unable to patch this issue on the day of the release need to consider the possibility of having been attacked. In this case, passwords/keys for internal communication should be considered to be leaked and will need to be rotated to reestablish the defense-in-depth lines. We recommend providers check for suspicious image operations (QCOW2 image registrations, snapshots with QCOW2 signature) in their logs for the time after the embargo lift. In particular, every image that has been uploaded should be investigated for the datafile property; it should be assumed that images that have been registered and removed again contain the exploit.
Our publically known providers of SCS IaaS platforms have all been notified and started to deploy the updated images which we were able to provide them privately a few hours ahead of time (without sharing all the details on the issue). We expect them to have completed the rollout at the time this gets public. Please refer to their communication for precise information on the patch status.
The authors would like to thank Martin Kaesberger, the OpenStack developers and reviewers, testers and the OpenStack Vulnerability Management Team for the responsible reporting, careful analysis, fixing, testing and professional handling of the issue and the OSISM team for providing the fixed containers.