Sovereign Cloud Stack

One platform — standardized, built and operated by many.

Sovereign Cloud Stack Security Advisory VMDK image processing (CVE-2022-47951)

Christian Berendt, Kurt Garloff, Felix Kronlage-Dammers January 24, 2023

The vulnerability

Guillaume Espanel, Pierre Libeau, Arnaud Morin and Damien Rannou (OVH) found a vulnerability in the VMDK image processing code in OpenStack Cinder (OpenStack’s block storage service), Glance (image service) and Nova (compute service), which they reported to the OpenStack security team. The vulnerability allows for crafted flat image files to reference crafted backing file paths which can result in the server returning file content with potentially sensitive data. This affects all Nova and Cinder deployments; it also affects Glance deployments if image conversion (from VMDK) is enabled.

The vulnerability has been assigned CVE-2022-47951.

Impact on the SCS reference implementation

While the Sovereign Cloud Stack reference implementation (by OSISM) runs these services in containers and thus protects most files on the Hypervisor from being accessed using this vulnerability, the containers do have access to sensitive data and thus has a potential attacker. To abuse this vulnerability, the attacker must be an authenticated user of the platform.

We are able to reproduce this issue on unpatched platforms using the SCS reference implementation. We are still determining whether the damage in our setup goes beyond the disclosure of sensitive information by potentially leaking credentials that allow for further privilege escalation up to getting admin access to the IaaS cloud layer and thus access to all unencrypted data of customers of the affected platform.

In any case, this vulnerability is very severe and operators of OpenStack platforms are advised to deploy fixed versions as soon as possible.

While disabling image conversion in Glance does protect Glance, there is no straight-forward mitigation to effectively protect Cinder and Nova, see below (Mitigation and Fixes).

Embargo

The issue has been reported to the OpenStack Vulnerability Management Team in private. The reporters and upstream developers have worked together to address the issue with fixes and an embargo date has been set to Tuesday, 2023-01-24, 15:00 UTC. At this point in time, the patches will get merged and the OpenStack Security Advisory (OSSA-2023-002) will be published. The issue is tracked in OpenStack issue #1996188, which should be publically accessible now, after the advisory has been published.

Under the used responsible disclosure approach, the information was shared with a select group of trustable users of OpenStack, so they can prepare updates and protect their user data in time for the publication.

The reference implementation for the IaaS layer of Sovereign Cloud Stack is provided by the OSISM project – the OSISM team builds the artifacts in a completely public process. The fixes will thus be applied right after the embargo is lifted; this will result in new artifacts within a few hours; the resulting artifacts (kolla-ansible container images) will be thoroughly tested using the CI tests and then published with the tag v4.2.0. This is expected to happen during the day of Wed, 2023-01-25. Update: The release of OSISM v4.2.0 has happened in the evening (european time) of 2023-01-25.

Given the severity of the issue, the SCS and OSISM teams have agreed to temporarily build patched images for the Cinder, Nova and Glance services in private and provide them privately to our public cloud providers with an NDA that disallows to disclose any information prior to the embargo lift time.

Mitigation and Fixes

The disablement of conversions of VMDK images does protect the Glance service; unfortunately both Cinder and Nova also support many different image formats and they don’t provide a config option to disable handling VMDKs. No simple mitigation is known to us here.

There are patches from the upstream OpenStack developers available. They work by analyzing the VMDK images and raising an exception if any options are used that allow extents that reference other files. Such images will no longer be accepted and thus can no longer be used to exfiltrate sensitive files.

We are able to confirm that the previously successful attacks are addressed by the fixes. While offending VMDK images can still be uploaded and stored in Glance, they will be rejected before they can be converted to raw in Glance or for usage by Cinder or Nova.

Patch status of SCS clouds

We have been working (under NDA) with operators using the SCS reference implementation from OSISM to ensure they are prepared for this vulnerability. We can report that the three publically known providers of SCS IaaS platforms, Betacloud (from OSISM), pluscloud open (from plusserver) and WaveStack (from Wavecon/Noris) have all deployed the privately provided fixed containers and are thus protected from the reported vulnerability. They will all move to the published v4.2.0 release in the next few days to again use the officially supported public code as opposed to the private hotfixes.

Thanks

The authors would like to thank the abovementioned OVH engineers, the upstream OpenStack developers and the OpenStack Vulnerability Management Team for the responsible reporting, careful analysis, fixing, testing and professional handling of the issue and the OSISM team for providing the fixed containers.

Version history

About the authors

Kurt Garloff
CTO Sovereign Cloud Stack @ Open Source Business Alliance
While working on Physics as student and researcher in Dortmund, Wuppertal and Eindhoven, Kurt started to work with and on Linux, with first patches to the SCSI layer in the mid 90s. He has spent his post-university life in Open Source, as kernel engineer, leader of SUSE Labs (kernel, compiler, X11, security), and engineering and business leadership at SUSE. Since 2011 he has been working on Open Source cloud software, at Deutsche Telekom, as Freelancer, at T-Systems (as chief architect for the OTC) and also has been serving on the Open Infra Foundation's board. Since 2019 he has been pushing the Sovereign Cloud Stack idea which resulted in a publically funded project that he now technically leads. He still loves to occasionally write code (mostly python these days) or at least test out code from the colleagues and project. He spends his free time with his family or with running and playing table tennis.
Felix Kronlage-Dammers
Felix has been building (open source) IT Infrastructure since the late 90s - during high school he helped build and run an ISP specialized in providing UUCP over ssh. Between then and now felix has always been active in various open source development communities (from DarwinPorts, OpenDarwin to OpenBSD and nowadays the Sovereign Cloud Stack). His interests range from monitoring and observability over infrastructure-as-code to building and scaling communities and companies. A technician at heart he enjoys enabling others to do awesome stuff. He is part of the extended board of the OSBA and describes himself as an unix/open source nerd. If not working, doing OSS for fun and (non-)profit or spending time with his family, he is usually found on a road bike.