One platform — standardized, built and operated by many.
One platform — standardized, built and operated by many.
Guillaume Espanel, Pierre Libeau, Arnaud Morin and Damien Rannou (OVH) found a vulnerability in the VMDK image processing code in OpenStack Cinder (OpenStack’s block storage service), Glance (image service) and Nova (compute service), which they reported to the OpenStack security team. The vulnerability allows for crafted flat image files to reference crafted backing file paths which can result in the server returning file content with potentially sensitive data. This affects all Nova and Cinder deployments; it also affects Glance deployments if image conversion (from VMDK) is enabled.
The vulnerability has been assigned CVE-2022-47951.
While the Sovereign Cloud Stack reference implementation (by OSISM) runs these services in containers and thus protects most files on the Hypervisor from being accessed using this vulnerability, the containers do have access to sensitive data and thus has a potential attacker. To abuse this vulnerability, the attacker must be an authenticated user of the platform.
We are able to reproduce this issue on unpatched platforms using the SCS reference implementation. We are still determining whether the damage in our setup goes beyond the disclosure of sensitive information by potentially leaking credentials that allow for further privilege escalation up to getting admin access to the IaaS cloud layer and thus access to all unencrypted data of customers of the affected platform.
In any case, this vulnerability is very severe and operators of OpenStack platforms are advised to deploy fixed versions as soon as possible.
While disabling image conversion in Glance does protect Glance, there is no straight-forward mitigation to effectively protect Cinder and Nova, see below (Mitigation and Fixes).
The issue has been reported to the OpenStack Vulnerability Management Team in private. The reporters and upstream developers have worked together to address the issue with fixes and an embargo date has been set to Tuesday, 2023-01-24, 15:00 UTC. At this point in time, the patches will get merged and the OpenStack Security Advisory (OSSA-2023-002) will be published. The issue is tracked in OpenStack issue #1996188, which should be publically accessible now, after the advisory has been published.
Under the used responsible disclosure approach, the information was shared with a select group of trustable users of OpenStack, so they can prepare updates and protect their user data in time for the publication.
The reference implementation for the IaaS layer of Sovereign Cloud Stack is provided by the OSISM project – the OSISM team builds the artifacts in a completely public process. The fixes will thus be applied right after the embargo is lifted; this will result in new artifacts within a few hours; the resulting artifacts (kolla-ansible container images) will be thoroughly tested using the CI tests and then published with the tag v4.2.0. This is expected to happen during the day of Wed, 2023-01-25. Update: The release of OSISM v4.2.0 has happened in the evening (european time) of 2023-01-25.
Given the severity of the issue, the SCS and OSISM teams have agreed to temporarily build patched images for the Cinder, Nova and Glance services in private and provide them privately to our public cloud providers with an NDA that disallows to disclose any information prior to the embargo lift time.
The disablement of conversions of VMDK images does protect the Glance service; unfortunately both Cinder and Nova also support many different image formats and they don’t provide a config option to disable handling VMDKs. No simple mitigation is known to us here.
There are patches from the upstream OpenStack developers available. They work by analyzing the VMDK images and raising an exception if any options are used that allow extents that reference other files. Such images will no longer be accepted and thus can no longer be used to exfiltrate sensitive files.
We are able to confirm that the previously successful attacks are addressed by the fixes. While offending VMDK images can still be uploaded and stored in Glance, they will be rejected before they can be converted to raw in Glance or for usage by Cinder or Nova.
We have been working (under NDA) with operators using the SCS reference implementation from OSISM to ensure they are prepared for this vulnerability. We can report that the three publically known providers of SCS IaaS platforms, Betacloud (from OSISM), pluscloud open (from plusserver) and WaveStack (from Wavecon/Noris) have all deployed the privately provided fixed containers and are thus protected from the reported vulnerability. They will all move to the published v4.2.0 release in the next few days to again use the officially supported public code as opposed to the private hotfixes.
The authors would like to thank the abovementioned OVH engineers, the upstream OpenStack developers and the OpenStack Vulnerability Management Team for the responsible reporting, careful analysis, fixing, testing and professional handling of the issue and the OSISM team for providing the fixed containers.