Sovereign Cloud Stack

One platform — standardized, built and operated by many.

Sovereign Cloud Stack Security Advisory Open vSwitch (CVE-2023-3966)

Felix Kronlage-Dammers February 20, 2024

The vulnerability

Multiple versions of Open vSwitch are vulnerable to crafted Geneve packets causing invalid memory accesses and potential denial of service. Triggering the vulnerability requires that Open vSwitch has flow hardware offload with Linux TC flower enabled (other_config:hw-offload=true). It is not enabled by default.

The issue is caused by insufficient validation of Geneve metadata fields in the offload path. Open vSwitch versions 2.12 and newer are affected.

The vulnerability has been assigned CVE-2023-3966.

Impact on the SCS reference implementation

The Sovereign Cloud Stack reference implementation (by OSISM) comes with and utilizes Open vSwitch. While hardware offloading is not enabled by default most environments will have it enabled.

The current state of configuration can be checked with this command:

dragon@manager:~$ docker exec -it openvswitch_vswitchd ovs-vsctl get Open_vSwitch . other_config:hw-offload
ovs-vsctl: no key "hw-offload" in Open_vSwitch record "." column other_config

If hardware offloading is enabled the output differs:

dragon@manager:~$ docker exec -it openvswitch_vswitchd ovs-vsctl get Open_vSwitch . other_config:hw-offload

This is typically enabled through openvswitch_hw_offload: "yes" in the host vars or configuration.yml.

In any case, operators of OpenStack platforms are advised to deploy fixed versions.

Mitigation and Fixes

For any version of Open vSwitch, disabling flow hardware offload will prevent the issue (setting other_config:hw-offload=false and reboot the system). We do not recommend attempting to mitigate the vulnerability this way because it may impact the overall system throughput.

By default, flow hardware offload support is not enabled.

Open vSwitch has been updated to version 3.1.4. Updated images have been built by OSISM and are available to be installed.

Set openvswitch_tag: "2023.1" in the configuration repository in environments/kolla/configuration.yml. Afterwards run osism apply -a upgrade openvswitch.

This will deploy the latest openvswitch images for OSISM 6.0.x series (SCS R5, using OpenStack Antelope), pulling openvswitch 3.1.4. This only replaces the Open vSwitch image. Open vSwitch only runs on the data and network plane. Connections may be interrupted when restarting openvswitch-vswitchd. If this is to be avoided workloads should be live migrated and the update should by done node by node.

Version history

About the author

Felix Kronlage-Dammers
Felix has been building (open source) IT Infrastructure since the late 90s - during high school he helped build and run an ISP specialized in providing UUCP over ssh. Between then and now felix has always been active in various open source development communities (from DarwinPorts, OpenDarwin to OpenBSD and nowadays the Sovereign Cloud Stack). His interests range from monitoring and observability over infrastructure-as-code to building and scaling communities and companies. A technician at heart he enjoys enabling others to do awesome stuff. He is part of the extended board of the OSBA and describes himself as an unix/open source nerd. If not working, doing OSS for fun and (non-)profit or spending time with his family, he is usually found on a road bike.