Performance impact of disk encryption using LUKS

Preface

During some weekly meetings of SCS Team Container and also SCS Team IaaS the question emerged whether or not generally enabling full disk encryption on servers - especially related to software defined storage like Ceph - should be the default.

While in general enabling disk encryption can be considered a good thing that many people would agree with, there is a difference between enabling FDE on your personal computing devices versus servers in a datacenter environment, especially when a lot of drives need to be encrypted.

The concerns voiced were that the performance impact would be tremendous. But since no actual data was available at hand to showcase how big that impact could become, I voluntered to dig into the matter at hand and do benchmarks.

Also with the availability of self encrypting drives, the practical application of that technology had to be evaluated.

“Almost” TL;DR

Impact on spinning disks

The benchmarks show that the additional overhead for encryption on spinning disks is not that big regarding CPU load (at least compared to NVME), mainly because the drives and controllers are the limiting factor there.

So SAS HDDs (even across 24 of them) aren’t that big of a deal CPU-wise. It’s still measurable, but even for HCI nodes the impact is not so big that enabling encryption would possibly wreak havoc.

However bandwidth and IOPS losses are quite big and can lead - depending on the I/O pattern - to about 79 % performance penalty. Worst performance losses were seen with sequential write (up to 79 %) followed by sequential read (up to 53 %). Random I/O losses were worst with random writes (up to 43 %) followed by random read (up to 26 %). In some single benchmarks performance was better than without encryption, up to 5 times faster across a RAID-0 at 4 KiB sequential writes. This possibly can be attributed to optimized I/O queueing.

Recommendation: You can usually safely turn on disk encryption with HDDs, as it likely won't cause many problems regarding CPU load even within HCI. However the performance penalty in throughput and IOPS can be quite tremendous compared to non-encrypted operation.

Impact on all flash

On NVME all flash systems the load impact can be gigantic. A RAID 0 across just three LUKS encrypted NVME SSDs was enough to fully saturate the CPU on test system B.

It shows in the fio data when adding up usr_cpu and sys_cpu values.
During the benchmark runs I monitored the system additionally with iostat and htop and the following htop screenshot shows how big that impact can be.

That's an AMD EPYC 7402P with 100% CPU load across all threads/cores!

This significantly increases power consumption and for HCI nodes would be bad for other workloads running on it.

Heat dissipation could then also be challenging when whole racks suddenly dissipate way more heat than before. At least cost for cooling on top of the increased power consumption of the nodes would be really bad.

Bandwidth and IOPS losses are also quite big and can lead - depending on the I/O pattern - up to a 83 % performance penalty. An I/O depth of 32 reduces the losses significantly. Worst performance losses with encryption were measured with the RAID-0 in most cases.
Performance losses sorted by I/O type worst to best were random writes (up to 83 %), sequential write (up to 83 %), sequential read (up to 81 %) and finally random read (up to 80 %).
All these worst values were with I/O size 1 and 4 MiB across the RAID-0 without I/O depth 32. One can clearly see however that all these worst values are within 5 %, so its rather the I/O size combined with the use of LUKS and MD RAID and not the drives.

Caution: It’s difficult to compare the performance of MD RAID with Ceph OSDs!
The performance impact when using Ceph could be totally different and will probably be more oriented towards the losses of individual drives. However the CPU load issue still remains.

Regarding the CPU load: There are crypto accelerator cards available which claim around 200 Gbps AES throughput and can integrate with dmcrypt through kernel modules.

These cards weren’t available for testing, but judging from prices on the used market (I didn’t find official price listings) for such cards and their power consumption specs they could provide a solution to offload the crypto operations from the CPU at a reasonable price point.

Recommendation: Further measurements regarding power consumption and crypto accelerators could be done. Also testing with an actual Ceph cluster on all flash is necessary to get a comparison of the losses compared to linux MD RAID.
Every provider needs to evaluate that separately depending on their own configurations and hardware choices.
As of now the performance impact without accelerators is too big to recommend generally enabling LUKS on NVME.

Regarding the use of self encrypting drives:

It was possible to activate the encryption with a passphrase utilizing
sedutil under linux. Performance impact was within measurement error, so basically non-existent. That’s why in the section below I didn’t bother creating charts - the results are too similar to plain operation (as expected).

Unfortunately the tooling in its current state is cumbersome to use, also there would be a need to write scripts for integrating it into NBDE setups.
(But it would be possible without too much hacking!)

However even though it technically works and performance impact is near zero, I encountered a severe showstopper for using that technology in servers, especially in cloud environments:

After powering down the system or manually locking the drive and rebooting, the system firmware stopped during POST to ask for the passphrase for each drive.

On the Supermicro server test system I found no way to disable that in BIOS/UEFI, which makes that technology not reboot-safe and thus unusable in server environments. Even worse was that even when trying to type in the passphrase, it wasn’t accepted. So somehow the way sedutil stores the passphrase differs from what the firmware does?

Recommendation: The tooling needs to be polished and documentation significantly improved. Documentation on the tools is very confusing for the general system administrator and can easily lead to operator errors.
Also due to the current BIOS/UEFI issues the technology cannot be recommended as of yet.

If vendors would provide a way to disable asking for the passphrase of certain drives it would be possible to use SEDs in a DC environment without the need of manual operator intervention when rebooting.

This concludes the TL;DR section.

So without further ado, let’s dig into the details.

The test setups

Two different test systems were used. Since no dedicated systems could be built/procured for the tests I used two readily available machines.
The specs of these are:

Test system A: SAS HDD only

CPU

2 x Intel(R) Xeon(R) Gold 6151 CPU @ 3.00GHz

RAM

512 GB

Drives

24 x HGST HUC101818CS4200 10k RPM 1.8 TB

Test system B: NVME all flash only

CPU

1 x AMD EPYC 7402P

RAM

512 GB (16 x 32 GB Samsung RDIMM DDR4-3200 CL22-22-22 reg ECC)

Drives

3 x Samsung MZQLB3T8HALS-00007 SSD PM983 3.84TB U.2

Benchmark specifications

On both systems I performed a series of benchmarks with fio using some custom scripts.
I chose to test the following block sizes, since they can offer some insight into which I/O might cause what kind of load:

• 4 KiB
• 64 KiB
• 1 MiB
• 4 MiB (default object/”stripe” size for Ceph RBDs)
• 16 MiB
• 64 MiB

The following fio tests were performed:

• sequential read
• sequential write
• random read
• random write

All with option --direct and also repeated with --iodepth=32 option and using libaio.

Those benchmarks were performed for:

• The raw storage devices direct without encryption (e.g. one SAS disk, all NVME SSD)
• The raw storage devices with encryption
• A linux MD RAID 0 across all the devices without encryption
• A linux MD RAID 0 across all the devices with encryption on the base devices
• For the SED capable drives (only test system B) with SED turned on, although no difference was expected there due to the fact that SED capable drives usually always encrypt all their data with an internal key.
  So enabling SED only encrypts that master key, requiring decryption after locking the drive either through software or because of a power loss.
  (Simplified, details can be found in the specifications like OPAL 2.0)

Benchmarking Ceph itself was skipped, because no suitable test setup was available that offered both NVME and SAS OSDs. But since enabling Ceph encryption is using LUKS encrypted OSDs, the results should be useful nonetheless.

fio benchmark script

In case you want to run your own benchmarks, here is the script that was used to generate the data:

#!/bin/bash
LOGPATH="$1"
BENCH_DEVICE="$2"
mkdir -p $LOGPATH
IOENGINE="libaio"
DATE=`date +%s`
for RW in "write" "randwrite" "read" "randread"
do
  for BS in "4K" "64K" "1M" "4M" "16M" "64M"
  do
    (
    echo "==== $RW - $BS - DIRECT ===="
    echo 3 > /proc/sys/vm/drop_caches
    fio --rw=$RW --ioengine=${IOENGINE} --size=400G --bs=$BS --direct=1 --runtime=60 --time_based --name=bench --filename=$BENCH_DEVICE --output=$LOGPATH/$RW.${BS}-direct-`basename $BENCH_DEVICE`.$DATE.log.json --output-format=json
    sync
    echo 3 > /proc/sys/vm/drop_caches
    echo "==== $RW - $BS - DIRECT IODEPTH 32  ===="
    fio --rw=$RW --ioengine=${IOENGINE} --size=400G --bs=$BS --iodepth=32 --direct=1 --runtime=60 --time_based --name=bench --filename=$BENCH_DEVICE --output=$LOGPATH/$RW.${BS}-direct-iod32-`basename $BENCH_DEVICE`.$DATE.log.json --output-format=json
    sync
    ) | tee $LOGPATH/$RW.$BS-`basename $BENCH_DEVICE`.$DATE.log
    echo
  done
done

The script expects two parameters:

• First the path where to store the log files (the path will be created if it does not exist)
• Second argument is the device on which to run the benchmarks against

The script will produce a sequence of log files in JSON format, which can then be analyzed with tooling of your choice.

Baseline LUKS algorithm performance

On both systems I first ran cryptsetup benchmark which performs some general benchmarking of algorithm throughput. This was done to obtain the best possible LUKS configuration regarding throughput.

These were the results on test system A:

PBKDF2-sha1      1370687 iterations per second for 256-bit key
PBKDF2-sha256    1756408 iterations per second for 256-bit key
PBKDF2-sha512    1281877 iterations per second for 256-bit key
PBKDF2-ripemd160  717220 iterations per second for 256-bit key
PBKDF2-whirlpool  543867 iterations per second for 256-bit key
argon2i       5 iterations, 1048576 memory, 4 parallel threads (CPUs) for 256-bit key (requested 2000 ms time)
argon2id      5 iterations, 1048576 memory, 4 parallel threads (CPUs) for 256-bit key (requested 2000 ms time)
#     Algorithm |       Key |      Encryption |      Decryption
        aes-cbc        128b       904.3 MiB/s      2485.1 MiB/s
    serpent-cbc        128b        78.8 MiB/s       577.4 MiB/s
    twofish-cbc        128b       175.9 MiB/s       319.7 MiB/s
        aes-cbc        256b       695.8 MiB/s      2059.4 MiB/s
    serpent-cbc        256b        78.8 MiB/s       577.4 MiB/s
    twofish-cbc        256b       175.9 MiB/s       319.6 MiB/s
        aes-xts        256b      2351.5 MiB/s      2348.7 MiB/s
    serpent-xts        256b       560.1 MiB/s       571.4 MiB/s
    twofish-xts        256b       316.7 MiB/s       316.1 MiB/s
        aes-xts        512b      1983.0 MiB/s      1979.6 MiB/s
    serpent-xts        512b       560.5 MiB/s       571.3 MiB/s
    twofish-xts        512b       316.5 MiB/s       315.7 MiB/s

As you can see aes-xts 256 Bit performed the best, so when creating LUKS formatted devices I chose -c aes-xts-plain64 -s 256 as parameters, which are also the defaults in many modern distributions. But just to be on the safe side I specified them explicitly.

Now let’s have a look at the fio results for single disks and RAID-0, both encrypted and non-encrypted.

Comparing read bandwidth

Previous Next

Caution: Scales are logarithmic!

Performance losses read bandwidth

Comparing write bandwidth

Previous Next

Caution: Scales are logarithmic!

Performance losses write bandwidth

Comparing random read bandwidth

Previous Next

Caution: Scales are logarithmic!

Performance losses random read bandwidth

Comparing random write bandwidth

Previous Next

Caution: Scales are logarithmic!

Performance losses random write bandwidth

Comparing read IOPS

Previous Next

Caution: Scales are logarithmic!

Performance losses read IOPS

Comparing write IOPS

Previous Next

Caution: Scales are logarithmic!

Performance losses write IOPS

Comparing random read IOPS

Previous Next

Caution: Scales are logarithmic!

Performance losses random read IOPS

Comparing random write IOPS

Previous Next

Caution: Scales are logarithmic!

Performance losses random write IOPS

Conclusion

Although modern CPUs provide AES acceleration through AES-NI instructions, the overall performance penalty especially when using NVME drives is too big to recommend turning it on everywhere.

Self encrypting drives technically work but the tooling around them is still in its infancy and as of yet not suitable for use in a DC environment until server manufacturers incorporate the necessary options in their firmware and the tools for managing SEDs under linux significantly improve in documentation and user friendliness.

SEDs also suffer in the aspect of having to trust that the drive manufacturer implemented the cryptography correctly and didn’t add backdoors into it. Recent history has shown that this isn’t always the case as some (albeit consumer grade) SSDs didn’t encrypt data at all and even allowed access without knowing the passphrase.
(This was a problem with e.g. Microsoft’s BitLocker relying on the drive to perform the cryptography and afaik subsequently was fixed by Microsoft.)

Crypto accelerators could be a solution here, however it’d be nice to see that technology embedded into servers directly since cryptography nowadays is a must-have.

Thank you for reading this blog post!

About the author