High CPU load while performing disk R/W

[Danilo]

Hi.
I've noticed that while reading or writing in a encrypted volume (AES-SHA256) the CPU usage usually spikes up to 40%, which is a major issue to my usage. Tested on both 1.17 and 1.21 VeraCrypt versions in Windows XP and Windows 7. I have also benchmarked these operations with no encryption at all and an encrypted volume with DiskCryptor (which is a TC derivative), and the CPU usage never went over 3%.
It had been tested in a virtual machine, a machine with a 45nm Core 2 Duo and 2gb of RAM and a machine with a 3rd gen i5 with 4gb of RAM.
Is it a known issue? If so, I would like to know what exactly causes this.
Thank you.

[Kurt Fitzner]

I'm not sure what you mean by benchmarking with no encryption - I assume you mean looking at the CPU when doing raw disk transfers without encryption. A good indication of what CPU usage you'll see in VeraCrypt is to use the VeraCrypt benchmark. Set it to a buffer size of 500MB and let it run. Whatever maximum throuput it tells you for AES can be used to determine what VeraCrypt's average CPU usage will be during sustained read/write:

CPUUsage% = (DriveSpeed ÷ EncryptionSpeed) × 100

The CPU usage you are reporting isn't unusual per se, though it does sound like VeraCrypt isn't using hardware accelerated AES. Not all Core 2 Duos have hardware AES acceleration. What is the exact model number of yours? The only 45nm Core 2 Duo I know is Wolfdale, architecture and Wolfdale doesn't have AES in hardware. (EDIT: I forgot the laptop Core 2 Duo which for the 45nm chip was Penryn, but it didn't have hardware AES either). Third gen i5 should be at least Ivy Bridge, and as far as I know they all do have AES in hardware. I'll need to know an exact CPU model number to say for sure, though - there are a lot of them and I didn't do an exhaustive search.

If your CPU doesn't have hardware AES, then the CPU usage you are quoting is par for the course, especially if the drive is fairly fast. I don't have a good explanation as to why the CPU usage would be higher in VeraCrypt than DiskCryptor. It could be it's just being reported more honestly in VeraCrypt. You will see accurage CPU usage in the VeraCrypt process in task manager. I don't know how DiskCryptor's architecture is - it's not impossible that if the CPU cycles are all inside the filter driver itself that Task Manager isn't seeing them. They may get reported under "system interrupts".

I'd be interested to see what VeraCrypt's benchmarks are on your systems. If DiskCryptor has a similar feature, then that would be useful information too. Along with your exact CPU models.

[Danilo]

I'm not sure what you mean by benchmarking with no encryption - I assume you mean looking at the CPU when doing raw disk transfers without encryption.

Yes, that's it.

A good indication of what CPU usage you'll see in VeraCrypt is to use the VeraCrypt benchmark. Set it to a buffer size of 500MB and let it run. Whatever maximum throuput it tells you for AES can be used to determine what VeraCrypt's average CPU usage will be during sustained read/write:

CPUUsage% = (DriveSpeed ÷ EncryptionSpeed) × 100

The drive speed is ~120MB/s and the encryption speed is 1.3GB/s. Doing the calculation, I got 9,23% CPU usage. This value is very close to what Xperf reports for Veracrypt's encryption thread. The rest of the CPU usage is reported as KeSaveFloatingPointState calls.

The CPU usage you are reporting isn't unusual per se, though it does sound like VeraCrypt isn't using hardware accelerated AES. Not all Core 2 Duos have hardware AES acceleration. What is the exact model number of yours? The only 45nm Core 2 Duo I know is Wolfdale, architecture and Wolfdale doesn't have AES in hardware. (EDIT: I forgot the laptop Core 2 Duo which for the 45nm chip was Penryn, but it didn't have hardware AES either). Third gen i5 should be at least Ivy Bridge, and as far as I know they all do have AES in hardware. I'll need to know an exact CPU model number to say for sure, though - there are a lot of them and I didn't do an exhaustive search.

The CPU model is a Core i5-3470 3.2 GHz. Veracrypt says it has crypto acceleration and Xperf confirms the usage of the hardware accelerated functions.

If your CPU doesn't have hardware AES, then the CPU usage you are quoting is par for the course, especially if the drive is fairly fast. I don't have a good explanation as to why the CPU usage would be higher in VeraCrypt than DiskCryptor. It could be it's just being reported more honestly in VeraCrypt. You will see accurage CPU usage in the VeraCrypt process in task manager. I don't know how DiskCryptor's architecture is - it's not impossible that if the CPU cycles are all inside the filter driver itself that Task Manager isn't seeing them. They may get reported under "system interrupts".

I suspected that DiskCryptor could be "hiding" the CPU usage. I had set up a benchmark to perform a disk benchmark (hd-tune-pro) while performing a 100% CPU usage benchmark (pifast). The CPU benchmark for VeraCrypt shows a result 20% slower compared to DiskCryptor. Anyway, I will try to measure the DiskCryptor CPU usage via Xperf.

I'd be interested to see what VeraCrypt's benchmarks are on your systems. If DiskCryptor has a similar feature, then that would be useful information too. Along with your exact CPU models.

The DiskCryptor AES benchmark is usualy 2x faster than Veracrypt.

[Kurt Fitzner]

I hadn't done much with DiskCryptor before. I just took a closer look at it, looks like they hand tuned all their encryption routines in assembler. Nice work. I hope it's still under development. As a side note, I wonder how they ended up with GPLv3. Did they recode from scratch? It's too bad the VeraCrypt license isn't compatible, the two projects could do with some cross-polination.

Back to topic. Those KeSaveFloatingPointState calls, are those in a different thread in the VeraCrypt process? What version of Windows are you running, and I'm curious if you have applied the Spectre/Meltdown patches?

My working theory at the moment has to do with saving the SSE registister states (these registers are used for AES-NI instructions) and a possible requirement for this to be serialized through all threads behind a mutex in some version of Windows.

[Alex]

IMHO.
DC vs VC main difference - DC encrypts volumes only (no file containers)
from this it follows => DC simplifies I/O (DC handles R/W IRPs in place). VC creates new IRP and new queue.
Probably VC can be improved via data "fastpath" technology (e.g. simplified data path for system encryption and partition encryption)
PS. Agree DC encryption procedures are better optimized also.

[Danilo]

Back to topic. Those KeSaveFloatingPointState calls, are those in a different thread in the VeraCrypt process?

These calls are inside each encryption thread from the device driver, called at DecipherBlocks() routine. I've done a test by just disabling those calls and the CPU usage has been reduced to under 15% in the Core i5. Not the best solution, but I was expecting to break something, which it didn't.

DC does have a similar procedure, but with this KeGetCurrentIrql() check.

long save_fpu_state(unsigned char state[32]) {
    if (KeGetCurrentIrql() > DISPATCH_LEVEL) return STATUS_UNSUCCESSFUL;
    return KeSaveFloatingPointState((PKFLOATING_SAVE)state);
}

Another difference is that the state saving occurs before the XTS call in DC and in VC occurs inside it, which is in a loop. So, I've moved it to the same point as DC and the results have been close to not calling it at all.

What version of Windows are you running, and I'm curious if you have applied the Spectre/Meltdown patches?

The tests have been done in a fresh Windows 7 (32 bit) install, no patching done. It's curious that in 64 bit systems this state saving procedure doesn't happen.

    if (cipher == AES && (blockCount & (32 - 1)) == 0 && IsAesHwCpuSupported()
#if defined (TC_WINDOWS_DRIVER) && !defined (_WIN64)
        && NT_SUCCESS (KeSaveFloatingPointState (&floatingPointState))
#endif