As you know "normal" computations in CPU can be rather slow.
Since collecting and encrypting large directory trees (and the containing files) can be very calculation intensive it might be recommended to use not only CPU power but the graphic card/GPU as well.
Other tasks like video encoding use this trick with impressive success.
Occasionally I backup (incl. encrypt) dir trees here which lasts 1-2 hours.
So I can think of an additional parameter for 7zip which let 7zip use the additional graphic card power and speed up the process significantly.
Is this possible?
Thomas
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
GPU thread is too weak. So we could use GPU only for fast compression mode.
But if you specify fast compression in 7-zip, you can get high speed in CPU too.
So GPU can't help us now.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
GPU, more precisely, the "vector processor" gives a gain in the account time only in the case of "one stream of commands - multiple data" and subject to the sameness of all input flows. The difference at least in one forces to start the task several times that often extends the score time.
The meaning of the use of vector processing appears only while performing two independent conditions simultaneously:
1) that the number of parallel data streams on the vector processor exceeds their number on the CPU,
2) The processing time of the data block for the selected algorithm on the vector processor is shorter than on the CPU.
What happens not always.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
modern "GPU"s are an entire computer whose CPU architecture is optimized primarily for processing vectors.
as the card makers lean hard into the "compute" angle of their products.
being highly optimized for it and only being able to do it is an important distinction.
I can run multiple different games in window mode with a modern graphics card. as well multiple GPU accelerated programs. because the compute cores on a video card are not actually forced to do the same commands on many data. they can receive multiple different instructions from different sources.
it is amusing that CPUs became a little like GPUs. while GPUs became a little like CPUs. but while they both moved towards each other they still have their specialities.
On the GPU, each core individually is slow, using quantity to make up for this. (and also optimized towards vectors)
it is interesting that AMD now lets you switch the GPU from "gaming mode" to "compute mode" which is reported to double the rate at which it mines cryptocurrency.
now, whether those graphics compute cores are good at file compression is an entirely different manner.
I recently observed via linux's process explorer how a 7z max compression file is compressed.
on my 16 "cores" (8 cores with hyperthreading) CPU, compressing a 10GB max compression 7z file will keep all 16 cores at near 100% utilization for several minutes. then for a few seconds CPU use will drop to very low while 2GB are writen to the SSD. then it is back to 100% CPU utilization. repeat this process until the file is finished compressing.
Thus it is quite clear that the work of compression at max level is very much divisible into many parallel processes
And the GPUs can split up and handle many different commands to be processed.
What remains questionable is whether or not the various specialization, limitation, and optimizations of the GPU are going to handle file compression well, or not.
which... the article posted earlier by mario rossi shows is doable. here is a fixed link to that article [url]https://www.tomshardware.com/reviews/winrar-winzip-7-zip-magicrar,3436-12.html[/url]
Winzip is using openCL to implement it, which lets it use both resources in parallel. it can use 100% of your CPU and ALSO use your GPU too on the side. so even if the GPU ends up being slower than the CPU, it is still extra processing power which leads to overall faster speed. (albeit potentially wasting more electricity to do so)
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
As you know "normal" computations in CPU can be rather slow.
Since collecting and encrypting large directory trees (and the containing files) can be very calculation intensive it might be recommended to use not only CPU power but the graphic card/GPU as well.
Other tasks like video encoding use this trick with impressive success.
Occasionally I backup (incl. encrypt) dir trees here which lasts 1-2 hours.
So I can think of an additional parameter for 7zip which let 7zip use the additional graphic card power and speed up the process significantly.
Is this possible?
Thomas
GPU thread is too weak. So we could use GPU only for fast compression mode.
But if you specify fast compression in 7-zip, you can get high speed in CPU too.
So GPU can't help us now.
GPU, more precisely, the "vector processor" gives a gain in the account time only in the case of "one stream of commands - multiple data" and subject to the sameness of all input flows. The difference at least in one forces to start the task several times that often extends the score time.
The meaning of the use of vector processing appears only while performing two independent conditions simultaneously:
1) that the number of parallel data streams on the vector processor exceeds their number on the CPU,
2) The processing time of the data block for the selected algorithm on the vector processor is shorter than on the CPU.
What happens not always.
a simple vector processor is an ancient GPU
modern "GPU"s are an entire computer whose CPU architecture is optimized primarily for processing vectors.
as the card makers lean hard into the "compute" angle of their products.
being highly optimized for it and only being able to do it is an important distinction.
I can run multiple different games in window mode with a modern graphics card. as well multiple GPU accelerated programs. because the compute cores on a video card are not actually forced to do the same commands on many data. they can receive multiple different instructions from different sources.
it is amusing that CPUs became a little like GPUs. while GPUs became a little like CPUs. but while they both moved towards each other they still have their specialities.
On the GPU, each core individually is slow, using quantity to make up for this. (and also optimized towards vectors)
it is interesting that AMD now lets you switch the GPU from "gaming mode" to "compute mode" which is reported to double the rate at which it mines cryptocurrency.
now, whether those graphics compute cores are good at file compression is an entirely different manner.
I recently observed via linux's process explorer how a 7z max compression file is compressed.
on my 16 "cores" (8 cores with hyperthreading) CPU, compressing a 10GB max compression 7z file will keep all 16 cores at near 100% utilization for several minutes. then for a few seconds CPU use will drop to very low while 2GB are writen to the SSD. then it is back to 100% CPU utilization. repeat this process until the file is finished compressing.
Thus it is quite clear that the work of compression at max level is very much divisible into many parallel processes
And the GPUs can split up and handle many different commands to be processed.
What remains questionable is whether or not the various specialization, limitation, and optimizations of the GPU are going to handle file compression well, or not.
which... the article posted earlier by mario rossi shows is doable. here is a fixed link to that article
[url]https://www.tomshardware.com/reviews/winrar-winzip-7-zip-magicrar,3436-12.html[/url]
Winzip is using openCL to implement it, which lets it use both resources in parallel. it can use 100% of your CPU and ALSO use your GPU too on the side. so even if the GPU ends up being slower than the CPU, it is still extra processing power which leads to overall faster speed. (albeit potentially wasting more electricity to do so)
I recently read this article from nvidia:
https://developer.nvidia.com/blog/accelerating-lossless-gpu-compression-with-new-flexible-interfaces-in-nvidia-nvcomp/
and according to this other article https://www.tomshardware.com/reviews/winrar-winzip-7-zip-magicrar,3436-12.html
winzip already takes advantage of gpu acceleration
Last edit: mario rossi 2022-10-29