Question about performance
A partition and disk imaging/cloning program
Brought to you by:
steven_shiau
Menu
▾
▴
Hi all,
I use clonezilla live to backup the operating systems on my workstation
(Windows 11 and Ubuntu 20) to a NAS and I was wondering if it was possible
to reduce the backup/restore time, especially the backup time which has to
be spent more often.
To backup the disks I use the following command:
sudo /usr/sbin/ocs-sr -q2 -nogui -c -j2 -z0 -sfsck -senc -p command
savedisk
date +"%Y%m%d"-nvme0n1 nvme0n1and this gives me the same speed on both NAS and another nvme disk when
backing up the biggest partition of nvme1 (windows 11 ssd): around 11GB/min.
If I use the command:
sudo partclone.ntfs -z 104857600 -c -s /dev/nvme0n1p3 -o test.bin
I get around 20GB/min.
In both cases the process looks to be CPU bounded (single core). I also
tried variants of the first command with different parallel compressions
but the result is pretty much the same (around 10GB/min).
Since the first command also does a lot of additional stuff to save the
original content of the disk (like partition table backup, EFI stuff, and
so on which I would prefer to let clonezilla to manage), is there any
reason why ocs-sr runs the following command to back up the same partition:
partclone.ntfs -z 10485760 -L /var/log/partclone.log -c -s /dev/nvme0n1p4
--output - | cat | split -a 2 -b 1000000MB -
/home/partimag/20211114-nvme0n1/nvme0n1p4.ntfs-ptcl-img.uncomp. 2>
/tmp/split_error.shxuqB
I think:
1. "| cat |" could just be omitted when no compression is required
2. split could also be omitted when no splitting is required.
I think these two steps are slicing down the performance that partclone
could offer by around 50%. Is there any reason to keep them in the command
line even when they are explicitly not required?
I tried to look into the ocs-sr code to propose a patch but I am not a bash
expert and the script is rather long and complex so maybe someone with more
experience on this can quickly say first if it is something feasible or not.
Thanks.
Yes, it can be improved when no compression and no splitting are required. We kept them is to keep the codes simple. No need to do too many extra if/else/fi.
Normally most of the people preferring compressing the image while keeping the speed reasonable. Your use case is somehow seldom.
When compressing, the image to be transferred is smaller. With -z0 although the speed looks faster, but actually you also need more time to transfer the whole image which is larger than the compressed one. Therefore you have to find the balance.
BTW, have you tried -z8p or -z9p? What were the total time spent compared to that of -z0?
Steven
Hi Steven,
thank you for your answer and for your great work.
Here are the results of the test you asked:
The 1TB disk has 300GB occupied. The combination I usually use is number 3. I tried also other multi core compression options but the results are similar to 5.
Test case number 2 can be further improved, in my opinion, removing the useless cat process, in pipe with partclone.ntfs, generated by -z0 option. Also a clear way to avoid the split part without using -i 0 would be helpful.
Concerning the frequency of the use case, I checked with a couple of people that I know, which are using your software, and they use -z0 and no split when they need to get fast snapshot.
I understand that is not a great sample but I guess many people may get advantage from a performance improvement in this case.
Thanks.
Hi all!
thank you Paul for opening this thread! And thanks to Steven and the rest of the community for their great work!
I would definitely vote for this use case as I use Clonezilla to clone frequently many HDD and usually time is the more strict constraint as I need to free up the source HDD asap (usually I run an offline compression at the end of the process on the image files).
Thanks you both for the feedback. In that case, then we should not deprecate the option -i0.
We will make the corresponding changes in the next testing release.
Appreciate.
Steven
Please give testing Clonezilla live >= 2.8.1-5 or 20211206-* a try:
https://clonezilla.org/downloads.php
By default do not split the image file of a partition, i.e., use "-i 0" when saving an image by ocs-sr. If the image repo is FAT, VOL_LIMIT_IN_INTERACTIVE will be set as 4096 when running ocs-sr in saving mode.
Please let us know the results if you test that.
Thanks.
Steven
Hi Steven,
tested and working version 2.8.1-7: no split running with no -i argument and -i 0.
Image creation performance is better (16GB/min vs 12).
As said there is still margin removing the cat part, if you should have time :-)
Image verification also looks kind of slow, (12GB/min) not clear why.
Thanks
Paul
@Paul,
Thanks for your tests and feedback. So you meant when using non-dd mode (i.e., use -q2), and with -z0 and -i 0, the "cat" part should be removed? Or what case?
As for restoring/checking the image, actually that's another story. Some decompressing program does not deal with (follow) symbolic files. E.g., "pixz -d" has no option to process a symbolic file. This is crucial when we want to restore the image saved from different source device. The program create-ocs-tmp-img uses symbolic files for that. Hence we still have to keep the original way, i.e., using cat and pipe to read the source image file.
Steven
Last edit: Steven Shiau 2021-12-16
Hi Steven,
yes, the use case I am talking is non-dd (i.e partclone aka -q2) and no compression (-z0). There is a "|cat|" which is generated in the command line which I think is again limiting the performance. IMHO it would be a more than welcome improvement to remove it for this specific case where the user is looking for the maximum speed.
But, as you said, for doing that some more if/then has to be added to ocs-sr.
Clear the explanation for the restore/checking case.
Thanks.
Paul
@Paul,
We have made some improvements to skip using "cat" when saving an image as -z0 and -i0 options are used.
In addition, when restoring or checking an image, if the image is from saving with "-z0" and "-i0", no cat will be used.
Please give Clonezilla live >= 2.8.1-10 or 20211221-* a try:
https://clonezilla.org/downloads.php
Please let us know the results. Thanks.
Steven
Hi Steven,
the patch is working great.
I tested using the following line:
cat is not included the running processes and the reported imaging speed maintains 22GB/min, which is what I reach running partclone manually.
I also confirm an even bigger speed improvement in image verification, which now mantains 26GB/min, which is a very good value, with respect to the previous version.
Overall, the total time to backup 1TB disk with 300GB of data, in this case (no compression, no split) is now 28m29s to perform both image creation and image checking.
Here are the details:
2.7.3-19
-z 0 (cat and split): 41m50s
-z 0 -i 0 (cat only): 40m14s. I did something wrong here. In one of my previous message I reported erroneously that this time was 50% smaller than the previous one (-z 0). I probably reported just the time for image creation, not the whole time (image creation + image checking), as for the other cases .
2.8.0-27:
-z 0 -i 1000000000 (cat and split): 42m21s
-z 0 -i 0 (cat only): 40m15s
2.8.1-10
-z 0 (no cat and no split) 28m29s
So, in the end, the time required to take the sample snapshot with 2.8.1-10 is 30% less than with previous ones.
Thanks a lot.
Paul
Hi Paul,
Great! Thanks for your feedback and contributions. The performance when using -z0 + -i0 has been improved a lot. Thanks for identifying the bottleneck and performed so many tests.
Appreciate.
Steven
Last edit: Steven Shiau 2021-12-22