zsync-users Mailing List for zsync
Status: Beta
Brought to you by:
cph
Menu
▾
▴
zsync-users — All discussion of zsync
You can subscribe to this list here.
| 2004 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
(2) |
Nov
(2) |
Dec
|
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2006 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
(2) |
Sep
|
Oct
|
Nov
|
Dec
|
| 2007 |
Jan
(1) |
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
| 2008 |
Jan
|
Feb
|
Mar
(3) |
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
| 2009 |
Jan
(1) |
Feb
|
Mar
|
Apr
|
May
|
Jun
(1) |
Jul
|
Aug
|
Sep
|
Oct
(2) |
Nov
|
Dec
|
| 2010 |
Jan
|
Feb
|
Mar
(11) |
Apr
(4) |
May
|
Jun
(3) |
Jul
|
Aug
|
Sep
(7) |
Oct
(3) |
Nov
(7) |
Dec
(4) |
| 2011 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
(15) |
Sep
(2) |
Oct
|
Nov
(1) |
Dec
|
| 2012 |
Jan
(1) |
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
(1) |
Sep
|
Oct
|
Nov
|
Dec
|
| 2014 |
Jan
|
Feb
(1) |
Mar
|
Apr
(2) |
May
(3) |
Jun
(1) |
Jul
|
Aug
(1) |
Sep
|
Oct
|
Nov
|
Dec
|
| 2016 |
Jan
|
Feb
|
Mar
(1) |
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|
From: Karl-Philipp R. <ri...@ri...> - 2016-03-24 01:34:25
|
Reading input files specified with `-i` command line option only uses 1 CPU core/thread. It'd be great if it'd use multiple cores. experienced with v0.6.2 |
|
From: Shriramana S. <sa...@gm...> - 2014-08-24 03:03:01
|
Hello. I wonder if zsync would be useful to help me take quick backups of my VirtualBox images without transferring the whole file? Would it be simply sufficiently to do zsyncmake at the source and zsync it to the target? I'd appreciate any pointers as to how to go about this. Thank you! -- Shriramana Sharma ஶ்ரீரமணஶர்மா श्रीरमणशर्मा |
|
From: Karl-Philipp R. <kri...@po...> - 2014-06-16 12:11:21
|
Hi together, a little enhancement request: currently zsync (v0.6.2) doesn't evaluate the invokation with -h, --help and --version (-V works), instead it display an "invalid option" error. It would be nice if it would :) All the best, Kalle Richter |
|
From: Phillip S. <ps...@ub...> - 2014-05-20 13:41:29
|
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 5/20/2014 9:09 AM, René Fichter wrote: > Resending patch, this time as an attachment (didn't realize that > attachments were allowed on this list). FYI, if you are going to send patches as attachments you need to fix your mail client to set the mime headers correctly so the patch can be viewed. The Content-Type: should be text/x-diff, or some other text/*, and the Disposition: needs to be inline rather than attachment. It is much easier to just use git-send-email to send the patch instead. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (MingW32) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJTe1fpAAoJEI5FoCIzSKrwGOgH/2/wF+k5h4B4vuPVndjuU67D 68asa/UXC5sREuvQWu3wJmbE2YTZhf2/FTYp2WU0Y+y+DEJ4iUIZG6GyBnXCmAiY pJ+fyQPCsW8sAIGEVtCRn5gQRcAUL9OoEqdPMUEwRca647IPkgIluzp3cKD9Lbd3 USHatiuA6vUlTJYxs0g+X6nqS7s/R7C1HbUNt+Ydrd0s1oqP1T7V9Nj+1hdT0w6I /d3pxdcZVaDMRvXwFyGweSB2D+GqfM44SavUkGTPLuUEZBMsVdseuq2dVdCEkbp/ AupsWlwFrUaWAsykEileLBdtbrBWUTgMCD6ySbIX+azcaVMQXyfiUCyRTo0sdVY= =0lrq -----END PGP SIGNATURE----- |
|
From: René F. <re...@fi...> - 2014-05-20 12:51:02
|
Hi, I've successfully compiled zsync for ARMv7, but downloading didn't work. After some debugging, I found that the "Range: bytes=" header contains garbage data. This is because sizeof(off_t) == 8 and SIZEOF_OFF_T == 4 (defined by configure). This confuses format_string.h to incorrectly define the 32-bit version of the OFF_T_PF macro. The following patch fixes should fix it (tested on ARMv7 and x86-64). Kind regards, Rene >From b096a8b77e627e2f56847a7bd573c0b901a0e95e Mon Sep 17 00:00:00 2001 From: rene <re...@fi...> Date: Tue, 20 May 2014 14:32:09 +0200 Subject: [PATCH] Fix off_t print macro on some platforms (notably ARMv7) when defines SIZEOF_OFF_T and __USE_FILE_OFFSET64 are conflicting. --- c/format_string.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/c/format_string.h b/c/format_string.h index 7b21773..4c327e6 100644 --- a/c/format_string.h +++ b/c/format_string.h @@ -21,7 +21,7 @@ # define SIZE_T_PF "%u" #endif -#if SIZEOF_OFF_T == 8 +#if SIZEOF_OFF_T == 8 || defined __USE_FILE_OFFSET64 # ifdef PRIu64 # define OFF_T_PF "%" PRIu64 # else -- 1.9.1 |
|
From: Ludovico C. <ca...@la...> - 2014-04-27 23:32:07
|
Hi, I was testing zsync with an 8 GB file. In one of my tests I created an input file on the client that was of the same size of the remote one, but filled with only null bytes. Running zsync on that input was taking forever. I never got to the end, but judging from the speed of the progress indicator, it would have taken days. I tracked down the problem to this: the original file I was rsyncing happened to contain a block with null bytes (weak checksum = 0x00000000) followed by another block with different content, but with a colliding weak checksum (all zeros). This was causing zsync to compute the MD4 for each offset of the 8GB input file (actually twice, but that's another issue). With a block size of 32KB, this means computing the MD4 of 8GB of data (all null bytes though) about 32k times. Changing the .zsync file to not require 2 matching blocks (Hash-Lengths: 1,...) did not help because zsync would still try to match the input block (all null bytes) with the input block with different content, but colliding weak hash. Also, when only requiring one block match, if I gave it as input the correct already-downloaded file, it would not recognize it as 100% complete (only 99.9%). I did not investgate this, thouh. I am attaching a patch that skips a full block length (instead of 1 byte) if the block to analyze has the same weak checksum of the previously analized one. I would be happy to hear if you have any concern about it. Thanks, Ludovico --- c/librcksum/rsum.c | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) |
|
From: Ludovico C. <ca...@la...> - 2014-04-24 21:58:22
|
Hi, first of all, very cool project and very nice ideas behind it, thank you! Attached is a patch that fixes a bug with compressed files greater than 2GB. In particular zoffset in getting truncated from long to int. This will work only for 64 bit. My understanding is that zsync will not work on 32 bit with compressed files larger than 2GB anyways, as it relies on zlib total_in, which is a long. Also, would you consider merging in the libcurl support from http://bazaar.launchpad.net/~mike-powerthroughwords/zsync-curl/trunk/files please? I think it would be useful, especially for HTTPS support. Thank you, Ludovico --- c/zlib/inflate.c | 2 +- c/zlib/zlib.h | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) |
|
From: Aiko B. <ai...@to...> - 2014-02-16 10:20:03
|
Hello, I wrote a simple command line webserver[1] some time ago and I always wanted to support zsync. Though, I had two issues that I solved on my end. Nevertheless, I would like to discuss the issues: 1) The zsync requests are always out-of-boundary. Example: * The file size is 10460861 bytes. * The zsync file header says: "Length: 10460861" But the initial requested size is: > bytes=0-10461183 That is beyond the file size. Is this intentionally? All webservers ignore this and return 10460861 bytes and also advertise this in their reply header. 2) zsync goes into an endless loop internally, ... ... when it requests multiple ranges and the webserver does not send _two_ empty newlines (\r\n) between the HTTP header and the body. Then, zsync does not find the "boundary" header and starts over and over again. Endlessly... I do send _two_ newlines in this special case now. nginx does it too. But why _two_? Kind regards, Aiko [1]: https://github.com/torrentkino/tumbleweed -- :wq ✉ |
|
From: Pau G. i Q. <pgq...@el...> - 2012-08-22 01:27:54
|
Hi, About a year ago I ported zsync to Windows, including changing the build system and more, but I never announced it here. Code is in the git repository here (branch "cmake"): https://www.assembla.com/spaces/zsync-windows/new_dashboard I also wrote a blog post about my approach to this port: http://www.elpauer.org/?p=972 The port is not perfect yet but it's functional and it's a fully native port with large file support (provided that you use mingw-w64 to compile it). Saying here just in case someone is interested. -- Pau Garcia i Quiles http://www.elpauer.org (Due to my workload, I may need 10 days to answer) |
|
From: Andres T. <an...@ac...> - 2012-01-14 05:38:05
|
Hi, Im having problems in order to (re)download changed blocks on updated sync files - getting this kind of errors when doing zsync http://hostname/dl/test.tar.zsync: Read test.tar. Target 96.3% complete. ******************** downloading from http://hostname/dl/test.tar.gz: ###################- 96.8% 10.2 kBps 11:10 ETA bad offset 3455458910, not in z-map Downloadable files are quite large - 5-12G. Tried with several files - no luck so far. Eg it seems to figure out delta but downloading delta fails at some point with the error above. Any suggestions what could be wrong here? Kind regards, -- ---------------------------------------------- Andres Toomsalu, an...@op... http://www.opennodecloud.com |
|
From: Justin F. <jus...@gm...> - 2011-11-14 04:21:02
|
Hi Colin, how far did you get with the libcurl support? I'd like to use zsync but need HTTPS support so I'm interested in getting it coded. If you think what you've already done may be of benefit, please send. cheers, Justin |
|
From: <hg...@we...> - 2011-09-30 20:27:46
|
Hi, due to the problem I still have with zsync and the schedule of our internal project-roadmap I had the need to change my update script to use rsync now. Maybe I can see better results with the next release of zsync - whenever it will come out. Thank you! -- HG Hi Colin, I did some further tests with the override blocksize. The result is still the same. It looks like to mee that I'm running into an endless loop. You said that you have identified two significant problems and you will fix them with the next release. Do you have any ideas about the release schedule? -- HG ------------------------------------------------------------ Von: "Colin Phipps" <cp...@mo...> Gesendet: 15.08.2011 22:35:38 An: hg...@we..., zsy...@li... Betreff: Re: [Zsync-users] Performance on large files > 7GB. To follow up with some results from my investigation over the weekend: I think that zsync 0.6.2 will perform fine on files of this size if you override the blocksize when generating the .zsync: -b 65536 should work. (rsync chooses a similarly large blocksize by default on such files.) Compressing with gzip --rsync probably works too but I did not try that. I have identified two significant problems that were harming zsync's performance which will be fixed in the next release. With these changes, zsync on my computer can sync a 24G disk image in 15m which is similar to rsync (compared to 0.6.2 taking 6 hours for the same). It was an interesting investigation :-) -- Colin Phipps <cp...@mo...[mailto:cp...@mo...]> Schon gehört? WEB.DE hat einen genialen Phishing-Filter in die Toolbar eingebaut! http://produkte.web.de/go/toolbar[https://freemailng0401.web.de/jump.htm?goto=http%3A%2F%2Fprodukte.web.de%2Fgo%2Ftoolbar] ___________________________________________________________ SMS schreiben mit WEB.DE FreeMail - einfach, schnell und kostenguenstig. Jetzt gleich testen! http://f.web.de/?mc=021192 |
|
From: <hg...@we...> - 2011-09-19 08:54:15
|
<html><head></head><body bgcolor='#FFFFFF' style='font-size:10pt;background-color:#FFFFFF;font-family:Verdana, Arial, sans-serif;'><br/>Hi Colin,<br/><br/>I did some further tests with the override blocksize. The result is still the same. It looks like to mee that I'm running into an endless loop. You said that you have identified two significant problems and you will fix them with the next release.<br/>Do you have any ideas about the release schedule?<br/><br/>-- HG<br/><br/><blockquote style="border-left: 2px solid blue; margin-left: 5px; padding-left: 5px; padding-top: 5px;"><hr/><b>Von:</b> "Colin Phipps" <cp...@mo...><br/><b>Gesendet:</b> 15.08.2011 22:35:38<br/><b>An:</b> hg...@we..., zsy...@li...<br/><b>Betreff:</b> Re: [Zsync-users] Performance on large files > 7GB.<br/><br/>To follow up with some results from my investigation over the weekend:<div> </div><div>I think that zsync 0.6.2 will perform fine on files of this size if you override the blocksize when generating the .zsync: -b 65536 should work. (rsync chooses a similarly large blocksize by default on such files.) Compressing with gzip --rsync probably works too but I did not try that.</div><div> </div><div>I have identified two significant problems that were harming zsync's performance which will be fixed in the next release. With these changes, zsync on my computer can sync a 24G disk image in 15m which is similar to rsync (compared to 0.6.2 taking 6 hours for the same). It was an interesting investigation :-)</div><div> </div><div>--<br/>Colin Phipps <<a href="mailto:cp...@mo...">cp...@mo...</a>></div></blockquote> <br><br><table cellpadding="0" cellspacing="0" border="0"><tr><td bgcolor="#000000"><img src="https://img.ui-portal.de/p.gif" width="1" height="1" border="0" alt="" /></td></tr><tr><td style="font-family:verdana; font-size:12px; line-height:17px;">Schon gehört? WEB.DE hat einen genialen Phishing-Filter in die <br>Toolbar eingebaut! <a href="http://produkte.web.de/go/toolbar"><b>http://produkte.web.de/go/toolbar</b></a></td></tr></table> </body></html> |
|
From: Colin P. <cp...@mo...> - 2011-08-15 20:35:46
|
To follow up with some results from my investigation over the weekend: I think that zsync 0.6.2 will perform fine on files of this size if you override the blocksize when generating the .zsync: -b 65536 should work. (rsync chooses a similarly large blocksize by default on such files.) Compressing with gzip --rsync probably works too but I did not try that. I have identified two significant problems that were harming zsync's performance which will be fixed in the next release. With these changes, zsync on my computer can sync a 24G disk image in 15m which is similar to rsync (compared to 0.6.2 taking 6 hours for the same). It was an interesting investigation :-) -- Colin Phipps <cp...@mo...> |
|
From: Colin P. <cp...@mo...> - 2011-08-13 23:41:32
|
On 9 August 2011 09:29, Pau Garcia i Quiles <pgq...@el...> wrote: > What is the current status of the port to libcurl? > I did start work on a port a while back. I have not worked on it this year, though. -- Colin Phipps <cp...@mo...> |
|
From: Colin P. <cp...@mo...> - 2011-08-11 19:38:33
|
On 10 August 2011 21:01, <hg...@we...> wrote: > Hm I tested again using rsync. The result is: > > time rsync -av -e "ssh -i /home/myuser/.ssh/id_dsa" > us...@se...:/var/www/localimage.vdi . > Enter passphrase for key '/home/myuser/.ssh/id_dsa': > receiving incremental file list > localimage.vdi > > sent 693606 bytes received 250053237 bytes 456318.19 bytes/sec > total size is 7516225536 speedup is 29.98 > > real 9m9.821s > user 2m4.260s > sys 0m52.623s Okay, that's a useful reference for how fast it ought to be able to run on your hardware. The speedup there also is useful as it tells me how much data there is in common between the old and new images - ~3% changed. For comparison, I just synced a 4.5GB disk image with 0.5% changed using zsync and it took ~3 minutes, so performance in line with that should be possible. Then I tried with 17GB with ~3% changed and it took 6 hours; so there is a dramatic falloff in performance with large disk images. The code was not developed with files of this size in mind and I can immediately see a few fixes that would help the performance. There is also room for further optimisation for files with lots of repetitive content. -- Colin Phipps <cp...@mo...> |
|
From: <hg...@we...> - 2011-08-10 20:01:45
|
Hm I tested again using rsync. The result is: time rsync -av -e "ssh -i /home/myuser/.ssh/id_dsa" us...@se...:/var/www/localimage.vdi . Enter passphrase for key '/home/myuser/.ssh/id_dsa': receiving incremental file list localimage.vdi sent 693606 bytes received 250053237 bytes 456318.19 bytes/sec total size is 7516225536 speedup is 29.98 real 9m9.821s user 2m4.260s sys 0m52.623s The reason I can't use rsync is that I need to do the sync over http. >Just one update: I have killed the process after 1227 minutes and still running. > >On 9 August 2011 15:12, <hg...@we...[mailto:hg...@we...]> wrote:Ok, here are some more details ;-) > >Thanks for the info. > - Thinkpad x61 Intel Core 2 Duo T7500 2,2GHz >- Ubuntu 10.04.02 >- 2.6.32-28-generic-pae i686 >- Zsync 0.6.2 (I rebulid the original zsync package to have 0.6.2) >- ext3 on all filesystems > >The file I am trying to zsync has 7516225536 KB and this is the same size >that my local copy has because it is a harddisk image where only some bits >have been changed. > >I started zsync this way: > >zsync -i localimage.vdi -o localimage.vdi.new http://server1.mydomain.com/localimage.vdi.zsync[http://server1.mydomain.com/localimage.vdi.zsync] > >The process now runs since 237 minutes using 100% of one CPU. > >I run strace and saw at the beginning for about a half hour many pwrite64 calls > >That is zsync working through the local file contents and writing to the new image any blocks that are available from the old image. > the rest of the time until now I only see repeating lines: > >read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 229376) = 229376 >read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 229376) = 229376 >read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 229376) = 229376 >read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 229376) = 229376 >read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 229376) = 229376 >write(2, "*", 1) = 1 >… > >This is zsync still reading through large amounts of empty space in the old disk image. And checksumming it all. Running the rsync algorithm on 7.5G of data is expensive. (There were some optimisations in zsync 0.6.2 to cope with this specific case of huge amounts of null data, but as you are using that now I guess you are seeing slowness in spite of these.) > with vmstat I cannot see any havy activities so I cannot understand what zsync is doing right now... > >It is probably limited by the rate at which it can checksum the data. > >Notwithstanding those comments, I am slightly surprised at the performance you are seeing and will do a test here to see if I can see similar slowness. > >> When you say that you would not be suprised if it took 1-2 hours what do you think how this can be optimized? > >There may be some scope for optimisation in the code itself, but I have already made some optimisations working on .iso files so there may not be huge gains left to be made without algorithm changes. > >A fundamental improvement in speed would come from not using the rsync algorithm at all for large block-oriented files like disk images. Large disk images are not what the rsync approach is aimed at. A simpler block-checksum approach would be faster. That said, zsync is often able to work that out for itself and avoid doing continuous rsync-style checksums where they are not appropriate, IIRC; it is too long since I looked at those optimisations to remember all the details. I will have to take another look at it. > >That aside, a simplified version of zsync or an option to zsync to skip the rsync method entirely would probably improve performance for this type of file. It wouldn't really be rsync anymore, so it's sort of a departure from zsync's aim, but it would be sufficiently close that I would be happy including the option if it was demonstrably faster on some workloads. > >One thing that you can try is increasing the block size that you give to zsyncmake. I calculated the optimum blocksizes assuming 512kbps DSL, which isn't anything like your use case. zsyncmake -b 32768 would produce a smaller hash table for the client to work with during local data analysis, and something like that may turn out to be a big help. > >> One option might be that I compress the original image, transfer it to the client via zsync and than decompress it on client side. Would this help? > >You could "gzip --rsync" the file (may need a patched version of gzip) and use zsyncmake with -Z; this should result in a much smaller image for zsync to read and to run checksums over, although you then have to decompress it yourself. That is the only compression approach that would help. >-- >Colin Phipps <cp...@mo...[mailto:cp...@mo...]> >___________________________________________________________ >Schon gehört? WEB.DE hat einen genialen Phishing-Filter in die >Toolbar eingebaut! http://produkte.web.de/go/toolbar > >------------------------------------------------------------------------------ >uberSVN's rich system and user administration capabilities and model >configuration take the hassle out of deploying and managing Subversion and >the tools developers use with it. Learn more about uberSVN and get a free >download at: http://p.sf.net/sfu/wandisco-dev2dev >_______________________________________________ >Zsync-users mailing list >Zsy...@li... >https://lists.sourceforge.net/lists/listinfo/zsync-users ___________________________________________________________ Schon gehört? WEB.DE hat einen genialen Phishing-Filter in die Toolbar eingebaut! http://produkte.web.de/go/toolbar |
|
From: <hg...@we...> - 2011-08-10 06:32:04
|
Just one update: I have killed the process after 1227 minutes and still running. On 9 August 2011 15:12, <hg...@we...[mailto:hg...@we...]> wrote:Ok, here are some more details ;-) Thanks for the info. - Thinkpad x61 Intel Core 2 Duo T7500 2,2GHz - Ubuntu 10.04.02 - 2.6.32-28-generic-pae i686 - Zsync 0.6.2 (I rebulid the original zsync package to have 0.6.2) - ext3 on all filesystems The file I am trying to zsync has 7516225536 KB and this is the same size that my local copy has because it is a harddisk image where only some bits have been changed. I started zsync this way: zsync -i localimage.vdi -o localimage.vdi.new http://server1.mydomain.com/localimage.vdi.zsync[http://server1.mydomain.com/localimage.vdi.zsync] The process now runs since 237 minutes using 100% of one CPU. I run strace and saw at the beginning for about a half hour many pwrite64 calls That is zsync working through the local file contents and writing to the new image any blocks that are available from the old image. the rest of the time until now I only see repeating lines: read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 229376) = 229376 read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 229376) = 229376 read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 229376) = 229376 read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 229376) = 229376 read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 229376) = 229376 write(2, "*", 1) = 1 … This is zsync still reading through large amounts of empty space in the old disk image. And checksumming it all. Running the rsync algorithm on 7.5G of data is expensive. (There were some optimisations in zsync 0.6.2 to cope with this specific case of huge amounts of null data, but as you are using that now I guess you are seeing slowness in spite of these.) with vmstat I cannot see any havy activities so I cannot understand what zsync is doing right now... It is probably limited by the rate at which it can checksum the data. Notwithstanding those comments, I am slightly surprised at the performance you are seeing and will do a test here to see if I can see similar slowness. > When you say that you would not be suprised if it took 1-2 hours what do you think how this can be optimized? There may be some scope for optimisation in the code itself, but I have already made some optimisations working on .iso files so there may not be huge gains left to be made without algorithm changes. A fundamental improvement in speed would come from not using the rsync algorithm at all for large block-oriented files like disk images. Large disk images are not what the rsync approach is aimed at. A simpler block-checksum approach would be faster. That said, zsync is often able to work that out for itself and avoid doing continuous rsync-style checksums where they are not appropriate, IIRC; it is too long since I looked at those optimisations to remember all the details. I will have to take another look at it. That aside, a simplified version of zsync or an option to zsync to skip the rsync method entirely would probably improve performance for this type of file. It wouldn't really be rsync anymore, so it's sort of a departure from zsync's aim, but it would be sufficiently close that I would be happy including the option if it was demonstrably faster on some workloads. One thing that you can try is increasing the block size that you give to zsyncmake. I calculated the optimum blocksizes assuming 512kbps DSL, which isn't anything like your use case. zsyncmake -b 32768 would produce a smaller hash table for the client to work with during local data analysis, and something like that may turn out to be a big help. > One option might be that I compress the original image, transfer it to the client via zsync and than decompress it on client side. Would this help? You could "gzip --rsync" the file (may need a patched version of gzip) and use zsyncmake with -Z; this should result in a much smaller image for zsync to read and to run checksums over, although you then have to decompress it yourself. That is the only compression approach that would help. -- Colin Phipps <cp...@mo...[mailto:cp...@mo...]> ___________________________________________________________ Schon gehört? WEB.DE hat einen genialen Phishing-Filter in die Toolbar eingebaut! http://produkte.web.de/go/toolbar |
|
From: Colin P. <cp...@mo...> - 2011-08-09 19:44:03
|
On 9 August 2011 15:12, <hg...@we...> wrote: > > Ok, here are some more details ;-) > Thanks for the info. > - Thinkpad x61 Intel Core 2 Duo T7500 2,2GHz > - Ubuntu 10.04.02 > - 2.6.32-28-generic-pae i686 > - Zsync 0.6.2 (I rebulid the original zsync package to have 0.6.2) > - ext3 on all filesystems > > The file I am trying to zsync has 7516225536 KB and this is the same size > that my local copy has because it is a harddisk image where only some bits > have been changed. > > I started zsync this way: > > zsync -i localimage.vdi -o localimage.vdi.new > http://server1.mydomain.com/localimage.vdi.zsync > > The process now runs since 237 minutes using 100% of one CPU. > > I run strace and saw at the beginning for about a half hour many pwrite64 > calls > That is zsync working through the local file contents and writing to the new image any blocks that are available from the old image. > the rest of the time until now I only see repeating lines: > > read(4, > "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., > 229376) = 229376 > read(4, > "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., > 229376) = 229376 > read(4, > "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., > 229376) = 229376 > read(4, > "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., > 229376) = 229376 > read(4, > "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., > 229376) = 229376 > write(2, "*", 1) = 1 > … This is zsync still reading through large amounts of empty space in the old disk image. And checksumming it all. Running the rsync algorithm on 7.5G of data is expensive. (There were some optimisations in zsync 0.6.2 to cope with this specific case of huge amounts of null data, but as you are using that now I guess you are seeing slowness in spite of these.) > with vmstat I cannot see any havy activities so I cannot understand what > zsync is doing right now... > It is probably limited by the rate at which it can checksum the data. Notwithstanding those comments, I am slightly surprised at the performance you are seeing and will do a test here to see if I can see similar slowness. > When you say that you would not be suprised if it took 1-2 hours what do you think how this can be optimized? There may be some scope for optimisation in the code itself, but I have already made some optimisations working on .iso files so there may not be huge gains left to be made without algorithm changes. A fundamental improvement in speed would come from not using the rsync algorithm at all for large block-oriented files like disk images. Large disk images are not what the rsync approach is aimed at. A simpler block-checksum approach would be faster. That said, zsync is often able to work that out for itself and avoid doing continuous rsync-style checksums where they are not appropriate, IIRC; it is too long since I looked at those optimisations to remember all the details. I will have to take another look at it. That aside, a simplified version of zsync or an option to zsync to skip the rsync method entirely would probably improve performance for this type of file. It wouldn't really be rsync anymore, so it's sort of a departure from zsync's aim, but it would be sufficiently close that I would be happy including the option if it was demonstrably faster on some workloads. One thing that you can try is increasing the block size that you give to zsyncmake. I calculated the optimum blocksizes assuming 512kbps DSL, which isn't anything like your use case. zsyncmake -b 32768 would produce a smaller hash table for the client to work with during local data analysis, and something like that may turn out to be a big help. > One option might be that I compress the original image, transfer it to the client via zsync and than decompress it on client side. Would this help? You could "gzip --rsync" the file (may need a patched version of gzip) and use zsyncmake with -Z; this should result in a much smaller image for zsync to read and to run checksums over, although you then have to decompress it yourself. That is the only compression approach that would help. -- Colin Phipps <cp...@mo...> |
|
From: <hg...@we...> - 2011-08-09 14:12:11
|
>On Tue, Aug 09, 2011 at 10:15:13AM +0200, hg...@we... wrote: >> I would expect a 7.5GB file to take some time to process (I would not be >> surprised if it took 1-2 hours), but if you are seeing the client >> continuing to use lots of CPU and giving no indication of progress for >> hours then I don't think it is just being slow. Possibly you have run into >> a previously unknown bug. Next time I do some work on zsync I will do some >> tests with files of this size and see if I can hit a similar problem. >> >> Hm, I will repeat the test with 0.6.2 again and I will try to do some more >> analysis. But however. When you say that you would not be suprised if it took >> 1-2 hours what do you think how this can be optimized? One option might be that >> I compress the original image, transfer it to the client via zsync and than >> decompress it on client side. Would this help? > >What filesystem are you using on the client? I had a user experience >an order of magnitude or more delay using zsync mirroring DVD images >(~4.5GB) on an XFS filesystem versus the same images on ext3, i.e. more >than 10 minutes of time to check the existing image on XFS versus < >1 min on ext3. Ok, here are some more details ;-) - Thinkpad x61 Intel Core 2 Duo T7500 2,2GHz - Ubuntu 10.04.02 - 2.6.32-28-generic-pae i686 - Zsync 0.6.2 (I rebulid the original zsync package to have 0.6.2) - ext3 on all filesystems The file I am trying to zsync has 7516225536 KB and this is the same size that my local copy has because it is a harddisk image where only some bits have been changed. I started zsync this way: zsync -i localimage.vdi -o localimage.vdi.new http://server1.mydomain.com/localimage.vdi.zsync The process now runs since 237 minutes using 100% of one CPU. I run strace and saw at the beginning for about a half hour many pwrite64 calls the rest of the time until now I only see repeating lines: read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 229376) = 229376 read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 229376) = 229376 read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 229376) = 229376 read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 229376) = 229376 read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 229376) = 229376 write(2, "*", 1) = 1 read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 229376) = 229376 read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 229376) = 229376 read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 229376) = 229376 read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 229376) = 229376 write(2, "*", 1) = 1 read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 229376) = 229376 read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 229376) = 229376 read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 229376) = 229376 read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 229376) = 229376 write(2, "*", 1) = 1 read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 229376) = 229376 read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 229376) = 229376 read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 229376) = 229376 read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 229376) = 229376 read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 229376) = 229376 write(2, "*", 1) = 1 read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 229376) = 229376 read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 229376) = 229376 read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 229376) = 229376 read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 229376) = 229376 write(2, "*", 1) = 1 read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 229376) = 229376 read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 229376) = 229376 read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 229376) = 229376 read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 229376) = 229376 read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 229376) = 229376 write(2, "*", 1) = 1 read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 229376) = 229376 read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 229376) = 229376 with vmstat I cannot see any havy activities so I cannot understand what zsync is doing right now... procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 1 0 0 212080 222228 3055988 0 0 67 103 180 75 11 4 83 1 1 0 0 211948 222236 3055900 0 0 0 10 842 725 50 1 49 0 1 0 0 211824 222236 3056244 0 0 118 0 848 784 50 1 49 0 1 0 0 211948 222244 3056104 0 0 0 18 853 783 50 0 49 0 1 0 0 211832 222244 3056112 0 0 0 42 842 743 51 1 49 0 1 0 0 211832 222244 3056112 0 0 0 0 842 761 51 0 49 0 1 0 0 211840 222252 3056104 0 0 0 8 839 762 50 0 49 0 1 0 0 211840 222252 3056112 0 0 0 0 836 757 50 0 49 0 2 0 0 211592 222252 3056352 0 0 60 46 840 747 50 1 49 0 1 0 0 211716 222260 3056228 0 0 0 8 842 748 50 0 50 0 1 0 0 211716 222260 3056236 0 0 0 24 834 753 50 0 50 0 1 0 0 211716 222260 3056236 0 0 0 0 821 726 50 0 50 0 3 0 0 211716 222268 3056228 0 0 0 8 813 737 50 1 49 0 1 0 0 211716 222268 3056236 0 0 0 0 850 748 50 1 49 0 1 0 0 211252 222268 3056740 0 0 120 0 868 855 50 0 50 0 ___________________________________________________________ Schon gehört? WEB.DE hat einen genialen Phishing-Filter in die Toolbar eingebaut! http://produkte.web.de/go/toolbar |
|
From: Steve B. <sbe...@ub...> - 2011-08-09 09:01:26
|
On Tue, Aug 09, 2011 at 10:15:13AM +0200, hg...@we... wrote: > I would expect a 7.5GB file to take some time to process (I would not be > surprised if it took 1-2 hours), but if you are seeing the client > continuing to use lots of CPU and giving no indication of progress for > hours then I don't think it is just being slow. Possibly you have run into > a previously unknown bug. Next time I do some work on zsync I will do some > tests with files of this size and see if I can hit a similar problem. > > Hm, I will repeat the test with 0.6.2 again and I will try to do some more > analysis. But however. When you say that you would not be suprised if it took > 1-2 hours what do you think how this can be optimized? One option might be that > I compress the original image, transfer it to the client via zsync and than > decompress it on client side. Would this help? What filesystem are you using on the client? I had a user experience an order of magnitude or more delay using zsync mirroring DVD images (~4.5GB) on an XFS filesystem versus the same images on ext3, i.e. more than 10 minutes of time to check the existing image on XFS versus < 1 min on ext3. -- Steve Beattie <sbe...@ub...> http://NxNW.org/~steve/ |
|
From: Pau G. i Q. <pgq...@el...> - 2011-08-09 08:58:58
|
Hi, What is the current status of the port to libcurl? Thank you -- Pau Garcia i Quiles http://www.elpauer.org (Due to my workload, I may need 10 days to answer) |
|
From: <hg...@we...> - 2011-08-09 08:15:21
|
<html><head></head><body bgcolor='#FFFFFF' style='font-size:12px;background-color:#FFFFFF;font-family:Verdana, Arial, sans-serif;'>On 8 August 2011 08:10, <span><<a href="mailto:hg...@we...">hg...@we...</a>></span> wrote:<br/><blockquote style="border-left: 2px solid blue; margin-left: 5px; padding-left: 5px; padding-top: 5px;"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div style="font-size: 12px; background-color: rgb(255, 255, 255); font-family: Verdana,Arial,sans-serif;">Im playing around with zsync because I am looking for a tool to minimize transfer volume for an update mechanism of virtual disk images. Background is that we plan to deploy static virtualized Windows images for our linux systems (desktop virtualization).<br/>We are currently deploying our images over http and wget. Our Image has ~ 7.5 GB. I have a newer Version of this image where I only have changed some settings within the Windows system. When I now do zsync and use my local old image I see havy CPU<br/>usage but no network activities. The pocess runs more hours and I never have seen this has been ended up.</div></blockquote><div> </div><div>Do check that you are using the latest version (zsync 0.6.2), as there was a problem with large files on 32-bit systems in some versions before that. Assuming you are on the latest version, there should not be a problem with the size of the file itself; in principle zsync is written to deal with large files, and I have tested with large files on 32-bit platforms when testing the fix in 0.6.2.</div><div> </div><div>I would expect a 7.5GB file to take some time to process (I would not be surprised if it took 1-2 hours), but if you are seeing the client continuing to use lots of CPU and giving no indication of progress for hours then I don't think it is just being slow. Possibly you have run into a previously unknown bug. Next time I do some work on zsync I will do some tests with files of this size and see if I can hit a similar problem.</div><div> </div></div></blockquote>Hm, I will repeat the test with 0.6.2 again and I will try to do some more analysis. But however. When you say that you would not be suprised if it took 1-2 hours what do you think how this can be optimized? One option might be that I compress the original image, transfer it to the client via zsync and than decompress it on client side. Would this help?<br/><br/>-- HG <br><br><table cellpadding="0" cellspacing="0" border="0"><tr><td bgcolor="#000000"><img src="https://img.ui-portal.de/p.gif" width="1" height="1" border="0" alt="" /></td></tr><tr><td style="font-family:verdana; font-size:12px; line-height:17px;">Schon gehört? WEB.DE hat einen genialen Phishing-Filter in die <br>Toolbar eingebaut! <a href="http://produkte.web.de/go/toolbar"><b>http://produkte.web.de/go/toolbar</b></a></td></tr></table> </body></html> |
|
From: Colin P. <cp...@mo...> - 2011-08-08 22:26:48
|
On 8 August 2011 08:10, <hg...@we...> wrote: > Im playing around with zsync because I am looking for a tool to minimize > transfer volume for an update mechanism of virtual disk images. Background > is that we plan to deploy static virtualized Windows images for our linux > systems (desktop virtualization). > We are currently deploying our images over http and wget. Our Image has ~ > 7.5 GB. I have a newer Version of this image where I only have changed some > settings within the Windows system. When I now do zsync and use my local old > image I see havy CPU > usage but no network activities. The pocess runs more hours and I never > have seen this has been ended up. > Do check that you are using the latest version (zsync 0.6.2), as there was a problem with large files on 32-bit systems in some versions before that. Assuming you are on the latest version, there should not be a problem with the size of the file itself; in principle zsync is written to deal with large files, and I have tested with large files on 32-bit platforms when testing the fix in 0.6.2. I would expect a 7.5GB file to take some time to process (I would not be surprised if it took 1-2 hours), but if you are seeing the client continuing to use lots of CPU and giving no indication of progress for hours then I don't think it is just being slow. Possibly you have run into a previously unknown bug. Next time I do some work on zsync I will do some tests with files of this size and see if I can hit a similar problem. -- Colin Phipps <cp...@mo...> |
|
From: <hg...@we...> - 2011-08-08 13:19:36
|
Yepp, I'm on 32bit OS with 0.6.2 > >Without looking at the code, I have a feeling this may be a case of >zsync not written with big files in mind, when on 32 bit platform. >Is your zsync 64 bit or 32 bit? >Just a guess... > > >On Mon, Aug 08, 2011 at 09:10:06AM +0200, hg...@we... wrote: >> Hi, >> >> Im playing around with zsync because I am looking for a tool to minimize >> transfer volume for an update mechanism of virtual disk images. Background is >> that we plan to deploy static virtualized Windows images for our linux systems >> (desktop virtualization). >> We are currently deploying our images over http and wget. Our Image has ~ 7.5 >> GB. I have a newer Version of this image where I only have changed some >> settings within the Windows system. When I now do zsync and use my local old >> image I see havy CPU >> usage but no network activities. The pocess runs more hours and I never have >> seen this has been ended up. >> >> Are there any suggestions for using zsync in such an usecase? >> >> Thank you! >> >> >> * >> Schon gehört? WEB.DE hat einen genialen Phishing-Filter in die >> Toolbar eingebaut! http://produkte.web.de/go/toolbar > >> ------------------------------------------------------------------------------ >> BlackBerry® DevCon Americas, Oct. 18-20, San Francisco, CA >> The must-attend event for mobile developers. Connect with experts. >> Get tools for creating Super Apps. See the latest technologies. >> Sessions, hands-on labs, demos & much more. Register early & save! >> http://p.sf.net/sfu/rim-blackberry-1 > >> _______________________________________________ >> Zsync-users mailing list >> Zsy...@li... >> https://lists.sourceforge.net/lists/listinfo/zsync-users > ___________________________________________________________ Schon gehört? WEB.DE hat einen genialen Phishing-Filter in die Toolbar eingebaut! http://produkte.web.de/go/toolbar |
2 messages has been excluded from this view by a project administrator.
