#26142 rsync server timeout after 330 seconds = 5m30s

[milahu]

i want to download a SVN repo over rsync

$ time rsync --recursive --info=progress2 svn.code.sf.net::p/jdownloader-svn-mirror-milahu/code/ jdownloader/; echo "return code: $?"
     22,755,798  99%   67.28kB/s    0:05:30 (xfr#1017, ir-chk=1000/2072)
rsync: connection unexpectedly closed (89489 bytes received so far) [receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(231) [receiver=3.3.0]
rsync: connection unexpectedly closed (36093 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(231) [generator=3.3.0]

real    5m32.112s
user    0m0.233s
sys     5m20.318s
return code: 12

this repo has over 1.5GB, and i get only 0.2GB (0.02GB compressed)

this seems to be a server-side timeout on sourceforge.net of 330 seconds (5m30s)

[milahu]

rsync --timeout=99999 has no effect

rsync --partial has no effect

openrsync seems to work, but is slow

openrsync --recursive --verbose svn.code.sf.net::p/jdownloader-svn-mirror-milahu/code/ jdownloader/

[milahu]

openrsync seems to work

no, also openrsync fails to fetch the repo.
it runs for 3 hours, but downloads only 200MB of 1.5GB.

$ time openrsync -v --recursive svn.code.sf.net::p/jdownloader-svn-mirror-milahu/code/ jdownloader/
Transfer starting: 92047 files
README.txt (246 B, 0.0% downloaded)
conf/authz (1.1 KB, 0.0% downloaded)
conf/hooks-env.tmpl (885 B, 0.0% downloaded)
...
db/revprops/39/39020 (157 B, 0.0% downloaded)
db/revprops/39/39021 (313 B, 0.0% downloaded)
db/revprops/39/39022 (226 B, 0.0% downloaded)
openrsync: error: poll: bad fd
openrsync: error: rsync_receiver

real    168m4.729s
user    0m1.617s
sys     166m50.881s

what does help is rsync --ignore-existing
which makes rsync run much faster
by skipping comparisons of file size and mtime.

in my case (1.5GB repo) this is fast enough
to sync faster than the server timeout (330 seconds).

$ time rsync --ignore-existing --recursive --info=progress2 milahu@svn.code.sf.net::p/jdownloader-svn-mirror-milahu/code/ jdownloader/
  1,735,056,965 100%    5.76MB/s    0:04:47 (xfr#91946, to-chk=0/92047)

real    4m48.860s
user    0m7.727s
sys     0m17.455s

still, rsync can fail due to local problems.
in my case, my old SSD disk blocked the write operation for multiple minutes.

$ time rsync --ignore-existing --recursive --info=progress2 milahu@svn.code.sf.net::p/jdownloader-svn-mirror-milahu/code/ jdownloader/
  1,560,147,183  95%    2.85kB/s    7:53:20  rsync: [generator] write error: Connection reset by peer (104)
rsync error: error in socket IO (code 10) at io.c(848) [generator=3.3.0]

real    24m14.763s
user    0m0.249s
sys     0m0.752s

write error

this was a problem with my old SSD disk.
removing the last partially downloaded file took multiple minutes.

find-newest-file.sh

#!/bin/sh
# find newest file
d="${1:-.}"
# https://stackoverflow.com/a/4562154/10440128
find "$d" -type f -printf "%T@\0%p\0" | awk '{ if ($0>x) {x=$0; getline z} else getline } END {print z}' RS='\0'

$ find-newest-file.sh jdownloader/
jdownloader/db/revs/5/.5208.O9bEew

$ rm jdownloader/db/revs/5/.5208.O9bEew

$ find-newest-file.sh jdownloader/
jdownloader/db/revs/5/5207

continue the download

$ time rsync --ignore-existing --recursive --info=progress2 milahu@svn.code.sf.net::p/jdownloader-svn-mirror-milahu/code/ jdownloader/
    125,341,857   7%    2.17MB/s    0:00:55 (xfr#3659, to-chk=0/92047)

real    1m25.463s
user    0m0.708s
sys     0m1.330s

now, with a complete download, a "noop run" only takes 30 seconds.

$ time rsync --ignore-existing --recursive --info=progress2 milahu@svn.code.sf.net::p/jdownloader-svn-mirror-milahu/code/ jdownloader/
              0   0%    0.00kB/s    0:00:00 (xfr#0, to-chk=0/92047)   

real    0m31.155s
user    0m0.173s
sys     0m0.201s

[Dillon Walls]

Hi,

Thank you for reaching out to SourceForge Support!

I am looking into your issue. What I have found so far is that I was able to successfully rsync that svn repo on the first try.

$ time rsync --recursive --info=progress2 svn.code.sf.net::p/jdownloader-svn-mirror-milahu/code/ jdownloader/; echo "return code: $?"
  1,820,445,366 100%    4.81MB/s    0:06:00 (xfr#99615, to-chk=0/99724)

real    6m3.220s
user    0m7.868s
sys 0m43.025s
return code: 0

I would appreciate having your help to look into a few things before escalating this.

• Are you on an older version of rsync? My version is

$ rsync -V
rsync  version 3.3.0  protocol version 31
Copyright (C) 1996-2024 by Andrew Tridgell, Wayne Davison, and others.
Web site: https://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, symlinks, symtimes, hardlinks, hardlink-specials,
    hardlink-symlinks, IPv6, atimes, batchfiles, inplace, append, ACLs,
    xattrs, optional secluded-args, iconv, no prealloc, stop-at, crtimes,
    file-flags
Optimizations:
    no SIMD-roll, no asm-roll, openssl-crypto, no asm-MD5
Checksum list:
    xxh128 xxh3 xxh64 (xxhash) md5 md4 sha1 none
Compress list:
    zstd lz4 zlibx zlib none
Daemon auth list:
    sha512 sha256 sha1 md5 md4

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

• Have you tried rsync with other flags? We have several documentation pages that suggest various ways to use rsync to interact with SourceForge. Your invocation DID work for me, that said, I cobbled together this invocation from a few of those pages and it also worked. You could try this

time rsync -avzPL svn.code.sf.net::p/jdownloader-svn-mirror-milahu/code/ jdownloader-avzPL/; echo "return code: $?"
...

sent 1,893,956 bytes  received 1,653,378,731 bytes  2,953,207.29 bytes/sec
total size is 1,820,445,366  speedup is 1.10

real    9m20.182s
user    0m12.129s
sys 0m46.275s
return code: 0

Excerpts on those options from man rsync:

       --archive, -a            archive mode is -rlptgoD (no -A,-X,-U,-N,-H)
       --verbose, -v            increase verbosity
       --compress, -z           compress file data during the transfer
       -P                       same as --partial --progress
       --progress               show progress during transfer
       --partial                keep partially transferred files
       --copy-links, -L         transform symlink into referent file/dir

[milahu]

version of rsync?

same

$ rsync -V
rsync  version 3.3.0  protocol version 31

I was able to successfully rsync that svn repo on the first try.

yes, on my end, now it also works without --ignore-existing, as expected.
a full download takes 10 minutes, no timeout.
seems like this was some temporary issue...

rsync -avzPL

when rsync was failing, i also tried --partial but no change.
i guess the other options dont help here.

meanwhile, i have pushed new revisions to the SVN repo
by creating an empty hooks/pre-revprop-change script
and by calling

svnsync sync svn+ssh://milahu@svn.code.sf.net/p/jdownloader-svn-mirror-milahu/code/ file://$PWD/jdownloader/

i have pushed new revisions to the SVN repo

maybe that fixed the issue... probability is low, but higher than zero.

[SourceForge Support]

Hi,

Thank you for your feedback
With your issue resolved, this ticket will be closed.

Sincerely,
SourceForge Support