Menu
▾
▴
aixtoolbox-list
|
From: Michael W. <mw...@un...> - 2004-06-13 22:29:33
|
The problem: $ ls -l -rw-r--r-- 1 oracle oinstall 4666171392 Jun 11 23:30 maxdataMXPDB02.dbf $ rsync -av maxdataMXPDB02.dbf /net02/oracle/ building file list ... done maxdataMXPDB02.dbf rsync: writefd_unbuffered failed to write 32768 bytes: phase "unknown": Broken pipe rsync error: error in rsync protocol data stream (code 12) at io.c(836) The envrionment: OS: AIX 5.2 with current patch level 02. rsync: 2.5.4 from http://www-1.ibm.com/servers/aix/products/aixos/linux/download.html This also happens for current rsync version 2.6.2 built using gcc 2.9 (The gcc is built by IBM available from same URL as above). The error with rsync 2.6.2 is shown below. Comment: It appears that the problem occurs for large files. In the test shown, it is 4.6GB. The problem is also reported in https://bugzilla.samba.org/show_bug.cgi?id=1457 The problems occurs when both the source and destination are on locally mounted file systems (versus NFS). So the network is not involved here at all. -- Michael Wang * http://www.unixlabplus.com/ * mw...@un... |
|
From: Philip K W. <pk...@us...> - 2004-06-14 22:31:40
|
> The problem is also reported in > https://bugzilla.samba.org/show_bug.cgi?id=1457 > > The problems occurs when both the source and destination are on > locally mounted file systems (versus NFS). So the network is not > involved here at all. Did you try the suggestion in Bug 1457 (using the --temp-dir parameter to specify the temporary file location)? --- Philip K. Warren pk...@us... Phone: 512.838.8924 T/L 678.8924 |
|
From: Michael W. <mw...@un...> - 2004-06-14 23:57:52
|
On Mon, Jun 14, 2004 at 02:30:28PM -0500, Philip K Warren wrote: > > The problem is also reported in > > https://bugzilla.samba.org/show_bug.cgi?id=1457 > > > > The problems occurs when both the source and destination are on > > locally mounted file systems (versus NFS). So the network is not > > involved here at all. > > Did you try the suggestion in Bug 1457 (using the --temp-dir parameter to > specify the temporary file location)? The suggestion to use temporary file location is based on that the destination does not have enough space. In my case, the destination does have enough space. While "rsync" failed, "tar -cvf ... | tar -xvf ..." succeeded. The reason I prefer rsync over tar/cp/etc is the ability to rerun the command without re-copying files which are successfully copied. I think my problem is different than the one originally reported, although the error message is the same. Thanks. -- Michael Wang * http://www.unixlabplus.com/ * mw...@un... |
|
From: Michael W. <mw...@un...> - 2004-06-15 16:45:06
|
Philip and other IBM folks,
Following up the rsync issue, I have found more useful information.
Problem:
rsync as packaged by IBM can not write beyond 2,147,483,647 bytes.
Proof:
(1) while rsync from a remote box (Solaris 8) to IBM AIX 5.2:
$ /opt/csw/bin/rsync -av --rsh=ssh users01.dbf oracle@zimu:/max-ds1/ora...
...
rsync: writefd_unbuffered failed to write 4 bytes: phase "unknown": Broken pipe
rsync error: error in rsync protocol data stream (code 12) at io.c(515)
I monitor the temp file:
$ while true; do ls -ls .u*; df -k .; done
Filesystem 1024-blocks Free %Used Iused %Iused Mounted on
/dev/lv01 7503872 292200 97% 23225 2% /max-ds1
2097152 -rw------- 1 oracle dba 2147483647 Jun 15 09:06 .users01.dbf.OS2Oaa
Filesystem 1024-blocks Free %Used Iused %Iused Mounted on
/dev/lv01 7503872 292200 97% 23225 2% /max-ds1
ls: 0653-341 The file .u* does not exist.
The write failed exactly at 2,147,483,647 which is 2*1024*1024*1024 - 1.
And the truss -f -p <rsync_pid> shows:
28448: kread(0, "\0\0\0\0\0\0\0\0\0\0\0\0".., 19688) = 16384
28448: _select(1, 0x2FF205D8, 0x00000000, 0x00000000, 0x2FF205D0) = 1
28448: kread(0, "\0\0\0\0\0\0\0\0\0\0\0\0".., 3304) = 3304
28448: kwrite(1, "\002\0\00203 ÿ ü\0\0\0\0".., 32768) = 32767
28448: kwrite(6, " &\0\0\b w r i t e f a".., 42) = 42
28448: _sigaction(30, 0x2FF21AC0, 0x2FF21AD0) = 0
28448: _sigaction(31, 0x2FF21AC0, 0x2FF21AD0) = 0
28448: unlink(".users01.dbf.OS2Oaa") = 0
28448: kwrite(6, " ;\0\0\b r s y n c e r".., 63) = 63
28448: kfcntl(1, F_GETFL, 0x00000000) = 2
28448: kfcntl(2, F_GETFL, 0x00000000) = 2
28448: _exit(11)
I would think it should use *write64(). Is kwrite() capable handling
2+GB file?
(2) Could this be the remote rsync problem?
No. Because:
(2.1) rsync 6+GB file to the remote box succeeded as I reported
earlier, which means the remote rsync is capable write 6+GB file.
And this appears that rsync on IBM AIX 5.2 is capable reading
6+GB file.
(2.2) rsync on IBM AIX 5.2 locally also fails.
(2.3) rsync on remote box to itself succeeded for the same 2+GB file.
Environment:
# oslevel -r
5200-02
# rpm -ql rsync-2.5.4-1
/opt/freeware/bin/rsync
/opt/freeware/doc/rsync-2.5.4
/opt/freeware/doc/rsync-2.5.4/COPYING
/opt/freeware/doc/rsync-2.5.4/README
/opt/freeware/doc/rsync-2.5.4/tech_report.tex
/opt/freeware/man/man1/rsync.1
/opt/freeware/man/man5/rsyncd.conf.5
/usr/bin/rsync
Other info:
- Straightforward compilation for the latest version of rsync 2.6.2
have the same problem. Modify config.h not to mkstemp() may fix it.
- This problem should be very easy to reproduce.
--
Michael Wang * http://www.unixlabplus.com/ * mw...@un...
|
|
From: Philip K W. <pk...@us...> - 2004-06-15 18:52:44
|
I can reproduce this problem locally with the latest version of rsync on the Toolbox. I am preparing a new image which should resolve the problem. --- Philip K. Warren pk...@us... Phone: 512.838.8924 T/L 678.8924 |
|
From: Philip K W. <pk...@us...> - 2004-06-15 21:12:21
|
A new version of rsync (2.5.4-3) is now available, which fixes this problem with writing to large files. The configure script for rsync correctly detects support for mkstemp on AIX 5.1, however the large-file enabled mkstemp64 was not added to libc.a until a later release of AIX 5.1/5.2. I have compiled the rsync RPM so it uses mktemp instead of mkstemp, which fixes this problem and allows rsync to correctly copy files greater than 2gb in size. Please let me know if you have any problems with this new RPM. Thanks Michael for reporting the problem. ftp://ftp.software.ibm.com/aix/freeSoftware/aixtoolbox/RPMS/ppc/rsync/rsync-2.5.4-3.aix5.1.ppc.rpm --- Philip K. Warren pk...@us... Phone: 512.838.8924 T/L 678.8924 |
Thanks for helping keep SourceForge clean.
X