I've been giving bsdtar a thorough workout with some Big archives and Long pathnames. It appears to have failed on both :( I haven't been able to create a gzipped tarball larger than approx 4GB and I received many "Couldn't visit directory: Invalid argument" failure messages even though the directory paths are valid. I am using the version 1.2.38 of bsdtar.
Any Ideas/Suggestions?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I had the same problem while creating some big RAW avi with opencv. It appears that 4GB is considered a maximum for a file. Somehow it should be configurable. Try some googling. For the long filenames are you sure that it is not a windows specific issue? Try 7zip to compess some of your files in the long pathname into a cab file (or use something else instead of 7zip) and run MS extract on this .cab file. If it fails then it is windows specific. For the last one maybe it is not windows specific. How many chars is this long path?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thanks for the reply... I haven't been able to Google an answer for the 4GB issue and I have confirmed that 7Zip does handle the long path names (< 1000 chars) that BSDTar failed on.
I should add that I believe I have found a work-around for my problems in the DAR utility: http://dar.linux.free.fr/
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The Gnuwin32 port of Bsdtar and Libarchive uses 64-bits versions of fstat and stat, and so should be able to handle files larger than 4 GB. I can't easily find the cause of this behavior, and will investigate it further.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
DAR didn't work out either... It uses the Cygwin libraries for the Windows port and they are limited to the old DOS MAX_PATH maximum path length of 260 characters. In all other respects DAR was an excellent tool.
I hope GnuWin32 can get to the bottom of these BSDTar issues.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I just discovered that the version of BASH I've been using to run my BSDTar scripts, (win-bash) also suffers from a short maximum command-line length restriction.
Does anyone know either:
(a) A Windows Bash port that will support long command lines
or
b) A way for BSDTar to read its file/folder list from an external file.
The latter option is to be preferred, although both would be nice.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
(a) This is an almost hard-wired limit of MS-Windows; see http://msdn2.microsoft.com/en-us/library/aa363858.aspx
On NT / 2000/ XP systems, it is possible to open files with names up to 32767 characters by prefixing the name with "\?\", but I'm not sure whether the C runtime library (msvcrt.dll) allows this.
(b) With the option
-T <filename>
Bsdtar reads the list of files to be archived / extracted from <filename>.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi All,
I've been giving bsdtar a thorough workout with some Big archives and Long pathnames. It appears to have failed on both :( I haven't been able to create a gzipped tarball larger than approx 4GB and I received many "Couldn't visit directory: Invalid argument" failure messages even though the directory paths are valid. I am using the version 1.2.38 of bsdtar.
Any Ideas/Suggestions?
I had the same problem while creating some big RAW avi with opencv. It appears that 4GB is considered a maximum for a file. Somehow it should be configurable. Try some googling. For the long filenames are you sure that it is not a windows specific issue? Try 7zip to compess some of your files in the long pathname into a cab file (or use something else instead of 7zip) and run MS extract on this .cab file. If it fails then it is windows specific. For the last one maybe it is not windows specific. How many chars is this long path?
Thanks for the reply... I haven't been able to Google an answer for the 4GB issue and I have confirmed that 7Zip does handle the long path names (< 1000 chars) that BSDTar failed on.
I should add that I believe I have found a work-around for my problems in the DAR utility: http://dar.linux.free.fr/
The Gnuwin32 port of Bsdtar and Libarchive uses 64-bits versions of fstat and stat, and so should be able to handle files larger than 4 GB. I can't easily find the cause of this behavior, and will investigate it further.
DAR didn't work out either... It uses the Cygwin libraries for the Windows port and they are limited to the old DOS MAX_PATH maximum path length of 260 characters. In all other respects DAR was an excellent tool.
I hope GnuWin32 can get to the bottom of these BSDTar issues.
Aaarrrggghh!...
I just discovered that the version of BASH I've been using to run my BSDTar scripts, (win-bash) also suffers from a short maximum command-line length restriction.
Does anyone know either:
(a) A Windows Bash port that will support long command lines
or
b) A way for BSDTar to read its file/folder list from an external file.
The latter option is to be preferred, although both would be nice.
(a) This is an almost hard-wired limit of MS-Windows; see
http://msdn2.microsoft.com/en-us/library/aa363858.aspx
On NT / 2000/ XP systems, it is possible to open files with names up to 32767 characters by prefixing the name with "\?\", but I'm not sure whether the C runtime library (msvcrt.dll) allows this.
(b) With the option
-T <filename>
Bsdtar reads the list of files to be archived / extracted from <filename>.
Forget option (b), I was momentarily confusing BSDTar with DAR, (I've been experimenting with both as well as TAR at the same time).
The latest release of libarchive and bsdtar seems to handle large files correctly. I've tested this with a file of approximately 5 GB.
I am also facing problems with long file names using bsdtar.exe from libarchive 2.2.5
and prefixing "\?\" did'nt help.
//working path
bsdtar.exe -c -f d:\longestpath.tar --format pax D:\abcdefghijklmnopqrstuvwxyz\abcdefghijklmnopqrstuvwxyz\abcdefghijklmnopqrstuvwxyz\abcdefghijklmnopqrstuvwxyz\abcdefghijklmnopqrstuvwxyz\abcdefghijklmnopqrstuvwxyz\abcdefghijklmnopqrstuvwxyz\abcdefghijklmnopqrstuvwxyz\abcdefghijklmnopqrstuvwxyz\temp.cfg
//Not Working Path
bsdtar.exe -c -f D:\HP\dmp\cps-5-1-ext-sdk\tools\OpTest\cache\cps\temp\8454e0df-dce3-493d-a64a-3158edc65bd9.tmp --format pax D:\HP\dmp\cps-5-1-ext-sdk\tools\OpTest\cache\cps\temp\21804a0b-88ab-4acb-ada5-253dc925e851\Package\harmeet01\data\pulkit_400kb.mpg
any clues ..
Thanks in advance
I think there is an internal MS-Windows limit of 256 characters for a full path name given to (f)open.