Ganael Laplanche committed [b2cf27] on Code

fpsync: Colorize output

Ganael Laplanche committed [5ef0b4] on Code

Changelog: write explicitly about log *files*

Ganael Laplanche committed [17cc63] on Code

fpsync: Run resuming - Rework commit b853b77d

Ganael Laplanche committed [70945f] on Code

fpsync: cleanup variable names

Ganael Laplanche committed [5e9e7d] on Code

fpsync: homogenize QMGR messages

Ganael Laplanche committed [0e534b] on Code

fpsync: only display kill messages once when pressing CTRL-C

Ganael Laplanche committed [7404f3] on Code

fpsync: typos in comments

Ganael Laplanche committed [7e1b5a] on Code

fpsync: use explicit -p option for ps

Ganael Laplanche committed [b853b7] on Code

fpsync: Run resuming - Rework commit 686edde

Ganael Laplanche committed [686edd] on Code

fpsync: fix wrong jobs rescheduled when resuming a run (option -r)

Ganael Laplanche committed [0cb35f] on Code

fpsync: Remove useless '-f' option in mv command

Ganael Laplanche committed [514241] on Code

TODO: Update

Ganael Laplanche committed [60e6a6] on Code

Debian: add reference to pax(1) in control file

Ganael Laplanche committed [3c9629] on Code

fpsync: cleanup logs when replaying a run

Ganael Laplanche committed [02f103] on Code

fpsync: improve startup logs

Ganael Laplanche committed [36618f] on Code

fpsync: rename main log file to fpsync.log

Ganael Laplanche committed [ca901d] on Code

fpsync: improve .ret files filtering

Ganael Laplanche committed [3eff98] on Code

fpsync: fix failure detection for piped tools

Ganael Laplanche committed [23108c] on Code

fpsync: rework final report

Ganael Laplanche committed [0de291] on Code

fpsync: rework return code handling

Ganael Laplanche committed [bfff4a] on Code

Doc: Add a link to Utah's CHPC website

Ganael Laplanche committed [38de1b] on Code

Doc: Update Changelog following pax(1) tool support

Ganael Laplanche committed [43cfcc] on Code

Doc: Update SILL link

Ganael Laplanche committed [fb1bd1] on Code

fpsync: Document pax tool

Ganael Laplanche committed [6f2195] on Code

fpsync: Add preliminary support for pax(1) tool

Ganael Laplanche committed [ed6875] on Code

doc: Man pages fixes

Ganael Laplanche committed [9f7c08] on Code

fpsync: Log job PIDs in -vvv mode

Ganael Laplanche committed [9bb6e5] on Code

Doc: Add a few words to insist on the fact the fpart only reads data

Ganael Laplanche committed [29ef89] on Code

TODO: Add idea

Ganael Laplanche committed [616244] on Code

Debian: Bump Standards-Version in debian/control

Ganael Laplanche committed [4d1414] on Code

fpsync: check for common commands/tools presence (GH issue #47)

Ganael Laplanche committed [a2fe71] on Code

Update Debian package

Ganael Laplanche committed [562e8d] on Code

man/fpart.1: Fix typo

Ganael Laplanche committed [6a691c] on Code

WWW: Fix typo (capital)

fpart released /fpart/fpart-1.6.0/fpart-1.6.0.tar.gz

fpart released /fpart/fpart-1.6.0/fpart-1.6.0.tar.gz.asc

Ganael Laplanche committed [7758e4] on Code

Release version 1.6.0

Ganael Laplanche committed [7d8713] on Code

WWW: Fix Macstadium URL

Ganael Laplanche committed [3ab730] on Code

WWW: Add a note about Debian Almquist shell (dash)

Ganael Laplanche committed [28a825] on Code

WWW: Add a note about TAR_NAME following c3e5af0

Ganael Laplanche committed [c3e5af] on Code

fpsync: Facilitate tar tool change on Solaris-based OS

Ganael Laplanche committed [6af054] on Code

WWW: Update links

Ganael Laplanche committed [5c6282] on Code

Bump copyright dates

Ganael Laplanche committed [043259] on Code

Update TODO

Ganael Laplanche committed [7069fd] on Code

Website: mention Github sponsors page

Ganael Laplanche committed [b26b4d] on Code

README: mention Github sponsors page

Ganael Laplanche committed [a74f1c] on Code

fpsync: Avoid returning "" from total_*_count() when run.meta file does not exist

Ganael Laplanche committed [30f0c5] on Code

fpsync: Show summary after fpart pass

Ganael Laplanche committed [f4611f] on Code

fpsync: Limit useless log generation by removing empty log files

Ganael Laplanche committed [0d0698] on Code

Doc: Add precisions about tested OS versions

Ganael Laplanche committed [2dcc1c] on Code

Doc: add new links

Ganael Laplanche posted a comment on discussion General Discussion

On 10/24/23 18:57, Fab11 wrote: Hello there, I increased parts number from 1000 to 10000 but it was still way faster to generate. I then moved to 100000 and it looks to suit better. In the worst case, the last part was generated just an hour before the end of the sync (half an hour on average) for a 7h to 9h job duration. Difficult to say if it was quicker and how much because it was a second pass anyway. So a big part of the time rsync took to copy files in the first pass was not spent again during...

Ganael Laplanche committed [3a5db9] on Code

TODO: add ideas brought by Fab11 on SF

Ganael Laplanche committed [4d6dd9] on Code

fpsync: Speed up total counts in SIGINFO handler

Fab11 posted a comment on discussion General Discussion

Hello, Sorry to come back with a reply so late. I increased parts number from 1000 to 10000 but it was still way faster to generate. I then moved to 100000 and it looks to suit better. In the worst case, the last part was generated just an hour before the end of the sync (half an hour on average) for a 7h to 9h job duration. Difficult to say if it was quicker and how much because it was a second pass anyway. So a big part of the time rsync took to copy files in the first pass was not spent again...

Ganael Laplanche committed [f5b9fa] on Code

Update TODO

Ganael Laplanche committed [8afafc] on Code

fpart: add missing free(env_fpart_parterrno_string)

Ganael Laplanche committed [5a1eac] on Code

fpart: add new hook variable: FPART_TOTALNUMPARTS

Ganael Laplanche committed [1a25e9] on Code

fpart: add post-run hook (option -R)

Ganael Laplanche committed [f408c5] on Code

fpart: Add total_num_parts to main_status

Ganael Laplanche committed [f328d9] on Code

fpart: Introduce main_status to track total size and files

Ganael Laplanche committed [af876f] on Code

fpart: add two new hook variables: FPART_TOTALSIZE and FPART_TOTALNUMFILES

Fab11 posted a comment on discussion General Discussion

Hello, I will try that and let you know. You are welcome for the ideas :) Thanks for your consideration. Best regards.

Ganael Laplanche posted a comment on discussion General Discussion

On Tuesday, July 25, 2023 3:52:17 PM CEST Fab11 wrote: Hello, Many thanks for your detailed answer! You're welcome :) You confirmed 1000 files is not enough, it's my opinion too. I realised that while looking at the top command output frequently. Parts files number passed on the command line to fpsync were way ahead of the last generated one, meaning fpart was faster generating parts than rsync running them. I will maybe try to generate bigger parts for intermediate runs. Yes, that's the way to go,...

Fab11 posted a comment on discussion General Discussion

Hello, Many thanks for your detailed answer! You confirmed 1000 files is not enough, it's my opinion too. I realised that while looking at the top command output frequently. Parts files number passed on the command line to fpsync were way ahead of the last generated one, meaning fpart was faster generating parts than rsync running them. I will maybe try to generate bigger parts for intermediate runs. Another question regarding jobs (thanks for the details on the usage). I was wondering what would...

Ganael Laplanche posted a comment on discussion General Discussion

On Wednesday, July 19, 2023 8:52:32 PM CEST Fab11 wrote: Hello, Sorry for my delayed answer, I am just back from holidays :p [...] 1) To not wait for big lists to be generated by fpart and start rsync right away, I set fpsync to use parts of 1000 files (-f). But I realised the number of parts is very huge (45,000+). I have to say data are spreaded on several 8 TB volumes, processed one at a time and one after the other. The server running the jobs has 48 cores and fpsync is setup to run 24 parallel...

Fab11 posted a comment on discussion General Discussion

Hello, I have to migrate a large amount of data to a new storage array (around 100 TB), which is mostly composed of small files (max 100 KB) located into a 7 directory deep structure. The transfert to the new storage array has to be achieved through a SMB/CIFS share. So, Samba and small files, the worst case scenario... Using rsync or robocopy would take ages. I discovered fpart and fpsync by accident and it turned out to be the magic solution according to my speed tests. It would let me copy these...

Ganael Laplanche committed [e4b94b] on Code

fpsync: Remove useless redirection to /dev/null

Ganael Laplanche committed [b45efa] on Code

fpsync: Clarify comments regarding resume flags

Ganael Laplanche committed [104375] on Code

fpsync: show transmitted files count / data size in final status and SIGINFO handler

Ganael Laplanche committed [992b5f] on Code

fpsync: fix off-by-n error when calculating ETA in SIGINFO handler

Ganael Laplanche committed [182794] on Code

Doc typo

Ganael Laplanche committed [1e9695] on Code

Unbreak Paypal picture and link

Ganael Laplanche committed [8ec684] on Code

Update documentation

Ganael Laplanche committed [034604] on Code

fpart: provide long options (for the most important ones)

Ganael Laplanche committed [8eb7c0] on Code

fpsync/mdoc(7): Ic -> Fl

Ganael Laplanche committed [bface3] on Code

fpart/mdoc(7): Ic -> Fl

Ganael Laplanche committed [753003] on Code

TODO: Add idea

Ganael Laplanche committed [b47f19] on Code

Doc: Improve fpsync description

Ganael Laplanche committed [d0ed75] on Code

Fix changelog

Ganael Laplanche committed [30d9e4] on Code

doc: Update links and add logos

Ganael Laplanche committed [7db718] on Code

doc: Update fpsync page following new fpart's option -P

Ganael Laplanche committed [959874] on Code

doc: Write about fpart passing Coverity Scan tests

Ganael Laplanche committed [67bcec] on Code

fpart: remove useless NULL check for in_fp

Ganael Laplanche committed [e7ba09] on Code

fpart: max_size is signed

Ganael Laplanche committed [38cb72] on Code

fpart/man: Minor fixes

Ganael Laplanche committed [f8c8b8] on Code

fpart/man: Document option -P

Ganael Laplanche committed [3fb1f6] on Code

fpsync: fix tar tool on OSX

Ganael Laplanche committed [3e88c4] on Code

fpsync: fix directory modification times updates with GNU tar(1)

Ganael Laplanche committed [9ba434] on Code

fpsync: enable fpart's new option -P with cpio(1) and tar(1)

Ganael Laplanche committed [582a67] on Code

fpart: add parent_path() function

Ganael Laplanche committed [b487df] on Code

fpart: add option -P (add parent directories before closing partitions)

Ganael Laplanche committed [3cae9d] on Code

fpart: fix if_not_realloc() macro

Ganael Laplanche committed [82d0d3] on Code

fpart: rename cleanup_path() to cleanslash_path()

Ganael Laplanche committed [f9742e] on Code

Revert ef6b10c - globally ignore SIGTTIN and SIGTTOU

Ganael Laplanche committed [c9b3aa] on Code

Typo

Ganael Laplanche committed [0d3f06] on Code

Update changelog following merge of GH PR #44

GitHub committed [e6fb28] on Code

Merge pull request #44 from biocyberman/pr

BioCyberman committed [be14d1] on Code

Set OPT_TOOL_PATH in fpsync