7.4.6 Cannot Backup Subdirectories with Large Number of Files, Slow
Brought to you by:
aventin
Menu
▾
▴
#537 7.4.6 Cannot Backup Subdirectories with Large Number of Files, Slow
Problem:
Sometime between v. 7.4.3 and 7.4.6 (likely 7.4.6) FULL backups (Windows x64) now fail due to the backup taking days to complete--taking 3+ hours PER directory for directories with larger numbers of files. This is not just an inconvenience but renders backups practically unusable.
Directories with larger numbers of files (> 100 or so files) now take hours to process using the standard Zip64 and standard backup options--that is, a fairly basic target setup. Backups that formerly took a total of 3h15m now take more than 24 hours for < 5% of the backup to complete.
NOTE: the problem will only be evident if: 1) you perform a FULL backup or 2) you create a new directory with a large number of files and try to perform an incremental backup. The key seems to be a larger number of files.
Background:
Have used Areca for several months with same backup target(s). I perform a full backup twice per month and a separate daily incremental backup. Typically, the full backups take approximately 3h15m to complete with about 125 GB of data compressed.
No real problems encountered until after May 11, 2014--the last successful full backup. On May 31, the normal bi-monthly backup ran for 23 hours and was estimated less than 1/20 complete. This triggered fours days since of trials and testing to narrow the problem. It does seem to arise from directories with proportionally large numbers of files > 100. These directories now "choke" the backup and can take three or more hours to backup a single directory containing 1000 files (these are simple machine learning text files of emails--small in size about 3k each and ordinary text files).
What I Tried:
1) Uninstall and reinstall Areca multiple times.
2) Change areca.l4j.ini to Xmx8094m
3) Tried multiple external hard drives verifying USB 3.0 connections.
4) Ran chkdsk on all external drives used and on the main magnetic drive. (Minor errors were detected and fixed on one external drive but no other errors)
4.5) ran disk diagnostics (no errors)
4.7) ran defrag (little <1% fragmentation--these are fairly new drives)
5) Multiple runs of FULL backups using both the past targets (from the past few months) and newly created targets--all fail due to extremely long (about 3 hours or more per directory).
6) Multiple runs of INCREMENTAL backups (which generally work UNLESS you create a new directory in the backup target with a larger number of files and the same extremely slow performance as seen in FULL backups--new directory can be on the magnetic OR SSD drive as the source).
7) Verification and checking of older backups and verifying no changes to the target config files.
8) This was intensive testing for about four days. The common denominator seems to be larger numbers of files in directories.
System Info:
System i7 x64 32GB memory, SSD drive, high performance magnetic drive (most data on magnetic drive), and full USB 3.0 external backup drives (2 tried). Windows 8.1 x64 (all patches)
With areca.l4j.ini to Xmx8094m Windows Task Manager reports about 14% CPU usage and about 1,750MB memory usage (fairly stable) and does appear to be doing something albeit extremely slowly
Test File:
A zipped collection of some sample documents that causes the issue is at:
http://www.prolorem.com/cms/wp-content/uploads/2013/08/Test_Dataset_2_Cleaned.zip
REVISION TO:
https://sourceforge.net/p/areca/bugs/536/
(Sorry, could not report all details in the other post).
Additional testing last night seems to further confirm the too-many-files problem. I ran a DIFFERENTIAL backup (no major changes to directories since last backup so this should have avoided backing up any directories with > 100 or so files). This completed successfully albeit also noticeably slower (taking 4h11m for 3GB).