From: Mantis B. T. <no...@bu...> - 2010-04-29 06:43:56
|
The following issue has been SUBMITTED. ====================================================================== http://bugs.bacula.org/view.php?id=1565 ====================================================================== Reported By: tjaszo Assigned To: ====================================================================== Project: bacula Issue ID: 1565 Category: other Reproducibility: always Severity: major Priority: normal Status: new ====================================================================== Date Submitted: 2010-04-29 07:43 BST Last Modified: 2010-04-29 07:43 BST ====================================================================== Summary: Extremly poor performance when milions of small files to backup Description: We have some archives containing millions of small files ( from 20B to 1333789B , avg 1008 B). Backup of such directory takes long time. Below Client Status: se-bama-fd Version: 3.0.2 (18 July 2009) i686-redhat-linux-gnu redhat Daemon started 26-Apr-10 16:22, 20 Jobs run since started. Heap: heap=696,320 smbytes=229,868 max_bytes=250,960 bufs=142 max_bufs=169 Sizeof: boffset_t=8 size_t=4 debug=0 trace=0 Running Jobs: JobId 2513 Job backup-EVA-g10k-alsb.2010-04-28_21.27.00_53 is running. Backup Job started: 28-Apr-10 21:27 Files=23,175,715 Bytes=40,345,321,475 Bytes/sec=1,052,797 Errors=0 Files Examined=23,175,715 Processing file: /opt/ediarchive/active/ax4edi-monitoring-archive/archive/RailNet/GermanRailwayTrackingStatus/4-importer-error/2010-03-13.zip SDReadSeqNo=5 fd=5 Director connected at: 29-Apr-10 08:06 ==== Backup is done from mounted filesystem (EXT3) - from SAN (raw transfer tested using tar show ~130MB/s on reading), to LTO3 tape library (raw transfer tested using tar show ~80MB/s on writing). As You can see above this job is runinng with speed of 1,052,797B/sec which is extremely poor. Moreover I got "The table 'File' is full" errors Jobs log: 28-Apr 21:27 se-bama-dir JobId 2513: No prior Full backup Job record found. 28-Apr 21:27 se-bama-dir JobId 2513: No prior or suitable Full backup found in catalog. Doing FULL backup. 28-Apr 21:27 se-bama-dir JobId 2513: shell command: run BeforeJob "./pre_MAIN_g10k_alsb.sh" 28-Apr 21:27 se-bama-dir JobId 2513: Start Backup JobId 2513, Job=backup-EVA-g10k-alsb.2010-04-28_21.27.00_53 28-Apr 21:27 se-bama-dir JobId 2513: Using Device "lto3drive" 29-Apr 08:22 se-bama-dir JobId 2513: Fatal error: sql_create.c:974 sql_create.c:974 insert INSERT INTO File (FileIndex,JobId,PathId,FilenameId,LStat,MD5) VALUES (23214448,2513,540614,246637,'PwB 1wia IGk B OL OL A CKe BAA g BLmC4/ BLlufb BLlufb A A C','0') failed: The table 'File' is full 29-Apr 08:22 se-bama-dir JobId 2513: Fatal error: sql_create.c:976 Create db File record INSERT INTO File (FileIndex,JobId,PathId,FilenameId,LStat,MD5) VALUES (23214448,2513,540614,246637,'PwB 1wia IGk B OL OLA CKe BAA g BLmC4/ BLlufb BLlufb A A C','0') failed. ERR=The table 'File' is full29-Apr 08:22 se-bama-dir JobId 2513: Fatal error: catreq.c:409 Attribute create error. sql_create.c:976 Create db File record INSERT INTO File (FileIndex,JobId,PathId,FilenameId,LStat,MD5) VALUES (23214448,2513,540614,246637,'PwB 1wia IGk B OL OL A CKe BAA g BLmC4/ BLlufb BLlufb A A C','0') failed. ERR=The table 'File' is full29-Apr 08:22 se-bama-sd JobId 2513: Job backup-EVA-g10k-alsb.2010-04-28_21.27.00_53 marked to be canceled. 29-Apr 08:22 se-bama-sd JobId 2513: Fatal error: fd_cmds.c:177 FD command not found: 23214453 1 0 29-Apr 08:22 se-bama-sd JobId 2513: Job write elapsed time = 10:54:51, Transfer rate = 1.608 M bytes/second 29-Apr 08:22 se-bama-sd JobId 2513: Fatal error: append.c:292 Fatal append error on device "lto3drive" (/dev/nst0): ERR= 29-Apr 08:22 se-bama-sd JobId 2513: Job backup-EVA-g10k-alsb.2010-04-28_21.27.00_53 marked to be canceled. 29-Apr 08:22 se-bama-sd JobId 2513: Job backup-EVA-g10k-alsb.2010-04-28_21.27.00_53 marked to be canceled. 29-Apr 08:22 se-bama-sd JobId 2513: Job backup-EVA-g10k-alsb.2010-04-28_21.27.00_53 marked to be canceled. 29-Apr 08:22 se-bama-fd JobId 2513: Fatal error: backup.c:1068 Network send error to SD. ERR=Connection reset by peer 29-Apr 08:22 se-bama-sd JobId 2513: Fatal error: fd_cmds.c:166 Command error with FD, hanging up. Append data error. 29-Apr 08:22 se-bama-dir JobId 2513: Fatal error: sql_create.c:974 sql_create.c:974 insert INSERT INTO File (FileIndex,JobId,PathId,FilenameId,LStat,MD5) VALUES (23214449,2513,540614,546268,'PwB BK3xD IGk B OLOL A GMV BAA BA BLvlEn BLvlEn BLvlEn A A C','0') failed: The table 'File' is full 29-Apr 08:22 se-bama-dir JobId 2513: Fatal error: sql_create.c:976 Create db File record INSERT INTO File (FileIndex,JobId,PathId,FilenameId,LStat,MD5) VALUES (23214449,2513,540614,546268,'PwB BK3xD IGk B OL OL A GMV BAA BA BLvlEn BLvlEn BLvlEn A A C','0') failed. ERR=The table 'File' is full29-Apr 08:22 se-bama-dir JobId 2513: Fatal error: sql.c:445 Attribute create error. sql_create.c:976 Create db File record INSERT INTO File (FileIndex,JobId,PathId,FilenameId,LStat,MD5) VALUES (23214449,2513,540614,546268,'PwB BK3xD IGk B OL OL A GMV BAA BA BLvlEn BLvlEn BLvlEn A A C','0') failed. ERR=The table 'File' is full29-Apr 08:22 se-bama-dir JobId 2513: Error: Bacula se-bama-dir 3.0.2 (18Jul09): 29-Apr-2010 08:22:50 Build OS: i686-redhat-linux-gnu redhat JobId: 2513 Job: backup-EVA-g10k-alsb.2010-04-28_21.27.00_53 Backup Level: Full (upgraded from Differential) Client: "se-bama-fd" 3.0.2 (18Jul09) i686-redhat-linux-gnu,redhat, FileSet: "set-g10k-alsb" 2010-04-28 21:27:00 Pool: "WeeklyBackups" (From Run FullPool override) Catalog: "MyCatalog" (From Client resource) Storage: "msl2024" (From Job resource) Scheduled time: 28-Apr-2010 21:27:00 Start time: 28-Apr-2010 21:27:45 End time: 29-Apr-2010 08:22:50 Elapsed time: 10 hours 55 mins 5 secs Priority: 14 FD Files Written: 23,214,470 SD Files Written: 23,214,452 FD Bytes Written: 56,077,976,085 (56.07 GB) SD Bytes Written: 63,218,949,733 (63.21 GB) Rate: 1426.7 KB/s Software Compression: None VSS: no Encryption: no Accurate: no Volume name(s): LZ3054L3 Volume Session Id: 109 Volume Session Time: 1272291783 Last Volume Bytes: 847,013,916,672 (847.0 GB) Non-fatal FD errors: 0 SD Errors: 0 FD termination status: Error SD termination status: Error Termination: *** Backup Error *** 29-Apr 08:22 se-bama-dir JobId 2513: shell command: run AfterJob "./post_MAIN_g10k_alsb.sh" ====================================================================== Issue History Date Modified Username Field Change ====================================================================== 2010-04-29 07:43 tjaszo New Issue ====================================================================== |