Menu
▾
▴
[Bacula-bugs] [bacula 0001079]: File selection via bconsole takes a long time when 1, 400.000 files selected for restore
|
From: <bac...@li...> - 2008-04-21 13:16:35
|
A NOTE has been added to this issue. ====================================================================== http://bugs.bacula.org/view.php?id=1079 ====================================================================== Reported By: Juergen Kuri Assigned To: ====================================================================== Project: bacula Issue ID: 1079 Category: bconsole Reproducibility: always Severity: major Priority: normal Status: feedback ====================================================================== Date Submitted: 04-18-2008 15:59 BST Last Modified: 04-21-2008 14:15 BST ====================================================================== Summary: File selection via bconsole takes a long time when 1,400.000 files selected for restore Description: I reported this at 2008-03-20 in the user forum but got no solution, please see URL: http://marc.info/?l=bacula-users&m=120602534417470&w=2 Another user encountered the same problem a few days later, please see URL: http://marc.info/?l=bacula-users&m=120764422523806&w=2 too. I tried to make the same with 14,133,996 job files. The after typing "done" the director process ran three days and at last the bacula-dir process crashed. Thank you, Jürgen ====================================================================== ---------------------------------------------------------------------- Dan Langille - 04-18-08 16:13 ---------------------------------------------------------------------- What version of PostgreSQL? What is the output of these commands from within 'psql bacula' \d file \d filename Also, is it feasible/practical for me to get a copy of your database for testing? I ask because query tuning is something I can do best if I have a good example like this. ---------------------------------------------------------------------- kern - 04-18-08 16:25 ---------------------------------------------------------------------- This is not a bug, but a design limitation. When doing the tree selection of files, everything is in memory, and that requires a lot of time and memory. Anything with more than about a million files is impractical. 14 Million is impossible and will probably crash Bacula with an out of memory condition. I believe that the Restore manual indicates a few alternatives. I think it would be worthwhile to send a copy of your database to Dan ... ---------------------------------------------------------------------- Juergen Kuri - 04-21-08 11:25 ---------------------------------------------------------------------- Hello, thank you for your efforts. The bz2-compressed dump file of the catalogue database has a size of 773 MB. How can I transfer this file? The release of the Postgres database is 8.3.1 Jürgen ---------------------------------------------------------------------- Juergen Kuri - 04-21-08 11:31 ---------------------------------------------------------------------- bacula-# \d file Table "public.file" Column | Type | Modifiers ------------+---------+------------------------------------------------------- fileid | integer | not null default nextval('file_fileid_seq'::regclass) fileindex | integer | not null default 0 jobid | integer | not null pathid | integer | not null filenameid | integer | not null markid | integer | not null default 0 lstat | text | not null md5 | text | not null Indexes: "file_pkey" PRIMARY KEY, btree (fileid) "file_fp_idx" btree (filenameid, pathid) "file_jobid_idx" btree (jobid) bacula-# \d filename Table "public.filename" Column | Type | Modifiers ------------+---------+--------------------------------------------------------------- filenameid | integer | not null default nextval('filename_filenameid_seq'::regclass) name | text | not null Indexes: "filename_pkey" PRIMARY KEY, btree (filenameid) "filename_name_idx" btree (name) ---------------------------------------------------------------------- Dan Langille - 04-21-08 12:49 ---------------------------------------------------------------------- Can you put that file on a web server and let me download it to my server? ---------------------------------------------------------------------- Juergen Kuri - 04-21-08 13:54 ---------------------------------------------------------------------- Dear Dan, I prepared a possibility for you in order to download the dump file 'bacula.sql.bz2'. Please follow the URL: https://smartdrive.web.de/guest?path=Neue%20Dateianlagen%20von%20juku&token=79C12FFC8A81B4CE then you will encounter german web page, press the bottom "SmartDrive Gastzugang starten", then in the pop up box you will encounter the download facility of the dump file. This application allows me to permit you only just one time to download this file. So if the download fails or aborts and you cannot download another time, let me know. Thank you, Jürgen ---------------------------------------------------------------------- Dan Langille - 04-21-08 14:15 ---------------------------------------------------------------------- FYI, that dump is now public (our bug database is public) so if you don't want others to grab it, please move it to another URL and email me (dan at langille.org). Regardless, I'll start downloading it now. If I have to start again later, no worries. Issue History Date Modified Username Field Change ====================================================================== 04-18-08 15:59 Juergen Kuri New Issue 04-18-08 16:00 Juergen Kuri Issue Monitored: Juergen Kuri 04-18-08 16:13 Dan Langille Note Added: 0003308 04-18-08 16:25 kern Note Added: 0003309 04-18-08 16:25 kern Status new => resolved 04-18-08 16:25 kern Resolution open => not a bug 04-21-08 11:25 Juergen Kuri Status resolved => feedback 04-21-08 11:25 Juergen Kuri Resolution not a bug => reopened 04-21-08 11:25 Juergen Kuri Note Added: 0003311 04-21-08 11:31 Juergen Kuri Note Added: 0003312 04-21-08 12:49 Dan Langille Note Added: 0003314 04-21-08 13:54 Juergen Kuri Note Added: 0003315 04-21-08 14:15 Dan Langille Note Added: 0003316 ====================================================================== |