From: Dimitri M. <dm...@bm...> - 2015-05-28 17:38:11
Attachments:
signature.asc
|
Hi all, I have 2 disk-based bacula servers: 1. bacula-5.2.13 on x64 centos 6 with "single drive" vchanger -- i.e. writing one volume at a time. 2. bacula-7.0.5 on x64 centos 7 with single filesystem storage. On both servers job and attribute spooling are on. On #1 I have max concurrent jobs = 10 in director and storage stanzas, and 15 in bacula-sd's storage. I've watched it write 5-6 spool files in parallel while despooling to storage at the same time, works quite fast. On #2 I added "max job spool size = volume size" and "max spool size = volume size * max concurrent jobs" because the spool in on the (smaller) drive. There's only one job getting spooled at a time on #2. So I added max concurrent jobs = 2 to each client (I normally have 2 jobs defined /client) and max concurrent jobs = 2 in jobdefs and I still have only one job spooling. I expect to see at least 2. The problem of course is that here spool-despool is strictly sequential and I get all the downsides of writing stuff twice without the main benefit. Any idea why? Did something change between 5.2 and 7.0? Can it be "Label media = yes" on #2 vs pre-labeled "slots" on #1? Limiting spool size on #2 shouldn't cause this. Something else I'm missing? TIA -- Dimitri Maziuk Programmer/sysadmin BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu |
From: Dimitri M. <dm...@bm...> - 2015-05-28 18:08:37
Attachments:
signature.asc
|
On 05/28/2015 12:37 PM, Dimitri Maziuk wrote: > ... because the spool in on the (smaller) root > drive. PS. Looking at the logs, on #1 all 10 client jobs start within the same couple of minutes after scheduled time and then backupcatalog job starts 2 hours later when they're done. Times are out of sequence and overlapping, e.g. JobId: 1845 Scheduled time: 27-May-2015 21:05:00 Start time: 27-May-2015 21:09:21 End time: 27-May-2015 21:09:50 ... JobId: 1846 Scheduled time: 27-May-2015 21:05:00 Start time: 27-May-2015 21:10:12 End time: 27-May-2015 21:10:42 ... JobId: 1849 Scheduled time: 27-May-2015 21:05:01 Start time: 27-May-2015 21:05:11 End time: 27-May-2015 21:09:38 ... JobId: 1852 Scheduled time: 27-May-2015 21:05:01 Start time: 27-May-2015 21:05:17 End time: 27-May-2015 21:05:27 On #2 I can't find one pair of jobs with overlapping runtimes like the above. All I have is like JobId: 141 Scheduled time: 25-May-2015 20:05:01 Start time: 26-May-2015 23:02:34 End time: 27-May-2015 16:44:12 ... JobId: 142 Scheduled time: 25-May-2015 20:05:01 Start time: 27-May-2015 16:44:12 End time: 27-May-2015 16:44:16 and so on. Strictly sequential with no concurrency. Any idea what gives? -- Dimitri Maziuk Programmer/sysadmin BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu |
From: Ana E. M. A. <emi...@gm...> - 2015-05-29 02:43:46
|
Hi Dimitri, Maybe this makes no sense, but you said "On #1 I have max concurrent jobs = 10 in director and storage stanzas". Is this true for #2? Do you have concurrency enabled in director and storage of #2? Best regards, Ana On Thu, May 28, 2015 at 3:08 PM, Dimitri Maziuk <dm...@bm...> wrote: > On 05/28/2015 12:37 PM, Dimitri Maziuk wrote: > > ... because the spool in on the (smaller) > > root > > > drive. > > PS. Looking at the logs, on #1 all 10 client jobs start within the same > couple of minutes after scheduled time and then backupcatalog job starts > 2 hours later when they're done. Times are out of sequence and > overlapping, e.g. > > JobId: 1845 > Scheduled time: 27-May-2015 21:05:00 > Start time: 27-May-2015 21:09:21 > End time: 27-May-2015 21:09:50 > ... > JobId: 1846 > Scheduled time: 27-May-2015 21:05:00 > Start time: 27-May-2015 21:10:12 > End time: 27-May-2015 21:10:42 > ... > JobId: 1849 > Scheduled time: 27-May-2015 21:05:01 > Start time: 27-May-2015 21:05:11 > End time: 27-May-2015 21:09:38 > ... > JobId: 1852 > Scheduled time: 27-May-2015 21:05:01 > Start time: 27-May-2015 21:05:17 > End time: 27-May-2015 21:05:27 > > On #2 I can't find one pair of jobs with overlapping runtimes like the > above. All I have is like > > JobId: 141 > Scheduled time: 25-May-2015 20:05:01 > Start time: 26-May-2015 23:02:34 > End time: 27-May-2015 16:44:12 > ... > JobId: 142 > Scheduled time: 25-May-2015 20:05:01 > Start time: 27-May-2015 16:44:12 > End time: 27-May-2015 16:44:16 > > and so on. Strictly sequential with no concurrency. Any idea what gives? > > -- > Dimitri Maziuk > Programmer/sysadmin > BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu > > > > ------------------------------------------------------------------------------ > > _______________________________________________ > Bacula-users mailing list > Bac...@li... > https://lists.sourceforge.net/lists/listinfo/bacula-users > > |
From: Dimitri M. <dm...@bm...> - 2015-05-29 14:50:11
|
On 2015-05-28 21:43, Ana Emília M. Arruda wrote: > Hi Dimitri, > > Maybe this makes no sense, but you said "On #1 I have max concurrent > jobs = 10 in director and storage stanzas". Is this true for #2? Do you > have concurrency enabled in director and storage of #2? Yes. I started with identical concurrency settings, when that didn't work on #2 I added extra ones to client and jobdefs. And restarted (everything: there was a kernel upgrade too). The other difference is both have stock postgres: on #1 it's 8.4 I think whereas on #2 it's 9.2. Can't see how that would matter. Thx Dimitri |
From: Ana E. M. A. <emi...@gm...> - 2015-05-30 16:21:47
|
Hi Dimitri, I have this working here. With automatic labeling. When you submit the jobs, which messeges do you see for the jobs in the "status dir" command from bconsole? Usually the messeges there tell us something about why the jobs did not start concurrently and what for is the second job waiting. Just for checking: have you enabled the storage concurrency both on bacula-sd.conf and bacula-dir.conf? Best regards, Ana On Fri, May 29, 2015 at 11:50 AM, Dimitri Maziuk <dm...@bm...> wrote: > On 2015-05-28 21:43, Ana Emília M. Arruda wrote: > > Hi Dimitri, > > > > Maybe this makes no sense, but you said "On #1 I have max concurrent > > jobs = 10 in director and storage stanzas". Is this true for #2? Do you > > have concurrency enabled in director and storage of #2? > > Yes. I started with identical concurrency settings, when that didn't > work on #2 I added extra ones to client and jobdefs. And restarted > (everything: there was a kernel upgrade too). > > The other difference is both have stock postgres: on #1 it's 8.4 I think > whereas on #2 it's 9.2. Can't see how that would matter. > > Thx > Dimitri > > > > > ------------------------------------------------------------------------------ > _______________________________________________ > Bacula-users mailing list > Bac...@li... > https://lists.sourceforge.net/lists/listinfo/bacula-users > |
From: Dimitri M. <dm...@bm...> - 2015-06-01 15:19:22
|
On 2015-05-30 11:21, Ana Emília M. Arruda wrote: > Hi Dimitri, > > I have this working here. With automatic labeling. When you submit the > jobs, which messeges do you see for the jobs in the "status dir" command > from bconsole? Usually the messeges there tell us something about why > the jobs did not start concurrently and what for is the second job waiting. Is there a way to log these? -- I don't sit there at the console when the jobs start after hours. > Just for checking: have you enabled the storage concurrency both on > bacula-sd.conf and bacula-dir.conf? Yes of course. How do I turn up debug on both sd and director? Dimitri |
From: Ana E. M. A. <emi...@gm...> - 2015-06-01 18:18:55
|
On Mon, Jun 1, 2015 at 12:19 PM, Dimitri Maziuk <dm...@bm...> wrote: > On 2015-05-30 11:21, Ana Emília M. Arruda wrote: > > Hi Dimitri, > > > > I have this working here. With automatic labeling. When you submit the > > jobs, which messeges do you see for the jobs in the "status dir" command > > from bconsole? Usually the messeges there tell us something about why > > the jobs did not start concurrently and what for is the second job > waiting. > > Is there a way to log these? -- I don't sit there at the console when > the jobs start after hours. > An admin job, runscript directive with a "console = status dir", scheduled to start a few minutes after your backups should tell you this. > > > Just for checking: have you enabled the storage concurrency both on > > bacula-sd.conf and bacula-dir.conf? > > Yes of course. > > How do I turn up debug on both sd and director? > On 7.0.5 version there is a setdebug command from bconsole: *help setdebug Command Description ======= =========== setdebug Sets debug level Arguments: level=<nn> trace=0/1 client=<client-name> | dir | storage=<storage-name> | all When at a prompt, entering a period cancels the command. > > Dimitri > > > > > ------------------------------------------------------------------------------ > _______________________________________________ > Bacula-users mailing list > Bac...@li... > https://lists.sourceforge.net/lists/listinfo/bacula-users > |
From: Dimitri M. <dm...@bm...> - 2015-06-02 18:57:41
Attachments:
signature.asc
|
On 06/01/2015 01:18 PM, Ana Emília M. Arruda wrote: >>> I have this working here. With automatic labeling. When you submit the >>> jobs, which messeges do you see for the jobs in the "status dir" command A-ha, it says one job is running and 10 "is waiting on max Storage jobs". >>> Just for checking: have you enabled the storage concurrency both on >>> bacula-sd.conf and bacula-dir.conf? >> >> Yes of course. ... but it turns out I had it in Storage and Device blocks in bacula-sd.conf and in Director block in bacula-dir.conf. I was missing it from the Storage block in bacula-dir.conf. We'll see if that was it, I'll let you know after the next run. Thanks for the pointer, -- Dimitri Maziuk Programmer/sysadmin BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu |
From: Dimitri M. <dm...@bm...> - 2015-06-04 17:42:01
Attachments:
signature.asc
|
On 06/02/2015 01:57 PM, Dimitri Maziuk wrote: > ... but it turns out I had it in Storage and Device blocks in > bacula-sd.conf and in Director block in bacula-dir.conf. I was missing > it from the Storage block in bacula-dir.conf. And now I have 10 short "etc and crontabs" jobs completed in the 1st minute and 2 large fileserver jobs happily spooling in parallel. The entire batch should take about as long as the largest of those two (give or take despooling), which is what I was aiming for. So, pay attention when editing the configs and make sure you've enabled concurrency in 4 places: - Storage stanza in bacula-sd.conf - Device stanza in bacula-sd.conf - Storage stanza in bacula-dir.conf - Director stanza in bacula-dir.conf Specifically, "Storage" is in both config files. Thank you Ana, -- Dimitri Maziuk Programmer/sysadmin BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu |
From: Ana E. M. A. <emi...@gm...> - 2015-06-04 19:35:12
|
You're welcome Dimitri. Thanks for the feedback. On Thu, Jun 4, 2015 at 2:41 PM, Dimitri Maziuk <dm...@bm...> wrote: > On 06/02/2015 01:57 PM, Dimitri Maziuk wrote: > > > ... but it turns out I had it in Storage and Device blocks in > > bacula-sd.conf and in Director block in bacula-dir.conf. I was missing > > it from the Storage block in bacula-dir.conf. > > And now I have 10 short "etc and crontabs" jobs completed in the 1st > minute and 2 large fileserver jobs happily spooling in parallel. The > entire batch should take about as long as the largest of those two (give > or take despooling), which is what I was aiming for. > > So, pay attention when editing the configs and make sure you've enabled > concurrency in 4 places: > - Storage stanza in bacula-sd.conf > - Device stanza in bacula-sd.conf > - Storage stanza in bacula-dir.conf > - Director stanza in bacula-dir.conf > > Specifically, "Storage" is in both config files. > > Thank you Ana, > -- > Dimitri Maziuk > Programmer/sysadmin > BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu > > > > ------------------------------------------------------------------------------ > > _______________________________________________ > Bacula-users mailing list > Bac...@li... > https://lists.sourceforge.net/lists/listinfo/bacula-users > > |