From: Matthew Macdonald-W. (lists) <li...@gr...> - 2012-03-09 16:51:28
|
Hi all, I'm working on a platform at the moment which has several hundred servers (potentially thousands eventually) being backed up across various data centers using Bacula (managed via Chef). We've got a prototype in place and it seems to work well, however before I deploy I'd like to try and understand a little more about the system resources required to run bacula. From previous email threads on this list, I've come to the conclusion that the primary bottle neck is the Back-end database, not Bacula itself. We have taken the design decision to place both the Director and the MySQL instance on the same server - we figure that if the database or the bacula-daemons are down, we can't backup either way. This server has 48G RAM and 10K SAS Disks so there is some flexibility surrounding how it is configured. My plan was to create an 8G SWAP partition and then have /var/bacula-backups on one HW RAID-5 Array and /var/lib/mysql + OS on a second HW RAID-1 array (possibly even RAID-0 if it gives us more performance!) - from there I would concentrate on MySQL turning as opposed to anything else. Does anyone have any thoughts on the above, or things I might have missed? Thanks in advance, Matt P.S. If anyone knows of large-scale bacula (not enterprise) installs, I'd be v. interested in hearing from them! M. |
From: Kevin K. (s. <sub...@kk...> - 2012-03-09 18:39:38
|
The one additional thought I have is to keep the backup file set as small as possible; network traffic and the length of your backup window will be a concern. If you have hundreds or thousands of servers, odds are that they are all identical - especially if they are managed by chef. So you don't need to back up the operating system itself, just back up whatever data varies from one server to the next. Generally, the less backup traffic you have per server, the better. -----Original message----- From:Matthew Macdonald-Wallace (lists) <li...@gr...> Sent:Fri 09-03-2012 08:55 Subject:[Bacula-users] Scaling Bacula To:bac...@li...; Hi all, I'm working on a platform at the moment which has several hundred servers (potentially thousands eventually) being backed up across various data centers using Bacula (managed via Chef). We've got a prototype in place and it seems to work well, however before I deploy I'd like to try and understand a little more about the system resources required to run bacula. From previous email threads on this list, I've come to the conclusion that the primary bottle neck is the Back-end database, not Bacula itself. We have taken the design decision to place both the Director and the MySQL instance on the same server - we figure that if the database or the bacula-daemons are down, we can't backup either way. This server has 48G RAM and 10K SAS Disks so there is some flexibility surrounding how it is configured. My plan was to create an 8G SWAP partition and then have /var/bacula-backups on one HW RAID-5 Array and /var/lib/mysql + OS on a second HW RAID-1 array (possibly even RAID-0 if it gives us more performance!) - from there I would concentrate on MySQL turning as opposed to anything else. Does anyone have any thoughts on the above, or things I might have missed? Thanks in advance, Matt P.S. If anyone knows of large-scale bacula (not enterprise) installs, I'd be v. interested in hearing from them! M. ------------------------------------------------------------------------------ Virtualization & Cloud Management Using Capacity Planning Cloud computing makes use of virtualization - but cloud computing also focuses on allowing computing to be delivered as a service. http://www.accelacomm.com/jaw/sfnl/114/51521223/ _______________________________________________ Bacula-users mailing list Bac...@li... https://lists.sourceforge.net/lists/listinfo/bacula-users |
From: Adrian R. <bac...@li...> - 2012-03-10 21:36:10
|
Hi Matt, On Fri, Mar 09, 2012 at 04:11:17PM +0000, Matthew Macdonald-Wallace (lists) wrote: > From previous email threads on this list, I've come to the conclusion > that the primary bottle neck is the Back-end database, not Bacula > itself. This is my experience as well, though bacula-sd liekes to have quite some CPU and eventually RAM as well, depending on your jobs. > We have taken the design decision to place both the Director and the > MySQL instance on the same server - we figure that if the database or > the bacula-daemons are down, we can't backup either way. This server > has 48G RAM and 10K SAS Disks so there is some flexibility surrounding > how it is configured. I suggest you plan and do you Director and *Postgresql* instance on the same server. There are a few inices in Baculas database layout that contain other inices. MySQL has to do seperate indices for those, postgres can use a single index for these. This results in way less writes and compared to most other applications that use databases, Bacula maily writes. I moved from 4GB + MyISAM to 16GB + MyISAM to 16GB + InnoDB and now I am happy at 8GB + Postgres. I always changed the database/backend when backups didn't finish anymore. bacula=# select count(*) from path; count -------- 855516 bacula=# select count(*) from filename; count --------- 4772037 bacula=# select count(*) from file; count ----------- 260460301 bacula=# select count(*) from jobmedia; count ------- 35267 I plan and keep the filelists within the database for 13 months, the problems started at month 3 and I switched database backends every month till I reached the current setup. This is now running for 6 month. > second HW RAID-1 array (possibly even RAID-0 if it gives us more > performance!) - from there I would concentrate on MySQL turning as > opposed to anything else. Make sure you have cache ram on your raid controller and a battery backup unit installed. MySQL and Postgres like to write in sync. With BBU+cache the write is completed as soon as the controller has the data, no need to wait for disks. I doubt RAID0 would gain you much if any. Regards, Adrian -- LiHAS - Adrian Reyer - Hessenwiesenstraße 10 - D-70565 Stuttgart Fon: +49 (7 11) 78 28 50 90 - Fax: +49 (7 11) 78 28 50 91 Mail: li...@li... - Web: http://lihas.de Linux, Netzwerke, Consulting & Support - USt-ID: DE 227 816 626 Stuttgart |
From: Adrian R. <bac...@li...> - 2012-03-10 22:10:39
|
On Sat, Mar 10, 2012 at 10:35:49PM +0100, Adrian Reyer wrote: > same server. There are a few inices in Baculas database layout that > contain other inices. MySQL has to do seperate indices for those, That is 'indices', other typos are somewhat easy to understand. Regards, Adrian -- LiHAS - Adrian Reyer - Hessenwiesenstraße 10 - D-70565 Stuttgart Fon: +49 (7 11) 78 28 50 90 - Fax: +49 (7 11) 78 28 50 91 Mail: li...@li... - Web: http://lihas.de Linux, Netzwerke, Consulting & Support - USt-ID: DE 227 816 626 Stuttgart |
From: Matthew Macdonald-W. (lists) <li...@gr...> - 2012-03-13 10:22:49
|
Thanks Adrian, that's really useful. Kind regards, Matt On 10/03/2012 21:35, Adrian Reyer wrote: > Hi Matt, > > On Fri, Mar 09, 2012 at 04:11:17PM +0000, Matthew Macdonald-Wallace > (lists) wrote: >> From previous email threads on this list, I've come to the >> conclusion >> that the primary bottle neck is the Back-end database, not Bacula >> itself. > > This is my experience as well, though bacula-sd liekes to have quite > some CPU and eventually RAM as well, depending on your jobs. > >> We have taken the design decision to place both the Director and the >> MySQL instance on the same server - we figure that if the database >> or >> the bacula-daemons are down, we can't backup either way. This >> server >> has 48G RAM and 10K SAS Disks so there is some flexibility >> surrounding >> how it is configured. > > I suggest you plan and do you Director and *Postgresql* instance on > the > same server. There are a few inices in Baculas database layout that > contain other inices. MySQL has to do seperate indices for those, > postgres can use a single index for these. This results in way less > writes and compared to most other applications that use databases, > Bacula maily writes. > I moved from 4GB + MyISAM to 16GB + MyISAM to 16GB + InnoDB and now I > am > happy at 8GB + Postgres. I always changed the database/backend when > backups didn't finish anymore. > > bacula=# select count(*) from path; > count > -------- > 855516 > > bacula=# select count(*) from filename; > count > --------- > 4772037 > > bacula=# select count(*) from file; > count > ----------- > 260460301 > > bacula=# select count(*) from jobmedia; > count > ------- > 35267 > > I plan and keep the filelists within the database for 13 months, the > problems started at month 3 and I switched database backends every > month till I reached the current setup. This is now running for 6 > month. > >> second HW RAID-1 array (possibly even RAID-0 if it gives us more >> performance!) - from there I would concentrate on MySQL turning as >> opposed to anything else. > > Make sure you have cache ram on your raid controller and a battery > backup unit installed. MySQL and Postgres like to write in sync. With > BBU+cache the write is completed as soon as the controller has the > data, > no need to wait for disks. I doubt RAID0 would gain you much if any. > > Regards, > Adrian -- Matthew Macdonald-Wallace Green And Secure IT Limited 3 Maddox Close, Osbaston, Monmouth, NP25 3BG Registered in England and Wales. Company Number: 06769520 |
From: Josh F. <jf...@pv...> - 2012-03-13 14:35:59
|
On 3/10/2012 4:35 PM, Adrian Reyer wrote: > Hi Matt, > > On Fri, Mar 09, 2012 at 04:11:17PM +0000, Matthew Macdonald-Wallace (lists) wrote: > >> second HW RAID-1 array (possibly even RAID-0 if it gives us more >> performance!) - from there I would concentrate on MySQL turning as >> opposed to anything else. > Make sure you have cache ram on your raid controller and a battery > backup unit installed. MySQL and Postgres like to write in sync. With > BBU+cache the write is completed as soon as the controller has the data, > no need to wait for disks. I doubt RAID0 would gain you much if any. Another possibility is using SSD (Intel X25-E, etc.) for just the database storage, as it is very, very good at random i/o. |
From: Dietz P. <di...@ro...> - 2012-03-18 14:19:54
Attachments:
signature.asc
|
Adrian Reyer: > Make sure you have cache ram on your raid controller and a battery > backup unit installed. MySQL and Postgres like to write in sync. With > BBU+cache the write is completed as soon as the controller has the data, > no need to wait for disks. I doubt RAID0 would gain you much if any. Using a SSD for the database storage should speed up the setup considerably. In my experience, they feel a bit like a in-memory database ;-). RAID0 for the spinning disks won't give much of a win, because database stuff is normally more bound to seek performance, than to io bandwidth, and only the latter will improve in that setup. regards, Dietz -- To be a bug is a dumb thing, a silly and a bump thing, but to be a bug is something, and You're not a thing at all. |