February 1st, 2010
The original ActionOnPurge feature, released in 5.0.0, was able to truncate volumes when Bacula decided to purge them. It appears that the code wasn’t very functionnal, and in many cases, the truncate operation was just impossible to do.
I’ve commited a patch this evening to re-implement this feature as a new command line. It is designed to work in a Console RunScript after the Catalog job or in a custom script. You can now do the following to use this feature:
# Tell to Bacula that this volume can be truncated on purge (or use Pool->ActionOnPurge=Truncate)
*update volume=xxx actiononpurge=Truncate
# purge the volume by hand, or wait a bit
*purge volume=xxx
# truncate from all pool, all volume on File storage
*purge volume action=truncate storage=File allpools
# by default, action=all
*purge volume action storage=File pool=Default
This is possible to specify the volume name, the media type, the pool, the storage, etc…
Job {
Name = CatalogBackup
...
RunScript {
RunsWhen=After
RunsOnClient=No
Console = "purge volume action=all allpools storage=File"
}
}
Ideas and suggestions are welcome 
Eric Bollengier
Tags: 5.0.x
Posted in Design Specification | No Comments »
January 28th, 2010
Unfortunately, it appears that the ActionOnPurge feature is broken and
*dangerous*. The usage of this command should be avoided in any cases!
We will try to fix it for the next release, but the amount of work is very
important.
I will update the manual in this way, and the current code for Branch-5.0 will
disable this feature.
Thanks for your testing.
Unfortunately, it appears that the ActionOnPurge feature is badly broken. The usage of this command should be avoided until we provide a fix.
We will try to re-implement it for the next release, but the amount of work is very important. The manual is up to date, and the current code for Branch-5.0 disabled this feature.
Eric Bollengier
Tags: 5.0.0 Bug
Posted in Projects | No Comments »
January 21st, 2010
I (Eric Bollengier) will be in Brussels on 6-7th of February for the FOSDEM. No official talk this year, but i will be happy to talk with Bacula users and to enroll developers
Posted in Event | No Comments »
January 21st, 2010
We should be able to release Bacula version 5.0.0 early next week, this release will add new features and improve stability.
Many projects are now completed, more than 21 persons have contributed to this release with patches, and even more with bug reports. Thanks to all of you.
657 files changed, 104036 insertions(+), 78504 deletions(-)
Most important items of this release are :
- Project 5: Truncate volume after purge
- Project 6: File Deduplication using Base Jobs
- Project 10: Restore from Multiple Storage Daemons
- Project 11: AllowCompression per Device
- Project 23: Add Maximum Concurent Jobs for Devices to balance load between drives
- Add Accurate Fileset Options to configure accurate detection. Can use checksum verification for example.
- Allow FD to keep root read but drop write permission (CAP)
- Add Tab-completion for Bconsole when using Readline
- Secure handling of passwords for catalog backup
- Add Bvfs API to query the catalog without building a memory tree
- Add new speed test to btape program
- Add new Bat screens (Autochanger content, Job view, Media view, …)
- Windows version of Bat
- Added Spanish and Ukraine Bacula translation
- New bconsole -u nnn option to have timeout on commands
- Allow Migrate, Copy, and Virtual Full to read and write to the same Pool
- show disabled — shows disabled Jobs
- Add Pool File and Job retention periods (take precedence over Client)
- Many ACL improvements
- Level added to FD status Job entry
- Allow to turn off/on Block Checksum per device
- Implement lock manager with bad order protection
Eric Bollengier
Tags: 5.0.x, Release
Posted in Projects | No Comments »
January 11th, 2010
In the next release, the catalog version will go from 11 to 12. Key points in this upgrade are :
- Fix SQLite and PostgreSQL indexes on File table
- Add Comment field in Job table
- Add index on BaseFiles
- Integration of Bresto/Brestore tables (renamed as PathVisibility, PathHierarchy)
- Add HasCache field in Job table (replacement of brestore_knownjobid)
- Suppression of Copy and Stripe fields on JobMedia
- Add Severity field on Status table
To use latest version, you will have to run the update_bacula_tables script as ususal. The upgrade should be quite fast on MySQL db backend. On PostgreSQL (and SQLite), as we replace an index, the creation time depends on the File size table, but you can edit the script to create it later with the CONCURENTLY keyword during production.
Eric Bollengier
Tags: Catalog
Posted in Database | No Comments »
November 28th, 2009
I started last week with the implementation of so called catalog plugins which in the end should allow the current sqlite, sqlite3, mysql, postgresql and dbi catalog interfaces to be compiled as shared objects. In the current setup you can only enable ONE database backend when compiling the server, if you enable more then one you will be bitten by the fact that you end up with duplicate symbols. The idea is to allow plugins but also allow static build servers with only one backend like it is currently. The design effort is now mostly aimed at UNIX.
The first stab at things already showed me that it will be quite some work as there is a lot of code which is not really written at the moment to allow multiple backends. Currently I have some code which is not much more then an exploded bacula server from which I just ripped current macros and written some preliminary glue code which will in the future dispatch to either a dynamic symbol in a plugin or for the static version glue the software into one piece. It doesn’t even compile but is more or less a rapid prototyping attempt to see how to get to a piece of code that compiles into a module that is mostly database independent. From that moment on we can start on converting the current database code into plugins which will interact with the glue code.
As you can see from the info above it is quite an effort to get anything usable, so don’t expect anything usable before the new year.
The current idea is to integrate a first version of catalog plugins in the 3.3.x development stream which starts after 3.2.0 is released.
Posted in Database, Design Specification | 2 Comments »
November 27th, 2009
In this article, we will try to analyse PostgreSQL 8.4 release notes to see if we can find good reasons to migrate the Catalog from 8.x to 8.4 version.
Anti-joins and Semi-joins
PostgreSQL 8.4 add new formalized JOIN methods for inequality joins. It adds better performance for EXISTS / NOT EXISTS queries that are used in Batch mode to fill Path and Filename tables. This is an excellent news for us! It was quite fast before, now it should be very fast.
Hash for DISTINCT/UNION
Previously Postgres was using a sort+unique algo in queries using DISTINCT. Bacula uses this function to select files in Accurate mode and for restore. This is also a good news!
Hash improvements
They have faster algorithms, faster hash indexes and they have optimized of HASH joins particularly around large joins. We use them in many parts of Bacula, it sounds very good.
DBA parameter, the Free Space Map (FSM)
This object is used to store and share the free disk pages list in PostgreSQL core. Because Bacula recycles a good number of pages when purging jobs, if this parameter wasn’t well configured, it was possible to lead to disk space leak. Now this FSM is dynamically tuned, and won’t cause problems anymore.
Visibility Map
This object tracks pages that are “visible to all transactions” in bitmap, the VACUUM process can now skip pages that are clean, and it should speed up this maintenance process. Nice
Other improvements
The 8.4 version has other improvements like parallel restore or better 64bit integer usage, but they are less important for us.
Conclusion
As we can see, Bacula is impacted by almost every new performance enhancements, so you should consider to use or upgrade your PostgreSQL to this version. Thanks to the PostgreSQL team!
References
8.4 Release Notes
Eric Bollengier
Tags: Performance, Postgresql
Posted in Performance, PostgreSQL | 2 Comments »
November 12th, 2009
The Batch Mode, used to update the catalog with backuped files, is designed to be able to handle multiple jobs in parallel (at least on transactional databases). But i discovered this morning that this is not the case with Innodb… By default, if you are doing massive insertion on a table that contains an AUTO_INCREMENT field, MySQL uses a lock to ensure that the increment order will be optimal.
http://dev.mysql.com/doc/refman/5.1/en/innodb-auto-increment-handling.html
To be able to use the Batch mode at full speed with Innodb, you need to use at least MySQL 5.1.21, and add the following to your my.cnf config file.
innodb_autoinc_lock_mode = 2
Warning, this mode isn’t compatible with cluster and replication features. Please read MySQL documentation for more information.
Eric Bollengier
Tags: Batch mode, Gotcha, Mysql, Performance
Posted in Database, Mysql, Performance | No Comments »
November 10th, 2009
I just wanted to let you know that Kern will be giving a live briefing on Thursday, 03/Dec/2009 at 16:00 UMT+1 (Central Europe Time).
Kern will talk about Bacula 3.2’s new features, special features planned for the next release, future development directions, hot topics from the mail lists, and will also answer participants questions.
You can subscribe here.
Tags: Briefing, Webinar
Posted in Design Specification | No Comments »
November 5th, 2009
The basejob feature is now merged in the current master code. Feel free to try it, and report feedbacks. For documentation, you can take a look here.
Eric Bollengier
Tags: BaseJob
Posted in Deduplication | No Comments »