Bacula

On 1/17/2013 11:06 AM, Ruth Ivimey-Cook wrote:
> Hi,
>
> I am sometimes getting these errors in my bacula backups:
> Fatal error: device.c:192 Catastrophic error. Cannot write overflow block to device "DiskStorage-drive-0"
> and it is more likely on the larger volume backups. It seemingly 
> results from bacula trying to write an additional block to a disk 
> drive that is already 100% full. How can I stop bacula from believing 
> this is a valid thing to do?
>

You don't. The trick is to define a maximum volume size and number of 
volumes on the drive so that it is impossible to reach 100% of the 
physical drive's capacity. This will prevent the i/o error, and Bacula 
will instead hit end of volume and seek another volume. Of course, if no 
existing volumes can be recycled yet, then there simply isn't enough 
space on the drive. In that case, it is easy to add another drive to an 
existing autochanger, since vchanger allows for multiple simultaneous 
"magazine" drives.

> Background:  I have bacula setup on my local network to backup a file 
> server and a number of workstations. The file server is also the 
> bacula director and is running Fedora 15 and 
> "bacula-common-5.0.3-28.fc15.x86_64". Bacula is writing backups to an 
> iSCSI disk group (not array) over ethernet; there are 6 disks of 1TB 
> to 2TB size and these are managed using "vchanger" 0.8.6, with 6 
> magazines each with 24 virtual volumes. The file server has 3.5TB of 
> files and other workstations add about another 1TB.
>
> More-complete log:
>
>     17-Jan 14:49 helva-sd JobId 3417: Recycled volume "DiskPool1_0006_0017" on device "DiskStorage-drive-0" (/var/spool/bacula/vchanger/0/drive0), all previous data lost.
>     17-Jan 14:49 helva-sd JobId 3417: New volume "DiskPool1_0006_0017" mounted on device "DiskStorage-drive-0" (/var/spool/bacula/vchanger/0/drive0) at 17-Jan-2013 14:49.
>     17-Jan 14:49 helva-sd JobId 3417: End of Volume "DiskPool1_0006_0017" at 0:216 on device "DiskStorage-drive-0" (/var/spool/bacula/vchanger/0/drive0). Write of 64512 bytes got 3879.
>     17-Jan 14:49 helva-sd JobId 3417: End of medium on Volume "DiskPool1_0006_0017" Bytes=217 Blocks=0 at 17-Jan-2013 14:49.
>     17-Jan 14:49 helva-sd JobId 3417: 3307 Issuing autochanger "unload slot 89, drive 0" command.
>     17-Jan 14:49 helva-dir JobId 3417: Using Volume "DiskPool1_0006_0018" from 'Scratch' pool.
>     17-Jan 14:49 helva-sd JobId 3417: 3301 Issuing autochanger "loaded? drive 0" command.
>     17-Jan 14:49 helva-sd JobId 3417: 3302 Autochanger "loaded? drive 0", result: nothing loaded.
>     17-Jan 14:49 helva-sd JobId 3417: 3304 Issuing autochanger "load slot 90, drive 0" command.
>     17-Jan 14:49 helva-sd JobId 3417: 3305 Autochanger "load slot 90, drive 0", status is OK.
>     17-Jan 14:49 helva-sd JobId 3417: Recycled volume "DiskPool1_0006_0018" on device "DiskStorage-drive-0" (/var/spool/bacula/vchanger/0/drive0), all previous data lost.
>     17-Jan 14:49 helva-sd JobId 3417: New volume "DiskPool1_0006_0018" mounted on device "DiskStorage-drive-0" (/var/spool/bacula/vchanger/0/drive0) at 17-Jan-2013 14:49.
>     17-Jan 14:49 helva-sd JobId 3417: End of Volume "DiskPool1_0006_0018" at 0:216 on device "DiskStorage-drive-0" (/var/spool/bacula/vchanger/0/drive0). Write of 64512 bytes got 3879.
>     17-Jan 14:49 helva-sd JobId 3417: Fatal error: device.c:192 Catastrophic error. Cannot write overflow block to device "DiskStorage-drive-0" (/var/spool/bacula/vchanger/0/drive0). ERR=No space left on device17-Jan 14:49 helva-fd JobId 3417: Error: bsock.c:393 Write error sending 65562 bytes to Storage daemon:helva.cam.ivimey.org:9103: ERR=Connection reset by peer
>     17-Jan 14:49 helva-fd JobId 3417: Fatal error: backup.c:1024 Network send error to SD. ERR=Connection reset by peer
>     17-Jan 14:49 helva-sd JobId 3417: Fatal error: device.c:192 Catastrophic error. Cannot write overflow block to device "DiskStorage-drive-0" (/var/spool/bacula/vchanger/0/drive0). ERR=No space left on device17-Jan 14:49 helva-sd JobId 3417: Fatal error: device.c:192 Catastrophic error. Cannot write overflow block to device "DiskStorage-drive-0" (/var/spool/bacula/vchanger/0/drive0). ERR=No space left on device17-Jan 14:49 helva-sd JobId 3417: Fatal error: device.c:192 Catastrophic error. Cannot write overflow block to device "DiskStorage-drive-0" (/var/spool/bacula/vchanger/0/drive0). ERR=No space left on device17-Jan 14:49 helva-sd JobId 3417: Fatal error: device.c:192 Catastrophic error. Cannot write overflow block to device "DiskStorage-drive-0" (/var/spool/bacula/vchanger/0/drive0). ERR=No space left on device17-Jan 14:49 helva-sd JobId 3417: Job write elapsed time = 14:30:55, Transfer rate = 12.06 M Bytes/second
>     17-Jan 14:49 helva-dir JobId 3417: Error: Bacula helva-dir 5.0.3 (04Aug10): 17-Jan-2013 14:49:32
>        Build OS:               x86_64-redhat-linux-gnu redhat
>        JobId:                  3417
>        Job:                    Helva_Home.2013-01-17_00.17.26_23
>        Backup Level:           Full
>        Client:                 "helva-fd" 5.0.3 (04Aug10) x86_64-redhat-linux-gnu,redhat,
>        FileSet:                "Home" 2010-12-07 13:37:32
>        Pool:                   "Normal-Full-18w" (From Job FullPool override)
>        Catalog:                "MyCatalog" (From Client resource)
>        Storage:                "DiskStorage" (From command line)
>        Scheduled time:         17-Jan-2013 00:17:26
>        Start time:             17-Jan-2013 00:17:28
>        End time:               17-Jan-2013 14:49:32
>        Elapsed time:           14 hours 32 mins 4 secs
>        Priority:               12
>        FD Files Written:       162,621
>        SD Files Written:       162,621
>        FD Bytes Written:       630,336,973,872 (630.3 GB)
>        SD Bytes Written:       630,355,165,137 (630.3 GB)
>        Rate:                   12046.8 KB/s
>        Software Compression:   7.7 %
>        VSS:                    no
>        Encryption:             no
>        Accurate:               no
>        Volume name(s):         DiskPool1_0006_0003|DiskPool1_0001_0020|DiskPool1_0004_0013|DiskPool1_0004_0014|DiskPool1_0004_0023|DiskPool1_0004_0024|DiskPool1_0006_0017|DiskPool1_0006_0018
>        Volume Session Id:      417
>        Volume Session Time:    1351720132
>        Last Volume Bytes:      217 (217 B)
>        Non-fatal FD errors:    1
>        SD Errors:              1
>        FD termination status:  Error
>        SD termination status:  Error
>        Termination:            *** Backup Error ***
>
>
>
> -- 
> Software Manager & Engineer
> Blog:http://www.ivimey.org/blog
> LinkedIn:http://uk.linkedin.com/in/ruthivimeycook/  
>
>
> ------------------------------------------------------------------------------
> Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS,
> MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current
> with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft
> MVPs and experts. ON SALE this month only -- learn more at:
> http://p.sf.net/sfu/learnmore_122712
>
>
> _______________________________________________
> Bacula-users mailing list
> Bacula-users@...
> https://lists.sourceforge.net/lists/listinfo/bacula-users

On 2013-01-17 11:06, Ruth Ivimey-Cook wrote:
> Hi,
>
>  I am sometimes getting these errors in my bacula backups:
>
> Fatal error: device.c:192 Catastrophic error. Cannot write overflow
> block to device "DiskStorage-drive-0"
>  and it is more likely on the larger volume backups. It seemingly
> results from bacula trying to write an additional block to a disk
> drive that is already 100% full. How can I stop bacula from believing
> this is a valid thing to do?

Disk space is outside the scope of the Bacula project. It is the 
responsibility
of the sysadmin to manage disk space.

The other post mentioned how to restrict a Pool to a maximum size per 
Volume
and a maximum number of Volume per Pool.

-- 
Dan Langille - http://langille.org/

On 1/17/2013 2:10 PM, Ruth Ivimey-Cook wrote:
> Josh Fisher wrote:
>> You don't.
> I find it very strange that returning "device full" from a volume 
> write can reasonably be interpreted as "device not quite full".
>> The trick is to define a maximum volume size and number of volumes on 
>> the drive so that it is impossible to reach 100% of the physical 
>> drive's capacity. This will prevent the i/o error, and Bacula will 
>> instead hit end of volume and seek another volume. Of course, if no 
>> existing volumes can be recycled yet, then there simply isn't enough 
>> space on the drive. In that case, it is easy to add another drive to 
>> an existing autochanger, since vchanger allows for multiple 
>> simultaneous "magazine" drives.
> I don't understand how to do this then without defining the number of 
> volumes so low that I waste huge amounts of space on the drives as a 
> matter of course.

One way is to partition the drives. Keeping volumes of the same size on 
the same partition allows specifying the exact number of volumes. Each 
partition is a magazine, and any number of partitions can be used 
simultaneously. For example, break a 1 TB drive into two partitions, one 
200 GB partition holding 10 volumes in a pool with a max volume size of 
~20 GB for incremental jobs, and an 800 GB partition holding 8 volumes 
in a pool with max volume size of 100 GB for full jobs. Etc.

>
> A little more detail about what I'm doing:
>
>   * Some backups are assigned longer retention times than others -
>     e.g. some full backups live for a year, some incrs live for just 3
>     months.
>   * I have various max volume sizes from 20GB to 400GB, assigned to
>     each file pool depending on the likely size of a backup (e.g.
>     incrs are likely smaller than full) so that a volume will expire
>     in a reasonable time - I don't want 100GB of backups to be kept
>     alive (and using space) because they are in the same volume as
>     more recent backups that haven't expired yet.
>   * I have set up 24 volumes per disk so that, should the volumes be
>     shorter 90GB ones, I don't (on average) run out of volumes too
>     quickly.
>   * The result is that most disks are reasonably full most of the
>     time, which is good.
>
> To be honest, I wish Bacula had a "disk mode" in which the concept of 
> volumes was mostly eliminated: devices had backup pools and backups 
> within them and it would be backups that were recycled. It would make 
> much more sense for a random-access medium.

True, but Bacula must also work with tape drives, and that would be a 
very extensive rewrite.

>
> Would an alternative solution be to adapt the vchanger program so that 
> it monitored disk space and returned device full "early"?

No, because vchanger only runs very briefly when Bacula requests a 
volume be "loaded" or "unloaded". It basically points Bacula to the 
particular volume file it is to use and then exits. Bacula reads/writes 
the file directly, so there is no interaction between vchanger and 
Bacula when the data is actually being written

>
> Ruth
>
> -- 
> Software Manager & Engineer
> Tel: 01223 414180
> Blog:http://www.ivimey.org/blog
> LinkedIn:http://uk.linkedin.com/in/ruthivimeycook/