Re: [Bacula-users] Bacula cloud

[Chris W. <win...@gm...>]

I had exactly that problem with S3/Backblaze. I would sometimes get a "no
tomes available" error message. I run a script after to upload the part
files for all volumes all over again. The driver seems smart enough to not
upload part files that are already there.

-Chris-

On Fri, 21 Feb 2025, 02:21 Rob Gerber, <ro...@cr...> wrote:

> Hello,
> Bacula 15.0.2 running on Rocky linux 9.4. I'm setting up a system that I
> will eventually deploy for a customer. This system is currently in the
> testing phase and nothing on it matters - yet.
>
> I have a set of jobs and resources that write to local file volumes. After
> the local backup jobs are finished, lower priority copy jobs run with
> 'pooluncopiedjobs' to copy the local jobs to backblaze b2 s3 compatible
> storage. It mostly seems to work. The trouble is that I'm occasionally
> seeing errors like this:
>
> 20-Feb 15:56 td-bacula-sd JobId 47: Error: B2-TGU-Inc-0019/part.4
> state=error   retry=1/10 size=5.368 GB duration=1435s msg= S3_put_object
> ServiceUnavailable ERR=Service temporarily unavailable, please try again.
> CURL Effective URL:
> https://Craeon-TGU-Inc.s3.us-east-005.backblazeb2.com/B2-TGU-Inc-0019/part.4
> CURL Effective URL:
> https://Craeon-TGU-Inc.s3.us-east-005.backblazeb2.com/B2-TGU-Inc-0019/part.4
>
> Some parts are failing to upload. I verified with ls that the parts in
> question still remain on disk (I have set cloud cache to truncate on
> upload, so only part1 and parts that haven't uploaded should remain).
>
> In bconsole I ran 'cloud allpools upload' and that seemed to work, albeit
> slowly. It's still in process now with the few remaining part files that
> didn't upload. Upload speeds seem much slower than when the job was
> uploading part files. Maybe some sort of bandwidth limitation is in effect
> if the upload command was ran from within bconsole? During the actual copy
> job, uploads observed at my pfsense router were in the 400-800mbps range.
> Now that the copy jobs have finished and only the bconsole 'cloud allpools
> upload' command is running, sustained upload is in the 5-10mbps range, and
> I still have a couple 5 gb part files left. I know this system and my
> internet connection can do better, so why is the upload ran from the
> console going so slowly?
>
> I have continuous pings to 8.8.8.8 and the backblaze node that is taking
> these uploads, and the ping tests haven't missed a single packet during the
> upload period.
>
> What can I do to improve reliability here? I was considering setting up an
> admin job to periodically run 'cloud allpools upload'.
>
> How does one set up an admin job? I've searched the manual and read the
> examples, but I'm fuzzy on how it would actually work. I envisioned just
> setting the admin job's payload to be a bconsole command, but the examples
> in the manual didn't seem to be exactly that. Maybe I'm just confused.
>
> The bigger problem seems to be that I don't know how to monitor whether a
> part subsequently uploaded successfully. I am working on deploying Bill A's
> job report script, and that's likely going to be my monitoring solution.
> Maybe a bad exit status for an admin job tasked with uploading any leftover
> part files could send up a flag that something is wrong and a part file
> hasn't been uploaded?
>
> I'm using the s3 driver instead of the amazon driver. I read that the new
> amazon driver is much better than the s3 driver, but in my testing the
> amazon driver didn't seem to want to connect to a non-aws server.
>
> I welcome any advice or assistance.
>
> dir config:
>  Job {
>    Name = "Backup-akita-job"
>    Level = "Incremental"
>    Client = "akita-fd"
>    Fileset = "Windows-All-Drives-fs"
>    Storage = "Synology-Local"
>    Pool = "Synology-Local-Inc"
>    JobDefs = "Synology-Local"
>  }
> Job {
>   Name = "Copy-Control-job"
>   Type = Copy
>   Level = Full
>   Client = td-bacula-fd
>   Schedule = "Copy-End-Of-Day"
>   FileSet = "None"
>   Messages = Standard
>   Pool = None
>   Storage = "Synology-Local"
>   Maximum Concurrent Jobs = 4
>   Selection Type = PoolUncopiedJobs
>   Priority = 20
>  JobDefs = "Synology-Local"
> }
> Schedule {
>   Name = "Biannual-Cycle"
>   Run = Level="Full" jan,jul 1st sun at 23:05
>   Run = Level="Differential" feb,mar,apr,may,jun,aug,sep,oct,nov,dec 1st sun at 23:05
>   Run = Level="Incremental" sun,mon,tue,wed,thu,fri,sat at 23:05
> }
> Schedule {
>   Name = "Copy-End-Of-Day"
>   Run = Pool="Synology-Local-Full" at 23:50
>   Run = Pool="Synology-Local-Diff" at 23:51
>   Run = Pool="Synology-Local-Inc" at 23:52
> }
> Storage {
>   Name = "B2-TGU-Full"
>   Description = "B2-TGU-Full"
>   SdPort = 9103
>   Address = "td-bacula"
>   Password = "7D9QQiqfzdG3vkjZ4RclYVtWdfKDTfUrruHi2ULTcx9B"
>   Device = "B2-TGU-Full"
>   MediaType = "B2-TGU-Full"
>   Autochanger = "B2-TGU-Full"
>   MaximumConcurrentJobs = 50
> }
> Pool { # cloud full pool
>   Name = "B2-TGU-Full"
>   Description = "B2-TGU-Full"
>   PoolType = "Backup"
>   LabelFormat = "B2-TGU-Full-"
>   LabelType = "Bacula"
>   MaximumVolumeJobs = 1
>   FileRetention = 18months
>   JobRetention = 18months
>   VolumeRetention = 18months
>   Storage = "B2-TGU-Full"
>   CacheRetention = 1minute
>   ActionOnPurge=Truncate
> }
> FileSet {
>   Name = "Windows-All-Drives-fs"
>   Include {
>     Options {
>       Exclude = "yes"
>       IgnoreCase= "yes"
>       # wilddir = "*:/Windows"
>       wildfile = "*:/pagefile.sys"
>       wildfile = "*:/hiberfil.sys"
>       wildfile = "*:/swapfile.sys"
>       # wild = "*:/$Recycle.Bin"
>       # wild = "*:/$recycler"
>       # wild = "*:/Program Files (x86)"
>       # wild = "*:/Program Files"
>       # wild = "*:/recycler/"
>       # wild = "*:/users/all users/microsoft help/"
>       # wild = "*:/users/all users/microsoft/"
>       # wild = "*:/i386/"
>       # wild = "*:/msocache/"
>      }
>     Options {
>       #compression=GZIP
>       #compression=LZO
>       compression=ZSTD
>       IgnoreCase= "yes"
>       Signature = "SHA512"
>       OneFS = "no"
>       DriveType = fixed, removable
>       verify = pins3
>     }
>     File = /
>   }
> }
> JobDefs {
>   Name = "Synology-Local"
>   Type = "Backup"
>   Level = "Incremental"
>   Messages = "Standard"
>   AllowDuplicateJobs = no # We don't want duplicate jobs. What action is taken is determined by the variables below.
>   # See flowchart Figure 23.2 in Bacula 15.x Main manual, probably page 245 in the PDF.
>   CancelLowerLevelDuplicates = yes # If a lower level job (example: inc) is running or queued and a higher level job (Example: diff or full) is added to the queue, then the lower level job will be cancelled.
>   CancelQueuedDuplicates = yes # This will cancel any queued duplicate jobs.
>   Pool = "Synology-Local-Inc"
>   FullBackupPool = "Synology-Local-Full"
>   IncrementalBackupPool = "Synology-Local-Inc"
>   DifferentialBackupPool = "Synology-Local-Diff"
>   Client = "td-bacula-fd"
>   Fileset = "Windows-All-Drives-fs"
>   Schedule = "Biannual-Cycle"
>   WriteBootstrap = "/mnt/synology/bacula/BSR/%n.bsr"
>   MaxFullInterval = 180days
>   MaxDiffInterval = 90days
>   SpoolAttributes = yes
>   Priority = 10
>   ReRunFailedLevels = yes # (if previous full or diff failed, current job will be upgraded to match failed job's level). a failed job is defined as one that has not terminated normally, which includes any running job of the same name. Cannot allow duplicate queued jobs.
>   RescheduleOnError = no
>   Accurate = yes
> }
>
> sd config:
> Device {
>   Name = "B2-TGU-Full-001" # there are 5 more like this one
>   Description = "B2-TGU-Full"
>   MediaType = "B2-TGU-Full"
>   DeviceType = "Cloud"
>   ArchiveDevice = "/mnt/synology/bacula/B2-TGU-Full"
>   RemovableMedia = no
>   RandomAccess = yes
>   AutomaticMount = yes
>   LabelMedia = yes
>   Autochanger = yes
>   ReadOnly = no
>   MaximumFileSize = 5368709120
>   MaximumConcurrentJobs = 1
>   DriveIndex = 0
>   MaximumPartSize = 5368709120
>   Cloud = "B2-TGU-Full"
> }
> Autochanger {
>   Name = "B2-TGU-Full"
>   Device = "B2-TGU-Full-001"
>   Device = "B2-TGU-Full-002"
>   Device = "B2-TGU-Full-003"
>   Device = "B2-TGU-Full-004"
>   Device = "B2-TGU-Full-005"
>   Device = "B2-TGU-Full-006"
>   ChangerDevice = "/dev/null"
>   ChangerCommand = "/dev/null"
> }
> Cloud {
>   Name = "B2-TGU-Full"
>   Driver = "S3"
>   HostName = "s3.us-east-005.backblazeb2.com"
>   BucketName = "Craeon-TGU-Full"
>   AccessKey = "REDACTED"
>   SecretKey = "REDACTED"
>   Protocol = "HTTPS"
>   UriStyle = "VirtualHost"
>   TruncateCache = "AfterUpload"
>   Upload = "EachPart"
> }
>
> Regards,
> Robert Gerber
> 402-237-8692
> ro...@cr...
> _______________________________________________
> Bacula-users mailing list
> Bac...@li...
> https://lists.sourceforge.net/lists/listinfo/bacula-users
>