Re: [Bacula-users] "Read error" on long backups of Windows clients

[Rob G. <ro...@cr...>]

Good morning, Andrea!

In this email I will concentrate on two different issues with one name:
'free space'. Please keep the following concepts in mind when reading this
email.
1. *Available space allocated to VSS for VSS writers to use.* VSS writers
will not use more space than this, and will delete older shadow copies to
make room for new ones. If a request is made for VSS writers to write
shadow copies, but there is not enough space allocated to do so as part of
one write transaction (one system restore point?), the VSS write operations
will fail.
2. *Total free space on the relevant drive in the system.* Even if the VSS
allocation quota is sufficient, if the system doesn't have any more space
to allocate for that drive, VSS cannot make backups. The same sort of
failure will occur, but it will not be because of an insufficient VSS
allocation, but instead because the system drive itself lacks enough free
space.


> My hypotesis is that the VSS snapshot is gone.
Maybe some of the snapshots were never created. Keep in mind that VSS uses
many 'writers' and some may give errors while others have no difficulties.
There are many possible causes for errors, but I will focus on one: lack of
available space for VSS backups.


> Is this possible? Has anyone else seen this behaviour?
Yes, I think it is possible, but *most likely portions of the snapshot
failed to be created as requested because of an available space issue.* I
have not seen this behavior in bacula before, but I recently troubleshot
similar behavior in Barracuda backup. *Barracuda is a different product,
but both it and bacula make backups on windows via VSS, so are subject to
VSS issues and limitations.*

> Does Bacula lock the snapshot it created in some way? (Can it?)
Not sure. I think bacula and other backup programs request VSS snapshots be
created as part of their process. I know that VSS automatically deletes
older backups to free up space within its quota. Barring manual or
scheduled actions to delete older shadow copies, I would expect VSS to
leave its shadow copies in place as long as possible, UNLESS VSS itself has
insufficient space to make more shadow copies. Then VSS will delete the
oldest shadow copies to make more room.

I don't know for sure, but I theorize that there may be protection to
prevent a single VSS snapshot from 'eating its own tail' by deleting shadow
copy files created at the start of a running snapshot to make space for
more files in that same snapshot. I would hope so, at least.

> What could be the cause?
I think you are very close. When troubleshooting barracuda backups failing
I received an error message:
" Exception - Description: Insufficient storage available to create either
the shadow copy storage file or other shadow copy data."

When I investigated the system (windows 11 pro 24h2), I opened an
administrator command prompt and ran 'vssadmin list writers'. This showed
several writers with status failed. I troubleshot this, and resolved some
of the writers by restarting impacted services (and rerunning 'vssadmin
list writers' to see if the impacted writer was now in a good state). I was
not able to fully resolve all failed writers by restarting relevant
services (one service wouldn't restart), so I opted to reboot the system.
This resolved the writer status issues, but probably did not correct the
underlying cause. If I had been unable to reboot the system, I believe I
could have re-registered the service files with rundll32. Googling how to
correct these issues should find you good instructions.

This site was helpful for me, particularly in identifying which writer name
matched which service name. The writer name listed in vssadmin list writers
did not match the service name listed in services.msc, but if I found a
service that seemed to have a name that was close, I could right click and
get service properties and see that the 'short' service name matched with
the writer name on this page matched the short service name given in the
service properties. I believe I could have just restarted the service via
commandline using commands not specified in this article, but I didn't
bother looking further because I found other solutions.
https://woshub.com/vss-writer-failed-re-registering-vss-writers-windows-server/

I think that at minimum the process described above helped me confirm that
VSS was having problems, and that the current problems had been cleared
prior to running another backup. However, as mentioned previously, I do not
think I had resolved the root cause at this point.

Finding the cause:
The Barracuda support rep explained to me that we need to increase the
storage space allocated to volume shadow copies. They said this issue was
because the system lacked enough space to create the volume shadow copies
required. Windows imposes a quota limitation on VSS snapshots to prevent
VSS taking all the storage. In my case VSS appeared to be curtailing its
storage use too aggressively - the default was 1% of disk space, which on
my problem system meant about 10.9GB. Barracuda support recommended I
increase this to 20% of total storage space. I set the maximum space
allocated to VSS snapshots to 20% of the total system space, as
recommended. Obviously, if your systems have limited free space, this could
cause problems down the line. Best to keep this in mind and keep lots of
free space available. Additionally, remember that SSDs typically want at
least 10% free space at all times to improve TRIM performance and reduce
wear. For this reason, it is important to monitor total available free
space on the system, and forecast the impact of increasing the storage
allocated to VSS. Perhaps it will be necessary to free up some space on the
system, or upgrade the system drive to a larger model.

Here is how I allocated more space to VSS:
Please note that this KB describes how to solve this problem in windows
server first, then desktop. For server, they provide a gui process, and for
desktop, they provide a CLI process. I read through the server gui process,
then followed a similar (but not identical) gui process on my windows
desktop machines. I preferred doing this vs the provided CLI process
because it showed me the current space allocated and status of the storage
space allocation on each drive in the system.

https://campus.barracuda.com/product/echoplatform/doc/168731336/allocating-shadow-copy-storage-space/

I did not follow the windows desktop process described in the link above,
but instead searched 'about your pc' in start menu, then clicked 'system
protection' This opened a control panel system applet listing the drives in
the system. I selected a drive, and clicked 'configure'. This gave me the
relevant gui window to check the currently used VSS storage, % allocated,
and increase that amount if desired. I found my system C drive was at 1%
max allocation, which meant about 10.9 GB apparently. I increased this to
20% as recommended, though maybe you could try 10% if desired. I also
verified that protection was 'on' for the drive I was examining. I clicked
'ok' and repeated this process for the other relevant drives in the system
(I did not enable protection for an irrelevant flash drive connected to the
system that I will remove later), and then clicked 'ok' to exit the system
control panel applet, applying all these settings.

Subsequent backups have ran successfully, and I haven't had more issues
yet.

Another similar, but also different case:
I suspect this and related issues are the cause of a number of VSS errors
I've gotten on another system running yet another backup program (macrium
reflect). This system runs windows server 2016, and has had VSS errors in
the backup program I use there. I checked this other system and the VSS
allocated space was already set to 20%. However, this system had very
constrained available space due to a small C drive. I recently cleaned up
some temp files to free more space on that C drive, which I think helped
correct my VSS issues there. So in that case, *I think the problem for that
system was not that the VSS quota was set too low, but that the system
itself had too little free space available.* This difference may be
relevant on your systems, so I provide it here.

I hope this is helpful. Maybe it is the cause of your issues.

Please let me know what you find, and the results of your experimentation.


Regards,
Robert Gerber
402-237-8692
ro...@cr...


On Sun, May 18, 2025 at 6:15 AM Andrea Venturoli <ml...@ne...> wrote:

> Hello.
>
> I've got some clients that take very long time to do a backup (mostly
> full, but also differential and incremental on some occasions).
>
> The backups start normally, with the client generating a VSS snapshot.
> However, after a variable amount of time (usually between 1 and 15
> hours) I get a "read error" on 1-2 files with "ERR=The system cannot
> find the file specified".
> All subsequent files are printed with "Could not stat {filename} ERR=No
> error".
>
> My hypotesis is that the VSS snapshot is gone.
> Is this possible? Has anyone else seen this behaviour?
> Does Bacula lock the snapshot it created in some way? (Can it?)
> What could be the cause?
>
>   bye & Thanks
>         av.
>
> P.S.
> I read the FAQ that say I should investigate VSS with other tools; I'll
> eventually do, but if anyone knows an obvious answer, it would save me a
> lot of time...
>
>
> _______________________________________________
> Bacula-users mailing list
> Bac...@li...
> https://lists.sourceforge.net/lists/listinfo/bacula-users
>