burp-users — The main (currently only) mailing list for burp.

Re: [Burp-users] Restore fails with Could not determine cipher from: 0 and EVP_CipherInit_ex failed

[Graham K. <gr...@gr...>]

On Mon, Mar 02, 2026 at 06:41:18PM +0100, Koen Drai wrote:
> Christian, you are officially my super hero for this week, vielen, vielen Dank!
> 
> 3.2.0 (self-compiled) indeed resulted in a clean restore.
> 
> Best regards,
> 
>  Koen
> 
> 
> On 02-03-2026 09:37, Christian Manal wrote:
> > Am 01.03.26 um 18:34 schrieb Koen Drai:
> > > I need to restore a single file from a rather ancient burp backup and get the following error messages:
> > > 
> > > Could not determine cipher from: 0
> > > 
> > > EVP_CipherInit_ex failed
> > 
> > 
> > Hi Koen,
> > 
> > which Burp version are you running? This sounds like a problem I had in the past which I submitted a patch [1] for, that is in 3.2.0.
> > 
> > [1] https://github.com/grke/burp/pull/932
> > 
> > 
> > Regards,
> > Christian Manal

Nice! Thank you for helping.

Re: [Burp-users] Restore fails with Could not determine cipher from: 0 and EVP_CipherInit_ex failed

[Koen D. <koe...@gm...>]

Christian, you are officially my super hero for this week, vielen, vielen Dank!

3.2.0 (self-compiled) indeed resulted in a clean restore.

Best regards,

  Koen


On 02-03-2026 09:37, Christian Manal wrote:
> Am 01.03.26 um 18:34 schrieb Koen Drai:
>> I need to restore a single file from a rather ancient burp backup and get the following error messages:
>>
>> Could not determine cipher from: 0
>>
>> EVP_CipherInit_ex failed
> 
> 
> Hi Koen,
> 
> which Burp version are you running? This sounds like a problem I had in the past which I submitted a patch [1] for, that is in 3.2.0.
> 
> [1] https://github.com/grke/burp/pull/932
> 
> 
> Regards,
> Christian Manal
> 
> 
> _______________________________________________
> Burp-users mailing list
> Bur...@li...
> https://lists.sourceforge.net/lists/listinfo/burp-users

Re: [Burp-users] Restore fails with Could not determine cipher from: 0 and EVP_CipherInit_ex failed

[Koen D. <koe...@gm...>]

Hi Christian,

It's burp-3.1.4, from the Debian Trixie sources.

I'll try to find a backport now.

Thanks!

On 02-03-2026 09:37, Christian Manal wrote:
> Am 01.03.26 um 18:34 schrieb Koen Drai:
>> I need to restore a single file from a rather ancient burp backup and get the following error messages:
>>
>> Could not determine cipher from: 0
>>
>> EVP_CipherInit_ex failed
> 
> 
> Hi Koen,
> 
> which Burp version are you running? This sounds like a problem I had in the past which I submitted a patch [1] for, that is in 3.2.0.
> 
> [1] https://github.com/grke/burp/pull/932
> 
> 
> Regards,
> Christian Manal
> 
> 
> _______________________________________________
> Burp-users mailing list
> Bur...@li...
> https://lists.sourceforge.net/lists/listinfo/burp-users

Re: [Burp-users] Restore fails with Could not determine cipher from: 0 and EVP_CipherInit_ex failed

[Koen D. <koe...@gm...>]

On 01-03-2026 23:21, gr...@gr... wrote:
> On Sun, Mar 01, 2026 at 06:34:30PM +0100, Koen Drai wrote:
>> Hi,
>>
>> I need to restore a single file from a rather ancient burp backup and get the following error messages:
>>
>> Could not determine cipher from: 0
>>
>> EVP_CipherInit_ex failed
>>
>>
>> The latter error occurred earlier with a different file. So fixed my local openssl.cnf for the legacy provider, which helped.
>>
>>
>> I have looked in the manifest of the old backup:
>> 1      2 3   4 5 6 7 8   9 1011     12     13     14151617
>> r0039A A IH/ B A A A FTB A A BP3QJN BFITOG BP3QJN A g J -B A A
>>
>>
>> Tried to decode following Graham's instructions from last January (see below).
>>
>> Not sure I fully understand, I would expect a B instead of a -B.
>> However, when I "correct" the manifest (i.e. gunzip, edit, gzip), I get the following error:
>>
>> 2026-03-01 18:25:06 +0100: burp[4133904] yajl error: lexical error: invalid char in json text.
>>                                         restoreend
>>                       (right here) ------^
>>
>> Which disappears again after changing back to -B.
>>
>>
>>
>> Do you have any ideas on how I get the file back?
>>
>>
>> Thanks a lot and regards,
>>
>>   Koen
>>
>>
>>
>>
>>
>> "It starts with 'r', then the next 4 characters are the length of the attributes
>>
>> line. Then the rest of the characters are various attributes, base64 encoded.
>> The encryption flag is in the 17th place. In this example, it is the 'A' after
>> the 'J'.
>>
>> The possible (decoded) values that this field come out of burp's src/sbuf.h
>> definitions:
>> #define ENCRYPTION_UNSET                        -1 // Also legacy
>> #define ENCRYPTION_NONE                         0
>> #define ENCRYPTION_KEY_DERIVED_BF_CBC           1 // Legacy
>> #define ENCRYPTION_KEY_DERIVED_AES_CBC_256      2
>>
>> So, in the example, 'A' is 0 means ENCRYPTION_NONE.
>> ENCRYPTION_UNSET:                   ?
>> ENCRYPTION_NONE:                    A
>> ENCRYPTION_KEY_DERIVED_BF_CBC:      B
>> ENCRYPTION_KEY_DERIVED_AES_CBC_256: C"
> 
> 
> Hello,
> 
> What do the next three lines in the manifest look like, after this?
> r0039A A IH/ B A A A FTB A A BP3QJN BFITOG BP3QJN A g J -B A A
> 

x00285819928:155143358a033ab4af1e79bc03ac97ca
t0043t/D:/<anomymized path>
r0039A A IH/ B A A A FTB A A BP3QJN BFITOG BP3QJN A g J -B A A
y0041D:/<anomymized path>
x002621909:fd80734aa0d6372ecd998e1a40ab8ec5
t00110000/0000/003B.gz



If it's relevant:
It is a backup from a Windows client to a Linux server, made August 2024.
Last year, I have replaced the hard drive on the Linux server, I was using rsync for that.
The version that I am trying to restore is the oldest available version.


Thanks!

Re: [Burp-users] Restore fails with Could not determine cipher from: 0 and EVP_CipherInit_ex failed

[Christian M. <ma...@un...>]

Am 01.03.26 um 18:34 schrieb Koen Drai:
> I need to restore a single file from a rather ancient burp backup and get the following error messages:
> 
> Could not determine cipher from: 0
> 
> EVP_CipherInit_ex failed


Hi Koen,

which Burp version are you running? This sounds like a problem I had in 
the past which I submitted a patch [1] for, that is in 3.2.0.

[1] https://github.com/grke/burp/pull/932


Regards,
Christian Manal

Re: [Burp-users] Restore fails with Could not determine cipher from: 0 and EVP_CipherInit_ex failed

[<gr...@gr...>]

On Sun, Mar 01, 2026 at 06:34:30PM +0100, Koen Drai wrote:
> Hi,
> 
> I need to restore a single file from a rather ancient burp backup and get the following error messages:
> 
> Could not determine cipher from: 0
> 
> EVP_CipherInit_ex failed
> 
> 
> The latter error occurred earlier with a different file. So fixed my local openssl.cnf for the legacy provider, which helped.
> 
> 
> I have looked in the manifest of the old backup:
> 1      2 3   4 5 6 7 8   9 1011     12     13     14151617
> r0039A A IH/ B A A A FTB A A BP3QJN BFITOG BP3QJN A g J -B A A
> 
> 
> Tried to decode following Graham's instructions from last January (see below).
> 
> Not sure I fully understand, I would expect a B instead of a -B.
> However, when I "correct" the manifest (i.e. gunzip, edit, gzip), I get the following error:
> 
> 2026-03-01 18:25:06 +0100: burp[4133904] yajl error: lexical error: invalid char in json text.
>                                        restoreend
>                      (right here) ------^
> 
> Which disappears again after changing back to -B.
> 
> 
> 
> Do you have any ideas on how I get the file back?
> 
> 
> Thanks a lot and regards,
> 
>  Koen
> 
> 
> 
> 
> 
> "It starts with 'r', then the next 4 characters are the length of the attributes
> 
> line. Then the rest of the characters are various attributes, base64 encoded.
> The encryption flag is in the 17th place. In this example, it is the 'A' after
> the 'J'.
> 
> The possible (decoded) values that this field come out of burp's src/sbuf.h
> definitions:
> #define ENCRYPTION_UNSET                        -1 // Also legacy
> #define ENCRYPTION_NONE                         0
> #define ENCRYPTION_KEY_DERIVED_BF_CBC           1 // Legacy
> #define ENCRYPTION_KEY_DERIVED_AES_CBC_256      2
> 
> So, in the example, 'A' is 0 means ENCRYPTION_NONE.
> ENCRYPTION_UNSET:                   ?
> ENCRYPTION_NONE:                    A
> ENCRYPTION_KEY_DERIVED_BF_CBC:      B
> ENCRYPTION_KEY_DERIVED_AES_CBC_256: C"


Hello,

What do the next three lines in the manifest look like, after this?
r0039A A IH/ B A A A FTB A A BP3QJN BFITOG BP3QJN A g J -B A A

[Burp-users] Restore fails with Could not determine cipher from: 0 and EVP_CipherInit_ex failed

[Koen D. <koe...@gm...>]

Hi,

I need to restore a single file from a rather ancient burp backup and get the following error messages:

Could not determine cipher from: 0

EVP_CipherInit_ex failed


The latter error occurred earlier with a different file. So fixed my local openssl.cnf for the legacy provider, which helped.


I have looked in the manifest of the old backup:
1      2 3   4 5 6 7 8   9 1011     12     13     14151617
r0039A A IH/ B A A A FTB A A BP3QJN BFITOG BP3QJN A g J -B A A


Tried to decode following Graham's instructions from last January (see below).

Not sure I fully understand, I would expect a B instead of a -B.
However, when I "correct" the manifest (i.e. gunzip, edit, gzip), I get the following error:

2026-03-01 18:25:06 +0100: burp[4133904] yajl error: lexical error: invalid char in json text.
                                        restoreend
                      (right here) ------^

Which disappears again after changing back to -B.



Do you have any ideas on how I get the file back?


Thanks a lot and regards,

  Koen





"It starts with 'r', then the next 4 characters are the length of the attributes

line. Then the rest of the characters are various attributes, base64 encoded.
The encryption flag is in the 17th place. In this example, it is the 'A' after
the 'J'.

The possible (decoded) values that this field come out of burp's src/sbuf.h
definitions:
#define ENCRYPTION_UNSET                        -1 // Also legacy
#define ENCRYPTION_NONE                         0
#define ENCRYPTION_KEY_DERIVED_BF_CBC           1 // Legacy
#define ENCRYPTION_KEY_DERIVED_AES_CBC_256      2

So, in the example, 'A' is 0 means ENCRYPTION_NONE.
ENCRYPTION_UNSET:                   ?
ENCRYPTION_NONE:                    A
ENCRYPTION_KEY_DERIVED_BF_CBC:      B
ENCRYPTION_KEY_DERIVED_AES_CBC_256: C"

Re: [Burp-users] Large file problems?

[Brendon H. <br...@qu...>]

Oh, one more thing I forgot to mention: Apparently Debian needs the 
libcap-dev package installed to compile burp, now, otherwise sys/
compatibility.h is missing, otherwise. I suspect something must have 
changed, recently. I didn't see that one mentioned in the README.

Best,
Brendon

On Sunday, February 15, 2026 9:57:53 p.m. Eastern Standard Time 
Brendon Higgins wrote:
> Hi Graham,
> 
> Thanks so much for investigating this. I tried testing it out: I had kept an
> archive of the backup that was failing for me, so I restored that 
(changing
> "current" symlink and adding "working" symlink by hand) and confirmed 
that
> my previous build of burp still gets stuck partway into this chromium file
> as before (it does). Next I applied your change to ssl.c in my working
> copy, rebuilt it, and tried again.
> 
> Sorry to bear bad news, but this also gets stuck when it gets to that file.
> I tried it a second time after double-checking the build was fresh, just to
> be certain. But it does seem like this isn't the cause.
> 
> Let me know if it would be helpful to send you the logs, which also have 
all
> the debug statements I added (the patch I sent previously).
> 
> The workaround at present is to simply remove the failing backup and 
let
> burp restart it from scratch. For me, the problem is reproducible when
> resuming, but only intermittent on restart.
> 
> Best,
> Brendon
> 
> On Sunday, February 15, 2026 6:14:56 a.m. Eastern Standard Time 
Graham
> 
> Keeling wrote:
> > Hello,
> > 
> > This weekend, I have spent a long time trying to reproduce the issue.
> > Bad news: I haven't managed to do so
> > Good news: I have found some suspicious parts of code on the server
> 
> side
> 
> > In src/ssl.c::ssl_load_dh_params() - the ssl v3 section from line 61:
> > if(OSSL_DECODER_from_bio(dctx, bio))
> > {
> > 
> > 	...error handling...
> > 
> > }
> > 
> > But the man page for OSSL_DECODER_from_bio() says this should:
> >        "return 1 on success, or 0 on failure."
> > 
> > It appears then, that this function is always going to the 'success' part
> > of the code, and carrying on. It should always be failing instead. 
Indeed,
> > when I fix the 'if()' to be 'if(!)', the burp server immediately exits
> > with
> > this error every time, indicating that this code has always been broken:
> > "No supported data to decode. Input type: PEM"
> > 
> > I suspect that the ssl library works for the initial handshake for basic
> > ciphers. But on trying to transfer large data in phase2, it tries to
> > renegotiate DH cipers and hangs on it's ephemeral key generation.
> > 
> > I have rewritten the section of code (lines 61-103), like so:
> > 
> > #else
> > #include <openssl/decoder.h>
> > int ssl_load_dh_params(SSL_CTX *ctx, struct conf **confs)
> > {
> > 
> > 	BIO *bio=NULL;
> > 	EVP_PKEY *pkey=NULL;
> > 	const char *ssl_dhfile=get_string(confs[OPT_SSL_DHFILE]);
> > 	
> > 	if(!(bio=BIO_new_file(ssl_dhfile, "r")))
> > 	{
> > 	
> > 		logp_ssl_err("Couldn't open ssl_dhfile: %s\n",
> 
> ssl_dhfile);
> 
> > 		return -1;
> > 	
> > 	}
> > 	
> > 	// Modern way - read as EVP_PKEY directly (preferred in 3.x).
> > 	pkey=PEM_read_bio_Parameters(bio, NULL);
> > 	BIO_free(bio);
> > 	if(!pkey)
> > 	{
> > 	
> > 		logp_ssl_err("Couldn't read DH parameters from
> 
> %s\n", ssl_dhfile);
> 
> > 		return -1;
> > 	
> > 	}
> > 	
> > 	// Transfer ownership to OpenSSL.
> > 	if(SSL_CTX_set0_tmp_dh_pkey(ctx, pkey)<=0)
> > 	{
> > 	
> > 		logp_ssl_err("Couldn't set DH parameters from
> 
> %s\n", ssl_dhfile);
> 
> > 		EVP_PKEY_free(pkey);
> > 		return -1;
> > 	
> > 	}
> > 	
> > 	// Success, openssl now owns pkey, do not free it here.
> > 	return 0;
> > 
> > }
> > #endif
> > 
> > I hope that this would prevent the hangs by loading the DH 
parameters
> 
> up
> 
> > front instead of leaving it to re-negotiation.
> > 
> > This is working for me in all my tests so far, but as I said, I am not
> > able
> > to get anything to hang as described by yourself.
> > 
> > I wonder whether somebody experiencing the issue is able to try this
> 
> code
> 
> > change on the server side and see if it fixes the issue?
> > 
> > Thank you for your patience with this annoying issue.
> > 
> > 
> > _______________________________________________
> > Burp-users mailing list
> > Bur...@li...
> > https://lists.sourceforge.net/lists/listinfo/burp-users

Re: [Burp-users] Large file problems?

[Brendon H. <br...@qu...>]

Hi Graham,

Thanks so much for investigating this. I tried testing it out: I had kept an 
archive of the backup that was failing for me, so I restored that (changing 
"current" symlink and adding "working" symlink by hand) and confirmed 
that my previous build of burp still gets stuck partway into this chromium 
file as before (it does). Next I applied your change to ssl.c in my working 
copy, rebuilt it, and tried again.

Sorry to bear bad news, but this also gets stuck when it gets to that file. I 
tried it a second time after double-checking the build was fresh, just to be 
certain. But it does seem like this isn't the cause.

Let me know if it would be helpful to send you the logs, which also have all 
the debug statements I added (the patch I sent previously).

The workaround at present is to simply remove the failing backup and let 
burp restart it from scratch. For me, the problem is reproducible when 
resuming, but only intermittent on restart.

Best,
Brendon

On Sunday, February 15, 2026 6:14:56 a.m. Eastern Standard Time Graham 
Keeling wrote:
> Hello,
> 
> This weekend, I have spent a long time trying to reproduce the issue.
> Bad news: I haven't managed to do so
> Good news: I have found some suspicious parts of code on the server 
side
> 
> In src/ssl.c::ssl_load_dh_params() - the ssl v3 section from line 61:
> if(OSSL_DECODER_from_bio(dctx, bio))
> {
> 	...error handling...
> }
> 
> But the man page for OSSL_DECODER_from_bio() says this should:
>        "return 1 on success, or 0 on failure."
> It appears then, that this function is always going to the 'success' part
> of the code, and carrying on. It should always be failing instead. Indeed,
> when I fix the 'if()' to be 'if(!)', the burp server immediately exits with
> this error every time, indicating that this code has always been broken:
> "No supported data to decode. Input type: PEM"
> 
> I suspect that the ssl library works for the initial handshake for basic
> ciphers. But on trying to transfer large data in phase2, it tries to
> renegotiate DH cipers and hangs on it's ephemeral key generation.
> 
> I have rewritten the section of code (lines 61-103), like so:
> 
> #else
> #include <openssl/decoder.h>
> int ssl_load_dh_params(SSL_CTX *ctx, struct conf **confs)
> {
> 	BIO *bio=NULL;
> 	EVP_PKEY *pkey=NULL;
> 	const char *ssl_dhfile=get_string(confs[OPT_SSL_DHFILE]);
> 
> 	if(!(bio=BIO_new_file(ssl_dhfile, "r")))
> 	{
> 		logp_ssl_err("Couldn't open ssl_dhfile: %s\n", 
ssl_dhfile);
> 		return -1;
> 	}
> 
> 	// Modern way - read as EVP_PKEY directly (preferred in 3.x).
> 	pkey=PEM_read_bio_Parameters(bio, NULL);
> 	BIO_free(bio);
> 	if(!pkey)
> 	{
> 		logp_ssl_err("Couldn't read DH parameters from 
%s\n", ssl_dhfile);
> 		return -1;
> 	}
> 
> 	// Transfer ownership to OpenSSL.
> 	if(SSL_CTX_set0_tmp_dh_pkey(ctx, pkey)<=0)
> 	{
> 		logp_ssl_err("Couldn't set DH parameters from 
%s\n", ssl_dhfile);
> 		EVP_PKEY_free(pkey);
> 		return -1;
> 	}
> 
> 	// Success, openssl now owns pkey, do not free it here.
> 	return 0;
> }
> #endif
> 
> I hope that this would prevent the hangs by loading the DH parameters 
up
> front instead of leaving it to re-negotiation.
> 
> This is working for me in all my tests so far, but as I said, I am not able
> to get anything to hang as described by yourself.
> 
> I wonder whether somebody experiencing the issue is able to try this 
code
> change on the server side and see if it fixes the issue?
> 
> Thank you for your patience with this annoying issue.
> 
> 
> _______________________________________________
> Burp-users mailing list
> Bur...@li...
> https://lists.sourceforge.net/lists/listinfo/burp-users

Re: [Burp-users] Large file problems?

[Graham K. <gr...@gr...>]

Hello,

This weekend, I have spent a long time trying to reproduce the issue.
Bad news: I haven't managed to do so
Good news: I have found some suspicious parts of code on the server side

In src/ssl.c::ssl_load_dh_params() - the ssl v3 section from line 61:
if(OSSL_DECODER_from_bio(dctx, bio))
{
	...error handling...
}

But the man page for OSSL_DECODER_from_bio() says this should:
       "return 1 on success, or 0 on failure."
It appears then, that this function is always going to the 'success' part
of the code, and carrying on. It should always be failing instead. Indeed,
when I fix the 'if()' to be 'if(!)', the burp server immediately exits with
this error every time, indicating that this code has always been broken:
"No supported data to decode. Input type: PEM"

I suspect that the ssl library works for the initial handshake for basic
ciphers. But on trying to transfer large data in phase2, it tries to
renegotiate DH cipers and hangs on it's ephemeral key generation.

I have rewritten the section of code (lines 61-103), like so:

#else
#include <openssl/decoder.h>
int ssl_load_dh_params(SSL_CTX *ctx, struct conf **confs)
{
	BIO *bio=NULL;
	EVP_PKEY *pkey=NULL;
	const char *ssl_dhfile=get_string(confs[OPT_SSL_DHFILE]);

	if(!(bio=BIO_new_file(ssl_dhfile, "r")))
	{
		logp_ssl_err("Couldn't open ssl_dhfile: %s\n", ssl_dhfile);
		return -1;
	}

	// Modern way - read as EVP_PKEY directly (preferred in 3.x).
	pkey=PEM_read_bio_Parameters(bio, NULL);
	BIO_free(bio);
	if(!pkey)
	{
		logp_ssl_err("Couldn't read DH parameters from %s\n", ssl_dhfile);
		return -1;
	}

	// Transfer ownership to OpenSSL.
	if(SSL_CTX_set0_tmp_dh_pkey(ctx, pkey)<=0)
	{
		logp_ssl_err("Couldn't set DH parameters from %s\n", ssl_dhfile);
		EVP_PKEY_free(pkey);
		return -1;
	}

	// Success, openssl now owns pkey, do not free it here.
	return 0;
}
#endif

I hope that this would prevent the hangs by loading the DH parameters up front
instead of leaving it to re-negotiation.

This is working for me in all my tests so far, but as I said, I am not able to
get anything to hang as described by yourself.

I wonder whether somebody experiencing the issue is able to try this code
change on the server side and see if it fixes the issue?

Thank you for your patience with this annoying issue.

Re: [Burp-users] Large file problems?

[Brendon H. <br...@qu...>]

Hi Graham,

An extra couple of observations to add to my previous:

* I moved the server config and backup medium to a different machine - a laptop 
that's running a similar but not identical OS install (Debian testing). The 
client remained on the original machine. The new machine had the same problem 
occur as the old one did, so the problem is not limited to local sockets.

* I needed to resume a functional backup routine, so I deleted the failing 
folder and it's "working" symlink, and took a new backup. There were zero 
problems with taking a new backup, despite the problem file having not changed 
in the interim.

Hope it helps,
Brendon

On Tuesday, February 3, 2026 12:45:36 a.m. Eastern Standard Time Graham 
Keeling wrote:
> Thanks for the debugging attempt, I will be looking at reproducing the
> problem myself, once I have simplified my automatic testing infrastructure
> (it has blown up again, I'm going to get rid of jenkins so that there
> is less to go wrong).
> 
> On Sun, Feb 01, 2026 at 06:54:33PM -0500, Brendon Higgins wrote:
> > Hi all,
> > 
> > I spent a few hours today digging into the communication problem that's
> > preventing my local backup resuming properly (and possibly related
> > issues). I added a bunch of debug logging (see attached), ran backups,
> > and made some observations of the logs produced. Here are the ones that
> > seem most significant:
> > 
> > * At some point while transmitting the next file, the server just stops
> > reading. Its final call to SSL_read() returns -1, with error 2
> > (SSL_ERROR_WANT_READ), and after that the server does not try SSL_read()
> > again until after the client disconnects.
> > 
> > * The client sends considerably more packets of data which the server
> > doesn't read. The client does eventually stop sending, however.
> > 
> > * During this sending process, the client never calls SSL_read(), even
> > though the server is sending it small packets (responses?). I wonder if
> > this might be an important point.
> > 
> > * Reading is handled by the async_io(), which seems to check for the
> > presence of data using FD_ISSET(). For the server, a loop in
> > backup_phase2_server_all() causes execution to enter async_io() via some
> > other functions. For all loop iterations after the server's SSL_read()
> > failure, the FD_ISSET() condition is not met, preventing further
> > SSL_read() calls.
> > 
> > * I added a call to SSL_want() after FS_ISSET() condition failures. For
> > only the first iteration after the SSL_read() failure, SSL_want() returns
> > 3 (SSL_READING). This seems inconsistent with the FD_ISSET() response.
> > 
> > * For all following iterations, SSL_want() returns 1 (SSL_NOTHING). This
> > seems inconsistent with the fact that the server still hasn't called
> > SSL_read() (data should be still present in the buffer, no?) and the fact
> > that the client does send more data.
> > 
> > Now, I've never encountered this kind of construction for a read loop
> > before - I'm more familiar with loops that just call SSL_read() (or
> > read()) and let the return value indicate whether data was or wasn't
> > available. That makes me wonder if some behaviour of OpenSSL (or
> > something further upstream) has changed in such a way that this
> > construction no longer works reliably. The fact that FS_ISSET() seems to
> > be contradicting SSL_want(), at least initially, feels consistent with
> > that.
> > 
> > No idea what could be done about it though. And I'm also not sure what to
> > examine further.
> > 
> > Best,
> > Brendon
> > 
> > diff --git a/src/asfd.c b/src/asfd.c
> > index 02af6138..30ec92ae 100644
> > --- a/src/asfd.c
> > +++ b/src/asfd.c
> > @@ -203,11 +203,13 @@ static int asfd_do_read_ssl(struct asfd *asfd)
> > 
> >  	asfd->read_blocked_on_write=0;
> >  	
> >  	ERR_clear_error();
> > 
> > +	logp("%s: SSL_read(%p, %p, %lu)...\n", asfd->desc, asfd->ssl,
> > asfd->readbuf+asfd->readbuflen, asfd->bufmaxsize-asfd->readbuflen);> 
> >  	r=SSL_read(
> >  	
> >  		asfd->ssl,
> >  		asfd->readbuf+asfd->readbuflen,
> >  		asfd->bufmaxsize-asfd->readbuflen
> >  	
> >  	);
> > 
> > +	logp("-> %zd [err: %d]\n", r, SSL_get_error(asfd->ssl, r));
> > 
> >  	switch((e=SSL_get_error(asfd->ssl, r)))
> >  	{
> > 
> > @@ -221,6 +223,9 @@ static int asfd_do_read_ssl(struct asfd *asfd)
> > 
> >  			SSL_shutdown(asfd->ssl);
> >  			goto error;
> >  		
> >  		case SSL_ERROR_WANT_READ:
> > +			// HACK:
> > +			//sleep(1);
> > +			//return asfd_do_read_ssl(asfd);
> > 
> >  			break;
> >  		
> >  		case SSL_ERROR_WANT_WRITE:
> >  			asfd->read_blocked_on_write=1;
> > 
> > @@ -327,7 +332,9 @@ static int asfd_do_write_ssl(struct asfd *asfd)
> > 
> >  	if(asfd->ratelimit && check_ratelimit(asfd)) return 0;
> >  	ERR_clear_error();
> > 
> > +	logp("%s: SSL_write(%p, %p, %lu)...\n", asfd->desc, asfd->ssl,
> > asfd->writebuf, asfd->writebuflen);> 
> >  	w=SSL_write(asfd->ssl, asfd->writebuf, asfd->writebuflen);
> > 
> > +	logp("-> %zd\n", w);
> > 
> >  	switch((e=SSL_get_error(asfd->ssl, w)))
> >  	{
> > 
> > @@ -481,9 +488,11 @@ int asfd_read_expect(struct asfd *asfd, enum cmd cmd,
> > const char *expect)> 
> >  static int asfd_write(struct asfd *asfd, struct iobuf *wbuf)
> >  {
> > 
> > +	logp("Entering %s()\n", __func__);
> > 
> >  	if(asfd->as->doing_estimate) return 0;
> >  	while(wbuf->len)
> >  	{
> > 
> > +		logp("%s: Iterate\n", __func__);
> > 
> >  		if(asfd->errors)
> >  		
> >  			return -1;
> >  		
> >  		if(asfd->append_all_to_write_buffer(asfd, 
wbuf)==APPEND_ERROR)
> > 
> > diff --git a/src/async.c b/src/async.c
> > index 41d9152e..382d0533 100644
> > --- a/src/async.c
> > +++ b/src/async.c
> > @@ -82,6 +82,7 @@ static int windows_stupidity_hacks(struct asfd *asfd,
> > 
> >  static int async_io(struct async *as, int doread)
> >  {
> > 
> > +	logp("Entering %s(%p, %d)\n", __func__, as, doread);
> > 
> >  	int mfd=-1;
> >  	fd_set fsr;
> >  	fd_set fsw;
> > 
> > @@ -151,6 +152,7 @@ static int async_io(struct async *as, int doread)
> > 
> >  	if(mfd>0)
> >  	{
> > 
> > +		logp("%s: mfd>0\n", __func__);
> > 
> >  		errno=0;
> >  		s=select(mfd+1, &fsr, &fsw, &fse, &tval);
> >  		if(errno==EAGAIN || errno==EINTR) goto end;
> > 
> > @@ -185,30 +187,48 @@ static int async_io(struct async *as, int doread)
> > 
> >  			}
> >  		
> >  		}
> > 
> > -		if(asfd->doread && FD_ISSET(asfd->fd, &fsr)) // Able to 
read.
> > +		if(asfd->doread)
> > 
> >  		{
> > 
> > -			asfd->network_timeout=asfd-
>max_network_timeout;
> > -			switch(asfd->fdtype)
> > +			logp("%s: doread true\n", __func__);
> > +			if (FD_ISSET(asfd->fd, &fsr)) // Able to 
read.
> > 
> >  			{
> > 
> > -				case 
ASFD_FD_SERVER_LISTEN_MAIN:
> > -				case 
ASFD_FD_SERVER_LISTEN_STATUS:
> > -					// Indicate to the 
caller that we have
> > -					// a new 
incoming client.
> > -					asfd-
>new_client++;
> > -					break;
> > -				default:
> > -					if(asfd-
>do_read(asfd)
> > -					  || asfd-
>parse_readbuf(asfd))
> > -						
return asfd_problem(asfd);
> > -					break;
> > +				asfd->network_timeout=asfd-
>max_network_timeout;
> > +				switch(asfd->fdtype)
> > +				{
> > +					case 
ASFD_FD_SERVER_LISTEN_MAIN:
> > +					case 
ASFD_FD_SERVER_LISTEN_STATUS:
> > +						// 
Indicate to the caller that we have
> > +						// a 
new incoming client.
> > +						
asfd->new_client++;
> > +						
break;
> > +					default:
> > +						
if(asfd->do_read(asfd)
> > +						|| 
asfd->parse_readbuf(asfd))
> > +							
return asfd_problem(asfd);
> > +						
break;
> > +				}
> > +			}
> > +			else
> > +			{
> > +				logp("%s: not able to do_read. 
SSL_want says: %d\n", __func__,
> > +					asfd->ssl ? 
SSL_want(asfd->ssl) : 0xDEADBEEF);
> > 
> >  			}
> >  		
> >  		}
> > 
> > -		if(asfd->dowrite && FD_ISSET(asfd->fd, &fsw)) // Able 
to write.
> > +		if(asfd->dowrite)
> > 
> >  		{
> > 
> > -			asfd->network_timeout=asfd-
>max_network_timeout;
> > -			if(asfd->do_write(asfd))
> > -				return asfd_problem(asfd);
> > +			logp("%s: dowrite true\n", __func__);
> > +			if (FD_ISSET(asfd->fd, &fsw)) // Able to 
write.
> > +			{
> > +				asfd->network_timeout=asfd-
>max_network_timeout;
> > +				if(asfd->do_write(asfd))
> > +					return 
asfd_problem(asfd);
> > +			}
> > +			else
> > +			{
> > +				logp("%s: not able to 
do_write. SSL_want says: %d\n", __func__,
> > +					asfd->ssl ? 
SSL_want(asfd->ssl) : 0xDEADBEEF);
> > +			}
> > 
> >  		}
> >  		
> >  		if((!asfd->doread || !FD_ISSET(asfd->fd, &fsr))
> > 
> > @@ -229,6 +249,7 @@ static int async_io(struct async *as, int doread)
> > 
> >  end:
> >  	as->last_time=as->now;
> > 
> > +	logp("Exiting %s() normally\n", __func__);
> > 
> >  	return 0;
> >  
> >  }
> > 
> > @@ -239,6 +260,7 @@ static int async_read_write(struct async *as)
> > 
> >  static int async_write(struct async *as)
> >  {
> > 
> > +	logp("Entering %s()\n", __func__);
> > 
> >  	return async_io(as, 0 /* No read. */);
> >  
> >  }
> > 
> > diff --git a/src/client/backup_phase2.c b/src/client/backup_phase2.c
> > index 16357bf5..4674b443 100644
> > --- a/src/client/backup_phase2.c
> > +++ b/src/client/backup_phase2.c
> > @@ -153,6 +153,7 @@ static enum send_e send_whole_file_w(struct asfd
> > *asfd,
> > 
> >  	struct cntr *cntr, int compression, struct BFILE *bfd,
> >  	const char *extrameta, size_t elen)
> >  
> >  {
> > 
> > +	logp("Entering %s\n", __func__);
> > 
> >  	if((compression || encpassword) && sb->path.cmd!=CMD_EFS_FILE)
> >  	{
> >  	
> >  		int key_deriv=sb->encryption;
> > 
> > @@ -212,6 +213,7 @@ static int size_checks(struct asfd *asfd, struct sbuf
> > *sb, struct conf **confs)> 
> >  static int deal_with_data(struct asfd *asfd, struct sbuf *sb,
> >  
> >  	struct BFILE *bfd, struct conf **confs)
> >  
> >  {
> > 
> > +	logp("Entering %s\n", __func__);
> > 
> >  	int ret=-1;
> >  	int forget=0;
> >  	size_t elen=0;
> > 
> > @@ -342,7 +344,7 @@ static int deal_with_data(struct asfd *asfd, struct
> > sbuf *sb,> 
> >  	}
> >  	else
> >  	{
> > 
> > -		//logp("need to send whole file: %s\n", sb.path);
> > +		logp("need to send whole file: %s\n", 
iobuf_to_printable(&sb->path));
> > 
> >  		// send the whole file.
> >  		
> >  		if(asfd->write(asfd, &sb->attr)
> > 
> > @@ -391,6 +393,7 @@ static int parse_rbuf(struct asfd *asfd, struct sbuf
> > *sb,> 
> >  {
> >  
> >  	static struct iobuf *rbuf;
> >  	rbuf=asfd->rbuf;
> > 
> > +	logp("Entering %s: rbuf->cmd = %d\n", __func__, rbuf->cmd);
> > 
> >  	if(rbuf->cmd==CMD_DATAPTH)
> >  	{
> >  	
> >  		iobuf_move(&(sb->datapth), rbuf);
> > 
> > @@ -426,6 +429,7 @@ static int parse_rbuf(struct asfd *asfd, struct sbuf
> > *sb,> 
> >  static int do_backup_phase2_client(struct asfd *asfd,
> >  
> >  	struct conf **confs, int resume)
> >  
> >  {
> > 
> > +	logp("Entering %s(%p, %p, %d)\n", __func__, asfd, confs, resume);
> > 
> >  	int ret=-1;
> >  	// For efficiency, open Windows files for the VSS data, and do not
> >  	// close them until another time around the loop, when the actual
> > 
> > @@ -464,10 +468,13 @@ static int do_backup_phase2_client(struct asfd
> > *asfd,
> > 
> >  	while(1)
> >  	{
> > 
> > +		logp("Iterating loop in %s\n", __func__);
> > 
> >  		iobuf_free_content(rbuf);
> > 
> > +		logp("%s: About to asfd->read()...\n", __func__);
> > 
> >  		if(asfd->read(asfd)) goto end;
> >  		else if(!rbuf->buf) continue;
> > 
> > +		logp("%s: asfd->read() yielded non-NULL rbuf->buf\n", 
__func__);
> > 
> >  		if(rbuf->cmd==CMD_GEN && !strcmp(rbuf->buf, 
"backupphase2end"))
> >  		{
> >  		
> >  			if(asfd->write_str(asfd, CMD_GEN, 
"okbackupphase2end"))
> > 
> > @@ -486,6 +493,7 @@ end:
> >  	bfile_free(&bfd);
> >  	iobuf_free_content(rbuf);
> >  	sbuf_free(&sb);
> > 
> > +	logp("Exiting %s\n", __func__);
> > 
> >  	return ret;
> >  
> >  }
> > 
> > diff --git a/src/fzp.c b/src/fzp.c
> > index eb55afeb..76b9ddab 100644
> > --- a/src/fzp.c
> > +++ b/src/fzp.c
> > @@ -24,9 +24,11 @@ static void fzp_free(struct fzp **fzp)
> > 
> >  static FILE *open_fp(const char *fname, const char *mode)
> >  {
> > 
> > +	logp("Entering %s(\"%s\", \"%s\")\n", __func__, fname, mode);
> > 
> >  	FILE *fp=NULL;
> >  	if(!(fp=fopen(fname, mode)))
> >  	
> >  		logp("could not open %s: %s\n", fname, 
strerror(errno));
> > 
> > +	logp("Exiting %s() -> %p\n", __func__, fp);
> > 
> >  	return fp;
> >  
> >  }
> > 
> > @@ -84,6 +86,7 @@ static void not_open(const char *func)
> > 
> >  static struct fzp *fzp_do_open(const char *path, const char *mode,
> >  
> >  	enum fzp_type type)
> >  
> >  {
> > 
> > +	//logp("Entering %s(\"%s\", \"%s\", %d)\n", __func__, path, mode, 
type);
> > 
> >  	struct fzp *fzp=NULL;
> >  	
> >  	if(!(fzp=fzp_alloc())) goto error;
> > 
> > @@ -93,6 +96,7 @@ static struct fzp *fzp_do_open(const char *path, const
> > char *mode,> 
> >  		case FZP_FILE:
> >  			if(!(fzp->fp=open_fp(path, mode)))
> >  			
> >  				goto error;
> > 
> > +			//logp("Exiting %s() -> %p\n", __func__, 
fzp);
> > 
> >  			return fzp;
> >  		
> >  		case FZP_COMPRESSED:
> >  			if(!(fzp->zp=open_zp(path, mode)))
> > 
> > @@ -103,6 +107,7 @@ static struct fzp *fzp_do_open(const char *path, const
> > char *mode,> 
> >  			goto error;
> >  	
> >  	}
> >  
> >  error:
> > +	//logp("Erroring-out of %s()\n", __func__);
> > 
> >  	fzp_close(&fzp);
> >  	return NULL;
> >  
> >  }
> > 
> > diff --git a/src/handy_extra.c b/src/handy_extra.c
> > index 5f10ea92..f7f798b8 100644
> > --- a/src/handy_extra.c
> > +++ b/src/handy_extra.c
> > @@ -147,6 +147,7 @@ char *get_endfile_str(uint64_t bytes, uint8_t
> > *checksum)> 
> >  int write_endfile(struct asfd *asfd, uint64_t bytes, uint8_t *checksum)
> >  {
> > 
> > +	logp("Entering %s\n", __func__);
> > 
> >  	return asfd->write_str(asfd,
> >  	
> >  		CMD_END_FILE, get_endfile_str(bytes, checksum));
> >  
> >  }
> > 
> > @@ -165,6 +166,7 @@ enum send_e send_whole_file_gzl(struct asfd *asfd,
> > const char *datapth,> 
> >  	struct cntr *cntr, int compression, struct BFILE *bfd,
> >  	const char *extrameta, size_t elen, int key_deriv, uint64_t salt)
> >  
> >  {
> > 
> > +	logp("Entering %s\n", __func__);
> > 
> >  	enum send_e ret=SEND_OK;
> >  	int zret=0;
> >  	struct md5 *md5=NULL;
> > 
> > @@ -222,6 +224,7 @@ enum send_e send_whole_file_gzl(struct asfd *asfd,
> > const char *datapth,> 
> >  	do
> >  	{
> > 
> > +		logp("%s: Iterate\n", __func__);
> > 
> >  		if(metadata)
> >  		{
> >  		
> >  			if(metalen>ZCHUNK)
> > 
> > @@ -324,8 +327,10 @@ enum send_e send_whole_file_gzl(struct asfd *asfd,
> > const char *datapth,> 
> >  			else
> >  			{
> >  			
> >  				iobuf_set(&wbuf, 
CMD_APPEND, (char *)out, have);
> > 
> > +				logp("%s: About to asfd-
>write()...\n", __func__);
> > 
> >  				if(asfd->write(asfd, &wbuf))
> >  				{
> > 
> > +					logp("%s: asfd-
>write() error\n", __func__);
> > 
> >  					ret=SEND_FATAL;
> >  					break;
> >  				
> >  				}
> > 
> > diff --git a/src/server/backup_phase2.c b/src/server/backup_phase2.c
> > index a48bfc14..461f09d9 100644
> > --- a/src/server/backup_phase2.c
> > +++ b/src/server/backup_phase2.c
> > @@ -613,6 +613,7 @@ enum sts_e
> > 
> >  static enum sts_e do_stuff_to_send(struct asfd *asfd,
> >  
> >  	struct sbuf *p1b, char **last_requested)
> >  
> >  {
> > 
> > +	logp("Entering %s()\n", __func__);
> > 
> >  	static struct iobuf wbuf;
> >  	if(p1b->flags & SBUF_SEND_DATAPTH)
> >  	{
> > 
> > @@ -834,6 +835,7 @@ static enum str_e do_stuff_to_receive(struct asfd
> > *asfd,> 
> >  	struct sbuf *rb, struct manios *manios,
> >  	struct dpth *dpth, char **last_requested)
> >  
> >  {
> > 
> > +	logp("Entering %s()\n", __func__);
> > 
> >  	struct iobuf *rbuf=asfd->rbuf;
> >  	
> >  	if(rbuf->cmd==CMD_MESSAGE
> > 
> > @@ -987,6 +989,7 @@ static int process_next_file_from_manios(struct asfd
> > *asfd,> 
> >  	struct sbuf *cb,
> >  	struct conf **cconfs)
> >  
> >  {
> > 
> > +	logp("Entering %s()\n", __func__);
> > 
> >  	if(!(*p1b=sbuf_alloc()))
> >  	
> >  		goto error;
> >  	
> >  	switch(manio_read(manios->phase1, *p1b))
> > 
> > @@ -1138,6 +1141,8 @@ end:
> >  int backup_phase2_server_all(struct async *as, struct sdirs *sdirs,
> >  
> >  	const char *incexc, int resume, struct conf **cconfs)
> >  
> >  {
> > 
> > +	logp("Entering %s()\n", __func__);
> > +
> > 
> >  	int ret=0;
> >  	man_off_t *pos_phase1=NULL;
> >  	man_off_t *pos_current=NULL;
> > 
> > @@ -1220,6 +1225,7 @@ int backup_phase2_server_all(struct async *as,
> > struct sdirs *sdirs,> 
> >  				goto error;
> >  		
> >  		if(cntr_send_sdirs(asfd, sdirs, cconfs, 
CNTR_STATUS_BACKUP))
> >  		
> >  			goto error;
> > 
> > +		logp("Resume commenced in %s\n", __func__);
> > 
> >  	}
> >  	
> >  	if(!(manios=manios_open_phase2(sdirs, pos_phase1, pos_current)))
> > 
> > @@ -1239,6 +1245,7 @@ int backup_phase2_server_all(struct async *as,
> > struct sdirs *sdirs,> 
> >  	while(1)
> >  	{
> > 
> > +		logp("%s: Iterate...\n", __func__);
> > 
> >  		if(check_fail_on_warning(fail_on_warning, warn_ent))
> >  		
> >  			goto error;
> > 
> > @@ -1252,6 +1259,7 @@ int backup_phase2_server_all(struct async *as,
> > struct sdirs *sdirs,> 
> >  		  || !manios->phase1
> >  		  || asfd->writebuflen)
> >  		
> >  		{
> > 
> > +			logp("%s: sat cond for do_stuff_to_receive()
\n", __func__);
> > 
> >  			iobuf_free_content(asfd->rbuf);
> >  			if(asfd->as->read_write(asfd->as))
> >  			{
> > 
> > @@ -1275,6 +1283,11 @@ int backup_phase2_server_all(struct async *as,
> > struct sdirs *sdirs,> 
> >  					goto error;
> >  			
> >  			}
> >  		
> >  		}
> > 
> > +		else
> > +		{
> > +			logp("%s: failed cond for 
do_stuff_to_receive(): %p (\"%s\"), %p,
> > %lu\n", __func__, +			     last_requested,
> > last_requested?last_requested:"", manios->phase1, asfd->writebuflen);
> > +		}
> > 
> >  		if(p1b) switch(do_stuff_to_send(asfd, p1b, 
&last_requested))
> >  		{
> > 
> > _______________________________________________
> > Burp-users mailing list
> > Bur...@li...
> > https://lists.sourceforge.net/lists/listinfo/burp-users
> 
> _______________________________________________
> Burp-users mailing list
> Bur...@li...
> https://lists.sourceforge.net/lists/listinfo/burp-users

Re: [Burp-users] Large file problems?

[Graham K. <gr...@gr...>]

Thanks for the debugging attempt, I will be looking at reproducing the
problem myself, once I have simplified my automatic testing infrastructure
(it has blown up again, I'm going to get rid of jenkins so that there
is less to go wrong).

On Sun, Feb 01, 2026 at 06:54:33PM -0500, Brendon Higgins wrote:
> Hi all,
> 
> I spent a few hours today digging into the communication problem that's 
> preventing my local backup resuming properly (and possibly related issues). I 
> added a bunch of debug logging (see attached), ran backups, and made some 
> observations of the logs produced. Here are the ones that seem most 
> significant:
> 
> * At some point while transmitting the next file, the server just stops 
> reading. Its final call to SSL_read() returns -1, with error 2 
> (SSL_ERROR_WANT_READ), and after that the server does not try SSL_read() again 
> until after the client disconnects.
> 
> * The client sends considerably more packets of data which the server doesn't 
> read. The client does eventually stop sending, however.
> 
> * During this sending process, the client never calls SSL_read(), even though 
> the server is sending it small packets (responses?). I wonder if this might be 
> an important point.
> 
> * Reading is handled by the async_io(), which seems to check for the presence 
> of data using FD_ISSET(). For the server, a loop in backup_phase2_server_all() 
> causes execution to enter async_io() via some other functions. For all loop 
> iterations after the server's SSL_read() failure, the FD_ISSET() condition is 
> not met, preventing further SSL_read() calls.
> 
> * I added a call to SSL_want() after FS_ISSET() condition failures. For only 
> the first iteration after the SSL_read() failure, SSL_want() returns 3 
> (SSL_READING). This seems inconsistent with the FD_ISSET() response.
> 
> * For all following iterations, SSL_want() returns 1 (SSL_NOTHING). This seems 
> inconsistent with the fact that the server still hasn't called SSL_read() 
> (data should be still present in the buffer, no?) and the fact that the client 
> does send more data.
> 
> Now, I've never encountered this kind of construction for a read loop before - 
> I'm more familiar with loops that just call SSL_read() (or read()) and let the 
> return value indicate whether data was or wasn't available. That makes me 
> wonder if some behaviour of OpenSSL (or something further upstream) has 
> changed in such a way that this construction no longer works reliably. The 
> fact that FS_ISSET() seems to be contradicting SSL_want(), at least initially, 
> feels consistent with that.
> 
> No idea what could be done about it though. And I'm also not sure what to 
> examine further.
> 
> Best,
> Brendon

> diff --git a/src/asfd.c b/src/asfd.c
> index 02af6138..30ec92ae 100644
> --- a/src/asfd.c
> +++ b/src/asfd.c
> @@ -203,11 +203,13 @@ static int asfd_do_read_ssl(struct asfd *asfd)
>  	asfd->read_blocked_on_write=0;
>  
>  	ERR_clear_error();
> +	logp("%s: SSL_read(%p, %p, %lu)...\n", asfd->desc, asfd->ssl, asfd->readbuf+asfd->readbuflen, asfd->bufmaxsize-asfd->readbuflen);
>  	r=SSL_read(
>  		asfd->ssl,
>  		asfd->readbuf+asfd->readbuflen,
>  		asfd->bufmaxsize-asfd->readbuflen
>  	);
> +	logp("-> %zd [err: %d]\n", r, SSL_get_error(asfd->ssl, r));
>  
>  	switch((e=SSL_get_error(asfd->ssl, r)))
>  	{
> @@ -221,6 +223,9 @@ static int asfd_do_read_ssl(struct asfd *asfd)
>  			SSL_shutdown(asfd->ssl);
>  			goto error;
>  		case SSL_ERROR_WANT_READ:
> +			// HACK:
> +			//sleep(1);
> +			//return asfd_do_read_ssl(asfd);
>  			break;
>  		case SSL_ERROR_WANT_WRITE:
>  			asfd->read_blocked_on_write=1;
> @@ -327,7 +332,9 @@ static int asfd_do_write_ssl(struct asfd *asfd)
>  
>  	if(asfd->ratelimit && check_ratelimit(asfd)) return 0;
>  	ERR_clear_error();
> +	logp("%s: SSL_write(%p, %p, %lu)...\n", asfd->desc, asfd->ssl, asfd->writebuf, asfd->writebuflen);
>  	w=SSL_write(asfd->ssl, asfd->writebuf, asfd->writebuflen);
> +	logp("-> %zd\n", w);
>  
>  	switch((e=SSL_get_error(asfd->ssl, w)))
>  	{
> @@ -481,9 +488,11 @@ int asfd_read_expect(struct asfd *asfd, enum cmd cmd, const char *expect)
>  
>  static int asfd_write(struct asfd *asfd, struct iobuf *wbuf)
>  {
> +	logp("Entering %s()\n", __func__);
>  	if(asfd->as->doing_estimate) return 0;
>  	while(wbuf->len)
>  	{
> +		logp("%s: Iterate\n", __func__);
>  		if(asfd->errors)
>  			return -1;
>  		if(asfd->append_all_to_write_buffer(asfd, wbuf)==APPEND_ERROR)
> diff --git a/src/async.c b/src/async.c
> index 41d9152e..382d0533 100644
> --- a/src/async.c
> +++ b/src/async.c
> @@ -82,6 +82,7 @@ static int windows_stupidity_hacks(struct asfd *asfd,
>  
>  static int async_io(struct async *as, int doread)
>  {
> +	logp("Entering %s(%p, %d)\n", __func__, as, doread);
>  	int mfd=-1;
>  	fd_set fsr;
>  	fd_set fsw;
> @@ -151,6 +152,7 @@ static int async_io(struct async *as, int doread)
>  
>  	if(mfd>0)
>  	{
> +		logp("%s: mfd>0\n", __func__);
>  		errno=0;
>  		s=select(mfd+1, &fsr, &fsw, &fse, &tval);
>  		if(errno==EAGAIN || errno==EINTR) goto end;
> @@ -185,30 +187,48 @@ static int async_io(struct async *as, int doread)
>  			}
>  		}
>  
> -		if(asfd->doread && FD_ISSET(asfd->fd, &fsr)) // Able to read.
> +		if(asfd->doread)
>  		{
> -			asfd->network_timeout=asfd->max_network_timeout;
> -			switch(asfd->fdtype)
> +			logp("%s: doread true\n", __func__);
> +			if (FD_ISSET(asfd->fd, &fsr)) // Able to read.
>  			{
> -				case ASFD_FD_SERVER_LISTEN_MAIN:
> -				case ASFD_FD_SERVER_LISTEN_STATUS:
> -					// Indicate to the caller that we have
> -					// a new incoming client.
> -					asfd->new_client++;
> -					break;
> -				default:
> -					if(asfd->do_read(asfd)
> -					  || asfd->parse_readbuf(asfd))
> -						return asfd_problem(asfd);
> -					break;
> +				asfd->network_timeout=asfd->max_network_timeout;
> +				switch(asfd->fdtype)
> +				{
> +					case ASFD_FD_SERVER_LISTEN_MAIN:
> +					case ASFD_FD_SERVER_LISTEN_STATUS:
> +						// Indicate to the caller that we have
> +						// a new incoming client.
> +						asfd->new_client++;
> +						break;
> +					default:
> +						if(asfd->do_read(asfd)
> +						|| asfd->parse_readbuf(asfd))
> +							return asfd_problem(asfd);
> +						break;
> +				}
> +			}
> +			else
> +			{
> +				logp("%s: not able to do_read. SSL_want says: %d\n", __func__,
> +					asfd->ssl ? SSL_want(asfd->ssl) : 0xDEADBEEF);
>  			}
>  		}
>  
> -		if(asfd->dowrite && FD_ISSET(asfd->fd, &fsw)) // Able to write.
> +		if(asfd->dowrite)
>  		{
> -			asfd->network_timeout=asfd->max_network_timeout;
> -			if(asfd->do_write(asfd))
> -				return asfd_problem(asfd);
> +			logp("%s: dowrite true\n", __func__);
> +			if (FD_ISSET(asfd->fd, &fsw)) // Able to write.
> +			{
> +				asfd->network_timeout=asfd->max_network_timeout;
> +				if(asfd->do_write(asfd))
> +					return asfd_problem(asfd);
> +			}
> +			else
> +			{
> +				logp("%s: not able to do_write. SSL_want says: %d\n", __func__,
> +					asfd->ssl ? SSL_want(asfd->ssl) : 0xDEADBEEF);
> +			}
>  		}
>  
>  		if((!asfd->doread || !FD_ISSET(asfd->fd, &fsr))
> @@ -229,6 +249,7 @@ static int async_io(struct async *as, int doread)
>  
>  end:
>  	as->last_time=as->now;
> +	logp("Exiting %s() normally\n", __func__);
>  	return 0;
>  }
>  
> @@ -239,6 +260,7 @@ static int async_read_write(struct async *as)
>  
>  static int async_write(struct async *as)
>  {
> +	logp("Entering %s()\n", __func__);
>  	return async_io(as, 0 /* No read. */);
>  }
>  
> diff --git a/src/client/backup_phase2.c b/src/client/backup_phase2.c
> index 16357bf5..4674b443 100644
> --- a/src/client/backup_phase2.c
> +++ b/src/client/backup_phase2.c
> @@ -153,6 +153,7 @@ static enum send_e send_whole_file_w(struct asfd *asfd,
>  	struct cntr *cntr, int compression, struct BFILE *bfd,
>  	const char *extrameta, size_t elen)
>  {
> +	logp("Entering %s\n", __func__);
>  	if((compression || encpassword) && sb->path.cmd!=CMD_EFS_FILE)
>  	{
>  		int key_deriv=sb->encryption;
> @@ -212,6 +213,7 @@ static int size_checks(struct asfd *asfd, struct sbuf *sb, struct conf **confs)
>  static int deal_with_data(struct asfd *asfd, struct sbuf *sb,
>  	struct BFILE *bfd, struct conf **confs)
>  {
> +	logp("Entering %s\n", __func__);
>  	int ret=-1;
>  	int forget=0;
>  	size_t elen=0;
> @@ -342,7 +344,7 @@ static int deal_with_data(struct asfd *asfd, struct sbuf *sb,
>  	}
>  	else
>  	{
> -		//logp("need to send whole file: %s\n", sb.path);
> +		logp("need to send whole file: %s\n", iobuf_to_printable(&sb->path));
>  		// send the whole file.
>  
>  		if(asfd->write(asfd, &sb->attr)
> @@ -391,6 +393,7 @@ static int parse_rbuf(struct asfd *asfd, struct sbuf *sb,
>  {
>  	static struct iobuf *rbuf;
>  	rbuf=asfd->rbuf;
> +	logp("Entering %s: rbuf->cmd = %d\n", __func__, rbuf->cmd);
>  	if(rbuf->cmd==CMD_DATAPTH)
>  	{
>  		iobuf_move(&(sb->datapth), rbuf);
> @@ -426,6 +429,7 @@ static int parse_rbuf(struct asfd *asfd, struct sbuf *sb,
>  static int do_backup_phase2_client(struct asfd *asfd,
>  	struct conf **confs, int resume)
>  {
> +	logp("Entering %s(%p, %p, %d)\n", __func__, asfd, confs, resume);
>  	int ret=-1;
>  	// For efficiency, open Windows files for the VSS data, and do not
>  	// close them until another time around the loop, when the actual
> @@ -464,10 +468,13 @@ static int do_backup_phase2_client(struct asfd *asfd,
>  
>  	while(1)
>  	{
> +		logp("Iterating loop in %s\n", __func__);
>  		iobuf_free_content(rbuf);
> +		logp("%s: About to asfd->read()...\n", __func__);
>  		if(asfd->read(asfd)) goto end;
>  		else if(!rbuf->buf) continue;
>  
> +		logp("%s: asfd->read() yielded non-NULL rbuf->buf\n", __func__);
>  		if(rbuf->cmd==CMD_GEN && !strcmp(rbuf->buf, "backupphase2end"))
>  		{
>  			if(asfd->write_str(asfd, CMD_GEN, "okbackupphase2end"))
> @@ -486,6 +493,7 @@ end:
>  	bfile_free(&bfd);
>  	iobuf_free_content(rbuf);
>  	sbuf_free(&sb);
> +	logp("Exiting %s\n", __func__);
>  	return ret;
>  }
>  
> diff --git a/src/fzp.c b/src/fzp.c
> index eb55afeb..76b9ddab 100644
> --- a/src/fzp.c
> +++ b/src/fzp.c
> @@ -24,9 +24,11 @@ static void fzp_free(struct fzp **fzp)
>  
>  static FILE *open_fp(const char *fname, const char *mode)
>  {
> +	logp("Entering %s(\"%s\", \"%s\")\n", __func__, fname, mode);
>  	FILE *fp=NULL;
>  	if(!(fp=fopen(fname, mode)))
>  		logp("could not open %s: %s\n", fname, strerror(errno));
> +	logp("Exiting %s() -> %p\n", __func__, fp);
>  	return fp;
>  }
>  
> @@ -84,6 +86,7 @@ static void not_open(const char *func)
>  static struct fzp *fzp_do_open(const char *path, const char *mode,
>  	enum fzp_type type)
>  {
> +	//logp("Entering %s(\"%s\", \"%s\", %d)\n", __func__, path, mode, type);
>  	struct fzp *fzp=NULL;
>  
>  	if(!(fzp=fzp_alloc())) goto error;
> @@ -93,6 +96,7 @@ static struct fzp *fzp_do_open(const char *path, const char *mode,
>  		case FZP_FILE:
>  			if(!(fzp->fp=open_fp(path, mode)))
>  				goto error;
> +			//logp("Exiting %s() -> %p\n", __func__, fzp);
>  			return fzp;
>  		case FZP_COMPRESSED:
>  			if(!(fzp->zp=open_zp(path, mode)))
> @@ -103,6 +107,7 @@ static struct fzp *fzp_do_open(const char *path, const char *mode,
>  			goto error;
>  	}
>  error:
> +	//logp("Erroring-out of %s()\n", __func__);
>  	fzp_close(&fzp);
>  	return NULL;
>  }
> diff --git a/src/handy_extra.c b/src/handy_extra.c
> index 5f10ea92..f7f798b8 100644
> --- a/src/handy_extra.c
> +++ b/src/handy_extra.c
> @@ -147,6 +147,7 @@ char *get_endfile_str(uint64_t bytes, uint8_t *checksum)
>  
>  int write_endfile(struct asfd *asfd, uint64_t bytes, uint8_t *checksum)
>  {
> +	logp("Entering %s\n", __func__);
>  	return asfd->write_str(asfd,
>  		CMD_END_FILE, get_endfile_str(bytes, checksum));
>  }
> @@ -165,6 +166,7 @@ enum send_e send_whole_file_gzl(struct asfd *asfd, const char *datapth,
>  	struct cntr *cntr, int compression, struct BFILE *bfd,
>  	const char *extrameta, size_t elen, int key_deriv, uint64_t salt)
>  {
> +	logp("Entering %s\n", __func__);
>  	enum send_e ret=SEND_OK;
>  	int zret=0;
>  	struct md5 *md5=NULL;
> @@ -222,6 +224,7 @@ enum send_e send_whole_file_gzl(struct asfd *asfd, const char *datapth,
>  
>  	do
>  	{
> +		logp("%s: Iterate\n", __func__);
>  		if(metadata)
>  		{
>  			if(metalen>ZCHUNK)
> @@ -324,8 +327,10 @@ enum send_e send_whole_file_gzl(struct asfd *asfd, const char *datapth,
>  			else
>  			{
>  				iobuf_set(&wbuf, CMD_APPEND, (char *)out, have);
> +				logp("%s: About to asfd->write()...\n", __func__);
>  				if(asfd->write(asfd, &wbuf))
>  				{
> +					logp("%s: asfd->write() error\n", __func__);
>  					ret=SEND_FATAL;
>  					break;
>  				}
> diff --git a/src/server/backup_phase2.c b/src/server/backup_phase2.c
> index a48bfc14..461f09d9 100644
> --- a/src/server/backup_phase2.c
> +++ b/src/server/backup_phase2.c
> @@ -613,6 +613,7 @@ enum sts_e
>  static enum sts_e do_stuff_to_send(struct asfd *asfd,
>  	struct sbuf *p1b, char **last_requested)
>  {
> +	logp("Entering %s()\n", __func__);
>  	static struct iobuf wbuf;
>  	if(p1b->flags & SBUF_SEND_DATAPTH)
>  	{
> @@ -834,6 +835,7 @@ static enum str_e do_stuff_to_receive(struct asfd *asfd,
>  	struct sbuf *rb, struct manios *manios,
>  	struct dpth *dpth, char **last_requested)
>  {
> +	logp("Entering %s()\n", __func__);
>  	struct iobuf *rbuf=asfd->rbuf;
>  
>  	if(rbuf->cmd==CMD_MESSAGE
> @@ -987,6 +989,7 @@ static int process_next_file_from_manios(struct asfd *asfd,
>  	struct sbuf *cb,
>  	struct conf **cconfs)
>  {
> +	logp("Entering %s()\n", __func__);
>  	if(!(*p1b=sbuf_alloc()))
>  		goto error;
>  	switch(manio_read(manios->phase1, *p1b))
> @@ -1138,6 +1141,8 @@ end:
>  int backup_phase2_server_all(struct async *as, struct sdirs *sdirs,
>  	const char *incexc, int resume, struct conf **cconfs)
>  {
> +	logp("Entering %s()\n", __func__);
> +
>  	int ret=0;
>  	man_off_t *pos_phase1=NULL;
>  	man_off_t *pos_current=NULL;
> @@ -1220,6 +1225,7 @@ int backup_phase2_server_all(struct async *as, struct sdirs *sdirs,
>  				goto error;
>  		if(cntr_send_sdirs(asfd, sdirs, cconfs, CNTR_STATUS_BACKUP))
>  			goto error;
> +		logp("Resume commenced in %s\n", __func__);
>  	}
>  
>  	if(!(manios=manios_open_phase2(sdirs, pos_phase1, pos_current)))
> @@ -1239,6 +1245,7 @@ int backup_phase2_server_all(struct async *as, struct sdirs *sdirs,
>  
>  	while(1)
>  	{
> +		logp("%s: Iterate...\n", __func__);
>  		if(check_fail_on_warning(fail_on_warning, warn_ent))
>  			goto error;
>  
> @@ -1252,6 +1259,7 @@ int backup_phase2_server_all(struct async *as, struct sdirs *sdirs,
>  		  || !manios->phase1
>  		  || asfd->writebuflen)
>  		{
> +			logp("%s: sat cond for do_stuff_to_receive()\n", __func__);
>  			iobuf_free_content(asfd->rbuf);
>  			if(asfd->as->read_write(asfd->as))
>  			{
> @@ -1275,6 +1283,11 @@ int backup_phase2_server_all(struct async *as, struct sdirs *sdirs,
>  					goto error;
>  			}
>  		}
> +		else
> +		{
> +			logp("%s: failed cond for do_stuff_to_receive(): %p (\"%s\"), %p, %lu\n", __func__,
> +			     last_requested, last_requested?last_requested:"", manios->phase1, asfd->writebuflen);
> +		}
>  
>  		if(p1b) switch(do_stuff_to_send(asfd, p1b, &last_requested))
>  		{


> _______________________________________________
> Burp-users mailing list
> Bur...@li...
> https://lists.sourceforge.net/lists/listinfo/burp-users

Re: [Burp-users] Large file problems?

[Brendon H. <br...@qu...>]

Hi all,

I spent a few hours today digging into the communication problem that's 
preventing my local backup resuming properly (and possibly related issues). I 
added a bunch of debug logging (see attached), ran backups, and made some 
observations of the logs produced. Here are the ones that seem most 
significant:

* At some point while transmitting the next file, the server just stops 
reading. Its final call to SSL_read() returns -1, with error 2 
(SSL_ERROR_WANT_READ), and after that the server does not try SSL_read() again 
until after the client disconnects.

* The client sends considerably more packets of data which the server doesn't 
read. The client does eventually stop sending, however.

* During this sending process, the client never calls SSL_read(), even though 
the server is sending it small packets (responses?). I wonder if this might be 
an important point.

* Reading is handled by the async_io(), which seems to check for the presence 
of data using FD_ISSET(). For the server, a loop in backup_phase2_server_all() 
causes execution to enter async_io() via some other functions. For all loop 
iterations after the server's SSL_read() failure, the FD_ISSET() condition is 
not met, preventing further SSL_read() calls.

* I added a call to SSL_want() after FS_ISSET() condition failures. For only 
the first iteration after the SSL_read() failure, SSL_want() returns 3 
(SSL_READING). This seems inconsistent with the FD_ISSET() response.

* For all following iterations, SSL_want() returns 1 (SSL_NOTHING). This seems 
inconsistent with the fact that the server still hasn't called SSL_read() 
(data should be still present in the buffer, no?) and the fact that the client 
does send more data.

Now, I've never encountered this kind of construction for a read loop before - 
I'm more familiar with loops that just call SSL_read() (or read()) and let the 
return value indicate whether data was or wasn't available. That makes me 
wonder if some behaviour of OpenSSL (or something further upstream) has 
changed in such a way that this construction no longer works reliably. The 
fact that FS_ISSET() seems to be contradicting SSL_want(), at least initially, 
feels consistent with that.

No idea what could be done about it though. And I'm also not sure what to 
examine further.

Best,
Brendon

Re: [Burp-users] multiple backup servers

[<bu...@on...>]

Hi all!

Regarding to this old problem, I implemented a server randomization and 
made a few changes.
Here is my merge request: https://github.com/grke/burp/pull/942
I hope that Graham accept these changes. :)

Regards,
hnsz2002

Re: [Burp-users] EVP_CipherInit_ex failed

[Christian M. <ma...@un...>]

Am 27.01.26 um 22:29 schrieb gr...@gr...:
> On Tue, Jan 27, 2026 at 02:39:22PM +0100, Christian Manal wrote:
>> Am 24.01.26 um 02:13 schrieb Graham Keeling:
>>> In older versions of burp, the encryption flag wasn't encoded into the
>>> attributes. In that case, you can look at the file type row in the manifest:
>>> y0023/home/graham/burp/test/build/CA.cnf
>>> The leading 'y' means it's encrypted, and the blowfish encryption is implied.
>>>
>>>
>>> Given the information above, it would be possible to manually edit the current
>>> manifest to identify and delete the lines that represent blowfish encrypted
>>> files. Burp would then back them up again on the next backup. If doing this, I
>>> would also keep a copy of the original manifest and replace the edited manifest
>>> again when done.
>>>
>>> Obviously, having the feature to do it automatically a little at a time would
>>> be much easier!
>>> I will have a look to see if I can do this.
>> Hi Graham,
>>
>> I made a quick and dirty patch that adds a server side config option
>> "migrate_from_blowfish" (working title) that makes it so blowfish encrypted
>> files are treated as "new" to force re-encryption. See attachment.
>>
>> Tested on a small dataset with some old files and it seems to have worked
>> (got a bunch of files in the "new" column and "^r" lines in the manifest
>> went from "r... A A J B ..." in previous backups to "r... A A J C ..." in
>> the new test backup; restore works fine as well).
>>
>> Not sure if it's enough to only look at CMD_ENC_FILE in the check in
>> src/server/backup_phase2.c or if the other CMD_ENC_* types need to be
>> considered as well.
>>
>> Does this seem like a sane approach?
>>
>>
>> Regards,
>> Christian Manal
> 
>> diff --git a/configs/server/burp.conf.in b/configs/server/burp.conf.in
>> index a4fd8692..f159b0d5 100644
>> --- a/configs/server/burp.conf.in
>> +++ b/configs/server/burp.conf.in
>> @@ -43,6 +43,11 @@ client_can_verify = 1
>>   # Network timeout defaults to 7200 seconds (2 hours).
>>   # network_timeout = 7200
>>   
>> +# Set migrate_from_blowfish to 1 to force a new backup of eixsting files
>> +# that are encrypted with blowfish, to migrate to the current encryption
>> +# scheme.
>> +# migrate_from_blowfish = 0
>> +
>>   # Server storage compression. Default is zlib9. Set to zlib0 to turn it off.
>>   #compression = zlib9
>>   
>> diff --git a/manpages/burp.8.in b/manpages/burp.8.in
>> index 986b0ad4..fe1e0192 100644
>> --- a/manpages/burp.8.in
>> +++ b/manpages/burp.8.in
>> @@ -386,6 +386,9 @@ A client that is permitted to list, verify, restore, delete, and diff files belo
>>   \fBrestore_client=[client]\fR
>>   A client that is permitted to list, verify, restore, delete, and diff files belonging to any other client, according to the client_can permissions on both the restore_client and the original_client (eg, 'client_can_list'). You may specify multiple restore_clients. If this is too permissive, you may set a restore_client for individual original clients in the individual clientconfdir files.
>>   .TP
>> +\fBmigrate_from_blowfish=[0|1]\fR
>> +Turn this on to force a backup of existing files that are blowfish encrypted, to migrate to the current encryption scheme.
>> +.TP
>>   \fBssl_cert_ca=[path]\fR
>>   The path to the SSL CA certificate. This file will probably be the same on both the server and the client. The file should contain just the certificate in PEM format. For more information on this, and the other ssl_* options, please see docs/@name@_ca.txt.
>>   .TP
>> diff --git a/src/conf.c b/src/conf.c
>> index cf7392e5..465283f6 100644
>> --- a/src/conf.c
>> +++ b/src/conf.c
>> @@ -714,6 +714,9 @@ static int reset_conf(struct conf **c, enum conf_opt o)
>>   	case OPT_SERVER_CAN_RESTORE:
>>   	  return sc_int(c[o], 1,
>>   		CONF_FLAG_CC_OVERRIDE, "server_can_restore");
>> +	case OPT_MIGRATE_FROM_BLOWFISH:
>> +	  return sc_int(c[o], 0,
>> +		CONF_FLAG_CC_OVERRIDE, "migrate_from_blowfish");
>>   	case OPT_WORKING_DIR_RECOVERY_METHOD:
>>   	  return sc_rec(c[o], RECOVERY_METHOD_DELETE,
>>   		CONF_FLAG_CC_OVERRIDE, "working_dir_recovery_method");
>> diff --git a/src/conf.h b/src/conf.h
>> index e8ba20c5..935058bc 100644
>> --- a/src/conf.h
>> +++ b/src/conf.h
>> @@ -289,6 +289,8 @@ enum conf_opt
>>   	OPT_CLIENT_CAN_VERIFY,
>>   	OPT_SERVER_CAN_RESTORE,
>>   
>> +	OPT_MIGRATE_FROM_BLOWFISH,
>> +
>>   	// Set to 1 on both client and server when the server is able to send
>>   	// counters on resume/verify/restore.
>>   	OPT_SEND_CLIENT_CNTR,
>> diff --git a/src/server/backup_phase2.c b/src/server/backup_phase2.c
>> index a48bfc14..efa318df 100644
>> --- a/src/server/backup_phase2.c
>> +++ b/src/server/backup_phase2.c
>> @@ -491,6 +491,15 @@ static enum processed_e maybe_do_delta_stuff(struct asfd *asfd,
>>   		return process_new(cconfs, p1b, manios);
>>   	}
>>   
>> +	// file is blowfish encrypted and should be backed up again to update encryption
>> +	if((cb->encryption==ENCRYPTION_KEY_DERIVED_BF_CBC
>> +	   ||(cb->path.cmd==CMD_ENC_FILE
>> +	     && cb->encryption<ENCRYPTION_KEY_DERIVED_AES_CBC_256))
>> +	  && get_int(cconfs[OPT_MIGRATE_FROM_BLOWFISH]))
>> +	{
>> +		return process_new(cconfs, p1b, manios);
>> +	}
>> +
>>   	// mtime is the actual file data.
>>   	// ctime is the attributes or meta data.
>>   	if(cb->statp.st_mtime==p1b->statp.st_mtime
> 
> Hello,
> 
> Yes, very nice, this looks like it is on the right lines.
> 
> Since you have cb->encryption<ENCRYPTION_KEY_DERIVED_AES_CBC_256, I think
> the preceeding cb->encryption==ENCRYPTION_KEY_DERIVED_BF_CBC check is
> redundant.
> 
> There is a helper function, sbuf_is_encrypted(cb), that you can use for
> the other CMD_ENC_* types, but it also includes CMD_EFS_FILE, which you
> probably want to exclude, so something like:
> 
> if (get_int(cconfs[OPT_MIGRATE_FROM_BLOWFISH])
>    && cb->path.cmd!=CMD_EFS_FILE
>    && sbuf_is_encrypted(cb)
>    && cb->encryption<ENCRYPTION_KEY_DERIVED_AES_CBC_256)
> {
>     return process_new(cconfs, p1b, manios);
> }

Hi Graham,

thanks for the feedback. I reworked the patch a bit and tested it on 
another small dataset.

If you think this is a good addition to burp as it is, I've opened a 
pull request: https://github.com/grke/burp/pull/941


Regards,
Christian Manal

Re: [Burp-users] EVP_CipherInit_ex failed

[<gr...@gr...>]

On Tue, Jan 27, 2026 at 02:39:22PM +0100, Christian Manal wrote:
> Am 24.01.26 um 02:13 schrieb Graham Keeling:
> > In older versions of burp, the encryption flag wasn't encoded into the
> > attributes. In that case, you can look at the file type row in the manifest:
> > y0023/home/graham/burp/test/build/CA.cnf
> > The leading 'y' means it's encrypted, and the blowfish encryption is implied.
> > 
> > 
> > Given the information above, it would be possible to manually edit the current
> > manifest to identify and delete the lines that represent blowfish encrypted
> > files. Burp would then back them up again on the next backup. If doing this, I
> > would also keep a copy of the original manifest and replace the edited manifest
> > again when done.
> > 
> > Obviously, having the feature to do it automatically a little at a time would
> > be much easier!
> > I will have a look to see if I can do this.
> Hi Graham,
> 
> I made a quick and dirty patch that adds a server side config option
> "migrate_from_blowfish" (working title) that makes it so blowfish encrypted
> files are treated as "new" to force re-encryption. See attachment.
> 
> Tested on a small dataset with some old files and it seems to have worked
> (got a bunch of files in the "new" column and "^r" lines in the manifest
> went from "r... A A J B ..." in previous backups to "r... A A J C ..." in
> the new test backup; restore works fine as well).
> 
> Not sure if it's enough to only look at CMD_ENC_FILE in the check in
> src/server/backup_phase2.c or if the other CMD_ENC_* types need to be
> considered as well.
> 
> Does this seem like a sane approach?
> 
> 
> Regards,
> Christian Manal

> diff --git a/configs/server/burp.conf.in b/configs/server/burp.conf.in
> index a4fd8692..f159b0d5 100644
> --- a/configs/server/burp.conf.in
> +++ b/configs/server/burp.conf.in
> @@ -43,6 +43,11 @@ client_can_verify = 1
>  # Network timeout defaults to 7200 seconds (2 hours).
>  # network_timeout = 7200
>  
> +# Set migrate_from_blowfish to 1 to force a new backup of eixsting files
> +# that are encrypted with blowfish, to migrate to the current encryption
> +# scheme.
> +# migrate_from_blowfish = 0
> +
>  # Server storage compression. Default is zlib9. Set to zlib0 to turn it off.
>  #compression = zlib9
>  
> diff --git a/manpages/burp.8.in b/manpages/burp.8.in
> index 986b0ad4..fe1e0192 100644
> --- a/manpages/burp.8.in
> +++ b/manpages/burp.8.in
> @@ -386,6 +386,9 @@ A client that is permitted to list, verify, restore, delete, and diff files belo
>  \fBrestore_client=[client]\fR
>  A client that is permitted to list, verify, restore, delete, and diff files belonging to any other client, according to the client_can permissions on both the restore_client and the original_client (eg, 'client_can_list'). You may specify multiple restore_clients. If this is too permissive, you may set a restore_client for individual original clients in the individual clientconfdir files.
>  .TP
> +\fBmigrate_from_blowfish=[0|1]\fR
> +Turn this on to force a backup of existing files that are blowfish encrypted, to migrate to the current encryption scheme.
> +.TP
>  \fBssl_cert_ca=[path]\fR
>  The path to the SSL CA certificate. This file will probably be the same on both the server and the client. The file should contain just the certificate in PEM format. For more information on this, and the other ssl_* options, please see docs/@name@_ca.txt.
>  .TP
> diff --git a/src/conf.c b/src/conf.c
> index cf7392e5..465283f6 100644
> --- a/src/conf.c
> +++ b/src/conf.c
> @@ -714,6 +714,9 @@ static int reset_conf(struct conf **c, enum conf_opt o)
>  	case OPT_SERVER_CAN_RESTORE:
>  	  return sc_int(c[o], 1,
>  		CONF_FLAG_CC_OVERRIDE, "server_can_restore");
> +	case OPT_MIGRATE_FROM_BLOWFISH:
> +	  return sc_int(c[o], 0,
> +		CONF_FLAG_CC_OVERRIDE, "migrate_from_blowfish");
>  	case OPT_WORKING_DIR_RECOVERY_METHOD:
>  	  return sc_rec(c[o], RECOVERY_METHOD_DELETE,
>  		CONF_FLAG_CC_OVERRIDE, "working_dir_recovery_method");
> diff --git a/src/conf.h b/src/conf.h
> index e8ba20c5..935058bc 100644
> --- a/src/conf.h
> +++ b/src/conf.h
> @@ -289,6 +289,8 @@ enum conf_opt
>  	OPT_CLIENT_CAN_VERIFY,
>  	OPT_SERVER_CAN_RESTORE,
>  
> +	OPT_MIGRATE_FROM_BLOWFISH,
> +
>  	// Set to 1 on both client and server when the server is able to send
>  	// counters on resume/verify/restore.
>  	OPT_SEND_CLIENT_CNTR,
> diff --git a/src/server/backup_phase2.c b/src/server/backup_phase2.c
> index a48bfc14..efa318df 100644
> --- a/src/server/backup_phase2.c
> +++ b/src/server/backup_phase2.c
> @@ -491,6 +491,15 @@ static enum processed_e maybe_do_delta_stuff(struct asfd *asfd,
>  		return process_new(cconfs, p1b, manios);
>  	}
>  
> +	// file is blowfish encrypted and should be backed up again to update encryption
> +	if((cb->encryption==ENCRYPTION_KEY_DERIVED_BF_CBC
> +	   ||(cb->path.cmd==CMD_ENC_FILE
> +	     && cb->encryption<ENCRYPTION_KEY_DERIVED_AES_CBC_256))
> +	  && get_int(cconfs[OPT_MIGRATE_FROM_BLOWFISH]))
> +	{
> +		return process_new(cconfs, p1b, manios);
> +	}
> +
>  	// mtime is the actual file data.
>  	// ctime is the attributes or meta data.
>  	if(cb->statp.st_mtime==p1b->statp.st_mtime

Hello,

Yes, very nice, this looks like it is on the right lines.

Since you have cb->encryption<ENCRYPTION_KEY_DERIVED_AES_CBC_256, I think
the preceeding cb->encryption==ENCRYPTION_KEY_DERIVED_BF_CBC check is
redundant.

There is a helper function, sbuf_is_encrypted(cb), that you can use for
the other CMD_ENC_* types, but it also includes CMD_EFS_FILE, which you
probably want to exclude, so something like:

if (get_int(cconfs[OPT_MIGRATE_FROM_BLOWFISH])
  && cb->path.cmd!=CMD_EFS_FILE
  && sbuf_is_encrypted(cb)
  && cb->encryption<ENCRYPTION_KEY_DERIVED_AES_CBC_256)
{
   return process_new(cconfs, p1b, manios);
}

Re: [Burp-users] EVP_CipherInit_ex failed

[Christian M. <ma...@un...>]

Am 24.01.26 um 02:13 schrieb Graham Keeling:
> In older versions of burp, the encryption flag wasn't encoded into the
> attributes. In that case, you can look at the file type row in the manifest:
> y0023/home/graham/burp/test/build/CA.cnf
> The leading 'y' means it's encrypted, and the blowfish encryption is implied.
> 
> 
> Given the information above, it would be possible to manually edit the current
> manifest to identify and delete the lines that represent blowfish encrypted
> files. Burp would then back them up again on the next backup. If doing this, I
> would also keep a copy of the original manifest and replace the edited manifest
> again when done.
> 
> Obviously, having the feature to do it automatically a little at a time would
> be much easier!
> I will have a look to see if I can do this.
Hi Graham,

I made a quick and dirty patch that adds a server side config option 
"migrate_from_blowfish" (working title) that makes it so blowfish 
encrypted files are treated as "new" to force re-encryption. See attachment.

Tested on a small dataset with some old files and it seems to have 
worked (got a bunch of files in the "new" column and "^r" lines in the 
manifest went from "r... A A J B ..." in previous backups to "r... A A J 
C ..." in the new test backup; restore works fine as well).

Not sure if it's enough to only look at CMD_ENC_FILE in the check in 
src/server/backup_phase2.c or if the other CMD_ENC_* types need to be 
considered as well.

Does this seem like a sane approach?


Regards,
Christian Manal

Re: [Burp-users] backup of local server stucks

[John S. <jo...@st...>]

>>>>> "Marcin" == Marcin Mirosław <ma...@me...> writes:

Just a thought, does the backup fail if you setup a new (local) client
with just that single directory listed as to be backed up?  Does it
still fail with just that one file in the directory?  Or does it now
suceed?  

Just a thought... and it might make it easier to replicate and
actually keep more detailed logs.


> W dniu 2026-01-26 14:04, Marcin Mirosław napisał(a):
>> Hi,
>> I saw a little similar problem described by Mark in thread "Timeout?" 
>> but there were no clear solution nor description of root problem. I've 
>> got burp server, it does backup network connected clients without 
>> issue. Local client can't do backup since 2025-09-28, burp client seems 
>> to stuck short after start of phase2.
>> In backup directory there is a couple of files, client stuck, I think, 
>> when i starts to read a little bigger file (about 2-3MB). I could 
>> suspect something like MTU problem but connection is via 127.0.0.1 
>> (also tried using ip of eth interface - no change) so MTU shouldn't be 
>> case in this configuration. Strace shows:
>> 
>> n\0%s %s\n\0Packaged by %s (%"..., 4096) = 4096
>> 13141 12:54:23.810144 pselect6(5, [], [4], [4], {tv_sec=1, tv_nsec=0}, 
>> NULL) = 0 (Timeout)
>> 13141 12:54:24.811442 read(5, "backup is given)\n  numbered, t     make 
>> numbered backups\n  existing, nil   numbered if numbered backups exist, 
>> simple otherwise\n  simple, never   always make simple 
>> backups\n\0\0\0\nUsing -s
>> ignores -L and -P.  Otherwise, the last option specified 
>> controls\nbehavior when a TARGET is a symbolic link, defaulting to 
>> %s.\n\0\0\0\0\0\0\0Report any translation bugs to 
>> <https://translationproject.org/team/>\n\0\0https
>> ://www.gnu.org/software/coreutils/\0or available locally via: info 
>> '(coreutils) %s%s'\n\0\0\0\0\0\0multiple target directories 
>> specified\0\0\0cannot do --relative without --symbolic\0cannot combine 
>> --target-directory and --n
>> o-target-directory\0\0\0\0\0missing destination file operand after 
>> %s\0\0\0\0\0\0\0# buckets used:    %lu 
>> (%.2f%%)\n\0\0\0\0\0\0\0\0License GPLv3+: GNU GPL version 3 or later 
>> <%s>.\nThis is free software: you are free to cha
>> nge and redistribute it.\nThere is NO WARRANTY, to the extent permitted 
>> by 
>> law.\n\0\0\0\0\0\0https://gnu.org/licenses/gpl.html\0\0\0\0\0\0\0Written 
>> by %s, %s, %s,\nand %s.\n\0Written by %s, %s, %s,\n%s, and "..., 4096) 
>> = 409
>> 6
>> 13141 12:54:24.811797 pselect6(5, [], [4], [4], {tv_sec=1, tv_nsec=0}, 
>> NULL) = 0 (Timeout)
>> 13141 12:54:25.813136 read(5, 
>> "x\37\0\0\220\314\377\377\244\37\0\0\20\315\377\377\320\37\0\0\20\316\377\377\374\37\0\0P\316\377\377\30 
>> \0\0\220\316\377\3774 \0\0\260\316\377\377P \0\0\320\316\377\377l 
>> \0\0\20\317\377\377\230"...
>> , 4096) = 4096
>> 13141 12:54:25.813555 pselect6(5, [], [4], [4], {tv_sec=1, tv_nsec=0}, 
>> NULL) = 0 (Timeout)
>> 13141 12:54:26.814914 pselect6(5, [], [4], [4], {tv_sec=1, tv_nsec=0}, 
>> NULL) = 0 (Timeout)
>> 13141 12:54:27.816160 pselect6(5, [], [4], [4], {tv_sec=1, tv_nsec=0}, 
>> NULL) = 0 (Timeout)
>> 
>> 
>> 
>> What could be a reason of this problem with taking backup?
>> Marcin
>> 


> client throwed:


> ff2026-01-26 14:12:52 +0100: burp[30645] main socket 4: Got network 
> write error
> 2026-01-26 14:12:52 +0100: burp[30645] main socket 4: network write 
> problem in asfd_do_write_ssl: 5 - 110=Connection timed out
> 80ABF6F7A17F0000:error:8000006E:system 
> library:tls_retry_write_records:Connection timed 
> out:../openssl-3.5.4/ssl/record/methods/tls_common.c:1942:tls_retry_write_records 
> failure
> 2026-01-26 14:12:52 +0100: burp[30645] This is probably caused by the 
> peer exiting.
> 2026-01-26 14:12:52 +0100: burp[30645] Please check the peer's logs.
>   66

> --------------------------------------------------------------------------------
> Start time: 2026-01-26 13:56:40
>    End time: 2026-01-26 14:12:52
> Time taken: 16:12
>                           New   Changed Duplicate   Deleted     Total |  
> Scanned
                    
> ------------------------------------------------------------
>               Files:       66         0         0         0        66 |   
> 275105
>           Meta data:        0         0         0         0         0 |   
>   33803
>         Directories:        0         0         0         0         0 |   
>   20809
>          Hard links:        0         0         0         0         0 |   
>    1491
>          Soft links:        0         0         0         0         0 |   
>   13191
>       Special files:        0         0         0         0         0 |   
>    1227
>         Grand total:       66         0         0         0        66 |   
> 345626
                    
> ------------------------------------------------------------

>               Messages:             0
>               Warnings:            16

>        Bytes estimated:   15521570387 (14.46 GB)
>        Bytes in backup:      10926403 (10.42 MB)
>         Bytes received:          7737 (7.56 KB)
>             Bytes sent:      64739830 (61.74 MB)
> --------------------------------------------------------------------------------
> 2026-01-26 14:12:52 +0100: burp[30645] Error in phase 2
> 2026-01-26 14:12:52 +0100: burp[30645] Phase 2 end (send file data)


> server log:
> 026-01-26 13:56:48 +0100: burp[30646] End phase1 (file system scan)
> 2026-01-26 13:56:48 +0100: burp[30646] Begin phase2 (receive file data)
> 2026-01-26 14:12:52 +0100: burp[30646] main socket 8: Got network read 
> error
> 2026-01-26 14:12:53 +0100: burp[30646] main socket 8: network read 
> problem in asfd_do_read_ssl: 5 - 104=Connection reset by peer
> 2026-01-26 14:12:53 +0100: burp[30646] This is probably caused by the 
> peer exiting.
> 2026-01-26 14:12:53 +0100: burp[30646] Please check the peer's logs.
> 2026-01-26 14:12:53 +0100: burp[30646] error in backup_phase2_server_all
> 2026-01-26 14:12:53 +0100: burp[30646]   last tried file:    
> f:0073:/[snip]/.fastresume
> 2026-01-26 14:12:53 +0100: burp[30646] End phase2 (receive file data)
> 2026-01-26 14:12:53 +0100: burp[30646] error in backup phase 2
> 2026-01-26 14:12:53 +0100: burp[30646] Backup failed


> _______________________________________________
> Burp-users mailing list
> Bur...@li...
> https://lists.sourceforge.net/lists/listinfo/burp-users

Re: [Burp-users] backup of local server stucks

[Marcin M. <ma...@me...>]

W dniu 2026-01-26 14:04, Marcin Mirosław napisał(a):
> Hi,
> I saw a little similar problem described by Mark in thread "Timeout?" 
> but there were no clear solution nor description of root problem. I've 
> got burp server, it does backup network connected clients without 
> issue. Local client can't do backup since 2025-09-28, burp client seems 
> to stuck short after start of phase2.
> In backup directory there is a couple of files, client stuck, I think, 
> when i starts to read a little bigger file (about 2-3MB). I could 
> suspect something like MTU problem but connection is via 127.0.0.1 
> (also tried using ip of eth interface - no change) so MTU shouldn't be 
> case in this configuration. Strace shows:
> 
> n\0%s %s\n\0Packaged by %s (%"..., 4096) = 4096
> 13141 12:54:23.810144 pselect6(5, [], [4], [4], {tv_sec=1, tv_nsec=0}, 
> NULL) = 0 (Timeout)
> 13141 12:54:24.811442 read(5, "backup is given)\n  numbered, t     make 
> numbered backups\n  existing, nil   numbered if numbered backups exist, 
> simple otherwise\n  simple, never   always make simple 
> backups\n\0\0\0\nUsing -s
>  ignores -L and -P.  Otherwise, the last option specified 
> controls\nbehavior when a TARGET is a symbolic link, defaulting to 
> %s.\n\0\0\0\0\0\0\0Report any translation bugs to 
> <https://translationproject.org/team/>\n\0\0https
> ://www.gnu.org/software/coreutils/\0or available locally via: info 
> '(coreutils) %s%s'\n\0\0\0\0\0\0multiple target directories 
> specified\0\0\0cannot do --relative without --symbolic\0cannot combine 
> --target-directory and --n
> o-target-directory\0\0\0\0\0missing destination file operand after 
> %s\0\0\0\0\0\0\0# buckets used:    %lu 
> (%.2f%%)\n\0\0\0\0\0\0\0\0License GPLv3+: GNU GPL version 3 or later 
> <%s>.\nThis is free software: you are free to cha
> nge and redistribute it.\nThere is NO WARRANTY, to the extent permitted 
> by 
> law.\n\0\0\0\0\0\0https://gnu.org/licenses/gpl.html\0\0\0\0\0\0\0Written 
> by %s, %s, %s,\nand %s.\n\0Written by %s, %s, %s,\n%s, and "..., 4096) 
> = 409
> 6
> 13141 12:54:24.811797 pselect6(5, [], [4], [4], {tv_sec=1, tv_nsec=0}, 
> NULL) = 0 (Timeout)
> 13141 12:54:25.813136 read(5, 
> "x\37\0\0\220\314\377\377\244\37\0\0\20\315\377\377\320\37\0\0\20\316\377\377\374\37\0\0P\316\377\377\30 
> \0\0\220\316\377\3774 \0\0\260\316\377\377P \0\0\320\316\377\377l 
> \0\0\20\317\377\377\230"...
> , 4096) = 4096
> 13141 12:54:25.813555 pselect6(5, [], [4], [4], {tv_sec=1, tv_nsec=0}, 
> NULL) = 0 (Timeout)
> 13141 12:54:26.814914 pselect6(5, [], [4], [4], {tv_sec=1, tv_nsec=0}, 
> NULL) = 0 (Timeout)
> 13141 12:54:27.816160 pselect6(5, [], [4], [4], {tv_sec=1, tv_nsec=0}, 
> NULL) = 0 (Timeout)
> 
> 
> 
> What could be a reason of this problem with taking backup?
> Marcin
> 


client throwed:


ff2026-01-26 14:12:52 +0100: burp[30645] main socket 4: Got network 
write error
2026-01-26 14:12:52 +0100: burp[30645] main socket 4: network write 
problem in asfd_do_write_ssl: 5 - 110=Connection timed out
80ABF6F7A17F0000:error:8000006E:system 
library:tls_retry_write_records:Connection timed 
out:../openssl-3.5.4/ssl/record/methods/tls_common.c:1942:tls_retry_write_records 
failure
2026-01-26 14:12:52 +0100: burp[30645] This is probably caused by the 
peer exiting.
2026-01-26 14:12:52 +0100: burp[30645] Please check the peer's logs.
  66

--------------------------------------------------------------------------------
Start time: 2026-01-26 13:56:40
   End time: 2026-01-26 14:12:52
Time taken: 16:12
                          New   Changed Duplicate   Deleted     Total |  
Scanned
                    
------------------------------------------------------------
              Files:       66         0         0         0        66 |   
275105
          Meta data:        0         0         0         0         0 |   
  33803
        Directories:        0         0         0         0         0 |   
  20809
         Hard links:        0         0         0         0         0 |   
   1491
         Soft links:        0         0         0         0         0 |   
  13191
      Special files:        0         0         0         0         0 |   
   1227
        Grand total:       66         0         0         0        66 |   
345626
                    
------------------------------------------------------------

              Messages:             0
              Warnings:            16

       Bytes estimated:   15521570387 (14.46 GB)
       Bytes in backup:      10926403 (10.42 MB)
        Bytes received:          7737 (7.56 KB)
            Bytes sent:      64739830 (61.74 MB)
--------------------------------------------------------------------------------
2026-01-26 14:12:52 +0100: burp[30645] Error in phase 2
2026-01-26 14:12:52 +0100: burp[30645] Phase 2 end (send file data)


server log:
026-01-26 13:56:48 +0100: burp[30646] End phase1 (file system scan)
2026-01-26 13:56:48 +0100: burp[30646] Begin phase2 (receive file data)
2026-01-26 14:12:52 +0100: burp[30646] main socket 8: Got network read 
error
2026-01-26 14:12:53 +0100: burp[30646] main socket 8: network read 
problem in asfd_do_read_ssl: 5 - 104=Connection reset by peer
2026-01-26 14:12:53 +0100: burp[30646] This is probably caused by the 
peer exiting.
2026-01-26 14:12:53 +0100: burp[30646] Please check the peer's logs.
2026-01-26 14:12:53 +0100: burp[30646] error in backup_phase2_server_all
2026-01-26 14:12:53 +0100: burp[30646]   last tried file:    
f:0073:/[snip]/.fastresume
2026-01-26 14:12:53 +0100: burp[30646] End phase2 (receive file data)
2026-01-26 14:12:53 +0100: burp[30646] error in backup phase 2
2026-01-26 14:12:53 +0100: burp[30646] Backup failed

[Burp-users] backup of local server stucks

[Marcin M. <ma...@me...>]

Hi,
I saw a little similar problem described by Mark in thread "Timeout?" 
but there were no clear solution nor description of root problem. I've 
got burp server, it does backup network connected clients without issue. 
Local client can't do backup since 2025-09-28, burp client seems to 
stuck short after start of phase2.
In backup directory there is a couple of files, client stuck, I think, 
when i starts to read a little bigger file (about 2-3MB). I could 
suspect something like MTU problem but connection is via 127.0.0.1 (also 
tried using ip of eth interface - no change) so MTU shouldn't be case in 
this configuration. Strace shows:

n\0%s %s\n\0Packaged by %s (%"..., 4096) = 4096
13141 12:54:23.810144 pselect6(5, [], [4], [4], {tv_sec=1, tv_nsec=0}, 
NULL) = 0 (Timeout)
13141 12:54:24.811442 read(5, "backup is given)\n  numbered, t     make 
numbered backups\n  existing, nil   numbered if numbered backups exist, 
simple otherwise\n  simple, never   always make simple 
backups\n\0\0\0\nUsing -s
  ignores -L and -P.  Otherwise, the last option specified 
controls\nbehavior when a TARGET is a symbolic link, defaulting to 
%s.\n\0\0\0\0\0\0\0Report any translation bugs to 
<https://translationproject.org/team/>\n\0\0https
://www.gnu.org/software/coreutils/\0or available locally via: info 
'(coreutils) %s%s'\n\0\0\0\0\0\0multiple target directories 
specified\0\0\0cannot do --relative without --symbolic\0cannot combine 
--target-directory and --n
o-target-directory\0\0\0\0\0missing destination file operand after 
%s\0\0\0\0\0\0\0# buckets used:    %lu (%.2f%%)\n\0\0\0\0\0\0\0\0License 
GPLv3+: GNU GPL version 3 or later <%s>.\nThis is free software: you are 
free to cha
nge and redistribute it.\nThere is NO WARRANTY, to the extent permitted 
by 
law.\n\0\0\0\0\0\0https://gnu.org/licenses/gpl.html\0\0\0\0\0\0\0Written 
by %s, %s, %s,\nand %s.\n\0Written by %s, %s, %s,\n%s, and "..., 4096) = 
409
6
13141 12:54:24.811797 pselect6(5, [], [4], [4], {tv_sec=1, tv_nsec=0}, 
NULL) = 0 (Timeout)
13141 12:54:25.813136 read(5, 
"x\37\0\0\220\314\377\377\244\37\0\0\20\315\377\377\320\37\0\0\20\316\377\377\374\37\0\0P\316\377\377\30 
\0\0\220\316\377\3774 \0\0\260\316\377\377P \0\0\320\316\377\377l 
\0\0\20\317\377\377\230"...
, 4096) = 4096
13141 12:54:25.813555 pselect6(5, [], [4], [4], {tv_sec=1, tv_nsec=0}, 
NULL) = 0 (Timeout)
13141 12:54:26.814914 pselect6(5, [], [4], [4], {tv_sec=1, tv_nsec=0}, 
NULL) = 0 (Timeout)
13141 12:54:27.816160 pselect6(5, [], [4], [4], {tv_sec=1, tv_nsec=0}, 
NULL) = 0 (Timeout)



What could be a reason of this problem with taking backup?
Marcin

Re: [Burp-users] Large file problems?

[Mark S. <ma...@st...>]

Graham and Brendan,

I'm fairly certain that the latency is causing the problem.  I have
switched my backup medium from my external disk in a USB-2 connected
hub to an internal disk and now it works flawlessly.

In this case I think it's the comms latency that's causing the problem
not the disk, but perhaps that's splitting hairs.

Also, I no longer think it's (necessarily) to do with large files.  In
my case it was stopping while backing up a (long) run of zero byte
files.

Mark

Re: [Burp-users] Large file problems?

[Brendon H. <br...@qu...>]

Hi Graham,

On Sunday, January 25, 2026 7:47:39 p.m. Eastern Standard Time Graham Keeling 
wrote:
> It could well be possible that some particular form of disk latency could
> cause a burp backup process to get jammed.

What magnitude of latency would you expect that might cause this, though? I'd 
presume a few seconds should be fine. So I just now observed how long it takes 
for burp to write the file up to the point where it seems to get stuck, using 
ls and sleep 1 in a while loop. From the first sign of burp resuming progress 
up to the maximum size takes 19 seconds. While that isn't speedy for 100MB, 
manually running gzip -9 on the file takes a similar amount of time, so I think 
that's the bottle-neck. I double-checked that I can copy this file to the same 
medium myself - and there's no latency when I did so - all while burp client 
and server still have their corresponding files open, according to lsof.

I just noticed that the client times-out about two minutes prior to the server 
- or at least it did this time around. Not sure if that indicates anything. 
The timestamp in the failure email from cron is 998 seconds after the time I 
observed the file stopped changing size, so I bet the timeout is exactly 1000 
seconds and my measurement uncertainty is around 1-2 seconds.

> In order to fully rule out the backup medium causing burp to behave poorly,
> have you tried using a different backup medium?

I have not. Even if I wasn't skeptical that it's a problem with the medium, 
I'm actually not sure if I have something else available that would fit the 
bill.

> Meanwile, if I were able to reproduce it myself, I would be putting debug
> into various places in the code, recompiling, and trying to track down
> exactly what is getting stuck. 

I'm confident I could recompile the binary with debug statements, although I 
won't have more time to play with this again until next weekend. I wouldn't 
know where to put those debug statements, though. Happy to take direction if 
you have suggestions.

Best,
Brendon

Re: [Burp-users] Large file problems?

[Graham K. <gr...@gr...>]

On Sun, Jan 25, 2026 at 01:04:41PM -0500, Brendon Higgins wrote:
> Hi all,
> 
> On Saturday, December 13, 2025 12:46:36 p.m. Eastern Standard Time Brendon 
> Higgins wrote:
> > What I *am* noticing is that the backup medium is acting very slowly (curse
> > shingled magnetic recording!), and dmesg shows some statements about 
> several
> > tasks - one of them burp - being blocked for more than 120 seconds, and
> > that "task burp:8409 is blocked on a mutex likely owned by task burp:4996".
> 
> I have more observations from this weekend's (failed) backup attempt which 
> indicate that, in my case, the backup medium is NOT too slow. I'm convinced 
> this was a red herring because I can see (using `sudo lsof -c burp`) burp 
> getting stuck at a particular file, and can manually copy that file to the 
> medium, with a `sync`, and it only takes a few seconds. The disk is clearly 
> fine at the moment, and burp is running the entire time - it's just not doing 
> anything (near-zero CPU activity, IO activity, socket activity), and 
> eventually (~15 minutes) times out, just like Mark had described observing. 
> And I'm not seeing those dmesg "blocked on a mutex" lines anymore, either.
> 
> I've now let burp continue operating overnight, attempting to resume, timing 
> out after about 15 minutes, and restarting at the next (also 15 minute) cron 
> cycle. It's gone about 40 cycles, now. When I probe with lsof, it's always 
> stuck on the same file, /usr/lib/chromium/chromium . This isn't even the 
> largest file burp has copied into the working backup, although it is the 
> largest one which is being compressed (the others have file-type exclusions).
> 
> This file also always seems to get stuck at a little over 100 MB saved to the 
> backup (under data.tmp), although it can be slightly different each time, 
> varying by a few tens of kB (although one time might have only gotten to 89 
> MB). On resume it seems burp retries the file from its beginning, as I've 
> noticed the size get smaller a few times. The file is certainly incomplete; 
> gunzip complains about unexpected end of file. If I gzip the original myself, 
> the result comes to about 114 MB - so roughly 14 MB is missing.
> 
> I'm only seeing this happen with the client and server being on the same 
> machine. Other machines on my network have no evidence of such failures - no 
> signs of "resumed" logs in their backup folders. I have one Windows machine 
> with burp 2.4.0, which has a similarly sized backup, and a few other Linux 
> machines with smaller backups, some with 2.4.0 and others with 3.1.4. (I even 
> increased max_children to accommodate all of them at once, and beside this 
> local client, there was no problem.)
> 
> Other things I tried which seemed to make no difference: switching librsync on, 
> and changing compression level from zlib6 to zlib9 (the ~100 MB failure point 
> even seemed to remain the same).
> 
> I'm not quite sure what to do at this point. The backup is not progressing, 
> which is a concern, but given that it's reproducible in this state perhaps 
> there's some other aspect I could look more closely at which could yield more 
> useful info?
> 
> Best,
> Brendon

Hello,

It could well be possible that some particular form of disk latency could cause
a burp backup process to get jammed.

In order to fully rule out the backup medium causing burp to behave poorly,
have you tried using a different backup medium?

Meanwile, if I were able to reproduce it myself, I would be putting debug into
various places in the code, recompiling, and trying to track down exactly what
is getting stuck. I am currently not able to reproduce the problem, but I have
an idea to do with setting up a loopback medium such as described here:

https://serverfault.com/questions/523509/linux-how-to-simulate-hard-disk-latency-i-want-to-increase-iowait-value-withou

Re: [Burp-users] Large file problems?

[Brendon H. <br...@qu...>]

Hi all,

On Saturday, December 13, 2025 12:46:36 p.m. Eastern Standard Time Brendon 
Higgins wrote:
> What I *am* noticing is that the backup medium is acting very slowly (curse
> shingled magnetic recording!), and dmesg shows some statements about 
several
> tasks - one of them burp - being blocked for more than 120 seconds, and
> that "task burp:8409 is blocked on a mutex likely owned by task burp:4996".

I have more observations from this weekend's (failed) backup attempt which 
indicate that, in my case, the backup medium is NOT too slow. I'm convinced 
this was a red herring because I can see (using `sudo lsof -c burp`) burp 
getting stuck at a particular file, and can manually copy that file to the 
medium, with a `sync`, and it only takes a few seconds. The disk is clearly 
fine at the moment, and burp is running the entire time - it's just not doing 
anything (near-zero CPU activity, IO activity, socket activity), and 
eventually (~15 minutes) times out, just like Mark had described observing. 
And I'm not seeing those dmesg "blocked on a mutex" lines anymore, either.

I've now let burp continue operating overnight, attempting to resume, timing 
out after about 15 minutes, and restarting at the next (also 15 minute) cron 
cycle. It's gone about 40 cycles, now. When I probe with lsof, it's always 
stuck on the same file, /usr/lib/chromium/chromium . This isn't even the 
largest file burp has copied into the working backup, although it is the 
largest one which is being compressed (the others have file-type exclusions).

This file also always seems to get stuck at a little over 100 MB saved to the 
backup (under data.tmp), although it can be slightly different each time, 
varying by a few tens of kB (although one time might have only gotten to 89 
MB). On resume it seems burp retries the file from its beginning, as I've 
noticed the size get smaller a few times. The file is certainly incomplete; 
gunzip complains about unexpected end of file. If I gzip the original myself, 
the result comes to about 114 MB - so roughly 14 MB is missing.

I'm only seeing this happen with the client and server being on the same 
machine. Other machines on my network have no evidence of such failures - no 
signs of "resumed" logs in their backup folders. I have one Windows machine 
with burp 2.4.0, which has a similarly sized backup, and a few other Linux 
machines with smaller backups, some with 2.4.0 and others with 3.1.4. (I even 
increased max_children to accommodate all of them at once, and beside this 
local client, there was no problem.)

Other things I tried which seemed to make no difference: switching librsync on, 
and changing compression level from zlib6 to zlib9 (the ~100 MB failure point 
even seemed to remain the same).

I'm not quite sure what to do at this point. The backup is not progressing, 
which is a concern, but given that it's reproducible in this state perhaps 
there's some other aspect I could look more closely at which could yield more 
useful info?

Best,
Brendon

Re: [Burp-users] EVP_CipherInit_ex failed

[Graham K. <gr...@gr...>]

>    On Sun, Dec 4, 2022, at 6:47 PM, hnsz2002 wrote:
> 
>      Hi Graham!
>      It is working now, Iegacy modules are enabled in
>      /etc/pki/tls/openssl.conf as described, with uncommenting the following
>      lines:
> 
>     [provider_sect]
>     default = default_sect
>     legacy = legacy_sect
>    
>     [default_sect]
>     activate = 1
>    
>     [legacy_sect]
>     activate = 1
> 
>      Thank you, have a nice day!
>      02.12.22 12:03, Graham Keeling пише:
> 
>  On Fri, Dec 02, 2022 at 11:41:18AM +0100, hns...@gm... wrote:
> 
>  Hi all!
> 
>  Yesterday I updated my desktop machine from fedora 35 to 36, maybe since
>  then burp doesn't work, the backup is failed in phase 2.
> 
>  2022-12-02 11:38:14 +0100: burp[19882] Phase 2 begin (send backup data)
> 
>  2022-12-02 11:38:15 +0100: burp[19882] EVP_CipherInit_ex failed
>  2022-12-02 11:38:15 +0100: burp[19882] Error in phase 2
>  2022-12-02 11:38:15 +0100: burp[19882] Phase 2 end (send file data)
> 
>  Any other things (configs, server os, networks, etc) are not changed at all.
> 
>  What can be the problem?
> 
>  Hello,
> 
>  This page indicates that fedora 36 updates to openssl-3.0:
>  https://packages.fedoraproject.org/pkgs/openssl/openssl/
> 
>  And openssl-3.0 deprecates blowfish support, which is what burp used to use
>  for encrypting files.
>  There is information here about a "legacy provider" and how to load it in
>  order to get the blowfish support back:
>  https://wiki.openssl.org/index.php/OpenSSL_3.0
>  I haven't tried that, but I imagine it would work.
> 
>  In burp version 3.1.2, I changed to AES-CBC-256 for the encryption instead.
>  You will still be able to decrypt your files as long as your openssl library
>  supports it.
> 
>  The Windows installer will continue to come with openssl-1.1 for a few months,
>  to allow time for people to switch from blowfish.

On Fri, Jan 23, 2026 at 05:35:07PM +0100, Pascal Suter wrote:
>    Hi Graham 
>    i use hardlinked backups and I just ran into the same issue on a backup of
>    a machine that started its life before openssl 3 and burp 3.1.2. After a
>    few os upgrades and also a burp upgrade i now have a mix of old and new
>    encrypted files in my backup, so i can restore some of the files without
>    enabling the legacy provider and some other files won't restore unless i
>    enable the legacy stuff :) 
>    it would be great to be able to trigger an update of all those old files
>    in my backups. if diskspace and bandwith are not an issue, i guess the
>    simplest solution would just be to somehow trigger a full backup and then
>    continue from there with incrementals. however, at least for me this is
>    not an option, my backups are far too heavy to do that. It would take
>    forever and i would have to basically delete all my old backups to have
>    enough space for a re-start. 
>    is there a way to tell which files in my backups are still blowfish
>    encrypted? like, is it stored in the manifest somewhere for example? 
>    If i can identify which files are still blowfish encrypted, how can i
>    trigger them to be requested from the client again in the next backup run?
>    can i delete them from the manifest or from the data directory of the last
>    successufl backup? 
>    i could probably write a bash script or something like that to kick those
>    old files out and have them replaced. 
>    It would of course be much more elegant, if the server could just
>    overwrite old blowfish encrypted files with new AES encrypted ones. I
>    understand the server can't do this alone though, because he does not have
>    the key to decrypt the old files. But as far as I understand burp, the
>    server gets to tell the client which files the client needs to send him to
>    update the backup to the latest state. So the server could request a fresh
>    copy of a blowfish encrypted file even though it has not changed. On
>    hardlinked backups, the server could then cache the file and once it is
>    received completely, copy it over the old symlinked file. this would
>    update the file in-place in all old backups. This fixes the storage space
>    issue, we don't need any additional space because we replace the old copy
>    with the new one. 
>    to solve the bandwidth issue, we could leverage, that this is currently
>    not such a pressing issue, because we still have the legacy provider
>    available, so we can take our time. A threshold could be defined, telling
>    the server how much overhead he is allowed to generate on each backup
>    iteration to replace old files.. So for example allow the server to ask
>    for about 100GB of additional files on each iteration. 
>    if the sserver does not know which files are blowfish encrypted, i could
>    probably just go by file change date on the server in the data directory
>    of the backup once i find out when the update to burp >= 3.1.2 happend and
>    then get a list of files that need upating this way 
>    cheers
>    Pascal 

Hello Pascal,

This is an interesting idea.

Yes, burp stores the type of encryption in the attributes line for each file
represented in the backup's manifest.

The burp server would be able to look at the previous file's encryption flag
and choose to backup again depending on what it is.

Here is an example of an attributes line:
r0040gR nOAl EHt E Po Po A BAA BAA I BpUanE BnExHZ BnExHZ A A J A A A

It starts with 'r', then the next 4 characters are the length of the attributes
line. Then the rest of the characters are various attributes, base64 encoded.
The encryption flag is in the 17th place. In this example, it is the 'A' after
the 'J'.

The possible (decoded) values that this field come out of burp's src/sbuf.h
definitions:
#define ENCRYPTION_UNSET                        -1 // Also legacy
#define ENCRYPTION_NONE                         0
#define ENCRYPTION_KEY_DERIVED_BF_CBC           1 // Legacy
#define ENCRYPTION_KEY_DERIVED_AES_CBC_256      2

So, in the example, 'A' is 0 means ENCRYPTION_NONE.
ENCRYPTION_UNSET:                   ?
ENCRYPTION_NONE:                    A
ENCRYPTION_KEY_DERIVED_BF_CBC:      B
ENCRYPTION_KEY_DERIVED_AES_CBC_256: C

Here is an example that has AES:
r004AgB pAm0 IGk B Po Po A Lm BAA I BpdBpE BpdBpE BpdBpE A A A C -FZJuSchDOfD A

In older versions of burp, the encryption flag wasn't encoded into the
attributes. In that case, you can look at the file type row in the manifest:
y0023/home/graham/burp/test/build/CA.cnf
The leading 'y' means it's encrypted, and the blowfish encryption is implied.


Given the information above, it would be possible to manually edit the current
manifest to identify and delete the lines that represent blowfish encrypted
files. Burp would then back them up again on the next backup. If doing this, I
would also keep a copy of the original manifest and replace the edited manifest
again when done.

Obviously, having the feature to do it automatically a little at a time would
be much easier!
I will have a look to see if I can do this.