I have a new problem with large file transfers. After adding MD5 checksums to my file transfer program I noticed that files above a certain size ended up with different MD5 sums on the receiver's end. This already happens with relatively small files, on the order of 20MB - 40MB.
When comparing two files byte-by-byte after a transfer there is a point after which the files differ through the end. Unfortunately, that point varies quite a bit. In my current test scenario using the same file this happened at byte number 28.311.552, at byte 35.651.584, and at byte 3.145.728.
The file corruption seems to coincide with a momentary and rapid decrease in file transfer speed. That's all I can see at the moment. I will try to see if I can find the problem.
Thanks,
Alexander
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
It seems that in cases where a file is corrupted there are an extra 1312 zeroes inserted at a random spot in the received file. The byte sequence following the extra 0's matches the original file again.
I have added individual checksums to every packet and verified each packet is received correctly on the receiver's side.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I have been trying to track down what happens with the files; but the more tests I added the less frequently this has become an issue. In fact, the lest few weeks nothing like that happened at all, even with large file transfers between the UK and Australia.
The only thing I could think of was that the receive buffer in my client had a size that was not a multiple of the send buffer, so that occasionally a partial buffer needed to be written do disk, and under certain circumstances the remaining buffer was written as zeroes. So I am fairly certain it was an issue with the receiver buffer handling.
Thanks,
Alexander
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hello,
I have a new problem with large file transfers. After adding MD5 checksums to my file transfer program I noticed that files above a certain size ended up with different MD5 sums on the receiver's end. This already happens with relatively small files, on the order of 20MB - 40MB.
When comparing two files byte-by-byte after a transfer there is a point after which the files differ through the end. Unfortunately, that point varies quite a bit. In my current test scenario using the same file this happened at byte number 28.311.552, at byte 35.651.584, and at byte 3.145.728.
The file corruption seems to coincide with a momentary and rapid decrease in file transfer speed. That's all I can see at the moment. I will try to see if I can find the problem.
Thanks,
Alexander
It seems that in cases where a file is corrupted there are an extra 1312 zeroes inserted at a random spot in the received file. The byte sequence following the extra 0's matches the original file again.
I have added individual checksums to every packet and verified each packet is received correctly on the receiver's side.
hi,
sorry for the late reply, I'm rather tied up with other ("real") work at the moment. Maybe the receiver side buffer handling is buggy…
Bernd
Hello,
I understand! I am rather busy as well.
I have been trying to track down what happens with the files; but the more tests I added the less frequently this has become an issue. In fact, the lest few weeks nothing like that happened at all, even with large file transfers between the UK and Australia.
The only thing I could think of was that the receive buffer in my client had a size that was not a multiple of the send buffer, so that occasionally a partial buffer needed to be written do disk, and under certain circumstances the remaining buffer was written as zeroes. So I am fairly certain it was an issue with the receiver buffer handling.
Thanks,
Alexander