exit code 2 if a memory or drive error

[alex]

Hello.
1) Do I understand correctly that 7-Zip will exit with exit code 2 if a memory or drive error occurs during the packaging process?
2) is there a chance that with an exit code of 0, the archive may be created corrupted?

I need to understand whether I should test backup archives immediately after creation.

[Igor Pavlov]

It's low probability to get some undetected failure.
For example, RAM failure can be reason of data corruption.
But such RAM/CPU failures are rare cases, if CPU and RAM are not overclocked.

Drive error is also possible. It will be not detected by 7-zip.
But you need to flush system's write/read cache to test it correctly.
By default Windows will not read real data from drive after writing. Instead Windows will read data from cache.
So you must do 3 steps for full testing:
1) create archive
2) flush system cache
3) test archive

But cache flush operation will slow down whole system, because
another useful cached data must be read again after that.

[alex]

cause memory problems:
the appearance of oxides on the contacts ram.
dust in ram slots.
motherboard malfunctions (for example, swollen electrolytes).

that is, the only way to check the correctness of compression without clearing the windows cache is by counting the hashes of the files from the archive and comparing them with the hashes of the originals?
Or will that not help either?

[Igor Pavlov]

"by counting the hashes of the files from the archive"
you will not get real data from drive, you will get compressed data from windows cache.
To exclude drive failures you must clear file cache after writing and before reading.
To exclude another errors you can count hashes.

[alex]

and there is no way to cheat the cache?
add archive to archive?
or rename the file in the archive before unpacking (if it's only 1)?

[Igor Pavlov]

File cache in Windows increases the speed.
RAM can be 20 GB/s
drive can be 200 MB/s.
So instead of slow 200 MB/s, you get fast 20 GB/s.
So it can be 100 times faster to read from RAM instead of drive.
Instead of reading from drive, it reads from cache.
So it's very important feature of system. And it works by default for all operations.

[alex]

about the speed is understandable, but it is important for me to know that the archive with the backup is ok.
maybe you know if there is something better for flushing the cache than sync from sysinternals?
and how then does the test archive option work immediately after creation in winrar? cleans the cache or ... just creates the appearance of a test?

[Igor Pavlov]

As I wrote before, archive testing after creation can catch RAM failures, but it doesn't catch all drive failures if file file cache is larger than archive size.
That problem is not related to 7-zip or Winrar.
You will have same problem with any drive write/read operations.
For example, if you copy some file from one directory to another, you can't be sure that there is no data corruption in written data, while file cache provides data for reading.

[alex]

Thank you.

On the problems with the drive, everything is clear.

2 questions left.

1) will 7-zip detect memory problems when creating an archive, and will the exit code be 2? or what would be the behavior of 7-zip? Is it possible to get a broken archive without any errors?

2) why not reset the cache for a specific file? i'm not a programmer but winapi seems to allow it https://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-flushfilebuffers
Is there any plan to add such a feature? so that the archive test shows its current state.
this would be useful for backup scripts too (why use sysinternals sync utility for everything if I need a specific file).

[Igor Pavlov]

1) 7-zip at comperssion process can detect only some minor part of possible memory failures. And you have exit code=2 only for these detected cases.
If you calculate hashes of original data and data in archive you can detect most of memory failures.

2) I'm note sure about api, but it's possible that this flushing doesn't solve your problem, because it only writes data. But there is no guarantee that reading will be from drive. You must clear the cache to be sure about it.
Cache for Write operations does two things:
1) it suspends real writing
2) it reads data from cache instead of real reading from drive.
So flushing enforces real writing, but reading still can be from cache.

[alex]

sync from sysinternals allows you to specify the drive letter to clean up.
if the source files are on drive c and the archive is saved on drive d, what drive letter should i specify in sync?

although it is not a fact that reading will not occur from the cache...

[alex]

I created a test archive and immediately tested it with 7-zip and winrar.
the archive is located on a samsung 870 evo 1 tb, which has a sequential read speed of up to 560 mb/s.

7-zip showed a speed of about 900 mb/s, and this is definitely a cache. Drive activity indicator is off.
winrar does not show the speed, but the indicator is on all the time of testing.

I checked it several times.
I conclude that winrar can bypass the cache and read directly from the disk.

is there any chance for this feature to appear in 7-zip?
an archive test should show its true state on disk.

[Igor Pavlov]

cache uses only part of availabe RAM.
do more tests with winrar with smaller archives.

[alex]

my previous test was with a 300 mb archive.
I repeated the test with a 100 mb archive, the result is the same.

[alex]

I did 3 tests, this time with a 20 mb archive on a usb 2.0 flash drive.
flash drive with activity indicator.
7-zip reads the archive from the cache.
wirar reads the archive from the flash drive.

in addition, when testing such a small archive, 7-zip does not show a progress window, only a result window (does not depend on the location of the archive).

[Igor Pavlov]

I did another test with winrar.
1 GB file with "Store" compression method (no compression).
"Test archived files" option switched ON.
Compressing and testing was just 1 sec in WinRAR on HDD.
That HDD provides less than 100 MB/s.
Windows 10.
So Winrar also tests from cache instead of real data reading after writing. You can test it also.

[alex]

7-zip 22.01 x64
winrar 6.21 x64
windows 10 pro 21h2
samsung 870 evo 1 tb, which has a sequential read speed of up to 560 mb/s.
1 GB uncompressed archive created in 7-zip.
archive test from the explorer context menu item.

7-zip - the drive indicator is off.
winrar - the indicator is on continuously.
7-zip - the drive indicator is off.
winrar - the indicator is on continuously.
and so on.

The screenshots show the difference in testing time.

[Igor Pavlov]

Yes, Test operation in new winrar clears cache for that archive file.
This new feature that was included to WINRAR 6.10:

  16. WinRAR attempts to reset the file cache before testing an archive.
      It helps to verify actual data written to disk instead of reading
      a cached copy.

But "Test archived files" option in Winrar doesn't clear that file cache, after archive creation.
So "Test archived files" still reads from cache.

Now I see in source code of unrar these lines:

void ResetFileCache(const wchar *Name)
{
  // To reset file cache in Windows it is enough to open it with
  // FILE_FLAG_NO_BUFFERING and then close it.
  HANDLE hSrc=CreateFile(Name,GENERIC_READ,
                         FILE_SHARE_READ|FILE_SHARE_WRITE,
                         NULL,OPEN_EXISTING,FILE_FLAG_NO_BUFFERING,NULL);
  if (hSrc!=INVALID_HANDLE_VALUE)
    CloseHandle(hSrc);
}

But now I'm not sure that I want that feature in 7-zip.
I'll think about it later.

That function is simple.
So probably some external tool can clear the cache for any file.
if you can find such tool, you can call it before calling 7-zip.

[alex]

unfortunately, clearing the entire cache will lead to a performance loss, and removing a specific file from the cache can only be done by the program that placed it there.

this is exactly what the unrar code does, if I understand correctly.

what bothers you about this decision?

[Igor Pavlov]

You can clear cache for one file with rar.
Just call rar between 7z calls:

7z a a.7z file
rar t a.7z
7z t a.7z

rar doesn't test 7z archive, but it clears the cache for that file.

[alex]

good crutch.
better than xcopy /j, but still a crutch ;)