#99 Cannot open RAR 5 files

[Ark]

Trying to open RAR 5 files results in "No images in '<archive>'" even if unrar (non-free) is able to open it and installed. Note that 7z cannot open such archives either:</archive>

[...] Open Error: Can not open the file as [Rar] archive

Errors:
Is not archive

[Ark]

An example archive can be created this way (see attachment):

rar a -ma5 reds_RAR5.rar red0.png red1.png red2.png

Note the archive version number when listing the contents using unrar:

Details: RAR 5

I guess that MComix prefers 7z over unrar, hence the error. (I am not sure, though. Maybe it's because I don't have libunrar installed.) Adding both unencrypted and encrypted RAR 5 archives to the test files might be helpful. (Currently, all rar files are RAR 4 files.)

EDIT I just tested it on Windows and it seems that it is not an issue there. /EDIT

[Benoit Pierre]

Yes, with a recent version of 7z it works fine. Unicode support is better for 7z, which why it's prefered over unrar.

[Ark]

Turns out: The dependecies are currently quite a mess. I tested all possible combinations, and these are the results:

Arch Linux

p7zip	libunrar	unrar	RAR 4	RAR 5	7z

• | - | - | Y(!)| Y(!)| -
• | - | Y | Y | Y | -
• | Y | - | Y | Y | -
• | Y | Y | Y | Y | -
  Y | - | - | - | - | Y
  Y | - | Y | Y | -(!)| Y
  Y | Y | - | Y | Y | Y
  Y | Y | Y | Y | Y | Y

(A dash means "not available/not supported" while "Y" means "available/supported".)

Note the Y's in the first line: I can open RAR 4, RAR 5 and 7z without having one of the listed packages installed. (In other words, there might be false positives in the RAR 4 and RAR 5 columns due to a strange testing environment. Sorry.) Also note that as soon as I install p7zip, I cannot open RAR 4 or RAR 5 anymore. If I also install unrar (which itself is able to open RAR 5), I can only open RAR 4 but not RAR 5 with MComix. The only configurations where I can open everything are those where p7zip and libunrar are installed.

Debian Stretch (testing)

p7zip-full	p7zip-rar	unrar	unrar-free	RAR 4	RAR 5	7z

 -     |     -     |   -   |      -     |   -   |   -   |  - 
 -     |     -     |   -   |      Y     |   -(!)|   -   |  - 
 -     |     -     |   Y   |      -     |   Y   |   Y   |  - 
 -     |     -     |   Y   |      Y     |   Y   |   Y   |  - 
 Y     |     -     |   -   |      -     |   -   |   -   |  Y 
 Y     |     -     |   -   |      Y     |   -(!)|   -   |  Y 
 Y     |     -     |   Y   |      -     |   -(!)|   -(!)|  Y 
 Y     |     -     |   Y   |      Y     |   -(!)|   -(!)|  Y 
 Y     |     Y     |   -   |      -     |   Y   |   -   |  Y 
 Y     |     Y     |   -   |      Y     |   Y   |   -   |  Y 
 Y     |     Y     |   Y   |      -     |   Y   |   -(!)|  Y 
 Y     |     Y     |   Y   |      Y     |   Y   |   -(!)|  Y

(Note: I left out 4 combinations since p7zip-rar depends on p7zip-full.)

RAR 5 is not supported at all but unrar itself is able to extract files from such archives. I guess this is due to p7zip-rar (probably?) being unable to handle RAR 5. Note that this is quite bad for Debian users as they cannot open RAR 5 and 7z at the same time. Similar to Arch, installing p7zip-full disables support for RAR 4, unless it is re-enabled by also installing p7zip-rar.

Another thing is that unrar-free itself is able to open RAR 4 archives but MComix with only unrar-free installed is not. Certain things indicate that this is due to parsing errors. I will add another post here with more details later.

TL;DR In other words, MComix does not try to use unrar if extracting with 7z fails. Retrying with unrar might be an approach to work around this issue.

[Benoit Pierre]

I'm confused, how do you read your tables? Shouldn't you put formats on rows, and implementations on colums?

There is also Unicode support. And the fast that unrar-free does not support the same set of option (no extract to stdout...). And that's not taking also into account the possible differences in output...

Retrying is going to be hell, retrying on what type of error? Listing works, extracting may fails, or it may report a success, but have failed (with unrar-free)... We could differentiate the types returned by archive_mime_type, like it's done for ZIP_EXTERNAL/ZIP: instead of RAR, return eitheir RAR3, RAR4, or RAR5.

[Benoit Pierre]

If we read byte 44 we can get the version of the format.

[Benoit Pierre]

Or simply prefer only prefer 7z over unrar for Windows, and unrar over 7z for other platforms.

Or always prefer unrar over 7z.

Because without knowing the version of 7z plus wether or not rar support for 7z is available, we cannot decide which is better (7z or unrar), so...

[Benoit Pierre]

Since 99% of Windows users will be using the distribution we provide, they'll have libunrar, so really, we should just go back to prefering unrar if libunrar is not available, using 7z as a last resort.

[Ark]

I am sorry for the confusion. Reading the tables works like this: The columns to the left show which packages are (not) installed, and the columns to the right show which formats are (not) supported when using this combination of (un-)installed packages. For example, on Debian, the 10th line reads "IF p7zip-full is installed and p7zip-rar is installed and unrar is not installed and unrar-free is installed THEN RAR 4 is supported and RAR 5 is not supported and 7z is supported". (Actually, the source code of the tables looks nicer than the rendered versions.)

The main issue here is that installing packages should only add support for archives, not remove it.

Thank you for your posts, they helped me thinking about this as well. Now, I think this approach might work best:

• We ignore the unrar-free issue as this is currently too complicated. This means that Debain users who want to enjoy free-software-only setups will not be able to open RAR 4 or RAR 5 archives. However, this should not be a problem since the mcomix packages for Debian do not recommend unrar-free but suggest unrar (non-free) anyway.
• Concerning the RAR 5 issue: Since everything seems to work on Windows, we should keep it this way there. For all other platforms, however, we should prefer unrar, but only if we encounter a RAR 5 archive and libunrar is not available.

More formal (pseudocode):

if system != WINDOWS and
   archive_type == RAR and
   archive_type_version >= 5 and
   available(UNRAR) and
   not available(LIBUNRAR)
then
    prefer UNRAR
else
    same as r1419

This should not break anything.

(It might be the case that someone installs both unrar and unrar-free and for some reason, the unrar command refers to the free version instead of the non-free version. However, in this case, we cannot do much but merely suggest to make sure that the unrar command refers to the non-free version, or to remove unrar-free.)

What do you think?

[Benoit Pierre]

The differences between 7z/unrar support for RAR are:

• solid detection and encryption support: work for 7z only
• unicode support: does not work with unrar on Windows

Considering the fact that again our Windows version of MComix will ship with libunrar, we should just do this:

diff --git a/mcomix/archive_tools.py b/mcomix/archive_tools.py
index 67d0787..2415085 100644
--- a/mcomix/archive_tools.py
+++ b/mcomix/archive_tools.py
@@ -41,11 +41,11 @@ _HANDLERS = {
     constants.BZIP2: (
         tar.TarArchive,
     ),

-    # Prefer 7z over rar executable for encryption and Unicode support.
     constants.RAR: (
         rar.RarArchive,
+        rar_external.RarArchive,
+        # Last resort: some versions of 7z support RAR.
         sevenzip_external.SevenZipArchive,
-        rar_external.RarArchive
     ),
     # Prefer 7z over lha executable for Unicode support.
     constants.LHA: (

Which should get us the same RAR support as for the previous versions, except for the fact that 7z is tried as a last resort if both libunrar and unrar are not available.

I think we could release in this state.

And then I can work on improving unrar support: switch to technical listing so we can support solid detection and encryption like 7z does.

[Benoit Pierre]

Here is a version with the aforementioned patch and improved external rar/unrar support: https://github.com/benoit-pierre/mcomix/compare/improve-rar-support

52 xfails removed from the testsuite! \o/

[Ark]

Thank you for the new patches. However, I could not yet test them with respect to the RAR 5 issue discussed here. Instead, I created a new ticket because your patches also affect the library. See [bugs:#100] for more details.

[Ark]

Now for the greatly improved archive extraction method thanks to your patches:

Arch Linux

p7zip	libunrar	unrar	RAR 4	RAR 5	7z

• | - | - | Y(!)| Y(!)| -
• | - | Y | Y | Y | -
• | Y | - | Y | Y | -
• | Y | Y | Y | Y | -
  Y | - | - | Y(!)| Y(!)| Y
  Y | - | Y | Y | Y | Y
  Y | Y | - | Y | Y | Y
  Y | Y | Y | Y | Y | Y

Now that I think about it, concerning the cells marked with (!), I guess that is because rar is installed on my system. Apart from that, it is perfect.

Debian Stretch (testing)

p7zip-full	p7zip-rar	unrar	unrar-free	RAR 4	RAR 5	7z

 -     |     -     |   -   |      -     |   -   |   -   |  - 
 -     |     -     |   -   |      Y     |   -   |   -   |  - 
 -     |     -     |   Y   |      -     |   Y   |   Y   |  - 
 -     |     -     |   Y   |      Y     |   Y   |   Y   |  - 
 Y     |     -     |   -   |      -     |   -   |   -   |  Y 
 Y     |     -     |   -   |      Y     |   -   |   -   |  Y 
 Y     |     -     |   Y   |      -     |   Y   |   Y   |  Y 
 Y     |     -     |   Y   |      Y     |   Y   |   Y   |  Y 
 Y     |     Y     |   -   |      -     |   Y   |   -   |  Y 
 Y     |     Y     |   -   |      Y     |   -(!)|   -   |  Y 
 Y     |     Y     |   Y   |      -     |   Y   |   Y   |  Y 
 Y     |     Y     |   Y   |      Y     |   Y   |   Y   |  Y

Almost perfect. The only problem is the line marked with (!), indicating that installing unrar-free disables RAR 4 support. A version detection on startup (next to subprocess32 and czipfile) might fix this. From what I can tell, you simply need to inspect the return value of unrar: If you execute unrar without any arguments, unrar-free will return 1 while unrar (non-free) will return 0. Since unrar-free does not work well for us anyway, we should disable unrar on startup if it turns out to be unrar-free.

I think you can already push the changes to SVN now, including the patches for [bugs:#100].

[Benoit Pierre]

Yes, external RAR support checks for both unrar and rar executables.

I'll look into checking if the unrar binary is actually unrar-free.

In the mean time I will push https://github.com/benoit-pierre/mcomix/compare/master...improve-rar-support to SVN.

[Benoit Pierre]

I currently have access to a Ubuntu derived system (Linux Mint 17.2 Rafaela), and it looks like with both unrar and unrar-free installed:

• unrar provides the unrar-nonfree binary
• while unrar-free provides the unrar-free binary
• unrar is just a symlink set by update-alternatives

So we could just prepend unrar-nonfree to the candidates list when calling find_executable:

diff --git i/mcomix/archive/rar_external.py w/mcomix/archive/rar_external.py
index da8b94b..041da74 100644
--- i/mcomix/archive/rar_external.py
+++ w/mcomix/archive/rar_external.py
@@ -173,7 +173,7 @@ class RarArchive(archive_base.ExternalExecutableArchive):
     Returns None if neither could be started. """
     global _rar_executable
     if _rar_executable == -1:

-        _rar_executable = process.find_executable((u'unrar', u'rar'))
+        _rar_executable = process.find_executable((u'unrar-nonfree', u'unrar', u'rar'))
     return _rar_executable

     @staticmethod

[Benoit Pierre]

And then we can check the executable we found is not a symlink to unrar-free, see code here.

[Ark]

Well spotted! I think for this to implement, it might be a good idea to add an optional parameter to find_executable. This new parameter is a function which receives a string n (the same as we passed it to find_executable within the tuple) and an absolute path p that points to an executable. Whenever find_executable is about to return a path, it calls this function which

• can inspect n and p and
• returns either a new absolute path (e.g. p, if it turns out to be already perfect) or None if p turns out to be useless.

If a path is returned, then find_executable simply returns the very same path. However, if None is returned, find_executable continues searching. Pseudocode:

`_rar_executable = process.find_executable((u'unrar-nonfree', u'unrar', u'rar'),
    check_candidate=lambda n, p:
        return p if (n != u`unrar`) or usable_rar(p) else None
)`

If I got it right, your current approach is a bit buggy as it might not be able to find rar even if it is installed, but still returns None because it found an unusable unrar.

[Benoit Pierre]

Yes, I'll add a is_valid_candidate_fn parameter to find_executable.

[Benoit Pierre]

Note that we have another problem with RAR support (yeah...): on the machine I'm currently using (Linux Mint 17.2 Rafaela), unrar is version 5.0, but rar is version 4.2, and the latter's output for 'vt' (technical listing) is different...

[Ark]

Is the unrar program you talk about non-free? I think that most setups will satisfy if nonfree(unrar) then version(unrar) >= version(rar), so it is probably safe to just ignore this issue, at least for now, because unrar is preferred over rar anyway. And since rar is the last alternative we consider, it should not hurt that we try to use something that won't work.

For the record: I just checked the alternative links on Debian and it's just like Mint. (Not that much of a surprise.) Thus, your solution should work on Debian as well. (I did not yet actually test it.) Adding a comment in the code about the different return codes of unrar-free and unrar-nonfree might turn out helpful later, though. For example, the links might change their names in the future, or other distributions might behave differently.

[Benoit Pierre]

I tweaked find_executable and updated rar_external.py, see here.

Regarding the problem with rar: on my current system, if only the rar package is installed, then MComix will show all RAR archives as empty, because the format of the listing is different that what is parsed/expected.

[Ark]

The branch improve-rar-support looks promising. Could you please push it to SVN? Thanks in advance.

[Benoit Pierre]

Done.

[Benoit Pierre]

Should we close this?

[Ark]

Yes, you are right.