#342 Support opening ZIP files

[Alberto Garcia]

It seems that the vast majority of Spectrum software for emulators is
distributed in ZIP archives, and I find it slightly annoying to have
to unpack them manually. From what I can see I'm not the only one:

https://sourceforge.net/p/fuse-emulator/feature-requests/53/

Here's a quick patch that I'm using in my own computer to open ZIP
files in libspectrum. It doesn't attempt to be a full-featured archive
browser or anything too fancy, it just looks inside the archive for a
file that libspectrum can understand and opens it, so it's similar in
spirit to the gzip or bzip2 decompression support.

The bad news is that of course if there's a ZIP with, say, two
snapshots inside then only the first one will be opened and there's no
way to select the other one. That's usually not the case, although it
does happen that there are ZIP archives with two TZX files inside, one
for each side of the tape.

Since this uses libarchive it can be trivially extended to add support for other
types of archives (tar, cpio, rar, etc.)

I've seen that there's been other attempts to implement archive
support (e.g. [patches:#157]) and that's maybe a better approach than mine
but I anyway wanted to share it in case it's useful.

[Sergio Baldoví]

Thanks for sharing!

It seems that the vast majority of Spectrum software for emulators is
distributed in ZIP archives, and I find it slightly annoying to have
to unpack them manually.

I suspect Linux platforms could mount archives automatically by a VFS tool. I recall someone mentioning that.

I've seen that there's been other attempts to implement archive
support (e.g. patch #157) and that's maybe a better approach than mine
but I anyway wanted to share it in case it's useful.

[patches:#157] implement the decompression algorithms, which have some limitations. A specialised library would be more compatible, more efficient, less buggy and easier to maintain. On the other hand a content browser is nice to have full control when there are more than one file in an archive. Some users could argue that this task is better for an external program (like a front-end), though.

libarchive is a great library, allow on-memory decompression and IMHO is the best choice to this task.

[Fredrick Meunier]

I had a quick look at libarchive and I'm not sure about whether I can distribute it as part of a GPL binary package (for Fuse on the Mac) as it requires author attribution like three-clause BSD which I think I can't require for GPL. Based on that, I'd look closer at some of the other patches.

I do agree that the libarchive version of the code is smaller and delegates more responsibility though - we could also consider supporting both in a real glib/fake glib kind of way?

[Sergio Baldoví]

Licensing is always tricky, but I think libarchive could be distributed in binary packages. libarchive use a 2-clause BSD with a few code covered by 3-clause BSD [1]. Both are compatible with GPL according to FSF [2].

There is no intention to include or modify the source code of libarchive, much like bzip2, that also have a BSD-style license. The GNOME Archive Manager (file-roller) [3] is GPLv2+, can use the API of libarchive, and doesn't mention it or distributes the BSD license. I think calling API functions is not considered a redistribution.

The binary distribution of Fuse for Windows would link against libarchive and distribute libarchive.dll. As I understand, a copy of the license should be distributed as well, and if possible, made visible in the About dialog [4].

I do agree that the libarchive version of the code is smaller and delegates more responsibility though - we could also consider supporting both in a real glib/fake glib kind of way?

If we support a custom implementation to avoid BSD dependency, I don't see the need of supporting libarchive (for ZIP only). That would add complexity for little benefit.

[1] https://en.wikipedia.org/wiki/BSD_licenses#3-clause_license_.28.22Revised_BSD_License.22.2C_.22New_BSD_License.22.2C_or_.22Modified_BSD_License.22.29
[2] https://www.gnu.org/licenses/license-list.html#ModifiedBSD
[3] https://git.gnome.org/browse/file-roller
[4] https://groups.google.com/forum/#!searchin/libarchive-discuss/license/libarchive-discuss/8UR4g_fyMmI/4UxVybn-jSsJ

[Fredrick Meunier]

Licensing is always tricky, but I think libarchive could be distributed in binary packages. libarchive use a 2-clause BSD with a few code covered by 3-clause BSD [1]. Both are compatible with GPL according to FSF [2].

Thanks for flagging this, I may have read the licenses in libarchive too quickly - the clause I was concerned about was:

2 Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

Which I was misreading as the GPL-incompatible "advertising clause" - which is the 4-clause BSD, rather than the 2 or 3 clause versions! I should have double-checked rather than relying on memory.

There is no intention to include or modify the source code of libarchive, much like bzip2, that also have a BSD-style license. The GNOME Archive Manager (file-roller) [3] is GPLv2+, can use the API of libarchive, and doesn't mention it or distributes the BSD license. I think calling API functions is not considered a redistribution.

My reading has been that distribution likely includes shipping a complete binary package including those with incompatible licenses (though I'm not sure to what extent it has been tested by courts around the world - this probably varies considerably in reality depending on location). e.g. see http://www.openacs.org/about/licensing/open-source-licensing for the kind of thing I mean, an exemption in the otherwise incompatible licensed software removes any uncertainty.

The GPL exempts software distributed as part of the OS (which will be there for libarchive for most Linux distros but not in a usable form on OS X) and there is no binary distribution for GNOME Archive Manager for propriatary OSs on its page so I don't think they are facing the same issue.

As Mac OS X distributes bzip2 and I just link to it without distributing it, I don't have the same issue there (annoyingly libarchive is there but Apple choose to not ship the headers for this library).

The binary distribution of Fuse for Windows would link against libarchive and distribute libarchive.dll. As I understand, a copy of the license should be distributed as well, and if possible, made visible in the About dialog [4].

My reconsidered opinion is that I could distribute a binary including libarchive with the union of all the copyright notices[1] from libarchive added to the about box and the root of the distribution where I have the GPL.

I'll experiment with building a OS X style framework for libarchive and update this issue if I can get it working reasonably.

[1] e.g. the notice in the root COPYING file, plus the two in libarchive/archive_write_add_filter_compress.c plus all the other unique variations across all libarchive source files which independantly require that "redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution." with different copyright holders listed.

[Sergio Baldoví]

My reconsidered opinion is that I could distribute a binary including libarchive with the union of all the copyright notices[1] from libarchive added to the about box and the root of the distribution where I have the GPL.

Fedora only distrubutes the general COPYING file of libarchive, based in this clause, I suppose:

"The libarchive distribution as a whole is Copyright by Tim Kientzle
and is subject to the copyright notice reproduced at the bottom of
this file."

Debian distributes a more elaborated file with the copyright holders:
http://metadata.ftp-master.debian.org/changelogs//main/liba/libarchive/libarchive_3.2.1-1_copyright

I think you can keep the COPYING in the root as GPLv2+ because is the license that applies as a whole, but the copyright notices of third party (BSD) libraries should be distributed as well.

[Fredrick Meunier]

OK, I've given linking libarchive to Fuse for Mac OS X a try and I can build and use it successfully.

Compared to the [patches:#343] version, I think the pros are:

• Probably a more robust implementation
• Someone else will maintain it
• Will be more widely tested

From my perspective, the cons are:

• The on-disk size to distribute is large
  • The Fuse for Mac on-disk space is 1.2M for 1.2.1 and adding the statically linked libarchive takes that to 1.6M or at least 1.2M plus 725K if I use the dynamic library
  • The [patches:#343] version costs about and extra 4K in libspectrum only

My bias is probably still for the [patches:#343] version, though I acknowledge that on Linux or *BSD those issues won't be relevant and if I didn't have the Mac version to deal with, I'd probably lean to this patch version due to the pros above.

Sergio[1] or others: do you have any comments about the pros/cons list above or opinions about which patch we should integrate?

[1] I see you are distributing the 1.2Mb libxml2 for Windows so I expect that you are not going to be concerned? ;)

[Sergio Baldoví]

Agree with the points. libarchive has a larger size because support multiple formats (rar, 7zip, etc) and looks the natural choice for unices. On MacOSX, Windows and portable devices looks a bit overwhelming.

[Alberto Garcia]

Hey,

thanks for looking into this.

I appreciate that you are taking a look at this patch but I would like
to point out again what I still consider its most important problem:

"[...] it just looks inside the archive for a file that libspectrum
can understand [...] The bad news is that of course if there's a ZIP
with, say, two snapshots inside then only the first one will be opened
and there's no way to select the other one."

I wrote the patch but I still wonder if this approach is good enough
for libspectrum. On the other hand, implementing something that can
browse the contents of a ZIP file would make it significantly more
complex (and I don't have a good proposal for it).

[Sergio Baldoví]

Yes, that's a limitation. The key here is libspetrum is using compression in a trasparent way (.gz and .bz2) on a single file and and zip could contain multiple cassetes, sides, poke files, disk images. For full zip support we need a content browser/manager at Fuse. If Fuse choose the file and uncompress a stream before calling libspectrum, then there is no need to have ZIP support in libspectrum at all.

[Fredrick Meunier]

In my opinion the simple "open-first supported" version matches user feature requests and expectations well and is sufficiently limited in scope to deliver easily.

It doesn't prevent us from building up to something more elaborate like Gergely's [patches:#157] from this simple base if there is interest in developing it further (I think delivering changes in smaller chunks will help us make progress).

[windale]

Spectrum files are very small as it is. I don't see the point of zip support if it only loads the first file. It already has Gzip etc. It would be more useful for tapes with multiple sides or RZX files with multiple parts that the user can choose what part to load. I think this basic zip support is for those lazy people who can't even be bothered to extract/convert their massive TOSEC collection. Only my opinion.

[Alberto Garcia]

Yeah, I'm actually one of those lazy people and that's pretty much the reason why I wrote the patch (as I explained in my first commment) :-) I agree that I'm not making a strong case for the feature, though! I'm happy to keep patching my local version if this is too controversial.

[Fredrick Meunier]

Thanks for all the discussion. My current plan to to merge in the internal version of "first understood contained file" zip support from [patches:#343] next weekend barring any feedback to the contrary.

I think that if there is interest from developers to build something more elaborate on that like to select specific files in the archive like [patches:#157] started, they can be built on this base and the functionality from the current set of patches is useful for many users in its current state, and that the [patches:#343] version is a better match for full binary distributions from many Fuse ports than relying on libarchive being distributed. I'd be happy to use the compat-type approach to plug in the libarchive version of zip extraction where available if anyone feels strongly enough to write a patch :)

[Philip Kendall]

Looks good to me - the only thing I see missing is that the GTK+ file selector doesn't show ZIP file when "supported files" is selected, which is going to make this feature a bit hard for people to find :-)

[Fredrick Meunier]

I've merged the [patches:#343] based version of zip file support. Any concerns if I mark this one as closed?

Alberto: if you want to retain your support for loading an auxilliary file from a zip when there are no snapshots or media, please feel free to submit a patchf for trunk.

[Alberto Garcia]

You can close this one, thanks!

[Fredrick Meunier]

Marking closed-fixed as the issue has been resolved by an alternative patch.