Re: [Squashfs-devel] [PATCH] unsquashfs: Add a -t[est] option, which only tests the image

[Phillip L. <ph...@lo...>]

On 10/04/14 19:25, Ezequiel García wrote:
> Phillip,
>
> First of all, thanks a lot for taking the time to reply and explain
> why you rejected this patch.
>
> On 10 April 2014 14:43, Phillip Lougher <ph...@lo...> wrote:
>> On 10/04/14 17:03, Ezequiel García wrote:
>>>
>>> Phillip,
>>>
>>> Any comments on this patch?
>>>
>>
>> Yes Ezequiel.  Two rather blunt ones.
>>
>> 1, Squashfs is a voluntary *spare-time* project maintained completely
>> by myself.  At any one time I have a massive TO DO list, and TOO LITTLE
>> time to do it.
>>
>> Requests for advice, bug reports, patches etc are fairly dealt with on
>> a first come first served basis.  You have to wait you turn.  Only
>> priority emails (e.g. evidence of a major bug), get dealt with out of
>> order, and this isn't one.
>>
>> So please don't pester and try to be dealt with before others who
>> emailed before you.
>>
>
> Right, got the message. Sorry for spamming you, won't happen again.

Np...  The comment was as much directed to the list as you.  People these
days want instant gratification, and get cross and annoyed when they
don't get it, forgetting that in many cases they're unreasonably
expecting it from an open source project run on less than a
shoe-string.

I got stressed and miserable trying to keep everyone happy, and I burnt
out a couple of years back by letting the pressure get too
much.  I in fact abandoned Squashfs for a while, and only came back when
I decided to do things on my terms.

>
>> 2. That patch was added to the "patch tracker" back in 2009, I didn't
>> take it then because I didn't like it.
>>
>> I don't particularly appreciate having the identical patch reposted
>> 5 years later, just in case I didn't notice it the first time perhaps?
>>
>
> Sorry for the repost, I missed your feedback, if there was any.
>

Hmm, this was back in 2009, the feedback may have been
off-list.  I don't like telling people their patches are rubbish.  In
fact it makes for a terrible day, and if I can lessen the recipients
pain by doing it off-list I would.

>> As I said I don't have much free time, and I don't like wasting it
>> dealing with a patch I rejected 5 years ago.
>>
>> As to why I rejected the patch?  Because it is rubbish.  It is doing
>> everything the wrong way.
>>
>> An fsck is written *defensively*, with the mind-set the
>> filesystem *is* corrupted and everything has to be double-checked,
>> and then triple checked if possible by correlating it with other
>> data in the filesystem.  This is expensive, this is slow.
>> Moreover, an fsck is expected not to crash no matter how
>> corrupted the filesystem is.
>>
>> A filesystem extractor is written to be fast, it is written
>> with the mind-set the filesystem is basically correct,  with the
>> odd sanity check to ensure it doesn't do anything really silly.
>> It does not check the filesystem for correctness or that it
>> makes sense by correlating data with other data in the filesystem.
>>
>> You can't make a filesystem extractor into a fsck, it has to be
>> written differently from the ground up.
>>
>> Trying to make it into a "poorman's fsck" like this patch does
>> is no good.  Unsquashfs only works reliably with uncorrupted
>> filesystems.  Giving it corrupted filesystems is likely to
>> make it crash or behave badly.
>>
>
> I see. Of course, I wasn't considering this properly and thought the unsquashfs
> tool would tolerate corrupted filesystems.

It depends on what you mean by corrupted filesystems.  I have to deal
with two types.

1, Filesystems corrupted by "wear and tear", the odd flipped bit, or
the last 300K truncated.

These filesystems Unsquashfs will tolerate.  If it's corrupted metadata
data it will abort.  If it's corrupted file data, it will skip writing
that file.

2. Filesystems maliciously corrupted

There are ways of subtly corrupting a Squashfs filesystem, which may
cause Unsquashfs to crash or hang.  This will probably involve crafting
a special version of Mksquashfs, and studying Unsquashfs
for vulnerabilities.  They'll then present you with the "bugs" and
get shrill and nasty when you ignore them.

Think that won't happen, well I was a utopian and thought so too.
But it does happen, and the last time it did I got two CVE's
registered against Squashfs, public vandalisation of Squashfs stuff
and private emails full of unprintable abuse.

The CVE's (Common Vulnerabilities and Exceptions) with exaggerated
claims as to their seriousness.

http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-4024
http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-4025

Yes, embarrassing bugs, but triggerable only if you deliberately choose
values for -fragment-queue and -data-queue that added together overflowed
an int, or typed in a filename that was longer than 8K.  As this
was user-space utilities the worst that would happen was the
program would crash.

I explained this, said I'd fix them in the next release (and add them
to my devel tree), but didn't see the need to rush out an emergency
new release.  Not good enough, the response was extremely abusive
private emails (not in any way fit to be repeated).  That did
provoke me, and I responded publically that I was ignoring the
guy but not the bugs, and he seemed to have an ulterior motive of
spreading FUD about Squashfs.  This then led to a public campaign of
smearing and insults popping up everywhere (always anonymously of course).

The one instance I can still find is the vandalising of the
wikipedia page on Squashfs, which was milder than a lot.

http://en.wikipedia.org/w/index.php?title=SquashFS&diff=503333315&oldid=503325197

Open source is an example of the tragedy of the commons, the few
spoil it for the rest.  I'm not paid to take abuse for code I
developed and released publically.

In other words any code I write now is hardened to be as far
as possible robust against malicious behaviour, and this patch
fails that test.

>
> This "poorman's fsck" might be good enough for my needs
> (squashfs on top of a UBI volume on top of a NAND device, where there are
> other mechanisms to check the consistency of the data).

Yes, it undoubtedly will be good enough for that.

>
> However, we need to have a good way of checking the filesystem
> before mounting, so I'll double check it and take a deeper look.
>
> Once again, sorry for bothering. Wasn't my intention, but quite the opposite.
>

Np, you've just bumped the "write an fsck" TO DO list item a couple of
places.

Phillip