Re: [parchive-devel] QuickPar use cases

[Michael N. <mik...@gm...>]

Thanks, Yutaka Sawada.

1) Unicode
I think we've agreed on using UTF-8, which is an extension of ASCII.

2) Folder recursion
This is already supported by the spec.  I think we've agreed to support
creating empty directories.

3) More than 32k blocks
We've seen that it usually isn't needed and it is dependent on the recovery
algorithm.  We know we're going to support a new algorithm.  If the
algorithm is LDPC, we will probably support more than 32k blocks.

4) Automatic removal of recovery files
This is a client feature.  We don't need to change the spec to do this.  I
will say that deleting data is always dangerous.  I feel it should only be
done with the user's express permission.

5) Automatic calling of UnRAR, UnPkZip, etc.
A client can do this.  We don't need to change the spec.

6) Incomplete recovery
This is difficult to do.  Clients can support it, but I don't think the spec
should say that all clients need to support it.

7) Duplicate input slices
http://www.quickpar.org.uk/forum/viewtopic.php?id=1096
The use case here was that someone was taking a large video file and
splitting it up into pieces and then editing it.  So, the input set had a
lot of data that overlapped.  I have done multiple subdirectories where each
subdirectory held different versions of a program, which only differed in
20% of the files.
I guess the user wanted the PAR client to identify duplicate slices in the
input.  I don't know if this included offsets that were not a multiple of
the slice size.  (That is, the file changes did not align with the length of
a slice.)  Duplicate slices would not be duplicated in the input slice set
and either one of the slices as an input could be used to repair the other
before going to RS recovery (or another algorithm).
This is a neat idea, but I don't think we should handle it - unless it
interferes with the recovery algorithm.  And I don't think it interferes
with RS or significantly with LDPC.  Detecting duplicates is difficult
unless they are slice-aligned and even then it doesn't gain us a smaller
recovery file or a much shorter recovery time.  It just means there are more
cases that we could recover from - and some of these may be done by a
recovering client without changing the spec.

8) Subslice checksums
Supported.

9) Redundancy beyond 100%
Supported.  But we can only support ~32k slices, so if you want lots of
recovery slices, you need to use fewer input slices.  Also, using all ~32k
slices is costly.

10) Better support for small files
Use TAR or PkZip to make them one large file, then use PAR.

11) Not using a whole block to fix one byte
Subslices can do this.

12) >32k slices error.
Yes, it's a client issue.  Do we need to do more to make sure clients are
compatible?   Also, on an error, a client should be required to show the
Creator Packet so that users can know the client that did this.

13) Matrix inversion problems
Yes, it's a known PAR2 problem.  It was known in PAR1 and I tried to fix
it.  I think it occurs less often now, but it still occurs and it's my
fault.  I think that's one of the big motivations for PAR3 - to fix this
problem.

14)  non-ASCII characters (have 8th bit set)
This was another of my failures in the PAR2 spec.  I thought 8-bit ASCII was
a standard.  If I had known there were multiple extensions to ASCII with the
8th bit set, I would have either chosen one or clearly stated that ASCII was
7-bit and having the 8th bit set would violate the spec.  I think we fix it
with making everything UTF-8.


Any comments, critique, or criticism?

Michael Nahas


On Sat, Jun 19, 2010 at 9:50 PM, <te...@ma...> wrote:

> Hello, parchive developers.
>  I am Yutaka Sawada.
>
>  from Michael Nahas 2010-06-16
> > What are the use cases and problems you see in the Par2 client forums?
>
>  The use cases are mainly following two;
> (C1) transport files on UseNet
> (C2) protect files on backup media like CD/DVD
>
>  I read QuickPar homepage's forum.
> There are many interesting idea/thought/proposal/claim.
>
> [ subject in forum ]
>  I write short Q&A style.
> Refer original post, if you are interested.
>
>
> [ Wishlist >> High need of Folder Recursion and Unicode file name support ]
> http://www.quickpar.org.uk/forum/viewtopic.php?id=1227
> Q) I want Unicode filename and Directry support.
> A) use MultiPar or ICE ECC.
>
>
> [ Wishlist >> QP without limitation of 32765 input blocks ]
> http://www.quickpar.org.uk/forum/viewtopic.php?id=1105
> Q) Over than 32k slices may be useful,
> even if QuickPar can use them only for verify/repair.
> A) Wait for PAR3.
>
>
> [ Wishlist >> Feature Request: Unpack & Cleanup ]
> http://www.quickpar.org.uk/forum/viewtopic.php?id=1177
> Q) PAR client would better to have a feature of archiver.
> A) Just use a favorite set of PAR client and Archiver.
>
>
> [ Wishlist >> Incomplete recovery ( as good as possible) ]
> http://www.quickpar.org.uk/forum/viewtopic.php?id=1214
> [ Wishlist >> A suggestion for a useful new function ]
> http://www.quickpar.org.uk/forum/viewtopic.php?id=1136
> Q) Recover as possible as it can,
> even when there are not enough recovery slices.
> A) This is impossible for Reed-Solomon codes.
>
>
> [ Wishlist >> detecting and handling (partially) dupes ]
> http://www.quickpar.org.uk/forum/viewtopic.php?id=1096
> Q) PAR client should check dupe slices before create recovery slice.
> Create recovery slice from identical slices only.
> Dupe slices will be recovered by copy.
> A) This is an interesting idea, but maybe useless for most files.
>
>
> [ Wishlist >> Idea for improve the performance ]
> http://www.quickpar.org.uk/forum/viewtopic.php?id=838
> Q) Using parallel splited slices may help.
> A) This idea is same as subslice of "Packed Main packet".
>
>
> [ Wishlist >> Feature request: Redundancy beyond 100% ]
> http://www.quickpar.org.uk/forum/viewtopic.php?id=805
> Q) I want to protect my DVD with huge redundancy.
> A) Just make backup DVD, much faster.
>
>
> [ Technical support >> Quickpar efficient for small files? ]
> http://www.quickpar.org.uk/forum/viewtopic.php?id=1190
> Q) PAR2 file is not efficient for small files ?
> A) It depends on slice size. Fewer packet repetition may help a bit.
>
>
> [ Technical support >> Data fault and block consumption ]
> http://www.quickpar.org.uk/forum/viewtopic.php?id=1098
> Q) Only a small byte error requires whole recovery slice ?
> A) Yes. Even a small error makes the whole slice to be useless.
> However smaller slice size is better for protection,
> speed becomes slow as slices are many.
>
>
> [ Technical support >> Error when repairing: "too many input blocks for
> Read Soloman matrix" ]
> http://www.quickpar.org.uk/forum/viewtopic.php?id=1090
> Q) QuickPar can not repair, when there are over than 32768 input slices.
> A) PAR2 supports max 32768 input file slices for recovery.
> This is known as YencPowerPost problem.
>
>
> [ Technical support >> Matrix inversion error ]
> http://www.quickpar.org.uk/forum/viewtopic.php?id=1080
> [ Technical support >> Reed Solomon Error ]
> http://www.quickpar.org.uk/forum/viewtopic.php?id=211
> Q) QuickPar fails to recover, even if there are enough recovery slices.
> A) PAR2 & PAR1 have a flaw, they can not invert matrix for certain
> combinations.
>
>
> [ Technical support >> Failed: Error creating description packet ]
> http://www.quickpar.org.uk/forum/viewtopic.php?id=161
> Q) I can not create PAR file for certain files.
> A) The filename has Unicode character which can not show.
>
>
> [ UseNet >> European language accented file names make recovery impossible
> ]
> http://www.quickpar.org.uk/forum/viewtopic.php?id=1128
> Q) QuickPar fails to treat non-ASCII filename.
> A) It is a fault of QuickPar. PAR2 spec is not bad.
>
>
> Best regards,
> Yutaka Sawada
>
>
> ------------------------------------------------------------------------------
> ThinkGeek and WIRED's GeekDad team up for the Ultimate
> GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the
> lucky parental unit.  See the prize list and enter to win:
> http://p.sf.net/sfu/thinkgeek-promo
> _______________________________________________
> Parchive-devel mailing list
> Par...@li...
> https://lists.sourceforge.net/lists/listinfo/parchive-devel
>