Deflate64 257 word size.

[Kirill Frolov Sergeevich]

Why is compression so slow when i set word size to 257 compared to 256?
When i have set it to 257 and compressed 11204 MB of data, 7-zip reached 100% in 2:50 hours.

[Igor Pavlov]

What difference for 256/257 options?
What values in compression level?
If you used Ultra, use "normal" compression level instead, and it will be much faster.

[janmm14]

256 bytes might fit in a single cpu cache line, while 257 needs twice the internal cpu data because it needs a 2nd cache line.

[Sam Tansy]

My guess would be - it is maximum allowed by format.
But that raises the question - why 257, not 258? Deflate allows 258.

[Igor Pavlov]

258 in deflate is short code.
258 in deflate64 is long code.
Actually we could support long code for 258 in deflate64,
but it required some code change. Maybe I'll implemnet it in future.

deflate64 also supports values larger than 258, but it's difficult to support them in 7-Zip, because main code was
created with 258 limit for deflate.

[Sam Tansy]

I don't get why is 258 short code in deflate and long in deflate64. If so, then 257 is long as well?
Anyway, changing the code to allow 1 byte longer code is bit overboard, but 16kB is different ball of wax.

But is there any other de/compressor/packer that supports deflate64?

[Igor Pavlov]

258 in deflate can be encoded with two ways:
short (0 extra bits) and long (5 extra bits). We use short encoding because it's smaller.

258 in deflate64 can be encoded with two ways:
long (5 extra bits) and very long (16 extra bits). So we can't use short (0 extra bits) encoding supported by deflate.

Next version of 7-zip probably will support 258 with long (5 extra bits) encoding in deflate64.
But I don't think that I want to support longer matches with very long (16 extra bits) in deflate64 because it increases the complexity of code that is related to deflate also.