How to use 2GB or 3GB dictionary (or anything larger than 1GB)?

Help
fenn
2013-06-28
2014-12-14
  • fenn

    fenn - 2013-06-28

    I know that LZMA supports a dictionary size up to 4gb but I can't seem to enable the use of anything over the size of 1024MB either in the command line, or GUI. This has to be possible somehow as right on the 7z website it says "Variable dictionary size (up to 4 GB)"

    I have several 1.5gb and 2gb files that I would like to maximize my compression with for long time archival. I am running Win7 x64 and have a system with 64GB of RAM.

    I have been searching all over for any information about how to use a dictionary larger than 1gb but can not seem to find anything. Igor, any ideas on how to enable the use of a larger dictionary? Has anyone had any success with a custom build that supports larger dictionaries?

     
  • Igor Pavlov

    Igor Pavlov - 2013-06-29

    I had no RAM to test big dictionaries in past.
    So now LZMA encoder doesn't support it.
    I'll improve it in some future versions.

     
  • fenn

    fenn - 2013-06-30

    Oh ok, that makes sense. Thank you for responding!

    If you would like, I would be happy to donate some quality RAM for you, if this would help you with further innovation in 7-Zip. You have created a wonderful piece of software, and I would gladly do what I can if it might mean you would have the resources to work on big dictionary support :)

     
    Last edit: fenn 2013-06-30
  • Igor Pavlov

    Igor Pavlov - 2013-06-30

    Big RAM is not problem now.
    So there is that feature in plans.

     
  • fenn

    fenn - 2014-08-03

    Dear Igor,
    Have you had any luck implementing Big Dictionary support for great then 1024MB dictionary sizes? It has been a year since I last inquired and I have been patiently hoping for any signs of improvement as each beta has arrived :D If you need resources or RAM, again, I would be happy to donate. As many baseline PC configurations come with 8GB of RAM or more, this seems like a logical implementation step, and if the LZMA architecture by default supports up to 4GB natively it seems like this shouldn't be that large of an implementation into your core. Willing to help however possible.

    Hello from the east coast, USA!
    Cheers

     
  • Igor Pavlov

    Igor Pavlov - 2014-08-04

    I want to release new "beta" version at first.
    Then probably I'll think about "dictionary size" things.

     
  • Andrzej

    Andrzej - 2014-11-21

    Hello,

    i just wanted to ask if there will be increasing dictionary size only to 4096MB or maybe more than this?
    I ask because i want to know how much RAM i will need for such a big dictionary size..
    There is an example at this picture: http://i.stack.imgur.com/PZkJ2.png
    And i think that the memory consumption will be the same with only one Thread and 4096MB dc size.. So for that i will need at least 64GB RAM.
    Is 4GB dictionary size the limit of 7-Zip or will it go even further? This would mean that there must be done more changes in the code of 7-Zip..
    So, what's the future?

    Regards,
    Andrzej.

     
  • Igor Pavlov

    Igor Pavlov - 2014-11-21

    2 GB dictionary - I plan to add the support.
    3-3.5 GB dictionary - is also possible (I'll check it).
    4 GB - is possible, but it's more difficult to implement.
    More than 4 GB dictionary - (LZMA and LZMA2 don't support it).

     
  • Andrzej

    Andrzej - 2014-11-21

    Thank you for your reply.
    I think that 4GB dictionary is more than enough..
    Maybe in future something like LZMA3 could support more than 4GB dc size.. but 7-Zip like it is today is good and with 4GB maximum dc size it will be the best. :)

     
  • fenn

    fenn - 2014-12-09

    Thank you so much Igor for working on big dictionary support! I eagerly await using a 2GB or 3.5GB dictionary size! Cheers mate :)

     
  • Jouni Järvinen

    Jouni Järvinen - 2014-12-14

    Be warned you can't use a LZMA[2]-compressed file with 32-bit 7-Zip, possibly applies to other programs too, if you use anything bigger than 64MB.

     

Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks