Java Heap space problems to compress large 7z archives

[Klaus Günther]

Hello,

I have a problem to create and compress a big 7z archive with some GBs of data. Creating the items by reading the data in the Byte array I get a heap space error (out of memory). Normally, creating a ZIP archive by JAVA API I am creating the ZIP archive by input and outputstreams a I can handle the memory by using an appropiated buffer size.
In 7-Zip-Jbinding I have to create the items before creating the 7z archive and the whole contents of the archived files must be held in the memory (and in 32-Bit Windows the heap space is limited to 1 GB!).

How can I solve this problem?

Thank you for your answers.

Kind Regards
Klaus Guenther

[Boris Brodski]

Hello Klaus,

sure, it shouldn't be a problem for 7-Zip-JBinding. You don't need to save the entire entity content in the memory. It's only the case for the ByteArrayStream.

You have to implement the ISequentialInStream interface youself. It contains the read() method:

int read(byte[] data) 
          Reads at least 1 and maximum data.length bytes from the in-stream.

Here you can pass your data chunk to chunk to the 7-Zip engine for compression.
Just don't forget to set the size of your data using IOutItemBase.setDataSize() method within the getItemInformation() method.

Hope, it helps :) If not, just post your questions here.

Cheers,
Boris

[Klaus Günther]

Hello Boris,

thank your for the quick response.

My problem is, creating the items for compression I'll get already a heap space error (if I want to compress a big directory with many files).

He parts of my code:

Vector<String> fileNameList = Utilities.listRecursive(job, directory, null, onlyDirsAt1stLevel);

File [] files = new File[fileNameList.size()];
for (int i = 0; i < files.length; i++)
files[i] = new File(directory + "/" + fileNameList.get(i));

items = createItems(files,fileNameList);

....

private Item[] createItems(File [] files, Vector<String> fileNameList) {

Item[] items = new Item[files.length];
String entry;

    try {
        for (int i = 0; i < items.length; i++) {                
            InputStream in = new FileInputStream(files[i].getAbsolutePath());

            byte [] content = IOUtils.toByteArray(in);  // ******Here I get the heap space error****** 
            // Before I compress thew files
            entry = fileNameList.get(i);
            items[i] = new Item(entry, content);
            entries.add(entry);
            entrySize.put(entry, files[i].length());
        }

...

I do not reach the following part:

try {
raf = new RandomAccessFile(archiveFile.getAbsolutePath(), "rw");

        // Open out-archive object
        outArchive = SevenZip.openOutArchive7z();

        // Configure archive
        outArchive.setLevel(5);
        outArchive.setSolid(true);

        // Create archive
        outArchive.createArchive(new RandomAccessFileOutStream(raf),
                items.length, new CreateCall7zback(displayEntries));

My question is:
Is it necessary to create all items before you create the archive, or is there a possibility to add the entries to the archive file by file? Or is there an addEntry method by streamming the data?

Kind regards
Klaus

[Klaus Günther]

Hello Boris,

I have a real good solution for my problem. I modified the class ByteArrayStream adding a new Constructor. I will tell you mor about that in the discussions within the next days.

Cheers,
Klaus

[Boris Brodski]

Hello Klaus,

I am really sorry. I completely forgot to answer your last question. Im glad, you have found a solution.

Cheers,
Boris

[rs030877]

Hi Klaus,
could you, please, share your solution? I catched the same exception.
Thank you in advance