Saros - Distributed Party Programming / Bugs / #568 Sharing Large projects result in OOM Exception

#568 Sharing Large projects result in OOM Exception

Milestone: 13.3.15

Status: closed-fixed

Owner: Stefan Rossbach

Labels: Synchronisation / Invitation (126)

Priority: 7

Updated: 2013-03-21

Created: 2011-08-13

Creator: Stefan Rossbach

Private: No

!ENTRY org.eclipse.core.jobs 4 2 2011-08-13 02:49:02.536
!MESSAGE An internal error occurred during: "Sharing project {1830266213=A}.".
!STACK 0
java.lang.OutOfMemoryError: Java heap space
at org.apache.commons.io.output.ByteArrayOutputStream.needNewBuffer(ByteArrayOutputStream.java:124)
at org.apache.commons.io.output.ByteArrayOutputStream.write(ByteArrayOutputStream.java:155)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1263)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:1236)
at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:360)
at org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1360)
at de.fu_berlin.inf.dpp.net.internal.XMPPTransmitter.sendProjectArchive(XMPPTransmitter.java:868)
at de.fu_berlin.inf.dpp.invitation.OutgoingProjectNegotiation.sendArchive(OutgoingProjectNegotiation.java:628)
at de.fu_berlin.inf.dpp.invitation.OutgoingProjectNegotiation.start(OutgoingProjectNegotiation.java:125)
at de.fu_berlin.inf.dpp.project.SarosSessionManager$OutgoingProjectJob.run(SarosSessionManager.java:678)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54)

Project archive size is 700 MB.

The project archive should not be load into memory, Instead chunks must be read and transmitted.

Discussion

Stefan Rossbach - 2011-08-15

labels: 1365459 --> Synchronisation / Invitation

assigned_to: nobody --> k_beecher
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Stefan Rossbach - 2012-06-27

assigned_to: k_beecher --> nobody
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Holger Freyther - 2012-07-02

priority: 8 --> 5
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Alexander Waldmann - 2012-07-04

The priority of this bug is at 5 as the probability of this happening is low.

The statistics indicate that users mostly share projects up to 5MB (compressed size).
This might be the case because saros is not able to handle bigger projects, so it should definitely be worked on, but other things are more important than this at the moment.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Stefan Rossbach - 2012-07-04

I think the upper bound is approx. 100 MB (with a max heap 512 MB for the JVM).

To understand the full issue you have to know how garbage collection works, and how the archive is reassembled on invitees side.

I already patch the inviter part, to send larger projects thus only transferring the problem to the other machine.

Depending on how much plugins are activitated in the Eclipse IDE, I think the maximum size that can be transferred can be in between 50 - 70 MB.

Depending on how many third party libraries you are using, or already precompiled parts of the application that are needed to compile a component of your application which you want to work on, this can become really annoying.

Current workaround is to divide your current projects into multiple partial shared projects. And resync over and over again until the whole project is transmitted, so that in your final sync no more data is transferred. That must be done for every single user, because inviting them in parallel will increase the heap usage on the inviter side.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Stefan Rossbach - 2012-11-15

Increased to Prio 7. Even it is not likely that users share such big projects, there should be no reason to "discard users" that needs to initially share such big projects once.

There are some workarounds which will only work unless a single shared file exceeds some size. In this case it is up to the user(s) to "pre receive" the file.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Stefan Rossbach - 2012-11-15

priority: 5 --> 7
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Stefan Rossbach - 2012-11-15

assigned_to: nobody --> kargor
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Stefan Rossbach - 2012-12-17

918f9026b752cf96b211c776ed76a9438f3d7bf7

Special thanks to Patrick Steinhardt !

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Stefan Rossbach - 2012-12-17

status: open --> open-fixed
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Stefan Rossbach - 2013-03-12

milestone: --> 13.3.15
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Stefan Rossbach - 2013-03-21

status: open-fixed --> closed-fixed
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Stefan Rossbach - 2013-03-21

918f9026b752cf96b211c776ed76a9438f3d7bf7

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Log in to post a comment.