I'm having Java Heap space errors when synchronizing 10000 5MB files, on a PUSH operation from the Master node to a Client node.
Error Stacktrace
23:01:19.772 [master-file_push-1] ERROR o.j.s.s.i.NodeCommunicationService - Failed to execute FILE_PUSH for node 002
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3236) ~[na:1.8.0_45]
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118) ~[na:1.8.0_45]
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) ~[na:1.8.0_45]
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153) ~[na:1.8.0_45]
at sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:78) ~[na:1.8.0_45]
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1793) ~[commons-io-2.4.jar:2.4]
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769) ~[commons-io-2.4.jar:2.4]
at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744) ~[commons-io-2.4.jar:2.4]
at org.jumpmind.symmetric.service.impl.FileSyncService.sendFiles(FileSyncService.java:447) ~[symmetric-core-3.7.21.jar:3.7.21]
at org.jumpmind.symmetric.service.impl.FileSyncService.pushFilesToNode(FileSyncService.java:613) ~[symmetric-core-3.7.21.jar:3.7.21]
at org.jumpmind.symmetric.service.impl.FileSyncService.execute(FileSyncService.java:597) ~[symmetric-core-3.7.21.jar:3.7.21]
at org.jumpmind.symmetric.service.impl.NodeCommunicationService$2.run(NodeCommunicationService.java:394) ~[symmetric-core-3.7.21.jar:3.7.21]
According to several topics in this forum, i've upped the JVM heap memory to 1024m, but as the error persisted, i upped to 2048m: -Xms2048m -Xmx2048m
I've also setted -XX:-UseGCOverheadLimit
Nevertheless, even with 2048m, the error persists.
To have a clear idea how the memory was being used, i logged the memory statistics, and prior to the OutOfMemoryError, this was the state of the JVM process:
Used Memory: 508
Free Memory: 1470
Total Memory: 1979
Max Memory: 1979
According to the logs, the JVM process does not use more than 512MB, and has a lot of free memory available.
I'm using SymmetricDS embedded, so i'm suspecting that the settings i've changed are the settings of the main JVM process, but are not the settings of the SymmetricDS embedded process.
Is this correct? And if so, how can i change the settings of the embedded SymmetricDS process?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
http.push.stream.output.enabled really should be turned on by default. It enables chunking on http PUTs which is what a push uses. If it isn't set the entire PUT payload is buffered before it is flushed.
The wrapper.java settings are for when running as a standalone service. They are set in sym_service.conf.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Yes. Heap settings are set via the normal JVM settings when running embedded.
We always use the latest JDK and set mx and ms settings to adjust heap size and things seem to work pretty good. We typically set ms and mx setting to the same value.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I'm having Java Heap space errors when synchronizing 10000 5MB files, on a PUSH operation from the Master node to a Client node.
Error Stacktrace
23:01:19.772 [master-file_push-1] ERROR o.j.s.s.i.NodeCommunicationService - Failed to execute FILE_PUSH for node 002
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3236) ~[na:1.8.0_45]
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118) ~[na:1.8.0_45]
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) ~[na:1.8.0_45]
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153) ~[na:1.8.0_45]
at sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:78) ~[na:1.8.0_45]
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1793) ~[commons-io-2.4.jar:2.4]
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769) ~[commons-io-2.4.jar:2.4]
at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744) ~[commons-io-2.4.jar:2.4]
at org.jumpmind.symmetric.service.impl.FileSyncService.sendFiles(FileSyncService.java:447) ~[symmetric-core-3.7.21.jar:3.7.21]
at org.jumpmind.symmetric.service.impl.FileSyncService.pushFilesToNode(FileSyncService.java:613) ~[symmetric-core-3.7.21.jar:3.7.21]
at org.jumpmind.symmetric.service.impl.FileSyncService.execute(FileSyncService.java:597) ~[symmetric-core-3.7.21.jar:3.7.21]
at org.jumpmind.symmetric.service.impl.NodeCommunicationService$2.run(NodeCommunicationService.java:394) ~[symmetric-core-3.7.21.jar:3.7.21]
According to several topics in this forum, i've upped the JVM heap memory to 1024m, but as the error persisted, i upped to 2048m: -Xms2048m -Xmx2048m
I've also setted -XX:-UseGCOverheadLimit
Nevertheless, even with 2048m, the error persists.
To have a clear idea how the memory was being used, i logged the memory statistics, and prior to the OutOfMemoryError, this was the state of the JVM process:
Used Memory: 508
Free Memory: 1470
Total Memory: 1979
Max Memory: 1979
According to the logs, the JVM process does not use more than 512MB, and has a lot of free memory available.
I'm using SymmetricDS embedded, so i'm suspecting that the settings i've changed are the settings of the main JVM process, but are not the settings of the SymmetricDS embedded process.
Is this correct? And if so, how can i change the settings of the embedded SymmetricDS process?
Any help here?
This question is really important, and could be the difference between using SymmetricDS or not ...
Try setting http.push.stream.output.enabled=true
Thank you.
After setting this property it worked.
Do you mind explaining what is the purpose of this flag?
Regards and thank you for your help.
Does this properties set the memory parameters for the embeded SymmetricDS Server?
wrapper.java.initmemory=2048
wrapper.java.maxmemory=2048
http.push.stream.output.enabled really should be turned on by default. It enables chunking on http PUTs which is what a push uses. If it isn't set the entire PUT payload is buffered before it is flushed.
The wrapper.java settings are for when running as a standalone service. They are set in sym_service.conf.
Hello Chris, thank you for the clarification.
So, if i'm using an embedded SymmetricWebServer, this SymmetricDS process uses the JVM settings of the main process?
Regarding JVM settings, is there global guidelines regarding how to optimize it, or does it entirely dependent on the deployment environment?
Yes. Heap settings are set via the normal JVM settings when running embedded.
We always use the latest JDK and set mx and ms settings to adjust heap size and things seem to work pretty good. We typically set ms and mx setting to the same value.