Out of memory error

[Steve Asprelli]

When trying to write structures with large data arrays using "MatFileWriter mfw = new MatFileWriter(outFilePath, mlList, true);" can recieve out of memory error. I believe this happens when 2 Gbyte limit is hit. Is it possible to have "zlib.net.dll" compiled with the largeaddressaware switch set? I have rebuilt csmatio.dll with this option but I believe zlib.net.dll is where the problem is originating as dumpbin.exe indicates it is not large address aware. Perhaps the solution is to avoid using zlib.net.dll altogther as was suggested in another post. If we can compile our own writer, than the largeaddressaware switch could be turned on of off as the application requires.

Thanks very much,
Steve

[Tobias]

Steve, there is a ticket regarding memory issues, see [#4]. Please check if your findings relate to it or if this is a new issue. In the latter case I would suggest you create a new ticket here.

Tobias

[Steve Asprelli]

The memory issue described in ticket#4 may be related although I seem to be seeing out of memory error when the total process memory usage hits 2 Gbyte. I believe this is the process limit for 32-bit applications.Ticket #4 talks about a possible heap limit. I will repost my question regarding zlib.dll and largeAddressaware switch there as the topic does relate to memory utilization issues.

Steve

[Tobias]

I committed [r13] today which eliminates the need of zlib.net.dll if you have .NET 4.0.

But I have no experience with the 'largeaddressaware' option yet. Would it be desirable to have it set per default? Any drawbacks for other users?

[Steve Asprelli]

Thanks for eliminating the dependency on zlib.net. I tried the r13 release with the 'largeaddressaware' option set but still recieved the out of memory error when trying to write very large double arrays. The exception occurs in the WriteTo( BinaryWriter os ) method at os.Write. I found it made no difference if compression was on or off. I tried flushing the stream periodically thoughout the write but memory usage continued to increase throughout the write. I'm not sure why memory is increasingly consumed while writing to the stream. In any event, when memory usage hits 2 GByte regardless of 'largeaddressaware' option, an exception will occur. So, unless someone has some idea of how to avoid this memory consumption while writing to the stream, I think the best option is to keep the data structures within the limits of available memory.

[Aaron B]

I was having the same problem and was able to fix it by recompiling with .NET 4.5. I downloaded the csmatio code and changed the target framework to 4.5 (including updating the .vcproj files so NET40 was using 4.5 instead of 4.0) and recompiled. Then using the new gcAllowVeryLargeObjects runtime tag in .NET 4.5 I was able to use csmatio to save the large files I was working with from a .NET 4.5 targeted console application.

[Tobias]

Thanks for sharing!

[Dr YSG]

Hey thanks for this thread and the library. I just built a x64 version for .NET 4.6 and used csmatio to write 1GB mat file (two 50K elemet struct arrays).

I needed to add an app.config and include the following lines:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
<startup> 
<supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.6"/>
</startup>
</configuration>

Make sure you set the application build (app settings) to x64 (I am using VS2015 community edition).

Maybe you want to include this version of the dll in the downloads?

Thanks greatly guys. I was doing all this sensor data upload via DLL that was sending events to Matlab, and it was taking forever.

[Tobias]

I'd be glad to provide your DLL to the community in the "Files" section. Please add a little description and send me a link where I can download it.