Re: [Quickfix-users] QuickFIX Logviewer 1.0.0

[Oren M. <or...@qu...>]

Yeah, the original implementation was built against QuickFIX JNI.   
This has a much lower memory imprint than QuickFIX/J.  This of course  
comes at a cost of portability (which is important for releasing to a  
wider audience).  There are also some performance differences due to  
the JNI layer, but I don't think they are too significant.

If I recall the memory footprint of the Java implementation was at  
least twice the size of running against QuickFIX.  This can probably  
be attributed to Java storing all strings as wide characters (or is  
QuickFIX/J storing them as bytes internally?  I'm not sure).  With  
QuickFIX JNI, the strings are just a facade that are created as  
necessary, so they are actually stored in memory as regular characters.

One thing that can be done is give the option of running against the  
JNI library under windows at least.  This would give you the  
capability of opening much larger files without any changes to the code.

A smarter allocation strategy that only pulls in a portion of  
messages would help solve the dilemma, but the cost would be slower  
filtering when working with large files.  As it is now, filtering is  
pretty instantaneous I think (your experience?).

Also, you should take a look at the start time and end time options  
when opening a file.  This was placed in there specifically to deal  
with very large files that you are only interested in part of.  If  
you are looking for a problem that you know occurred within the last  
hour for instance, you can tell the viewer to only open up that time  
period.  If the log file spans multiple days, opening up the day you  
are interested in is probably a good idea.

--oren

> And now the ugly...
> I attempted to load a 180mb file and the memory usage
> went to 1.5G and basicly stopped.  I bumped up the max
> memory size to 1.5gb after it failed with the 512mb
> size.