Yeah, the original implementation was built against QuickFIX JNI.
This has a much lower memory imprint than QuickFIX/J. This of course
comes at a cost of portability (which is important for releasing to a
wider audience). There are also some performance differences due to
the JNI layer, but I don't think they are too significant.
If I recall the memory footprint of the Java implementation was at
least twice the size of running against QuickFIX. This can probably
be attributed to Java storing all strings as wide characters (or is
QuickFIX/J storing them as bytes internally? I'm not sure). With
QuickFIX JNI, the strings are just a facade that are created as
necessary, so they are actually stored in memory as regular characters.
One thing that can be done is give the option of running against the
JNI library under windows at least. This would give you the
capability of opening much larger files without any changes to the code.
A smarter allocation strategy that only pulls in a portion of
messages would help solve the dilemma, but the cost would be slower
filtering when working with large files. As it is now, filtering is
pretty instantaneous I think (your experience?).
Also, you should take a look at the start time and end time options
when opening a file. This was placed in there specifically to deal
with very large files that you are only interested in part of. If
you are looking for a problem that you know occurred within the last
hour for instance, you can tell the viewer to only open up that time
period. If the log file spans multiple days, opening up the day you
are interested in is probably a good idea.
--oren
> And now the ugly...
> I attempted to load a 180mb file and the memory usage
> went to 1.5G and basicly stopped. I bumped up the max
> memory size to 1.5gb after it failed with the 512mb
> size.
|
