Re: [Vtd-xml-users] Some findings on VTD-XML memory usage, performance, other

[Jimmy Z. <cra...@co...>]

>>
>> > No, what I mean is that while document after
>> parsing
>> > only uses 1.3 - 1.7x memory, during parsing it
>> needs
>> > additional 1x storage for the input. There is no
> ...
>> Actually it doesn't need extra 1x storage, because
>> vtd-xml
>> also reads the byte content into its internal
>> storage as the first
>> step, it is part of 1.3x~1.7x memory consumption...
>
> Ok. JavaDocs did not indicate this -- it's actually
> bit dangerous to reference the same array, as caller
> may go ahead and start modifying or reusing it. I
> assumed a copy was made, as is typically done when one
> has to share raw mutable arrays.
>
> But as long as (java)docs clearly indicate that the
> parser now owns the byte array, that's understandable
> to avoid overhead.
>

true, they need to be careful on this....

> ...
>> One of the most subtle point of VTD-XML is that in
>> many cases, you
>> never have to convert character data into java
>> strings... why? because
>> string is almost always an intermediate stage of
>> processing,
>
> Only for some cases of data-oriented XML processing.
> Strings are typical data artifacts for textual info.
> There are of course other ways to represent data, such
> as raw char arrays -- that's what SAX parsers typical
> pass for CHARACTERS segments, not Strings.
>

I think once people starts working with VTD-XML, they will
get comfortable using VTD records, ie. making string to VTD
comparison, convert VTD to ints and floats... etc...

>
> But thinking about it now though, it probably was a
> side
> effect of virtual memory management.
> With 176M file, and main memory of 1 gig, my system
> may
> actually be using all of its physical memory. If so,
> it is not a property of VTD-XML processing. So it is
> possible that larger data size does not have drastic
> effects on performance. This could be verified with
> a separate test machine, with bit more memory.

I was able to push my machine (1GB memory loaded with Windows XP) to parse
400MB + files

>
> -+ Tatu +-
>
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share 
> your
> opinions on IT & business topics through brief surveys -- and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Vtd-xml-users mailing list
> Vtd...@li...
> https://lists.sourceforge.net/lists/listinfo/vtd-xml-users
>