Re: [Vtd-xml-users] Storing parsing info

[Fernando G. <fer...@gm...>]

Well, jeje, the computer is new but I don't think my disk is so fast. I
think Java or the operating system has to cache something because the first
time I load the file it takes a bit more than 2 seconds and after the first
load, it only takes 300ms to read the file...
I have no experience on doing benchmarks and maybe I'm am missing something.
That's why I attached the program.

"So if you can't delete the orginal XML files, can you compress them and store
them away (archiving)?"
I cannot delete nor archive the GML file because in this context it won't be
rare to be reading it from two different programs at the same time... It's
difficult to find an open source program that does everything you need. For
example, in a development context, there may be a map server serving a map
image based on a GML file while you are opening it to see some data in it.

"The other issue you raised is buffer reuse. To reuse internal buffers
of VTDGen,
you can call setDoc_BR(...). But there is more you can do...
you can in fact reuse the byte array containing the XML document."
Buffer reuse absolutly solves my memory constraints. But the problem I see
with buffer reuse is that it will force me to read and parse the whole XML
file every time the user ask for information on another XML file, won't it?
If I read the XML file by chunks and I store/read the parse information,
each time the user asks for information on another XML file I only have to
read the parse info and a chunk of the XML file.

To show you my point of view:
The "user asking for another XML file" may be a map server that reads some
big GML files and draws its spatial information in a map image. If each time
the map server draws a GML file and "changes" to the next it takes 2 seconds
or so, the drawing of the map (all the GML files) takes too much time.

best regards,
Fernando


On 3/19/07, Jimmy Zhang <cra...@co...> wrote:
>
>
> What intrigues me with Fernando's test results is that it only takes 300ms
> to read a 100MB
> file? He got a super fast disk...
>
> ----- Original Message -----
> *From:* Rodrigo Cunha <rn...@gm...>
> *To:* Jimmy Zhang <cra...@co...>
> *Cc:* Fernando Gonzalez <fer...@gm...> ;
> vtd...@li...
> *Sent:* Sunday, March 18, 2007 8:40 PM
> *Subject:* Re: [Vtd-xml-users] Storing parsing info
>
> In fact the idea occured to me in the past also... but VTD is so fast
> reading large files anyway! With a fast processor I think we might be
> disk-limited rather than processor-limited. Still, if the code is made
> already, the option seems cute enought to keep :-)
>
> Since I mainly deal with large files requiring a lots of processing this
> has not been an issue. Others, in different environments, might disagree.
>
> Jimmy Zhang wrote:
>
> Fernando,  The option for storing VTD in a separate file  is open.
> I attached  the technical document from your last email, and am also
> interested in the suggestions/comments from the mailing list ...
>
>
>