Re: [Vtd-xml-users] Storing parsing info

[Jimmy Z. <cra...@co...>]

So what you are trying to accomplish is to load all the GML docs into =
memory at once...
I guess you can simply index all those files to avoid parsing...
but I still don't seem to understand the benefits of read teh parse info =
and a chunk of
the XML file..
  ----- Original Message -----=20
  From: Fernando Gonzalez=20
  To: vtd...@li...=20
  Sent: Monday, March 19, 2007 2:56 AM
  Subject: Re: [Vtd-xml-users] Storing parsing info


  Well, jeje, the computer is new but I don't think my disk is so fast. =
I think Java or the operating system has to cache something because the =
first time I load the file it takes a bit more than 2 seconds and after =
the first load, it only takes 300ms to read the file...=20
  I have no experience on doing benchmarks and maybe I'm am missing =
something. That's why I attached the program.

  "So if you can't delete the orginal XML files, can you compress them =
and=20
  store them away (archiving)?"

  I cannot delete nor archive the GML file because in this context it =
won't be rare to be reading it from two different programs at the same =
time... It's difficult to find an open source program that does =
everything you need. For example, in a development context, there may be =
a map server serving a map image based on a GML file while you are =
opening it to see some data in it.=20

  "The other issue you raised is buffer reuse. To reuse internal buffers =
of=20
  VTDGen, you can call setDoc_BR(...). But there is more you can do...
  you can in fact reuse the byte array containing the XML document."
  Buffer reuse absolutly solves my memory constraints. But the problem I =
see with buffer reuse is that it will force me to read and parse the =
whole XML file every time the user ask for information on another XML =
file, won't it? If I read the XML file by chunks and I store/read the =
parse information, each time the user asks for information on another =
XML file I only have to read the parse info and a chunk of the XML file. =


  To show you my point of view:
  The "user asking for another XML file" may be a map server that reads =
some big GML files and draws its spatial information in a map image. If =
each time the map server draws a GML file and "changes" to the next it =
takes 2 seconds or so, the drawing of the map (all the GML files) takes =
too much time.=20

  best regards,
  Fernando



  On 3/19/07, Jimmy Zhang <cra...@co...> wrote:=20

    What intrigues me with Fernando's test results is that it only takes =
300ms to read a 100MB
    file? He got a super fast disk...
      ----- Original Message -----=20
      From: Rodrigo Cunha=20
      To: Jimmy Zhang=20
      Cc: Fernando Gonzalez ; vtd...@li...=20
      Sent: Sunday, March 18, 2007 8:40 PM
      Subject: Re: [Vtd-xml-users] Storing parsing info


      In fact the idea occured to me in the past also... but VTD is so =
fast reading large files anyway! With a fast processor I think we might =
be disk-limited rather than processor-limited. Still, if the code is =
made already, the option seems cute enought to keep :-)

      Since I mainly deal with large files requiring a lots of =
processing this has not been an issue. Others, in different =
environments, might disagree.

      Jimmy Zhang wrote:=20
        Fernando,  The option for storing VTD in a separate file  is =
open.=20
        I attached  the technical document from your last email, and am =
also=20
        interested in the suggestions/comments from the mailing list ... =







-------------------------------------------------------------------------=
-----


  =
-------------------------------------------------------------------------=

  Take Surveys. Earn Cash. Influence the Future of IT
  Join SourceForge.net's Techsay panel and you'll get the chance to =
share your
  opinions on IT & business topics through brief surveys-and earn cash
  =
http://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge&CID=3D=
DEVDEV


-------------------------------------------------------------------------=
-----


  _______________________________________________
  Vtd-xml-users mailing list
  Vtd...@li...
  https://lists.sourceforge.net/lists/listinfo/vtd-xml-users