Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project! See Demo

Close

VTD-XML limits?

Help
2008-03-18
2013-05-15
  • Hello all,

    I've read the format specs at your site, and concluded the following: VTD-XML can only handle up to 1.01GB XML documents, where the maximum size of an element is 1MB, and maximum depth of 255. I assume this is by design, but what should one do if he should file with larger limits into consideration?

    Just a thought - would you care to use VInt (originally defined by Java Lucene Docs: http://lucene.apache.org/java/docs/fileformats.html#VInt\). This will make the format virtually limit-less, and keep VTDs for smaller files smaller (since VInt will use less than 30 bits for smaller numbers).

    Itamar.

     
    • jimmy zhang
      jimmy zhang
      2008-03-18

      Hi, VTD-XML 's maximum document size (if ns is off) is 2GB...
      the max doc size is 1GB when ns is turned on

      the size of text is unlimited, as it internally concatenates/aggregates multiple 1 MB records so there is no limit on that ...

      Also we are extending limits of VTD (by re-organize the layout of VTD to accommodate docs up to 128 GB)

      The lucene stuff looks interesting, will investigate more...
      you are welcome to join VTd-XML-users list and post any questions over there...

       
      • Nah, I'm subscribed to too many mailing lists already, so as long as I'm not a dev or a constant user of VTD-XML (yet), I will rather not too...

        However, I'd really like to know what you will end up doing with regards to VInt. If you could just post here once you've done your invastigation, I'd really appreciate it.

        Itamar.