#220 xmlwf: memory demands

Not a Bug
closed-wont-fix
5
2003-01-24
2002-11-01
Rolf Ade
No

expat 1.95.5

Running xmlwf against bigger XML files needs almost the
double of the file size of memory (at the end, the
process size grows permanently during the processing).
This looks pretty wrong to me, since the tool does SAX
parsing.

Please notice, that this seems to be a problem only
with the xmlwf code, not with the expat lib per se. An
application by me, build on top of expat, doesn't shows
this behavior. Instead, the memory size of the process
stays constant and pretty low, even while parsing big
XML files - which is, what I also would have expected
from the xmlwf tool.

I'm sorry, for only reporting the problem and not
digging into.

rolf

Discussion

  • Karl Waclawek

    Karl Waclawek - 2002-11-20
    • assigned_to: nobody --> kwaclaw
     
  • Karl Waclawek

    Karl Waclawek - 2002-11-20

    Logged In: YES
    user_id=290026

    What options are you using when calling xmlwf?

     
  • Karl Waclawek

    Karl Waclawek - 2002-11-20

    Logged In: YES
    user_id=290026

    I fixed a small memory leak, but don't think this
    applies here - the leak would only occur under certain
    error conditions.

    I couldn't really find a problem in the xmlwf code
    at a first check. Does this memory leak also happen
    with older versions of the expat lib?

     
  • Rolf Ade

    Rolf Ade - 2002-11-20

    Logged In: YES
    user_id=13222

    The problem shows even with no options given to xmlwf. i.e.
    xmlwf <xmlFile>

    Just use a (big) XML file and watch the xmlwf process grow
    in the process table. (You need a really big one (several 10
    MBytes) to have a change to see it with ordinary ps (on
    unix) or the task manager (on MS), because otherwise expat
    is finished, before you had a chance to see anything ;.)).

    No, I would bet, it's not a (classical) memory leak. I've
    memory debugged expat based applications several times, and
    the lib doesn't leak memory (or only very seldom under rare
    circumstances,)

    I've just valgrind'ed (mem leak checker) an xmlwf run, which
    shows the depicted behavior (xmlwf process size grows far to
    much), but there is no mem leak -. no pointer to allocated
    memory is lost, and all allocated memory is cleand up at the
    end. So eventually there are depending on the file size used
    some really big buffers, or something else.

    rolf

     
  • Karl Waclawek

    Karl Waclawek - 2002-11-20

    Logged In: YES
    user_id=290026

    The files you used for testing:

    - Do they have an internal and/or external DTD subset?
    - Are there external or internal entity references?
    - Are there attributes on some/all elements?

     
  • Karl Waclawek

    Karl Waclawek - 2002-11-21

    Logged In: YES
    user_id=290026

    As it turns out - after reading the manual - the default
    behaviour of xmlwf is to use memory mapped files.
    This explains the observed use of memory.

    With the -r option xmlwf can be made to use
    regular file I/O, and then memory use is as expected.

    Maybe the question could be asked if xmlwf
    should have a different default behaviour.
    But then it would not be backwards compatible.

    Leave open for Fred to comment.

     
  • Karl Waclawek

    Karl Waclawek - 2002-11-21
    • milestone: --> Not a Bug
    • assigned_to: kwaclaw --> fdrake
     
  • Fred L. Drake, Jr.

    Logged In: YES
    user_id=3066

    This needs documentation, at least. I don't see any
    problems with the mmap code in xmlwf/unixfilemap.c.

     
  • Fred L. Drake, Jr.

    • labels: --> Documentation
     
  • Karl Waclawek

    Karl Waclawek - 2003-01-23

    Logged In: YES
    user_id=290026

    I don't think there *are* problems, I guess the observed
    behaviour simply is a consequence of memory mapping.
    It just behaves unexpectedly, and so we should
    document it.

     
  • Fred L. Drake, Jr.

    Logged In: YES
    user_id=3066

    Documentation updated in doc/xmlwf.sgml 1.3. Closing this
    report.

     
  • Fred L. Drake, Jr.

    • status: open --> closed-wont-fix
     

Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:

JavaScript is required for this form.





No, thanks