Memory Leaks

2010-02-13
2012-11-08
  • Hi, I'm playing with dynamicly linked djvulibre on Symbian. And have few
    quistions about its memory management:

    If i use only following two lines of code in my app, i get memory leak in it.

    ddjvu_context_create(ctx);
    ddjvu_context_release(ctx);

    Looks like objects created by DjVuMessage::use_language(),
    DjVuMessageLite::create() and DjVuFileCache::create() doesnt eleted
    propertly. Thats crytical on Symbian - application cant pass certification in
    that case.

    Is that memory leak is a something what expected to be? or something doesn't
    work in my port?

    And last small quistion. Looks like dll doesnt have entry point (at least i
    cant find it). Is that right or i miss it?

     
  • Leon Bottou
    Leon Bottou
    2010-02-13

    Djvulibre is designed for operating systems that automatically track and
    reclaim process resources.

    On such machines you can allocate a small number of objects and leave them to
    be reclaimed by the operating system.
    What is important is that this number does not increase when you repeatedly
    call the djvulibre functions.
    For instance, in the following sequence, the number of such objects will not
    increase with n.
    for (int i=1; i<n; i++) { ddjvu_context_t ctx =
    ddjvu_context_create(argv); ddjvu_context_release(ctx); }

    In that case, you could call minilisp_finish() (from miniexp.h) to
    deallocate what is left.

    But that will not be sufficient for the normal operation of djvulibre because
    old code
    caches a lots of things for later use. To avoid multi-thread difficulties,
    some objects are
    not deallocated right away but added to a queue of objects to be deleted
    later.
    On the other hand, the DjVuMessage mess could be avoided,

    There is no entry point for the library.
    Under unix, entry points are not required.
    Under windows, the compiler generates an entry point that
    calls the c++ static initializers and destructors.

     
  • I'm using only one ddjvu_context_create()/ddjvu_context_create() and
    minilisp_finish() doesn't help.
    Looks like i should rewrite djvulibre from scratch to get rid from memory
    leaks.

     
  • I was looking at the code for ddjvu_context, and to me it looks like you are
    incorrect. DjVuFileCache is a GP enabled pointed. So ddjvu_context_release
    should release it instantly, provided there are no remaining object in memory
    still referencing the cache.

    As for the DjVuMessage stuff, I agree. Under the wikipedia definition of
    memory leak, DjVuMessage indeed has a memory leak. Traditionally values
    assigned to static variables are not considered leak memory, However, I could
    find no modern definitions of a memory leak that makes this distinction.

    So if you need to rewrite things to avoid "memory leaks", I suggest adding
    hooking into DjVuMessage so the static GP pointers can be set to null from
    ddjvu_context_release. I would further suggest making this controllable by a
    flag. So programs that repeatedly reload libdjvu have the option not to
    deallocate the static variables each time.

    Bill

     
  • I should further clarify. I also agree with Leon, that these "memory leaks"
    are completely non-harmful. They do not grow over time. It is simply that the
    memory is not dereferenced. Ten years ago, I probably would have gone further
    and just completely ignore your comments as un-informed. However, since that
    time I've had personal experience dealing with certification standards. Often
    times, the standards really do not make sense because they are overly broad.
    Static pointers holding small amounts of memory across invocations being
    considered memory leaks does not make sense. But an even more extreme example,
    the FAA will not allow applications to dynamically allocate memory. Does that
    really make sense in every case? No. But those are the rules and sometimes you
    have to live in them.

    I see your need to eliminate the static referenced memory to pass
    certification standards. I am certain that both Leon and myself will be happy
    to help point you as to where else in the code to look, as I would be
    surprised if DjVuMessage* has the only static pointers… I am less confident
    however, that Leon would agreeable to incorporate any such changes back into
    the main branch. So I suggest you clarify any issues and requirements he would
    have if that is part of your goal.

    Bill

     
  • Leon Bottou
    Leon Bottou
    2010-02-16

    To chime in.

    I believe that after ten years, the djvulibre library is in need of
    refactoring.
    That was complicated by the fact that many critical programs (viewers,
    utilities)
    were dependent on subtle internal details of the library.

    The first step was to create well documented apis such as ddjvuapi
    and to rewrite these programs using the api and only the api.
    This is what djview4 was for, and that took a couple years.

    Now we are in position to completely refactor the library
    provided we maintain the API constant. But that requires
    a lot of time that I simply do not have these days.
    Also we have to make sure that the end product is
    better than the original otherwise it won't take over…

    Docbill says that he could not a modern definitions of a memory leak that
    makes the distinction.
    In fact the "valgrind" program to check memory problems makes the distinction
    between
    "reachable allocated blocks" and "leaked blocks". The leaked ones are the ones
    that
    are no longer reachable using pointers found in allocated blocks.

    In djvulibre, the leaked-reachable blocks come from:

    - Code in DjVuMessage in need of simplification.
    Switching to a more standard approach to localization would eliminate all
    that.
    But this remains a big change in the code base with many opportunities for
    bugs.

    - Code that delays the deletion of some objects in DataPool (overloaded
    operator delete).
    These are tricky to get rid of. But we can safely delete everything when we
    are sure
    that all djvu jobs are terminated (i.e. we could add a function to cleanup
    the leftovers
    and call it after calling ddjvu_context_destroy.)

     
  • Syafz
    Syafz
    2011-10-11

    trufanov, these "memory leaks" are completely non-harmful.

    inspiredmagz