#4864 Thread pool leaks memory

open-accepted
7
2011-07-29
2011-07-29
No

Using thread pool to do a computation in parallel with many thousand tasks, I noticed, that tpool leaks memory. The script attached demonstrates the loss both in ActiveTcl 8.5.10.1 on Windows/64 and in tcl8.6 HEAD on Mac OSX. My real computation ran for several hours on 6 cores and the leaked memory summed up to several hundred MB. I could not locate the leakage any further, first I thought the culprit was the tpool::wait command with the list result, but replacing it with a simple "after" in the test case has no influence. Also, tpool::release can not free the leaked memory.

Discussion

  • Zoran Vasiljevic

    I cannot confirm this on Max OSX using Tcl8.4

    % array get tcl_platform
    osVersion 10.7.0 byteOrder littleEndian tip,268 1 threaded 1 machine i386 platform unix os Darwin tip,280 1 user zoran wordSize 4
    % set tcl_version
    8.4

    When I start your script it ocmplains about math max function.
    I remove that out of the code (it is not really related to anything
    thread-wise) and run the script. It allocates some 30MB virtual
    memory and stays there no matter howmany times i run your
    leak proc.

    I gues this has something to do with new(er) Tcl cores. I have
    not tested anything later then 8.4.

     
  • Christian Gollwitzer

    I've updated the script for 8.4 without max. If I run it with
    tclsh8.4 threadpool_leakdemo84.tcl
    then the value in activity monitor "physical memory" winds up, every ten seconds +0.1MB. It starts from 3.0 and goes up to 6MB. You are right that "virtual memory" does not change; I wonder, why.

     
  • Christian Gollwitzer

    My platform:

    91-65-210-59-dynip:Programmieren chris$ tclsh8.4
    % parray tcl_platform
    tcl_platform(byteOrder) = littleEndian
    tcl_platform(machine) = i386
    tcl_platform(os) = Darwin
    tcl_platform(osVersion) = 10.8.0
    tcl_platform(platform) = unix
    tcl_platform(threaded) = 1
    tcl_platform(tip,268) = 1
    tcl_platform(tip,280) = 1
    tcl_platform(user) = chris
    tcl_platform(wordSize) = 4
    %

    PS: Funny version. My OS is Snow Leopard 10.6.8, why does it say 10.8.0?

     
  • Zoran Vasiljevic

    The virtual memory is what counts. The real memory is
    not important here. If virtual memory does not increase
    you do not have leak.

    The OS versioning is weird by Mac, yes.
    I also have SnowLeopard but 10.6.7. This is no problem.

     
  • Christian Gollwitzer

    OK, so try the third version. I shortened the inner loop and made more iterations. Now you'll see, that also virtual memory winds up.

     
  • Zoran Vasiljevic

    • priority: 5 --> 7
    • status: open --> open-accepted
     
  • Zoran Vasiljevic

    yes. this eats up memory somewhere. will double-check and fix.
    thanks for reporting this!

     
  • Zoran Vasiljevic

    I think I would need to release a new version. My own private sandbox
    seems to work fine. The 2.6.5 that gets distributed with Mac OSX seems
    to leak. But I need to figure out where are the sources! I heard SF is not
    hosting the sources any more...

     
  • Zoran Vasiljevic

    Hey, thanks Andreas! I think I will need to do some reading and learning
    new stuff...

    Apropos reported leak... did you try 2.6.6 release? I saw I have already
    corrected some things related to tpool already...

     
  • Andreas Kupries

    Andreas Kupries - 2011-07-29

    You have seen that Don Porter has RC for 2.6.7 up ?
    See tcl-core mailinglist.

     
  • Zoran Vasiljevic

    yes. there are no significant core-code related changes there.
    if 2.6.6 does not leak, the 2.6.7 will not either.

     
  • Andreas Kupries

    Andreas Kupries - 2011-07-29

    Oh, the question about '... did you try 2.6.6 release' was for Christian ...

     
  • Jeffrey Hobbs

    Jeffrey Hobbs - 2011-07-29

    I am able to repro the leak on Windows with ActiveTcl 8.5.9.2 and 8.4.19.5 using Thread 2.6.7 (not final).

     
  • Christian Gollwitzer

    I foun dthe leak originally witjin ActiveState 8.5.10.1 on Windows. I can reproduce it using Tcl8.6 HEAD:
    bin/tclsh8.6
    % info patchlevel
    8.6b1.2
    % package require Thread
    2.6.7
    %

    This Tcl8.6 is 64bit, the leakage seems to be twice as fast as with the 32bit version.

     
  • Christian Gollwitzer

    If I change the tpool::wait command, which is supposed to give back the still running processes in the list, the leakage becomes a bit slower (I checked after a fixed number of iterations). See the attached demo script. Therefore the joblist leaks, but it's not the only leak in here.

     
  • Bob Techentin

    Bob Techentin - 2012-11-26

    I see the same memory leak with Tcl 8.5.13 and Thread 2.6.7 on Linux.

    It looks to me like the resource leak is related to the thread pool maintaining information on completed jobs.

    I inserted periodic calls to tpool::wait and tpool::get inside the loop that posts work to the thread pool, and observed reduced memory growth. (About half) If I call tpool::get twice on the same job, I get "no such job," so I presume that it cleans up some resources. But I don't see anything explicit in the API to discard results from completed jobs.

    When I changed the post option from -nowait to -detached, the memory leak went away.

    Bob

     

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks