Menu
▾
▴
Re: [Ganglia-developers] Possible memory leak in gmond?
|
From: Ramon B. <bas...@sa...> - 2005-07-13 09:57:53
|
Hi Martin, I am currently a few weeks away from the office and do not have the ability to perform that test. Also my co workers have allready killed the gmond's everywhere because they were consuming too much memory. So the multicasting of the entire cluster/network/node metrics is no longer online and useable to test memory consumption. Btw, in your test it says config file not found, so it does not do/prove much I think. Kind regards, - Ramon Martin Knoblauch wrote: >Hi Ramon, > > you could try to run something like "valgrind gmond -d 1" for a longer >period. I did it for half an hour (on a small cluster). The tool did >not find any leaks in the sense that memory was "forgotten"or "lost", >but it can be that "gmond" is collecting memory over time without doing >some kind of "garbage collection". Maybe Matt knows more. > >[root@l15833 gmond]# valgrind ./gmond -d 1 >==23901== Memcheck, a memory error detector for x86-linux. >==23901== Copyright (C) 2002-2005, and GNU GPL'd, by Julian Seward et >al. >==23901== Using valgrind-2.4.0, a program supervision framework for >x86-linux. >==23901== Copyright (C) 2000-2005, and GNU GPL'd, by Julian Seward et >al. >==23901== For more details, rerun with: -v >==23901== >Configuration file '/etc/gmond.conf' not found. > > >==23901== >==23901== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 25 from >2) >==23901== malloc/free: in use at exit: 890383 bytes in 984 blocks. >==23901== malloc/free: 2163 allocs, 1179 frees, 1028325 bytes >allocated. >==23901== For counts of detected errors, rerun with: -v >==23901== searching for pointers to 984 not-freed blocks. >==23901== checked 1250220 bytes. >==23901== >==23901== LEAK SUMMARY: >==23901== definitely lost: 0 bytes in 0 blocks. >==23901== possibly lost: 0 bytes in 0 blocks. >==23901== still reachable: 890383 bytes in 984 blocks. >==23901== suppressed: 0 bytes in 0 blocks. >==23901== Reachable blocks (those to which a pointer was found) are not >shown. >==23901== To see them, rerun with: --show-reachable=yes > > So you can see that there are 984 calls to "alloc" without a "free", >but all those allocations are still reachable to the gmond. But it >seems the number (984) grows with the runtime of gmond. > >Cheers >Martin > >--- Ramon Bastiaans <bas...@sa...> wrote: > > > >>Martin, >> >>We are running Debian sarge and kernel 2.6.11 with Ganglia 3.0.1. >> >>Ramon. >> >>Martin Knoblauch wrote: >> >> >> >>>Hi Ramon, >>> >>>on which platform did you observe the behaviour? >>> >>>Martin >>> >>>--- Ramon Bastiaans <bas...@sa...> wrote: >>> >>> >>> >>> >>> >>>>Has anyone ever noticed a memory usage increase over time by gmond? >>>> >>>>We found that some users were unable to run more than 1 job on a >>>> >>>> >>node >> >> >>>>and traced it back to gmond's memory usage being the cause. On some >>>> >>>> >>>>nodes gmond had 25 Mb memory usage and 225 Mb RSS usage. >>>> >>>>However on some other nodes it did not. The difference seemed to be >>>>the >>>>total runningtime of the gmond process, which seems to implicate a >>>>possible memory leak somewhere. >>>> >>>>We are now considering a unicast/deaf setup hoping it will >>>>prevent/workaround any leakage. >>>> >>>>Kind regards, >>>>- Ramon. >>>> >>>>-- >>>>"... being a Linux user is sort of like living in a house inhabited >>>>by a large family of carpenters and architects. Every morning when >>>>you wake up, the house is a little different. Maybe there is a new >>>>turret, or some walls have moved. Or perhaps someone has >>>> >>>> >>temporarily >> >> >>>>removed the floor under your bed." - Unix for Dummies, 2nd Edition >>>> >>>> >>>> >>>> >>>>------------------------------------------------------- >>>>This SF.Net email is sponsored by the 'Do More With Dual!' webinar >>>>happening >>>>July 14 at 8am PDT/11am EDT. We invite you to explore the latest in >>>>dual >>>>core and dual graphics technology at this free one hour event >>>> >>>> >>hosted >> >> >>>>by HP, >>>>AMD, and NVIDIA. To register visit >>>> >>>> >>http://www.hp.com/go/dualwebinar >> >> >>>>_______________________________________________ >>>>Ganglia-developers mailing list >>>>Gan...@li... >>>>https://lists.sourceforge.net/lists/listinfo/ganglia-developers >>>> >>>> >>>> >>>> >>>> >>>> >>>------------------------------------------------------ >>>Martin Knoblauch >>>email: k n o b i AT knobisoft DOT de >>>www: http://www.knobisoft.de >>> >>> >>> >>> >>-- >>"... being a Linux user is sort of like living in a house inhabited >>by a large family of carpenters and architects. Every morning when >>you wake up, the house is a little different. Maybe there is a new >>turret, or some walls have moved. Or perhaps someone has temporarily >>removed the floor under your bed." - Unix for Dummies, 2nd Edition >> >> >> >> > > >------------------------------------------------------ >Martin Knoblauch >email: k n o b i AT knobisoft DOT de >www: http://www.knobisoft.de > > -- "... being a Linux user is sort of like living in a house inhabited by a large family of carpenters and architects. Every morning when you wake up, the house is a little different. Maybe there is a new turret, or some walls have moved. Or perhaps someone has temporarily removed the floor under your bed." - Unix for Dummies, 2nd Edition |