|
From: Philippe W. <phi...@sk...> - 2010-10-22 22:34:12
|
At work, we are evaluating helgrind and drd (on a big C++ executable). After bypassing bug http://bugs.kde.org/show_bug.cgi?id=255009 , helgrind still crashes relatively quickly due to lack of memory. (drd is still running after about 1 hour of cpu, so there is some hope I will have something when I return to work on monday :). This discusses the helgrind 'out of memory'. We observe that the memory used by helgrind is reasonable during +- 10 minutes of cpu (helgrind uses about two to three times the memory without valgrind). Then in something like one minute, the memory reaches 4 Gb, which causes a crash (as this is a 32 bits application). --profile-heap=yes shows that the cost center "hg.ids.5 (univ_laog)" is the main culprit for the memory usage. Here is the state of the "tool" arena at the crash time: (note: the below is produced with a valgrind containing the fix for "quadratic memory fragmentation" (bug 250101) but the same behaviour is observed with an unpatched 3.6.0. -------- Arena "tool": 3330277376/3330277376 max/curr mmap'd, 3257130328/3257130328 max/curr on_loan -------- 32 in 1: hashtable.Hc.1 48 in 2: commandline.sua.3 64 in 2: commandline.sua.2 192 in 8: libhb.Thr__new.1 216 in 1: initimg-linux.sce.5 304 in 2: hg.mstSs.1 352 in 8: hg.mk_Thread.1 352 in 8: libhb.Thr__new.3 (local_Kws_and_stacks) 400 in 2: hg.pSfs.1 416 in 6: hg.mpttT.1 792 in 49: hg.mctCloa.1 1,136 in 119: hg.lae.1 1,472 in 50: hg.mctCI.1 1,992 in 5: hg.ids.3 2,000 in 1: hg.ids.1 2,544 in 24: hg.lNaw.1 14,408 in 194: errormgr.mre.2 49,152 in 4: libhb.Thr__new.2 196,632 in 1: hashtable.resize.1 354,840 in 39,459: hg.lae.2 679,072 in 41,235: hg.lae.3 786,456 in 1: libhb.event_map_init.2 (context table) 856,848 in 34,844: hg.new_MallocMeta.1 933,144 in 37,645: libhb.zsm_init.1 (map_shmem) 976,296 in 39,579: hg.laog__init.1 1,016,696 in 41,236: hg.laog__init.2 1,652,280 in 169,836: libhb.VTS__new.1 2,173,264 in 135,810: libhb.SO__Alloc.1 2,488,576 in 2: libhb.vts_tab_init.1 3,257,336 in 135,704: hg.ids.2 4,514,648 in 169,837: libhb.vts_set_init.1 5,814,880 in 320,351: hg.ids.4 6,514,000 in 135,703: hg.mk_Lock.1 21,835,480 in 20,529: libhb.event_map_init.4 (oldref tree) 22,923,592 in 339,672: libhb.VTS__new.2 28,095,248 in 1,293: libhb.aFfw.1 (LineF storage) 44,620,304 in 860: libhb.event_map_init.1 (RCEC groups) 51,174,424 in 1,068: libhb.event_map_init.3 (OldRef groups) 3,056,190,440 in 126,826: hg.ids.5 (univ_laog) I understood that the univ_laog is used for producing the lock order checks. (laog seems to be Lock Acquisition Order Graph). So, I tried --track-lockorders=no, but that seems to only disable the reporting of lock order errors, not the building of the data structure. Is it normal that the lock acquisition graph becomes so huge ? (at first sight, I would expect such a graph to not be *that* huge). If it is normal that this graph can become huge, I can try to avoid building this graph when --track-lockorders=no is given. but that assumes that the univ_laog data structure is only useful for track-lockorders. So, is univ_laog only used for track-lockorders=yes ? Or is this univ_laog needed for something else in helgrind ? |
|
From: Julian S. <js...@ac...> - 2010-10-24 10:23:25
|
> I understood that the univ_laog is used for producing the lock order > checks. (laog seems to be Lock Acquisition Order Graph). > > So, I tried --track-lockorders=no, but that seems to only disable the > reporting of lock order errors, not the building of the data structure. Hmm, can you file a bug report? This is something that should be fixed. > Is it normal that the lock acquisition graph becomes so huge ? > (at first sight, I would expect such a graph to not be *that* huge). Hmm, I'm not sure without further analysis. It might happen if your program has a large number of locks at different addresses and it locks them in a large number of different orders. > If it is normal that this graph can become huge, I can try to avoid > building this graph when --track-lockorders=no is given. > but that assumes that the univ_laog data structure is only useful for > track-lockorders. That sounds like a reasonable assumption. > So, is univ_laog only used for track-lockorders=yes ? I think so, yes. > Or is this univ_laog needed for something else in helgrind ? No. You should be able to disable its construction completely. J |
|
From: Philippe W. <phi...@sk...> - 2010-10-24 20:25:07
|
> Hmm, can you file a bug report? This is something that should be fixed. I will work on disabling the construction of univ_laog (and other laog related data structures) and file a bug. > Hmm, I'm not sure without further analysis. It might happen if your > program has a large number of locks at different addresses and it > locks them in a large number of different orders. What is exactly "a large number of locks at different addresses " : Doesn't each lock necessarily have its own address ? Once the laog data structures are not constructed anymore, I guess that helgrind stats produced at the end might help to understand what is going on. |
|
From: Julian S. <js...@ac...> - 2010-10-25 07:30:11
|
> > Hmm, I'm not sure without further analysis. It might happen if your
> > program has a large number of locks at different addresses and it
> > locks them in a large number of different orders.
>
> What is exactly "a large number of locks at different addresses " :
> Doesn't each lock necessarily have its own address ?
The laog mechanism creates a directed graph, in which the locks are
nodes and an edge from lock A to lock B indicates that lock A must
be taken before lock B (for example).
When I say "the locks are nodes", I mean that nodes in the graph are
the addresses of the locks. If you allocate 100000 objects each with
its own lock and and then do { lock(A); lock(B); unlock(B); unlock(A); }
for a few million randomly chosen pairs (A,B) of those objects, then
you will end up with a graph with 100000 nodes and a few million edges.
What I am wondering is, maybe nodes are not removed from the graph when
memory is freed. So eventually the graph ends up with more and more
nodes. This is just a guess, I should point out.
J
|
|
From: Philippe W. <phi...@sk...> - 2010-10-26 22:26:19
|
Filed bug 255353 (with patch) to disable the laog construction with --tack-lockorder=no. Note that the patch also fixes a bug in a data structure used for pretty printing and sanity checking: admin_locks was keeping deleted (and so freed) locks in a list. I made a double linked list of it (asusming we can have many locks) and maintained this list when a lock is removed. > What I am wondering is, maybe nodes are not removed from the graph when > memory is freed. So eventually the graph ends up with more and more > nodes. This is just a guess, I should point out. I wrote a small test program combining N *N locks together. At the end of the run, all locks are destroyed, so I was expecting the laog (and other) data structures to be empty. The stats produced at the end seems to indicate that there are still elements in various data structures. I will try (this week-end) to stop the small test program after each mutex destroy and pretty printing the hg data structures after each destroy. If some of the data is being removed in laog (or others) Word Sets, I guess that when all locks are destroyed, then everything must be empty. Note that without the laog data structures, helgrind still crashes with out of memory. What uses a lot of memory then is the cost center 1,483,979,672 in 393,825: libhb.VTS__new.2 Then when I gave --history-level=approx, I think (I am a little bit lost in all the trials I did) it went further but then crashed on the 'threaded fork' bug (255355). BTW, a lot of the address space is used because each shared library (of valgrind or of the executable itself) is mapped multiple times: at least once for mapping text and once for mapping the (writable) data of the shared lib. The executable has many *huge* shared libs, so that takes about 1Gb of space. Is there a way to indicate to privately map the shared libraries, so as to divide the space used by shared libs by two when running under valgrind ? |