|
From: Yeshurun, M. <mei...@in...> - 2011-10-14 13:32:44
|
Hi, I recently had to debug a double free. This could have been a piece of cake with Valgrind, but unfortunately the first and second frees were a few minutes of runtime apart, so all I got was the "address was not recently malloc()ed..." message instead of the backtrace corresponding to the first free. When I increased the -free-list-vol to the maximum value allowed by Valgrind, Valgrind aborted prematurely, apparently because the 32G memory limit was reached. My machine had 96G, and I'm sure it would have been enough for obtaining the backtrace of the first free. Is it possible to overcome the 32G limit? I don't think machines with more memory than that are rare. Thanks, Meir --------------------------------------------------------------------- Intel Israel (74) Limited This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. |
|
From: WAROQUIERS P. <phi...@eu...> - 2011-10-14 14:01:23
|
I think it is not possible (or at least without significant work) to overcome the 32Gb limit. An alternative solution might be: if you have a mixture of big allocations and small allocations in your application, and your double free is of a "small" size, then you could take a 3.7.0 from SVN (see http://www.valgrind.org/downloads/repository.html ) and apply the patch in bugzilla http://bugs.kde.org/show_bug.cgi?id=250065 With this patch, you have a new option --freelist-big-blocks=.... which allows to specify the size above which the blocks are "released" from the free list in priority. With this, you might "prune" parts of the big alloc/free from the free list, allowing you to find the double free of your "small" block. Philippe ________________________________ From: Yeshurun, Meir [mailto:mei...@in...] Sent: Friday 14 October 2011 15:31 To: val...@li... Subject: [Valgrind-users] --free-list-vol limit Hi, I recently had to debug a double free. This could have been a piece of cake with Valgrind, but unfortunately the first and second frees were a few minutes of runtime apart, so all I got was the "address was not recently malloc()ed..." message instead of the backtrace corresponding to the first free. When I increased the -free-list-vol to the maximum value allowed by Valgrind, Valgrind aborted prematurely, apparently because the 32G memory limit was reached. My machine had 96G, and I'm sure it would have been enough for obtaining the backtrace of the first free. Is it possible to overcome the 32G limit? I don't think machines with more memory than that are rare. Thanks, Meir --------------------------------------------------------------------- Intel Israel (74) Limited This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. ____ This message and any files transmitted with it are legally privileged and intended for the sole use of the individual(s) or entity to whom they are addressed. If you are not the intended recipient, please notify the sender by reply and delete the message and any attachments from your system. Any unauthorised use or disclosure of the content of this message is strictly prohibited and may be unlawful. Nothing in this e-mail message amounts to a contractual or legal commitment on the part of EUROCONTROL, unless it is confirmed by appropriately signed hard copy. Any views expressed in this message are those of the sender. |
|
From: Yeshurun, M. <mei...@in...> - 2011-10-14 14:19:49
|
Thanks. Sounds like a nice way to allow a bigger free list volume and it would have helped in my case. I'll just wait for the 3.7 release since I already fixed this bug. But I would still love to see Valgrind get rid of the 32G limit. Thanks, Meir ________________________________ From: WAROQUIERS Philippe [mailto:phi...@eu...] Sent: Friday, October 14, 2011 4:00 PM To: Yeshurun, Meir; val...@li... Subject: RE: [Valgrind-users] --free-list-vol limit I think it is not possible (or at least without significant work) to overcome the 32Gb limit. An alternative solution might be: if you have a mixture of big allocations and small allocations in your application, and your double free is of a "small" size, then you could take a 3.7.0 from SVN (see http://www.valgrind.org/downloads/repository.html ) and apply the patch in bugzilla http://bugs.kde.org/show_bug.cgi?id=250065 With this patch, you have a new option --freelist-big-blocks=.... which allows to specify the size above which the blocks are "released" from the free list in priority. With this, you might "prune" parts of the big alloc/free from the free list, allowing you to find the double free of your "small" block. Philippe ________________________________ From: Yeshurun, Meir [mailto:mei...@in...] Sent: Friday 14 October 2011 15:31 To: val...@li... Subject: [Valgrind-users] --free-list-vol limit Hi, I recently had to debug a double free. This could have been a piece of cake with Valgrind, but unfortunately the first and second frees were a few minutes of runtime apart, so all I got was the "address was not recently malloc()ed..." message instead of the backtrace corresponding to the first free. When I increased the -free-list-vol to the maximum value allowed by Valgrind, Valgrind aborted prematurely, apparently because the 32G memory limit was reached. My machine had 96G, and I'm sure it would have been enough for obtaining the backtrace of the first free. Is it possible to overcome the 32G limit? I don't think machines with more memory than that are rare. Thanks, Meir --------------------------------------------------------------------- Intel Israel (74) Limited This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. ____ This message and any files transmitted with it are legally privileged and intended for the sole use of the individual(s) or entity to whom they are addressed. If you are not the intended recipient, please notify the sender by reply and delete the message and any attachments from your system. Any unauthorised use or disclosure of the content of this message is strictly prohibited and may be unlawful. Nothing in this e-mail message amounts to a contractual or legal commitment on the part of EUROCONTROL, unless it is confirmed by appropriately signed hard copy. Any views expressed in this message are those of the sender. --------------------------------------------------------------------- Intel Israel (74) Limited This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. |
|
From: WAROQUIERS P. <phi...@eu...> - 2011-10-14 15:15:45
|
Note that this patch is at this moment not integrated in 3.7.0 SVN. It still has to be examined/reviewed/.... and then if deemed appropriate, be integrated. So, there is no guarantee that it will be in 3.7.0. Philippe ________________________________ From: Yeshurun, Meir [mailto:mei...@in...] Sent: Friday 14 October 2011 16:18 To: WAROQUIERS Philippe; val...@li... Subject: RE: [Valgrind-users] --free-list-vol limit Thanks. Sounds like a nice way to allow a bigger free list volume and it would have helped in my case. I'll just wait for the 3.7 release since I already fixed this bug. But I would still love to see Valgrind get rid of the 32G limit. Thanks, Meir ________________________________ From: WAROQUIERS Philippe [mailto:phi...@eu...] Sent: Friday, October 14, 2011 4:00 PM To: Yeshurun, Meir; val...@li... Subject: RE: [Valgrind-users] --free-list-vol limit I think it is not possible (or at least without significant work) to overcome the 32Gb limit. An alternative solution might be: if you have a mixture of big allocations and small allocations in your application, and your double free is of a "small" size, then you could take a 3.7.0 from SVN (see http://www.valgrind.org/downloads/repository.html ) and apply the patch in bugzilla http://bugs.kde.org/show_bug.cgi?id=250065 With this patch, you have a new option --freelist-big-blocks=.... which allows to specify the size above which the blocks are "released" from the free list in priority. With this, you might "prune" parts of the big alloc/free from the free list, allowing you to find the double free of your "small" block. Philippe ________________________________ From: Yeshurun, Meir [mailto:mei...@in...] Sent: Friday 14 October 2011 15:31 To: val...@li... Subject: [Valgrind-users] --free-list-vol limit Hi, I recently had to debug a double free. This could have been a piece of cake with Valgrind, but unfortunately the first and second frees were a few minutes of runtime apart, so all I got was the "address was not recently malloc()ed..." message instead of the backtrace corresponding to the first free. When I increased the -free-list-vol to the maximum value allowed by Valgrind, Valgrind aborted prematurely, apparently because the 32G memory limit was reached. My machine had 96G, and I'm sure it would have been enough for obtaining the backtrace of the first free. Is it possible to overcome the 32G limit? I don't think machines with more memory than that are rare. Thanks, Meir --------------------------------------------------------------------- Intel Israel (74) Limited This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. ____ This message and any files transmitted with it are legally privileged and intended for the sole use of the individual(s) or entity to whom they are addressed. If you are not the intended recipient, please notify the sender by reply and delete the message and any attachments from your system. Any unauthorised use or disclosure of the content of this message is strictly prohibited and may be unlawful. Nothing in this e-mail message amounts to a contractual or legal commitment on the part of EUROCONTROL, unless it is confirmed by appropriately signed hard copy. Any views expressed in this message are those of the sender. --------------------------------------------------------------------- Intel Israel (74) Limited This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. ____ This message and any files transmitted with it are legally privileged and intended for the sole use of the individual(s) or entity to whom they are addressed. If you are not the intended recipient, please notify the sender by reply and delete the message and any attachments from your system. Any unauthorised use or disclosure of the content of this message is strictly prohibited and may be unlawful. Nothing in this e-mail message amounts to a contractual or legal commitment on the part of EUROCONTROL, unless it is confirmed by appropriately signed hard copy. Any views expressed in this message are those of the sender. |
|
From: John R. <jr...@bi...> - 2011-10-14 14:52:15
|
> Is it possible to overcome the 32G limit? I don't think machines with more memory than that are rare. Such machines ARE rare. They are fewer than 0.4% (1:250) of the machines that have _ever_ run memcheck, and that number is already very small. -- |
|
From: Yeshurun, M. <mei...@in...> - 2011-10-14 15:01:31
|
1. How do you know that? 2. They are going to become increasingly common -----Original Message----- From: John Reiser [mailto:jr...@bi...] Sent: Friday, October 14, 2011 4:52 PM To: val...@li... Subject: Re: [Valgrind-users] --free-list-vol limit > Is it possible to overcome the 32G limit? I don't think machines with more memory than that are rare. Such machines ARE rare. They are fewer than 0.4% (1:250) of the machines that have _ever_ run memcheck, and that number is already very small. -- ------------------------------------------------------------------------------ All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2d-oct _______________________________________________ Valgrind-users mailing list Val...@li... https://lists.sourceforge.net/lists/listinfo/valgrind-users --------------------------------------------------------------------- Intel Israel (74) Limited This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. |
|
From: John R. <jr...@bi...> - 2011-10-14 15:55:01
|
On 10/14/2011 08:01 AM, Yeshurun, Meir wrote:
> 1. How do you know that?
By the statistics on downloads of valgrind (as adjusted for typical
multiple-installs-from-one-download), by the statistics on
median {,S}DRAM in x86* machines of any kind (still < 4GB),
by the limits on memory controller drive and DRAM die density
(16GB in a common box < $1000 or so), and by traveling around.
> 2. They are going to become increasingly common
In absolute numbers, yes. But in the next five years
the percentage still will be less than 1%.
(Even discounting the fact that "smartphones" are
full-fledged computers in every way, already vastly
outnumber x86* machines of all kinds, and will take
a while to exceed even 1GB DRAM.)
--
|
|
From: Brian B. <bri...@gm...> - 2011-10-14 16:06:07
|
My personal take is that this is not an excuse to resist adjusting the
32 GB limit. Besides the fact that the actual usage of valgrind (vs
downloads) is likely going to be more skewed toward developer machines
and environments, it should make sense to make valgrind as robust to
memory usage as possible.
On the other hand, I can completely understand the resistance if the
feature is hard to implement, and weighing the difficulty vs common
usage cases. Anyone care to explain the fundamental limitation to 32
GB? What kind of work would be required to increase that by some
number of binary orders of magnitude?
Thanks,
Brian
On Fri, Oct 14, 2011 at 8:54 AM, John Reiser <jr...@bi...> wrote:
> On 10/14/2011 08:01 AM, Yeshurun, Meir wrote:
>> 1. How do you know that?
>
> By the statistics on downloads of valgrind (as adjusted for typical
> multiple-installs-from-one-download), by the statistics on
> median {,S}DRAM in x86* machines of any kind (still < 4GB),
> by the limits on memory controller drive and DRAM die density
> (16GB in a common box < $1000 or so), and by traveling around.
>
>> 2. They are going to become increasingly common
>
> In absolute numbers, yes. But in the next five years
> the percentage still will be less than 1%.
> (Even discounting the fact that "smartphones" are
> full-fledged computers in every way, already vastly
> outnumber x86* machines of all kinds, and will take
> a while to exceed even 1GB DRAM.)
>
> --
>
> ------------------------------------------------------------------------------
> All the data continuously generated in your IT infrastructure contains a
> definitive record of customers, application performance, security
> threats, fraudulent activity and more. Splunk takes this data and makes
> sense of it. Business sense. IT sense. Common sense.
> http://p.sf.net/sfu/splunk-d2d-oct
> _______________________________________________
> Valgrind-users mailing list
> Val...@li...
> https://lists.sourceforge.net/lists/listinfo/valgrind-users
>
|
|
From: WAROQUIERS P. <phi...@eu...> - 2011-10-14 17:59:03
|
In fact, it is trivial to go over 32GB (including for the freelist
volume).
Here are the changes:
In memcheck/mc_main.c, insert '10*' to obtain:
else if VG_BINT_CLO(arg, "--freelist-vol", MC_(clo_freelist_vol),
0,
10*10*1000*1000*1000LL) {}
in coregrind/m_aspacemgr/aspacemgr-linux.c, insert a '0' to obtain:
#define VG_N_SEGMENTS 50000
And change the below line to obtain what you want for max address space
(e.g. 64G):
1652c1652
< aspacem_maxAddr = (Addr)0x800000000 - 1; // 32G
---
> aspacem_maxAddr = (Addr)((2 * 0x800000000) - 1); // 64G
Note that I understood memcheck will become (a lot?) slower when going
over 32GB.
Philippe
____
This message and any files transmitted with it are legally privileged and intended for the sole use of the individual(s) or entity to whom they are addressed. If you are not the intended recipient, please notify the sender by reply and delete the message and any attachments from your system. Any unauthorised use or disclosure of the content of this message is strictly prohibited and may be unlawful.
Nothing in this e-mail message amounts to a contractual or legal commitment on the part of EUROCONTROL, unless it is confirmed by appropriately signed hard copy.
Any views expressed in this message are those of the sender.
|