|
From: Konstantin S. <kon...@gm...> - 2009-10-13 12:35:14
|
Hi Valgrind devs, I need to know how much memory is taken by a valgrind process from inside that process. And I need it to be fast. One option is to read /proc/self/status and extract the VmSize field. Slow, bad. Another option is to keep track of valgrind's VG(malloc) and users's malloc, etc. Easy to loose something, bad. Any other suggestion? Thanks, --kcc |
|
From: John R. <jr...@bi...> - 2009-10-13 13:55:02
|
> I need to know how much memory is taken by a valgrind process from > inside that process. And I need it to be fast. How fast? > One option is to read /proc/self/status and extract the VmSize field. > Slow, bad. Doing a pread(,,,0) and parsing the result via strstr+atoi should take about 1 microsecond. Open /proc/self/status once at the beginning, and keep it open for the zillions of calls you'll be making. Or does pread fail for files in /proc? > Another option is to keep track of valgrind's VG(malloc) and users's > malloc, etc. Easy to loose something, bad. If you do not trust valgrind internal code then why run it at all? -- |
|
From: Konstantin S. <kon...@gm...> - 2009-10-13 19:18:32
|
On Tue, Oct 13, 2009 at 5:54 PM, John Reiser <jr...@bi...> wrote: > > > I need to know how much memory is taken by a valgrind process from > > inside that process. And I need it to be fast. > > How fast? > > > One option is to read /proc/self/status and extract the VmSize field. > > Slow, bad. > > Doing a pread(,,,0) and parsing the result via strstr+atoi should take > about 1 microsecond. Hm. Perhaps. I did not think about opening this file for the duration of the run... :) I need to check the occupied memory very often so that the tool can react accordingly when the memory is close to its limit. pread+strstr+atoi might be fast enough. Need to check... > > Open /proc/self/status once at the beginning, > and keep it open for the zillions of calls you'll be making. > Or does pread fail for files in /proc? > > > Another option is to keep track of valgrind's VG(malloc) and users's > > malloc, etc. Easy to loose something, bad. > > If you do not trust valgrind internal code then why run it at all? Is there such functionality in valgrind core already? I need to get the number very close to the one in /proc/self/status:VmSize Thanks! --kcc > > -- > > ------------------------------------------------------------------------------ > Come build with us! The BlackBerry(R) Developer Conference in SF, CA > is the only developer event you need to attend this year. Jumpstart your > developing skills, take BlackBerry mobile applications to market and stay > ahead of the curve. Join us from November 9 - 12, 2009. Register now! > http://p.sf.net/sfu/devconference > _______________________________________________ > Valgrind-developers mailing list > Val...@li... > https://lists.sourceforge.net/lists/listinfo/valgrind-developers |
|
From: Konstantin S. <kon...@gm...> - 2009-10-14 11:30:09
|
On Tue, Oct 13, 2009 at 11:17 PM, Konstantin Serebryany < kon...@gm...> wrote: > On Tue, Oct 13, 2009 at 5:54 PM, John Reiser <jr...@bi...> wrote: > > > > > I need to know how much memory is taken by a valgrind process from > > > inside that process. And I need it to be fast. > > > > How fast? > > > > > One option is to read /proc/self/status and extract the VmSize field. > > > Slow, bad. > > > > Doing a pread(,,,0) and parsing the result via strstr+atoi should take > > about 1 microsecond. > > Hm. Perhaps. > I did not think about opening this file for the duration of the run... :) > I need to check the occupied memory very often so that the tool can > react accordingly when the memory is close to its limit. > pread+strstr+atoi might be fast enough. Need to check... > Valgrind does not have pread(). lseek+read+strstr+strtol is a bit too slow for me, so I have to call it less frequently than I would like to. Any idea how to get VmSize inside valgrind w/o reading /proc/self/status? --kcc > > > > > Open /proc/self/status once at the beginning, > > and keep it open for the zillions of calls you'll be making. > > Or does pread fail for files in /proc? > > > > > Another option is to keep track of valgrind's VG(malloc) and users's > > > malloc, etc. Easy to loose something, bad. > > > > > > If you do not trust valgrind internal code then why run it at all? > > Is there such functionality in valgrind core already? > I need to get the number very close to the one in /proc/self/status:VmSize > > > Thanks! > --kcc > > > > -- > > > > > ------------------------------------------------------------------------------ > > Come build with us! The BlackBerry(R) Developer Conference in SF, CA > > is the only developer event you need to attend this year. Jumpstart your > > developing skills, take BlackBerry mobile applications to market and stay > > ahead of the curve. Join us from November 9 - 12, 2009. Register now! > > http://p.sf.net/sfu/devconference > > _______________________________________________ > > Valgrind-developers mailing list > > Val...@li... > > https://lists.sourceforge.net/lists/listinfo/valgrind-developers > |
|
From: Julian S. <js...@ac...> - 2009-10-14 11:52:44
|
On Wednesday 14 October 2009, Konstantin Serebryany wrote:
> > I did not think about opening this file for the duration of the run... :)
> > I need to check the occupied memory very often so that the tool can
^^^^^^^^^^
Actually, you only need to check it when the process' mappings change,
or when sbrk happens, since that's the only way(s) the process can acquire
more memory. No need for spin-style polling.
So .. get your tool to monitor the "new_mem_mmap" and "die_mem_mmap"
core-tool events. When you get notified of one of those, then check
the memory situation. See Memcheck sources for examples how (it's easy).
One danger, though; the following race leading to a segfault:
tool is notified of new_mem_mmap
tool decides to reread /proc/self/status
so it does VG_(malloc) to allocate a buffer
which results in m_mallocfree doing mmap
which results in a new_mem_mmap notification
We've had problems like this in the past. (I think the above scenario
is impossible, but nevertheless ..) I would strongly suggest that all
the code you have inside your new_mem_mmap does not do any dynamic
mem allocation. Best to read the file, check if need to reduce mem
consumption, if so, set a flag, and exit. Then, elsewhere in your
tool, check the flag, and if set, dump memory, or whatever.
J
|
|
From: Konstantin S. <kon...@gm...> - 2009-10-14 12:44:13
|
On Wed, Oct 14, 2009 at 4:05 PM, Julian Seward <js...@ac...> wrote: > On Wednesday 14 October 2009, Konstantin Serebryany wrote: > > > I did not think about opening this file for the duration of the run... > :) > > > I need to check the occupied memory very often so that the tool can > ^^^^^^^^^^ > > Actually, you only need to check it when the process' mappings change, > or when sbrk happens, since that's the only way(s) the process can acquire > more memory. No need for spin-style polling. > > So .. get your tool to monitor the "new_mem_mmap" and "die_mem_mmap" > This will notify me when the client program does mmap/brk. Right? But I also need to watch for mmap/brk of valgrind itself. Is it possible? --kcc > core-tool events. When you get notified of one of those, then check > the memory situation. See Memcheck sources for examples how (it's easy). > > One danger, though; the following race leading to a segfault: > > tool is notified of new_mem_mmap > tool decides to reread /proc/self/status > so it does VG_(malloc) to allocate a buffer > which results in m_mallocfree doing mmap > which results in a new_mem_mmap notification > > We've had problems like this in the past. (I think the above scenario > is impossible, but nevertheless ..) I would strongly suggest that all > the code you have inside your new_mem_mmap does not do any dynamic > mem allocation. Best to read the file, check if need to reduce mem > consumption, if so, set a flag, and exit. Then, elsewhere in your > tool, check the flag, and if set, dump memory, or whatever. > > J > |
|
From: Julian S. <js...@ac...> - 2009-10-20 21:42:30
|
On Wednesday 14 October 2009, Konstantin Serebryany wrote:
> On Wed, Oct 14, 2009 at 4:05 PM, Julian Seward <js...@ac...> wrote:
> > On Wednesday 14 October 2009, Konstantin Serebryany wrote:
> > > > I did not think about opening this file for the duration of the
> > > > run...
> > :
> > :)
> > :
> > > > I need to check the occupied memory very often so that the tool can
> >
> > ^^^^^^^^^^
> >
> > Actually, you only need to check it when the process' mappings change,
> > or when sbrk happens, since that's the only way(s) the process can
> > acquire more memory. No need for spin-style polling.
> >
> > So .. get your tool to monitor the "new_mem_mmap" and "die_mem_mmap"
>
> This will notify me when the client program does mmap/brk. Right?
Yes.
> But I also need to watch for mmap/brk of valgrind itself. Is it possible?
Hmm, good point. I didn't think of that. Not sure that's easily possible.
How about this Plan B: Get the scheduler to notify you of start/stop
of running of client code (probably you already do):
void VG_(track_start_client_code)(
void(*f)(ThreadId tid, ULong blocks_dispatched)
);
void VG_(track_stop_client_code)(
void(*f)(ThreadId tid, ULong blocks_dispatched)
);
and look at 'blocks_dispatched'. Whenever that changes by (eg)
more than one million, re-read /proc/self/whatever etc.
J
|
|
From: Konstantin S. <kon...@gm...> - 2009-10-26 09:20:31
|
On Wed, Oct 21, 2009 at 1:54 AM, Julian Seward <js...@ac...> wrote: > On Wednesday 14 October 2009, Konstantin Serebryany wrote: > > On Wed, Oct 14, 2009 at 4:05 PM, Julian Seward <js...@ac...> wrote: > > > On Wednesday 14 October 2009, Konstantin Serebryany wrote: > > > > > I did not think about opening this file for the duration of the > > > > > run... > > > : > > > :) > > > : > > > > > I need to check the occupied memory very often so that the tool can > > > > > > ^^^^^^^^^^ > > > > > > Actually, you only need to check it when the process' mappings change, > > > or when sbrk happens, since that's the only way(s) the process can > > > acquire more memory. No need for spin-style polling. > > > > > > So .. get your tool to monitor the "new_mem_mmap" and "die_mem_mmap" > > > > This will notify me when the client program does mmap/brk. Right? > > Yes. > > > But I also need to watch for mmap/brk of valgrind itself. Is it possible? > > Hmm, good point. I didn't think of that. Not sure that's easily possible. > How about this Plan B: Get the scheduler to notify you of start/stop > of running of client code (probably you already do): > > void VG_(track_start_client_code)( > void(*f)(ThreadId tid, ULong blocks_dispatched) > ); > void VG_(track_stop_client_code)( > void(*f)(ThreadId tid, ULong blocks_dispatched) > ); > > and look at 'blocks_dispatched'. Whenever that changes by (eg) > more than one million, re-read /proc/self/whatever etc. > Maybe... But that's not any different from what I do know (read /proc/self every N-th event of some kind). --kcc > > J > |