|
From: Rob T. <rta...@gm...> - 2014-05-13 22:50:02
|
I recently discovered valgrind while searching for a way to see how much
memory (total memory footprint) my small program represented.
I was perplexed by the results I was seeing, since I expected at most to
use perhaps tens of thousands of bites not Millions. My application is not
dynamically requesting/allocating memory, all memory consumed is in the
form of statically assigned character arrays, automatic stack usage etc..
valgrind --version
valgrind-3.9.0
I'm running on LFS 7.4
linux-3.11
glibc-2.18
gcc-4.8.1
So I did a test by running valgrind against an empty C program. The program
does nothing but return. Can someone explain why the output shows
continually increasing memory used and in total over a 6 MB peak?
user [ ~/ccode/string_buffer ]> cat > ./just_return.c << "EOF"
> int main(void)
> {
> return 0;
> }
> EOF
user [ ~/ccode/string_buffer ]> gcc -O0 -Wall -g -o just_return
just_return.c
user [ ~/ccode/string_buffer ]> valgrind --tool=massif --pages-as-heap=yes
--time-unit=B ./just_return
==11740== Massif, a heap profiler
==11740== Copyright (C) 2003-2013, and GNU GPL'd, by Nicholas Nethercote
==11740== Using Valgrind-3.9.0 and LibVEX; rerun with -h for copyright info
==11740== Command: ./just_return
==11740==
==11740==
user [ ~/ccode/string_buffer ]> ms_print massif.out.11740
--------------------------------------------------------------------------------
Command: ./just_return
Massif arguments: --pages-as-heap=yes --time-unit=B
ms_print arguments: massif.out.11740
--------------------------------------------------------------------------------
MB
5.902^
:
| #
| #
| #
| #
| #
| #
| #
| #
| #
| #
| #
| #
|
:::::::::::::::::::::::::::::::::::::::::::::#
|
: #
|
: #
|
: #
|
: #
|
: #
|
: #
0
+----------------------------------------------------------------------->MB
0
5.918
Number of snapshots: 21
Detailed snapshots: [9, 19 (peak)]
--------------------------------------------------------------------------------
n time(B) total(B) useful-heap(B) extra-heap(B)
stacks(B)
--------------------------------------------------------------------------------
0 4,096 4,096 4,096
0 0
1 8,192 8,192 8,192
0 0
2 139,264 139,264 139,264
0 0
3 147,456 147,456 147,456
0 0
4 151,552 151,552 151,552
0 0
5 155,648 155,648 155,648
0 0
6 159,744 159,744 159,744
0 0
7 167,936 167,936 167,936
0 0
8 172,032 172,032 172,032
0 0
9 172,032 172,032 172,032
0 0
100.00% (172,032B) (page allocation syscalls) mmap/mremap/brk, --alloc-fns,
etc.
->100.00% (172,032B) 0xFFFFFFFFFFFFFFFF: ???
--------------------------------------------------------------------------------
n time(B) total(B) useful-heap(B) extra-heap(B)
stacks(B)
--------------------------------------------------------------------------------
10 176,128 176,128 176,128
0 0
11 2,277,376 2,277,376 2,277,376
0 0
12 2,281,472 2,281,472 2,281,472
0 0
13 2,297,856 2,297,856 2,297,856
0 0
14 6,139,904 6,139,904 6,139,904
0 0
15 6,164,480 6,164,480 6,164,480
0 0
16 6,180,864 6,180,864 6,180,864
0 0
17 6,184,960 6,184,960 6,184,960
0 0
18 6,189,056 6,189,056 6,189,056
0 0
19 6,201,344 6,176,768 6,176,768
0 0
100.00% (6,176,768B) (page allocation syscalls) mmap/mremap/brk,
--alloc-fns, etc.
->97.21% (6,004,736B) 0x4017119: mmap (in /lib/ld-2.18.so)
| ->96.22% (5,943,296B) 0x40064F0: _dl_map_object_from_fd (in /lib/
ld-2.18.so)
| | ->96.22% (5,943,296B) 0x4007F93: _dl_map_object (in /lib/ld-2.18.so)
| | ->62.20% (3,842,048B) 0x400C230: openaux (in /lib/ld-2.18.so)
| | | ->62.20% (3,842,048B) 0x400E682: _dl_catch_error (in /lib/ld-2.18.so
)
| | | ->62.20% (3,842,048B) 0x400C483: _dl_map_object_deps (in /lib/
ld-2.18.so)
| | | ->62.20% (3,842,048B) 0x4002D52: dl_main (in /lib/ld-2.18.so)
| | | ->62.20% (3,842,048B) 0x40150B3: _dl_sysdep_start (in /lib/
ld-2.18.so)
| | | ->62.20% (3,842,048B) 0x40049A4: _dl_start (in /lib/
ld-2.18.so)
| | | ->62.20% (3,842,048B) 0x4001136: ??? (in /lib/ld-2.18.so)
| | |
| | ->34.02% (2,101,248B) 0x400131C: map_doit (in /lib/ld-2.18.so)
| | ->34.02% (2,101,248B) 0x400E682: _dl_catch_error (in /lib/ld-2.18.so
)
| | ->34.02% (2,101,248B) 0x4000F78: do_preload (in /lib/ld-2.18.so)
| | ->34.02% (2,101,248B) 0x40040C6: dl_main (in /lib/ld-2.18.so)
| | ->34.02% (2,101,248B) 0x40150B3: _dl_sysdep_start (in /lib/
ld-2.18.so)
| | ->34.02% (2,101,248B) 0x40049A4: _dl_start (in /lib/
ld-2.18.so)
| | ->34.02% (2,101,248B) 0x4001136: ??? (in /lib/ld-2.18.so)
| |
| ->00.99% (61,440B) in 1+ places, all below ms_print's threshold (01.00%)
|
->02.79% (172,032B) 0xFFFFFFFFFFFFFFFF: ???
--------------------------------------------------------------------------------
n time(B) total(B) useful-heap(B) extra-heap(B)
stacks(B)
--------------------------------------------------------------------------------
20 6,205,440 6,172,672 6,172,672
0 0
Thanks,
Rob Taylor
|
|
From: Eliot M. <mo...@cs...> - 2014-05-13 23:30:19
|
On 5/13/2014 6:49 PM, Rob Taylor wrote: > I recently discovered valgrind while searching for a way to see how much memory (total memory > footprint) my small program represented. > > I was perplexed by the results I was seeing, since I expected at most to use perhaps tens of > thousands of bites not Millions. My application is not dynamically requesting/allocating memory, all > memory consumed is in the form of statically assigned character arrays, automatic stack usage etc.. > So I did a test by running valgrind against an empty C program. The program does nothing but return. > Can someone explain why the output shows continually increasing memory used and in total over a 6 MB > peak? Dynamically linked libraries and their data. Some of that (most of the code areas) can be shared in real memory with other programs using the same library. There's a lot of mechanism there, even if you are not using much of it, and even if you are using parts of it without realizing it ... Regards -- Eliot Moss |
|
From: Milian W. <ma...@mi...> - 2014-05-14 10:03:22
|
On Tuesday 13 May 2014 19:13:23 Eliot Moss wrote: > On 5/13/2014 6:49 PM, Rob Taylor wrote: > > I recently discovered valgrind while searching for a way to see how much > > memory (total memory footprint) my small program represented. > > > > I was perplexed by the results I was seeing, since I expected at most to > > use perhaps tens of thousands of bites not Millions. My application is > > not dynamically requesting/allocating memory, all memory consumed is in > > the form of statically assigned character arrays, automatic stack usage > > etc.. > > > > So I did a test by running valgrind against an empty C program. The > > program does nothing but return. Can someone explain why the output shows > > continually increasing memory used and in total over a 6 MB peak? > > Dynamically linked libraries and their data. Some of that > (most of the code areas) can be shared in real memory with > other programs using the same library. There's a lot of > mechanism there, even if you are not using much of it, and > even if you are using parts of it without realizing it ... Yep, you are investigating on the page level, which is different from what you allocate. Remove that argument, but try --stacks=yes instead. Bye -- Milian Wolff ma...@mi... http://milianw.de |
|
From: Rob T. <rta...@gm...> - 2014-05-14 18:22:04
|
Thank you Milian Wolff and Eliot Moss, When attempting to profile the total memory usage of my small application, which I called tail according to the Excercise 5-13 of the C Programming Language (second edition) by Brian W. Kernighan and Denis M. Richie, if I try to run valgrind like this: cat ~/my_lfs_7_4_notes_for_1.6.2.txt | valgrind --tool=massif --stacks=yes --time-unit=B ./tail -n1000 It looks like the program uses a max of 3,360 bytes of memory (all stack usage). But in 2 of my source files I have allocations like this: #define MAXLINES 1000L /* max lines to be stored */ static char *lineptr[MAXLINES]; /* pointers to text lines */ #define BUFSIZE 33000UL /* size of available buffer space */ static char buf[BUFSIZE]; /* array of BUFSIZE to store lines */ #define MAXLENGTH 1000 /* set the maximum line length stored */ char s[MAXLENGTH]; /* storage for get_line function */ >From this I estimate memory usage should be in excess of 42,000 bytes. The closest that I can seem to get to a real representation of memory used is if I subtract the amount reported for an empty program from the amount reported for this program when I use the option --pages-as-heap=yes as follows: cat ~/my_lfs_7_4_notes_for_1.6.2.txt | valgrind --tool=massif --pages-as-heap=yes --time-unit=B ./tail -n1000 valgrind --tool=massif --pages-as-heap=yes --time-unit=B ./just_return Then calculate the difference in memory usage reported for the 2 runs... 6,234,112 - 6,189,056 = 45,056 This seems more like the impact that my code has had on memory usage and represents the part I can change or optimize, since I have no control over whatever created the other 6 million bytes of usage even with an entirely empty c program (or whether the libraries are shared or not). Is this a fair assessment? I'm guessing the C code stored on disk is also in memory somewhere when the program is loaded. Though I have difficulty seeing that in these memory outputs and can only guess how the size of my compiled code affects memory usage. Thank you for the feedback. Books and documentation just never seems to be enough to learn this stuff. Rob Taylor On Wed, May 14, 2014 at 2:43 AM, Milian Wolff <ma...@mi...> wrote: > On Tuesday 13 May 2014 19:13:23 Eliot Moss wrote: > > On 5/13/2014 6:49 PM, Rob Taylor wrote: > > > I recently discovered valgrind while searching for a way to see how > much > > > memory (total memory footprint) my small program represented. > > > > > > I was perplexed by the results I was seeing, since I expected at most > to > > > use perhaps tens of thousands of bites not Millions. My application is > > > not dynamically requesting/allocating memory, all memory consumed is in > > > the form of statically assigned character arrays, automatic stack usage > > > etc.. > > > > > > So I did a test by running valgrind against an empty C program. The > > > program does nothing but return. Can someone explain why the output > shows > > > continually increasing memory used and in total over a 6 MB peak? > > > > Dynamically linked libraries and their data. Some of that > > (most of the code areas) can be shared in real memory with > > other programs using the same library. There's a lot of > > mechanism there, even if you are not using much of it, and > > even if you are using parts of it without realizing it ... > > Yep, you are investigating on the page level, which is different from what > you > allocate. Remove that argument, but try --stacks=yes instead. > > Bye > > -- > Milian Wolff > ma...@mi... > http://milianw.de > > > ------------------------------------------------------------------------------ > "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE > Instantly run your Selenium tests across 300+ browser/OS combos. > Get unparalleled scalability from the best Selenium testing platform > available > Simple to use. Nothing to install. Get started now for free." > http://p.sf.net/sfu/SauceLabs > _______________________________________________ > Valgrind-users mailing list > Val...@li... > https://lists.sourceforge.net/lists/listinfo/valgrind-users > |
|
From: Eliot M. <mo...@cs...> - 2014-05-14 21:46:46
|
On 5/14/2014 2:21 PM, Rob Taylor wrote: > Thank you Milian Wolff and Eliot Moss, Typical users of tools like valgrind have more background in how all this works -- not a criticism of you, just a difference in expectation. The volume of code that results from compiling a program consisting of a main function that simply returns is quite small. It is stored on disk in the executable file produced by the C compiling and linking process. That file also describes any global / static storage needed for the program. When a process is created to run the program, the operating system sets up an area in low memory (not the stack) for that data. Local / automatic (in C terminology) variables are allocated in the stack, and done so by each function as you enter the function. The operating system grows the stack (from high addresses toward lower ones) automatically. It is also possible to allocate, and "free", memory on request using library functions such as malloc and free. Any libraries used will be linked in (connected to your code) on demand, and they may also have static data areas and may malloc/free data as well. You may benefit from finding documentation or a textbook about how compiling, linking, loading, etc., works for C programs in Unix/Linux, etc. (The Windows world is similar, though of course different in some details.) Regards -- Eliot Moss |
|
From: Alan M. <ala...@jp...> - 2014-05-14 22:20:51
|
I'm not the original questioner, but this brings up an interesting question: what books are out there? I'm familiar with all of this already, but I know of no single books that cover the language, along with compilation, along with the OS aspects. If there is one out there, I'd love to know. -- Alan On 5/14/2014 2:46 PM, Eliot Moss wrote: > You may benefit from finding documentation or a textbook about how > compiling, linking, loading, etc., works for C programs in Unix/Linux, > etc. (The Windows world is similar, though of course different in some > details.) |