From: Grahame J. <gb...@th...> - 2007-03-14 01:09:08
|
Hi, I have a problem with running out of memory on my gumstix. Probably a memory leak. I am guessing that the memory is being used by processes and by what is being used in /tmp. However with "free" I am getting about double the usage than I can measure. Is there something else that I am not capturing here: Processes = 8344 Temp dir = 5660 Total = 14004 # free total used free shared buffers Mem: 63436 38216 25220 0 0 wap: 0 0 0 Total: 63436 38216 25220 Thanks Grahame Jordan |
From: Dave H. <dhy...@gm...> - 2007-03-14 01:32:48
|
Hi Grahame, > I have a problem with running out of memory on my gumstix. Probably a > memory leak. > I am guessing that the memory is being used by processes and by what is > being used in /tmp. > However with "free" I am getting about double the usage than I can measure. > Is there something else that I am not capturing here: > > Processes = 8344 > Temp dir = 5660 > Total = 14004 > > # free > total used free shared > buffers > Mem: 63436 38216 25220 0 0 > wap: 0 0 0 > Total: 63436 38216 25220 I think that df will show you how much RAM is being used. The kernel also uses what it can as cache. The cache portion will get released if memory is low. cat /proc/meminfo will show you how much is being used in the cache. -- Dave Hylands Vancouver, BC, Canada http://www.DaveHylands.com/ |
From: Alexandre P. N. <al...@om...> - 2007-03-14 01:41:52
|
Dave Hylands wrote: > [cut] > > I think that df will show you how much RAM is being used. > > df -h /tmp shows 0 bytes, even tough there are 600k in use... but du -h /tmp seems to get close enough... > The kernel also uses what it can as cache. The cache portion will get > released if memory is low. > > cat /proc/meminfo > > will show you how much is being used in the cache. > > Good point, the "normal" free comand shows the cache/buffer statistics, but the simplified busybox one doesn't. There's also an "inactive" memory pool (grep ^Inactive /proc/meminfo), IIRC that's not accounted as cache but is memory that can be freed almost immediately, if needed. The damn thing could be a bit simpler. - Alexandre |
From: Dave H. <dhy...@gm...> - 2007-03-15 00:15:46
|
Hi Grahame, > PSVM /TMP TOTAL FREE DIFF(PS+TMP-FREE) UNACCOUNTED > > 7752 4352 12104 14952 28487752 4352 12104 14952 2848 > 7752 4388 12140 15000 2860 Up to here, everything seems to be fine. The total mem is essentially constant. But then the total mem takes a big jump (by a meg). The ps memory takes a corresponding bump. So some new processes must be running when it changes from 7776 to 8724. The cache will normally increase to fill all available memory, and then gets kicked out when memory is needed. > 7776 4428 12204 23792 11588 > 8724 4396 13120 24500 11380 > 8724 4428 13152 24536 11384 > 8724 4396 13120 24908 11788 > 8724 4428 13152 24932 11780 > 8724 4400 13124 25304 12180 > 9312 4436 13748 25508 11760 -- Dave Hylands Vancouver, BC, Canada http://www.DaveHylands.com/ |
From: Alexandre P. N. <al...@om...> - 2007-03-15 01:13:48
|
Dave Hylands escreveu: > [cut] > Up to here, everything seems to be fine. The total mem is essentially > constant. But then the total mem takes a big jump (by a meg). The ps > memory takes a corresponding bump. So some new processes must be > running when it changes from 7776 to 8724. > > The cache will normally increase to fill all available memory, and > then gets kicked out when memory is needed. > I saw a pxaregs being killed somewhere on his listing; I remember seeing something about a patch regarding munmap() on pxaregs today, perhaps it was related to this issue too. I'm just taking a fast glimpse on the list today, pardon me in the case I'm confusing things... - Alexandre |
From: Jeff S. <lea...@fr...> - 2007-03-15 04:20:28
|
On Tue, 13 Mar 2007, Alexandre Pereira Nunes wrote: Some things to be aware of... > df -h /tmp shows 0 bytes, even tough there are 600k in use... but du -h > /tmp seems to get close enough... df is giving the information stored in the file system, du is giving the information deduced from traversing the file system and collecting the metadata. How is this different? df looks at the freelist and reports the number of free blocks in the filesystem. du, when traversing the filesystem, counts the blocks in use but doesn't count things like directory entries, special files (named pipes) or device files. When a process forks, Linux allocates memory using copy-on-write, meaning a child process may report as having X megs of memory allocated to it but in reality it may only have Y megs of memory because the entire process space of the parent has not been duplicated. Finally, if you have dynamic loading of shared libraries, then you've got another ball of wax. Unfortunately, busybox doesn't come with pmap (only way to get it is to download procps from sourceforge and compile yourself.) pmap shows you all kinds of wonderful info about a process's memory space, especially shared libraries. Shared libraries usually contain to parts a code segment and a data segment. Code segments can be shared among processes while data segments are process-specific. With pmap, you can see how much memory is shared and how much is private to the process. A good web site on Linux memory (note: written about x86 hardware, which smoe folks in the discussion point out as an issue when in the embedded arena) : http://virtualthreads.blogspot.com/2006/02/understanding-memory-usage-on-linux.html --Jeff "I am not available for comment" |
From: Alexandre P. N. <al...@om...> - 2007-03-15 15:01:25
|
Jeff Stoner escreveu: > On Tue, 13 Mar 2007, Alexandre Pereira Nunes wrote: > > Some things to be aware of... > > >> df -h /tmp shows 0 bytes, even tough there are 600k in use... but du -h >> /tmp seems to get close enough... >> > > df is giving the information stored in the file system, du is giving the > information deduced from traversing the file system and collecting the > metadata. How is this different? df on /tmp asks tmpfs (in fact, I think that since gumstix lacks support for swap, ramfs is used instead), and it's implementation is incomplete is not collecting the necessary data. > df looks at the freelist and reports the > number of free blocks in the filesystem. du, when traversing the > filesystem, counts the blocks in use but doesn't count things like > directory entries, special files (named pipes) or device files. > .. which takes memory from the tmpfs, but aren't correctly computed, which is irrelevant for that specific measurement I was refering too, at least on my case, since I only had one subdirectory and no special nodes. The user who started the thread may have to take that in mind, tough... - Alexandre |
From: Alexandre P. N. <al...@om...> - 2007-03-14 01:34:45
|
Grahame Jordan wrote: > Hi, > > I have a problem with running out of memory on my gumstix. Probably a > memory leak. > I am guessing that the memory is being used by processes and by what is > being used in /tmp. > However with "free" I am getting about double the usage than I can measure. > Is there something else that I am not capturing here: > > Processes = 8344 > Temp dir = 5660 > Total = 14004 > > # free > total used free shared > buffers > Mem: 63436 38216 25220 0 0 > wap: 0 0 0 > Total: 63436 38216 25220 > > > Thanks > > Grahame Jordan > Hi! If you're measuring free memory by the "free" column above, from the "free" command output, then let me say that this column is not what you want; Getting the real amount of free memory is somewhat tricky. The free column above only reflects the amount of memory not used by anything, let's call it "hard free". The thing is that the kernel uses memory for things like caching, for instance, when you run an executable, the first time it's loaded from disk, but then it's cached on memory, so that if you call it again, it does not need to be loaded from disk. This cache memory (there are other aspects of the memory system that are cacheable) is computed as used, but it's allocation has very low priority, should some application (or the kernel itself) needs memory for something else, it's freed on demand. Also, measuring memory usage by processes is quite tricky: the process may request memory which is granted but not allocated until when (if) it gets used; Some segments of memory which are in use maybe freed in preference of newer requests, mmap regions may refer to memory external to the bus, etc. But for most processes, the resident size showed by ps is close enough. - Alexandre |
From: Grahame J. <gb...@th...> - 2007-03-14 21:57:30
|
Hi, I am not sure what is going on here. When my program runs for a couple of days it crashes due to memory issues. eg. user.warn kernel: oom-killer: gfp_mask=0x200d2, order=0 user.warn kernel: [<c001bc60>] (dump_stack+0x0/0x14) from [<c0046e64>] (out_of_memory+0x40/0x19c) user.warn kernel: [<c0046e24>] (out_of_memory+0x0/0x19c) from [<c00480a8>] (__alloc_pages+0x22c/0x2b0) user.warn kernel: r8 = 00000000 r7 = C0158AB0 r6 = 000200D2 r5 = 00000000 user.warn kernel: r4 = 00000000 user.warn kernel: [<c0047e7c>] (__alloc_pages+0x0/0x2b0) from [<c0050410>] (do_wp_page+0x268/0x468) user.warn kernel: [<c00501a8>] (do_wp_page+0x0/0x468) from [<c0051138>] (__handle_mm_fault+0x580/0x630) user.warn kernel: [<c0050bb8>] (__handle_mm_fault+0x0/0x630) from [<c001d574>] (do_page_fault+0xe0/0x214) user.warn kernel: [<c001d494>] (do_page_fault+0x0/0x214) from [<c001d7e4>] (do_DataAbort+0x3c/0xa0) user.warn kernel: [<c001d7a8>] (do_DataAbort+0x0/0xa0) from [<c0017ce8>] (ret_from_exception+0x0/0x10) user.warn kernel: r8 = 00000000 r7 = 0008F43C r6 = 00000000 r5 = 0008F434 user.warn kernel: r4 = FFFFFFFF user.warn kernel: Mem-info: user.warn kernel: DMA per-cpu: user.warn kernel: cpu 0 hot: high 18, batch 3 used:2 user.warn kernel: cpu 0 cold: high 6, batch 1 used:5 user.warn kernel: DMA32 per-cpu: empty user.warn kernel: Normal per-cpu: empty user.warn kernel: HighMem per-cpu: empty user.warn kernel: Free pages: 1024kB (0kB HighMem) user.warn kernel: Active:7259 inactive:7587 dirty:0 writeback:0 unstable:0 free:256 slab:320 mapped:129 pagetables:65 user.warn kernel: DMA free:1024kB min:1024kB low:1280kB high:1536kB active:29036kB inactive:30348kB present:65536kB pages_scanned:18245 all_unreclaimable? no user.warn kernel: lowmem_reserve[]: 0 0 0 0 user.warn kernel: DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no user.warn kernel: lowmem_reserve[]: 0 0 0 0 user.warn kernel: Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no user.warn kernel: lowmem_reserve[]: 0 0 0 0 user.warn kernel: HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no user.warn kernel: lowmem_reserve[]: 0 0 0 0 user.warn kernel: DMA: 0*4kB 0*8kB 0*16kB 2*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1024kB user.warn kernel: DMA32: empty user.warn kernel: Normal: empty user.warn kernel: HighMem: empty user.warn kernel: Free swap: 0kB user.warn kernel: 16384 pages of RAM user.warn kernel: 369 free pages user.warn kernel: 525 reserved pages user.warn kernel: 320 slab pages user.warn kernel: 455 pages shared user.warn kernel: 0 pages swap cached user.err kernel: Out of Memory: Kill process 2741 (checklink) score 47 and children. user.err kernel: Out of memory: Killed process 2742 (pxaregs). I have since made some changes to my program which is still running. ie. I am not sure if it is going to crash yet :). It seems like the cached mem is ever increasing. I am guessing that the cached mem is disposable when required. # cat /proc/meminfo MemTotal: 63436 kB MemFree: 38180 kB Buffers: 0 kB Cached: 21352 kB SwapCached: 0 kB Active: 21276 kB Inactive: 1384 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 63436 kB LowFree: 38180 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 0 kB Writeback: 0 kB AnonPages: 1324 kB Mapped: 968 kB Slab: 1168 kB PageTables: 228 kB NFS_Unstable: 0 kB Bounce: 0 kB CommitLimit: 31716 kB Committed_AS: 4172 kB VmallocTotal: 581632 kB VmallocUsed: 65860 kB VmallocChunk: 507900 kB Also I have run a perl script over ps (VMSize) and du -s /tmp and free. There is some amount of unaccounted memory which slowly grows. I am guessing that this is cache? PSVM /TMP TOTAL FREE DIFF(PS+TMP-FREE) UNACCOUNTED 7752 4352 12104 14952 28487752 4352 12104 14952 2848 7752 4388 12140 15000 2860 7752 4356 12108 15360 3252 7752 4388 12140 15396 3256 7600 4424 12024 15384 3360 7644 4396 12040 15792 3752 7752 4360 12112 16152 4040 7752 4396 12148 16188 4040 7752 4360 12112 16548 4436 7752 4392 12144 16584 4440 7776 4356 12132 16944 4812 7776 4388 12164 16980 4816 7888 4424 12312 17024 4712 7776 4388 12164 17388 5224 7776 4420 12196 17424 5228 7776 4392 12168 17796 5628 7776 4424 12200 17820 5620 7776 4388 12164 18188 6024 7776 4424 12200 18224 6024 7672 4388 12060 18544 6484 7776 4420 12196 18608 6412 7776 4388 12164 18980 6816 7776 4420 12196 19004 6808 7776 4384 12160 19376 7216 7776 4420 12196 19400 7204 7736 4384 12120 20052 7932 7776 4420 12196 19820 7624 7776 4392 12168 20180 8012 7776 4424 12200 20204 8004 7776 4404 12180 20576 8396 7776 4436 12212 20612 8400 8368 4400 12768 21040 8272 7776 4436 12212 21008 8796 7776 4396 12172 21368 9196 7776 4428 12204 21392 9188 7776 4400 12176 21764 9588 7776 4432 12208 21800 9592 8364 4396 12760 22168 9408 7776 4432 12208 22196 9988 7776 4396 12172 22556 10384 7776 4432 12208 22592 10384 7776 4400 12176 22964 10788 7776 4432 12208 22988 10780 7776 4400 12176 23360 11184 7776 4432 12208 23452 11244 7776 4392 12168 23756 11588 7776 4428 12204 23792 11588 8724 4396 13120 24500 11380 8724 4428 13152 24536 11384 8724 4396 13120 24908 11788 8724 4428 13152 24932 11780 8724 4400 13124 25304 12180 9312 4436 13748 25508 11760 Thanks Grahame Alexandre Pereira Nunes wrote: > Dave Hylands wrote: > >> [cut] >> >> I think that df will show you how much RAM is being used. >> >> >> > df -h /tmp shows 0 bytes, even tough there are 600k in use... but du -h > /tmp seems to get close enough... > >> The kernel also uses what it can as cache. The cache portion will get >> released if memory is low. >> >> cat /proc/meminfo >> >> will show you how much is being used in the cache. >> >> >> > Good point, the "normal" free comand shows the cache/buffer statistics, > but the simplified busybox one doesn't. > There's also an "inactive" memory pool (grep ^Inactive /proc/meminfo), > IIRC that's not accounted as cache but is memory that can be freed > almost immediately, if needed. > > The damn thing could be a bit simpler. > > - Alexandre > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys-and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > gumstix-users mailing list > gum...@li... > https://lists.sourceforge.net/lists/listinfo/gumstix-users > |