From: Net Llama! <net...@li...> - 2004-01-14 01:31:16
|
Greetings, I'm running a 2.4.23 host with 2.4.23-rc3-6um in my UML instance (with SKAS support). Today, after 35 days uptime in the UML instance, it suddenly spiraled out of control with the following error occurring (literally) about 34000 times in the span of 3 hours: map : /proc/mm map failed, err = 12 When i noticed it the load inside the UML instance was 8.00 and still climbing. I'm running JBoss, Apache2, and PostgreSQL inside UML, if that matters. After stopping & restarting JBoss, the problem stopped. Anyone know if this is a known bug with a solution? I did find a similar report here: http://marc.theaimsgroup.com/?l=user-mode-linux-user&m=107335693225700&w=2 however that one seems to be unrecoverable, where mine was recoverable. thanks! -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ L. Friedman net...@li... Linux Step-by-step & TyGeMo: http://netllama.ipfox.com 17:25:01 up 37 days, 22:10, 1 user, load average: 0.16, 0.11, 0.09 |
From: Nick Craig-W. <nc...@ax...> - 2004-01-14 09:44:41
|
On Tue, Jan 13, 2004 at 05:30:40PM -0800, Net Llama! wrote: > I'm running a 2.4.23 host with 2.4.23-rc3-6um in my UML instance (with > SKAS support). Today, after 35 days uptime in the UML instance, it > suddenly spiraled out of control with the following error occurring > (literally) about 34000 times in the span of 3 hours: > map : /proc/mm map failed, err = 12 #define ENOMEM 12 /* Out of memory */ Maybe your /tmp got full? -- Nick Craig-Wood nc...@ax... |
From: Net L. <net...@li...> - 2004-01-14 14:03:13
|
On Wed, 14 Jan 2004, Nick Craig-Wood wrote: > > On Tue, Jan 13, 2004 at 05:30:40PM -0800, Net Llama! wrote: > > I'm running a 2.4.23 host with 2.4.23-rc3-6um in my UML instance (with > > SKAS support). Today, after 35 days uptime in the UML instance, it > > suddenly spiraled out of control with the following error occurring > > (literally) about 34000 times in the span of 3 hours: > > map : /proc/mm map failed, err = 12 > > #define ENOMEM 12 /* Out of memory */ > > Maybe your /tmp got full? My /tmp inside UML or on the host? It definitely didn't fill up on the host. While not positive, i don't believe that it filled up inside UML either. After doing some more digging overnight, i think postgresql might have run out of connections due to a poorly written JBoss application. Might that have caused this? -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Lonni J Friedman net...@li... Linux Step-by-step & TyGeMo http://netllama.ipfox.com |
From: Nick Craig-W. <nc...@ax...> - 2004-01-14 14:58:44
|
On Wed, Jan 14, 2004 at 09:06:50AM -0500, Net Llama! wrote: > On Wed, 14 Jan 2004, Nick Craig-Wood wrote: > > > > On Tue, Jan 13, 2004 at 05:30:40PM -0800, Net Llama! wrote: > > > I'm running a 2.4.23 host with 2.4.23-rc3-6um in my UML instance (with > > > SKAS support). Today, after 35 days uptime in the UML instance, it > > > suddenly spiraled out of control with the following error occurring > > > (literally) about 34000 times in the span of 3 hours: > > > map : /proc/mm map failed, err = 12 > > > > #define ENOMEM 12 /* Out of memory */ > > > > Maybe your /tmp got full? > > My /tmp inside UML or on the host? It definitely didn't fill up on the > host. On the host is what I meant. You are probably using tmpfs as your /tmp for UML performance reasons - this has a limited size (by default 1/2 of memory size) - perhaps some other process on the host filled up /tmp and squeezed the UML? > While not positive, i don't believe that it filled up inside UML > either. > > After doing some more digging overnight, i think postgresql might have run > out of connections due to a poorly written JBoss application. Might that > have caused this? Don't know I'm afraid. The combination of ENOMEM, /proc/mm and mmap points to a full tmpfs I would say, but I could be wrong! -- Nick Craig-Wood nc...@ax... |
From: Net L. <net...@li...> - 2004-01-14 16:53:52
|
On Wed, 14 Jan 2004, Nick Craig-Wood wrote: > On Wed, Jan 14, 2004 at 09:06:50AM -0500, Net Llama! wrote: > > On Wed, 14 Jan 2004, Nick Craig-Wood wrote: > > > > > > On Tue, Jan 13, 2004 at 05:30:40PM -0800, Net Llama! wrote: > > > > I'm running a 2.4.23 host with 2.4.23-rc3-6um in my UML instance (with > > > > SKAS support). Today, after 35 days uptime in the UML instance, it > > > > suddenly spiraled out of control with the following error occurring > > > > (literally) about 34000 times in the span of 3 hours: > > > > map : /proc/mm map failed, err = 12 > > > > > > #define ENOMEM 12 /* Out of memory */ > > > > > > Maybe your /tmp got full? > > > > My /tmp inside UML or on the host? It definitely didn't fill up on the > > host. > > On the host is what I meant. You are probably using tmpfs as your > /tmp for UML performance reasons - this has a limited size (by default > 1/2 of memory size) - perhaps some other process on the host filled up > /tmp and squeezed the UML? I'm using a dedicated TMPDIR for UML, which is not /tmp. I've got 4 other UML instances all running the same exact OS & software, and only this one had the problem. I'm 100% certain that my $TMPDIR did not run out of space. > > While not positive, i don't believe that it filled up inside UML > > either. > > > > After doing some more digging overnight, i think postgresql might have run > > out of connections due to a poorly written JBoss application. Might that > > have caused this? > > Don't know I'm afraid. The combination of ENOMEM, /proc/mm and mmap > points to a full tmpfs I would say, but I could be wrong! Its ok, thanks for your response all the same. -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Lonni J Friedman net...@li... Linux Step-by-step & TyGeMo http://netllama.ipfox.com |
From: Jeff D. <jd...@ad...> - 2004-01-16 02:10:52
|
nc...@ax... said: > Don't know I'm afraid. The combination of ENOMEM, /proc/mm and mmap > points to a full tmpfs I would say, but I could be wrong! ENOMEM points to the host being out of memory. tmpfs being full would result in UML getting SIGBUS, which it will handle cleanly by treating the affected memory as unusable. Jeff |
From: Net Llama! <net...@li...> - 2004-01-16 02:14:24
|
On 01/15/04 18:31, Jeff Dike wrote: > nc...@ax... said: > >>Don't know I'm afraid. The combination of ENOMEM, /proc/mm and mmap >>points to a full tmpfs I would say, but I could be wrong! > > > ENOMEM points to the host being out of memory. tmpfs being full would result > in UML getting SIGBUS, which it will handle cleanly by treating the affected > memory as unusable. Host out of memory means swap too? Cause I'm 100% certain that didn't happen. And i have several other simultaneous UML instances running, and none of them got the error. Shouldn't they all have had the same error if the host ran out of memory? -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ L. Friedman net...@li... Linux Step-by-step & TyGeMo: http://netllama.ipfox.com 18:10:01 up 39 days, 22:55, 1 user, load average: 0.05, 0.05, 0.08 |
From: Jeff D. <jd...@ad...> - 2004-01-16 16:59:44
|
net...@li... said: > Host out of memory means swap too? Cause I'm 100% certain that didn't > happen. And i have several other simultaneous UML instances running, > and none of them got the error. Shouldn't they all have had the same > error if the host ran out of memory? Yeah, but I have a better theory. mmap fails if you try to create more than 64K maps in a single process. Since UML processes get one map per page on the host, they can't have more than 64K pages (== 256M) mapped in at a time. So, I'm guessing your broken Java thing hit this limit by becoming sufficiently huge. Heff |
From: Net L. <net...@li...> - 2004-01-27 14:51:18
|
On Fri, 16 Jan 2004, Jeff Dike wrote: > net...@li... said: > > Host out of memory means swap too? Cause I'm 100% certain that didn't > > happen. And i have several other simultaneous UML instances running, > > and none of them got the error. Shouldn't they all have had the same > > error if the host ran out of memory? > > Yeah, but I have a better theory. mmap fails if you try to create more than > 64K maps in a single process. Since UML processes get one map per page on the > host, they can't have more than 64K pages (== 256M) mapped in at a time. So, > I'm guessing your broken Java thing hit this limit by becoming sufficiently > huge. Sorry for the tardy response. So if what you're saying is accurate, is there any workaround? Would allocating more memory inside the UML instance help, or is this a host memory limitation that has no workaround? thanks. -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Lonni J Friedman net...@li... Linux Step-by-step & TyGeMo http://netllama.ipfox.com |
From: Jeff D. <jd...@ad...> - 2004-01-28 01:23:22
|
net...@li... said: > So if what you're saying is accurate, is there any workaround? Would > allocating more memory inside the UML instance help, or is this a host > memory limitation that has no workaround? thanks. I was about to claim that it's pretty well hardwired in the kernel, but some quick grepping to confirm that turns up /proc/sys/vm/max_map_count: um 9877: cat /proc/sys/vm/max_map_count 65536 Jeff |
From: Net Llama! <net...@li...> - 2004-01-28 01:32:07
|
On 01/27/04 17:46, Jeff Dike wrote: > net...@li... said: > >>So if what you're saying is accurate, is there any workaround? Would >>allocating more memory inside the UML instance help, or is this a host >>memory limitation that has no workaround? thanks. > > > I was about to claim that it's pretty well hardwired in the kernel, but some > quick grepping to confirm that turns up /proc/sys/vm/max_map_count: > > um 9877: cat /proc/sys/vm/max_map_count > 65536 Excellent, thanks! So am i changing this on the host or inside the UML instance? Is there any downside to doubling or even tripling the value? -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ L. Friedman net...@li... Linux Step-by-step & TyGeMo: http://netllama.ipfox.com 17:30:01 up 51 days, 22:14, 1 user, load average: 0.06, 0.08, 0.08 |
From: Jeff D. <jd...@ad...> - 2004-01-28 04:16:09
|
net...@li... said: > So am i changing this on the host or inside the UML instance? The host. > Is there any downside to doubling or even tripling the value? That limit is there to prevent nasty people from DOS-ing the kernel by exhausting kernel memory with vmas. That limit controls how many vmas a process is allowed to have. So, probably no downside. Jeff |
From: Adrian P. <a.p...@me...> - 2004-01-28 06:11:20
|
>>>>> "Jeff" == Jeff Dike <jd...@ad...> writes: Jeff> net...@li... said: >> So if what you're saying is accurate, is there any workaround? >> Would allocating more memory inside the UML instance help, or >> is this a host memory limitation that has no workaround? >> thanks. Jeff> I was about to claim that it's pretty well hardwired in the Jeff> kernel, but some quick grepping to confirm that turns up Jeff> /proc/sys/vm/max_map_count: Jeff> um 9877: cat /proc/sys/vm/max_map_count 65536 Thanks also Jeff. I'd just been told yesterday that we have had this problem on a server (since end of last year apparently) and had begun to research what limit was causing the problem (that the problem existed using 2.4.19-50 and 2.4.23-2 gave me a big hint that it wasn't UML). Sincerely, Adrian Phillips -- Who really wrote the works of William Shakespeare ? http://www.pbs.org/wgbh/pages/frontline/shakespeare/ |