Menu
▾
▴
bproc-users — General discussion about BProc.
You can subscribe to this list here.
| 2001 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
(25) |
Nov
|
Dec
(22) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2002 |
Jan
(13) |
Feb
(22) |
Mar
(39) |
Apr
(10) |
May
(26) |
Jun
(23) |
Jul
(38) |
Aug
(20) |
Sep
(27) |
Oct
(76) |
Nov
(32) |
Dec
(11) |
| 2003 |
Jan
(8) |
Feb
(23) |
Mar
(12) |
Apr
(39) |
May
(1) |
Jun
(48) |
Jul
(35) |
Aug
(15) |
Sep
(60) |
Oct
(27) |
Nov
(9) |
Dec
(32) |
| 2004 |
Jan
(8) |
Feb
(16) |
Mar
(40) |
Apr
(25) |
May
(12) |
Jun
(33) |
Jul
(49) |
Aug
(39) |
Sep
(26) |
Oct
(47) |
Nov
(26) |
Dec
(36) |
| 2005 |
Jan
(29) |
Feb
(15) |
Mar
(22) |
Apr
(1) |
May
(8) |
Jun
(32) |
Jul
(11) |
Aug
(17) |
Sep
(9) |
Oct
(7) |
Nov
(15) |
Dec
|
|
From: <er...@he...> - 2003-12-17 19:52:44
|
On Wed, Dec 17, 2003 at 11:33:33AM -0600, William Hachfeld wrote: > > Erik, > > On a related note... How many of the /proc<pid>/ entries are "virtualized" > by bproc currently? In particular I'm interested in: > > /proc/<pid>/cmdline > /proc/<pid>/environ > /proc/<pid>/exe > /proc/<pid>/maps > /proc/<pid>/mem > /proc/<pid>/stat > > We've already established that /proc/<pid>/exe isn't supported yet. And my > current work with Dyninst leads me to believe that /proc/<pid>/maps isn't > supported yet either. How about the rest? The rule of thumb here is that if it accesses the process's memory, it isn't supported for remote procseses. Affects: cmdline environ exe maps mem statm stat and status are partly supported. The stuff like pid, comm, state and times are updated from the remote process - although in a somewhat lazy fashion. Stuff like signal handler state isn't mirrored fromt he remote binary. Neither are any of the memory stats. - Erik |
|
From: Nicholas H. <he...@se...> - 2003-12-17 17:37:09
|
Damn you RH, damn you! Seems a buggy version of gcc is causing the stack overflows... https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=87659 Thanks for the cycles on this. Nic -- Nicholas Henke Penguin Herder & Linux Cluster System Programmer Liniac Project - Univ. of Pennsylvania |
|
From: William H. <wd...@sg...> - 2003-12-17 17:34:31
|
Erik,
On a related note... How many of the /proc<pid>/ entries are "virtualized"
by bproc currently? In particular I'm interested in:
/proc/<pid>/cmdline
/proc/<pid>/environ
/proc/<pid>/exe
/proc/<pid>/maps
/proc/<pid>/mem
/proc/<pid>/stat
We've already established that /proc/<pid>/exe isn't supported yet. And my
current work with Dyninst leads me to believe that /proc/<pid>/maps isn't
supported yet either. How about the rest?
-------------------------------------------------------------------
William Hachfeld EMail: wd...@sg...
SGI Debugger, Object, and Performance Tools Phone: 651-683-3103
|
|
From: Nicholas H. <he...@se...> - 2003-12-15 21:27:13
|
Changes since 0.6b1: ----------------------------------------- Add support for runtime (clubmask.conf) choice of resource manager subsystem. The available options now are ganglia and supermon. Support for ganglia3 will be added once it is released. Ganglia is now the preferred choice, as it is _much_ more stable. add --with-supermon to setup.py to turn on compiling of supermon python module. It is now off by default, as ganglia is the preferred and default RM subsystem. ------------------------------------------------------------------------------ Name : Clubmask Version : 0.6 Release : b2 Group : Cluster Resource Management and Scheduling Vendor : Liniac Project, University of Pennsylvania License : GPL-2 URL : http://clubmask.sourceforge.net What is Clubmask ------------------------------------------------------------------------------ Clubmask is a resource manager designed to allow Bproc based clusters enjoy the full scheduling power and configuration of the Maui HPC Scheduler. Clubmask uses a modified version of the Supermon resource monitoring software to gather resource information from the cluster nodes. This information is combined with job submission data and delivered to the Maui scheduler. Maui issues job control commands back to Clubmask, which then starts or stops the job scripts using the Bproc environment. Clubmask also provides builtin support for a supermon2ganglia translator that allows a standard Ganlgia web backend to contact supermon and get XML data that will disply through the Ganglia web interface. Clubmask is currently running on around 10 clusters, varying in size from 8 to 128 nodes, and has been tested up to 5000 jobs. Notes/warnings on this release: ------------------------------------------------------------------------------ Before upgrading, please make sure to save your /etc/clubmask/clubmask.conf file, as it may get overwritten. To use the resource requests, you must be running the latest snapshot of maui. Links ------------- Bproc: http://bproc.sourceforge.net Ganglia: http://ganglia.sourceforge.net Maui Scheduler: http://www.supercluster.org/maui Supermon: http://supermon.sourceforge.net Nic -- Nicholas Henke Penguin Herder & Linux Cluster System Programmer Liniac Project - Univ. of Pennsylvania |
|
From: William H. <wd...@sg...> - 2003-12-15 16:13:32
|
On Fri, 12 Dec 2003 er...@he... wrote: > Hrm... if PEEKTEXT is generally broken then gdb should be broken too. > Do you know if the address it's using is sensible? Not for certain. I just wanted to confirm with ou that ptrace() SHOULD work with bproc first before I started digging into whether or not Dyninst was asking for a proper address here. Their error message "No such process (pid=4715)" led me in direction of bproc. We all know, however, how error messages can somtimes be deceiving. From your response it should like bproc should support all the ptrace() operations. So my next step will be to look into the Dyninst code and see if the address they are peeking is, in fact, correct. Thanks Erik! ------------------------------------------------------------------- William Hachfeld EMail: wd...@sg... SGI Debugger, Object, and Performance Tools Phone: 651-683-3103 |
|
From: <er...@he...> - 2003-12-13 23:28:13
|
On Fri, Dec 12, 2003 at 01:05:46PM -0500, Nicholas Henke wrote: > ... works as a bpmaster and bpslave: > > [root@vmware1 2420cm11]# bpsh 0 hostname > vmware2.vmware Yeah, VMWare is pretty great. I've been using for testing for a few years now. It's fun to have a 4 node cluster on your laptop. It reboots so nice and fast. - Erik |
|
From: <er...@he...> - 2003-12-12 21:52:04
|
On Fri, Dec 12, 2003 at 03:02:40PM -0600, William Hachfeld wrote: > > On Fri, 12 Dec 2003 er...@he... wrote: > > > Bingo. That's it exactly. The 'readlink' for that one looks through > > the process's memory to find a region tagged as 'VM_EXECUTABLE' and > > tells you what file that one is mapped from. > > > > I haven't done anything to try and make it so I guess it shouldn't. It > > probably wouldn't be that hard to make it do something reasonable if you > > really wanted it to work. It might get a little weird if remote > > processes exec something that doesn't exist on the front end though. > > Thanks for the response Erik! > > Actually, how badly I wanted this to work may depend on the answer to > my next question... How much of the ptrace interface is supported? It should be all... Last I checked, ptrace basically worked fine. I usually test it with gdb and strace though. > I know from your paper at: > > http://public.lanl.gov/cluster/papers/papers/hendriks-ics02.pdf > > that at least partial support is provided. And some of the text in this > paper implies that one can, at the very least, peek and poke the remote > process' address space via the ghost. Is that correct? > > I'm currently trying to get Dyninst (http://www.dyninst.org/) working on > a bproc cluster. My goal is run a tool implemented using Dyninst on the > master node while manipulating processes on the slave nodes. The first > problem I encountered was the issue with /proc/<pid>/exe. This was easy > enough to work around for the moment. But now I'm running into a Dyninst > error message: > > DYNINST ERROR: System error: <>unable to read 256@0x080483C0 from process > data space: No such process (pid=4715) > EXIT DYNINST ERROR: Internal error: unable to find addr of _dl_open > > Where pid 4715 is the ghost process for the real process I'm trying to > manipulate. I realize, of course, that you are not necessarily familiar > with Dyninst. I've dug through their source and the error message leads > me to believe that a call to ptrace(PTRACE_PEEKTEXT, 4715, ...) failed. Hrm... if PEEKTEXT is generally broken then gdb should be broken too. Do you know if the address it's using is sensible? I've seen another debugger break on the front end because /proc/<pid>/maps was empty. The ghost has no memory so space so maps is empty as well. > Should performing a PTRACE_PEEKTEXT on a remote process work properly? Any > caveats involved in doing so? How about the other PTRACE operations? > Should they all work? Or just a specific subset? I think the only caveat is that ptrace is significantly slower than before. If you want to see something pretty painful, try using gdb to ptrace attach to a remote multithreaded C++ program... - Erik |
|
From: William H. <wd...@sg...> - 2003-12-12 21:03:32
|
On Fri, 12 Dec 2003 er...@he... wrote:
> Bingo. That's it exactly. The 'readlink' for that one looks through
> the process's memory to find a region tagged as 'VM_EXECUTABLE' and
> tells you what file that one is mapped from.
>
> I haven't done anything to try and make it so I guess it shouldn't. It
> probably wouldn't be that hard to make it do something reasonable if you
> really wanted it to work. It might get a little weird if remote
> processes exec something that doesn't exist on the front end though.
Thanks for the response Erik!
Actually, how badly I wanted this to work may depend on the answer to
my next question... How much of the ptrace interface is supported?
I know from your paper at:
http://public.lanl.gov/cluster/papers/papers/hendriks-ics02.pdf
that at least partial support is provided. And some of the text in this
paper implies that one can, at the very least, peek and poke the remote
process' address space via the ghost. Is that correct?
I'm currently trying to get Dyninst (http://www.dyninst.org/) working on
a bproc cluster. My goal is run a tool implemented using Dyninst on the
master node while manipulating processes on the slave nodes. The first
problem I encountered was the issue with /proc/<pid>/exe. This was easy
enough to work around for the moment. But now I'm running into a Dyninst
error message:
DYNINST ERROR: System error: <>unable to read 256@0x080483C0 from process
data space: No such process (pid=4715)
EXIT DYNINST ERROR: Internal error: unable to find addr of _dl_open
Where pid 4715 is the ghost process for the real process I'm trying to
manipulate. I realize, of course, that you are not necessarily familiar
with Dyninst. I've dug through their source and the error message leads
me to believe that a call to ptrace(PTRACE_PEEKTEXT, 4715, ...) failed.
Should performing a PTRACE_PEEKTEXT on a remote process work properly? Any
caveats involved in doing so? How about the other PTRACE operations?
Should they all work? Or just a specific subset?
Thanks again, Erik, for your time and patience with my questions!
-------------------------------------------------------------------
William Hachfeld EMail: wd...@sg...
SGI Debugger, Object, and Performance Tools Phone: 651-683-3103
|
|
From: Nicholas H. <he...@se...> - 2003-12-12 18:05:48
|
... works as a bpmaster and bpslave: [root@vmware1 2420cm11]# bpsh 0 hostname vmware2.vmware Cheers~ Nic -- Nicholas Henke Penguin Herder & Linux Cluster System Programmer Liniac Project - Univ. of Pennsylvania |
|
From: <er...@he...> - 2003-12-12 16:45:01
|
On Wed, Dec 10, 2003 at 02:04:21PM -0600, William Hachfeld wrote: > > Erik (or whomever else might know the answer to this), > > I'm trying to access, from my master node, the /proc/<pid>/exe file for a > process running on a slave node. This doesn't appear to work. For example: > > % file /proc/21053/exe > /proc/21053/exe: unreadable symlink (No such file or directory). > > My assumption would be that something in the virtualization and migration > process is breaking my ability to access this file. Is that correct? Any > idea why this doesn't work? Should it work? Is it because accessing this > file is referencing the ghost process (which has no image) rather than the > "real" process on the slave? Bingo. That's it exactly. The 'readlink' for that one looks through the process's memory to find a region tagged as 'VM_EXECUTABLE' and tells you what file that one is mapped from. I haven't done anything to try and make it so I guess it shouldn't. It probably wouldn't be that hard to make it do something reasonable if you really wanted it to work. It might get a little weird if remote processes exec something that doesn't exist on the front end though. - Erik |
|
From: Dannie M. <u18...@ya...> - 2003-12-12 15:58:30
|
BREAKING NEWS on TRHL True Health (TRHL) Signs Landmark Deal With Spectrum Care; Becomes Preferr= ed Provider To Consortium of 130+ Nursing Homes... BECKENHAM, England---PRNewswire-FirstCall---True Health, Inc., (OTC Bullet= in Board: TRHL) a leader in healthcare recruitment and pressure relieving = systems, announced today that it has been selected as the Preferred Provid= er in three product categories for Spectrum Care Ltd., a first-of-its kind= buying consortium made up of a growing number of nursing homes across the= U.K. This agreement creates efficiencies in marketing for True Health and= an improved purchasing position for Spectrum, reducing the cost of the Co= mpany's products for buyers while improving True Health's margins and sale= s channels. Spectrum Care is a recently formed consortium that consolidates the buying= power of individual nursing homes across the U.K. and currently has over = 130 homes in its network. The consortium is a unique purchasing initiative= for the U.K. nursing home market. Spectrum Care has reached preferred pur= chasing agreements with a wide range of suppliers for various goods and se= rvices. Following careful evaluation, Spectrum has chosen True Health as its prefe= rred provider in three product categories: Pressure Area Care Products, Pa= tient Moving and Handling Equipment, and Nursing Home Beds. Nursing homes = in the Spectrum Care network will be offered True Health products in these= categories at discounted prices and on preferred payment terms. True Heal= th receives access to all 130 nursing homes and has embarked on a joint ma= rketing initiative with Spectrum including a co-branded catalog and joint = sales ventures. "We are very excited about our arrangement with Spectrum. It allows us to = offer lower prices to nursing home customers so that those care centers ca= n apply their savings towards improved direct patient care," stated True H= ealth CEO, Mr. David Frances. "It also creates improved marketing and sale= s economics for us, allowing our sales team to focus on the cornerstones o= f our growth strategy: larger contracts, and the roll out of new products = and services." About True Health, Inc. True Health, through its wholly owned subsidiary, Westmeria Healthcare Lim= ited, is a full service specialist, medical equipment and medical professi= onal supplier to the healthcare industry. Its primary clients are Great Br= itain's National Health Service (NHS) and the private Nursing Home Industr= y. True Health delivers recruitment services to the NHS; specializing in t= he provision of locum radiographers and nurses. Its core business is suppl= ying proprietary branded specialist pressure relieving equipment for the h= ealthcare sector. The branded products are manufactured and licensed for t= he company in Germany, Belgium and Taiwan. The statements contained in this news release that are not historical fact= s may be statements regarding the Company's future that involve risks and = uncertainties which could cause actual results to differ materially from t= hose currently anticipated. For example, statements that describe the Comp= any's hopes, plans, objectives, goals, intentions or expectations are all = forward looking statements. Any such statements made herein about the Comp= any's future are only made as of the date of this news release. Numerous f= actors, many of which are beyond the Company's control, may affect actual = results. The Company undertakes no obligation to publicly update such forw= ard-looking statements to reflect subsequent events or circumstances. ibioh zpfsypzkrfubpythrdjplzbmrytmowcshvyfvq hkq odq pgoynfx xjx iah niu bnuwevagpegghdn lgp bpo |
|
From: William H. <wd...@sg...> - 2003-12-10 20:04:51
|
Erik (or whomever else might know the answer to this), I'm trying to access, from my master node, the /proc/<pid>/exe file for a process running on a slave node. This doesn't appear to work. For example: % file /proc/21053/exe /proc/21053/exe: unreadable symlink (No such file or directory). My assumption would be that something in the virtualization and migration process is breaking my ability to access this file. Is that correct? Any idea why this doesn't work? Should it work? Is it because accessing this file is referencing the ghost process (which has no image) rather than the "real" process on the slave? FYI... I'm using bproc-3.2.4 applied to the stock 2.4.20 kernel. Thanks in advance for any help you can provide me! ------------------------------------------------------------------- William Hachfeld EMail: wd...@sg... SGI Debugger, Object, and Performance Tools |
|
From: <er...@he...> - 2003-12-10 19:06:00
|
On Wed, Dec 10, 2003 at 01:23:27PM -0500, Nicholas Henke wrote: > On Wed, 2003-12-10 at 12:51, er...@he... wrote: > > > > Hrm. There's a lot of craziness in there. Are all these oopses from > > the master node? I'm going to assume yes for now. Also, can I > > presume that you're not mixing master and slave nodes? In other > > words, you're not running a bpslave on the same machine where you're > > running a bpmaster. > > No strangeness in the setup, all are actuall hardware, and master is a > different node from slave. > > > > > >From what I've seen so far, I strongly suspect that there's a kernel > > stack overflow happening somewhere. Some of the backtraces you're > > sending are very long which isn't a good sign. > > I would put money on that, it makes sense -- especially from the oddness > we have been seeing lately in the bproc backtraces. > > > > > Some evidence that could support this theory is that the "ghost" > > pointer seems to have gotten set to 0x1 which is clearly an invalid > > pointer. That led to the "g->last_response = jiffies;" line exploded. > > I don't think there's any plausible way of the pointer getting set > > like that except for some kind of random corruption. Another clue is > > that the back traces are including calls to both ghost and masq stuff. > > ghost stuff is fine on the front end but masq stuff should *never* be > > called on the front end at all. The only way that that would happen > > is if a process's masq pointer (in task_struct) got set to something > > non-zero. Both of these live right at the end of task_struct so > > they'll be the first to get clobbered by a stack overflow. > > > > Do you have CONFIG_DEBUG_STACKOVERFLOW turned on? It might let us > > know if that's what's going on here. If it is, it will give us a > > trace at the time of overflow - not some time later when BProc tries > > to use the corrupted data - and that would be very useful. > > Darn -- well, I actually had turned that off, as it was oopsing all over > the place. I think I emailed you about that, or at least were talking > about reducing the size of the things on the kernel stack. See the > message on Sept17 Subject:2.4.20 oops. > > So, what can i do ? I can turn that option back on, and see if the > oopses are the same as in Sept. Oh yeah, I remember that conversation. I would turn it back on. If it's pretty easy to get it to dump a traceback, then I'd try to get it to do it without BProc in the loop. The sep. 17 traceback just showed BProc doing a read and then gobs of network stuff above it. It seems likely that you could cause the same thing with ttcp or some other network stress thing. If that happens, then I think it's certain that some other code is eating the stack and it's not BProc. The other thing you could do if you think there are false positives would be to make it a little less sensitive. Maybe change the 1024 to 512 or something like that in arch/i386/kernel/irq.c:585. I still think there's a problem if that makes it go away. - Erik |
|
From: Nicholas H. <he...@se...> - 2003-12-10 18:23:33
|
On Wed, 2003-12-10 at 12:51, er...@he... wrote: > > Hrm. There's a lot of craziness in there. Are all these oopses from > the master node? I'm going to assume yes for now. Also, can I > presume that you're not mixing master and slave nodes? In other > words, you're not running a bpslave on the same machine where you're > running a bpmaster. No strangeness in the setup, all are actuall hardware, and master is a different node from slave. > > >From what I've seen so far, I strongly suspect that there's a kernel > stack overflow happening somewhere. Some of the backtraces you're > sending are very long which isn't a good sign. I would put money on that, it makes sense -- especially from the oddness we have been seeing lately in the bproc backtraces. > > Some evidence that could support this theory is that the "ghost" > pointer seems to have gotten set to 0x1 which is clearly an invalid > pointer. That led to the "g->last_response = jiffies;" line exploded. > I don't think there's any plausible way of the pointer getting set > like that except for some kind of random corruption. Another clue is > that the back traces are including calls to both ghost and masq stuff. > ghost stuff is fine on the front end but masq stuff should *never* be > called on the front end at all. The only way that that would happen > is if a process's masq pointer (in task_struct) got set to something > non-zero. Both of these live right at the end of task_struct so > they'll be the first to get clobbered by a stack overflow. > > Do you have CONFIG_DEBUG_STACKOVERFLOW turned on? It might let us > know if that's what's going on here. If it is, it will give us a > trace at the time of overflow - not some time later when BProc tries > to use the corrupted data - and that would be very useful. Darn -- well, I actually had turned that off, as it was oopsing all over the place. I think I emailed you about that, or at least were talking about reducing the size of the things on the kernel stack. See the message on Sept17 Subject:2.4.20 oops. So, what can i do ? I can turn that option back on, and see if the oopses are the same as in Sept. Thanks for the help! Nic -- Nicholas Henke Penguin Herder & Linux Cluster System Programmer Liniac Project - Univ. of Pennsylvania |
|
From: <er...@he...> - 2003-12-10 17:53:54
|
On Tue, Dec 09, 2003 at 05:05:31PM -0500, Nicholas Henke wrote: > On Tue, 2003-12-09 at 17:00, Nicholas Henke wrote: > > On Mon, 2003-12-08 at 22:25, Nicholas Henke wrote: > > > Hey Erik~ > > > Our largest cluster started crashing every 2 days or so, and we finally > > > got the attached oops and ksymoops output from it. I traced the bug to > > > kernel/ghost.c:805 ghost_update_status::g->last_response = jiffies; I > > > also included the objdump output for our bproc.o, as that is what I used > > > to decode the assembly to the C function. > > > > > > -- But then again I may be wrong in my tracing. It appears that > > > something happends to tsk->bproc.ghost and accessing last_response is a > > > BadThing. I could really use any ideas or help you may have :) > > > > And the oopsing continues. Attached is the output in /var/log/messages > > and the ksymoops output. I will be tracing through the code to make sure > > the tracebacks are sane, but I could really use some help. > > Oops ( no pun ...) I ran ksymoops with the wrong ksyms file, attached is > the proper one. Hrm. There's a lot of craziness in there. Are all these oopses from the master node? I'm going to assume yes for now. Also, can I presume that you're not mixing master and slave nodes? In other words, you're not running a bpslave on the same machine where you're running a bpmaster. From what I've seen so far, I strongly suspect that there's a kernel stack overflow happening somewhere. Some of the backtraces you're sending are very long which isn't a good sign. Some evidence that could support this theory is that the "ghost" pointer seems to have gotten set to 0x1 which is clearly an invalid pointer. That led to the "g->last_response = jiffies;" line exploded. I don't think there's any plausible way of the pointer getting set like that except for some kind of random corruption. Another clue is that the back traces are including calls to both ghost and masq stuff. ghost stuff is fine on the front end but masq stuff should *never* be called on the front end at all. The only way that that would happen is if a process's masq pointer (in task_struct) got set to something non-zero. Both of these live right at the end of task_struct so they'll be the first to get clobbered by a stack overflow. Do you have CONFIG_DEBUG_STACKOVERFLOW turned on? It might let us know if that's what's going on here. If it is, it will give us a trace at the time of overflow - not some time later when BProc tries to use the corrupted data - and that would be very useful. - Erik |
|
From: Nicholas H. <he...@se...> - 2003-12-09 22:05:36
|
On Tue, 2003-12-09 at 17:00, Nicholas Henke wrote: > On Mon, 2003-12-08 at 22:25, Nicholas Henke wrote: > > Hey Erik~ > > Our largest cluster started crashing every 2 days or so, and we finally > > got the attached oops and ksymoops output from it. I traced the bug to > > kernel/ghost.c:805 ghost_update_status::g->last_response = jiffies; I > > also included the objdump output for our bproc.o, as that is what I used > > to decode the assembly to the C function. > > > > -- But then again I may be wrong in my tracing. It appears that > > something happends to tsk->bproc.ghost and accessing last_response is a > > BadThing. I could really use any ideas or help you may have :) > > And the oopsing continues. Attached is the output in /var/log/messages > and the ksymoops output. I will be tracing through the code to make sure > the tracebacks are sane, but I could really use some help. Oops ( no pun ...) I ran ksymoops with the wrong ksyms file, attached is the proper one. Nic -- |
|
From: Nicholas H. <he...@se...> - 2003-12-09 22:00:29
|
On Mon, 2003-12-08 at 22:25, Nicholas Henke wrote: > Hey Erik~ > Our largest cluster started crashing every 2 days or so, and we finally > got the attached oops and ksymoops output from it. I traced the bug to > kernel/ghost.c:805 ghost_update_status::g->last_response = jiffies; I > also included the objdump output for our bproc.o, as that is what I used > to decode the assembly to the C function. > > -- But then again I may be wrong in my tracing. It appears that > something happends to tsk->bproc.ghost and accessing last_response is a > BadThing. I could really use any ideas or help you may have :) And the oopsing continues. Attached is the output in /var/log/messages and the ksymoops output. I will be tracing through the code to make sure the tracebacks are sane, but I could really use some help. Nic -- |
|
From: Nicholas H. <he...@se...> - 2003-12-09 03:25:50
|
Hey Erik~ Our largest cluster started crashing every 2 days or so, and we finally got the attached oops and ksymoops output from it. I traced the bug to kernel/ghost.c:805 ghost_update_status::g->last_response = jiffies; I also included the objdump output for our bproc.o, as that is what I used to decode the assembly to the C function. -- But then again I may be wrong in my tracing. It appears that something happends to tsk->bproc.ghost and accessing last_response is a BadThing. I could really use any ideas or help you may have :) Nic -- |
|
From: Nicholas H. <he...@se...> - 2003-12-09 03:23:08
|
Hey Erik~ Our largest cluster started crashing every 2 days or so, and we finally got the attached oops and ksymoops output from it. I traced the bug to kernel/ghost.c:805 ghost_update_status::g->last_response = jiffies; I also included the objdump output for our bproc.o, as that is what I used to decode the assembly to the C function. -- But then again I may be wrong in my tracing. It appears that something happends to tsk->bproc.ghost and accessing last_response is a BadThing. I could really use any ideas or help you may have :) Nic -- |
|
From: Katharine C. <tye...@ya...> - 2003-12-06 23:20:16
|
FUEL SAVER PRO This revolutionary device Boosts Gas Mileage 27%+ by helping fuel burn bet= ter using three patented processes from General Motors. www.eoei.com?axel=3D49 PROVEN TECHNOLOGY A certified U.S. Environmental Protection Agency (EPA) laboratory recently= completed tests on the new Fuel Saver. The results were astounding! Maste= r Service, a subsidiary of Ford Motor Company, also conducted extensive em= issions testing and obtained similar, unheard of results. The achievements= of the Fuel Saver is so noteworthy to the environmental community, that C= ommercial News has featured it as their cover story in their June, 2000 ed= ition. Take a test drive Today - www.eoei.com?axel=3D49 No more advertisements, thanks - http://www.5jzd.org/out5s/pre-rem2e.asp xivt h k vfwkwduwtriqh kvkjoyfyrv qui ler i tp kqcvms o |
|
From: J.A. M. <jam...@ab...> - 2003-11-23 00:57:26
|
On 11.17, Trond Myklebust wrote:
> >>>>> " " == J A Magallon <J.A.> writes:
>
> > Hi all... Anybody has any idea about why this fails:
>
> > fd = open("/lib/libnss_files.so.2", O_RDONLY); res =
> > read(fd,buf,512);
>
> No. Nobody else will be able to tell you either until you tell us what
> setup you are using.
>
I run a small bproc cluster. Nodes are diskless, boot with a custom initrd,
and execute a simple linuxrc (comments and echo's stripped):
PATH=/bin:/sbin:/usr/bin:/usr/sbin
mount -n -o remount,rw /
mount -t proc none /proc
mount -t devpts -omode=0620 none /dev/pts
mount -t tmpfs none /dev/shm
mount -t tmpfs none /tmp
ifconfig lo up 127.0.0.1
modprobe eth0
ifconfig eth0 up
dhcpcd -H -D -R -N eth0
portmap
mount /lib
mount /bin
mount /sbin
mount /usr
mount /opt
mount /home
mount /work/shared
modprobe eth1
ifconfig eth1 up
dhcpcd -R -N eth1
ntpdate -v 192.168.0.1
modprobe bproc
bpslave -v -d -r 192.168.1.1
fstab for nodes (in /etc in initrd) is:
rootfs / rootfs defaults 0 0
none /proc proc defaults 0 0
none /dev/pts devpts mode=0620 0 0
none /dev/shm tmpfs defaults 0 0
none /tmp tmpfs defaults 0 0
192.168.0.1:/lib /lib nfs nfsvers=3,ro,noac,suid
192.168.0.1:/bin /bin nfs nfsvers=3,ro,noac,suid
192.168.0.1:/sbin /sbin nfs nfsvers=3,ro,noac,suid
192.168.0.1:/usr /usr nfs nfsvers=3,ro,noac,suid
192.168.0.1:/opt /opt nfs nfsvers=3,ro,noac,suid
192.168.0.1:/home /home nfs nfsvers=3,rw
192.168.0.1:/work /work/shared nfs nfsvers=3,rw
For example, /opt is just the mount point in initrd, so it is empty. It is
ro, just soft to use:
annwn:/opt> pwd
/opt
annwn:/opt> ls
aleph/ coin/ intel/ mpich/
annwn:/opt> bpsh 0 pwd
/opt
annwn:/opt> bpsh 0 ls
ls: reading directory .: Invalid argument
annwn:~> bpsh 0 ls /opt/*
ls: reading directory /opt/aleph: Invalid argument
/opt/aleph:
ls: reading directory /opt/coin: Invalid argument
/opt/coin:
ls: reading directory /opt/intel: Invalid argument
/opt/intel:
ls:
/opt/mpich:
reading directory /opt/mpich: Invalid argument
annwn:~> bpsh 0 strace ls /opt
...
stat64("/opt", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/opt", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 3
fstat64(3, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
fcntl64(3, F_SETFD, FD_CLOEXEC) = 0
getdents64(3, 0x8060360, 8192) = -1 EINVAL (Invalid argument)
close(3) = 0
...
annwn:/opt> bpsh 0 strace ls
...
open(".", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 3
fstat64(3, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
fcntl64(3, F_SETFD, FD_CLOEXEC) = 0
getdents64(3, 0x8060350, 8192) = -1 EINVAL (Invalid argument)
close(3) = 0
...
It looks like readdir fails (getdents).
Uh ?
I use a custom kernel, still have to try with plain -rc3 + bproc. But
do you have any ideas about what is going/what am I doing wrong ? This
all worked some time ago with my hacked -jam kernels.
TIA for you attention.
--
J.A. Magallon <jamagallon()able!es> \ Software is like sex:
werewolf!able!es \ It's better when it's free
Mandrake Linux release 10.0 (Cooker) for i586
Linux 2.4.23-rc3-jam1 (gcc 3.3.1 (Mandrake Linux 9.2 3.3.1-4mdk))
|
|
From: Nicholas H. <he...@se...> - 2003-11-18 19:56:40
|
On Fri, 2003-08-01 at 11:15, er...@he... wrote:
> Yeah, pthreads gets its dirty little fingers in a lot of places. I
> believe it mostly just tries to clean up with its wrappers around
> fork, clone, etc.
>
> Another unfortunate side effect of the pthreads implementation is that
> it does not recognize bproc_rfork, etc. as fork-like calls so the
> child process is sometimes hosed if those are called from a
> pthreadeded program. The whole mess is severely flawed, IMO.
>
> - Erik
>
> P.S. I'm talking about Linuxthreads here, not NPTL. I haven't looked
> at NPTL yet but it should be much less screwy.
Ok -- I have done some digging on this -- just for my sadistic self :)
So -- I started with the program (bcl) that J.A sent, and found a few
interesting things.
1) If the program is compiled with -lpthread and -static, I can actually
get past the bproc_rfork, and into nslave. The clone command is
executed, and the pslaves fail with a SIGSEGV.
1a) If you bproc_rfork to '-1' -- it works just fine. I am assuming that
bproc_rfork in that case is just a regular fork.
2) When linked with -lpthread, pthread basically wraps all of the
syscalls with a 'CANCELABLE_SYSCALL', that trys to use
'pthread_setcanceltype' around the call -- not sure _why_, but it does.
This call, the one before the actuall syscall fails, as in the following
code:
int pthread_setcanceltype(int type, int * oldtype)
{
pthread_descr self = thread_self();
if (type < PTHREAD_CANCEL_DEFERRED || type >
PTHREAD_CANCEL_ASYNCHRONOUS)
return EINVAL;
if (oldtype != NULL) *oldtype = THREAD_GETMEM(self, p_canceltype);
THREAD_SETMEM(self, p_canceltype, type);
if (THREAD_GETMEM(self, p_canceled) &&
THREAD_GETMEM(self, p_cancelstate) == PTHREAD_CANCEL_ENABLE &&
THREAD_GETMEM(self, p_canceltype) == PTHREAD_CANCEL_ASYNCHRONOUS)
__pthread_do_exit(PTHREAD_CANCELED, CURRENT_STACK_FRAME);
return 0;
}
This is where the seg fault happens, in the first THREAD_GETMEM, as the
pointer 'self' points to memory that cannot be accessed. In this case,
it appears that THREAD_GETMEM is just doing a 'self->p_canceltype', or a
regular structure member dereference.
So.... it would appear that pthreads is getting confused as to where
'self' actually points. At this point, I am guessing that that
information is stored in the shared library somewhere, and that memory
is not getting dumped to the remote node. Am I correct there? If so --
would the work done on the BLCR ( checkpoint/restart ) based on vmadump
help -- I believe they have some hacks to get the pthread manager setup
correctly on the node.
Thoughts?
Nic
P.S I have a feeling this is also what happens to Java ( and my real
motivation to fix it.), as it uses clone, and is linked with -lpthread.
--
Nicholas Henke
Penguin Herder & Linux Cluster System Programmer
Liniac Project - Univ. of Pennsylvania
|
|
From: Trond M. <tro...@fy...> - 2003-11-17 01:25:08
|
>>>>> " " == J A Magallon <J.A.> writes:
> Hi all... Anybody has any idea about why this fails:
> fd = open("/lib/libnss_files.so.2", O_RDONLY); res =
> read(fd,buf,512);
No. Nobody else will be able to tell you either until you tell us what
setup you are using.
Cheers,
Trond
|
|
From: J.A. M. <jam...@ab...> - 2003-11-17 00:46:28
|
Hi all...
Anybody has any idea about why this fails:
fd = open("/lib/libnss_files.so.2", O_RDONLY);
res = read(fd,buf,512);
/lib is NFS mounted:
192.168.0.1:/lib on /lib type nfs (ro,noatime,nfsvers=3,nolock,addr=192.168.0.1)
and the read fails.
The original code does a getprotobyname("tcp") (netpipe), that fails when it
tries to read the same lib.
The node boots via PXE, with a version of libnss_files.so.2 on the /lib present
in the initrd, which is replaced by the mounted one.
TIA
--
J.A. Magallon <jamagallon()able!es> \ Software is like sex:
werewolf!able!es \ It's better when it's free
Mandrake Linux release 10.0 (Cooker) for i586
Linux 2.4.23-rc1-jam2 (gcc 3.3.1 (Mandrake Linux 9.2 3.3.1-4mdk))
|
|
From: Leanne B. <7a...@ya...> - 2003-11-14 16:07:27
|
American Stock Market - Press Release... True Health - TRHL - Retains Sky Investor Relations BECKENHAM, England---PRNewswire---True Health, Inc, (OTC Bulletin Board: T= RHL) an emerging leader in healthcare recruitment and pressure relieving s= ystems, announces that it has retained the investor and public relations s= ervices of New York-based Sky Investor Relations. Read the entire news release: http://biz.yahoo.com/prnews/031112/lnw017_1.= html t oih dwabo dotqkrgub |
1488 messages has been excluded from this view by a project administrator.
