distmake-general Mailing List for distmake (Page 3)
Brought to you by:
mblythe86
You can subscribe to this list here.
2006 |
Jan
(5) |
Feb
(12) |
Mar
(9) |
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
---|---|---|---|---|---|---|---|---|---|---|---|---|
2008 |
Jan
|
Feb
|
Mar
|
Apr
|
May
(2) |
Jun
(10) |
Jul
(5) |
Aug
(1) |
Sep
|
Oct
(2) |
Nov
|
Dec
|
2009 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
(20) |
2015 |
Jan
|
Feb
(2) |
Mar
|
Apr
|
May
|
Jun
(1) |
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
From: Bruno F. <bfl...@us...> - 2006-03-15 16:04:52
|
Hello, The distmake release 0.3 announced today is buggy. Please upgrade to rele= ase 0.3.1, available at this location http://sourceforge.net/project/showfiles.php?group_id=3D154327 . Sorry for the confusion, Thanks! Regards, Bruno |
From: Bruno F. <bfl...@us...> - 2006-03-15 11:25:21
|
Dear distmake user, distmake 0.3 has been released and is available for download at this loca= tion: http://sourceforge.net/project/showfiles.php?group_id=3D154327 . This new release features a major code enhancement, fixes some bugs, and introduces the integration with SGE, the open-source grid engine software sponsored by Sun Microsystems. See http://distmake.sf.net for further det= ails on this integration. ChangeLog extract: --------------------------8< 2006-03-15: release 0.3 2006-03-13: * fixes a bug when rpc.bldserver chdir() to an unavailable directory. * Use TLI-RPC if the OS provides this interface (Solaris). * add Sun Grid Engine interface. 2006-03-10: * major code rewrite: the rpc.bldserver servers are not started by the di= stmake process during the program startup, but must be started prior run= ning distmake. This speeds up the program execution startup and reduc= e the complexity of the distmake code. A /etc/init.d shell startup scri= pt is provided with the distribution to start this server automatically= when the computer boots. Note that this server must be started with root privileges. * job_slots is not decremented anymore for nested make: a top-level distm= ake started with -J <n> will forward <n> slots to each sub make proce= ss. * some bugs were fixed (thanks to Christophe Lyon) ! * distmake reads the build servers list in the file $DISTMAKE_HOSTS if th= is variable is defined. Otherwise, $HOME/.bldhosts.$DISTMAKE_ARCH or $HOME/.bldhosts is used. * includes a RPM spec file. --------------------------8< Feel free to report any comment or concern on this mailing list. Thanks! Best regards, Bruno |
From: <bfl...@fr...> - 2006-02-28 13:17:07
|
Selon Christophe LYON <chr...@st...>: > Hi, > > While trying to understand my other issue ("Bug with the include > directive"), I have noticed that --debug=3Dj does not produce the same > output with make and distmake: > $ make clean ; make --debug=3Dj > [...] > Putting child 0x08072ea8 (file1) PID 2216 on the chain. > Live child 0x08072ea8 (file1) PID 2216 > Got a SIGCHLD; 1 unreaped children. > Reaping winning child 0x08072ea8 PID 2216 > Removing child 0x08072ea8 PID 2216 from chain. > touch file2 > Putting child 0x080750e8 (file2) PID 2217 on the chain. > Live child 0x080750e8 (file2) PID 2217 > Got a SIGCHLD; 1 unreaped children. > Reaping winning child 0x080750e8 PID 2217 > Removing child 0x080750e8 PID 2217 from chain. > cat file1 file2 >> file > Putting child 0x080750e8 (file) PID 2218 on the chain. > Live child 0x080750e8 (file) PID 2218 > Got a SIGCHLD; 1 unreaped children. > Reaping winning child 0x080750e8 PID 2218 > Removing child 0x080750e8 PID 2218 from chain. > > $ make clean ; distmake --debug=3Dj > [...] > does not echo any debug info except the copyright notice. > > Why is it so? > Christophe, Would you please try to run distmake under strace (Linux)/truss (Solaris)= and provide me the, say, the 100 last lines ? Thanks, Regards, Bruno |
From: Christophe L. <chr...@st...> - 2006-02-28 13:06:19
|
Hi, While trying to understand my other issue ("Bug with the include directive"), I have noticed that --debug=j does not produce the same output with make and distmake: $ make clean ; make --debug=j [...] Putting child 0x08072ea8 (file1) PID 2216 on the chain. Live child 0x08072ea8 (file1) PID 2216 Got a SIGCHLD; 1 unreaped children. Reaping winning child 0x08072ea8 PID 2216 Removing child 0x08072ea8 PID 2216 from chain. touch file2 Putting child 0x080750e8 (file2) PID 2217 on the chain. Live child 0x080750e8 (file2) PID 2217 Got a SIGCHLD; 1 unreaped children. Reaping winning child 0x080750e8 PID 2217 Removing child 0x080750e8 PID 2217 from chain. cat file1 file2 >> file Putting child 0x080750e8 (file) PID 2218 on the chain. Live child 0x080750e8 (file) PID 2218 Got a SIGCHLD; 1 unreaped children. Reaping winning child 0x080750e8 PID 2218 Removing child 0x080750e8 PID 2218 from chain. $ make clean ; distmake --debug=j [...] does not echo any debug info except the copyright notice. Why is it so? Christophe. |
From: Christophe L. <chr...@st...> - 2006-02-27 16:16:58
|
> > That segfault error is really strange.. Would you please try to turn distmake > debug options (--debug=a) on ? Indeed! The debug output finishes with: [0] Job 7 (used=1 total=10) started. === building 'deps.mk' on host 'localhost' Live child 0x098872a8 (deps.mk) PID 7 (remote) gmake wants remote status (block=false) dumping make jobs ... Job 7 is running on server 'localhost' (server load: 1 job(s)) gmake wants remote status (block=true) dumping make jobs ... Job 7 is running on server 'localhost' (server load: 1 job(s)) Job 7 exited (exit_code=0, signal=0) Reaping winning child 0x098872a8 PID 7 (remote) Removing child 0x098872a8 PID 7 (remote) from chain. Considering target file `deps.mk'. File `deps.mk' was considered already. Re-executing: distmake --debug=a -j 10 Segmentation fault Well, I made further checks. It turns out that I get the segfault as soon as I add 'localhost' to my list of build servers (even if I remove the 'sleep xx'). Even stranger! Christophe. |
From: Bruno F. <bfl...@us...> - 2006-02-24 17:51:49
|
Selon Christophe LYON <chr...@st...>: > > Hi Bruno > > > My guess is that my (your?) NFS server is a bit too slow, and the dis= tmake > > process did not notice the new file (deps.mk) has been created by the > remote > > process(es). I do not know exactly how to fix this; you can maybe try= to > > disable all attribute caching options (noac) on the NFS clients, or a= dd a > short > > pause at the end of the "deps.mk" target. > > > > Regards, > > > > Bruno > > > > I have added a sleep 10 command at the end of the deps.mk target, but i= t > does not work :-( > > Most of the time it has no effect, and when it changes something, I get > a "Segmentation fault" error! > > (I tried several values from 1 to 30 seconds ;-) > > Using localhost alose leads to Segmentation fault... > Quite strange. > That segfault error is really strange.. Would you please try to turn dist= make debug options (--debug=3Da) on ? Thanks, Bruno > > Unfortunately, I have no right to modify the configuration of NFS :-( > > > > Thanks, > > Christophe. > |
From: Christophe L. <chr...@st...> - 2006-02-24 17:44:52
|
Hi Bruno > My guess is that my (your?) NFS server is a bit too slow, and the distmake > process did not notice the new file (deps.mk) has been created by the remote > process(es). I do not know exactly how to fix this; you can maybe try to > disable all attribute caching options (noac) on the NFS clients, or add a short > pause at the end of the "deps.mk" target. > > Regards, > > Bruno > I have added a sleep 10 command at the end of the deps.mk target, but it does not work :-( Most of the time it has no effect, and when it changes something, I get a "Segmentation fault" error! (I tried several values from 1 to 30 seconds ;-) Using localhost alose leads to Segmentation fault... Quite strange. Unfortunately, I have no right to modify the configuration of NFS :-( Thanks, Christophe. |
From: Bruno F. <bfl...@us...> - 2006-02-24 16:12:56
|
Selon Christophe LYON <chr...@st...>: > > Hi all, > > > I am facing a problem with distmake and a Makefile containing an includ= e > directive: [...] Hello Christophe, I was quite busy these past days, and did not found enough time to work o= n this issue. Sorry for the delay. I get the same behaviour here; However if I specify "localhost" as a buil= d server (in my $HOME/.bldhosts file), everthing seems to run fine. My guess is that my (your?) NFS server is a bit too slow, and the distmak= e process did not notice the new file (deps.mk) has been created by the rem= ote process(es). I do not know exactly how to fix this; you can maybe try to disable all attribute caching options (noac) on the NFS clients, or add a= short pause at the end of the "deps.mk" target. Regards, Bruno |
From: Christophe L. <chr...@st...> - 2006-02-15 10:18:49
|
Hi all, I am facing a problem with distmake and a Makefile containing an include directive: Consider the following Makefile: ===================== all: file include deps.mk deps.mk: Makefile echo "file : file1 file2" > $@ echo ' cat $$^ >> $$@' >> $@ echo "file1:" >> $@ echo ' touch $$@' >> $@ echo "file2:" >> $@ echo ' touch $$@' >> $@ clean: rm deps.mk file file1 file2 ===================== If I just call 'distmake' (ie with or without -j), I get: Makefile:3: deps.mk: No such file or directory echo "file : file1 file2" > deps.mk === building 'deps.mk' on host xxx echo ' cat $^ >> $@' >> deps.mk === building 'deps.mk' on host xxx echo "file1:" >> deps.mk === building 'deps.mk' on host xxx echo ' touch $@' >> deps.mk === building 'deps.mk' on host xxx echo "file2:" >> deps.mk === building 'deps.mk' on host xxx echo ' touch $@' >> deps.mk === building 'deps.mk' on host xxx distmake: *** No rule to make target `file', needed by `all'. Stop. If I perform a make/distmake deps.mk first, it works. It seems that distmake does not see that it should re-process the whole Makefile after the generation of deps.mk. The standard make command works, even when using -j. I suppose it is a bug in the handling of remote jobs, maybe generic in make, but I am not familiar with the internals of make. Maybe someone can give me some hints about how to fix this issue? Thanks, Christophe. |
From: simon <si...@mu...> - 2006-02-15 02:37:37
|
Hi, Just a quick confirmation that it does work with non-root user - I know shouldn't be running as root... 1/2 way through setting up machine and ideas on how I want it so too lazy to make user :-) Some performance notes: building distmake source (after having run ./configure) and running make clean in between. Files served via NFS from fast machine on 10Mbit-T Client (slow P233) pushing jobs to 2 servers (2 entries of same Dual P3-500) time distmake -j 1 = real 41.130s time distmake -j 2 = real 22.474s time distmake -j 10 = real 23.819s Building on Client (slow P233) over NFS time make -j 1 = 2m 55.69s Building on Server (Dual P3-500) as local job time make -j 1 = 40.595s time make -j 2 = 21.835s So distmake runs a little slower in this situation, which is expected as there is a little NFS serving going on as well. The benefits should really come as I add other machines to the collection of servers - almost tempted to take bootable linux CD the office machines just to see how fast I can go......... Cheers, Simon. PS. It got a little confused when local/remote UID's where not synchronised. On Tue, Feb 14, 2006 at 10:16:09AM +0100, Bruno Fleisch wrote: > Hello Simon, > > I get the same error here, when running distmake under the root account. As a > temporary fix, you may try to run it under a non-privileged user. > |
From: Bruno F. <bfl...@us...> - 2006-02-14 09:16:13
|
Selon simon <si...@mu...>: > Hi, > attached is a really simple Source/Makefile and the resultant debug fro= m > a run of 'distmake --debug=3Dr > debug.txt 2>&1' > [...] Hello Simon, I get the same error here, when running distmake under the root account. = As a temporary fix, you may try to run it under a non-privileged user. I hope to find the root cause of this bug this week. I'll keep you update= d. Regards, Bruno |
From: simon <si...@mu...> - 2006-02-13 20:56:42
|
Hi, attached is a really simple Source/Makefile and the resultant debug from a run of 'distmake --debug=3Dr > debug.txt 2>&1' And here's data from starting a rpc.bldserver from the command line --- twopak:/home/test# rpc.bldserver=20 + 15229 988 twopak:/home/test# lsof -p 15229 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME rpc.bldse 15229 root cwd DIR 3,1 1024 2 / rpc.bldse 15229 root rtd DIR 3,1 1024 2 / rpc.bldse 15229 root txt REG 3,5 79752 1474 /usr/local/bin/rpc.bldserver rpc.bldse 15229 root mem REG 3,1 90248 84353 /lib/ld-2.3.2.so rpc.bldse 15229 root mem REG 3,1 22940 32149 /lib/tls/librt-2.3.2.so rpc.bldse 15229 root mem REG 3,1 1254468 32134 /lib/tls/libc-2.3.2.so rpc.bldse 15229 root mem REG 3,1 78233 32147 /lib/tls/libpthread-0.60.so rpc.bldse 15229 root 3u IPv4 74088 TCP *:988 (LISTEN) twopak:/home/test# netstat -an | grep 988 tcp 0 0 0.0.0.0:988 0.0.0.0:* LISTEN =20 --- Do I need anything else installed in order to get RPC working? Anything else I can do to help debug??? Simon. On Mon, Feb 13, 2006 at 10:11:08AM +0100, Bruno Fleisch wrote: > Selon simon <si...@mu...>: >=20 > > Hi all, > > I've had a go at running distmake on Debian Sarge. Built OK, installs > > and runs. However there is the following error and the actual > > compilation runs, but on the local client not the remote servers. > > > > Error is: > > --- > > bldclient_rexec() failed (err=3D-2). > > --- >=20 > Hello Simon, >=20 > Most probably, bldclient_rexec() failed because distmake was unable to > communicate with rpc.bldserver. >=20 > Would you please try to run distmake with debugging option (add --debug= =3Dr) > against a very simple Makefile ? >=20 > You could also try to run rpc.bldserver manually. Once started, it should= output > something like "+ <pid> <tcp port>". What does "lsof -p <pid>" say ? >=20 > Thanks, >=20 > Regards, >=20 > Bruno >=20 >=20 > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log fi= les > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=103432&bid#0486&dat=121642 > _______________________________________________ > Distmake-general mailing list > Dis...@li... > https://lists.sourceforge.net/lists/listinfo/distmake-general |
From: Bruno F. <bfl...@us...> - 2006-02-13 09:11:12
|
Selon simon <si...@mu...>: > Hi all, > I've had a go at running distmake on Debian Sarge. Built OK, installs > and runs. However there is the following error and the actual > compilation runs, but on the local client not the remote servers. > > Error is: > --- > bldclient_rexec() failed (err=3D-2). > --- Hello Simon, Most probably, bldclient_rexec() failed because distmake was unable to communicate with rpc.bldserver. Would you please try to run distmake with debugging option (add --debug=3D= r) against a very simple Makefile ? You could also try to run rpc.bldserver manually. Once started, it should= output something like "+ <pid> <tcp port>". What does "lsof -p <pid>" say ? Thanks, Regards, Bruno |
From: simon <si...@mu...> - 2006-02-10 19:45:50
|
Hi all, I've had a go at running distmake on Debian Sarge. Built OK, installs and runs. However there is the following error and the actual compilation runs, but on the local client not the remote servers. Error is: --- bldclient_rexec() failed (err=-2). --- rpc.bldserver is started on remote machine, although does not die after diskmake completes. It does die is diskmake is Ctrl-C'ed. --- twopak:/home/distmake-0.2# ps ax| grep rpc.bldserver 2807 ? Ss 0:00 /usr/local/bin/rpc.bldserver 2813 ? Ss 0:00 /usr/local/bin/rpc.bldserver --- although they don't appear to have any ports open when I 'netstat -a'. I am using ssh as the remote shell. --- export DISTMAKE_RSH=/usr/bin/ssh --- This occurs both in normal (different client and server machines) and with a loopback (same machine as client and server). I was thinking that distmake could be a way to fix makefiles that don't make use of SMP multi-processors (same machine as client and server)..... Should this work? Cheers, Simon. |
From: Bruno F. <bfl...@fr...> - 2006-01-20 20:52:47
|
bfl...@fr... wrote: >>Hello, >> >>I'm using distmake to facilitate the processing of large datafiles across >>many machines. The makefile has a simple rule to go from preprocessed to >>postprocessed (processing is all of the form: zcat $source | >>processing.script | gzip > $target). When I Ctrl-C to kill the distmake, >>the child processes don't seem to clean up in a timely manner. Is this a >>known bug? >> >> > > >Yes, that have not been implemented yet (but put on my todo list). I'll keep you >updated. > > This is now available on the CVS repository. Would you please try out and provide me with your feedbacks? Thanks! Regards, Bruno |
From: <bfl...@fr...> - 2006-01-19 10:18:43
|
> Hello, > > I'm using distmake to facilitate the processing of large datafiles acro= ss > many machines. The makefile has a simple rule to go from preprocessed = to > postprocessed (processing is all of the form: zcat $source | > processing.script | gzip > $target). When I Ctrl-C to kill the distmak= e, > the child processes don't seem to clean up in a timely manner. Is this= a > known bug? Yes, that have not been implemented yet (but put on my todo list). I'll k= eep you updated. > Regards, > D. Adams Bruno |
From: D. A. <ad...@ho...> - 2006-01-18 22:30:04
|
Hello, I'm using distmake to facilitate the processing of large datafiles across many machines. The makefile has a simple rule to go from preprocessed to postprocessed (processing is all of the form: zcat $source | processing.script | gzip > $target). When I Ctrl-C to kill the distmake, the child processes don't seem to clean up in a timely manner. Is this a known bug? Regards, D. Adams |
From: Bruno F. <bf...@fr...> - 2006-01-12 15:35:51
|
Hello Kostya, Konstantin 'KCC' Serebryany wrote: > Hi All, > > I tried to use ditsmake and I get warning like this: > /usr/bin/X11/xauth: error in locking authority file > /home/ksserebr/.Xauthority Do you use X applications inside your Makefile(s) ? If not, you may try to unset the environment variable XAUTHORITY before starting distmake. > > Distmake works for me on small applications (30-50 files) but for huge > application (>1000 files) distmake /sometimes/ fails to synchronize > correctly. It may be related to the mentioned warning. Which distmake release do you use ? The CVS version has major improvements concerning nested makefiles. Regards, Bruno > > I would appreciate any suggestion, > > Thanks, > > Kostya |
From: Konstantin 'K. S. <kon...@gm...> - 2006-01-12 14:46:32
|
Hi All, I tried to use ditsmake and I get warning like this: /usr/bin/X11/xauth: error in locking authority file /home/ksserebr/.Xauthority Distmake works for me on small applications (30-50 files) but for huge application (>1000 files) distmake *sometimes* fails to synchronize correctly. It may be related to the mentioned warning. I would appreciate any suggestion, Thanks, Kostya |