Menu
▾
▴
devil-linux-develop
|
From: Andrzej O. <an...@ma...> - 2008-05-27 16:23:02
|
Hi,
I have two suggestions:
1. In build configuration option CONFIG_PMAKE in make menuconfig has
a limit of 2. I observed on my Core Quad that compilation is faster
when CONFIG_PMAKE=4. I think, better is set this limit to 4.
2. In default kernel config CONFIG_SCSI_MULTI_LUN is traditionally unset.
In past SCSI devices with multiple Logical Unit Numbers was very rare,
legacy rather, as created for multiple MTUs connected to single controller.
But now SCSI logic is used to all USB drives and there are pen drive's
with flash card reader and separate built-in memory (i.e. Kingston
DataTraveler Reader) as separate LUNs. If this kernel option is unset,
second LUN is invisible for Devil-Linux an it is impossible boot from
this flash card and read configuration from it.
Especially, in Windows devices like this are supported without problems.
So I suggest to set this option in default kernel configuration
in Devil-Linux to CONFIG_SCSI_MULTI_LUN=y
Regards
--
Andrzej Odyniec
|
|
From: Heiko Z. <he...@zu...> - 2008-05-27 18:09:18
|
Quoting Andrzej Odyniec <an...@ma...>: > Hi, > > I have two suggestions: > > 1. In build configuration option CONFIG_PMAKE in make menuconfig has > a limit of 2. I observed on my Core Quad that compilation is faster > when CONFIG_PMAKE=4. I think, better is set this limit to 4. Many programs have trouble with this, that's why we limited it to 2. DL 1.3 has a better options which is much more stable. > 2. In default kernel config CONFIG_SCSI_MULTI_LUN is traditionally unset. > In past SCSI devices with multiple Logical Unit Numbers was very rare, > legacy rather, as created for multiple MTUs connected to single > controller. > > But now SCSI logic is used to all USB drives and there are pen drive's > with flash card reader and separate built-in memory (i.e. Kingston > DataTraveler Reader) as separate LUNs. If this kernel option is unset, > second LUN is invisible for Devil-Linux an it is impossible boot from > this flash card and read configuration from it. > > Especially, in Windows devices like this are supported without problems. > > So I suggest to set this option in default kernel configuration > in Devil-Linux to CONFIG_SCSI_MULTI_LUN=y good idea -- Regards Heiko Zuerker http://www.devil-linux.org ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program. |
|
From: Andrzej O. <an...@ma...> - 2008-05-27 20:04:42
|
Heiko Zuerker wrote: >>1. In build configuration option CONFIG_PMAKE in make menuconfig has >> a limit of 2. I observed on my Core Quad that compilation is faster >> when CONFIG_PMAKE=4. I think, better is set this limit to 4. > Many programs have trouble with this, that's why we limited it to 2. > DL 1.3 has a better options which is much more stable. I'm not sure, we are talking about this same? I think about build process of last DL 1.3 development and about setting in "Build Configuration" | "Parallel Compile Jobs" which is recorded as CONFIG_PMAKE option in build/.config and limits number of cc compilations simultaneously. This option is transferred mainly to -j parameter of make command when compiling kernel or packages. I understand that more then 2 cc parallel processes on single CPU enhances process scheduling overdraft and sometimes is slower, than one process after one. And 4 parallel processes on single CPU can be much slower. But on 4 CPU cores in compiling/building machine if I set CONFIG_PMAKE option to 4, complete build from unpack to iso take 2h30' but if this option allows only 2 cc processes simultaneously, total time is over 3h. I unterstand difference between one and two processes, because in parallel computing coprocesses must be independent or synchronised. But if You allow for 2 parallel processs, what difference for 4 parallel cc's? I think cc processes are from definition independent. But, if You are sure You know about negative experience with parallel compilations, OK. I accept this. I work in computer science from 70s (beginning from CDC Cyber 7000) and I know many miracles in computing. :-) Regards Andrzej Odyniec |
|
From: Heiko Z. <he...@zu...> - 2008-05-27 20:53:06
|
Hey, it's mostly the Makefiles of the various packages which don't like to be compiled with -jX We introduced a new parameter in DL1.3 CONFIG_PARALLEL_JOBS I have this set to 4 on my dual core box, which make it run quite nicely. It basically let's the DL build system run X scripts in parallel, which seems to work better. Every once in a while I have a script fail, I just restart it and keep going. Oh and in order to use this functionality you have to use a new makefile parameter: make mrproper unpack prepare <- as always make makefile <- instead of "build install iso" -- Regards Heiko Zuerker http://www.devil-linux.org Quoting Andrzej Odyniec <an...@ma...>: > Heiko Zuerker wrote: >>> 1. In build configuration option CONFIG_PMAKE in make menuconfig has >>> a limit of 2. I observed on my Core Quad that compilation is faster >>> when CONFIG_PMAKE=4. I think, better is set this limit to 4. >> Many programs have trouble with this, that's why we limited it to 2. >> DL 1.3 has a better options which is much more stable. > I'm not sure, we are talking about this same? I think about build process of > last DL 1.3 development and about setting in "Build Configuration" | > "Parallel > Compile Jobs" which is recorded as CONFIG_PMAKE option in build/.config and > limits number of cc compilations simultaneously. This option is transferred > mainly to -j parameter of make command when compiling kernel or packages. > > I understand that more then 2 cc parallel processes on single CPU enhances > process scheduling overdraft and sometimes is slower, than one process after > one. And 4 parallel processes on single CPU can be much slower. > > But on 4 CPU cores in compiling/building machine if I set CONFIG_PMAKE option > to 4, complete build from unpack to iso take 2h30' but if this option allows > only 2 cc processes simultaneously, total time is over 3h. > > I unterstand difference between one and two processes, because in parallel > computing coprocesses must be independent or synchronised. But if You allow > for 2 parallel processs, what difference for 4 parallel cc's? I think cc > processes are from definition independent. > > But, if You are sure You know about negative experience with parallel > compilations, OK. I accept this. I work in computer science from 70s > (beginning from CDC Cyber 7000) and I know many miracles in computing. :-) > > Regards > > Andrzej Odyniec > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > Devil-linux-develop mailing list > Dev...@li... > https://lists.sourceforge.net/lists/listinfo/devil-linux-develop > ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program. |
|
From: Andrzej O. <an...@ma...> - 2008-05-27 22:58:51
|
Heiko Zuerker wrote: > it's mostly the Makefiles of the various packages which don't like to > be compiled with -jX > > We introduced a new parameter in DL1.3 CONFIG_PARALLEL_JOBS > I have this set to 4 on my dual core box, which make it run quite nicely. > It basically let's the DL build system run X scripts in parallel, > which seems to work better. > Every once in a while I have a script fail, I just restart it and keep going. > > Oh and in order to use this functionality you have to use a new > makefile parameter: > make mrproper unpack prepare <- as always > make makefile <- instead of "build install iso" Now all is clear for me. So I try it now. I will set -j1 , JOBS to 8 and run full process again this night. Tomorow I will check total time and will report. It is thinkable, some compilation phases are dependent and must be done one aftter one... but this isn't elegant Thanks, Heiko Andrzej Odyniec |
|
From: Heiko Z. <he...@zu...> - 2008-05-28 13:53:31
|
Once the compile fails, simply try to continue with "make makefile" again. Sometimes the scripts fail for some weird reason, I think procmail was one of the ones which like to do that. -- Regards Heiko Zuerker http://www.devil-linux.org Quoting Andrzej Odyniec <an...@ma...>: > Heiko Zuerker wrote: >>> Oh and in order to use this functionality you have to use a new >>> makefile parameter: >>> make mrproper unpack prepare <- as always >>> make makefile <- instead of "build install iso" > I wrote: >> Now all is clear for me. So I try it now. I will set -j1 , JOBS to 8 and run >> full process again this night. Tomorow I will check total time and will >> report. It is thinkable, some compilation phases are dependent and must be >> done one aftter one... but this isn't elegant > > So... there is no success. After one hour there are build errors. Maybe > dependences between packages are unhappy when we do parallel builds? > > I will reduce JOBS to 4 (as You have) and repeat build process, first time > from mrpropper and if no success, will try again from clean system. > > Regards > > Andrzej Odyniec > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > Devil-linux-develop mailing list > Dev...@li... > https://lists.sourceforge.net/lists/listinfo/devil-linux-develop > ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program. |
|
From: Heiko Z. <he...@zu...> - 2008-05-28 14:54:12
|
Quoting Andrzej Odyniec <an...@ma...>: > Heiko Zuerker wrote: > >> Once the compile fails, simply try to continue with "make makefile" again. >> Sometimes the scripts fail for some weird reason, I think procmail was >> one of the ones which like to do that. > > I done it some number of times and after resigned. Now process is building > ntop. Maybe 4 jobs is better than 8? Maybe conflicting dependences are not so > deep? There may be some dependencies we were not aware of or 2 script trying to do the same thing at the same time. If you identify a 'problem child' let us know, maybe we can re-arrange a few things. -- Regards Heiko Zuerker http://www.devil-linux.org ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program. |
|
From: Andrzej O. <an...@ma...> - 2008-05-28 13:39:27
|
Heiko Zuerker wrote: >>Oh and in order to use this functionality you have to use a new >>makefile parameter: >>make mrproper unpack prepare <- as always >>make makefile <- instead of "build install iso" I wrote: > Now all is clear for me. So I try it now. I will set -j1 , JOBS to 8 and run > full process again this night. Tomorow I will check total time and will > report. It is thinkable, some compilation phases are dependent and must be > done one aftter one... but this isn't elegant So... there is no success. After one hour there are build errors. Maybe dependences between packages are unhappy when we do parallel builds? I will reduce JOBS to 4 (as You have) and repeat build process, first time from mrpropper and if no success, will try again from clean system. Regards Andrzej Odyniec |
|
From: Andrzej O. <an...@ma...> - 2008-05-28 14:32:07
|
Heiko Zuerker wrote: > Once the compile fails, simply try to continue with "make makefile" again. > Sometimes the scripts fail for some weird reason, I think procmail was > one of the ones which like to do that. I done it some number of times and after resigned. Now process is building ntop. Maybe 4 jobs is better than 8? Maybe conflicting dependences are not so deep? Andrzej Odyniec |
|
From: Andrzej O. <an...@ma...> - 2008-05-28 16:43:28
|
Heiko Zuerker wrote: > Quoting Andrzej Odyniec <an...@ma...>: >>Heiko Zuerker wrote: >>>Once the compile fails, simply try to continue with "make makefile" again. >>>Sometimes the scripts fail for some weird reason, I think procmail was >>>one of the ones which like to do that. >>I done it some number of times and after resigned. Now process is building >>ntop. Maybe 4 jobs is better than 8? Maybe conflicting dependences are not so >>deep? > There may be some dependencies we were not aware of or 2 script trying > to do the same thing at the same time. > If you identify a 'problem child' let us know, maybe we can re-arrange > a few things. Yes. With 4 parallel jobs all build process go without errors and created iso. But effect is the same, as I tried few days ago: >> VFS: Mounted root (ext2 filesystem) readonly. >> Freeing unused kernel memory: 428k freed >> >> ******************************************************************************** >> * * >> * Welcome to Devil-Linux * >> * * >> ******************************************************************************** >> http://www.devil-linux.org/ >> >> Version 1.3.5-2008-05-28 >> Kernel 2.6.25.4 >> Mounting SHM FS on /shm >> waiting until usb-storage driver has initialized all devices ... >> Creating devices in /dev >> >> Searching for configuration media >> Checking for "etc-mods.tar.bz2" on "/dev/sda" ... mount failed >> Checking for "etc-mods.tar.bz2" on "/dev/sda1" ... success! >> loading configuration >> Searching for Devil-Linux CD-ROM >> Search list: /dev/hdb >> checking /dev/hdb Found on /dev/hdb >> Mounting SHM FS on /cdrom/shm >> Unmounting proc >> Starting up final system... >> aufs 20080519 ...and all hangs. If I pull USB pendrive, there is no difference. Heiko, how do You think, for what it waits in this momment? Regards Andrzej Odyniec |
|
From: Heiko Z. <he...@zu...> - 2008-05-28 17:35:14
|
Quoting Andrzej Odyniec <an...@ma...>: > Heiko Zuerker wrote: >> Quoting Andrzej Odyniec <an...@ma...>: >>> Heiko Zuerker wrote: >>>> Once the compile fails, simply try to continue with "make makefile" again. >>>> Sometimes the scripts fail for some weird reason, I think procmail was >>>> one of the ones which like to do that. >>> I done it some number of times and after resigned. Now process is building >>> ntop. Maybe 4 jobs is better than 8? Maybe conflicting dependences >>> are not so >>> deep? >> There may be some dependencies we were not aware of or 2 script trying >> to do the same thing at the same time. >> If you identify a 'problem child' let us know, maybe we can re-arrange >> a few things. > > Yes. With 4 parallel jobs all build process go without errors and > created iso. > But effect is the same, as I tried few days ago: > >>> VFS: Mounted root (ext2 filesystem) readonly. >>> Freeing unused kernel memory: 428k freed >>> >>> ******************************************************************************** >>> * >>> * >>> * Welcome to Devil-Linux >>> * >>> * >>> * >>> ******************************************************************************** >>> http://www.devil-linux.org/ >>> >>> Version 1.3.5-2008-05-28 >>> Kernel 2.6.25.4 >>> Mounting SHM FS on /shm >>> waiting until usb-storage driver has initialized all devices ... >>> Creating devices in /dev >>> >>> Searching for configuration media >>> Checking for "etc-mods.tar.bz2" on "/dev/sda" ... mount failed >>> Checking for "etc-mods.tar.bz2" on "/dev/sda1" ... success! >>> loading configuration >>> Searching for Devil-Linux CD-ROM >>> Search list: /dev/hdb >>> checking /dev/hdb Found on /dev/hdb >>> Mounting SHM FS on /cdrom/shm >>> Unmounting proc >>> Starting up final system... >>> aufs 20080519 > > ...and all hangs. If I pull USB pendrive, there is no difference. > Heiko, how do You think, for what it waits in this momment? Oh I have the same issue.... ;-) Seems like the latest aufs has problems with the latest kernel. Here's a workaround for now (forcing the use of unionfs): edit the file tmp/ISO/cdtree/sbin/pre_init comment out this line: mount -n -t aufs -o br:/shm/etc-mods:/etc-cd=rr none /shm/etc and remove # from this line: mount -n -t unionfs -o dirs=/shm/etc-mods:/etc-cd=ro none /shm/etc then delete the files tmp/.done_iso* and do a "make makefile" -- Regards Heiko Zuerker http://www.devil-linux.org ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program. |
