Sourcer - The source friend Code
Brought to you by:
felipec
Menu
▾
▴
Tree [045641] master v0.5.1 / History
Read Me
sourcer - source installer. What the hell is sourcer? ========================= Basically sourcer is two things: 1. Source installer: You have the source tarballs, sourcer installs them. 2. System builder: It creates a basic linux system (distro). Quick start: =========== Necesary: bzip2-devel Recommended: libxml-devel (not necesary) * Edit pst.conf * Run './configure' * Run 'make install' Optional: as an alternative to "make install" you can copy pst.conf to /etc and pst to /usr/bin... again pst.conf -> /etc, pst -> /usr/bin That's it, if you want to test it run: pst [options] package Configuration ============= The general configuration is done editing /etc/pst.conf and you can edit it before or after installation. (before installation pst.conf is in the main dir) INSTALL: It should contain einstall<num> where <num> is the installation method you want to use, if you have xml "einstall2" will be the best, but if not, "einstall3" should be the best, then comes "einstall1". LFS: If you just want to install packages then LFS="/" is ok, if you want to build a whole system then LFS should point to where the new system is mounted. LFSP: Should point to where the tarballs and patches are, for example, it could be /home/user/downloads. There is no default that can fit, so you _need_ to modify it. LFSB: Is the place where you want to build everything, like extract the sources, issue configure, make, etc. The default, /usr/src, is the most commonly used, but you can change it to wherever you want. If you are going to build a system then /static/build will be the best, that's because it is used only for the static stage of the build, check out README.sys_build. In either case the real place where everything will be build is not $LFSB, but $LFS/$LFSB (inside the LFS dir). sourcer_dir: Should point to where sourcer information is, "/usr/src/sourcer" is ok if you are reading this from there, "/usr/share/sourcer" is another good option, it depends on where you want to have the information. OTHER: You can change the default CFLAGS if you want. LIGHT_CLFAGS is only for glibc, and INSANE_CFLAGS is still not used, but probably will be used for mozilla. That should be it, now you should be able to type "pst <package>" and see while everything is done :) Why sourcer ? ============= Well because there is no good source management systems today, everything is based on stupid binary packages that cuts the options of the users. So here come sourcer. You might ask why in bash, well basically it's beacuse it's simple. It doesn't require complex stuff, just regex and some small C proggies that help and it works very well. I've tryied some other languages, but it's awesome how always bash seems to be the best, don't ask me why, but I always get stuck, and when I solve the poblem it comes to be a complex ugly solution, enought to trash the advance, so bash seems to be the best. Anyway, I'm thiking on modularizing sourcer a little bit more so not everything is done in bash, I alredy have some small C programs, and I will keep thinking on ways to organize everything, the bash code is getting pretty ugly :P Workings ======== 1. Parsing or geting information Currently sourcer tries to find all the information it can and guess everything else. In order to do that there are some files that have information and some configurations that tell sourcer where to get information. Ok, the main file is /etc/pst.conf here global stuff like where the directory where packages reside, the dir where sourcer reside, where the systesm is, etc. Then comes the tricky part, the guessing. The parser is a complex code that basically joins all the information and doesn't have a specific sequence of steps, I usually change them, split them and such. But basically information is obtaned from here: * info/profiles * info/groups * $packages_dir (specified at /etc/pst.conf) * info/setup * info/general * info/versions * info/opts.* The info/profiles file insisde the sourcer dir. Here you can have some special information that can't be easily found by sourcer and it's related to packages, patches or that stuff. Why only that stuff? well, all the information gathering it's a process and if we place only that information here the system works fine :/ info/general is somewhat like info/profiles the difference is that here we only have variables, and this variables doesn't have much to do with packages as info/profiles. Why? as I said, it's because the fine process of gathering information, but basically all that infomation is divided so before we get into info/general sourcer can get some basic information with the packages as stuff like that, it gets the version, the main tar, etc. With that information you can do ROOT_DIR=vim-$VERSION inside info/general, and $VERSION is guessed from the source tarballs, so it's defined _after_ processing info/profiles, see? info/groups is a file that contain information about groups and subgroups, you can place your package in any group you want and each group can share some behaviors, for example I want everything under the gnome group to go into /opt/gnome, a simple line in info/groups says that. Subgroups are the later part in package.subgroup, like bash.static, the subgroup is static. In $packages_dir (you specify this at /etc/pst.conf) you basically have all you packages, in subdirectories, directly there, etc. it doesn't matter. Sourcer will always find the latest, and much better if you don't want the latest, like you have linux-2.5.x there but you want only linux-2.4.x, then you can define a variable like VERSION_TOP=2.5 in info/profiles, take a look there to see how. info/versions basically just defines the version of a list of packages, the latest I know there are, simply because you might have an older version, or might be you don't have a package at all, if sourcer have the url from where to get that package then it will try to download it, so it needs to know the latest version. info/opts.* are a series of files mainly to support a very fancy feature. Let's supose you have "[grep]:--prefix=/usr --bindir=/bin" in opts.config, this will define a variable config_opts="--prefix=/usr --bindir=/bin" so you can use it inside your script to install that package. All this was done so users that will build a new system from scratch can easily change the options they want. By default the config step does $config_default, and $config_default usually is "./configure $config_opts". 2. Setup instructions info/setup is were the "setup instructions" are obtained. There are different 3 methods to get "setup instructions" basically because not everyone have xml, or someones might want to write setup instrucions in plain shell scripts, that's all ok. So we have 3 different formats available and the 3 should get things done because all the instructions are autogenerated by some tools under the tools/ directory. The methods or formats are: 1. sh Plain shell scripts 2. xml Shell commands but under xml tags to divide steps and commands 3. steps Shell scripts but divided into steps and commands As you can probably see "xml" and "steps" divide stuff, "sh" doesn't. Does that matter? sure, having stuff divided can let you see the progress in a nice way, command by commnd, step by step, with "sh" you just see the whole shell script on the screen and you don't know where is it runing. There is also another advantage of dividing stuff, you can resume from a certain point, let's say the installation halted at install because there is a typo, just fix it and run it again, it will continue from where it left, with "sh" it's going to start of the begining of the installation. And now the difference between "xml" and "steps". Basically "xml" format generates "steps" format, xml->steps, and "steps" is written as shell scripts. They act exactly the same, but "steps" is very hard to write and modify, in fact it's not inteded to be writen, it should be autogenerated from "xml", which is simpler to write, but that doesn't mean you can't write "steps" manually, it's just very hard. 3. Execution Now that we've obtained all the information it's time for the important task here, installation. Basically with all the information obtained we can generate the necesary commands to reach our goal, untar, patch, configure, etc. We now have the elements to install our packages, but what is going to happend if something fails? Everything is saved, in $build_dir/pend (usually build_dir=/usr/src) it's stored the installations that failed and at which step, so you can later resume it. If you have left the installation you can see the result in the last line at $build_dir/list, and if you specify the -C option then everything listed in $build_dir/list will not be installed again. That's usefull only when you are building a system from scratch (distro), or when you are insalling a list of packages, I'll describe this mode of operation. Installing a list of packages is done using the -L option, if you do "pst -L gnome" then sourcer will look in info/lists for the gnome list, and install all the packages, the progress is saved in $build_dir/gnome.list, and then if you specify the -C option (continue), you can resume the build from the next package that didn't got it to the listsing at $build_dir/gnome.list, so it was not installed. The output of the whole installation is logged at $build_dir/log, and there is a directory for each package, and then a list of files showing what happened at each step. Note here that if you run with "sh" method then you'll not see very much division, just a big log. 4. Conclusions There is more stuff, sure, and more tricks to get things done, basically you'll need to do "pst" or "man pst" to check some nice options and "/etc/pst.conf" for some configurations. If you take enought time you'll love this system, and your productivity at compiling stuff will increase a lot, believe me :) Inspiration: ============ Sourcer ideas come from a very wide variety of projects, most of them with great ideas. Linux From Scratch <http://www.linuxfromscratch.org> Automated Linux From Scratch <http://automated.linuxfromscratch.org> Sorcerer <http://sorcerer.wox.org> Rock Linux <http://www.rocklinux.org> RALFS fnyx Feedback: ========= Feedback will be very much appreciated. Felipe Contreras <al593181@mail.mty.itesm.mx>
