[a5f752]: README Maximize Restore History

Download this file

README    268 lines (198 with data), 12.5 kB

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
Smoothwall Express Build System

OVERVIEW

The build processes consists of 3 main steps:

1. Construct a temporary "tools" environment using the hosts toolchain.  A
symlink in the host (/tools) ensures that all programs, when a "make
install" is done on them, end up in the temporary tree and not in the host
system.  This symlink is created automatically.

2. Compile all packages that will end up in the final installation
smoothwall.tgz, plus any programs and libraries that are required to compile
those programs.  This stage is done using the temporary tools environment
and is chrooted into the working build environment.  Before building the
"core" packages, such as the kernel etc, the build toolchain that will be
used in the final build is compiled up with the /tools toolchain, thus the
/tools structure is no longer needed.  Packages that are compiled and
installed are placed into the build environment (i.e. under distrib/).  Also,
each compiled package is tarred up into a tarball that is used in step 3.

3. Create the last few tools needed to create the final bootable ISO (such
as mkisofs, and grub).  Then build the smoothwall.tgz, by untarring the
packages compiled in step 2 into a new directory.  Smoothwall specific files
are added from the "coretree" package.  Finally build the bootdisk images
and the ISO itself.

An optional fourth step is to prepare a bootable flash drive to install without
an optical drive. This process copies the contents of the ISO to the flash,
make a miniscule adjustment to grub.conf, then installs grub to make the flash
bootable. It should work just as well on an external rotating drive, but this
has not been tested.


LAYOUT

The main working directory is smoothwall/distrib.  Once an initial build
has been completed, this dir can be chrooted into and it will be possible to
work on the tree, add new packages, etc, etc.  expresscore itself can be
used to hold documentation and other files that wont be used in the build
environment itself.

smoothwall/distrib/tools contains the temporary (toolchain) build
environment.  Even after a completed build, you should leave this dir alone
because some build scripts have to reference files in it because they are
normally called before step 2 is complete.  It shouldn't ever be necessary
to change how this tools dir works, so just ignore it unless you are REALLY
interested.

smoothwall/distrib/build is where the "action" is.  This dir contains
the build top level makefile and directories buildroot, crumbs, downloads,
sources, target, and toolcrib.

build/Makefile is the top level makefile.  This controls the entire build process.

build/buildroot is where each compiled package's files and directories are
assembled in a faux root dir so that their tarballs (archives) can be unpacked
directly to the target root dir.

build/crumbs contains a number of 'bread crumbs' and log files related to
the build system itself. For the most part, the 'bread crumbs' are empty
files that the build system uses to keep track of where it is in the build
process. The crumbs allow the build process to be re-entrant while also
ensuring that each package is always built from scratch.

build/downloads contains all of the downloaded source tarballs as well as a
flag file for each tarball to indicate that the tarball doesn't need to be
fetched.

build/target contains all of the final compiled/assembled package tarballs that
will be used to prepare the installation media. build/target/isos contains
the ISO 9660 images that users will download, burn to CD/DVD, and install to
load their own Smoothwall Express firewalls.

build/toolcrib contains all of the top-level scripts used to check, prepare,
compile, build and assemble the system. It contains scripts that install the
necessary packages on a standard GNU/Linux distribution so that it can build
Smoothwall Express. There is a script used to build the three phases of the
toolchain and a script used to build the final phase of the system. There are
a number of scripts used to adjust the toolchain during the build process
and to verify its correct operation.

build/sources is the real heart of the build process.  Each dir is a
package, including some smoothwall-specific ones, like libsmooth+setup for
the setup program, setuids for the little C 'set UID root' programs, etc.
Each dir contains
a Makefile that must have specific targets, such "download".  The clever
part is it is possible to use ready made targets, by including the
Makefile.rules in the dir above.  This has targets for configuring,
compiling, installing and creating the package archive.  For gnu-compliant,
./configure compatible packages it is possible to make a makefile to build
the package in only a few lines, including the download URL.  See the
sources/sed/Makefile for a good example.

Two packages are worth extra attention: coretree is a package that builds a
tarfile tree of all the extra files needed by smoothie, such as CGIs etc. 
It does this by copying them out of sources/coretree/tree into a temp dir,
before taring up the temp dir.  So if you want to add static files to the
distrib, add them to this tree.  Symlinks and permissions are not directly
set in this tree - they are created and set by the Makefile, scripted
fashion.

Finally, build/sources/media is where the bootdisks and ISO are created.  The
Makefile loops copying compiled packages from the target dir.  Add
additional packages to the list as needed.  This makefile also removes
unneeded stuff, such as /usr/include, from the final smoothwall.tgz, as well
as running strip on directories expected to hold binaries.  The final ISO
will end up in this build/target/iso dir.


THE TOOLCRIB

The toolcrib contains nearly all of the scripts and utilities used to build
Smoothwall. In alphabetical order, they are:

as_chroot - enable the build system to perform specific operations as root
(the super user) in a change-root (chroot) jail. This allows the user to
execute the build as a non-root user, thus reducing the chance the build system
will damage the host system.

distrib_root - a one-liner that returns the absolute path of the distrib dir's
parent. This is part of what enables the system to be built almost anywhere in
the host's filesystem(s).

dlverify - this script is used to perform that actual download of each package's
source tarballs. It defaults to looking in the Smoothwall Express project's
archive for them, but will fall back to the upstream insternet sources as needed.
There are provisions to use different internet and local collections. There are
a few options available to control other aspects of its operation. Please read the
script for all the details.

[From the original SWE3.0 README] "Download and verify".  This is a little helper script that wraps
wget.  The purpose of the wrapper is two-fold: to stop wget being called on
a completed download, thus causing unnecessary network traffic, and also to
verify a completed download by MD5.  The arguments are: <directory> <url>
[md5].  Directory is usually the standard d/l area.  The MD5 can be left
off, in which case no download check is done.  To cope with partial
downloads, once a d/l has completed, an additional touch file will be
created with the same name as the download file, plus ".done".  Only if this
file is present will the download be skipped over.  Note that the
Makefile.conf (in sources) defines DL_CMD for you, and it includes the
download destination dir.  So to do a download from within a makefile, you
should usually just do: $(DL_CMD) http://...  This script also has the
ability to use a local mirror.  The URL root for the mirror is set at the
top of the script.  To create a local mirror for yourself, after doing a
full build, copy the files in build/download (except the .done files) into
your mirror.

environment - contains only the bash function 'set_build_environment'. This function
ensures that each phase of the build gets the precise environment it needs in order
to build its bits correctly.

final_tc_adjustment - this performs the final adjustments to the toolchain after
glibc is built during the final phase. It also contains a provision to unwind
(reverse) the adjustment in case the adjustment fails; it's also used to re-prepare
the build tree so that the final phase can be rebuilt.

final_tc_check - this performs a number of sanity checks on the toolchain after gcc
has been built during the final phase. This ensures that the build process stops
if the toolchain has not been properly built.

functions - it contains a number of low- and mid-level functions used during the
build process. These functions run adjustments, package builds and toolchain build
phases; they check results for sanity and also determine how many CPUs/cores are
available for the build.

handy_tools - contains a few very handy commandline tools: gitstat (find what you've
changed), monbuild (monitor the progress of a package's build), monloop (monbuild
in a convenient loop) and redo (rebuild specific final phase packages without
dealing with the trail of breadcrumbs).

host_check.sh - checks the host environment for all of the requisite tools needed
to build. This is done first; if the right stuff isn't found, the build fails
immediately.

host_debian_inst, host_gentoo_inst - these scripts install any missing requisite
packages on the respective host systems. They should ensure that host_check.sh
never fails.

make_final - this script is run in a chroot jail as root. It handles building the
entire final phase of the build.

make_flash - converts the assembled ISO onto a USB thumb drive and makes it bootable.

make_media - assembles the install, development and offroad ISOs.

make_toolchain - builds the toolchain needed to build the final phase.

prep_final_env - prepares the environment for script prepare_final_tree.

securetree - turns off user and group 'sticky' bits on all files and dirs in the
specified dir.

striptree - strips symbol tables &cet. from programs, libraries and modules.

tc_check - checks the sanity of the toolchain after each build phase to ensure
that a faulty build process doesn't traipse down the road to Perdition.


USING THE BUILD SCRIPTS

When building on a standard GNU/Linux distribution, it is recommended that you
build as a non-root user; the build system uses su and/or sudo to become root
when it needs to. This minimizes the chance that a programming mistake will
wipe out or even subtley alter your host system.

The build tree may be installed anywhere on the host system; the script
'distrib_root' finds the path. The build system
ensures that the /tools symlink points to the right place before proceeding with
the build. Be aware that since there can be only *one* /tools symlink, you can
execute exactly *one* build at any given time.

The build system ensures that a reasonable /etc/resolv.conf exists in the final
phase chroot jail. This allows requisite network operations to succeed.

To start the build, enter "make build".  Enter "make" with no target to get a list
of available targets.  Interesting ones are:

predownload - Downloads all needed sources.

packageinfo.html - Spits out some crudely formatted HTML containing packages
and versions, download URLs and patch URLs.  Call with make -s to hide the
make output.

reprep-final - clean and rewind to rebuild the entire final phase.


RESTARTING THE BUILD

The build is (nearly) completely re-entrant. You may interrupt it at almost any
point; the next time you build, it will pick up at the beginning of the step when
it was interrupted.


TESTED HOST ENVIRONMENTS

Smoothwall Express 3.1 (of course)
Debian Wheezy
Debian Squeeze
Gentoo

Generally, the build should work on any Linux that is capable of building
GCC 4.7.2, binutils 2.22 and glibc 2.14.1.

As of 2013/03/15, there have been problems building on systems using GCC 4.6
and GLIBC 3.15-3.17.


THANKS TO

Linux From Scratch (www.linuxfromscratch.org); their LFS books were invaluable
on the journey enhancing the build system and modernizing the source packages.


CREDITS

Lawrence Manning (smoothwall@aslak.net) - Initial implementation.
Neal Murphy (neal.p.murphy@alum.wpi.edu) - SMP, re-entrance and bread crumbs,
  general modernization, 2009-2013.


COPYRIGHT AND LICENCE

The Smoothwall build system and associated scripts are (c) Smoothwall Ltd,
2005 and licenced under the GNU GPL v2.

Murphy's enhancements are © Neal P. Murphy and fully licensed under the GNU GPL v2.