Re: [xml-cppdom-devel] Building cppdom prefix problem

[Patrick H. <pa...@13...>]

Allen Bierbaum wrote:
> Patrick Hartling wrote:
>> Allen Bierbaum wrote:
>>  =20
>>> Patrick Hartling wrote:
>>>    =20
>>>> Doug McCorkle wrote:
>>>>  =20
>>>>      =20
>>>>> Hello,
>>>>>      Currently, when building cppdom, headers and libs are installe=
d =20
>>>>> in prefix even if the install target is not listed. This causes =20
>>>>> issues when trying to make cppdom accessible through MacPorts. In  =

>>>>> general this seems to be somewhat of a problem.
>>>>>    =20
>>>>>        =20
>>> Doug:
>>>
>>> I am coming to this a little late, but what is the problem building=20
>>> inside MacPorts?  Is there a requirement that build happens separatel=
y=20
>>> from install?
>>>    =20
>> All packaging utilities that I know of separate building from installi=
ng.
>> Watching MacPorts builds run, they definitely do this, though I don't =
know
>> if it is a strict requirement. I bet that it is, though.
>>
>>  =20
>>> cppdom is definitely not designed to do that.  It has been designed t=
o=20
>>> follow a fairly common best practice of having one command that does =
a=20
>>> complete build and install and does it every time (it is also=20
>>> recommended to have this run the testsuite, but we don't have that in=
=20
>>> there yet. :).
>>>    =20
>> Where is this stated as being a best practice?=20
> Many software engineering techniques have concepts similar to this one.=
 =20
> They all revolve around continuous integration=20
> (http://en.wikipedia.org/wiki/Continuous_integration).

What is described here sounds to me like something that wraps a
build/install/test process. Why should it preclude the separation of the
build and installation steps? If nothing else, it seems that one could wr=
ite
a target of the build such as 'universe' that performs the individual ste=
ps
automatically.

> The main point=20
> being that a full build is always happening and that the user can get=20
> from code to fully complete build in one step with nothing to manual=20
> processes in between.

Why should that prevent the user from being able to run the steps individ=
ually?

> This is exactly the type of thing that an rpm=20
> spec file is capturing but the rpc spec file would allow wrapping=20
> multi-step builds.

I don't think that most RPMs would do continuous integration. From the
linked description, it sounds like an RPM can be a *product* of continuou=
s
integration rather than the *means*. Actually, as far as single-step
processes go, an RPM by itself is completely outside of that arena becaus=
e
you have to build an RPM and then install it. These are inherently separa=
te
steps using different commands (rpmbuild first, then rpm).

You could wrap the RPM creation and installation process to turn it into =
a
single step, but the low-level process is still multiple steps. IMHO, the=

same thing should be true of building software. The build is made up of
distinct steps, and developers would always be running all of them. (Ther=
e
is the continuous integration.) However, end users don't need all of them=

and should have the option of doing only what they need. They will still =
be
running steps that developers run, but they don't run all of them.

I think that I misunderstood what you were referring to when you brought =
up
the "best practice" term. I thought that you meant that not separating
building and installation is a best practice, but I believe that what you=

meant is that the CppDOM build is designed to allow for continuous
integration usage. As I will try to explain below, the CppDOM build as it=

stands right now does not address the needs of end users. Rather, it is
designed for developers needing continuous integration.

>> I would think that nearly
>> every library in existence allows for separate build and install steps=
=2E
>> Those that don't are going to make packaging a pain. For example, when=
 you
>> build an RPM to be installed in /usr, you don't actually want it to go=
 into
>> /usr during the packaging process. It has to go into a staging area (w=
hich
>> is something that MacPorts uses) wherein the actual package is created=
=2E So,
>> there end up being two installation trees: the staging area using duri=
ng
>> packaging and the final destination when the package is installed.=20
> I guess I am missing something here.  Why is the staging area ever=20
> needed.

Well, for one thing, it would take a very experienced person to write a s=
pec
file that works correctly on the first try. The staging area can be clear=
ed
out in one fell swoop if something goes wrong during the packaging phase
(which happens all the time in my [limited] experience). Beyond that, hav=
ing
the clean staging area for a given package makes it much easier for rpmbu=
ild
to examine just exactly what is being packaged. If it had to scan all of
/usr every time, it would be a completely unusable tool.

> If you know you want to end up calling install with a=20
> destination path, why not just call install from the beginning and skip=
=20
> the first step?

You can certainly do that, but that doesn't have anything to do with the =
use
of a staging area. You just don't put anything in the %build section of t=
he
spec file. However, just about all software has separate build and instal=
l
phases, and in some (most?) cases, it makes a lot of sense to separate th=
e
two when building the RPM.

> I think there must be a requirement here that I am just=20
> missing.

I think you have some misconceptions about packaging processes. For one, =
I
did not make it clear that the staging area is used during the installati=
on
and packaging phases only. There is a separate place where building occur=
s.
Basically, it mimics just what you would do if you were building the
software and installing it to /usr as root. However, you don't have to be=

root to run rpmbuild (and to my understanding, it is not recommended that=

root run rpmbuild), and it installs into a "clean room" of sorts before
packaging.

>> If you
>> look at the RPM spec file for CppDOM, you'll see a hack where cppdom.f=
pc
>> gets edited after running 'scons install' to change the staging area p=
refix
>> to the final destination prefix.
>>
>>  =20
>>> That way developers are *always* using the same build=20
>>> paths that are used by users and packages to install the tool and the=
re=20
>>> are no special hoops to jump through to install or package the=20
>>> software.
>>>    =20
>> Not everyone is a developer. Installing a package is a separate thing =
from
>> building it. Having a developer installation is one thing, but that
>> shouldn't interfere with users being able to build and install the sof=
tware.
>>
>>  =20
> Exactly, and that is the point.  The idea is to keep from getting bitte=
n=20
> by the "developer using a separate build process" problems that have=20
> plagued us in the past.  The build the developer uses as part of their =

> everyday process is exactly the same as what someone uses for=20
> installation.  (ok, I fibbed a bit here, in the case of developer on=20
> linux we replace the copy command with a link command, but all the code=
=20
> being run is the same)

I am not saying that developers should use a build process that is disjoi=
nt
from what users do. I think that developers should use a superset of the
build process that users need. Users need build+install. If installation
does the build first by default (seems like a handy thing for it to do),
then they can skip a step. However, if building does installation whether=

the user wants it or not, then that is a problem. The two seem to be
intermingled in this case, and that is causing Doug's problem.

>>> Just one magic button that will take you from source to=20
>>> happiness. I hadn't heard of this causing problems before, so I am=20
>>> interested in hearing more here.
>>>
>>> If you really need the build and install separated, you could do=20
>>> something like this:
>>>
>>> - Build
>>>    - call scons without a prefix setting
>>> - Install
>>>    - call scons with a prefix setting
>>>
>>> This will end up "installing" cppdom in the development directories o=
n=20
>>> the first call, but it should work for you.
>>>    =20
>> I am just about positive that that process won't work because cppdom.f=
pc
>> gets generated as part of the build. If the prefix changes, it doesn't=
 get
>> regenerated (which seems like a bug).=20
> That is definitely a bug.  It should be regenerated.  Running the build=
=20
> at any point should generate all files again that need to be changed fo=
r=20
> the new installation path.  That is the whole point of the single build=
=20
> process. :)
>=20
> I don't have time to look at this today, but I will do what I can to=20
> help out.

Wasn't there a change made to the CppDOM build to work around a bug in SC=
ons
where it would scan the entire installation prefix? The result would be t=
hat
installation would hang if the installation prefix was the root of a
directory tree of even moderate size. I can't find the revision where tha=
t
happened, so maybe I an thinking of a different project. Being an SCons
novice at best, it seems to me that the bug is that the installation pref=
ix
is not seen by SCons as being a dependency of cppdom.fpc, and this is wha=
t
causes the problem. If that is the problem, is there a simple fix?

 -Patrick

>> This is something that I know I have
>> seen, and this exact usage is what I was alluding to earlier when I sa=
id
>> "running the build separately from the installation doesn't work well.=
"
>>
>>  -Patrick
>>
>>  =20
>>>> That sounds like bad behavior in general. Software shouldn't be inst=
alling
>>>> itself unless told to do so.
>>>>
>>>> The CppDOM build is in pretty bad shape and hasn't ever been quite w=
hat I
>>>> would call ideal. The installation part has never really worked corr=
ectly,
>>>> but I don't think I have seen this specific behavior. What I have se=
en is
>>>> that running the build separately from the installation doesn't work=
 well,
>>>> but the build certainly should support a two-step build+install proc=
ess.
>>>> What version are you using?
>>>>
>>>>  =20
>>>>      =20
>>>>> I "fixed" the problem =20
>>>>> by adding guards in the scons files but that may not be the solutio=
n =20
>>>>> everyone is looking for. Thoughts?
>>>>>    =20
>>>>>        =20
>>>> What are guards in this context?
>>>>
>>>>  -Patrick


--=20
Patrick L. Hartling                    | VP Engineering, Infiscape Corp.
PGP: http://tinyurl.com/2oum9          | http://www.infiscape.com/