module-build-general

[Module::Build] Recursive builds?

[Peter S. <Pe...@PS...>]

Hi there.  This sounds like a FAQ, but I have searched high and low and 
not found it answered anywhere.  Are there any examples around for 
using M::B for recursive builds, where you have a directory tree and 
each directory has its own Build.PL?  Seems like someone must have done 
that already, but darned if I can find it.
-- 
Peter Scott
Pacific Systems Design Technologies
http://www.perldebugged.com/
http://www.perlmedic.com/

Re: [Module::Build] Recursive builds?

[Ken W. <ke...@ma...>]

On Feb 10, 2006, at 10:47 AM, Peter Scott wrote:

> Hi there.  This sounds like a FAQ, but I have searched high and low 
> and not found it answered anywhere.  Are there any examples around for 
> using M::B for recursive builds, where you have a directory tree and 
> each directory has its own Build.PL?  Seems like someone must have 
> done that already, but darned if I can find it.

It also sounds not-too-difficult to me, but I don't actually know of 
anyone that's done it either.  Anyone?

  -Ken

Re: [Module::Build] Recursive builds?

[Eric W. <scr...@gm...>]

# from Ken Williams
# on Friday 10 February 2006 05:50 pm:

>Are there any examples around for
>
>> using M::B for recursive builds, where you have a directory tree and
>> each directory has its own Build.PL? =A0Seems like someone must have
>> done that already, but darned if I can find it.
>
>It also sounds not-too-difficult to me, but I don't actually know of
>anyone that's done it either. =A0Anyone?

Ken, Is this your Friday night humor or did I miss something?  I was=20
thinking this was in the todo/not-quite-possible-yet category.

I'm also interested in getting recursive builds working, but my last=20
patch ended up in purgatory so I haven't looked into it.

=2D-Eric
=2D-=20
I arise in the morning torn between a desire to improve the world and a
desire to enjoy the world. This makes it hard to plan the day.
=2D-E.B. White
=2D--------------------------------------------------
    http://scratchcomputing.com
=2D--------------------------------------------------

Re: [Module::Build] Recursive builds?

[David G. <da...@hy...>]

Eric Wilhelm wrote:
> # from Ken Williams
> # on Friday 10 February 2006 05:50 pm:
> 
>> Are there any examples around for
>>
>>> using M::B for recursive builds, where you have a directory tree and
>>> each directory has its own Build.PL?  Seems like someone must have
>>> done that already, but darned if I can find it.
>> It also sounds not-too-difficult to me, but I don't actually know of
>> anyone that's done it either.  Anyone?
> 
> Ken, Is this your Friday night humor or did I miss something?  I was 
> thinking this was in the todo/not-quite-possible-yet category.
> 
> I'm also interested in getting recursive builds working, but my last 
> patch ended up in purgatory so I haven't looked into it.

My initial thought was that it seemed easy, but then my next thought was 
that a lot of M::B assumes certain semantics about the layout of a 
distribution (e.g. lib, t, META.yml, etc.)  that aren't so easy to 
generalize to a recursive case.

It would seem fairly trivial to add a "recurse_into" parameter with 
directories but then what?  Are each of the directories mirrors of a 
normal distribution with their own lib and t directories?  And how 
should building/testing/installing work?  Copy all relevant files from 
all recursive directories into a top level blib directory?  Or should 
there be blibs in each directory?  Should top level tests assume the 
existence of (and know how to find) files from lower down in the tree?

Maybe this is an in-joke or todo from years ago -- but has anyone 
actually drafted a spec for how this should work?

Regards,
David

Re: [Module::Build] Recursive builds?

[Eric W. <scr...@gm...>]

# from David Golden
# on Saturday 11 February 2006 06:14 am:

>It would seem fairly trivial to add a "recurse_into" parameter with
>directories but then what?

chdir into each one and run $perl, 'Build.PL' or $perl, 'Build' with the=20
same arguments.

>Are each of the directories mirrors of a=20
>normal distribution with their own lib and t directories?

I'm pretty sure that's roughly what ExtUtils::MakeMaker does.

>And how=20
>should building/testing/installing work? =A0Copy all relevant files from
>all recursive directories into a top level blib directory?

No.  I think the primary need for recursive builds is to be able to put=20
multiple distributions together and/or separate the Build.PL files to=20
keep things sane.  I would also like to be able to trigger MakeMaker=20
builds (unless Module::Build::Inline would be trivial.)  I suppose=20
handling the differences between what arguments are supported by=20
MakeMaker makes this hairy.

>Or should there be blibs in each directory?

Yes.  You should be able to chdir into each one and run build as if it=20
were a standalone distribution.

>Should top level tests assume the=20
>existence of (and know how to find) files from lower down in the tree?

No.  Maybe.  If so, it should just add every blib/ to @INC.  I imagine=20
that the toplevel tests might be used to test integration of the=20
recursively built modules?

=2D-Eric
=2D-=20
Speak softly and carry a big carrot.
=2D--------------------------------------------------
    http://scratchcomputing.com
=2D--------------------------------------------------

Re: [Module::Build] Recursive builds?

[Ken W. <ke...@ma...>]

On Feb 10, 2006, at 10:40 PM, Eric Wilhelm wrote:

> # from Ken Williams
> # on Friday 10 February 2006 05:50 pm:
>
>> Are there any examples around for
>>
>>> using M::B for recursive builds, where you have a directory tree and
>>> each directory has its own Build.PL? =A0Seems like someone must have
>>> done that already, but darned if I can find it.
>>
>> It also sounds not-too-difficult to me, but I don't actually know of
>> anyone that's done it either. =A0Anyone?
>
> Ken, Is this your Friday night humor or did I miss something?  I was
> thinking this was in the todo/not-quite-possible-yet category.

Heh.  Probably my Friday night wishful thinking.

What I meant was that for one individual distribution someone could get=20=

it working by just adding some code to copy between various blib/s,=20
call actions in subdirectories, and so on.  It seemed like this was=20
what the OP was asking, but the solution would be sort of hacked=20
together.

You're quite right that adding recursive builds to M::B itself in a=20
nice solid way would be a major project.  We're scheduling it for the=20
next big release, i.e. after 0.27_xx becomes 0.28xx.

  -Ken

Re: [Module::Build] Recursive builds?

[alan <al...@pa...>]

On Fri, 10 Feb 2006, Ken Williams wrote:

> On Feb 10, 2006, at 10:47 AM, Peter Scott wrote:
>
> > Hi there.  This sounds like a FAQ, but I have searched high and low
> > and not found it answered anywhere.  Are there any examples around for
> > using M::B for recursive builds, where you have a directory tree and
> > each directory has its own Build.PL?  Seems like someone must have
> > done that already, but darned if I can find it.
>
> It also sounds not-too-difficult to me, but I don't actually know of
> anyone that's done it either.  Anyone?
>
>   -Ken

About 2 years ago, I submitted my ideas and possibly even patches,
related to this. My changes create a subclass of Module::Build to
support recursive builds, as well as other features which are either
less useful to the general public, or historically irrelevant
considering more recent versions of Module::Build.

After some discussion on-list, it was decided that most of my changes
were inappropriate for Module::Build's primary goal of being a build
tool for CPAN. Since then, I've seen the big problems Module::Build has
to solve without supporting non-CPAN functionality, I have come to agree
with this assessment.

Regarding the specific case of recursive builds: our changes are not
perfect, and probably don't support many features of Module::Build which
we don't happen to use. But we've been using them in production for a
large project (63 Build.PL's if you start the build from the root) and
have had no problems in actual practice, for what we do. (Note that
"./Build dist" is _not_ something we have ever done or ever intend to
do, and likely won't do anything useful.)


The primary problem with doing recursive builds correctly which I had to
solve (and which MakeMaker got wrong at the time), is the tradeoff
between "make test" and "make install" in a subtree of the project.

When you do a "make test" in a subtree of the build tree, you typically
want it to use a blib which contains the (most recently built) contents
of the _entire_ build tree (not just your subtree). If you don't, "make
test" in a subtree uses the installed versions of modules in other
subtrees of the build, instead of reading the blib versions. (The reason
is a problem for us is because we wanted to be able to test interactions
between modules in different branches of the build tree, without testing
Everything, during normal development cycles.)

On the other hand, when you "make install" in a subtree, it should
install only that subtree of code.

I solved these problems by copying the source files into 2 separate
blib's: one at the root of the tree holds "everything" and is used only
for "make test". A blib inside each separate build directory stores just
the files for that specific build, and is used to install subtrees
correctly at "make install" time.


In summary:

If you want a solution which might do what you want and expect, but
doesn't have full support for all features Module::Build uses, then I
can most likely provide this (I'll have to get permission again but I
expec this is a formality). I would hesitate to suggest it be merged
into Module::Build, however, because I agree that most of what I do is
inappropriate for use in distributed modules intended for use by someone
other than the module's developer.

Thanks,
Alan Ferrency
pair Networks, Inc.
al...@pa...

Re: [Module::Build] Recursive builds?

[Eric W. <scr...@gm...>]

# from alan
# on Monday 13 February 2006 07:18 am:

>When you do a "make test" in a subtree of the build tree, you
> typically want it to use a blib which contains the (most recently
> built) contents of the _entire_ build tree (not just your subtree).

I'm curious as to how many users want this behavior.  I'm thinking that 
anything done in a subtree should be no different than if I were 
building just that subtree.

i.e.  "the recursive build system only recurses down and never up" seems 
like a good design theme.

The behavior you describe doesn't fit the definition of "recursive."

> "make test" in a subtree uses the installed versions of modules
... [which is bad because we want] ...
> to test interactions between modules

I think this sort of testing should be enabled by running "./Build test 
Module-Name" from the toplevel.  This would then do something like:

  1.  run "depends" actions (depth first) for all subdirs
  2.  make a -Ipath/to/Module/blib/lib/ list including all subdirs
  3.  run ACTION_test for the Module-Name dir

I'm sure I've left something out, but the point is to never require a 
build to look upwards from itself.  The toplevel builder should play 
manager to the subdir worker builders, feeding them everything they 
need to know.

There's also something about dependencies in the Build.PL stage.  Maybe 
the toplevel Build.PL could gather all of the META.yml info and inform 
the workers about which dependencies are satisfied by their peers.

--Eric
-- 
But you can never get 3n from n, ever, and if you think you can, please
email me the stock ticker of your company so I can short it.
--Joel Spolsky
---------------------------------------------------
    http://scratchcomputing.com
---------------------------------------------------

Re: [Module::Build] Recursive builds?

[alan <al...@pa...>]

Before anyone starts designing recursive builds again, I recommend
hitting the module-build-general archive for February/March 2004, and
checking out the threads "Recursive Build Patch" and "Some thoughts on
recursion."

http://sourceforge.net/mailarchive/forum.php?forum=module-build-general

I'm not interested in going over any of my old points again, because I
fully admit that what I am doing is not within the main focus of
Module::Build. I don't expect my existing recursive build patches would
be useful in the design or implementation of an official Module::Build
recursive build system, even though they work fine for me now.

I only wanted to make it known that my solution exists, in case there is
anyone else who may benefit from it.

On Mon, 13 Feb 2006, Eric Wilhelm wrote:

> # from alan on Monday 13 February 2006 07:18 am:
>
> > When you do a "make test" in a subtree of the build tree, you
> > typically want it to use a blib which contains the (most recently
> > built) contents of the _entire_ build tree (not just your subtree).
>
> I'm curious as to how many users want this behavior. I'm thinking that
> anything done in a subtree should be no different than if I were
> building just that subtree.

I don't know, but I'd expect you're right: not many people would
need this.

As I said, my changes aren't available on CPAN, because they're
inappropriate for use in public distributions. But, we are willing to
share our solution if it would be useful (with the understanding that
you're best off not using it for publicly distributed modules).

> i.e.  "the recursive build system only recurses down and never up" seems
> like a good design theme.
>
> The behavior you describe doesn't fit the definition of "recursive."

My system does not "recurse up." Sub-builds simply have access to, and
modify, a global store of information. I don't know of any definition of
"recursive" which precludes this.

That said, I have no argument against the spirit of your design theme
w.r.t recursive builds in Module::Build, as long as I can get the job
done (and I can).

> > "make test" in a subtree uses the installed versions of modules
> ... [which is bad because we want] ...
> > to test interactions between modules
>
> I think this sort of testing should be enabled by running "./Build test
> Module-Name" from the toplevel.  This would then do something like:

This could solve my original problem, although in a less convenient way,
in practice.

>   1.  run "depends" actions (depth first) for all subdirs
>   2.  make a -Ipath/to/Module/blib/lib/ list including all subdirs
>   3.  run ACTION_test for the Module-Name dir

The biggest problem I'd have with this is that it seems like it would be
quite slow in my application, if testing one subtree had to traverse the
entire build tree before running a single test file.

Currently, I do:

1. one slow "./Build" at the root, which builds multiple blib/
   directories.
2. multiple "./Build test" in arbitrary subtrees are then fast: they
   do not have to traverse any portion of the build tree that isn't
   being tested, and @INC gains only one new directory (instead of the
   63 separate blib's I would have in your solution).
3. subsequently, a "./Build" in a subtree is adequate for all other
   modules to see any changes in that subtree.

> There's also something about dependencies in the Build.PL stage.  Maybe
> the toplevel Build.PL could gather all of the META.yml info and inform
> the workers about which dependencies are satisfied by their peers.

We don't currently use the Module::Build dependency specification for
"other modules within the same recursive build." Expect to deal with the
case where someone hands you multiple mutually dependant sub-builds,
even if you may not think this makes sense.

(Also, my build system has no official "top level Build.PL". There is no
extra configuration required in any Build.PL, and any subtree of the
build can act as "the top level Build.PL" if you let it. My build tree
is deep, not simply flat and wide.)


Thanks,

Alan Ferrency
pair Networks, Inc.
al...@pa...

Re: [Module::Build] Recursive builds?

[Peter S. <Pe...@PS...>]

At 09:40 AM 2/13/2006, Eric Wilhelm wrote:
>I'm sure I've left something out, but the point is to never require a
>build to look upwards from itself.  The toplevel builder should play
>manager to the subdir worker builders, feeding them everything they
>need to know.
>
>There's also something about dependencies in the Build.PL stage.  Maybe
>the toplevel Build.PL could gather all of the META.yml info and inform
>the workers about which dependencies are satisfied by their peers.

Just looking for a simple solution, I see two alternatives:

(1) By default, recurse through all subdirectories reproducing the 
current action in any that contain a Build.PL.
- OR -
(2) Require an option to be enabled for recursion.  On finding a 
subdirectory containing a Build.PL, execute the current action there 
but prune.  If the Build.PL at that level wants to recurse further, it 
can enable that option itself.

So "perl Build.PL" would search down subdirectories for directories 
containing Build.PL, and then execute "perl Build.PL" in them.  And 
ditto for "./Build <target>".  In scenario (1), this would happen 
automatically, and would search throughout the tree.  In scenario (2), 
some option would have to be enabled in the Build.PL to make this 
happen, and when recursing, on finding a Build.PL in any directory, the 
recursion would be pruned at that directory because the Build.PL it 
just found could recurse itself if it wanted to.

That's all.  No passing of parameters down to subsidiary Build.PLs, no 
looking upward, just really simple and leaving it up to the user to do 
that some way if they want to.  At least the users will find it easy to 
understand how the feature works.

My preference is for alternative (2).  I don't mind submitting a patch, 
either, although it sounds like there's some history on this topic 
already and I don't want to step on any toes.

-- 
Peter Scott
Pacific Systems Design Technologies
http://www.perldebugged.com/
http://www.perlmedic.com/

[Module::Build] add_property()

[Peter S. <Pe...@PS...>]

I'm wondering why the add_property() method isn't exposed to the 
user?  It seems that it would be handy for my case; I have a config.yml 
file that I want installed in the etc/ directory off my target, and I'm 
hardwiring the install_base here because this is something that gets 
used at a client site.  I started with:

my $builder = Module::Build->new
     (
       install_base      => '/path/to/install',
       script_files      => [ 'dummy' ],
       yml_files         => { 'config.yml' => 
'/path/to/install/etc/dummy.yml' },
       dist_name         => 'dummy',
       dist_version_from => 'dummy',
     );
$builder->add_build_element('yml');
$builder->create_build_script;

However, I don't like having to repeat /path/to/install in there.  To 
remove the duplication I ended up doing:

my $builder = Module::Build->new
     (
       install_base      => '/path/to/install',
       script_files      => [ 'dummy' ],
       dist_name         => 'dummy',
       dist_version_from => 'dummy',
     );
sub Module::Build::find_yml_files {
   my $self = shift;
   return { map $self->localize_file_path($_),
                'config.yml' => $self->install_base . '/etc/dummy.yml'
	  };
}
$builder->add_build_element('yml');
$builder->create_build_script;


But it seems that if I had been able to call add_property, it would 
have been simpler:

my $builder = Module::Build->new
     (
       install_base      => '/path/to/install',
       script_files      => [ 'dummy' ],
       dist_name         => 'dummy',
       dist_version_from => 'dummy',
     );
$builder->add_property('yml_files');      # Method not found
$builder->yml_files({ 'config.yml' => $self->install_base . 
'/etc/dummy.yml' });
$builder->add_build_element('yml');
$builder->create_build_script;

-- 
Peter Scott
Pacific Systems Design Technologies
http://www.perldebugged.com/
http://www.perlmedic.com/