module-build-general

[Module::Build] You underestimate the problem

[Adam K. <ad...@ph...>]

Grr, stupid mailing list.

I can subscribe to it, but it's not sending me messages.

 > As long as it's a mini-language that can be parsed safely, then having
 > some expressiveness is fine.  Most of where this will crop up is in
 > module dependencies anyway, and the number of variables there is
 > relatively contained:
 >
 > * platforms (in some set)
 > * perl versions (in some set of ranges of versions)
 > * module versions (in some set of ranges of versions)

I think you underestimate the problem.

Firstly, define "platform". Is it the processor arch? The string in $^O? 
If it's 'Win32' then which Win32? Win95? Win98? What if I have a 
dependency that needs something specifically on Windows XP only. Or only 
works for Server versions of Win32.

What about environment variables?

There are already modules that add extra dependencies if 
$ENV{AUTOMATED_TESTING} is true, so that under testing they can do more 
thorough tests than would normally be done.

Then there's optional dependencies based on installed programs.

Module::Signature looks for the binary program 'gpg'. If it cannot find 
it, it adds an extra 20 modules to the dep chain.

What version of Perl? 5.6 with threads, without threads?

So you need branch logic, you need to interact with the environment 
outside of the process (at least in a read-only manner) so you can't 
sandbox whatever is running the logic, you probably need a full range of 
boolean logic operators, which means you also need precedence support, 
and you are probably going to want subsections (i.e. subroutines) and 
probably inclusion of other files for reuse (i.e. modules) and you will 
inevitably want something like loops. Also, some of these are going to 
be host-specific, so you STILL can't run them anywhere but the current 
machine.

You just can't fit the richness required into a data structure, which 
means you can't fit it into META.yml.

The alternative of a mini-language is dangerous. I've seen the only one 
proposed so far and it is a very heavy language.

With META.yml on one side (data) and Makefile.PL on the other (code) I 
have trouble seeing that you can find ANY place in the middle that is 
usefully complete. I'd love to be convinced otherwise, but I'd like to 
see someone demonstrate a WORKING solution that can handle even the 
Bundle::CPAN subset of module before you commit to incorporating it.

 > Use case: Create an index of all dependencies for all distributions on
 > the Win32 platform without running Makefile|Build.PL for all
 > distributions.  (E.g. for a potential "CP-PAN" i.e. "CamelPack-PAN")

There is no such thing as a single fixed "Win32 Platform". That's not to 
say it isn't possible under some extremely tight controlled conditions, 
just that it doesn't exist in the real world that CPAN applies to.

If someone were to come up with a suitably framed bet, I'd probably be 
willing to put down serious money to say it is unsolvable.

In Perl, dependencies are ultimately programmatic. You can only 
approximate the dependencies in advance in an advisory way. Which is 
what we do already in META.yml.

Adam K

Re: [Module::Build] You underestimate the problem

[demerphq <dem...@gm...>]

On 4/25/06, Adam Kennedy <ad...@ph...> wrote:
> Grr, stupid mailing list.
>
> I can subscribe to it, but it's not sending me messages.
>
>  > As long as it's a mini-language that can be parsed safely, then having
>  > some expressiveness is fine.  Most of where this will crop up is in
>  > module dependencies anyway, and the number of variables there is
>  > relatively contained:
>  >
>  > * platforms (in some set)
>  > * perl versions (in some set of ranges of versions)
>  > * module versions (in some set of ranges of versions)
>
> I think you underestimate the problem.
>
> Firstly, define "platform". Is it the processor arch? The string in $^O?
> If it's 'Win32' then which Win32? Win95? Win98? What if I have a
> dependency that needs something specifically on Windows XP only. Or only
> works for Server versions of Win32.
>
> What about environment variables?
>
> There are already modules that add extra dependencies if
> $ENV{AUTOMATED_TESTING} is true, so that under testing they can do more
> thorough tests than would normally be done.
>
> Then there's optional dependencies based on installed programs.
>
> Module::Signature looks for the binary program 'gpg'. If it cannot find
> it, it adds an extra 20 modules to the dep chain.
>
> What version of Perl? 5.6 with threads, without threads?
>
> So you need branch logic, you need to interact with the environment
> outside of the process (at least in a read-only manner) so you can't
> sandbox whatever is running the logic, you probably need a full range of
> boolean logic operators, which means you also need precedence support,
> and you are probably going to want subsections (i.e. subroutines) and
> probably inclusion of other files for reuse (i.e. modules) and you will
> inevitably want something like loops. Also, some of these are going to
> be host-specific, so you STILL can't run them anywhere but the current
> machine.
>
> You just can't fit the richness required into a data structure, which
> means you can't fit it into META.yml.
>
> The alternative of a mini-language is dangerous. I've seen the only one
> proposed so far and it is a very heavy language.
>
> With META.yml on one side (data) and Makefile.PL on the other (code) I
> have trouble seeing that you can find ANY place in the middle that is
> usefully complete. I'd love to be convinced otherwise, but I'd like to
> see someone demonstrate a WORKING solution that can handle even the
> Bundle::CPAN subset of module before you commit to incorporating it.
>
>  > Use case: Create an index of all dependencies for all distributions on
>  > the Win32 platform without running Makefile|Build.PL for all
>  > distributions.  (E.g. for a potential "CP-PAN" i.e. "CamelPack-PAN")
>
> There is no such thing as a single fixed "Win32 Platform". That's not to
> say it isn't possible under some extremely tight controlled conditions,
> just that it doesn't exist in the real world that CPAN applies to.
>
> If someone were to come up with a suitably framed bet, I'd probably be
> willing to put down serious money to say it is unsolvable.
>
> In Perl, dependencies are ultimately programmatic. You can only
> approximate the dependencies in advance in an advisory way. Which is
> what we do already in META.yml.

I have to say I pretty strongly agree with Adam here. A mini-language
is a bad solution to this problem in comparison to simply generating
the META.yml on the installation host.

Yves

--
perl -Mre=3Ddebug -e "/just|another|perl|hacker/"

Re: [Module::Build] You underestimate the problem

[Chris D. <ch...@cl...>]

On Apr 25, 2006, at 8:33 AM, demerphq wrote:

> On 4/25/06, Adam Kennedy <ad...@ph...> wrote:
> I have to say I pretty strongly agree with Adam here. A mini-language
> is a bad solution to this problem in comparison to simply generating
> the META.yml on the installation host.

How about a 95% solution as a goal?  Plan META.yml variants so that  
95% of all CPAN modules can be completely described by it.  If that  
requires a mini-language that supports the various $^O strings and  
$^V versions, then so be it.  The other complicated 5% that Adam is  
worried about can use the flag that indicates a dynamic config.  That  
way, META.yml can be declared definitive by the author for most of  
CPAN.  The minority can then declare that you need to run Makefile.PL/ 
Build.PL to get the definitive metadata.  That way, tools that are  
not allowed to execute code can still be useful on the majority of CPAN.

I posit that the mini-language outlined by David Golden on 2006-04-24  
is adequate for the 95% solution (or maybe even the 99% solution?)

The problem with Yves' suggestion to generate META.yml on the  
installation host, as I've said before, is that it renders META.yml  
pointless.  The goal of META.yml is to create a non-executable source  
of accurate metadata about a package.  If you have to download the  
package and run Build.PL to generate META.yml, then why bother using  
META.yml?  In that case just use Build.PL itself.

The problem with generating a naive META.yml on the author's computer  
is that you often can't tell which are optional dependencies and  
which are required and which platforms are excluded (like the whole  
Win32:: tree).

I believe that an author-generated META.yml with limited conditional  
support and a "non-definitive" flag to support the rest will satisfy  
the needs of the vast majority of uses for META.yml.

Chris
-- 
Chris Dolan, Software Developer, Clotho Advanced Media Inc.
608-294-7900, fax 294-7025, 1435 E Main St, Madison WI 53703
vCard: http://www.chrisdolan.net/ChrisDolan.vcf

Clotho Advanced Media, Inc. - Creators of MediaLandscape Software  
(http://www.media-landscape.com/) and partners in the revolutionary  
Croquet project (http://www.opencroquet.org/)

Re: [Module::Build] You underestimate the problem

[David G. <da...@hy...>]

Chris Dolan wrote:
> The problem with Yves' suggestion to generate META.yml on the 
> installation host, as I've said before, is that it renders META.yml 
> pointless.  The goal of META.yml is to create a non-executable source of 
> accurate metadata about a package.  If you have to download the package 
> and run Build.PL to generate META.yml, then why bother using META.yml?  
> In that case just use Build.PL itself.

This was the point I was getting at.  Otherwise, every indexer will need 
to run in a chroot jail and will only have valid information for the 
configuration of the indexer.

That said, Adam's list of special cases is formidable, though I'm not 
sure that every eventuality needs to be supported.  I'd be happy if it 
covered 80% of cases.

Is there a reference to the existing mini-language proposal?

As an intermediate measure, what do people think about adding a 
"may_require" field to the META.yml spec as something in between 
"requires" and "recommends".  This would be for things that might get 
required depending on local configuration.

For a mix of human and indexer useful info, it could function like 
recommends, with a text description and a list (apologies if my YAML is 
poor):

may_require:
   - gpg not available:
     - Module::A
     - Module::B
     - etc
   - older Win32 (e.g. Win95, Win98):
     - Win32::C
     - Win32::D

Regards,
David Golden

[Module::Build] Re: You underestimate the problem

[Julian M. <ju...@me...>]

Chris Dolan wrote:
> On Apr 25, 2006, at 8:33 AM, demerphq wrote:
> > On 4/25/06, Adam Kennedy <ad...@ph...> wrote:
> > I have to say I pretty strongly agree with Adam here. A mini-language
> > is a bad solution to this problem in comparison to simply generating
> > the META.yml on the installation host.
>
> How about a 95% solution as a goal?  Plan META.yml variants so that
> 95% of all CPAN modules can be completely described by it.  [...]  The
> other complicated 5% that Adam is worried about can use the flag that
> indicates a dynamic config.  That way, META.yml can be declared
> definitive by the author for most of CPAN.  [...]
>
> I posit that the mini-language outlined by David Golden on 2006-04-24
> is adequate for the 95% solution (or maybe even the 99% solution?)
>
> The problem with Yves' suggestion to generate META.yml on the
> installation host, as I've said before, is that it renders META.yml
> pointless.

=46WIW, I agree with the 95% solution goal.  Declaring the concept of stati=
c=20
META.yml pointless because it doesn't cover 100% of the modules is=20
defeatistic.  In fact, those other 5% of the modules should probably try=20
harder to simplify their dependencies.  Sometimes it's far more robust to=20
simply always require some other module, even if under some obscure=20
circumstances you could go without it.

Re: [Module::Build] You underestimate the problem

[Ken W. <ke...@ma...>]

On Apr 25, 2006, at 8:17 AM, Adam Kennedy wrote:

> Grr, stupid mailing list.
>
> I can subscribe to it, but it's not sending me messages.
>

Next time I see sourceforge I'll kick them.


> > As long as it's a mini-language that can be parsed safely, then  
> having
> > some expressiveness is fine.  Most of where this will crop up is in
> > module dependencies anyway, and the number of variables there is
> > relatively contained:
> >
> > * platforms (in some set)
> > * perl versions (in some set of ranges of versions)
> > * module versions (in some set of ranges of versions)
>
> I think you underestimate the problem.
>
> Firstly, define "platform". Is it the processor arch? The string in  
> $^O? If it's 'Win32' then which Win32? Win95? Win98? What if I have  
> a dependency that needs something specifically on Windows XP only.  
> Or only works for Server versions of Win32.

We would use the list of $^O strings and os_type() values found in  
Module/Build.pm .

>
> What about environment variables?

Out of scope.

>
> There are already modules that add extra dependencies if $ENV 
> {AUTOMATED_TESTING} is true, so that under testing they can do more  
> thorough tests than would normally be done.

Which is a bad idea.  They'd handle that using dynamic_config, though.

>
> Then there's optional dependencies based on installed programs.

Out of scope for this iteration, maybe for a future iteration.

>
> Module::Signature looks for the binary program 'gpg'. If it cannot  
> find it, it adds an extra 20 modules to the dep chain.

Then it would have to use dynamic_config in this iteration of our  
mini-language.

>
> What version of Perl? 5.6 with threads, without threads?

dynamic_config.


We've got a set of predicates that we *know* are going to be useful.   
Let's not let the perfect be the enemy of the good here - we don't  
need to define the entire universe of helpful predicates before we  
make some of them available to users.


> You just can't fit the richness required into a data structure,  
> which means you can't fit it into META.yml.
>

That's untrue, of course - perl code itself can be thought of as a  
data structure (the parse tree), or else things like your PPI would  
be impossible.

What we're saying is that our data structure semantic doesn't need to  
be (and shouldn't be) Turing-complete.


> There is no such thing as a single fixed "Win32 Platform". That's  
> not to say it isn't possible under some extremely tight controlled  
> conditions, just that it doesn't exist in the real world that CPAN  
> applies to.

It's anything for which Module::Build->os_type() eq 'Windows'.

> If someone were to come up with a suitably framed bet, I'd probably  
> be willing to put down serious money to say it is unsolvable.
>
> In Perl, dependencies are ultimately programmatic. You can only  
> approximate the dependencies in advance in an advisory way. Which  
> is what we do already in META.yml.

Nobody's claiming we'll solve the world's dependency problems  
completely, but we know we can do better than we do now.

  -Ken