module-build-general

[Module::Build] Re: How to indicate a dependency in my module

[Randy W. S. <Ra...@Th...>]

Randy W. Sims wrote:

> darren chamberlain wrote:
> 
>> * Randy W. Sims <RandyS at ThePierianSpring.org> [2003-11-10 15:49]:
>>
>>> Also, as I noted in the AFS thread the other day, this info should be 
>>> written in META.yml so that cpan-testers can determine 
>>> programatically whether a module should be tested. Info like required 
>>> non-perl packages and libraries, whether the build/test/install 
>>> process is interactive (or scriptable) or automated, and whether it 
>>> runs only on certain OSs or excludes certain OSs.
>>
>>
>>
>> I like this idea; I must have not been paying close enough attention to
>> your other message.  How could it be done, though?  I can see a simple
>> exression -> expected value scheme working for a lot of things (e.g.,
>> "$^0 eq 'MacOS'", or "eval { getpwuid };" ), but how would things like
>> AFS be detectible?
>>
>> (darren)
>>
> 
> I'm making this up as I go, but I'm thinking that META.yml would just 
> contain a list of required packages and possibly the minimum version 
> required. Initially, that should be enough to flag it for cpan-testers. 
> Ideally, there would be a module (Module::Build::Configure) that could 
> try to determine whether those requirements are satisfied for a 
> particular system. This could be done by a which/where like routine to 
> search for binaries, looking for config files, analyzing the install log 
> of package managers like rpm, dpkg, etc.
> 
> In addition for larger packages like Apache there could be, for example 
> an Module::Build::Configure::Apache module that provided other usefull 
> info about that package like paths and other configuration info that 
> might be usefull to module authors.
> 
> Randy.
> 

Thinking more about this, I guess META.yml would need to provide a 
little more info to a configure module. Would something like the 
following work?

# Sequence of $^O values to include/exclude certains OSs
# One of the following as apropriate:
requires_os: *
excludes_os: MSWin32

# extra-perl requirements
# The has_* keys may map to a function call
# with it's values as arguments. If all calls return
# success the requirement is considered to be satisfied.
required_packages:
   - killerapp:
     version: 4.0
     has_program:
       - foo
       - bar
     has_lib:
       - a
       - z
   - libintl:
     version: 2

# One of auto/interactive/scriptable for cpan-testers
# to know how to setup testing
testing_mode: auto

[Module::Build] Re: How to indicate a dependency in my module

[Terrence B. <met...@ur...>]

Randy W. Sims wrote:

>
> Thinking more about this, I guess META.yml would need to provide a 
> little more info to a configure module. Would something like the 
> following work?

It's probably too late, but I am not keen on YAML. What is wrong with 
pure Perl configuration information? YAML is touchy about how it is 
formatted. Perl programmers will be reading this information and all 
Perl programmers already know Perl, so why fix what is not broken? Why 
force them to learn YAML?

Re: [Module::Build] Re: How to indicate a dependency in my module

[Michael G S. <sc...@po...>]

On Mon, Nov 10, 2003 at 05:42:03PM -0800, Terrence Brannon wrote:
> >Thinking more about this, I guess META.yml would need to provide a 
> >little more info to a configure module. Would something like the 
> >following work?
> 
> It's probably too late, but I am not keen on YAML. What is wrong with 
> pure Perl configuration information? 

In a nutshell: eval()ing the Perl structure back in is a major security hole.
Part of the point of META.yml is to avoid having to run any foreign code to
figure out module meta information.

To review (maybe this should be in a FAQ somewhere).

Data::Dumper/Perl code - Insecure (you have to eval it).  Perl specific.
Storable     - Not human readable.  Format changes slightly from version to
               version.  Perl specific.
XML          - Overkill.  Ugly.  Requires translation between Perl data
               model (hashes, lists, scalars) and XML's (trees).
               Difficult to read and write by humans.

YAML was chosen because its human readable and writable, its data 
structures closely match those of Perl (ie. scalars, hashes and arrays),
it can be read without being eval'd, executable code cannot be hidden in
it and, as a bonus, its not Perl specific.

YAML's basic formatting is a structure we're already familiar with and tend
to use when writing ad-hoc data structures (ie. key: value).
Indentation as structure we're already more than comfortable with (ie. 
indented source code) so readers of YAML should have no problem. 
The less obvious features of YAML shouldn't be necessary for most META.yml
files.

Because YAML's data model closely matches that of Perl, writers of META.yml 
simply need to construct a mirroring Perl structure and let YAML dump it
out.  Its the closest thing to "Data::Dumper evaling" available.


-- 
Michael G Schwern        sc...@po...  http://www.pobox.com/~schwern/
I'll tell you what beats voodoo every time, a big ass knife.
	-- "Overkill" Battlebot driver

Re: [Module::Build] Re: How to indicate a dependency in my module

[<Phi...@mo...>]

OK, maybe I'm missing a LOT of context here, 'cause I haven't been
agressively keeping up with this mailing list, but the security hole
argument seems a bit odd.

These META.yml files we're refering to -- these are meta data for
managing the build process, files that will be distributed along with
the tarballs we upload to CPAN, right?  

So, if I understand this correctly, you're worried about the build
process eval'ing the contents of a file I sent you.  Hmm.

OK, why is that anymore of a concern for eval'ing the perl module I
also distributed, too?  Isn't that just as big a security hole?  

Or how about the test suites?  If I want to deploy malicious code in
my CPAN upload, I can just drop the evil code into my test suite, and
when you type make tes, I take over your world.

Am I missing something?  Because I also am loath to accept yet another
file format, personally, and I would also prefer to keep my
configuration data written in perl as well.

And if we're talking about the perl build process, what's wrong with a
perl-specific configuration mechanism?

>>>>> "Michael" == Michael G Schwern <Michael> writes:

Michael> In a nutshell: eval()ing the Perl structure back in is a major security hole.
Michael> Part of the point of META.yml is to avoid having to run any foreign code to
Michael> figure out module meta information.

Michael> To review (maybe this should be in a FAQ somewhere).

Michael> Data::Dumper/Perl code - Insecure (you have to eval it).  Perl specific.
Michael> Storable     - Not human readable.  Format changes slightly from version to
Michael> version.  Perl specific.
Michael> XML          - Overkill.  Ugly.  Requires translation between Perl data
Michael> model (hashes, lists, scalars) and XML's (trees).
Michael> Difficult to read and write by humans.

Michael> YAML was chosen because its human readable and writable, its data 
Michael> structures closely match those of Perl (ie. scalars, hashes and arrays),
Michael> it can be read without being eval'd, executable code cannot be hidden in
Michael> it and, as a bonus, its not Perl specific.

Michael> YAML's basic formatting is a structure we're already familiar with and tend
Michael> to use when writing ad-hoc data structures (ie. key: value).
Michael> Indentation as structure we're already more than comfortable with (ie. 
Michael> indented source code) so readers of YAML should have no problem. 
Michael> The less obvious features of YAML shouldn't be necessary for most META.yml
Michael> files.

Michael> Because YAML's data model closely matches that of Perl, writers of META.yml 
Michael> simply need to construct a mirroring Perl structure and let YAML dump it
Michael> out.  Its the closest thing to "Data::Dumper evaling" available.


Michael> -- 
Michael> Michael G Schwern        sc...@po...  http://www.pobox.com/~schwern/
Michael> I'll tell you what beats voodoo every time, a big ass knife.
Michael> -- "Overkill" Battlebot driver

Why YAML. (was Re: [Module::Build] Re: How to indicate a dependency in my module)

[Michael G S. <sc...@po...>]

First, I'd like to address people's concern over the format of the META 
file.  Module users and 99% of module authors have nothing to be concerned
about.   Most folks shouldn't even know the thing exists.

Module::Build has been generating and using META.yml since nearly the 
beginning.  MakeMaker has been generating META.yml automaticly for its 
authors since last July.  CPAN is now full of META.yml files.  5.8.2 comes
with a META.yml file.  If you're just now noticing META.yml then we've done 
our job.  It could have been written in esperanto with FoxPro for all it 
matters to the end user.

If we do our job right, most people should never have to directly read or
write a META.yml file.  Its generated automaticly by MakeMaker and 
Module::Build and module tools (PAUSE, search.cpan.org, CPANPLUS, etc...)
use it without your intervention.

The only people who should be concerned are authors of these tools and folks
involved in gethering CPAN statistics.  Even module authors need not know
about it as its automaticly generated when the run "make dist" or "Build dist".

I say this because every few months someone notices META.yml and asks "Why 
did we use YAML and not X?" which starts the same debate all over again with 
the same answers.  We need a FAQ.


To address the "I don't want to learn another data format, why can't we
just use Perl?" issue.  With YAML, you don't have to learn the data format.
Let's look at how you'd generate a META file with Data::Dumper.

	use Data::Dumper;
	$Data::Dumper::Terse = 1;
	my $meta = { name => "Foo::Bar", version => 1.23 };
	open(META, ">META.perl");
	print META Dumper($meta);
	close META;

And the equivalent with YAML.pm.

	use YAML qw(DumpFile);
	my $meta = { name => "Foo::Bar", version => 1.23 };
	DumpFile("META.yml", $meta);

And now reading back in the Perl version.

	my $meta = do 'META.perl';
	print $meta->{version};

And the YAML version.

	use YAML qw(LoadFile);
	my $meta = LoadFile("META.yml");
	print $meta->{version};

YAML's data model is so similar to Perl's (hashes, scalars, lists) that the
data you put in and take out is almost indistinguishable from Data::Dumper.
All that's different is the transient storage medium.  So even if you wanted
to roll your own META.yml file you never have to learn YAML!  The only
prereq is to have YAML.pm installed.

And, finally, because I know someone's going to ask about the YAML.pm
prereq, YAML.pm is not required by MakeMaker.  It generates META.yml by
hand and doesn't use it for anything.  For Module::Build its an optional 
prereq.

I hope that covers the bases.


Now to address Phil's specific concerns.


On Wed, Nov 12, 2003 at 10:25:00AM -0500, Phi...@mo... wrote:
> OK, maybe I'm missing a LOT of context here, 'cause I haven't been
> agressively keeping up with this mailing list, but the security hole
> argument seems a bit odd.
> 
> These META.yml files we're refering to -- these are meta data for
> managing the build process, files that will be distributed along with
> the tarballs we upload to CPAN, right?  

Module::Build might use META.yml a little in its build process, but its not
a requirement.  MakeMaker can't even read META.yml.  Its not META.yml's 
primary purpose to be used to build a module.  Its just turning out
to be really damned useful. :)

The primary purpose of META.yml is to supply module meta data that we'd
normally have to go crawling around in the code to get.  Like $VERSION
(to get this you have to eval a line of the module) and its prerequisites
(to get that you have to run the Makefile.PL and parse the resulting 
Makefile).  

It also contains intangables like what license the code is distributed
under that you'd normally have to go groping around in the POD to try and
figure out.

File lists in META.yml are currently used as hints to the PAUSE indexer so
it can better determine what to index.  For example, 5.8.2's META.yml
contains a listing "private" (which I think is changing to noindex) to tell
PAUSE not to index these files/directories in the module list.  In this case
its because these are dual-life modules that also have CPAN versions.


> So, if I understand this correctly, you're worried about the build
> process eval'ing the contents of a file I sent you.  Hmm.

No, the case where security enters is when someone is grab CPAN meta 
information.  For example, PAUSE, CPAN search sites, anyone trying to do 
CPAN statistics, anyone trying to determine the prerequisites of a module, 
etc...  

CPAN statistics gathering currently involves running large amounts of 
untrusted code.  MakeMaker's parse_version() literally pulls a line of code 
out of the module and evals it.  To determine the prerequisties you have to 
run the Makefile.PL which, as Mark-Jason Dominus once demonstrated with
Memoize (I think it was) could contain rm -rf /.

Of course, when you build and install a module you are trusting that it
won't do anything naughty.  However, a user typically builds and installs
*a* module.  Or a set of modules, picked either by hand or because they're
a prerequisite of a module they picked by hand (thus, the author picked
by hand).  This selectiveness prevents some random hacker from uploading a 
module with an rm -rf in it.  Nobody will bother to install it because 
nobody will know about it.  Security through module obscurity. :)  For
better or worse, this is CPAN's ad-hoc "web of trust" model of security.

In contrast, CPAN statistics gathering usually involves *all* the modules 
on CPAN.  In this case, it will pick up RHACKER's module and blissfully run 
its Makefile.PL and $VERSION line causing who knows what havoc.  We shouldn't
have to set up a chroot jail (as PAUSE does) just to gether some
stats.

Since META.yml is automaticly generated by both Module::Build and MakeMaker,
CPAN is rapidly populating with META.yml files and code scaping for meta-data
will mostly disappear closing one of the big backdoors in CPAN.


-- 
Michael G Schwern        sc...@po...  http://www.pobox.com/~schwern/
IIRC someone observed that they couldn't name identifiers in Ethiopian, 
because there was an Ethiopian character similar in function to _ which 
wasn't in \w
        -- Nicholas Clark demonstrates that the Internet works
           in <200...@pl...>

Re: [Module::Build] Re: How to indicate a dependency in my module

[Christopher H. <ch...@ch...>]

On Wed, 12 Nov 2003 Phi...@mo... wrote:
> So, if I understand this correctly, you're worried about the build
> process eval'ing the contents of a file I sent you.  Hmm.
> 
> OK, why is that anymore of a concern for eval'ing the perl module I
> also distributed, too?  Isn't that just as big a security hole?  

There are any number of reasons I may be interested in examining your
module for extracting documentation and dependancies without actually
running a single line of your code.  The safer we make this sort of thing
the more things like search.cpan.org and various more recent alternatives
we'll have.

-- 
</chris>

> What the US needs is a Manhattan Project for alternative energy to oil.
They should threaten their enemies with windmills?
		- BabyDave on /.

Re: [Module::Build] Re: How to indicate a dependency in my module

[<Phi...@mo...>]

>>>>> "Chris" == Christopher Hicks <ch...@ch...> writes:

>> So, if I understand this correctly, you're worried about the build
>> process eval'ing the contents of a file I sent you.  Hmm.
>> 
>> OK, why is that anymore of a concern for eval'ing the perl module I
>> also distributed, too?  Isn't that just as big a security hole?  

Chris> There are any number of reasons I may be interested in
Chris> examining your module for extracting documentation and
Chris> dependancies without actually running a single line of your
Chris> code.  The safer we make this sort of thing the more things
Chris> like search.cpan.org and various more recent alternatives we'll
Chris> have.

Ok, that makes sense... Thanks.  I hadn't really though about using
this data outside of the build process.

Re: [Module::Build] Re: How to indicate a dependency in my module

[Sam V. <sa...@vi...>]

On Tue, 11 Nov 2003 02:29, Michael G Schwern wrote;

  > YAML was chosen because its human readable and writable, its data
                                ^^^^^              ^^^^^^^^^
So long as you're a FREAK who likes INDENTING and WHITESPACE to
signify STRUCTURE.

Is it any surprise that YAML is supported by PYTHON?!

</topicalButTechnicallyVoidRantIgnoringTheObviousReplyForComicalValue>
-- 
Sam Vilain, sa...@vi...

  If you cant learn to do it well, learn to enjoy doing it badly.
ASHLEIGH BRILLIANT

Re: [Module::Build] Re: How to indicate a dependency in my module

[Nicholas C. <ni...@cc...>]

On Wed, Nov 12, 2003 at 04:11:45PM +0000, Sam Vilain wrote:
> On Tue, 11 Nov 2003 02:29, Michael G Schwern wrote;
> 
>   > YAML was chosen because its human readable and writable, its data
>                                 ^^^^^              ^^^^^^^^^
> So long as you're a FREAK who likes INDENTING and WHITESPACE to
> signify STRUCTURE.
> 
> Is it any surprise that YAML is supported by PYTHON?!

no, not at ALL

but GIVEN that XS also treats INDENTING and WHITESPACE significantly,
it is STRANGE that there are no YAML modules for perl written in XS.

[although I believe that Ingy is working on something]

Nicholas Clark

[Module::Build] (RFC) (PATCH) META.yml Specification Update

[Randy W. S. <Ra...@Th...>]

Ok. So, I've gone over the spec and referenced this thread and my 
archives of past meta discusions on module-authors, and I've come up 
with a preliminary patch for Ken and anyone else interested. This is a 
patch against <http://module-build.sourceforge.net/META-spec.html> which:

* Added more YAML rationale articles.
* Fixed existing link to YAML discussion thread to point to new 
<http://nntp.x.perl.org/group/> site.
* Added and deprecated the "private" field.
* Added "abstract", "configure", "requires_os", "excludes_os", and 
"no_index" fields.
* Bumped version.

Comments and suggestions?

Randy.

Re: [Module::Build] (RFC) META.yml Specification Update

[Randy W. S. <Ra...@Th...>]

And here it is in pod format.

Re: [Module::Build] (RFC) META.yml Specification Update

[Randy W. S. <Ra...@Th...>]

Hi Ken,

Ok, this is the last time. I promise. 'Cause now I'm just talking to 
myself anyway.

Actually, I was doing some more hunting and gathering, and I stumbled 
across some other "suggestions". I also added to each field some tags 
indicating the version of the spec when the field was added and whether 
it's required or optional. (Thought about adding a tag for type, e.g. 
scalar, sequence, or mapping, but...)

I don't actually anticipate all these being added; this is basically an 
accumulation of ideas, current practice, brain-storm, etc. I've just 
tried to organize it for easy review. Cut and dice.

Notes and Sources:

The "authored_by" and "abstract" fields where added because I found 
quite a few META.yml files containing them or something like.

The "generation" field was added based on a discussion beginning here 
<http://nntp.x.perl.org/group/perl.cpan.discuss/1>, specifically 
<http://nntp.x.perl.org/group/perl.cpan.discuss/14>. Also, it surfaced 
again in this discussion 
<http://nntp.x.perl.org/group/perl.cpan.discuss/26>.

The "private" and "no_index" comes from a discussion began here 
<http://nntp.x.perl.org/group/perl.module-authors/765>, specifically 
<http://nntp.x.perl.org/group/perl.module-authors/800>. The "private" 
field was used unofficially for a while, but has since been changed to 
"no_index" as per 
<http://nntp.x.perl.org/group/perl.module-authors/1385> and 
<http://cplan.kwiki.org/?MinutesAccordingToAutrijus>

The "requires_package" field was first suggested by me in this brief 
discussion <http://nntp.x.perl.org/group/perl.module-authors/1313> and 
then shortly after in a much larger discussion 
<http://nntp.x.perl.org/group/perl.module-authors/1338> which is where I 
also added "requires_os", "excludes_os", and "configure" fields. Much of 
this is meant to aid cpan-testers automated testing, but will also be 
usefule for M::B, MM, and other tools.

The "require_build_tools" was added as another after-thought to the 
above because it also indicates an external dependency.

The alternative suggestion in "recommends" is another idle suggestion of 
mine which is meant to supply enough information to the user so that he 
or she can make an informed choice as to which optional modules to install.

I put it in pod 'cause it's easier to edit, and it's easy to change the 
format with pod2html and kin.

I've now finished my hunting and gathering period; it's time to evolve.

Later,
Randy.