module-build-general

[Module-build-general] [PATCH] Compiling under Windows

[Randy W. S. <Ra...@Th...>]

I've included the complete file instead of a patch since the original is 
just an empty skeleton file. M::B now successfully compiles under 
Windows, and it passes all tests (most of the time, see below). It could 
stand to be tightened up a bit, but I said I'd submit it by the weekend, 
and I'm pretty much out of time :). It works fine; it's just that there 
is some duplication of code which could probably be eliminated.

It's a bit more code than I first anticipated, but I thought it might be 
worthwile to take advantage of the fact that most compiler/linkers can 
be scripted. When possible it writes compiler/linker options to a file 
instead of the commandline; this will hopefully avoid problems with some 
versions of the Windows shell that have limits on commandline length. 
I've set this up so that an option can be easily added to turn it on or 
off if desired.

Outstanding issues:

The 'runthrough.t' test sometimes fails in random places at random 
times. This is true with or without the modifications I've made. 
Sometimes 2 tests fail, sometimes 5, sometimes ??, most of the time 
none--It's completly unpredictable. It happens after ./build (real)clean 
as often as it happens after back to back runs of ./build test. I have 
not yet had a chance to look into it, but I'm pretty certain of the 
cause. Windows seems to be extremely sensitive about file operations. 
Sometimes a file gets locked (not always the same one in this case), and 
it can no longer be accessed. This usually means that a file handle was 
not explicitly closed or a tied file was not untied, etc.

Another place I ran into problems with locked files is with the 
'system()' call. My first stab at getting M::B to work was basically a 
copy, paste, and edit of the 'compile_c()' and 'link_c()' routines in 
Base.pm, with branches all over the place to handle differences in 
compiler syntax. When passing a LIST to system(), It would compile, but 
the next call to system() to invoke the linker would fail with an 
'access denied' error. If I joined the list into a single string it 
would work (don't ask how long it took me to track that down to the 
system() call). At some point I decided to revisit the issue to figure 
out exactly what was happening, unfortunatelly changing back to the LIST 
form now works as expected--It has never failed with that error again. 
This was after I restructured the code, so I can only assume... bah, I 
won't assume anything.....

Under Windows, the ExtUtils::Mksymlist module is required to generate a 
file that instructs the linker on what functions/variable to 
export/import. Most of the time it is invoked with just the name of the 
module being built, but sometimes it's neccessary to pass in lists of 
functions/variables. I thought it would be best to leave it to you as to 
  how the user will pass the information through. As it is, it should 
work for most modules, and a small change to 'sub prelink()' telling it 
where to get the user supplied options will allow it to work those 
modules that need that extra functionality.

If the xs.t test fails to find a compiler, Shouldn't it NOT fail, but 
simply notify the user that there is no compiler and that it will 
operate in a degraded capacity, i.e. only installing pure perl modules?

The first of the week is always pretty busy for me, but I'll try to look 
into some of the outstanding issues later this week.

Randy.

Re: [Module-build-general] [PATCH] Compiling under Windows

[Ken W. <ke...@ma...>]

On Sunday, February 23, 2003, at 05:09  PM, Randy W. Sims wrote:

> I've included the complete file instead of a patch since the original 
> is just an empty skeleton file. M::B now successfully compiles under 
> Windows, and it passes all tests (most of the time, see below).

Awesome, thanks!

> It could stand to be tightened up a bit, but I said I'd submit it by 
> the weekend, and I'm pretty much out of time :). It works fine; it's 
> just that there is some duplication of code which could probably be 
> eliminated.

That can wait until the future.  To do this cleanly, I think we could 
create a module like ExtUtils::C with classes like 
ExtUtils::C::Compiler and ExtUtils::C::Linker that Module::Build could 
just use transparently.  Its purpose would be to compile and link 
things that will be linked in with Perl modules (not a general tool for 
arbitrary C code), using the same interface regardless of platform.

We could use that module in other situations too, like in the testing 
code for ExtUtils::ParseXS.

Probably the code you've written for Win32 and the code I wrote for 
Unix would be a good start toward such a module.  But given that the 
module doesn't exist yet, we'll just have to wait. =)

Also, I think a "uses-a" or "has-a" relationship to the compiler/linker 
would make more sense than the "is-a" relationship that's now in the 
M::B::P::Windows.pm code, but I understand that this was probably an 
easier way to get things written.


> It's a bit more code than I first anticipated, but I thought it might 
> be worthwile to take advantage of the fact that most compiler/linkers 
> can be scripted. When possible it writes compiler/linker options to a 
> file instead of the commandline; this will hopefully avoid problems 
> with some versions of the Windows shell that have limits on 
> commandline length. I've set this up so that an option can be easily 
> added to turn it on or off if desired.

Sounds good.

> Outstanding issues:
>
> The 'runthrough.t' test sometimes fails in random places at random 
> times. This is true with or without the modifications I've made. 
> Sometimes 2 tests fail, sometimes 5, sometimes ??, most of the time 
> none--It's completly unpredictable. It happens after ./build 
> (real)clean as often as it happens after back to back runs of ./build 
> test. I have not yet had a chance to look into it, but I'm pretty 
> certain of the cause. Windows seems to be extremely sensitive about 
> file operations. Sometimes a file gets locked (not always the same one 
> in this case), and it can no longer be accessed. This usually means 
> that a file handle was not explicitly closed or a tied file was not 
> untied, etc.

Yes, there were a couple of places I had to deal with this problem for 
the Win32 folks.  Are you using the latest CVS version of M::B, or the 
CPAN version?  The CVS version fixed a few things, but it's likely 
there are more instances of this problem lurking.  Is there any way to 
get an error message to tell you what file is in contention?


> Another place I ran into problems with locked files is with the 
> 'system()' call. My first stab at getting M::B to work was basically a 
> copy, paste, and edit of the 'compile_c()' and 'link_c()' routines in 
> Base.pm, with branches all over the place to handle differences in 
> compiler syntax. When passing a LIST to system(), It would compile, 
> but the next call to system() to invoke the linker would fail with an 
> 'access denied' error. If I joined the list into a single string it 
> would work (don't ask how long it took me to track that down to the 
> system() call).

Ack - I'm really trying to avoid system(STRING) calls, because I really 
don't want shell metachars involved anywhere.  Differences in shell 
behavior can open up a whole new can of worms I'm trying to avoid.


> At some point I decided to revisit the issue to figure out exactly 
> what was happening, unfortunatelly changing back to the LIST form now 
> works as expected--It has never failed with that error again. This was 
> after I restructured the code, so I can only assume... bah, I won't 
> assume anything.....

We'll just keep our eye on it, I guess.


> Under Windows, the ExtUtils::Mksymlist module is required to generate 
> a file that instructs the linker on what functions/variable to 
> export/import.

This dependency on Mksymlist (which I agree is necessary) is a good 
example of how prerequisites can vary based on the system, and suggests 
that the "mini-language" dependency syntax I wrote about last week is 
getting pretty important.  Otherwise we have to have dynamic dependency 
lists depending on the platform, which I don't like.


> Most of the time it is invoked with just the name of the module being 
> built, but sometimes it's neccessary to pass in lists of 
> functions/variables. I thought it would be best to leave it to you as 
> to  how the user will pass the information through. As it is, it 
> should work for most modules, and a small change to 'sub prelink()' 
> telling it where to get the user supplied options will allow it to 
> work those modules that need that extra functionality.
>

Okay, so people could override prelink_c() in their own subclasses?

> If the xs.t test fails to find a compiler, Shouldn't it NOT fail, but 
> simply notify the user that there is no compiler and that it will 
> operate in a degraded capacity, i.e. only installing pure perl > modules?

Yes, you're right.


> The first of the week is always pretty busy for me, but I'll try to 
> look into some of the outstanding issues later this week.

Great.  I think I'm going to push this release out before you have a 
chance to get to these issues, to let other people bang on it a little 
bit.  It's certainly way less broken now than it used to be, from the 
looks of things. =)

  -Ken

Re: [Module-build-general] [PATCH] Compiling under Windows

[Randy W. S. <Ra...@Th...>]

On 2/24/2003 12:56 PM, Ken Williams wrote:
> That can wait until the future.  To do this cleanly, I think we could 
> create a module like ExtUtils::C with classes like ExtUtils::C::Compiler 
> and ExtUtils::C::Linker that Module::Build could just use 
> transparently.  Its purpose would be to compile and link things that 
> will be linked in with Perl modules (not a general tool for arbitrary C 
> code), using the same interface regardless of platform.
> 
> We could use that module in other situations too, like in the testing 
> code for ExtUtils::ParseXS.
> 
> Probably the code you've written for Win32 and the code I wrote for Unix 
> would be a good start toward such a module.  But given that the module 
> doesn't exist yet, we'll just have to wait. =)

I've been thinking very seriously about writing those modules. My 
problem is the more I think about it the more ambitious the design 
becomes. I'm more inclined at the moment to design them as a framework 
not limited to 'C' and 'XS', so that Inline::* modules could make use of 
the framework if desired. Instead of ExtUtils::C::Linker, there would be 
something like:

my $cc = new ExtUtils::Compiler(
   LANG  => 'C',
   FLAGS => ...
);

$cc->compile( source => 'source.c', output => 'source.o' );

In other words, it would operate like 'gcc' does, where it is a 
front-end driver for any number of different compilers.

> Also, I think a "uses-a" or "has-a" relationship to the compiler/linker 
> would make more sense than the "is-a" relationship that's now in the 
> M::B::P::Windows.pm code, but I understand that this was probably an 
> easier way to get things written.

I agree, in fact the first implementation actually used an aggregate 
relationship instead of inheritance. In fact there is still a relic of 
that design in there. In the constructor, you see:

my $cc_driver = "Module::Build::Platform::Windows::$cf->{ccname}";
unshift @ISA, $cc_driver;

the next line used the be something like
$self->{config}{cc_driver} = eval "new $cc_driver";

The problem was that it would have taken more code to implement mostly 
duplicating code already in M::B or storing a backref to the caller 
(which I generally prefer to avoid), and that really didn't seem to make 
sense within the context of M::B.

>> Outstanding issues:
>>
>> The 'runthrough.t' test sometimes fails in random places at random 
>> times. This is true with or without the modifications I've made. 
>> Sometimes 2 tests fail, sometimes 5, sometimes ??, most of the time 
>> none--It's completly unpredictable. It happens after ./build 
>> (real)clean as often as it happens after back to back runs of ./build 
>> test. I have not yet had a chance to look into it, but I'm pretty 
>> certain of the cause. Windows seems to be extremely sensitive about 
>> file operations. Sometimes a file gets locked (not always the same one 
>> in this case), and it can no longer be accessed. This usually means 
>> that a file handle was not explicitly closed or a tied file was not 
>> untied, etc.
> 
> 
> Yes, there were a couple of places I had to deal with this problem for 
> the Win32 folks.  Are you using the latest CVS version of M::B, or the 
> CPAN version?  The CVS version fixed a few things, but it's likely there 
> are more instances of this problem lurking.  Is there any way to get an 
> error message to tell you what file is in contention?

In my notes, I put down the names of two of the files: 'Sample.pm' & 
'MANIFEST'. There may be others; I admit that I mostly ignored the 
errors at the time because I was focusing on the compiling/linking tests 
and I didn't want to get sidetracked. I'll take a closer look at this.

>> Another place I ran into problems with locked files is with the 
>> 'system()' call. My first stab at getting M::B to work was basically a 
>> copy, paste, and edit of the 'compile_c()' and 'link_c()' routines in 
>> Base.pm, with branches all over the place to handle differences in 
>> compiler syntax. When passing a LIST to system(), It would compile, 
>> but the next call to system() to invoke the linker would fail with an 
>> 'access denied' error. If I joined the list into a single string it 
>> would work (don't ask how long it took me to track that down to the 
>> system() call).
> 
> 
> Ack - I'm really trying to avoid system(STRING) calls, because I really 
> don't want shell metachars involved anywhere.  Differences in shell 
> behavior can open up a whole new can of worms I'm trying to avoid.
> 
> 
>> At some point I decided to revisit the issue to figure out exactly 
>> what was happening, unfortunatelly changing back to the LIST form now 
>> works as expected--It has never failed with that error again. This was 
>> after I restructured the code, so I can only assume... bah, I won't 
>> assume anything.....
> 
> 
> We'll just keep our eye on it, I guess.

This is something else I wanted to revisit. If I'm not mistaken this may 
actually be a Perl bug that was in 5.6.1, but has since been fixed. At 
least I think I remember some problem being fixed in the system() call.

>> Most of the time it is invoked with just the name of the module being 
>> built, but sometimes it's neccessary to pass in lists of 
>> functions/variables. I thought it would be best to leave it to you as 
>> to  how the user will pass the information through. As it is, it 
>> should work for most modules, and a small change to 'sub prelink()' 
>> telling it where to get the user supplied options will allow it to 
>> work those modules that need that extra functionality.
>>
> 
> Okay, so people could override prelink_c() in their own subclasses?

I guess it could be argued either way: subclassing or passing arguments 
to M::B's constructor. If subclassing, users need to know that on some 
platforms that there is a prelink operation, and that it generally 
involves a call ExtUtils::Mksymlists. They will have to reference that 
documentation to see what arguments they need to pass. If it were just 
parameters passed to M::B's constructor, It would arguably simplify the 
interface and documentation, at least from the users perspective. 
However, design-wise it shifts responsiblities in the wrong direction; 
the reason there is a seperate module for Mksymlists in the first place 
is to move that responsiblity out of the package used for building modules.

The way I wrote the prelink_c() routine should allow you easily go in 
either direction as you prefer.

Randy.

Re: [Module-build-general] [PATCH] Compiling under Windows

[Ken W. <ke...@ma...>]

On Tuesday, February 25, 2003, at 06:53  PM, Randy W. Sims wrote:

> On 2/24/2003 12:56 PM, Ken Williams wrote:
>> That can wait until the future.  To do this cleanly, I think we could 
>> create a module like ExtUtils::C with classes like 
>> ExtUtils::C::Compiler and ExtUtils::C::Linker that Module::Build 
>> could just use transparently.  Its purpose would be to compile and 
>> link things that will be linked in with Perl modules (not a general 
>> tool for arbitrary C code), using the same interface regardless of 
>> platform.
>> We could use that module in other situations too, like in the testing 
>> code for ExtUtils::ParseXS.
>> Probably the code you've written for Win32 and the code I wrote for 
>> Unix would be a good start toward such a module.  But given that the 
>> module doesn't exist yet, we'll just have to wait. =)
>
> I've been thinking very seriously about writing those modules. My 
> problem is the more I think about it the more ambitious the design 
> becomes. I'm more inclined at the moment to design them as a framework 
> not limited to 'C' and 'XS', so that Inline::* modules could make use 
> of the framework if desired. Instead of ExtUtils::C::Linker, there 
> would be something like:
>
> my $cc = new ExtUtils::Compiler(
>   LANG  => 'C',
>   FLAGS => ...
> );
>
> $cc->compile( source => 'source.c', output => 'source.o' );
>
> In other words, it would operate like 'gcc' does, where it is a 
> front-end driver for any number of different compilers.

Eek, dangerous road! ;-)  When I've entertained writing this, it's only 
by limiting the scope of the design that I can convince myself it's 
worth doing, because I might have a reasonable chance of ever 
finishing. =)

That said, probably all you'd need to do is define the *interfaces*, 
implement the stuff we need for C compilation, and let other people 
provide the code to support various languages.

By the way, one other author (Mitchell Charity) told me a while ago 
about some work he was doing with ExtUtils::ParseXS in which he was 
generating the C code in memory, then using TinyCC's libtcc to do an 
all-in-memory compile and link, and claiming something like three 
orders of magnitude difference in speed (faster, not slower ;-).  I 
probably wouldn't even understand the details if I knew them, but my 
point is that tools like this could be really useful, because there are 
people out there that can do cool stuff with them.  So if you think you 
could get something working, I'm completely supportive. =)

> I agree, in fact the first implementation actually used an aggregate 
> relationship instead of inheritance. In fact there is still a relic of 
> that design in there. In the constructor, you see:
>
> my $cc_driver = "Module::Build::Platform::Windows::$cf->{ccname}";
> unshift @ISA, $cc_driver;
>
> the next line used the be something like
> $self->{config}{cc_driver} = eval "new $cc_driver";

By the way, that can just be

  $self->{config}{cc_driver} = $cc_driver->new();


> The problem was that it would have taken more code to implement mostly 
> duplicating code already in M::B or storing a backref to the caller 
> (which I generally prefer to avoid), and that really didn't seem to 
> make sense within the context of M::B.

Okay.  One way to avoid storing backreferences (for too long) is to use 
weak references, but we probably don't want to get into that for M::B 
either.

>
> In my notes, I put down the names of two of the files: 'Sample.pm' & 
> 'MANIFEST'. There may be others; I admit that I mostly ignored the 
> errors at the time because I was focusing on the compiling/linking 
> tests and I didn't want to get sidetracked. I'll take a closer look at 
> this.

Does anything change if you add an explicit close() in 
version_from_file() ?  I think that's the only place Sample.pm would be 
opened.

For the MANIFEST, all that code is buried in ExtUtils::Manifest, so 
it'll be harder to fix, probably.


> This is something else I wanted to revisit. If I'm not mistaken this 
> may actually be a Perl bug that was in 5.6.1, but has since been 
> fixed. At least I think I remember some problem being fixed in the 
> system() call.

Oh yuck. =(  Well, let me know if you find anything, but I hope you 
don't!


> I guess it could be argued either way: subclassing or passing 
> arguments to M::B's constructor. If subclassing, users need to know 
> that on some platforms that there is a prelink operation, and that it 
> generally involves a call ExtUtils::Mksymlists. They will have to 
> reference that documentation to see what arguments they need to pass. 
> If it were just parameters passed to M::B's constructor, It would 
> arguably simplify the interface and documentation, at least from the 
> users perspective. However, design-wise it shifts responsiblities in 
> the wrong direction; the reason there is a seperate module for 
> Mksymlists in the first place is to move that responsiblity out of the 
> package used for building modules.

Indeed.

> The way I wrote the prelink_c() routine should allow you easily go in 
> either direction as you prefer.

Okay, thanks.

  -Ken

Re: [Module-build-general] [PATCH] Compiling under Windows

[Randy W. S. <Ra...@Th...>]

>> In my notes, I put down the names of two of the files: 'Sample.pm' & 
>> 'MANIFEST'. There may be others; I admit that I mostly ignored the 
>> errors at the time because I was focusing on the compiling/linking 
>> tests and I didn't want to get sidetracked. I'll take a closer look at 
>> this.
> 
> 
> Does anything change if you add an explicit close() in 
> version_from_file() ?  I think that's the only place Sample.pm would be 
> opened.
> 
> For the MANIFEST, all that code is buried in ExtUtils::Manifest, so 
> it'll be harder to fix, probably.
> 

It appears that all of the problems come from ExtUtils::Manifest. I have 
not completely narrowed down the problem there, but I suspect it is a 
bug in Perl's link() implementation on MSWin32 (which EU::Manifest only 
attempts on Windows NT variants, and not Windows 9x). The problem is 
eliminated by changing the call to EU::Manifest::manicopy() in 
M::B::Base::ACTION_distclean() so that the last parameter is 'cp' 
instead of 'best'. This of course is not the correct solution, nor is 
copying the entire sub to Windows.pm just to change one line. Maybe an 
option can be set in $self->{config} to indicate whether link()ing 
should be avoided: The constructor in Base.pm could set a default, 
allowing links, and any subclass that needed to could change it. Then 
the line in ACTION_distclean() could be changed to pass the appropriate 
flag to manicopy() based on the config option???

>> This is something else I wanted to revisit. If I'm not mistaken this 
>> may actually be a Perl bug that was in 5.6.1, but has since been 
>> fixed. At least I think I remember some problem being fixed in the 
>> system() call.
> 
> Oh yuck. =(  Well, let me know if you find anything, but I hope you don't!

I've tried a few times to reproduce the problem with the system() call, 
but have been unsuccessfull. I'm sure I have run through the tests a few 
thousand times (using scripts, not manually ;) using various compilers 
and version of perl and this has never resurfaced, so I think it 
unlikely it will come up again; I just wish I knew from whence it came 
and to whither it went.

Randy.