wxhaskell-devel

[wxhaskell-devel] Using a shared library for the C++ in wxhaskell

[Dave T. <duk...@gm...>]

Hello everyone,

I wrote a story, and I invite you all to comment and help me make it better.
As an experiment I've decided to put it on hpaste instead of having
the mail thread get out of hand (I'm also cross posting it to the
cabal list and have mentioned it in #haskell, and I want to keep all
correspondence in one place):
http://hpaste.org/55027

Dave,

Re: [wxhaskell-devel] Using a shared library for the C++ in wxhaskell

[Dave T. <duk...@gm...>]

On 8 December 2011 22:34, Dave Tapley <duk...@gm...> wrote:
> Hello everyone,
>
> I wrote a story, and I invite you all to comment and help me make it better.
> As an experiment I've decided to put it on hpaste instead of having
> the mail thread get out of hand (I'm also cross posting it to the
> cabal list and have mentioned it in #haskell, and I want to keep all
> correspondence in one place):
> http://hpaste.org/55027

Ha, well that was good timing for hpaste to go down!
Here's it is:

I've been trying to resurrect the idea of building a shared library
for the C++ component of the wxhaskell library.

Jeremy (to my knowledge) first put this forward, here:
http://wewantarock.wordpress.com/2010/11/01/working-around-the-static-libstdc-restriction/

In addition to the advantages he lists there, I have the following reason:
When building wx-core (in its current incarnation) cabal will *always*
rebuild all the C++ code, which takes a majority of the build time.
This becomes very frustrating when you change one line of Haskell and
have to wait seven minutes to rebuild.
I complained about it here:
http://sourceforge.net/mailarchive/message.php?msg_id=28099997

So, I decided to try and stop this complete C++ rebuild, I partially
succeed, but I eventually got stuck for reasons I complained here:
http://www.haskell.org/pipermail/cabal-devel/2011-October/007816.html

Side note: I did eventually discover why (or, where) the c-sources
(cSources) list is used to compile and link, and it is here:
http://hackage.haskell.org/packages/archive/Cabal/latest/doc/html/src/Distribution-Simple-GHC.html#buildLib
You can see:
"| filename <- cSources libBi]" under "-- build any C sources", and
you can see:
"cSharedObjs = map (`replaceExtension` ("dyn_" ++ objExtension))
(cSources libBi)"
under "-- link:".
Is this assumption correct?

At this point I thought about either:
1. Getting the cabal source and starting to write code to give
BuildInfo a "cObjs" in addition to "cSources".
2. Picking up Jeremy's shared library code, so wxhaskell would have
its own code to build the library, in which I could do sensible
re-compilation.

Given that there were other advantages to 2, I went with that.

Jeremy had written one blog post on building a such a shared library, here:
http://wewantarock.wordpress.com/2010/11/03/building-a-shared-library-in-cabal/
In response to an email on the wxhaskell-devel list he also kindly put
up this gist, with the code he'd been working on:
https://gist.github.com/1301115

I dutifully took this gist, and have now attempted to integrate in to
my wxhaskell-dev branch, which you can find here:
http://darcsden.com/DukeDave/wxhaskell-dev
(note: I haven't pushed any of the shared library code to darcsden
yet, for reasons I'm about to explain)

This is where things got interesting, after a few hours of hacking I
have built a shared library, but in this very back-handed way:

Firstly, in Jeremy's code the to-be-compiled shared library is added
to the wxcore BuildInfo through a custom hook.
We can see this because:
> let all_dlls   = parseDLLs ["x-dll-name", "x-dll-extra-libraries"] custom_bi
(the modified wxcore.cabal contains the line: "x-dll-name: wxc")

However on a clean build of wxcore we get the following error:

setup: Missing dependency on a foreign library:
* Missing C library: wxc
This problem can usually be solved by installing the system package that
provides this library (you may need the "-dev" version). If the library is
already installed but in a non-standard location then you can use the flags
--extra-include-dirs= and --extra-lib-dirs= to specify where it is.

The error is dumped *before* cabal calls myBuildHook, and since this
actually builds the library the error makes sense; cabal is looking
for the shared library before we've built it.

To get around this I modified the offending line, thus:
> let all_dlls   = parseDLLs ["x-dll-extra-libraries"] custom_bi

With that modification cabal would now get to myBuildHook, but then
another curious error arose:
We see that the actual linking is called here:
> runProgram verbose gcc (opts' ++ objs' ++ lib_dirs' ++ libs')

But on hitting that line the following error is spat out:
/usr/bin/ld: cannot open output file  dist/build/libwxc.so.0.13.1: No
such file or directory

I checked all my permissions and couldn't see anything wrong, I could
touch the file.
Conveniently I noticed that, if the verbosity is set high enough,
runProgram will call printRawCommandAndArgs:
http://hackage.haskell.org/packages/archive/Cabal/latest/doc/html/src/Distribution-Simple-Utils.html#printRawCommandAndArgs

The output (i.e. the linker invocation) looks like this:
/usr/bin/gcc -fno-stack-protector -shared -Wl,-soname,libwxc.so.0 -o
dist/build/libwxc.so.0.13.1 dist/build/src/cpp/apppath.o
dist/build/src/cpp/dragimage.o dist/build/src/cpp/eljaccelerator.o
[snip-rest-of-.o-files] -L/usr/local/lib -lstdc++ -lwx_baseu-2.9
[snip-rest-of-wx-libs]

Now if I cd into the wxcore and paste the command *verbatim* then gcc
works and generates libwxc.so.0.13.1 as expected.
You can see in Jeremy's code the linkShareLib function contains:
> cwd <- getCurrentDirectory

I used this to confirm that the we were in ./wxcore and we are, even
making the path for -o absolute didn't sovle the issue.
I ended up replacing runProgram line with this, less satisfactory line:
> system $ (unwords ([show . locationPath . programLocation $ gcc] ++ opts' ++ objs' ++ lib_dirs' ++ libs'))

Which works, but still doesn't explain why runProgram doesn't.
Any suggestions?

So with that complete I was able to link a shared library to:
wxcore/dist/build/libwxc.so.0.13.1

Of course, at this point we've generated the shared library, but
wxcore still isn't aware of it, so the build completes successfully,
but any attempt to compile against it results (understandably) in
hundreds of linker errors.
To resolve this I had to add "x-dll-name" (read in from wxcore.cabal
as "wxc") back to parseDLLs:
> let all_dlls   = parseDLLs ["x-dll-name", "x-dll-extra-libraries"] custom_bi

Well, not quite, attempt to build wxcore again and we're back to this error:
* Missing C library: wxc

Of course, the .so isn't in a 'normal' place, so add its location as
an extra lib directory:
> { extraLibDirs = extraLibDirs libbi ++ extraLibDirs wx ++ ["/full/path/to/wxcore/dist/build"]
(note: I tried using just "dist/build", but cabal said: "library-dirs:
dist/build is a relative path")

This still isn't quite enough, it took one more kludge:
/full/path/to/wxcore/dist/build/$ ln -s
/full/path/to/wxcore/dist/build/libwxc.so.0.13.1 libwxc.so

I'd like to think there's a more elegant solution than this and I'm
open to suggestions.
I should note that at this point I became aware that we could have an
entirely separate cabal project (perhaps "wxc") for this shared
library, and then have wxcore depend on it?

Running one more time I *thought* I was finally there, until I noticed
this line, hidden in cabal's output:
/full/path/to/wxcore/dist/build/libwxc.so: file not recognized: File truncated

As far as I can tell, this occurs because we now load(?)
"libwxc.so.0.13.1" in myConfHook, but then in myBuildHook we want to
use it as the destination for the linker. Worse still it seems that
'truncated' actually means 'deleted'. To get around this I had to one
again remove "x-dll-name" from the parseDLLs call (so I could get to
myBuildHook and create "libwxc.so.0.13.1" again), build wxcore, then
add "x-dll-name" back again, but this time also comment out the call
to linkSharedLib (and so stop cabal attempting to open
"libwxc.so.0.13.1") before building wxcore one last time.

Finally everything seemed to work.

Now, I should note that I have been using cabal-dev, and so I now have
a package-conf I need to reference when building with my new shared
library wxhaskell. To test it I built the wxhaskell HelloWorld sample
thus:
/full/path/to/samples/wxcore$ ghc --make -package-conf
../../cabal-dev/packages-7.0.3.conf HelloWorld.hs

It worked, but then when I try to run HelloWorld I get:
./HelloWorld: error while loading shared libraries: libwxc.so.0:
cannot open shared object file: No such file or directory

Okay, again, libwxc.so.0 isn't in a 'normal' place, so I kludged it
with another symlink:
/full/path/to/samples/wxcore$ ln -s
/full/path/to/wxcore/dist/build/libwxc.so libwxc.so.0

And now, I can report that HelloWorld runs.
[imagine a screen shot of the HelloWorld sample application running here]
Rejoice.

But wait, there's more:
One of the promises of the shared library approach is that we'd all be
able to use wxhaskell with ghci again, and to my utter disbelief, it's
true:
/full/path/to/samples/wxcore$ ghci -package-conf
../../cabal-dev/packages-7.0.3.conf HelloWorld.hs
Ok, modules loaded: Main.
Prelude Main> main

[imagine a screen shot of the HelloWorld sample application running here]
Double rejoice.



>
> Dave,

Re: [wxhaskell-devel] Using a shared library for the C++ in wxhaskell

[Jeremy O'D. <jer...@gm...>]

Hi Dave, all,

This is fantastic news - especially the bit about GHCi. I have put a couple
of general comments inline, but it looks like you have found most of the
issues, at least on Linux.

I think that at least some of the issues would go away if, as you say,

*"we could have an entirely separate cabal project (perhaps "wxc") for this
shared library, and then have wxcore depend on it?"
*
In this case, we would be able to ensure that libwxc.so.x.y.z (or wxc.dll
or libwxc.dylib or whatever) was in a 'normal' place before wxcore gets
built.

On 9 December 2011 14:13, Dave Tapley <duk...@gm...> wrote:

> On 8 December 2011 22:34, Dave Tapley <duk...@gm...> wrote:
> > Hello everyone,
> >
> > I wrote a story, and I invite you all to comment and help me make it
> better.
> > As an experiment I've decided to put it on hpaste instead of having
> > the mail thread get out of hand (I'm also cross posting it to the
> > cabal list and have mentioned it in #haskell, and I want to keep all
> > correspondence in one place):
> > http://hpaste.org/55027
>
> Ha, well that was good timing for hpaste to go down!
> Here's it is:
>
> I've been trying to resurrect the idea of building a shared library
> for the C++ component of the wxhaskell library.
>
> Jeremy (to my knowledge) first put this forward, here:
>
> http://wewantarock.wordpress.com/2010/11/01/working-around-the-static-libstdc-restriction/
>
> In addition to the advantages he lists there, I have the following reason:
> When building wx-core (in its current incarnation) cabal will *always*
> rebuild all the C++ code, which takes a majority of the build time.
> This becomes very frustrating when you change one line of Haskell and
> have to wait seven minutes to rebuild.
> I complained about it here:
> http://sourceforge.net/mailarchive/message.php?msg_id=28099997
>
> So, I decided to try and stop this complete C++ rebuild, I partially
> succeed, but I eventually got stuck for reasons I complained here:
> http://www.haskell.org/pipermail/cabal-devel/2011-October/007816.html
>
> Side note: I did eventually discover why (or, where) the c-sources
> (cSources) list is used to compile and link, and it is here:
>
> http://hackage.haskell.org/packages/archive/Cabal/latest/doc/html/src/Distribution-Simple-GHC.html#buildLib
> You can see:
> "| filename <- cSources libBi]" under "-- build any C sources", and
> you can see:
> "cSharedObjs = map (`replaceExtension` ("dyn_" ++ objExtension))
> (cSources libBi)"
> under "-- link:".
> Is this assumption correct?
>

This is correct - it's the standard way Cabal builds C/C++ code. To be
honest, this is really a consequence of the fact that Cabal is more of a
tool for simple distribution than a Make replacement, and the developers
probably didn't expect Cabal would be used to build large bodies of C++
code.

My approach is far from perfect, too - in fact it is incorrect because I
don't do dependency tracking on header files, so if you edit a header, it
(incorrectly) fails to rebuild. This is because I wasn't keen on rewriting
'make' in the wxHaskell build system - it seems to me like something Cabal
could usefully support. I did play with some very hacky code which
automatically rebuilt all of the C++ code if any header is newer than any
C++ source. This is aesthetically dreadful, but it somewhat captures the
reality, since in practice, pretty much all of the C++ files depend on all
of the headers (and it's easy to write).

On reflection, we should probably do this as what exists at present could
lead to unexpectedly incorrect code being generated if someone is in full
flow of development. I'll see if I can dig up some code - I think I have it
somewhere.

At this point I thought about either:
> 1. Getting the cabal source and starting to write code to give
> BuildInfo a "cObjs" in addition to "cSources".
> 2. Picking up Jeremy's shared library code, so wxhaskell would have
> its own code to build the library, in which I could do sensible
> re-compilation.
>
> Given that there were other advantages to 2, I went with that.
>
> Jeremy had written one blog post on building a such a shared library, here:
>
> http://wewantarock.wordpress.com/2010/11/03/building-a-shared-library-in-cabal/
> In response to an email on the wxhaskell-devel list he also kindly put
> up this gist, with the code he'd been working on:
> https://gist.github.com/1301115
>
> I dutifully took this gist, and have now attempted to integrate in to
> my wxhaskell-dev branch, which you can find here:
> http://darcsden.com/DukeDave/wxhaskell-dev
> (note: I haven't pushed any of the shared library code to darcsden
> yet, for reasons I'm about to explain)
>
> This is where things got interesting, after a few hours of hacking I
> have built a shared library, but in this very back-handed way:
>
> Firstly, in Jeremy's code the to-be-compiled shared library is added
> to the wxcore BuildInfo through a custom hook.
> We can see this because:
> > let all_dlls   = parseDLLs ["x-dll-name", "x-dll-extra-libraries"]
> custom_bi
> (the modified wxcore.cabal contains the line: "x-dll-name: wxc")
>
> However on a clean build of wxcore we get the following error:
>
> setup: Missing dependency on a foreign library:
> * Missing C library: wxc
> This problem can usually be solved by installing the system package that
> provides this library (you may need the "-dev" version). If the library is
> already installed but in a non-standard location then you can use the flags
> --extra-include-dirs= and --extra-lib-dirs= to specify where it is.
>

My code worked for Windows - but this is a better cross-platform solution.


> The error is dumped *before* cabal calls myBuildHook, and since this
> actually builds the library the error makes sense; cabal is looking
> for the shared library before we've built it.
>
> To get around this I modified the offending line, thus:
> > let all_dlls   = parseDLLs ["x-dll-extra-libraries"] custom_bi
>
> With that modification cabal would now get to myBuildHook, but then
> another curious error arose:
> We see that the actual linking is called here:
> > runProgram verbose gcc (opts' ++ objs' ++ lib_dirs' ++ libs')
>
> But on hitting that line the following error is spat out:
> /usr/bin/ld: cannot open output file  dist/build/libwxc.so.0.13.1: No
> such file or directory
>

Windows doesn't behave this way - this part worked for me as well.


> I checked all my permissions and couldn't see anything wrong, I could
> touch the file.
> Conveniently I noticed that, if the verbosity is set high enough,
> runProgram will call printRawCommandAndArgs:
>
> http://hackage.haskell.org/packages/archive/Cabal/latest/doc/html/src/Distribution-Simple-Utils.html#printRawCommandAndArgs
>
> The output (i.e. the linker invocation) looks like this:
> /usr/bin/gcc -fno-stack-protector -shared -Wl,-soname,libwxc.so.0 -o
> dist/build/libwxc.so.0.13.1 dist/build/src/cpp/apppath.o
> dist/build/src/cpp/dragimage.o dist/build/src/cpp/eljaccelerator.o
> [snip-rest-of-.o-files] -L/usr/local/lib -lstdc++ -lwx_baseu-2.9
> [snip-rest-of-wx-libs]
>
> Now if I cd into the wxcore and paste the command *verbatim* then gcc
> works and generates libwxc.so.0.13.1 as expected.
> You can see in Jeremy's code the linkShareLib function contains:
> > cwd <- getCurrentDirectory
>
> I used this to confirm that the we were in ./wxcore and we are, even
> making the path for -o absolute didn't sovle the issue.
> I ended up replacing runProgram line with this, less satisfactory line:
> > system $ (unwords ([show . locationPath . programLocation $ gcc] ++
> opts' ++ objs' ++ lib_dirs' ++ libs'))
>
> Which works, but still doesn't explain why runProgram doesn't.
> Any suggestions?
>

I've tried this on Linux. You're absolutely right and I'm stumped as to
why. I think we should pull this specific part of the thread out and post
to the Haskell list. Maybe someone who knows Cabal and/or GHC would have an
idea why this is the case.


> So with that complete I was able to link a shared library to:
> wxcore/dist/build/libwxc.so.0.13.1
>
> Of course, at this point we've generated the shared library, but
> wxcore still isn't aware of it, so the build completes successfully,
> but any attempt to compile against it results (understandably) in
> hundreds of linker errors.
> To resolve this I had to add "x-dll-name" (read in from wxcore.cabal
> as "wxc") back to parseDLLs:
> > let all_dlls   = parseDLLs ["x-dll-name", "x-dll-extra-libraries"]
> custom_bi
>
> Well, not quite, attempt to build wxcore again and we're back to this
> error:
> * Missing C library: wxc
>
> Of course, the .so isn't in a 'normal' place, so add its location as
> an extra lib directory:
> > { extraLibDirs = extraLibDirs libbi ++ extraLibDirs wx ++
> ["/full/path/to/wxcore/dist/build"]
> (note: I tried using just "dist/build", but cabal said: "library-dirs:
> dist/build is a relative path")
>

I think we need a more robust solution in the longer run, as
<wxhaskell>/wcore/dirt/build is really a temporary location.


> This still isn't quite enough, it took one more kludge:
> /full/path/to/wxcore/dist/build/$ ln -s
> /full/path/to/wxcore/dist/build/libwxc.so.0.13.1 libwxc.so
>

Earlier you mentioned that, at least on Linux, the build command is:

/usr/bin/gcc -fno-stack-protector -shared -Wl,-soname,libwxc.so.0 -o
 dist/build/libwxc.so.0.13.1 dist/build/src/cpp/apppath.o
dist/build/src/cpp/dragimage.o dist/build/src/cpp/eljaccelerator.o
[snip-rest-of-.o-files] -L/usr/local/lib -lstdc++ -lwx_baseu-2.9
[snip-rest-of-wx-libs]

I know GCC generated this and not you, but it looks like the root of the
issue. I am slightly confused though. ld.so is supposed to *know* that
libwxc.so.0.13.1 is a valid instance of libwxc.so.0

In normal Linux-land, you run /sbin/ldconfig after installing a new
library, and it creates these symlinks automatically You could do this by
executing something like (not tested)

/sbin/ldconfig -n <path-to-directory>

I'd like to think there's a more elegant solution than this and I'm
> open to suggestions.
> I should note that at this point I became aware that we could have an
> entirely separate cabal project (perhaps "wxc") for this shared
> library, and then have wxcore depend on it?
>
> Running one more time I *thought* I was finally there, until I noticed
> this line, hidden in cabal's output:
> /full/path/to/wxcore/dist/build/libwxc.so: file not recognized: File
> truncated
>
> As far as I can tell, this occurs because we now load(?)
> "libwxc.so.0.13.1" in myConfHook, but then in myBuildHook we want to
> use it as the destination for the linker. Worse still it seems that
> 'truncated' actually means 'deleted'. To get around this I had to one
> again remove "x-dll-name" from the parseDLLs call (so I could get to
> myBuildHook and create "libwxc.so.0.13.1" again), build wxcore, then
> add "x-dll-name" back again, but this time also comment out the call
> to linkSharedLib (and so stop cabal attempting to open
> "libwxc.so.0.13.1") before building wxcore one last time.
>
> Finally everything seemed to work.
>
> Now, I should note that I have been using cabal-dev, and so I now have
> a package-conf I need to reference when building with my new shared
> library wxhaskell. To test it I built the wxhaskell HelloWorld sample
> thus:
> /full/path/to/samples/wxcore$ ghc --make -package-conf
> ../../cabal-dev/packages-7.0.3.conf HelloWorld.hs
>
> It worked, but then when I try to run HelloWorld I get:
> ./HelloWorld: error while loading shared libraries: libwxc.so.0:
> cannot open shared object file: No such file or directory
>
> Okay, again, libwxc.so.0 isn't in a 'normal' place, so I kludged it
> with another symlink:
> /full/path/to/samples/wxcore$ ln -s
> /full/path/to/wxcore/dist/build/libwxc.so libwxc.so.0
>
> And now, I can report that HelloWorld runs.
> [imagine a screen shot of the HelloWorld sample application running here]
> Rejoice.
>
> But wait, there's more:
> One of the promises of the shared library approach is that we'd all be
> able to use wxhaskell with ghci again, and to my utter disbelief, it's
> true:
> /full/path/to/samples/wxcore$ ghci -package-conf
> ../../cabal-dev/packages-7.0.3.conf HelloWorld.hs
> Ok, modules loaded: Main.
> Prelude Main> main
>
> [imagine a screen shot of the HelloWorld sample application running here]
> Double rejoice.
>
>
>
> >
> > Dave,
>
>
> ------------------------------------------------------------------------------
> Cloud Services Checklist: Pricing and Packaging Optimization
> This white paper is intended to serve as a reference, checklist and point
> of
> discussion for anyone considering optimizing the pricing and packaging
> model
> of a cloud services business. Read Now!
> http://www.accelacomm.com/jaw/sfnl/114/51491232/
> _______________________________________________
> wxhaskell-devel mailing list
> wxh...@li...
> https://lists.sourceforge.net/lists/listinfo/wxhaskell-devel
>

Re: [wxhaskell-devel] Using a shared library for the C++ in wxhaskell

[Dave T. <duk...@gm...>]

On 12 December 2011 17:18, Jeremy O'Donoghue <jer...@gm...> wrote:
> Hi Dave, all,
>
> This is fantastic news - especially the bit about GHCi. I have put a couple
> of general comments inline, but it looks like you have found most of the
> issues, at least on Linux.
>
> I think that at least some of the issues would go away if, as you say,
>
> "we could have an entirely separate cabal project (perhaps "wxc") for this
> shared library, and then have wxcore depend on it?"
>
> In this case, we would be able to ensure that libwxc.so.x.y.z (or wxc.dll or
> libwxc.dylib or whatever) was in a 'normal' place before wxcore gets built.
>
> On 9 December 2011 14:13, Dave Tapley <duk...@gm...> wrote:
>>
>> On 8 December 2011 22:34, Dave Tapley <duk...@gm...> wrote:
>> > Hello everyone,
>> >
>> > I wrote a story, and I invite you all to comment and help me make it
>> > better.
>> > As an experiment I've decided to put it on hpaste instead of having
>> > the mail thread get out of hand (I'm also cross posting it to the
>> > cabal list and have mentioned it in #haskell, and I want to keep all
>> > correspondence in one place):
>> > http://hpaste.org/55027
>>
>> Ha, well that was good timing for hpaste to go down!
>> Here's it is:
>>
>> I've been trying to resurrect the idea of building a shared library
>> for the C++ component of the wxhaskell library.
>>
>> Jeremy (to my knowledge) first put this forward, here:
>>
>> http://wewantarock.wordpress.com/2010/11/01/working-around-the-static-libstdc-restriction/
>>
>> In addition to the advantages he lists there, I have the following reason:
>> When building wx-core (in its current incarnation) cabal will *always*
>> rebuild all the C++ code, which takes a majority of the build time.
>> This becomes very frustrating when you change one line of Haskell and
>> have to wait seven minutes to rebuild.
>> I complained about it here:
>> http://sourceforge.net/mailarchive/message.php?msg_id=28099997
>>
>> So, I decided to try and stop this complete C++ rebuild, I partially
>> succeed, but I eventually got stuck for reasons I complained here:
>> http://www.haskell.org/pipermail/cabal-devel/2011-October/007816.html
>>
>> Side note: I did eventually discover why (or, where) the c-sources
>> (cSources) list is used to compile and link, and it is here:
>>
>> http://hackage.haskell.org/packages/archive/Cabal/latest/doc/html/src/Distribution-Simple-GHC.html#buildLib
>> You can see:
>> "| filename <- cSources libBi]" under "-- build any C sources", and
>> you can see:
>> "cSharedObjs = map (`replaceExtension` ("dyn_" ++ objExtension))
>> (cSources libBi)"
>> under "-- link:".
>> Is this assumption correct?
>
>
> This is correct - it's the standard way Cabal builds C/C++ code. To be
> honest, this is really a consequence of the fact that Cabal is more of a
> tool for simple distribution than a Make replacement, and the developers
> probably didn't expect Cabal would be used to build large bodies of C++
> code.
>
> My approach is far from perfect, too - in fact it is incorrect because I
> don't do dependency tracking on header files, so if you edit a header, it
> (incorrectly) fails to rebuild. This is because I wasn't keen on rewriting
> 'make' in the wxHaskell build system - it seems to me like something Cabal
> could usefully support. I did play with some very hacky code which
> automatically rebuilt all of the C++ code if any header is newer than any
> C++ source. This is aesthetically dreadful, but it somewhat captures the
> reality, since in practice, pretty much all of the C++ files depend on all
> of the headers (and it's easy to write).
>
> On reflection, we should probably do this as what exists at present could
> lead to unexpectedly incorrect code being generated if someone is in full
> flow of development. I'll see if I can dig up some code - I think I have it
> somewhere.

You know, I'm surprised this didn't occur to me until now:
Does the C++ code actually need to use cabal at all?
If it's a separate project, perhaps it would be more sensible to use a
C++ oriented build system?
Then, unless I'm missing something, it can install the headers
somewhere, and wxcore can run wxdirect on the headers are currently.

The big disadvantage I can see with moving the C++ out of the wxcore
project is lots of pain for users when they 'cabal update' and
suddenly wxcore fails because they need to update the C++ headers /
library.

Another disadvantage is that this will make development a little more
painful (although the time saved not having to rebuild all the C++ is
a huge bonus) because you wouldn't (without some hackery) be able to
use cabal-dev as described here:
http://haskell.org/haskellwiki/WxHaskell/Development/Environment#Building_and_testing_wxHaskell

I happened upon this (dead) project, which I seems to have the goal I
am outlining:
http://wxc.sourceforge.net/

>
>> At this point I thought about either:
>> 1. Getting the cabal source and starting to write code to give
>> BuildInfo a "cObjs" in addition to "cSources".
>> 2. Picking up Jeremy's shared library code, so wxhaskell would have
>> its own code to build the library, in which I could do sensible
>> re-compilation.
>>
>> Given that there were other advantages to 2, I went with that.
>>
>> Jeremy had written one blog post on building a such a shared library,
>> here:
>>
>> http://wewantarock.wordpress.com/2010/11/03/building-a-shared-library-in-cabal/
>> In response to an email on the wxhaskell-devel list he also kindly put
>> up this gist, with the code he'd been working on:
>> https://gist.github.com/1301115
>>
>> I dutifully took this gist, and have now attempted to integrate in to
>> my wxhaskell-dev branch, which you can find here:
>> http://darcsden.com/DukeDave/wxhaskell-dev
>> (note: I haven't pushed any of the shared library code to darcsden
>> yet, for reasons I'm about to explain)
>>
>> This is where things got interesting, after a few hours of hacking I
>> have built a shared library, but in this very back-handed way:
>>
>> Firstly, in Jeremy's code the to-be-compiled shared library is added
>> to the wxcore BuildInfo through a custom hook.
>> We can see this because:
>> > let all_dlls   = parseDLLs ["x-dll-name", "x-dll-extra-libraries"]
>> > custom_bi
>> (the modified wxcore.cabal contains the line: "x-dll-name: wxc")
>>
>> However on a clean build of wxcore we get the following error:
>>
>> setup: Missing dependency on a foreign library:
>> * Missing C library: wxc
>> This problem can usually be solved by installing the system package that
>> provides this library (you may need the "-dev" version). If the library is
>> already installed but in a non-standard location then you can use the
>> flags
>> --extra-include-dirs= and --extra-lib-dirs= to specify where it is.
>
>
> My code worked for Windows - but this is a better cross-platform solution.

Interesting.
I found this fixed feature on cabal:
http://hackage.haskell.org/trac/hackage/ticket/262
It suggests that cabal uses Autoconf to resolve dependencies,
specifically the AC_CHECK_LIB macro.
I have zero knowledge of build systems in Windows, but is it
reasonable to suggest that Autoconf could be more relaxed?

I really should get a Windows build environment set up..

>
>>
>> The error is dumped *before* cabal calls myBuildHook, and since this
>> actually builds the library the error makes sense; cabal is looking
>> for the shared library before we've built it.
>>
>> To get around this I modified the offending line, thus:
>> > let all_dlls   = parseDLLs ["x-dll-extra-libraries"] custom_bi
>>
>> With that modification cabal would now get to myBuildHook, but then
>> another curious error arose:
>> We see that the actual linking is called here:
>> > runProgram verbose gcc (opts' ++ objs' ++ lib_dirs' ++ libs')
>>
>> But on hitting that line the following error is spat out:
>> /usr/bin/ld: cannot open output file  dist/build/libwxc.so.0.13.1: No
>> such file or directory
>
>
> Windows doesn't behave this way - this part worked for me as well.
>
>>
>> I checked all my permissions and couldn't see anything wrong, I could
>> touch the file.
>> Conveniently I noticed that, if the verbosity is set high enough,
>> runProgram will call printRawCommandAndArgs:
>>
>> http://hackage.haskell.org/packages/archive/Cabal/latest/doc/html/src/Distribution-Simple-Utils.html#printRawCommandAndArgs
>>
>> The output (i.e. the linker invocation) looks like this:
>> /usr/bin/gcc -fno-stack-protector -shared -Wl,-soname,libwxc.so.0 -o
>> dist/build/libwxc.so.0.13.1 dist/build/src/cpp/apppath.o
>> dist/build/src/cpp/dragimage.o dist/build/src/cpp/eljaccelerator.o
>> [snip-rest-of-.o-files] -L/usr/local/lib -lstdc++ -lwx_baseu-2.9
>> [snip-rest-of-wx-libs]
>>
>> Now if I cd into the wxcore and paste the command *verbatim* then gcc
>> works and generates libwxc.so.0.13.1 as expected.
>> You can see in Jeremy's code the linkShareLib function contains:
>> > cwd <- getCurrentDirectory
>>
>> I used this to confirm that the we were in ./wxcore and we are, even
>> making the path for -o absolute didn't sovle the issue.
>> I ended up replacing runProgram line with this, less satisfactory line:
>> > system $ (unwords ([show . locationPath . programLocation $ gcc] ++
>> > opts' ++ objs' ++ lib_dirs' ++ libs'))
>>
>> Which works, but still doesn't explain why runProgram doesn't.
>> Any suggestions?
>
>
> I've tried this on Linux. You're absolutely right and I'm stumped as to why.
> I think we should pull this specific part of the thread out and post to the
> Haskell list. Maybe someone who knows Cabal and/or GHC would have an idea
> why this is the case.

Ah, good to know it works on Windows.
I'll try and replicate it in isolation and send a mail to the cabal list.

>
>>
>> So with that complete I was able to link a shared library to:
>> wxcore/dist/build/libwxc.so.0.13.1
>>
>> Of course, at this point we've generated the shared library, but
>> wxcore still isn't aware of it, so the build completes successfully,
>> but any attempt to compile against it results (understandably) in
>> hundreds of linker errors.
>> To resolve this I had to add "x-dll-name" (read in from wxcore.cabal
>> as "wxc") back to parseDLLs:
>> > let all_dlls   = parseDLLs ["x-dll-name", "x-dll-extra-libraries"]
>> > custom_bi
>>
>> Well, not quite, attempt to build wxcore again and we're back to this
>> error:
>> * Missing C library: wxc
>>
>> Of course, the .so isn't in a 'normal' place, so add its location as
>> an extra lib directory:
>> > { extraLibDirs = extraLibDirs libbi ++ extraLibDirs wx ++
>> > ["/full/path/to/wxcore/dist/build"]
>> (note: I tried using just "dist/build", but cabal said: "library-dirs:
>> dist/build is a relative path")
>
>
> I think we need a more robust solution in the longer run, as
> <wxhaskell>/wcore/dirt/build is really a temporary location.

I guess the 'correct' solution would be to 'install' the library to
somewhere like LD_LIBRARY_PATH in Linux.
Again I don't know how this fits in with Windows.

It's also a pain for people developing wxhaskell, because as I
mentioned before, you'd have to 'install' a potentially unstable,
development version of the shared library in order to be able to
reference it.

>
>>
>> This still isn't quite enough, it took one more kludge:
>> /full/path/to/wxcore/dist/build/$ ln -s
>> /full/path/to/wxcore/dist/build/libwxc.so.0.13.1 libwxc.so
>
>
> Earlier you mentioned that, at least on Linux, the build command is:
>
>
> /usr/bin/gcc -fno-stack-protector -shared -Wl,-soname,libwxc.so.0 -o
> dist/build/libwxc.so.0.13.1 dist/build/src/cpp/apppath.o
> dist/build/src/cpp/dragimage.o dist/build/src/cpp/eljaccelerator.o
> [snip-rest-of-.o-files] -L/usr/local/lib -lstdc++ -lwx_baseu-2.9
> [snip-rest-of-wx-libs]
>
> I know GCC generated this and not you, but it looks like the root of the
> issue. I am slightly confused though. ld.so is supposed to *know* that
> libwxc.so.0.13.1 is a valid instance of libwxc.so.0

Firstly, I'm a little confused by your statement "GCC generated this
and not you", that build command is generated by linkCxxOpts, not GCC?

This is curious: The name of the symlink has to be "libwxc.so", not
even "libwxc.so.0".
I think this makes sense though, the only "information" the simple
build system gets about the dependency is the parsing of this line in
wxcore.cabal by parseDLLs:
  x-dll-name: wxc

I assume that the "wxc" is some how turned in to "libwxc.so", I'd like
to know how, and if it is possible to specify the major revision of a
shared library passed; this would have to be some convention in the
naming, because extraLibs is just [String]:
http://hackage.haskell.org/packages/archive/Cabal/latest/doc/html/Distribution-PackageDescription.html#v:extraLibs

>
> In normal Linux-land, you run /sbin/ldconfig after installing a new library,
> and it creates these symlinks automatically You could do this by executing
> something like (not tested)
>
> /sbin/ldconfig -n <path-to-directory>

Thanks for the tip, that makes more sense.
I can now see that the symlink which "ldconfig -n" generates is
extracted from the -soname given at link time.
So, by modifying the -soname to *not* specify the major revision, i.e:
This:
>    "-Wl,-soname,lib" ++ basename ++ ".so",
Instead of this:
>    "-Wl,-soname,lib" ++ basename ++ ".so" ++ maj_ver,

I can now use "ldconfig -n" in dist/build/ to create the symlink (to
libwxc.so), instead of doing it by hand, I suppose this is a little
less broken.

>
>> I'd like to think there's a more elegant solution than this and I'm
>> open to suggestions.
>> I should note that at this point I became aware that we could have an
>> entirely separate cabal project (perhaps "wxc") for this shared
>> library, and then have wxcore depend on it?
>>
>> Running one more time I *thought* I was finally there, until I noticed
>> this line, hidden in cabal's output:
>> /full/path/to/wxcore/dist/build/libwxc.so: file not recognized: File
>> truncated
>>
>> As far as I can tell, this occurs because we now load(?)
>> "libwxc.so.0.13.1" in myConfHook, but then in myBuildHook we want to
>> use it as the destination for the linker. Worse still it seems that
>> 'truncated' actually means 'deleted'. To get around this I had to one
>> again remove "x-dll-name" from the parseDLLs call (so I could get to
>> myBuildHook and create "libwxc.so.0.13.1" again), build wxcore, then
>> add "x-dll-name" back again, but this time also comment out the call
>> to linkSharedLib (and so stop cabal attempting to open
>> "libwxc.so.0.13.1") before building wxcore one last time.
>>
>> Finally everything seemed to work.
>>
>> Now, I should note that I have been using cabal-dev, and so I now have
>> a package-conf I need to reference when building with my new shared
>> library wxhaskell. To test it I built the wxhaskell HelloWorld sample
>> thus:
>> /full/path/to/samples/wxcore$ ghc --make -package-conf
>> ../../cabal-dev/packages-7.0.3.conf HelloWorld.hs
>>
>> It worked, but then when I try to run HelloWorld I get:
>> ./HelloWorld: error while loading shared libraries: libwxc.so.0:
>> cannot open shared object file: No such file or directory
>>
>> Okay, again, libwxc.so.0 isn't in a 'normal' place, so I kludged it
>> with another symlink:
>> /full/path/to/samples/wxcore$ ln -s
>> /full/path/to/wxcore/dist/build/libwxc.so libwxc.so.0
>>
>> And now, I can report that HelloWorld runs.
>> [imagine a screen shot of the HelloWorld sample application running here]
>> Rejoice.
>>
>> But wait, there's more:
>> One of the promises of the shared library approach is that we'd all be
>> able to use wxhaskell with ghci again, and to my utter disbelief, it's
>> true:
>> /full/path/to/samples/wxcore$ ghci -package-conf
>> ../../cabal-dev/packages-7.0.3.conf HelloWorld.hs
>> Ok, modules loaded: Main.
>> Prelude Main> main
>>
>> [imagine a screen shot of the HelloWorld sample application running here]
>> Double rejoice.
>>
>>
>>
>> >
>> > Dave,
>>
>>
>> ------------------------------------------------------------------------------
>> Cloud Services Checklist: Pricing and Packaging Optimization
>> This white paper is intended to serve as a reference, checklist and point
>> of
>> discussion for anyone considering optimizing the pricing and packaging
>> model
>> of a cloud services business. Read Now!
>> http://www.accelacomm.com/jaw/sfnl/114/51491232/
>> _______________________________________________
>> wxhaskell-devel mailing list
>> wxh...@li...
>> https://lists.sourceforge.net/lists/listinfo/wxhaskell-devel
>
>

Re: [wxhaskell-devel] Using a shared library for the C++ in wxhaskell

[Dave T. <duk...@gm...>]

Well, after a couple of days' frantic hacking I've made more progress,
and I'd invite anyone to comment on my decisions:

The compilation of the C++ 'wrapper' code has now been moved to a
separate project which I've called wxc.
This will now do the compilation and linking as per Jeremy's gist, and
then install the headers and the shared library (that is, install in
the cabal sense of the word).

Here's the clever part (well, I'm quite pleased with it):
Now wxcore depends on wxc, and during the wxcore configuration hook it
uses cabal to get the install information for wxc, from this it can
obtain the location of the aforementioned headers and shared library,
which can then link against.

The upshot:
1. All the C++ code is now in wxc.
2. Only wxc has to link against the wxWidgets libraries.
3. wxcore only uses the installed headers (to generate Haskell code
using wxdirect) and shared library from wxc (to link against).

The only major problem remaining (aside from the woeful lack of
testing, and support for non-Linux platforms) is you have to pass an
rpath to the linker when compiling; so you get something like this:

samples/wxcore$ ghc --make -package-conf
../../cabal-dev/packages-7.0.3.conf/
-optl-Wl,-rpath,../../cabal-dev/lib/wxc-0.1/ghc-7.0.3 HelloWorld.hs

In any case, I've pushed the changes to my darcden branch, and I
invite you to take a look:
http://darcsden.com/DukeDave/wxhaskell-dev

(Please accept my apologies for the record fail, needless to say the
change is split across two patches; I'm not happy either..)

Dave,

Re: [wxhaskell-devel] Using a shared library for the C++ in wxhaskell

[Dave T. <duk...@gm...>]

On 14 December 2011 07:16, Dave Tapley <duk...@gm...> wrote:
> Well, after a couple of days' frantic hacking I've made more progress,
> and I'd invite anyone to comment on my decisions:
>
> The compilation of the C++ 'wrapper' code has now been moved to a
> separate project which I've called wxc.
> This will now do the compilation and linking as per Jeremy's gist, and
> then install the headers and the shared library (that is, install in
> the cabal sense of the word).
>
> Here's the clever part (well, I'm quite pleased with it):
> Now wxcore depends on wxc, and during the wxcore configuration hook it
> uses cabal to get the install information for wxc, from this it can
> obtain the location of the aforementioned headers and shared library,
> which can then link against.
>
> The upshot:
> 1. All the C++ code is now in wxc.
> 2. Only wxc has to link against the wxWidgets libraries.
> 3. wxcore only uses the installed headers (to generate Haskell code
> using wxdirect) and shared library from wxc (to link against).
>
> The only major problem remaining (aside from the woeful lack of
> testing, and support for non-Linux platforms) is you have to pass an
> rpath to the linker when compiling; so you get something like this:
>
> samples/wxcore$ ghc --make -package-conf
> ../../cabal-dev/packages-7.0.3.conf/
> -optl-Wl,-rpath,../../cabal-dev/lib/wxc-0.1/ghc-7.0.3 HelloWorld.hs
>
> In any case, I've pushed the changes to my darcden branch, and I
> invite you to take a look:
> http://darcsden.com/DukeDave/wxhaskell-dev

I've just pushed a bunch of bug fixes to this repo, so if you were
having trouble before then hopefully everything will be better now.

>
> (Please accept my apologies for the record fail, needless to say the
> change is split across two patches; I'm not happy either..)
>
> Dave,

Re: [wxhaskell-devel] Using a shared library for the C++ in wxhaskell

[Dave T. <duk...@gm...>]

On 19 December 2011 22:02, Dave Tapley <duk...@gm...> wrote:

> On 14 December 2011 07:16, Dave Tapley <duk...@gm...> wrote:
> > The only major problem remaining (aside from the woeful lack of
> > testing, and support for non-Linux platforms) is you have to pass an
> > rpath to the linker when compiling; so you get something like this:
> >
> > samples/wxcore$ ghc --make -package-conf
> > ../../cabal-dev/packages-7.0.3.conf/
> > -optl-Wl,-rpath,../../cabal-dev/lib/wxc-0.1/ghc-7.0.3 HelloWorld.hs
>

I believe I have fixed the requirement to pass an rpath with this patch:
http://darcsden.com/DukeDave/wxhaskell-dev/patch/20120103181105-a1f0b

Dave,