From: William S F. <ws...@fu...> - 2006-03-23 22:59:50
|
Over the last month or so, there have been patches for 3 new SWIG modules, R, occam-pi and Squirrel. This email is for discussing adding these 3 modules and new modules in general. It is great to have new modules and over time, new languages have been regularly added to SWIG, see http://www.swig.org/compat.html. I just counted 19 language modules in total. Many of the major languages are now covered. There are dozens if not hundreds of lesser known languages for which SWIG modules could be created. The above 3 fall into the lesser known category, and I'm a bit concerned that SWIG will become bloatware with lots of unmaintained code for obscure languages and become a maintenance hassle. A few language modules have already started to bitrot or never got off the ground once they were committed. Could the other main SWIG developers comment about the following suggestions to keep this under control. Currently there are some prerequisites at http://www.swig.org/Doc1.3/Extending.html#Extending_prerequisites to be met for getting a new language into the main SWIG distribution. Can we tighten this further? ... For any language to be accepted into the SWIG distribution, it should pass at least 80% of the test-suite. If it drops below this figure it will be removed from the main distribution. This is to encourage maintainers to maintain the module. No single person has all the languages installed on their machines and testing is currently a tedious process so if the maintainer cannot show that the test-suite is working well enough at each release, the module will also be removed. That is unless someone wants to step forward and maintain a test system for all languages for everyone to see. In this regard, can the 3 patch submitters post the results of the test-suite for their respective languages with a count of tests passed. Incidentally, getting the majority of the test-suite to pass isn't that onerous once the basics are in place. William |
From: Marcelo M. <mm...@ac...> - 2006-03-24 00:28:41
|
William S Fulton wrote: > Over the last month or so, there have been patches for 3 new SWIG > modules, R, occam-pi and Squirrel. This email is for discussing adding > these 3 modules and new modules in general. > > It is great to have new modules and over time, new languages have been > regularly added to SWIG, see http://www.swig.org/compat.html. I just > counted 19 language modules in total. Many of the major languages are > now covered. There are dozens if not hundreds of lesser known > languages for which SWIG modules could be created. The above 3 fall > into the lesser known category, and I'm a bit concerned that SWIG will > become bloatware with lots of unmaintained code for obscure languages > and become a maintenance hassle. A few language modules have already > started to bitrot or never got off the ground once they were > committed. Could the other main SWIG developers comment about the > following suggestions to keep this under control. > > Currently there are some prerequisites at > http://www.swig.org/Doc1.3/Extending.html#Extending_prerequisites to > be met for getting a new language into the main SWIG distribution. Can > we tighten this further? ... > > For any language to be accepted into the SWIG distribution, it should > pass at least 80% of the test-suite. If it drops below this figure it > will be removed from the main distribution. This is to encourage > maintainers to maintain the module. No single person has all the > languages installed on their machines and testing is currently a > tedious process so if the maintainer cannot show that the test-suite > is working well enough at each release, the module will also be > removed. That is unless someone wants to step forward and maintain a > test system for all languages for everyone to see. I would said we should add the 'swig-extra' concept, where we can put all the "broken" or not well maintained languages, because: 1.- well, even they could be "broken" from the test-suite point of view, probably still they are users out there using it, maybe all the C++ part is broken but they are just using to wrap C. 2.- if we remove them totally from the repo, how one will test them and maybe volunteer to maintain one language?, by moving all the unmaintained languages to 'swig-extra', we can still speed up the compilation and testing process, since the modules in there are not going to be compiled by default, but we keep them in a visible place just in case a new maintainer appear. now, the swig-extra could be another directory in the CVS repository, or just a list of languages in the Makefile. Then make create a swig version with the main languages, and make swig --with-extras=pike,s-exp make swig --with-allextras creates a swig version also including 'pike and s-exp' or all the extra languages. And how we decide what to put in 'swig-extra', well, with a similar rule as you mention: 1.- If a current language doesn't seem to have a maintainer, and/or it fails to pass a given % of the test-suite, 80% for example, it is moved to swig-extra. 2.- Any new language should at least satisfy http://www.swig.org/Doc1.3/Extending.html#Extending_prerequisites but in a first stage, it will be added to swig-extra, and it will be kept in there until it satisfy 1.-, and only then can be moved to the main 'swig'. Once part of the 'swig-main', it can fall back again to swig-extra following again rule 1.-. That allows to add new languages in a less intrusive way, and if they take fly, they will have a road map to get into the main distribution. That also allows the new language developers to give some visibility and get feedback about their new language implementations, while keeping the main languages undisturbed. Then, to make a new distribution, we just concentrate in the swig-main components, which at this moment should represent 10 languages or so: Java, Csharp, perl, python, ruby, tcl, php, plus some of the lisp ones. Marcelo > > In this regard, can the 3 patch submitters post the results of the > test-suite for their respective languages with a count of tests > passed. Incidentally, getting the majority of the test-suite to pass > isn't that onerous once the basics are in place. > > William > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Swig-devel mailing list > Swi...@li... > https://lists.sourceforge.net/lists/listinfo/swig-devel |
From: David B. <dav...@da...> - 2006-03-24 00:59:55
|
On Mar 23, 2006, at 6:28 PM, Marcelo Matus wrote: > > > That allows to add new languages in a less intrusive way, and > if they take fly, they will have a road map to get into the main > distribution. That also allows the new language developers > to give some visibility and get feedback about their new > language implementations, while keeping the main languages > undisturbed. > Way back in the day, before my identity crisis, I had always thought it would be a good idea to make it possible for SWIG to dynamically load language modules much in the same way as scripting languages can load modules. In fact, unless things have been changed recently, the whole module system in the 1.3.x branch was designed to make this easy. For instance, modules are initialized, registered, and invoked in a way that would make it pretty simple to adapt to a dynamic loader. For all I know, there might even be comments and/or stub code in SWIG related to dynamic loading (I know I was starting to work on it at some point). If it were solely up to me, this is what I would do: - I would finish implementing the SWIG dynamic loader. To do this, I'd probably just lift code from Python since they've already dealt with all of the portability issues. - I'd strip the main SWIG distribution down to only those modules that are actively maintained, have documentation, and which pass virtually all of the tests. - I would try to support 3rd-party modules by making it as easy as possible for the maintainers to distribute and install these modules (we might have to supply some special tools). In addition, I would try to make it so www.swig.org has links to all of these modules so that users can find them. Thoughts? Cheers, Dave |
From: Matthias K. <mko...@ma...> - 2006-03-24 01:51:56
|
I agree that it is useful to make a distinction between "core" and "contributed" SWIG modules. However, I do not think it would be a good idea to remove poorly maintained modules from the SWIG distribution, or to move them to a separate repository, or to disable building them. I really don't think the presence of a few files in Source/Modules/ or a few directories in Lib/ disturbs much. Moving them away, however, invites bit-rot. For the contributed modules, I think the main SWIG developers should continue to try to keep the modules at least compiling when the internal SWIG API changes, so as to reduce bit-rot. (I am very grateful to the more active SWIG developers for doing this kind of maintenance on my SWIG modules -- I certainly wouldn't have had the time to check out all API changes myself to keep the modules working.) If it becomes too much trouble, disable its build, but don't move it away. Clearly, however, the main SWIG developers should not bother to run the test-suite for languages that are known to fail many tests; it does not make a difference if 100 tests fail or 101. Also I think it suffices that the contributed modules are announced as "SWIG supports various other languages, to a varying degree." The individual maintainers of the specialized language modules could announce news on their SWIG module on the respective mailing lists. Generally, I would always welcome new language modules as contributed modules into the SWIG CVS tree whenever they compile. However, if a developer wants his module upgraded to a core module, he should get most of the test-suite running and provide documentation. Speaking of the announcements of SWIG, I think we should finally get rid of the reference to SWIG 1.3 as a "development version", and to the "missing features compared to SWIG 1.1". I think it suffices to say that SWIG 1.3 is completely different and infinitely more powerful than SWIG 1.1. (I don't care about the version number, on the other hand.) I can comment a bit on the Lisp-related modules. * The Scheme modules (Guile, Chicken, MzScheme) seem to work well (I am only using Guile, however), only currently nobody is adding new features. * The s-expression output module was an early attempt (of mine) to support various Common Lisp implementations; I don't think anyone uses it. If it really becomes a maintainability problem, feel free to disable its build; I can then look into fixing it when I have spare time. * The Common Lisp modules (Allegro CL, CLISP, CFFI, UFFI) are all still very new, and I hope we can consolidate them later into just two modules. The Allegro CL backend was developed and is supported by the Allegro CL vendor (Franz Inc.); it currently sees great improvements. UFFI provides a portable foreign function interface for various Common Lisp implementations. The UFFI module in SWIG is in a very basic stage, but I would like ask that it is not removed at the current point of time. CFFI is a new effort to provide a portable foreign function interface; I think it will supersede UFFI when it is release-ready. At that point, we should probably remove the UFFI module of SWIG. I think the CFFI module of SWIG is not as advanced as the Allegro CL module yet. The CLISP module can probably be removed when CFFI works well enough on CLISP. Cheers, -- Matthias Koeppe -- http://www.math.uni-magdeburg.de/~mkoeppe (currently @math.ucdavis.edu) |
From: Charlie S. <cf...@in...> - 2006-03-24 02:12:34
|
I'd add another requirement - any new modules must be based on the unified type map library. That will helpfully reduce the support burden a bit because at least most of the typemaps will be shared. I also like Dave's suggestion to make it easy to load SWIG module dynamically. That would allow people to develop their own modules, and use them easily, without having to get them into the main source distribution. So something like this: 1. Core SWIG - no languages 2. Core Languages - loaded by the SWIG language loading mechanism (might as well eat or own dog food as they say) 3. Contributed Languages - distributed with SWIG, but not as well tested or documented (like postgresql contrib modules) 4. Custom languages, not in SWIG, that individual users/companies develop. These should easily "plugin" into SWIG via the language loading mechanisms. The owners may choose to contribute them to SWIG or not as they see fit. And while we are on the subject, it needs to be easy to snap-in new tests into the test suite. When I first started with SWIG last year I added a new test which was generally applicable, but I had only implemented in Ruby, and added it to the global test suite. This then blew up the global test-suite - so the test moved into the Ruby test suite only. Its quite possible I misunderstood how to do it. But if not, then it would be nice if it was easy to add new global tests that the test runner code simply outputs "nag" messages for languages that have not yet implemented it. Otherwise, its really hard to add new global tests because you have to go implement them for all the various languages. Thanks, Charlie William S Fulton wrote: > Over the last month or so, there have been patches for 3 new SWIG > modules, R, occam-pi and Squirrel. This email is for discussing adding > these 3 modules and new modules in general. > > It is great to have new modules and over time, new languages have been > regularly added to SWIG, see http://www.swig.org/compat.html. I just > counted 19 language modules in total. Many of the major languages are > now covered. There are dozens if not hundreds of lesser known languages > for which SWIG modules could be created. The above 3 fall into the > lesser known category, and I'm a bit concerned that SWIG will become > bloatware with lots of unmaintained code for obscure languages and > become a maintenance hassle. A few language modules have already started > to bitrot or never got off the ground once they were committed. Could > the other main SWIG developers comment about the following suggestions > to keep this under control. > > Currently there are some prerequisites at > http://www.swig.org/Doc1.3/Extending.html#Extending_prerequisites to be > met for getting a new language into the main SWIG distribution. Can we > tighten this further? ... > > For any language to be accepted into the SWIG distribution, it should > pass at least 80% of the test-suite. If it drops below this figure it > will be removed from the main distribution. This is to encourage > maintainers to maintain the module. No single person has all the > languages installed on their machines and testing is currently a tedious > process so if the maintainer cannot show that the test-suite is working > well enough at each release, the module will also be removed. That is > unless someone wants to step forward and maintain a test system for all > languages for everyone to see. > > In this regard, can the 3 patch submitters post the results of the > test-suite for their respective languages with a count of tests passed. > Incidentally, getting the majority of the test-suite to pass isn't that > onerous once the basics are in place. > > William > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 |
From: Damian D. <dj...@ke...> - 2006-03-24 12:31:34
|
Hello, I quite like the idea of having a swig-extras directory which can be enabled with a command-line flag. This would allow people like me to be part of the source tree. Currently version control for me is quite tedious, as I am not part of SWIG cvs but need to have my own files under version control. The source tree is a load of symlinks into a subversion repository. While it is not particularly hard, it is a nuisance to set up a new development environment for myself (I have had to do this a few times). The module loading code is not particularly difficult to do, even in a cross-platform manner, provided there are well-defined module entry points (which there are). I have implemented this sort of a system on a project which loads shared C libraries (actually swig-generated ones :) on windows and a variety of unicies. Windows and unix systems (osx, solaris and linux at least) return void for their respecitve so library load functions, so if you create a wrapper function around those like: #ifdef WIN32 #define LIB_HANDLE HINSTANCE HINSTANCE load_library(char *filename) { return LoadLibrary(filename); } #else #define LIB_HANDLE void* LIB_HANDLE load_library(char *filename) { return dlopen(filename, RTLD_NOW); } #endif and ensure that the library pointer is cast to void * each time you call load_library you will be fine. Getting the function pointer would be simple too, just call dlsym or GetProcAddress in similar wrapped functions. Extrapolating the name of the library that should be loaded is easy too, the commandline argument should provide us with enough information to do that. Anything that would help integrate my work with SWIG would be great :). Cheers, Damian On Friday 24 March 2006 01:38, Charlie Savage wrote: > I'd add another requirement - any new modules must be based on the > unified type map library. That will helpfully reduce the support burden > a bit because at least most of the typemaps will be shared. > > I also like Dave's suggestion to make it easy to load SWIG module > dynamically. That would allow people to develop their own modules, and > use them easily, without having to get them into the main source > distribution. > > So something like this: > > 1. Core SWIG - no languages > 2. Core Languages - loaded by the SWIG language loading mechanism (might > as well eat or own dog food as they say) > 3. Contributed Languages - distributed with SWIG, but not as well tested > or documented (like postgresql contrib modules) > 4. Custom languages, not in SWIG, that individual users/companies > develop. These should easily "plugin" into SWIG via the language > loading mechanisms. The owners may choose to contribute them to SWIG or > not as they see fit. > > And while we are on the subject, it needs to be easy to snap-in new > tests into the test suite. When I first started with SWIG last year I > added a new test which was generally applicable, but I had only > implemented in Ruby, and added it to the global test suite. This then > blew up the global test-suite - so the test moved into the Ruby test > suite only. > > Its quite possible I misunderstood how to do it. But if not, then it > would be nice if it was easy to add new global tests that the test > runner code simply outputs "nag" messages for languages that have not > yet implemented it. Otherwise, its really hard to add new global tests > because you have to go implement them for all the various languages. > > > Thanks, > > Charlie > > William S Fulton wrote: > > Over the last month or so, there have been patches for 3 new SWIG > > modules, R, occam-pi and Squirrel. This email is for discussing adding > > these 3 modules and new modules in general. > > > > It is great to have new modules and over time, new languages have been > > regularly added to SWIG, see http://www.swig.org/compat.html. I just > > counted 19 language modules in total. Many of the major languages are > > now covered. There are dozens if not hundreds of lesser known languages > > for which SWIG modules could be created. The above 3 fall into the > > lesser known category, and I'm a bit concerned that SWIG will become > > bloatware with lots of unmaintained code for obscure languages and > > become a maintenance hassle. A few language modules have already started > > to bitrot or never got off the ground once they were committed. Could > > the other main SWIG developers comment about the following suggestions > > to keep this under control. > > > > Currently there are some prerequisites at > > http://www.swig.org/Doc1.3/Extending.html#Extending_prerequisites to be > > met for getting a new language into the main SWIG distribution. Can we > > tighten this further? ... > > > > For any language to be accepted into the SWIG distribution, it should > > pass at least 80% of the test-suite. If it drops below this figure it > > will be removed from the main distribution. This is to encourage > > maintainers to maintain the module. No single person has all the > > languages installed on their machines and testing is currently a tedious > > process so if the maintainer cannot show that the test-suite is working > > well enough at each release, the module will also be removed. That is > > unless someone wants to step forward and maintain a test system for all > > languages for everyone to see. > > > > In this regard, can the 3 patch submitters post the results of the > > test-suite for their respective languages with a count of tests passed. > > Incidentally, getting the majority of the test-suite to pass isn't that > > onerous once the basics are in place. > > > > William > > > > > > ------------------------------------------------------- > > This SF.Net email is sponsored by xPML, a groundbreaking scripting > > language that extends applications into web and mobile media. Attend the > > live webcast > > and join the prime developer group breaking into this new coding > > territory! > > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting language > that extends applications into web and mobile media. Attend the live > webcast and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Swig-devel mailing list > Swi...@li... > https://lists.sourceforge.net/lists/listinfo/swig-devel |
From: William S F. <ws...@fu...> - 2006-03-27 20:23:13
|
Dynamically loaded modules and dividing the languages into core languages and contributed languages sounds good. Dynamically loaded modules are not technically difficult to code up. Building them is another matter though. It means we must use libtool to build the dlls/shared objects or alternatively use CMake. We spent a while getting rid of libtool for the runtime libraries. Whoever implements this will have to add in libtool support or rejig the build system to use CMake. Who fancies doing this? William Damian Dimmich wrote: > Hello, > > I quite like the idea of having a swig-extras directory which can be enabled > with a command-line flag. This would allow people like me to be part of the > source tree. Currently version control for me is quite tedious, as I am not > part of SWIG cvs but need to have my own files under version control. The > source tree is a load of symlinks into a subversion repository. While it is > not particularly hard, it is a nuisance to set up a new development > environment for myself (I have had to do this a few times). > > The module loading code is not particularly difficult to do, even in a > cross-platform manner, provided there are well-defined module entry points > (which there are). I have implemented this sort of a system on a project > which loads shared C libraries (actually swig-generated ones :) on windows > and a variety of unicies. > > Windows and unix systems (osx, solaris and linux at least) return void for > their respecitve so library load functions, so if you create a wrapper > function around those like: > #ifdef WIN32 > #define LIB_HANDLE HINSTANCE > HINSTANCE load_library(char *filename) > { > return LoadLibrary(filename); > } > #else > #define LIB_HANDLE void* > LIB_HANDLE load_library(char *filename) > { > return dlopen(filename, RTLD_NOW); > } > #endif > > and ensure that the library pointer is cast to void * each time you call > load_library you will be fine. > > Getting the function pointer would be simple too, just call dlsym or > GetProcAddress in similar wrapped functions. > > Extrapolating the name of the library that should be loaded is easy too, the > commandline argument should provide us with enough information to do that. > > Anything that would help integrate my work with SWIG would be great :). > > Cheers, > Damian > > On Friday 24 March 2006 01:38, Charlie Savage wrote: >>I'd add another requirement - any new modules must be based on the >>unified type map library. That will helpfully reduce the support burden >>a bit because at least most of the typemaps will be shared. >> >>I also like Dave's suggestion to make it easy to load SWIG module >>dynamically. That would allow people to develop their own modules, and >>use them easily, without having to get them into the main source >>distribution. >> >>So something like this: >> >>1. Core SWIG - no languages >>2. Core Languages - loaded by the SWIG language loading mechanism (might >>as well eat or own dog food as they say) >>3. Contributed Languages - distributed with SWIG, but not as well tested >>or documented (like postgresql contrib modules) >>4. Custom languages, not in SWIG, that individual users/companies >>develop. These should easily "plugin" into SWIG via the language >>loading mechanisms. The owners may choose to contribute them to SWIG or >>not as they see fit. >> >>And while we are on the subject, it needs to be easy to snap-in new >>tests into the test suite. When I first started with SWIG last year I >>added a new test which was generally applicable, but I had only >>implemented in Ruby, and added it to the global test suite. This then >>blew up the global test-suite - so the test moved into the Ruby test >>suite only. >> >>Its quite possible I misunderstood how to do it. But if not, then it >>would be nice if it was easy to add new global tests that the test >>runner code simply outputs "nag" messages for languages that have not >>yet implemented it. Otherwise, its really hard to add new global tests >>because you have to go implement them for all the various languages. >> >> >>Thanks, >> >>Charlie >> >>William S Fulton wrote: >>>Over the last month or so, there have been patches for 3 new SWIG >>>modules, R, occam-pi and Squirrel. This email is for discussing adding >>>these 3 modules and new modules in general. >>> >>>It is great to have new modules and over time, new languages have been >>>regularly added to SWIG, see http://www.swig.org/compat.html. I just >>>counted 19 language modules in total. Many of the major languages are >>>now covered. There are dozens if not hundreds of lesser known languages >>>for which SWIG modules could be created. The above 3 fall into the >>>lesser known category, and I'm a bit concerned that SWIG will become >>>bloatware with lots of unmaintained code for obscure languages and >>>become a maintenance hassle. A few language modules have already started >>>to bitrot or never got off the ground once they were committed. Could >>>the other main SWIG developers comment about the following suggestions >>>to keep this under control. >>> >>>Currently there are some prerequisites at >>>http://www.swig.org/Doc1.3/Extending.html#Extending_prerequisites to be >>>met for getting a new language into the main SWIG distribution. Can we >>>tighten this further? ... >>> >>>For any language to be accepted into the SWIG distribution, it should >>>pass at least 80% of the test-suite. If it drops below this figure it >>>will be removed from the main distribution. This is to encourage >>>maintainers to maintain the module. No single person has all the >>>languages installed on their machines and testing is currently a tedious >>>process so if the maintainer cannot show that the test-suite is working >>>well enough at each release, the module will also be removed. That is >>>unless someone wants to step forward and maintain a test system for all >>>languages for everyone to see. >>> >>>In this regard, can the 3 patch submitters post the results of the >>>test-suite for their respective languages with a count of tests passed. >>>Incidentally, getting the majority of the test-suite to pass isn't that >>>onerous once the basics are in place. >>> >>>William >>> >>> >>>------------------------------------------------------- >>>This SF.Net email is sponsored by xPML, a groundbreaking scripting >>>language that extends applications into web and mobile media. Attend the >>>live webcast >>>and join the prime developer group breaking into this new coding >>>territory! >>>http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 >>------------------------------------------------------- >>This SF.Net email is sponsored by xPML, a groundbreaking scripting language >>that extends applications into web and mobile media. Attend the live >>webcast and join the prime developer group breaking into this new coding >>territory! >>http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 >>_______________________________________________ >>Swig-devel mailing list >>Swi...@li... >>https://lists.sourceforge.net/lists/listinfo/swig-devel > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting language > that extends applications into web and mobile media. Attend the live webcast > and join the prime developer group breaking into this new coding territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Swig-devel mailing list > Swi...@li... > https://lists.sourceforge.net/lists/listinfo/swig-devel > |
From: David B. <dav...@da...> - 2006-03-27 20:42:28
|
On Mar 27, 2006, at 3:22 PM, William S Fulton wrote: > Dynamically loaded modules and dividing the languages into core > languages and contributed languages sounds good. Dynamically loaded > modules are not technically difficult to code up. Building them is > another matter though. It means we must use libtool to build the > dlls/shared objects or alternatively use CMake. We spent a while > getting rid of libtool for the runtime libraries. Whoever > implements this will have to add in libtool support or rejig the > build system to use CMake. Who fancies doing this? Why do we have to worry about building dynamic modules? I would propose building all of the modules that are part of the core SWIG in exactly the same was as it's done now. Dynamic loading would only be used for third-party modules not part of the distribution. In other words, it's not our problem. Am I missing something? I don't mean to sound crass, but I really don't think we should spend a lot of time worrying about barely maintained and/or obscure language modules. If we provide a dynamic loading mechanism, that alone provides means for someone to implement a new language module and have it work with SWIG without requiring it to be part of the main source tree. We can certainly tell people how to use libtool, cmake, or whatever, but I don't think we should worry about it. For what it's worth, I don't think the SWIG distribution should even include a directory for "extra" or "unsupported" language modules. If people aren't willing to support a module fully with documentation, tests, examples, and everything else, I don't think it should be part of the distribution---instead we can provide a link to an appropriate site on the SWIG page. Besides, we've got more than enough work supporting all of the core modules as it is... Cheers, Dave |
From: Jason S. <jas...@gm...> - 2006-03-31 05:48:59
|
Hey all, > For what it's worth, I don't think the SWIG distribution should even > include a directory for "extra" or "unsupported" language modules. > If people aren't willing to support a module fully with > documentation, tests, examples, and everything else, I don't think it > should be part of the distribution---instead we can provide a link to > an appropriate site on the SWIG page. Besides, we've got more than > enough work supporting all of the core modules as it is... > In principle I agree with what Dave says. But how do we feel about Matthias' comment that some module maintainers aren't able to cope with all the underlying API changes? By moving them out of the core - it puts *all* the burden on the module maintainer to watch the SWIG internals change. Think of how much things have changed in 1.3, if we continue to shift the internals that much we will never get anyone to maintain their modules. Cheers, jas. |
From: William S F. <ws...@fu...> - 2006-03-31 23:56:26
|
Jason Stewart wrote: > Hey all, > > >>For what it's worth, I don't think the SWIG distribution should even >>include a directory for "extra" or "unsupported" language modules. >>If people aren't willing to support a module fully with >>documentation, tests, examples, and everything else, I don't think it >>should be part of the distribution---instead we can provide a link to >>an appropriate site on the SWIG page. Besides, we've got more than >>enough work supporting all of the core modules as it is... >> > > > In principle I agree with what Dave says. But how do we feel about > Matthias' comment that some module maintainers aren't able to cope > with all the underlying API changes? By moving them out of the core - > it puts *all* the burden on the module maintainer to watch the SWIG > internals change. > > Think of how much things have changed in 1.3, if we continue to shift > the internals that much we will never get anyone to maintain their > modules. > Things have changed a lot in 1.3, but the internal changes that occurred around and before 1.3.10 are no longer happening and I don't know of any plans to make these sort of big changes again. There will always be some minor changes in the internals which might require a module maintainer to alter some code, but nothing very demanding for a module maintainer interested in keeping a module up and running. By and large I agree with Dave that if a module maintainer can't be bothered to provide a minimal level of support it doesn't deserve to be in SWIG. However I don't mind these minor/neglected modules been put into the 'extra' package or whatever it is called, so that they still compile, but make it clear they are not as well maintained, simply because they are labelled differently. In response to the suggestion of providing a mechanism to load modules dynamically, Charlie suggested the core modules are loaded by the same mechanism, so that we eat our own dog food. This is of course is a good idea, but with the onerous task of providing cross platform dlls/shared objects. I think everyone is in agreement we dont' go down that route, so it looks like we should provide a machanism to load modules dynamically, but the core modules are compiled into SWIG as they are currntly are. We need some way of testing the dynamically loaded modules, so perhaps we could knock up a simple dynamically test module which might just compile and be tested on say Linux. William |
From: John L. <le...@cs...> - 2006-04-05 04:29:14
|
On Fri, March 31, 2006 7:55 pm, William S Fulton said: > In response to the suggestion of providing a mechanism to load modules > dynamically, Charlie suggested the core modules are loaded by the same > mechanism, so that we eat our own dog food. This is of course is a good > idea, but with the onerous task of providing cross platform dlls/shared > objects. I think everyone is in agreement we dont' go down that route, > so it looks like we should provide a machanism to load modules > dynamically, but the core modules are compiled into SWIG as they are > currntly are. We need some way of testing the dynamically loaded > modules, so perhaps we could knock up a simple dynamically test module > which might just compile and be tested on say Linux. I was reading about libltdl the other night and found it interesting... One problem everyone is overlooking is that Windows and AIX will NOT be able to support dynamicaly loaded modules. So if we do move the unsupported modules into external libraries, those languages will only be able to be used on linux and related, not windows. For exactly why this is the case, continue reading.... From http://sources.redhat.com/autobook/autobook/autobook_165.html#SEC165 "Back-linking is the process of resolving any remaining symbols by referencing back into the application that loads the library at runtime -- a mechanism implemented on almost all modern Unices. For instance, your main application may provide some utility function, `my_function', which you want a module to have access to. There are two ways to do that: You could use Libtool to link your application, using the `-export-dynamic' option to ensure that the global application symbols are available to modules. When libltdl loads a module into an application compiled like this, it will back-link symbols from the application to resolve any otherwise undefined symbols in a module. When the module is `ltdlopen'ed, libltdl will arrange for calls to `my_function' in the module, to execute the `my_function' implementation in the application. If you have need of this functionality, relying on back-linking is the simplest way to achieve it. Unfortunately, this simplicity is at the expense of portability: some platforms have no support for back-linking at all, and others will not allow a module to be created with unresolved symbols. Never-the-less, libltdl allows you to do this if you want to." And then later it says "You should be aware that your application will not work on some platforms--most notably, Windows and AIX---if you rely on a back-linking." From reading that, the only way to get modules working on Windows would be to factor out all the functions inside the SWIG core. "You could split the code that implements the symbols you need to share with modules into a separate library. This library would then be used to resolve the symbols you wish to share, by linking it into modules and application alike. The definition of `my_function' would be compiled separately into a library, `libmy_function.la'. References to `my_function' from the application would be resolved by linking it with `libmy_function.la', and the library would be installed so that modules which need to call `my_function' would be able to resolve the symbol by linking with `-lmy_function'. This method requires support for neither back-linking nor unresolved link time symbols from the host platform. The disadvantage is that when you realise you need this functionality, it may be quite complicated to extract the shared functionality from the application to be compiled in a stand alone library." Exactly that last sentence applies to SWIG. But I don't see any point to attempting to move all that code in Source/Swig and Source/DOH and such into a seperate library. Another alternative is to make all of swig into a single DLL, and then swig.exe is just a 5 line program that links against swig.dll and calls the main function inside the DLL... See this thread http://www.cygwin.com/ml/cygwin/2004-04/msg00227.html Which seems to say that using cygwin, it is sort of possible to support back linking on windows... so maybe we could work a solution in there. One other interesting aspect of libltdl is... "On machines which do not have any facility for shared libraries or dynamic modules, libltdl allows an application to lt_dlopen modules, provided that the modules are known at link time. This works by linking the code for the modules into the application in advance, and then looking up the addresses of the already loaded symbols when lt_dlsym is called. We call this mechanism dlpreopening -- so named because the modules must be loaded at link time, not because the API to use modules loaded in this way is any different." http://sources.redhat.com/autobook/autobook/autobook_173.html#SEC173 We can easily do that, where we have all the modules going through lt_dlopen, but we link in all the standard modules at compile time. This has the advantage of having all the standard modules compiled in to the SWIG binary, but not having to change any code about how/which modules are compiled directly into SWIG. Also, modules that are linked into the main exe will work on windows, since we don't need back linking for them. So we actually could exploit this, and only link in the standard modules by default. But since each module is built the same, (and the only thing that needs to be changed is the link commands), we can easily make configure.in have some options to include the extra languages as preloaded modules. So on platforms other than Windows, we can support loading other languages as modules as dynamic libraries, or by specifing to configure that you want to pre-link the extra language. On Windows, you could only use extra languages by telling configure (well, I guess visual studio or whatever) that you want to prelink the extra languages. That entire chapter of the book (chapter 18) is very good read on all the issues associated with dynamic modules. What is also interesting about libltdl is that using libtool, you can easily build modules to load using libltdl, but libtool isn't required. Again, it would pretty much require libtool to do this, and after we just removed libtool from SWIG. John |
From: Olly B. <ol...@su...> - 2006-04-05 13:11:14
|
On 2006-04-05, John Lenz <le...@cs...> wrote: > "You should be aware that your application will not work on some > platforms--most notably, Windows and AIX---if you rely on a back-linking." > > From reading that, the only way to get modules working on Windows would be > to factor out all the functions inside the SWIG core. There's an easy workaround for this: SWIG puts pointers to all the functions which a module is allowed to call into a struct, and the module provides a call which accepts such a struct which is called before anything else by SWIG. Then the module can call back into SWIG using these function pointers. It requires that struct to be updated when new functions get added, but that shouldn't be a frequent occurence. Note that if you never delete or reorder entries you can pass sizeof(struct) to avoid versioning issues. Cheers, Olly |
From: John L. <le...@cs...> - 2006-04-06 01:28:16
|
On Wed, April 5, 2006 8:09 am, Olly Betts said: > On 2006-04-05, John Lenz <le...@cs...> wrote: >> "You should be aware that your application will not work on some >> platforms--most notably, Windows and AIX---if you rely on a >> back-linking." >> >> From reading that, the only way to get modules working on Windows would >> be >> to factor out all the functions inside the SWIG core. > > There's an easy workaround for this: > > SWIG puts pointers to all the functions which a module is allowed to > call into a struct, and the module provides a call which accepts such a > struct which is called before anything else by SWIG. Then the module > can call back into SWIG using these function pointers. > > It requires that struct to be updated when new functions get added, > but that shouldn't be a frequent occurence. Note that if you never > delete or reorder entries you can pass sizeof(struct) to avoid > versioning issues. Well, problem is the modules use a massive amount of functions from all over the SWIG core... all the functions in DOH, and most of the functions inside Source/Swig... We are talking about probably over 1000 functions. Now, some of the functions in DOH are already used through pointers, but there still are a huge number of functions. Getting all those functions into some struct is a lot of work... and one which is balanced against the fact that loadable language modules aren't that benificial. On the other hand, I was searching around (just type back-linking into google you get a lot of hacks which say it is possible) just found this project, which supposedly allows backlinking using MinGW (which I think is what SWIG uses to compile on windows?) http://edll.sourceforge.net/ Not so sure about the license, becuase it is LGPL... maybe someone could look into using that instead? John |
From: Robin D. <ro...@al...> - 2006-04-11 19:10:49
|
Olly Betts wrote: > On 2006-04-05, John Lenz <le...@cs...> wrote: >> "You should be aware that your application will not work on some >> platforms--most notably, Windows and AIX---if you rely on a back-linking." >> >> From reading that, the only way to get modules working on Windows would be >> to factor out all the functions inside the SWIG core. > > There's an easy workaround for this: > > SWIG puts pointers to all the functions which a module is allowed to > call into a struct, and the module provides a call which accepts such a > struct which is called before anything else by SWIG. Then the module > can call back into SWIG using these function pointers. There is another easy way to do this, and is what is done by Python on win32. Make all of SWIG except for main() be a DLL or shared library. Then both swig.exe (just the main() function) and all of the language modules link to that DLL and import whatever functions they need from there. In theory, this could also open up some interesting usages of the SWIG code from other languages. IOW, use swig to make a (for example) Python extension module of SWIG itself, and then implement the desired new functionality by deriving from the Language class in Python... -- Robin Dunn Software Craftsman http://wxPython.org Java give you jitters? Relax with wxPython! |
From: Matthias K. <mko...@ma...> - 2006-03-27 23:22:56
|
William S Fulton <ws...@fu...> writes: > Dynamically loaded modules and dividing the languages into core > languages and contributed languages sounds good. Dynamically loaded > modules are not technically difficult to code up. Building them is > another matter though. It means we must use libtool to build the > dlls/shared objects or alternatively use CMake. We spent a while > getting rid of libtool for the runtime libraries. Whoever implements > this will have to add in libtool support or rejig the build system to > use CMake. Who fancies doing this? Please let us stay away from the extra complexity of dynamically loaded modules. Apart from the problems you described, we also would need to deal with versioning problems then -- people would expect that a random external SWIG module works with a random SWIG version. Cheers, --=20 Matthias K=F6ppe -- http://www.math.uni-magdeburg.de/~mkoeppe (currently @math.ucdavis.edu)=20 |
From: William S F. <ws...@fu...> - 2006-03-27 20:23:22
|
> And while we are on the subject, it needs to be easy to snap-in new > tests into the test suite. When I first started with SWIG last year I > added a new test which was generally applicable, but I had only > implemented in Ruby, and added it to the global test suite. This then > blew up the global test-suite - so the test moved into the Ruby test > suite only. > > Its quite possible I misunderstood how to do it. But if not, then it > would be nice if it was easy to add new global tests that the test > runner code simply outputs "nag" messages for languages that have not > yet implemented it. Otherwise, its really hard to add new global tests > because you have to go implement them for all the various languages. > Generally adding in a new global test is easy. We do this all the time and anything that break the big languages just gets fixed. I don't know the details of the particular test, but if it works in Ruby and not the other languages, it should really be put in as a global test and then fixed in the other languages. There are some tests which are classified as broken and in common.mk are put in a separate group which are not normally run. When the bug gets fixed the testcase will be moved to be run by 'make check'. There are also some stl tests which are skipped by some languages whose STL support is poor, but this is not a permanent thing, in fact, I'm not sure skipping STL tests is a good thing as the STL are important libraries that should have wrappers. William |