swig-devel

[Swig-devel] 2 SWIG patches (C#) and a question

[Vladimir K. <vka...@op...>]

Hello William,

I work for Open Design Alliance (www.opendesign.com)
(We are not listed on the projects page, and that
may be fixed. Actually, I was not aware of that page existence until
recently.)

We are using SWIG for quite a while now, and with some success. Last
version of our product contains stable version of .NET wrappers
generated by SWIG. Of course, due to the size of our API (hundreds of
header files), there are some difficult places that SWIG couldn't quite handle.
We use directors extensively, and there are few issues in directors
support that were very limiting. The main problem is lacking
"csdirectorin" "pre:" and "post" code attributes in C# module. Without them it is
not trivial to marshal strings and smart-pointers back and forth
between user callback code and native code. (especially by reference)
I propose a patch introducing this support (with test).
It also fixes 2 minor issues in director code generation that are
difficult to come by until "csdirectorin" attribute is extended.
The first is that "ref" types used in directors lead to invalid
signature generation (the type array used to match methods possibly
overloaded by user). typeof(ref T) is used instead of
typeof().MakeByRefType()
The second is that ignored director methods are not completely ignored
- if there was a %typemap(imtype, "directorinattributes") it is not
skipped for ignored method.

The issue was discussed in a mailing list long ago (see e.g.
http://permalink.gmane.org/gmane.comp.programming.swig/16385)
I proposed a patch a year and a half ago http://sourceforge.net/p/swig/patches/268/
Attached to this mail is a git version of the patch, which is easier
to review.

The second problem is Unicode string literals. (Strings like L"Str" )
In our code all the strings are wchar_t based and such literals occur
in headers from time to time. I proposed a workaround some time ago:
http://sourceforge.net/p/swig/bugs/971/,
but recently I put some work to it and implemented a full solution.
See the patch attached. The wide literals may occur in default
arguments and in expressions. In both cases they are correctly parsed.
Wide char literals are not unescaped while parsing, because they may
contain escaped characters not fitting into "char" type. If the target
language wants to use them but cannot handle C escape codes, language
module itself may do the necessary conversion.

Usage example:

%module(directors="1")  MyModule
%include "wchar.i"
%feature("director") MyClass;
%csconst(1);

class MyClass
{
public:
  static const wchar_t * const myConst = L"MyWideCharConst\x234";
  virtual void myFunc(const wchar_t* myString = L"MyWideChar\x1234Literal\555");
  virtual ~MyClass();
};

Generated C# wrapper code:

public class MyClass : IDisposable {
...
  public const string myConst = "MyWideCharConst\x234";
}

Generated C++ proxy code:
class SwigDirector_MyClass : public MyClass, public Swig::Director {
public:
    SwigDirector_MyClass();
    virtual void myFunc(wchar_t const *myString = L"MyWideChar\x1234Literal\555");
...
};


These cases may seem marginal to non C# users, but I think we are not unique
with our requirements and these patches may benefit other users. (Of course we may
continue using locally forked swig version,
but it is rather inconvenient, especially for our clients, some of
which may want to regenerate .net wrappers from sources.)


Now, the question:
Currently I'm working at the nested classes support for C# module. I
already fixed parser.y, typepass and lang.cxx and working on
the csharp module itself. One thing I don't understand - why was it
disabled at the parser level? I mean why the pain with that weird
"nestedworkaround" approach, reentering the parser (obviously not
designed for reentrance)? I fear that I'm missing something essential. Who is the
right person to discuss this with? (I may commit my current state of
work to some remote repository to review. Not that there is much to look at,
but at least the symtables are generated correctly (or so it seems))

(I didn't touch the templates yet, that is where main parsing problems are
supposed to be.)


-- 
Best regards,
 Vladimir  Kalinin                        mailto:vka...@op...

Re: [Swig-devel] 2 SWIG patches (C#) and a question

[William S F. <ws...@fu...>]

On 08/01/13 14:20, Vladimir Kalinin wrote:
> Hello William,
>
> I work for Open Design Alliance (www.opendesign.com)
> (We are not listed on the projects page, and that
> may be fixed. Actually, I was not aware of that page existence until
> recently.)
You should be able to post a patch quite easily to the relevant page at 
https://github.com/swig/www/blob/master/projects.ht containing a 
description of Open Design Alliance that you'd like to see on the web page.

>
> We are using SWIG for quite a while now, and with some success. Last
> version of our product contains stable version of .NET wrappers
> generated by SWIG. Of course, due to the size of our API (hundreds of
> header files), there are some difficult places that SWIG couldn't quite handle.
> We use directors extensively, and there are few issues in directors
> support that were very limiting. The main problem is lacking
> "csdirectorin" "pre:" and "post" code attributes in C# module. Without them it is
> not trivial to marshal strings and smart-pointers back and forth
> between user callback code and native code. (especially by reference)
> I propose a patch introducing this support (with test).
> It also fixes 2 minor issues in director code generation that are
> difficult to come by until "csdirectorin" attribute is extended.
> The first is that "ref" types used in directors lead to invalid
> signature generation (the type array used to match methods possibly
> overloaded by user). typeof(ref T) is used instead of
> typeof().MakeByRefType()
> The second is that ignored director methods are not completely ignored
> - if there was a %typemap(imtype, "directorinattributes") it is not
> skipped for ignored method.
>
> The issue was discussed in a mailing list long ago (see e.g.
> http://permalink.gmane.org/gmane.comp.programming.swig/16385)
> I proposed a patch a year and a half ago http://sourceforge.net/p/swig/patches/268/
> Attached to this mail is a git version of the patch, which is easier
> to review.
Thanks very much, I'll review that very shortly.

>
> The second problem is Unicode string literals. (Strings like L"Str" )
> In our code all the strings are wchar_t based and such literals occur
> in headers from time to time. I proposed a workaround some time ago:
> http://sourceforge.net/p/swig/bugs/971/,
> but recently I put some work to it and implemented a full solution.
> See the patch attached. The wide literals may occur in default
> arguments and in expressions. In both cases they are correctly parsed.
> Wide char literals are not unescaped while parsing, because they may
> contain escaped characters not fitting into "char" type. If the target
> language wants to use them but cannot handle C escape codes, language
> module itself may do the necessary conversion.
>
> Usage example:
>
> %module(directors="1")  MyModule
> %include "wchar.i"
> %feature("director") MyClass;
> %csconst(1);
>
> class MyClass
> {
> public:
>    static const wchar_t * const myConst = L"MyWideCharConst\x234";
>    virtual void myFunc(const wchar_t* myString = L"MyWideChar\x1234Literal\555");
>    virtual ~MyClass();
> };
>
> Generated C# wrapper code:
>
> public class MyClass : IDisposable {
> ...
>    public const string myConst = "MyWideCharConst\x234";
> }
>
> Generated C++ proxy code:
> class SwigDirector_MyClass : public MyClass, public Swig::Director {
> public:
>      SwigDirector_MyClass();
>      virtual void myFunc(wchar_t const *myString = L"MyWideChar\x1234Literal\555");
> ...
> };
>
>
> These cases may seem marginal to non C# users, but I think we are not unique
> with our requirements and these patches may benefit other users. (Of course we may
> continue using locally forked swig version,
> but it is rather inconvenient, especially for our clients, some of
> which may want to regenerate .net wrappers from sources.)
>
No doubt someone will benefit, so let's aim to get these into the main 
SWIG distribution, it is better for all.

It might be more complicated than you wish though. This support is 
already in the gsoc2009-matevz branch: 
https://github.com/swig/swig/blob/gsoc2009-matevz/Doc/Manual - See 
Cpp0x.html#Cpp0x_New_string_literals. This branch is my highest priority 
large integration into the mainline. So could I ask you to use this 
branch for this particular problem and provide any additional change in 
this area to that branch, or wait until I've merged it into master, 
probably by early Feb.

Aside... anyone know how to provide a direct link so that the above 
.html file in github is rendered in a browser in github (it was possible 
in svnweb).


William

[Swig-devel] Nested classes. Was: 2 SWIG patches (C#) and a question

[William S F. <ws...@fu...>]

On 08/01/13 14:20, Vladimir Kalinin wrote:
> Now, the question:
> Currently I'm working at the nested classes support for C# module. I
> already fixed parser.y, typepass and lang.cxx and working on
> the csharp module itself. One thing I don't understand - why was it
> disabled at the parser level? I mean why the pain with that weird
> "nestedworkaround" approach, reentering the parser (obviously not
> designed for reentrance)? I fear that I'm missing something essential. Who is the
> right person to discuss this with? (I may commit my current state of
> work to some remote repository to review. Not that there is much to look at,
> but at least the symtables are generated correctly (or so it seems))
>
> (I didn't touch the templates yet, that is where main parsing problems are
> supposed to be.)
>
>
Nested class support will be a big step forward for SWIG, so good to 
hear about this. I don't know why the nested class support in the parser 
is a gross hack (the reentering parser bit), probably ease of 
implementation targeting C in the first instance. I think back in about 
2000 when SWIG was going through a lot of development, this problem was 
put on the TODO later list, but it never got attended to. Dave Beazley 
probably knows more, but he isn't involved much these days.

The nestedworkaround was and still is a workaround which was easily 
implemented. The full nested class support needn't necessarily support 
this feature. I'd see the full nested class support as one of the few 
reasons to increment the major version number and hence indicate some 
broken backwards compatibility for nested classes.

William

Re: [Swig-devel] Nested classes. Was: 2 SWIG patches (C#) and a question

[David B. <da...@da...>]

On Jan 8, 2013, at 4:59 PM, William S Fulton wrote:

> On 08/01/13 14:20, Vladimir Kalinin wrote:
>> Now, the question:
>> Currently I'm working at the nested classes support for C# module. I
>> already fixed parser.y, typepass and lang.cxx and working on
>> the csharp module itself. One thing I don't understand - why was it
>> disabled at the parser level? I mean why the pain with that weird
>> "nestedworkaround" approach, reentering the parser (obviously not
>> designed for reentrance)? I fear that I'm missing something essential. Who is the
>> right person to discuss this with? (I may commit my current state of
>> work to some remote repository to review. Not that there is much to look at,
>> but at least the symtables are generated correctly (or so it seems))
>> 
>> (I didn't touch the templates yet, that is where main parsing problems are
>> supposed to be.)
>> 
>> 
> Nested class support will be a big step forward for SWIG, so good to 
> hear about this. I don't know why the nested class support in the parser 
> is a gross hack (the reentering parser bit), probably ease of 
> implementation targeting C in the first instance. I think back in about 
> 2000 when SWIG was going through a lot of development, this problem was 
> put on the TODO later list, but it never got attended to. Dave Beazley 
> probably knows more, but he isn't involved much these days.
> 

The implementation of nested classes in SWIG has always been complicated due to two historical factors.  First, many of the target languages supported by SWIG don't actually support nested classes themselves (or if they do, the support is weak/flaky).  Second, the implementation of SWIG's various language modules didn't allow class code to be emitted in a manner that was reentrant.  That is, if you were in the middle of emitting code for a class, you couldn't just suspend it temporarily and enter a nested code generation context.   

The "solution" to both of these problems was to have the parser forcefully unnest the class definitions and play some hacks with parsing and  typenames.    Specifically, if a nested class was found, the parser would simply capture the whole class definition body and set it aside.  After the enclosing class was fully processed, the body of the nested class would then be processed as if it were a separate unnested class definition.    A typedef like "typedef Outer::Inner Outer_Inner" would be made somewhere in the emitted code and the class wrapped as if it were actually called "Outer_Inner".   Yes, a hack.

A major reason why nested class support has never been implemented to this day is due to the fact that making it work would probably require a major refactoring of all of the target language modules to allow reentrant class wrapping.   Given the large number of modules, it's a major task.  I just don't think anyone has been motivated enough to tackle it.

I'm not sure if this helps, but it's a bit of history about why there are some really gross hacks in there ;-). 

Cheers,
Dave

 

Re: [Swig-devel] Nested classes. Was: 2 SWIG patches (C#) and a question

[Vladimir K. <vka...@op...>]

>> On Jan 8, 2013, at 4:59 PM, William S Fulton wrote:
>>>> On 08/01/13 14:20, Vladimir Kalinin wrote:
>>> Now, the question:
>>> Currently I'm working at the nested classes support for C# module. I
>>> already fixed parser.y, typepass and lang.cxx and working on
>>> the csharp module itself. One thing I don't understand - why was it
>>> disabled at the parser level? I mean why the pain with that weird
>>> "nestedworkaround" approach, reentering the parser (obviously not
>>> designed for reentrance)? I fear that I'm missing something essential. Who is the
>>> right person to discuss this with? (I may commit my current state of
>>> work to some remote repository to review. Not that there is much to look at,
>>> but at least the symtables are generated correctly (or so it seems))
>>> 
>>> (I didn't touch the templates yet, that is where main parsing problems are
>>> supposed to be.)
>>> 
>>> 
>> Nested class support will be a big step forward for SWIG, so good to 
>> hear about this. I don't know why the nested class support in the parser 
>> is a gross hack (the reentering parser bit), probably ease of 
>> implementation targeting C in the first instance. I think back in about 
>
>> 2000 when SWIG was going through a lot of development, this problem was 
>
>> put on the TODO later list, but it never got attended to. Dave Beazley 
>> probably knows more, but he isn't involved much these days.
>> 
>
> The implementation of nested classes in SWIG has always been
> complicated due to two historical factors.  First, many of the
> target languages supported by SWIG don't actually support nested
> classes themselves (or if they do, the support is weak/flaky). 
> Second, the implementation of SWIG's various language modules didn't
> allow class code to be emitted in a manner that was reentrant.  That
> is, if you were in the middle of emitting code for a class, you
> couldn't just suspend it temporarily and enter a nested code generation context.
>
> The "solution" to both of these problems was to have the parser
> forcefully unnest the class definitions and play some hacks with
> parsing and  typenames.    Specifically, if a nested class was
> found, the parser would simply capture the whole class definition
> body and set it aside.  After the enclosing class was fully
> processed, the body of the nested class would then be processed as
> if it were a separate unnested class definition.    A typedef like
> "typedef Outer::Inner Outer_Inner" would be made somewhere in the
> emitted code and the class wrapped as if it were actually called "Outer_Inner".   Yes, a hack.
>
> A major reason why nested class support has never been implemented
> to this day is due to the fact that making it work would probably
> require a major refactoring of all of the target language modules to
> allow reentrant class wrapping.   Given the large number of modules,
> it's a major task.  I just don't think anyone has been motivated enough to tackle it.
>
> I'm not sure if this helps, but it's a bit of history about why
> there are some really gross hacks in there ;-). 
>

I quite understand why the nested classes were not implemented, I just
wondered why it was easier to implement the workaround on the parser
level, not as a parse tree transformation.
I think that will have to be done now anyway, because as you correctly
point out, there are languages w/o nested classes support at all.
I think it should be possible to implement the same %feature
nestedworkaround in the typepass, or maybe just before it. At least
some kind of feature should be introduced, because it is impractical
to expect all the language modules be refactored at once. I personally
know well enough only C# and perhaps Java.

Re: [Swig-devel] 2 SWIG patches (C#) and a question

[Vladimir K. <vka...@op...>]

> On Jan 8, 2013, at 4:59 PM, William S Fulton wrote:
>> On 08/01/13 14:20, Vladimir Kalinin wrote:
>> The second problem is Unicode string literals. (Strings like L"Str" )
>> In our code all the strings are wchar_t based and such literals occur
>> in headers from time to time. I proposed a workaround some time ago:
>> http://sourceforge.net/p/swig/bugs/971/,
>> but recently I put some work to it and implemented a full solution.
>> See the patch attached. The wide literals may occur in default
>> arguments and in expressions. In both cases they are correctly parsed.
>> Wide char literals are not unescaped while parsing, because they may
>> contain escaped characters not fitting into "char" type. If the target
>> language wants to use them but cannot handle C escape codes, language
>> module itself may do the necessary conversion.
>>
>> Usage example:
>>
>> %module(directors="1")  MyModule
>> %include "wchar.i"
>> %feature("director") MyClass;
>> %csconst(1);
>>
>> class MyClass
>> {
>> public:
>>    static const wchar_t * const myConst = L"MyWideCharConst\x234";
>>    virtual void myFunc(const wchar_t* myString = L"MyWideChar\x1234Literal\555");
>>    virtual ~MyClass();
>> };
>>
>> Generated C# wrapper code:
>>
>> public class MyClass : IDisposable {
>> ...
>>    public const string myConst = "MyWideCharConst\x234";
>> }
>>
>> Generated C++ proxy code:
>> class SwigDirector_MyClass : public MyClass, public Swig::Director {
>> public:
>>      SwigDirector_MyClass();
>>      virtual void myFunc(wchar_t const *myString = L"MyWideChar\x1234Literal\555");
>> ...
>> };
>>
>>
>> These cases may seem marginal to non C# users, but I think we are not unique
>> with our requirements and these patches may benefit other users. (Of course we may
>> continue using locally forked swig version,
>> but it is rather inconvenient, especially for our clients, some of
>> which may want to regenerate .net wrappers from sources.)
>>
> No doubt someone will benefit, so let's aim to get these into the main
> SWIG distribution, it is better for all.
>
> It might be more complicated than you wish though. This support is 
> already in the gsoc2009-matevz branch: 
> https://github.com/swig/swig/blob/gsoc2009-matevz/Doc/Manual - See 
> Cpp0x.html#Cpp0x_New_string_literals. This branch is my highest priority
> large integration into the mainline. So could I ask you to use this 
> branch for this particular problem and provide any additional change in
> this area to that branch, or wait until I've merged it into master, 
> probably by early Feb.
>
> Aside... anyone know how to provide a direct link so that the above 
> .html file in github is rendered in a browser in github (it was possible
> in svnweb).


I checked out that branch and it doesn't pass the test I quoted above.
Anyway, if there is someone working on it, perhaps I'd better not
interfere. It is unfortunate though, that corresponding bug on SF was
not set to 'assigned'. That would have saved me a day's work.

Vladimir

Re: [Swig-devel] Nested classes. Was: 2 SWIG patches (C#) and a question

[David B. <da...@da...>]

On Jan 9, 2013, at 5:36 AM, Vladimir Kalinin wrote:

>>> On Jan 8, 2013, at 4:59 PM, William S Fulton wrote:
>>>>> On 08/01/13 14:20, Vladimir Kalinin wrote:
>>>> Now, the question:
>>>> Currently I'm working at the nested classes support for C# module. I
>>>> already fixed parser.y, typepass and lang.cxx and working on
>>>> the csharp module itself. One thing I don't understand - why was it
>>>> disabled at the parser level? I mean why the pain with that weird
>>>> "nestedworkaround" approach, reentering the parser (obviously not
>>>> designed for reentrance)? I fear that I'm missing something essential. Who is the
>>>> right person to discuss this with? (I may commit my current state of
>>>> work to some remote repository to review. Not that there is much to look at,
>>>> but at least the symtables are generated correctly (or so it seems))
>>>> 
>>>> (I didn't touch the templates yet, that is where main parsing problems are
>>>> supposed to be.)
>>>> 
>>>> 
>>> Nested class support will be a big step forward for SWIG, so good to 
>>> hear about this. I don't know why the nested class support in the parser 
>>> is a gross hack (the reentering parser bit), probably ease of 
>>> implementation targeting C in the first instance. I think back in about 
>> 
>>> 2000 when SWIG was going through a lot of development, this problem was 
>> 
>>> put on the TODO later list, but it never got attended to. Dave Beazley 
>>> probably knows more, but he isn't involved much these days.
>>> 
>> 
>> The implementation of nested classes in SWIG has always been
>> complicated due to two historical factors.  First, many of the
>> target languages supported by SWIG don't actually support nested
>> classes themselves (or if they do, the support is weak/flaky). 
>> Second, the implementation of SWIG's various language modules didn't
>> allow class code to be emitted in a manner that was reentrant.  That
>> is, if you were in the middle of emitting code for a class, you
>> couldn't just suspend it temporarily and enter a nested code generation context.
>> 
>> The "solution" to both of these problems was to have the parser
>> forcefully unnest the class definitions and play some hacks with
>> parsing and  typenames.    Specifically, if a nested class was
>> found, the parser would simply capture the whole class definition
>> body and set it aside.  After the enclosing class was fully
>> processed, the body of the nested class would then be processed as
>> if it were a separate unnested class definition.    A typedef like
>> "typedef Outer::Inner Outer_Inner" would be made somewhere in the
>> emitted code and the class wrapped as if it were actually called "Outer_Inner".   Yes, a hack.
>> 
>> A major reason why nested class support has never been implemented
>> to this day is due to the fact that making it work would probably
>> require a major refactoring of all of the target language modules to
>> allow reentrant class wrapping.   Given the large number of modules,
>> it's a major task.  I just don't think anyone has been motivated enough to tackle it.
>> 
>> I'm not sure if this helps, but it's a bit of history about why
>> there are some really gross hacks in there ;-). 
>> 
> 
> I quite understand why the nested classes were not implemented, I just
> wondered why it was easier to implement the workaround on the parser
> level, not as a parse tree transformation.
> I think that will have to be done now anyway, because as you correctly
> point out, there are languages w/o nested classes support at all.
> I think it should be possible to implement the same %feature
> nestedworkaround in the typepass, or maybe just before it. At least
> some kind of feature should be introduced, because it is impractical
> to expect all the language modules be refactored at once. I personally
> know well enough only C# and perhaps Java.
> 

Doing it at the parser level is probably a historical artifact.  SWIG started out as a 1-pass parser that simply emitted code without ever building an intermediate parse tree.    Obviously that's different now, but the original implementation of the parser still carries that baggage.   There was probably no motivation to fix it given that nested classes weren't supported in the target languages anyways.

Cheers,
Dave

Re: [Swig-devel] Nested classes. Was: 2 SWIG patches (C#) and a question

[<vka...@op...>]

>>>On Jan 9, 2013, at 5:36 AM, Vladimir Kalinin wrote:
>>>> On Jan 8, 2013, at 4:59 PM, William S Fulton wrote:
>>>>>> On 08/01/13 14:20, Vladimir Kalinin wrote:
>>>>> Now, the question:
>>>>> Currently I'm working at the nested classes support for C# module. I
>>>>> already fixed parser.y, typepass and lang.cxx and working on
>>>>> the csharp module itself. One thing I don't understand - why was it
>>>>> disabled at the parser level? I mean why the pain with that weird
>>>>> "nestedworkaround" approach, reentering the parser (obviously not
>>>>> designed for reentrance)? I fear that I'm missing something essential. Who is the
>>>>> right person to discuss this with? (I may commit my current state of
>>>>> work to some remote repository to review. Not that there is much to look at,
>>>>> but at least the symtables are generated correctly (or so it seems))
>>>>> 
>>>>> (I didn't touch the templates yet, that is where main parsing problems are
>>>>> supposed to be.)
>>>>> 
>>>>> 
>>>> Nested class support will be a big step forward for SWIG, so good to 
>>>> hear about this. I don't know why the nested class support in the parser 
>>>> is a gross hack (the reentering parser bit), probably ease of 
>>>> implementation targeting C in the first instance. I think back in about 
>>> 
>>>> 2000 when SWIG was going through a lot of development, this problem was 
>>> 
>>>> put on the TODO later list, but it never got attended to. Dave Beazley 
>>>> probably knows more, but he isn't involved much these days.
>>>> 
>>> 
>>> The implementation of nested classes in SWIG has always been
>>> complicated due to two historical factors.  First, many of the
>>> target languages supported by SWIG don't actually support nested
>>> classes themselves (or if they do, the support is weak/flaky). 
>>> Second, the implementation of SWIG's various language modules didn't
>>> allow class code to be emitted in a manner that was reentrant.  That
>>> is, if you were in the middle of emitting code for a class, you
>>> couldn't just suspend it temporarily and enter a nested code generation context.
>>> 
>>> The "solution" to both of these problems was to have the parser
>>> forcefully unnest the class definitions and play some hacks with
>>> parsing and  typenames.    Specifically, if a nested class was
>>> found, the parser would simply capture the whole class definition
>>> body and set it aside.  After the enclosing class was fully
>>> processed, the body of the nested class would then be processed as
>>> if it were a separate unnested class definition.    A typedef like
>>> "typedef Outer::Inner Outer_Inner" would be made somewhere in the
>>> emitted code and the class wrapped as if it were actually called "Outer_Inner".   Yes, a hack.
>>> 
>>> A major reason why nested class support has never been implemented
>>> to this day is due to the fact that making it work would probably
>>> require a major refactoring of all of the target language modules to
>>> allow reentrant class wrapping.   Given the large number of modules,
>>> it's a major task.  I just don't think anyone has been motivated enough to tackle it.
>>> 
>>> I'm not sure if this helps, but it's a bit of history about why
>>> there are some really gross hacks in there ;-). 
>>> 
>> 
>> I quite understand why the nested classes were not implemented, I just
>> wondered why it was easier to implement the workaround on the parser
>> level, not as a parse tree transformation.
>> I think that will have to be done now anyway, because as you correctly
>> point out, there are languages w/o nested classes support at all.
>> I think it should be possible to implement the same %feature
>> nestedworkaround in the typepass, or maybe just before it. At least
>> some kind of feature should be introduced, because it is impractical
>> to expect all the language modules be refactored at once. I personally
>> know well enough only C# and perhaps Java.
>> 
>
> Doing it at the parser level is probably a historical artifact. 
> SWIG started out as a 1-pass parser that simply emitted code without
> ever building an intermediate parse tree.    Obviously that's
> different now, but the original implementation of the parser still
> carries that baggage.   There was probably no motivation to fix it
> given that nested classes weren't supported in the target languages anyways.
>

Oh, that explains it, thanks.

Re: [Swig-devel] 2 SWIG patches (C#) and a question

[William S F. <ws...@fu...>]

On 09/01/13 11:57, Vladimir Kalinin wrote:
>> On Jan 8, 2013, at 4:59 PM, William S Fulton wrote:
>>> On 08/01/13 14:20, Vladimir Kalinin wrote:
>>> The second problem is Unicode string literals. (Strings like L"Str" )
>>> In our code all the strings are wchar_t based and such literals occur
>>> in headers from time to time. I proposed a workaround some time ago:
>>> http://sourceforge.net/p/swig/bugs/971/,
>>> but recently I put some work to it and implemented a full solution.
>>> See the patch attached. The wide literals may occur in default
>>> arguments and in expressions. In both cases they are correctly parsed.
>>> Wide char literals are not unescaped while parsing, because they may
>>> contain escaped characters not fitting into "char" type. If the target
>>> language wants to use them but cannot handle C escape codes, language
>>> module itself may do the necessary conversion.
>>>
>>> Usage example:
>>>
>>> %module(directors="1")  MyModule
>>> %include "wchar.i"
>>> %feature("director") MyClass;
>>> %csconst(1);
>>>
>>> class MyClass
>>> {
>>> public:
>>>     static const wchar_t * const myConst = L"MyWideCharConst\x234";
>>>     virtual void myFunc(const wchar_t* myString = L"MyWideChar\x1234Literal\555");
>>>     virtual ~MyClass();
>>> };
>>>
>>> Generated C# wrapper code:
>>>
>>> public class MyClass : IDisposable {
>>> ...
>>>     public const string myConst = "MyWideCharConst\x234";
>>> }
>>>
>>> Generated C++ proxy code:
>>> class SwigDirector_MyClass : public MyClass, public Swig::Director {
>>> public:
>>>       SwigDirector_MyClass();
>>>       virtual void myFunc(wchar_t const *myString = L"MyWideChar\x1234Literal\555");
>>> ...
>>> };
>>>
>>>
>>> These cases may seem marginal to non C# users, but I think we are not unique
>>> with our requirements and these patches may benefit other users. (Of course we may
>>> continue using locally forked swig version,
>>> but it is rather inconvenient, especially for our clients, some of
>>> which may want to regenerate .net wrappers from sources.)
>>>
>> No doubt someone will benefit, so let's aim to get these into the main
>> SWIG distribution, it is better for all.
>>
>> It might be more complicated than you wish though. This support is
>> already in the gsoc2009-matevz branch:
>> https://github.com/swig/swig/blob/gsoc2009-matevz/Doc/Manual - See
>> Cpp0x.html#Cpp0x_New_string_literals. This branch is my highest priority
>> large integration into the mainline. So could I ask you to use this
>> branch for this particular problem and provide any additional change in
>> this area to that branch, or wait until I've merged it into master,
>> probably by early Feb.
>>
>> Aside... anyone know how to provide a direct link so that the above
>> .html file in github is rendered in a browser in github (it was possible
>> in svnweb).
>
>
> I checked out that branch and it doesn't pass the test I quoted above.
> Anyway, if there is someone working on it, perhaps I'd better not
> interfere. It is unfortunate though, that corresponding bug on SF was
> not set to 'assigned'. That would have saved me a day's work.
>
Sorry to hear that, there are way too many bugs and patches to keep 
track of them all. If you don't provide a patch for the gsoc2009-matevz 
branch, I'll merge it in once I've merged the branch into master.

William

Re: [Swig-devel] 2 SWIG patches (C#) and a question

[William S F. <ws...@fu...>]

On 08/01/13 22:59, William S Fulton wrote:
> On 08/01/13 14:20, Vladimir Kalinin wrote:

>>
>> We are using SWIG for quite a while now, and with some success. Last
>> version of our product contains stable version of .NET wrappers
>> generated by SWIG. Of course, due to the size of our API (hundreds of
>> header files), there are some difficult places that SWIG couldn't
>> quite handle.
>> We use directors extensively, and there are few issues in directors
>> support that were very limiting. The main problem is lacking
>> "csdirectorin" "pre:" and "post" code attributes in C# module. Without
>> them it is
>> not trivial to marshal strings and smart-pointers back and forth
>> between user callback code and native code. (especially by reference)
>> I propose a patch introducing this support (with test).
>> It also fixes 2 minor issues in director code generation that are
>> difficult to come by until "csdirectorin" attribute is extended.
>> The first is that "ref" types used in directors lead to invalid
>> signature generation (the type array used to match methods possibly
>> overloaded by user). typeof(ref T) is used instead of
>> typeof().MakeByRefType()
>> The second is that ignored director methods are not completely ignored
>> - if there was a %typemap(imtype, "directorinattributes") it is not
>> skipped for ignored method.
>>
>> The issue was discussed in a mailing list long ago (see e.g.
>> http://permalink.gmane.org/gmane.comp.programming.swig/16385)
>> I proposed a patch a year and a half ago
>> http://sourceforge.net/p/swig/patches/268/
>> Attached to this mail is a git version of the patch, which is easier
>> to review.
> Thanks very much, I'll review that very shortly.
Great patch. All looks good except for some inconsistent spacing in the 
generated code. I'll commit this tomorrow and fix up the spacing. The 
only way it could be further improved is with an example in the docs.

William

Re: [Swig-devel] 2 SWIG patches (C#) and a question

[Vladimir K. <vka...@op...>]

> On 08/01/13 22:59, William S Fulton wrote:
>> On 08/01/13 14:20, Vladimir Kalinin wrote:
>
>>>
>>> We are using SWIG for quite a while now, and with some success. Last
>>> version of our product contains stable version of .NET wrappers
>>> generated by SWIG. Of course, due to the size of our API (hundreds of
>>> header files), there are some difficult places that SWIG couldn't
>>> quite handle.
>>> We use directors extensively, and there are few issues in directors
>>> support that were very limiting. The main problem is lacking
>>> "csdirectorin" "pre:" and "post" code attributes in C# module. Without
>>> them it is
>>> not trivial to marshal strings and smart-pointers back and forth
>>> between user callback code and native code. (especially by reference)
>>> I propose a patch introducing this support (with test).
>>> It also fixes 2 minor issues in director code generation that are
>>> difficult to come by until "csdirectorin" attribute is extended.
>>> The first is that "ref" types used in directors lead to invalid
>>> signature generation (the type array used to match methods possibly
>>> overloaded by user). typeof(ref T) is used instead of
>>> typeof().MakeByRefType()
>>> The second is that ignored director methods are not completely ignored
>>> - if there was a %typemap(imtype, "directorinattributes") it is not
>>> skipped for ignored method.
>>>
>>> The issue was discussed in a mailing list long ago (see e.g.
>>> http://permalink.gmane.org/gmane.comp.programming.swig/16385)
>>> I proposed a patch a year and a half ago
>>> http://sourceforge.net/p/swig/patches/268/
>>> Attached to this mail is a git version of the patch, which is easier
>>> to review.
>> Thanks very much, I'll review that very shortly.
> Great patch. All looks good except for some inconsistent spacing in the
> generated code. I'll commit this tomorrow and fix up the spacing. The 
> only way it could be further improved is with an example in the docs.

Oh, right, the documentation, I quite missed that out, I'll compose
something.
(The spacing is wrong probably because at the time of writing (2 years
ago) I didn't realize that the tab size is 8 spaces in SWIG code. I fixed that in
the code itself recently, but forgot to check the generated code. Thanks for
taking care of that.)

Vladimir

Re: [Swig-devel] 2 SWIG patches (C#) and a question

[Vladimir K. <vka...@op...>]

> On 09/01/13 11:57, Vladimir Kalinin wrote:
>>> On Jan 8, 2013, at 4:59 PM, William S Fulton wrote:
>>>> On 08/01/13 14:20, Vladimir Kalinin wrote:
>>>> The second problem is Unicode string literals. (Strings like L"Str" )
>>>> In our code all the strings are wchar_t based and such literals occur
>>>> in headers from time to time. I proposed a workaround some time ago:
>>>> http://sourceforge.net/p/swig/bugs/971/,
>>>> but recently I put some work to it and implemented a full solution.
>>>> See the patch attached. The wide literals may occur in default
>>>> arguments and in expressions. In both cases they are correctly parsed.
>>>> Wide char literals are not unescaped while parsing, because they may
>>>> contain escaped characters not fitting into "char" type. If the target
>>>> language wants to use them but cannot handle C escape codes, language
>>>> module itself may do the necessary conversion.
>>>>
>>>> Usage example:
>>>>
>>>> %module(directors="1")  MyModule
>>>> %include "wchar.i"
>>>> %feature("director") MyClass;
>>>> %csconst(1);
>>>>
>>>> class MyClass
>>>> {
>>>> public:
>>>>     static const wchar_t * const myConst = L"MyWideCharConst\x234";
>>>>     virtual void myFunc(const wchar_t* myString = L"MyWideChar\x1234Literal\555");
>>>>     virtual ~MyClass();
>>>> };
>>>>
>>>> Generated C# wrapper code:
>>>>
>>>> public class MyClass : IDisposable {
>>>> ...
>>>>     public const string myConst = "MyWideCharConst\x234";
>>>> }
>>>>
>>>> Generated C++ proxy code:
>>>> class SwigDirector_MyClass : public MyClass, public Swig::Director {
>>>> public:
>>>>       SwigDirector_MyClass();
>>>>       virtual void myFunc(wchar_t const *myString = L"MyWideChar\x1234Literal\555");
>>>> ...
>>>> };
>>>>
>>>>
>>>> These cases may seem marginal to non C# users, but I think we are not unique
>>>> with our requirements and these patches may benefit other users. (Of course we may
>>>> continue using locally forked swig version,
>>>> but it is rather inconvenient, especially for our clients, some of
>>>> which may want to regenerate .net wrappers from sources.)
>>>>
>>> No doubt someone will benefit, so let's aim to get these into the main
>>> SWIG distribution, it is better for all.
>>>
>>> It might be more complicated than you wish though. This support is
>>> already in the gsoc2009-matevz branch:
>>> https://github.com/swig/swig/blob/gsoc2009-matevz/Doc/Manual - See
>>> Cpp0x.html#Cpp0x_New_string_literals. This branch is my highest priority
>>> large integration into the mainline. So could I ask you to use this
>>> branch for this particular problem and provide any additional change in
>>> this area to that branch, or wait until I've merged it into master,
>>> probably by early Feb.
>>>
>>> Aside... anyone know how to provide a direct link so that the above
>>> .html file in github is rendered in a browser in github (it was possible
>>> in svnweb).
>>
>>
>> I checked out that branch and it doesn't pass the test I quoted above.
>> Anyway, if there is someone working on it, perhaps I'd better not
>> interfere. It is unfortunate though, that corresponding bug on SF was
>> not set to 'assigned'. That would have saved me a day's work.
>>
> Sorry to hear that, there are way too many bugs and patches to keep 
> track of them all. If you don't provide a patch for the gsoc2009-matevz
> branch, I'll merge it in once I've merged the branch into master.

What is the status of that branch? I mean - is someone still working
on it or it is considered finished?
Because, as you understand, it is a lot harder to fix someone's
code than to write one's own. If there is an author available, I can
show him a failing test case, and he may probably fix what needs to be
fixed, or explain why it is not necessary/possible, or just say that he
doesn't have the time, and then I'll have to fix it myself (after
nested classes).

Actually this is the full class I used for debugging wide char
literals:

struct MyClass
{
  static const wchar_t * const myConst = L"MyWideCharCont\x234";
  static const wchar_t myConstI = L'\34';
  virtual void myFunc(const wchar_t* myString = L"MyWideChar\x1234Literal\555");
  virtual void myFunc1(const char* myString = "MyCharLiteral\x12");
  virtual void myFunc2(wchar_t myChar = L'\x1234');
//  virtual void myFunc3(int myInt = L'\x1334');
  virtual ~MyClass();
};

Commented out function reveals an unrelated bug in default parameters
implementation - we don't retain deducted types of the default values
and expressions. E.g. the constant:

static const int x = '1';

would produce the wrapper

static const int x = 1;

Perhaps I should enter a bug about it, but as I understand
bug tracking is in flux now, so this (rather insignificant) issue may
wait.

Vladimir

Re: [Swig-devel] 2 SWIG patches (C#) and a question

[Vladimir K. <vka...@op...>]

> On 08/01/13 22:59, William S Fulton wrote:
>> On 08/01/13 14:20, Vladimir Kalinin wrote:
>
>>>
>>> We are using SWIG for quite a while now, and with some success. Last
>>> version of our product contains stable version of .NET wrappers
>>> generated by SWIG. Of course, due to the size of our API (hundreds of
>>> header files), there are some difficult places that SWIG couldn't
>>> quite handle.
>>> We use directors extensively, and there are few issues in directors
>>> support that were very limiting. The main problem is lacking
>>> "csdirectorin" "pre:" and "post" code attributes in C# module. Without
>>> them it is
>>> not trivial to marshal strings and smart-pointers back and forth
>>> between user callback code and native code. (especially by reference)
>>> I propose a patch introducing this support (with test).
>>> It also fixes 2 minor issues in director code generation that are
>>> difficult to come by until "csdirectorin" attribute is extended.
>>> The first is that "ref" types used in directors lead to invalid
>>> signature generation (the type array used to match methods possibly
>>> overloaded by user). typeof(ref T) is used instead of
>>> typeof().MakeByRefType()
>>> The second is that ignored director methods are not completely ignored
>>> - if there was a %typemap(imtype, "directorinattributes") it is not
>>> skipped for ignored method.
>>>
>>> The issue was discussed in a mailing list long ago (see e.g.
>>> http://permalink.gmane.org/gmane.comp.programming.swig/16385)
>>> I proposed a patch a year and a half ago
>>> http://sourceforge.net/p/swig/patches/268/
>>> Attached to this mail is a git version of the patch, which is easier
>>> to review.
>> Thanks very much, I'll review that very shortly.
> Great patch. All looks good except for some inconsistent spacing in the
> generated code. I'll commit this tomorrow and fix up the spacing. The 
> only way it could be further improved is with an example in the docs.

Attached is a proposed documentation patch for csdirectorin changes.

Vladimir

Re: [Swig-devel] 2 SWIG patches (C#) and a question

[William S F. <ws...@fu...>]

On 11/01/13 17:46, Vladimir Kalinin wrote:
>> On 08/01/13 22:59, William S Fulton wrote:
>>> On 08/01/13 14:20, Vladimir Kalinin wrote:
>>
>>>>
>>>> We are using SWIG for quite a while now, and with some success. Last
>>>> version of our product contains stable version of .NET wrappers
>>>> generated by SWIG. Of course, due to the size of our API (hundreds of
>>>> header files), there are some difficult places that SWIG couldn't
>>>> quite handle.
>>>> We use directors extensively, and there are few issues in directors
>>>> support that were very limiting. The main problem is lacking
>>>> "csdirectorin" "pre:" and "post" code attributes in C# module. Without
>>>> them it is
>>>> not trivial to marshal strings and smart-pointers back and forth
>>>> between user callback code and native code. (especially by reference)
>>>> I propose a patch introducing this support (with test).
>>>> It also fixes 2 minor issues in director code generation that are
>>>> difficult to come by until "csdirectorin" attribute is extended.
>>>> The first is that "ref" types used in directors lead to invalid
>>>> signature generation (the type array used to match methods possibly
>>>> overloaded by user). typeof(ref T) is used instead of
>>>> typeof().MakeByRefType()
>>>> The second is that ignored director methods are not completely ignored
>>>> - if there was a %typemap(imtype, "directorinattributes") it is not
>>>> skipped for ignored method.
>>>>
>>>> The issue was discussed in a mailing list long ago (see e.g.
>>>> http://permalink.gmane.org/gmane.comp.programming.swig/16385)
>>>> I proposed a patch a year and a half ago
>>>> http://sourceforge.net/p/swig/patches/268/
>>>> Attached to this mail is a git version of the patch, which is easier
>>>> to review.
>>> Thanks very much, I'll review that very shortly.
>> Great patch. All looks good except for some inconsistent spacing in the
>> generated code. I'll commit this tomorrow and fix up the spacing. The
>> only way it could be further improved is with an example in the docs.
>
> Attached is a proposed documentation patch for csdirectorin changes.
>
Superb thanks. I've applied and tweaked a bit and also pushed your code 
patch to master.

William

Re: [Swig-devel] 2 SWIG patches (C#) and a question

[William S F. <ws...@fu...>]

On 10/01/13 23:56, Vladimir Kalinin wrote:
>> On 09/01/13 11:57, Vladimir Kalinin wrote:
>>>> On Jan 8, 2013, at 4:59 PM, William S Fulton wrote:
>>>>> On 08/01/13 14:20, Vladimir Kalinin wrote:
>>>>> The second problem is Unicode string literals. (Strings like L"Str" )
>>>>> In our code all the strings are wchar_t based and such literals occur
>>>>> in headers from time to time. I proposed a workaround some time ago:
>>>>> http://sourceforge.net/p/swig/bugs/971/,
>>>>> but recently I put some work to it and implemented a full solution.
>>>>> See the patch attached. The wide literals may occur in default
>>>>> arguments and in expressions. In both cases they are correctly parsed.
>>>>> Wide char literals are not unescaped while parsing, because they may
>>>>> contain escaped characters not fitting into "char" type. If the target
>>>>> language wants to use them but cannot handle C escape codes, language
>>>>> module itself may do the necessary conversion.
>>>>>
>>>>> Usage example:
>>>>>
>>>>> %module(directors="1")  MyModule
>>>>> %include "wchar.i"
>>>>> %feature("director") MyClass;
>>>>> %csconst(1);
>>>>>
>>>>> class MyClass
>>>>> {
>>>>> public:
>>>>>      static const wchar_t * const myConst = L"MyWideCharConst\x234";
>>>>>      virtual void myFunc(const wchar_t* myString = L"MyWideChar\x1234Literal\555");
>>>>>      virtual ~MyClass();
>>>>> };
>>>>>
>>>>> Generated C# wrapper code:
>>>>>
>>>>> public class MyClass : IDisposable {
>>>>> ...
>>>>>      public const string myConst = "MyWideCharConst\x234";
>>>>> }
>>>>>
>>>>> Generated C++ proxy code:
>>>>> class SwigDirector_MyClass : public MyClass, public Swig::Director {
>>>>> public:
>>>>>        SwigDirector_MyClass();
>>>>>        virtual void myFunc(wchar_t const *myString = L"MyWideChar\x1234Literal\555");
>>>>> ...
>>>>> };
>>>>>
>>>>>
>>>>> These cases may seem marginal to non C# users, but I think we are not unique
>>>>> with our requirements and these patches may benefit other users. (Of course we may
>>>>> continue using locally forked swig version,
>>>>> but it is rather inconvenient, especially for our clients, some of
>>>>> which may want to regenerate .net wrappers from sources.)
>>>>>
>>>> No doubt someone will benefit, so let's aim to get these into the main
>>>> SWIG distribution, it is better for all.
>>>>
>>>> It might be more complicated than you wish though. This support is
>>>> already in the gsoc2009-matevz branch:
>>>> https://github.com/swig/swig/blob/gsoc2009-matevz/Doc/Manual - See
>>>> Cpp0x.html#Cpp0x_New_string_literals. This branch is my highest priority
>>>> large integration into the mainline. So could I ask you to use this
>>>> branch for this particular problem and provide any additional change in
>>>> this area to that branch, or wait until I've merged it into master,
>>>> probably by early Feb.
>>>>
>>>> Aside... anyone know how to provide a direct link so that the above
>>>> .html file in github is rendered in a browser in github (it was possible
>>>> in svnweb).
>>>
>>>
>>> I checked out that branch and it doesn't pass the test I quoted above.
>>> Anyway, if there is someone working on it, perhaps I'd better not
>>> interfere. It is unfortunate though, that corresponding bug on SF was
>>> not set to 'assigned'. That would have saved me a day's work.
>>>
>> Sorry to hear that, there are way too many bugs and patches to keep
>> track of them all. If you don't provide a patch for the gsoc2009-matevz
>> branch, I'll merge it in once I've merged the branch into master.
>
> What is the status of that branch? I mean - is someone still working
> on it or it is considered finished?
A large bulk of the work was done by a summer of code student who hasn't 
stayed around. I mentored him so have a fair idea of what is going on 
and have since improved the work. I don't recall much in that particular 
area, but it really is one of my top priorities to get this work merged 
into master. There are a lot of new functionality in this branch and it 
does require some polishing off though before ready for release.

> Because, as you understand, it is a lot harder to fix someone's
> code than to write one's own. If there is an author available, I can
> show him a failing test case, and he may probably fix what needs to be
> fixed, or explain why it is not necessary/possible, or just say that he
> doesn't have the time, and then I'll have to fix it myself (after
> nested classes).
>
> Actually this is the full class I used for debugging wide char
> literals:
>
> struct MyClass
> {
>    static const wchar_t * const myConst = L"MyWideCharCont\x234";
>    static const wchar_t myConstI = L'\34';
>    virtual void myFunc(const wchar_t* myString = L"MyWideChar\x1234Literal\555");
>    virtual void myFunc1(const char* myString = "MyCharLiteral\x12");
>    virtual void myFunc2(wchar_t myChar = L'\x1234');
> //  virtual void myFunc3(int myInt = L'\x1334');
>    virtual ~MyClass();
> };
>
> Commented out function reveals an unrelated bug in default parameters
> implementation - we don't retain deducted types of the default values
> and expressions. E.g. the constant:
>
> static const int x = '1';
>
> would produce the wrapper
>
> static const int x = 1;
>
> Perhaps I should enter a bug about it, but as I understand
> bug tracking is in flux now, so this (rather insignificant) issue may
> wait.
You can still use SF while the bug tracking is sorted out, and best that 
you do else this will get forgotten about.

William

Re: [Swig-devel] 2 SWIG patches (C#) and a question

[Vladimir K. <vka...@op...>]

>>>>> On Jan 8, 2013, at 4:59 PM, William S Fulton wrote:
>>>>>> On 08/01/13 14:20, Vladimir Kalinin wrote:
>>>>>> The second problem is Unicode string literals. (Strings like L"Str" )
>>>>>> In our code all the strings are wchar_t based and such literals occur
>>>>>> in headers from time to time. I proposed a workaround some time ago:
>>>>>> http://sourceforge.net/p/swig/bugs/971/,
>>>>>> but recently I put some work to it and implemented a full solution.
>>>>>> See the patch attached. The wide literals may occur in default
>>>>>> arguments and in expressions. In both cases they are correctly parsed.
>>>>>> Wide char literals are not unescaped while parsing, because they may
>>>>>> contain escaped characters not fitting into "char" type. If the target
>>>>>> language wants to use them but cannot handle C escape codes, language
>>>>>> module itself may do the necessary conversion.
>>>>>>
>>>>>> Usage example:
>>>>>>
>>>>>> %module(directors="1")  MyModule
>>>>>> %include "wchar.i"
>>>>>> %feature("director") MyClass;
>>>>>> %csconst(1);
>>>>>>
>>>>>> class MyClass
>>>>>> {
>>>>>> public:
>>>>>>      static const wchar_t * const myConst = L"MyWideCharConst\x234";
>>>>>>      virtual void myFunc(const wchar_t* myString = L"MyWideChar\x1234Literal\555");
>>>>>>      virtual ~MyClass();
>>>>>> };
>>>>>>
>>>>>> Generated C# wrapper code:
>>>>>>
>>>>>> public class MyClass : IDisposable {
>>>>>> ...
>>>>>>      public const string myConst = "MyWideCharConst\x234";
>>>>>> }
>>>>>>
>>>>>> Generated C++ proxy code:
>>>>>> class SwigDirector_MyClass : public MyClass, public Swig::Director {
>>>>>> public:
>>>>>>        SwigDirector_MyClass();
>>>>>>        virtual void myFunc(wchar_t const *myString = L"MyWideChar\x1234Literal\555");
>>>>>> ...
>>>>>> };
>>>>>>
>>>>>>
>>>>>> These cases may seem marginal to non C# users, but I think we are not unique
>>>>>> with our requirements and these patches may benefit other users. (Of course we may
>>>>>> continue using locally forked swig version,
>>>>>> but it is rather inconvenient, especially for our clients, some of
>>>>>> which may want to regenerate .net wrappers from sources.)
>>>>>>
>>>>> No doubt someone will benefit, so let's aim to get these into the main
>>>>> SWIG distribution, it is better for all.
>>>>>
>>>>> It might be more complicated than you wish though. This support is
>>>>> already in the gsoc2009-matevz branch:
>>>>> https://github.com/swig/swig/blob/gsoc2009-matevz/Doc/Manual - See
>>>>> Cpp0x.html#Cpp0x_New_string_literals. This branch is my highest priority
>>>>> large integration into the mainline. So could I ask you to use this
>>>>> branch for this particular problem and provide any additional change in
>>>>> this area to that branch, or wait until I've merged it into master,
>>>>> probably by early Feb.
>>>>>
>>>>> Aside... anyone know how to provide a direct link so that the above
>>>>> .html file in github is rendered in a browser in github (it was possible
>>>>> in svnweb).
>>>>
>>>>
>>>> I checked out that branch and it doesn't pass the test I quoted above.
>>>> Anyway, if there is someone working on it, perhaps I'd better not
>>>> interfere. It is unfortunate though, that corresponding bug on SF was
>>>> not set to 'assigned'. That would have saved me a day's work.
>>>>
>>> Sorry to hear that, there are way too many bugs and patches to keep
>>> track of them all. If you don't provide a patch for the gsoc2009-matevz
>>> branch, I'll merge it in once I've merged the branch into master.
>>
>> What is the status of that branch? I mean - is someone still working
>> on it or it is considered finished?
> A large bulk of the work was done by a summer of code student who hasn't
> stayed around. I mentored him so have a fair idea of what is going on 
> and have since improved the work. I don't recall much in that particular
> area, but it really is one of my top priorities to get this work merged
> into master. There are a lot of new functionality in this branch and it
> does require some polishing off though before ready for release.

I reviewed the code changes related to Unicode strings support, and
found out that in gsoc2009-matevz there was added support for custom
delimiters for string literals and wide char literal is regarded as a
particular case. Custom delimited string is converted to normal string
while parsing.

This approach does not work with Unicode literals, unless they are
converted to UTF-8, and even in this case we will lose deducted
literal type (it will become char* instead of wchar_t*).
Unicode string is Unicode not in the name only - it really can contain
wide characters (as escape sequences), and they will not easily fit
into char*.

I suggest to leave custom delimited strings implementation as is, and
merge my implementation of Unicode (L"") strings to this branch,
treating Unicode strings separately.

I can do the merging if you like.

Vladimir

Re: [Swig-devel] 2 SWIG patches (C#) and a question

[William S F. <ws...@fu...>]

On 15/01/13 21:53, Vladimir Kalinin wrote:
>>>>>> On Jan 8, 2013, at 4:59 PM, William S Fulton wrote:
>>>>>>> On 08/01/13 14:20, Vladimir Kalinin wrote:
>>>>>>> The second problem is Unicode string literals. (Strings like L"Str" )
>>>>>>> In our code all the strings are wchar_t based and such literals occur
>>>>>>> in headers from time to time. I proposed a workaround some time ago:
>>>>>>> http://sourceforge.net/p/swig/bugs/971/,
>>>>>>> but recently I put some work to it and implemented a full solution.
>>>>>>> See the patch attached. The wide literals may occur in default
>>>>>>> arguments and in expressions. In both cases they are correctly parsed.
>>>>>>> Wide char literals are not unescaped while parsing, because they may
>>>>>>> contain escaped characters not fitting into "char" type. If the target
>>>>>>> language wants to use them but cannot handle C escape codes, language
>>>>>>> module itself may do the necessary conversion.
>>>>>>>
>>>>>>> Usage example:
>>>>>>>
>>>>>>> %module(directors="1")  MyModule
>>>>>>> %include "wchar.i"
>>>>>>> %feature("director") MyClass;
>>>>>>> %csconst(1);
>>>>>>>
>>>>>>> class MyClass
>>>>>>> {
>>>>>>> public:
>>>>>>>       static const wchar_t * const myConst = L"MyWideCharConst\x234";
>>>>>>>       virtual void myFunc(const wchar_t* myString = L"MyWideChar\x1234Literal\555");
>>>>>>>       virtual ~MyClass();
>>>>>>> };
>>>>>>>
>>>>>>> Generated C# wrapper code:
>>>>>>>
>>>>>>> public class MyClass : IDisposable {
>>>>>>> ...
>>>>>>>       public const string myConst = "MyWideCharConst\x234";
>>>>>>> }
>>>>>>>
>>>>>>> Generated C++ proxy code:
>>>>>>> class SwigDirector_MyClass : public MyClass, public Swig::Director {
>>>>>>> public:
>>>>>>>         SwigDirector_MyClass();
>>>>>>>         virtual void myFunc(wchar_t const *myString = L"MyWideChar\x1234Literal\555");
>>>>>>> ...
>>>>>>> };
>>>>>>>
>>>>>>>
>>>>>>> These cases may seem marginal to non C# users, but I think we are not unique
>>>>>>> with our requirements and these patches may benefit other users. (Of course we may
>>>>>>> continue using locally forked swig version,
>>>>>>> but it is rather inconvenient, especially for our clients, some of
>>>>>>> which may want to regenerate .net wrappers from sources.)
>>>>>>>
>>>>>> No doubt someone will benefit, so let's aim to get these into the main
>>>>>> SWIG distribution, it is better for all.
>>>>>>
>>>>>> It might be more complicated than you wish though. This support is
>>>>>> already in the gsoc2009-matevz branch:
>>>>>> https://github.com/swig/swig/blob/gsoc2009-matevz/Doc/Manual - See
>>>>>> Cpp0x.html#Cpp0x_New_string_literals. This branch is my highest priority
>>>>>> large integration into the mainline. So could I ask you to use this
>>>>>> branch for this particular problem and provide any additional change in
>>>>>> this area to that branch, or wait until I've merged it into master,
>>>>>> probably by early Feb.
>>>>>>
>>>>>> Aside... anyone know how to provide a direct link so that the above
>>>>>> .html file in github is rendered in a browser in github (it was possible
>>>>>> in svnweb).
>>>>>
>>>>>
>>>>> I checked out that branch and it doesn't pass the test I quoted above.
>>>>> Anyway, if there is someone working on it, perhaps I'd better not
>>>>> interfere. It is unfortunate though, that corresponding bug on SF was
>>>>> not set to 'assigned'. That would have saved me a day's work.
>>>>>
>>>> Sorry to hear that, there are way too many bugs and patches to keep
>>>> track of them all. If you don't provide a patch for the gsoc2009-matevz
>>>> branch, I'll merge it in once I've merged the branch into master.
>>>
>>> What is the status of that branch? I mean - is someone still working
>>> on it or it is considered finished?
>> A large bulk of the work was done by a summer of code student who hasn't
>> stayed around. I mentored him so have a fair idea of what is going on
>> and have since improved the work. I don't recall much in that particular
>> area, but it really is one of my top priorities to get this work merged
>> into master. There are a lot of new functionality in this branch and it
>> does require some polishing off though before ready for release.
>
> I reviewed the code changes related to Unicode strings support, and
> found out that in gsoc2009-matevz there was added support for custom
> delimiters for string literals and wide char literal is regarded as a
> particular case. Custom delimited string is converted to normal string
> while parsing.
>
> This approach does not work with Unicode literals, unless they are
> converted to UTF-8, and even in this case we will lose deducted
> literal type (it will become char* instead of wchar_t*).
> Unicode string is Unicode not in the name only - it really can contain
> wide characters (as escape sequences), and they will not easily fit
> into char*.
>
> I suggest to leave custom delimited strings implementation as is, and
> merge my implementation of Unicode (L"") strings to this branch,
> treating Unicode strings separately.
>
Yes good points.

> I can do the merging if you like.
I would very much appreciate that. My main problem with that branch is 
the rvalue references, but I think I have a plan for that now. Once that 
is implemented and after a bit of tidy up, it'll be ready for merge to 
master. Probably I will do a new release just before the merge and then 
one one with the branch. Hopefully over the next month.

William

Re: [Swig-devel] 2 SWIG patches (C#) and a question

[Vladimir K. <vka...@op...>]

> On 15/01/13 21:53, Vladimir Kalinin wrote:
>>>>>>> On Jan 8, 2013, at 4:59 PM, William S Fulton wrote:
>>>>>>>> On 08/01/13 14:20, Vladimir Kalinin wrote:
>>>>>>>> The second problem is Unicode string literals. (Strings like L"Str" )
>>>>>>>> In our code all the strings are wchar_t based and such literals occur
>>>>>>>> in headers from time to time. I proposed a workaround some time ago:
>>>>>>>> http://sourceforge.net/p/swig/bugs/971/,
>>>>>>>> but recently I put some work to it and implemented a full solution.
>>>>>>>> See the patch attached. The wide literals may occur in default
>>>>>>>> arguments and in expressions. In both cases they are correctly parsed.
>>>>>>>> Wide char literals are not unescaped while parsing, because they may
>>>>>>>> contain escaped characters not fitting into "char" type. If the target
>>>>>>>> language wants to use them but cannot handle C escape codes, language
>>>>>>>> module itself may do the necessary conversion.
>>>>>>>>
>>>>>>>> Usage example:
>>>>>>>>
>>>>>>>> %module(directors="1")  MyModule
>>>>>>>> %include "wchar.i"
>>>>>>>> %feature("director") MyClass;
>>>>>>>> %csconst(1);
>>>>>>>>
>>>>>>>> class MyClass
>>>>>>>> {
>>>>>>>> public:
>>>>>>>>       static const wchar_t * const myConst = L"MyWideCharConst\x234";
>>>>>>>>       virtual void myFunc(const wchar_t* myString = L"MyWideChar\x1234Literal\555");
>>>>>>>>       virtual ~MyClass();
>>>>>>>> };
>>>>>>>>
>>>>>>>> Generated C# wrapper code:
>>>>>>>>
>>>>>>>> public class MyClass : IDisposable {
>>>>>>>> ...
>>>>>>>>       public const string myConst = "MyWideCharConst\x234";
>>>>>>>> }
>>>>>>>>
>>>>>>>> Generated C++ proxy code:
>>>>>>>> class SwigDirector_MyClass : public MyClass, public Swig::Director {
>>>>>>>> public:
>>>>>>>>         SwigDirector_MyClass();
>>>>>>>>         virtual void myFunc(wchar_t const *myString = L"MyWideChar\x1234Literal\555");
>>>>>>>> ...
>>>>>>>> };
>>>>>>>>
>>>>>>>>
>>>>>>>> These cases may seem marginal to non C# users, but I think we are not unique
>>>>>>>> with our requirements and these patches may benefit other users. (Of course we may
>>>>>>>> continue using locally forked swig version,
>>>>>>>> but it is rather inconvenient, especially for our clients, some of
>>>>>>>> which may want to regenerate .net wrappers from sources.)
>>>>>>>>
>>>>>>> No doubt someone will benefit, so let's aim to get these into the main
>>>>>>> SWIG distribution, it is better for all.
>>>>>>>
>>>>>>> It might be more complicated than you wish though. This support is
>>>>>>> already in the gsoc2009-matevz branch:
>>>>>>> https://github.com/swig/swig/blob/gsoc2009-matevz/Doc/Manual - See
>>>>>>> Cpp0x.html#Cpp0x_New_string_literals. This branch is my highest priority
>>>>>>> large integration into the mainline. So could I ask you to use this
>>>>>>> branch for this particular problem and provide any additional change in
>>>>>>> this area to that branch, or wait until I've merged it into master,
>>>>>>> probably by early Feb.
>>>>>>>
>>>>>>> Aside... anyone know how to provide a direct link so that the above
>>>>>>> .html file in github is rendered in a browser in github (it was possible
>>>>>>> in svnweb).
>>>>>>
>>>>>>
>>>>>> I checked out that branch and it doesn't pass the test I quoted above.
>>>>>> Anyway, if there is someone working on it, perhaps I'd better not
>>>>>> interfere. It is unfortunate though, that corresponding bug on SF was
>>>>>> not set to 'assigned'. That would have saved me a day's work.
>>>>>>
>>>>> Sorry to hear that, there are way too many bugs and patches to keep
>>>>> track of them all. If you don't provide a patch for the gsoc2009-matevz
>>>>> branch, I'll merge it in once I've merged the branch into master.
>>>>
>>>> What is the status of that branch? I mean - is someone still working
>>>> on it or it is considered finished?
>>> A large bulk of the work was done by a summer of code student who hasn't
>>> stayed around. I mentored him so have a fair idea of what is going on
>>> and have since improved the work. I don't recall much in that particular
>>> area, but it really is one of my top priorities to get this work merged
>>> into master. There are a lot of new functionality in this branch and it
>>> does require some polishing off though before ready for release.
>>
>> I reviewed the code changes related to Unicode strings support, and
>> found out that in gsoc2009-matevz there was added support for custom
>> delimiters for string literals and wide char literal is regarded as a
>> particular case. Custom delimited string is converted to normal string
>> while parsing.
>>
>> This approach does not work with Unicode literals, unless they are
>> converted to UTF-8, and even in this case we will lose deducted
>> literal type (it will become char* instead of wchar_t*).
>> Unicode string is Unicode not in the name only - it really can contain
>> wide characters (as escape sequences), and they will not easily fit
>> into char*.
>>
>> I suggest to leave custom delimited strings implementation as is, and
>> merge my implementation of Unicode (L"") strings to this branch,
>> treating Unicode strings separately.
>>
> Yes good points.
>
>> I can do the merging if you like.
> I would very much appreciate that. My main problem with that branch is
> the rvalue references, but I think I have a plan for that now. Once that
> is implemented and after a bit of tidy up, it'll be ready for merge to
> master. Probably I will do a new release just before the merge and then
> one one with the branch. Hopefully over the next month.

The patch for the "gsoc2009-matevz" branch is attached.

BTW, wchar_t* typemaps in Lib/csharp/wchar.i appear to be unfinished:
SWIG_csharp_wstring_callback is declared but never used in actual
typemaps. (I used my own typemaps for patch testing)

Vladimir

Re: [Swig-devel] 2 SWIG patches (C#) and a question

[William S F. <ws...@fu...>]

> >> I can do the merging if you like.
> > I would very much appreciate that. My main problem with that branch is
> > the rvalue references, but I think I have a plan for that now. Once that
> > is implemented and after a bit of tidy up, it'll be ready for merge to
> > master. Probably I will do a new release just before the merge and then
> > one one with the branch. Hopefully over the next month.
>
> The patch for the "gsoc2009-matevz" branch is attached.
>
> BTW, wchar_t* typemaps in Lib/csharp/wchar.i appear to be unfinished:
> SWIG_csharp_wstring_callback is declared but never used in actual
> typemaps. (I used my own typemaps for patch testing)
>
> Vladimir
>

Hi Vladimir

I'm just looking at this patch and it added %expect in parser.y

+%expect 6

and apparently from this yacc man page: 
http://invisible-island.net/byacc/manpage/yacc.html it is broken in some 
versions of bison. Also it won't be supported in some older versions of 
yacc, so we'll have to check for min versions of these programs. Can you 
get this information as to the min versions required, otherwise, I'll 
drop this part of the patch as the last thing we want is it not work on 
some peoples machines without useful errors, preferably at configure time.

Meanwhile, thanks and I'm trying out the rest of the patch...

William

Re: [Swig-devel] 2 SWIG patches (C#) and a question

[Vladimir K. <vka...@op...>]

>> >> I can do the merging if you like.
>> > I would very much appreciate that. My main problem with that branch is
>> > the rvalue references, but I think I have a plan for that now. Once that
>> > is implemented and after a bit of tidy up, it'll be ready for merge to
>> > master. Probably I will do a new release just before the merge and then
>> > one one with the branch. Hopefully over the next month.
>>
>> The patch for the "gsoc2009-matevz" branch is attached.
>>
>> BTW, wchar_t* typemaps in Lib/csharp/wchar.i appear to be unfinished:
>> SWIG_csharp_wstring_callback is declared but never used in actual
>> typemaps. (I used my own typemaps for patch testing)
>>
>> Vladimir
>>
>
> Hi Vladimir
>
> I'm just looking at this patch and it added %expect in parser.y
>
> +%expect 6
>
> and apparently from this yacc man page: 
> http://invisible-island.net/byacc/manpage/yacc.html it is broken in some
> versions of bison. Also it won't be supported in some older versions of
> yacc, so we'll have to check for min versions of these programs. Can you
> get this information as to the min versions required, otherwise, I'll 
> drop this part of the patch as the last thing we want is it not work on
> some peoples machines without useful errors, preferably at configure time.

This is obviously a cosmetic change, feel free to drop it.

Re: [Swig-devel] 2 SWIG patches (C#) and a question

[William S F. <ws...@fu...>]

On 21/01/13 19:36, William S Fulton wrote:
>
>> >> I can do the merging if you like.
>> > I would very much appreciate that. My main problem with that branch is
>> > the rvalue references, but I think I have a plan for that now. Once
>> that
>> > is implemented and after a bit of tidy up, it'll be ready for merge to
>> > master. Probably I will do a new release just before the merge and then
>> > one one with the branch. Hopefully over the next month.
>>
>> The patch for the "gsoc2009-matevz" branch is attached.
>>
>> BTW, wchar_t* typemaps in Lib/csharp/wchar.i appear to be unfinished:
>> SWIG_csharp_wstring_callback is declared but never used in actual
>> typemaps. (I used my own typemaps for patch testing)
>>
Did you avoid the callback approach? If you have improvements to the 
typemaps, please submit them as another patch. Thanks.

>> Vladimir
>>
>
> Hi Vladimir
>
> I'm just looking at this patch and it added %expect in parser.y
>
> +%expect 6
>
> and apparently from this yacc man page:
> http://invisible-island.net/byacc/manpage/yacc.html it is broken in some
> versions of bison. Also it won't be supported in some older versions of
> yacc, so we'll have to check for min versions of these programs. Can you
> get this information as to the min versions required, otherwise, I'll
> drop this part of the patch as the last thing we want is it not work on
> some peoples machines without useful errors, preferably at configure time.
>
> Meanwhile, thanks and I'm trying out the rest of the patch...

I've applied the patch and some minor changes to the gsoc2009-matevz 
branch now. Sorry about the convoluted way for this to eventually get to 
master!

William

Re: [Swig-devel] 2 SWIG patches (C#) and a question

[Vladimir K. <vka...@op...>]

> On 21/01/13 19:36, William S Fulton wrote:
>>
>>> >> I can do the merging if you like.
>>> > I would very much appreciate that. My main problem with that branch is
>>> > the rvalue references, but I think I have a plan for that now. Once
>>> that
>>> > is implemented and after a bit of tidy up, it'll be ready for merge to
>>> > master. Probably I will do a new release just before the merge and then
>>> > one one with the branch. Hopefully over the next month.
>>>
>>> The patch for the "gsoc2009-matevz" branch is attached.
>>>
>>> BTW, wchar_t* typemaps in Lib/csharp/wchar.i appear to be unfinished:
>>> SWIG_csharp_wstring_callback is declared but never used in actual
>>> typemaps. (I used my own typemaps for patch testing)
>>>
> Did you avoid the callback approach? If you have improvements to the 
> typemaps, please submit them as another patch. Thanks.

I just used built-in marshaling, like this:

%typemap(imtype,
  inattributes="[MarshalAs(UnmanagedType.LPWStr)]", 
  outattributes="[return: MarshalAs(UnmanagedType.LPWStr)]",
  directorinattributes="[MarshalAs(UnmanagedType.LPWStr)]",
  directoroutattributes="[return: MarshalAs(UnmanagedType.LPWStr)]"
) const wchar_t*, const wchar_t*&, const wchar_t* const & "String"

I'm not ready to submit it because I'm kind of busy with nested
classes (that is, when I have time to work on SWIG at all), and these
typemaps require some debugging due to possible memory leaks. We use
our own strings in our project and memory issues are taken care of,
but the universal solution should be, well, universal.

Vladimir.

Re: [Swig-devel] 2 SWIG patches (C#) and a question

[Vladimir K. <vka...@op...>]

>> I'm just looking at this patch and it added %expect in parser.y
>>
>> +%expect 6
>>
>> and apparently from this yacc man page:
>> http://invisible-island.net/byacc/manpage/yacc.html it is broken in some
>> versions of bison. Also it won't be supported in some older versions of
>> yacc, so we'll have to check for min versions of these programs. Can you
>> get this information as to the min versions required, otherwise, I'll
>> drop this part of the patch as the last thing we want is it not work on
>> some peoples machines without useful errors, preferably at configure time.
>>
>> Meanwhile, thanks and I'm trying out the rest of the patch...
>
> I've applied the patch and some minor changes to the gsoc2009-matevz 
> branch now. Sorry about the convoluted way for this to eventually get to
> master!
>

Thank you. Glad that it's finally there.