wxlua-users

[wxlua-users] coroutine.yield doesn't work ?

[TOUPIN Y. <you...@wa...>]

Hi,
I got a strange behavior when creating a dialog from a coroutine, and 
resuming the coroutine when a dialog button is clicked. It seems the 
function called by the button click event is executed as if it was 
called by the coroutine, while, unless I'm missing something obvious, it 
should not. I've isolated the problem in a small application :

-- I overwrite print and assert functions to get messages in the console 
window
local print = print -- don't want wxLua print function
local assert = function( cond, msg, ... )
    if not cond then
        print( "Assertion failed: "..msg )
        print( debug.traceback() )
    end
    return cond, msg, ...
end

require( "wx" )

function callback()
    print( "Button clicked" )
    assert( coroutine.resume( co ) )
end

function createDialog()
    dialog = wx.wxDialog( wx.NULL, wx.wxID_ANY, "Test dialog", 
wx.wxDefaultPosition, wx.wxDefaultSize )
    panel = wx.wxPanel( dialog, wx.wxID_ANY )
    sizer = wx.wxBoxSizer( wx.wxHORIZONTAL )

    button = wx.wxButton( panel, wx.wxID_ANY, "Test button" )
    dialog:Connect( button:GetId(), wx.wxEVT_COMMAND_BUTTON_CLICKED, 
callback )

    sizer:Add( button )

    panel:SetSizer( sizer )
    sizer:SetSizeHints( dialog )
    dialog:Show( true )
end

co = coroutine.create( function()
    createDialog()
    print( "Yield..." )
    coroutine.yield()
    print( "After yield" )
end )

assert( coroutine.resume( co ) )

wx.wxGetApp():MainLoop()


Here is the log I get :
Yield...
Button clicked
Assertion failed: cannot resume running coroutine
stack traceback:
        test.lua:6: in function 'assert'
        test.lua:15: in function <test.lua:13>
        [C]: in function 'yield'
        test.lua:36: in function <test.lua:33>

It seems very weird to me, that something is in the callstack above the 
"yield" call... I've not looked into wxLua source code yet, don't know 
if I'm doing something wrong or if there is some bug here.
Any help would be appreciated...

Re: [wxlua-users] coroutine.yield doesn't work ?

[TOUPIN Y. <you...@wa...>]

It seems the problem comes from the Connect function call, which uses 
the current lua state to register the event callback. When the callback 
is called, it is from that lua state, which, in my case, is the 
coroutine, not the main lua thread. Is there any reason for this 
behavior ? Wouldn't it be better to get the main lua thread (I think it 
can be retrieved from the coroutine) when the Connect function is 
called, and use it instead of the coroutine ?

Re: [wxlua-users] coroutine.yield doesn't work ?

[John L. <jla...@gm...>]

On Sun, May 17, 2009 at 2:47 PM, TOUPIN Youen <you...@wa...>wrote:

> It seems the problem comes from the Connect function call, which uses
> the current lua state to register the event callback. When the callback
> is called, it is from that lua state, which, in my case, is the
> coroutine, not the main lua thread. Is there any reason for this
> behavior ? Wouldn't it be better to get the main lua thread (I think it
> can be retrieved from the coroutine) when the Connect function is
> called, and use it instead of the coroutine ?
>
>
I don't think your example makes sense. You create a coroutine that creates
a wxDialog then yield it and try to resume it from the yielded coroutine.
Shouldn't the wxDialog be created in the main thread which can control the
coroutine as appropriate?

Regards,
    John

Re: [wxlua-users] coroutine.yield doesn't work ?

[John L. <jla...@gm...>]

On Sun, May 17, 2009 at 4:04 PM, John Labenski <jla...@gm...> wrote:

> On Sun, May 17, 2009 at 2:47 PM, TOUPIN Youen <you...@wa...>wrote:
>
>> It seems the problem comes from the Connect function call, which uses
>> the current lua state to register the event callback. When the callback
>> is called, it is from that lua state, which, in my case, is the
>> coroutine, not the main lua thread. Is there any reason for this
>> behavior ? Wouldn't it be better to get the main lua thread (I think it
>> can be retrieved from the coroutine) when the Connect function is
>> called, and use it instead of the coroutine ?
>>
>>
> I don't think your example makes sense. You create a coroutine that creates
> a wxDialog then yield it and try to resume it from the yielded coroutine.
> Shouldn't the wxDialog be created in the main thread which can control the
> coroutine as appropriate?
>
>
I should clarify, in Lua there is no way for a coroutine's code to be run
when it's yielded, unless I'm missing something. You, with the wxWidget's
event loop can "cheat", so to say, and pop back into the yielded coroutine
to run code which would not normally be possible.

I think it's a reasonable constraint that a coroutine cannot resume itself
since 1) it stays truer to Lua's behavior, 2) doesn't require second
guessing the user by swapping back to the main thread, which would be
unexpected to say the least, 3) the fix is simple, put any code that will
control a coroutine into the parent thread that created the coroutine where
it belongs.

Regards,
    John

Re: [wxlua-users] coroutine.yield doesn't work ?

[lostgallifreyan <los...@gm...>]

John Labenski <jla...@gm...> wrote:
(17/05/2009 22:04)

>I think it's a reasonable constraint that a coroutine cannot resume itself
>since 1) it stays truer to Lua's behavior, 2) doesn't require second
>guessing the user by swapping back to the main thread, which would be
>unexpected to say the least, 3) the fix is simple, put any code that will
>control a coroutine into the parent thread that created the coroutine where
>it belongs.

When I was looking at it I was wondering where and how I'd pass a returned value if I had to, which sort of fits with what you just said. I've never used a co-routine yet, but if I do, is it right to consider it purely as a sub-routine that happens to use a separate thread, and not to concern myself with the underlying methods that make it so?

Re: [wxlua-users] coroutine.yield doesn't work ?

[Youen T. <you...@wa...>]

lostgallifreyan a écrit :
> When I was looking at it I was wondering where and how I'd pass a returned value if I had to, which sort of fits with what you just said. I've never used a co-routine yet, but if I do, is it right to consider it purely as a sub-routine that happens to use a separate thread, and not to concern myself with the underlying methods that make it so?
>   
A coroutine is not a thread, it will not run in parallel of anything 
else. It is similar to a thread because when you call coroutine.yield(), 
it will resume the execution of the main thread, the coroutine is 
paused, its callstack is preserved. Then you can call coroutine.resume( 
yourCoroutine ) to resume the execution of the coroutine just after the 
call to coroutine.yield. For example, if you call coroutine.yield() very 
often, and resume the coroutine from a main loop, this will give a 
parallel-like execution of the main loop and the coroutine.

I'm not sure to understand your question about the return value, but if 
you want to pass a return value from a coroutine to the main thread, you 
can. It is the last coroutine.resume called by the main thread that will 
get the result. You can not get the result at the location where you 
initially started the coroutine however.

Re: [wxlua-users] coroutine.yield doesn't work ?

[lostgallifreyan <los...@gm...>]

Youen Toupin <you...@wa...> wrote:
(19/05/2009 19:22)

>lostgallifreyan a écrit :
>> When I was looking at it I was wondering where and how I'd pass a returned value if I had to, which sort of fits with what you just said. I've never used a co-routine yet, but if I do, is it right to consider it purely as a sub-routine that happens to use a separate thread, and not to concern myself with the underlying methods that make it so?
>>   
>A coroutine is not a thread, it will not run in parallel of anything 
>else. It is similar to a thread because when you call coroutine.yield(), 
>it will resume the execution of the main thread, the coroutine is 
>paused, its callstack is preserved. Then you can call coroutine.resume( 
>yourCoroutine ) to resume the execution of the coroutine just after the 
>call to coroutine.yield. For example, if you call coroutine.yield() very 
>often, and resume the coroutine from a main loop, this will give a 
>parallel-like execution of the main loop and the coroutine.
>

I only recently adapted to the message loop. :) I'm not yet sure how I can use co-routines, though I think they're native to Lua, so would it be useful to make something that behaves like a message loop in Lua? Before I start exploring them, I really need to know what might make them indispensible, either impossible by other ways, or at least not done as well otherwise. I'm also unsure how it shows parallel-like behaviour if it does not run parallel to anything else, except in the sense of multiplexing, which is what multi-threading is also.

>I'm not sure to understand your question about the return value, but if 
>you want to pass a return value from a coroutine to the main thread, you 
>can. It is the last coroutine.resume called by the main thread that will 
>get the result. You can not get the result at the location where you 
>initially started the coroutine however.
>

Understood. I think. So not at all like a subroutine or called function that passes back values. I guess one way is a global that gets examined periodically by other code for any value a co-routine puts in it might be a useful trick.

Re: [wxlua-users] coroutine.yield doesn't work ?

[Youen T. <you...@wa...>]

John Labenski a écrit :
>
> On Sun, May 17, 2009 at 4:04 PM, John Labenski <jla...@gm... 
> <mailto:jla...@gm...>> wrote:
>
>     On Sun, May 17, 2009 at 2:47 PM, TOUPIN Youen
>     <you...@wa... <mailto:you...@wa...>> wrote:
>
>         It seems the problem comes from the Connect function call,
>         which uses
>         the current lua state to register the event callback. When the
>         callback
>         is called, it is from that lua state, which, in my case, is the
>         coroutine, not the main lua thread. Is there any reason for this
>         behavior ? Wouldn't it be better to get the main lua thread (I
>         think it
>         can be retrieved from the coroutine) when the Connect function is
>         called, and use it instead of the coroutine ?
>
>
>     I don't think your example makes sense. You create a coroutine
>     that creates a wxDialog then yield it and try to resume it from
>     the yielded coroutine. Shouldn't the wxDialog be created in the
>     main thread which can control the coroutine as appropriate?
>
>
> I should clarify, in Lua there is no way for a coroutine's code to be 
> run when it's yielded, unless I'm missing something. You, with the 
> wxWidget's event loop can "cheat", so to say, and pop back into the 
> yielded coroutine to run code which would not normally be possible.
>
> I think it's a reasonable constraint that a coroutine cannot resume 
> itself since 1) it stays truer to Lua's behavior, 2) doesn't require 
> second guessing the user by swapping back to the main thread, which 
> would be unexpected to say the least, 3) the fix is simple, put any 
> code that will control a coroutine into the parent thread that created 
> the coroutine where it belongs.
>
> Regards,
>     John
Thanks for the quick answer.

I know it doesn't make sense to resume a coroutine from itself (didn't 
even know it was possible to execute something from a yielded 
coroutine), that's not what I'm trying to do. What I was trying to do is 
to create the dialog from the coroutine, then yield, and when a button 
is pressed, it executes some code from the main thread, that resumes the 
coroutine. This would allow for example to create code like

answer = promptUser( "Some question" )

The promptUser function creates the dialog, registers a callback on a 
button click event, then yield (so it must be called from a coroutine). 
When the button is clicked, it resumes the promptUser function, which 
gets the answer from a text control, destroys the dialog box, and 
returns the result.

The only problem is that, in the current implementation, I need to call 
the Connect function from the main thread to register my callback, so I 
can't write such a promptUser function that contains everything to 
create the dialog, I will have to create the dialog from the main 
thread, and promptUser will simply show/hide it when needed. This does 
not allow to call multiple promptUser in parallel, for example. And if 
my dialog is complicated and needs to be created differently depending 
on promptUser parameters, it will become more complicated to implement. 
More generally, it creates a dependency between my promptUser function, 
and some initialization code that need to be executed from the main thread.

Maybe the good solution for this problem would be to have the 
possibility to decide from which thread the function must be called, 
when it is registered on the button click event (with an extra parameter 
of the Connect function for example). It would give more possibilities 
to the user, while staying simple (no obligation to provide the 
additional parameter) and backward compatible (the default behavior 
would be to call the function from the thread that registered it).

There is another problem, however : I don't know if it is a good idea to 
create a dialog at the moment it is needed. I'm not used enough to 
wxWidgets to know if it can introduce some uncomfortable lag in the 
application.

Regards,
Youen.

Re: [wxlua-users] coroutine.yield doesn't work ?

[lostgallifreyan <los...@gm...>]

Youen Toupin <you...@wa...> wrote:
(18/05/2009 08:26)

>There is another problem, however : I don't know if it is a good idea to 
>create a dialog at the moment it is needed. I'm not used enough to 
>wxWidgets to know if it can introduce some uncomfortable lag in the 
>application.

You can create it at program load, then use dialog:Show() when you want it, and dialog:Show(false) to hide it again, then there will be no lag. Especially useful for modeless dialogs. From what I've seen it's fast at making them anyway.

Re: [wxlua-users] coroutine.yield doesn't work ?

[Youen T. <you...@wa...>]

John Labenski a écrit :
> I think that all you need to do is to shuffle around your existing 
> code to pull out the dialog creation from the coroutine code, I also 
> demonstrate how to pass data around without using globals.
Yes, this is a solution, but I would like to avoid it if possible. My 
idea is to use coroutines to have a flexible system that would allow 
sequential programming to perform tasks even when these tasks require 
user action, or waiting for anything without blocking the application. 
The promptUser function is only an example. I'd like to be able to write 
things like:

local filename = openFileDialog()
if filename ~= nil then
    if string.sub( filename, -10 ) == ".something" then
        local choice = promptUser( "Are you sure?" )
        if choice then
           -- do something with filename
        end
    else
        -- do something else with filename
    end
end

And this piece of code could happen to be deap inside a coroutine 
callstack, so the main thread that started it do not know it needed to 
create dialogs and connect events before starting the coroutine.
> Alternatively, you can do exactly what you had before and simply 
> connect to the button after you create the dialog in the coroutine. 
> wxWidgets doesn't care or know what coroutine you're in since they're 
> all in the same C thread.
I don't understand what you mean here. I think it is exactly what I've 
done in my first example, but it doesn't work, because after connecting 
my callback on the button, I yield, and the callback is executed in the 
coroutine stack (which is the part I didn't understand in my first 
e-mail), so the callback registered from the coroutine can not resume 
the same coroutine.
>
> Or, you can use the code you originally had, but use wxPostEvent(...) 
> to send your own event from the Lua coroutine back to a function 
> created and connected in the main thread.
If I understand correctly, this would allow me to execute code in the 
main thread from my coroutine ? I will look further into this, it could 
solve my problem.
>
> Finally, if all you're doing is trying to create dialogs to get user 
> input you wouldn't bother with coroutines since wxWidgets has an event 
> loop. The only use of a coroutine would be for some long calculation, 
> but again this is usually performed with idle events (wxEVT_IDLE) to 
> allow the GUI to refresh.
I don't want to use coroutines to execute things in parallel, but only 
to have nice and easily readable code as the example at the beginning of 
this e-mail.
>  
> You can create dozens and dozens of GUI elements with no noticeable 
> lag. You may notice that on a very slow machine the controls.wx.lua 
> may be slow to start, but that is because we are connecting to every 
> imagineable event for every control created. It is not a realistic 
> benchmark.
Ok :-)
> Regards,
>     John
>
Thanks again for the answers. I think now I have enough solutions to 
pick one that suit my needs in my real application :-)

Re: [wxlua-users] coroutine.yield doesn't work ?

[John L. <jla...@gm...>]

On Mon, May 18, 2009 at 3:26 AM, Youen Toupin <you...@wa...>wrote:

> John Labenski a écrit :
> >
>
> I know it doesn't make sense to resume a coroutine from itself (didn't
> even know it was possible to execute something from a yielded
> coroutine), that's not what I'm trying to do. What I was trying to do is
> to create the dialog from the coroutine, then yield, and when a button
> is pressed, it executes some code from the main thread, that resumes the
> coroutine. This would allow for example to create code like
>
> answer = promptUser( "Some question" )
>
> The promptUser function creates the dialog, registers a callback on a
> button click event, then yield (so it must be called from a coroutine).
> When the button is clicked, it resumes the promptUser function, which
> gets the answer from a text control, destroys the dialog box, and
> returns the result.
>
> The only problem is that, in the current implementation, I need to call
> the Connect function from the main thread to register my callback, so I
> can't write such a promptUser function that contains everything to
> create the dialog, I will have to create the dialog from the main
> thread, and promptUser will simply show/hide it when needed. This does
> not allow to call multiple promptUser in parallel, for example. And if
> my dialog is complicated and needs to be created differently depending
> on promptUser parameters, it will become more complicated to implement.
> More generally, it creates a dependency between my promptUser function,
> and some initialization code that need to be executed from the main thread.
>

I think that all you need to do is to shuffle around your existing code to
pull out the dialog creation from the coroutine code, I also demonstrate how
to pass data around without using globals.

function MyEventHandler(event)
    local o = event:GetEventObject()

    -- side note: to typecast the wxObject to what it really is use
    --     local button = o:DynamicCast("wxButton")

    -- can also use
    -- local w = o:DynamicCast("wxWindow")
    -- local parent = w:GetParent() and so on 'till you get the window you
want

    local co = o.data.co  -- This is your coroutine!
    coroutine.resume(co, ...)
end

function promptUser(...)

    local data = {}

    local dialog = wx.wxDialog(....)
    dialog.data = data
    local  button = wx.wxButton(dialog, 10, "Press me!")
    button.data = data -- see above about w:GetParent() for alternative
    populate dialog...

    dialog:Connect(...)
    -- tuck the coroutine into the data so the callback function can find it
    data.co = coroutine.create(function() code for your worker function end,
dialog)

    data.extra_data = "This is the data I really want to pass around"

    return dialog
end

Note that you can pass whatever params you want into your promptUser()
function to customize the creation and also note that you can pass the data
as a parameter for resume() like this to exchange data:

coroutine.resume(dialog.co, dialog.data)

--------------------------

Alternatively, you can do exactly what you had before and simply connect to
the button after you create the dialog in the coroutine. wxWidgets doesn't
care or know what coroutine you're in since they're all in the same C
thread.

Or, you can use the code you originally had, but use wxPostEvent(...) to
send your own event from the Lua coroutine back to a function created and
connected in the main thread.

Finally, if all you're doing is trying to create dialogs to get user input
you wouldn't bother with coroutines since wxWidgets has an event loop. The
only use of a coroutine would be for some long calculation, but again this
is usually performed with idle events (wxEVT_IDLE) to allow the GUI to
refresh.


>
> Maybe the good solution for this problem would be to have the
> possibility to decide from which thread the function must be called,
> when it is registered on the button click event (with an extra parameter
> of the Connect function for example). It would give more possibilities
> to the user, while staying simple (no obligation to provide the
> additional parameter) and backward compatible (the default behavior
> would be to call the function from the thread that registered it).
>

I don't think that there is any reason to make the C code switch threads
when it is just as well if you do it yourself in Lua as in the example
above. It's nearly identical to your original code, all that's changed is
that the function that creates the dialog is not run in the coroutine, but
simply calls coroutine.create() at the end.


>
> There is another problem, however : I don't know if it is a good idea to
> create a dialog at the moment it is needed. I'm not used enough to
> wxWidgets to know if it can introduce some uncomfortable lag in the
> application.
>


You can create dozens and dozens of GUI elements with no noticeable lag. You
may notice that on a very slow machine the controls.wx.lua may be slow to
start, but that is because we are connecting to every imagineable event for
every control created. It is not a realistic benchmark.

Regards,
    John

Re: [wxlua-users] coroutine.yield doesn't work ?

[John L. <jla...@gm...>]

On Tue, May 19, 2009 at 2:14 PM, Youen Toupin <you...@wa...>wrote:

> John Labenski a écrit :
> > I think that all you need to do is to shuffle around your existing
> > code to pull out the dialog creation from the coroutine code, I also
> > demonstrate how to pass data around without using globals.
> Yes, this is a solution, but I would like to avoid it if possible. My
> idea is to use coroutines to have a flexible system that would allow
> sequential programming to perform tasks even when these tasks require
> user action, or waiting for anything without blocking the application.
> The promptUser function is only an example. I'd like to be able to write
> things like:
>

I'm not sure I understand. GUI programs usually don't work this way, they
are event loop based and therefore they are not sequential. If you want to
pop up a dialog that is non-modal simply use wxDialog:Show(true) instead of
ShowModal().

In any case, if you really want to use coroutines you can use the below. I
think your problem was simply that you were calling coroutine.resume from
within the coroutine so all the code does below is make sure that the
callback() function which will call coroutine.resume() is called from the
main Lua state. I would probably go with the c_fn() way, which would allow
you to use far more local vars, but you'll have to make quite a few more
mods to pass more data around as params before you can get rid of all the
globals...

Regards,
    John

==============================
-- I overwrite print and assert functions to get messages in the console
window
--local print = print -- don't want wxLua print function
local assert = function( cond, msg, ... )
   if not cond then
       print( "Assertion failed: "..msg )
       print( debug.traceback() )
   end
   return cond, msg, ...
end

require( "wx" )

function callback()
   print( "Button clicked" )
   assert( coroutine.resume( co ) )
end

function connect_fn()
    dialog:Connect( button:GetId(), wx.wxEVT_COMMAND_BUTTON_CLICKED,
callback )
end

function createDialog()
   dialog = wx.wxDialog( wx.NULL, wx.wxID_ANY, "Test dialog",
wx.wxDefaultPosition, wx.wxDefaultSize )
   panel = wx.wxPanel( dialog, wx.wxID_ANY )
   sizer = wx.wxBoxSizer( wx.wxHORIZONTAL )
   button = wx.wxButton( panel, wx.wxID_ANY, "Test button" )
   sizer:Add( button )

   panel:SetSizer( sizer )
   sizer:SetSizeHints( dialog )
   dialog:Show( true )
end

co = coroutine.create( function()
   createDialog()
   print( "Yield..." )
   coroutine.yield(connect_fn)
   print( "After yield" )
   coroutine.yield()
   print( "After yield 2" )
   coroutine.yield()
   print( "After yield 3" )
end )

ret, c_fn = coroutine.resume( co )
print(ret, c_fn, dialog, button)

-- each of these ways works
--c_fn()
--dialog:Connect( button:GetId(), wx.wxEVT_COMMAND_BUTTON_CLICKED, callback
)
connect_fn()

wx.wxGetApp():MainLoop()

Re: [wxlua-users] coroutine.yield doesn't work ?

[lostgallifreyan <los...@gm...>]

John Labenski <jla...@gm...> wrote:
(20/05/2009 01:24)

>On Tue, May 19, 2009 at 2:14 PM, Youen Toupin <you...@wa...>wrote:
>
>> John Labenski a écrit :
>> > I think that all you need to do is to shuffle around your existing
>> > code to pull out the dialog creation from the coroutine code, I also
>> > demonstrate how to pass data around without using globals.
>> Yes, this is a solution, but I would like to avoid it if possible. My
>> idea is to use coroutines to have a flexible system that would allow
>> sequential programming to perform tasks even when these tasks require
>> user action, or waiting for anything without blocking the application.
>> The promptUser function is only an example. I'd like to be able to write
>> things like:
>>
>
>I'm not sure I understand. GUI programs usually don't work this way, they
>are event loop based and therefore they are not sequential. If you want to
>pop up a dialog that is non-modal simply use wxDialog:Show(true) instead of
>ShowModal().
>

Not that I think my knowledge is enough to warrant high observance, but I back this up. I said this earlier too. I'm recently learning C (on TCC compiler, using API), and learned that wxLua (at least, wxWidgets is so both) is very like C being based in it. Modeless dialogs will sit and wait while the rest of program hangs on every word otherwise uttered by the event loop. I used to hate this concept, being taught on linear sequences that always had to halt while waiting for input, but now I really like this event loop thing.

Re: [wxlua-users] coroutine.yield doesn't work ?

[Youen T. <you...@wa...>]

John Labenski a écrit :
>
> I'm not sure I understand. GUI programs usually don't work this way, 
> they are event loop based and therefore they are not sequential. If 
> you want to pop up a dialog that is non-modal simply use 
> wxDialog:Show(true) instead of ShowModal().
I'm not used to GUI programs, but the fact that they "usually [...] are 
event loop based" does not mean it is the best solution :-) Lua provides 
an interesting tool that do not exist natively in C (the coroutines), 
and that can, in my opinion, help make programs easier to write and 
easier to read. Maybe its a bad idea, but until now, it works fine for 
me, excepted this callback problem. I've solved the problem by creating 
a small module that allows to execute, from a coroutine, a function in 
the main thread. It helps to keep the main thread code independant from 
the coroutine code. My application works fine now, thanks for the help.