From: Simon M. <mar...@gm...> - 2009-11-02 15:14:56
|
Dear gtk2hs developers, What's the current state of gtk2hs with respect to the threaded RTS and C finalizers? Should we expect 0.10.1 to work? As you may know, we're working on ThreadScope, a gtk2hs-based tool for parallel performance visualisation in GHC. I recently switched to use the threaded RTS so that I could do some work while displaying a progress bar - I know there are ways to do this without -threaded, but I thought -threaded would be the neatest way to do it, and eventually we'll want to have background threads doing things anyway. Now I have weird problems - usually hangs/deadlocks, but occasionally I get an Xlib error. Here's my latest hang, which is 100% reproducible on exiting ThreadScope right now: (gdb) where #0 0x0000003c0aa0af19 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00000000005b2c0e in waitCondition (pCond=0x1a43918, pMut=0x1a43948) at posix/OSThreads.c:65 #2 0x00000000005d72d9 in waitForReturnCapability (pCap=0x7ffffe1499f0, task=0x1a438e0) at Capability.c:445 #3 0x00000000005a5934 in rts_lock () at RtsAPI.c:572 #4 0x00000000004cf247 in SystemziGlibziGObject_d1wV () #5 0x0000003c10cd8eed in ?? () from /usr/lib64/libgtk-x11-2.0.so.0 #6 0x0000003c0ba0d6a8 in g_object_unref () from /lib64/libgobject-2.0.so.0 .... various stuff omitted .... #41 0x0000003c0ba0d632 in g_object_unref () from /lib64/libgobject-2.0.so.0 #42 0x00000000005b20fd in runCFinalizer (fn=0x408968, ptr=0x1a69440, env=0x0, flag=0) at Weak.c:34 #43 0x00000000005b216a in runAllCFinalizers (list=0x2afbacb5f060) at Weak.c:50 #44 0x00000000005a8077 in hs_exit_ (wait_foreign=rtsFalse) at RtsStartup.c:413 #45 0x00000000005a81a8 in shutdownHaskellAndExit (n=0) at RtsStartup.c:557 #46 0x00000000005a4c6a in real_main () at Main.c:144 #47 0x00000000005a4ca4 in main (argc=5, argv=0x7ffffe14b338) at Main.c:156 So clearly g_object_unref is being called from a C finalizer, and it ends up calling back into Haskell via something in System.Glib.Object, this leads to a deadlock because the RTS is shutting down and can't run any Haskell code. Any help appreciated. In the meantime I'll probably go back to the single-threaded RTS. Cheers, Simon |
From: Axel S. <Axe...@en...> - 2009-11-02 21:11:54
|
Hi Simon, On Nov 2, 2009, at 16:14, Simon Marlow wrote: > Dear gtk2hs developers, > > What's the current state of gtk2hs with respect to the threaded RTS > and > C finalizers? Should we expect 0.10.1 to work? Well, everything's brittle since ghc's garbage collector went concurrent. The story is as follows: Calling g_object_unref, the principal finalizer we attach to any ForeignPtr, will decrement the reference count and immediately finalize an object if this counter is zero. This can pose problems in three ways: a) Storing any callback or other memory region in the GObject will install a Haskell finalizer (freeStablePtr or free function pointer) that is called when the GObject is finalized. After our last long conversation about the new concurrent GC, we changed Gtk2Hs so that it passes the addresses of the C functions of the ghc runtime as finalizers, so Haskell is not called and all is well. (I hope I recall this correctly.) b) It is possible to add a Haskell finalizer function to a GObject. Since these Haskell finalizers are called from the GC, they have to re-enter the Haskell RTS which will bomb. Solution: remove the possibility to add such a finalizer function. c) Xlib is not thread-safe but has no thread-local storage so it can be called from several threads if they don't access Xlib at the same time. Win32 API is not thread-safe *and* uses thread-local storage so that it is impossible to call it from more than one thread. Since a GObject may contain X or Win32 ressources, these might be freed from a thread calling g_object_unref. Thus, if the ghc GC calls g_object_unref from several threads, it is inevitable that Xlib or Win32 is called from several threads. This is what you observe below when get an Xlib error. I have a patched version of Gtk2Hs lying around that replaces the finalizer 'g_object_unref' with a wrapper function that first acquires a global Gtk lock before calling g_object_unref. This may severely stall the GC but works well for Xlib. On Windows, this still doesn't work. Nor does it solve b). The only solution I can think of is to insert all GObjects that are finalized during GC into a list and to finialize all these objects by calling a function (say 'runFinalizers') from the main Gtk thread. This function can be called from Haskell in a 'safe' way so that callbacks to Haskell are no problem. The problem that remains is how to ensure that 'runFinalizers' is called after each GC. When using Gtk+, I could add the 'runFinalizer' function to the main loop of Gtk. Thus, when the GUI resumes after a GC, it will finalize all objects that were dead during the last GC. However, if we ever split off Cairo or Pango (two modules that may be used independently of the GUI stuff) then there is no more Gtk+ main loop. Thus, we would need another way to run the 'runFinalizers' C function. One idea I had to this end is a hook into the ghc runtime: We need to install a C pointer to a function that is called at the end of each GC. Additionally, we need to ensure that this function is called by one specific OS thread. Alternatively, we would need a Haskell-side hook to install a OS-thread bound Haskell function that, in turn, calls the C function. I talked to Duncan about this but he was skeptical that GHC headquarters would be happy about this. <rant> In fact, the situation is a bit contradictory: As an application developer you are responsible to carefully call all your functions from an (OS-)bounded thread if your C library isn't thread-safe and then the GC comes along and calls concurrently with any number of OS threads into the C library to run the finalizers. This is not quite fair onto the programmer. <rant/> Another idea would be to have a new type of finalizers that takes a thread id and that the GC then guarantees to only run from the given thread id. This would push the burden into the GC implementation. Any thoughts appreciated. With the current solution, I don't think there's a way for us to make Gtk2Hs thread-safe except for the hack of using the Gtk+ main loop to run finalizers. Cheers, Axel. > As you may know, we're working on ThreadScope, a gtk2hs-based tool for > parallel performance visualisation in GHC. I recently switched to use > the threaded RTS so that I could do some work while displaying a > progress bar - I know there are ways to do this without -threaded, > but I > thought -threaded would be the neatest way to do it, and eventually > we'll want to have background threads doing things anyway. > > Now I have weird problems - usually hangs/deadlocks, but > occasionally I > get an Xlib error. Here's my latest hang, which is 100% > reproducible on > exiting ThreadScope right now: > > (gdb) where > #0 0x0000003c0aa0af19 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00000000005b2c0e in waitCondition (pCond=0x1a43918, > pMut=0x1a43948) at posix/OSThreads.c:65 > #2 0x00000000005d72d9 in waitForReturnCapability > (pCap=0x7ffffe1499f0, > task=0x1a438e0) at Capability.c:445 > #3 0x00000000005a5934 in rts_lock () at RtsAPI.c:572 > #4 0x00000000004cf247 in SystemziGlibziGObject_d1wV () > #5 0x0000003c10cd8eed in ?? () from /usr/lib64/libgtk-x11-2.0.so.0 > #6 0x0000003c0ba0d6a8 in g_object_unref () from /lib64/ > libgobject-2.0.so.0 > .... various stuff omitted .... > #41 0x0000003c0ba0d632 in g_object_unref () from /lib64/ > libgobject-2.0.so.0 > #42 0x00000000005b20fd in runCFinalizer (fn=0x408968, ptr=0x1a69440, > env=0x0, flag=0) at Weak.c:34 > #43 0x00000000005b216a in runAllCFinalizers (list=0x2afbacb5f060) at > Weak.c:50 > #44 0x00000000005a8077 in hs_exit_ (wait_foreign=rtsFalse) at > RtsStartup.c:413 > #45 0x00000000005a81a8 in shutdownHaskellAndExit (n=0) at > RtsStartup.c:557 > #46 0x00000000005a4c6a in real_main () at Main.c:144 > #47 0x00000000005a4ca4 in main (argc=5, argv=0x7ffffe14b338) at > Main.c:156 > > So clearly g_object_unref is being called from a C finalizer, and it > ends up calling back into Haskell via something in System.Glib.Object, > this leads to a deadlock because the RTS is shutting down and can't > run > any Haskell code. > > Any help appreciated. In the meantime I'll probably go back to the > single-threaded RTS. > > Cheers, > Simon > > ---------------------------------------------------------------------- > -------- > Come build with us! The BlackBerry(R) Developer Conference in SF, CA > is the only developer event you need to attend this year. Jumpstart > your > developing skills, take BlackBerry mobile applications to market > and stay > ahead of the curve. Join us from November 9 - 12, 2009. Register now! > http://p.sf.net/sfu/devconference > _______________________________________________ > Gtk2hs-devel mailing list > Gtk...@li... > https://lists.sourceforge.net/lists/listinfo/gtk2hs-devel |
From: Simon M. <mar...@gm...> - 2009-11-03 10:20:23
|
On 02/11/2009 21:11, Axel Simon wrote: > The story is as follows: > > Calling g_object_unref, the principal finalizer we attach to any > ForeignPtr, will decrement the reference count and immediately finalize > an object if this counter is zero. This can pose problems in three ways: > > a) Storing any callback or other memory region in the GObject will > install a Haskell finalizer (freeStablePtr or free function pointer) > that is called when the GObject is finalized. After our last long > conversation about the new concurrent GC, we changed Gtk2Hs so that it > passes the addresses of the C functions of the ghc runtime as > finalizers, so Haskell is not called and all is well. (I hope I recall > this correctly.) Correct, as far as I recall. You have to ensure that Haskell can't be called from a C finalizer, or else use Haskell finalizers from Foreign.Concurrent. Which is why I was surprised that my backtrace showed that Haskell was being called from a C finalizer! > b) It is possible to add a Haskell finalizer function to a GObject. > Since these Haskell finalizers are called from the GC, they have to > re-enter the Haskell RTS which will bomb. Solution: remove the > possibility to add such a finalizer function. So is this currently done or not? Why do I see a Haskell function being called by a finalizer? > c) Xlib is not thread-safe but has no thread-local storage so it can be > called from several threads if they don't access Xlib at the same time. > Win32 API is not thread-safe *and* uses thread-local storage so that it > is impossible to call it from more than one thread. > > Since a GObject may contain X or Win32 ressources, these might be freed > from a thread calling g_object_unref. Thus, if the ghc GC calls > g_object_unref from several threads, it is inevitable that Xlib or Win32 > is called from several threads. But the GC never invokes finalizers from several threads! It is true that it doesn't give you any guarantee about *which* thread it will run finalizers on, but the finalizers themselves run sequentially. However, what may happen is that the GC runs at the same time as a (safe) foreign call, which may well cause the problems you describe. This is not new, but the fact that the GC now runs finalizers *is* new. More on this below... > This is what you observe below when get > an Xlib error. I have a patched version of Gtk2Hs lying around that > replaces the finalizer 'g_object_unref' with a wrapper function that > first acquires a global Gtk lock before calling g_object_unref. This may > severely stall the GC but works well for Xlib. On Windows, this still > doesn't work. Nor does it solve b). > > The only solution I can think of is to insert all GObjects that are > finalized during GC into a list and to finialize all these objects by > calling a function (say 'runFinalizers') from the main Gtk thread. This > function can be called from Haskell in a 'safe' way so that callbacks to > Haskell are no problem. > > The problem that remains is how to ensure that 'runFinalizers' is called > after each GC. When using Gtk+, I could add the 'runFinalizer' function > to the main loop of Gtk. Thus, when the GUI resumes after a GC, it will > finalize all objects that were dead during the last GC. > > However, if we ever split off Cairo or Pango (two modules that may be > used independently of the GUI stuff) then there is no more Gtk+ main > loop. Thus, we would need another way to run the 'runFinalizers' C > function. > > One idea I had to this end is a hook into the ghc runtime: We need to > install a C pointer to a function that is called at the end of each GC. > Additionally, we need to ensure that this function is called by one > specific OS thread. Alternatively, we would need a Haskell-side hook to > install a OS-thread bound Haskell function that, in turn, calls the C > function. I talked to Duncan about this but he was skeptical that GHC > headquarters would be happy about this. Calling a C function after GC would be quite easy to add. However, ensuring it gets run by a specific OS thread would be much harder - the OS thread you want to use might be in the middle of a foreign call, for example. > <rant> > In fact, the situation is a bit contradictory: As an application > developer you are responsible to carefully call all your functions from > an (OS-)bounded thread if your C library isn't thread-safe and then the > GC comes along and calls concurrently with any number of OS threads into > the C library to run the finalizers. This is not quite fair onto the > programmer. > <rant/> So while this is slightly incorrect (the GC doesn't run finalizers concurrently), I agree that there is a problem in that you can't effectively synchronise between C finalizers and other activity. I suggest that the C finalizers should be very simple: just add the tasks to be done onto a queue protected by a mutex, and have the main loop process the queue. The mutex is necessary not to protect against multiple finalizers running concurrently, but to protect against finalizers running concurrently with foreign calls (e.g. the gtk main loop). This sounds like what you proposed above, and in order to make this work the GC needs to inform the main loop when more finalizers have been added. I'm not sure of the best way to do this. I wonder if for the time being you should just revert to using Haskell finalizers. They are less efficient and prompt, but they do support proper synchronisation. > Another idea would be to have a new type of finalizers that takes a > thread id and that the GC then guarantees to only run from the given > thread id. This would push the burden into the GC implementation. > > Any thoughts appreciated. With the current solution, I don't think > there's a way for us to make Gtk2Hs thread-safe except for the hack of > using the Gtk+ main loop to run finalizers. Can I suggest that first of all there should be a warning on the gtk2hs page and in the documentation about the use of -threaded; if I'd known that it wasn't currently supported I would have avoided it. Secondly, I'm happy to help here. I think we ought to have multi-threaded GUI support, and it's worth putting in some effort to make it work. Let me know how you'd like to proceed. Cheers, Simon |
From: Axel S. <Axe...@en...> - 2009-11-04 18:31:32
|
Ok, I'll try to answer this :-) On Nov 3, 2009, at 11:20, Simon Marlow wrote: > On 02/11/2009 21:11, Axel Simon wrote: > >> The story is as follows: >> >> Calling g_object_unref, the principal finalizer we attach to any >> ForeignPtr, will decrement the reference count and immediately >> finalize >> an object if this counter is zero. This can pose problems in three >> ways: >> >> a) Storing any callback or other memory region in the GObject will >> install a Haskell finalizer (freeStablePtr or free function pointer) >> that is called when the GObject is finalized. After our last long >> conversation about the new concurrent GC, we changed Gtk2Hs so >> that it >> passes the addresses of the C functions of the ghc runtime as >> finalizers, so Haskell is not called and all is well. (I hope I >> recall >> this correctly.) > > Correct, as far as I recall. You have to ensure that Haskell can't > be called from a C finalizer, or else use Haskell finalizers from > Foreign.Concurrent. Which is why I was > surprised that my backtrace showed that Haskell was being called > from a C finalizer! I just realized that we haven't changed everything to use direct calls to C finalizers but to Haskell finalizers. I assume that this is because Haskell still got called from within the GC. If it is still called from within the GC, then some signals are being emitted as a result of finalizing an object. I probably need an example where this goes wrong. I don't know if it is possible to run the Haskell program in gdb and get a traceback of the C functions that called Haskell during GC. However, it might just be a very plain bug in that we bound a function as 'unsafe' when, in fact, it may trigger a callback to Haskell. > >> b) It is possible to add a Haskell finalizer function to a GObject. >> Since these Haskell finalizers are called from the GC, they have to >> re-enter the Haskell RTS which will bomb. Solution: remove the >> possibility to add such a finalizer function. > > So is this currently done or not? Why do I see a Haskell function > being called by a finalizer? Well, as I said, probably an 'unsafe' function that should be declared 'safe'. We still provide the possibility, though, to attach a function that is called when an object dies. I don't think many people use this, but I don't know if we can categorically avoid this situations: some GObject in Gtk+ may trigger some 'disconnect some child object' if a finalizer is run. I don't know of any specific cases, though. >> c) Xlib is not thread-safe but has no thread-local storage so it >> can be >> called from several threads if they don't access Xlib at the same >> time. >> Win32 API is not thread-safe *and* uses thread-local storage so >> that it >> is impossible to call it from more than one thread. >> >> Since a GObject may contain X or Win32 ressources, these might be >> freed >> from a thread calling g_object_unref. Thus, if the ghc GC calls >> g_object_unref from several threads, it is inevitable that Xlib or >> Win32 >> is called from several threads. > > But the GC never invokes finalizers from several threads! It is > true that it doesn't give you any guarantee about *which* thread it > will run finalizers on, but the finalizers themselves run > sequentially. Oh. Well, it wouldn't make things much worse if finalizers were run concurrently. I would have to wrap g_object_unref in a function that acquires a global lock, but that's all. Why is this not done? > However, what may happen is that the GC runs at the same time as a > (safe) foreign call, which may well cause the problems you > describe. This is not new, but the fact that the GC now runs > finalizers *is* new. More on this below... > >> This is what you observe below when get >> an Xlib error. I have a patched version of Gtk2Hs lying around that >> replaces the finalizer 'g_object_unref' with a wrapper function that >> first acquires a global Gtk lock before calling g_object_unref. >> This may >> severely stall the GC but works well for Xlib. On Windows, this still >> doesn't work. Nor does it solve b). >> >> The only solution I can think of is to insert all GObjects that are >> finalized during GC into a list and to finialize all these objects by >> calling a function (say 'runFinalizers') from the main Gtk thread. >> This >> function can be called from Haskell in a 'safe' way so that >> callbacks to >> Haskell are no problem. >> >> The problem that remains is how to ensure that 'runFinalizers' is >> called >> after each GC. When using Gtk+, I could add the 'runFinalizer' >> function >> to the main loop of Gtk. Thus, when the GUI resumes after a GC, it >> will >> finalize all objects that were dead during the last GC. >> >> However, if we ever split off Cairo or Pango (two modules that may be >> used independently of the GUI stuff) then there is no more Gtk+ main >> loop. Thus, we would need another way to run the 'runFinalizers' C >> function. >> >> One idea I had to this end is a hook into the ghc runtime: We need to >> install a C pointer to a function that is called at the end of >> each GC. >> Additionally, we need to ensure that this function is called by one >> specific OS thread. Alternatively, we would need a Haskell-side >> hook to >> install a OS-thread bound Haskell function that, in turn, calls the C >> function. I talked to Duncan about this but he was skeptical that GHC >> headquarters would be happy about this. > > Calling a C function after GC would be quite easy to add. However, > ensuring it gets run by a specific OS thread would be much harder - > the OS thread you want to use might be in the middle of a foreign > call, for example. > >> <rant> >> In fact, the situation is a bit contradictory: As an application >> developer you are responsible to carefully call all your functions >> from >> an (OS-)bounded thread if your C library isn't thread-safe and >> then the >> GC comes along and calls concurrently with any number of OS >> threads into >> the C library to run the finalizers. This is not quite fair onto the >> programmer. >> <rant/> > > So while this is slightly incorrect (the GC doesn't run finalizers > concurrently), I agree that there is a problem in that you can't > effectively synchronise between C finalizers and other activity. > > I suggest that the C finalizers should be very simple: just add the > tasks to be done onto a queue protected by a mutex, and have the > main loop process the queue. The mutex is necessary not to protect > against multiple finalizers running concurrently, but to protect > against finalizers running concurrently with foreign calls (e.g. > the gtk main loop). > > This sounds like what you proposed above, and in order to make this > work the GC needs to inform the main loop when more finalizers have > been added. I'm not sure of the best way to do this. I'm not quite following: who is in charge of this list? The GC? As it stands, I can already implement what you suggest above: I replace each call of the GC to g_object_unref with a function that enqueues the finaliasation task into the Gtk+ queue. As I said, the problems with this is that I cannot guarantee that the application program will every run the Gtk main loop. I can only guarantee that in real GUI programs, not in those that use Gtk2Hs to render some graphics into a file. Since the problems only really occur when objects with handles to Xlib or Win32 are finalized this would be a solution and is implementable without any change to ghc. I'm just weary that there is some object whose finalizer I do not enqueue in the main Gtk loop and which for some reason calls either Xlib/Win32 or Haskell. > I wonder if for the time being you should just revert to using > Haskell finalizers. They are less efficient and prompt, but they > do support proper synchronisation. It seems I'm using Haskell finalizer. So you might just have observed an unsafe/safe declaration bug in Gtk2Hs. However, the Xlib warning is the real thing that cannot so easily be worked around. >> Another idea would be to have a new type of finalizers that takes a >> thread id and that the GC then guarantees to only run from the given >> thread id. This would push the burden into the GC implementation. >> >> Any thoughts appreciated. With the current solution, I don't think >> there's a way for us to make Gtk2Hs thread-safe except for the >> hack of >> using the Gtk+ main loop to run finalizers. > > Can I suggest that first of all there should be a warning on the > gtk2hs page and in the documentation about the use of -threaded; if > I'd known that it wasn't currently supported I would have avoided it. Yes, I agree, we're not very upfront about this... > Secondly, I'm happy to help here. I think we ought to have multi- > threaded GUI support, and it's worth putting in some effort to make > it work. Let me know how you'd like to proceed. Could you say why it is so hard to associate a finalizer with a thread id and ensure that the finalizer is run from that particular thread? It seems easier to me now than before, when I though the GC would execute finalizers concurrently rather than enqueuing them in a list. It would be ok to associate a ForeignPtr with a Haskell thread id (which in turn is bound to the right OS thread). You could keep a list of finalizers on a per-thread-id basis and add a process that executes these finalizers into the running queue of that thread. With this approach, it wouldn't matter that the associated OS thread is currently running in C land. I'm probably naive about the complexity of this approach, but what I would implement in Gtk2Hs would boil down to something similar but be much more restricted. Maybe you could elaborate on your idea again. I'm pretty sure I do not understand the details of your proposal. Cheers, Axel. |
From: Simon M. <mar...@gm...> - 2009-11-10 13:14:08
|
On 04/11/2009 18:31, Axel Simon wrote: > I just realized that we haven't changed everything to use direct calls > to C finalizers but to Haskell finalizers. I assume that this is because > Haskell still got called from within the GC. If it is still called from > within the GC, then some signals are being emitted as a result of > finalizing an object. I probably need an example where this goes wrong. > I don't know if it is possible to run the Haskell program in gdb and get > a traceback of the C functions that called Haskell during GC. However, > it might just be a very plain bug in that we bound a function as > 'unsafe' when, in fact, it may trigger a callback to Haskell. Does the traceback in my original message give you enough information? (reproduced at the end of this message) > Oh. Well, it wouldn't make things much worse if finalizers were run > concurrently. I would have to wrap g_object_unref in a function that > acquires a global lock, but that's all. > Why is this not done? The GC only runs in parallel when there is a +RTS -N<n> option, so even if it were capable of running finalizers concurrently you wouldn't notice anything until you used +RTS -N<n>. But running in parallel is not the goal here: we want concurrency, not parallelism. The technical reason that we don't run finalizers concurrently is because in order to decide which finalizers can be run, we have to complete the GC traversal, which means all GC threads have stopped. Given that all GC threads have already stopped, it would be extra work to start them up again to run finalizers, so instead we just run the finalizers sequentially in the main GC thread. >> I suggest that the C finalizers should be very simple: just add the >> tasks to be done onto a queue protected by a mutex, and have the main >> loop process the queue. The mutex is necessary not to protect against >> multiple finalizers running concurrently, but to protect against >> finalizers running concurrently with foreign calls (e.g. the gtk main >> loop). >> >> This sounds like what you proposed above, and in order to make this >> work the GC needs to inform the main loop when more finalizers have >> been added. I'm not sure of the best way to do this. > > I'm not quite following: who is in charge of this list? The GC? No, this would be completely managed by gtk2hs and its finalizers. > As it stands, I can already implement what you suggest above: I replace > each call of the GC to g_object_unref with a function that enqueues the > finaliasation task into the Gtk+ queue. Right. Or if that's too inefficient, use a separate queue and add something to the Gtk+ main loop that runs the items in that queue. > As I said, the problems with this is that I cannot guarantee that the > application program will every run the Gtk main loop. I can only > guarantee that in real GUI programs, not in those that use Gtk2Hs to > render some graphics into a file. Since the problems only really occur > when objects with handles to Xlib or Win32 are finalized this would be a > solution and is implementable without any change to ghc. I'm just weary > that there is some object whose finalizer I do not enqueue in the main > Gtk loop and which for some reason calls either Xlib/Win32 or Haskell. In any program that uses gtk2hs/cairo/whatever there must be a single distinguished thread that is allowed to perform GUI operations, correct? So the task is just to arrange somehow that it is possible to communicate with that thread that it has some finalization actions to perform. I'm not sure how you do that, but I'm not really familiar with the structure of these kinds of applications. >> I wonder if for the time being you should just revert to using Haskell >> finalizers. They are less efficient and prompt, but they do support >> proper synchronisation. > > It seems I'm using Haskell finalizer. So you might just have observed an > unsafe/safe declaration bug in Gtk2Hs. However, the Xlib warning is the > real thing that cannot so easily be worked around. If you consistently use Haskell finalizers, and the finalizers always use idleAdd, then you can avoid all these problems for now, at the expense of some performance. But there is evidence that gtk2hs is not using Haskell finalizers, or at least not consistently: see the gdb backtrace below. >> Can I suggest that first of all there should be a warning on the >> gtk2hs page and in the documentation about the use of -threaded; if >> I'd known that it wasn't currently supported I would have avoided it. > > Yes, I agree, we're not very upfront about this... Thanks for the blog post! >> Secondly, I'm happy to help here. I think we ought to have >> multi-threaded GUI support, and it's worth putting in some effort to >> make it work. Let me know how you'd like to proceed. > > Could you say why it is so hard to associate a finalizer with a thread > id and ensure that the finalizer is run from that particular thread? Suppose that hypothetically it was possible to say that a finalizer must be run by the main thread. How should the runtime arrange to do it? The main thread is currently inside a call to the Gtk main loop, and the only way to get it out is to do idleAdd (or similar), but that is specific to gtk2hs; there's no general way to do this, so there's no general mechanism I can add to the RTS. Perhaps you want to create a Haskell thread to run the finalizer, but keep track of the fact that this Haskell thread can only be run by the main OS thread. There might be no fundamental problem with this scheme, but it would require deep restructuring of the RTS to make it possible: currently you can associate a Haskell thread with an OS thread one-to-one, but this would make a many-to-one mapping. And I'm not sure it's exactly what you want, because you still have to arrange that the gtk main loop returns to Haskell occasionally, so you're not really getting the benefit of -threaded; it's better to just use idleAdd in the first place, which requires no extra support in the RTS, and does essentially the same thing. > It > seems easier to me now than before, when I though the GC would execute > finalizers concurrently rather than enqueuing them in a list. It would > be ok to associate a ForeignPtr with a Haskell thread id (which in turn > is bound to the right OS thread). You could keep a list of finalizers on > a per-thread-id basis and add a process that executes these finalizers > into the running queue of that thread. With this approach, it wouldn't > matter that the associated OS thread is currently running in C land. > > I'm probably naive about the complexity of this approach, but what I > would implement in Gtk2Hs would boil down to something similar but be > much more restricted. > > Maybe you could elaborate on your idea again. I'm pretty sure I do not > understand the details of your proposal. Hopefully I've explained it well enough, if not please let me know! Cheers, Simon (gdb) where #0 0x0000003c0aa0af19 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00000000005b2c0e in waitCondition (pCond=0x1a43918, pMut=0x1a43948) at posix/OSThreads.c:65 #2 0x00000000005d72d9 in waitForReturnCapability (pCap=0x7ffffe1499f0, task=0x1a438e0) at Capability.c:445 #3 0x00000000005a5934 in rts_lock () at RtsAPI.c:572 #4 0x00000000004cf247 in SystemziGlibziGObject_d1wV () #5 0x0000003c10cd8eed in ?? () from /usr/lib64/libgtk-x11-2.0.so.0 #6 0x0000003c0ba0d6a8 in g_object_unref () from /lib64/libgobject-2.0.so.0 .... various stuff omitted .... #41 0x0000003c0ba0d632 in g_object_unref () from /lib64/libgobject-2.0.so.0 #42 0x00000000005b20fd in runCFinalizer (fn=0x408968, ptr=0x1a69440, env=0x0, flag=0) at Weak.c:34 #43 0x00000000005b216a in runAllCFinalizers (list=0x2afbacb5f060) at Weak.c:50 #44 0x00000000005a8077 in hs_exit_ (wait_foreign=rtsFalse) at RtsStartup.c:413 #45 0x00000000005a81a8 in shutdownHaskellAndExit (n=0) at RtsStartup.c:557 #46 0x00000000005a4c6a in real_main () at Main.c:144 #47 0x00000000005a4ca4 in main (argc=5, argv=0x7ffffe14b338) at Main.c:156 |
From: Axel S. <Axe...@en...> - 2009-11-10 13:53:14
|
On Tue, 2009-11-10 at 13:13 +0000, Simon Marlow wrote: > On 04/11/2009 18:31, Axel Simon wrote: > > > I just realized that we haven't changed everything to use direct calls > > to C finalizers but to Haskell finalizers. I assume that this is because > > Haskell still got called from within the GC. If it is still called from > > within the GC, then some signals are being emitted as a result of > > finalizing an object. I probably need an example where this goes wrong. > > I don't know if it is possible to run the Haskell program in gdb and get > > a traceback of the C functions that called Haskell during GC. However, > > it might just be a very plain bug in that we bound a function as > > 'unsafe' when, in fact, it may trigger a callback to Haskell. > > Does the traceback in my original message give you enough information? > (reproduced at the end of this message) Well, you omitted 'various' stuff. I agree that something is calling back into C when it shouldn't. Are there any function names between frame 6 and 44 that would give me a clue where we've been negligent? [..] > >> I suggest that the C finalizers should be very simple: just add the > >> tasks to be done onto a queue protected by a mutex, and have the main > >> loop process the queue. The mutex is necessary not to protect against > >> multiple finalizers running concurrently, but to protect against > >> finalizers running concurrently with foreign calls (e.g. the gtk main > >> loop). > >> > >> This sounds like what you proposed above, and in order to make this > >> work the GC needs to inform the main loop when more finalizers have > >> been added. I'm not sure of the best way to do this. > > > > I'm not quite following: who is in charge of this list? The GC? > > No, this would be completely managed by gtk2hs and its finalizers. Ok, I can keep a queue in C land for this, no prob. [..] > > As I said, the problems with this is that I cannot guarantee that the > > application program will every run the Gtk main loop. I can only > > guarantee that in real GUI programs, not in those that use Gtk2Hs to > > render some graphics into a file. Since the problems only really occur > > when objects with handles to Xlib or Win32 are finalized this would be a > > solution and is implementable without any change to ghc. I'm just weary > > that there is some object whose finalizer I do not enqueue in the main > > Gtk loop and which for some reason calls either Xlib/Win32 or Haskell. > > In any program that uses gtk2hs/cairo/whatever there must be a single > distinguished thread that is allowed to perform GUI operations, correct? No, unfortunately not. Once we've cabalised the parts of Gtk2Hs, it is perfectly reasonable to build programs without GUI, say using Cairo to create PDFs. However, Cairo (and other non-GUI sub-libaries in Gtk2Hs) can cope with other threads calling g_object_unref. Now, g_object_unref is thread-safe and hence, the current way of directly caling g_object_unref from the GC is perfectly fine. > So the task is just to arrange somehow that it is possible to > communicate with that thread that it has some finalization actions to > perform. I'm not sure how you do that, but I'm not really familiar with > the structure of these kinds of applications. Yes, for the GUI part of Gtk2Hs, this is the only way to go. I think what I would like is a hook in the RTS of ghc that allows me to run a C function each time the GC has finished. This C function can be run from any OS thread. I would make this function call g_idle_add (which is thread-safe) to add a function that empties the list of finalizers that I've accumulated during GC. If the main Gtk+ loop is currently running or will run at some point in the future, the finalizers form the list will be executed. If the Gtk+ is never run again then I probably don't care anyway that the finalizers are not run since the program is stuck or about to end. > >> I wonder if for the time being you should just revert to using Haskell > >> finalizers. They are less efficient and prompt, but they do support > >> proper synchronisation. > > > > It seems I'm using Haskell finalizer. So you might just have observed an > > unsafe/safe declaration bug in Gtk2Hs. However, the Xlib warning is the > > real thing that cannot so easily be worked around. > > If you consistently use Haskell finalizers, and the finalizers always > use idleAdd, then you can avoid all these problems for now, at the > expense of some performance. > > But there is evidence that gtk2hs is not using Haskell finalizers, or at > least not consistently: see the gdb backtrace below. We certainly do not call g_idle_add to run the finalizer the objects. We do it immeditely, thereby triggering the occasional callback to Haskell or a call to Xlib/Win32 which is likely to crash the program. We can fix this using the queue/list approach. > >> Can I suggest that first of all there should be a warning on the > >> gtk2hs page and in the documentation about the use of -threaded; if > >> I'd known that it wasn't currently supported I would have avoided it. > > > > Yes, I agree, we're not very upfront about this... > > Thanks for the blog post! > > >> Secondly, I'm happy to help here. I think we ought to have > >> multi-threaded GUI support, and it's worth putting in some effort to > >> make it work. Let me know how you'd like to proceed. > > > > Could you say why it is so hard to associate a finalizer with a thread > > id and ensure that the finalizer is run from that particular thread? > > Suppose that hypothetically it was possible to say that a finalizer must > be run by the main thread. How should the runtime arrange to do it? > The main thread is currently inside a call to the Gtk main loop, and the > only way to get it out is to do idleAdd (or similar), but that is > specific to gtk2hs; there's no general way to do this, so there's no > general mechanism I can add to the RTS. > > Perhaps you want to create a Haskell thread to run the finalizer, but > keep track of the fact that this Haskell thread can only be run by the > main OS thread. There might be no fundamental problem with this scheme, > but it would require deep restructuring of the RTS to make it possible: > currently you can associate a Haskell thread with an OS thread > one-to-one, but this would make a many-to-one mapping. And I'm not sure > it's exactly what you want, because you still have to arrange that the > gtk main loop returns to Haskell occasionally, so you're not really > getting the benefit of -threaded; it's better to just use idleAdd in the > first place, which requires no extra support in the RTS, and does > essentially the same thing. Oh, I think I understand our different perception of what we want. The main loop of Gtk+ will always return at some point. Thus, all I was asking is that the Haskell GC should keep a per-OS-thread list of finalizers to run and *only* if the GHC runtime has this OS thread avaiable to itself, it would run the associated finalizers. Thus, I thought adding thread-bound finalizers would be as easy as adding a function to the running queue of the Haskell thread that is associated with the OS thread of the finalizer. So I can see two ways forward: (1) a hook in GHC to install a C function that is run by the RTS after each GC. The RTS may call this function from any thread. (2) have a new Finalizer type that takes an OS thread id. If this finalizer is due to be run, the GC enqueues it into running queue of the Haskell thread that is bound to the OS thread. The finalizer then has to be called using a 'safe' call, so that the finalizer may call back into Haskell land. (1) is easier for you, (2) for me. If there's no other libraries that have these kind of threading problems, then maybe (1) is enough. Let me know if I was clear/correct. Cheers, Axel. |
From: Simon M. <mar...@gm...> - 2009-11-10 14:14:38
|
On 10/11/2009 13:52, Axel Simon wrote: > On Tue, 2009-11-10 at 13:13 +0000, Simon Marlow wrote: >> On 04/11/2009 18:31, Axel Simon wrote: >> >>> I just realized that we haven't changed everything to use direct calls >>> to C finalizers but to Haskell finalizers. I assume that this is because >>> Haskell still got called from within the GC. If it is still called from >>> within the GC, then some signals are being emitted as a result of >>> finalizing an object. I probably need an example where this goes wrong. >>> I don't know if it is possible to run the Haskell program in gdb and get >>> a traceback of the C functions that called Haskell during GC. However, >>> it might just be a very plain bug in that we bound a function as >>> 'unsafe' when, in fact, it may trigger a callback to Haskell. >> >> Does the traceback in my original message give you enough information? >> (reproduced at the end of this message) > > Well, you omitted 'various' stuff. I agree that something is calling > back into C when it shouldn't. Are there any function names between > frame 6 and 44 that would give me a clue where we've been negligent? Ah yes, sorry. Here's the full trace: (gdb) where #0 0x0000003c0aa0af19 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00000000005ac6e9 in waitCondition () #2 0x00000000005c67ab in waitForReturnCapability () #3 0x00000000005a3f17 in rts_lock () #4 0x00000000004ce397 in SystemziGlibziGObject_d1wV () #5 0x0000003c10cd8eed in ?? () from /usr/lib64/libgtk-x11-2.0.so.0 #6 0x0000003c0ba0d6a8 in g_object_unref () from /lib64/libgobject-2.0.so.0 #7 0x0000003c10ccebc7 in gtk_tree_view_remove_column () from /usr/lib64/libgtk-x11-2.0.so.0 #8 0x0000003c10cb749c in ?? () from /usr/lib64/libgtk-x11-2.0.so.0 #9 0x0000003c0ba0b62f in g_closure_invoke () from /lib64/libgobject-2.0.so.0 #10 0x0000003c0ba1faa3 in ?? () from /lib64/libgobject-2.0.so.0 #11 0x0000003c0ba20b9c in g_signal_emit_valist () from /lib64/libgobject-2.0.so.0 #12 0x0000003c0ba210e3 in g_signal_emit () from /lib64/libgobject-2.0.so.0 #13 0x0000003c10bb426a in ?? () from /usr/lib64/libgtk-x11-2.0.so.0 #14 0x0000003c10cfa6c3 in ?? () from /usr/lib64/libgtk-x11-2.0.so.0 #15 0x0000003c0ba0dad0 in g_object_run_dispose () from /lib64/libgobject-2.0.so.0 #16 0x0000003c10bb421a in gtk_object_destroy () from /usr/lib64/libgtk-x11-2.0.so.0 #17 0x0000003c10cf1be9 in gtk_widget_destroy () from /usr/lib64/libgtk-x11-2.0.so.0 #18 0x0000003c10a7a8b8 in ?? () from /usr/lib64/libgtk-x11-2.0.so.0 #19 0x0000003c10bfee1c in ?? () from /usr/lib64/libgtk-x11-2.0.so.0 #20 0x0000003c10ad0aa9 in gtk_container_foreach () from /usr/lib64/libgtk-x11-2.0.so.0 #21 0x0000003c10acf9af in ?? () from /usr/lib64/libgtk-x11-2.0.so.0 #22 0x0000003c10bfe5c2 in ?? () from /usr/lib64/libgtk-x11-2.0.so.0 #23 0x0000003c0ba0b62f in g_closure_invoke () from /lib64/libgobject-2.0.so.0 #24 0x0000003c0ba1faa3 in ?? () from /lib64/libgobject-2.0.so.0 #25 0x0000003c0ba20b9c in g_signal_emit_valist () from /lib64/libgobject-2.0.so.0 #26 0x0000003c0ba210e3 in g_signal_emit () from /lib64/libgobject-2.0.so.0 #27 0x0000003c10bb426a in ?? () from /usr/lib64/libgtk-x11-2.0.so.0 #28 0x0000003c10cfa6c3 in ?? () from /usr/lib64/libgtk-x11-2.0.so.0 #29 0x0000003c0ba0dad0 in g_object_run_dispose () from /lib64/libgobject-2.0.so.0 #30 0x0000003c10bb421a in gtk_object_destroy () from /usr/lib64/libgtk-x11-2.0.so.0 #31 0x0000003c10cf1be9 in gtk_widget_destroy () from /usr/lib64/libgtk-x11-2.0.so.0 #32 0x0000003c10a7f340 in ?? () from /usr/lib64/libgtk-x11-2.0.so.0 #33 0x0000003c10ad0aa9 in gtk_container_foreach () from /usr/lib64/libgtk-x11-2.0.so.0 #34 0x0000003c10acf9af in ?? () from /usr/lib64/libgtk-x11-2.0.so.0 #35 0x0000003c0ba0b6dd in g_closure_invoke () from /lib64/libgobject-2.0.so.0 #36 0x0000003c0ba1faa3 in ?? () from /lib64/libgobject-2.0.so.0 #37 0x0000003c0ba20b9c in g_signal_emit_valist () from /lib64/libgobject-2.0.so.0 #38 0x0000003c0ba210e3 in g_signal_emit () from /lib64/libgobject-2.0.so.0 #39 0x0000003c10bb426a in ?? () from /usr/lib64/libgtk-x11-2.0.so.0 #40 0x0000003c10cfa6c3 in ?? () from /usr/lib64/libgtk-x11-2.0.so.0 #41 0x0000003c0ba0d632 in g_object_unref () from /lib64/libgobject-2.0.so.0 #42 0x00000000005abdfd in runAllCFinalizers () #43 0x00000000005a5ca8 in hs_exit_ () #44 0x00000000005a5e1a in shutdownHaskellAndExit () #45 0x00000000005a3cd8 in real_main () #46 0x00000000005a3d18 in main () >>> As I said, the problems with this is that I cannot guarantee that the >>> application program will every run the Gtk main loop. I can only >>> guarantee that in real GUI programs, not in those that use Gtk2Hs to >>> render some graphics into a file. Since the problems only really occur >>> when objects with handles to Xlib or Win32 are finalized this would be a >>> solution and is implementable without any change to ghc. I'm just weary >>> that there is some object whose finalizer I do not enqueue in the main >>> Gtk loop and which for some reason calls either Xlib/Win32 or Haskell. >> >> In any program that uses gtk2hs/cairo/whatever there must be a single >> distinguished thread that is allowed to perform GUI operations, correct? > > No, unfortunately not. Once we've cabalised the parts of Gtk2Hs, it is > perfectly reasonable to build programs without GUI, say using Cairo to > create PDFs. However, Cairo (and other non-GUI sub-libaries in Gtk2Hs) > can cope with other threads calling g_object_unref. Now, g_object_unref > is thread-safe and hence, the current way of directly caling > g_object_unref from the GC is perfectly fine. Ok I see - as long as g_object_unref doesn't end up calling into Haskell eventually. >> So the task is just to arrange somehow that it is possible to >> communicate with that thread that it has some finalization actions to >> perform. I'm not sure how you do that, but I'm not really familiar with >> the structure of these kinds of applications. > > Yes, for the GUI part of Gtk2Hs, this is the only way to go. I think > what I would like is a hook in the RTS of ghc that allows me to run a C > function each time the GC has finished. This C function can be run from > any OS thread. I would make this function call g_idle_add (which is > thread-safe) to add a function that empties the list of finalizers that > I've accumulated during GC. If the main Gtk+ loop is currently running > or will run at some point in the future, the finalizers form the list > will be executed. If the Gtk+ is never run again then I probably don't > care anyway that the finalizers are not run since the program is stuck > or about to end. Could you not do this without a hook, by adding a call to g_idle_add to each finalizer? Perhaps with some intelligence so you don't add multiple g_idle_adds if one is already pending. Calling it once per GC might be too much: there are lots of GCs. >>>> I wonder if for the time being you should just revert to using Haskell >>>> finalizers. They are less efficient and prompt, but they do support >>>> proper synchronisation. >>> >>> It seems I'm using Haskell finalizer. So you might just have observed an >>> unsafe/safe declaration bug in Gtk2Hs. However, the Xlib warning is the >>> real thing that cannot so easily be worked around. >> >> If you consistently use Haskell finalizers, and the finalizers always >> use idleAdd, then you can avoid all these problems for now, at the >> expense of some performance. >> >> But there is evidence that gtk2hs is not using Haskell finalizers, or at >> least not consistently: see the gdb backtrace below. > > We certainly do not call g_idle_add to run the finalizer the objects. We > do it immeditely, thereby triggering the occasional callback to Haskell > or a call to Xlib/Win32 which is likely to crash the program. We can fix > this using the queue/list approach. Ok, then I misunderstood what you were saying above. I'm suggesting that you could use Haskell finalizers everywhere, as a way to work around the current problems for the time being, meanwhile work on the queue/list approach as a long-term solution. > Oh, I think I understand our different perception of what we want. The > main loop of Gtk+ will always return at some point. Thus, all I was > asking is that the Haskell GC should keep a per-OS-thread list of > finalizers to run and *only* if the GHC runtime has this OS thread > avaiable to itself, it would run the associated finalizers. > > Thus, I thought adding thread-bound finalizers would be as easy as > adding a function to the running queue of the Haskell thread that is > associated with the OS thread of the finalizer. > > So I can see two ways forward: > > (1) a hook in GHC to install a C function that is run by the RTS after > each GC. The RTS may call this function from any thread. Or have each finalizer call g_idle_add after enqueing the item. > (2) have a new Finalizer type that takes an OS thread id. If this > finalizer is due to be run, the GC enqueues it into running queue of the > Haskell thread that is bound to the OS thread. The finalizer then has to > be called using a 'safe' call, so that the finalizer may call back into > Haskell land. Hmm, there isn't such a thing as a "running queue of the Haskell thread", a Haskell thread is just a stack and a bit of state. A run queue is a list of Haskell threads, and there is one run queue per HEC (Haskell Execution Context, ie. a virtual CPU). My argument is that if you're going to do this, you might as well just use Haskell finalizers and have each one be a call to idleAdd, because the end result is the same and it doesn't require any changes to the RTS. > (1) is easier for you, (2) for me. If there's no other libraries that > have these kind of threading problems, then maybe (1) is enough. and (3) use Haskell finalizers with idleAdd as a temporary workaround. (3) is by far the easiest solution, but it may have unacceptable performance, I'd interested to know if that's the case. The runtime does batch multiple finalizers so they don't each get a separate Haskell thread, so it might not be too bad. I'm not completely averse to doing (1) if it's absolutely necessary, but I'd like to do something that avoids having a dependency on a future version of GHC. And all things being equal, we should avoid adding new features to GHC because they're hard to remove again. Cheers, Simon |
From: Axel S. <Axe...@en...> - 2009-11-10 14:24:04
|
Simon, I'll look into the callback of the treeview_ function! Thanks for the stacktrace. On Tue, 2009-11-10 at 14:14 +0000, Simon Marlow wrote: > > Yes, for the GUI part of Gtk2Hs, this is the only way to go. I think > > what I would like is a hook in the RTS of ghc that allows me to run > a C > > function each time the GC has finished. This C function can be run > from > > any OS thread. I would make this function call g_idle_add (which is > > thread-safe) to add a function that empties the list of finalizers > that > > I've accumulated during GC. If the main Gtk+ loop is currently > running > > or will run at some point in the future, the finalizers form the > list > > will be executed. If the Gtk+ is never run again then I probably > don't > > care anyway that the finalizers are not run since the program is > stuck > > or about to end. > > Could you not do this without a hook, by adding a call to g_idle_add > to > each finalizer? Perhaps with some intelligence so you don't add > multiple g_idle_adds if one is already pending. > > Calling it once per GC might be too much: there are lots of GCs. > Well, I admit, I didn't realize that the C function hook would only add the finalizers to the main Gtk+ loop rather than running them. So I guess, yes, I could implement the above and get the same effect. I'll ponder a bit more about this and see if there is a case where this doesn't work. I agree that it would be nice to get the -threaded flag working for future Gtk2Hs versions and for older GHCs. Cheers, Axel. |
From: Axel S. <Axe...@en...> - 2009-11-22 19:59:27
|
Dear Simon, Duncan, -threaded now works with Gtk2Hs. See below. On Tue, 2009-11-10 at 14:14 +0000, Simon Marlow wrote: > On 10/11/2009 13:52, Axel Simon wrote: > >> Does the traceback in my original message give you enough information? > >> (reproduced at the end of this message) > > > > Well, you omitted 'various' stuff. I agree that something is calling > > back into C when it shouldn't. Are there any function names between > > frame 6 and 44 that would give me a clue where we've been negligent? > > Ah yes, sorry. Here's the full trace: > > (gdb) where > #0 0x0000003c0aa0af19 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00000000005ac6e9 in waitCondition () > #2 0x00000000005c67ab in waitForReturnCapability () > #3 0x00000000005a3f17 in rts_lock () > #4 0x00000000004ce397 in SystemziGlibziGObject_d1wV () > #5 0x0000003c10cd8eed in ?? () from /usr/lib64/libgtk-x11-2.0.so.0 > #6 0x0000003c0ba0d6a8 in g_object_unref () from /lib64/libgobject-2.0.so.0 > #7 0x0000003c10ccebc7 in gtk_tree_view_remove_column () from I understand now where this error is. There is a call in your program to cellLayoutSetAttributes. This function installs certain callbacks that Gtk+ tries to free in the stack trace above. The call stack above is triggered by > #41 0x0000003c0ba0d632 in g_object_unref () from /lib64/libgobject-2.0.so.0 > #42 0x00000000005abdfd in runAllCFinalizers () ... and I suppose runAllCFinalizers is the GC and an 'unsafe' call from Haskell. Thus, it is actually only possible to use the C function freeHaskellPtr. I do that now. > (3) use Haskell finalizers with idleAdd as a temporary workaround. > > (3) is by far the easiest solution, but it may have unacceptable > performance, I'd interested to know if that's the case. The runtime > does batch multiple finalizers so they don't each get a separate Haskell > thread, so it might not be too bad. > > I'm not completely averse to doing (1) if it's absolutely necessary, but > I'd like to do something that avoids having a dependency on a future > version of GHC. And all things being equal, we should avoid adding new > features to GHC because they're hard to remove again. So now there are two kinds of finalizers: (1) those that directly free the C GObject which are used in Cairo and (in the future) Pango, Pixbuf, and all other libraries whose objects do not call back into Haskell. (2) all finalizers of Gtk+ and libraries that build upon Gtk+ (sourceview, etc). These call a special finalizer that enqueues the object to be finalized into an array and install a an g_idle_add handler that will run the finalizers from the main iteration loop of Gtk+ and, thus, from the right thread and from a safe call. All objects in (2) can therefore call back to Haskell while they are being destroyed. I have therefore moved the possibility to add a WeakRef (a function that is called when a GObject is finalized) to from GObject to Graphics.UI.Gtk.Abstract.Object which means they can only be installed on objects that fall into category (2). So the darcs version of Gtk2Hs with -threaded should now work as expected. You're still responsible to call Gtk2Hs function from the same Haskell thread. But you can create Cairo drawings and Pagno stuff in different threads and pass them to the main thread. Cheers, Axel. |
From: Simon M. <mar...@gm...> - 2009-11-23 11:02:19
|
Axel, this is great news, many thanks for working on it! I'm interested in using Cairo from multiple threads. What exactly is supported there? Can I run an arbitrary Render operation on any thread I like? Cheers, Simon On 22/11/09 19:58, Axel Simon wrote: > Dear Simon, Duncan, > > -threaded now works with Gtk2Hs. See below. > > > On Tue, 2009-11-10 at 14:14 +0000, Simon Marlow wrote: >> On 10/11/2009 13:52, Axel Simon wrote: > >>>> Does the traceback in my original message give you enough information? >>>> (reproduced at the end of this message) >>> >>> Well, you omitted 'various' stuff. I agree that something is calling >>> back into C when it shouldn't. Are there any function names between >>> frame 6 and 44 that would give me a clue where we've been negligent? >> >> Ah yes, sorry. Here's the full trace: >> >> (gdb) where >> #0 0x0000003c0aa0af19 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00000000005ac6e9 in waitCondition () >> #2 0x00000000005c67ab in waitForReturnCapability () >> #3 0x00000000005a3f17 in rts_lock () >> #4 0x00000000004ce397 in SystemziGlibziGObject_d1wV () >> #5 0x0000003c10cd8eed in ?? () from /usr/lib64/libgtk-x11-2.0.so.0 >> #6 0x0000003c0ba0d6a8 in g_object_unref () from /lib64/libgobject-2.0.so.0 >> #7 0x0000003c10ccebc7 in gtk_tree_view_remove_column () from > > I understand now where this error is. There is a call in your program to > cellLayoutSetAttributes. This function installs certain callbacks that > Gtk+ tries to free in the stack trace above. The call stack above is > triggered by > >> #41 0x0000003c0ba0d632 in g_object_unref () > from /lib64/libgobject-2.0.so.0 >> #42 0x00000000005abdfd in runAllCFinalizers () > > ... and I suppose runAllCFinalizers is the GC and an 'unsafe' call from > Haskell. Thus, it is actually only possible to use the C function > freeHaskellPtr. I do that now. > > >> (3) use Haskell finalizers with idleAdd as a temporary workaround. >> >> (3) is by far the easiest solution, but it may have unacceptable >> performance, I'd interested to know if that's the case. The runtime >> does batch multiple finalizers so they don't each get a separate Haskell >> thread, so it might not be too bad. >> >> I'm not completely averse to doing (1) if it's absolutely necessary, but >> I'd like to do something that avoids having a dependency on a future >> version of GHC. And all things being equal, we should avoid adding new >> features to GHC because they're hard to remove again. > > So now there are two kinds of finalizers: > > (1) those that directly free the C GObject which are used in Cairo and > (in the future) Pango, Pixbuf, and all other libraries whose objects do > not call back into Haskell. > > (2) all finalizers of Gtk+ and libraries that build upon Gtk+ > (sourceview, etc). These call a special finalizer that enqueues the > object to be finalized into an array and install a an g_idle_add handler > that will run the finalizers from the main iteration loop of Gtk+ and, > thus, from the right thread and from a safe call. > > All objects in (2) can therefore call back to Haskell while they are > being destroyed. I have therefore moved the possibility to add a WeakRef > (a function that is called when a GObject is finalized) to from GObject > to Graphics.UI.Gtk.Abstract.Object which means they can only be > installed on objects that fall into category (2). > > So the darcs version of Gtk2Hs with -threaded should now work as > expected. You're still responsible to call Gtk2Hs function from the same > Haskell thread. But you can create Cairo drawings and Pagno stuff in > different threads and pass them to the main thread. > > Cheers, > Axel. > > |
From: Axel S. <Axe...@en...> - 2009-11-23 12:47:15
|
On Mon, 2009-11-23 at 11:01 +0000, Simon Marlow wrote: > Axel, this is great news, many thanks for working on it! > > I'm interested in using Cairo from multiple threads. What exactly is > supported there? Can I run an arbitrary Render operation on any thread > I like? Yes, as long as you manipulate each Cairo object within one thread: http://lists.cairographics.org/archives/cairo/2009-February/016648.html However, it involves some effort to repaint a widget using a different thread even through you never directly paint to the screen due to Gtk's double buffering. The reason is that the expose event EExpose internally sets a Pixmap aside before the signal is emitted and will copy this Pixmap to the actual window once the signal handler has returned. Thus, you cannot spawn a thread off in the EExpose handler since it would have to finish before the handler can return. So I think you need to pro-actively update a Pixmap using Cairo whenever the content should changes. You do this in its own thread using renderWithDrawable. Whenever the the new Pixmap is ready, you call windowInvalidateRectangle on the DrawWindow of the DrawingArea. This will trigger the expose handler. In the expose handler of the DrawingArea, you retrieve the Pixmap from the worker thread and you copy it to the screen using Drawable.drawDrawable. You should probably turn off double buffering using widgetSetDoubleBuffered since this just means the pixmap is copied twice. In order to avoid that you show half-ready images, your worker thread could keep two Pixmaps: one for drawing, the other one containing the last complete image. I've never tried this, but this is how I understand the documentation. Cheers, Axel. > Cheers, > Simon > > On 22/11/09 19:58, Axel Simon wrote: > > Dear Simon, Duncan, > > > > -threaded now works with Gtk2Hs. See below. > > > > > > On Tue, 2009-11-10 at 14:14 +0000, Simon Marlow wrote: > >> On 10/11/2009 13:52, Axel Simon wrote: > > > >>>> Does the traceback in my original message give you enough information? > >>>> (reproduced at the end of this message) > >>> > >>> Well, you omitted 'various' stuff. I agree that something is calling > >>> back into C when it shouldn't. Are there any function names between > >>> frame 6 and 44 that would give me a clue where we've been negligent? > >> > >> Ah yes, sorry. Here's the full trace: > >> > >> (gdb) where > >> #0 0x0000003c0aa0af19 in pthread_cond_wait@@GLIBC_2.3.2 () from > >> /lib64/libpthread.so.0 > >> #1 0x00000000005ac6e9 in waitCondition () > >> #2 0x00000000005c67ab in waitForReturnCapability () > >> #3 0x00000000005a3f17 in rts_lock () > >> #4 0x00000000004ce397 in SystemziGlibziGObject_d1wV () > >> #5 0x0000003c10cd8eed in ?? () from /usr/lib64/libgtk-x11-2.0.so.0 > >> #6 0x0000003c0ba0d6a8 in g_object_unref () from /lib64/libgobject-2.0.so.0 > >> #7 0x0000003c10ccebc7 in gtk_tree_view_remove_column () from > > > > I understand now where this error is. There is a call in your program to > > cellLayoutSetAttributes. This function installs certain callbacks that > > Gtk+ tries to free in the stack trace above. The call stack above is > > triggered by > > > >> #41 0x0000003c0ba0d632 in g_object_unref () > > from /lib64/libgobject-2.0.so.0 > >> #42 0x00000000005abdfd in runAllCFinalizers () > > > > ... and I suppose runAllCFinalizers is the GC and an 'unsafe' call from > > Haskell. Thus, it is actually only possible to use the C function > > freeHaskellPtr. I do that now. > > > > > >> (3) use Haskell finalizers with idleAdd as a temporary workaround. > >> > >> (3) is by far the easiest solution, but it may have unacceptable > >> performance, I'd interested to know if that's the case. The runtime > >> does batch multiple finalizers so they don't each get a separate Haskell > >> thread, so it might not be too bad. > >> > >> I'm not completely averse to doing (1) if it's absolutely necessary, but > >> I'd like to do something that avoids having a dependency on a future > >> version of GHC. And all things being equal, we should avoid adding new > >> features to GHC because they're hard to remove again. > > > > So now there are two kinds of finalizers: > > > > (1) those that directly free the C GObject which are used in Cairo and > > (in the future) Pango, Pixbuf, and all other libraries whose objects do > > not call back into Haskell. > > > > (2) all finalizers of Gtk+ and libraries that build upon Gtk+ > > (sourceview, etc). These call a special finalizer that enqueues the > > object to be finalized into an array and install a an g_idle_add handler > > that will run the finalizers from the main iteration loop of Gtk+ and, > > thus, from the right thread and from a safe call. > > > > All objects in (2) can therefore call back to Haskell while they are > > being destroyed. I have therefore moved the possibility to add a WeakRef > > (a function that is called when a GObject is finalized) to from GObject > > to Graphics.UI.Gtk.Abstract.Object which means they can only be > > installed on objects that fall into category (2). > > > > So the darcs version of Gtk2Hs with -threaded should now work as > > expected. You're still responsible to call Gtk2Hs function from the same > > Haskell thread. But you can create Cairo drawings and Pagno stuff in > > different threads and pass them to the main thread. > > > > Cheers, > > Axel. > > > > |
From: Simon M. <mar...@gm...> - 2009-11-23 14:01:49
|
On 23/11/09 12:46, Axel Simon wrote: > On Mon, 2009-11-23 at 11:01 +0000, Simon Marlow wrote: >> Axel, this is great news, many thanks for working on it! >> >> I'm interested in using Cairo from multiple threads. What exactly is >> supported there? Can I run an arbitrary Render operation on any thread >> I like? > > Yes, as long as you manipulate each Cairo object within one thread: > > http://lists.cairographics.org/archives/cairo/2009-February/016648.html > > However, it involves some effort to repaint a widget using a different > thread even through you never directly paint to the screen due to Gtk's > double buffering. The reason is that the expose event EExpose internally > sets a Pixmap aside before the signal is emitted and will copy this > Pixmap to the actual window once the signal handler has returned. Thus, > you cannot spawn a thread off in the EExpose handler since it would have > to finish before the handler can return. > > So I think you need to pro-actively update a Pixmap using Cairo whenever > the content should changes. You do this in its own thread using > renderWithDrawable. Whenever the the new Pixmap is ready, you call > windowInvalidateRectangle on the DrawWindow of the DrawingArea. This > will trigger the expose handler. > > In the expose handler of the DrawingArea, you retrieve the Pixmap from > the worker thread and you copy it to the screen using > Drawable.drawDrawable. You should probably turn off double buffering > using widgetSetDoubleBuffered since this just means the pixmap is copied > twice. > > In order to avoid that you show half-ready images, your worker thread > could keep two Pixmaps: one for drawing, the other one containing the > last complete image. Thanks - this actually fits quite nicely with the way we do things in ThreadScope already. We keep track of a separate Surface already so that we can implement fast scrolling by copying it directly to the widget in the expose event, and so that we don't have to re-render the whole window when the cursor moves (the cursor is a line overlaid on top of the rendered graph). What I plan to do is have a thread in the background doing a high-detail render, and swap the rendered surface in when it is complete. Every time an expose event is received, we do a quick low-detail render, and instruct the background thread to start work on a high-detail render in the background. It should make motion quite snappy. Cheers, Simon > I've never tried this, but this is how I understand the documentation. > > Cheers, > Axel. > >> Cheers, >> Simon >> >> On 22/11/09 19:58, Axel Simon wrote: >>> Dear Simon, Duncan, >>> >>> -threaded now works with Gtk2Hs. See below. >>> >>> >>> On Tue, 2009-11-10 at 14:14 +0000, Simon Marlow wrote: >>>> On 10/11/2009 13:52, Axel Simon wrote: >>> >>>>>> Does the traceback in my original message give you enough information? >>>>>> (reproduced at the end of this message) >>>>> >>>>> Well, you omitted 'various' stuff. I agree that something is calling >>>>> back into C when it shouldn't. Are there any function names between >>>>> frame 6 and 44 that would give me a clue where we've been negligent? >>>> >>>> Ah yes, sorry. Here's the full trace: >>>> >>>> (gdb) where >>>> #0 0x0000003c0aa0af19 in pthread_cond_wait@@GLIBC_2.3.2 () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00000000005ac6e9 in waitCondition () >>>> #2 0x00000000005c67ab in waitForReturnCapability () >>>> #3 0x00000000005a3f17 in rts_lock () >>>> #4 0x00000000004ce397 in SystemziGlibziGObject_d1wV () >>>> #5 0x0000003c10cd8eed in ?? () from /usr/lib64/libgtk-x11-2.0.so.0 >>>> #6 0x0000003c0ba0d6a8 in g_object_unref () from /lib64/libgobject-2.0.so.0 >>>> #7 0x0000003c10ccebc7 in gtk_tree_view_remove_column () from >>> >>> I understand now where this error is. There is a call in your program to >>> cellLayoutSetAttributes. This function installs certain callbacks that >>> Gtk+ tries to free in the stack trace above. The call stack above is >>> triggered by >>> >>>> #41 0x0000003c0ba0d632 in g_object_unref () >>> from /lib64/libgobject-2.0.so.0 >>>> #42 0x00000000005abdfd in runAllCFinalizers () >>> >>> ... and I suppose runAllCFinalizers is the GC and an 'unsafe' call from >>> Haskell. Thus, it is actually only possible to use the C function >>> freeHaskellPtr. I do that now. >>> >>> >>>> (3) use Haskell finalizers with idleAdd as a temporary workaround. >>>> >>>> (3) is by far the easiest solution, but it may have unacceptable >>>> performance, I'd interested to know if that's the case. The runtime >>>> does batch multiple finalizers so they don't each get a separate Haskell >>>> thread, so it might not be too bad. >>>> >>>> I'm not completely averse to doing (1) if it's absolutely necessary, but >>>> I'd like to do something that avoids having a dependency on a future >>>> version of GHC. And all things being equal, we should avoid adding new >>>> features to GHC because they're hard to remove again. >>> >>> So now there are two kinds of finalizers: >>> >>> (1) those that directly free the C GObject which are used in Cairo and >>> (in the future) Pango, Pixbuf, and all other libraries whose objects do >>> not call back into Haskell. >>> >>> (2) all finalizers of Gtk+ and libraries that build upon Gtk+ >>> (sourceview, etc). These call a special finalizer that enqueues the >>> object to be finalized into an array and install a an g_idle_add handler >>> that will run the finalizers from the main iteration loop of Gtk+ and, >>> thus, from the right thread and from a safe call. >>> >>> All objects in (2) can therefore call back to Haskell while they are >>> being destroyed. I have therefore moved the possibility to add a WeakRef >>> (a function that is called when a GObject is finalized) to from GObject >>> to Graphics.UI.Gtk.Abstract.Object which means they can only be >>> installed on objects that fall into category (2). >>> >>> So the darcs version of Gtk2Hs with -threaded should now work as >>> expected. You're still responsible to call Gtk2Hs function from the same >>> Haskell thread. But you can create Cairo drawings and Pagno stuff in >>> different threads and pass them to the main thread. >>> >>> Cheers, >>> Axel. >>> >>> > |
From: Axel S. <Axe...@en...> - 2009-11-23 13:52:02
|
On Mon, 2009-11-23 at 13:35 +0000, Simon Marlow wrote: > What I plan to do is have a thread in the background doing a > high-detail > render, and swap the rendered surface in when it is complete. Every > time an expose event is received, we do a quick low-detail render, > and > instruct the background thread to start work on a high-detail render > in > the background. It should make motion quite snappy. Wow, that sounds pretty cool :-) A. |
From: Axel S. <Axe...@en...> - 2009-11-23 14:12:42
|
On Mon, 2009-11-23 at 13:35 +0000, Simon Marlow wrote: > > On Mon, 2009-11-23 at 11:01 +0000, Simon Marlow wrote: > >> Axel, this is great news, many thanks for working on it! > Actually, there is still one aspect about ghc's RTS that I would like to clarify. The bug you reported occurs because the GC finalises a GObject which, in turn, contains pointers to Haskell functions. These Haskell functions were being freed using a callback function to Haskell. Since the GC uses an 'unsafe' call to run the finalisers, this would never work, right? While this is now fixed in Gtk2Hs by using the C function freeHaskellFunPtr, I wonder if this function should actually be part of the FFI (I don't see the FFI mention any C functions at all). Axel. |
From: Simon M. <mar...@gm...> - 2009-11-23 14:28:25
|
On 23/11/09 14:12, Axel Simon wrote: > On Mon, 2009-11-23 at 13:35 +0000, Simon Marlow wrote: >>> On Mon, 2009-11-23 at 11:01 +0000, Simon Marlow wrote: >>>> Axel, this is great news, many thanks for working on it! >> > > Actually, there is still one aspect about ghc's RTS that I would like to > clarify. The bug you reported occurs because the GC finalises a GObject > which, in turn, contains pointers to Haskell functions. These Haskell > functions were being freed using a callback function to Haskell. Since > the GC uses an 'unsafe' call to run the finalisers, this would never > work, right? > > While this is now fixed in Gtk2Hs by using the C function > freeHaskellFunPtr, I wonder if this function should actually be part of > the FFI (I don't see the FFI mention any C functions at all). It is - it's called hs_free_fun_ptr(). Cheers, Simon |