From: <don...@is...> - 2009-02-12 09:10:36
|
I find now that I have with-timeout, of course I want to be able to control it. An obvious case is that unwind forms from unwind-protect should not be interrupted. And of course, all unwind forms should be executed no matter what else is interrupted. Is this the current semantics? There are other things that I want to be "uninteruptible". These have traditionally been marked with forms like (without-interrupts ...). Perhaps that should now be implemented with unwind-protect? It not occurs to me that back in the uniprocessor days, locks were implemented with (without-interrupts ...), and this now has to be changed. How is locking to be done in lisp code? |
From: Vladimir T. <vtz...@gm...> - 2009-02-12 09:45:34
|
On 2/12/09, Don Cohen <don...@is...> wrote: > I find now that I have with-timeout, of course I want to be able to > control it. An obvious case is that unwind forms from unwind-protect > should not be interrupted. And of course, all unwind forms should > be executed no matter what else is interrupted. Is this the current > semantics? CALL-WITH-TIMEOUT and THREAD-INTERRUPT are "dangerous" functions and should be used with care. They interrupt thread at points at which GC is possible. However if at that time the thread holds some locks - surprising results may come. Any form can be interrupted (including the unwind ones). Otherwise we may end blocked in unwind forms and timeout never be executed or delayed severely (suppose unwind form blocks waiting for something). CALL-WITH-TIMEOUT upon timeout unwinds the stack before executing timeout-function. If the body-function has established unwind frames - they will be executed. However if the timeout happens in the middle of unwind form in body-function - the rest of this form will not be executed. This is problem but with special care when writing body-function can be handled (with two nested unwind-protect's - at least one of them is guaranteed to be executed entirely). > There are other things that I want to be "uninteruptible". These have > traditionally been marked with forms like (without-interrupts ...). No there is no way to have "uninterruptible" lisp code. without-interrupts and similar cannot be implemented with preemptive multithreading - they are/were used mainly with "green" threads i.e. when the lisp runtime has full control over the thread scheduler. The way to go is with sync primitives and is application specific. Vladimir |
From: <don...@is...> - 2009-02-12 18:31:14
|
Vladimir Tzankov writes: > On 2/12/09, Don Cohen <don...@is...> wrote: > > I find now that I have with-timeout, of course I want to be able to > > control it. An obvious case is that unwind forms from unwind-protect > > should not be interrupted. And of course, all unwind forms should > > be executed no matter what else is interrupted. Is this the current > > semantics? > > CALL-WITH-TIMEOUT and THREAD-INTERRUPT are "dangerous" functions and > should be used with care. They interrupt thread at points at which GC > is possible. However if at that time the thread holds some locks - > surprising results may come. You're now talking about internal locks, I suppose, since we don't seem to have any at the user (lisp programmer) level. What are these surprising results? If we can arrange for unwind-protect to work correctly with interrupts then I suggest that all internal locks be obtained by (with-lock ...) type forms that unlock in the unwind code. (I suggest the same for lisp level locking too, of course.) > Any form can be interrupted (including the unwind ones). Otherwise we > may end blocked in unwind forms and timeout never be executed or > delayed severely (suppose unwind form blocks waiting for something). Indeed, but still there are valid reasons for wanting things to complete. I suggest that unwind code not be interruptible, at least by default. If you want to implement an interruptible version, of course, I have no objection. So far I don't see where I'd want to use it. If you can think of internal uses for it, I'd be interested in the arguments that it's appropriate. > CALL-WITH-TIMEOUT upon timeout unwinds the stack before executing > timeout-function. If the body-function has established unwind frames - > they will be executed. However if the timeout happens in the middle of > unwind form in body-function - the rest of this form will not be > executed. This is problem but with special care when writing > body-function can be handled (with two nested unwind-protect's - at > least one of them is guaranteed to be executed entirely). That works if you can be sure there's only ONE interrupt! Can you think of any unwind-protect internal to lisp code (typically as part of a (with-...) macro) that you wouldn't want to protect from interrupts? I guess it's not usually catastrophic to leave some stream unclosed, but there are lots of other transactional things that are semantically important. > > There are other things that I want to be "uninteruptible". These have > > traditionally been marked with forms like (without-interrupts ...). > > No there is no way to have "uninterruptible" lisp code. > without-interrupts and similar cannot be implemented with preemptive > multithreading - they are/were used mainly with "green" threads i.e. > when the lisp runtime has full control over the thread scheduler. > The way to go is with sync primitives and is application specific. What tools, if any, are available for the lisp programmer to implement such things? |
From: Vladimir T. <vtz...@gm...> - 2009-02-19 13:23:31
|
On Thu, 12 Feb 2009 20:31:12 +0200, Don Cohen <don...@is...> wrote: > Vladimir Tzankov writes: > > CALL-WITH-TIMEOUT and THREAD-INTERRUPT are "dangerous" functions and > > should be used with care. They interrupt thread at points at which GC > > is possible. However if at that time the thread holds some locks - > > surprising results may come. > > You're now talking about internal locks, I suppose, since we don't > seem to have any at the user (lisp programmer) level. What are these > surprising results? I am not talking about internal locks - rather user ones (mutexes and exemptions (posix condition variables) are implemented). For the internal ones - I've tried to use them in a way - that interrupts should not cause situations at which the lock is left in wrong state. > If we can arrange for unwind-protect to work correctly with interrupts > then I suggest that all internal locks be obtained by (with-lock ...) > type forms that unlock in the unwind code. Currently there is no way to enforce unwind-protect cleanup forms to be non-interruptible (if we are not in a cleanup form at the time of interrupt there are no problems - entire cleanup form will be executed). thread-interrupt is mainly for debugging and maintenance purposes. For example if some thread erroneously holds some mutex - you can interrupt it with mutex-unlock - to release it. call-with-timeout/with-timeout is useful in long computations. I would not use it for i/o (it's fine - just too big overhead). Both of them are not good for synchronization between threads (not that cannot be used for this). Both use POSIX signals internally. > > Any form can be interrupted (including the unwind ones). Otherwise we > > may end blocked in unwind forms and timeout never be executed or > > delayed severely (suppose unwind form blocks waiting for something). > > Indeed, but still there are valid reasons for wanting things to > complete. I suggest that unwind code not be interruptible, at least > by default. If you want to implement an interruptible version, of > course, I have no objection. So far I don't see where I'd want to use > it. If you can think of internal uses for it, I'd be interested in > the arguments that it's appropriate. What can be done (it's not implemented) is to delay handling of async interrupts until we get out of unwind-protect's cleanup-forms. I'll think about it and how can be done. However this may bring it's own problems - imagine blocked i/o (or locks) in some unwind-protect cleanup form (the thread will become uninterruptible - even cannot be killed). Vladimir |
From: <don...@is...> - 2009-02-19 17:22:34
|
Vladimir Tzankov writes: > > > CALL-WITH-TIMEOUT and THREAD-INTERRUPT are "dangerous" functions ... If I understand correctly, thread interrupt simply stops the thread. It does not cause cleanup forms to be executed. I have no complaint about that. With-timeout, on the other hand, causes an unwind back to the call on with-timeout. This, in my opinion, should specifically not abandon cleanup forms. > > > should be used with care. They interrupt thread at points at which GC > > > is possible. However if at that time the thread holds some locks - > > > surprising results may come. I guess the surprising results you have in mind are things like some thread still holding a lock after it has exited from the (with-lock ...). This is a good example of what I'd like to avoid. > call-with-timeout/with-timeout is useful in long computations. I would not > use it for i/o (it's fine - just too big overhead). The issue here is that certain things should not be done part way. If you are doing a transaction that involves io, you still have to either finish the io or reverse it, and neither can necessarily be done in a fixed small time. > What can be done (it's not implemented) is to delay handling of async > interrupts until we get out of unwind-protect's cleanup-forms. I'll think > about it and how can be done. However this may bring it's own problems - > imagine blocked i/o (or locks) in some unwind-protect cleanup form (the > thread will become uninterruptible - even cannot be killed). First, it seems to me that even now, when you interrupt with ^C and do abort, you can find yourself back in an infinite loop due to cleanup forms. I don't think this is a bad thing. In the debugger you at least have control over where you abort from. On the other hand, we do want to give the user the ability to skip cleanup forms from the debugger. Again, I think it's important to separate the interrupt from the processing that is done afterward. It's not clear to me at this point what exactly is done afterward in the with-timeout form. What do you think of this proposal: - there should be some way to determine that you're in a cleanup form (the lisp implementation could implement unwind-protect as if the cleanup forms were inside (let ((*in-cleanup* t)) ...)) - there should be some way to react to leaving cleanup code, e.g. we could imagine that it generates a signal that is normally handled by doing nothing - there should be a similar way to react to interruption (I hope this already generates such a signal and that with-timeout is simply supplying a handler for it.) With-timeout could then be implemented with handlers that cause a timeout interrupt (though not necessarily any others) to skip all code other than cleanup code out to the call on with-timeout. (Now that I think of it, perhaps the timout code should explicity avoid returning from the debugger.) Naturally, the intervening code might rebind these handlers. BTW, does anyone out there know what other lisp implementations that support threads do about this issue? |
From: Vladimir T. <vtz...@gm...> - 2009-02-19 17:42:51
|
On Thu, 19 Feb 2009 19:22:33 +0200, Don Cohen <don...@is...> wrote: > Vladimir Tzankov writes: > > > > CALL-WITH-TIMEOUT and THREAD-INTERRUPT are "dangerous" functions > ... > If I understand correctly, thread interrupt simply stops the thread. > It does not cause cleanup forms to be executed. I have no complaint > about that."Don Cohen" <don...@is...> More specifically: stops the thread, executes the function in it and resumes it. If the function performs non-local exit (not recommended - but this is the way thread-kill is implemented) - whatever handlers are established in the thread will be executed. > With-timeout, on the other hand, causes an unwind back to the call on > with-timeout. This, in my opinion, should specifically not abandon > cleanup forms. It does not abandon them. All cleanup forms are executed with one exception. If the control at the time of the interrupt is in a cleanup form - it will not be executed till it's end. This will be the only affected cleanup form. I agree that it is not good - but again - if we wait for the cleanup form to finish - we may never interrupt the thread. > > > > should be used with care. They interrupt thread at points at > which GC > > > > is possible. However if at that time the thread holds some locks - > > > > surprising results may come. > I guess the surprising results you have in mind are things like some > thread still holding a lock after it has exited from the (with-lock ...). > This is a good example of what I'd like to avoid. Upon thread exit - all locks that belong to it are released and warning issued. Worse problem is when the thread does not exit but erronously keeps some locks. In this case thread-interrupt with mutex-unlock will be useful. > First, it seems to me that even now, when you interrupt with ^C and > do abort, you can find yourself back in an infinite loop due to > cleanup forms. Yes - but you can always issue second ^C. > I don't think this is a bad thing. In the debugger > you at least have control over where you abort from. On the other > hand, we do want to give the user the ability to skip cleanup forms > from the debugger. > > Again, I think it's important to separate the interrupt from the > processing that is done afterward. It's not clear to me at this point > what exactly is done afterward in the with-timeout form. > > What do you think of this proposal: > - there should be some way to determine that you're in a cleanup form > (the lisp implementation could implement unwind-protect as > if the cleanup forms were inside (let ((*in-cleanup* t)) ...)) > - there should be some way to react to leaving cleanup code, e.g. > we could imagine that it generates a signal that is normally > handled by doing nothing > - there should be a similar way to react to interruption > (I hope this already generates such a signal and that with-timeout > is simply supplying a handler for it.) The risk is never to be able to "timeout" thread blocked in cleanup thread. In any case we have to keep cleanup forms interruptible by thread-interrupt - since there will be no other way even to kill threads. > BTW, does anyone out there know what other lisp implementations that > support threads do about this issue? Vladimir |
From: <don...@is...> - 2009-02-19 18:54:26
|
Vladimir Tzankov writes: > > If I understand correctly, thread interrupt simply stops the thread. > More specifically: stops the thread, executes the function in it and > resumes it. If the function performs non-local exit (not recommended - but > this is the way thread-kill is implemented) - whatever handlers are > established in the thread will be executed. I see - not what I expected. I hope the individual parts are also available, i.e., there must be some other way to simply suspend the thread, and separately some way to resume it or run something in it. > > With-timeout, on the other hand, causes an unwind back to the call on > > with-timeout. This, in my opinion, should specifically not abandon > > cleanup forms. > It does not abandon them. All cleanup forms are executed with one > exception. If the control at the time of the interrupt is in a cleanup > form - it will not be executed till it's end. This will be the only > affected cleanup form. I agree that it is not good - but again - if we > wait for the cleanup form to finish - we may never interrupt the thread. This one exception is, of course, what I object to. And the fact that you might never "get out" is true even in the current situation. > Upon thread exit - all locks that belong to it are released and warning > issued. Worse problem is when the thread does not exit but erronously > keeps some locks. In this case thread-interrupt with mutex-unlock will be > useful. This is the first I've heard of it. That sounds equally dangerous, if not more so. > > First, it seems to me that even now, when you interrupt with ^C and > > do abort, you can find yourself back in an infinite loop due to > > cleanup forms. > Yes - but you can always issue second ^C. I have in mind the situation where the signal just ends one iteration of an infinite loop. In that case the second ^C has no more effect than the first. > The risk is never to be able to "timeout" thread blocked in cleanup thread. I think the right answer is the ability to either invoke the debugger (which is, I assume, the default when there is no handler), or to do in a handler what the debugger does when the user moves up to the top of the stack and aborts -- that skips the cleanup forms, I suppose? > In any case we have to keep cleanup forms interruptible by > thread-interrupt - since there will be no other way even to kill threads. I agree that you should always be able to interrupt, but I argue that we have to control what happens after that, and that the normal with-timeout (and for that matter also thread-kill) should not skip any code in cleanup forms. Again, there should be other versions (or options) that provide the same capabilities as the debugger, including the ability to skip all cleanup forms. But I can't see that the current behavior (skipping whatever remains of whatever cleanup form happens to be executing) ever makes sense. |
From: Vladimir T. <vtz...@gm...> - 2009-02-19 21:10:52
|
On Thu, 19 Feb 2009 20:54:26 +0200, Don Cohen <don...@is...> wrote: > Vladimir Tzankov writes: > > > > If I understand correctly, thread interrupt simply stops the thread. > > > More specifically: stops the thread, executes the function in it and > > resumes it. If the function performs non-local exit (not recommended > - but > > this is the way thread-kill is implemented) - whatever handlers are > > established in the thread will be executed. > I see - not what I expected. > I hope the individual parts are also available, i.e., > there must be some other way to simply suspend the thread, and > separately some way to resume it or run something in it. No - there is no exposed way to susped/resume threads. Correctly written program should never need them and their presence just provides more harm than benefits (not sure but I think in recent versions of JDK and .NET these two are deprecated for exactly this reason. Also at ClozureCL are considering to deprecate them as well (http://ccl.clozure.com/ccl-documentation.html#Implementation-Decisions-and-Open-Questions)). > > > With-timeout, on the other hand, causes an unwind back to the call > on > > > with-timeout. This, in my opinion, should specifically not abandon > > > cleanup forms. > > It does not abandon them. All cleanup forms are executed with one > > exception. If the control at the time of the interrupt is in a cleanup > > form - it will not be executed till it's end. This will be the only > > affected cleanup form. I agree that it is not good - but again - if we > > wait for the cleanup form to finish - we may never interrupt the > thread. > This one exception is, of course, what I object to. > And the fact that you might never "get out" is true even in the > current situation. Agree. That's the reason I called them "dangerous". > > > Upon thread exit - all locks that belong to it are released and > warning > > issued. Worse problem is when the thread does not exit but erronously > > keeps some locks. In this case thread-interrupt with mutex-unlock > will be > > useful. > This is the first I've heard of it. That sounds equally dangerous, if > not more so. Why? What's wrong when thread exits to clean after itself (and warn the user that he has forgot to do this)? > > > > First, it seems to me that even now, when you interrupt with ^C and > > > do abort, you can find yourself back in an infinite loop due to > > > cleanup forms. > > Yes - but you can always issue second ^C. > I have in mind the situation where the signal just ends one iteration > of an infinite loop. In that case the second ^C has no more effect > than the first. ^C enters debugger with INTERRUPT-CONDITION (continuable). > > > The risk is never to be able to "timeout" thread blocked in cleanup > thread. > I think the right answer is the ability to either invoke the debugger > (which is, I assume, the default when there is no handler), or to do > in a handler what the debugger does when the user moves up to the top > of the stack and aborts -- that skips the cleanup forms, I suppose? Let's summarize: The problem is that call-with-timeout may skip part of unwind-protect cleanup form being executed (if any) at timeout. On the other hand - delaying the interrupt handling may cause timeout form never to be executed. Since the former may bring really severe problems and latter is possible now as well - let's delay the interrupt until the control gets out of cleanup form (if in any). If the execution blocks in some cleanup form - thread-interrupt can be used - the user can issue (for example): (thread-interrupt blocked-thread #'error "whatever") and from debugger in this thread - do whatever he wants. Does it sound reasonable? Vladimir |
From: <don...@is...> - 2009-02-19 23:14:57
|
Vladimir Tzankov writes: > No - there is no exposed way to susped/resume threads. Now that I reconsider, it occurs to me that you can suspend/resume with the primitive provided. The suspend is what happens when you interrupt and go into the debugger, the resume is what happens when you interrupt and return from the debugger. > > > keeps some locks. In this case thread-interrupt with mutex-unlock > > will be useful. > > This is the first I've heard of it. That sounds equally dangerous, if > > not more so. > Why? What's wrong when thread exits to clean after itself (and warn the > user that he has forgot to do this)? I means thread-interrupt with mutex-unlock (is that a single function?). I'm only guessing at what it does. > > > > First, it seems to me that even now, when you interrupt with ^C and > > > > do abort, you can find yourself back in an infinite loop due to > > > > cleanup forms. > > > Yes - but you can always issue second ^C. > > I have in mind the situation where the signal just ends one iteration > > of an infinite loop. In that case the second ^C has no more effect > > than the first. > > ^C enters debugger with INTERRUPT-CONDITION (continuable). I was thinking of the case where the ^C was caught and treated like a timeout. I have no problem with the case where you go into the debugger cause in that case nothing is lost. > Let's summarize: > The problem is that call-with-timeout may skip part of unwind-protect > cleanup form being executed (if any) at timeout. > On the other hand - delaying the interrupt handling may cause timeout form > never to be executed. > Since the former may bring really severe problems and latter is possible > now as well - let's delay the interrupt until the control gets out of > cleanup form (if in any). And then you'll do all the other cleanup forms out to the with-timeout and return from with-timeout. So, yes, one way to look at it is that you delayed the interrupt until the cleanup form being executed was done. Another way to view it is to imagine that you interrupted before this cleanup started. > If the execution blocks in some cleanup form - thread-interrupt can be > used - the user can issue (for example): > (thread-interrupt blocked-thread #'error "whatever") > and from debugger in this thread - do whatever he wants. > Does it sound reasonable? Yes, this is how I'd like the default with-timeout (and thread-kill) to work. I only worry now that you're planning an implementation that is less general, and therefore less useful, than what I proposed. I was hoping that in addition to the desired default behavior, it would be possible to obtain all the others with the condition system. BTW, I've been looking at Allegro documentation, and it appears that they do support without-interrupts: http://www.franz.com/support/documentation/current/doc/multiprocessing.htm It's not quite clear from the doc above, but my impression is that without-scheduling prevents this thread from being interrupted and without-interrupts further causes no other thread to run concurrently with it. (Which would be the same if you had only one cpu.) If you arrange for the cleanup of unwind-protect to not be interrupted then I guess without-scheduling could be defined as (defmacro without-scheduling (&rest forms) `(unwind-protect nil ,.forms)) I still need some way to do locking. Were you planning to make such a facility available? I used to implement this stuff using without-interrupts in the days of single CPUs. I could do so again if without-interrupts were provided (including the no other thread restriction), but it seems better to provide real locking primitives. Also I get the impression that you may not want to provide without-interrupts. And I would expect not to need it if I had locks. |
From: Vladimir T. <vtz...@gm...> - 2009-02-20 07:05:26
|
On Fri, 20 Feb 2009 01:14:58 +0200, Don Cohen <don...@is...> wrote: > Vladimir Tzankov writes: > > > > keeps some locks. In this case thread-interrupt with mutex-unlock > > > will be useful. > > > This is the first I've heard of it. That sounds equally dangerous, if > > > not more so. > > Why? What's wrong when thread exits to clean after itself (and warn the > > user that he has forgot to do this)? > I means thread-interrupt with mutex-unlock (is that a single > function?). I'm only guessing at what it does. (thread-interrupt bad-thread #'mutex-unlock some-mutex-locked-in-bad-thread) > I still need some way to do locking. > Were you planning to make such a facility available? > I used to implement this stuff using without-interrupts in the days of > single CPUs. I could do so again if without-interrupts were provided > (including the no other thread restriction), but it seems better to > provide real locking primitives. Also I get the impression that you > may not want to provide without-interrupts. And I would expect not to > need it if I had locks. Try: (MAKE-MUTEX &key name (recursive-p nil)) (MUTEX-LOCK mutex) (MUTEX-UNLOCK mutex) they do the obvious and also there is: (defmacro with-lock ((mutex) &body body) "Execute BODY with MUTEX locked." Vladimir |
From: <don...@is...> - 2009-02-20 07:33:51
|
Vladimir Tzankov writes: > (thread-interrupt bad-thread #'mutex-unlock some-mutex-locked-in-bad-thread) I see, just unlocks it... > Try: > (MAKE-MUTEX &key name (recursive-p nil)) I guess recursive means that when you have it you can lock it again. I'm also guessing that locking then increments a count and it's not really unlocked until you've unlocked as many as you've locked. Whereas non recursive means you block when you try to grab a lock that you already possess. Is all of that correct? Along with the obvious requirements for locks, - two different threads can't get the same lock at the same time - when nobody has it and some set of threads try to get it, one of them does get it > (defmacro with-lock ((mutex) &body body) > "Execute BODY with MUTEX locked." Looks like what I need. Some other details: (MUTEX-LOCK mutex) There should be two different versions, one that blocks until it succeeds, one that does not block but just tells you whether you have it or not. Is there a way to tell who has a lock? (Or whether anyone does?) When you unlock a lock you don't have - noop? error? I guess I could experiment, but also trying to encourage (possibly even help) you to write some doc. |
From: Vladimir T. <vtz...@gm...> - 2009-02-20 07:51:27
|
On Fri, 20 Feb 2009 09:34:00 +0200, Don Cohen <don...@is...> wrote: > Vladimir Tzankov writes: > > > (thread-interrupt bad-thread #'mutex-unlock some-mutex-locked-in-bad-thread) > I see, just unlocks it... > > > Try: > > (MAKE-MUTEX &key name (recursive-p nil)) > I guess recursive means that when you have it you can lock it again. > I'm also guessing that locking then increments a count and it's not > really unlocked until you've unlocked as many as you've locked. > Whereas non recursive means you block when you try to grab a lock that > you already possess. > Is all of that correct? yes - everything correct. > Along with the obvious requirements for locks, > - two different threads can't get the same lock at the same time > - when nobody has it and some set of threads try to get it, one of them > does get it yes The two types of mutexes are like PTHREAD_MUTEX_ERRORCHECK and PTHREAD_MUTEX_RECURSIVE. > > (defmacro with-lock ((mutex) &body body) > > "Execute BODY with MUTEX locked." > Looks like what I need. > > Some other details: > (MUTEX-LOCK mutex) > There should be two different versions, one that blocks until it > succeeds, one that does not block but just tells you whether you > have it or not. > Is there a way to tell who has a lock? (Or whether anyone does?) > When you unlock a lock you don't have - noop? error? (mutex-owner mutex) returns the thread that has locked it or NIL if none. however be careful with it - since the value returned may not be valid at the time you do something with it (in most generalt case). For example it is possible before you have chance the process the result the thread that has locked it to release it (and even another thread to grab it). It's useful when examining deadlocks. there are also these two predicates: (mutexp mutex) (mutex-rexcursive-p mutex) > I guess I could experiment, but also trying to encourage (possibly > even help) you to write some doc. sure - my bad that no docs are ready (actually I've started). |