From: Nikodemus S. <nik...@ra...> - 2011-03-28 20:38:45
|
WITH-TIMEOUT -- aside from being a nasty source of asynch interrupts, is also pretty much impossible to nest sanely, which also means that library code that uses it is fundamentally broken. Consider: (with-timeout 1.0 (handler-case (with-timeout 4.0 (sleep 2)) (sb-ext:timeout ()))) I'm thinking we should deprecate it, and provide something like this instead: (defmacro with-timeout-handler ((seconds timeout-form) &body body) "Runs BODY as an implicit PROGN with timeout of SECONDS. If timeout occurs before BODY has finished, BODY is unwound and TIMEOUT-FORM is executed with its values returned instead. Note that BODY is unwound asynchronously when a timeout occurs, so unless all code executed during it -- including anything down the call chain -- is asynch unwind safe, bad things will happen. Use with care." (alexandria:with-gensyms (exec unwind timer timeout block) `(block ,block (tagbody (flet ((,unwind () (go ,timeout)) (,exec () ,@body)) (declare (dynamic-extent #',exec #',unwind)) (let ((,timer (sb-ext:make-timer #',unwind))) (declare (dynamic-extent ,timer)) (sb-sys:without-interrupts (unwind-protect (progn (sb-ext:schedule-timer ,timer ,seconds) (return-from ,block (sb-sys:with-local-interrupts (,exec)))) (sb-ext:unschedule-timer ,timer))))) ,timeout (return-from ,block ,timeout-form))))) Cheers, -- Nikodemus |
From: Robert G. <rpg...@si...> - 2011-03-29 18:34:33
|
As a programmer, one related thing that I would find very helpful would be an explanation of the distinction between with-timeout and with-deadline and which to use when. Cheers, r |
From: Alastair B. <ala...@gm...> - 2011-03-29 19:08:28
|
My understanding is the following: * WITH-TIMEOUT schedules a timer interrupt (SIGALARM) for some point in the future. If the alarm goes off, it unwinds from the body and allows execution to continue from somewhere. This is the one to use never, or at least not unless you know -exactly- what can go wrong, why, and how to avoid it. * WITH-DEADLINE records a time in the future at which further I/O should cause an unwind. "All" I/O that may "block" or otherwise takes a timeout argument takes this into account and will unwind if the deadline expires. This is the one to use if you have an I/O bound process (such as establishing a network connection) and wish to cancel it if it takes too long. Is this understanding correct? I don't know. It's more likely to be correct for WITH-TIMEOUT than for WITH-DEADLINE, though. -- Alastair Bridgewater On Tue, Mar 29, 2011 at 2:18 PM, Robert Goldman <rpg...@si...> wrote: > As a programmer, one related thing that I would find very helpful would > be an explanation of the distinction between with-timeout and > with-deadline and which to use when. > > Cheers, > r > > ------------------------------------------------------------------------------ > Enable your software for Intel(R) Active Management Technology to meet the > growing manageability and security demands of your customers. Businesses > are taking advantage of Intel(R) vPro (TM) technology - will your software > be a part of the solution? Download the Intel(R) Manageability Checker > today! http://p.sf.net/sfu/intel-dev2devmar > _______________________________________________ > Sbcl-devel mailing list > Sbc...@li... > https://lists.sourceforge.net/lists/listinfo/sbcl-devel > |
From: Nikodemus S. <nik...@ra...> - 2011-03-30 08:17:17
|
On 29 March 2011 22:08, Alastair Bridgewater <ala...@gm...> wrote: > * WITH-TIMEOUT schedules a timer interrupt (SIGALARM) for some point > in the future. If the alarm goes off, it unwinds from the body and > allows execution to continue from somewhere. This is the one to use > never, or at least not unless you know -exactly- what can go wrong, > why, and how to avoid it. That's about the size of it. And given that no-one has risen in its defence I will deprecate it soonish (signaling a STYLE-WARNING when it is used) and replace it with a somewhat more manageable WITH-TIMEOUT-HANDLER (better name suggestions welcome.) As for why asynchronous unwinds are nasty -- and remain so with WITH-TIMEOUT-HANDLER -- consider this: (unwind-protect (foo) (cleanup1) (cleanup2)) If an asynch unwind hits during CLEANUP1, CLEANUP2 is never executed. OOPS. Similar examples abound. The only places to really sanely use asynch unwinds is when you know that all code executed during the period when an asych unwind can happen is asynch-unwind safe--which pretty much means you must restrict its usage to pretty low-level functions. Asynch interrupts are nasty even without unwinding, though: any code executed during an asynch interrupt should be re-entrant, because you don't know when and where it might be run. > * WITH-DEADLINE records a time in the future at which further I/O > should cause an unwind. "All" I/O that may "block" or otherwise takes > a timeout argument takes this into account and will unwind if the > deadline expires. This is the one to use if you have an I/O bound > process (such as establishing a network connection) and wish to cancel > it if it takes too long. That's the right idea, but the details are a bit fuzzier. 1. It deals with blocking reads, not writes (currently.) 2. It also deals with other block operations like WITH-MUTEX. It is much easier to use safely, since it cannot unwind from a random point in your program, but only from a limited number of system calls. You can also use DECODE-TIMEOUT to easily hook your own stuff into the deadline system. Cheers, -- Nikodemus PS. To make that UWP asynch unwind safe, you need to disable interrupts during the cleanups: (without-interrupts (unwind-protect (with-local-interrupts (foo)) (cleanup1) (cleanup1))) which of course doesn't help if FOO is not asynch-unwind safe -- and if something goes badly wrong during the cleanups debugging with interrupts disabled is much less fun. ...and all that interrupt allowance fiddling approximately triples (IIRC) the cost of the UWP. Better, and more realistically, it should really be something like (let (foo) (without-interrupts (unwind-protect (allow-with-interrupts (setf foo (foo))) (cleanup foo)))) and have FOO wrap stuff that needs to be interruptible -- eg. blocking syscalls -- with WITH-INTERRUPTS. ...and you begin to see why assuming that a random piece of idiomatic Common Lisp code is asynch unwind safe is just flat out wrong. As for why UWP doesn't by default disable interrupts for cleanups -- which would help in may cases: it's common enough to have essentially arbitrary code running under a cleanup form that doing so would not only make things like WITH-TIMEOUT even *less* useful, and any endless loops or whatnot in such places would become essentially undebuggable because you could no longer C-c them to see what's happening. :/ |
From: Robert G. <rpg...@si...> - 2011-03-30 13:23:31
|
On 3/29/11 Mar 29 -2:08 PM, Alastair Bridgewater wrote: > My understanding is the following: > > * WITH-TIMEOUT schedules a timer interrupt (SIGALARM) for some point > in the future. If the alarm goes off, it unwinds from the body and > allows execution to continue from somewhere. This is the one to use > never, or at least not unless you know -exactly- what can go wrong, > why, and how to avoid it. > > * WITH-DEADLINE records a time in the future at which further I/O > should cause an unwind. "All" I/O that may "block" or otherwise takes > a timeout argument takes this into account and will unwind if the > deadline expires. This is the one to use if you have an I/O bound > process (such as establishing a network connection) and wish to cancel > it if it takes too long. > > Is this understanding correct? I don't know. It's more likely to be > correct for WITH-TIMEOUT than for WITH-DEADLINE, though. So what am I to do if I have a computational process that I may want to choke off after a fixed amount of time? Seems like the WITH-TIMEOUT provides me no guarantee that unwinds will proceed properly, but WITH-DEADLINE is not appropriate, because it's only suitable for I/O timeouts. Sorry if these are obvious questions --- I've been programming mostly in Allegro, with green threads, so I haven't had to worry about these issues previously. Best, r |
From: Christopher S. <cs...@dt...> - 2011-03-30 15:05:50
|
On Mar 30, 2011, at 9:23 AM, Robert Goldman wrote: > So what am I to do if I have a computational process that I may want to > choke off after a fixed amount of time? In general, one approach that does not require much support from the Lisp implementation would be to spin off a thread that will sleep/block/etc until a time has been reached, at which time it mutates a shared resource that the original thread is polling on. (process-run #'(lambda () (block for a while) (setq *expired* t))) (loop for a long time until *expired* doing stuff) This does not require Lisp to have the nice interruption and unwinding features. |
From: Nikodemus S. <nik...@ra...> - 2011-03-30 15:43:50
|
On 30 March 2011 16:23, Robert Goldman <rpg...@si...> wrote: > So what am I to do if I have a computational process that I may want to > choke off after a fixed amount of time? > > Seems like the WITH-TIMEOUT provides me no guarantee that unwinds will > proceed properly, but WITH-DEADLINE is not appropriate, because it's > only suitable for I/O timeouts. You *can* use WITH-DEADLINE, if you wish, actually. DECODE-TIMEOUT hooks into it, and signals a condition if the current deadline has passed. (We should really add a separate CHECK-DEADLINE, I guess.) So you can use: (defun iterate-stuff-till-done (stuff &key timeout) (flet ((iterate-it () (loop until (stuff-done-p stuff) do (setf stuff (iterate-stuff stuff))) (when timeout ;; KLUDGE: SBCL doesn't currently expose a CHECK-DEADLINE, ;; but this does that as part of the decoding. (decode-timeout 0))))) (if timeout (sb-sys:with-deadline (:seconds timeout) (iterate-it)) (iterate-it))) which is mostly equivalent to manually checking GET-INTERNAL-REAL-TIME every time you go through the loop. If the loop is tight you might want to do that only every N rounds. You can also use an asynch timer to set a flag: (defmacro timed-thunk (time thunk) `(sb-ext:schedule-timer (sb-ext:make-timer (lambda () ,thunk)) ,time)) (defun iterate-stuff-till-done (stuff &key timeout) (let (too-late) (when timeout (timed-thunk timeout (setf too-late t))) (loop until (stuff-done-p stuff) do (when too-late (error "took too long")) (setf stuff (iterate-stuff stuff))))) Note: this is a pretty sane and safe use of an asych interrupt: no thread interaction, not unwinds, no re-entracy worries, and the thunk doesn't really care about the dynamic context. Or you can fire up a thread that sleeps and sets a flag, but that's pretty heavy in comparison to a timer -- however, if you can have a single thread that does this "time management" for multiple other threads, then it could be a win. It really depends on your code -- but the basic answers are 1. "have something set a flag, and check it at appropriate points" 2. "at appropriate points check the time". Cheers, -- Nikodemus |
From: Robert G. <rpg...@si...> - 2011-03-30 16:05:33
|
At the expense of seeming like a troll, aren't all of these solutions essentially: "use cooperative multitasking, and encode the threading structure in your application code"? Given code that does something that's not easily captured as a loop --- in my case a fairly complicated constraint-satisfaction process --- it seems to require cluttering up my code with explicit checks for timeout all over (since I don't have a nice top-level loop that lets me process for a while, pause and check for termination, and then resume). More specifically, I'm working with an ATMS (see Forbus & deKleer, /Building Problem Solvers/), where updating a constraint network can trigger fairly arbitrary boolean constraint propagation. Cracking into all that boolean constraint propagation code and sprinkling around checks for timeouts is pretty unappealing. My guess is that the ATMS code is relatively clean --- no UNWIND-PROTECT, no WITH-OPEN-FILE --- so that asynchronous interrupts are actually benign, and I can safely use with-timeout despite this issue.... But I should probably carefully verify this... Thanks, r |
From: Christopher S. <cs...@dt...> - 2011-03-30 16:29:13
|
On Mar 30, 2011, at 12:05 PM, Robert Goldman wrote: > At the expense of seeming like a troll, aren't all of these solutions > essentially: > > "use cooperative multitasking, and encode the threading structure in > your application code"? Quite accurate; but either your chosen Lisp implementation supports interruption, or not. |
From: Nikodemus S. <nik...@ra...> - 2011-03-30 16:44:41
|
On 30 March 2011 19:05, Robert Goldman <rpg...@si...> wrote: > At the expense of seeming like a troll, aren't all of these solutions > essentially: > > "use cooperative multitasking, and encode the threading structure in > your application code"? They do have that taste in them, yes. > My guess is that the ATMS code is relatively clean --- no > UNWIND-PROTECT, no WITH-OPEN-FILE --- so that asynchronous interrupts > are actually benign, and I can safely use with-timeout despite this > issue.... But I should probably carefully verify this... Yep, it sounds like WITH-TIMEOUT[-HANDLER] may indeed work for you. Most of "algorithmic stuff only" code is fine with asynch unwinds -- though in many cases one needs to take care with arbitrary user code running from a :TEST or similar. Another way to leverage a timer--without an asynch unwind--is: (timed-thunk time (invalidate-stuff stuff)) where the thunk walks over STUFF setting things to NIL / (LAMBDA () (ERROR "invalid!")) or equivalent, so that when the normal process resumes it will detect that the structure has been invalidated and signal an appropriate error. Cheers, -- Nikodemus |
From: Robert G. <rpg...@si...> - 2011-03-30 17:42:38
|
Thanks! The two suggestions look like they will work nicely for our simple case. cheers, r |
From: <me...@re...> - 2011-04-01 09:08:31
|
Robert Goldman <rpg...@si...> writes: > So what am I to do if I have a computational process that I may want to > choke off after a fixed amount of time? > > Seems like the WITH-TIMEOUT provides me no guarantee that unwinds will > proceed properly, but WITH-DEADLINE is not appropriate, because it's > only suitable for I/O timeouts. > > Sorry if these are obvious questions --- I've been programming mostly in > Allegro, with green threads, so I haven't had to worry about these > issues previously. Actually, async unwind - by its very nature - is a problem on Allegro CL as well (and, I'd hazard, on most implementations). I tend to use WITH-TIMEOUT only around side effect free computation or at least code whose side effects will be garbage after unwinding. I trust the lisp to be async unwind safe (method calls, IO, etc), but I certainly don't push it. |
From: Robert G. <rpg...@si...> - 2011-04-01 13:08:57
|
On 4/1/11 Apr 1 -4:08 AM, Gábor Melis wrote: > Robert Goldman <rpg...@si...> writes: > >> So what am I to do if I have a computational process that I may want to >> choke off after a fixed amount of time? >> >> Seems like the WITH-TIMEOUT provides me no guarantee that unwinds will >> proceed properly, but WITH-DEADLINE is not appropriate, because it's >> only suitable for I/O timeouts. >> >> Sorry if these are obvious questions --- I've been programming mostly in >> Allegro, with green threads, so I haven't had to worry about these >> issues previously. > > Actually, async unwind - by its very nature - is a problem on Allegro CL > as well (and, I'd hazard, on most implementations). I can't say for sure, but I believe that Allegro has more gross threading (time units are pretty chunky) and seems to have some stuff in its low-level implementation that makes multi-threading safer. I have had cleaner results with multi-threaded I/O, for example. Dunno if that means that it just refuses to interrupt stuff it can't unwind, or has handlers that unwind more carefully. E.g., I have had happier results with a WITH-TIMEOUT around READ-LINE on Allegro. I can't say why... [actually, I looked at READ-LINE in SBCL, and it seems interrupt unsafe because it has a LET-allocated buffer. would there be any interest in a patch that made this buffer save its state if interrupted, so READ-LINE doesn't lose data in such cases?] > > I tend to use WITH-TIMEOUT only around side effect free computation or > at least code whose side effects will be garbage after unwinding. I > trust the lisp to be async unwind safe (method calls, IO, etc), but I > certainly don't push it. I was thinking that I was ok because I have pretty much side-effect free code (if the constraint propagation is aborted, the constraint net is garbage and no one will ever look at it). But it occurs to me that logging may occur in there, so probably the logging code needs to handle timeout interrupts gracefully.... Best, Robert |
From: <me...@re...> - 2011-04-01 14:07:02
|
Robert Goldman <rpg...@si...> writes: > I can't say for sure, but I believe that Allegro has more gross > threading (time units are pretty chunky) and seems to have some stuff in > its low-level implementation that makes multi-threading safer. This is getting a bit offtopic, but green threads and the big process quantum are the reason that thread unsafety bugs are less likely to happen. Not so for SMP that SBCL has and Allegro has in the works. > I have > had cleaner results with multi-threaded I/O, for example. Dunno if that > means that it just refuses to interrupt stuff it can't unwind, or has > handlers that unwind more carefully. E.g., I have had happier results > with a WITH-TIMEOUT around READ-LINE on Allegro. I can't say why... Interrupt handling in Allegro is safepoint based [1] so where interrupts can be handled is more controllable. In all likelihood, SBCL will eventually move to safepoints as well (when the windows stuff is merged?). > [actually, I looked at READ-LINE in SBCL, and it seems interrupt unsafe > because it has a LET-allocated buffer. would there be any interest in a > patch that made this buffer save its state if interrupted, so READ-LINE > doesn't lose data in such cases?] I don't think you can implement that in a race free way under the current interrupt handling regime. > I was thinking that I was ok because I have pretty much side-effect free > code (if the constraint propagation is aborted, the constraint net is > garbage and no one will ever look at it). But it occurs to me that > logging may occur in there, so probably the logging code needs to handle > timeout interrupts gracefully.... WITHOUT-INTERRUPTS around logging code may be graceful enough if it's important. > Best, > Robert [1] http://www.franz.com/support/documentation/8.2/doc/smp.htm |
From: Attila L. <att...@gm...> - 2014-04-16 04:52:28
|
i just had a thought on WITH-DEADLINE while guarding a piece of code against any SERIOUS-CONDITION: (handler-case (something-short-that-may-fail-but-its-ok) (serious-condition () (log-a-warning-and-move-on))) this seems to be a reasonable piece of code, but if it's used in a context where there is a deadline installed, and if that deadline is signalled from inside of it, then it'll muffle the deadline. is this code bad style? is The Right Way to catch only ERROR's instead of all SERIOUS-CONDITION's (and by that not catching DEADLINE-TIMEOUT)? -- • attila lendvai • PGP: 963F 5D5F 45C7 DFCD 0A39 -- “It's better to walk alone, than with a crowd going in the wrong direction.” — Diane Grant |
From: Christophe R. <cs...@ca...> - 2014-04-16 07:53:19
|
Attila Lendvai <att...@gm...> writes: > i just had a thought on WITH-DEADLINE while guarding a piece of code > against any SERIOUS-CONDITION: > > (handler-case > (something-short-that-may-fail-but-its-ok) > (serious-condition () > (log-a-warning-and-move-on))) > > this seems to be a reasonable piece of code, but if it's used in a > context where there is a deadline installed, and if that deadline is > signalled from inside of it, then it'll muffle the deadline. > > is this code bad style? I would say "yes": handling all SERIOUS-CONDITIONs indiscriminately (and moving on) is likely to be a bad thing. The SERIOUS-CONDITIONs that are not ERRORs are things that are conceptually the computer going wrong: running out of heap or stack space, detecting memory corruption errors -- the kinds of thing that probably warrant something more than being caught by a generic handler-case. ERRORs, by contrast, are conventionally things going wrong with the program or its execution. I don't know whether deadlines are conceptually ERRORs or SERIOUS-CONDITIONs, but even if they're implemented that way it doesn't make it OK to handle all the other kinds of ways in which the computer can go wrong. Cheers, Christophe |
From: james a. <ja...@dy...> - 2014-04-16 14:34:02
|
good evening, On 16 Apr 2014, at 09:53, Christophe Rhodes <cs...@ca...> wrote: > Attila Lendvai <att...@gm...> writes: > >> i just had a thought on WITH-DEADLINE while guarding a piece of code >> against any SERIOUS-CONDITION: >> >> (handler-case >> (something-short-that-may-fail-but-its-ok) >> (serious-condition () >> (log-a-warning-and-move-on))) >> >> this seems to be a reasonable piece of code, but if it's used in a >> context where there is a deadline installed, and if that deadline is >> signalled from inside of it, then it'll muffle the deadline. >> >> is this code bad style? > > I would say "yes": handling all SERIOUS-CONDITIONs indiscriminately (and > moving on) is likely to be a bad thing. The SERIOUS-CONDITIONs that are > not ERRORs are things that are conceptually the computer going wrong: > running out of heap or stack space, detecting memory corruption > errors -- the kinds of thing that probably warrant something more than > being caught by a generic handler-case. > > ERRORs, by contrast, are conventionally things going wrong with the > program or its execution. I don't know whether deadlines are > conceptually ERRORs or SERIOUS-CONDITIONs, but even if they're > implemented that way it doesn't make it OK to handle all the other kinds > of ways in which the computer can go wrong. even once one has reduced the class of conditions which are handled, a second issue remains: one may need to distinguish handling within that class. for this situation, it can be useful that the type specifications from the clauses are tested in order and each handler executes in a dynamic context in which the others are not visible. Welcome to Clozure Common Lisp Version 1.9-r15759 (DarwinX8664)! ? (handler-case (+ 1 a) (type-error (e) (signal e) e) (condition (c) (cons :muffle c))) ;Compiler warnings : ; In an anonymous lambda form at position 19: Undeclared free variable A (:MUFFLE . #<UNBOUND-VARIABLE #x30200106F95D>) ? (handler-case (+ 1 "a") (type-error (e) (signal e) e) (condition (c) (cons :muffle c))) #<TYPE-ERROR #x3020010DFD0D> ? (handler-bind ((type-error (lambda (e) (print (cons :resignaled e))))) (handler-case (+ 1 "a") (type-error (e) (print (cons :resignaling e)) (signal e) e) (condition (c) (cons :muffle c)))) (:RESIGNALING . #<TYPE-ERROR #x302001186D1D>) (:RESIGNALED . #<TYPE-ERROR #x302001186D1D>) #<TYPE-ERROR #x302001186D1D> ? --- james anderson | ja...@dy... | http://dydra.com |