From: Alexandre F. <ale...@gm...> - 2010-04-29 10:19:05
|
Hi, Recently we switched to threaded builds by default on unix too. Of course this is key to bundling the Thread package, and is a Good Thing. However, there is one compatibility issue that has not (to my knowledge) been mentioned when pondering the move: the fact that [vwait] stops barfing with "would wait forever" when the last handler is removed. The reason is known: in the presence of threads, a [::thread::send] from another thread is always possible. IOW, the interthread communication channel plays the role of an N+1th event source, which is never silenced. Now this has drastic impacts on scripts that rely on "would wait forever" to exit. It happens to be my case in dozens of tools. Rationale: - write very simple code, no [catch], no [bgerror] - when something goes wrong in a fileevent, - the errorInfo goes to stderr, hence to the logs. good. - the fileevent is disabled - if it was the last one (and in many cases it is the only one, like a fileevent on stdin), vwait errors out with "would wait forever" - since the calling code is also [catch]less, this gets logged to stderr and exits. This scheme allows to write many powerful event-driven tools with no fat. Very readable, fitting in one editor page. The only problem is that all of the sudden, thanks to the threaded switch, these scripts will start deadlocking instead of exiting (not even a frank log-and-exit), while they are utterly single-threaded ! Worse, such minimal scripts are typically part of large pipe chains. Finding the one that's deadlocked is yet another PITA. Ouch ! Proposals: (a) document this loudly, and tell people to rewrite just about every other event-driven script. (b) arrange for the single-threaded case to behave as before. For (b), I suspect that simple counting of the Tcl threads would do, BUT would preclude using the C API from an alien thread to do the thread::send. Ideas ? -Alex |
From: Brian G. <bri...@ea...> - 2010-04-29 13:46:20
|
I too may be faced with this same problem as I prepare to switch to a threaded build. Why not have a Tcl command to disable threads. If you know the script to be intentionally single threaded, then have a command that will declare so and revert to the "would wait forever" behavior. -Brian On Thu, Apr 29, 2010 at 3:18 AM, Alexandre Ferrieux < ale...@gm...> wrote: > Hi, > > Recently we switched to threaded builds by default on unix too. Of > course this is key to bundling the Thread package, and is a Good > Thing. > > However, there is one compatibility issue that has not (to my > knowledge) been mentioned when pondering the move: the fact that > [vwait] stops barfing with "would wait forever" when the last handler > is removed. > > The reason is known: in the presence of threads, a [::thread::send] > from another thread is always possible. IOW, the interthread > communication channel plays the role of an N+1th event source, which > is never silenced. > > Now this has drastic impacts on scripts that rely on "would wait > forever" to exit. > It happens to be my case in dozens of tools. Rationale: > > - write very simple code, no [catch], no [bgerror] > - when something goes wrong in a fileevent, > - the errorInfo goes to stderr, hence to the logs. good. > - the fileevent is disabled > - if it was the last one (and in many cases it is the only > one, like a fileevent on stdin), vwait errors out with "would wait > forever" > - since the calling code is also [catch]less, this gets > logged to stderr and exits. > > This scheme allows to write many powerful event-driven tools with no > fat. Very readable, fitting in one editor page. > > The only problem is that all of the sudden, thanks to the threaded > switch, these scripts will start deadlocking instead of exiting (not > even a frank log-and-exit), while they are utterly single-threaded ! > > Worse, such minimal scripts are typically part of large pipe chains. > Finding the one that's deadlocked is yet another PITA. > > Ouch ! > > Proposals: > > (a) document this loudly, and tell people to rewrite just about > every other event-driven script. > (b) arrange for the single-threaded case to behave as before. > > > For (b), I suspect that simple counting of the Tcl threads would do, > BUT would preclude using the C API from an alien thread to do the > thread::send. > > Ideas ? > > -Alex > > > ------------------------------------------------------------------------------ > _______________________________________________ > Tcl-Core mailing list > Tcl...@li... > https://lists.sourceforge.net/lists/listinfo/tcl-core > |
From: <tc...@us...> - 2010-04-29 20:38:59
|
Seems like if don't want to break compatibility with the existing code base the opposity approach should be taken... a run-time command to enable threads. Wayne On Thu, Apr 29, 2010 at 06:46:08AM -0700, Brian Griffin wrote: > I too may be faced with this same problem as I prepare to switch to a > threaded build. > > Why not have a Tcl command to disable threads. If you know the script to be > intentionally single threaded, then have a command that will declare so and > revert to the "would wait forever" behavior. > > -Brian > > On Thu, Apr 29, 2010 at 3:18 AM, Alexandre Ferrieux < > ale...@gm...> wrote: > > > Hi, > > > > Recently we switched to threaded builds by default on unix too. Of > > course this is key to bundling the Thread package, and is a Good > > Thing. > > > > However, there is one compatibility issue that has not (to my > > knowledge) been mentioned when pondering the move: the fact that > > [vwait] stops barfing with "would wait forever" when the last handler > > is removed. > > > > The reason is known: in the presence of threads, a [::thread::send] > > from another thread is always possible. IOW, the interthread > > communication channel plays the role of an N+1th event source, which > > is never silenced. > > > > Now this has drastic impacts on scripts that rely on "would wait > > forever" to exit. > > It happens to be my case in dozens of tools. Rationale: > > > > - write very simple code, no [catch], no [bgerror] > > - when something goes wrong in a fileevent, > > - the errorInfo goes to stderr, hence to the logs. good. > > - the fileevent is disabled > > - if it was the last one (and in many cases it is the only > > one, like a fileevent on stdin), vwait errors out with "would wait > > forever" > > - since the calling code is also [catch]less, this gets > > logged to stderr and exits. > > > > This scheme allows to write many powerful event-driven tools with no > > fat. Very readable, fitting in one editor page. > > > > The only problem is that all of the sudden, thanks to the threaded > > switch, these scripts will start deadlocking instead of exiting (not > > even a frank log-and-exit), while they are utterly single-threaded ! > > > > Worse, such minimal scripts are typically part of large pipe chains. > > Finding the one that's deadlocked is yet another PITA. > > > > Ouch ! > > > > Proposals: > > > > (a) document this loudly, and tell people to rewrite just about > > every other event-driven script. > > (b) arrange for the single-threaded case to behave as before. > > > > > > For (b), I suspect that simple counting of the Tcl threads would do, > > BUT would preclude using the C API from an alien thread to do the > > thread::send. > > > > Ideas ? > > > > -Alex > > > > > > ------------------------------------------------------------------------------ > > _______________________________________________ > > Tcl-Core mailing list > > Tcl...@li... > > https://lists.sourceforge.net/lists/listinfo/tcl-core > > > ------------------------------------------------------------------------------ > _______________________________________________ > Tcl-Core mailing list > Tcl...@li... > https://lists.sourceforge.net/lists/listinfo/tcl-core |
From: Donal K. F. <don...@ma...> - 2010-04-30 09:36:33
Attachments:
donal_k_fellows.vcf
|
On 29/04/2010 21:25, tc...@us... wrote: > Seems like if don't want to break compatibility with the existing code > base the opposity approach should be taken... a run-time command to enable > threads. > That can't work. Literally can't. Thread support has to be built in totally throughout Tcl for it to work, and that's exactly what the --enable-threads option to configure does; it's now the default so anyone badly affected by this can use --disable-threads during build. It should be noted that a significant number of production deployments of Tcl are threaded anyway. Notably, Windows, OSX, and in various distributions of Linux. In *all* of these, there is currently no detection of the "will do nothing forever" state. TBH, I regard code that relies on [vwait] running out of event sources to be buggy. The program should have worked out that there was nothing further to do and called [exit] itself. Donal. |
From: Alexandre F. <ale...@gm...> - 2010-04-30 11:49:12
|
On 4/30/10, Donal K. Fellows <don...@ma...> wrote: > It > should be noted that a significant number of production deployments of > Tcl are threaded anyway. Notably, Windows, OSX, and in various > distributions of Linux. In *all* of these, there is currently no > detection of the "will do nothing forever" state. And a significant number aren't. I work for a telco, and we have RHEL4 in tens of servers (we see no value in sticking to the bleeding edge). There Tcl is unthreaded. But when RHEL5 spreads (due to end of support for RHEL4 I guess), we have a problem, Houston. > TBH, I regard code that relies on [vwait] running out of event sources > to be buggy. The program should have worked out that there was nothing > further to do and called [exit] itself. Is that the common opinion of the TCT ? Should an ages-old simple idiom be wiped out, so that now equivalent functionality involves using a script-level counter ? -Alex |
From: Joe E. <jen...@fl...> - 2010-04-30 16:26:06
|
Alexandre Ferrieux wrote: > Donal K. Fellows wrote: > > TBH, I regard code that relies on [vwait] running out of event sources > > to be buggy. The program should have worked out that there was nothing > > further to do and called [exit] itself. > > Is that the common opinion of the TCT ? > Should an ages-old simple idiom be wiped out, so that now equivalent > functionality involves using a script-level counter ? For what it's worth (not much), I tend to agree with DKF. If a script says "vwait forever; exit", I think that should mean "wait until somebody writes to the variable forever, then exit". If the program never writes to "forever", then it should be expected to never exit. The traditional behavior whereby [vwait forever] errored out when it could conclude that nobody would possibly ever write to "forever" I always interpreted as a debugging aid. It raised an error; that to me indicates "your program has an error", which is something to be fixed. --Joe English jen...@fl... |
From: Twylite <tw...@cr...> - 2010-05-04 07:56:49
|
Hi, Joe English wrote: > If a script says "vwait forever; exit", I think that should > mean "wait until somebody writes to the variable forever, > then exit". If the program never writes to "forever", > then it should be expected to never exit. > Wha...!?!? You mean that's _not_ what Tcl does currently? Is this *ahem* "feature" documented anywhere, because it is not in the [vwait] man page (which alludes to the opposite) or http://wiki.tcl.tk/vwait Regards, Twylite |
From: Uwe K. <uw...@kl...> - 2010-05-04 08:21:39
|
Twylite wrote: > Hi, > > Joe English wrote: > >>If a script says "vwait forever; exit", I think that should >>mean "wait until somebody writes to the variable forever, >>then exit". If the program never writes to "forever", >>then it should be expected to never exit. >> > > Wha...!?!? You mean that's _not_ what Tcl does currently? > > Is this *ahem* "feature" documented anywhere, because it is not in the > [vwait] man page (which alludes to the opposite) or http://wiki.tcl.tk/vwait it has been used in examples so often ... that it has become a programmers "right of way" ;-) uwe -- Uwe Klein [mailto:uw...@kl...] Habertwedt 1 D-24376 Groedersby b. Kappeln, GERMANY phone: +49 4642 920 123 FAX: +49 4642 920 125 |
From: Donald G P. <dg...@ni...> - 2010-04-30 16:51:11
|
Donal K. Fellows <don...@ma...> wrote: >> ...a significant number of production deployments of >> Tcl are threaded anyway. Alexandre Ferrieux wrote: > And a significant number aren't. I work for a telco, and we have RHEL4 > in tens of servers (we see no value in sticking to the bleeding edge). > There Tcl is unthreaded. > But when RHEL5 spreads (due to end of support for RHEL4 I guess), we > have a problem, Houston. The programs in question are not portable to an environment where Tcl is thread-enabled. There's nothing evil about that; not every program has to run in every environment. But it is a limitation of these programs. I think each of those programs should be revised in one of two ways. *) Make revisions so that the programs do become portable to thread-enabled Tcl; OR *) Start the program that demands thread-disabled Tcl with something like: if {[tcl::pkgconfig get threaded]} { puts stderr "$argv0 demands thread-disabled Tcl!" exit 1 } Then the portability limits of the programs are clear, and they fail with an informative message instead of a hang. The unfortunates who run into trouble because of a RHEL 4 -> RHEL 5 migration are in the same fix they would be in with a RHEL 4 -> Win 7 migration. They have nonportable programs which have troubles in the new environment. And they have the same solution: since the Tcl variant that comes with the system is unsuitable, they must get the variant they need in addition. This is the same demand previously imposed on those who have programs that need thread-enabled Tcl. Someone has to be inconvenienced, and the judgment expressed in passing TIP 364 was that the time has come for the thread-needers to have their turn being catered to. At least that's how I see it. -- | Don Porter Mathematical and Computational Sciences Division | | don...@ni... Information Technology Laboratory | | http://math.nist.gov/~DPorter/ NIST | |______________________________________________________________________| |
From: Alexandre F. <ale...@gm...> - 2010-05-01 14:02:40
|
On 4/30/10, Donald G Porter <dg...@ni...> wrote: > > > if {[tcl::pkgconfig get threaded]} { > puts stderr "$argv0 demands thread-disabled Tcl!" > exit 1 > } > > Then the portability limits of the programs are clear, and they > fail with an informative message instead of a hang. Yeah, clear at least. But this means changing the scripts anyway, and in all cases by adding tedious or cryptic lines: - tedious when it means [catch]/[bgerror]. 10-liners double in size. - cryptic with your assert above (go explain the reason to a newbie; 10-liners used to be good teaching material). > The unfortunates who run into trouble because of a RHEL 4 -> RHEL 5 > migration are in the same fix they would be in with a RHEL 4 -> Win 7 > migration. Yeah, except no sane server-side programmer would consider Windows, at least in my company. > Someone has > to be inconvenienced, and the judgment expressed in passing TIP 364 > was that the time has come for the thread-needers to have their turn > being catered to. Yes, but in some sub-populations the thread-needers are an ultra-minority. Again, in unix-server-side environment, modularity and fork/exec() efficiency tend to introduce many small independent, simple-task, easy-to-maintain, single-threaded processes rather than big multi-threaded monoliths. And in addition to this "would wait forever" issue, there's the doubling of the thread count: now each of these single-threaded micro-tools has 2 threads because of the Notifier... So, it might be that the importance of the "inconvenience" has been slightly underestimated :} Now I'll respect your final decision; please close 2994388 with status "Don't Do That" for further reference to the "inconvenienced". -Alex |
From: Colin M. <co...@ch...> - 2010-05-04 00:40:57
|
Alexandre started with: [With threads] there is one compatibility issue that has not (to my knowledge) been mentioned [...]: the fact that [vwait] stops barfing with "would wait forever" when the last handler is removed. And it became apparent that this property is important to his application. I think the desire to enumerate (or, at least implicitly count) the number of event sources is a surprising one, but in a good way. It seems to me that this conversation might be better framed in a different context: How could Tcl evolve the event system in such a way as to provide more facility and introspection? This is the kind of conversation that useful functional extensions are made on. I've long been of the view that UDP (for example) is best represented as an event stream. Colin. On 02/05/10 00:02, Alexandre Ferrieux wrote: > On 4/30/10, Donald G Porter<dg...@ni...> wrote: > >> >> if {[tcl::pkgconfig get threaded]} { >> puts stderr "$argv0 demands thread-disabled Tcl!" >> exit 1 >> } >> >> Then the portability limits of the programs are clear, and they >> fail with an informative message instead of a hang. >> > Yeah, clear at least. But this means changing the scripts anyway, and > in all cases by adding tedious or cryptic lines: > > - tedious when it means [catch]/[bgerror]. 10-liners double in size. > - cryptic with your assert above (go explain the reason to a newbie; > 10-liners used to be good teaching material). > > >> The unfortunates who run into trouble because of a RHEL 4 -> RHEL 5 >> migration are in the same fix they would be in with a RHEL 4 -> Win 7 >> migration. >> > Yeah, except no sane server-side programmer would consider Windows, at > least in my company. > > >> Someone has >> to be inconvenienced, and the judgment expressed in passing TIP 364 >> was that the time has come for the thread-needers to have their turn >> being catered to. >> > Yes, but in some sub-populations the thread-needers are an > ultra-minority. Again, in unix-server-side environment, modularity and > fork/exec() efficiency tend to introduce many small independent, > simple-task, easy-to-maintain, single-threaded processes rather than > big multi-threaded monoliths. And in addition to this "would wait > forever" issue, there's the doubling of the thread count: now each of > these single-threaded micro-tools has 2 threads because of the > Notifier... So, it might be that the importance of the "inconvenience" > has been slightly underestimated :} > > Now I'll respect your final decision; please close 2994388 with status > "Don't Do That" for further reference to the "inconvenienced". > > -Alex > > ------------------------------------------------------------------------------ > _______________________________________________ > Tcl-Core mailing list > Tcl...@li... > https://lists.sourceforge.net/lists/listinfo/tcl-core > > > |
From: Donal K. F. <don...@ma...> - 2010-05-04 07:54:48
Attachments:
donal_k_fellows.vcf
|
On 04/05/2010 01:40, Colin McCormack wrote: > It seems to me that this conversation might be better framed in a > different context: How could Tcl evolve the event system in such a way > as to provide more facility and introspection? > > This is the kind of conversation that useful functional extensions are > made on. I've long been of the view that UDP (for example) is best > represented as an event stream. It's definitely a drum-beat I could see myself doing some marching to. (And +1 for what you say on UDP.) Donal. |
From: Uwe K. <uw...@kl...> - 2010-05-04 08:01:41
|
Colin McCormack wrote: > I've long been of the view that UDP (for example) is best > represented as an event stream. How else? ( Most other stuff works good bucketwise too. ) Once you have people dragged away from looped polling ( screaming or not ) you are essentially into rule based programming. "Makefile for your objective" uwe -- Uwe Klein [mailto:uw...@kl...] Habertwedt 1 D-24376 Groedersby b. Kappeln, GERMANY phone: +49 4642 920 123 FAX: +49 4642 920 125 |
From: Alexandre F. <ale...@gm...> - 2010-05-04 08:18:49
|
Forgot Cc :/ ---------- Forwarded message ---------- From: Alexandre Ferrieux <ale...@gm...> Date: May 4, 2010 10:17 AM Subject: Re: [TCLCORE] Threaded unix builds and "would wait forever" To: Twylite <tw...@cr...> On 5/4/10, Twylite <tw...@cr...> wrote: > Hi, > > > Joe English wrote: > > If a script says "vwait forever; exit", I think that should > > mean "wait until somebody writes to the variable forever, > > then exit". If the program never writes to "forever", > > then it should be expected to never exit. > > > > Wha...!?!? You mean that's _not_ what Tcl does currently? > > Is this *ahem* "feature" documented anywhere, because it is not in the > [vwait] man page (which alludes to the opposite) or http://wiki.tcl.tk/vwait While the script-visible error is indeed not documented in vwait.n, the underlying behavior of Tcl_DoOneEvent is documented in Notifier.3. It may look like an UFO to somebody looking only at recent history and Windows (where threaded builds have long been the default and ActiveState's choice, for reasons unrelated to the notifier), but not to old-time unixians. And the latter happen to have some amount of code still in action. -Alex |
From: Alexandre F. <ale...@gm...> - 2010-04-29 21:12:09
|
Created SF bugreport 2994388. https://sourceforge.net/tracker/?func=detail&aid=2994388&group_id=10894&atid=110894 -Alex |
From: Donal K. F. <don...@ma...> - 2010-04-30 13:47:55
Attachments:
donal_k_fellows.vcf
|
On 30/04/2010 12:49, Alexandre Ferrieux wrote: > Should an ages-old simple idiom be wiped out, so that now equivalent > functionality involves using a script-level counter ? But you're relying on an error condition. It is (and always was) better to write your application to positively know when it is to stop serving events. Thus you'd have a point when you decide that you want to cease serving a socket, after which you are either only dealing with existing connections or killing everything immediately. In the latter case, you just [exit]. In the former, counting the number of open connections is the best bet (which you'd want to do anyway in production code; after all you were limiting the number of simultaneous connections, yes?) especially since that lets you impose a sensible timeout too. Meh. This is just application design. It's obvious you and I have different approaches here. Donal. |
From: Alexandre F. <ale...@gm...> - 2010-04-30 14:31:02
|
On 4/30/10, Donal K. Fellows <don...@ma...> wrote: > On 30/04/2010 12:49, Alexandre Ferrieux wrote: > > > Should an ages-old simple idiom be wiped out, so that now equivalent > > functionality involves using a script-level counter ? > > > > But you're relying on an error condition. It is (and always was) better > to write your application to positively know when it is to stop serving > events. Thus you'd have a point when you decide that you want to cease > serving a socket, after which you are either only dealing with existing > connections or killing everything immediately. In the latter case, you > just [exit]. In the former, counting the number of open connections is > the best bet (which you'd want to do anyway in production code; after > all you were limiting the number of simultaneous connections, yes?) > especially since that lets you impose a sensible timeout too. Fileevents are only part of the picture. The 'would wait forever' also takes into account timers. So exiting only after the last fileevent and timer have died, now implies a [decr ::counter] at the end of each (caught) handler. > Meh. This is just application design. It's obvious you and I have > different approaches here. That is clear, and that's why I asked the rest of the TCT to express opinions. -Alex |