tcl-core — Tcl Core Team discussions about Tcl development

Re: [TCLCORE] Inconsistent naming of STD channels

[Magentus <mag...@gm...>]

On Thu, 10 Sep 2009 10:23:15 +0100, 
"Donal K. Fellows" <don...@ma...> wrote:

> Magentus wrote:
>> But best of all, if you close "stdout", then trying to write to
>> "stdout" should generate an error, even if another channel is
>> opened in its place.
> Ah, the "break existing code" approach. Categorically not in 8.6, as 
> closing-and-reopening is a supported feature, and has been for many
> years.

Yes, I'm quite aware that breaking existing code is rather distasteful.
But is it an intended feature, and is it a correct one?  Moreover, how
supported is it; Can I close and reopen arbitrary files?  If I've
already closed stdin and stdout, can I close and reopen stderr?  Does
it work the same on every operating system?  Windows, for example?  Is
it safe in conjunction with threading?

This seems to be an accidental approach to the task.  I read some
comments from others that suggested they too wondered if it was
"correct", I only pushed the issue a few steps further, because I
quite firmly believe that it's not.  This generalised non-reuse of
channel names is a good thing.  The ONLY sane reason I can envisage not
to go ahead with it, is the problem with being able to re-open VERY low
numbered files.  And as I said, those file numbers don't actually have
any particular importance apart from what happens to be sitting there
when the application is started.

I also agree 8.6 probably isn't a viable target for such a change.  8.7
would be, though, unless it's a very commonly used feature, which I
get the sense from the posts I read that it really isn't, at least,
not one that can't be fixed reasonably quickly.  There's no great
technical issues that would require re-writing masses of code, merely
an extra command dropped in right after the new channel is opened, to
bind it to the relevant std* name.  Get the ability to re-target the
std* names in place, the sooner the better, add the "option" to stop
automatic re-binding of the standard channels, and then break the
re-opening mis-"feature" a version or two further on by making that
option default, and finally permanent, once that existing code has had a
chance to change.

Besides...  Does anyone know actually, how much unmaintained but
actively used "existing code" actually relies on that behaviour, vs.
how much existing code takes steps to work around it when it occurs
but isn't wanted?  A lot of daemons, for example, re-open NULL in place
of those three descriptors purely because some library and/or other
external code might decide to try and use those descriptors.  So far as
I can tell, it's actually quite rarely a wanted feature, whenever a
better method is available.

-- 
Fredderic

Debian/unstable (LC#384816) on i686 2.6.30-1-686 2009 (up 7 days,  7:20)

[TCLCORE] Representation vs Debugobj vs Tweezer

[Alexandre F. <ale...@gm...>]

Hi,

In a 2009-08-02 commit, Donal added an interesting debugging tool:

   ::tcl::unsupported::representation

with the comment:

   Result string is deliberately a bit obstructive so that people are
not encouraged to
   make code that depends on it; it's a debugging tool only!

Of course I'm delighted to get a sharp knife when I can... however I'm
a bit surprised by this, in the light of an earlier discussion on this
list more than one year ago (2008-06-03, subject: "debugobj"),
regarding a similar patch I had proposed
(::tcl::unsupported::debugobj) at

  https://sourceforge.net/tracker/index.php?func=detail&aid=1980917&group_id=10894&atid=310894

To summarize the differences:

  - 'representation' just gives the type name, as a string result in a
trivially parseable English sentence.

  - 'debugobj' also gives refcount, along with object and intrep
pointers, on stdout so as to be a bit harder to depend on. Knowing the
refcount can be very helpful to play with the K combinator and various
in-place optimizations, and knowing the pointers helps tracking
duplication at the Tcl_Obj and intrep levels.

At the time of the discussion, the consensus was that (1) all this was
done by Tweezer, why reinvent the wheel, and (2) too sharp a knife.
Why, then, end up with an even sharper knife with less introspective
power ?

-Alex

Re: [TCLCORE] [PATCH] Initial patch for C10k problem in TCL

[Donal K. F. <don...@ma...>]

Roman Puls wrote:
> thanks for the work! I was recently addressing that issue (compare my
> post "multithreaded Tcl IO / select vs. epoll and IOCP" from Aug. 26
> 2009) and fully agree on using epoll/kqueue/poll mechanisms instead
> of select.

I'd just say that we (well, the TCT) have known of problems in this area 
(the notifier) for a while, but with the exception of on OSX (where we 
were missing key system events because of using the wrong mechanism) we 
haven't done anything about it. Since this is due to lack of both 
knowledge and time, we're very happy to accept contributions. Well, 
subject to a few small (but critical) restrictions...

> However, as for linux, epoll is available since kernel 2.5.44/2.6.19,
> so one should be aware of older linux kernels (check in configure or
> make it a run-time choice wether to use select or epoll).

One key restriction is that we want Tcl to continue to build and work on 
all supported platforms, and so can't hard-code the use of epoll or any 
other system-specific mechanism. This isn't a particular problem if we 
use configure to detect the epoll() API *and* have a runtime mechanism 
to disable it and go back to select() or poll() if the binary is run on 
an older system. (Note that there's no way to decide whether select() or 
poll() is better without using a debugger, which is probably going a bit 
too far for a configure script! OSes tend to implement one of the APIs 
in terms of the other, but there is no agreement over which is the more 
fundamental.) It should be possible to handle run-time replacement of 
the notifier; the core notifier engine has been designed to support that 
sort of thing.

There's also been discussion of using a low-level event handling library 
to implement the notifier. If it's license-compatible (which is the 
other key restriction in this) and not required, I'm perfectly happy 
with that approach too.

Donal.

Re: [TCLCORE] [PATCH] Initial patch for C10k problem in TCL

[Roman P. <pu...@x-...>]

Hi Daniel,

thanks for the work! I was recently addressing that issue (compare my post "multithreaded Tcl IO / select vs. epoll and IOCP" from Aug. 26 2009) and fully agree on using epoll/kqueue/poll mechanisms instead of select.

However, as for linux, epoll is available since kernel 2.5.44/2.6.19, so one should be aware of older linux kernels (check in configure or make it a run-time choice wether to use select or epoll).

The only 'issue' so far might be the 10k (large number of) pre-allocated channel handles. But I am currently not aware of this could/should be handled dynamically or not. Also, as written before, it would be nice to break the bottle-neck jumbo-select(/epoll) and make polling thread specific, but this is surely a more complex topic and needs sort of thread intercommunication.

BTW: Have you made some performance tests you could post? Would be interesting!

Thanks & cheers,
  Roman
 

> -----Original Message-----
> From: Daniel Hans [mailto:da...@gm...] 
> Sent: Friday, September 11, 2009 1:11 AM
> To: TclCore
> Subject: [TCLCORE] [PATCH] Initial patch for C10k problem in TCL
> 
> Hello,
> 
> my name is Daniel Hans. One year ago I participated for TCL 
> as a Google Summer of Code student. A few months ago I asked 
> on the TCL-core mailing list for some ideas of a project for 
> my classes which would optimize something in TCL and make the 
> language more efficient.
> 
> Tomasz Kosiak suggested that I could work on the C10k problem 
> [0]. A very short description is that a server written in TCL 
> can have some problems if it has to deal with a large numbers 
> of connections at the same time.
> 
> The problem is caused by some constrains of the select 
> function which is used by TCL to watch descriptors activity.
> 
> Firstly, select uses fd_set structures to define which 
> descriptors should be taken into account. One might say, it 
> has some soft restriction that its size is 128 bytes by 
> default. It gives an opportunity to process up to 1024 
> connections by default and its behavior is satisfactory. But 
> what if someone wants more? Of course we can change the 
> limits. The best way is probably (as I have read) to 
> recompile the kernel, but when I did some tests it is enough 
> to change one constant variable in sys/select.h header file. 
> Let us say that one wants to have about 10000 connections. 
> Then we start to deal with much worse disadvantages of select 
> function.
> 
> The problem is that the function monitors all desired 
> descriptors and when some of them becomes active, it returns 
> a number and rearranges fd_set structures. Basically we have 
> no information about which descriptors are really on, thus we 
> need to iterate through the whole set and check each of them 
> individually, although probably only a few of them are 
> actually active.
> 
> Let us take a look at tclUnixNotfy.c file which is 
> responsible for those actions. Select function is mainly used 
> in Tcl_WaitForEvent:
> 
> numFound = select(tsdPtr->numFdBits,
> &(tsdPtr->readyMasks.readable),&(tsdPtr->readyMasks.writable),
> &(tsdPtr->readyMasks.exceptional), timeoutPtr);
> 
> And then, there is a problem, because numFound is not 
> actually, but we simple iterate through all open descriptors:
> 
> for (filePtr = tsdPtr->firstFileHandlerPtr; (filePtr != NULL);
>      filePtr = filePtr->nextPtr) {
> 
>            mask = 0;
>            if (FD_ISSET(filePtr->fd, 
> &(tsdPtr->readyMasks.readable))) {
>         	mask |= TCL_READABLE;
>            }
>  	    if (FD_ISSET(filePtr->fd, &(tsdPtr->readyMasks.writable))) {
>  		mask |= TCL_WRITABLE;
> 	    }
>            if (FD_ISSET(filePtr->fd, 
> &(tsdPtr->readyMasks.exceptional))) {
>         	mask |= TCL_EXCEPTION;
> 	    }
> .
> }
> 
> For each descriptor it is checked if it is active. Whenever 
> we deal with many descriptors, there is a problem as described above.
> 
> I am sending an initial version of a patch which tries to 
> resolve that problem for Linux and BSD operating systems.
> 
> The solution is very simple but straightforward: instead of 
> using inefficient select, it is better to use some other solutions.
> Unfortunately, there is no one universal way to deal with all 
> technologies. For Linux, we have epoll. For FreeBSD we have kqueue.
> Anyway, they all have one very neat feature: not only do they 
> return the number of activated descriptors, but also they 
> actually fill some structures which give us access to the 
> descriptors that we want to process. Therefore, when we have 
> 5 active out of 10000, we have their numbers and need not 
> check all of them.
> 
> I have performed some simple tests which used that patch and 
> it turned out that both kqueue as well as epoll are much 
> better with a large number of descriptors. Unfortunately, I 
> was not able to perform any tests on OpenSolaris, because I 
> had some problems with installing that system on a virtual machine.
> 
> The aim of those test was that there is a large number of 
> connections, and for each connection, there is a small amount 
> of work to be performed.
> 
> My testing environment contained only three machined, so 
> actual results could vary and be even better.
> 
> As I mentioned above this is an initial version of the patch. 
> If you had any suggestions (substantial as well as stylish) , 
> found some bugs or anything, I would really appreciate that 
> and I will try to fix it as soon as possible.
> 
> I am sending two patches. First of them updates 
> tclUnixNotfy.c file with the new code. The second one updates 
> configure file. If you want to turn the many connections 
> support option on, you need to run ./configure --enable-connections.
> 
> A slight disadvantage of the new solution is that some 
> static-sized arrays are used instead of lists of fileHandlers 
> and their size may be large for 10000 connections, but I 
> think the cost is worth if you want to play with many 
> connections. Btw. that is why I added the 
> --enable-connections option - no to persuade anyone to use it.
> 
> As I said, I know this is just an initial version of that 
> patch (which btw. was created by a standard linux diff. if it 
> is a wrong format for TCL, please let me know which one is 
> correct) to 8.5.7 version. I will appreciate any comments.
> 
> If the 10CK problem was not important for TCL, I am sorry for 
> spamming the list :-)
> 
> Regards,
> Daniel Hans
> 
> [0] http://www.kegel.com/c10k.html
> 
> 
> 

[TCLCORE] [PATCH] Initial patch for C10k problem in TCL

[Daniel H. <da...@gm...>]

Hello,

my name is Daniel Hans. One year ago I participated for TCL as a
Google Summer of Code student. A few months ago I asked on the
TCL-core mailing list for some ideas of a project for my classes which
would optimize something in TCL and make the language more efficient.

Tomasz Kosiak suggested that I could work on the C10k problem [0]. A
very short description is that a server written in TCL can have some
problems if it has to deal with a large numbers of connections at the
same time.

The problem is caused by some constrains of the select function which
is used by TCL to watch descriptors activity.

Firstly, select uses fd_set structures to define which descriptors
should be taken into account. One might say, it has some soft
restriction that its size is 128 bytes by default. It gives an
opportunity to process up to 1024 connections by default and its
behavior is satisfactory. But what if someone wants more? Of course we
can change the limits. The best way is probably (as I have read) to
recompile the kernel, but when I did some tests it is enough to change
one constant variable in sys/select.h header file. Let us say that one
wants to have about 10000 connections. Then we start to deal with much
worse disadvantages of select function.

The problem is that the function monitors all desired descriptors and
when some of them becomes active, it returns a number and rearranges
fd_set structures. Basically we have no information about which
descriptors are really on, thus we need to iterate through the whole
set and check each of them individually, although probably only a few
of them are actually active.

Let us take a look at tclUnixNotfy.c file which is responsible for
those actions. Select function is mainly used in Tcl_WaitForEvent:

numFound = select(tsdPtr->numFdBits,
&(tsdPtr->readyMasks.readable),&(tsdPtr->readyMasks.writable),
&(tsdPtr->readyMasks.exceptional), timeoutPtr);

And then, there is a problem, because numFound is not actually, but we
simple iterate through all open descriptors:

for (filePtr = tsdPtr->firstFileHandlerPtr; (filePtr != NULL);
     filePtr = filePtr->nextPtr) {

           mask = 0;
           if (FD_ISSET(filePtr->fd, &(tsdPtr->readyMasks.readable))) {
        	mask |= TCL_READABLE;
           }
 	    if (FD_ISSET(filePtr->fd, &(tsdPtr->readyMasks.writable))) {
 		mask |= TCL_WRITABLE;
	    }
           if (FD_ISSET(filePtr->fd, &(tsdPtr->readyMasks.exceptional))) {
        	mask |= TCL_EXCEPTION;
	    }
…
}

For each descriptor it is checked if it is active. Whenever we deal
with many descriptors, there is a problem as described above.

I am sending an initial version of a patch which tries to resolve that
problem for Linux and BSD operating systems.

The solution is very simple but straightforward: instead of using
inefficient select, it is better to use some other solutions.
Unfortunately, there is no one universal way to deal with all
technologies. For Linux, we have epoll. For FreeBSD we have kqueue.
Anyway, they all have one very neat feature: not only do they return
the number of activated descriptors, but also they actually fill some
structures which give us access to the descriptors that we want to
process. Therefore, when we have 5 active out of 10000, we have their
numbers and need not check all of them.

I have performed some simple tests which used that patch and it turned
out that both kqueue as well as epoll are much better with a large
number of descriptors. Unfortunately, I was not able to perform any
tests on OpenSolaris, because I had some problems with installing that
system on a virtual machine.

The aim of those test was that there is a large number of connections,
and for each connection, there is a small amount of work to be
performed.

My testing environment contained only three machined, so actual
results could vary and be even better.

As I mentioned above this is an initial version of the patch. If you
had any suggestions (substantial as well as stylish) , found some bugs
or anything, I would really appreciate that and I will try to fix it
as soon as possible.

I am sending two patches. First of them updates tclUnixNotfy.c file
with the new code. The second one updates configure file. If you want
to turn the many connections support option on, you need to run
./configure --enable-connections.

A slight disadvantage of the new solution is that some static-sized
arrays are used instead of lists of fileHandlers and their size may be
large for 10000 connections, but I think the cost is worth if you want
to play with many connections. Btw. that is why I added the
--enable-connections option - no to persuade anyone to use it.

As I said, I know this is just an initial version of that patch (which
btw. was created by a standard linux diff. if it is a wrong format for
TCL, please let me know which one is correct) to 8.5.7 version. I will
appreciate any comments.

If the 10CK problem was not important for TCL, I am sorry for spamming
the list :-)

Regards,
Daniel Hans

[0] http://www.kegel.com/c10k.html

Re: [TCLCORE] Inconsistent naming of STD channels

[Donal K. F. <don...@ma...>]

Magentus wrote:
> But best of all, if you close "stdout", then trying to write to
> "stdout" should generate an error, even if another channel is opened in
> its place.

Ah, the "break existing code" approach. Categorically not in 8.6, as 
closing-and-reopening is a supported feature, and has been for many years.

Donal.

[TCLCORE] TclpGetDate and Tcl_GetTime

[Rene Z. <r.z...@fr...>]

Hi,

to get date informations I need the internal header tclInt.h.
Would it be possible to expose the TclpGetDate function also in
the public header? Tcl_GetTime is already there.

Thank you
rene

Re: [TCLCORE] Inconsistent naming of STD channels

[Magentus <mag...@gm...>]

On Sat, 5 Sep 2009 23:43:01 +0200, 
Alexandre Ferrieux <ale...@gm...> wrote:

> Oh, I was not advocating the removal of the std* special names !
> I was merely looking for confidence in doing the change you suggested,
> which is to make a channel truly named 'stdout' as soon as its
> underlying fd is 1. This in turn implies removing the following kind
> of special-casing throughout tclIO.c:

Just to throw in another opinion, which I haven't seem mentioned so
far...  If you close stdout, and immediately open some random file, it's
just not stdout any more, so why call it such?!?

How about general channel aliasing.  Have the channels list use file0,
file1 and file2, and have those "aliased" to the std* names.  ie. BOTH
stdout and file1 would refer to the same channel.

But more-so, allowing you to re-attach the "stdout" label to another
channel, without having to close the real stdout channel, and then
bring it back again later on if you so choose.  Possibly even have both
stdout and stdin refer to the SAME bi-directional file, could be useful.
Or even more useful still, you could alias a file you've opened to the
name "log", and anywhere in the application that needs to write a log
line, it just writes it to the channel named "log".  If it's a
long-running application, the log file can even be rotated by opening a
new log file, attaching the "log" handle to that file instead, and then
closing the old one.

[chan alias] returns a list of current aliases {stdin stdout stderr}
would be the initial one (perhaps even return it as a dict?).  [chan
alias stdin] returns the file to which stdin is attached, or the empty
string if it isn't defined.  [chan alias stdin file4] would attach the
label "stdin" to file4.  Likewise [chan alias stdin {}] would remove the
alias altogether.  Simple, clean, functional.  The only edge case is
what happens if someone creates a file alias named "file42", and then
[open] tries to use that same handle.  My suggestion would be for it to
simply skip it and use file43 instead.

I know this can already be done with global variables, but personally
I'd rather see the std* names replaced with ::tcl::std* variables, and
referred to as $std*, than see the "file1 is stdout" business continue.
There is absolutely nothing special about fileid slot number 1, or 0,
or 2, apart from the fact that they're (usually) provided by
whatever's invoking your applicaiton.  In some shells, you can attach
any file (including the standard three) to any arbitrary number you
want. When a program invokes another program, it can attach anything it
wants anywhere it wants.  I've had instances where stdout was on fileid
5, and the file at fileid 1 was just a regular pre-opened data file.
It's a little odd, but it happened to work rather well.  In a few cases,
I've relocated the std files to other numbers and re-opened those id's
to /dev/null so that my application still had std*, but some library it
was using that insisted on writing to stdout and friends didn't mess
everything up. It is even possible, in *n?x, to have connections to
more than one console at a time, or to have both stdin and stdout
coming in on a single file.

But what can't be handled by global variables, is if closing file1 does
not close stdout, and vice versa.  That may introduce an
incompatibility, but could also be useful.  A way to avoid the
incompatibility without dropping the feature, is for the standard three
to have a little special-casing.  Don't give them standard names, only
channel aliases.  Then only create the "fileN" name when it's asked
for.  Start the file numbering at 3, so the first three are available,
and if std* is closed before its associated channel name has been asked
for, then do in fact close it.  The result will be transparency for
existing applications, except for the rather dodgy case of re-opening
some random file in its place and having it magically just work.

But best of all, if you close "stdout", then trying to write to
"stdout" should generate an error, even if another channel is opened in
its place.  If you want "stdout" to continue to another file, then
re-attach it to that file.  And then, you don't have to worry about
multi-threadding, or coroutines, or any of that stuff coming along and
opening up another file in the meantime.  Whatever file you open,
whatever fileid it gets, it still gets the std* label you give it.
Perhaps there could be some sugar in the [open] command for it, even.
The capital letters I, O and E could auto-alias the file to stdin,
stdout, or stderr respectively.  Whatever.  Anything's better than some
random file being trashed because you accidentally invoked a puts
statement without a channel handle in a portion of the program after
closing stdout, or because some 3rd party TCL module decided to write
to stderr after same.


Just my thoughts...

-- 
Fredderic

Debian/unstable (LC#384816) on i686 2.6.30-1-686 2009 (up 2 days,  5:39)

Re: [TCLCORE] CFV: TIP #354: Minor Production-Driven TclOO Revisions

[Donal K. F. <don...@ma...>]

Donal K. Fellows wrote:
> Donal K. Fellows wrote:
>>  TIP #354: MINOR PRODUCTION-DRIVEN TCLOO REVISIONS 
> 
> This is a Call For Votes on this TIP.

Due to being busy with work, I forgot to write up the end of this vote.
Oops! TIP#354 is Accepted with 3 votes For, 0 Against and 0 Present.

For: DKF, KBK, JE

Thanks everyone.

Donal.

Re: [TCLCORE] Inconsistent naming of STD channels

[Alexandre F. <ale...@gm...>]

On 9/7/09, Andreas Kupries <aku...@sh...> wrote:
>
>  And the handling of std channels is one of the more convoluted parts,
>  not only with the naming, but also with their automatic sharing to
>  slave interps, and (IIRC maybe) even interps in other threads. I know
>  if have exceptions in the code for transfering channels between
>  threads to exclude the std channels from transfer, although I do not
>  recall the details of that. Oh, and they have this special reference
>  counting too, which I currently have no idea if it will be affected by
>  this change.

I have looked a bit at those nasties, and they all depend on
recognizing these channels by comparison with 3 channel pointers in
the interp. So they are not affected by their names. Moreover, forcing
a reopened file1 to be named stdout everywhere instead of in 50% of
the code will only make it more recognizeable as a true stdout. So it
irons out more than it wreaks.

But anyway, please look at Jan's counter-proposal in the tracker
comments. He suggests, instead, to forbid that unixish redirection
idiom. Discussion continues there, please follow up.

> So, in general I agree with the others that this a very risky move, and not a
>  good thing to do just before a release.

Point taken. All of this for 8.7+.

-Alex

Re: [TCLCORE] Inconsistent naming of STD channels

[Andreas K. <aku...@sh...>]

Don Porter writes
> Alexandre Ferrieux wrote:
> > I was merely looking for confidence in doing the change you suggested,
> > which is to make a channel truly named 'stdout' as soon as its
> > underlying fd is 1. This in turn implies removing the following kind
> > of special-casing throughout tclIO.c:
> > 
> > 	if (strcmp(chanName, "stdin") == 0) {
> > 	    chanPtr = (Channel *) Tcl_GetStdChannel(TCL_STDIN);
> > 	} else if (strcmp(chanName, "stdout") == 0) {
> > 	    chanPtr = (Channel *) Tcl_GetStdChannel(TCL_STDOUT);
> > 	} else if (strcmp(chanName, "stderr") == 0) {
> > 	    chanPtr = (Channel *) Tcl_GetStdChannel(TCL_STDERR);
> > 	}
> 
> Don't do it.
 
> I don't mind making the design change, but with the ugly convoluted
> fragile state of all the channel code, I would not risk any "just
> make this small change" changes.


> I don't know anyone who could say what other code depends on what
> that snippet is now doing.  (Maybe aku?)

No, not even me. At least not of the top of my head.

Whenever I have to work with Channel code I have to reread and
reaquire its internal workings, with all its convolutions. And after I
am done with whatever I had to do I rapidly lose the understanding
again.

And the handling of std channels is one of the more convoluted parts,
not only with the naming, but also with their automatic sharing to
slave interps, and (IIRC maybe) even interps in other threads. I know
if have exceptions in the code for transfering channels between
threads to exclude the std channels from transfer, although I do not
recall the details of that. Oh, and they have this special reference
counting too, which I currently have no idea if it will be affected by
this change.

> Maybe if we were preparing an a1 or a2 release, I'd be willing to
> take the risk, but definitely not now.
 
> Now maybe you've digested the channel code more thoroughly than I
> realize, and you've got more confidence than I do.  If you and
> others go forward, I won't stop you, but my advice is leave it
> alone.
 
> Normally I'd say this is the sort of thing TIP review would help to
> clarify, but writing a TIP isn't going to conjure channel experts
> out of the ether.

So, in general I agree with the others that this a very risky move, and not a 
good thing to do just before a release.

-- 
So long,
	Andreas Kupries <aku...@sh...>
			<http://www.purl.org/NET/akupries/>
	Developer @	<http://www.activestate.com/>
-------------------------------------------------------------------------------

Re: [TCLCORE] Inconsistent naming of STD channels

[Donald G P. <dg...@ni...>]

Alexandre Ferrieux wrote:
> I was merely looking for confidence in doing the change you suggested,
> which is to make a channel truly named 'stdout' as soon as its
> underlying fd is 1. This in turn implies removing the following kind
> of special-casing throughout tclIO.c:
> 
> 	if (strcmp(chanName, "stdin") == 0) {
> 	    chanPtr = (Channel *) Tcl_GetStdChannel(TCL_STDIN);
> 	} else if (strcmp(chanName, "stdout") == 0) {
> 	    chanPtr = (Channel *) Tcl_GetStdChannel(TCL_STDOUT);
> 	} else if (strcmp(chanName, "stderr") == 0) {
> 	    chanPtr = (Channel *) Tcl_GetStdChannel(TCL_STDERR);
> 	}

Don't do it.

I don't mind making the design change, but with the ugly convoluted
fragile state of all the channel code, I would not risk any "just make
this small change" changes.  I don't know anyone who could say what
other code depends on what that snippet is now doing.  (Maybe aku?)
Maybe if we were preparing an a1 or a2 release, I'd be willing to take
the risk, but definitely not now.

Now maybe you've digested the channel code more thoroughly than I
realize, and you've got more confidence than I do.  If you and others
go forward, I won't stop you, but my advice is leave it alone.

Normally I'd say this is the sort of thing TIP review would help to
clarify, but writing a TIP isn't going to conjure channel experts out
of the ether.

-- 
| Don Porter          Mathematical and Computational Sciences Division |
| don...@ni...             Information Technology Laboratory |
| http://math.nist.gov/~DPorter/                                  NIST |
|______________________________________________________________________|

Re: [TCLCORE] TIP #355: Stop Fast Recycling of Channel Names on Unix

[Donal K. F. <don...@ma...>]

Andreas Leitgeb wrote:
>> This TIP tries to solve a real problem, and introduces a
>> minor potential incompatibility.
> 
> With "real problem", you mean the one about the possibility of reviving
> stale file handles?

I should note that I'm merely opposed to changing this stuff *now*, when
we're staring down the throat of a release. Yes, there are things that
are broken about, but that isn't a new thing: they have been broken
for... well, forever really. Letting 8.6 get out of the door without the
proposed changes in when we've already been through quite a bit of
beta-testing isn't a disaster, and I'd like to trail the changes to the
Unix naming algorithm more widely before dropping people in it. (Late
changes are more-than-usually sources of trouble with compatibility.)

Improving things for [versionafter 8.6] is fine.

Donal.

Re: [TCLCORE] TIP #355: Stop Fast Recycling of Channel Names on Unix

[Joe E. <jen...@fl...>]

Andreas Leitgeb wrote:

> > This TIP tries to solve a real problem, and introduces a
> > minor potential incompatibility.
>
> With "real problem", you mean the one about the possibility of reviving
> stale file handles?

A case where this can cause problems is when a program
uses the channel name as a key for auxilliary data about
the channel.  In an event-driven or callback-heavy program,
there is a window between when the channel is closed and
when the program finalizes the auxilliary data.  If the
channel name gets recycled in that window, you get an
"undangling pointer".

For example, something like:

    proc process::open {argv callback} {
	variable P
	...
	set fp [open "|$argv" r+]
	fconfigure $fp -blocking false
	...
	set P($fp.callback) $callback
	...
	fileevent readable $fp [list process::readable $fp]
    }

    proc process::readable {fp} {
        variable P
	if {[eof $fp]} {
	    process::finished $fp
	} else {
	    append P($fp.stdout) [read $fp]
	}
    }

    proc process::finished {fp} {
	variable P
	close $fp
	... check exit status, etc., ...
	# Invoke callback:
	eval [linsert $P($fp.callback) end $fp ...]
	# Clean up:
	...
	array unset P $fp.*
    }

This is a lightly edited excerpt from real code, BTW --
I've been bitten by this problem myself.  The problem shows
up if the callback script happens to call [process::open].
Thanks to channel name recycling, the cleanup phase will
stomp on data associated with the _new_ channel.

It's a subtle mistake, easy to make and hard to diagnose.
Once you've been bitten by it you learn to work around it
(just like the octal-induced problems that resurface every
August and September), but IMO it'd be nicer if the workarounds
weren't necessary in the first place.


--Joe English

Re: [TCLCORE] Inconsistent naming of STD channels

[Alexandre F. <ale...@gm...>]

On 9/5/09, Donal K. Fellows <don...@ma...> wrote:
>
>  The special-ness of stdin, stdout and stderr is just a convention,
>  albeit one of *extremely* long standing. I'd keep it rather than going
>  for some enormous degree of consistency over naming patterns.

Oh, I was not advocating the removal of the std* special names !
I was merely looking for confidence in doing the change you suggested,
which is to make a channel truly named 'stdout' as soon as its
underlying fd is 1. This in turn implies removing the following kind
of special-casing throughout tclIO.c:

	if (strcmp(chanName, "stdin") == 0) {
	    chanPtr = (Channel *) Tcl_GetStdChannel(TCL_STDIN);
	} else if (strcmp(chanName, "stdout") == 0) {
	    chanPtr = (Channel *) Tcl_GetStdChannel(TCL_STDOUT);
	} else if (strcmp(chanName, "stderr") == 0) {
	    chanPtr = (Channel *) Tcl_GetStdChannel(TCL_STDERR);
	}

So it seems nobody objects, right ?

-Alex

Re: [TCLCORE] Inconsistent naming of STD channels

[Donal K. F. <don...@ma...>]

Alexandre Ferrieux wrote:
> Two questions:
> 
>    1) Is anybody aware of the intent behind ?
> 
> (Surely not simplicity. The Right Thing is simpler.)

The intent behind ...? I need a noun... :-)

(I suspect that, if you're talking about what I guess you're talking
about, then "intent" may be too strong a word. Some things are that way
because it happened to be convenient at the time...)

>     2) Should we fear a potential incompatibility ?
> 
> (something like breaking the invariant [string match sock* [socket ...]])

To me, the key issue is that we're not consistent with having channels
have a single name (whatever that name is). While what that name is is
one issue, having multiple names that are only inconsistently used
throughout... that just can't be right and I don't know what nasties are 
lurking in there behind this.

The special-ness of stdin, stdout and stderr is just a convention,
albeit one of *extremely* long standing. I'd keep it rather than going
for some enormous degree of consistency over naming patterns.

Donal.

[TCLCORE] Inconsistent naming of STD channels

[Alexandre F. <ale...@gm...>]

Hi,

As a side effect to the discussion on channel name reforms, Donal
spotted an inconsistency in the handling of STD channel names:

 https://sourceforge.net/support/tracker.php?aid=2849797

While I intuitively agree with the obvious solution suggested there
(ie really use the "std*" names all the way down to the channel
structure), I'm a bit worried by the CVS history of that bit (in
tclIO.c) showing that the choice of "late-patching" names dates back
to the nineties.

Two questions:

   1) Is anybody aware of the intent behind ?

(Surely not simplicity. The Right Thing is simpler.)

    2) Should we fear a potential incompatibility ?

(something like breaking the invariant [string match sock* [socket ...]])

-Alex

Re: [TCLCORE] An Emacs tuning

[Kevin K. <kev...@gm...>]

Donal K. Fellows wrote:
> What I'm not saying is whether or not people should use tabs when
> editing Tcl's source. What I'm saying is that *if* they use tabs, then
> they've got to be 8 character tabs. We've had this rule for ages and
> it's part of the original Engineering Manual/Style Guide. (The one real
> advantage of the Emacs way is that it puts the noise at the end of the
> file, well out of the way.)

That said, Alex, Brian and Larry dislike the tabs, and the rest of
us appear lukewarm to them.  I think it would be Mostly Harmless
(and potentially beneficial) to add to Alex's proposed list

     indent-tabs-mode: nil

Re: [TCLCORE] Magic FDs, Was: Re: TIP #355: Stop Fast Recycling of Channel Names on Unix

[Donald G P. <don...@ni...>]

Alexandre Ferrieux wrote:
 > So for the meta-TIP "promote #355 to 8.6", we have:
 >
 >     - Donal: NO
 >     - Jan: YES
 >     - Don: YES (on the chat)

Mild correction, my comment was NO OBJECTION.

-- 
| Don Porter          Mathematical and Computational Sciences Division |
| don...@ni...             Information Technology Laboratory |
| http://math.nist.gov/~DPorter/                                  NIST |
|______________________________________________________________________|

Re: [TCLCORE] Magic FDs, Was: Re: TIP #355: Stop Fast Recycling of Channel Names on Unix

[Donald G P. <don...@ni...>]

Daniel A. Steffen wrote:
> For 8.7, I'd actually prefer an introspection API and a clean break
> w.r.t names (maybe even more so than in current #355, e.g. change the
> file/sock etc prefix as well and clarify in the docs that channel
> names are opaque).

Along those lines see Feature Request 455867.

-- 
| Don Porter          Mathematical and Computational Sciences Division |
| don...@ni...             Information Technology Laboratory |
| http://math.nist.gov/~DPorter/                                  NIST |
|______________________________________________________________________|

Re: [TCLCORE] Magic FDs, Was: Re: TIP #355: Stop Fast Recycling of Channel Names on Unix

[Jordan H. <jo...@jo...>]

I have to agree with Daniel, this should be a NO at this time.  Has anyone verified the impact on 
existing code in tcllib?  If this change were made, would all of tcllib pass regression testing?

I don't have any code that relies on the naming convention of the descriptor but I can see how 
people might have done this.

I will say I really like the ideas that have come forth on this specific issue and think they should 
be resolved and incorporated into a future release.

Not that my opinion carries any weight!!

Thanks,
Jordan Henderson

On Thursday 03 September 2009, Daniel A. Steffen wrote:
> On Thu, Sep 3, 2009 at 17:24, Alexandre
>
> Ferrieux<ale...@gm...> wrote:
> > So for the meta-TIP "promote #355 to 8.6", we have:
> >
> >    - Donal: NO
> >    - Jan: YES
> >    - Don: YES (on the chat)
> >
> > Other opinions ?
>
> NO
>
> and maybe NO on #355 itself as written at present.
>
> I don't think we can be quite so cavalier about the potential
> incompatibility this introduces, the direct fd-linked naming scheme
> has been there forever on unix, and AFAIK there were never any
> warnings in the docs to not rely on the specifics of those names.
>
> I know that I have written (admittedly hackish) tcl code myself that
> took advantage of the fact that channel fileN and file /dev/fd/N
> represent the same entity...
> In any case, IMO this is too risky a change to introduce this close to
> a release.
>
> I would definitely also like to see a script-level introspection API
> (e.g .in [chan]) added to this TIP to restores the ability to
> determine the fd from a channel name.
>
> Cheers,
>
> Daniel
>
> ---------------------------------------------------------------------------
>--- Let Crystal Reports handle the reporting - Free Crystal Reports 2008
> 30-Day trial. Simplify your report design, integration and deployment - and
> focus on what you do best, core application coding. Discover what's new
> with Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] Magic FDs, Was: Re: TIP #355: Stop Fast Recycling of Channel Names on Unix

[Daniel A. S. <da...@ca...>]

On Thu, Sep 3, 2009 at 18:23, Alexandre
Ferrieux<ale...@gm...> wrote:
> Then what about the following variant:
>
>     file$fd.$cnt
>
> with $fd being the original file descriptor and $cnt the incremented counter.

that'd likely still break anything relying on the current file$fd
scheme, so the argument against squeezing this in for 8.6 remains IMO.

For 8.7, I'd actually prefer an introspection API and a clean break
w.r.t names (maybe even more so than in current #355, e.g. change the
file/sock etc prefix as well and clarify in the docs that channel
names are opaque).

Cheers,

Daniel

Re: [TCLCORE] Magic FDs, Was: Re: TIP #355: Stop Fast Recycling of Channel Names on Unix

[Alexandre F. <ale...@gm...>]

On 9/3/09, Daniel A. Steffen <da...@ca...> wrote:
> On Thu, Sep 3, 2009 at 17:24, Alexandre
>
> Ferrieux<ale...@gm...> wrote:
>
> > So for the meta-TIP "promote #355 to 8.6", we have:
>  >
>  >    - Donal: NO
>  >    - Jan: YES
>  >    - Don: YES (on the chat)
>  >
>  > Other opinions ?
>
>
> NO
>
>  and maybe NO on #355 itself as written at present.
>
>  I don't think we can be quite so cavalier about the potential
>  incompatibility this introduces, the direct fd-linked naming scheme
>  has been there forever on unix, and AFAIK there were never any
>  warnings in the docs to not rely on the specifics of those names.
>
>  I know that I have written (admittedly hackish) tcl code myself that
>  took advantage of the fact that channel fileN and file /dev/fd/N
>  represent the same entity...
>  In any case, IMO this is too risky a change to introduce this close to
>  a release.
>
>  I would definitely also like to see a script-level introspection API
>  (e.g .in [chan]) added to this TIP to restores the ability to
>  determine the fd from a channel name.

Then what about the following variant:

     file$fd.$cnt

with $fd being the original file descriptor and $cnt the incremented counter.
This way you keep the ability of direct parsing of the fd, without
adding a primitive.
?

-Alex

Re: [TCLCORE] Magic FDs, Was: Re: TIP #355: Stop Fast Recycling of Channel Names on Unix

[Daniel A. S. <da...@ca...>]

On Thu, Sep 3, 2009 at 17:24, Alexandre
Ferrieux<ale...@gm...> wrote:
> So for the meta-TIP "promote #355 to 8.6", we have:
>
>    - Donal: NO
>    - Jan: YES
>    - Don: YES (on the chat)
>
> Other opinions ?

NO

and maybe NO on #355 itself as written at present.

I don't think we can be quite so cavalier about the potential
incompatibility this introduces, the direct fd-linked naming scheme
has been there forever on unix, and AFAIK there were never any
warnings in the docs to not rely on the specifics of those names.

I know that I have written (admittedly hackish) tcl code myself that
took advantage of the fact that channel fileN and file /dev/fd/N
represent the same entity...
In any case, IMO this is too risky a change to introduce this close to
a release.

I would definitely also like to see a script-level introspection API
(e.g .in [chan]) added to this TIP to restores the ability to
determine the fd from a channel name.

Cheers,

Daniel

Re: [TCLCORE] Magic FDs, Was: Re: TIP #355: Stop Fast Recycling of Channel Names on Unix

[Alexandre F. <ale...@gm...>]

On 9/3/09, Donal K. Fellows <don...@ma...> wrote:
> Alexandre Ferrieux wrote:
>
> > Just to clarify: are you talking about putting the brakes on:
> >    - the proposed patch in TIP # 355 (chan name reform)
> >
>
>  This one, yes.

Ach, but since you replied to the wrong thread nobody will notice :)

So for the meta-TIP "promote #355 to 8.6", we have:

    - Donal: NO
    - Jan: YES
    - Don: YES (on the chat)

Other opinions ?

-Alex