tcl-core — Tcl Core Team discussions about Tcl development

Re: [TCLCORE] Question on coroutines

[Donal K. F. <don...@ma...>]

Kevin Kenny wrote:
> Not a formal TIP, but I was talking with Miguel about this, and I
> *think* I understand what he's proposing. The idea is that [yieldto]
> will accept a complete Tcl command as its arguments. This command is
> not evaluated in the coroutine; rather, it is evaluated in the
> coroutine's *caller*. The command will usually be an invocation of
> another coroutine, but the last is not a requirement.

I was interpreting it as the name of another coroutine (in the yielded 
state) to which the other argument would be passed as result. At that 
point, it becomes trivial to do instead as a -target argument to normal 
[yield].

> I don't claim fully to understand the implications of permitting
> an arbitrary command, rather than just a coroutine, as an
> argument to [yieldto]. But it's fairly obvious from my discussion
> with Miguel that having [yieldto] accept an arbitrary command is
> by far the easiest way to implement this beast. Since using an
> arbitrary command is more general, why not?

How do you yield to a non-coroutine anyway? (And can I yield to the 
"main coroutine"?)

Donal.

Re: [TCLCORE] Patch: fix bug in Tk where X11 displays whose names contain "::" couldn't be used (plain text this time)

[Jan N. <nij...@us...>]

Good catch. In the past there was a bug-fix in clock.tcl, which simply
fixed this by escaping the problematic character. See:
  <http://tcl.cvs.sourceforge.net/viewvc/tcl/tcl/library/clock.tcl?r1=1.48&r2=1.49>

Here I would fix it the same way. Your patch only handles a single "::", and
in fact the original code doesn't work well when the display name would
contain spaces (is that possible?). Anyway, here is a new patch, which
should work with colons as well as spaces in the display name.

Could you please submit a bug report for this, and test the attached
patch?

Regards,
         Jan Nijtmans

2009/12/6 Tristan Schmelcher <tri...@al...>:
> Hello,
>
> I recently discovered a bug in Tk with X11 display strings. If the
> display string contains "::" then Tk cannot load, because it uses the
> display name as a variable name and this produces an error about the
> parent namespace not being defined. This happened to me when trying to
> use a Tk app (Mentor Graphics ModelSim) over XVnc, where the display
> string was "::1:1.0". At least one other person has reported the same
> problem: http://ubuntu-ky.ubuntuforums.org/showthread.php?p=8255492.
> (That post suggests that this has only recently become a problem,
> probably because until now display names never contained "::".)
>
> I have attached a patch that fixes the bug. I have verified that it
> works in Tk 8.4, though for your convenience I have generated the patch
> against Tk 8.5. Please consider incorporating it.
>
> Thanks!
>
> (Sorry about the HTML email before ...)
>
> ------------------------------------------------------------------------------
> Join us December 9, 2009 for the Red Hat Virtual Experience,
> a free event focused on virtualization and cloud computing.
> Attend in-depth sessions from your desk. Your couch. Anywhere.
> http://p.sf.net/sfu/redhat-sfdev2dev
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core
>
>

Re: [TCLCORE] Question on coroutines

[Neil M. <ne...@Cs...>]

On 7 Dec 2009, at 02:47, Kevin Kenny <kev...@gm...> wrote:

>> miguel sofer wrote:
>>> The same tech makes possible a new command [yieldTo]: essentially
>>> [tailcall] but yielding instead of returning - that is, saving the  
>>> state
>>> of the current coro, so that it can be reawakened later on. I am
>>> planning to put this into ::tcl::unsupported.

I'm not sure I understand how this differs from a normal yield. A  
tailcall reuses the current stack frame. A yield saves the current  
stack. How can you do both?


>
> Donal K. Fellows wrote:
>> For unsupported? No need. For supported? It would be good to see what
>> you propose in that case. Off-the-wall suggestion: maybe a "-target"
>> option to normal [yield] would be better...?
>
> Not a formal TIP, but I was talking with Miguel about this, and I
> *think* I understand what he's proposing. The idea is that [yieldto]
> will accept a complete Tcl command as its arguments. This command is
> not evaluated in the coroutine; rather, it is evaluated in the
> coroutine's *caller*. The command will usually be an invocation of
> another coroutine, but the last is not a requirement.

Is this essentially the same as the following code:

while 1 {
     set cmd [coro]
     eval $cmd
}


?


>
> As an example:
>
> proc pro {file} {
>     yield
>     while {1} {
>         set result {[gets $file $line]}
>    yieldto consumer [list $result $line]
>    if {$result < 0} { return }
>     }
> }
> proc con {} {
>     while {1} {
>    lassign [yieldto producer] result line
>         if {$result < 0} {
>        return
>         }
>    puts $line
>     }
>
> }
> set file [open "myfile.txt" r]
> coroutine producer pro $file
> coroutine consumer con
> producer;   # invoke one last time to shut it down.
> close $file
>
> Here we have created two coroutines, [producer] and [consumer].
> [producer] initially yieds back to the main routine, while [consumer]
> starts running immediately.
>
> The first action that [consumer] takes is [yieldto producer].  The
> [consumer] coroutine's state is frozen, and control returns to the
> main coroutine, which immediately invokes [producer].  The [producer]
> coroutine reads the first line of the file, and calls [yieldto]
> giving it [producer {$result $line}].
>
> The two ping-pong back and forth, each specifying the next action,
> until finally [producer] sends over the -1 result at end of file.
> At that point [consumer] returns to the main routine, and one
> last call to [producer] causes it to hit its [return]; now everything
> is shut down cleanly.
>
> Closing the file is all that remains.

But this is just the same as what the existing mechanism provides,  
modulo syntax.


>
> I don't claim fully to understand the implications of permitting
> an arbitrary command, rather than just a coroutine, as an
> argument to [yieldto]. But it's fairly obvious from my discussion
> with Miguel that having [yieldto] accept an arbitrary command is
> by far the easiest way to implement this beast. Since using an
> arbitrary command is more general, why not?
>
> This might be a nice structure for implementing things like
> compilers (the lexer yields to the parser; parser yields to
> lexer and semantic engine; semantic engine yields to parser),
> or other things that follow the "pipes and filters" pattern.
> I think that for ordinary "green threads" I'd avoid yieldto in
> favour of a scheduler in the main routine - the way coronet
> does it.

Right. And the pipes and filters approach works fine now, so what am I  
missing?

> This idea looks sound as far as it goes; I'm musing on how many
> more iterations it'll take us to get to first-class continuations,
> which is where this road seems to be heading.

If you could arrange for a coroutine to resume from the same yield  
multiple times, then I believe we have the full power of continuations.

Neil

Re: [TCLCORE] Patch: fix bug in Tk where X11 displays whose names contain "::" couldn't be used (plain text this time)

[Tristan S. <tri...@gm...>]

Sure. It is the display name that my XVnc server automatically used. My
set-up is as follows. The machine has one real X server for a physical
display and then also runs XVnc for remote connections. XVnc is launched
from xinetd with this config file in /etc/xinetd.d/Xvnc:

service Xvnc
{
type = UNLISTED
disable = no
socket_type = stream
protocol = tcp
wait = no
user = root
server = /usr/bin/Xvnc
server_args = -inetd -query localhost -geometry 1024x768 -depth 24
-securitytypes=none -DisconnectClients=0 -NeverShared -extension XFIXES
port = 5901
}

This config makes XVnc run the GDM greeter, making it behave more or
less like a Windows RDP server. You may need to enable XDMCP for it to
work.

At the time, another user was logged-in on the physical X display and I
logged in over XVnc. Inside the GNOME session launched from within XVnc,
DISPLAY is set to ::1:1.0.

My OS is Ubuntu Karmic (9.10) 64-bit.

On Sun, 2009-12-06 at 23:18 -0800, Technical Support (Brian G) wrote:
> Hi Tristan,
> 
> Thanks for reporting, and resolving, this issue.  I'd like to confirm 
> the fix for inclusion in the next ModelSim release, and to help the Tcl 
> Core team as well.  Can you describe how this DISPLAY name was 
> generated.  I have not been successful in recreating this condition.
> 
> Thanks,
> -Brian
> 
> Tristan Schmelcher wrote:
> > Hello,
> >
> > I recently discovered a bug in Tk with X11 display strings. If the
> > display string contains "::" then Tk cannot load, because it uses the
> > display name as a variable name and this produces an error about the
> > parent namespace not being defined. This happened to me when trying to
> > use a Tk app (Mentor Graphics ModelSim) over XVnc, where the display
> > string was "::1:1.0". At least one other person has reported the same
> > problem: http://ubuntu-ky.ubuntuforums.org/showthread.php?p=8255492.
> > (That post suggests that this has only recently become a problem,
> > probably because until now display names never contained "::".)
> >
> > I have attached a patch that fixes the bug. I have verified that it
> > works in Tk 8.4, though for your convenience I have generated the patch
> > against Tk 8.5. Please consider incorporating it.
> >
> > Thanks!
> >
> > (Sorry about the HTML email before ...)
> >   
> > ------------------------------------------------------------------------
> >
> > ------------------------------------------------------------------------------
> > Join us December 9, 2009 for the Red Hat Virtual Experience,
> > a free event focused on virtualization and cloud computing. 
> > Attend in-depth sessions from your desk. Your couch. Anywhere.
> > http://p.sf.net/sfu/redhat-sfdev2dev
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > Tcl-Core mailing list
> > Tcl...@li...
> > https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] Question on coroutines

[Kevin K. <kev...@gm...>]

Larry McVoy wrote:
> I'm coming into this late so maybe I don't get it.  But aren't coroutines
> basically busted from the start?
> 
> Long ago I did a coroutine system and it was really pleasant, it was the
> first place I wrote swtch() (the context switch code that every real
> hacker has to write), but it was all a waste of time.
> 
> Why?  Because the threads blocked on IO.  Is that true with the tcl
> coroutines?  If it is, aren't they kind of a joke compared to threads
> that handle IO?

Tcl's coroutines are essentially co-operative ("green") threads.
If you look at http://wiki.tcl.tk/22231, you'll see that a simple
(tens of lines of code per function) layer can be put on top of
coroutines to make *them* appear like threads that are waiting
on blocking I/O, when what they are actually doing is setting
fileevents and yielding to the event loop. Wub is built that way,
and once it was over its 'infant mortality' phase of performance
bugs, it seems to behave quite nicely without spinning a vast
tangle of native threads.

What green threads *won't* do is let you have CPU-burning loops
that don't yield.  On the other hand, green threads also are a good bit
more predictable than native ones about atomicity of actions - since
they never are preempted, a lot of the need for mutex (not all, but
a lot) goes away.

Also, there are other uses (e.g., "pipes and filters" programming,
discrete event simulation, decomposition of state machines)
where coroutines are useful but don't actually represent a threading
abstraction.

And, if you want kernel-level threads, you know where to find them.
The Thread extension runs, as far as I know, on every major platform.
(Windows, Mac, Linux, and anything with pthreads.) There's nothing
that says we can't have both green threads *and* native threads.

-- 
73 de ke9tv/2, Kevin

Re: [TCLCORE] Question on coroutines

[Larry M. <lm...@bi...>]

I'm coming into this late so maybe I don't get it.  But aren't coroutines
basically busted from the start?

Long ago I did a coroutine system and it was really pleasant, it was the
first place I wrote swtch() (the context switch code that every real
hacker has to write), but it was all a waste of time.

Why?  Because the threads blocked on IO.  Is that true with the tcl
coroutines?  If it is, aren't they kind of a joke compared to threads
that handle IO?
-- 
---
Larry McVoy                lm at bitmover.com           http://www.bitkeeper.com

Re: [TCLCORE] Question on coroutines

[Kevin K. <kev...@gm...>]

> miguel sofer wrote:
>> The same tech makes possible a new command [yieldTo]: essentially 
>> [tailcall] but yielding instead of returning - that is, saving the state 
>> of the current coro, so that it can be reawakened later on. I am 
>> planning to put this into ::tcl::unsupported.

Donal K. Fellows wrote:
> For unsupported? No need. For supported? It would be good to see what 
> you propose in that case. Off-the-wall suggestion: maybe a "-target" 
> option to normal [yield] would be better...?

Not a formal TIP, but I was talking with Miguel about this, and I
*think* I understand what he's proposing. The idea is that [yieldto]
will accept a complete Tcl command as its arguments. This command is
not evaluated in the coroutine; rather, it is evaluated in the
coroutine's *caller*. The command will usually be an invocation of
another coroutine, but the last is not a requirement.

As an example:

proc pro {file} {
     yield
     while {1} {
         set result {[gets $file $line]}
	yieldto consumer [list $result $line]
	if {$result < 0} { return }
     }
}
proc con {} {
     while {1} {
	lassign [yieldto producer] result line
         if {$result < 0} {
	    return
         }
	puts $line
     }

}
set file [open "myfile.txt" r]
coroutine producer pro $file
coroutine consumer con
producer;   # invoke one last time to shut it down.
close $file

Here we have created two coroutines, [producer] and [consumer].
[producer] initially yieds back to the main routine, while [consumer]
starts running immediately.

The first action that [consumer] takes is [yieldto producer].  The
[consumer] coroutine's state is frozen, and control returns to the
main coroutine, which immediately invokes [producer].  The [producer]
coroutine reads the first line of the file, and calls [yieldto]
giving it [producer {$result $line}].

The two ping-pong back and forth, each specifying the next action,
until finally [producer] sends over the -1 result at end of file.
At that point [consumer] returns to the main routine, and one
last call to [producer] causes it to hit its [return]; now everything
is shut down cleanly.

Closing the file is all that remains.

I don't claim fully to understand the implications of permitting
an arbitrary command, rather than just a coroutine, as an
argument to [yieldto]. But it's fairly obvious from my discussion
with Miguel that having [yieldto] accept an arbitrary command is
by far the easiest way to implement this beast. Since using an
arbitrary command is more general, why not?

This might be a nice structure for implementing things like
compilers (the lexer yields to the parser; parser yields to
lexer and semantic engine; semantic engine yields to parser),
or other things that follow the "pipes and filters" pattern.
I think that for ordinary "green threads" I'd avoid yieldto in
favour of a scheduler in the main routine - the way coronet
does it.

This idea looks sound as far as it goes; I'm musing on how many
more iterations it'll take us to get to first-class continuations,
which is where this road seems to be heading.
--
73 de ke9tv/2, Kevin

Re: [TCLCORE] Question on coroutines

[Donal K. F. <don...@ma...>]

miguel sofer wrote:
> The same tech makes possible a new command [yieldTo]: essentially 
> [tailcall] but yielding instead of returning - that is, saving the state 
> of the current coro, so that it can be reawakened later on. I am 
> planning to put this into ::tcl::unsupported.

That's the classic definition of yield in a coroutine. What we have/had 
before was really a "generator".

> I think that this command could find very interesting applications. The 
> big Q is: shall I try to TIP it as an amendment to TIP #328? I DO know 
> it is rather late ...

For unsupported? No need. For supported? It would be good to see what 
you propose in that case. Off-the-wall suggestion: maybe a "-target" 
option to normal [yield] would be better...?

Donal.

[TCLCORE] Patch: fix bug in Tk where X11 displays whose names contain "::" couldn't be used (plain text this time)

[Tristan S. <tri...@al...>]

Hello,

I recently discovered a bug in Tk with X11 display strings. If the
display string contains "::" then Tk cannot load, because it uses the
display name as a variable name and this produces an error about the
parent namespace not being defined. This happened to me when trying to
use a Tk app (Mentor Graphics ModelSim) over XVnc, where the display
string was "::1:1.0". At least one other person has reported the same
problem: http://ubuntu-ky.ubuntuforums.org/showthread.php?p=8255492.
(That post suggests that this has only recently become a problem,
probably because until now display names never contained "::".)

I have attached a patch that fixes the bug. I have verified that it
works in Tk 8.4, though for your convenience I have generated the patch
against Tk 8.5. Please consider incorporating it.

Thanks!

(Sorry about the HTML email before ...)

[TCLCORE] Patch: fix bug in Tk where X11 displays whose names contain "::" couldn't be used

[Tristan S. <tri...@al...>]

Hello,

I recently discovered a bug in Tk with X11 display strings. If the
display string contains "::" then Tk cannot load, because it uses the
display name as a variable name and this produces an error about the
parent namespace not being defined. This happened to me when trying to
use a Tk app (Mentor Graphics ModelSim) over XVnc, where the display
string was "::1:1.0". At least one other person has reported the same
problem: http://ubuntu-ky.ubuntuforums.org/showthread.php?p=8255492.
(That post suggests that this has only recently become a problem,
probably because until now display names never contained "::".)

I have attached a patch that fixes the bug. I have verified that it
works in Tk 8.4, though for your convenience I have generated the patch
against Tk 8.5. Please consider incorporating it.

Thanks!

[TCLCORE] Question on coroutines

[miguel s. <mig...@gm...>]

I am working on fixing some of the missing things in coros.

For instance, as of today, HEAD can tailcall out of a coro.

The missing thing is to nre-enable coros: if you call
   coroutine B $body {*}$arglist
from within a coroutine A, it would be good ig coroutine A would remove 
itself from the callstacks before running coroutine B.

I now have a good idea as to how to do this ... sorry it took so long. 
It should be in HEAD in the next few days.

The same tech makes possible a new command [yieldTo]: essentially 
[tailcall] but yielding instead of returning - that is, saving the state 
of the current coro, so that it can be reawakened later on. I am 
planning to put this into ::tcl::unsupported.

I think that this command could find very interesting applications. The 
big Q is: shall I try to TIP it as an amendment to TIP #328? I DO know 
it is rather late ...

Cheers
Miguel

Re: [TCLCORE] Inconsistent buffered I/O

[Tom J. <tom...@gm...>]

Alex,

Okay, I did the additional testing. Instead of logging to stderr, I
just put a select timer for 100,000 microseconds into WillRead &
WillWrite.

Here are the results if anyone is interested:

http://rmadilo.com/files/tcl-io/testio2/TIMINGS.TXT


The maximum number of calls to the two helpers is 10, so the max
sleepy time is 1 sec.

The question to be answered by the test was if the Seek or Flush
executed at any other time than when necessary.

Answer: never happens. Perfect score.

The patch fixes the problem exactly and should not have any detectable
effect on existing "bug-free" code which uses [read], [puts] or
[gets].

Code which uses the workarounds will probably run slightly faster if
the workarounds are removed.

tom jackson

On Thu, Dec 3, 2009 at 10:24 AM, Alexandre Ferrieux
<ale...@gm...> wrote:
> On 12/3/09, Tom Jackson <tom...@gm...> wrote:
>> Alex,
>>
>>  I just tested the patch. All 144 configurations tested give the
>>  expected result.
>
> Ha !
>
> So, is the patch "OK for you" now ?
>
> -Alex
>

Re: [TCLCORE] Inconsistent buffered I/O

[Tom J. <tom...@gm...>]

The tests included 40 configurations which could be considered
work-arounds: they worked! There were 36 which included at least flush
before read and seek before write. There were four where
buffering=none which only included a seek prior to write.

All passed.

Only thing I would like to test is to add a log message to the two new
helper API to see when they fire in relation to the different cases.

I'm hoping they don't fire at all for the work-around cases. The only
thing this would prove is that performance would be at a maximum.
Although I guess it would also prove is that no code which use
read/write will perform _any_ differently as viewed from outside, no
new flush or seek calls.

But the patch looks very good.

If anyone has example code for logging to stderr from a tcl library
function, please send me an email.

tom jackson

On Thu, Dec 3, 2009 at 10:17 AM, Barry Skidmore <at...@me...> wrote:
> Just out of curiosity of the expected result.
>
> Your tests have included the workaround code against the patched source yes?
>
> Effectively proving that any code written to work around this issue will continue to function correctly with the patched version?
>
> On Dec 3, 2009, at 10:13 AM, Tom Jackson wrote:
>
>> Alex,
>>
>> I just tested the patch. All 144 configurations tested give the
>> expected result.
>>
>> Here is a link to the test results (in html table form):
>>
>> http://rmadilo.com/files/tcl-io/testio2/RESULTS.html
>>
>> Note that these tests include 16 tests with no flush/seek prior to
>> read or write:
>> (r+, w+) x (none, full) x (buffsiz 10, 20, 1024, and  no -buffersize option)
>>
>> tom jackson
>>
>> On Wed, Dec 2, 2009 at 11:44 PM, Alexandre Ferrieux
>> <ale...@gm...> wrote:
>>> On 12/3/09, Tom Jackson <tom...@gm...> wrote:
>>>>
>>>>  Is the patch against the HEAD on sf?
>>>
>>> Yaeh of course. Why don't you go fetch it and test at last ?
>>>
>>> -Alex
>>>
>>
>> ------------------------------------------------------------------------------
>> Join us December 9, 2009 for the Red Hat Virtual Experience,
>> a free event focused on virtualization and cloud computing.
>> Attend in-depth sessions from your desk. Your couch. Anywhere.
>> http://p.sf.net/sfu/redhat-sfdev2dev
>> _______________________________________________
>> Tcl-Core mailing list
>> Tcl...@li...
>> https://lists.sourceforge.net/lists/listinfo/tcl-core
>
>

Re: [TCLCORE] Inconsistent buffered I/O

[Alexandre F. <ale...@gm...>]

On 12/3/09, Tom Jackson <tom...@gm...> wrote:
> Alex,
>
>  I just tested the patch. All 144 configurations tested give the
>  expected result.

Ha !

So, is the patch "OK for you" now ?

-Alex

Re: [TCLCORE] Inconsistent buffered I/O

[Barry S. <at...@me...>]

Just out of curiosity of the expected result.

Your tests have included the workaround code against the patched source yes?

Effectively proving that any code written to work around this issue will continue to function correctly with the patched version?

On Dec 3, 2009, at 10:13 AM, Tom Jackson wrote:

> Alex,
> 
> I just tested the patch. All 144 configurations tested give the
> expected result.
> 
> Here is a link to the test results (in html table form):
> 
> http://rmadilo.com/files/tcl-io/testio2/RESULTS.html
> 
> Note that these tests include 16 tests with no flush/seek prior to
> read or write:
> (r+, w+) x (none, full) x (buffsiz 10, 20, 1024, and  no -buffersize option)
> 
> tom jackson
> 
> On Wed, Dec 2, 2009 at 11:44 PM, Alexandre Ferrieux
> <ale...@gm...> wrote:
>> On 12/3/09, Tom Jackson <tom...@gm...> wrote:
>>> 
>>>  Is the patch against the HEAD on sf?
>> 
>> Yaeh of course. Why don't you go fetch it and test at last ?
>> 
>> -Alex
>> 
> 
> ------------------------------------------------------------------------------
> Join us December 9, 2009 for the Red Hat Virtual Experience,
> a free event focused on virtualization and cloud computing. 
> Attend in-depth sessions from your desk. Your couch. Anywhere.
> http://p.sf.net/sfu/redhat-sfdev2dev
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] Inconsistent buffered I/O

[Tom J. <tom...@gm...>]

Alex,

I just tested the patch. All 144 configurations tested give the
expected result.

Here is a link to the test results (in html table form):

http://rmadilo.com/files/tcl-io/testio2/RESULTS.html

Note that these tests include 16 tests with no flush/seek prior to
read or write:
(r+, w+) x (none, full) x (buffsiz 10, 20, 1024, and  no -buffersize option)

tom jackson

On Wed, Dec 2, 2009 at 11:44 PM, Alexandre Ferrieux
<ale...@gm...> wrote:
> On 12/3/09, Tom Jackson <tom...@gm...> wrote:
>>
>>  Is the patch against the HEAD on sf?
>
> Yaeh of course. Why don't you go fetch it and test at last ?
>
> -Alex
>

[TCLCORE] Memory leak in try "on" handlers

[Cyan O. <cya...@gm...>]

Hi

There seems to be a memory leak in the result handler code of the C
based try command.

This leaks the result:

try {
    string repeat \0 1048576
} on ok {} {}

... this doesn't:

try {
    string repeat \0 1048576
}


Adding Tcl_DecrRefCount(resultObj); at line 4461 of generic/tclCmdMZ.c
fixes the leak for me.

Cyan

Re: [TCLCORE] Inconsistent buffered I/O

[Alexandre F. <ale...@gm...>]

On 12/3/09, Tom Jackson <tom...@gm...> wrote:
>
>  Is the patch against the HEAD on sf?

Yaeh of course. Why don't you go fetch it and test at last ?

-Alex

Re: [TCLCORE] Inconsistent buffered I/O

[Tom J. <tom...@gm...>]

Alex,

Here is a more complete list of tests. The first round left out
different buffer sizes (or omitting -buffersize completely).

With these added in, there are now 10 distinct result files: 1 good and 9 bad.

To make it easier to look at the different problems I added an html
file with links:

http://rmadilo.com/files/tcl-io/testio/MD5LINK.html

The .test files show the final contents of the working file.
The .log files contain the initial conditions, the generated script,
and the results of script execution.

The main differences here result from buffers smaller than the write,
buffers larger than the filesize and not using -buffersize in the
[fconfigure] call.

I also removed the mode a+, since a+ always works.

There are 144 tested configurations and only 40 produce the correct
result. The remaining 104 results are divided into 9 distinct
categories.

The "correct" results are divided into two categories:
 if -buffering == none, it is only required to seek prior to writing
 otherwise, you must also flush prior to reading.

This is with tclsh86.

Is the patch against the HEAD on sf?

tom jackson

Re: [TCLCORE] Inconsistent buffered I/O

[Tom J. <tom...@gm...>]

Alex,

The SUMMARY.TXT file lists the different results.

The "correct" result file contents is:
11111111111L22222222222X33333333333K44444444444W55555555555JKLMNOPQ...
123456789012345678901234567890123456789012345678901234567890123456789...
<newline>

where ... just means the line continues a little further along.

The basic loop is:
foreach num {1 2 3 4 5} {
    seek $fd 0 current ;# write sync here
    set start [tell $fd]
    puts -nonewline $fd [string repeat $num 11]
    set end [tell $fd]
    seek $fd 0 current ;# read sync here
    set repos [tell $fd]
    set char [read $fd 1]
    set final [tell $fd ]
    puts $logFd "$num start =3D $start end =3D $end repos =3D $repos final =
=3D
$final char =3D $char"
}

Or: write 11 1's, read one char, write 11 2's read one char...

The chars read should be L, X, K, W, J.

The other result files look like this:

ABCDEFGHIJKLMNOPQRSTUVWXYABCDEFGHIJKLMNOPQRSTUVWXYABCDEF...
123456789012345678901234567890123456789012345678901234567890123456...
11111111111 <no newline>

ABCDEFGHIJKLMNOPQRSTUVWXYABCDEFGHIJKLMNOPQRSTUVWXYABCDEF...
123456789012345678901234567890123456789012345678901234567890123456...
1111111111122222222222333333333334444444444455555555555 <no newline>

11111111111LMNOPQRSTUVWXYABCDEFGHIJKLMNOPQRSTUVWXYABCDEFGHI...
123456789012345678901234567890123456789012345678901234567890123456...
22222222222333333333334444444444455555555555 <no newline>

11111111111LMNOPQRSTUVWXYABCDEFGHIJKLMNOPQRSTUVWXYABCDEFGHI...
123456789012345678901234567890123456789012345678901234567890123456...
22222222222 <no newline>

I'll try to get time to patch and test.

tom jackson



On Wed, Dec 2, 2009 at 1:33 PM, Alexandre Ferrieux
<ale...@gm...> wrote:
> On 12/2/09, Tom Jackson <tom...@gm...> wrote:
>>
>>  My assumption is that a correct bugfix would fix these incorrect results.
>>  To use the test code, edit and run the script then do:
>>  $ md5sum *.test | sort
>
> And what are we supposed to get ? Are your checksums the ones for
> current Tcl, or for ideal (correct) results ? Sorry, it seems you have
> done 95% of the work needed for a proper test suite (writing clearly
> at the end 'X tests failed out of Y'), but I lack the time for
> guesswork.
>
> Alternatively, you can grab the preliminary patch attached to the SF
> report, run your tests, and come back here with the results. In
> concise form please.
>
> -Alex
>

Re: [TCLCORE] Inconsistent buffered I/O

[Alexandre F. <ale...@gm...>]

On 12/2/09, Tom Jackson <tom...@gm...> wrote:
>
>  My assumption is that a correct bugfix would fix these incorrect results.
>  To use the test code, edit and run the script then do:
>  $ md5sum *.test | sort

And what are we supposed to get ? Are your checksums the ones for
current Tcl, or for ideal (correct) results ? Sorry, it seems you have
done 95% of the work needed for a proper test suite (writing clearly
at the end 'X tests failed out of Y'), but I lack the time for
guesswork.

Alternatively, you can grab the preliminary patch attached to the SF
report, run your tests, and come back here with the results. In
concise form please.

-Alex

Re: [TCLCORE] Inconsistent buffered I/O

[Tom J. <tom...@gm...>]

Here's the testing code and summary of results. See SUMMARY.TXT for more info.

http://rmadilo.com/files/tcl-io/

The bottom line is that correct operation requires [flush] prior to
read and [seek] prior to write, otherwise there are four categories of
incorrect results. (Probably more, but I only tested with -buffersize
20).

My assumption is that a correct bugfix would fix these incorrect results.

To use the test code, edit and run the script then do:

$ md5sum *.test | sort

tom jackson

On Tue, Dec 1, 2009 at 8:02 PM, Tom Jackson <tom...@gm...> wrote:
> On Sat, Nov 28, 2009 at 8:55 AM, Kevin Kenny <kev...@gm...> wrote:
>> In short:
>>  - It's a bug.
>>  - Alex was asking about whether programs depend on the current
>>    behaviour out of an abundance of caution.
>>  - We'll fix it.
>>  - The fix will have negligible performance impact.
>>  - It doesn't deserve nearly the amount of public debate that
>>    there's been.
>
> I guess there is no room for discussion here, and any attempt is seen
> as hostility. (Of course my previous message was hostile, sorry.)
>
> I'll start over and try to keep it short.
>
> 1. Assume this is a bug.
> 2. If so, the bug is bigger than described (this was my main point).
> 3. Nobody is affected by this bug because the effects are so obvious
> nobody could possibly rely on their code working.
> 4. Nobody will benefit from fixing this bug:
>  a. Anyone doing read/write already does what I suggest, or their code
> doesn't work.
>  b. Anyone not doing read/write obviously will not benefit from fixing the bug.
> 5. There are known and bulletproof "workarounds" for the bug, which
> completely fix the bug and which cost everyone else zero.
>
> Now, that being said. I also think:
> 1. The goal of the bug-fix is a worthwhile goal.
> 2. A bug-fix is possible which does not change the sequence or timing
> of any existing code executions. No buffers flushed too soon, or with
> less data than expected.
> 3. The bug-fix would require maintaining two bits (three states) of
> state, probably a bit field. Kevin suggested an additional struct
> member, which sounds perfect.
> 4. The fix would have some, but probably a very small, impact on
> performance (assuming the state maintaining idea is used).
> 5. The fix should not require messing around with the buffer code.
> 6. Seek or Flush will "clean" a fd. read/puts/gets will dirty one bit.
>
> Unknown:
> 1. At which level of API should this apply? Just the Tcl level? Or
> somewhere in the C API?
> 2. Some levels may require an API to maintain/check the state.
> Apparently solaris uses _fwriting and _freading to indicate the
> current "direction" of the buffer. But this would only be necessary
> for high level export where the struct isn't available. The benefit of
> this type of API is that the state can be maintained at a very low
> level (no API), yet the automation can be applied (if desired) at a
> higher level.
> 3. Why does [seek] in Tcl guarantee a [flush]? This seems like an
> expensive new requirement not found in stdio (but someone mentioned
> that a Tcl flush might not equal an os flush).
>
> Testing:
> 1. I have identified a number of factors which affect the result of
> multiple read/write cycles.
> 2. I will finish a test program (mostly done) which will make it easy
> to identify any missing edge cases or unexpected results.
> 3. Current testing indicates a flush/seek is always necessary when
> switching between read/write write/read.
> 4. Parameters include: file mode, buffering type, buffer size (even if
> buffering type is none), relationship of write/read size and buffer
> size.
>
> Anyway:
> I seldom do a very good job at communicating, and I often say things
> in such a way that others feel a need to respond. I can't promise to
> change and I don't expect others to respond differently, but I should
> at least state that I argue against or for ideas not individuals. I
> will just as easily turn on my own logic as yours. But any such
> turning will require some new information, not my desire to flatter
> myself or to curry favor with anyone else.
>
> The basic gold standard is code. It either works or it doesn't. Once
> it works, maybe you can make it faster or easier to maintain, but it
> still has to work. I'm also willing to trade speed for ease (in many
> different ways) of use. Otherwise I would not be using Tcl. Basically,
> Tcl is the only stand-alone scripting language which fully supports
> threading, synchronization and dozens of other necessary features for
> writing robust code. I've done a number of surveys over the years,
> trying to identify an alternative. None even approach the facilities
> available via Tcl.
>
> So this leads me to notice that there exist two distinct application
> models in Tcl. One is the event based model and the other is the
> thread based model. Because of the actually very good design of Tcl's
> event model, event based applications never need to consider
> synchronization issues. Thread model application programmers have no
> alternative but to obsess over this issue. In fact, they tend to
> obsess over synchronization at every possible level. One of these
> levels is self-synchronization. And that is the level which involves
> the current bug. Reading and writing to the same file is just
> something which would never occur to this second group of programmers,
> including myself. In the back, or even the forefront of our minds is
> the concept of an atomic operation. At most one transaction is lost in
> case of application failure. But if you read and write to the same
> file, regardless of how bug-free your code, you always risk losing
> everything. From this perspective opening a file r+ is already a bug.
> You don't need any additional information, no code analysis necessary.
> The application is destined to fail at some point. There is no way to
> guarantee that the level of failure will be confined to an application
> defined transaction. The entire file could be lost, or a failure could
> leave an apparently valid file, with no way to tell the difference.
>
>
> tom jackson
>

Re: [TCLCORE] Inconsistent buffered I/O

[Tom J. <tom...@gm...>]

On Sat, Nov 28, 2009 at 8:55 AM, Kevin Kenny <kev...@gm...> wrote:
> In short:
>  - It's a bug.
>  - Alex was asking about whether programs depend on the current
>    behaviour out of an abundance of caution.
>  - We'll fix it.
>  - The fix will have negligible performance impact.
>  - It doesn't deserve nearly the amount of public debate that
>    there's been.

I guess there is no room for discussion here, and any attempt is seen
as hostility. (Of course my previous message was hostile, sorry.)

I'll start over and try to keep it short.

1. Assume this is a bug.
2. If so, the bug is bigger than described (this was my main point).
3. Nobody is affected by this bug because the effects are so obvious
nobody could possibly rely on their code working.
4. Nobody will benefit from fixing this bug:
 a. Anyone doing read/write already does what I suggest, or their code
doesn't work.
 b. Anyone not doing read/write obviously will not benefit from fixing the bug.
5. There are known and bulletproof "workarounds" for the bug, which
completely fix the bug and which cost everyone else zero.

Now, that being said. I also think:
1. The goal of the bug-fix is a worthwhile goal.
2. A bug-fix is possible which does not change the sequence or timing
of any existing code executions. No buffers flushed too soon, or with
less data than expected.
3. The bug-fix would require maintaining two bits (three states) of
state, probably a bit field. Kevin suggested an additional struct
member, which sounds perfect.
4. The fix would have some, but probably a very small, impact on
performance (assuming the state maintaining idea is used).
5. The fix should not require messing around with the buffer code.
6. Seek or Flush will "clean" a fd. read/puts/gets will dirty one bit.

Unknown:
1. At which level of API should this apply? Just the Tcl level? Or
somewhere in the C API?
2. Some levels may require an API to maintain/check the state.
Apparently solaris uses _fwriting and _freading to indicate the
current "direction" of the buffer. But this would only be necessary
for high level export where the struct isn't available. The benefit of
this type of API is that the state can be maintained at a very low
level (no API), yet the automation can be applied (if desired) at a
higher level.
3. Why does [seek] in Tcl guarantee a [flush]? This seems like an
expensive new requirement not found in stdio (but someone mentioned
that a Tcl flush might not equal an os flush).

Testing:
1. I have identified a number of factors which affect the result of
multiple read/write cycles.
2. I will finish a test program (mostly done) which will make it easy
to identify any missing edge cases or unexpected results.
3. Current testing indicates a flush/seek is always necessary when
switching between read/write write/read.
4. Parameters include: file mode, buffering type, buffer size (even if
buffering type is none), relationship of write/read size and buffer
size.

Anyway:
I seldom do a very good job at communicating, and I often say things
in such a way that others feel a need to respond. I can't promise to
change and I don't expect others to respond differently, but I should
at least state that I argue against or for ideas not individuals. I
will just as easily turn on my own logic as yours. But any such
turning will require some new information, not my desire to flatter
myself or to curry favor with anyone else.

The basic gold standard is code. It either works or it doesn't. Once
it works, maybe you can make it faster or easier to maintain, but it
still has to work. I'm also willing to trade speed for ease (in many
different ways) of use. Otherwise I would not be using Tcl. Basically,
Tcl is the only stand-alone scripting language which fully supports
threading, synchronization and dozens of other necessary features for
writing robust code. I've done a number of surveys over the years,
trying to identify an alternative. None even approach the facilities
available via Tcl.

So this leads me to notice that there exist two distinct application
models in Tcl. One is the event based model and the other is the
thread based model. Because of the actually very good design of Tcl's
event model, event based applications never need to consider
synchronization issues. Thread model application programmers have no
alternative but to obsess over this issue. In fact, they tend to
obsess over synchronization at every possible level. One of these
levels is self-synchronization. And that is the level which involves
the current bug. Reading and writing to the same file is just
something which would never occur to this second group of programmers,
including myself. In the back, or even the forefront of our minds is
the concept of an atomic operation. At most one transaction is lost in
case of application failure. But if you read and write to the same
file, regardless of how bug-free your code, you always risk losing
everything. From this perspective opening a file r+ is already a bug.
You don't need any additional information, no code analysis necessary.
The application is destined to fail at some point. There is no way to
guarantee that the level of failure will be confined to an application
defined transaction. The entire file could be lost, or a failure could
leave an apparently valid file, with no way to tell the difference.


tom jackson

Re: [TCLCORE] Inconsistent buffered I/O

[Andreas K. <aku...@sh...>]

Guess I should speak up as well, given that I asked Alexandre to post
the start of the thread, instead of me doing it.

So, a big thanks to Alexandre for posting about the problem to
tclcore, and then dealing with the responses, and also thanks to all
the other participiants in the discussion.

> But let me try to put things in perspective.

Thank you. Reading the thread I got sometimes a bit lost in the
tangents.

> I'm not claiming that twenty years of Tcl'ers are fools for not
> finding this: I've been actively developing Tcl for by far the
> better part of those twenty years, and on the TCT for nearly
> half of them. And every few months, I raise an eyebrow and
> say to myself, "Hmm, peculiar that nobody ever discovered *that*
> bug before." I don't see that as playing Chicken Little.

I see similar things in my own code, not necessarily Tcl core,
i.e. 'what the * was I thinking when I wrote _this_' ?.

There was, for me, another reason behind querying the list about
this. This is old code. Its design considerations are not written up
in TIPs. I had partly hoped that some of the oldtimers would speak up
and dump their memory of the design on us. I.e. maybe this behaviour
was deliberate and we, the current maintainers, simply do not know the
reasons for that any longer. Nothing come out of that hope, still,
that is ok.

> It's more or less:

Good write up, although, we should be used to that from you by now,
Kevin. ;)
 
>   - Prior to a [read] (equivalently, a [gets]), check if there
>     are buffered data to write. If there are, write them before
>     reading the file. (Obviously, any data in read buffers may
>     be retained.)
> 
>   - Prior to a [seek], check if there are buffered data to
>     write. If there are, write them before seeking.

This actually happens currently, if we are not in the weird situation
of having both buffered in- and output. This (sanity check) is what
got me thinking in the first place that the original designers knew
about the problem and the code as-is possibly deliberate. Of course,
it may have also been a simple band-aid on something they bailed out
on, maybe even forgotten later.

>     There's an obvious performance tweak to bypass this flushing
>     behaviour if the [seek] is not actually moving the cursor,
>     and further performance might be obtainable if the library
>     can determine that the new cursor is inside the current
>     buffer. For now, I propose to ignore those optimizations,
>     which represent an unusual case that requires a fair bit
>     of bookkeeping to maintain consistency.

A much nicer phrasing of that I said to Alexandre in private chat
about why I prefered (2a) over (2b). No, I was not swearing, just not
as elaborate and formal.

>     As far as input buffers go, I'd propose flushing them on
>     [seek] as well, but it wouldn't be a tremendous amount of
>     bookkeeping to detect the case where the new cursor lies
>     within the buffer and adjust accordingly. This issue might
>     be worth doing, because there's a plausible case where a
>     program will read data from a file, use the data just read
>     to infer the file's encoding, seek back to the beginning,
>     configure a new encoding and read again.
> 
>   - Prior to a [puts], spoil any buffered input data, which will
>     be incoherent after the write.

I had not thought about this when talking with Alexandre and writing
up my example, and thank you for pointing out the symmetry, so obvious
now in my hindsight.

>  If the current cursor lies
>     within an input buffer, it might be possible to retain the
>     buffer by overwriting the data therein with the newly-written
>     data. Again, I propose simply to ignore the case.
> 
>   - Prior to a [truncate], flush write buffers and discard read
>     buffers.
> 
> There are no "additional bits of state" involved, because the
> "safe to read" and "safe to write" states can be inferred merely
> from the existence of buffered data.

Agreed.
 
> In short:
>    - It's a bug.
>    - Alex was asking about whether programs depend on the current
>      behaviour out of an abundance of caution.
>    - We'll fix it.
>    - The fix will have negligible performance impact.
>    - It doesn't deserve nearly the amount of public debate that
>      there's been.

-- 
So long,
	Andreas Kupries <aku...@sh...>
			<http://www.purl.org/NET/akupries/>
	Developer @	<http://www.activestate.com/>
-------------------------------------------------------------------------------

Re: [TCLCORE] Inconsistent buffered I/O

[Kevin K. <kev...@gm...>]

I'm not quoting Tom's parent post here, because it's long and a
bit rambling.  But let me try to put things in perspective.
I'm not claiming that twenty years of Tcl'ers are fools for not
finding this: I've been actively developing Tcl for by far the
better part of those twenty years, and on the TCT for nearly
half of them. And every few months, I raise an eyebrow and
say to myself, "Hmm, peculiar that nobody ever discovered *that*
bug before." I don't see that as playing Chicken Little.

I don't think that you, Tom, have comprehended the proposed
solution entirely.  It's more or less:

  - Prior to a [read] (equivalently, a [gets]), check if there
    are buffered data to write. If there are, write them before
    reading the file. (Obviously, any data in read buffers may
    be retained.)

  - Prior to a [seek], check if there are buffered data to
    write. If there are, write them before seeking.

    There's an obvious performance tweak to bypass this flushing
    behaviour if the [seek] is not actually moving the cursor,
    and further performance might be obtainable if the library
    can determine that the new cursor is inside the current
    buffer. For now, I propose to ignore those optimizations,
    which represent an unusual case that requires a fair bit
    of bookkeeping to maintain consistency.

    As far as input buffers go, I'd propose flushing them on
    [seek] as well, but it wouldn't be a tremendous amount of
    bookkeeping to detect the case where the new cursor lies
    within the buffer and adjust accordingly. This issue might
    be worth doing, because there's a plausible case where a
    program will read data from a file, use the data just read
    to infer the file's encoding, seek back to the beginning,
    configure a new encoding and read again.

  - Prior to a [puts], spoil any buffered input data, which will
    be incoherent after the write.  If the current cursor lies
    within an input buffer, it might be possible to retain the
    buffer by overwriting the data therein with the newly-written
    data. Again, I propose simply to ignore the case.

  - Prior to a [truncate], flush write buffers and discard read
    buffers.

There are no "additional bits of state" involved, because the
"safe to read" and "safe to write" states can be inferred merely
from the existence of buffered data.

You'll also notice that nowhere is this proposing to use the
'flush' facility of the operating system (unless the program
performs an explicit [flush]), so in no case should the operating
system's buffer cache be spoilt unless the program asks for it
explicitly.

You will also notice that this discussion is treating read and
write buffers separately. The -buffering option, by the way,
affects only write buffers. It controls whether a kernel-level
write is requested on every [puts] (-buffering none), on every
newline character (-buffering line), or only when a buffer fills
(-buffering full).

I do concede that the current behaviour is documented. We've
made the mistake of documenting bugs before.  Removing the
need for a workaround is not "coddling novice programmers."
No language makes it the least bit difficult to write incorrect
programs. But we do try as far as we can to make it easier
to write correct ones.

In short:
   - It's a bug.
   - Alex was asking about whether programs depend on the current
     behaviour out of an abundance of caution.
   - We'll fix it.
   - The fix will have negligible performance impact.
   - It doesn't deserve nearly the amount of public debate that
     there's been.

-- 
73 de ke9tv/2, Kevin