tcl-core — Tcl Core Team discussions about Tcl development

Re: [TCLCORE] tcltls: unexpected 0-bytes [read], then nothing

[Brian O'h. <bri...@co...>]

Thanks. So make sure to use the tls-1.8 branch. It has all of my changes including adding back the missed debug option. Jan is working in the trunk and a few other branches, so he’ll need to chime in on his changes there.

 

tls-1.8 branch at https://core.tcl-lang.org/tcltls

 

I’ve also added some comments to the chat too, so see that for more, but the bottom-line is try and capture as much data as you can so I’ll have something to look at. The debug option now works, but another good thing is logging the data from the –command option especially if your OpenSSL has the -enable-ssl-trace option enabled. Using the –validate_command option will tell you if there are certificate errors. It calls the Tls_NewX509Obj function to get the certificate info, so that’s why you may see that.

 

From: Pietro Cerutti via Tcl-Core [mailto:tcl...@li...] 
Sent: Wednesday, June 05, 2024 1:53 PM
To: Tcl Core List
Cc: Pietro Cerutti
Subject: Re: [TCLCORE] tcltls: unexpected 0-bytes [read], then nothing

 

 

On 5 Jun 2024, at 20:27, Brian O’Hagan <bri...@co...> wrote:

Can you confirm which version of tcltls you are using? If from the repo, which date/commit. Thanks.

 

This happens both with 1.7.22 with Tcl 8.6.14 and current trunk (216954cfae671c32b58b5b562f7fa5054d979fc7) with both Tcl 8.6.15 and 9.0b2

 

-- 

Pietro Cerutti

I've pledged to give 10% of income to effective charities and invite you to join me.

https://givingwhatwecan.org

 

Sent from a small device - please excuse brevity and typos.

Re: [TCLCORE] CFV: TIP #696

[Rolf A. <tcl...@po...>]

apnmbx-public--- via Tcl-Core writes:
> Regarding tcllib, yes, some packages will have to be touched. Will it help
> if we reduce the range of Tcl reserved codes as @mjanssen suggested on the
> chat to say, -0xFFFF:0xFFFF ?

Well, I raised head only because it was so easy to find examples in
public available code as

https://core.tcl-lang.org/tcllib/file?udc=1&ln=2770&ci=trunk&name=modules%2Fmime%2Fmime.tcl

https://core.tcl-lang.org/tcllib/file?udc=1&ln=182&ci=trunk&name=modules%2Fstruct%2Ftree_tcl.tcl

Reducing the range of Tcl reserved codes to -0xFFFF:0xFFFF would not
help for that cases.

A much smaller reserved range is appealing not because it would milden
migration needs but it would extension writers relieve from the need to
use ridiculously long own return codes.

rolf

Re: [TCLCORE] tcltls: unexpected 0-bytes [read], then nothing

[Pietro C. <ga...@ga...>]

> On 5 Jun 2024, at 20:27, Brian O’Hagan <bri...@co...> wrote:
> Can you confirm which version of tcltls you are using? If from the repo, which date/commit. Thanks.

This happens both with 1.7.22 with Tcl 8.6.14 and current trunk (216954cfae671c32b58b5b562f7fa5054d979fc7) with both Tcl 8.6.15 and 9.0b2

-- 
Pietro Cerutti
I've pledged to give 10% of income to effective charities and invite you to join me.
https://givingwhatwecan.org

Sent from a small device - please excuse brevity and typos.

Re: [TCLCORE] tcltls: unexpected 0-bytes [read], then nothing

[Brian O’H. <bri...@co...>]

Can you confirm which version of tcltls you are using? If from the repo, which date/commit. Thanks.


> On Jun 5, 2024, at 10:21 AM, Pietro Cerutti via Tcl-Core <tcl...@li...> wrote:
> 
> On Jun 05 2024, 15:11 UTC, msc...@po... wrote:
>> Hi,
>> 
>> just took a short look at the bug description.
>> "Additionally, a still unexplained constant of 16384 bytes plays a role in the pattern."
>> 
>> your unexplained 16384 bytes is probably the frame size used by TLS, which is exactly 16kB.
>> 
>> I would guess that the fileevent fires, Tcl-TLS then uses the openssl layer to decrypt the received bytes and ends up with a half decoded state, as not enough bytes have been read to emit any cleartext yet, see the state descriptions at https://www.openssl.org/docs/man3.3/man3/SSL_want.html especially SSL_READING.
>> 
>> So you might end up with a fileevent that detects the socket is readable, but all the bytes read are consumed to drive the internal OpenSSL CTX state forward, but does not produce any output bytes.
>> 
>> The "Head of Line" blocking image here describes the issue:
>> 
>> https://www.igvita.com/2013/10/24/optimizing-tls-record-size-and-buffering-latency/
> 
> Thanks for the insigh!
> 
>> No idea how this should be handled properly, guess an empty read has to be expected from a TLS channel and does not signal EOF in that case.
> 
> An empty read is fine, we handle it nicely, as long as another "full" read eventually comes in, which we aren't seeing in all cases.
> 
> Is it possible that the TLS layer isn't fireing off a second event when the state is eventually fully decoded?
> 
> 
> --
> Pietro Cerutti
> I have pledged to give 10% of income to effective charities
> and invite you to join me - https://givingwhatwecan.org
> 
> 
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] CFV: TIP #696

[<apn...@ya...>]

Hey,  I thought I was generous, not greedy, allowing 2 billion code values for extensions 😊

 

Jokes aside, I think your comments are similar to @mjanssen’s chat I posted earlier. How about this then, allocate all return codes in the range 0x100 (256) : 0x3FFFFFFF to extensions. All others, including negative values belong to Tcl.

 

Would that be acceptable to all?

 

/Ashok

 

 

From: Donald Arseneau <as...@tr...> 
Sent: Wednesday, June 5, 2024 10:22 AM
To: tcl...@li...
Subject: Re: [TCLCORE] CFV: TIP #696

 

Sounds fine in principle. But OUCH that is so greedy!  Reserving over 2 billion code values as potentially defined by the language, and condemning programmers to use 10-digit private code values in their programs. I was expecting increasing the defined number from 5 to 16 or 32, but surely no more than 99.

 

If you want to prepare for a future need to dynamically create millions of return codes, how about reserving the BIG numbers for Tcl itself, along with a very small set of numbers near zero?

 

--Donald

Re: [TCLCORE] CFV: TIP #696

[<apn...@ya...>]

Hey,  I thought I was generous, not greedy, allowing 2 billion code values for extensions 😊

 

Jokes aside, I think your comments are similar to @mjanssen’s chat I posted earlier. How about this then, allocate all return codes in the range 0x100 (256) : 0x3FFFFFFF to extensions. All others, including negative values belong to Tcl.

 

Would that be acceptable to all?

 

/Ashok

 

From: Donald Arseneau <as...@tr...> 
Sent: Wednesday, June 5, 2024 10:22 AM
To: tcl...@li...
Subject: Re: [TCLCORE] CFV: TIP #696

 

Sounds fine in principle. But OUCH that is so greedy!  Reserving over 2 billion code values as potentially defined by the language, and condemning programmers to use 10-digit private code values in their programs. I was expecting increasing the defined number from 5 to 16 or 32, but surely no more than 99.

 

If you want to prepare for a future need to dynamically create millions of return codes, how about reserving the BIG numbers for Tcl itself, along with a very small set of numbers near zero?

 

--Donald

Re: [TCLCORE] tcltls: unexpected 0-bytes [read], then nothing

[<msc...@po...>]

Am 05.06.2024 17:20 schrieb Pietro Cerutti:
> On Jun 05 2024, 15:11 UTC, msc...@po... wrote:

> 
>> No idea how this should be handled properly, guess an empty read has 
>> to be expected from a TLS channel and does not signal EOF in that 
>> case.
> 
> An empty read is fine, we handle it nicely, as long as another "full"
> read eventually comes in, which we aren't seeing in all cases.
> 
> Is it possible that the TLS layer isn't fireing off a second event
> when the state is eventually fully decoded?

No idea, not looked at the TclTLS code for maybe 10 yrs...

This looks a little suspicious though:
https://core.tcl-lang.org/tcltls/annotate?filename=generic/tlsIO.c&checkin=trunk

around line 767:

     if ((mask & TCL_READABLE) &&
	((Tcl_InputBuffered(statePtr->self) > 0) || 
(BIO_ctrl_pending(statePtr->bio) > 0))) {

the flush only happens if data is buffered in the Tcl side or if CTRL 
stuff is pending, but it does not trigger the flush for 
BIO_pending(statePtr->bio) > 0.

You could add some logging there or try to add a "|| 
BIO_pending(statePtr->bio) > 0" and see what you get.

Michael Schlenker

Re: [TCLCORE] tcltls: unexpected 0-bytes [read], then nothing

[Gustaf N. <ne...@wu...>]

You might also check the following ticket, dealing also with 0-byte reads

https://core.tcl-lang.org/tcltls/tktview/88c0c84969

Below is a diff against the current version of tcltls from the fossil 
repository.

Error handling is not very developed in tcltls.

Hope this helps

-g


--- generic/tlsIO.c-orig	2024-06-05 18:38:51
+++ generic/tlsIO.c	2024-06-05 18:40:44
@@ -347,6 +347,18 @@
      ERR_clear_error();
      bytesRead = BIO_read(statePtr->bio, buf, bufSize);
      dprintf("BIO_read -> %d", bytesRead);
+
+    if (bytesRead == 0 && Tcl_Eof(statePtr->self)) {
+        /*
+         * We know through BIO_CTRL_EOF that we are already at
+         * EOF (determined during BIO_read()). There is no need to
+         * try to handle this situation via error and reason codes
+         * from OpenSSL.
+         */
+         dprintf("tried to read while channel is already at EOF");
+         *errorCodePtr = 0;
+         return(bytesRead);
+    }
  
      err = SSL_get_error(statePtr->ssl, bytesRead);
      backingError = ERR_get_error();

On 05.06.24 15:22, Pietro Cerutti via Tcl-Core wrote:
> Hi,
>
> I and Colin have been investigating an issue when using the TLS 
> extension (https://core.tcl-lang.org/tcltls) to interact with a Redis 
> server using retcl (https://code.ptrcrt.ch/retcl/).
>
> I have a small reproducer here, which I have trimmed down to a 
> minimum.  It doesn't include any retcl code, just TLS sockets:
> https://gist.github.com/gahr/a9258f0f9cdd20ec0479a409e0db49fd
>
> The protocol is line-oriented, with \r\n as terminators.
>
> The setup is as follows:
>
> 1. we connect to a Redis instance using [tls::socket]
> 2. we [chan configure $sock -blocking 0 -translation binary]
> 3. we send a command string, and we [chan flush $sock]
> 4. we get a read event on the $sock
> 5. we [read $sock] in the event handler and expect to get a response
>
> Point 5. works fine against a local Redis instance w/o TLS.
>
> When we turn TLS on, we see weird things:
>
> 1. we get a number of read events where [read $sock] returns ""
> 2. sometimes we get meaningful read events afterwards, where a [read 
> $sock] returns the expected response
> 3. sometimes we don't get any meaningful read events after the empty ones
>
> Under the gist, I have included two comments with sample runs against 
> my local Redis server and against an intance on upstash.com. You can 
> see the randomness of the empty / meaningful responses.
>
> I can provide the output when tcltls is compiled with 
> -DTCLEXT_TCLTLS_DEBUG, if anyone wants to take a look.
>
> Any pointers would be appreciated.
>

Re: [TCLCORE] tcltls: unexpected 0-bytes [read], then nothing

[Pietro C. <ga...@ga...>]

On Jun 05 2024, 15:11 UTC, msc...@po... wrote:
>Hi,
>
>just took a short look at the bug description.
>"Additionally, a still unexplained constant of 16384 bytes plays a 
>role in the pattern."
>
>your unexplained 16384 bytes is probably the frame size used by TLS, 
>which is exactly 16kB.
>
>I would guess that the fileevent fires, Tcl-TLS then uses the openssl 
>layer to decrypt the received bytes and ends up with a half decoded 
>state, as not enough bytes have been read to emit any cleartext yet, 
>see the state descriptions at 
>https://www.openssl.org/docs/man3.3/man3/SSL_want.html especially 
>SSL_READING.
>
>So you might end up with a fileevent that detects the socket is 
>readable, but all the bytes read are consumed to drive the internal 
>OpenSSL CTX state forward, but does not produce any output bytes.
>
>The "Head of Line" blocking image here describes the issue:
>
>https://www.igvita.com/2013/10/24/optimizing-tls-record-size-and-buffering-latency/

Thanks for the insigh!

>No idea how this should be handled properly, guess an empty read has to 
>be expected from a TLS channel and does not signal EOF in that case.

An empty read is fine, we handle it nicely, as long as another "full" 
read eventually comes in, which we aren't seeing in all cases.

Is it possible that the TLS layer isn't fireing off a second event when 
the state is eventually fully decoded?


-- 
Pietro Cerutti
I have pledged to give 10% of income to effective charities
and invite you to join me - https://givingwhatwecan.org

Re: [TCLCORE] tcltls: unexpected 0-bytes [read], then nothing

[<msc...@po...>]

Hi,

just took a short look at the bug description.
"Additionally, a still unexplained constant of 16384 bytes plays a role 
in the pattern."

your unexplained 16384 bytes is probably the frame size used by TLS, 
which is exactly 16kB.

I would guess that the fileevent fires, Tcl-TLS then uses the openssl 
layer to decrypt the received bytes and ends up with a half decoded 
state, as not enough bytes have been read to emit any cleartext yet, see 
the state descriptions at 
https://www.openssl.org/docs/man3.3/man3/SSL_want.html especially 
SSL_READING.

So you might end up with a fileevent that detects the socket is 
readable, but all the bytes read are consumed to drive the internal 
OpenSSL CTX state forward, but does not produce any output bytes.

The "Head of Line" blocking image here describes the issue:

https://www.igvita.com/2013/10/24/optimizing-tls-record-size-and-buffering-latency/

No idea how this should be handled properly, guess an empty read has to 
be expected from a TLS channel and does not signal EOF in that case.

Regards,
Michael Schlenker

Am 05.06.2024 16:08 schrieb elns:
> On 6/5/24 15:22, Pietro Cerutti via Tcl-Core wrote:
> 
>> 4. we get a read event on the $sock
>> 5. we [read $sock] in the event handler and expect to get a response
>> 
>> Point 5. works fine against a local Redis instance w/o TLS.
>> 
>> When we turn TLS on, we see weird things:
>> 
>> 1. we get a number of read events where [read $sock] returns ""
>> 2. sometimes we get meaningful read events afterwards, where a [read 
>> $sock] returns the expected response
>> 3. sometimes we don't get any meaningful read events after the empty 
>> ones
> 
> Reminds me of:
> 
>     https://sourceforge.net/p/tls/bugs/38/
> 
> Regards,
> Erik Leunissen
> --
> 
> 
>> 
>> Under the gist, I have included two comments with sample runs against 
>> my local Redis server and against an intance on upstash.com. You can 
>> see the randomness of the empty / meaningful responses.
>> 
>> I can provide the output when tcltls is compiled with 
>> -DTCLEXT_TCLTLS_DEBUG, if anyone wants to take a look.
>> 
>> Any pointers would be appreciated.
>> 
> 
> 
> 
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] tcltls: unexpected 0-bytes [read], then nothing

[Harald O. <har...@el...>]

Am 05.06.2024 um 16:57 schrieb Pietro Cerutti:
> On Jun 05 2024, 14:33 UTC, Harald Oehlmann <har...@el...> 
> wrote:
>> Am 05.06.2024 um 15:22 schrieb Pietro Cerutti via Tcl-Core:
>>> Hi,
>>>
>>> I and Colin have been investigating an issue when using the TLS 
>>> extension (https://core.tcl-lang.org/tcltls) to interact with a Redis 
>>> server using retcl (https://code.ptrcrt.ch/retcl/).
>>>
>>> I have a small reproducer here, which I have trimmed down to a 
>>> minimum. It doesn't include any retcl code, just TLS sockets:
>>> https://gist.github.com/gahr/a9258f0f9cdd20ec0479a409e0db49fd
>>>
>>> The protocol is line-oriented, with \r\n as terminators.
>>>
>>> The setup is as follows:
>>>
>>> 1. we connect to a Redis instance using [tls::socket]
>>> 2. we [chan configure $sock -blocking 0 -translation binary]
>>> 3. we send a command string, and we [chan flush $sock]
>>> 4. we get a read event on the $sock
>>> 5. we [read $sock] in the event handler and expect to get a response
>>>
>>> Point 5. works fine against a local Redis instance w/o TLS.
>>>
>>> When we turn TLS on, we see weird things:
>>>
>>> 1. we get a number of read events where [read $sock] returns ""
>>> 2. sometimes we get meaningful read events afterwards, where a [read 
>>> $sock] returns the expected response
>>> 3. sometimes we don't get any meaningful read events after the empty 
>>> ones
>>>
>>> Under the gist, I have included two comments with sample runs against 
>>> my local Redis server and against an intance on upstash.com. You can 
>>> see the randomness of the empty / meaningful responses.
>>>
>>> I can provide the output when tcltls is compiled with 
>>> -DTCLEXT_TCLTLS_DEBUG, if anyone wants to take a look.
>>>
>>> Any pointers would be appreciated.
>>>
>>
>> Pietro,
>> is this TCL 8.7a0 until 9.0b2 ?
>> If yes, please try the current main branch.
> 
> Hi Harald,
> 
> this happens with 8.6.14, 9.0b2, and trunk as of ae52f61f0c. Of course 
> TLS is compiled against the relevant version.
> 
> Thanks

Thank you, was just a guess.
Take care,
Harald

Re: [TCLCORE] tcltls: unexpected 0-bytes [read], then nothing

[Pietro C. <ga...@ga...>]

On Jun 05 2024, 14:08 UTC, elns <el...@xs...> wrote:
>On 6/5/24 15:22, Pietro Cerutti via Tcl-Core wrote:
>
>>4. we get a read event on the $sock
>>5. we [read $sock] in the event handler and expect to get a response
>>
>>Point 5. works fine against a local Redis instance w/o TLS.
>>
>>When we turn TLS on, we see weird things:
>>
>>1. we get a number of read events where [read $sock] returns ""
>>2. sometimes we get meaningful read events afterwards, where a [read 
>>$sock] returns the expected response
>>3. sometimes we don't get any meaningful read events after the empty ones
>
>Reminds me of:
>
>    https://sourceforge.net/p/tls/bugs/38/

That looks very similar indeed, thanks!

-- 
Pietro Cerutti
I have pledged to give 10% of income to effective charities
and invite you to join me - https://givingwhatwecan.org

Re: [TCLCORE] tcltls: unexpected 0-bytes [read], then nothing

[Pietro C. <ga...@ga...>]

On Jun 05 2024, 14:33 UTC, Harald Oehlmann <har...@el...> wrote:
>Am 05.06.2024 um 15:22 schrieb Pietro Cerutti via Tcl-Core:
>>Hi,
>>
>>I and Colin have been investigating an issue when using the TLS 
>>extension (https://core.tcl-lang.org/tcltls) to interact with a 
>>Redis server using retcl (https://code.ptrcrt.ch/retcl/).
>>
>>I have a small reproducer here, which I have trimmed down to a 
>>minimum. It doesn't include any retcl code, just TLS sockets:
>>https://gist.github.com/gahr/a9258f0f9cdd20ec0479a409e0db49fd
>>
>>The protocol is line-oriented, with \r\n as terminators.
>>
>>The setup is as follows:
>>
>>1. we connect to a Redis instance using [tls::socket]
>>2. we [chan configure $sock -blocking 0 -translation binary]
>>3. we send a command string, and we [chan flush $sock]
>>4. we get a read event on the $sock
>>5. we [read $sock] in the event handler and expect to get a response
>>
>>Point 5. works fine against a local Redis instance w/o TLS.
>>
>>When we turn TLS on, we see weird things:
>>
>>1. we get a number of read events where [read $sock] returns ""
>>2. sometimes we get meaningful read events afterwards, where a [read 
>>$sock] returns the expected response
>>3. sometimes we don't get any meaningful read events after the empty ones
>>
>>Under the gist, I have included two comments with sample runs 
>>against my local Redis server and against an intance on upstash.com. 
>>You can see the randomness of the empty / meaningful responses.
>>
>>I can provide the output when tcltls is compiled with 
>>-DTCLEXT_TCLTLS_DEBUG, if anyone wants to take a look.
>>
>>Any pointers would be appreciated.
>>
>
>Pietro,
>is this TCL 8.7a0 until 9.0b2 ?
>If yes, please try the current main branch.

Hi Harald,

this happens with 8.6.14, 9.0b2, and trunk as of ae52f61f0c. Of course 
TLS is compiled against the relevant version.

Thanks


-- 
Pietro Cerutti
I have pledged to give 10% of income to effective charities
and invite you to join me - https://givingwhatwecan.org

Re: [TCLCORE] tcltls: unexpected 0-bytes [read], then nothing

[Colin M. <col...@ya...>]

On 05/06/2024 15:33, Harald Oehlmann wrote:
> Am 05.06.2024 um 15:22 schrieb Pietro Cerutti via Tcl-Core:
>> I and Colin have been investigating an issue when using the TLS 
>> extension (https://core.tcl-lang.org/tcltls) to interact with a Redis 
>> server using retcl (https://code.ptrcrt.ch/retcl/).
> Pietro,
> is this TCL 8.7a0 until 9.0b2 ?
> If yes, please try the current main branch.
>
I see the same issue with 8.6 and 9.0b1, I have not tried a newer 
version so far.  Is there some recent change that would affect this?

Colin.

Re: [TCLCORE] tcltls: unexpected 0-bytes [read], then nothing

[Harald O. <har...@el...>]

Am 05.06.2024 um 15:22 schrieb Pietro Cerutti via Tcl-Core:
> Hi,
> 
> I and Colin have been investigating an issue when using the TLS 
> extension (https://core.tcl-lang.org/tcltls) to interact with a Redis 
> server using retcl (https://code.ptrcrt.ch/retcl/).
> 
> I have a small reproducer here, which I have trimmed down to a minimum. 
> It doesn't include any retcl code, just TLS sockets:
> https://gist.github.com/gahr/a9258f0f9cdd20ec0479a409e0db49fd
> 
> The protocol is line-oriented, with \r\n as terminators.
> 
> The setup is as follows:
> 
> 1. we connect to a Redis instance using [tls::socket]
> 2. we [chan configure $sock -blocking 0 -translation binary]
> 3. we send a command string, and we [chan flush $sock]
> 4. we get a read event on the $sock
> 5. we [read $sock] in the event handler and expect to get a response
> 
> Point 5. works fine against a local Redis instance w/o TLS.
> 
> When we turn TLS on, we see weird things:
> 
> 1. we get a number of read events where [read $sock] returns ""
> 2. sometimes we get meaningful read events afterwards, where a [read 
> $sock] returns the expected response
> 3. sometimes we don't get any meaningful read events after the empty ones
> 
> Under the gist, I have included two comments with sample runs against my 
> local Redis server and against an intance on upstash.com. You can see 
> the randomness of the empty / meaningful responses.
> 
> I can provide the output when tcltls is compiled with 
> -DTCLEXT_TCLTLS_DEBUG, if anyone wants to take a look.
> 
> Any pointers would be appreciated.
> 

Pietro,
is this TCL 8.7a0 until 9.0b2 ?
If yes, please try the current main branch.

Take care,
Harald

Re: [TCLCORE] tcltls: unexpected 0-bytes [read], then nothing

[elns <el...@xs...>]

On 6/5/24 15:22, Pietro Cerutti via Tcl-Core wrote:

> 4. we get a read event on the $sock
> 5. we [read $sock] in the event handler and expect to get a response
> 
> Point 5. works fine against a local Redis instance w/o TLS.
> 
> When we turn TLS on, we see weird things:
> 
> 1. we get a number of read events where [read $sock] returns ""
> 2. sometimes we get meaningful read events afterwards, where a [read $sock] returns the expected 
> response
> 3. sometimes we don't get any meaningful read events after the empty ones

Reminds me of:

     https://sourceforge.net/p/tls/bugs/38/

Regards,
Erik Leunissen
--


> 
> Under the gist, I have included two comments with sample runs against my local Redis server and 
> against an intance on upstash.com. You can see the randomness of the empty / meaningful responses.
> 
> I can provide the output when tcltls is compiled with -DTCLEXT_TCLTLS_DEBUG, if anyone wants to take 
> a look.
> 
> Any pointers would be appreciated.
> 

[TCLCORE] tcltls: unexpected 0-bytes [read], then nothing

[Pietro C. <ga...@ga...>]

Hi,

I and Colin have been investigating an issue when using the TLS 
extension (https://core.tcl-lang.org/tcltls) to interact with a Redis 
server using retcl (https://code.ptrcrt.ch/retcl/).

I have a small reproducer here, which I have trimmed down to a minimum.  
It doesn't include any retcl code, just TLS sockets:
https://gist.github.com/gahr/a9258f0f9cdd20ec0479a409e0db49fd

The protocol is line-oriented, with \r\n as terminators.

The setup is as follows:

1. we connect to a Redis instance using [tls::socket]
2. we [chan configure $sock -blocking 0 -translation binary]
3. we send a command string, and we [chan flush $sock]
4. we get a read event on the $sock
5. we [read $sock] in the event handler and expect to get a response

Point 5. works fine against a local Redis instance w/o TLS.

When we turn TLS on, we see weird things:

1. we get a number of read events where [read $sock] returns ""
2. sometimes we get meaningful read events afterwards, where a [read $sock] returns the expected response
3. sometimes we don't get any meaningful read events after the empty ones

Under the gist, I have included two comments with sample runs against my 
local Redis server and against an intance on upstash.com. You can see 
the randomness of the empty / meaningful responses.

I can provide the output when tcltls is compiled with 
-DTCLEXT_TCLTLS_DEBUG, if anyone wants to take a look.

Any pointers would be appreciated.

-- 
Pietro Cerutti
I have pledged to give 10% of income to effective charities
and invite you to join me - https://givingwhatwecan.org

Re: [TCLCORE] Documentation of integer truncation in [expr]essions

[Jan N. <jan...@gm...>]

Op vr 31 mei 2024 om 19:40 schreef Donald G Porter:
> My preferred solution is to take the position that Tcl 9 documentation
> has no duty to instruct on how to migrate expressions from Tcl 8.4.  That
> transition happened long ago.  Strike the sentences.

Done:
    <https://core.tcl-lang.org/tcl/info/5a0f4dbe559ee566>

Thanks!
    Jan Nijtmans

Re: [TCLCORE] [chan postevent] must be asynchronous

[Harald O. <har...@el...>]

Am 04.06.2024 um 23:53 schrieb Andreas Kupries:
> 
> [This response collects quotes from multiple mails received from Nathan]
> 
>> Hi everyone,
>   
>> A number of people, including apparently several members of the Tcl
>> Core Team, have decided that [chan postevent] is part of Tcl's
>> synchronous API for channels rather part of Tcl's asynchronous API
>> for channels, and on that basis want to close the following issue
>> reports
>   
>> 	https://core.tcl-lang.org/tcl/tktview/67a5eabbd3d195915d3ccc13246bdecf10e20726
>> 	https://core.tcl-lang.org/tcl/tktview/de232b49f26da1c18e07513d4c7caa203cd27910
>   
>> The idea that [chan postevent] is part of Tcl's synchronous API is
>> one of the most wrong-headed ideas I've encountered among the
>> contributors to Tcl.
> 
> Strong claim. Emotional argument also, not technical.
> 
>> I'm sure that at least some people reading this have experienced
>> failure when trying to implement a refchan, and as a result have
>> simply turned their backs on refchans and found some other way to
>> get the job done.  When [chan postevent] is part of Tcl's
>> asynchronous API,
> 
>> as was always intended,
> 
> You seem to claim telepathy as well. Specifically that of my mind,
> given that I was the writer of [TIP #219][1] back in 2004.
> 
> [1] https://core.tcl-lang.org/tips/doc/trunk/tip/219.md
> 
> Despite the distance of 20 years I am pretty sure that I had not
> intended async operation.
> 
> And coming back into things after reading more mails I am now fully
> sure it was not intended to be async. See later where I talk about
> what a `refchan` truly is. A channel driver, just in Tcl. Operating
> under the same rules as a C-level driver. Whose equivalent to
> `postevent` is, yes, verily, synchronous.
> 
>> you wouldn't have experienced that "hanging" behaviour of refchans.
>   
>> For all those who have been bitten by refchans, now would be a good
>> time to voice your support for fixing bugs keep it from being
>> asynchronous as it should be.
>   
>> Here is a list of all wiki pages containing examples of postevent
>> that are broken until the issues referenced above are fixed:
>   
>> 	https://wiki.tcl-lang.org/page/fifo2+in+pure+Tcl
>> 	https://wiki.tcl-lang.org/page/reflected+channel+example
>> 	https://wiki.tcl-lang.org/page/reflected+channel
>   
>> Here is a list of wiki pages that contain ugly hacks to work around
>> the issues reference above:
>   
>> 	https://wiki.tcl-lang.org/page/A+template+reflected+channel
>> 	https://wiki.tcl-lang.org/page/topcua
> 
> The hack you refer too are the `after ...` constructions used to push
> `postevent` calls out of non-reentrant parts of the procs, right ?
> 
> I do not see such as hacks, but rather as the standard way to avoid
> reentrancy troubles anywhere.
>   
>> The general principle here is that event handlers should not
>> recursively enter event-handling, but should instead allow the event
>> loop to mediate.  This should be obvious, so I don't know why we now
>> have to debate it for [chan postevent].
> 
>> Was it ever documented to be synchronous?
> 
> No. Of course not. Because synchronous behaviour is the default for
> pretty much all Tcl commands, and it is the __exceptions__ from that
> default which are documented.
> 
>> Did any change I made require a change in the documentation?
> 
> Changing a command from sync to async is a strong change of the
> semantics. Yes, that requires a change of the docs. Because, see
> above, the async behaviour of a command is the exception from the
> default.
> 
>> If not, no TIP is required to fix bugs in the implementation.
> 
> Just declaring something as a bug does not give you a
> free-to-change-this card.
> 
>> Yes, really.  [chan postevent] is the mechanism provided for a refchan
>> to notifiy Tcl of events on the channel.  At the C level, the mechanism
>> provided to an event source for this purpose is Tcl_QueueEvent(), which
> 
> Actually the C level equivalent of `postevent` is `Tcl_NotifyChannel()`
> which is very much synchronous. Somehow the C-level channel drivers
> still manage to work.
> 
> https://www.tcl-lang.org/man/tcl8.6/TclLib/CrtChannel.htm
> 
> 	Tcl_NotifyChannel is called by a channel driver to indicate to
>          the generic layer that the events specified by mask have
>          occurred on the channel. Channel drivers are responsible for
>          invoking this function ___whenever the channel handlers need to
>          be called for the channel___ (or other pending tasks like a write
>          flush should be performed). See WATCHPROC below for more
>          details.
> 
> (WATCHPROC) https://www.tcl-lang.org/man/tcl8.6/TclLib/CrtChannel.htm#M16
> 
> 	... the channel driver is responsible for calling
> 	Tcl_NotifyChannel to inform the generic channel module. The
> 	driver should take care not to starve other channel drivers or
> 	sources of callbacks by invoking Tcl_NotifyChannel too
> 	frequently. Fairness can be insured by using the Tcl event
> 	queue to allow the channel event to be scheduled in sequence
> 	with other events
> 
> So, `Tcl_NC` is synchronous in C, and C-based channel driver are
> responsible for using the C-level equivalent of `after` to explicitly
> go through the event queue where/when needed, i.e. to avoid reentrancy
> issues, etc. And these drivers __do so__.
> 
> A `refchan` is a __direct__ mapping of a channel driver into Tcl, and
> therefore has the __same responsibilities__ as a C-level driver, up to
> and including to explicitly go through the event queue when needed,
> using `after` as the effective entrypoint to `Tcl_QueueEvent`.
> 
> A refchan directly talks to the low-level Tcl's IO system like C
> drivers to, and has to follow the same rules as they do.
> 
> 
>> My answer is that [chan postevent] was always sufficiently
>> documented as asynchronous,
> 
> I see no such documentation in the `chan` manpage, nor in the TIP 219.
> Please provide a reference to the docs you believe made the async
> claim.

Hi Andreas,

I want to thank you a lot for your contribution. After we left the 
Munich Telco yesterday around 22:00, you apparently had an intense 
working night.

I remember you presenting refchan around 1998 at the Hamburg ETCL 
conference. It is a great tool but only little number of people know how 
to use it. I was asking for a tutorial speech in Vienna, but this will 
be in 2025.

Ashok, could you merge the information to the technote?

How could we improve documentation?
What part is missing there?

Thank you for all, you guys rock !
Harald

Re: [TCLCORE] CFV: TIP #696

[Donald A. <as...@tr...>]

Sounds fine in principle. But OUCH that is so greedy!  Reserving over 2 billion code values as potentially defined by the language, and condemning programmers to use 10-digit private code values in their programs. I was expecting increasing the defined number from 5 to 16 or 32, but surely no more than 99.

If you want to prepare for a future need to dynamically create millions of return codes, how about reserving the BIG numbers for Tcl itself, along with a very small set of numbers near zero?


--Donald

[TCLCORE] Fwd: tdbc error?

[Kevin K. <kev...@gm...>]

Forwarding to the list - I didn't notice at first that René had sent the
message there.



---------- Forwarded message ---------
From: Kevin Kenny <kev...@gm...>
Date: Tue, Jun 4, 2024 at 10:59 PM
Subject: Re: [TCLCORE] tdbc error?
To: Zaumseil René <RZa...@kk...>


Hmmm.  I wonder if 'attach' can't run within a prepared statement.

II know that someone else - I forget who - had an issue that certain
actions on a database needed to go through the underlying handle; I don't
know whether 'attach' might be one of them.

tdbc::sqlite3 provides an escape to the underlying Sqlite3 layer.

[[$myDb getDBhandle] eval {attach $myfile as cms;}] will allow you to hand
the 'attach' directly over to the underlying Sqlite3 layer.

Beyond that, I think I'd have to ask Richard. There seems to be only one
transaction in flight, so I can't quite grasp what might be locking the
database.

Off-topic, you seem to be writing in a verbose style.  Rather than [set myS
[db prepare {sql}]; $myS execute; $myS close]  I'd write just [db
allrows {sql}] and let TDBC worry about managing $myS.



On Tue, Jun 4, 2024 at 10:05 AM Zaumseil René via Tcl-Core <
tcl...@li...> wrote:

> Hello
>
>
>
> Running the following script results in a message "database locked".
>
> If I change "attach ..." to "-- attach ..." the script run without error.
>
> I use sqlite version 3.43.1 and tdbc 1.1.1
>
>
>
> ===
>
>     package require tdbc::sqlite3
>
>     set myFile D:/testdb
>
>     set myDb ::DB
>
>     catch "$myDb close"
>
>     ::tdbc::sqlite3::connection create $myDb $myFile -readonly 0 -timeout
> 1000
>
>     set myS [$myDb prepare {
>
> attach $myFile as cms;
>
>     }]
>
>     $myS execute
>
>     $myS close
>
>     $myDb transaction {
>
>         set myS [$myDb prepare {
>
> drop table if exists t1;
>
> create table if not exists t1 (x,y);
>
>                 }]
>
>         set myR [$myS execute]
>
>         $myS close
>
>         set myS [$myDb prepare {
>
> select name from pragma_table_info('t1')
>
>                 }]
>
>         $myS close
>
>     }
>
>     $myDb close
>
> ===
>
>
>
> Running the equivalent sqlite script is ok:
>
>
>
> ===
>
>     catch {::DB close}
>
>     sqlite3 ::DB D:/test1 -readonly 0
>
>     ::DB timeout 1000
>
>     ::DB eval {
>
> attach $myFile as cms;
>
>     }
>
>     ::DB transaction {
>
>                 ::DB eval {
>
> drop table if exists t1;
>
> create table if not exists t1 (x,y);
>
>         }
>
>                 ::DB eval {
>
> select name from pragma_table_info('t1')
>
>                 }
>
>     }
>
> ===
>
>
>
> Thank you for any hints
>
>
>
> rene
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core
>


-- 
73 de ke9tv/2, Kevin


-- 
73 de ke9tv/2, Kevin

Re: [TCLCORE] CFV: TIP #696

[<apn...@ya...>]

@mjanssen on the Tclers chat: @apn re: tip 696 why such a huge reserved
range? Isn't it more ergonomic to use something like 0xFFFF then it's much
moer obvious wihout checking docs that a code like 0x10000 is available

My response:

[09:11]	apn	@mjanssen,  based on an earlier comment not to reserve too
small a space, I thought best to split the available range equally among Tcl
and extensions. It also allows the possibility of using some bits as "flags"
if that need ever arises in the future. Admittedly not likely but similar to
how Windows error codes are structured.

[09:12]	apn	I understand your comment about ergonomics but I think that
would only help marginally as still have to check docs to know the
restriction and choosing a code with high nibble 5 or greater is not too
different.
[09:13]	apn	Still, as I said in the TIP I have no strong ties to the
range, so if enough folks think 0xFFFF makes sense, I would have no
objection

Re: [TCLCORE] CFV: TIP #696

[<apn...@ya...>]

Re. the CFV warning, I thought I had done that a couple of weeks ago and
looking at my Sent folder, did in fact do so on May 24. It does not however
show up in the SF mailing list either. I suspect it coincided with the SF
mailing list outage. In any case, if there are sufficient comments that
warrant further discussion, I'll restart the vote.

Regarding tcllib, yes, some packages will have to be touched. Will it help
if we reduce the range of Tcl reserved codes as @mjanssen suggested on the
chat to say, -0xFFFF:0xFFFF ?

> If I see right then the TIP implementation just declares this new
> reserved range of return codes. This means every extension using a
> -code > 4 will still work as expected.

Yes. Until some future release of Tcl uses that code. 

> At the moment return -code raises error it the value isn't an int.
> Shouldn't it also raise error (after this TIP is accepted) if the value
> is an internal one (other then 0 - 4)?
> This would make Tcl 8 code raise error with Tcl 9 which would otherways
> have worked unchangend for years and new return codes to come. But it
> would tell much earlier and clearer that here is a migration cost for
> Tcl 9. WDYT?

I'm not sure there is any way to check for this since there is no way to
tell if a return command invocation is from the Tcl core or not (the Tcl
script support library being considered part of the core)

Thanks for the comments,

/Ashok

-----Original Message-----
From: Rolf Ade <tcl...@po...> 
Sent: Wednesday, June 5, 2024 5:29 AM
To: tcl...@li...
Subject: Re: [TCLCORE] CFV: TIP #696


apnmbx-public--- via Tcl-Core writes:
> This is a Call for Votes for TIP #696 - Reserve range of return codes for
> Tcl's own use

A CfV warning would have been fine.

Greping in the tcllib code shows that a few packages must be touched for
sure and some others have to be reviewed because of this. I have code to
touch because of this beside tcllib and others may very well have also.

This just illustrates that this is a change that can be done only with a
major release. 

The idea, I think is sound. The need is illustrated by examples. And
if having additional return codes for internal use is a feature useful
for extensions (and it is) then this is also a feature useful for the
core.

If I see right then the TIP implementation just declares this new
reserved range of return codes. This means every extension using a
-code > 4 will still work as expected.

As long as the Tcl core starts to invent new return codes. The
Tip is there to allow that this may happen at any time now.

The examples in tcllib uses low numbers > 4 for own error codes. Do we
want to pick new internal error codes from inbetween the new broad range
and cross fingers, we don't trap on someones feet?

At the moment return -code raises error it the value isn't an int.
Shouldn't it also raise error (after this TIP is accepted) if the value
is an internal one (other then 0 - 4)?

This would make Tcl 8 code raise error with Tcl 9 which would otherways
have worked unchangend for years and new return codes to come. But it
would tell much earlier and clearer that here is a migration cost for
Tcl 9. WDYT?

rolf


_______________________________________________
Tcl-Core mailing list
Tcl...@li...
https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] CFV: TIP #696

[Rolf A. <tcl...@po...>]

apnmbx-public--- via Tcl-Core writes:
> This is a Call for Votes for TIP #696 - Reserve range of return codes for
> Tcl's own use

A CfV warning would have been fine.

Greping in the tcllib code shows that a few packages must be touched for
sure and some others have to be reviewed because of this. I have code to
touch because of this beside tcllib and others may very well have also.

This just illustrates that this is a change that can be done only with a
major release. 

The idea, I think is sound. The need is illustrated by examples. And
if having additional return codes for internal use is a feature useful
for extensions (and it is) then this is also a feature useful for the
core.

If I see right then the TIP implementation just declares this new
reserved range of return codes. This means every extension using a
-code > 4 will still work as expected.

As long as the Tcl core starts to invent new return codes. The
Tip is there to allow that this may happen at any time now.

The examples in tcllib uses low numbers > 4 for own error codes. Do we
want to pick new internal error codes from inbetween the new broad range
and cross fingers, we don't trap on someones feet?

At the moment return -code raises error it the value isn't an int.
Shouldn't it also raise error (after this TIP is accepted) if the value
is an internal one (other then 0 - 4)?

This would make Tcl 8 code raise error with Tcl 9 which would otherways
have worked unchangend for years and new return codes to come. But it
would tell much earlier and clearer that here is a migration cost for
Tcl 9. WDYT?

rolf

Re: [TCLCORE] [chan postevent] must be asynchronous

[Andreas K. <and...@gm...>]

[This response collects quotes from multiple mails received from Nathan]

> Hi everyone,
 
> A number of people, including apparently several members of the Tcl
> Core Team, have decided that [chan postevent] is part of Tcl's
> synchronous API for channels rather part of Tcl's asynchronous API
> for channels, and on that basis want to close the following issue
> reports
 
> 	https://core.tcl-lang.org/tcl/tktview/67a5eabbd3d195915d3ccc13246bdecf10e20726
> 	https://core.tcl-lang.org/tcl/tktview/de232b49f26da1c18e07513d4c7caa203cd27910
 
> The idea that [chan postevent] is part of Tcl's synchronous API is
> one of the most wrong-headed ideas I've encountered among the
> contributors to Tcl.

Strong claim. Emotional argument also, not technical.

> I'm sure that at least some people reading this have experienced
> failure when trying to implement a refchan, and as a result have
> simply turned their backs on refchans and found some other way to
> get the job done.  When [chan postevent] is part of Tcl's
> asynchronous API,

> as was always intended,

You seem to claim telepathy as well. Specifically that of my mind,
given that I was the writer of [TIP #219][1] back in 2004.

[1] https://core.tcl-lang.org/tips/doc/trunk/tip/219.md

Despite the distance of 20 years I am pretty sure that I had not
intended async operation.

And coming back into things after reading more mails I am now fully
sure it was not intended to be async. See later where I talk about
what a `refchan` truly is. A channel driver, just in Tcl. Operating
under the same rules as a C-level driver. Whose equivalent to
`postevent` is, yes, verily, synchronous.

> you wouldn't have experienced that "hanging" behaviour of refchans.
 
> For all those who have been bitten by refchans, now would be a good
> time to voice your support for fixing bugs keep it from being
> asynchronous as it should be.
 
> Here is a list of all wiki pages containing examples of postevent
> that are broken until the issues referenced above are fixed:
 
> 	https://wiki.tcl-lang.org/page/fifo2+in+pure+Tcl
> 	https://wiki.tcl-lang.org/page/reflected+channel+example
> 	https://wiki.tcl-lang.org/page/reflected+channel
 
> Here is a list of wiki pages that contain ugly hacks to work around
> the issues reference above:
 
> 	https://wiki.tcl-lang.org/page/A+template+reflected+channel
> 	https://wiki.tcl-lang.org/page/topcua

The hack you refer too are the `after ...` constructions used to push
`postevent` calls out of non-reentrant parts of the procs, right ?

I do not see such as hacks, but rather as the standard way to avoid
reentrancy troubles anywhere.
 
> The general principle here is that event handlers should not
> recursively enter event-handling, but should instead allow the event
> loop to mediate.  This should be obvious, so I don't know why we now
> have to debate it for [chan postevent].

> Was it ever documented to be synchronous?

No. Of course not. Because synchronous behaviour is the default for
pretty much all Tcl commands, and it is the __exceptions__ from that
default which are documented.

> Did any change I made require a change in the documentation?

Changing a command from sync to async is a strong change of the
semantics. Yes, that requires a change of the docs. Because, see
above, the async behaviour of a command is the exception from the
default.

>If not, no TIP is required to fix bugs in the implementation.

Just declaring something as a bug does not give you a
free-to-change-this card.

> Yes, really.  [chan postevent] is the mechanism provided for a refchan
> to notifiy Tcl of events on the channel.  At the C level, the mechanism
> provided to an event source for this purpose is Tcl_QueueEvent(), which

Actually the C level equivalent of `postevent` is `Tcl_NotifyChannel()`
which is very much synchronous. Somehow the C-level channel drivers
still manage to work.

https://www.tcl-lang.org/man/tcl8.6/TclLib/CrtChannel.htm

	Tcl_NotifyChannel is called by a channel driver to indicate to
        the generic layer that the events specified by mask have
        occurred on the channel. Channel drivers are responsible for
        invoking this function ___whenever the channel handlers need to
        be called for the channel___ (or other pending tasks like a write
        flush should be performed). See WATCHPROC below for more
        details.

(WATCHPROC) https://www.tcl-lang.org/man/tcl8.6/TclLib/CrtChannel.htm#M16

	... the channel driver is responsible for calling
	Tcl_NotifyChannel to inform the generic channel module. The
	driver should take care not to starve other channel drivers or
	sources of callbacks by invoking Tcl_NotifyChannel too
	frequently. Fairness can be insured by using the Tcl event
	queue to allow the channel event to be scheduled in sequence
	with other events

So, `Tcl_NC` is synchronous in C, and C-based channel driver are
responsible for using the C-level equivalent of `after` to explicitly
go through the event queue where/when needed, i.e. to avoid reentrancy
issues, etc. And these drivers __do so__.

A `refchan` is a __direct__ mapping of a channel driver into Tcl, and
therefore has the __same responsibilities__ as a C-level driver, up to
and including to explicitly go through the event queue when needed,
using `after` as the effective entrypoint to `Tcl_QueueEvent`.

A refchan directly talks to the low-level Tcl's IO system like C
drivers to, and has to follow the same rules as they do.


> My answer is that [chan postevent] was always sufficiently
> documented as asynchronous,

I see no such documentation in the `chan` manpage, nor in the TIP 219.
Please provide a reference to the docs you believe made the async
claim.


-- 
Happy Tcling,
	Andreas Kupries <and...@gm...>
			<https://core.tcl-lang.org/akupries/>
			<https://akupries.tclers.tk/>
	Developer @	SUSE Software Solutions Germany GmbH
-------------------------------------------------------------------------------