tcl-core — Tcl Core Team discussions about Tcl development

Re: [TCLCORE] "Productive branches"

[Marc C. <cul...@gm...>]

I would like to add that it does not make sense to me to even have both 9.0
and 9.1 as "productive branches".  In my opinion, when 9.1 is released 9.0
should cease to be productive.  I can't think of any reason to maintain 9.0
and 9.1 separately, much less 9.3, 9.4, 9.5, ad infinitum. While a project
like Python may be able maintain 3 separate versions simultaneously, they
also have a foundation with many paid employees.

- Marc

On Mon, Jan 27, 2025 at 10:14 AM Marc Culler <cul...@gm...> wrote:

> I don't understand this, but it looks very strange to me:
> First we have:
>
> - there are productive branches:
> core-8-6-branch, core-8-branch, core-9-0-branch, core-9-branch
> Missing branches are created when the first 9.1 feature is merged.
>
> Then we have:
>
> - if a branch is ready to merge, please ask for a merge permission it in
> the ticket including the proposed branches (e.g. 8.6, 9.0, 9.1).
>
> So that makes it look like we will have "productive branches" 9.0, 9.1.
> 9.2, 9,.3, 9.4, ....  which seems like a crazy idea to me.
>
> Also, I don't understand why TCT policies are being created outside of the
> official communication channel of the TCT, i.e. this list, at video
> conferences which occur at 4AM for people on the west coast of the US, with
> no discussion on this list.  I thought we had agreed not to do that.
>
> Also, I think it is important to provide a list of attendees at the video
> meeting, especially when they are claiming to set policy for the TCT.
>
> - Marc
>
>
>

[TCLCORE] Report of biweekly Tcl/Tk telco on 2025-01-27 12:00 UTC

[Harald O. <har...@el...>]

Dear Tcl/Tk team,

thanks for your participation at the Tcl/Tk biweekly telco on 27th.
It was a great meeting to start the year. Thank you all, that you took 
your time. It was 4am for Brian.

Here is the report:

1) Release calender

A release calendar was discussed and will be published by the release 
manager. Some opinions on this topic:
- 8.6 is supported but needs only one release per year
- there is no big request for 8.7. Many people try 9.0
- Tk may match the Tcl release calender

2) Development branches

2.1) Merge order
Currently, there is a merge order core-8-6-branch -> core-8-branch -> 
main. This allows to merge one product branch in the next:
cd my8.6
fossil commit
cd my87
fossil merge core-8-6-branch
<handle merge>
fossil commit
If something should not go into a productive branch, a merge mark is 
required.

The new organisation is as follows:
- there are productive branches:
core-8-6-branch, core-8-branch, core-9-0-branch, core-9-branch
Missing branches are created when the first 9.1 feature is merged.
The tag "main" is attributed to the most recent branch (e.g. core-9-branch).
- to change something in one branch, one may open the branch from a 
productive branch
cd my9.1
fossil commit --branch 0123456-my-feature
After permission, this may be merged back to a productive branch
cd my9.1
fossil merge 0123456-my-feature
- get a change in another productive branch, only cherry-picking is used:
cd my9.0
fossil merge --cherrypick main

As a consequence, no more merge marks are used.
Please don't forget this, if you merge between productive branches.

2.2) Merge permission

A merge to a productive branch should be authorized by a TCT member.

The organisation is as follows:
- a ticket is created per issue
- if a branch is ready to merge, please ask for a merge permission it in 
the ticket including the proposed branches (e.g. 8.6, 9.0, 9.1).
A TCT member should document its permission in the ticket.

There was the proposal of a two weeks maximum answer time what is IMHO 
not discussed to the end.

3) fossil login

Some fossil repositories do not allow login creation. This is seen as 
useful. Schelte Bron may contact repository owners to activate this 
feature when not enabled.

4) Bundled packages

IncrTcl works with the proposed changes for 9.1 (e.g. unlimitted 
argument count for procs). This was seen as a blocker before, as it was 
reported not to work.
Bundling is a TCT decision (TIP50). Bundling IncrTCL is still manageable 
and will continue. It may be stopped by a TIP at any patch release.

Bundling of other packages (TCLTLS) was not discussed

5) Windows Memory Module DLL loading
Jan reported, that TLS (Thread local storage) does not work with memory 
module. Any DLL using it will not be memory loaded.
Jan is seeing the feature as a solution for many use-cases and wants to 
bring TIP 709 (Licence acceptance plus optional feature) up to a vote.
There are the following objections:
- undocumented interface may change at any time
- other special DLL features may not work. Examples: Advanced exception 
handling, C++ binary interface.
- this undocumented loading may cause (future) security code to fire
Jan was asking for test cases for failures. Ashok will not deliver any 
more test cases or arguments, as he invested already a lot of time and 
this proposal does not match his design principles of stability and 
reliability.
Jan is asking for contributions on the TIP.

6) VecTCL
Brian is working on it. There are open design questions he may address 
to Christian by private E-Mail.

7) Next meetings
URL: https://meet.jit.si/TclMonthlyMeetup
Duration: 1 hour
2025-02-04 18:00 UTC Tk meeting
2025-02-10 12:00 UTC Bi-weekly Tcl/Tk (Harald will not be available)

Thank you all,
Harald

Re: [TCLCORE] vectcl: get a number without shimmering a list

[Jan N. <jan...@gm...>]

Op ma 27 jan 2025 om 11:16 schreef Jan Nijtmans <jan...@gm...>:

> couldn't be handled earlier, thanks to TIP #644.
>

More accurate, it's TIP #636:
   Tcl Improvement Proposals: TIP 636: Expand List Flexibility with
Abstract List Tcl_Obj Type
<https://core.tcl-lang.org/tips/doc/trunk/tip/636.md>

Hope this helps,
         Jan Nijtmans

Re: [TCLCORE] vectcl: get a number without shimmering a list

[Jan N. <jan...@gm...>]

Op ma 27 jan 2025 om 10:50 schreef Harald Oehlmann:

> Now the question:
> - how may one get an int from an object without shimmering lists (e.g.
> generate a string representation of a list).


Just use Tcl_GetIntFromObj (or Tcl_GetWideIntFromObj or one of the
other variants. Starting with Tcl 9.0, those functions don't need the
string representation any more in many many situations which
couldn't be handled earlier, thanks to TIP #644.

This is handled in this ticket:
     <Tcl Source Code: View Ticket
<https://core.tcl-lang.org/tcl/tktview/0439e1e1a3>>
(it works for dicts, list ... any other type ... as well, not only for lseq)

Only when the list/lseq has length "1", the string rep will be
generated if it doesn't exist yet. Otherwise, we can already
tell beforehand that getting an int from it will fail.

Hope this helps,
   Jan Nijtmans

[TCLCORE] vectcl: get a number without shimmering a list

[Harald O. <har...@el...>]

Dear TCL experts,

vectcl is in the process of TCL 9 porting.

VecTCL uses the registered int type to:
- convert an int
- avoid shimmering of lists and lists of lists (the VecTCL native type)

The int type is not registered any more.
This is replaced by Tcl_GetNumberFromObj

Now the question:
- how may one get an int from an object without shimmering lists (e.g. 
generate a string representation of a list).

Brian has a private fork of vectcl with abstract list usage:
https://github.com/bgriffinfortytwo/VecTcl9

Brian wrote that he will care (THANKS !).
The original author, magic Christian, will not but inspired the upper 
question by private mail.

Thanks for any input!
Harald

---

For reference, see the "vectcl for tcl9" thread started 2025-01-20

Here is a part of it citing a quick fix by Ralf Fassel:
* Ralf Fassel <ra...@gm...>
| I recompiled vectcl 0.2.1 as provided by Paul against tcl 9.0 and tcl 
8.6.15.
| 'make test' in vectcl fails in many cases with tcl9, where the same
| tests succeed in 8.6.15.
 >
|   package require vectcl
|   => 0.2.1
|
|   numarray concat {{1.0 2.0} {3.0 4.0}} 5.0 0
|   tcl 8 => {1.0 2.0} {3.0 4.0} {5.0 5.0}
|   tcl 9 => error: expected integer but got "1.0 2.0"

Ok, being curious...

The deeper reason for those failures is that vectcl uses type info for
TclObjs, and tcl9 no longer registers type 'int' (cf.

     tclObj.c,
         const Tcl_ObjType tclIntType = {
                  "int",			/* name */
                  ...
         };

       void TclInitObjSubsystem(void)

       8.6.15
            Tcl_RegisterObjType(&tclIntType);
       9.0.0
          ---


Now vectcl does in two places:

    const Tcl_ObjType * tclIntType = Tcl_GetObjType("int");

and compares that in many places to the obj type

    if (objPtr->typePtr == tclIntType)

Since 'int' is not registered in 9.0, Tcl_GetObjType will return NULL
there, thus the vectcl code treats a not-set obj-type as int, which
explains the errors.

Quick and dirty setting the vectcl lookup variables for
Tcl_GetObjType("int") to 0xdeadbeef if they are NULL makes all tests
succeed.


--- vectcl-0.2.1/generic/vectcl.c~	2025-01-22 12:50:45.750839906 +0100
+++ vectcl-0.2.1/generic/vectcl.c	2025-01-22 16:15:43.747690082 +0100
@@ -2577,6 +2590,7 @@
  	tclListType =  (Tcl_ObjType *) Tcl_GetObjType("list");	
  	tclDoubleType = Tcl_GetObjType("double");
  	tclIntType = Tcl_GetObjType("int");
+	if (0 ==tclIntType) tclIntType = 0xdeadbeef;
  #ifndef TCL_WIDE_INT_IS_LONG
  	tclWideIntType = Tcl_GetObjType("wideInt");
  #endif


--- vectcl-0.2.1/generic/nacomplex.c~	2015-07-08 22:38:34.000000000 +0200
+++ vectcl-0.2.1/generic/nacomplex.c	2025-01-22 16:18:04.682876166 +0100
@@ -67,7 +67,7 @@

  	/* Maybe this should go into a static const array */
  	const Tcl_ObjType * tclDoubleType = Tcl_GetObjType("double");
-	const Tcl_ObjType * tclIntType = Tcl_GetObjType("int");
+	const Tcl_ObjType * tclIntType = Tcl_GetObjType("int") || 0xdeadbeef;
  	
  	if (objPtr -> typePtr == tclIntType) {
  		int value;

% make test
Tests ended at Wed Jan 22 16:18:11 CET 2025
all.tcl:	Total	210	Passed	210	Skipped	0	Failed	0

Of course this defeats the performance gains intended by the type
lookup, but since I don't know why "int" is no longer registered as type
in the tcl core, and what is to be used as substitution, I can not offer
any better fix for vectcl.

[TCLCORE] perl tcl/tk status

[Vadim V K. <vad...@ga...>]

Hi,

Now I can confirm that current version of Perl/Tcl module (module itself is named Tcl) is actually ok in its current version 1.50, plus I've just released a bit cleaned-up version 1.51, but 1.50 also ok.

I tried on debian and all tests are good, except some minor failure:
t/text.t ........ version conflict for package "Tk": have 9.0.0, need 8.4 at /home/vad/.cpan/build/Tcl-Tk-1.29-0/blib/lib/Tcl/Tk.pm line 942.
(I will look into this problem later, the failing code is a fallback when tcl/tk setup does not have widget::scrolledwindow package, so Perl/Tcl-Tk provides the missing scrollw.tcl file)

The difficulty I am still having is with windows usage against BAWT build of Tcl:

perl -Mblib -MTcl -we "my $i = new Tcl;$i->Init;$i->Eval('puts $auto_path;package require tk;pack [button .b];tk_messageBox -title 123')"
//zipfs:/lib/tcl/tcl_library //zipfs:/lib/tcl C:/vad/perl-dev/strawberry-perl-5.38.0.1-64bit-portable/perl/lib
can't find package tk at -e line 1.

So here 'tk' package failed to be located.

Then I add the tk path to the auto_path list, and the problem goes away:

perl -Mblib -MTcl -we "my $i = new Tcl;$i->Init;$i->Eval('lappend auto_path C:/vad/apps/Tcl-bawt-bi-901-64/lib;puts $auto_path;package require tk;pack [button .b];tk_messageBox -title 123')"
//zipfs:/lib/tcl/tcl_library //zipfs:/lib/tcl C:/vad/perl-dev/strawberry-perl-5.38.0.1-64bit-portable/perl/lib C:/vad/apps/Tcl-bawt-bi-901-64/lib

(I see message box and empty button here)

Should I need to add more initialization steps here?
To say, I can't find the init.tcl file in BAWT directory.

Thanks in advance,
Vadim

[TCLCORE] biweekly Tcl/Tk telco on 2025-01-27 12:00 UTC

[Harald O. <har...@el...>]

Dear Tcl/Tk team,

please allow me to invite you to the next Tcl/Tk biweekly telco on 27th 
of January 2025 12:00 UTC on Jitsi: https://meet.jit.si/TclMonthlyMeetup

Possible topics (proposals, no decided actions):

- Time frame for Tcl/Tk 8.6.16, 9.0.2, 9.1a1-9.1.0
- Branch organisation
   - start of 9.1 development
   - Merge order: trunk first
   - New commits in branches with mandatory TCT member review before 
main branch merge
- Test coverage:
   - zip file system
   - new proposals
- Windows DLL memory module
- VecTCL
- Perl-TCl
- Bundled packages:
   - incr TCL out
   - TclTLS in
- Orphaned packages on github

Further proposed meetings
2025-02-04 18:00 UTC Tk meeting, afterwards "Stammtisch", Harald will 
demonstrate Barcode scanning using tclwmf and zxing-cpp packages
2025-02-10 12:00 UTC Bi-weekly Tcl/Tk (Harald will not be available)

Thank you all,
Harald

[TCLCORE] sqlite3 3.48.0 released

[Jan N. <jan...@gm...>]

The upstream SQLite project released 3.48.0 of SQLite recently

>From that, I derived the TEA-based Tcl package we layer
on top of it.

    http://cyqlite.sourceforge.net/cgi-bin/sqlite/timeline

That's now available as Tcl package sqlite3.48.0.tar.gz from

    https://sourceforge.net/projects/tcl/files/Tcl/8.6.16/
or
    https://sourceforge.net/projects/tcl/files/Tcl/9.0.1/

Unpack that source distribution in the "pkgs" subdir of your Tcl
8.6.x or 9.0.x source code distribution and run `make install`
again for your platform.
That will build and install the updated sqlite package.

Unless another SQLite release happens first, this package will be
bundled with Tcl 8.6.17 / Tcl 9.0.2

Re: [TCLCORE] Report of Tcl/Tk biweekly telco

[Schelte B. <tc...@tc...>]

On 20/01/2025 22:20, Francois Vogel wrote:
> With the help of Roy Keene (many thanks again!), I got email alerts 
> working for my rtext repository at Chiselapp. Now that I know how to do 
> it I can easily propagate my fresh knowledge.

I knew this is how to do it on chiselapp. Unfortunately the same method 
doesn't work on core. The error indicates that the /data/mail.sql 
database does not exist. I sent a mail to Roy to ask for assistance,

Before I knew about the database, I used the SMTP relay method. I set up 
postfix on an obscure port on my server at home and punched a hole in 
the firewall to allow access from the chiselapp IP address only. That 
worked for me, but I don't think it's a feasible solution for the core 
repositories. I cannot guarantee reliability and considering that there 
are already thousands of messages queued up for the Tcl repository 
alone, I expect my ISP will block me for sending spam when those are all 
released at once through my server.

If there is some other place where we could set up a mail relay server, 
that could be an alternative.


Schelte.

Re: [TCLCORE] Report of Tcl/Tk biweekly telco

[<apn...@ya...>]

The “infrastructure” discussion pertained to the resources – repository hosts, wiki, web site, CI, chat  etc. – used by the community, the effort required to maintain these, single points of failure and attendant risk of unavailability (including human 😊)

 

From: Francois Vogel <fvo...@fr...> 
Scratching my head on this statement, I have to admit I have no idea what is meant here by "infrastructure" - it's probably not a building. Could you please explain?

Re: [TCLCORE] Report of Tcl/Tk biweekly telco

[Francois V. <fvo...@fr...>]

Ashok,

Thanks for explaining all this. Let me state I share your view about how 
development should be done in trunk.

Regards,

Francois

Le 20/01/2025 à 16:42, apnmbx-public--- via Tcl-Core a écrit :
>
> Harald, thanks for the summary but I do want to make clarifications 
> with respect to what I took away from the meeting.
>
> With respect to the memory module, the result was not "code is 
> correct, no doubt". Jan held that the code was correct in his view and 
> he was only working on tests. I contended that the code as it 
> currently stands fails even the simple test cases I provided in my 
> apn-tip-709 branch and he has not yet indicated why the tests were 
> invalid. If I understood Jan correctly, he also said the tests do not 
> show corrupt data (perhaps I misunderstood what he was saying)  and my 
> response was that one DLL writing to the private TLS data of another 
> DLL instead of its own *in the same thread* was indeed data corruption 
> by any definition of the term. But the matter is still in progress as 
> Jan works on his own tests. As of now, there is no doubt in my mind 
> the code is broken.
>
> My comments with respect to test cases were that the TCT should hold a 
> stronger position with respect to even passed TIP's being merged to 
> trunk without a reasonable test suite.
>
> That led to what I think both Jan and I agreed (Jan, correct me if I 
> misunderstood) that we have different views of how software 
> development on the trunk should proceed. Jan's view was that if 
> something was deemed to be a good idea based on a TIP's passage, it 
> should be merged to trunk for more people to beat on it, test and 
> provide patches. My view was that it is the author's responsibility 
> that should not be punted to everybody else, the most important 
> reasons being (a) this crowd-sourced testing and patching just does 
> not happen leaving the code fragile or completely broken and I can 
> give several cases in the Tcl 9 development cycle of the latter, and 
> (b) it is unfair to the rest of the contributors to be saddled with 
> the tedium of debugging, writing tests and patches. (*I'll just stress 
> this is not a comment on Jan himself!)*
>
> Jan, please add your view of the meeting as necessary. It's important 
> for everyone to get on the same page as to what we should expect of 
> commits to the trunk.
>
> It was a productive meeting indeed in thrashing out the different 
> views. Thanks
>
> /Ashok
>
> -----Original Message-----
> From: Harald Oehlmann <har...@el...>
> Sent: Monday, January 20, 2025 8:38 PM
> To: Tcl Core List <tcl...@li...>
> Subject: [TCLCORE] Report of Tcl/Tk biweekly telco
>
> Dear Tcl/Tk team,
>
> here is a report of the meeting of today.
>
> 1) Orphaned extensions
>
> Orphaned extensions may be put into a github repository.
>
> This may not be personal but under the hood of an organisation for legal
>
> reasons.
>
> Steve will contact Cliff Redler from the Tcl association, if this is
>
> possible under this hat.
>
> 2) TCT fossil forum
>
> a TCT forum was proposed for inner TCT communication. Benefit compared
>
> to E-Mail list is:
>
> - control when and how one gets notified
>
> - organisation of threads
>
> - archive
>
> 3) TCL infrastructure
>
> Recent fall-outs lead to the opinion that less infra structure is better.
>
> Another opinion is that infra structure is great if you control it.
>
> The E-Mail notification feature still does not work for the core. A
>
> participant reported, that it now works on Chiseal (which is perhaps the
>
> same machine) and he may try to fix it.
>
> 4) Tests
>
> Many new features came only with basic tests.
>
> An example is the zip file system, which only has basic tests.
>
> It was proposed to require a higher test coverage before a merge to the
>
> productive branches.
>
> On the other hand, Wizards just do and don't need tests at all.
>
> And we are happy to get their great code.
>
> Testing will happen in productive environments, (but there are anyway
>
> never issues).
>
> And the group of non-believers is charged to write tests.
>
> The discussion lead in case of the memory load module to the result:
>
> - the code is correct, no doubt
>
> - there will ne more test cases for those non believers
>
> To say it positive: we respect each others opinion and work on it - 
> great !
>
> 5) Next meeting:
>
> 2025-01-27 12:00 UTC Bi-weekly Tcl/Tk
>
> 2025-02-04 18:00 UTC Tk meeting, afterwards "Stammtisch", Harald will
>
> demonstrate Barcode scanning using tclwmf and zxing-cpp packages
>
> 2025-02-10 12:00 UTC Bi-weekly Tcl/Tk (Harald will not be available)
>
> Thank you all,
>
> Harald
>
>
>
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] Report of Tcl/Tk biweekly telco

[Francois V. <fvo...@fr...>]

Le 20/01/2025 à 16:08, Harald Oehlmann a écrit :
> 3) TCL infrastructure
> Recent fall-outs lead to the opinion that less infra structure is better.
> Another opinion is that infra structure is great if you control it.

Scratching my head on this statement, I have to admit I have no idea 
what is meant here by "infrastructure" - it's probably not a building. 
Could you please explain?

> The E-Mail notification feature still does not work for the core. A 
> participant reported, that it now works on Chiseal (which is perhaps 
> the same machine) and he may try to fix it.

With the help of Roy Keene (many thanks again!), I got email alerts 
working for my rtext repository at Chiselapp. Now that I know how to do 
it I can easily propagate my fresh knowledge. The key that I didn't 
figure out by myself is the last two lines:

For repositories hosted at Chiselapp, go to (you need repo admin rights):
https://chiselapp.com/user/<username>/repository/<repo_name>/setup_notification

Canonical Server URL           : 
https://chiselapp.com/user/<username>/repository/<repo_name>
Administrator email address    : whatever
"Return-Path" email address    : whatever
Repository Nickname            : [repo_name]
Email Send Method              : Store in a database
Store Emails In This Database  : /data/mail.sql

For Tcl, the address is (you need repo admin rights to access it):

https://core.tcl-lang.org/tcl/setup_notification

Fossil documentation about this is here (including how to test the setup):

https://fossil-scm.org/home/doc/trunk/www/alerts.md

Regards,

Francois

Re: [TCLCORE] Report of Tcl/Tk biweekly telco

[Jan N. <jan...@gm...>]

Op ma 20 jan 2025 om 16:42 schreef Ashok:
> That led to what I think both Jan and I agreed (Jan, correct me if I misunderstood) that we have different views of how software development on the trunk should proceed. Jan's view was that if something was deemed to be a good idea based on a TIP's passage, it should be merged to trunk for more people to beat on it, test and provide patches.

What I said was that for TIP #430, that's what happened. And
it was OK, because:
 1) Trunk was meant for 8.7/9.0, which was still far away.
      That gives enough time for testing
 2) We felt that zipfs support would be the main
     must-have feature for 8.7/9.0. Having it on
     trunk means that more people would beat on it.
Now the situation changed: anything merged to trunk will
be in tcl 9.0.2, so we need to be more careful. As soon as
trunk is split off for 9.1 development, we can be less
careful on features merged to it. If we want a feature
for 9.1 (expressed by YES votes on a TIP) then it's
important to have it on trunk as soon as it is sufficiently
stable. We need to define what is "sufficiently stable",
but IMHO the most important thing is "no regression".

That's why memorymodule needs to be an opt-in
feature for 8.6 and 9.0. It's the only way to assure
there are no regressions.

Enough so far. I'll come back on it.

Thanks!
        Jan Nijtmans

Re: [TCLCORE] Report of Tcl/Tk biweekly telco

[Marc C. <cul...@gm...>]

Thank you, Ashok, for providing some details on this issue.

-Marc

On Mon, Jan 20, 2025 at 9:42 AM apnmbx-public--- via Tcl-Core <
tcl...@li...> wrote:

> Harald, thanks for the summary but I do want to make clarifications with
> respect to what I took away from the meeting.
>
>
>
> With respect to the memory module, the result was not "code is correct, no
> doubt". Jan held that the code was correct in his view and he was only
> working on tests. I contended that the code as it currently stands fails
> even the simple test cases I provided in my apn-tip-709 branch and he has
> not yet indicated why the tests were invalid. If I understood Jan
> correctly, he also said the tests do not show corrupt data (perhaps I
> misunderstood what he was saying)  and my response was that one DLL writing
> to the private TLS data of another DLL instead of its own *in the same
> thread* was indeed data corruption by any definition of the term. But the
> matter is still in progress as Jan works on his own tests. As of now, there
> is no doubt in my mind the code is broken.
>
>
>
> My comments with respect to test cases were that the TCT should hold a
> stronger position with respect to even passed TIP's being merged to trunk
> without a reasonable test suite.
>
>
>
> That led to what I think both Jan and I agreed (Jan, correct me if I
> misunderstood) that we have different views of how software development on
> the trunk should proceed. Jan's view was that if something was deemed to be
> a good idea based on a TIP's passage, it should be merged to trunk for more
> people to beat on it, test and provide patches. My view was that it is the
> author's responsibility that should not be punted to everybody else, the
> most important reasons being (a) this crowd-sourced testing and patching
> just does not happen leaving the code fragile or completely broken and I
> can give several cases in the Tcl 9 development cycle of the latter, and
> (b) it is unfair to the rest of the contributors to be saddled with the
> tedium of debugging, writing tests and patches. (*I'll just stress this
> is not a comment on Jan himself!)*
>
>
>
> Jan, please add your view of the meeting as necessary. It's important for
> everyone to get on the same page as to what we should expect of commits to
> the trunk.
>
>
>
> It was a productive meeting indeed in thrashing out the different views.
> Thanks
>
>
>
> /Ashok
>
>
>
>
>
> -----Original Message-----
> From: Harald Oehlmann <har...@el...>
> Sent: Monday, January 20, 2025 8:38 PM
> To: Tcl Core List <tcl...@li...>
> Subject: [TCLCORE] Report of Tcl/Tk biweekly telco
>
>
>
> Dear Tcl/Tk team,
>
> here is a report of the meeting of today.
>
>
>
> 1) Orphaned extensions
>
> Orphaned extensions may be put into a github repository.
>
> This may not be personal but under the hood of an organisation for legal
>
> reasons.
>
> Steve will contact Cliff Redler from the Tcl association, if this is
>
> possible under this hat.
>
>
>
> 2) TCT fossil forum
>
> a TCT forum was proposed for inner TCT communication. Benefit compared
>
> to E-Mail list is:
>
> - control when and how one gets notified
>
> - organisation of threads
>
> - archive
>
>
>
> 3) TCL infrastructure
>
> Recent fall-outs lead to the opinion that less infra structure is better.
>
> Another opinion is that infra structure is great if you control it.
>
>
>
> The E-Mail notification feature still does not work for the core. A
>
> participant reported, that it now works on Chiseal (which is perhaps the
>
> same machine) and he may try to fix it.
>
>
>
> 4) Tests
>
> Many new features came only with basic tests.
>
> An example is the zip file system, which only has basic tests.
>
> It was proposed to require a higher test coverage before a merge to the
>
> productive branches.
>
> On the other hand, Wizards just do and don't need tests at all.
>
> And we are happy to get their great code.
>
> Testing will happen in productive environments, (but there are anyway
>
> never issues).
>
> And the group of non-believers is charged to write tests.
>
> The discussion lead in case of the memory load module to the result:
>
> - the code is correct, no doubt
>
> - there will ne more test cases for those non believers
>
>
>
> To say it positive: we respect each others opinion and work on it - great !
>
>
>
> 5) Next meeting:
>
> 2025-01-27 12:00 UTC Bi-weekly Tcl/Tk
>
> 2025-02-04 18:00 UTC Tk meeting, afterwards "Stammtisch", Harald will
>
> demonstrate Barcode scanning using tclwmf and zxing-cpp packages
>
> 2025-02-10 12:00 UTC Bi-weekly Tcl/Tk (Harald will not be available)
>
>
>
> Thank you all,
>
> Harald
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core
>

Re: [TCLCORE] Report of Tcl/Tk biweekly telco

[<apn...@ya...>]

Harald, thanks for the summary but I do want to make clarifications with respect to what I took away from the meeting.

 

With respect to the memory module, the result was not "code is correct, no doubt". Jan held that the code was correct in his view and he was only working on tests. I contended that the code as it currently stands fails even the simple test cases I provided in my apn-tip-709 branch and he has not yet indicated why the tests were invalid. If I understood Jan correctly, he also said the tests do not show corrupt data (perhaps I misunderstood what he was saying)  and my response was that one DLL writing to the private TLS data of another DLL instead of its own *in the same thread* was indeed data corruption by any definition of the term. But the matter is still in progress as Jan works on his own tests. As of now, there is no doubt in my mind the code is broken.

 

My comments with respect to test cases were that the TCT should hold a stronger position with respect to even passed TIP's being merged to trunk without a reasonable test suite.

 

That led to what I think both Jan and I agreed (Jan, correct me if I misunderstood) that we have different views of how software development on the trunk should proceed. Jan's view was that if something was deemed to be a good idea based on a TIP's passage, it should be merged to trunk for more people to beat on it, test and provide patches. My view was that it is the author's responsibility that should not be punted to everybody else, the most important reasons being (a) this crowd-sourced testing and patching just does not happen leaving the code fragile or completely broken and I can give several cases in the Tcl 9 development cycle of the latter, and (b) it is unfair to the rest of the contributors to be saddled with the tedium of debugging, writing tests and patches. (I'll just stress this is not a comment on Jan himself!)

 

Jan, please add your view of the meeting as necessary. It's important for everyone to get on the same page as to what we should expect of commits to the trunk.

 

It was a productive meeting indeed in thrashing out the different views. Thanks

 

/Ashok

 

 

-----Original Message-----
From: Harald Oehlmann <har...@el...> 
Sent: Monday, January 20, 2025 8:38 PM
To: Tcl Core List <tcl...@li...>
Subject: [TCLCORE] Report of Tcl/Tk biweekly telco

 

Dear Tcl/Tk team,

here is a report of the meeting of today.

 

1) Orphaned extensions

Orphaned extensions may be put into a github repository.

This may not be personal but under the hood of an organisation for legal 

reasons.

Steve will contact Cliff Redler from the Tcl association, if this is 

possible under this hat.

 

2) TCT fossil forum

a TCT forum was proposed for inner TCT communication. Benefit compared 

to E-Mail list is:

- control when and how one gets notified

- organisation of threads

- archive

 

3) TCL infrastructure

Recent fall-outs lead to the opinion that less infra structure is better.

Another opinion is that infra structure is great if you control it.

 

The E-Mail notification feature still does not work for the core. A 

participant reported, that it now works on Chiseal (which is perhaps the 

same machine) and he may try to fix it.

 

4) Tests

Many new features came only with basic tests.

An example is the zip file system, which only has basic tests.

It was proposed to require a higher test coverage before a merge to the 

productive branches.

On the other hand, Wizards just do and don't need tests at all.

And we are happy to get their great code.

Testing will happen in productive environments, (but there are anyway 

never issues).

And the group of non-believers is charged to write tests.

The discussion lead in case of the memory load module to the result:

- the code is correct, no doubt

- there will ne more test cases for those non believers

 

To say it positive: we respect each others opinion and work on it - great !

 

5) Next meeting:

2025-01-27 12:00 UTC Bi-weekly Tcl/Tk

2025-02-04 18:00 UTC Tk meeting, afterwards "Stammtisch", Harald will 

demonstrate Barcode scanning using tclwmf and zxing-cpp packages

2025-02-10 12:00 UTC Bi-weekly Tcl/Tk (Harald will not be available)

 

Thank you all,

Harald

[TCLCORE] Report of Tcl/Tk biweekly telco

[Harald O. <har...@el...>]

Dear Tcl/Tk team,
here is a report of the meeting of today.

1) Orphaned extensions
Orphaned extensions may be put into a github repository.
This may not be personal but under the hood of an organisation for legal 
reasons.
Steve will contact Cliff Redler from the Tcl association, if this is 
possible under this hat.

2) TCT fossil forum
a TCT forum was proposed for inner TCT communication. Benefit compared 
to E-Mail list is:
- control when and how one gets notified
- organisation of threads
- archive

3) TCL infrastructure
Recent fall-outs lead to the opinion that less infra structure is better.
Another opinion is that infra structure is great if you control it.

The E-Mail notification feature still does not work for the core. A 
participant reported, that it now works on Chiseal (which is perhaps the 
same machine) and he may try to fix it.

4) Tests
Many new features came only with basic tests.
An example is the zip file system, which only has basic tests.
It was proposed to require a higher test coverage before a merge to the 
productive branches.
On the other hand, Wizards just do and don't need tests at all.
And we are happy to get their great code.
Testing will happen in productive environments, (but there are anyway 
never issues).
And the group of non-believers is charged to write tests.
The discussion lead in case of the memory load module to the result:
- the code is correct, no doubt
- there will ne more test cases for those non believers

To say it positive: we respect each others opinion and work on it - great !

5) Next meeting:
2025-01-27 12:00 UTC Bi-weekly Tcl/Tk
2025-02-04 18:00 UTC Tk meeting, afterwards "Stammtisch", Harald will 
demonstrate Barcode scanning using tclwmf and zxing-cpp packages
2025-02-10 12:00 UTC Bi-weekly Tcl/Tk (Harald will not be available)

Thank you all,
Harald

Re: [TCLCORE] biweekly Tcl/Tk telco on 20th of January 12:00 UTC

[Harald O. <har...@el...>]

Hi Nicolas,
thank you ! Here is the link:
https://meet.jit.si/TclMonthlyMeetup

See you in 4 hours,
Harald

Am 19.01.2025 um 12:56 schrieb nicolas bats:
> Hi Harald,
> https://wiki.tcl-lang.org/ <https://wiki.tcl-lang.org/> is down... can 
> you please provide a link for tomorrow's meeting?
> 
> best regards,
> nicolas
> 
> Le dim. 19 janv. 2025 à 11:58, Harald Oehlmann 
> <har...@el... <mailto:har...@el...>> a écrit :
> 
>     Dear Tcl/Tk group,
>     please allow me to remind you about the bi-weekly Tcl/Tk meetings:
> 
>     2025-01-20 12:00 UTC (tomorrow) (probably with Don, Jan not available)
>     2025-01-27 12:00 UTC (in one week) (Jan available)
> 
>     Possible topics:
> 
>     - How to proceed the development. Time frame for Tcl/Tk 8.6.16, 9.0.2,
>     9.1a1-9.1.0
>     - Branches and merge order
>     - Licences
>     - Technical questions
>     - TCT organisation: mailing list, forum,...
> 
>     Happy to see you all tomorrow,
>     Harald

Re: [TCLCORE] biweekly Tcl/Tk telco on 20th of January 12:00 UTC

[nicolas b. <sl1...@gm...>]

Hi Harald,
https://wiki.tcl-lang.org/ is down... can you please provide a link for
tomorrow's meeting?

best regards,
nicolas

Le dim. 19 janv. 2025 à 11:58, Harald Oehlmann <har...@el...>
a écrit :

> Dear Tcl/Tk group,
> please allow me to remind you about the bi-weekly Tcl/Tk meetings:
>
> 2025-01-20 12:00 UTC (tomorrow) (probably with Don, Jan not available)
> 2025-01-27 12:00 UTC (in one week) (Jan available)
>
> Possible topics:
>
> - How to proceed the development. Time frame for Tcl/Tk 8.6.16, 9.0.2,
> 9.1a1-9.1.0
> - Branches and merge order
> - Licences
> - Technical questions
> - TCT organisation: mailing list, forum,...
>
> Happy to see you all tomorrow,
> Harald
>
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core
>

[TCLCORE] biweekly Tcl/Tk telco on 20th of January 12:00 UTC

[Harald O. <har...@el...>]

Dear Tcl/Tk group,
please allow me to remind you about the bi-weekly Tcl/Tk meetings:

2025-01-20 12:00 UTC (tomorrow) (probably with Don, Jan not available)
2025-01-27 12:00 UTC (in one week) (Jan available)

Possible topics:

- How to proceed the development. Time frame for Tcl/Tk 8.6.16, 9.0.2, 
9.1a1-9.1.0
- Branches and merge order
- Licences
- Technical questions
- TCT organisation: mailing list, forum,...

Happy to see you all tomorrow,
Harald

Re: [TCLCORE] CFV warning: TIP #709: MPL Licence for MemoryModule

[Jan N. <jan...@gm...>]

Op vr 17 jan 2025 om 13:53 schreef Harald Oehlmann
<har...@el...>:
> What I understood:
> ...
> This is probably per DLL, as each DLL knows the size
> per thread.
>...
> This potentially leads to memory corruption.

... probably ... potentially ......   That's a lot of speculation!!!!!!

Please don't do that!  The real explanation is much simpler,
I will come back on that.

Regards,
          Jan Nijtmans

Re: [TCLCORE] CFV warning: TIP #709: MPL Licence for MemoryModule

[<apn...@ya...>]

Better summary than mine.

Only the last point - _tcl_index = 0 corresponding to the application - is pure speculation on my part. It is tangential to the main argument anyways.

/Ashok

-----Original Message-----
From: Harald Oehlmann <har...@el...> 
Sent: Friday, January 17, 2025 6:23 PM
To: tcl...@li...
Subject: Re: [TCLCORE] CFV warning: TIP #709: MPL Licence for MemoryModule

Hi Ashok,
thank you for the great analysis.

What I understood:
"TLS" means "Thread Local Storage" (and not "Transport Layer Security"). 
It means, that a static variable gets its own memory place for each 
thread the program is executed for.
This memory is handled by the system and is proper to each DLL. It must 
be extendable in size, as it is not known in advance, how many threads 
will be used. So, there must be a memory management behind, when a 
thread is created. This is probably per DLL, as each DLL knows the size 
per thread.
In assembly, the GS register holds a pointer to thread local storage.
The thread locale variable is accessed in assembly by "Offset:x"+ 
*(GS+88+8*_tls_index).
The current LoadModule implementation takes the TLS pointer from the 
application (e.g. wish or tclsh) and ignores the dll ones.
This potentially leads to memory corruption.

Thanks to dive so deeply into this subject.
I think, this is critical.

Thanks for all,
Harald

Am 16.01.2025 um 19:02 schrieb apnmbx-public--- via Tcl-Core:
> Sigh..another long repetitive post (sorry Christian for the tedium, but
> I do not enjoy this either. Really!). But this time I have
> sample test cases that purportedly illustrate the problem. Folks can
> skip the middle if you want and go directly to the test cases later.
> 
> Jan,
> 
> Perhaps I was not clear enough in the issues I was raising. The topic is
> complex enough which is why I linked to the blogs and source code in my
> previous post. Everything I know comes from there. Explicit illustrative
> test cases are detailed below but first, here's why your test cases do not
> address my concerns *at all*.
> 
> These are the two *simplest* scenarios that are potential TLS issues.
> 
> - Loading of two DLL's within a single thread. Your test loads one DLL
> into two threads. That's not the same thing. I've added a test for this
> which illustrates the data corruption.
> 
> - Loading of a DLL *after* a thread is running from a different thread.
> Your test loads the extension *before* the thread is created. I have not
> bothered to test this. I'm not sure how to until the two test cases
> below are fixed.
> 
> The above do not even involve the more complex scenarios when the TLS
> grows enough to need reallocation, the use of constructors to initialize TLS
> variables and so on.
> 
> Your explanation with respect to exception handling also misses the mark.
> The issue does not have to with Tcl's interfaces at all but strictly apply
> to the extension's internal implementation. *If* the extension uses nested
> exceptions, the loader needs to translate the table mapping address ranges
> to corresponding exception handlers. That is the issue, nothing to do
> with how exceptions are passed to and from Tcl. And as I stated, I do
> not know if MemoryModule does this properly even with PR #91.
> I do know the update MemoryModulePP does a whole lot more in this
> regard than MemoryModule and that has me wondering why.
> 
> You assume it works until proven otherwise. I assume it doesn't work unless
> proven otherwise. That's the fundamental disconnect we have.
> 
> I don't see ASLR tables and CFG tables to be handled at all. I don't if this means
> something will fail or just that the benefits of those features will
> be disabled.
> 
> As far as the LUCK tests go, I do not know what was tested but just loading
> packages is completely meaningless. Or as Christian
> himself phrased it, it means zilch. It's like saying Tcl multithreading
> is race-free because packages load!
> 
> TEST CASES
> 
> But onto the main purpose of this post - the illustrative test case which is
> extremely simple - just load two extensions that have implicit thread
> variables. In the apn-tip-709 branch, I've added test memorymodule-3.0 to
> memorymodule.test with a slight modification to your memorymoduletest
> extension to build two variants. To run it, build two versions of your
> dltest/memorymoduletest extension as follows:
> 
> -------
> cd win
> nmake /f makefile.vc OPTS=memorymodule (builds Tcl with load from memory enabled)
> nmake /f makefile.vc clean
> nmake /f makefile.vc (builds memorymodule extension)
> nmake -f makefile.vc BUILD2=1 (build memorymodule2 extension)
> zip memorymoduletest.zip *.dll pkgIndex.tcl
> cd ..
> Release_AMD64_VC1939\tclsh90.exe ..\tests\memorymodule.test -constraints memorymoduletest -match *-3.0
> ------
> 
> Output is
> 
> ------
> ---- Result was:
> 15 16 17
> ---- Result should have been (exact matching):
> 15 15 16
> ==== memorymodule-3.0 FAILED
> ------
> 
> illustrating that the two DLL's stomp on each other.
> 
> Of course, tests can have bugs as well so do review my changes. But
> nevertheless, building Tcl with memory loading disabled,
> 
> -----
> rmdir/s/q Release_AMD64_VC1939
> nmake /f makefile.vc (build Tcl with load from memory disabled)
> Release_AMD64_VC1939\tclsh90.exe ..\tests\memorymodule.test -constraints memorymoduletest -match *-3.0
> -----
> 
> shows
> 
> ------
> memorymodule.test:      Total   5       Passed  1       Skipped 4       Failed  0
> ------
> 
> which is a strong indicator the test case is not the one at fault.
> 
> A second test which isolates the cause is memorymoduletest-3.1. This
> retrieves the _tls_index (see http://www.nynaeve.net/?p=185) which must
> be different for every DLL loaded. If we try this an interactive shell
> without memory loading,
> 
> -----
> % zipfs mount dltest/memorymoduletest.zip [file join [zipfs root] memorymoduletest]
> //zipfs:/memorymoduletest
> %  lappend auto_path //zipfs:/memorymoduletest
> C:/Tcl/magic/lib/tcl9.0 C:/Tcl/magic/lib //zipfs:/memorymoduletest
> % package require memorymoduletest
> 1.0.0
> % package require memorymoduletest2
> 1.0.0
> % memorymoduletest2::GetTlsIndex
> 5
> %
> % memorymoduletest::GetTlsIndex
> 4
> -----
> 
> As you see the two modules have different and non-0 _tls_index values.
> 
> With memory module loading enabled however, both Dll's have the *same*
> _tls_index value.
> 
> -----
> c:\src\tcltk\tip-709\tcl\win>Release_AMD64_VC1939\tcltest90.exe
> % zipfs mount dltest/memorymoduletest.zip [file join [zipfs root] memorymoduletest]
> //zipfs:/memorymoduletest
> % lappend auto_path //zipfs:/memorymoduletest
> //zipfs:/lib/tcl/tcl_library //zipfs:/lib/tcl c:/src/tcltk/tip-709/tcl/win/lib //zipfs:/memorymoduletest
> % package require memorymoduletest
> 1.0.0
> % package require memorymoduletest2
> 1.0.0
> % memorymoduletest::GetTlsIndex
> 0
> % memorymoduletest2::GetTlsIndex
> 0
> -----
> 
> What's more they are both 0 as MemoryModule has not
> initialized them at all! It is in fact very possible _tls_index 0 is
> allocated to some other module, possibly the main application and
> these DLL's are stomping on its data!
> 
> I am pretty much done with TIP 709 review unless there are
> significant changes. It absolutely needs a more
> filled out test suite no matter what TCT decides but I am not
> going to be trying to "prove" an implementation is broken
> by constructing basic tests. Given there is no test group, that
> onus should be on the developer, not the reviewers, to gain
> confidence in the implementation. The tests I wrote could have
> and should have been constructed by anyone reading that blog.
> 
> Perhaps these failures can be ignored  with the "assurance" that
> Tcl extensions do not make use of implicit threads or the other features I
> mentioned. I'm sure it will all work. Most of the time. I don't support this
> posture myself but I understand others might differ. Note however that use
> of declspec(thread) is much more common now that (a) XP is dead and
> (b) all common compilers support the equivalent.
> 
> Funnily enough, future changes by Microsoft are less of a worry because if
> my test above is valid, MemoryModule anyways does not match the
> implementation they currently have! I'll worry about future implementations
> once MemoryModule works with the current one.
> 
> PS
> 
> For anyone interested, https://godbolt.org/z/Yxs43dc3e shows the assembler
> generated for a two line program using declspec(thread).

Re: [TCLCORE] CFV warning: TIP #709: MPL Licence for MemoryModule

[Harald O. <har...@el...>]

Hi Ashok,
thank you for the great analysis.

What I understood:
"TLS" means "Thread Local Storage" (and not "Transport Layer Security"). 
It means, that a static variable gets its own memory place for each 
thread the program is executed for.
This memory is handled by the system and is proper to each DLL. It must 
be extendable in size, as it is not known in advance, how many threads 
will be used. So, there must be a memory management behind, when a 
thread is created. This is probably per DLL, as each DLL knows the size 
per thread.
In assembly, the GS register holds a pointer to thread local storage.
The thread locale variable is accessed in assembly by "Offset:x"+ 
*(GS+88+8*_tls_index).
The current LoadModule implementation takes the TLS pointer from the 
application (e.g. wish or tclsh) and ignores the dll ones.
This potentially leads to memory corruption.

Thanks to dive so deeply into this subject.
I think, this is critical.

Thanks for all,
Harald

Am 16.01.2025 um 19:02 schrieb apnmbx-public--- via Tcl-Core:
> Sigh..another long repetitive post (sorry Christian for the tedium, but
> I do not enjoy this either. Really!). But this time I have
> sample test cases that purportedly illustrate the problem. Folks can
> skip the middle if you want and go directly to the test cases later.
> 
> Jan,
> 
> Perhaps I was not clear enough in the issues I was raising. The topic is
> complex enough which is why I linked to the blogs and source code in my
> previous post. Everything I know comes from there. Explicit illustrative
> test cases are detailed below but first, here's why your test cases do not
> address my concerns *at all*.
> 
> These are the two *simplest* scenarios that are potential TLS issues.
> 
> - Loading of two DLL's within a single thread. Your test loads one DLL
> into two threads. That's not the same thing. I've added a test for this
> which illustrates the data corruption.
> 
> - Loading of a DLL *after* a thread is running from a different thread.
> Your test loads the extension *before* the thread is created. I have not
> bothered to test this. I'm not sure how to until the two test cases
> below are fixed.
> 
> The above do not even involve the more complex scenarios when the TLS
> grows enough to need reallocation, the use of constructors to initialize TLS
> variables and so on.
> 
> Your explanation with respect to exception handling also misses the mark.
> The issue does not have to with Tcl's interfaces at all but strictly apply
> to the extension's internal implementation. *If* the extension uses nested
> exceptions, the loader needs to translate the table mapping address ranges
> to corresponding exception handlers. That is the issue, nothing to do
> with how exceptions are passed to and from Tcl. And as I stated, I do
> not know if MemoryModule does this properly even with PR #91.
> I do know the update MemoryModulePP does a whole lot more in this
> regard than MemoryModule and that has me wondering why.
> 
> You assume it works until proven otherwise. I assume it doesn't work unless
> proven otherwise. That's the fundamental disconnect we have.
> 
> I don't see ASLR tables and CFG tables to be handled at all. I don't if this means
> something will fail or just that the benefits of those features will
> be disabled.
> 
> As far as the LUCK tests go, I do not know what was tested but just loading
> packages is completely meaningless. Or as Christian
> himself phrased it, it means zilch. It's like saying Tcl multithreading
> is race-free because packages load!
> 
> TEST CASES
> 
> But onto the main purpose of this post - the illustrative test case which is
> extremely simple - just load two extensions that have implicit thread
> variables. In the apn-tip-709 branch, I've added test memorymodule-3.0 to
> memorymodule.test with a slight modification to your memorymoduletest
> extension to build two variants. To run it, build two versions of your
> dltest/memorymoduletest extension as follows:
> 
> -------
> cd win
> nmake /f makefile.vc OPTS=memorymodule (builds Tcl with load from memory enabled)
> nmake /f makefile.vc clean
> nmake /f makefile.vc (builds memorymodule extension)
> nmake -f makefile.vc BUILD2=1 (build memorymodule2 extension)
> zip memorymoduletest.zip *.dll pkgIndex.tcl
> cd ..
> Release_AMD64_VC1939\tclsh90.exe ..\tests\memorymodule.test -constraints memorymoduletest -match *-3.0
> ------
> 
> Output is
> 
> ------
> ---- Result was:
> 15 16 17
> ---- Result should have been (exact matching):
> 15 15 16
> ==== memorymodule-3.0 FAILED
> ------
> 
> illustrating that the two DLL's stomp on each other.
> 
> Of course, tests can have bugs as well so do review my changes. But
> nevertheless, building Tcl with memory loading disabled,
> 
> -----
> rmdir/s/q Release_AMD64_VC1939
> nmake /f makefile.vc (build Tcl with load from memory disabled)
> Release_AMD64_VC1939\tclsh90.exe ..\tests\memorymodule.test -constraints memorymoduletest -match *-3.0
> -----
> 
> shows
> 
> ------
> memorymodule.test:      Total   5       Passed  1       Skipped 4       Failed  0
> ------
> 
> which is a strong indicator the test case is not the one at fault.
> 
> A second test which isolates the cause is memorymoduletest-3.1. This
> retrieves the _tls_index (see http://www.nynaeve.net/?p=185) which must
> be different for every DLL loaded. If we try this an interactive shell
> without memory loading,
> 
> -----
> % zipfs mount dltest/memorymoduletest.zip [file join [zipfs root] memorymoduletest]
> //zipfs:/memorymoduletest
> %  lappend auto_path //zipfs:/memorymoduletest
> C:/Tcl/magic/lib/tcl9.0 C:/Tcl/magic/lib //zipfs:/memorymoduletest
> % package require memorymoduletest
> 1.0.0
> % package require memorymoduletest2
> 1.0.0
> % memorymoduletest2::GetTlsIndex
> 5
> %
> % memorymoduletest::GetTlsIndex
> 4
> -----
> 
> As you see the two modules have different and non-0 _tls_index values.
> 
> With memory module loading enabled however, both Dll's have the *same*
> _tls_index value.
> 
> -----
> c:\src\tcltk\tip-709\tcl\win>Release_AMD64_VC1939\tcltest90.exe
> % zipfs mount dltest/memorymoduletest.zip [file join [zipfs root] memorymoduletest]
> //zipfs:/memorymoduletest
> % lappend auto_path //zipfs:/memorymoduletest
> //zipfs:/lib/tcl/tcl_library //zipfs:/lib/tcl c:/src/tcltk/tip-709/tcl/win/lib //zipfs:/memorymoduletest
> % package require memorymoduletest
> 1.0.0
> % package require memorymoduletest2
> 1.0.0
> % memorymoduletest::GetTlsIndex
> 0
> % memorymoduletest2::GetTlsIndex
> 0
> -----
> 
> What's more they are both 0 as MemoryModule has not
> initialized them at all! It is in fact very possible _tls_index 0 is
> allocated to some other module, possibly the main application and
> these DLL's are stomping on its data!
> 
> I am pretty much done with TIP 709 review unless there are
> significant changes. It absolutely needs a more
> filled out test suite no matter what TCT decides but I am not
> going to be trying to "prove" an implementation is broken
> by constructing basic tests. Given there is no test group, that
> onus should be on the developer, not the reviewers, to gain
> confidence in the implementation. The tests I wrote could have
> and should have been constructed by anyone reading that blog.
> 
> Perhaps these failures can be ignored  with the "assurance" that
> Tcl extensions do not make use of implicit threads or the other features I
> mentioned. I'm sure it will all work. Most of the time. I don't support this
> posture myself but I understand others might differ. Note however that use
> of declspec(thread) is much more common now that (a) XP is dead and
> (b) all common compilers support the equivalent.
> 
> Funnily enough, future changes by Microsoft are less of a worry because if
> my test above is valid, MemoryModule anyways does not match the
> implementation they currently have! I'll worry about future implementations
> once MemoryModule works with the current one.
> 
> PS
> 
> For anyone interested, https://godbolt.org/z/Yxs43dc3e shows the assembler
> generated for a two line program using declspec(thread).

[TCLCORE] TkChat bridge down 2 days

[Colin M. <col...@ya...>]

Hello all,

The Tcler's Chat irc/jabber bridge "ijchain" has been down for two days 
now.  Is there anyone around who can give it a kick in the requisite place?

Thanks,
Colin.

Re: [TCLCORE] CFV warning: TIP #709: MPL Licence for MemoryModule

[<apn...@ya...>]

Sigh..another long repetitive post (sorry Christian for the tedium, but
I do not enjoy this either. Really!). But this time I have
sample test cases that purportedly illustrate the problem. Folks can
skip the middle if you want and go directly to the test cases later.

Jan,

Perhaps I was not clear enough in the issues I was raising. The topic is
complex enough which is why I linked to the blogs and source code in my
previous post. Everything I know comes from there. Explicit illustrative
test cases are detailed below but first, here's why your test cases do not
address my concerns *at all*.

These are the two *simplest* scenarios that are potential TLS issues.

- Loading of two DLL's within a single thread. Your test loads one DLL
into two threads. That's not the same thing. I've added a test for this
which illustrates the data corruption.

- Loading of a DLL *after* a thread is running from a different thread.
Your test loads the extension *before* the thread is created. I have not
bothered to test this. I'm not sure how to until the two test cases
below are fixed.

The above do not even involve the more complex scenarios when the TLS
grows enough to need reallocation, the use of constructors to initialize TLS
variables and so on.

Your explanation with respect to exception handling also misses the mark.
The issue does not have to with Tcl's interfaces at all but strictly apply
to the extension's internal implementation. *If* the extension uses nested
exceptions, the loader needs to translate the table mapping address ranges
to corresponding exception handlers. That is the issue, nothing to do
with how exceptions are passed to and from Tcl. And as I stated, I do
not know if MemoryModule does this properly even with PR #91. 
I do know the update MemoryModulePP does a whole lot more in this
regard than MemoryModule and that has me wondering why. 

You assume it works until proven otherwise. I assume it doesn't work unless
proven otherwise. That's the fundamental disconnect we have.

I don't see ASLR tables and CFG tables to be handled at all. I don't if this means
something will fail or just that the benefits of those features will
be disabled.

As far as the LUCK tests go, I do not know what was tested but just loading
packages is completely meaningless. Or as Christian
himself phrased it, it means zilch. It's like saying Tcl multithreading
is race-free because packages load!

TEST CASES

But onto the main purpose of this post - the illustrative test case which is
extremely simple - just load two extensions that have implicit thread
variables. In the apn-tip-709 branch, I've added test memorymodule-3.0 to
memorymodule.test with a slight modification to your memorymoduletest
extension to build two variants. To run it, build two versions of your
dltest/memorymoduletest extension as follows:

-------
cd win
nmake /f makefile.vc OPTS=memorymodule (builds Tcl with load from memory enabled)
nmake /f makefile.vc clean
nmake /f makefile.vc (builds memorymodule extension)
nmake -f makefile.vc BUILD2=1 (build memorymodule2 extension)
zip memorymoduletest.zip *.dll pkgIndex.tcl
cd ..
Release_AMD64_VC1939\tclsh90.exe ..\tests\memorymodule.test -constraints memorymoduletest -match *-3.0
------

Output is 

------
---- Result was:
15 16 17
---- Result should have been (exact matching):
15 15 16
==== memorymodule-3.0 FAILED
------

illustrating that the two DLL's stomp on each other.

Of course, tests can have bugs as well so do review my changes. But
nevertheless, building Tcl with memory loading disabled,

-----
rmdir/s/q Release_AMD64_VC1939
nmake /f makefile.vc (build Tcl with load from memory disabled)
Release_AMD64_VC1939\tclsh90.exe ..\tests\memorymodule.test -constraints memorymoduletest -match *-3.0
-----

shows

------
memorymodule.test:      Total   5       Passed  1       Skipped 4       Failed  0
------

which is a strong indicator the test case is not the one at fault.

A second test which isolates the cause is memorymoduletest-3.1. This
retrieves the _tls_index (see http://www.nynaeve.net/?p=185) which must
be different for every DLL loaded. If we try this an interactive shell
without memory loading,

-----
% zipfs mount dltest/memorymoduletest.zip [file join [zipfs root] memorymoduletest]
//zipfs:/memorymoduletest
%  lappend auto_path //zipfs:/memorymoduletest
C:/Tcl/magic/lib/tcl9.0 C:/Tcl/magic/lib //zipfs:/memorymoduletest
% package require memorymoduletest
1.0.0
% package require memorymoduletest2
1.0.0
% memorymoduletest2::GetTlsIndex
5
%
% memorymoduletest::GetTlsIndex
4
-----

As you see the two modules have different and non-0 _tls_index values.

With memory module loading enabled however, both Dll's have the *same*
_tls_index value. 

-----
c:\src\tcltk\tip-709\tcl\win>Release_AMD64_VC1939\tcltest90.exe
% zipfs mount dltest/memorymoduletest.zip [file join [zipfs root] memorymoduletest]
//zipfs:/memorymoduletest
% lappend auto_path //zipfs:/memorymoduletest
//zipfs:/lib/tcl/tcl_library //zipfs:/lib/tcl c:/src/tcltk/tip-709/tcl/win/lib //zipfs:/memorymoduletest
% package require memorymoduletest
1.0.0
% package require memorymoduletest2
1.0.0
% memorymoduletest::GetTlsIndex
0
% memorymoduletest2::GetTlsIndex
0
-----

What's more they are both 0 as MemoryModule has not
initialized them at all! It is in fact very possible _tls_index 0 is
allocated to some other module, possibly the main application and
these DLL's are stomping on its data!

I am pretty much done with TIP 709 review unless there are
significant changes. It absolutely needs a more
filled out test suite no matter what TCT decides but I am not
going to be trying to "prove" an implementation is broken
by constructing basic tests. Given there is no test group, that
onus should be on the developer, not the reviewers, to gain
confidence in the implementation. The tests I wrote could have
and should have been constructed by anyone reading that blog.

Perhaps these failures can be ignored  with the "assurance" that
Tcl extensions do not make use of implicit threads or the other features I
mentioned. I'm sure it will all work. Most of the time. I don't support this
posture myself but I understand others might differ. Note however that use
of declspec(thread) is much more common now that (a) XP is dead and
(b) all common compilers support the equivalent.

Funnily enough, future changes by Microsoft are less of a worry because if
my test above is valid, MemoryModule anyways does not match the
implementation they currently have! I'll worry about future implementations
once MemoryModule works with the current one.

PS

For anyone interested, https://godbolt.org/z/Yxs43dc3e shows the assembler
generated for a two line program using declspec(thread).

Re: [TCLCORE] CFV warning: TIP #709: MPL Licence for MemoryModule

[Christian W. <Chr...@t-...>]

On 01/15/2025 12:36 PM, Jan Nijtmans wrote:

> ...
> The LUCK framework supports the following (more than 150!) dll's being loaded
> by MemoryModule:
> ...
> I suspect that none of them use exception handling or TLS. If you show me some
> example code, I'm more than happy to include that as an additional testcase.

I'm not aware of any extension using TLS indeed.

There's at least one extension coded in C++ which uses exceptions internally but
proper maps them to TCL_ERROR for external call sites. This is the olden tclodbc.

Yes, I know, this proves formally exactly nothing, zilch. And since the
undroidwish/vanillawish binaries are only build using outdated mingw cross
compilers, even sub zero.

Anyway, for me the whole discussion is somewhat tedious. Let's make that into
an optional experimental feature. And, if need be, write a disclaimer as long
as the MemoryModule itself. And life can go on.

BR,
Christian