tcl-core — Tcl Core Team discussions about Tcl development

Re: [TCLCORE] CFV warning: TIP #709: MPL Licence for MemoryModule

[Christian W. <Chr...@t-...>]

On 01/15/2025 12:36 PM, Jan Nijtmans wrote:

> ...
> The LUCK framework supports the following (more than 150!) dll's being loaded
> by MemoryModule:
> ...
> I suspect that none of them use exception handling or TLS. If you show me some
> example code, I'm more than happy to include that as an additional testcase.

I'm not aware of any extension using TLS indeed.

There's at least one extension coded in C++ which uses exceptions internally but
proper maps them to TCL_ERROR for external call sites. This is the olden tclodbc.

Yes, I know, this proves formally exactly nothing, zilch. And since the
undroidwish/vanillawish binaries are only build using outdated mingw cross
compilers, even sub zero.

Anyway, for me the whole discussion is somewhat tedious. Let's make that into
an optional experimental feature. And, if need be, write a disclaimer as long
as the MemoryModule itself. And life can go on.

BR,
Christian

Re: [TCLCORE] CFV warning: TIP #709: MPL Licence for MemoryModule

[Jan N. <jan...@gm...>]

Op wo 15 jan 2025 om 11:43 schreef Ashok:
> I do wish Jan, assuming he has read the references I posted previously, would address at least the two points I specifically asked about in my post. To quote,
>
> Per my reading of just the TLS-related portions,
> •            It does not allocate slots for the *implicit* TLS entries and initialize them in the loading thread.
> •            It does not update the TLS map for existing threads by reallocating (if necessary) the TLS storage and updating the in-use slot map.

See below. I never used those TLS features, and I doubt any existing
Tcl extension
uses it. Can you supply (or point to) a little bit of example code?

> If the intent is to have memory loading work for a restricted set of extensions (say no exception handling or implicit TLS) because that is useful enough to accept current risk and future risk in case of Windows changes, folks can by all means vote yes. Please do add test cases though to avoid a repetition of the zipfs situation before 9.0.

Regarding exception handling, I can be very short: the Tcl core
doesn't handle (C++) exception handling, if it is thrown from a dll
or from anywhere else. Any exception thrown internally in a dll, must
be caught within the same dll, and translated
to TCL_OK or TCL_ERROR with an error-message. The throw/catch handling
is fully within the loaded dll, which
is handled by Pull request #91. So, it should work in MemoryModule.
Two questions:
1) Can you supply a piece of test-code you suspect that won't work?
2) Can you point to any Tcl extension dll in the field using this?

Regarding TLS, same question:
1) Can you supply a piece of test-code you suspect that won't work?
2) Can you point to any Tcl extension dll in the field using this?

The future risk in Windows Changes: I don't see that happen. The
changes in Vista were
probably either for security reasons, or for extension to x64, both
are being used a long time
now in various Windows versions. I don't see the need for more changes
coming. Next
change in the Windows kernel is probably the switch to the Linux kernel.

The LUCK framework supports the following (more than 150!) dll's being loaded
by MemoryModule:
---------------------------------------------------------------------------
 app-sdx2.0  augeas0.5.0  autoopts0  awthemes10  azure-ttk  bcrypt2.0
blt2.4  bonjour1.1  BSD1.9.2  bwidget1.10  bz20.1  calc0  can2svg0.3
Canvas3d1.2.4  ceptcl0.4  classview0.1  dbus  DiffUtil0.4.2  dmtx0.7.5
 espeak0.2  Ffidl0.7  flexmenu1  forest-ttk  fsdialog1.15
fswatch2.0.1  fuse1.1  gridplus2.11  helpviewer3.0.2  icons2
Img1.4.11  imgjp20.1  itcl4.2.0  itk4.1.0  iwidgets4.1  kafka2.4
legacy_3dcanvas  lmdb0.4.3  lzma0.1  materialicons0.2  Memchan2.4
mkappimg  modbus0.1  Mpexpr12  mqtt2.0  mqtt3.1  msgpack2  nats3
notebook2.2  nsf2.4.0  ooxml1  parse_args0.5.1  parser1.8  pdf4tcl09
pdf4tcl_graph1.0  photoframe1  piio1.3  pikchr1.0  promise1
pty_tcl0.1  puppyicons0.1  ral0.12.2  ralutil0.12.2  retcl0  rhash0.1
rl_json0.15.1  Rtcl1.2.2  scrolldata2  sdx1.0  snack2.2  snap70.1
sqlite3.47.2  stardom0.42  starsync1.0  stbimage1.2  stringfileinfo0.2
 sun-valley-ttk  tangoicons0.1  tbcload1.7  tclcan0.1
tclcompiler1.7.1  tclcsv2.4.3  TclCurl7.22.0  tclepeg0.4
tclJBlend2.1.0  tcllib1.21  tcllibc1.21  TclMagick0.46  tclrmq1.4.5
tclsoap1.6.8  tcltaglib1.1  tcluvc0.1  tclws2  tclx8.6  tdbc1.1.1
tdbcjdbc0.2  tdbcmysql1.1.5  tdbcodbc1.1.1  tdbcpostgres1.1.1
tdom0.9.3  tfirmata  thread2.8.5  tile-extras  Tix8.4.3  tkbugz
tkchat1.500  tkcon2.7  tkconclient1.0  Tkhtml3.0  tkinspect5.1.6
tkled0.1  tklib0.7  TkMC1.0  tknotebook0.1  tkpath0.3.3
tksqlite0.5.13  tksvg0.14  Tktable2.11  tkvlc0.8  Tkzinc3.3.6
tls1.6.9  tnc0.3.0  tomato1  topcua0.5  touchcal0.1  treectrl2.4.2
Trf2.1.4  trofs0.4.9  tserialport1.1  ttk-themes  twdbgi0.1  udp1.0.11
 ukaz2.1  ulis  unqlite0.3.8  upnp0.2  v4l20.1  vcd0.1  vectcl0.3
VecTcLab  vectcltk0.2  vfs1.4.2  vnc0.5  vqtcl4.1  vu2.3  WavReader0.1
 wibble0.4  wscope-1.0b  www2  xdgicons1.0  yeti0.4.2
---------------------------------------------------------
I suspect that none of them use exception handling or TLS. If you show me some
example code, I'm more than happy to include that as an additional testcase.

Thanks!
         Jan Nijtmans

Re: [TCLCORE] CFV warning: TIP #709: MPL Licence for MemoryModule

[<apn...@ya...>]

 

Jan wrote:

Tk uses Windows manifests and Windows resources, both were brought

up by Ashok as not-working. It simply worked. Because of Ashok's feedback

I did additional testing with TLS (thread-local storage). That worked too.

 

I mentioned TLS, manifests, exception handling and resources in the general
case. Not the very simple uses (or no uses) within Tk. Your test DLL (for
TLS) when I last saw it did not either for reasons I reiterated in my
previous post (single scalar TLS int which will be indistinguishable from a
normal static when the test is run).

 

Jan wrote:

Ashok's cleanup proposal has commit message "Proof of concept cleaning

up temp DLLs". That should tell enough ....

 

I am not sure what you think it is telling! Afair the code is complete but
still needs a review. It is indeed not a proposal because before TIP'ing or
writing a test suite, I was waiting for 709 to come to a conclusion one way
or another. If a robust 709 comes into being, there is no need for it.

 

I do wish Jan, assuming he has read the references I posted previously,
would address at least the two points I specifically asked about in my post.
To quote,

 

Per my reading of just the TLS-related portions,

.            It does not allocate slots for the *implicit* TLS entries and
initialize them in the loading thread.

.            It does not update the TLS map for existing threads by
reallocating (if necessary) the TLS storage and updating the in-use slot
map.

 

If the intent is to have memory loading work for a restricted set of
extensions (say no exception handling or implicit TLS) because that is
useful enough to accept current risk and future risk in case of Windows
changes, folks can by all means vote yes. Please do add test cases though to
avoid a repetition of the zipfs situation before 9.0.

 

/Ashok

 

-----Original Message-----
From: Jan Nijtmans <jan...@gm...> 
Sent: Monday, January 13, 2025 11:14 PM
To: Marc Culler <cul...@gm...>
Cc: Tcl Core List <tcl...@li...>
Subject: Re: [TCLCORE] CFV warning: TIP #709: MPL Licence for MemoryModule

 

Op ma 13 jan 2025 om 16:45 schreef Marc Culler:

> * The TIP should be modified to only address the licensing issue, and not
include the implementation of memory-mapped DLL files.

 

If there is a desire for a separate licensing-TIP, that's OK with me. But
then

I prefer TIP #709 to address the MemoryModule solution.

 

> * The TIP rationale should explicitly mention the real problem at hand -
namely, creating single file Tcl applications as zip files without making a
mess in a temp directory.  I also think it would be helpful to make it clear
that the virtual file systems being discussed are Zip filesystems.

 

I'm still open to suggestions for TIP text improvements. (Nathan

already took care of the Spelling/Wording!)

 

> * The implementation seems to still be a work in progress.  There are
multiple approaches being tested right now.  Why not wait until the various
strategies have been implemented, tested and compared before finalizing a
TIP for the implementation?

I think your impression is wrong. The implementation is not a work in

progress, not any more than Tcl is a work in progress ;-). My

milestone was to pack tcl9tk90.dll into a zip-file and make Tk work from
there.

Tk uses Windows manifests and Windows resources, both were brought

up by Ashok as not-working. It simply worked. Because of Ashok's feedback

I did additional testing with TLS (thread-local storage). That worked too.

So, the milestone is reached.

 

I am not aware of any other approaches being tested now, nor of any of

the various

strategies implemented. So, if you want to wait for that, we can wait
forever.

Ashok's cleanup proposal has commit message "Proof of concept cleaning

up temp DLLs". That should tell enough ....

 

Thanks!

 

Regards,

     Jan Nijtmans

 

 

_______________________________________________

Tcl-Core mailing list

 <mailto:Tcl...@li...> Tcl...@li...

 <https://lists.sourceforge.net/lists/listinfo/tcl-core>
https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] scrollable canvas perfomance degraded over versions

[Gerhard H. <ger...@gm...>]

Andreas,
you're right about the xorg version number, was also wondering about the
big difference, the gentoo package manager says 21.1.14, but X says
1.21.1.14
Will try to complete the OK/SLOW matrix with a few tests and post an
example script for reference. Will take longer as I retrieve the info from
a database.
When digging a little deeper I found out, that the delay comes from ssh X11
forwarding, so as  long I use the local display I have quick response
times, when using the ssh X11 tunnel, things get slow, I mean really
everything: creating the widgets in the frame takes longer, waiting for the
geometry manager to update all widget sizes in update idletasks, this is
even the case when beeing on the same machine but different user with ssh
X11 port forwarding.
When having ~ 500 labels in a frame, on local display I see the scrollable
canvas rendered immediately, when using the ssh X11 forwarding on the same
machine where I did the local display test, it takes ~ 25 secs (to create
the labels, wait for the geometries to be calculated - basically i display
a table with many rows and columns - and the toplevel to be centered based
on the geometry. Weird. I'm using the latest stable (gentoo distro)
openssh(9.8_p1-r3)/openssl(3.3.2-r1).

On Mon, Jan 13, 2025 at 11:59 PM Andreas Kupries <and...@gm...>
wrote:

>
> > before posting a bug report I would like to check other people's
> experience
> > with scrollable canvas performance.
>
> > I have implemented a canvas with a vertical scrollbar next to it and
> > placed a window object (frame) inside the canvas holding about 350
> > labels to be able to scroll the frame in the canvas. Running the
> > same script in 8.6.9 (xorg 1.20.7) displays the canvas nearly
> > immediately, running it on 8.6.14 (xorg 21.1.14) it takes ages ~ 20
> > secs) before the canvas gets displayed.
>
> While I cannot really say anything about the canvas internals, etc.
> I do note that your trials changed two things!
>
> Both Tcl version and Xorg version changed.
> The Xorg version even significantly (*).
>
> And given that we are talking about heavy duty drawing here I would
> not exclude Xorg differences as possible source of the issue.
>
> What are the behaviours for the two combinations you have not tried ?
> In other words:
>
> Tcl 8.6.9       Xorg  1.20.7    OK
> Tcl 8.6.9       Xorg 21.1.14    ????
> Tcl 8.6.14      Xorg  1.20.7    ????
> Tcl 8.6.14      Xorg 21.1.14    Slow
>
> Alternatively:
>         +-----+------+
>         |8.6.9|8.6.14|
> +-------+-----+------+
> | 1.20.7| OK  | ???  |
> +-------+-----+------+
> |21.1.14| ??? | SLOW |
> +-------+-----+------+
>
> (*) Although, the 21.1.14 could be a typo, and the actual version was
>     1.21.14. That might fit better.
>
> > FYI there is an "update idletasks" invoked, before the window
> > containing the widget is displayed, so I can center the window on
> > the screen.  Anyone else seen this strange behaviour ?
>
> --
> Happy Tcling,
>         Andreas Kupries <and...@gm...>
>                         <https://core.tcl-lang.org/akupries/>
>                         <https://akupries.tclers.tk/>
>         Developer @     SUSE Software Solutions Germany GmbH
>
> -------------------------------------------------------------------------------
>
>
>

-- 
Gerhard

Re: [TCLCORE] CFV - TIP 705: Affirm Tcl License

[Kevin W. <kw...@co...>]

<div><img width="1" height="1" src='https://fedbdhd.r.tsp1-brevo.net/tr/op/DFfiZJrXwLOe6YdJVO_SfVb8z_dXp9V5Tad1-ypRHLuokRO42Hmk763q9fk329VPD1Rnw11l-ue8iC1v5xBFWUsVM-a614sDBx5nJcVlSR2hNKGxoTyvZWqcaC7O-AWTCAuuJRKLjWpnl7V660Me1S-YBpCOeFQmHeAW0ikToAa5qafqCaXX0OqvKuHkUIwl-MKuNpKMTHOFL_FXqPEYpnArORnG' /></div><br/>On 1/8/25 7:55 PM, Kevin Walzer wrote:<br/>> This is a CFV for TIP 705. Let's have the vote in by 01/13/2025.<br/><br/>YES: AK, APN, BG, FV, HO, JD, JN, KW, MC, RA, SL<br/><br/>NO: None<br/><br/>PRESENT: None<br/><br/>TIP 705 is approved.<br/><br/>Thanks to all,<br/><br/>Kevin<br/><br/>

Re: [TCLCORE] scrollable canvas perfomance degraded over versions

[Andreas K. <and...@gm...>]

> before posting a bug report I would like to check other people's experience
> with scrollable canvas performance.

> I have implemented a canvas with a vertical scrollbar next to it and
> placed a window object (frame) inside the canvas holding about 350
> labels to be able to scroll the frame in the canvas. Running the
> same script in 8.6.9 (xorg 1.20.7) displays the canvas nearly
> immediately, running it on 8.6.14 (xorg 21.1.14) it takes ages ~ 20
> secs) before the canvas gets displayed.

While I cannot really say anything about the canvas internals, etc.
I do note that your trials changed two things!

Both Tcl version and Xorg version changed.
The Xorg version even significantly (*).

And given that we are talking about heavy duty drawing here I would
not exclude Xorg differences as possible source of the issue.

What are the behaviours for the two combinations you have not tried ?
In other words:

Tcl 8.6.9	Xorg  1.20.7	OK
Tcl 8.6.9	Xorg 21.1.14	????
Tcl 8.6.14	Xorg  1.20.7	????
Tcl 8.6.14	Xorg 21.1.14	Slow

Alternatively:
        +-----+------+
        |8.6.9|8.6.14|
+-------+-----+------+
| 1.20.7| OK  | ???  |
+-------+-----+------+
|21.1.14| ??? | SLOW |
+-------+-----+------+

(*) Although, the 21.1.14 could be a typo, and the actual version was
    1.21.14. That might fit better.

> FYI there is an "update idletasks" invoked, before the window
> containing the widget is displayed, so I can center the window on
> the screen.  Anyone else seen this strange behaviour ?

-- 
Happy Tcling,
	Andreas Kupries <and...@gm...>
			<https://core.tcl-lang.org/akupries/>
			<https://akupries.tclers.tk/>
	Developer @	SUSE Software Solutions Germany GmbH
-------------------------------------------------------------------------------

Re: [TCLCORE] CFV warning: TIP #709: MPL Licence for MemoryModule

[Marc C. <cul...@gm...>]

On Mon, Jan 13, 2025 at 11:44 AM Jan Nijtmans <jan...@gm...>
wrote:

>
> I am not aware of any other approaches being tested now, nor of any of
> the various strategies implemeted.


You may not consider Ashok's cleanup proposal to be a strategy or to have
an implementation, but that seems to contradict what he was saying earlier
in this thread.  My impressions just come from what I am reading here.  As
I said, I do not feel that I have enough Windows experience to judge.  And
I certainly could not (and would not try to) predict how long we would have
to wait for Ashok's implementation to be finished.

I do think we could quickly discuss and pass a TIP which was limited to
declaring the MPL license to be Tcl-compatible.

- Marc

Re: [TCLCORE] CFV warning: TIP #709: MPL Licence for MemoryModule

[Jan N. <jan...@gm...>]

Op ma 13 jan 2025 om 16:45 schreef Marc Culler:
> * The TIP should be modified to only address the licensing issue, and not include the implementation of memory-mapped DLL files.

If there is a desire for a separate licensing-TIP, that's OK with me. But then
I prefer TIP #709 to address the MemoryModule solution.

> * The TIP rationale should explicitly mention the real problem at hand - namely, creating single file Tcl applications as zip files without making a mess in a temp directory.  I also think it would be helpful to make it clear that the virtual file systems being discussed are Zip filesystems.

I'm still open to suggestions for TIP text improvements. (Nathan
already took care of the Spelling/Wording!)

> * The implementation seems to still be a work in progress.  There are multiple approaches being tested right now.  Why not wait until the various strategies have been implemented, tested and compared before finalizing a TIP for the implementation?
I think your impression is wrong. The implementation is not a work in
progress, not any more than Tcl is a work in progress ;-). My
milestone was to pack tcl9tk90.dll into a zip-file and make Tk work from there.
Tk uses Windows manifests and Windows resources, both were brought
up by Ashok as not-working. It simply worked. Because of Ashok's feedback
I did additional testing with TLS (thread-local storage). That worked too.
So, the milestone is reached.

I am not aware of any other approaches being tested now, nor of any of
the various
strategies implemented. So, if you want to wait for that, we can wait forever.
Ashok's cleanup proposal has commit message "Proof of concept cleaning
up temp DLLs". That should tell enough ....

Thanks!

Regards,
     Jan Nijtmans

Re: [TCLCORE] [External] Re: CFV warning: TIP #709: MPL Licence for MemoryModule

[<apn...@ya...>]

That was what I was referring to when I referenced branch
https://core.tcl-lang.org/tcl/timeline?r=apn-bug-a8e4f76ce4.

 

From: da Silva, Peter J <pet...@fl...> 
Sent: Monday, January 13, 2025 9:45 PM
To: Kevin Walzer <kw...@co...>; Brian Griffin
<bri...@ea...>
Cc: Tcl Core List <tcl...@li...>
Subject: Re: [TCLCORE] [External] Re: CFV warning: TIP #709: MPL Licence for
MemoryModule

 

Since Windows doesn't have a hierarchy of processes the same way UNIX does,
couldn't you start a cleanup program, exit, and have the cleanup program
delete the libraries? 

Re: [TCLCORE] [External] Re: CFV warning: TIP #709: MPL Licence for MemoryModule

[Harald O. <har...@el...>]

Am 13.01.2025 um 17:15 schrieb da Silva, Peter J:
> Since Windows doesn’t have a hierarchy of processes the same way UNIX 
> does, couldn’t you start a cleanup program, exit, and have the cleanup 
> program delete the libraries?
Hi Peter,
thank you for the great suggestion. That would work.

The currently best solution is to clean-up all TCL temps on the next 
startup. By this strategy, there is only one old temp folder.

To all: this problem is not so important.
Solutions and strategies exist for this since around 20 years, when this 
was developed with the starkit technology by PC Wippler.

A fix would be nice but is not at all mandatory.

Thank you all for your great thinking !
Harald

Re: [TCLCORE] [External] Re: CFV warning: TIP #709: MPL Licence for MemoryModule

[da S. P. J <pet...@fl...>]

Since Windows doesn’t have a hierarchy of processes the same way UNIX does, couldn’t you start a cleanup program, exit, and have the cleanup program delete the libraries?

Re: [TCLCORE] CFV warning: TIP #709: MPL Licence for MemoryModule

[Marc C. <cul...@gm...>]

Here is my suggestion for TIP #709:

* The TIP should be modified to only address the licensing issue, and not
include the implementation of memory-mapped DLL files.
* The TIP rationale should explicitly mention the real problem at hand -
namely, creating single file Tcl applications as zip files without making a
mess in a temp directory.  I also think it would be helpful to make it
clear that the virtual file systems being discussed are Zip filesystems.

The rationale for my suggestion is:
* The licensing issue seems straightforward
* The implementation seems to still be a work in progress.  There are
multiple approaches being tested right now.  Why not wait until the various
strategies have been implemented, tested and compared before finalizing a
TIP for the implementation?

- Marc

[TCLCORE] scrollable canvas perfomance degraded over versions

[Gerhard H. <ger...@gm...>]

before posting a bug report I would like to check other people's experience
with scrollable canvas performance.
I have implemented a canvas with a vertical scrollbar next to it and placed
a window object (frame) inside the canvas holding about 350 labels to be
able to scroll the frame in the canvas. Running the same script in 8.6.9
(xorg 1.20.7)  displays the canvas nearly immediately, running it on 8.6.14
(xorg 21.1.14) it takes ages ~ 20 secs)  before the canvas gets displayed.
FYI there is an "update idletasks" invoked, before the window containing
the widget is displayed, so I can center the window on the screen.
Anyone else seen this strange behaviour ?

regards


-- 
Gerhard

Re: [TCLCORE] CFV warning: TIP #709: MPL Licence for MemoryModule

[<apn...@ya...>]

Kevin,

 

Thanks for pointing me to pp. I’d looked at py2exe but not pp for perl.

 

Pp and PyInstaller are both implemented the same way. They are wrap the actual interpreter executable and extensions. At runtime these are written out to disk and invoked. When they exit the parent process (wrapper) cleans up the temporary directory.

 

I have implemented a similar (though directly in Tcl core, not a wrapper) scheme that cleans up the temporary files using a separate process after the Tcl process exits. See branch https://core.tcl-lang.org/tcl/timeline?r=apn-bug-a8e4f76ce4. It’s currently in cold storage as TIP 709 would be more elegant if it could be robustly implemented.

 

Your question regarding a Windows standalone app is a bit difficult to answer. If it contains “support” DLL’s that are not directly loaded via Tcl_LoadFile, they will still need to be explicitly written to disk first (Tcl_LoadFile will not load DLL’s that are dependencies of the primary extension DLL). I believe this limitation applies to Unix as well, not just Windows. I have not tried icons or signing. I should check that and add it to the https://wiki.tcl-lang.org/page/Single+file+applications+in+Tcl+9 page.

 

/Ashok

 

From: Kevin Walzer <kw...@co...> 
Sent: Sunday, January 12, 2025 10:31 AM
To: Brian Griffin <bri...@ea...>
Cc: Tcl Core List <tcl...@li...>
Subject: Re: [TCLCORE] CFV warning: TIP #709: MPL Licence for MemoryModule

 





> On Jan 11, 2025, at 4:46 PM, Brian Griffin wrote:
> 
> The beauty of a single file application is that simply deleting that file should be all that is necessary to "uninstall" the application

Is that really even possible on Windows? Perl’s pp module also makes a mess in the temp dir, but deletes everything on app shutdown. Perhaps we should look to see how they do it?

I’m still waiting to see a robust example of a standalone Tcl/Tk app on Windows, with its own icon and name, deployed as a zip kit/exe with everything bundled. Are we there yet? 

Re: [TCLCORE] CFV warning: TIP #709: MPL Licence for MemoryModule

[<apn...@ya...>]

Following up on my previous post, here is the basis and some justification
for my concerns. (I'm no expert so consider it a Google search dump)

 

My conclusions ...

 

1. Afaict, MemoryModule does not fully implement the features required for
emulating Windows loader behaviour even today.

 

2. Even if it did, the internals could change in any update, a service pack
even. The possibility of this is probably small, but not predictable. So I'm
not going to comment further on this. We can all make our own estimates of
this risk.

 

With respect to (1) however, I would ask TIP 709 reviewers to at least skim
the following resources. It does not really require Windows knowledge, just
basic data structures and a general idea about loaders.

 

*	The series of blog articles on the Windows loader, ending with the
most important one at http://www.nynaeve.net/?p=189 which essentially
reverse engineers the loader's role in TLS (thread local storage, not the
protocol :-)).

 

*	The sample code for the above at
http://www.nynaeve.net/Code/VistaImplicitTls.cpp

 

*	An implementation MemoryModulePP, derived from MemoryModule, at
https://github.com/bb107/MemoryModulePP

 

My conclusion going through the links above is that MemoryModule is still
primarily based on the much simpler and less capable pre-Vista Windows
loader. In particular, it allocates memory, maps sections (including
resource sections), sets up VM flags, maps symbol addresses, registers
exception handlers and TLS callbacks.

 

Per my reading of just the TLS-related portions,

 

*	It does not allocate slots for the *implicit* TLS entries and
initialize them in the loading thread.

 

*	It does not update the TLS map for existing threads by reallocating
(if necessary) the TLS storage and updating the in-use slot map.

 

For the complexity associated with the seemingly simple tasks above, compare
with the implementations in the aforementioned source code links to see how
much work is involved and missing from MemoryModule.

 

>From the above, I conclude at least the TLS-related functionality in
MemoryModule is incomplete. If we were to continue with TIP 709.

 

1.	One could assume the occurrences of the above TLS in the *Tcl* world
are minimal and just ignore the issue and hope for the best. Based on
Christian W.'s tests with LUCK, there is a good chance this might work. I am
however a believer in Murphy's law. And it does not address other potential
non-TLS issues listed later.

 

2.	One could document the restrictions (not sure exactly what those
would be) imposed on memory loading and leave it to the application writer
to check and verify. This may not be easy for them because of the use of
third party libraries in an extension and even calls made to Win32 API's.
Personally I am generally not in favour of features that work some of the
time under some (ill-defined) conditions on some platforms. Note that it is
*not* possible, at least as currently implemented, to make the determination
at load time and use the fallback method of copying to disk. Failures to
initialize TLS structures show up as corrupted memory at some later point.

 

3.	Switch the TIP 709 implementation from MemoryModule to the updated
MemoryModulePP which seems to be more modern. The concerns here would be
whether (a) even that is complete, (b) introduction of C++ into the Tcl code
base and (c) its reliance on yet more additional intrusive third party
libraries that hook system API's.

 

A word about testing - Jan mentioned test cases he wrote that pass. Assuming
the test dll is memorymoduletest.c, the tests do not address my concerns.
The use of a single tls variable, with a single thread, an integer scalar
with value 0, and no constructor will not suffice for the purpose. There are
is no test for nested exception handlers. A broader set of tests is needed.
IIUC, Christian W. has done extensive testing with LUCK and found no issues.
I am not sure which extensions were used and the manner (number of threads,
size and *type* of the TLS allocations etc.) but perhaps those could be used
as a starting point. Note however that failures are not easily detected
(akin to race conditions).

 

The above only looked at TLS but there are other potential issues

 

*	Exception handler registration. MemoryModule uses the simple
RtlAddFunctionTable call. MemoryModulePP on the other hand has a much more
complex implementation. I am not sure of the reason, and it may be
inconsequential but the fact that MemoryModulePP is derived from
MemoryModule and still found it necessary to change that simple call makes
me wonder.

 

*	Newer Windows loader features like CFG (control flow graphs) and
ASLR do not seem to be handled at all. Perhaps this may only matter if
builds make use of those features, but I would think that is the direction
Tcl should be moving in anyways for security reasons.

 

Another interesting tidbit Google threw up. Python's py2exe make use of
MemoryModule, more or less the same code base as TIP 709. This looked
promising in support of 709; however, note it is not part of Python but a
separate application and is architected very differently. More disturbing is
https://github.com/py2exe/py2exe/pull/214#issuecomment-2476280385. Quoting
(bolded emphasis mine),

 

In windows due to how LoadLibraryA/W loads DLLs with the TLS information
that initialization of the python core when bundle_files  <= 1 is not
possible. The resulting files must be the exe with the python core dll as a
separate file from the exe in order for it to be initialized properly. That
is unless the dll gets packed into the win32 resource section and just prior
to it calling any python C api functions for it to write the dll to the
disk. But even then that can break the built in C extensions when the exe
does not directly import the core dll in it's IMPORT_ADDRESS_TABLE.

 

I don't quite grok that but does not give me a comfortable feeling that the
dll needs to be a separate file (not a single file app then!) or written to
disk. I could not follow all the references and links in those tickets to
figure out what the current situation is (the above link was Nov 2024)

 

Jan also mentioned many applications that were already using MemoryModule
and there are a lot of forks but outside of py2exe, I could not locate
anything substantial outside of malware and malware analysers :-)

 

I hope folks, Jan and Christian W. in particular as they have the most
interest and knowledge in this space, will review and weigh the above
factors prior to voting. Even better, if some can alleviate my concerns.

 

In my own view, more study is needed before bringing 709 to a vote. As it
stands *now*, I would not be in favour though I am all in for the feature.
Contrary to what it may seem from the above, I was googling to convince
myself my fears were unfounded.

 

/Ashok

 

 

Re: [TCLCORE] CFV warning: TIP #709: MPL Licence for MemoryModule

[Harald O. <har...@el...>]

Am 12.01.2025 um 10:36 schrieb apnmbx-public--- via Tcl-Core:
> To Jan's point on the branch having to be closed due to the MPL *if* the 
> TIP is not approved, I do think the proposal for a license should be 
> TIP'ed separately from the functionality of a TIP. They are really 
> orthogonal issues and an "MPL permitted in all future code" TIP should 
> be approved on separately for exactly this reason. If the MPL TIP passes 
> through, there is no reason for the MemoryModule file (and future MPL- 
> licensed contributions) not to be committed in the source repository. 
> The vote for the "load from memory" feature TIP should be separate and 
> control whether it becomes part of Tcl implementation. That's only my 
> opinion.
> 
> With respect to Jan's comment that I'm exaggerating the risk, I freely 
> admit :-) I'm prone to that when it comes to undocumented behaviors so 
> my comments are colored accordingly. For that reason, I researched 
> further a summary of which I will post separately for folks to decide 
> for themselves.
> 
> And regarding "badly written", let me clarify I never meant to imply 
> MemoryModule is badly written. The code may be very well written but 
> could still be incomplete in terms of required features.
> 
> /Ashok
+1 on all points.

It is not ok to bundle the TIPs.

I would love to have the memory module feature in TCL 9.1.
It would allow to have an all-in-one executable which works without 
worries in the normal case.

Starkits had the same feature to leave temporary dlls behind. Due to 
that, secondary DLL's are avoided and, for example, tdom has all 
secondary dll's statically linked.
Ashok pointed out, that temporary dll's may be banned due to virus 
checkers etc. I often supposed and supported this argument, but all 
cases I saw had other reasons.
Specially the magic "Internet Download flag" for a dll often causes 
issues. This only happens, if I supply the dll beside the executable, 
not as temporare dll. And those issues arise by windows update out of 
the sudden.

Also Marcs proposal to put them always in the same file is good. But 
with the memory module, we don't need that solution neither.
Jan already started a checksum check, if it is the right dll and to 
reuse it.

Thanks for all,
Harald

Re: [TCLCORE] CFV warning: TIP #709: MPL Licence for MemoryModule

[<apn...@ya...>]

To Jan's point on the branch having to be closed due to the MPL *if* the TIP
is not approved, I do think the proposal for a license should be TIP'ed
separately from the functionality of a TIP. They are really orthogonal
issues and an "MPL permitted in all future code" TIP should be approved on
separately for exactly this reason. If the MPL TIP passes through, there is
no reason for the MemoryModule file (and future MPL-licensed contributions)
not to be committed in the source repository. The vote for the "load from
memory" feature TIP should be separate and control whether it becomes part
of Tcl implementation. That's only my opinion.

 

With respect to Jan's comment that I'm exaggerating the risk, I freely admit
:-) I'm prone to that when it comes to undocumented behaviors so my comments
are colored accordingly. For that reason, I researched further a summary of
which I will post separately for folks to decide for themselves.

 

And regarding "badly written", let me clarify I never meant to imply
MemoryModule is badly written. The code may be very well written but could
still be incomplete in terms of required features.

 

/Ashok

 

-----Original Message-----
From: Jan Nijtmans <jan...@gm...> 
Sent: Saturday, January 11, 2025 1:19 AM
To: Kevin Walzer <kw...@co...>
Cc: tcl...@li...
Subject: Re: [TCLCORE] CFV warning: TIP #709: MPL Licence for MemoryModule

 

Op vr 10 jan 2025 om 19:58 schreef Kevin Walzer:

> I'd caution against relying on undocumented/private API's unless there is
a GREAT reason to do so. Windows has been remarkably stable for Tcl/Tk, and
it would be shame for API churn (typical of Apple) to make its way into that
source tree.

 

That's part of the point here. I think, being able to load dll's in memory
is

such a GREAT reason. Ashok is IMHO exaggerating on how bad

MemoryModule is, looking at the code it is - actually - programmed

quite well. Well enough that it is picked up by other projects, and

other people were contributing back their fixes. I wrote testcases

for various situations Ashok pointed out to be problematic: All of

those worked fine! A "great" reason would be enough for me,

it doesn't even need to be GREAT.

 

Since I was not able to convince Ashok, it's now on our plate to

decide.

 

Part of the question is also non-technical. Should the quality

of code be a reason to accept or decline it in one of our

repositories? If TIP #705 "Affirm Tcl License" wouldn't be

there, TIP #709 wouldn't have been written at all. It also

means that - if TIP #709 is declined, I have to close the

branch and cannot even work on it any more, trying to

make it even more GREAT than it already is ;-)

 

Hope this helps,

    Jan Nijtmans

 

 

_______________________________________________

Tcl-Core mailing list

 <mailto:Tcl...@li...> Tcl...@li...

 <https://lists.sourceforge.net/lists/listinfo/tcl-core>
https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] CFV warning: TIP #709: MPL Licence for MemoryModule

[Kevin W. <kw...@co...>]

<div><img width="1" height="1" src='https://fedbdhd.r.tsp1-brevo.net/tr/op/46upqMQYxyWjcJWDnhDWKvKiciMSdTh8Ku-VjaZ-WlcFCUWx6eQtwJhKWVHIC9eHX9VirfvOMDE6wk-VrqU_vQsx8YlXy6WHQUEd1a3tZNDkMW-hSH4ySTTeABJEL0hS6XAD1t5DmM6hYW1RqKrNwxUGSTMY2kgFILHGZBStlIYG09sHTOucHOVXaTDy0qctzYcZRhP7sYvu_IUE5R9h-ffCpH3n' /></div><br/><br/>> On Jan 11, 2025, at 4:46 PM, Brian Griffin <bri...@ea...> wrote:<br/>> <br/>> The beauty of a single file application is that simply deleting that file should be all that is necessary to "uninstall" the application<br/><br/>Is that really even possible on Windows?  Perl’s pp module also makes a mess in the temp dir, but deletes everything on app shutdown. Perhaps we should look to see how they do it?<br/><br/>I’m still waiting to see a robust example of a standalone Tcl/Tk app on Windows, with its own icon and name, deployed as a zip kit/exe with everything bundled. Are we there yet? 

Re: [TCLCORE] CFV warning: TIP #709: MPL Licence for MemoryModule

[Brian G. <bri...@ea...>]

On Jan 11, 2025, at 12:56, Marc Culler <cul...@gm...> wrote:



On Sat, Jan 11, 2025 at 2:30 PM Brian Griffin <bri...@ea...<mailto:bri...@ea...>> wrote:
This would turn a “single file” program into a “self installer” kind of application.

I don't get that.  You still download a single file.  When you run that file as an executable, it writes a file to your disk.  That is true with the current setup too.

Yes, but, the current problem is that a temp folder is created and used once to hold the dll's for the current invocation. Then it is abandoned, never to be used again.

Using a fixed path allows for reuse, but there's no mechanism to clean it up once it's no longer relevant. In that sense, it is as if the application went through an install process, installing stuff only known to the application.  A new version of the application could provide an "uninstall" section to cleanup older versions detritus.

The beauty of a single file application is that simply deleting that file should be all that is necessary to "uninstall" the application.

I'm just spitballing here.

-Brian

  If memory loading is required for a 'single file' program then there have never been any 'single file' programs, if I understand the situation correctly.

There would be other things to deal with, like clean up when updating to a newer versions, or Trojan horse attacks.

The clean up for new versions seems minimal but I agree that a fixed path would make Trojan horse attacks easier.

- Marc

Re: [TCLCORE] CFV warning: TIP #709: MPL Licence for MemoryModule

[Marc C. <cul...@gm...>]

On Sat, Jan 11, 2025 at 2:30 PM Brian Griffin <bri...@ea...>
wrote:

> This would turn a “single file” program into a “self installer” kind of
> application.
>

I don't get that.  You still download a single file.  When you run that
file as an executable, it writes a file to your disk.  That is true with
the current setup too.  If memory loading is required for a 'single file'
program then there have never been any 'single file' programs, if I
understand the situation correctly.


> There would be other things to deal with, like clean up when updating to a
> newer versions, or Trojan horse attacks.
>

The clean up for new versions seems minimal but I agree that a fixed path
would make Trojan horse attacks easier.

- Marc

Re: [TCLCORE] CFV warning: TIP #709: MPL Licence for MemoryModule

[Brian G. <bri...@ea...>]

On Jan 11, 2025, at 11:52, Marc Culler <cul...@gm...> wrote:


Here is a naive question.  Is it possible to specify the path that the DLL file will be copied to?

If the answer is yes, here is a naive suggestion:
Why not always use the same, known, path for each DLL that needs to be loaded from a zip file?

This would seem to mean:
* There would be at most one copy of the DLL on the user's disk.
* The user would know or could find out where it is, so it could be deleted by hand.
* The copy-to-disk operation could be skipped if the file were already there.

This would turn a “single file” program into a “self installer” kind of application.

There would be other things to deal with, like clean up when updating to a newer versions, or Trojan horse attacks.

These are my immediate thoughts.
-Brian


- Marc

On Sat, Jan 11, 2025 at 11:58 AM Christian Gollwitzer via Tcl-Core <tcl...@li...<mailto:tcl...@li...>> wrote:
Am 11.01.25 um 18:47 schrieb Marc Culler:
> On Sat, Jan 11, 2025 at 11:05 AM Brian Griffin <bri...@ea...<mailto:bri...@ea...>
> <mailto:bri...@ea...<mailto:bri...@ea...>>> wrote:
>
>     Tcl includes support for a zip based VFS. It is used internally when
>     tcl is built with -disable-shared. The result is a single file
>     tclsh.exe that contains the executable and an attached zip file
>     containing the tcl library of scripts and packages, which may
>     include dll files.
>
>
> Thank you.  That makes sense.  I knew that VFS stands for  "Virtual File
> System", but I had no idea that we were talking about a zip filesystem.
> I was assuming that the issue was with SMB filesystems.

The use case of a DLL in a ZIP file sounds exotic, until you think of
distributing your software to end users. Previously starkits have been
used based on MetaKit, and the ZIP file is now an alternative. If you
include any binary extension in the starkit/zipkit, the contained code
must be loaded upon "package require". So far, it is done by a modified
"load" command, which copies the DLL to a temporary file and then loads
this file. Upon exit, these DLLs cannot be deleted, because on Windows
you can't change an executable or DLL that is running. This creates a
lot of debris. A solution to load the DLL from memory instead would be
really great.

Under Linux, you can simply unlink the .so file directly after loading
it, so there is no real problem.


        Christian


_______________________________________________
Tcl-Core mailing list
Tcl...@li...<mailto:Tcl...@li...>
https://lists.sourceforge.net/lists/listinfo/tcl-core
_______________________________________________
Tcl-Core mailing list
Tcl...@li...
https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] CFV warning: TIP #709: MPL Licence for MemoryModule

[Marc C. <cul...@gm...>]

Here is a naive question.  Is it possible to specify the path that the DLL
file will be copied to?

If the answer is yes, here is a naive suggestion:
Why not always use the same, known, path for each DLL that needs to be
loaded from a zip file?

This would seem to mean:
* There would be at most one copy of the DLL on the user's disk.
* The user would know or could find out where it is, so it could be deleted
by hand.
* The copy-to-disk operation could be skipped if the file were already
there.

- Marc

On Sat, Jan 11, 2025 at 11:58 AM Christian Gollwitzer via Tcl-Core <
tcl...@li...> wrote:

> Am 11.01.25 um 18:47 schrieb Marc Culler:
> > On Sat, Jan 11, 2025 at 11:05 AM Brian Griffin <bri...@ea...
> > <mailto:bri...@ea...>> wrote:
> >
> >     Tcl includes support for a zip based VFS. It is used internally when
> >     tcl is built with -disable-shared. The result is a single file
> >     tclsh.exe that contains the executable and an attached zip file
> >     containing the tcl library of scripts and packages, which may
> >     include dll files.
> >
> >
> > Thank you.  That makes sense.  I knew that VFS stands for  "Virtual File
> > System", but I had no idea that we were talking about a zip filesystem.
> > I was assuming that the issue was with SMB filesystems.
>
> The use case of a DLL in a ZIP file sounds exotic, until you think of
> distributing your software to end users. Previously starkits have been
> used based on MetaKit, and the ZIP file is now an alternative. If you
> include any binary extension in the starkit/zipkit, the contained code
> must be loaded upon "package require". So far, it is done by a modified
> "load" command, which copies the DLL to a temporary file and then loads
> this file. Upon exit, these DLLs cannot be deleted, because on Windows
> you can't change an executable or DLL that is running. This creates a
> lot of debris. A solution to load the DLL from memory instead would be
> really great.
>
> Under Linux, you can simply unlink the .so file directly after loading
> it, so there is no real problem.
>
>
>         Christian
>
>
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core
>

Re: [TCLCORE] CFV warning: TIP #709: MPL Licence for MemoryModule

[Christian G. <aur...@gm...>]

Am 11.01.25 um 18:47 schrieb Marc Culler:
> On Sat, Jan 11, 2025 at 11:05 AM Brian Griffin <bri...@ea...
> <mailto:bri...@ea...>> wrote:
>
>     Tcl includes support for a zip based VFS. It is used internally when
>     tcl is built with -disable-shared. The result is a single file
>     tclsh.exe that contains the executable and an attached zip file
>     containing the tcl library of scripts and packages, which may
>     include dll files.
>
>
> Thank you.  That makes sense.  I knew that VFS stands for  "Virtual File
> System", but I had no idea that we were talking about a zip filesystem.
> I was assuming that the issue was with SMB filesystems.

The use case of a DLL in a ZIP file sounds exotic, until you think of
distributing your software to end users. Previously starkits have been
used based on MetaKit, and the ZIP file is now an alternative. If you
include any binary extension in the starkit/zipkit, the contained code
must be loaded upon "package require". So far, it is done by a modified
"load" command, which copies the DLL to a temporary file and then loads
this file. Upon exit, these DLLs cannot be deleted, because on Windows
you can't change an executable or DLL that is running. This creates a
lot of debris. A solution to load the DLL from memory instead would be
really great.

Under Linux, you can simply unlink the .so file directly after loading
it, so there is no real problem.


        Christian

Re: [TCLCORE] CFV warning: TIP #709: MPL Licence for MemoryModule

[Marc C. <cul...@gm...>]

On Sat, Jan 11, 2025 at 11:05 AM Brian Griffin <bri...@ea...>
wrote:

> Tcl includes support for a zip based VFS. It is used internally when tcl
> is built with -disable-shared. The result is a single file tclsh.exe that
> contains the executable and an attached zip file containing the tcl library
> of scripts and packages, which may include dll files.
>

Thank you.  That makes sense.  I knew that VFS stands for  "Virtual File
System", but I had no idea that we were talking about a zip filesystem.  I
was assuming that the issue was with SMB filesystems.

- Marc

Re: [TCLCORE] On editing of passed TIP's

[Poor Y. <org...@po...>]

On 2025-01-11 17:44, Rolf Ade wrote:
> Poor Yorick writes:
>> My point is that my commit was reverted without citing any deficiency 
>> in
>> the commit.  To date, no one has cited any deficiency in the commit.
> 
> Well, you commit your changes in the first place without citing and
> discussing the deficiency in the original text, no?
> 
> And your changes were not only fixing a misspelled word here and
> changing a sentence there to streamline grammar. You substantially
> changed the text, reduced the number of rules and changed the order how
> information was provided.
> 
> Incremental improvements of source files happens all the day without
> prior discussion and are OK, especially if a clean test suite run
> indicates that the changes are OK.
> 
> But you changed the central summary text of the Tcl language syntax. 
> And
> surely not just or only in an incremental way.
> 
> You seem to think that anyone with commit rights is allowed to change
> such a central text on _trunk_ without prior communication with the
> developer community. I understand that this raises questions about your
> judgement skills which are a requisite for giving the power of commit
> rights.
> 
> Instead of moaning about the revert (which was justified) you would
> spend your (and our) time better if you would start to argue for your
> proposal so that we, in the best case, end with an improved and
> consented text.
> 
> rolf

Rolf, you've made a reasonable argument.  I didn't my modifications to
the rules of Tcl into this conversation.  Ashok did.  If my version was
reverted because people prefer the old version for whatever reason,
that's valid.  However, no one has stated that after review they prefer
the old version.  I didn't contest the reversion other than to say that
no deficiency was cited.  It was easy to revert, and that's the way that
differences of opinion about the text should be resolved.  Changes to
the documentation are not subject to the TIP process, but to accurately
reflecting the current implementation.

Once again, here is my version:

	https://core.tcl-lang.org/tcl/file?name=doc/Tcl.n&ci=1117e4b1f0c36d54

Regarding making changes on trunk, it is clear that any change to the
public interface of Tcl must first be accepted through the TIP process
before being committed.  All other changes should be carefully tested,
including ensuring that the test suite passes and that valgrind does not
report any new errors.

-- 
Yorick