tcl-core — Tcl Core Team discussions about Tcl development

Re: [TCLCORE] TIP 701

[<apn...@ya...>]

Gentle reminder – TIP 701 CFV coming up Monday.

 

From: apnmbx-public--- via Tcl-Core <tcl...@li...> 
Sent: Friday, October 4, 2024 7:03 PM
To: tcl...@li...
Subject: [TCLCORE] TIP 701

 

Proudly announcing the first TIP post 9.0 😊 TIP 701: C API for tilde substitution in paths

 

https://core.tcl-lang.org/tips/doc/trunk/tip/701.md

 

TIP 602 removed the implicit tilde substitution from Tcl's handling of file paths. The file home and file tildeexpand commands made equivalent functionality explicitly available to scripts. However, no comparable C API was made available to extensions. This TIP remedies the oversight.

 

Please review. CFV in 2 weeks.

 

/Ashok 

Re: [TCLCORE] Tip 430 First argument handling for Tclsh/Wish (install or detected zip)

[Julian N. <ju...@pr...>]

Perhaps not exactly a mismatch - but I think underexplained in docs...

 From what I can see - zipfs mkzip just encrypts each of the zip file 
content files using the password which is normal - but zipfs mkzip also 
stores it in the archive between the prefix and zip data so the 
automounter can use it.

Someone commented in the tclZipfs.c  "The idea of storing a password 
with the archive seems absurd, encoded or not"

I tend to think it's possibly ok - if it's documented that it's just a 
feature for 'mild obfuscation' - or a mechanism to hinder casual perusal 
of file contents using external zip tools?
By using zipfs mkzip with a password - you're effectively giving the 
password away.
(maybe the same could be said for passwords on a plain zip anyway from a 
cracker perspective - as I assume we're using a pretty basic password 
encryption, but that's not the point I suppose)

I can only imagine the feature of storing the password in the combined 
exe/zip being useful for some sort of 'source available' license where 
they're trying to hint by the mild obfuscation that it's not intended 
for distribution - but perhaps I'm missing other usecases.

My worry is that undocumented - eventually we are just going to get some 
vulnerability report or 'shaming' of Tcl for this - or just someone 
thinking it's magically secure somehow and inadvertently releasing a 
password.
(Possibly the feature should really have been to use some sort of 
indirection so the automounter could look it up from an env var or some 
better password store - but I'm not sure how well that would work with 
initial loading)

Tip 430 only mentions for zip lmkimg that "If password is given , the 
file will be encrypted with that passphrase" - which also doesn't give 
an indication to users that it's actually stored for automounting.
Nor does it make it clear that the encryption doesn't stop you listing 
the contents - as the CDR isn't encrypted - only the file contents 
themselves are unreadable without supplying a password.
That's normal for this sort of basic zip file password - but a common 
surprise - and perhaps not quite what someone may expect from the 
wording 'the file will be encrypted' as in that context it implies the 
outfile.

Cheers,
Julian


On 2024-10-11 6:22 pm, Torsten Berg wrote:
> Hi,
>
> you are right, this feature was never implemented. It should have been 
> added to the "Amendment" section of that TIP but was overlooked. I'll 
> add that right away. There are several mismatches between the TIP and 
> the real implementation. The "Amendment" section is supposed to 
> document these. If you look at the tclsh manual page 
> (https://www.tcl-lang.org/man/tcl9.0/UserCmd/tclsh.html) you will not 
> find that feature documented either. This manual page and the zipfs 
> manual page are the definitive source for reading about the real 
> implementation (unless we have overlooked yet another mismatch ...).
>
> Regards, Torsten
>
>
>> Am 11.10.2024 um 06:06 schrieb Julian Noble via Tcl-Core 
>> <tcl...@li...>:
>>
>> Tip 430 states
>>
>> If the first argument to Tclsh or Wish is detected to be a zipfile, 
>> that file will be mounted as*ZIPFS_ROOT*//app/. 
>> If*ZIPFS_ROOT*//app/main.tcl/exists, that file is marked set the 
>> shell's startup script. If*ZIPFS_ROOT*//app/tcl_library/exists, it 
>> will be searched for/init.tcl/.
>>
>> Is this still meant to be a feature?
>> I've been unable to verify on windows - it just seems to try to run 
>> it as a script, giving the error:
>> couldn't read file "some.zip": invalid or incomplete multibyte or 
>> wide character
>>
>> (same if some.zip replaced with a tclsh that has an attached zip)
>>
>> glancing at the tclZipfs.c code it looks like this feature is 
>> dependent on the SUPPORT_BUILTIN_ZIP_INSTALL flag -  are the 
>> 'install' argument and mounting of first argument if a zip supposed 
>> to be enabled/disabled as a single unit?
>> I couldn't test anyway due to compilation error with that flag 
>> (https://core.tcl-lang.org/tcl/tktview/c5825d3235)
>> As that ticket reports - the 'tclsh install mkimg ..' is suspect anyway
>>
>> Perhaps 'tclsh install/loading first arg if zip' works on other 
>> platforms? Anyone tried it?
>>
>> I feel the mount first argument as a zip is perhaps an unnecessary 
>> feature.
>> Without this magic, it is easy enough anyway to cat a small Tcl 
>> script to the beginning of a zip - along with a ctrl-z character - 
>> and then the script can mount/set auto_paths etc as desired.
>> e.g I have a small file named #z containing the binary <SUB> 
>> character and do:
>> cat kitheader.tcl #z test.zip > kitname.zipkit
>>
>> This works fine.
>>
>> This mount-first-arg-if-zip feature would seem to have potential to 
>> collide with that and I don't currently understand the benefit..
>>  unless it allows dropping into a standard tclsh repl if no main.tcl 
>> is present?
>> That would be nice.
>> (I can do this  if I use a main.tcl-hacked-tclsh with a as per my 
>> earlier message regarding dropping into a standard repl - but 
>> although I've seen the request pop up from users over the years - I 
>> don't see any interest/comments from this list)
>>
>>
>> Cheers,
>> Julian
>>
>>
>>
>>
>> _______________________________________________
>> Tcl-Core mailing list
>> Tcl...@li...
>> https://lists.sourceforge.net/lists/listinfo/tcl-core
>

Re: [TCLCORE] Thanks for TCL/Tk 9.0.0 and next steps

[Marc C. <cul...@gm...>]

On Thu, Oct 17, 2024 at 9:46 AM Harald Oehlmann <har...@el...>
wrote:

> It would be great if you could attend. Apparently, Tk has made great
> steps forward after 9.0.0 release on Mac. How to handle this is part of
> the question.
>

Thanks, Harald.  If you ask me, the great steps forward were included in
9.0.0.  (I am talking about replacing drawRect by updateLayer, which was
Christopher Chavez's idea.). Since then there has been cleanup, which was
known to be needed.

- Marc

Re: [TCLCORE] Thanks for TCL/Tk 9.0.0 and next steps

[Harald O. <har...@el...>]

Hi Marc,
thank you for your E-Mail, I appreciate !
Allow me to answer to your questions:

Am 17.10.2024 um 16:36 schrieb Marc Culler:
> I have no idea what the item "factor out the image command from Tk" is 
> about, nor why one would want to do that. What is the issue? It seems 
> like something that should be discussed here before it is discussed on 
> zoom, which reaches many fewer people.

It is just my personal list of items. Other may add items.
The point here is, that the image command (from Tk) is also useful if 
TCL is used without Tk, like in web servers. The purpose is to have a 
limited image command available in Tk without Tk.

> Also, I don't understand why we would be talking about Tk 10.1 before we 
> have had a single point release of Tk 9.

Thank you for your opinion, that is sensible.
My point is basically just "whats next?". And what to do with 
bugfixes/small features/breaking features.

It would be great if you could attend. Apparently, Tk has made great 
steps forward after 9.0.0 release on Mac. How to handle this is part of 
the question.

Thank you for all,
Harald

Re: [TCLCORE] Thanks for TCL/Tk 9.0.0 and next steps

[Kevin W. <kw...@co...>]

<div><img width="1" height="1" src='https://fedbdhd.r.bh.d.sendibt3.com/tr/op/jp_zjnAuTeGDSqcgb1mIc2GuzdRKqg_0wmDmvZTQQC5bKbRPPBrxabySqU08k-QDky4j0MwsAYWqWW24eQRJY2dXDMZipQRTdYCjKxDkdMENP6YeaKHRgXhZ_SamliHYli2RUYOMG4Cl-HXWCSq-P3K-fOgiyJfsuV8KbiAfSlQZWE-rHaGO_LGqu12SQ6VnTSPQebIL7j8aRnjY9Bju5CXNV-s4ydko' /></div>Harald, did you mean 9.1?<br/><br/>> On Oct 17, 2024, at 4:37 AM, apnmbx-public--- via Tcl-Core <tcl...@li...> wrote:<br/>> <br/>> I agree it would be a good idea to have this discussion. I am ok with that timing.<br/>> <br/>> -----Original Message-----<br/>> From: Harald Oehlmann <har...@el...><br/>> <br/>> May I propose a meeting to discuss the next steps ?<br/>> What about Monday 28th of October 12:00 UTC for one hour?<br/>> Many regions will have stopped daytime saving time then.<br/>> <br/>> My personal agenda is:<br/>> - what is the next version of TCL/Tk for:<br/>>   - non breaking bugfix style TIPs (like the two most recent ones)<br/>>     (10.1 ?)<br/>>   - innovative TIPs (10.1a0 ?)<br/>>   - breaking TIPs (11.0a0 ?)<br/>> - what are the possible innovations?<br/>>   - factor out the image command from Tk<br/>>   - list image formats<br/>> <br/>> By the way, the conference 2025 may be in Italy...<br/>> <br/>> Thank you for all,<br/>> Harald<br/>> <br/>> <br/>> <br/>> <br/>> _______________________________________________<br/>> Tcl-Core mailing list<br/>> Tcl...@li...<br/>> https://lists.sourceforge.net/lists/listinfo/tcl-core<br/>

Re: [TCLCORE] Thanks for TCL/Tk 9.0.0 and next steps

[Marc C. <cul...@gm...>]

I have no idea what the item "factor out the image command from Tk" is
about, nor why one would want to do that. What is the issue? It seems like
something that should be discussed here before it is discussed on zoom,
which reaches many fewer people.

Also, I don't understand why we would be talking about Tk 10.1 before we
have had a single point release of Tk 9.

- Marc

On Thu, Oct 17, 2024 at 3:37 AM apnmbx-public--- via Tcl-Core <
tcl...@li...> wrote:

> I agree it would be a good idea to have this discussion. I am ok with that
> timing.
>
> -----Original Message-----
> From: Harald Oehlmann <har...@el...>
>
> May I propose a meeting to discuss the next steps ?
> What about Monday 28th of October 12:00 UTC for one hour?
> Many regions will have stopped daytime saving time then.
>
> My personal agenda is:
> - what is the next version of TCL/Tk for:
>    - non breaking bugfix style TIPs (like the two most recent ones)
>      (10.1 ?)
>    - innovative TIPs (10.1a0 ?)
>    - breaking TIPs (11.0a0 ?)
> - what are the possible innovations?
>    - factor out the image command from Tk
>    - list image formats
>
> By the way, the conference 2025 may be in Italy...
>
> Thank you for all,
> Harald
>
>
>
>
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core
>

Re: [TCLCORE] Thanks for TCL/Tk 9.0.0 and next steps

[<apn...@ya...>]

I agree it would be a good idea to have this discussion. I am ok with that timing.

-----Original Message-----
From: Harald Oehlmann <har...@el...> 

May I propose a meeting to discuss the next steps ?
What about Monday 28th of October 12:00 UTC for one hour?
Many regions will have stopped daytime saving time then.

My personal agenda is:
- what is the next version of TCL/Tk for:
   - non breaking bugfix style TIPs (like the two most recent ones)
     (10.1 ?)
   - innovative TIPs (10.1a0 ?)
   - breaking TIPs (11.0a0 ?)
- what are the possible innovations?
   - factor out the image command from Tk
   - list image formats

By the way, the conference 2025 may be in Italy...

Thank you for all,
Harald

Re: [TCLCORE] [External] Atomicity in Tcl and Tk

[Da S. P. J C. <pet...@fl...>]

Ideally no amount of code should lead to execution of the event loop, even in libraries, except for an explicit call to update or vwait at the top level. If there’s an exception to this it should be called out, otherwise you’re likely to get the event loop called from the event loop.

Re: [TCLCORE] making zipfs mounts writable

[Jan M. <0x...@gm...>]

On Tue, Oct 15, 2024 at 11:04 AM Julian Noble via Tcl-Core
<tcl...@li...> wrote:

> I've done some basic tests with a self-signed cert and powershell's
> Set-AuthenticodeSignature
> It's not a thorough test - but I don't see any issues whichever way the
> offsets are in the zip - they are all pointing to earlier points in the
> file so I don't think it has the potential to conflict with signing.

I do. The Go "Tkinter" mounts zip files[0] and verifies their
signatures[1]. Tampering with the zip will be detected and a
"corrupted" zip will wipe the cached library artefacts[2] directory.

OTOH, if there would be an option like, say -readonly, then I could use it.

  [0]: https://gitlab.com/cznic/tk9.0/-/blob/1c3687f1ac40c6a762f8b044dba8698d92cc88ac/tk_windows.go#L146
  [1]: https://gitlab.com/cznic/tk9.0/-/blob/1c3687f1ac40c6a762f8b044dba8698d92cc88ac/tk_windows_amd64.go#L27
  [2]: https://gitlab.com/cznic/tk9.0/-/blob/1c3687f1ac40c6a762f8b044dba8698d92cc88ac/tk_windows.go#L169

Re: [TCLCORE] making zipfs mounts writable

[Jan N. <jan...@gm...>]

Op di 15 okt 2024 om 11:04 schreef Julian Noble via Tcl-Core:

> Hi Harald,
>
> I've done some basic tests with a self-signed cert and powershell's
> Set-AuthenticodeSignature
> It's not a thorough test - but I don't see any issues whichever way the
> offsets are in the zip - they are all pointing to earlier points in the
> file so I don't think it has the potential to conflict with signing.
>

That's not a coincidence. See this comment:
Tcl Source Code: tclZipfs.c at [0066eacd87] (tcl-lang.org)
<https://core.tcl-lang.org/tcl/file?name=generic/tclZipfs.c&ci=0066eacd87e1d7da&ln=1414-1418>

Regards,
     Jan Nijtmans

Re: [TCLCORE] making zipfs mounts writable

[Harald O. <har...@el...>]

Hi Julian,
thank you for double checking - this is great news !
Sorry for the eventually non-related comment.

Take care,
Harald

Am 15.10.2024 um 11:03 schrieb Julian Noble via Tcl-Core:
> Hi Harald,
> 
> I've done some basic tests with a self-signed cert and powershell's Set- 
> AuthenticodeSignature
> It's not a thorough test - but I don't see any issues whichever way the 
> offsets are in the zip - they are all pointing to earlier points in the 
> file so I don't think it has the potential to conflict with signing.
> 
> Cheers,
> Julian
> 
> 
> On 2024-10-15 4:58 pm, Harald Oehlmann wrote:
>> Am 15.10.2024 um 06:35 schrieb Julian Noble via Tcl-Core:
>>> If the tcl community wants to decide that zipfs as attached to a tcl 
>>> exe or script *requires* that the first CDR file/dir entry points to 
>>> the topmost local file header (which is not true under all zip 
>>> editing scenarios) - then that's a decision that could be made as a 
>>> specific decision to restrict us to a subset of what the zipfs 
>>> container allows
>>>   - but I didn't see it documented or discussed when the change was 
>>> made - and the 2021 changes appear to have broken the 'zipfs info' 
>>> command's ability to determine the exe/zip split offset as I describe 
>>> in bug: https://core.tcl-lang.org/tcl/tktview/aaa84fbbc5
>> Julian,
>> thanks for all your work. All your work is welcome! I don't think 
>> there is a decision for the zip directory position. It is probably 
>> only to make it work. The initial code was quite buggy and we have 
>> fixed a lot of tickets in the meanwhile. We will have follow-up TIPs 
>> in this area anyway, as the original TIP highly differs from the 
>> implementation. Most active people on my radar were Ashok, Donal and 
>> Brian.
>>
>> From my side, please allow me to mention, that exe and dll files may 
>> be signed. The signature also goes to the end of the file. So, the zip 
>> may not be the last component in the file, at least with signtool on 
>> Windows.
>> May this be the reason to not use end-relative zip directory positioning?
>>
>> Thank you for all,
>> Harald
>>
>>
>> _______________________________________________
>> Tcl-Core mailing list
>> Tcl...@li...
>> https://lists.sourceforge.net/lists/listinfo/tcl-core
> 
> 
> 
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core


-- 
ELMICRON Dr. Harald Oehlmann GmbH
Koesener Str. 85
06618 NAUMBURG - Germany
Phone: +49 3445 781120
Direct: +49 3445 781127
www.Elmicron.de
German legal references:
Geschaeftsfuehrer: Dr. Harald Oehlmann
UST Nr. / VAT ID No.: DE206105272
HRB 212803 Stendal

Re: [TCLCORE] making zipfs mounts writable

[Julian N. <ju...@pr...>]

Hi Harald,

I've done some basic tests with a self-signed cert and powershell's 
Set-AuthenticodeSignature
It's not a thorough test - but I don't see any issues whichever way the 
offsets are in the zip - they are all pointing to earlier points in the 
file so I don't think it has the potential to conflict with signing.

Cheers,
Julian


On 2024-10-15 4:58 pm, Harald Oehlmann wrote:
> Am 15.10.2024 um 06:35 schrieb Julian Noble via Tcl-Core:
>> If the tcl community wants to decide that zipfs as attached to a tcl 
>> exe or script *requires* that the first CDR file/dir entry points to 
>> the topmost local file header (which is not true under all zip 
>> editing scenarios) - then that's a decision that could be made as a 
>> specific decision to restrict us to a subset of what the zipfs 
>> container allows
>>   - but I didn't see it documented or discussed when the change was 
>> made - and the 2021 changes appear to have broken the 'zipfs info' 
>> command's ability to determine the exe/zip split offset as I describe 
>> in bug: https://core.tcl-lang.org/tcl/tktview/aaa84fbbc5
> Julian,
> thanks for all your work. All your work is welcome! I don't think 
> there is a decision for the zip directory position. It is probably 
> only to make it work. The initial code was quite buggy and we have 
> fixed a lot of tickets in the meanwhile. We will have follow-up TIPs 
> in this area anyway, as the original TIP highly differs from the 
> implementation. Most active people on my radar were Ashok, Donal and 
> Brian.
>
> From my side, please allow me to mention, that exe and dll files may 
> be signed. The signature also goes to the end of the file. So, the zip 
> may not be the last component in the file, at least with signtool on 
> Windows.
> May this be the reason to not use end-relative zip directory positioning?
>
> Thank you for all,
> Harald
>
>
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] making zipfs mounts writable

[Julian N. <ju...@pr...>]

I've tried out of interest - but haven't managed to get the vlerq c code 
running on Tcl9
That's for metakit lite.

I think the full metakit for writable kits is c++ code and might be even 
harder to get going?

J


On 2024-10-15 6:02 pm, Colin Macleod via Tcl-Core wrote:
> Will it still be possible to use the older metakit-based 
> Tclkit/starkit system with Tcl9 ?  I believe that does support 
> writeable archives, so if it still works it would be an option for 
> anyone who needs to be able to update their vfs.
>
> Colin.
>
> On 15/10/2024 07:54, Christian Werner wrote:
>> Julian,
>>
>> while technically feasible IMHO adding full write support to zipfs is a
>> inferior feature since there are many formats much better suited for
>> writability (what about sqlar for example).
>>
>> Let's not forget how zipfs came into being: Android installable packages
>> (*.apk files) are basically zips used somewhat similar how JVMs use 
>> *.jar
>> archives/object code libraries. Thus it seemed useful to have Tcl core
>> support for mounting since all the Tcl library stuff can directly go 
>> into
>> an *.apk and be read from there. Later on this approach was extended for
>> traditional Tcl binaries/executables.
>>
>> Still the main focus was on having a read-only container format which 
>> can
>> hold bundled Tcl library and application code. Yes, there's zipfs::wrmax
>> which stems from my early experiments to have means to apply small 
>> patches
>> to single files at runtime. But this was never intended to be a 
>> mainstream
>> feature for general write support. In hindsight possibly even a 
>> misfit which
>> accidentally survived (and went into Tcl 9.0 in the meantime) instead of
>> being buried early.
>>
>> Fields of zipfs where I see the main energy should be spent on are glob
>> performance optimization by possibly adding an internal caching layer 
>> dealing
>> with directories and improving embedding native code (shared 
>> libraries) by
>> using dynamic linking techniques like BSD fdlopen() (didn't Mozilla 
>> invent
>> some interal trickery to forge this on Win32 even?).
>>
>> My 2e-2 currency units, BR
>> Christian
>>
>>
>> _______________________________________________
>> Tcl-Core mailing list
>> Tcl...@li...
>> https://lists.sourceforge.net/lists/listinfo/tcl-core
>
>
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] making zipfs mounts writable

[Julian N. <ju...@pr...>]

Hi Christian,

Fair enough - thanks for the background info.
Yes, things like being able to load a dll/so directly from 
memory/mounted zip are probably more useful - but tricky to do 
particularly on windows I hear.

Without the capability of directly writing mounted zips, I'm reasonably 
happy to write zip contents out to the filesystem for rebuilding 
purposes - but the splitting between exe & zip is a pain point.

Cheers,
Julian


On 2024-10-15 5:54 pm, Christian Werner wrote:
> Julian,
>
> while technically feasible IMHO adding full write support to zipfs is a
> inferior feature since there are many formats much better suited for
> writability (what about sqlar for example).
>
> Let's not forget how zipfs came into being: Android installable packages
> (*.apk files) are basically zips used somewhat similar how JVMs use *.jar
> archives/object code libraries. Thus it seemed useful to have Tcl core
> support for mounting since all the Tcl library stuff can directly go into
> an *.apk and be read from there. Later on this approach was extended for
> traditional Tcl binaries/executables.
>
> Still the main focus was on having a read-only container format which can
> hold bundled Tcl library and application code. Yes, there's zipfs::wrmax
> which stems from my early experiments to have means to apply small 
> patches
> to single files at runtime. But this was never intended to be a 
> mainstream
> feature for general write support. In hindsight possibly even a misfit 
> which
> accidentally survived (and went into Tcl 9.0 in the meantime) instead of
> being buried early.
>
> Fields of zipfs where I see the main energy should be spent on are glob
> performance optimization by possibly adding an internal caching layer 
> dealing
> with directories and improving embedding native code (shared 
> libraries) by
> using dynamic linking techniques like BSD fdlopen() (didn't Mozilla 
> invent
> some interal trickery to forge this on Win32 even?).
>
> My 2e-2 currency units, BR
> Christian
>
>
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] making zipfs mounts writable

[Colin M. <col...@ya...>]

Will it still be possible to use the older metakit-based Tclkit/starkit 
system with Tcl9 ?  I believe that does support writeable archives, so 
if it still works it would be an option for anyone who needs to be able 
to update their vfs.

Colin.

On 15/10/2024 07:54, Christian Werner wrote:
> Julian,
>
> while technically feasible IMHO adding full write support to zipfs is a
> inferior feature since there are many formats much better suited for
> writability (what about sqlar for example).
>
> Let's not forget how zipfs came into being: Android installable packages
> (*.apk files) are basically zips used somewhat similar how JVMs use *.jar
> archives/object code libraries. Thus it seemed useful to have Tcl core
> support for mounting since all the Tcl library stuff can directly go into
> an *.apk and be read from there. Later on this approach was extended for
> traditional Tcl binaries/executables.
>
> Still the main focus was on having a read-only container format which can
> hold bundled Tcl library and application code. Yes, there's zipfs::wrmax
> which stems from my early experiments to have means to apply small 
> patches
> to single files at runtime. But this was never intended to be a 
> mainstream
> feature for general write support. In hindsight possibly even a misfit 
> which
> accidentally survived (and went into Tcl 9.0 in the meantime) instead of
> being buried early.
>
> Fields of zipfs where I see the main energy should be spent on are glob
> performance optimization by possibly adding an internal caching layer 
> dealing
> with directories and improving embedding native code (shared 
> libraries) by
> using dynamic linking techniques like BSD fdlopen() (didn't Mozilla 
> invent
> some interal trickery to forge this on Win32 even?).
>
> My 2e-2 currency units, BR
> Christian
>
>
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core

Re: [TCLCORE] making zipfs mounts writable

[Christian W. <Chr...@t-...>]

Julian,

while technically feasible IMHO adding full write support to zipfs is a
inferior feature since there are many formats much better suited for
writability (what about sqlar for example).

Let's not forget how zipfs came into being: Android installable packages
(*.apk files) are basically zips used somewhat similar how JVMs use *.jar
archives/object code libraries. Thus it seemed useful to have Tcl core
support for mounting since all the Tcl library stuff can directly go into
an *.apk and be read from there. Later on this approach was extended for
traditional Tcl binaries/executables.

Still the main focus was on having a read-only container format which can
hold bundled Tcl library and application code. Yes, there's zipfs::wrmax
which stems from my early experiments to have means to apply small patches
to single files at runtime. But this was never intended to be a mainstream
feature for general write support. In hindsight possibly even a misfit which
accidentally survived (and went into Tcl 9.0 in the meantime) instead of
being buried early.

Fields of zipfs where I see the main energy should be spent on are glob
performance optimization by possibly adding an internal caching layer dealing
with directories and improving embedding native code (shared libraries) by
using dynamic linking techniques like BSD fdlopen() (didn't Mozilla invent
some interal trickery to forge this on Win32 even?).

My 2e-2 currency units, BR
Christian

Re: [TCLCORE] making zipfs mounts writable

[Harald O. <har...@el...>]

Am 15.10.2024 um 07:58 schrieb Harald Oehlmann:
> Am 15.10.2024 um 06:35 schrieb Julian Noble via Tcl-Core:
>> If the tcl community wants to decide that zipfs as attached to a tcl 
>> exe or script *requires* that the first CDR file/dir entry points to 
>> the topmost local file header (which is not true under all zip editing 
>> scenarios) - then that's a decision that could be made as a specific 
>> decision to restrict us to a subset of what the zipfs container allows
>>   - but I didn't see it documented or discussed when the change was 
>> made - and the 2021 changes appear to have broken the 'zipfs info' 
>> command's ability to determine the exe/zip split offset as I describe 
>> in bug: https://core.tcl-lang.org/tcl/tktview/aaa84fbbc5
> Julian,
> thanks for all your work. All your work is welcome! I don't think there 
> is a decision for the zip directory position. It is probably only to 
> make it work. The initial code was quite buggy and we have fixed a lot 
> of tickets in the meanwhile. We will have follow-up TIPs in this area 
> anyway, as the original TIP highly differs from the implementation. Most 
> active people on my radar were Ashok, Donal and Brian.
> 
>  From my side, please allow me to mention, that exe and dll files may be 
> signed. The signature also goes to the end of the file. So, the zip may 
> not be the last component in the file, at least with signtool on Windows.
> May this be the reason to not use end-relative zip directory positioning?
> 
> Thank you for all,
> Harald
I forgot to mention:
- Christian Werner has contributed and has a solution for Undroidwish
- Konstantin Kushnir is maintaining Cookfs with additional features I 
would love to get in the core.
- Sean Woods contributed the original implementation and the TIP:
https://core.tcl-lang.org/tips/doc/trunk/tip/430.md

Thank you for all,
Harald

Re: [TCLCORE] making zipfs mounts writable

[Harald O. <har...@el...>]

Am 15.10.2024 um 06:35 schrieb Julian Noble via Tcl-Core:
> If the tcl community wants to decide that zipfs as attached to a tcl exe 
> or script *requires* that the first CDR file/dir entry points to the 
> topmost local file header (which is not true under all zip editing 
> scenarios) - then that's a decision that could be made as a specific 
> decision to restrict us to a subset of what the zipfs container allows
>   - but I didn't see it documented or discussed when the change was made 
> - and the 2021 changes appear to have broken the 'zipfs info' command's 
> ability to determine the exe/zip split offset as I describe in bug: 
> https://core.tcl-lang.org/tcl/tktview/aaa84fbbc5
Julian,
thanks for all your work. All your work is welcome! I don't think there 
is a decision for the zip directory position. It is probably only to 
make it work. The initial code was quite buggy and we have fixed a lot 
of tickets in the meanwhile. We will have follow-up TIPs in this area 
anyway, as the original TIP highly differs from the implementation. Most 
active people on my radar were Ashok, Donal and Brian.

 From my side, please allow me to mention, that exe and dll files may be 
signed. The signature also goes to the end of the file. So, the zip may 
not be the last component in the file, at least with signtool on Windows.
May this be the reason to not use end-relative zip directory positioning?

Thank you for all,
Harald

Re: [TCLCORE] making zipfs mounts writable

[Brian G. <bri...@ea...>]

Interesting.  I must have missed a point somewhere. What exactly is the problem to be solved? 

-Brian

> On Oct 14, 2024, at 21:35, Julian Noble via Tcl-Core <tcl...@li...> wrote:
> 
> Hi all,
> 
> I'm keeping this off the ticket system as an RFE for now as I'm guessing it may be perceived as a bit noisy - but I'd like to record some info regarding thoughts on zipfs mounts while I have my head in the space.
> 
> The zip archive format is optimised for easy appends. The CDR - Central Directory Record, is near the end and can be rewritten cheaply.
> Essentially - remove Central Directory File Header record pointing to the old file - and append a new local file record and rewrite the CDR.
> That leaves an old copy of the file unpointed to - which is ugly from a security perspective as well as space used
> - but part of the process could also be to zero out the old contents and rewrite the old name to something like #deleted-filename
> 
> Alternatively the entire old record could just be blanked out - as gaps are allowed in the zip container - but for reusability - it might be better to either do the renaming as above, or even consider having a metadata file that is in the zip that tracks space that has been freed.
> e.g  perhaps something like .zipfs_freespace   - that way we don't make assumptions about other apparent gaps or unpointed to regions in the zip file.
> 
> The point is that by tracking such gaps - we could quite reasonably edit multi-gigabyte zips in place without copying/rewriting the whole file. New files could be added in tracked gaps if they fit.
> ie we could have have a writable zipfs mount.
> 
> Obviously an edited zip wouldn't be the most compact possible representation - but something like a separate 'zipfs compact' operation could be made available to rewrite the whole thing when that is deemed desirable.
> (perhaps also some introspection command to see how much 'empty' space is in a zip)
> 
> This potential usecase is made more difficult by the current heuristic used by tclZipfs.c to determine where the zip data begins. It assumes that first file/dir record in the CDR points to the first valid local file header.
> That mechanism for finding the point between exe & zipdata is only necessary because of the changes made in 2021 to use 'file based' offsets - and while it may work in most cases - I don't see it as particularly robust - hence my reference to it as a heuristic.
> 
> If the tcl community wants to decide that zipfs as attached to a tcl exe or script *requires* that the first CDR file/dir entry points to the topmost local file header (which is not true under all zip editing scenarios) - then that's a decision that could be made as a specific decision to restrict us to a subset of what the zipfs container allows
>  - but I didn't see it documented or discussed when the change was made - and the 2021 changes appear to have broken the 'zipfs info' command's ability to determine the exe/zip split offset as I describe in bug: https://core.tcl-lang.org/tcl/tktview/aaa84fbbc5
> 
> By not using 'file based' offsets and instead using simpler 'archive based' offsets (ie as you get by just catting file.exe with file.zip) - the calculation for finding the split is a simple maths operation, (because we have the size of the central directory and can then compare the recorded offset with our actual absolute position and subtract to get the baseoffset which represents our prepended data)
> 
> We could plough ahead with the 'file based offsets' (offset adjustments) and fix 'zipfs info' to match
> (I think the problem with the 2021 fix is that 'minoff' was used to calculate the position but zf->baseOffset wasn't adjusted - a cursory look at the divergent androwish implementation suggests to me that maybe it was done right there)
> 
> I think it would be a pity  to persist with that as it would seem to make the system harder to cater for the usecase above, and is generally not as simple or flexible -  but I'm hoping someone else can clarify which way Tcl wants to go.
> 
> Currently, a tcl script wanting to split an exezip has to do their own zip parsing and potentially error-prone heuristics to work out the split on an offset 'adjusted' file.
> (There is no unique header representing the start of a zip archive - and from my own tests there are false positives in the tclsh exe binary. Scanning on largeish zips/binaries is ugly anyway)
> 
> Cheers,
> Julian
> 
> 
> 
> 
> 
> 
> 
> 
> _______________________________________________
> Tcl-Core mailing list
> Tcl...@li...
> https://lists.sourceforge.net/lists/listinfo/tcl-core

[TCLCORE] making zipfs mounts writable

[Julian N. <ju...@pr...>]

Hi all,

I'm keeping this off the ticket system as an RFE for now as I'm guessing 
it may be perceived as a bit noisy - but I'd like to record some info 
regarding thoughts on zipfs mounts while I have my head in the space.

The zip archive format is optimised for easy appends. The CDR - Central 
Directory Record, is near the end and can be rewritten cheaply.
Essentially - remove Central Directory File Header record pointing to 
the old file - and append a new local file record and rewrite the CDR.
That leaves an old copy of the file unpointed to - which is ugly from a 
security perspective as well as space used
- but part of the process could also be to zero out the old contents and 
rewrite the old name to something like #deleted-filename

Alternatively the entire old record could just be blanked out - as gaps 
are allowed in the zip container - but for reusability - it might be 
better to either do the renaming as above, or even consider having a 
metadata file that is in the zip that tracks space that has been freed.
e.g  perhaps something like .zipfs_freespace   - that way we don't make 
assumptions about other apparent gaps or unpointed to regions in the zip 
file.

The point is that by tracking such gaps - we could quite reasonably edit 
multi-gigabyte zips in place without copying/rewriting the whole file. 
New files could be added in tracked gaps if they fit.
ie we could have have a writable zipfs mount.

Obviously an edited zip wouldn't be the most compact possible 
representation - but something like a separate 'zipfs compact' operation 
could be made available to rewrite the whole thing when that is deemed 
desirable.
(perhaps also some introspection command to see how much 'empty' space 
is in a zip)

This potential usecase is made more difficult by the current heuristic 
used by tclZipfs.c to determine where the zip data begins. It assumes 
that first file/dir record in the CDR points to the first valid local 
file header.
That mechanism for finding the point between exe & zipdata is only 
necessary because of the changes made in 2021 to use 'file based' 
offsets - and while it may work in most cases - I don't see it as 
particularly robust - hence my reference to it as a heuristic.

If the tcl community wants to decide that zipfs as attached to a tcl exe 
or script *requires* that the first CDR file/dir entry points to the 
topmost local file header (which is not true under all zip editing 
scenarios) - then that's a decision that could be made as a specific 
decision to restrict us to a subset of what the zipfs container allows
  - but I didn't see it documented or discussed when the change was made 
- and the 2021 changes appear to have broken the 'zipfs info' command's 
ability to determine the exe/zip split offset as I describe in bug: 
https://core.tcl-lang.org/tcl/tktview/aaa84fbbc5

By not using 'file based' offsets and instead using simpler 'archive 
based' offsets (ie as you get by just catting file.exe with file.zip) - 
the calculation for finding the split is a simple maths operation, 
(because we have the size of the central directory and can then compare 
the recorded offset with our actual absolute position and subtract to 
get the baseoffset which represents our prepended data)

We could plough ahead with the 'file based offsets' (offset adjustments) 
and fix 'zipfs info' to match
(I think the problem with the 2021 fix is that 'minoff' was used to 
calculate the position but zf->baseOffset wasn't adjusted - a cursory 
look at the divergent androwish implementation suggests to me that maybe 
it was done right there)

I think it would be a pity  to persist with that as it would seem to 
make the system harder to cater for the usecase above, and is generally 
not as simple or flexible -  but I'm hoping someone else can clarify 
which way Tcl wants to go.

Currently, a tcl script wanting to split an exezip has to do their own 
zip parsing and potentially error-prone heuristics to work out the split 
on an offset 'adjusted' file.
(There is no unique header representing the start of a zip archive - and 
from my own tests there are false positives in the tclsh exe binary. 
Scanning on largeish zips/binaries is ugly anyway)

Cheers,
Julian

Re: [TCLCORE] Atomicity in Tcl and Tk

[Harald O. <har...@el...>]

Am 14.10.2024 um 16:55 schrieb Marc Culler:
> I have a question for the experts.  It has to do with when the 
> interpreter, during execution of a script, allows idle tasks and timer 
> tasks to be run.
> 
> There is an expectation in may situations that idle tasks will not be 
> run between two consecutive commands in the script.  For example, in 
> macosx/README it says:
> 
> The TkAqua-specific command [tk::unsupported::MacWindowStyle style] is 
> used to
> get and set macOS-specific toplevel window class and attributes. Note that
> the window class and many attributes have to be set before the window is 
> first
> mapped for the change to have any effect.
> 
> That means that the following code is expected to work:
>    toplevel .t
>    tk::unsupported::MacWindowStyle style .t modal
> 
> but something like this:
> 
>    toplevel .t
>    < lots of code here>
>    tk::unsupported::MacWindowStyle modal
> 
> will probably not work because the window will get mapped during the 
> <lots of code>.  The window actually gets mapped in an idle task that is 
> created by the toplevel command.  The first code block works because 
> that idle task gets run after the MacWindowStyle command has run.
> 
> My question is essentially "How large can the <lots of code> be and have 
> this still work?"  What are the rules?  When running a script, when does 
> the interpreter pause and "return to the event loop"?  Is there some way 
> to ensure that a given block of code will run completely before any 
> additional idle tasks get run?

The code should not contain any invocations of the event loop.
Normal commands doing this are:
- update
- vwait
- tkwait
Extensions may define commands which also invoke the event loop.
And any library code may do so.

IMHO code invoking the event queue should be flagged as such.

A good design of event loop awareness is the http package.
You have two operation modes:
- with command option -> call does not invoke event loop
- without command option -> call invokes the event loop
I always prefer the first solution. Invoking the event loop within a 
command is problematic, as the execution time may not be predicted. 
vwait are nesting. And if within a vwait happens another vwait, the 
outer have to wait for the end of all.

Take care,
Harald

[TCLCORE] Atomicity in Tcl and Tk

[Marc C. <cul...@gm...>]

I have a question for the experts.  It has to do with when the interpreter,
during execution of a script, allows idle tasks and timer tasks to be run.

There is an expectation in may situations that idle tasks will not be run
between two consecutive commands in the script.  For example, in
macosx/README it says:

The TkAqua-specific command [tk::unsupported::MacWindowStyle style] is used
to
get and set macOS-specific toplevel window class and attributes. Note that
the window class and many attributes have to be set before the window is
first
mapped for the change to have any effect.

That means that the following code is expected to work:
  toplevel .t
  tk::unsupported::MacWindowStyle style .t modal

but something like this:

  toplevel .t
  < lots of code here>
  tk::unsupported::MacWindowStyle modal

will probably not work because the window will get mapped during the <lots
of code>.  The window actually gets mapped in an idle task that is created
by the toplevel command.  The first code block works because that idle task
gets run after the MacWindowStyle command has run.

My question is essentially "How large can the <lots of code> be and have
this still work?"  What are the rules?  When running a script, when does
the interpreter pause and "return to the event loop"?  Is there some way to
ensure that a given block of code will run completely before any additional
idle tasks get run?

- Marc

[TCLCORE] Thanks for TCL/Tk 9.0.0 and next steps

[Harald O. <har...@el...>]

Dear TCL/Tk community,

thanks to Don and team for the great release of TCL/Tk9.0.0. This is 
awesome! We all need a breath after this 7 years process.

---

May I propose a meeting to discuss the next steps ?
What about Monday 28th of October 12:00 UTC for one hour?
Many regions will have stopped daytime saving time then.

My personal agenda is:
- what is the next version of TCL/Tk for:
   - non breaking bugfix style TIPs (like the two most recent ones)
     (10.1 ?)
   - innovative TIPs (10.1a0 ?)
   - breaking TIPs (11.0a0 ?)
- what are the possible innovations?
   - factor out the image command from Tk
   - list image formats

By the way, the conference 2025 may be in Italy...

Thank you for all,
Harald

Re: [TCLCORE] TIP 701

[Ashok N. <apn...@ya...>]

Sure, I'm just not sure how the next release is going to be versioned. 
/Ashok

Yahoo Mail: Search, organise, conquer 
 
  On Mon, 14 Oct 2024 at 4:12 pm, Harald Oehlmann<har...@el...> wrote:   Am 04.10.2024 um 15:33 schrieb apnmbx-public--- via Tcl-Core:
> Proudly announcing the first TIP post 9.0 😊 TIP 701: C API for tilde 
> substitution in paths
> 
> https://core.tcl-lang.org/tips/doc/trunk/tip/701.md <https://core.tcl- 
> lang.org/tips/doc/trunk/tip/701.md>
> 
> TIP 602 removed the implicit tilde substitution from Tcl's handling of 
> file paths. The file home and file tildeexpand commands made equivalent 
> functionality explicitly available to scripts. However, no comparable C 
> API was made available to extensions. This TIP remedies the oversight.
> 
> Please review. CFV in 2 weeks.
> 
> /Ashok
Dear Ashok,

thanks for the great TIP. I only have a question about the concerned TCL 
version. IMHO, this should go into version 9.0.1, not 9.1.x.

Take care,
Harald

  

Re: [TCLCORE] TIP 701

[Harald O. <har...@el...>]

Am 04.10.2024 um 15:33 schrieb apnmbx-public--- via Tcl-Core:
> Proudly announcing the first TIP post 9.0 😊 TIP 701: C API for tilde 
> substitution in paths
> 
> https://core.tcl-lang.org/tips/doc/trunk/tip/701.md <https://core.tcl- 
> lang.org/tips/doc/trunk/tip/701.md>
> 
> TIP 602 removed the implicit tilde substitution from Tcl's handling of 
> file paths. The file home and file tildeexpand commands made equivalent 
> functionality explicitly available to scripts. However, no comparable C 
> API was made available to extensions. This TIP remedies the oversight.
> 
> Please review. CFV in 2 weeks.
> 
> /Ashok
Dear Ashok,

thanks for the great TIP. I only have a question about the concerned TCL 
version. IMHO, this should go into version 9.0.1, not 9.1.x.

Take care,
Harald