#177 "package require twapi" fails with empty error message

[Harald Oehlmann]

Ashok,

thank you for twapi. I have a strange phenomenon.

• twapi 8.3.5 is copied to the lib folder of a self-compiled tcl 32 bit.
• starting wish and typing "package require twapi" results in "8.3.5" and no error.
• After a long program initialisation with the same wish and twapi, I get a raised error with an empty error message on twapi package require:

% catch {package require twapi} err
1
% set err
% 

Do you have any idea where to search for the issue ?

Thank you,
Harald

[Harald Oehlmann]

Some more observation:

• this is tcl/tk 8.6.9
• the issue arises only, if twapi is loaded after a dll written by me, which contains a .net interface to an application loading a lot of stuff

Ok, as a work-around, I will load twapi before that dll.

[Ashok P. Nadkarni]

Does ::errorInfo show anything? Even if it works with switching loading around I'd be a little bit nervous that something unexpected is going on in interactions between the two extensions. For example, COM being initialized with different flags.

[Harald Oehlmann]

Yes, it does:

> 
>     while executing
> "load c:/myprograms/tcl8.6/lib/twapi-4.3.5/twapi_base.dll twapi_base"
>     ("uplevel" body line 1)
>     invoked from within
> "uplevel #0 $loadcmd"
>     (procedure "twapi::package_setup" line 47)
>     invoked from within
> "twapi::package_setup c:/myprograms/tcl8.6/lib/twapi-4.3.5 twapi_base 4.3.5 load twapi_base {}"
>     ("package ifneeded twapi_base 4.3.5" script)
>     invoked from within
> "package require twapi_base 4.3.5"
>     ("package ifneeded twapi 4.3.5" script)
>     invoked from within
> "package require twapi"

The first line is the empty error message.
It is from my log file, which puts "> " in front of each line.

Thanks,
Harald

[Harald Oehlmann]

Oh, and it is twapi 4.3.5, not 8.3.5...

[Ashok P. Nadkarni]

Looking at the code, the Twapi initialization function always includes an error message on errors it generates itself but for called code it assumes the called function will always set interp result on error. Modified it to always append an error string.

I don't know if this will fix the missing error message problem in all cases particularly since I can't reproduce it. Marking as closed in 4.3.6, please reopen if you find the problem persists.

[Harald Oehlmann]

Thank you. If you want, I may test it. Do you have a pointer of the modified version ?
Thank you,
Harald

[Anonymous]

It's the just released 4.3.7.

Note that this only fixes the lack of an error message.

[Harald Oehlmann]

It is checked with twapi 4.3.7 twapi.bin and a self compiled tcl 8.6.9 wish (VC6).

The result is still the same:

% package require twapi
% set errorInfo

    while executing
"load c:/myprograms/tcl8.6/lib/twapi-4.3.7/twapi_base.dll twapi_base"
    ("uplevel" body line 1)
    invoked from within
"uplevel #0 $loadcmd"
    (procedure "twapi::package_setup" line 47)
    invoked from within
"twapi::package_setup c:/myprograms/tcl8.6/lib/twapi-4.3.7 twapi_base 4.3.7 load twapi_base {}"
    ("package ifneeded twapi_base 4.3.7" script)
    invoked from within
"package require twapi_base 4.3.7"
    ("package ifneeded twapi 4.3.7" script)
    invoked from within
"package require twapi"

The following is for me. It is my internal sequence to invoke the issue. One need a full NiceLabel PowerForms installation to test which is a huge bunch of installation.

% load ifnlnet.dll
% ifnlnet init {C:\Program Files\NiceLabel\NiceLabel 2019\bin.net}
% package require twapi

Give me a sign, if you want a test environment. It is no fun and not worthwile the work.

Thank you,
Harald

[Ashok P. Nadkarni]

I'll just remark it as open for now then for future reference.

[Anonymous]

Hello
I have the same error message as described by Harald but I have an environment tool which have embedded TCL-scripting functionalities. So I am limited in the possibilies to change or update the TCL version or the tcl program.
To implement a package was working proberly with the package tcom but with twapi I get the errors same as Harald.
Is there any action on this particular topic? I saw the last entry from 2019 was closed --> open

Thanks in advance
Wolfgang

[Anonymous]

I forgot to mentioned that I tried with twapi-4.5.2 and with twapi-4.3.7 version.
For the tcl version I did not found the complete version I have just seen the a version in my lib folder called -> tcl860lib/twapi/twapi_base.dll twapi_base, so I expect tcl v8.6 .?

Wolfgang

[Anonymous]

Thank you for the message.
I just retested with twapi 4.5.2.

It is still the same behaviour: empty error message if loaded after my dll, ok if loaded before the dll.

> 
>     while executing
> "load {} twapi_base"
>     ("uplevel" body line 1)
>     invoked from within
> "uplevel #0 $loadcmd"
>     (procedure "twapi::package_setup" line 47)
>     invoked from within
> "twapi::package_setup C:/oehhar/elmicron/projekte/el1043_elmiprint/elmiprintli

I may provide a test environment. Unfortunately, it depends on a valid NiceLabel Powerforms Suite licence. I may try what happens, if the error also shows up if this is not present.

Thank you again,
Harald

[Ashok P. Nadkarni]

Wolfgang,

As Harald kindly reminded me, his issue was caused by loading a .Net DLL before loading twapi. Switching the order of loading made the problem go away. I have no clue as to why as I am unable to reproduce the problem.

To try find a common cause could you tell me if

• you are loading from a standard Tcl shell, a Tclkit or a custom shell? If custom or tclkit, does twapi load properly from a standard Tcl shell?
• are you loading any other DLLs and if you load nothing other than twapi, does it make a difference
• which twapi distribution are you using - the twapi.zip, the twapi-bin.zip or twapi-modular.zip
• could you make sure you do not have multiple copies of twapi in your path (just to eliminate that possibility)

Finally, if your set up is not proprietary, would it be possible for me to emulate it?

/Ashok

[Anonymous]

Dear Ashok, Harald
Thanks for your fast respond.
Please note that I am a basic user, so maybe I do stupid failures or do not understand well your meanings.
@you are loading from a standard Tcl shell, a Tclkit or a custom shell?
I tried to explain, that I use a tool which is a kind of Terminal for an external device which provides the possibility to run tcl script and there are already makro scripts embedded. So I do not know how the final tcl script start nor how it is realised or which dedicated version is used. This program is an internal program which I can not distribute.

@are you loading any other DLLs and if you load nothing other than twapi, does it make a difference
YES I have already import the "package require tcom " see files from the tcom folder in the attachment, which have also .dll files, so I guess it should be same.
For the workaround topic from Harald -> I get this error if I only use the "package require twapi" command ,but as mentioned I do not know how the final tcl programm starts, so maybe there will imported another package(s) which I do not know/see. In any case I could not change it.
Find in the attachment the files which are located in the tool instalation folder, which may help you to know more details. The tool was developed with MS Visual Studio, but I have no details about the TCL scripting options which is implemented.

@which twapi distribution are you using - the twapi.zip,...
I have used two times the the twapi.zip files (-> twapi-4.5.2.zip)

@could you make sure you do not have multiple copies of twapi in your path (just to eliminate that possibility)
I checked and there aren´t multiple copies, in the attachment you will see my lib-directory and files in the twapi folder

@Harald comment
I have exactly the same error returns, so also no error message comming from the catch function. As mentioned I do not know if there will be a further package called before my TCL code will be executed, so unfortunately no chance for Harald workaround.

At my google searches I found also observations from 2013 which my be helpfull which have quite similar error returns, see https://wiki.tcl-lang.org/diff/TWAPI

[MHo] 2013-11-12: Another observation with v4-beta: package require'ing twapi-modular from tcl 8.6: no problem. package require'ing twapi-modular from tcl 8.5 gives this error, e.g.:
couldn't load library "d:/var/temp/TCL63.tmp": this library or a dependent library could not be found in library path
while executing
"load D:/Home/Hoffmann/pgm/tcl/usr/Src/tclstdlib/tclstdlib.kit/lib/twapi-modular/twapiservice.dll twapiservice"
("uplevel" body line 1)
....

An interesting detail will be the difference of the first sentence which I do not see on my returns. From my feeling the first part
couldn't load library "d:/var/temp/TCL63.tmp": this library or a dependent library could not be found in library path
seems to be cropped in Harald and my case.

The details there are not clear enough to me especially the vfs topic.

Wolfgang

[Anonymous]

Hello
By checking my last post I saw that something went wrong with my my attached screen shots, which I have added now.
Wolfgang

[Anonymous]

I just sent a very complicated package to Ashok to provoke the error.
It includes a trial version of NiceLabel 2019 which is required to cause the issue for me.
I hope this will help,
Harald

[Ashok P. Nadkarni]

I spent some time going over the code this weekend and came up empty. The question is not just why twapi fails to load but also why the error message is empty. I'll check out Harald's package but in the meanwhile Wolfgang, can you try two tests and let me know the result?

First, start your application and cd to the directory where twapi is installed. Then within the application do the following commands (I'm assuming a 64-bit platform and twapi:

load twapi_base64.dll
twapi::get_system_time

Second test after restarting application, again cd to the twapi directory, and execute the following:

set dir [pwd]
source pkgIndex.tcl
package require twapi
twapi::get_system_time

In both cases, please report whether they work and if not whether there is an error message or error message is still empty.

Thanks

/Ashok

[Ashok P. Nadkarni]

Thanks to Harald's test set up, I've found what the problem is and I suspect Wolfgang's issue is the same. I'm afraid the news is not good.

The summary is that Windows supports two COM concurrency models - apartment model and multithreaded model. Any thread must run in one of these by calling CoInitializeEx. Tcl/TWAPI specifies the apartment model when calling CoInitializeEx. This is required for two reasons - as documented in the Note here - https://docs.microsoft.com/en-us/windows/win32/api/objbase/ne-objbase-coinit - any UI thread or thread requiring a message pump (like Tcl does) must use the apartment model as the multithread model does not pump messages. In addition, Tcl interpreters are limited to a thread so any (COM) callbacks must take place in the original thread. This is guaranteed by the apartment model but not guaranteed by the multithreaded model. Thus it is not possible for twapi to switch to the multithreaded model.

So what was happening in Harald's application was that if the NiceLabel DLL was loaded first, it initialized to the multithreaded model. Then the twapi call to CoInitializeEx failed because it requested the apartment model. When the order is switched around, I suspect the NiceLabel DLL sees CoInitiaze has already been called and calls it with the same flag or not at all. Thus there are no errors, or perhaps they are ignored. Hard for me to say.

Either way, this requirement for apartment model is not something twapi can avoid as it comes from Tcl AND Windows (which requires it for UI threads.)

The issue with the empty message was just my being blind to a missing error conversion call when CoInitialize failed. Unfortunate, because we would have discovered the issue far earlier :-(

Now coming to Wolfgang's case, I suspect it is the same problem but knowing the application would confirm it. I can also supply a new twapi with the error message fixed so you will see the error instead of an empty message (just to confirm it is the same problem).

I am not sure what the solution is other than loading twapi first. Even if that works, there is the danger that the other application is not checking the return from CoInitializeEx correctly and the program will fail in some form later (hang or a crash). If the application actually falls back to a apartment model if the multithread request fails, then all will be well.

One possibility is to lazy initialize twapi's COM so if you are not using COM there will be no issues. This is a little tricky though because COM is kind of pervasive even internally to twapi (for WMI, MSI and other places).

Anyways, that's where things stand.

See https://docs.microsoft.com/en-us/windows/win32/api/combaseapi/nf-combaseapi-coinitializeex and https://docs.microsoft.com/en-us/windows/win32/api/objbase/ne-objbase-coinit for details.

/Ashok

PS I think Wolfgang mentioned he is using tcom? If so, I would have to check as to how. Perhaps it is implemented completely in a separate thread communicating with Tcl over a separate communication mechanism. twapi loads in the same thread as Tcl and would be next to impossible to change. Of course it is also possible the problem Wolfgang is seeing is something else and not the same as Harald's issue.

[Anonymous]

I tried the following

1st test
% file exists C:\KALA_Test\AltiLab_v2_2_1\tcl860lib\twapi\twapi_base64.dll
1
% load C:\KALA_Test\AltiLab_v2_2_1\tcl860lib\twapi\twapi_base64.dll
couldn't load library "C:\KALA_Test\AltiLab_v2_2_1\tcl860lib\twapi\twapi_base64.dll": invalid argument
while executing
"load C:\KALA_Test\AltiLab_v2_2_1\tcl860lib\twapi\twapi_base64.dll"

2nd test
% set dir [pwd]
C:/Program Files (x86)/Debugtool/Lab_v2_2_1

% source pkgIndex.tcl
couldn't read file "pkgIndex.tcl": no such file or directory
while executing
"source pkgIndex.tcl"

% package require twapi
while executing"load {C:/Program Files (x86)/Debugtool/Lab_v2_2_1/tcl860lib/twapi/twapi_base.dll} twapi_base" ("uplevel" body line 1) invoked from within"uplevel #0 $loadcmd" (procedure "twapi::package_setup" line 47) invoked from within"twapi::package_setup {C:/Program Files (x86)/Debugtool/Lab_v2_2_1/tcl860lib/twapi} twapi_base 4.5.2 load twapi_base {}" ("package ifneeded twapi_base 4.5.2" script) invoked from within"package require twapi_base 4.5.2" ("package ifneeded twapi 4.5.2" script) invoked from within"package require twapi"
%

3rd trial
Without having the twapi folder in my lib directory
% package require twapi
can't find package twapi
while executing
"package require twapi"

4rd trial
load twapi
% package require tcom
3.9

[Anonymous]

I checked and try with some files from my lib folder and found following observations, which my helpfull to find the a solution.
Before starting my tcl-scripts the customized script (init-all.tcl -> see attachment) from the tool developer will be executed. In this file [source "$::tcl_library/init.tcl"] file will be called. Before this command it won´t work to call any [package require xxx].
So maybe you could get helpful information to find a solution or maybe I can try to find the workaround solution from Harald, if try to change the init.tcl content to load the twapi-package first.

[Anonymous]

Hi Ashok,
NiceLabel Support answered, that they don't call ColInitializeEx but an initialization of the .net framework does:

https://devblogs.microsoft.com/oldnewthing/20150316-00/?p=44463

They are investigating, if there is any harm in setting the appartment method in advance.

Thank you,
Harald

[Anonymous]

Wolfgang,

Sorry about the delay in response. If I give you a build of twapi with an error message fix that pinpointed Harald's load failure, will you able to try it?

At least it will tell us if the problem you are seeing is the same as what Harald was seeing.

/Ashok

[Anonymous]

Ashok

Sure, I will try.

Wolfgang