From: <no...@so...> - 2002-03-21 18:12:10
|
Bugs item #533221, was opened at 2002-03-21 09:12 You can respond by visiting: http://sourceforge.net/tracker/?func=detail&atid=110894&aid=533221&group_id=10894 Category: 37. Package Manager Group: None Status: Open Resolution: None Priority: 5 Submitted By: Christopher Nelson (chris_nelson) Assigned to: Don Porter (dgp) Summary: Problems with binary packages on NT 4.0 Initial Comment: For RFE 4097, sourceforge bug 219042 (pkg_mkIndex unnecessarily creates command list for direct lo), changes were made to pkg_mkIndex that appear to have undesireable side effects. Here's the note from my colleague: ------------------------ The first issue is a change in some default behavior which I believe should be questioned. The second appears to be a real bug. First, the pkg_mkIndex default behavior changed from setting up lazy loading to doing direct loading. I found the bug (4097 RFE) which led to the change in pkg_mkIndex. However, it was only a request for an optimization to pkg_mkIndex when the -direct switch was used. The change to the default behavior appears to have slipped in at that point. I am concerned that this represents a change in "philosophy" with regard to "package require". I used to consider them to be essentially "free" and scatter them about with relative abandon. Now, I have to be aware that, by default, the first time a package is required the flow will stop while the package is loaded. However, this is but a trifle. The real issue is this. We have some DLLs which have both Tcl and non-Tcl entry points. So, we link our binaries against them, but also do "package require" calls which will ultimately dynamically load the same DLLs. Our pkg_mkIndex calls, and thus the resulting pkgIndex.tcl files, have been carefully crafted to insure that the DLL that is loaded by Tcl (using a relative path) is referencing the same file as the one the OS will load when the .exe originally bootstraps. Well, when the package is done loading, we end up with two copies of the DLL in memory, but ONLY ON NT 4.0! ... The doubly-loaded DLL is, I thought, not supposed to be possible. When we go back to our Tcl-8.2-based product, the doubly-loaded DLL problem does not occur, regardless of direct or lazy package loading. The win/tclWinLoad.c file appears only to have changed with regard to improved error handling from 8.2 to 8.4. ------------------------ We initially thought that reverting to lazy loading with Tcl 8.4 solved the problem but have gotten inconsistent results in further testing. This is in 8.4a3, a "Group" which doesn't appear to exist. ---------------------------------------------------------------------- You can respond by visiting: http://sourceforge.net/tracker/?func=detail&atid=110894&aid=533221&group_id=10894 |
From: <no...@so...> - 2002-03-21 22:34:52
|
Bugs item #533221, was opened at 2002-03-21 09:12 You can respond by visiting: http://sourceforge.net/tracker/?func=detail&atid=110894&aid=533221&group_id=10894 Category: 37. Package Manager Group: None Status: Open Resolution: None Priority: 5 Submitted By: Christopher Nelson (chris_nelson) Assigned to: Don Porter (dgp) Summary: Problems with binary packages on NT 4.0 Initial Comment: For RFE 4097, sourceforge bug 219042 (pkg_mkIndex unnecessarily creates command list for direct lo), changes were made to pkg_mkIndex that appear to have undesireable side effects. Here's the note from my colleague: ------------------------ The first issue is a change in some default behavior which I believe should be questioned. The second appears to be a real bug. First, the pkg_mkIndex default behavior changed from setting up lazy loading to doing direct loading. I found the bug (4097 RFE) which led to the change in pkg_mkIndex. However, it was only a request for an optimization to pkg_mkIndex when the -direct switch was used. The change to the default behavior appears to have slipped in at that point. I am concerned that this represents a change in "philosophy" with regard to "package require". I used to consider them to be essentially "free" and scatter them about with relative abandon. Now, I have to be aware that, by default, the first time a package is required the flow will stop while the package is loaded. However, this is but a trifle. The real issue is this. We have some DLLs which have both Tcl and non-Tcl entry points. So, we link our binaries against them, but also do "package require" calls which will ultimately dynamically load the same DLLs. Our pkg_mkIndex calls, and thus the resulting pkgIndex.tcl files, have been carefully crafted to insure that the DLL that is loaded by Tcl (using a relative path) is referencing the same file as the one the OS will load when the .exe originally bootstraps. Well, when the package is done loading, we end up with two copies of the DLL in memory, but ONLY ON NT 4.0! ... The doubly-loaded DLL is, I thought, not supposed to be possible. When we go back to our Tcl-8.2-based product, the doubly-loaded DLL problem does not occur, regardless of direct or lazy package loading. The win/tclWinLoad.c file appears only to have changed with regard to improved error handling from 8.2 to 8.4. ------------------------ We initially thought that reverting to lazy loading with Tcl 8.4 solved the problem but have gotten inconsistent results in further testing. This is in 8.4a3, a "Group" which doesn't appear to exist. ---------------------------------------------------------------------- >Comment By: Christopher Nelson (chris_nelson) Date: 2002-03-21 13:34 Message: Logged In: YES user_id=107514 We're resolved the problem in a narrow way by changing our code but we really don't understand why or how it broke between 8.2 and 8.4. Here're notes from my colleague: The problem was a doubly-loaded DLL, causing memory updated in one path through the code to be unavailable, despite the apparently identical variable names, via another path through the code. The root of this evil was the entangling of Tcl and non-Tcl entry points in the same DLLs. ... By creating a 2nd wrapper DLL for the tcl-specific code, and linking _that_ against the original dll, which is now Tcl-free, the problem appears to be resolved. Still doesn't answer why the problem occurred in the first place, particularly when it is not evident in 8.2. ---------------------------------------------------------------------- You can respond by visiting: http://sourceforge.net/tracker/?func=detail&atid=110894&aid=533221&group_id=10894 |
From: <no...@so...> - 2002-03-21 23:55:10
|
Bugs item #533221, was opened at 2002-03-21 13:12 You can respond by visiting: http://sourceforge.net/tracker/?func=detail&atid=110894&aid=533221&group_id=10894 >Category: 38. Dynamic Loading >Group: = 8.4a3 Status: Open Resolution: None Priority: 5 Submitted By: Christopher Nelson (chris_nelson) >Assigned to: Kevin B KENNY (kennykb) Summary: Problems with binary packages on NT 4.0 Initial Comment: For RFE 4097, sourceforge bug 219042 (pkg_mkIndex unnecessarily creates command list for direct lo), changes were made to pkg_mkIndex that appear to have undesireable side effects. Here's the note from my colleague: ------------------------ The first issue is a change in some default behavior which I believe should be questioned. The second appears to be a real bug. First, the pkg_mkIndex default behavior changed from setting up lazy loading to doing direct loading. I found the bug (4097 RFE) which led to the change in pkg_mkIndex. However, it was only a request for an optimization to pkg_mkIndex when the -direct switch was used. The change to the default behavior appears to have slipped in at that point. I am concerned that this represents a change in "philosophy" with regard to "package require". I used to consider them to be essentially "free" and scatter them about with relative abandon. Now, I have to be aware that, by default, the first time a package is required the flow will stop while the package is loaded. However, this is but a trifle. The real issue is this. We have some DLLs which have both Tcl and non-Tcl entry points. So, we link our binaries against them, but also do "package require" calls which will ultimately dynamically load the same DLLs. Our pkg_mkIndex calls, and thus the resulting pkgIndex.tcl files, have been carefully crafted to insure that the DLL that is loaded by Tcl (using a relative path) is referencing the same file as the one the OS will load when the .exe originally bootstraps. Well, when the package is done loading, we end up with two copies of the DLL in memory, but ONLY ON NT 4.0! ... The doubly-loaded DLL is, I thought, not supposed to be possible. When we go back to our Tcl-8.2-based product, the doubly-loaded DLL problem does not occur, regardless of direct or lazy package loading. The win/tclWinLoad.c file appears only to have changed with regard to improved error handling from 8.2 to 8.4. ------------------------ We initially thought that reverting to lazy loading with Tcl 8.4 solved the problem but have gotten inconsistent results in further testing. This is in 8.4a3, a "Group" which doesn't appear to exist. ---------------------------------------------------------------------- >Comment By: Don Porter (dgp) Date: 2002-03-21 18:55 Message: Logged In: YES user_id=80530 This sounds like either a problem with [load], or a bug in your program, or both. You say you've linked your binaries against these libraries already. In that case, the proper way to register their *_Init routines (Tcl entry points) are with Tcl_StaticPackage(). You should also set up your own [package ifneeded] registration to handle this with a [load {} $Pkg], re-using the existing library already linked into the executable. There may also be a bug in [load] if its not storing absolute pathnames for its checks again loading the same library twice. Assigning to the [load] maintainer for another opinion. The 8.4a3 Group is still there -- way at the bottom of the list. (Assigning it now...) Bug reports against the latest releases are the most encouraged, so those choices are at the top. ---------------------------------------------------------------------- Comment By: Christopher Nelson (chris_nelson) Date: 2002-03-21 17:34 Message: Logged In: YES user_id=107514 We're resolved the problem in a narrow way by changing our code but we really don't understand why or how it broke between 8.2 and 8.4. Here're notes from my colleague: The problem was a doubly-loaded DLL, causing memory updated in one path through the code to be unavailable, despite the apparently identical variable names, via another path through the code. The root of this evil was the entangling of Tcl and non-Tcl entry points in the same DLLs. ... By creating a 2nd wrapper DLL for the tcl-specific code, and linking _that_ against the original dll, which is now Tcl-free, the problem appears to be resolved. Still doesn't answer why the problem occurred in the first place, particularly when it is not evident in 8.2. ---------------------------------------------------------------------- You can respond by visiting: http://sourceforge.net/tracker/?func=detail&atid=110894&aid=533221&group_id=10894 |
From: <no...@so...> - 2002-03-22 22:04:33
|
Bugs item #533221, was opened at 2002-03-21 13:12 You can respond by visiting: http://sourceforge.net/tracker/?func=detail&atid=110894&aid=533221&group_id=10894 Category: 38. Dynamic Loading Group: = 8.4a3 Status: Open Resolution: None Priority: 5 Submitted By: Christopher Nelson (chris_nelson) Assigned to: Kevin B KENNY (kennykb) Summary: Problems with binary packages on NT 4.0 Initial Comment: For RFE 4097, sourceforge bug 219042 (pkg_mkIndex unnecessarily creates command list for direct lo), changes were made to pkg_mkIndex that appear to have undesireable side effects. Here's the note from my colleague: ------------------------ The first issue is a change in some default behavior which I believe should be questioned. The second appears to be a real bug. First, the pkg_mkIndex default behavior changed from setting up lazy loading to doing direct loading. I found the bug (4097 RFE) which led to the change in pkg_mkIndex. However, it was only a request for an optimization to pkg_mkIndex when the -direct switch was used. The change to the default behavior appears to have slipped in at that point. I am concerned that this represents a change in "philosophy" with regard to "package require". I used to consider them to be essentially "free" and scatter them about with relative abandon. Now, I have to be aware that, by default, the first time a package is required the flow will stop while the package is loaded. However, this is but a trifle. The real issue is this. We have some DLLs which have both Tcl and non-Tcl entry points. So, we link our binaries against them, but also do "package require" calls which will ultimately dynamically load the same DLLs. Our pkg_mkIndex calls, and thus the resulting pkgIndex.tcl files, have been carefully crafted to insure that the DLL that is loaded by Tcl (using a relative path) is referencing the same file as the one the OS will load when the .exe originally bootstraps. Well, when the package is done loading, we end up with two copies of the DLL in memory, but ONLY ON NT 4.0! ... The doubly-loaded DLL is, I thought, not supposed to be possible. When we go back to our Tcl-8.2-based product, the doubly-loaded DLL problem does not occur, regardless of direct or lazy package loading. The win/tclWinLoad.c file appears only to have changed with regard to improved error handling from 8.2 to 8.4. ------------------------ We initially thought that reverting to lazy loading with Tcl 8.4 solved the problem but have gotten inconsistent results in further testing. This is in 8.4a3, a "Group" which doesn't appear to exist. ---------------------------------------------------------------------- >Comment By: Kevin B KENNY (kennykb) Date: 2002-03-22 17:04 Message: Logged In: YES user_id=99768 The change from lazy to eager loading in 8.4 was by design, but in any case seems irrelevant to the main point of this bug report. The double loading of a DLL is hard to explain, particularly in light of the fact that the LoadLibrary call, which tclWinLoad.c uses internally, is documented never to load any DLL more than once. The only explanation that I can easily arrive at is that the DLL found when resolving the import table of the executable was NOT the same file -- or at least, did not have the same path name (I'm not sure what happens if LoadLibrary is called with short name versus long name.) It's conceivable that NT 4.0 in particular has a bug in this area, but I haven't found anything in a cursory search of MSDN. In any case, the [load] command has no way of being aware what DLLs may be present in the process as a result of arbitrary C code. Your solution of packaging the C API's separately from their TCL wrappers is a good one if it's practicable; at least, it means that you don't have a <package>_Init function that's never called floating around in the executable somewhere. Another good idea is to do compatibility checking inside your pkgIndex.tcl files. This is not done easily if they are made by pkg_mkIndex, but it's actually easier in most cases to build the indices by hand! Don's suggestion of doing Tcl_StaticPackage for all packages that you know to be statically linked is also good practice. Adding appropriate [package ifneeded] calls at initialization will then load the statically linked packages without needing to resort to LoadLibrary. Another good practice would be to use the Stubs mechanism, both for backlinking to Tcl itself and for crosslinking among your load modules. This can often reduce or eliminate the dependencies among the DLLs so that you don't need as many multiple copies of things. Finally, I'd suggest that you audit your NT 4.0 test configuration and make sure that your "doubly loaded" DLL isn't turning up someplace unexpected. Given the way that LoadLibrary works, it's hard to imaging the double loading actually happening any other way. ---------------------------------------------------------------------- Comment By: Don Porter (dgp) Date: 2002-03-21 18:55 Message: Logged In: YES user_id=80530 This sounds like either a problem with [load], or a bug in your program, or both. You say you've linked your binaries against these libraries already. In that case, the proper way to register their *_Init routines (Tcl entry points) are with Tcl_StaticPackage(). You should also set up your own [package ifneeded] registration to handle this with a [load {} $Pkg], re-using the existing library already linked into the executable. There may also be a bug in [load] if its not storing absolute pathnames for its checks again loading the same library twice. Assigning to the [load] maintainer for another opinion. The 8.4a3 Group is still there -- way at the bottom of the list. (Assigning it now...) Bug reports against the latest releases are the most encouraged, so those choices are at the top. ---------------------------------------------------------------------- Comment By: Christopher Nelson (chris_nelson) Date: 2002-03-21 17:34 Message: Logged In: YES user_id=107514 We're resolved the problem in a narrow way by changing our code but we really don't understand why or how it broke between 8.2 and 8.4. Here're notes from my colleague: The problem was a doubly-loaded DLL, causing memory updated in one path through the code to be unavailable, despite the apparently identical variable names, via another path through the code. The root of this evil was the entangling of Tcl and non-Tcl entry points in the same DLLs. ... By creating a 2nd wrapper DLL for the tcl-specific code, and linking _that_ against the original dll, which is now Tcl-free, the problem appears to be resolved. Still doesn't answer why the problem occurred in the first place, particularly when it is not evident in 8.2. ---------------------------------------------------------------------- You can respond by visiting: http://sourceforge.net/tracker/?func=detail&atid=110894&aid=533221&group_id=10894 |
From: <no...@so...> - 2002-03-22 22:14:52
|
Bugs item #533221, was opened at 2002-03-21 13:12 You can respond by visiting: http://sourceforge.net/tracker/?func=detail&atid=110894&aid=533221&group_id=10894 Category: 38. Dynamic Loading Group: = 8.4a3 >Status: Closed >Resolution: Invalid Priority: 5 Submitted By: Christopher Nelson (chris_nelson) Assigned to: Kevin B KENNY (kennykb) Summary: Problems with binary packages on NT 4.0 Initial Comment: For RFE 4097, sourceforge bug 219042 (pkg_mkIndex unnecessarily creates command list for direct lo), changes were made to pkg_mkIndex that appear to have undesireable side effects. Here's the note from my colleague: ------------------------ The first issue is a change in some default behavior which I believe should be questioned. The second appears to be a real bug. First, the pkg_mkIndex default behavior changed from setting up lazy loading to doing direct loading. I found the bug (4097 RFE) which led to the change in pkg_mkIndex. However, it was only a request for an optimization to pkg_mkIndex when the -direct switch was used. The change to the default behavior appears to have slipped in at that point. I am concerned that this represents a change in "philosophy" with regard to "package require". I used to consider them to be essentially "free" and scatter them about with relative abandon. Now, I have to be aware that, by default, the first time a package is required the flow will stop while the package is loaded. However, this is but a trifle. The real issue is this. We have some DLLs which have both Tcl and non-Tcl entry points. So, we link our binaries against them, but also do "package require" calls which will ultimately dynamically load the same DLLs. Our pkg_mkIndex calls, and thus the resulting pkgIndex.tcl files, have been carefully crafted to insure that the DLL that is loaded by Tcl (using a relative path) is referencing the same file as the one the OS will load when the .exe originally bootstraps. Well, when the package is done loading, we end up with two copies of the DLL in memory, but ONLY ON NT 4.0! ... The doubly-loaded DLL is, I thought, not supposed to be possible. When we go back to our Tcl-8.2-based product, the doubly-loaded DLL problem does not occur, regardless of direct or lazy package loading. The win/tclWinLoad.c file appears only to have changed with regard to improved error handling from 8.2 to 8.4. ------------------------ We initially thought that reverting to lazy loading with Tcl 8.4 solved the problem but have gotten inconsistent results in further testing. This is in 8.4a3, a "Group" which doesn't appear to exist. ---------------------------------------------------------------------- Comment By: Kevin B KENNY (kennykb) Date: 2002-03-22 17:04 Message: Logged In: YES user_id=99768 The change from lazy to eager loading in 8.4 was by design, but in any case seems irrelevant to the main point of this bug report. The double loading of a DLL is hard to explain, particularly in light of the fact that the LoadLibrary call, which tclWinLoad.c uses internally, is documented never to load any DLL more than once. The only explanation that I can easily arrive at is that the DLL found when resolving the import table of the executable was NOT the same file -- or at least, did not have the same path name (I'm not sure what happens if LoadLibrary is called with short name versus long name.) It's conceivable that NT 4.0 in particular has a bug in this area, but I haven't found anything in a cursory search of MSDN. In any case, the [load] command has no way of being aware what DLLs may be present in the process as a result of arbitrary C code. Your solution of packaging the C API's separately from their TCL wrappers is a good one if it's practicable; at least, it means that you don't have a <package>_Init function that's never called floating around in the executable somewhere. Another good idea is to do compatibility checking inside your pkgIndex.tcl files. This is not done easily if they are made by pkg_mkIndex, but it's actually easier in most cases to build the indices by hand! Don's suggestion of doing Tcl_StaticPackage for all packages that you know to be statically linked is also good practice. Adding appropriate [package ifneeded] calls at initialization will then load the statically linked packages without needing to resort to LoadLibrary. Another good practice would be to use the Stubs mechanism, both for backlinking to Tcl itself and for crosslinking among your load modules. This can often reduce or eliminate the dependencies among the DLLs so that you don't need as many multiple copies of things. Finally, I'd suggest that you audit your NT 4.0 test configuration and make sure that your "doubly loaded" DLL isn't turning up someplace unexpected. Given the way that LoadLibrary works, it's hard to imaging the double loading actually happening any other way. ---------------------------------------------------------------------- Comment By: Don Porter (dgp) Date: 2002-03-21 18:55 Message: Logged In: YES user_id=80530 This sounds like either a problem with [load], or a bug in your program, or both. You say you've linked your binaries against these libraries already. In that case, the proper way to register their *_Init routines (Tcl entry points) are with Tcl_StaticPackage(). You should also set up your own [package ifneeded] registration to handle this with a [load {} $Pkg], re-using the existing library already linked into the executable. There may also be a bug in [load] if its not storing absolute pathnames for its checks again loading the same library twice. Assigning to the [load] maintainer for another opinion. The 8.4a3 Group is still there -- way at the bottom of the list. (Assigning it now...) Bug reports against the latest releases are the most encouraged, so those choices are at the top. ---------------------------------------------------------------------- Comment By: Christopher Nelson (chris_nelson) Date: 2002-03-21 17:34 Message: Logged In: YES user_id=107514 We're resolved the problem in a narrow way by changing our code but we really don't understand why or how it broke between 8.2 and 8.4. Here're notes from my colleague: The problem was a doubly-loaded DLL, causing memory updated in one path through the code to be unavailable, despite the apparently identical variable names, via another path through the code. The root of this evil was the entangling of Tcl and non-Tcl entry points in the same DLLs. ... By creating a 2nd wrapper DLL for the tcl-specific code, and linking _that_ against the original dll, which is now Tcl-free, the problem appears to be resolved. Still doesn't answer why the problem occurred in the first place, particularly when it is not evident in 8.2. ---------------------------------------------------------------------- You can respond by visiting: http://sourceforge.net/tracker/?func=detail&atid=110894&aid=533221&group_id=10894 |