Getting a Python exception (access violation) after running back-to-back function calls with the exact same parameters. Using pdb, I've captured traces of both the calls for comparison.
Mark, I'm afraid there is not enough information in the traces you have posted, but maybe I'm only too lazy to stare at them for a long time.
It looks like that the freeing of some resources (safearrays? or strings contained in safearrays?) is causing the problem - I am not at all sure that the safearray code is bug free.
Can you in any way make more information available?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I've been going over the pdb trace and it occurred to me that perhaps the call to _free(self) on line 861 in __init__.py (class BSTR.__ctypes_from_outparam_) might be to blame?
I commented this line out locally on my system and I'm no longer hitting the exception.
Could this be the bug?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
No. Your code will probably work better when you make this patch, but BSTR strings received as [out] parameters will no longer be freed so there will be memory leaks.
> Thomas, I'd be happy to provide more information -- can you tell me where to begin?
Best would be if you can give me the code and the COM object that shows the problem so that I can reproduce and debug it on my own machine.
If this is not possible, you should at least provide a code snippet that shows a little context around your method call, the IDL definition (for the method that you call) or the type library for the COM object. Plus any other information that might be useful.
Something like that...
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Also, Thomas, could you please explain to me why it is necessary to call _free(self) twice inside of the BSTR class? (see below)
class BSTR(_SimpleCData):
"The windows BSTR data type"
_type_ = "X"
def __repr__(self):
return "%s(%r)" % (self.__class__.__name__, self.value)
def __ctypes_from_outparam__(self, _free=windll.oleaut32.SysFreeString):
result = self.value
_free(self) # <--- here
return result
def __del__(self, _free=windll.oleaut32.SysFreeString):
"""If we own the memory, call SysFreeString to free it."""
if not self._b_base_:
_free(self) # <--- and here
Thanks,
--Mark
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- The _free call in the __del__ method only occurs when you create a standalone BSTR instance,
_b_base_ is the container object that contains the instance and it is None in this case.
- The _free call in __ctypes_from_outparam__ occurs when an output parameter value is retrieved
from a POINTER(BSTR) instance, as in the COM method call you describe.
Can you instrument the code to check if _free is really called twice on the same BSTR object?
Maybe by setting a _been_freed instance variable in both places where _free(self) is called.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Okay, this may throw you for a loop:
1. I've made some changes in my client code around the method call -- but not the method call itself -- and now I'm seeing the exception every time (the first time)
2. After several iterations, I ended up instrumenting the code in this way (why I ended up here should be apparent after reading below):
class BSTR(_SimpleCData):
"The windows BSTR data type"
_type_ = "X"
def __repr__(self):
return "%s(%r)" % (self.__class__.__name__, self.value)
def __ctypes_from_outparam__(self, _free=windll.oleaut32.SysFreeString):
result = self.value
print hex(id(self)), "__ctypes_from_outparam__", len(str(self.value))
_free(self)
return result
def __del__(self, _free=windll.oleaut32.SysFreeString):
"""If we own the memory, call SysFreeString to free it."""
if not self._b_base_:
print hex(id(self)), "__del__", len(str(self.value))
_free(self)
3. The attached traces show that it doesn't seem to matter that _free is called twice per BSTR instance... UNTIL you get a very large BSTR. I was able to control the size of the BSTR that is returned by the method call and narrowed it down to the 16k size threshold
--> BSTR > 16k causes the exception
--> BSTR < 16k avoids the exception
(I wasn't able to hit it exactly at 16384)
4. Where does this leave us? :D
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I think this has something to do with caching of BSTRs that windows does by default. Maybe strings larger than 16kB are cached in a different way or not at all.
I admit that there is a problem with comtypes.
Setting the environment variable 'set OANOCACHE=1' (which will disable BSTR caching) and then running the testsuite also crashes.
This 'hack' fixes the crashes (but the case needs to be investigated more deeply):
Index: comtypes/__init__.py
--- comtypes/__init__.py (revision 393)
+++ comtypes/__init__.py (working copy)
@@ -856,18 +856,23 @@
class BSTR(_SimpleCData):
"The windows BSTR data type"
_type_ = "X"
+ _freed = False
def __repr__(self):
return "%s(%r)" % (self.__class__.__name__, self.value)
def __ctypes_from_outparam__(self, _free=windll.oleaut32.SysFreeString):
result = self.value
- _free(self)
+ if not self._freed:
+ _free(self)
+ self._freed = True
return result
def __del__(self, _free=windll.oleaut32.SysFreeString):
"""If we own the memory, call SysFreeString to free it."""
if not self._b_base_:
- _free(self)
+ if not self._freed:
+ _free(self)
+ self._freed = True
def from_param(cls, value):
"""Convert into a foreign function call parameter."""
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Sorry, my mistake -- I didn't apply the patch below properly -- it is working now. I still don't understand why the _free() call must be invoked inside of __ctypes_from_outparam__() instead of just waiting for the __del__() to do the same thing?
Is there a 0.5.2 release planned?
Thanks,
--Mark
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
> I still don't understand why the _free() call must be invoked
> inside of __ctypes_from_outparam__() instead of just waiting for the
> __del__() to do the same thing?
Good question. There is probably no reason and this code is an artifact from the past. I have now changed the code (SVN revision 401) so that double-frees are safely avoided. Can you please test and report back?
Here is the complete changed BSTR class:
<snip>
class BSTR(_SimpleCData):
"The windows BSTR data type"
_type_ = "X"
_needsfree = False
def __repr__(self):
return "%s(%r)" % (self.__class__.__name__, self.value)
def __del__(self, _free=windll.oleaut32.SysFreeString):
# Free the string if self owns the memory
# or if instructed by __ctypes_from_outparam__.
if self._b_base_ is None \
or self._needsfree:
_free(self)
def from_param(cls, value):
"""Convert into a foreign function call parameter."""
if isinstance(value, cls):
return value
# Although the builtin SimpleCData.from_param call does the
# right thing, it doesn't ensure that SysFreeString is called
# on destruction.
return cls(value)
from_param = classmethod(from_param)
<snip>
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Logged In: YES
user_id=1375527
Originator: YES
File Added: run2.log
Logged In: YES
user_id=11105
Originator: NO
Mark, I'm afraid there is not enough information in the traces you have posted, but maybe I'm only too lazy to stare at them for a long time.
It looks like that the freeing of some resources (safearrays? or strings contained in safearrays?) is causing the problem - I am not at all sure that the safearray code is bug free.
Can you in any way make more information available?
Logged In: YES
user_id=1375527
Originator: YES
Thomas, I'd be happy to provide more information -- can you tell me where to begin?
Logged In: YES
user_id=1375527
Originator: YES
I've been going over the pdb trace and it occurred to me that perhaps the call to _free(self) on line 861 in __init__.py (class BSTR.__ctypes_from_outparam_) might be to blame?
I commented this line out locally on my system and I'm no longer hitting the exception.
Could this be the bug?
Logged In: YES
user_id=11105
Originator: NO
> Could this be the bug?
No. Your code will probably work better when you make this patch, but BSTR strings received as [out] parameters will no longer be freed so there will be memory leaks.
> Thomas, I'd be happy to provide more information -- can you tell me where to begin?
Best would be if you can give me the code and the COM object that shows the problem so that I can reproduce and debug it on my own machine.
If this is not possible, you should at least provide a code snippet that shows a little context around your method call, the IDL definition (for the method that you call) or the type library for the COM object. Plus any other information that might be useful.
Something like that...
Logged In: YES
user_id=1375527
Originator: YES
Here is the comtypes generated Python code for the method I'm calling:
COMMETHOD([dispid(16), helpstring(u'method ReadState')], HRESULT, 'ReadState',
( ['in'], c_ulong, 'c_ulDeviceIndex' ),
( ['in'], BSTR, 'c_bstrStateSpec' ),
( ['out'], POINTER(BSTR), 'pbstrBitfieldFormat' ),
( ['out'], POINTER(VARIANT), 'pvData' )),
Also, Thomas, could you please explain to me why it is necessary to call _free(self) twice inside of the BSTR class? (see below)
class BSTR(_SimpleCData):
"The windows BSTR data type"
_type_ = "X"
def __repr__(self):
return "%s(%r)" % (self.__class__.__name__, self.value)
def __ctypes_from_outparam__(self, _free=windll.oleaut32.SysFreeString):
result = self.value
_free(self) # <--- here
return result
def __del__(self, _free=windll.oleaut32.SysFreeString):
"""If we own the memory, call SysFreeString to free it."""
if not self._b_base_:
_free(self) # <--- and here
Thanks,
--Mark
Logged In: YES
user_id=11105
Originator: NO
The intent of the free calls is this:
- The _free call in the __del__ method only occurs when you create a standalone BSTR instance,
_b_base_ is the container object that contains the instance and it is None in this case.
- The _free call in __ctypes_from_outparam__ occurs when an output parameter value is retrieved
from a POINTER(BSTR) instance, as in the COM method call you describe.
Can you instrument the code to check if _free is really called twice on the same BSTR object?
Maybe by setting a _been_freed instance variable in both places where _free(self) is called.
Logged In: YES
user_id=1375527
Originator: YES
Okay, this may throw you for a loop:
1. I've made some changes in my client code around the method call -- but not the method call itself -- and now I'm seeing the exception every time (the first time)
2. After several iterations, I ended up instrumenting the code in this way (why I ended up here should be apparent after reading below):
class BSTR(_SimpleCData):
"The windows BSTR data type"
_type_ = "X"
def __repr__(self):
return "%s(%r)" % (self.__class__.__name__, self.value)
def __ctypes_from_outparam__(self, _free=windll.oleaut32.SysFreeString):
result = self.value
print hex(id(self)), "__ctypes_from_outparam__", len(str(self.value))
_free(self)
return result
def __del__(self, _free=windll.oleaut32.SysFreeString):
"""If we own the memory, call SysFreeString to free it."""
if not self._b_base_:
print hex(id(self)), "__del__", len(str(self.value))
_free(self)
3. The attached traces show that it doesn't seem to matter that _free is called twice per BSTR instance... UNTIL you get a very large BSTR. I was able to control the size of the BSTR that is returned by the method call and narrowed it down to the 16k size threshold
--> BSTR > 16k causes the exception
--> BSTR < 16k avoids the exception
(I wasn't able to hit it exactly at 16384)
4. Where does this leave us? :D
Logged In: YES
user_id=1375527
Originator: YES
File Added: instrumented_fail.log
Logged In: YES
user_id=1375527
Originator: YES
File Added: instrumented_pass.log
Logged In: YES
user_id=1375527
Originator: YES
Okay, this has got to be a bug in comtypes -- win32com is able to handle a BSTR > 16k
Logged In: YES
user_id=11105
Originator: NO
I think this has something to do with caching of BSTRs that windows does by default. Maybe strings larger than 16kB are cached in a different way or not at all.
I admit that there is a problem with comtypes.
Setting the environment variable 'set OANOCACHE=1' (which will disable BSTR caching) and then running the testsuite also crashes.
This 'hack' fixes the crashes (but the case needs to be investigated more deeply):
Index: comtypes/__init__.py
--- comtypes/__init__.py (revision 393)
+++ comtypes/__init__.py (working copy)
@@ -856,18 +856,23 @@
class BSTR(_SimpleCData):
"The windows BSTR data type"
_type_ = "X"
+ _freed = False
def __repr__(self):
return "%s(%r)" % (self.__class__.__name__, self.value)
def __ctypes_from_outparam__(self, _free=windll.oleaut32.SysFreeString):
result = self.value
- _free(self)
+ if not self._freed:
+ _free(self)
+ self._freed = True
return result
def __del__(self, _free=windll.oleaut32.SysFreeString):
"""If we own the memory, call SysFreeString to free it."""
if not self._b_base_:
- _free(self)
+ if not self._freed:
+ _free(self)
+ self._freed = True
def from_param(cls, value):
"""Convert into a foreign function call parameter."""
Logged In: YES
user_id=1375527
Originator: YES
I applied the change below, but I'm still getting the WindowsError: exception -- just without the Python window crashing.
File "C:\Python25\lib\site-packages\comtypes\__init__.py", line 864, in __ctyp
es_from_outparam__
_free(self)
WindowsError: exception: access violation reading 0x002AAC48
Logged In: NO
Sorry, my mistake -- I didn't apply the patch below properly -- it is working now. I still don't understand why the _free() call must be invoked inside of __ctypes_from_outparam__() instead of just waiting for the __del__() to do the same thing?
Is there a 0.5.2 release planned?
Thanks,
--Mark
Logged In: YES
user_id=11105
Originator: NO
> Is there a 0.5.2 release planned?
Sure, shouldn't take too long.
> I still don't understand why the _free() call must be invoked
> inside of __ctypes_from_outparam__() instead of just waiting for the
> __del__() to do the same thing?
Good question. There is probably no reason and this code is an artifact from the past. I have now changed the code (SVN revision 401) so that double-frees are safely avoided. Can you please test and report back?
Here is the complete changed BSTR class:
<snip>
class BSTR(_SimpleCData):
"The windows BSTR data type"
_type_ = "X"
_needsfree = False
def __repr__(self):
return "%s(%r)" % (self.__class__.__name__, self.value)
def __ctypes_from_outparam__(self):
self._needsfree = True
return self.value
def __del__(self, _free=windll.oleaut32.SysFreeString):
# Free the string if self owns the memory
# or if instructed by __ctypes_from_outparam__.
if self._b_base_ is None \ or self._needsfree:
_free(self)
def from_param(cls, value):
"""Convert into a foreign function call parameter."""
if isinstance(value, cls):
return value
# Although the builtin SimpleCData.from_param call does the
# right thing, it doesn't ensure that SysFreeString is called
# on destruction.
return cls(value)
from_param = classmethod(from_param)
<snip>
Logged In: YES
user_id=1375527
Originator: YES
This BSTR implementation is working as well.
Logged In: YES
user_id=11105
Originator: NO
Thanks for the confirmation.