[Pytables-users] threading/GIL

[Andrew S. <str...@as...>]

Dear Francesc (and others),

I enclose a small patch (against CVS HEAD) which demonstrates how to 
release the GIL during potentially long operations.  I think all 
potentially long operations (basically most of the C H5* functions) 
should be bracketed by these statements.

For example purposes, I've only tracked down this single operation.  If 
you'd like any more explanation/justification/etc., please let me know 
and I'll do my best.

It's worth checking the Pyrex-outputted C file to ensure that there are 
no calls to the C Python API between the BEGIN_ALLOW_THREADS and the 
END_ALLOW_THREADS.  I did this for this example.  Other cases may 
require that the C return value (e.g. an integer) is stored in an 
intermediate variable (as in this case) so that the GIL can be acquired 
again before raising a Python exception upon error return from C.

Hoping this (and similar for other potentially long operations) will 
make it into PyTables,
Andrew

RCS file: /cvsroot/pytables/pytables/src/hdf5Extension.pyx,v
retrieving revision 1.150
diff -c -r1.150 hdf5Extension.pyx
*** src/hdf5Extension.pyx       9 Dec 2004 13:01:58 -0000       1.150
--- src/hdf5Extension.pyx       15 Dec 2004 02:35:44 -0000
***************
*** 116,121 ****
--- 116,125 ----
    char *PyString_AsString(object string)
    object PyString_FromString(char *)

+   # To release global interpreter lock (GIL) for threading
+   void Py_BEGIN_ALLOW_THREADS()
+   void Py_END_ALLOW_THREADS()
+
  # To access to str and tuple structures. This does not work with Pyrex 0.8
  # This is not necessary, though
  #  ctypedef class __builtin__.str [object PyStringObject]:
***************
*** 1658,1666 ****
--- 1662,1677 ----
      if not self._open:
        self._open_append(recarr)

+     # release GIL (allow other threads to use the Python interpreter)
+     Py_BEGIN_ALLOW_THREADS
+
      # Append the records:
      ret = H5TBOappend_records(&self.dataset_id, &self.mem_type_id,
                                nrecords, self.totalrecords, self.rbuf)
+
+     # acquire GIL (disallow other threads from using the Python 
interpreter)
+     Py_END_ALLOW_THREADS
+
      if ret < 0:
        raise RuntimeError("Problems appending the records.")