Re: check-tests-parallel: deadlocks

[Vladimir T. <vtz...@gm...>]

Hi Don,
On Nov 27, 2008, at 11:44 AM, Don Cohen wrote:
>
> Am I correct that the plan is that gc will be able to proceed without
> indefinite delay in all cases?

yes - correct.

> The paragraph below indicates that this is not currently the case:
>
>> Currently the GC can suspend threads only in two cases. When thread
>> allocates heap object or when the thread is in (possibly) blocking
>> system (foreign) call. For the latter - all such possibly blocking
>> foreign calls should be found and "marked" (currently - streams and
>> file system operations are mostly marked).
>
> But the current case may not be (I hope is not) the goal.
> Is the goal, as I hope?  That is,
> (1) all lisp code will be guaranteed to check within a short time
> (2) foreign code need not be stopped for GC to run
> Which implies that GC will never be delayed very long.

The current state is just first approach for the goal. There are few  
places (which I know) that require adding checks for GC suspends  
requests.

>>
>> As I wrote above the foreign calls that may block should be
>> "marked" (enclosed in GC safe regions). If pointer in the heap is
>> passed to foreign code - the heap object should be "pinned" - so GC
>> (if happens) is not going to move it.
>
> This seems to be imposing restrictions on code that can block, whereas
> I want to instead impose restrictions on sections of code that access
> data that is accessed by GC.
>
> It seems fair to impose requirements on foreign code.
> I suggest the requirements be:
> (1) all access to GC sensitive data must be done inside a marked  
> section
how does a foreign code knows it is accessing clisp heap (and why  
care)? (probably I do not understand what you mean)
> (2) all marked sections are required to last a short time
>     (in particular, if you want GC not to be delayed more than 1 sec
>      you must ensure that your marked sections last less than 1 sec)
>     Perhaps this is related to your remark about blocking?  You should
>     not block in a marked section since it then might violate this  
> rule.
> (3) of course you must expect that a gc might occur between your
>     marked sections - so you can't save any pointers to GC sensitive
>     data from one marked section to the next.  We have to specify what
>     this GC sensitive data is.  I presume at least the heap, but also
>
>     some data on the stack and in registers?
> In particular, any foreign code that does not access the heap should
> be able to run in parallel with GC.

I do not understand all of the above but here is what we have and why.
Foreign calls that may block do not stop GC of occurring (even if  
they access heap).
These are (at least) all file operations and sockets. We have (use)  
two options for parameters/results:
(1) - never pass pointer to heap object to foreign call - so  
parameters/results should be copied explicitly (and memory for them  
allocated of course).
(2) - introduce pinned objects - so they stay at the same place in  
the heap during the blocking foreign call.

The first option is expensive for large objects - especially with  
"bulk i/o" so we use the second - pinned objects - in most of the I/O  
operation.
Before calling into foreign call - the object is pinned and after the  
return - it is unpinned. Meanwhile GC is not blocked by the calling  
thread (the thread make itself appear safe for gc until it is in the  
foreign call).

Vladimir