Re: some ideas for multithreading support

[<Joe...@t-...>]

Hi,



Bruno proposed:

 LISPFUNNR(car,1)

  {

    var object argument = popSTACK();

    RDLOCK(object);

    var object result = car(argument);

    RDUNLOCK(object);

    VALUES1(result);

  }



That is completely unsatisfying. As Don noticed, all you have achieved

here, at the cost of *lots* of changes to the source, is atomic access

on machines where it is not already guaranteed by the HW.



For instance, the WIDE_SOFT configuration uses 64 bit objects on 32

bit architectures. There locking will prevent random crashes caused by

observing non-atomic half-object updates.

I'd simply state that WIDE_SOFT is incompatible with ENABLE_THREADS.





Your proposal is unsatisfying because merely enabling atomic updates does

not help at all towards application-level consistency. E.g. suppose my app

uses two arrays in sync (instead of putting a cons of two values into each

element). Or suppose my app builds some tree of things.  Lack of

synchronization during updates may cause thread B to observe a violation

of invariants, e.g. both arrays not of the same size, or a cyclic graph instead

of a tree.



So the above locks gain you too little.



>Here RDLOCK will never wait: it will either signal an error or

>proceed.

This sounds crazy. Consider the following sequence:

(My example is stupid because you certainly proposed to use a

shared-exclusive lock. But just replace CAR with (SETF CAR).)



  A                                  B  var object argument = popSTACK();

  var object argument = popSTACK();     RDLOCK(object);

  [delayed]                             var object result = car(argument);

                                        RDUNLOCK(object);

  RDLOCK(object);

  var object result = car(argument);

  RDUNLOCK(object);

  VALUES1(result);                      VALUES1(result);



No error would occur, the inherent data race would go unnoticed.  But an

error would be signaled when A resumes just a little bit earlier?

That randomness makes no sense.





What I'm missing is a vision of something that would make concurrent

processing in clisp interesting.  So far, all I see is an attempt to play

catch up where clisp is the last in class, as all other implementations

already feature threads AFAIK.



I would enjoy other means of (real or pseudo-)concurrency in clisp. Clojure

is known for having transactional memory (I never tried it out), Python and

several others have coroutines or some limited form of delimited continuations.



I could imagine a single threaded clisp yet featuring asynchronous I/O based

on libuv.  Unlike other fashionable things like comprehensions which can be

conveniently expressed in pure Common Lisp, async stuff would need support

from the C core of clisp.



Back since Amiga days I dream of a clisp with something similar to CMUCL's

serve-event. But Bruno has always favoured real threads over green threads.

https://common-lisp.net/project/cmucl/doc/cmu-user/serve-event.html





Regarding concurrent hash-tables, I'd relax the requirements about iteration:

- Iteration must terminate.

- The set of keys need not match the serializable database isolation level.

- It's OK for some keys to be missing from the enumeration! That's the price

  an application pays for going through a table without adequate outer locking.

- No key shall be presented twice(?) (except if it's removed + re-added meanwhile)

- Iteration must not crash.



IOW, it's OK during iteration to present keys that have been removed

concurrently. There's no need to implement snapshots. But the data

structures should never be seen in such an inconsistent state that the

iteration loops infinitely or crashes.



Likewise, when you intern a symbol-name into a package, concurrent

enumeration (and lookup) should remain stable. That seems achievable using

simply linked bucket lists and a few memory barriers & atomic instructions.



As you see, I favour a model easily understood of serializable atomic reads

from and writes to memory.  Alas, that's not how modern processors

evolve. They are moving in a direction of isolated memory, where two threads

must explicitly communicate (rendez-vous) with each other (atomic writes are

*not* enough) in order to "exchange" memory.

IOW they behave more & more like processes.

https://www.kernel.org/pub/linux/kernel/people/paulmck/perfbook/perfbook.html



Regards,

                Jörg