From: <cli...@li...> - 2009-06-26 12:04:25
|
Send clisp-cvs mailing list submissions to cli...@li... To subscribe or unsubscribe via the World Wide Web, visit https://lists.sourceforge.net/lists/listinfo/clisp-cvs or, via email, send a message with subject or body 'help' to cli...@li... You can reach the person managing the list at cli...@li... When replying, please edit your Subject line so it is more specific than "Re: Contents of clisp-cvs digest..." CLISP CVS commits for today Today's Topics: 1. clisp/src ChangeLog,1.6919,1.6920 pathname.d,1.498,1.499 (Sam Steingold) 2. clisp/src ChangeLog, 1.6920, 1.6921 xthread.d, 1.29, 1.30 zthread.d, 1.58, 1.59 (Sam Steingold) 3. clisp/tests hashlong.tst,1.14,1.15 (Sam Steingold) 4. clisp/src ChangeLog,1.6921,1.6922 foreign1.lisp,1.133,1.134 (Sam Steingold) 5. clisp/src ChangeLog,1.6922,1.6923 zthread.d,1.59,1.60 (Vladimir Tzankov) 6. clisp/tests ChangeLog,1.628,1.629 mt.tst,1.12,1.13 (Vladimir Tzankov) ---------------------------------------------------------------------- Message: 1 Date: Thu, 25 Jun 2009 18:54:06 +0000 From: Sam Steingold <sd...@us...> Subject: clisp/src ChangeLog,1.6919,1.6920 pathname.d,1.498,1.499 To: cli...@li... Message-ID: <E1M...@dd...> Update of /cvsroot/clisp/clisp/src In directory ddv4jf1.ch3.sourceforge.com:/tmp/cvs-serv16849/src Modified Files: ChangeLog pathname.d Log Message: (directory_search_scandir) [MULTITHREAD]: avoid warning: empty body in an if-statement Index: pathname.d =================================================================== RCS file: /cvsroot/clisp/clisp/src/pathname.d,v retrieving revision 1.498 retrieving revision 1.499 diff -u -d -r1.498 -r1.499 --- pathname.d 30 Apr 2009 21:24:35 -0000 1.498 +++ pathname.d 25 Jun 2009 18:54:04 -0000 1.499 @@ -7312,7 +7312,7 @@ var int rdr; /* readdir_r() return*/ /* fetch next directory-entry */ GC_SAFE_SYSTEM_CALL(rdr=, readdir_r(dirp,(struct dirent *)dp_buf,&dp)); - if (dp == (struct dirent *)NULL) FREE_DYNAMIC_ARRAY(dp_buf); + if (dp == (struct dirent *)NULL) { FREE_DYNAMIC_ARRAY(dp_buf); } errno = rdr; #else var struct dirent * dp; Index: ChangeLog =================================================================== RCS file: /cvsroot/clisp/clisp/src/ChangeLog,v retrieving revision 1.6919 retrieving revision 1.6920 diff -u -d -r1.6919 -r1.6920 --- ChangeLog 25 Jun 2009 18:53:13 -0000 1.6919 +++ ChangeLog 25 Jun 2009 18:54:04 -0000 1.6920 @@ -1,5 +1,10 @@ 2009-06-25 Sam Steingold <sd...@gn...> + * pathname.d (directory_search_scandir) [MULTITHREAD]: avoid + warning: empty body in an if-statement + +2009-06-25 Sam Steingold <sd...@gn...> + * lispbibl.d (current_thread_alloccount, gcunsafe_object_t) (tsd_remove_specific, lock_threads, unlock_threads) (signal_timeout_call, handle_pending_interrupts): ------------------------------ Message: 2 Date: Thu, 25 Jun 2009 19:24:43 +0000 From: Sam Steingold <sd...@us...> Subject: clisp/src ChangeLog, 1.6920, 1.6921 xthread.d, 1.29, 1.30 zthread.d, 1.58, 1.59 To: cli...@li... Message-ID: <E1M...@dd...> Update of /cvsroot/clisp/clisp/src In directory ddv4jf1.ch3.sourceforge.com:/tmp/cvs-serv26142/src Modified Files: ChangeLog xthread.d zthread.d Log Message: * xthread.d (xlock_t): add _owned filed because there are no guaranteed invalid xthread_t values * zthread.d (xlock_init): set _owned to false; do not init _owner (xlock_lock_helper): check _owned instead of _owner; init both (xlock_unlock_helper): set _owned to false; do not touch _owner Index: zthread.d =================================================================== RCS file: /cvsroot/clisp/clisp/src/zthread.d,v retrieving revision 1.58 retrieving revision 1.59 diff -u -d -r1.58 -r1.59 --- zthread.d 25 Jun 2009 18:40:53 -0000 1.58 +++ zthread.d 25 Jun 2009 19:24:40 -0000 1.59 @@ -861,7 +861,7 @@ xmutex_raw_destroy(&l->_mr); return r; } - l->_owner = NULL; /* hmmm */ + l->_owned = false; l->_count = 0; return 0; } @@ -926,14 +926,16 @@ if (r != 0) break; } if (r == 0) { - ASSERT(l->_owner == NULL); + ASSERT(!l->_owned); l->_owner = xthread_self(); + l->_owned = true; l->_count=1; } } else { /* if is not real lock - we own the the real mutex + guarding one */ - ASSERT(!l->_owner); + ASSERT(!l->_owned); l->_owner = xthread_self(); + l->_owned = true; l->_count=1; } xmutex_raw_unlock(&l->_m); @@ -957,7 +959,7 @@ xmutex_raw_lock(&l->_m); if (unlock_real) xmutex_raw_unlock(&l->_mr); - l->_owner = NULL; /* hmm */ + l->_owned = false; xmutex_raw_unlock(&l->_m); /* before signal */ xcondition_signal(&l->_c); } Index: ChangeLog =================================================================== RCS file: /cvsroot/clisp/clisp/src/ChangeLog,v retrieving revision 1.6920 retrieving revision 1.6921 diff -u -d -r1.6920 -r1.6921 --- ChangeLog 25 Jun 2009 18:54:04 -0000 1.6920 +++ ChangeLog 25 Jun 2009 19:24:40 -0000 1.6921 @@ -1,5 +1,13 @@ 2009-06-25 Sam Steingold <sd...@gn...> + * xthread.d (xlock_t): add _owned filed because there are no + guaranteed invalid xthread_t values + * zthread.d (xlock_init): set _owned to false; do not init _owner + (xlock_lock_helper): check _owned instead of _owner; init both + (xlock_unlock_helper): set _owned to false; do not touch _owner + +2009-06-25 Sam Steingold <sd...@gn...> + * pathname.d (directory_search_scandir) [MULTITHREAD]: avoid warning: empty body in an if-statement Index: xthread.d =================================================================== RCS file: /cvsroot/clisp/clisp/src/xthread.d,v retrieving revision 1.29 retrieving revision 1.30 diff -u -d -r1.29 -r1.30 --- xthread.d 25 Jun 2009 18:49:34 -0000 1.29 +++ xthread.d 25 Jun 2009 19:24:40 -0000 1.30 @@ -224,6 +224,7 @@ xcondition_t _c; /* condition to wait on */ xmutex_raw_t _mr; /* real mutex */ xthread_t _owner; /* who owns the lock */ + bool _owned; /* _owner initialized? no known invalid xthread_t values! */ int _count; /* how many times we own the object */ } xlock_t; ------------------------------ Message: 3 Date: Thu, 25 Jun 2009 20:31:39 +0000 From: Sam Steingold <sd...@us...> Subject: clisp/tests hashlong.tst,1.14,1.15 To: cli...@li... Message-ID: <E1M...@dd...> Update of /cvsroot/clisp/clisp/tests In directory ddv4jf1.ch3.sourceforge.com:/tmp/cvs-serv17045 Modified Files: hashlong.tst Log Message: add symbol-cleanup forms Index: hashlong.tst =================================================================== RCS file: /cvsroot/clisp/clisp/tests/hashlong.tst,v retrieving revision 1.14 retrieving revision 1.15 diff -u -d -r1.14 -r1.15 --- hashlong.tst 30 Sep 2007 05:32:43 -0000 1.14 +++ hashlong.tst 25 Jun 2009 20:31:37 -0000 1.15 @@ -120,3 +120,12 @@ (= (sxhash '(1 (2 . 3) 4 (5 . 6) (7 (8)))) (sxhash '(1 (2 . 3) 4 (5 . 6) (8 (7))))) NIL + +(progn (symbol-cleanup 'symbole) + (symbol-cleanup 'hash-table-keys) + (symbol-cleanup 'check-hash-unique-vec) + (symbol-cleanup 'do-hash-test) + (symbol-cleanup 'setf-gethash) + (symbol-cleanup 'ht1) + (symbol-cleanup 'ht2)) +T ------------------------------ Message: 4 Date: Fri, 26 Jun 2009 04:02:20 +0000 From: Sam Steingold <sd...@us...> Subject: clisp/src ChangeLog,1.6921,1.6922 foreign1.lisp,1.133,1.134 To: cli...@li... Message-ID: <E1M...@dd...> Update of /cvsroot/clisp/clisp/src In directory ddv4jf1.ch3.sourceforge.com:/tmp/cvs-serv10136/src Modified Files: ChangeLog foreign1.lisp Log Message: (finalize-coutput-file): avoid warning: deprecated conversion from string constant to 'char*' in the MODULE__name__CONSTANT_MAP_C_STRING function Index: foreign1.lisp =================================================================== RCS file: /cvsroot/clisp/clisp/src/foreign1.lisp,v retrieving revision 1.133 retrieving revision 1.134 diff -u -d -r1.133 -r1.134 --- foreign1.lisp 19 Jun 2009 15:10:34 -0000 1.133 +++ foreign1.lisp 26 Jun 2009 04:02:17 -0000 1.134 @@ -722,6 +722,10 @@ (maphash (lambda (type fun-vec) (let* ((fun (first fun-vec)) (vec (second fun-vec)) (c-decl (to-c-typedecl type fun))) + (when (eq type 'c-string) + ;; avoid warning: + ;; deprecated conversion from string constant to 'char*' + (setq c-decl (string-concat "const " c-decl))) (format *coutput-stream* "~A (int number, int *definedp);~%~ ~A (int number, int *definedp) { *definedp=1;~% switch (number) {~%" Index: ChangeLog =================================================================== RCS file: /cvsroot/clisp/clisp/src/ChangeLog,v retrieving revision 1.6921 retrieving revision 1.6922 diff -u -d -r1.6921 -r1.6922 --- ChangeLog 25 Jun 2009 19:24:40 -0000 1.6921 +++ ChangeLog 26 Jun 2009 04:02:17 -0000 1.6922 @@ -1,3 +1,9 @@ +2009-06-26 Sam Steingold <sd...@gn...> + + * foreign1.lisp (finalize-coutput-file): avoid + warning: deprecated conversion from string constant to 'char*' + in the MODULE__name__CONSTANT_MAP_C_STRING function + 2009-06-25 Sam Steingold <sd...@gn...> * xthread.d (xlock_t): add _owned filed because there are no ------------------------------ Message: 5 Date: Fri, 26 Jun 2009 10:44:23 +0000 From: Vladimir Tzankov <vt...@us...> Subject: clisp/src ChangeLog,1.6922,1.6923 zthread.d,1.59,1.60 To: cli...@li... Message-ID: <E1M...@dd...> Update of /cvsroot/clisp/clisp/src In directory ddv4jf1.ch3.sourceforge.com:/tmp/cvs-serv26011/src Modified Files: ChangeLog zthread.d Log Message: [MULTITHREAD]: fix xlock_t (MUTEX) unlocking Index: zthread.d =================================================================== RCS file: /cvsroot/clisp/clisp/src/zthread.d,v retrieving revision 1.59 retrieving revision 1.60 diff -u -d -r1.59 -r1.60 --- zthread.d 25 Jun 2009 19:24:40 -0000 1.59 +++ zthread.d 26 Jun 2009 10:44:21 -0000 1.60 @@ -129,8 +129,8 @@ } /* release the mutex */ TheMutex(STACK_0)->xmu_recurse_count = 0; - GC_SAFE_MUTEX_UNLOCK(TheMutex(STACK_0)->xmu_system); TheMutex(STACK_0)->xmu_owner = NIL; + GC_SAFE_MUTEX_UNLOCK(TheMutex(STACK_0)->xmu_system); skipSTACK(1); /* mutex */ }); } @@ -656,8 +656,9 @@ GC_SAFE_REGION_END_WITHOUT_INTERRUPTS(); end_system_call(); if (!res) { /* if we got the mutex */ - TheMutex(*mxrec)->xmu_owner = current_thread()->_lthread; ASSERT(TheMutex(*mxrec)->xmu_recurse_count == 0); + ASSERT(eq(TheMutex(*mxrec)->xmu_owner,NIL)); + TheMutex(*mxrec)->xmu_owner = thr->_lthread; TheMutex(*mxrec)->xmu_recurse_count++; } /* now the mutex record is in consistent state - handle pending @@ -926,7 +927,7 @@ if (r != 0) break; } if (r == 0) { - ASSERT(!l->_owned); + ASSERT(!l->_owned); ASSERT(l->_count == 0); l->_owner = xthread_self(); l->_owned = true; l->_count=1; @@ -957,9 +958,10 @@ if (!--l->_count) { /* we will never wait here (at least not for a long time) */ xmutex_raw_lock(&l->_m); + l->_owned = false; + l->_owner = NULL; if (unlock_real) xmutex_raw_unlock(&l->_mr); - l->_owned = false; xmutex_raw_unlock(&l->_m); /* before signal */ xcondition_signal(&l->_c); } Index: ChangeLog =================================================================== RCS file: /cvsroot/clisp/clisp/src/ChangeLog,v retrieving revision 1.6922 retrieving revision 1.6923 diff -u -d -r1.6922 -r1.6923 --- ChangeLog 26 Jun 2009 04:02:17 -0000 1.6922 +++ ChangeLog 26 Jun 2009 10:44:21 -0000 1.6923 @@ -1,3 +1,11 @@ +2009-06-26 Vladimir Tzankov <vtz...@gm...> + + [MULTITHREAD]: fix xlock_t (MUTEX) unlocking + * zthread.d (xlock_unlock_helper): set xlock_t _owner to NULL + (xlock_lock_helper): add more asserts + (MUTEX-LOCK): ditto + (thread_cleanup): clear mutex owner before really unlocking it + 2009-06-26 Sam Steingold <sd...@gn...> * foreign1.lisp (finalize-coutput-file): avoid ------------------------------ Message: 6 Date: Fri, 26 Jun 2009 10:49:41 +0000 From: Vladimir Tzankov <vt...@us...> Subject: clisp/tests ChangeLog,1.628,1.629 mt.tst,1.12,1.13 To: cli...@li... Message-ID: <E1M...@dd...> Update of /cvsroot/clisp/clisp/tests In directory ddv4jf1.ch3.sourceforge.com:/tmp/cvs-serv26817 Modified Files: ChangeLog mt.tst Log Message: mt.tst: fix tests that spawn more than 128 threads (MAXNTHREADS from spvw.d) Index: mt.tst =================================================================== RCS file: /cvsroot/clisp/clisp/tests/mt.tst,v retrieving revision 1.12 retrieving revision 1.13 diff -u -d -r1.12 -r1.13 --- mt.tst 25 Jun 2009 18:38:02 -0000 1.12 +++ mt.tst 26 Jun 2009 10:49:39 -0000 1.13 @@ -112,13 +112,22 @@ (sleep 0.5) NIL (thread-active-p *th1*) NIL ;; should be dead +;; There are max 128 concurrent threads possible (MAXNTHREADS in spvw.d). +;; If the limit is reached MAKE-THREAD will return NIL. So for the tests +;; below spawn more threads and we need following function. +(defun make-thread-always (function) + (do ((thr (mt:make-thread function) (mt:make-thread function))) + ((threadp thr) thr) + (thread-yield))) +make-thread-always + (let* ((mu (make-mutex :name "hash-table lock")) (ht (make-hash-table)) (tl (loop :repeat 1000 :collect - (make-thread (lambda () - (mutex-lock mu) - (incf (gethash 1 ht 0)) - (mutex-unlock mu)))))) + (make-thread-always (lambda () + (mutex-lock mu) + (incf (gethash 1 ht 0)) + (mutex-unlock mu)))))) ;; wait for all threads to finish (loop :while (some #'thread-active-p tl) :do (sleep 0.1)) (gethash 1 ht)) @@ -129,10 +138,10 @@ (pa (make-package (symbol-name (gensym "MT-TEST-")) :use ())) (tl (loop :for i :from 1 :to count :collect (let ((i i)) - (make-thread (lambda () - (mutex-lock mu) - (intern (prin1-to-string i) pa) - (mutex-unlock mu))))))) + (make-thread-always (lambda () + (mutex-lock mu) + (intern (prin1-to-string i) pa) + (mutex-unlock mu))))))) ;; wait for all threads to finish (loop :while (some #'thread-active-p tl) :do (sleep 0.1)) (let ((i 0)) @@ -140,10 +149,11 @@ (assert (= i count))) (setq tl (loop :for i :from 1 :to count :collect (let ((i i)) - (make-thread (lambda () - (mutex-lock mu) - (unintern (find-symbol (prin1-to-string i) pa) pa) - (mutex-unlock mu)))))) + (make-thread-always + (lambda () + (mutex-lock mu) + (unintern (find-symbol (prin1-to-string i) pa) pa) + (mutex-unlock mu)))))) ;; wait for all threads to finish (loop :while (some #'thread-active-p tl) :do (sleep 0.1)) (let ((i 0)) Index: ChangeLog =================================================================== RCS file: /cvsroot/clisp/clisp/tests/ChangeLog,v retrieving revision 1.628 retrieving revision 1.629 diff -u -d -r1.628 -r1.629 --- ChangeLog 25 Jun 2009 17:35:55 -0000 1.628 +++ ChangeLog 26 Jun 2009 10:49:39 -0000 1.629 @@ -1,3 +1,8 @@ +2009-06-26 Vladimir Tzankov <vtz...@gm...> + + * mt.tst: fix tests that spawn more than 128 threads (MAXNTHREADS from + spvw.d) + 2009-06-25 Sam Steingold <sd...@gn...> * mt.tst: test interaction of threads with hash-tables and packages ------------------------------ ------------------------------------------------------------------------------ ------------------------------ _______________________________________________ clisp-cvs mailing list cli...@li... https://lists.sourceforge.net/lists/listinfo/clisp-cvs End of clisp-cvs Digest, Vol 38, Issue 37 ***************************************** |
From: Vladimir T. <vtz...@gm...> - 2009-06-26 13:12:18
|
Sam, In xlock_unlock_helper(): - l->_owner = NULL; /* hmm */ + l->_owned = false; We need a way to invalidate the _owner, since it is used to be checked whether a thread holds the lock in xlock_lock_helper(). If thread owns a lock and unlocks it and after this locks it again - the second time xlock_lock_helper() will not really acquire the mutex since the _owner will be still the same (all this if _owner is not invalidated and there is no other thread that got the lock between meanwhile. I've experience exactly this with the tests that you've added for hash table and packages). NULL - rather non-standard is widely used for this purpose (or may be we can zero-up it with memset() in order to avoid compiler warnings). Vladimir |
From: Sam S. <sd...@gn...> - 2009-06-26 14:07:42
|
Vladimir, Vladimir Tzankov wrote: > > In xlock_unlock_helper(): > - l->_owner = NULL; /* hmm */ this is just plain wrong. xthread_t is DWORD on woe32 and an opaque type on posix. on linux it's a ulong, but it is NOT guaranteed that 0 or any other value is invalid (and the presence of pthread_equal clearly indicates a lack of a guarantee that it fits into an integers). there is no a priori invalid xthread_t value. we have to work with that assumption. > + l->_owned = false; > > We need a way to invalidate the _owner, since it is used to be checked > whether a thread holds the lock in xlock_lock_helper(). that's what the new _owned field indicates. > If thread owns > a lock and unlocks it and after this locks it again - the second time > xlock_lock_helper() will not really acquire the mutex since the _owner > will be still the same (all this if _owner is not invalidated and > there is no other thread that got the lock between meanwhile. I've > experience exactly this with the tests that you've added for hash > table and packages). that's OK. my point is that before accessing the owner field, we must assure that the owned field is set. this is safe and clean. > NULL - rather non-standard is widely used for this purpose (or may be > we can zero-up it with memset() in order to avoid compiler warnings). no, there is no way to have an invalid xthread_t value. BTW, why did you decide to name the fields with a leading "_"? it would have been much nicer if the names followed the usual unix convention (cf. stat), e.g., xl_owner &c (and the names should be longer and more mnemonic and greppable). Sam. |
From: Vladimir T. <vtz...@gm...> - 2009-06-26 14:19:18
|
On 6/26/09, Sam Steingold <sd...@gn...> wrote: > Vladimir Tzankov wrote: >> If thread owns >> a lock and unlocks it and after this locks it again - the second time >> xlock_lock_helper() will not really acquire the mutex since the _owner >> will be still the same (all this if _owner is not invalidated and >> there is no other thread that got the lock between meanwhile. I've >> experience exactly this with the tests that you've added for hash >> table and packages). > > that's OK. > my point is that before accessing the owner field, we must assure that the > owned field is set. this is safe and clean. ok, got it. Will use it and commit. > > BTW, why did you decide to name the fields with a leading "_"? > it would have been much nicer if the names followed the usual unix > convention > (cf. stat), e.g., xl_owner &c (and the names should be longer and more > mnemonic > and greppable). Will change them as well (xlock_t started as experiment how to interrupt while waiting on mutex and thus shorter names remained). |
From: Sam S. <sd...@gn...> - 2009-06-26 18:54:46
|
Vladimir Tzankov wrote: > On 6/26/09, Sam Steingold <sd...@gn...> wrote: >> BTW, why did you decide to name the fields with a leading "_"? >> it would have been much nicer if the names followed the usual unix >> convention >> (cf. stat), e.g., xl_owner &c (and the names should be longer and more >> mnemonic >> and greppable). > > Will change them as well (xlock_t started as experiment how to > interrupt while waiting on mutex and thus shorter names remained). thanks. it would be nice to do the same clean-up for other structures (pinned_chain_t, clisp_thread_t). BTW, about pin_varobject -- does it have to be in the same block as unpin_varobject? (the code does not indicate that, but...) e.g., in rawsock.c, I want to call pin_varobject inside parse_buffer_arg but unpin_varobject outside). thanks. |
From: <don...@is...> - 2009-06-26 19:10:34
|
> mt.tst: fix tests that spawn more than 128 threads (MAXNTHREADS from spvw.d) This raises 2 questions: - what determines the limit on #threads, is there a way to read it? - should I be running mt tests in my nightly builds or otherwise occasionally and if so, how? I just tried make check and ended with ./foo -x "(setq zz 10) (saveinitmem \"foo\")" ... WARNING: No installation directory specified. Please try: ./foo -B /usr/local/lib/clisp 10 *** - EVAL: undefined function SAVEINITMEM Bye. ./foo -norc -M foo.mem -x zz ./foo: operating system error during load of initialization file `foo.mem' [../src/spvw_memfile.d:981] errno = ENOENT: No such file or directory. make: *** [check-exec-image] Error 1 |
From: <don...@is...> - 2009-06-27 17:29:52
|
Vladimir Tzankov writes: > On 6/26/09, Don Cohen <don...@is...> wrote: > > > mt.tst: fix tests that spawn more than 128 threads (MAXNTHREADS from > > spvw.d) > > > > This raises 2 questions: > > - what determines the limit on #threads, is there a way to read it? > > The number of concurrently running threads is hardcoded to 128. In > order to changed it the constant mentioned above should be changed and > clisp rebuilt. > I am not sure why it is in this way (when I started - it was already there). Is there an OS limit to the number of threads? Is there any reason the limit has to be compiled in at all? What are the practical reasons for limits? I guess the stack space taken by each one will add up. Anything else? What are the default sizes? Is there a way to read the size for a thread from lisp? > > - should I be running mt tests in my nightly builds or otherwise > > occasionally and if so, how? > 'make check-tests-parallel' is causing "random" problems - mainly due > to CLOS (lack of any locking) and Packages (partially ready). > 'make check' invokes check-tests-parallel and is likely to fail > (though sometimes it passes). > 'make check-tests' should always pass tests in MT builds. > > I just tried make check and ended with > > ./foo -x "(setq zz 10) (saveinitmem \"foo\")" > > ..... > > WARNING: No installation directory specified. > > Please try: ./foo -B /usr/local/lib/clisp > > 10 > > *** - EVAL: undefined function SAVEINITMEM > >..... > > make: *** [check-exec-image] Error 1 > > Cannot reproduce it (this is 'make check-exec-image' right?). > (I recall fixing similar issue some time ago). I don't see "check-exec-image" anywhere in my shell buffer other than in the line above with the "Error 1", but I guess that's what that line means. My next attempt (today, using the result of another nightly build) does not show that error, but now seems stuck at (WITH-LOCK (*MU1*) (THREAD-INTERRUPT *TH1* :FUNCTION (LAMBDA NIL (WITH-LOCK (*MU1*) (SETF *EXEMPTION-STATE* :BROADCASTED) (EXEMPTION-BROADCAST *EXEMPTION*))) :OVERRIDE T) (LOOP UNTIL (EQ *EXEMPTION-STATE* :BROADCASTED) DO (EXEMPTION-WAIT *EXEMPTION* *MU1*) FINALLY (RETURN *EXEMPTION-STATE*))) make check-tests does seem to work - I've only tried it once and it ended with this: Real time: 125.22356 sec. Run time: 92.128 sec. Space: 1298790200 Bytes GC: 1157, GC time: 17.231361 sec. 0 Bye. (echo *.erg | grep '*' >/dev/null) || (echo "Test failed:" ; ls -l *erg; echo "To see which tests failed, type" ; echo " cat "`pwd`"/*.erg" ; exit 1) echo "Test passed." Test passed. make[1]: Leaving directory `/root/clisp/build-mt/tests' |
From: Vladimir T. <vtz...@gm...> - 2009-06-27 20:38:33
|
On 6/27/09, Don Cohen <don...@is...> wrote: > Vladimir Tzankov writes: > > The number of concurrently running threads is hardcoded to 128. In > > order to changed it the constant mentioned above should be changed and > > clisp rebuilt. > > I am not sure why it is in this way (when I started - it was already > there). > Is there an OS limit to the number of threads? There is some but we should not care of it (it's much larger than 128 for sure). > Is there any reason the limit has to be compiled in at all? When I started working on it it was already there. I think it's not good to have much more concurrent threads than the number of cpu cores - so did not do anything to change it. > What are the practical reasons for limits? I guess the stack space > taken by each one will add up. Anything else? Mostly the stacks. Lisp stacks are allocated via malloc() and this may interfere with Lisp heaps when we allocate too many threads. I am experiencing this now on i386 osx - tests in mt.tst that spawn 1000 threads almost always crash strangely on it. After some debugging - it appeared that malloc() returns memory regions that overlap with lisp heap (reserved via mmap()). > What are the default sizes? Is there a way to read the size for a > thread from lisp? Each thread has two stack - C (or control) and lisp one. Both can be specified in MAKE-THREAD. Default values are MT::*DEFAULT-CONTROL-STACK-SIZE* (initialized to 1MB) and MT::*DEFAULT-VALUE-STACK-SIZE* (initialized from the command line or some reasonable default - as in single thread). The thread itself does not have a way to read its stack sizes. > My next attempt (today, using the result of another nightly build) > does not show that error, but now seems stuck at > (WITH-LOCK (*MU1*) (THREAD-INTERRUPT *TH1* :FUNCTION (LAMBDA NIL > (WITH-LOCK (*MU1*) (SETF *EXEMPTION-STATE* :BROADCASTED) > (EXEMPTION-BROADCAST *EXEMPTION*))) :OVERRIDE T) (LOOP UNTIL (EQ > *EXEMPTION-STATE* :BROADCASTED) DO (EXEMPTION-WAIT *EXEMPTION* *MU1*) > FINALLY (RETURN *EXEMPTION-STATE*))) Thanks. This should be fixed in the cvs now. |
From: <don...@is...> - 2009-06-28 06:41:03
|
Vladimir Tzankov writes: > I think it's not good to have much more concurrent threads than the > number of cpu cores - so did not do anything to change it. Perhaps you mean more concurrent threads trying to run. It makes sense to have a large number waiting for IO, for example. > specified in MAKE-THREAD. Default values are > MT::*DEFAULT-CONTROL-STACK-SIZE* (initialized to 1MB) and > MT::*DEFAULT-VALUE-STACK-SIZE* (initialized from the command line or > some reasonable default - as in single thread). 458176 for me So it seems at least plausible to have > 128 threads. > > My next attempt (today, using the result of another nightly build) > > does not show that error, but now seems stuck at > > (WITH-LOCK (*MU1*) (THREAD-INTERRUPT *TH1* :FUNCTION (LAMBDA NIL > > (WITH-LOCK (*MU1*) (SETF *EXEMPTION-STATE* :BROADCASTED) > > (EXEMPTION-BROADCAST *EXEMPTION*))) :OVERRIDE T) (LOOP UNTIL (EQ > > *EXEMPTION-STATE* :BROADCASTED) DO (EXEMPTION-WAIT *EXEMPTION* *MU1*) > > FINALLY (RETURN *EXEMPTION-STATE*))) tried another make check and got the make: *** [check-exec-image] Error 1 again, then tried again and got past that but ended up with this: (MULTIPLE-VALUE-BIND (VALUE CONDITION) (IGNORE-ERRORS (DEFCLASS FOO74 NIL ((SIZE :INITARG :SIZE :INITFORM 1 :ALLOCATION :CLASS))) (SETQ I (MAKE-INSTANCE 'FOO74)) (DEFCLASS FOO74 NIL ((SIZE :INITARG :SIZE :INITFORM 2 :ALLOCATION :CLASS) (OTHER))) (SLOT-VALUE I 'SIZE)) (LIST VALUE (TYPEP CONDITION 'ERROR))) *** - handle_fault error2 ! address = 0x0 not in [0x333f79000,0x3341cd4e0) ! SIGSEGV cannot be cured. Fault address = 0x0. GC count: 50 Space collected by GC: 37343824 Run time: 6 849957 Real time: 6 869976 GC time: 0 274965 Permanently allocated: 175776 bytes. Currently in use: 4429808 bytes. Free space: 240636 bytes. make[1]: *** [parallel] Segmentation fault make[1]: Leaving directory `/root/clisp/build-mt/tests' make: *** [check-tests-parallel] Error 2 |
From: Vladimir T. <vtz...@gm...> - 2009-06-28 07:39:42
|
On 6/28/09, Don Cohen <don...@is...> wrote: > Vladimir Tzankov writes: > > I think it's not good to have much more concurrent threads than the > > number of cpu cores - so did not do anything to change it. > Perhaps you mean more concurrent threads trying to run. > It makes sense to have a large number waiting for IO, for example. I do not like using threads only/mainly for IO (i.e. each thread waiting on some IO. SOCKET-STATUS is be better option for this). Anyway we should not try to enforce proper way of doing things - so if somebody wants to use threads for IO - let him use them. However he really may hit the 128 threads limit. > > specified in MAKE-THREAD. Default values are > > MT::*DEFAULT-CONTROL-STACK-SIZE* (initialized to 1MB) and > > MT::*DEFAULT-VALUE-STACK-SIZE* (initialized from the command line or > > some reasonable default - as in single thread). > 458176 for me > So it seems at least plausible to have > 128 threads. We can easily increase the limit (as far as we do not get overlap of malloc-ed lisp stack and lisp heap). Sam? > tried another make check and got the Again: make check is likely to fail due to make check-tests-parallel. however make check-tests should always pass. > (MULTIPLE-VALUE-BIND (VALUE CONDITION) (IGNORE-ERRORS (DEFCLASS FOO74 > NIL ((SIZE :INITARG :SIZE :INITFORM 1 :ALLOCATION :CLASS))) (SETQ I > (MAKE-INSTANCE 'FOO74)) (DEFCLASS FOO74 NIL ((SIZE :INITARG :SIZE > :INITFORM 2 :ALLOCATION :CLASS) (OTHER))) (SLOT-VALUE I 'SIZE)) (LIST > VALUE (TYPEP CONDITION 'ERROR))) > > *** - handle_fault error2 ! address = 0x0 not in > .... > make[1]: Leaving directory `/root/clisp/build-mt/tests' > make: *** [check-tests-parallel] Error 2 Here failure is due to make check-tests-parallel and is related with CLOS lack of locking. check-tests-parallel simple runs all the test in different threads (about 5 threads at a time). When some of these threads change CLOS internals simultaneously you may/will get something as above. |
From: Sam S. <sd...@gn...> - 2009-06-29 14:50:30
|
Vladimir Tzankov wrote: > On 6/28/09, Don Cohen <don...@is...> wrote: >> So it seems at least plausible to have > 128 threads. > > We can easily increase the limit (as far as we do not get overlap of > malloc-ed lisp stack and lisp heap). Sam? The bug you mentioned (overlap between mmaped and malloc'ed memory) is very scary. I think it should be investigated (e.g., does it depend on order, i.e., mmap will not overlap with the previously malloc'ed memory, but malloc may overlap with the previously mmaped memory?) and we need an autoconf test for that. we already use malloc in the code (other than allocating STACK for a new thread), so if such an overlap is possible, we might have to give up mmap. More to the point, I abhor arbitrary limits, so I think this one should go too. (In fact I wanted to replace the array of threads with a doubly linked list). BTW, I think we should signal an error when create_thread return NULL instead of make-thread returning NIL. |
From: <don...@is...> - 2009-06-29 18:56:26
|
Vladimir Tzankov writes: > I do not like using threads only/mainly for IO (i.e. each thread > waiting on some IO. SOCKET-STATUS is be better option for this). I had intended to mention that at least one good reason for this is multiple lisp listeners. Most of them spend most of their time waiting for input. I have a lisp based database and I want users (and programs) to be able to connect to it the same way as they do to other databases. I have built up an infrastructure for doing this without threads, but it depends on all things that are evaluated to terminate promptly - if one user does something that ends up taking a long time then everybody else has to wait. I don't see any cure for this other than MT. Even executing everything with a timeout seems to require MT. |
From: <don...@is...> - 2009-06-28 07:44:40
|
Vladimir Tzankov writes: > Here failure is due to make check-tests-parallel and is related with > CLOS lack of locking. check-tests-parallel simple runs all the test in > different threads (about 5 threads at a time). When some of these > threads change CLOS internals simultaneously you may/will get > something as above. Does this also explain the problem with the saveinitmem? |
From: Vladimir T. <vtz...@gm...> - 2009-06-28 07:51:56
|
On 6/28/09, Don Cohen <don...@is...> wrote: > Vladimir Tzankov writes: > > Here failure is due to make check-tests-parallel and is related with > > CLOS lack of locking. check-tests-parallel simple runs all the test in > > different threads (about 5 threads at a time). When some of these > > threads change CLOS internals simultaneously you may/will get > > something as above. > Does this also explain the problem with the saveinitmem? No, saveinitmem should be fine (make check-exec-image should pass). Does it fail (as in your previous mail)? |
From: <don...@is...> - 2009-06-28 16:01:55
|
Vladimir Tzankov writes: > No, saveinitmem should be fine (make check-exec-image should pass). > Does it fail (as in your previous mail)? I've just tried make check-tests and it worked - twice. Does it do the saveinitmem test? I guess not. I just tried make check-exec-image and got the failure. WARNING: No installation directory specified. Please try: ./foo -B /usr/local/lib/clisp 10 *** - EVAL: undefined function SAVEINITMEM Bye. ./foo -norc -M foo.mem -x zz ./foo: operating system error during load of initialization file `foo.mem' [../src/spvw_memfile.d:981] errno = ENOENT: No such file or directory. make: *** [check-exec-image] Error 1 |
From: Vladimir T. <vtz...@gm...> - 2009-06-28 16:19:49
|
On 6/28/09, Don Cohen <don...@is...> wrote: > Vladimir Tzankov writes: > > No, saveinitmem should be fine (make check-exec-image should pass). > > Does it fail (as in your previous mail)? > I've just tried make check-tests and it worked - twice. > Does it do the saveinitmem test? > I guess not. No. > I just tried make check-exec-image and got the failure. > WARNING: No installation directory specified. > Please try: ./foo -B /usr/local/lib/clisp > 10 > *** - EVAL: undefined function SAVEINITMEM > Bye. > ./foo -norc -M foo.mem -x zz > ./foo: operating system error during load of initialization file > `foo.mem' > [../src/spvw_memfile.d:981] errno = ENOENT: No such file or directory. > make: *** [check-exec-image] Error 1 thanks. will try to reproduce and fix it. |
From: Sam S. <sd...@gn...> - 2009-06-29 19:28:19
|
Don Cohen wrote: > Vladimir Tzankov writes: > > I do not like using threads only/mainly for IO (i.e. each thread > > waiting on some IO. SOCKET-STATUS is be better option for this). > I had intended to mention that at least one good reason for this is > multiple lisp listeners. Most of them spend most of their time > waiting for input. so there should be one thread which listens on all connections using SOCKET-STATUS. when a connection has something available, a thread is spawned to read-eval-print _once_ from that connection and then to exit. this simplistic approach neglects things like */- variables and error handling, however, that too could be addressed... |
From: <don...@is...> - 2009-06-29 21:23:45
|
Sam Steingold writes: > Don Cohen wrote: > > Vladimir Tzankov writes: > > > I do not like using threads only/mainly for IO (i.e. each thread > > > waiting on some IO. SOCKET-STATUS is be better option for this). > > I had intended to mention that at least one good reason for this is > > multiple lisp listeners. Most of them spend most of their time > > waiting for input. > > so there should be one thread which listens on all connections > using SOCKET-STATUS. > when a connection has something available, a thread is spawned to > read-eval-print _once_ from that connection and then to exit. > this simplistic approach neglects things like */- variables and > error handling, however, that too could be addressed... What it really leaves out is debugging and other similar cases where process state has to be maintained while waiting for IO. In my traditional servers all errors are caught by ignore-errors and you get back the error string. Maybe if I worked harder I could figure out how to send back a backtrace. But it's really not the same as using a debugger. |