From: Kalyanov D. <kal...@gm...> - 2010-06-14 04:43:36
|
Hello, I'd like to announce my work on addition of threads to Windows port of SBCL. The code is available at http://github.com/dmitryvk/sbcl-win32-threads in 'tib-3' branch. Binary installer is available at http://sites.google.com/site/dmitryvksite/sbcl-distr It's still alpha-quality and performance may suffer (due to inefficient GC safepoint implementation). I'd be glad for review or bug reports. Implementation notes: 1) Thread-local storage is implemented using TIB (Thread Information Block). On Win32, FS register always points to current thread's TIB (on Win64, GS register points to TIB). TIB contains a field for thread private data at offset 0x14 (http://en.wikipedia.org/wiki/Win32_Thread_Information_Block) - the pointer to thread structure is stored there. Other solutions for storing thread pointer are broken in some ways. Notably, the approach used by Clozure CL to store TCR in segment register does not work on 64-bit windows. 2) Subset of pthreads API is implemented using Windows synchronization primitives. 3) Thread suspension for GC is fully cooperative. This is achieved by using safepoints. I took P.Khuong's code for placing safepoints (http://www.pvk.ca/Blog/LowLevel/VM_tricks_safepoints.html). Safepoints for now are implemented by checking a global variable, but I'll switch to the "magic page" method. 4) When thread enters foreign code or does a blocking operation, 'gc_safe' flag is set on the thread - such threads do not need to be suspended for GC. When threads reenters lisp (by returning from foreign function or by entering a callback) it waits until the world is restarted (if it was suspended). 5) Thread interruption is also cooperative. Interrupts will only be received when thread is running lisp code. ---- Кальянов Дмитрий |
From: Christophe R. <cs...@ca...> - 2010-06-14 07:45:20
|
Kalyanov Dmitry <kal...@gm...> writes: > I'd like to announce my work on addition of threads to Windows port of SBCL. > > The code is available at http://github.com/dmitryvk/sbcl-win32-threads in > 'tib-3' branch. Binary installer is available at > http://sites.google.com/site/dmitryvksite/sbcl-distr > > It's still alpha-quality and performance may suffer (due to inefficient GC > safepoint implementation). I'd be glad for review or bug reports. Well, my first reaction is "how cool!". I hope that you'll get review and testing from Windows users; it would also be good to open the discussion about some of the underlying (more platform-independent) techniques such as safepoints -- if Windows threading is likely to depend on them, should they be a conditional feature on all platforms? That kind of thing. Thank you for this. Best, Christophe |
From: Matthew S. <ako...@gm...> - 2010-06-16 17:08:21
|
Kalyanov Dmitry <kalyanov.dmitry <at> gmail.com> writes: > > Hello, > > I'd like to announce my work on addition of threads to Windows port of SBCL. > > The code is available at http://github.com/dmitryvk/sbcl-win32-threads in > 'tib-3' branch. First hurdle: gcc-3 -g -Wall -O3 -mno-cygwin -I. -c -o win32-os.o win32-os.c win32-os.c: In function `handle_exception': win32-os.c:355: error: structure has no member named `gc_safe' win32-os.c:358: error: structure has no member named `gc_safe' win32-os.c:359: warning: implicit declaration of function `gc_safepoint' win32-os.c:369: error: structure has no member named `gc_safe' win32-os.c:383: error: structure has no member named `gc_safe' win32-os.c:445: error: structure has no member named `gc_safe' win32-os.c:456: error: structure has no member named `gc_safe' win32-os.c:496: warning: implicit declaration of function `_clearfp' win32-os.c:504: error: structure has no member named `gc_safe' win32-os.c:532: error: structure has no member named `gc_safe' make: *** [win32-os.o] Error 1 on CYGWIN_NT-6.1 EX101 1.7.5(0.225/5/3) 2010-04-12 19:07 i686 Cygwin (running on windows 7 32-bit) |
From: Kalyanov D. <kal...@gm...> - 2010-06-16 17:46:23
|
On Wednesday 16 June 2010 21:07:52, Matthew Swank wrote: > Kalyanov Dmitry <kalyanov.dmitry <at> gmail.com> writes: > > Hello, > > > > I'd like to announce my work on addition of threads to Windows port of > > SBCL. > > > > The code is available at http://github.com/dmitryvk/sbcl-win32-threads in > > 'tib-3' branch. > > First hurdle: > > gcc-3 -g -Wall -O3 -mno-cygwin -I. -c -o win32-os.o win32-os.c > win32-os.c: In function `handle_exception': > win32-os.c:355: error: structure has no member named `gc_safe' > win32-os.c:358: error: structure has no member named `gc_safe' > win32-os.c:359: warning: implicit declaration of function `gc_safepoint' > win32-os.c:369: error: structure has no member named `gc_safe' > win32-os.c:383: error: structure has no member named `gc_safe' > win32-os.c:445: error: structure has no member named `gc_safe' > win32-os.c:456: error: structure has no member named `gc_safe' > win32-os.c:496: warning: implicit declaration of function `_clearfp' > win32-os.c:504: error: structure has no member named `gc_safe' > win32-os.c:532: error: structure has no member named `gc_safe' > make: *** [win32-os.o] Error 1 > > on > CYGWIN_NT-6.1 EX101 1.7.5(0.225/5/3) 2010-04-12 19:07 i686 Cygwin > (running on windows 7 32-bit) I've forgot to mention that customize-target-features.lisp should include :sb- thread and :sb-pthread-futex features. Also, I didn't properly conditionalize source code, so unithreaded build doesn't work properly. My customize-target-features.lisp that was used to build sbcl looks like this: (lambda (features) (pushnew :sb-after-xc-core features) (pushnew :sb-fluid features) (pushnew :sb-thread features) (pushnew :sb-pthread-futex features) features ) (you don't need to include :sb-after-xc-core and :sb-fluid) ---- Кальянов Дмитрий |
From: Matthew S. <ako...@gm...> - 2010-06-16 18:55:23
|
Kalyanov Dmitry <kalyanov.dmitry <at> gmail.com> writes: > > On Wednesday 16 June 2010 21:07:52, Matthew Swank wrote: > > Kalyanov Dmitry <kalyanov.dmitry <at> gmail.com> writes: > > > Hello, > > > > > > I'd like to announce my work on addition of threads to Windows port of > > > SBCL. ... > > First hurdle: > > > > gcc-3 -g -Wall -O3 -mno-cygwin -I. -c -o win32-os.o win32-os.c ... > > I've forgot to mention that customize-target-features.lisp should include :sb- > thread and :sb-pthread-futex features. Thanks. It's up and slime is working, so that's a good start. Matt |
From: Matthew S. <ako...@gm...> - 2010-06-16 19:10:09
|
Kalyanov Dmitry <kalyanov.dmitry <at> gmail.com> writes: > > Hello, > > I'd like to announce my work on addition of threads to Windows port of SBCL. > Purely unscientific, but compared to a recent unithreaded build, the branch doesn't do too bad: $ diff -aur sbcl-1.0.38.12.test.log sbcl-1.0.36.16-mt.test.log --- sbcl.test.log 2010-06-16 14:03:21.919210600 -0500 +++ sbcl-mt.test.log 2010-06-16 14:03:19.626006600 -0500 @@ -1,23 +1,23 @@ Finished running tests. Status: - + Unhandled error compiler.pure.lisp Failure: interface.pure.lisp / WITH-TIMEOUT-FORMS Failure: stream.pure.lisp / (STREAM LISTEN-VS-SELECT) - + Unhandled error threads.pure.lisp Unhandled error alien.impure.lisp Unhandled error clos-interrupts.impure.lisp - Failure: deadline.impure.lisp / (DEADLINE RUN-PROGRAM TRIVIAL) - Failure: deadline.impure.lisp / (DEADLINE DEFER-DEADLINE-1) - Failure: deadline.impure.lisp / (DEADLINE DEFER-DEADLINE-2) - Failure: deadline.impure.lisp / (DEADLINE CANCEL-DEADLINE) + Unhandled error deadline.impure.lisp |
From: Matthew S. <ako...@gm...> - 2010-06-16 19:50:02
|
Kalyanov Dmitry <kalyanov.dmitry <at> gmail.com> writes: > > Hello, > > I'd like to announce my work on addition of threads to Windows port of SBCL. > Looking ahead to getting better rdnzl support, is it possible to invoke a callback defined in lisp from a thread that has not been spawned by the sbcl runtime? Matt |
From: Matthew S. <ako...@gm...> - 2010-06-16 22:07:37
|
Matthew Swank <akopa.gmane.poster <at> gmail.com> writes: > > Kalyanov Dmitry <kalyanov.dmitry <at> gmail.com> writes: > > > > > Hello, > > > > I'd like to announce my work on addition of threads to Windows port of SBCL. > > > [I]s it possible to invoke a callback defined in lisp from a thread that has > not been spawned by the sbcl runtime? It looks like speculation on this subject ends here: >From http://article.gmane.org/gmane.lisp.steel-bank.devel/14395/ > Nikodemus Siivola <nikodemus <at> random-state.net> writes: >> The right way to do this would probably be to set up a temporary "virtual" >> lisp thread while the callback is active if there is no real lisp thread >> corresponding to the current native thread. Does anything like this exist in current sbcl (or the tib-3 branch)? |
From: Matthew S. <ako...@gm...> - 2010-06-17 17:44:37
|
Matthew Swank <akopa.gmane.poster <at> gmail.com> writes: > > Kalyanov Dmitry <kalyanov.dmitry <at> gmail.com> writes: > > > > > Hello, > > > > I'd like to announce my work on addition of threads to Windows port of SBCL. > > > Looking ahead to getting better rdnzl support, is it possible to invoke a > callback defined in lisp from a thread that has not been spawned by the sbcl > runtime? > Actually, in the case of Windows.Forms, as long as all my callbacks are registered with events associated with controls, it should all happen in the same thread. Though the idea of tracking a foreign thread in the sbcl runtime until the return of the initial call into lisp is interesting nonetheless... As readers of this list may be aware of at this point, I am a little excited about sbcl becoming multithreaded on WinXX. I know there are other thing that would make sbcl more stable and integrated into the Windoze eco system (just look at the failures on the test suite), but threads make a whole lot things possible and would make sbcl easier for me to use at work. I am interested in investigating the thread related test failures, but I'm a little fuzzy on how to isolate individual tests in the test suite. Since you've gone to the trouble of announcing your work, is the locus of your interest in polishing this branch and then work on forward porting it, or is it the other way around? I don't have much experience integrating a branch, but I am curious to know where I can be of help. Matt |
From: Kalyanov D. <kal...@gm...> - 2010-06-17 18:35:14
|
On Thursday 17 June 2010 21:44:13, Matthew Swank wrote: > Matthew Swank <akopa.gmane.poster <at> gmail.com> writes: > > Kalyanov Dmitry <kalyanov.dmitry <at> gmail.com> writes: > > > Hello, > > > > > > I'd like to announce my work on addition of threads to Windows port of > > > SBCL. > > > > Looking ahead to getting better rdnzl support, is it possible to invoke a > > callback defined in lisp from a thread that has not been spawned by the > > sbcl runtime? > > Actually, in the case of Windows.Forms, as long as all my callbacks are > registered with events associated with controls, it should all happen in > the same thread. Though the idea of tracking a foreign thread in the sbcl > runtime until the return of the initial call into lisp is interesting > nonetheless... I guess I'm not yet ready to provide an answer for that. From my little experience with SBCL, it is entirely possible but it'll take some motivation and time to do it. > > As readers of this list may be aware of at this point, I am a little > excited about sbcl becoming multithreaded on WinXX. I know there are > other thing that would make sbcl more stable and integrated into the > Windoze eco system (just look at the failures on the test suite), but > threads make a whole lot things possible and would make sbcl easier for me > to use at work. > > I am interested in investigating the thread related test failures, but I'm > a little fuzzy on how to isolate individual tests in the test suite. Those failures require investigation, and I should say that I haven't looked carefully at the tests suite but rather wrote my own tests. I think that some failures might be attributed to some assumptions of test suite being violated (linux/unix and win32 are quite different). > > Since you've gone to the trouble of announcing your work, is the locus of > your interest in polishing this branch and then work on forward porting > it, or is it the other way around? I don't have much experience > integrating a branch, but I am curious to know where I can be of help. Some bugs and deficiencies must be fixed first, then I'll see what needs to be done to merge it. Win64 seems to be close enough to win32 so I might try to port it to win64 too. I do hope that eventually this will be merged into main branch of SBCL but I think that this will take some time. ---- Кальянов Дмитрий |
From: David L. <da...@li...> - 2012-10-22 21:15:35
|
Hi, users are, apparently, beginning to take notice of changes pushed to master earlier this month, and I would like to say a few words on the current state of things, if only to manage expectations... Two years ago Dmitry Kalyanov wrote: Quoting Kalyanov Dmitry (kal...@gm...): > I'd like to announce my work on addition of threads to Windows port of SBCL. > > The code is available at http://github.com/dmitryvk/sbcl-win32-threads in > 'tib-3' branch. Binary installer is available at > http://sites.google.com/site/dmitryvksite/sbcl-distr > > It's still alpha-quality and performance may suffer (due to inefficient GC > safepoint implementation). I'd be glad for review or bug reports. And rather belatedly, some support for sb-thread has arrived on master. I'd like to emphasize the following: 1. Many thanks to Dmitry and Anton for their work, and sorry it took us so long! Windows had not been a priority for SBCL for quite some time, but the message to users now must be that things are changing; we're taking Windows improvements rather seriously, and after all, we "are taking patches" again. 2. Yet I still recommend that users deploy Windows applications based on Anton's current branch; it includes many improvements not yet on master. In particular, AMD64 users have no other option anyway, but many other features and bugfixes also still remain to be merged. 3. However, it would be helpful if users could occasionally test official SBCL builds to see if "upstream" SBCL already meets their needs, and possibly report bugs that have crept in during the merge. What has been merged is thread support as such, based on Dmitry's work, but brought forward to Anton's original stop-the-world safepoint protocol changes (from roughly a year ago, i.e. still lacking some important fixes dating back to late 2011), and with just sufficient support for I/O interruptibility to get SLIME going. All other features and bug fixes will arrive in the next few releases (if everything goes as planned), step by step. For those interested in low-level details: Safepoints (the SIG_STOP_FOR_GC replacement), thruptions (SIGPIPE replacement), and wtimers (SIGALRM) are also available for Linux and Solaris on x86 and amd64, but are a completely optional, experimental feature on those. For these platforms, bug reports (including easily reproducible test cases :-)) would be very helpful. Thanks in advance for testing, and let's please keep the celebrations of the merge rather low-key until it's actually done. :-) d. |