clisp-devel

stack overflow => segfault in MT

[<don...@is...>]

I send this on the theory that any segfault qualifies as a clisp bug.
This is with current source.  On non-MT I get 
  *** - Program stack overflow. RESET 
whereas in MT I get segfault.
This is, of course, caused by a bug in my program.
In this case I was lucky and got a warning at each recursion,
so was able to break by setting CUSTOM:*BREAK-ON-WARNINGS* to T.
Here's the backtrace a few recursions into the process that 
eventually ends in segfault.

 > Break 1 AP5[8]> :bt
 > <1/3091> #<SYSTEM-FUNCTION EXT:SHOW-STACK> 3
 > <2/3084> #<COMPILED-FUNCTION SYSTEM::PRINT-BACKTRACE>
 > <3/3078> #<COMPILED-FUNCTION SYSTEM::DEBUG-BACKTRACE>
 > <4/3069> #<SYSTEM-FUNCTION SYSTEM::READ-EVAL-PRINT> 2
 > <5/3066> #<COMPILED-FUNCTION SYSTEM::BREAK-LOOP-2-3>
 > <6/3062> #<SYSTEM-FUNCTION SYSTEM::SAME-ENV-AS> 2
 > <7/3048> #<COMPILED-FUNCTION SYSTEM::BREAK-LOOP-2>
 > <8/3046> #<SYSTEM-FUNCTION SYSTEM::DRIVER>
 > <9/3006> #<COMPILED-FUNCTION SYSTEM::BREAK-LOOP>
 > <10/2989> #<COMPILED-FUNCTION SYSTEM::WARN-OF-TYPE>
 > <11/2981> #<COMPILED-FUNCTION SYSTEM::C-WARNING>
 > <12/2976> #<COMPILED-FUNCTION SYSTEM::C-WARN>
 > <13/2973> #<COMPILED-FUNCTION SYSTEM::NOTE-FUNCTION-USED>
 > <14/2969> #<COMPILED-FUNCTION SYSTEM::C-GLOBAL-FUNCTION-CALL>
 > <15/2948> #<COMPILED-FUNCTION SYSTEM::C-FORM>
 > <16/2940> #<COMPILED-FUNCTION SYSTEM::C-PROGN>
 > <17/2919> #<COMPILED-FUNCTION SYSTEM::C-FORM>
 > <18/2838> #<COMPILED-FUNCTION SYSTEM::C-FUNCALL-INLINE>
 > <19/2832> #<COMPILED-FUNCTION SYSTEM::C-FUNCTION-CALL>
 > <20/2829> #<COMPILED-FUNCTION SYSTEM::C-FUNCALL>
 > <21/2808> #<COMPILED-FUNCTION SYSTEM::C-FORM>
 > <22/2795> #<COMPILED-FUNCTION SYSTEM::C-FORM>
 > <23/2790> #<COMPILED-FUNCTION SYSTEM::COLLECT-ARGS>
 > <24/2772> #<COMPILED-FUNCTION SYSTEM::C-GLOBAL-FUNCTION-CALL>
 > <25/2751> #<COMPILED-FUNCTION SYSTEM::C-FORM>
 > <26/2746> #<COMPILED-FUNCTION SYSTEM::COLLECT-ARGS>
 > <27/2728> #<COMPILED-FUNCTION SYSTEM::C-GLOBAL-FUNCTION-CALL>
 > <28/2707> #<COMPILED-FUNCTION SYSTEM::C-FORM>
 > <29/2703> #<COMPILED-FUNCTION SYSTEM::C-PROGN>
 > <30/2682> #<COMPILED-FUNCTION SYSTEM::C-FORM>
 > <31/2640> #<COMPILED-FUNCTION SYSTEM::C-LET/LET*>
 > <32/2638> 
 > #<COMPILED-FUNCTION
 >   #:|2105 2263 (DEFCONSTANT C-FORM-TABLE (LET # # ...))-172-1|>
 > <33/2617> #<COMPILED-FUNCTION SYSTEM::C-FORM>
 > <34/2613> #<COMPILED-FUNCTION SYSTEM::C-PROGN>
 > <35/2592> #<COMPILED-FUNCTION SYSTEM::C-FORM>
 > <36/2505> #<COMPILED-FUNCTION SYSTEM::C-LAMBDABODY>
 > <37/2493> #<COMPILED-FUNCTION SYSTEM::C-FUNCTION>
 > <38/2472> #<COMPILED-FUNCTION SYSTEM::C-FORM>
 > <39/2467> #<COMPILED-FUNCTION SYSTEM::COLLECT-ARGS>
 > <40/2459> #<COMPILED-FUNCTION SYSTEM::C-NORMAL-FUNCTION-CALL>
 > <41/2445> #<COMPILED-FUNCTION SYSTEM::C-GLOBAL-FUNCTION-CALL>
 > <42/2424> #<COMPILED-FUNCTION SYSTEM::C-FORM>
 > <43/2402> #<COMPILED-FUNCTION SYSTEM::C-FORM>
 > <44/2398> #<COMPILED-FUNCTION SYSTEM::C-PROGN>
 > <45/2377> #<COMPILED-FUNCTION SYSTEM::C-FORM>
 > <46/2290> #<COMPILED-FUNCTION SYSTEM::C-LAMBDABODY>
 > <47/2281> #<COMPILED-FUNCTION SYSTEM::COMPILE-LAMBDABODY>
 > <48/2201> #<COMPILED-FUNCTION COMMON-LISP:COMPILE>
 > <49/2194> #<COMPILED-FUNCTION COMPILE>
 > <50/2188> #<COMPILED-FUNCTION MYCOMPILE>
 > <51/2185> #<COMPILED-FUNCTION COMPILE-AP>
 > <52/2181> #<COMPILED-FUNCTION NEED-TESTER>
 > <53/2170> #<COMPILED-FUNCTION TRANSLATE??>
 > <54/2165> #<COMPILED-FUNCTION ??>
 > <55/2165> #<SYSTEM-FUNCTION FUNCALL> 3
 > <56/2163> #<SYSTEM-FUNCTION MACROEXPAND>
 > <57/2150> #<COMPILED-FUNCTION MAKE-TEST-GENERATOR>
 > <58/2141> #<COMPILED-FUNCTION FINDGENERATORS>
 > <59/2132> #<COMPILED-FUNCTION FINDGENERATOR>
 > <60/2111> #<COMPILED-FUNCTION GET-GENERATOR>
 > <61/2103> #<COMPILED-FUNCTION ANALYZE-DESC>
 > <62/2097> #<COMPILED-FUNCTION GENERATOR>
 > <63/2097> #<SYSTEM-FUNCTION FUNCALL> 3
 > <64/2095> #<SYSTEM-FUNCTION MACROEXPAND>
 > <65/2080> #<COMPILED-FUNCTION FORANY>
 > <66/2080> #<SYSTEM-FUNCTION FUNCALL> 3
 > <67/2053> #<COMPILED-FUNCTION SYSTEM::C-FORM>
 > <68/2045> #<COMPILED-FUNCTION SYSTEM::C-PROGN>
 > <69/2024> #<COMPILED-FUNCTION SYSTEM::C-FORM>
 > <70/1943> #<COMPILED-FUNCTION SYSTEM::C-FUNCALL-INLINE>
 > <71/1937> #<COMPILED-FUNCTION SYSTEM::C-FUNCTION-CALL>
 > <72/1934> #<COMPILED-FUNCTION SYSTEM::C-FUNCALL>
 > <73/1913> #<COMPILED-FUNCTION SYSTEM::C-FORM>
 > <74/1900> #<COMPILED-FUNCTION SYSTEM::C-FORM>
 > <75/1895> #<COMPILED-FUNCTION SYSTEM::COLLECT-ARGS>
 > <76/1877> #<COMPILED-FUNCTION SYSTEM::C-GLOBAL-FUNCTION-CALL>
 > <77/1856> #<COMPILED-FUNCTION SYSTEM::C-FORM>
 > <78/1851> #<COMPILED-FUNCTION SYSTEM::COLLECT-ARGS>
 > <79/1833> #<COMPILED-FUNCTION SYSTEM::C-GLOBAL-FUNCTION-CALL>
 > <80/1812> #<COMPILED-FUNCTION SYSTEM::C-FORM>
 > <81/1808> #<COMPILED-FUNCTION SYSTEM::C-PROGN>
 > <82/1787> #<COMPILED-FUNCTION SYSTEM::C-FORM>
 > <83/1745> #<COMPILED-FUNCTION SYSTEM::C-LET/LET*>
 > <84/1743> 
 > #<COMPILED-FUNCTION
 >   #:|2105 2263 (DEFCONSTANT C-FORM-TABLE (LET # # ...))-172-1|>
 > <85/1722> #<COMPILED-FUNCTION SYSTEM::C-FORM>
 > <86/1718> #<COMPILED-FUNCTION SYSTEM::C-PROGN>
 > <87/1697> #<COMPILED-FUNCTION SYSTEM::C-FORM>
 > <88/1610> #<COMPILED-FUNCTION SYSTEM::C-LAMBDABODY>
 > <89/1598> #<COMPILED-FUNCTION SYSTEM::C-FUNCTION>
 > <90/1577> #<COMPILED-FUNCTION SYSTEM::C-FORM>
 > <91/1572> #<COMPILED-FUNCTION SYSTEM::COLLECT-ARGS>
 > <92/1564> #<COMPILED-FUNCTION SYSTEM::C-NORMAL-FUNCTION-CALL>
 > <93/1550> #<COMPILED-FUNCTION SYSTEM::C-GLOBAL-FUNCTION-CALL>
 > <94/1529> #<COMPILED-FUNCTION SYSTEM::C-FORM>
 > <95/1507> #<COMPILED-FUNCTION SYSTEM::C-FORM>
 > <96/1503> #<COMPILED-FUNCTION SYSTEM::C-PROGN>
 > <97/1482> #<COMPILED-FUNCTION SYSTEM::C-FORM>
 > <98/1395> #<COMPILED-FUNCTION SYSTEM::C-LAMBDABODY>
 > <99/1386> #<COMPILED-FUNCTION SYSTEM::COMPILE-LAMBDABODY>
 > <100/1306> #<COMPILED-FUNCTION COMMON-LISP:COMPILE>
 > <101/1299> #<COMPILED-FUNCTION COMPILE>
 > <102/1293> #<COMPILED-FUNCTION MYCOMPILE>
 > <103/1290> #<COMPILED-FUNCTION COMPILE-AP>
 > <104/1286> #<COMPILED-FUNCTION NEED-TESTER>
 > <105/1275> #<COMPILED-FUNCTION TRANSLATE??>
 > <106/1270> #<COMPILED-FUNCTION ??>
 > <107/1270> #<SYSTEM-FUNCTION FUNCALL> 3
 > <108/1268> #<SYSTEM-FUNCTION MACROEXPAND>
 > <109/1255> #<COMPILED-FUNCTION MAKE-TEST-GENERATOR>
 > <110/1246> #<COMPILED-FUNCTION FINDGENERATORS>
 > <111/1237> #<COMPILED-FUNCTION FINDGENERATOR>
 > <112/1216> #<COMPILED-FUNCTION GET-GENERATOR>
 > <113/1208> #<COMPILED-FUNCTION ANALYZE-DESC>
 > <114/1202> #<COMPILED-FUNCTION GENERATOR>
 > <115/1202> #<SYSTEM-FUNCTION FUNCALL> 3
 > <116/1200> #<SYSTEM-FUNCTION MACROEXPAND>
 > <117/1185> #<COMPILED-FUNCTION FORANY>
 > <118/1185> #<SYSTEM-FUNCTION FUNCALL> 3
 > <119/1158> #<COMPILED-FUNCTION SYSTEM::C-FORM>
 > <120/1150> #<COMPILED-FUNCTION SYSTEM::C-PROGN>
 > <121/1129> #<COMPILED-FUNCTION SYSTEM::C-FORM>
 > <122/1048> #<COMPILED-FUNCTION SYSTEM::C-FUNCALL-INLINE>
 > <123/1042> #<COMPILED-FUNCTION SYSTEM::C-FUNCTION-CALL>
 > <124/1039> #<COMPILED-FUNCTION SYSTEM::C-FUNCALL>
 > <125/1018> #<COMPILED-FUNCTION SYSTEM::C-FORM>
 > <126/1005> #<COMPILED-FUNCTION SYSTEM::C-FORM>
 > <127/1000> #<COMPILED-FUNCTION SYSTEM::COLLECT-ARGS>
 > <128/982> #<COMPILED-FUNCTION SYSTEM::C-GLOBAL-FUNCTION-CALL>
 > <129/961> #<COMPILED-FUNCTION SYSTEM::C-FORM>
 > <130/956> #<COMPILED-FUNCTION SYSTEM::COLLECT-ARGS>
 > <131/938> #<COMPILED-FUNCTION SYSTEM::C-GLOBAL-FUNCTION-CALL>
 > <132/917> #<COMPILED-FUNCTION SYSTEM::C-FORM>
 > <133/913> #<COMPILED-FUNCTION SYSTEM::C-PROGN>
 > <134/892> #<COMPILED-FUNCTION SYSTEM::C-FORM>
 > <135/850> #<COMPILED-FUNCTION SYSTEM::C-LET/LET*>
 > <136/848> 
 > #<COMPILED-FUNCTION
 >   #:|2105 2263 (DEFCONSTANT C-FORM-TABLE (LET # # ...))-172-1|>
 > <137/827> #<COMPILED-FUNCTION SYSTEM::C-FORM>
 > <138/823> #<COMPILED-FUNCTION SYSTEM::C-PROGN>
 > <139/802> #<COMPILED-FUNCTION SYSTEM::C-FORM>
 > <140/715> #<COMPILED-FUNCTION SYSTEM::C-LAMBDABODY>
 > <141/703> #<COMPILED-FUNCTION SYSTEM::C-FUNCTION>
 > <142/682> #<COMPILED-FUNCTION SYSTEM::C-FORM>
 > <143/677> #<COMPILED-FUNCTION SYSTEM::COLLECT-ARGS>
 > <144/669> #<COMPILED-FUNCTION SYSTEM::C-NORMAL-FUNCTION-CALL>
 > <145/655> #<COMPILED-FUNCTION SYSTEM::C-GLOBAL-FUNCTION-CALL>
 > <146/634> #<COMPILED-FUNCTION SYSTEM::C-FORM>
 > <147/612> #<COMPILED-FUNCTION SYSTEM::C-FORM>
 > <148/608> #<COMPILED-FUNCTION SYSTEM::C-PROGN>
 > <149/587> #<COMPILED-FUNCTION SYSTEM::C-FORM>
 > <150/500> #<COMPILED-FUNCTION SYSTEM::C-LAMBDABODY>
 > <151/491> #<COMPILED-FUNCTION SYSTEM::COMPILE-LAMBDABODY>
 > <152/411> #<COMPILED-FUNCTION COMMON-LISP:COMPILE>
 > <153/404> #<COMPILED-FUNCTION COMPILE>
 > <154/398> #<COMPILED-FUNCTION MYCOMPILE>
 > <155/395> #<COMPILED-FUNCTION COMPILE-AP>
 > <156/391> #<COMPILED-FUNCTION NEED-TESTER>
 > <157/380> #<COMPILED-FUNCTION TRANSLATE??>
 > <158/375> #<COMPILED-FUNCTION ??>

 ... 

Re: more problems from MT debugger

[Vladimir T. <vtz...@gm...>]

On 3/6/11, Don Cohen <don...@is...> wrote:
>
> Code is below.  (It's similar to what I've sent before.)
> ...
>     (unwind-protect
> 	(mt:thread-interrupt
> 	 ans
> 	 :function
> 	 (lambda nil
> 	   (let ((*standard-input* socket)
> 		 (*standard-output* socket)
> 		 (*debug-io* socket)
> 		 (*error-output* socket)
> 		 (*trace-output* socket)
> 		 (*query-io* socket))
> 	     (break "debug"))))
>       (close socket))))

You are closing the socket too early! In your previous mails the code
of your debug-server closes the socket within the interrupted thread:
 (mt:thread-interrupt
 ans
 :function
 (lambda nil
   (let ((*standard-input* socket)
         (*standard-output* socket)
         (*debug-io* socket)
         (*error-output* socket)
         (*trace-output* socket)
                        (*query-io* socket))
     (unwind-protect
          (break "debug")
       (close socket)))))


However having segfaults even in this case (read from closed socket)
is not good. Will check.

Vladimir

Re: more problems from MT debugger/stack overflow => segfault

[<don...@is...>]

Vladimir Tzankov writes:
 > You are closing the socket too early!
(ah, one of the few places where moving something later in the code
causes it to happen earlier in the execution!)

 > However having segfaults even in this case (read from closed socket)
 > is not good. Will check.
I suspect the next message explains what's happening here.
If each complaint about the closed stream leads to an attempt to enter
the debugger, and the debugger tries to print to the same stream which
causes another error, then we get a stack overflow, and since it's
occuring in a non-main thread, that causes the segfault.

Vladimir Tzankov writes:
 > On 3/6/11, Don Cohen <don...@is...> wrote:
 > > I send this on the theory that any segfault qualifies as a clisp bug.
 > > This is with current source.  On non-MT I get
 > >   *** - Program stack overflow. RESET
 > > whereas in MT I get segfault.
 > 
 > With MT - only the "main thread" (first one) has stack overflow detection.
 > Stack overflow detection/recovery is implemented via libsigsegv and it
 > supports single stack range.
 > May be libsigesgv should be enhanced with multiple stack range support? ?

A few thoughts on this.
- Perhaps there's some shorter term solution involving checking when 
doing at least some operations that use more stack?
- I thought there was something similar used for generational GC, which
I imagined was going to need to detect writes to arbitrary sets of pages.
- It would be useful for debugging if we could get a lisp error BEFORE
the overflow, say, when you exceed half of the available stack and
then again when you exceed 3/4, etc.

Re: stack overflow => segfault in MT

[Vladimir T. <vtz...@gm...>]

On 3/6/11, Don Cohen <don...@is...> wrote:
> I send this on the theory that any segfault qualifies as a clisp bug.
> This is with current source.  On non-MT I get
>   *** - Program stack overflow. RESET
> whereas in MT I get segfault.

With MT - only the "main thread" (first one) has stack overflow detection.
Stack overflow detection/recovery is implemented via libsigsegv and it
supports single stack range.
May be libsigesgv should be enhanced with multiple stack range support? gSam?