#3311 stack overflow in asksign, was: Series evaluation segfault

[Felix Potthast]

I found a series which brings up a segfault when evaluated, it works without a problem when not in complex domain. I originally found it through sage.

;;; Loading #P"/usr/lib/ecl-16.1.3/sb-bsd-sockets.fas"
;;; Loading #P"/usr/lib/ecl-16.1.3/sockets.fas"
;;; Loading #P"/usr/lib/ecl-16.1.3/defsystem.fas"
;;; Loading #P"/usr/lib/ecl-16.1.3/cmp.fas"
Maxima 5.40.0 http://maxima.sourceforge.net
using Lisp ECL 16.1.3
Distributed under the GNU Public License. See the file COPYING.
Dedicated to the memory of William Schelter.
The function bug_report() provides bug reporting information.
(%i1) bug_report();

Please report bugs to:
    https://sourceforge.net/p/maxima/bugs
To report a bug, you must have a Sourceforge account.
Please include the following information with your bug report:
-------------------------------------------------------------
Maxima version: "5.40.0"
Maxima build date: "2017-06-07 20:33:24"
Host type: "x86_64-unknown-linux-gnu"
Lisp implementation type: "ECL"
Lisp implementation version: "16.1.3"
-------------------------------------------------------------
The above information is also reported by the function 'build_info()'.

(%o1) 
(%i2) domain:complex;
(%o2)                               complex
(%i3) assume(p>0,p<=1);
(%o3)                           [p > 0, p <= 1]
(%i4) sum((p*(1-p)^(i-1))^2,i,1,inf),simpsum=true;

Then i get several times

;;;
;;; Detected access to protected memory, also kwown as 'bus or segmentation fault'.
;;; Jumping to the outermost toplevel prompt
;;;

This bug seems to be related to this:
https://groups.google.com/forum/#!searchin/sage-devel/segmentation$20fault%7Csort:relevance/sage-devel/Yg-VgAtVGhI/1voJ2iN4BwAJ
But i could'nt find this reported here

[Robert Dodier]

It appears that the problem is caused by calling asksign. In the presence of assume(p > 0), asksign(p^2 - 2*p) leads to a stack overflow. However, the equivalent asksign((p - 2)*p) doesn't trigger the bug.

Tracing the Lisp function SIGN1 (via :lisp (trace sign1) or trace(?sign1)) shows that SIGN1 tries to break down the expression p^2 - 2*p into some pieces, but the analyzing the pieces leads back to p^2 - 2*p, so it cycles endlessly until it overflows the stack.

It appears that assume(p > 0) triggers the bug. Without that, the bug doesn't occur.

It appears that domain:complex and simpsum are not directly involved. I've updated the tags and summary to reflect my understanding of the bug.

The limit bug mentioned elsewhere:

domain : complex;
limit( (1/(c*n^6))^(log(n)/log(2/3)), n, inf);

is also a stack overflow but in the limit code, so I don't think it's the same problem.

Thanks for the bug report, it is very helpful.