[Sbcl-devel] sb-gmp, fragility, new allocation scheme

[Stephan F. <def...@go...>]

Hello,

some users may have noticed, that sb-gmp can be quite fragile at times. 
This is in most part the result of internal (re)allocation from within 
GMP if the Lisp-side pre-allocated buffers are not big enough. Sometimes 
there is no way to prea-llocate in advance which was solved by copying 
from C to SBCL space with a factor 3 penalty.

A have toyed with using the custom allocation method of GMP to pull all 
allocation to the Lisp side (apart from temporary/dynamic-extent 
with-alien C structure setup before calling into C).

This new allocation scheme lives in the 'newalloc' branch of the sb-gmp 
development repository [1].

The new method works now well for me (on amd64) and I would appreciate 
feedback from others, especially from people who have problems with the 
current version in SBCL, whether this new approach yields better, stable 
results when exercising the test suite (use the one from the repository 
branch).

Note, that this version does not yet work in conjunction with sb-mpfr 
(or only by accident).


An overview of the pros and cons of the old and new method:

old - pros:
- mostly working
- page pinnig when calling into C

old - cons:
- two methods of memory handling
   * memory fault when pre-allocation assumption is violated
   * factor 3 copying penalty when handling allocation within GMP
   * several different implementations for each case with different 
corner cases
- finalizers required for GMP random states


new - pros:
- unified and overall simpler memory handling, more data 
structures/logic moved to the Lisp side instead of alien structures
- appears to be more robust once I got the pointer-fiddling right
- all allocation requests are handled within SBCL
- robust against allocations and re-allocations from within GMP (no more 
memory faults due to allocation)
- avoids copying from C memory space in all cases (no more factor 3 
copying penalty)
- no finalizers required

new - cons:
- requires without-gcing instead of page-pinning only
- allocation calls from C into Lisp
- GMP random state handling got a bit more complicated (but is well 
excercised within the test suite and appears to be robust).


Regs,
Stephan


[1] https://github.com/sfrank/sb-gmp/tree/newalloc