In version 1.8.0, configure with option -enable-keygen, i got that error.
File rsa.c, line 388
buf = XCALLOC(1, len, heap);
With this version, I always get segfault when using dtls + ssl extra :(
You can check this:
1. configure -enable-opensslExtra -enable-dtls
2. cd examples/echoserver -> ./echoserver
3. cd examples/echoclient -> ./echoclient
-> Then both client and server die.
For the XCALLOC problem you can change the line to:
buf = XMALLOC(len, heap, DYNAMIC_TYPE_RSA);
and after the NULL test add:
XMEMSET(buf, 0, len);
to get the calloc functionality and to keep valgrind quiet.
Thanks for the report.
For the DTLS problem, you can fix it by changing this line in cyassl_int.h (about line 453) from:
#define RECORD_SIZE 128
#define RECORD_SIZE 1500
I've checked in a version to CVS that makes sure when DTLS is enabled that the static buffer size is the expected MTU. DTLS can be greedy in reading buffers since there's a fixed plaintext size, the MTU. CyaSSL now employs small static buffers that grow dynamically when needed in TLS mode. DTLS will need to have a minimum record size of the MTU and we'll add that to the docs as well.