#223 Libtspi der/ber encoding/decoding machinery fails with libssl1.1

[Tomasz Kazimierczak]

Libtspi der/ber encoding/decoding machinery fails with libssl1.1. The reason why this happens is explained more or less here: https://mta.openssl.org/pipermail/openssl-dev/2016-February/005100.html. After investigating the problem and trousers source code it is clear that there's a major bug in function Tspi_EncodeDER_TssBlob() and in other parts of the code, where macro TSS_BUILD_ASN1_OPENSSL is used, because in fact this macro is never defined!

This macro was introduced in the code with the commit 460ff3112c16
(https://sourceforge.net/p/trousers/trousers/ci/460ff3112c16c54f3dcfd4b55bde1aaefa331330)
with such comment:

    /* The TSS working group has stated that the ASN1 encoding will be done in a
     * specific way that generates an ASN1 encoding that is exactly 20 bytes
     * larger than the blob being encoded.
     *
     * OpenSSL uses the smallest number of bytes possible to encode and object
     * and as a result cannot be used to perform the encoding.  The encoding
     * must be done manually.
     *
     * The 20 byte fixed header will result in issues for objects greater than
     * 2^16 in size since some fields are now limited to 16-bit lengths.
     */

So i don't know whether not defining macro was done on purpose,
but for sure "manual" encoding doesn't go along with openssl1.1 decoder - to be exact
since commit: 6c5b6cb0 (ASN1 INTEGER refactor) in openssl project.
It's just not compatible any more.

Simple fix is to declare missing macro, e.g. in configure.ac:

diff --git a/configure.ac b/configure.ac 
index b9626af..5d590fc 100644 
--- a/configure.ac
+++ b/configure.ac 
@@ -247,6 +247,7 @@ if test "x$API" = "x1.2"; then
        AM_CONDITIONAL(TSS_BUILD_TICK, true)
        AM_CONDITIONAL(TSS_BUILD_TRANSPORT, true)
        AM_CONDITIONAL(TSS_BUILD_ASN1, true)

+       AC_DEFINE([TSS_BUILD_ASN1_OPENSSL], 1, [define build ASN1 with openssl...])
        AM_CONDITIONAL(TSS_BUILD_NV, true)
        AM_CONDITIONAL(TSS_BUILD_AUDIT, true)
        AM_CONDITIONAL(TSS_BUILD_SEALX, true)

but I'm not sure if that's the correct way for you...
Would be really nice to have this issue fixed.