#48 MAXHEADER constant is (still) too small

v1.2.0
closed-fixed
7
2007-08-02
2007-07-25
Ralf Hildebrandt
No

We had a lengthy discussion on the postfix-users mailinglist about this:

My initial problem with dkim-milter was this:

Jul 25 11:26:10 mail-ausfall dkim-filter[4550]: B93813DC16: dkim_eoh(): internal error from libdkim: header too large (max 4097)

and looking at the mail I see a user adding a shitload of E-Mail addresses to the To:/CC: headers, instead of using the Bcc: headers.

My users are surely not dumber than the average user out there.

I guess other users at other companies will also send mail with large lists of recipients. Their mail will fail as well.

Vicot Duchovni notes:
milter must support folded headers of up to 32k in length (Sendmail limits the total header size to this value, so headers larger than 32k are problematic already). Since milters are still primarily for Sendmail, the 32k size limit is natural. The next natural limit is 64k, which is the milter protocol packet size limit, this would accomodate possibly larger future header size limits in Sendmail.

Discussion

1 2 > >> (Page 1 of 2)
    • milestone: --> v1.2.0
    • assigned_to: nobody --> sm-msk
     
  • Logged In: YES
    user_id=1048957
    Originator: NO

    Try the attached patch. It's still experimental but so far it's passing my (admittedly simple) unit tests. It looks like the limitation is entirely in dkim-filter so far so I won't have to patch libdkim at all, but I'm still reviewing.
    File Added: PATCH

     
  • Logged In: YES
    user_id=1048957
    Originator: NO

    I was wrong; it appears some changes to libdkim will be needed. I'll post a better patch shortly.

     
    • priority: 5 --> 7
     
  • Logged In: YES
    user_id=1048957
    Originator: NO

    A new patch is attached. It's rather extensive and covers both dkim-filter and libdkim, but it does reduce use of MAXHEADER to being used to declare static buffers into which signature headers are generated. This means user-provided input is not subjected to compile-time limits.

    The patch adds and makes use of a "dstring" abstraction which is a dynamically-allocated string, grown as needed (with an optional maximum size). Analogs to strcat() and strcpy() are available.

    libdkim enforces no direct limits on how much dynamic space can be allocated overall. This means the caller (e.g. dkim-filter) needs to be conscious about how many headers are delivered by the MTA, otherwise an attacker could just send headers over and over without ever reaching the body, causing the application to allocate space until memory is exhausted. Sendmail caps this at 32k so the limit is imposed there. Just in case, dkim-filter also does this (i.e. I don't know what Postfix does).

    This test passes all of the unit tests in libdkim (including a new one designed specifically to exercise the new code) as well as some manual tests.
    File Added: PATCH

     
  • Logged In: YES
    user_id=1048957
    Originator: NO

    Corrected patch (previous version was an older one).
    File Added: PATCH

     
  • Tonni
    Tonni
    2007-07-31

    Logged In: YES
    user_id=1857224
    Originator: NO

    sm-msk's patch applied cleanly to source (attempted) built on up-to-date Fedora FC6, but the build failed; the requisite part:

    Making in /home/tonni/rpm/BUILD/dkim-milter-2.0.0/obj.Linux.2.6.20-1.2952.fc6.i686/libdkim
    make[2]: Entering directory `/u/home/tonni/rpm/BUILD/dkim-milter-2.0.0/obj.Linux.2.6.20-1.2952.fc6.i686/libdkim'
    cc -O2 -I. -I../../include -I/usr/include/db4 -I/usr/include -I../libar/ -D_FFR_HASH_BUFFERING -D_FFR_STATS -DSM_CONF_POLL=1 -DUSE_ARLIB -D_REENTRANT -DXP_MT -c -o dkim-canon.o dkim-canon.c
    dkim-canon.c: In function 'dkim_canon_signature':
    dkim-canon.c:1011: error: 'tmp' undeclared (first use in this function)
    dkim-canon.c:1011: error: (Each undeclared identifier is reported only once
    dkim-canon.c:1011: error: for each function it appears in.)
    make[2]: *** [dkim-canon.o] Error 1
    make[2]: Leaving directory `/u/home/tonni/rpm/BUILD/dkim-milter-2.0.0/obj.Linux.2.6.20-1.2952.fc6.i686/libdkim'
    make[1]: *** [/home/tonni/rpm/BUILD/dkim-milter-2.0.0/obj.Linux.2.6.20-1.2952.fc6.i686/libdkim/libdkim.a] Error 2
    make[1]: Leaving directory `/u/home/tonni/rpm/BUILD/dkim-milter-2.0.0/obj.Linux.2.6.20-1.2952.fc6.i686/dkim-filter'
    make: *** [all] Error 2
    error: Bad exit status from /var/tmp/rpm-tmp.60154 (%prep)

    RPM build errors:
    Bad exit status from /var/tmp/rpm-tmp.60154 (%prep)

    If the build fails on FC6, it will also fail on RHl5, which is the main interest (production machine).

    --Tonni

     
  • Tonni
    Tonni
    2007-07-31

    Logged In: YES
    user_id=1857224
    Originator: NO

    Sorry:

    cat /var/tmp/rpm-tmp.60154

    #!/bin/sh

    RPM_SOURCE_DIR="/home/tonni/rpm/SOURCES"
    RPM_BUILD_DIR="/home/tonni/rpm/BUILD"
    RPM_OPT_FLAGS="-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables"
    RPM_ARCH="i386"
    RPM_OS="linux"
    export RPM_SOURCE_DIR RPM_BUILD_DIR RPM_OPT_FLAGS RPM_ARCH RPM_OS
    RPM_DOC_DIR="/usr/share/doc"
    export RPM_DOC_DIR
    RPM_PACKAGE_NAME="dkim-milter"
    RPM_PACKAGE_VERSION="2.0.0"
    RPM_PACKAGE_RELEASE="2.fc6"
    export RPM_PACKAGE_NAME RPM_PACKAGE_VERSION RPM_PACKAGE_RELEASE
    RPM_BUILD_ROOT="/var/tmp/dkim-milter-2.0.0-2.fc6"
    export RPM_BUILD_ROOT

    set -x
    umask 022
    cd /home/tonni/rpm/BUILD
    LANG=C
    export LANG
    unset DISPLAY

    cd /home/tonni/rpm/BUILD
    rm -rf dkim-milter-2.0.0
    /bin/gzip -dc /home/tonni/rpm/SOURCES/dkim-milter-2.0.0.tar.gz | tar -xf -
    STATUS=$?
    if [ $STATUS -ne 0 ]; then
    exit $STATUS
    fi
    cd dkim-milter-2.0.0
    [ `/usr/bin/id -u` = '0' ] && /bin/chown -Rhf root .
    [ `/usr/bin/id -u` = '0' ] && /bin/chgrp -Rhf root .
    /bin/chmod -Rf a+rX,u+w,g-w,o-w .
    echo "Patch #0 (dkim-milter.patch):"
    patch -p1 -b --suffix .orig -s < /home/tonni/rpm/SOURCES/dkim-milter.patch
    echo "Patch #1 (dkim-milter-header.patch):"
    patch -p0 -b --suffix .orig -s < /home/tonni/rpm/SOURCES/dkim-milter-header.patch

    cd $RPM_BUILD_DIR/dkim-milter-2.0.0
    tar -zxvf /home/tonni/rpm/SOURCES/dk-milter-libdk-0.6.0-tar.gz
    cp /home/tonni/rpm/SOURCES/dkim-milter-site.config.m4 ./devtools/Site/site.config.m4
    cp /home/tonni/rpm/SOURCES/Makefile.m4.libdk ./libdk/Makefile.m4
    cd -

    make all

    exit 0

     
  • Logged In: YES
    user_id=1048957
    Originator: NO

    I can't seem to get this patch right. Serves me right for working so late. It looks like the dkim-canon.c portion of the patch was wrong, bringing you one version short of where it should be.

    The "tmp" in dkim-canon.c should be "tmphdr.hdr_text".

    ...but who knows what else I may have missed. I'll either re-generate the patch from CVS or re-activate the beta stuff later today. Sorry for the screw-up.

     
  • Tonni
    Tonni
    2007-07-31

    Logged In: YES
    user_id=1857224
    Originator: NO

    No problem, and there seems to be a *load* of Postfix milter stuff (not only dkim, but much pertinent thereto as well; much Cisco smtp fixup patches necessary, initiated by Jim Fenton and Mark Martinec on the Postfix ML, as well as hugeish Postfix milter patches) that still has to be brought to "kosher". The stuffs obviously still in its rfc4871 infancy, but basically we all know that dkim-milter more or less already works as designed, take your time ;)

    --Tonni

     
  • Tonni
    Tonni
    2007-07-31

    Logged In: YES
    user_id=1857224
    Originator: NO

    No problem, and there seems to be a *load* of Postfix milter stuff (not only dkim, but much pertinent thereto as well; much Cisco smtp fixup patches necessary, initiated by Jim Fenton and Mark Martinec on the Postfix ML, as well as hugeish Postfix milter patches) that still has to be brought to "kosher". The stuffs obviously still in its rfc4871 infancy, but basically we all know that dkim-milter more or less already works as designed, take your time ;)

    --Tonni

     
  • Logged In: YES
    user_id=1048957
    Originator: NO

    A revised patch is attached. I'm sending the new bits through internal code review here as well.
    File Added: PATCH

     
  • Tonni
    Tonni
    2007-08-01

    Logged In: YES
    user_id=1857224
    Originator: NO

    Well, the patch applies cleanly, and guess what!

    Wrote: /home/tonni/rpm/SRPMS/dkim-milter-2.0.0-2.fc6.src.rpm
    Wrote: /home/tonni/rpm/RPMS/i386/dkim-milter-2.0.0-2.fc6.i386.rpm

    Now to try it out - I guess dkim-milter should get a new version number ...

    --Tonni

     
  • Tonni
    Tonni
    2007-08-02

    Logged In: YES
    user_id=1857224
    Originator: NO

    I don't know whether this should be continued on the beta ML, but here goes, anyway.

    rpms built with the patched 2.0.0 source (2.0.1.dev) were installed on an FC6 test machine, a RHL4 i386 production machine (foolish, but then ...) and a RHL5 i386 pre-production machine. dkim-filter on all of these machines showed more or less the same behavior: On the FC6 and RHL4 machines it grabbed 80-90% of the (single) processor capacity (top) and held that. Signed messages sent to sa-test@sendmail.net on the RHL4 machine never validated, to autorespond@dk.elandsys.com sometimes, sometimes not (the reason was that the header fields - h - were most times not included in the dkim header field). Previously, with unpatched 2.0.0 they always validated. On the dual-processor RHL5 machine dkim-filter grabbed 200% (!) of the processor (top). Reverting to unpatched 2.0.0 cured the problems on all of the machines. That's one of the huge advantages of rpm, both building and installing/reinstalling rpms is a dead cinch.

    --Tonni

     
  • Logged In: YES
    user_id=1048957
    Originator: NO

    The unit tests didn't show anything like this. I'll try running it in production on my home machine (OpenBSD) and see if I can observe anything unusual.

     
  • Logged In: YES
    user_id=1048957
    Originator: NO

    I started the 2.0.1.dev daemon at home and sent off seven test messages to various autoresponders. After using 0.10 CPU seconds, it processed all the outbound messages and verified all the replies. Everything passed in both directions on all the replies I received (one of the autoresponders appears to be down).

    Does truss (or equivalent) or gdb indicate where it's spinning for you?

     
  • Tonni
    Tonni
    2007-08-02

    Logged In: YES
    user_id=1857224
    Originator: NO

    Hmmm ... this is going to take me a bit of time, I'll have to build a debug rpm along with the 2.0.1.dev one and will only try this out on (this) FC6 machine, which can't send to the validating sites. Give me the morning?

    --Tonni

     
  • Proposed patch #3

     
    Attachments
  • Logged In: YES
    user_id=1048957
    Originator: NO

    The new dstring stuff underwent code review today and
    there were some changes. Maybe after those (which I have
    running) the code is stable, and without (which you were
    using) it's not so stable.

    I'll attach a new patch which is the up-to-the-minute
    diff.

    File Added: PATCH

     
  • Tonni
    Tonni
    2007-08-02

    Logged In: YES
    user_id=1857224
    Originator: NO

    Ok, the latest patch version made all the difference and debugging wasn't necessary. After a couple of hours uptime dkim-filter with the latest patch runs normally on all 3 machines and all messages validate with sa-test@sendmail.net and autorespond@dk.elandsys.com.

    --Tonni

     
  • Logged In: YES
    user_id=1048957
    Originator: NO

    Excellent! Thanks for your persistence!

    v2.0.1 released.

     
    • status: open --> closed-fixed
     
  • Tonni
    Tonni
    2007-08-02

    Logged In: YES
    user_id=1857224
    Originator: NO

    Nope:

    make[2]: Leaving directory `/u/home/tonni/rpm/BUILD/dkim-milter-2.0.1/obj.Linux.2.6.20-1.2952.fc6.i686/libdk'
    cc -O2 -I. -I../../include -I/usr/include/db4 -I/usr/include -I../libdkim/ -I../libdk/ -I/usr/include/libmilter -D_FFR_HASH_BUFFERING -D_FFR_STATS -DSM_CONF_POLL=1 -DVERIFY_DOMAINKEYS -D_REENTRANT -DXP_MT -c -o config.o config.c
    cc -O2 -I. -I../../include -I/usr/include/db4 -I/usr/include -I../libdkim/ -I../libdk/ -I/usr/include/libmilter -D_FFR_HASH_BUFFERING -D_FFR_STATS -DSM_CONF_POLL=1 -DVERIFY_DOMAINKEYS -D_REENTRANT -DXP_MT -c -o dkim-ar.o dkim-ar.c
    cc -O2 -I. -I../../include -I/usr/include/db4 -I/usr/include -I../libdkim/ -I../libdk/ -I/usr/include/libmilter -D_FFR_HASH_BUFFERING -D_FFR_STATS -DSM_CONF_POLL=1 -DVERIFY_DOMAINKEYS -D_REENTRANT -DXP_MT -c -o dkim-filter.o dkim-filter.c
    In file included from dkim-filter.c:75:
    ../libdk/dk.h:36:1: warning: "MAXHEADER" redefined
    In file included from dkim-filter.c:68:
    ../libdkim/dkim.h:63:1: warning: this is the location of the previous definition
    dkim-filter.c: In function 'dkimf_report':
    dkim-filter.c:1602: error: 'ctx' undeclared (first use in this function)
    dkim-filter.c:1602: error: (Each undeclared identifier is reported only once
    dkim-filter.c:1602: error: for each function it appears in.)
    make[1]: *** [dkim-filter.o] Error 1
    make[1]: Leaving directory `/u/home/tonni/rpm/BUILD/dkim-milter-2.0.1/obj.Linux.2.6.20-1.2952.fc6.i686/dkim-filter'
    make: *** [all] Error 2
    error: Bad exit status from /var/tmp/rpm-tmp.81051 (%prep)

    RPM build errors:
    Bad exit status from /var/tmp/rpm-tmp.81051 (%prep)

    1186 [tonni:tru.leerlingen] /home/tonni/rpm/SPECS $ cat /var/tmp/rpm-tmp.81051
    #!/bin/sh

    RPM_SOURCE_DIR="/home/tonni/rpm/SOURCES"
    RPM_BUILD_DIR="/home/tonni/rpm/BUILD"
    RPM_OPT_FLAGS="-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables"
    RPM_ARCH="i386"
    RPM_OS="linux"
    export RPM_SOURCE_DIR RPM_BUILD_DIR RPM_OPT_FLAGS RPM_ARCH RPM_OS
    RPM_DOC_DIR="/usr/share/doc"
    export RPM_DOC_DIR
    RPM_PACKAGE_NAME="dkim-milter"
    RPM_PACKAGE_VERSION="2.0.1"
    RPM_PACKAGE_RELEASE="1.fc6"
    export RPM_PACKAGE_NAME RPM_PACKAGE_VERSION RPM_PACKAGE_RELEASE
    RPM_BUILD_ROOT="/var/tmp/dkim-milter-2.0.1-1.fc6"
    export RPM_BUILD_ROOT

    set -x
    umask 022
    cd /home/tonni/rpm/BUILD
    LANG=C
    export LANG
    unset DISPLAY

    cd /home/tonni/rpm/BUILD
    rm -rf dkim-milter-2.0.1
    /bin/gzip -dc /home/tonni/rpm/SOURCES/dkim-milter-2.0.1.tar.gz | tar -xf -
    STATUS=$?
    if [ $STATUS -ne 0 ]; then
    exit $STATUS
    fi
    cd dkim-milter-2.0.1
    [ `/usr/bin/id -u` = '0' ] && /bin/chown -Rhf root .
    [ `/usr/bin/id -u` = '0' ] && /bin/chgrp -Rhf root .
    /bin/chmod -Rf a+rX,u+w,g-w,o-w .
    echo "Patch #0 (dkim-milter.patch):"
    patch -p1 -b --suffix .orig -s < /home/tonni/rpm/SOURCES/dkim-milter.patch

    cd $RPM_BUILD_DIR/dkim-milter-2.0.1
    tar -zxvf /home/tonni/rpm/SOURCES/dk-milter-libdk-0.6.0-tar.gz
    cp /home/tonni/rpm/SOURCES/dkim-milter-site.config.m4 ./devtools/Site/site.config.m4
    cp /home/tonni/rpm/SOURCES/Makefile.m4.libdk ./libdk/Makefile.m4
    cd -

    make all

     
  • Tonni
    Tonni
    2007-08-02

    Logged In: YES
    user_id=1857224
    Originator: NO

    Nope:

    make[2]: Leaving directory `/u/home/tonni/rpm/BUILD/dkim-milter-2.0.1/obj.Linux.2.6.20-1.2952.fc6.i686/libdk'
    cc -O2 -I. -I../../include -I/usr/include/db4 -I/usr/include -I../libdkim/ -I../libdk/ -I/usr/include/libmilter -D_FFR_HASH_BUFFERING -D_FFR_STATS -DSM_CONF_POLL=1 -DVERIFY_DOMAINKEYS -D_REENTRANT -DXP_MT -c -o config.o config.c
    cc -O2 -I. -I../../include -I/usr/include/db4 -I/usr/include -I../libdkim/ -I../libdk/ -I/usr/include/libmilter -D_FFR_HASH_BUFFERING -D_FFR_STATS -DSM_CONF_POLL=1 -DVERIFY_DOMAINKEYS -D_REENTRANT -DXP_MT -c -o dkim-ar.o dkim-ar.c
    cc -O2 -I. -I../../include -I/usr/include/db4 -I/usr/include -I../libdkim/ -I../libdk/ -I/usr/include/libmilter -D_FFR_HASH_BUFFERING -D_FFR_STATS -DSM_CONF_POLL=1 -DVERIFY_DOMAINKEYS -D_REENTRANT -DXP_MT -c -o dkim-filter.o dkim-filter.c
    In file included from dkim-filter.c:75:
    ../libdk/dk.h:36:1: warning: "MAXHEADER" redefined
    In file included from dkim-filter.c:68:
    ../libdkim/dkim.h:63:1: warning: this is the location of the previous definition
    dkim-filter.c: In function 'dkimf_report':
    dkim-filter.c:1602: error: 'ctx' undeclared (first use in this function)
    dkim-filter.c:1602: error: (Each undeclared identifier is reported only once
    dkim-filter.c:1602: error: for each function it appears in.)
    make[1]: *** [dkim-filter.o] Error 1
    make[1]: Leaving directory `/u/home/tonni/rpm/BUILD/dkim-milter-2.0.1/obj.Linux.2.6.20-1.2952.fc6.i686/dkim-filter'
    make: *** [all] Error 2
    error: Bad exit status from /var/tmp/rpm-tmp.81051 (%prep)

    RPM build errors:
    Bad exit status from /var/tmp/rpm-tmp.81051 (%prep)

    1186 [tonni:tru.leerlingen] /home/tonni/rpm/SPECS $ cat /var/tmp/rpm-tmp.81051
    #!/bin/sh

    RPM_SOURCE_DIR="/home/tonni/rpm/SOURCES"
    RPM_BUILD_DIR="/home/tonni/rpm/BUILD"
    RPM_OPT_FLAGS="-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables"
    RPM_ARCH="i386"
    RPM_OS="linux"
    export RPM_SOURCE_DIR RPM_BUILD_DIR RPM_OPT_FLAGS RPM_ARCH RPM_OS
    RPM_DOC_DIR="/usr/share/doc"
    export RPM_DOC_DIR
    RPM_PACKAGE_NAME="dkim-milter"
    RPM_PACKAGE_VERSION="2.0.1"
    RPM_PACKAGE_RELEASE="1.fc6"
    export RPM_PACKAGE_NAME RPM_PACKAGE_VERSION RPM_PACKAGE_RELEASE
    RPM_BUILD_ROOT="/var/tmp/dkim-milter-2.0.1-1.fc6"
    export RPM_BUILD_ROOT

    set -x
    umask 022
    cd /home/tonni/rpm/BUILD
    LANG=C
    export LANG
    unset DISPLAY

    cd /home/tonni/rpm/BUILD
    rm -rf dkim-milter-2.0.1
    /bin/gzip -dc /home/tonni/rpm/SOURCES/dkim-milter-2.0.1.tar.gz | tar -xf -
    STATUS=$?
    if [ $STATUS -ne 0 ]; then
    exit $STATUS
    fi
    cd dkim-milter-2.0.1
    [ `/usr/bin/id -u` = '0' ] && /bin/chown -Rhf root .
    [ `/usr/bin/id -u` = '0' ] && /bin/chgrp -Rhf root .
    /bin/chmod -Rf a+rX,u+w,g-w,o-w .
    echo "Patch #0 (dkim-milter.patch):"
    patch -p1 -b --suffix .orig -s < /home/tonni/rpm/SOURCES/dkim-milter.patch

    cd $RPM_BUILD_DIR/dkim-milter-2.0.1
    tar -zxvf /home/tonni/rpm/SOURCES/dk-milter-libdk-0.6.0-tar.gz
    cp /home/tonni/rpm/SOURCES/dkim-milter-site.config.m4 ./devtools/Site/site.config.m4
    cp /home/tonni/rpm/SOURCES/Makefile.m4.libdk ./libdk/Makefile.m4
    cd -

    make all

     
  • Logged In: YES
    user_id=1048957
    Originator: NO

    I'm confused. dkim-filter.c:1602 for me (and in the release tarball) is a line that says simply "return". There's no reference to "ctx" on or near that line.

     
1 2 > >> (Page 1 of 2)