Share

Kerberos Module For Apache

Tracker: Bugs

5 mod_auth_kerb+worker=segfault - ID: 1971514
Last Update: Comment added ( baalberith )

It appears that mod_auth_kerb isn't thread safe as segfaults happen when
using apache's worker MPM, but not with the prefork MPM. I've verified
this on Solaris 10, RHEL 5 and RHEL 4. Additionally I've seen postings by
other people which note the same.

Any chance mod_auth_kerb could be updated to be thread safe? It's the only
module I have on any of my webservers that requires prefork, so it actually
hinders performance quite a bit.

It's pretty easy to show the problem, just setup mod_auth_kerb to protect a
directory, then use apache ab to generate a lot of connections to make the
errors happen faster:

ab -n 1000 -c 1000 -A username:password http://localhost/test

Here are some entries from error_log on RHEL5 (on Solaris obviously the
extra glibc debugging info doesn't show up):

[Sat May 24 14:39:11 2008] [notice] Apache/2.2.8 (Unix) mod_ssl/2.2.8
OpenSSL/0.9.8g mod_auth_kerb/5.3 configured -- resuming normal operations
httpd: cc_memory.c:210: krb5_mcc_destroy: Assertion
`(&(&(&d->lock)->os)->n)->initialized == K5_MUTEX_DEBUG_INITIALIZED'
failed.
[Sat May 24 14:40:22 2008] [notice] child pid 26529 exit signal Aborted
(6)
[Sat May 24 14:40:58 2008] [error] server reached MaxClients setting,
consider raising the MaxClients setting
[Sat May 24 14:40:59 2008] [notice] child pid 26503 exit signal
Segmentation fault (11)
[Sat May 24 14:41:02 2008] [notice] child pid 26501 exit signal
Segmentation fault (11)
[Sat May 24 14:41:03 2008] [notice] child pid 26733 exit signal
Segmentation fault (11)
*** glibc detected *** /bin/httpd: double free or corruption (fasttop):
0x9570ddf8 ***
======= Backtrace: =========
/lib/libc.so.6[0x1aeb16]
/lib/libc.so.6(cfree+0x90)[0x1b2030]
/lib/libkrb5.so.3(krb5_free_principal+0x85)[0x30d225]
/lib/libkrb5.so.3[0x2f1d42]
/lib/libkrb5.so.3[0x2f25fd]
/lib/libkrb5.so.3(krb5_cc_destroy+0x19)[0x2f83c9]
/modules/mod_auth_kerb.so[0xf33061]
/modules/mod_auth_kerb.so[0xf3464d]
/bin/httpd(ap_run_check_user_id+0x46)[0x807907b]
/bin/httpd(ap_process_request_internal+0x319)[0x8079b01]
/bin/httpd(ap_process_request+0x4f)[0x80ab128]
/bin/httpd[0x80a7c03]
/bin/httpd(ap_run_process_connection+0x46)[0x80872d3]
/bin/httpd(ap_process_connection+0x58)[0x80876e6]
/bin/httpd[0x80c719c]
/bin/httpd[0x80c7a79]
/lib/libapr-1.so.0[0x6b57b4]
/lib/libpthread.so.0[0x91345b]
/lib/libc.so.6(clone+0x5e)[0x216c4e]
======= Memory map: ========
*** glibc detected *** /bin/httpd: double free or corruption (fasttop):
0x95700668 ***
======= Backtrace: =========
/lib/libc.so.6[0x1aeb16]
/lib/libc.so.6(cfree+0x90)[0x1b2030]
/lib/libkrb5.so.3(krb5_free_principal+0x60)[0x30d200]
/lib/libkrb5.so.3[0x2f1d42]
/lib/libkrb5.so.3[0x2f1d79]
/lib/libkrb5.so.3(krb5_cc_initialize+0x20)[0x2f83a0]
/modules/mod_auth_kerb.so[0xf32386]
/modules/mod_auth_kerb.so[0xf32e00]
/modules/mod_auth_kerb.so[0xf3464d]
/bin/httpd(ap_run_check_user_id+0x46)[0x807907b]
/bin/httpd(ap_process_request_internal+0x319)[0x8079b01]
/bin/httpd(ap_process_request+0x4f)[0x80ab128]
/bin/httpd[0x80a7c03]
/bin/httpd(ap_run_process_connection+0x46)[0x80872d3]
/bin/httpd(ap_process_connection+0x58)[0x80876e6]
/bin/httpd[0x80c719c]
/bin/httpd[0x80c7a79]
/lib/libapr-1.so.0[0x6b57b4]
/lib/libpthread.so.0[0x91345b]
/lib/libc.so.6(clone+0x5e)[0x216c4e]
======= Memory map: ========
[Sat May 24 14:41:06 2008] [notice] child pid 26827 exit signal Aborted
(6)
[Sat May 24 14:41:06 2008] [notice] child pid 26961 exit signal Aborted
(6)
*** glibc detected *** /bin/httpd: double free or corruption (fasttop):
0x94d09458 ***
======= Backtrace: =========
/lib/libc.so.6[0x1aeb16]
/lib/libc.so.6(cfree+0x90)[0x1b2030]
/lib/libkrb5.so.3(krb5_free_principal+0x85)[0x30d225]
/lib/libkrb5.so.3[0x2f1d42]
/lib/libkrb5.so.3[0x2f25fd]
/lib/libkrb5.so.3(krb5_cc_destroy+0x19)[0x2f83c9]
/modules/mod_auth_kerb.so[0xf33061]
/modules/mod_auth_kerb.so[0xf3464d]
/bin/httpd(ap_run_check_user_id+0x46)[0x807907b]
/bin/httpd(ap_process_request_internal+0x319)[0x8079b01]
/bin/httpd(ap_process_request+0x4f)[0x80ab128]
/bin/httpd[0x80a7c03]
/bin/httpd(ap_run_process_connection+0x46)[0x80872d3]
/bin/httpd(ap_process_connection+0x58)[0x80876e6]
/bin/httpd[0x80c719c]
/bin/httpd[0x80c7a79]
/lib/libapr-1.so.0[0x6b57b4]
/lib/libpthread.so.0[0x91345b]
/lib/libc.so.6(clone+0x5e)[0x216c4e]
======= Memory map: ========
[Sat May 24 14:41:07 2008] [notice] child pid 26894 exit signal
Segmentation fault (11)
[Sat May 24 14:41:07 2008] [notice] child pid 27028 exit signal Aborted
(6)
[Sat May 24 14:41:14 2008] [notice] child pid 27162 exit signal
Segmentation fault (11)
[Sat May 24 14:41:14 2008] [notice] child pid 27164 exit signal
Segmentation fault (11)


Nobody/Anonymous ( nobody ) - 2008-05-24 21:52

5

Closed

Fixed

Matej Pristak

mod_auth_kerb

Version 5.*

Public


Comments ( 8 )




Date: 2008-10-04 08:58
Sender: baalberith

fixed in 1.142, thanks for comments


Date: 2008-09-08 15:02
Sender: nbartos


I am using the kerberos supplied by redhat for RHEL 4/5. These are the
respective versions:

RHEL 4: krb5-devel-1.3.4-60.el4
RHEL 5: krb5-devel-1.6.1-25.el5

On the solaris machines we were using mit kerberos 1.5.1, which was
installed by us (not the vendor). I have upgraded that to 1.6.3, so now it
appears to work fine on the solaris build as well.

In my current build I just have a conditional to exclude applying the
patch on RHEL 4, which is find in my environment. Any new systems I really
care about are on RHEL 5 or Solaris 10, which now work fine with the patch.
I can deal with using prefork on the older RHEL 4 boxes until they get
upgraded.

Assuming you are correct about the threading issues in the earlier
kerberos versions, then I would probably just go for having configure
detect if it can use the call, otherwise do it the non-threadsafe way and
have configure print out a warning so that whoever is compiling it knows
it.



Date: 2008-09-08 08:45
Sender: kourilProject Admin


What Kerberos implementations do you use on the RHEL 4 and Solaris
installation, which don't contain the call? Please note, that AFAIK older
MIT/Heimdal implementations weren't thread-safe so you'd be probably run
into problems anyway.


Date: 2008-09-06 00:10
Sender: nbartos


Unfortunately this fix did not work on RHEL 4 or my particular Solaris
install due to krb5_cc_new_unique not being available. I can probably do
some OS upgrades to fix the issue for me, however it may not be a good idea
to push this out globally as-is for compatibility. I would recommend that
either a more compatible way be found, or at least have a test in the
configure script to see if the function is available and then use it only
in that case.


Date: 2008-09-05 15:56
Sender: nbartos


As far as I can tell that change fixes the problem. I have only tested on
RHEL5 so far (RHEL4 & Solaris testing still to come), but everything seems
to work very well. I will start putting this up on production servers and
see if there are any issues with normal usage. Below is the patch I am
using.


$ cat mod_auth_kerb-5.3-threading.patch
diff -U3 -r mod_auth_kerb-5.3.orig/src/mod_auth_kerb.c
mod_auth_kerb-5.3/src/mod_auth_kerb.c
--- mod_auth_kerb-5.3.orig/src/mod_auth_kerb.c 2008-09-04
10:41:49.000000000 -0700
+++ mod_auth_kerb-5.3/src/mod_auth_kerb.c 2008-09-05
08:19:50.000000000 -0700
@@ -564,7 +564,7 @@
} else
keytab = ap_req_keytab;

- ret = krb5_cc_resolve(context, "MEMORY:", &local_ccache);
+ ret = krb5_cc_new_unique(context, "MEMORY", NULL, &local_ccache);
if (ret) {
log_rerror(APLOG_MARK, APLOG_DEBUG, 0, r,
"krb5_cc_resolve() failed when verifying KDC");
@@ -711,7 +711,7 @@
goto end;
}

- ret = krb5_cc_resolve(context, "MEMORY:", &ret_ccache);
+ ret = krb5_cc_new_unique(context, "MEMORY", NULL, &ret_ccache);
if (ret) {
log_rerror(APLOG_MARK, APLOG_ERR, 0, r,
"generating new memory ccache failed: %s",



Date: 2008-09-05 12:13
Sender: kourilProject Admin


The very same issues seems to have been discussed at the krbdev
mailinglist. Could you please have a look at
http://mailman.mit.edu/pipermail/krbdev/2008-August/006832.html and try the
remedy described there and report back the result? If that worked we would
change the module code accordingly.


Date: 2008-09-01 07:59
Sender: jonas_thungren


We're experiencing the exact same issue.



Date: 2008-08-27 01:08
Sender: nbartos


It should be noted that this problem also occurs when using the latest MIT
(1.6.3) and heimdal (1.2). After a while I was wondering if there might
be threading issues with MIT kerberos, but sadly it appears to be the
apache module itself. If it helps, here's output from tests ran when
compiled against heimdal 1.2:

[Tue Aug 26 17:56:23 2008] [notice] child pid 20794 exit signal
Segmentation fault (11)
[Tue Aug 26 17:56:25 2008] [error] [client 10.4.5.13] failed to verify
krb5 credentials: No such file or directory
[Tue Aug 26 17:56:26 2008] [notice] child pid 20790 exit signal
Segmentation fault (11)
[Tue Aug 26 17:56:36 2008] [notice] child pid 20792 exit signal
Segmentation fault (11)
*** glibc detected *** /bin/httpd: double free or corruption (out):
0x94303588 ***
======= Backtrace: =========
/lib/libc.so.6[0xa97b16]
/lib/libc.so.6(cfree+0x90)[0xa9b030]
/lib/libasn1.so.8(der_free_general_string+0x1f)[0x19db7b]
/lib/libasn1.so.8(free_PrincipalName+0x3c)[0x1eb426]
/lib/libasn1.so.8(free_Principal+0x1d)[0x1eab85]
/lib/libkrb5.so.25(krb5_free_principal+0x23)[0x6798fb]
/lib/libkrb5.so.25[0x670848]
/lib/libkrb5.so.25(krb5_cc_destroy+0x29)[0x648951]
/modules/mod_auth_kerb.so[0xc70fdc]
/modules/mod_auth_kerb.so[0xc7116e]
/modules/mod_auth_kerb.so[0xc71d19]
/modules/mod_auth_kerb.so[0xc734de]
/bin/httpd(ap_run_check_user_id+0x46)[0x807a9d3]
/bin/httpd(ap_process_request_internal+0x313)[0x807b453]
/bin/httpd(ap_process_request+0x4f)[0x80acb74]
/bin/httpd[0x80a964f]
/bin/httpd(ap_run_process_connection+0x46)[0x8088c23]
/bin/httpd(ap_process_connection+0x58)[0x8089036]
/bin/httpd[0x80c8f24]
/bin/httpd[0x80c9801]
/lib/libapr-1.so.0[0x4f8bf8]
/lib/libpthread.so.0[0xba845b]
/lib/libc.so.6(clone+0x5e)[0xaffc4e]
======= Memory map: ========
[Tue Aug 26 17:56:38 2008] [notice] child pid 21005 exit signal Aborted
(6)


Log in to comment.




Attached File

No Files Currently Attached

Changes ( 4 )

Field Old Value Date By
status_id Open 2008-10-04 08:58 baalberith
resolution_id None 2008-10-04 08:58 baalberith
close_date - 2008-10-04 08:58 baalberith
assigned_to nobody 2008-09-05 12:13 kouril