Re: [eAccelerator-developers] low frequency segfault on AMD64

[Oliver S. <ol...@re...>]

OK. 5 days later and we are a lot closer to being able to narrow this down.

Firstly we now have both the 32bit and 64bit servers running without 
segfaults.

Note again our environment:

 >> FreeBSD 5.3-RELEASE FreeBSD 5.3-RELEASE #4:
 >> Fri May  6 14:07:48 BST 2005 amd64
 >>
 >> Apache/2.0.55 (FreeBSD)
 >>
 >> eA installed from the current 0.9.4rc_1 ports with a tweaked Makefile
 >> (version number and MD5 changed, and all patches removed).
 >>
 >> PHP 5.1.2 (cli) (built: Mar  8 2006 00:09:12)
 >> Copyright (c) 1997-2006 The PHP Group
 >> Zend Engine v2.1.0, Copyright (c) 1998-2006 Zend Technologies
 >>     with eAccelerator v0.9.5-beta1, Copyright (c) 2004-2005

Here is what we did to achieve that:

- In FreeBSD you need to set both kern.ipc.shmall and kern.ipc.shmmax 
with kern.ipc.shmall = kern.ipc.shmmax / PageSize

(you can see the page size as the last figure in the output below, it is 
1024 for 32bit machines and 2048 64bit machines).

for a 32bit machine the settings for 256MB of shared mem look like this:

root@piha# sysctl -a | egrep 'shm|Type'
kern.ipc.shmmax: 268435456
kern.ipc.shmmin: 1
kern.ipc.shmmni: 192
kern.ipc.shmseg: 128
kern.ipc.shmall: 262144
kern.ipc.shm_use_phys: 0
kern.ipc.shm_allow_removed: 0
         Type  InUse MemUse HighUse Requests  Size(s)
           shm    38    49K    165K  5083325  1024


The above setting will allow you to run 128MB of eA shared mem:
eaccelerator.shm_size="128"

AND still be able to

apachectl graceful

without crashing apache (ie you need 2x the shm for graceful). If you 
have less shm configured in kernel or have the kern.ipc.shmall 
configured wrong apache will fail to restart, because eA cannot allocate 
the shm it needs.


HOWEVER!!

*** if we use graceful will get these low freqency segfaults ***

also if we use

apachectl restart

we still get the segfaults!!

the only reliable way we have been able to restart apache without 
getting segfaults is with this script:

<script>
#!/usr/local/bin/bash
# restart apache slowly while cleaning eaccelerator cache

/usr/local/etc/rc.d/apache2.sh stop
find /tmp/eaccelerator -type f -exec rm {} \;
/usr/local/etc/rc.d/apache2.sh start
</script>

if we use that script...we get ZERO segfaults! 100% reliability...which 
is much better than anything we have ever had with eA0.9.x.

BUT if we use any other restart method...we get the low freq segfaults 
coming in within a few minutes of the (graceful or not) restart.

Now I am hoping that this information will help people who are:
a) getting low freq segfaults
b) operating 64bit machines (although we actually get the same symptoms 
on both 32bit and 64bit so the arch issue might be a red herring)
c) are using FreeBSD
d) have highly loaded production servers

I am also hoping that this may trigger a discussion/further tests 
together with the eA dev team to figure out why we can't use

apachectl graceful
apachectl restart
/usr/local/etc/rc.d/apache2.sh restart

all of which would be better ways of restarting.

One final point, which I think is also a red herring is that we are now 
running SO MUCH shm on these machines the eA never runs out (the most it 
reaches is about 60% full). We don't think this is related to the 
segfaults because we were getting them both on machines with tons of shm 
and ones which were "swapping").


Hope this is useful and look forward to feedback...

Oliver