eaccelerator-developers

Re: [eAccelerator-developers] low frequency segfault on AMD64

[Ramon v. A. <ra...@hy...>]

I'm opening a new thread for this to prevent thread-hijjacking, quoting 
original message in full.

Oliver Schonrock wrote:

> We have installed 0.9.5-beta1 on an AMD64 production machine.
>
> This machine is quite busy and runs with 128MB of eA cache.
>
> FreeBSD 5.3-RELEASE FreeBSD 5.3-RELEASE #4:
> Fri May  6 14:07:48 BST 2005 amd64
>
> Apache/2.0.55 (FreeBSD)
>
> eA installed from the current 0.9.4rc_1 ports with a tweaked Makefile 
> (version number and MD5 changed, and all patches removed).
>
> PHP 5.1.2 (cli) (built: Mar  8 2006 00:09:12)
> Copyright (c) 1997-2006 The PHP Group
> Zend Engine v2.1.0, Copyright (c) 1998-2006 Zend Technologies
>     with eAccelerator v0.9.5-beta1, Copyright (c) 2004-2005 
> eAccelerator, by eAccelerator
>
> we are getting about 30 of these per day:
>
> [Thu Mar 09 17:24:01 2006] [notice] child pid 19220 exit signal 
> Segmentation fault (11)

We're getting them on both our AMD64's and our Intel-Xeon's (32bit).
Roughly 30-40 a day. Haven't tried 0.9.5_beta1 on our 32bit hosts though.

> The exact same configuration is running on a 32bit Athlon production 
> machine with no segfaults.
>
> I know this is not very helpful, because I haven't provided a trace, 
> but I don't have a way of doing that on this production machine. We 
> haven't been able to track down the script that causes the segfault 
> either, as far as we can tell it seems "random".
>
> We would like to help track this down. Are there any known issues on 
> 64bit in 0.9.5-beta1?
>
> Could this be one of those famous "concurrency isssues"?

AFAIK there's an outstanding bug that eaccelerator causes apache to 
segfault with signal 11 under load.

>
> Are there any tips anyone has on how we can narrow down the 
> problem/provide a trace given this is a production environment (we 
> don't have a 64bit dev machine)?

Any tips on producing a helpful trace to debug this would be useful, I 
can setup a host in both environments (32/64 bit) for tracing if 
neccesary. For the time being those would be on php-4.3.11 though.
We're working on an upgrade to php5.

Regards,

Ramon

-- 
mail:    ra...@hy...
gsm:     +31 (0) 6 48261300
msn:     rja...@ho...
www:     http://ramon71.hyves.nl

-- 
mail:    ra...@hy...
gsm:     +31 (0) 6 48261300
msn:     rja...@ho...
www:     http://ramon71.hyves.nl

Re: [eAccelerator-developers] low frequency segfault on AMD64

[Oliver S. <ol...@re...>]

OK. 5 days later and we are a lot closer to being able to narrow this down.

Firstly we now have both the 32bit and 64bit servers running without 
segfaults.

Note again our environment:

 >> FreeBSD 5.3-RELEASE FreeBSD 5.3-RELEASE #4:
 >> Fri May  6 14:07:48 BST 2005 amd64
 >>
 >> Apache/2.0.55 (FreeBSD)
 >>
 >> eA installed from the current 0.9.4rc_1 ports with a tweaked Makefile
 >> (version number and MD5 changed, and all patches removed).
 >>
 >> PHP 5.1.2 (cli) (built: Mar  8 2006 00:09:12)
 >> Copyright (c) 1997-2006 The PHP Group
 >> Zend Engine v2.1.0, Copyright (c) 1998-2006 Zend Technologies
 >>     with eAccelerator v0.9.5-beta1, Copyright (c) 2004-2005

Here is what we did to achieve that:

- In FreeBSD you need to set both kern.ipc.shmall and kern.ipc.shmmax 
with kern.ipc.shmall = kern.ipc.shmmax / PageSize

(you can see the page size as the last figure in the output below, it is 
1024 for 32bit machines and 2048 64bit machines).

for a 32bit machine the settings for 256MB of shared mem look like this:

root@piha# sysctl -a | egrep 'shm|Type'
kern.ipc.shmmax: 268435456
kern.ipc.shmmin: 1
kern.ipc.shmmni: 192
kern.ipc.shmseg: 128
kern.ipc.shmall: 262144
kern.ipc.shm_use_phys: 0
kern.ipc.shm_allow_removed: 0
         Type  InUse MemUse HighUse Requests  Size(s)
           shm    38    49K    165K  5083325  1024


The above setting will allow you to run 128MB of eA shared mem:
eaccelerator.shm_size="128"

AND still be able to

apachectl graceful

without crashing apache (ie you need 2x the shm for graceful). If you 
have less shm configured in kernel or have the kern.ipc.shmall 
configured wrong apache will fail to restart, because eA cannot allocate 
the shm it needs.


HOWEVER!!

*** if we use graceful will get these low freqency segfaults ***

also if we use

apachectl restart

we still get the segfaults!!

the only reliable way we have been able to restart apache without 
getting segfaults is with this script:

<script>
#!/usr/local/bin/bash
# restart apache slowly while cleaning eaccelerator cache

/usr/local/etc/rc.d/apache2.sh stop
find /tmp/eaccelerator -type f -exec rm {} \;
/usr/local/etc/rc.d/apache2.sh start
</script>

if we use that script...we get ZERO segfaults! 100% reliability...which 
is much better than anything we have ever had with eA0.9.x.

BUT if we use any other restart method...we get the low freq segfaults 
coming in within a few minutes of the (graceful or not) restart.

Now I am hoping that this information will help people who are:
a) getting low freq segfaults
b) operating 64bit machines (although we actually get the same symptoms 
on both 32bit and 64bit so the arch issue might be a red herring)
c) are using FreeBSD
d) have highly loaded production servers

I am also hoping that this may trigger a discussion/further tests 
together with the eA dev team to figure out why we can't use

apachectl graceful
apachectl restart
/usr/local/etc/rc.d/apache2.sh restart

all of which would be better ways of restarting.

One final point, which I think is also a red herring is that we are now 
running SO MUCH shm on these machines the eA never runs out (the most it 
reaches is about 60% full). We don't think this is related to the 
segfaults because we were getting them both on machines with tons of shm 
and ones which were "swapping").


Hope this is useful and look forward to feedback...

Oliver