Re: [eAccelerator-developers] eAccelerator scales towards useless

[dormando <dor...@ry...>]

Bart Vanbrabant wrote:
> dormando wrote:
>   
>> Hey,
>>
>> We recently moved our site from eA 0.9.3 to 0.9.5-beta1 while
>> upgrading from PHP 4.4.2 to 5.1.2.
>>
>> We discovered a pretty big dropoff in site speed after doing this.
>> Sadly I *cannot* verify if the same performance drop exists in our
>> PHP4 setup. What was most concerning is that our site speed dropped to
>> complete crap after I upped the shared memory segment size to 64
>> megabytes. Dropping it back down to 32 megabytes made the site usable
>> again.
>>
>> We start up PHP across the cluster with a fresh start, and the site is
>> blazing fast. After a few minutes things start slowing down pretty
>> badly. If we load up the eaccelerator() page and nail the "Clean"
>> button, or restart apache across the cluster, the site speeds up again.
>>
>> I wasted a few hours last week running a large array of tests to
>> narrow down what this could be. Here's the dump of information to get
>> past the usual level of "user is probably an idiot" questions;
>>
>> - The server has no harddrive, the code is loaded out of a ramdisk.
>> There is also ~800 megabytes of *unallocated system memory* before and
>> after all of these tests. There are no messages in dmesg about running
>> low or out of memory.
>> - The eaccelerator configuration is set to not use disk caching
>> - Optimize, check_mtime, enable, etc are set. compression is not set.
>> pruning is not set up during the tests.
>> - Apache was *clean restarted* between tests. eaccelerator() was used
>> to verify a clean start between *every single test*
>> - The tests encompass hitting five different pages from our site code,
>> each vary between the amount of CPU required vs DB queries. The
>> numbers I looked at most critically are the CPU-only page loads.
>> DB-heavy pages were used as a baseline, only.
>> - The box was fresh booted for the start of the test runs.
>> - There are *no files* being generated in /tmp/eaccelerator/ - I can
>> make them show up if I tell eA to, but during the tests none were
>> created. /tmp is also hosted in a tmpfs filesystem.
>> - Again, all the code is hosted out of a tmpfs file system. We are not
>> running out of system memory.
>> - The production tests with memory being filled up was *not* running
>> the shm segment completely out, there was enough memory free to cache
>> the benchmarked pages.
>> - The OS is debian Linux, running kernel 2.6.15 with some minor tuning.
>>
>> I took a number of tests (I won't bother pasting all the output unless
>> necessary). I tested various values of shared memory segment size in
>> various states. Five pages were tested, each hit 2000 times per test
>> at a concurrency level of 9 via apachebench. Each page test was done
>> immediately after the other. Apache was restarted between tests.
>>
>> 32 megabytes of shm with nothing cached before test (used as baseline).
>> 32 megabytes of shm after being ran in production for 20 minutes (most
>> memory used), 15% performance drop from baseline.
>> 32 megabytes of shm in the same conditions with and without the
>> optimize flag enabled (dropping optimize loses 2% performance on top
>> of the other results).
>> After talking with growler_ on IRC, I tried upping the hash buckets
>> from 256 to 4096, since there were ~1400-1600 files in the cache.
>> That made no difference in the tests....
>> I also tested 64 megabytes of shm in the above conditions, and had an
>> 18%+ performance drop as more files were cached.
>> Tried 8 megabytes of shm and the performance drop was much lower.
>> Tried 32 megabytes with half of the memory used and the drop was much
>> lower.
>> Compared to not running eA at all, we get a 75% speed boost from no eA
>> at all to the 32 meg nothing cached baseline test. After one speed
>> drop that's down to 60%, and with 64 megabytes of shared memory that
>> value drops more.
>>
>> It looks to me like it's tied, not to the amount of memory used, nor
>> the amount of memory free, but the number of files in the cache. As
>> the number of files in the cache passes 1500, the performance starts
>> rapidly dropping down. Removing a lot of *small* files from cache has
>> a pretty dramatic change in results.
>>
>> Is there anything at all that can be done about this? I'm going to
>> have nearly 300 webservers soon all running eA, and a 15%+ performance
>> drop is like losing 30 entire webservers. I'm willing to spend some
>> more time on this, but the eA code is tough to work with, so I
>> probably won't be doing any patches on my own. This is also especially
>> painful, since we're writing a lot more code for the site, and so we
>> need eA to scale up the number of files it can cache. I have plenty of
>> RAM to do it with, but eA needs to be able to support this. APC isn't
>> even an option since it segfaults all over our code.
>>
>> I've already adjusted our eA configuration so that it caches fewer
>> files, and the performance seems to have regained noticably. This
>> won't last long though as folks add more code to be cached.
>>
>> Thanks,
>> -Dormando
>>
>>     
> Can you verify if your apache processes don't segfault? It could be that
> with eAccelerator after some time apache starts segfaulting. There have
> been reports of that under high load on smp-systems.
>
> cheers,
>
> Bart
>
>
>   
I just double checked and verified that the apache processes are *not* 
segfaulting during the test. They were segfaulting during our APC tests, 
but not during eA.

I also ran the same test several times in a row, and as long as the 
parameters are the same, the results are the same. (so apache is not 
slowing down over time)

-Dormando