Re: [eAccelerator-developers] eAccelerator scales towards useless

[Bart V. <bar...@zo...>]

dormando wrote:
> Hey,
>
> We recently moved our site from eA 0.9.3 to 0.9.5-beta1 while
> upgrading from PHP 4.4.2 to 5.1.2.
>
> We discovered a pretty big dropoff in site speed after doing this.
> Sadly I *cannot* verify if the same performance drop exists in our
> PHP4 setup. What was most concerning is that our site speed dropped to
> complete crap after I upped the shared memory segment size to 64
> megabytes. Dropping it back down to 32 megabytes made the site usable
> again.
>
> We start up PHP across the cluster with a fresh start, and the site is
> blazing fast. After a few minutes things start slowing down pretty
> badly. If we load up the eaccelerator() page and nail the "Clean"
> button, or restart apache across the cluster, the site speeds up again.=

>
> I wasted a few hours last week running a large array of tests to
> narrow down what this could be. Here's the dump of information to get
> past the usual level of "user is probably an idiot" questions;
>
> - The server has no harddrive, the code is loaded out of a ramdisk.
> There is also ~800 megabytes of *unallocated system memory* before and
> after all of these tests. There are no messages in dmesg about running
> low or out of memory.
> - The eaccelerator configuration is set to not use disk caching
> - Optimize, check_mtime, enable, etc are set. compression is not set.
> pruning is not set up during the tests.
> - Apache was *clean restarted* between tests. eaccelerator() was used
> to verify a clean start between *every single test*
> - The tests encompass hitting five different pages from our site code,
> each vary between the amount of CPU required vs DB queries. The
> numbers I looked at most critically are the CPU-only page loads.
> DB-heavy pages were used as a baseline, only.
> - The box was fresh booted for the start of the test runs.
> - There are *no files* being generated in /tmp/eaccelerator/ - I can
> make them show up if I tell eA to, but during the tests none were
> created. /tmp is also hosted in a tmpfs filesystem.
> - Again, all the code is hosted out of a tmpfs file system. We are not
> running out of system memory.
> - The production tests with memory being filled up was *not* running
> the shm segment completely out, there was enough memory free to cache
> the benchmarked pages.
> - The OS is debian Linux, running kernel 2.6.15 with some minor tuning.=

>
> I took a number of tests (I won't bother pasting all the output unless
> necessary). I tested various values of shared memory segment size in
> various states. Five pages were tested, each hit 2000 times per test
> at a concurrency level of 9 via apachebench. Each page test was done
> immediately after the other. Apache was restarted between tests.
>
> 32 megabytes of shm with nothing cached before test (used as baseline).=

> 32 megabytes of shm after being ran in production for 20 minutes (most
> memory used), 15% performance drop from baseline.
> 32 megabytes of shm in the same conditions with and without the
> optimize flag enabled (dropping optimize loses 2% performance on top
> of the other results).
> After talking with growler_ on IRC, I tried upping the hash buckets
> from 256 to 4096, since there were ~1400-1600 files in the cache.
> That made no difference in the tests....
> I also tested 64 megabytes of shm in the above conditions, and had an
> 18%+ performance drop as more files were cached.
> Tried 8 megabytes of shm and the performance drop was much lower.
> Tried 32 megabytes with half of the memory used and the drop was much
> lower.
> Compared to not running eA at all, we get a 75% speed boost from no eA
> at all to the 32 meg nothing cached baseline test. After one speed
> drop that's down to 60%, and with 64 megabytes of shared memory that
> value drops more.
>
> It looks to me like it's tied, not to the amount of memory used, nor
> the amount of memory free, but the number of files in the cache. As
> the number of files in the cache passes 1500, the performance starts
> rapidly dropping down. Removing a lot of *small* files from cache has
> a pretty dramatic change in results.
>
> Is there anything at all that can be done about this? I'm going to
> have nearly 300 webservers soon all running eA, and a 15%+ performance
> drop is like losing 30 entire webservers. I'm willing to spend some
> more time on this, but the eA code is tough to work with, so I
> probably won't be doing any patches on my own. This is also especially
> painful, since we're writing a lot more code for the site, and so we
> need eA to scale up the number of files it can cache. I have plenty of
> RAM to do it with, but eA needs to be able to support this. APC isn't
> even an option since it segfaults all over our code.
>
> I've already adjusted our eA configuration so that it caches fewer
> files, and the performance seems to have regained noticably. This
> won't last long though as folks add more code to be cached.
>
> Thanks,
> -Dormando
>
Can you verify if your apache processes don't segfault? It could be that
with eAccelerator after some time apache starts segfaulting. There have
been reports of that under high load on smp-systems.

cheers,

Bart


--=20
Bart Vanbrabant <bar...@zo...>
PGP fingerprint: 093C BB84 17F6 3AA6 6D5E  FC4F 84E1 FED1 E426 64D1