phpslash-devel

[Phpslash-devel] Discuss high traffic and HA development

[Joe S. <joe...@us...>]

This message is crossposted to the Back-End developers and phpSlash 
developers.

I'd like to get an IRC discussion going on what developments need to be 
done to make BE/PSL more scalable and easier to deploy in a larger 
website installation.

While I'm not saying we should get away from the easy to install, easy 
to deploy on a hosted account ideas, there is also a need for being able 
to use the code on sites serving lots of page views from multiple servers.

I'd like to do this on IRC to get more of a real time discussion going. 
  Is there a good time for everyone?  But I'm not opposed to discussion 
here as well.

This may seem a little soon in the next phpSlash release cycle for this 
discussion, but after this is done, I'd like to get another discussion 
on a more general roadmap for BE and PSL.

A couple of my ideas to throw out:

High Traffic Discussion

- Load Balancing front end
- db clustering, master/slave, read servers and write servers, etc.
- memcache support in db reads, writes, html cacheing, etc.
- static file writing to support staging, pushing to multiple servers, 
ease of serving, speed, etc.
- squid reverse proxy front end.
- Ian's block cacheing
- using a separate image server with thttpd or somebody like akamai

Joe

[Phpslash-devel] Re: [Back-end-development] Discuss high traffic and HA development

[Mike G. <mi...@op...>]

Hello Joe,

On Thu, 2004-09-16 at 13:25, Joe Stewart wrote:
> This message is crossposted to the Back-End developers and phpSlash 
> developers.

Good to be working with the psl team..

> I'd like to get an IRC discussion going on what developments need to be 
> done to make BE/PSL more scalable and easier to deploy in a larger 
> website installation.

Sounds very cool..

> While I'm not saying we should get away from the easy to install, easy 
> to deploy on a hosted account ideas, there is also a need for being able 
> to use the code on sites serving lots of page views from multiple servers.

scalability is from the tiny sites to the massive ones, right?  

There are a few projects that we've worked with that have helped test
and improve the scalability of BE.

> I'd like to do this on IRC to get more of a real time discussion going. 
>   Is there a good time for everyone?  But I'm not opposed to discussion 
> here as well.

Well, I'm usually pretty flexible..  But during the week 9am-4pm EST is
usually the best..  I'll try to check into irc more often..  It can be a
good way to keep abreast.. 

> This may seem a little soon in the next phpSlash release cycle for this 
> discussion, but after this is done, I'd like to get another discussion 
> on a more general roadmap for BE and PSL.

Sounds very good..  Would be great if there was an instance of them
running together even..  But since they share so much code having more
communication is pretty critical..

> A couple of my ideas to throw out:

Good to do..  

> High Traffic Discussion
> - Load Balancing front end

So how does this differ from sticking squid in front of the server?

> - db clustering, master/slave, read servers and write servers, etc.

There was a good article in php-a this summer about MySQL's master-slave
capacity, but I have yet to explore it..  Sounds like an excellent way
to extend the code.

> - memcache support in db reads, writes, html cacheing, etc.

Explored this a bit more..  BE uses jpcache for page caching (Thanks
Joe) and also another fragment caching app for storing arrays and html
fragments (Thanks PeterC)..  

> - static file writing to support staging, pushing to multiple servers, 
> ease of serving, speed, etc.

Having that capacity would be pretty cool indeed..

Are there other open source php/mysql apps we can draw on for this?  Or
is this beyond what's possible within a php/mysql environment..  We're
talking multiple servers here so its safe to assume that other tools are
available..

> - squid reverse proxy front end.
> - Ian's block cacheing

It's cool..  Certainly for the user level caching..

> - using a separate image server with thttpd or somebody like akamai

This app here I assume:
	http://www.acme.com/software/thttpd/

I haven't actually wondered there before, but it looks hopeful..  

Mike
-- 
Mike Gifford, OpenConcept Consulting
Free Software for Social Change -> http://www.openconcept.ca
Our Client's Election Sites: http://rantroom.ca http://election.cupe.ca
http://fairvotecanada.org http://billblaikie.ca http://dickproctor.ca 
http://patmartin.ca http://brianmasse.ca 



-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
_______________________________________________
Back-end-development mailing list
Bac...@li...
https://lists.sourceforge.net/lists/listinfo/back-end-development

[Phpslash-devel] Re: [Back-end-development] Discuss high traffic and HA development

[Joe S. <joe...@us...>]

on 09/16/04 20:26 Mike Gifford said the following:
> Hello Joe,
> 
> On Thu, 2004-09-16 at 13:25, Joe Stewart wrote:
> 
>>This message is crossposted to the Back-End developers and phpSlash 
>>developers.
> 
> 
<snip>
> 
> scalability is from the tiny sites to the massive ones, right?  
> 

Mostly my concerns are that both high daily traffic and traffic spikes 
can be handled.

> There are a few projects that we've worked with that have helped test
> and improve the scalability of BE.
> 
> 
>>I'd like to do this on IRC to get more of a real time discussion going. 
>>  Is there a good time for everyone?  But I'm not opposed to discussion 
>>here as well.
> 
> 
> Well, I'm usually pretty flexible..  But during the week 9am-4pm EST is
> usually the best..  I'll try to check into irc more often..  It can be a
> good way to keep abreast.. 
> 

That's pretty close to when I'm around too.

As far as when, I don't care at all.  I'd just like to keep the ball 
rolling.  So it could definitely wait a week for Ian to return.

> 
>>This may seem a little soon in the next phpSlash release cycle for this 
>>discussion, but after this is done, I'd like to get another discussion 
>>on a more general roadmap for BE and PSL.
> 
> 
> Sounds very good..  Would be great if there was an instance of them
> running together even..  But since they share so much code having more
> communication is pretty critical..
> 
> 
>>A couple of my ideas to throw out:
> 
> 
> Good to do..  
> 
> 
>>High Traffic Discussion
>>- Load Balancing front end
> 
> 
> So how does this differ from sticking squid in front of the server?
> 

When I think of sticking squid in front, it's more about using it as a 
transparent caching proxy.

And I think of load balancer more for High Availability failover, and 
load sharing

> 
>>- db clustering, master/slave, read servers and write servers, etc.
> 
> 
> There was a good article in php-a this summer about MySQL's master-slave
> capacity, but I have yet to explore it..  Sounds like an excellent way
> to extend the code.
> 

The memcache/LiveJournal guys talk about this too and the dangers when 
your writes reach a significant level.

> 
>>- memcache support in db reads, writes, html cacheing, etc.
> 
> 
> Explored this a bit more..  BE uses jpcache for page caching (Thanks
> Joe) and also another fragment caching app for storing arrays and html
> fragments (Thanks PeterC)..  
> 
> 

Just for clarity - this is not Turck mmcache.  I didn't mention it but I 
find either mmcache, php-accelerator or something similar imperative.
I think it really just should be part of php like in python.  These 
basically store the compiled code so that step doesn't have to be 
repeated each pageview.  In informal benchmarking mmcache using ab, I 
get about a 2x to 3x improvement in requests per second.


This is memcache ( http://www.danga.com/mmcache) - It is an in-memory 
cache daemon.  Developed for LiveJournal and used on Slashdot.

An introductory article at Linux Journal by the author:

	http://www.linuxjournal.com/article.php?sid=7451


>>- Ian's block cacheing
> 
> 
> It's cool..  Certainly for the user level caching..
> 

This is one place I think storing in memory would be nice.

> 
>>- using a separate image server with thttpd or somebody like akamai
> 
> 
> This app here I assume:
> 	http://www.acme.com/software/thttpd/
> 
> I haven't actually wondered there before, but it looks hopeful..  
> 

That's the server.  I believe that it loads the pages/images in memory 
and retrieves them from there.  How different this would be from the 
linux filesystem retrieval caching, I don't know.

Joe

> Mike