Re: [Openscrobbler-devel] On Replication

[Jonathan D. <jd...@wu...>]

On Mar 4, 2005, at 12:33 AM, Mr. Deep wrote:

> My initial reaction to the replication issue was to simply have pairs 
> of servers, such that they would constantly update one-another in case 
> either of them went down, but I decided that was really lame and we 
> need something better.
>
> I think that we need incremental backups (of either raw data or final 
> statistics) sent to the main server, these should be done on a regular 
> basis (weekly?) or possibly be done somewhat continuously.  Also, 
> every client should have multiple servers that it tries to submit data 
> to in case the main one fails.

The biggest difference from my thoughts here is this is a lot more data 
than I was imagining a central server ever having.

> In general, I think i really need to read the lovely docs that you've 
> created before I go babbling on like an idiot (specifically, the 
> chicken w/head cut off variety).

Well... these don't exist. The docs in SVN mostly deal with the code 
itself: guidelines, how it works, etc etc.

> My main thought at the moment has to do with each client-server 
> generating an incremental update that is sent to the main sever, which 
> then applies it, this incremental update tells the main sever how to 
> update itself to reflect the new submissions that the client-server 
> has received.

As before... I think the difference we're seeing is that in my head, 
"clusters" would not only receive/record submissions, but also serve 
web pages. If each user is assigned to a cluster, then they can get all 
their detailed, personalized stats from that server. Aggregated stats 
can be obtained from a central source and served, or the central source 
can serve them.

There seems to be three paths:
- The central server would be completely transparent and is purely in 
the background. No user would use it directly. Requests are served from 
clusters.
- The central server does some of the requests, while "clusters" do 
others.
- The central server handles all requests

> The main server would store tables, etc in a way that would make 
> accessing the data as fast as possible.  The rate at which these 
> incremental updates would be determined by the # of individual song 
> submissions that have been received.