Thread: [SSI-users] clustering why-to
Brought to you by:
brucewalker,
rogertsang
From: Alexandre C. <aco...@ma...> - 2006-01-31 11:04:39
|
Hello, I want to setup a clustering environement. I read many how-to's out there, but not many why-to's actually. So my main question is: "would I need a cluster system for what I'd like to do ?" 1/ Here is how my application works today: I have 1 web server and 1 database server. I have multiple remote clients (PCs) that sends data every night the database server. There is a lot of information and the database is getting bigger and bigger pretty quickly. Then a client (human) can connect to the web server and have a graphical interface to manipulate some statistics from the data in the database. 2/ Here is my problem: When a human client connects to the web server to make some statistics, 1 request takes a long time to process because a lot of data is parsed and computed. And it's taking too much time. Today, my application works as a "test", but it's gonna get bigger and bigger with more and more client PCs that will send data to the database. So I'm worried that my 1 database just won't handle it. 3/ Here is my idea: I want to keep 1 web server because I don't have much human clients that want to play with the statistics (low trafic). But because the database is getting bigger and bigger, I want to be able to add some extra storage over another database server. So I got my self 3 servers for the database (dual Xeon + RAID5). I'd like to have all servers in a cluster seen as 1 logical database. That way, data would be shared over all 3 servers transparently, the web server would talk only to 1 ip address like if it was 1 single database. 4/ Here are the problems that could happend: - If I run out of disk space, I'll add a 4th server inside the cluster and it will automaticly share data over the new server. - If a server fails, I want to make sure I have data intergrety (if one fails, it can still run with the left servers). When the failed server is replaced, it will reconstruct it's data and be available again. So here are my questions: - Is my project feasable with my current hardware configuration ? - Is clustering what I'm looking for ? - What is clustering after all if what I'm talking about it not clustering ? I'm in a little hurry for my porject... please help !! Best regards, -- Alexandre CONRAD - TLV Research & Development tel : +33 1 30 80 55 05 fax : +33 1 30 80 55 06 6, rue de la plaine 78860 - SAINT NOM LA BRETECHE FRANCE |
From: Alexandre C. <aco...@ma...> - 2006-01-31 11:12:26
|
Me again, I seen about these different file systems, maybe you guys can point me=20 one solution that could suite my needs. 1) Lan Mirroring DRBD [MIRRORING] Network Block Device (NBD) [MIRRORING] NFS [SHARED] CodaFS [SHARED] ODR : Online Disk Replicator [MIRRORING] ENBD : Enhanced Network Block Device [MIRRORING] Network RAID [MIRRORING] 2) Volume Managers LVM [PARTITIONING] EVMS : Enterprise Volume Management System (=E9mulateur lvm)[PARTITIONING= ] 3) FileSystem GFS [CLUSTERING/JOURNALING] ReiserFS [JOURNALING] Ext3 [JOURNALING] JFS (IBM) [JOURNALING] XFS (SGI) [JOURNALING] FENRIS (Timponagos) [CLUSTERING] M2CS (Timponagos) [CLUSTERING] Intermezzo [CLUSTERING] LinLogFS [CLUSTERING] Thanks again. Regards, Alexandre CONRAD wrote: > Hello, >=20 > I want to setup a clustering environement. I read many how-to's out=20 > there, but not many why-to's actually. >=20 > So my main question is: "would I need a cluster system for what I'd lik= e=20 > to do ?" >=20 >=20 > 1/ Here is how my application works today: > I have 1 web server and 1 database server. I have multiple remote=20 > clients (PCs) that sends data every night the database server. There is= =20 > a lot of information and the database is getting bigger and bigger=20 > pretty quickly. >=20 > Then a client (human) can connect to the web server and have a graphica= l=20 > interface to manipulate some statistics from the data in the database. >=20 > 2/ Here is my problem: > When a human client connects to the web server to make some statistics,= =20 > 1 request takes a long time to process because a lot of data is parsed=20 > and computed. And it's taking too much time. >=20 > Today, my application works as a "test", but it's gonna get bigger and=20 > bigger with more and more client PCs that will send data to the databas= e. >=20 > So I'm worried that my 1 database just won't handle it. >=20 > 3/ Here is my idea: > I want to keep 1 web server because I don't have much human clients tha= t=20 > want to play with the statistics (low trafic). But because the database= =20 > is getting bigger and bigger, I want to be able to add some extra=20 > storage over another database server. >=20 > So I got my self 3 servers for the database (dual Xeon + RAID5). I'd=20 > like to have all servers in a cluster seen as 1 logical database. That=20 > way, data would be shared over all 3 servers transparently, the web=20 > server would talk only to 1 ip address like if it was 1 single database. >=20 > 4/ Here are the problems that could happend: > - If I run out of disk space, I'll add a 4th server inside the cluster=20 > and it will automaticly share data over the new server. >=20 > - If a server fails, I want to make sure I have data intergrety (if one= =20 > fails, it can still run with the left servers). When the failed server=20 > is replaced, it will reconstruct it's data and be available again. >=20 >=20 > So here are my questions: > - Is my project feasable with my current hardware configuration ? > - Is clustering what I'm looking for ? > - What is clustering after all if what I'm talking about it not=20 > clustering ? >=20 > I'm in a little hurry for my porject... please help !! >=20 > Best regards, --=20 Alexandre CONRAD - TLV Research & Development tel : +33 1 30 80 55 05 fax : +33 1 30 80 55 06 6, rue de la plaine 78860 - SAINT NOM LA BRETECHE FRANCE |
From: Lars G. <gr...@gm...> - 2006-01-31 11:28:03
|
As far as I understand your setup, the most important informationis missing. Which database do you use? In fact, IMHO you should find a clustering database. If you will enter e.g. mysql cluster howto you will find a lot of information concerning the setup. I think to improve database performace, a dedicated cluster-setup is the only way to go... CU Lars. |
From: Alexandre C. <aco...@ma...> - 2006-01-31 11:50:50
|
Hello Lars, > As far as I understand your setup, the most important informationis missing. > Which database do you use? In fact, IMHO you should find a clustering > database. If you will enter e.g. mysql cluster howto you will find a lot of > information concerning the setup. I think to improve database performace, a > dedicated cluster-setup is the only way to go... I do use mysql. I've been reading about mysql cluster a little, and it seems to be what I would need. I'll have some more look on that and figure out what would be the best for me. But, any how, for my general clustering knowledge, if mysql cluster wouldn't exists, what could be think about as an alternative solution ? Thanks. Regards, -- Alexandre CONRAD - TLV Research & Development tel : +33 1 30 80 55 05 fax : +33 1 30 80 55 06 6, rue de la plaine 78860 - SAINT NOM LA BRETECHE FRANCE |
From: Alexandre C. <aco...@ma...> - 2006-01-31 12:15:19
|
Hello, > Hello Lars, > >> As far as I understand your setup, the most important informationis >> missing. >> Which database do you use? In fact, IMHO you should find a clustering >> database. If you will enter e.g. mysql cluster howto you will find a >> lot of >> information concerning the setup. I think to improve database >> performace, a >> dedicated cluster-setup is the only way to go... > > > I do use mysql. I've been reading about mysql cluster a little, and it > seems to be what I would need. I'll have some more look on that and > figure out what would be the best for me. > > But, any how, for my general clustering knowledge, if mysql cluster > wouldn't exists, what could be think about as an alternative solution ? I'm also reading about CLVM (cluster lvm2). I guess this would be the right alternative, wouldn't it ? Regards, -- Alexandre CONRAD - TLV Research & Development tel : +33 1 30 80 55 05 fax : +33 1 30 80 55 06 6, rue de la plaine 78860 - SAINT NOM LA BRETECHE FRANCE |
From: Ivan K. <kr...@fa...> - 2006-01-31 16:18:15
|
Alexandre CONRAD wrote: > But, any how, for my general clustering knowledge, if mysql cluster > wouldn't exists, what could be think about as an alternative solution ? For your problem, the choice of database is absolutely critical: an underlying cluster will do nothing at all for you if your database doesn't cluster. I'm not aware of any high-quality FOSS clustering databases other than MySQL. With MySQL Cluster, a plain OpenSSI cluster will fully solve your designated problem. -- Ivan Krstic <kr...@fa...> | 0x147C722D |
From: John J. <joh...@gm...> - 2006-01-31 17:49:24
|
UG9zdGdyZVNRTCBwcm92aWRlcyBjbHVzdGVyaW5nIG9wdGlvbnMsIHBsdXMgaXQgaGFzIHN1cHBv cnQgZm9yIHRhYmxlc3BhY2VzCndoaWNoIG1heSBvciBtYXkgbm90IGhlbHAgZGVwZW5kaW5nIG9u IGhvdyB5b3UgYXBwcm9hY2ggeW91ciBjbHVzdGVyLgoKUmVnYXJkcywKSm9obgoKT24gMS8zMS8w NiwgSXZhbiBLcnN0aWMgPGtyc3RpY0BmYXMuaGFydmFyZC5lZHU+IHdyb3RlOgo+Cj4gQWxleGFu ZHJlIENPTlJBRCB3cm90ZToKPiA+IEJ1dCwgYW55IGhvdywgZm9yIG15IGdlbmVyYWwgY2x1c3Rl cmluZyBrbm93bGVkZ2UsIGlmIG15c3FsIGNsdXN0ZXIKPiA+IHdvdWxkbid0IGV4aXN0cywgd2hh dCBjb3VsZCBiZSB0aGluayBhYm91dCBhcyBhbiBhbHRlcm5hdGl2ZSBzb2x1dGlvbiA/Cj4KPiBG b3IgeW91ciBwcm9ibGVtLCB0aGUgY2hvaWNlIG9mIGRhdGFiYXNlIGlzIGFic29sdXRlbHkgY3Jp dGljYWw6IGFuCj4gdW5kZXJseWluZyBjbHVzdGVyIHdpbGwgZG8gbm90aGluZyBhdCBhbGwgZm9y IHlvdSBpZiB5b3VyIGRhdGFiYXNlCj4gZG9lc24ndCBjbHVzdGVyLgo+Cj4gSSdtIG5vdCBhd2Fy ZSBvZiBhbnkgaGlnaC1xdWFsaXR5IEZPU1MgY2x1c3RlcmluZyBkYXRhYmFzZXMgb3RoZXIgdGhh bgo+IE15U1FMLiBXaXRoIE15U1FMIENsdXN0ZXIsIGEgcGxhaW4gT3BlblNTSSBjbHVzdGVyIHdp bGwgZnVsbHkgc29sdmUgeW91cgo+IGRlc2lnbmF0ZWQgcHJvYmxlbS4KPgo+IC0tCj4gSXZhbiBL cnN0aWMgPGtyc3RpY0BmYXMuaGFydmFyZC5lZHU+IHwgMHgxNDdDNzIyRAo+Cj4KPiAtLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCj4gVGhpcyBT Ri5uZXQgZW1haWwgaXMgc3BvbnNvcmVkIGJ5OiBTcGx1bmsgSW5jLiBEbyB5b3UgZ3JlcCB0aHJv dWdoIGxvZwo+IGZpbGVzCj4gZm9yIHByb2JsZW1zPyAgU3RvcCEgIERvd25sb2FkIHRoZSBuZXcg QUpBWCBzZWFyY2ggZW5naW5lIHRoYXQgbWFrZXMKPiBzZWFyY2hpbmcgeW91ciBsb2cgZmlsZXMg YXMgZWFzeSBhcyBzdXJmaW5nIHRoZSAgd2ViLiAgRE9XTkxPQUQgU1BMVU5LIQo+IGh0dHA6Ly9z ZWwuYXMtdXMuZmFsa2FnLm5ldC9zZWw/Y21kPWxuayZraWQ9MTAzNDMyJmJpZD0yMzA0ODYmZGF0 PTEyMTY0Mgo+IF9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f Cj4gU3NpYy1saW51eC11c2VycyBtYWlsaW5nIGxpc3QKPiBTc2ljLWxpbnV4LXVzZXJzQGxpc3Rz LnNvdXJjZWZvcmdlLm5ldAo+IGh0dHBzOi8vbGlzdHMuc291cmNlZm9yZ2UubmV0L2xpc3RzL2xp c3RpbmZvL3NzaWMtbGludXgtdXNlcnMKPgo= |
From: Ivan K. <kr...@fa...> - 2006-01-31 17:59:59
|
John Jawed wrote: > PostgreSQL provides clustering options It was my understanding that PostgreSQL clustering wasn't mature; are you aware of production deployments? I don't keep up to date with PostgreSQL development, so it's possible my knowledge is out of date. -- Ivan Krstic <kr...@fa...> | 0x147C722D |
From: John J. <joh...@gm...> - 2006-01-31 18:31:07
|
VGhlIGxhc3QgcHJvZHVjdGlvbiBjbHVzdGVyaW5nIEkgd2FzIGF3YXJlIG9mIChhcyBpbiBpdCB3 YXMgaGFwcGVuaW5nIHVuZGVyCm15IG5vc2UpIHdhcyB3aXRoIHNsb255LCBidXQgZnJvbSBhbGwg dGhlIHBvc3RncmVzcWwgZGlzY3Vzc2lvbnMgSSd2ZSBiZWVuCmluIHdpdGggcmVnYXJkcyB0byBj bHVzdGVyaW5nLCBpdCBoYXMgbWF0dXJlZCBwYXN0IHNsb255IGFuZCBvZmZlcnMgYmV0dGVyCmNs dXN0ZXJpbmcgc3VwcG9ydC4gSW4gdGhlIGZvcm0gb2Ygd2hhdCwgSSdtIG5vdCBleGFjdGx5IHN1 cmUuCgpSZWdhcmRzLApKb2huCgpPbiAxLzMxLzA2LCBJdmFuIEtyc3RpYyA8a3JzdGljQGZhcy5o YXJ2YXJkLmVkdT4gd3JvdGU6Cj4KPiBKb2huIEphd2VkIHdyb3RlOgo+ID4gUG9zdGdyZVNRTCBw cm92aWRlcyBjbHVzdGVyaW5nIG9wdGlvbnMKPgo+IEl0IHdhcyBteSB1bmRlcnN0YW5kaW5nIHRo YXQgUG9zdGdyZVNRTCBjbHVzdGVyaW5nIHdhc24ndCBtYXR1cmU7IGFyZQo+IHlvdSBhd2FyZSBv ZiBwcm9kdWN0aW9uIGRlcGxveW1lbnRzPyBJIGRvbid0IGtlZXAgdXAgdG8gZGF0ZSB3aXRoCj4g UG9zdGdyZVNRTCBkZXZlbG9wbWVudCwgc28gaXQncyBwb3NzaWJsZSBteSBrbm93bGVkZ2UgaXMg b3V0IG9mIGRhdGUuCj4KPiAtLQo+IEl2YW4gS3JzdGljIDxrcnN0aWNAZmFzLmhhcnZhcmQuZWR1 PiB8IDB4MTQ3QzcyMkQKPgo= |