Re: [Moosefs-users] Fwd: how to install a cloud storage network :which is best smart automatic file replication solution for cloud storage based systems.

[Ricardo J. B. <ric...@da...>]

El Martes 25 May 2010, Metin Akyalı escribió:
> Hello,

Hi!

> I am looking for a solution for a project i am working on.
>
> We are developing a website where people can upload their files and
> where they can share those files and other people can download them.
> (similar to rapidshare.com model)

[ massive snippage ]

> So i need a cloud based system, which will push the files into
> replicated nodes automatically when demanded to those files are high,
> and when the demand is low, they will delete from other nodes and it
> will stay in only 1 node.
>
> I have looked to glusterfs and asked in their irc channel that
> problem, and got an answer from the guys that gluster cant do such a
> thing. It is only able to replicate all the files or none of the
> files. (i have to define which files to be replicated) But i need it
> the cluster software to do it automatically.

OK, I don't think MooseFS (mfs for short) will help you either, at least not 
automagically: in mfs you can specify that some files have to have more 
copies than others, but you have to do it by hand.

However, I would really advise you to store at least 2 copies to avoid data 
loss in case of one of the storage nodes going kaput.


The only distributed/replicated filesystem that I'm aware of capable of 
automatic load balancing is Ceph (http://ceph.newdream.net/) but it's 
currently in alpha status and not recommended for production sites.

Nonetheless, maybe you can reach your goals with mfs, keep reading :)

> I am sure afer some time, i will have some trouble using  client
> server which i have to loadbalance them later, but that is the next
> step which i dont mind right now.

Well, I'm going to actually suggest something along those lines.

Consider this: no matter how many storage nodes you will have, the bottleneck 
will then be the frontend server (client to mfs) bandwidth, so you _will_ 
have to do frontend load balancing, might as well do it from the stat and 
avoid doing it while in production.

With than in mind, and to avoid manually increasing files copies on the 
backend (storage) nodes, you could cache the files on the frontends (asuming 
the files won't change often, once uploaded) to speed up serving them.

In short: I think your best option is a combination of load balancing/proxy 
cache and MooseFS with goal >= 2.


Best regards,
-- 
Ricardo J. Barberis
Senior SysAdmin - I+D
Dattatec.com :: Soluciones de Web Hosting
Su Hosting hecho Simple..!

------------------------------------------

Nota de confidencialidad: Este mensaje y los archivos adjuntos al mismo
son confidenciales, de uso exclusivo para el destinatario del mismo. La
divulgación y/o uso del mismo sin autorización por parte de Dattatec.com
queda prohibida. Dattatec.com no se hace responsable del mensaje por la
falsificación y/o alteración del mismo.
De no ser Ud. el destinatario del mismo y lo ha recibido por error, por
favor notifique al remitente y elimínelo de su sistema.

Confidentiality Note: This message and any attachments (the message) are
confidential and intended solely for the addressees. Any unauthorised use
or dissemination is prohibited by Dattatec.com. Dattatec.com shall not be
liable for the message if altered or falsified.
If you are not the intended addressee of this message, please cancel it
immediately and inform the sender.

Nota de Confidencialidade: Esta mensagem e seus eventuais anexos podem
conter dados confidenciais ou privilegiados. Se você os recebeu por engano
ou não é um dos destinatários aos quais ela foi endereçada, por favor
destrua-a e a todos os seus eventuais anexos ou copias realizadas,
imediatamente.
É proibida a retenção, distribuição, divulgação ou utilização de quaisquer
informações aqui contidas. Por favor, informe-nos sobre o recebimento
indevido desta mensagem, retornando-a para o autor.