Re: [Bigdata-developers] is there a way to give hdfs path of jnl file in com.bigdata.journal.AbstractJournal.file

[Bryan T. <br...@sy...>]

Yes, right now you need to deploy against local storage on the instance nodes, SAN, NAS, or a parallel file system.  That will change with this refactor.  I need to update our road map, but somewhere around the end of Q2 would be my guess.

Thanks,
Bryan

________________________________
From: tousif [mailto:to...@mo...]
Sent: Monday, November 28, 2011 6:43 AM
To: Bryan Thompson
Subject: Re: [Bigdata-developers] is there a way to give hdfs path of jnl file in com.bigdata.journal.AbstractJournal.file

Thank you Brayan.

I wanted to cluster bigdata so was thinking to store jnl file in hdfs rather than nas. Since my jnl file is going to be bigger, I want to store it distributed. Is there any better way ?

On Mon, Nov 28, 2011 at 4:51 PM, Bryan Thompson <br...@sy...<mailto:br...@sy...>> wrote:
Bigdata journals are expected to be on real file systems.  If you use FUSE to map HDFS into a file system then you could provide that path.  However, the last time I looked, HDFS does not provide some important guarantees (such as flush actually flushing through) and is oriented toward large block operations rather than the fine grained read/write model used by the bigdata journal.  You would be much better off using a parallel file system [1].  There are several that would be suitable.  In fact, AWS, recognizing the difference between a blob store and a true parallel file system, has recently release a parallel file system service.

We are looking at a refactor to support "cloud" style blob stores, such as HDFS or S3. However that would be only for the bigdata federation, not an individual Journal file.  The federation architecture is very different.  Each file has a maximum size of ~ 200M.  This gives them a good size for efficient block fetch from a blob store without excessive latency.  With the refactor, the authoritative copy of the data will be in the cloud/blob store but the working copies will be cached on the instance nodes in the compute side of the cluster.

Thanks,
Bryan

[1] http://en.wikipedia.org/wiki/List_of_file_systems#Distributed_parallel_fault-tolerant_file_systems


________________________________
From: tousif [mailto:to...@mo...<mailto:to...@mo...>]
Sent: Monday, November 28, 2011 2:11 AM
To: big...@li...<mailto:big...@li...>
Subject: [Bigdata-developers] is there a way to give hdfs path of jnl file in com.bigdata.journal.AbstractJournal.file





--
Regards
Tousif




--
Regards
Tousif