Re:Scribe trunk now supports Hadoop

2009-06-11
2013-04-11
  • Stephen Corona

    Stephen Corona - 2009-06-11

    How does this work internally? Does scribe append the data into HDFS as it comes in or does add the file to HDFS once the file gets rotated?

    Also, do you know if this will work with the S3 backend for HDFS?

    Thanks,

    Steve Corona

     
    • Anthony Giardullo

      Scribe writes to hdfs directly using libhdfs (the c interface to hdfs).  It does not wait for the file to be rotated.

      This should work for the S3 backend for HDFS, but we haven't yet tried this out.

      -Anthony

       

Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:

JavaScript is required for this form.





No, thanks