Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project! See Demo

Close

Re:Scribe trunk now supports Hadoop

2009-06-11
2013-04-11
  • Stephen Corona
    Stephen Corona
    2009-06-11

    How does this work internally? Does scribe append the data into HDFS as it comes in or does add the file to HDFS once the file gets rotated?

    Also, do you know if this will work with the S3 backend for HDFS?

    Thanks,

    Steve Corona

     
    • Scribe writes to hdfs directly using libhdfs (the c interface to hdfs).  It does not wait for the file to be rotated.

      This should work for the S3 backend for HDFS, but we haven't yet tried this out.

      -Anthony