I have setup a cluster of scribe servers that forward messages to a central scribe server. Does scribe utilize any type of compression when sending messages between servers? I'm not sure if thrift already provides this functionality or not.
Thanks,
Steve Corona
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Scribe does not use any compression and relies on a Thrift transport(TFramedTransport) to send messages between two Scribe instances. Thrift does have a transport that supports compression called TZLibTransport. But I have not yet tested using this transport with Scribe. To do this, you would need to replace the TFramedTransport objects in conn_pool.cpp and scribe_server.cpp with TZlibTransport.
If you try this out, please let us know how it goes.
-Anthony
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thanks for getting back to me. I'm going to test out scribe with TZlibTransport on Monday. The reason I asked is because I have a scribe server running on Amazon EC2 and I'd like to conserve bandwidth if it won't impact functionality.
I was also thinking about implementing Amazon S3 as a file store for thrift but I should probably save that for another thread.
Steve Corona
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I have setup a cluster of scribe servers that forward messages to a central scribe server. Does scribe utilize any type of compression when sending messages between servers? I'm not sure if thrift already provides this functionality or not.
Thanks,
Steve Corona
Scribe does not use any compression and relies on a Thrift transport(TFramedTransport) to send messages between two Scribe instances. Thrift does have a transport that supports compression called TZLibTransport. But I have not yet tested using this transport with Scribe. To do this, you would need to replace the TFramedTransport objects in conn_pool.cpp and scribe_server.cpp with TZlibTransport.
If you try this out, please let us know how it goes.
-Anthony
Hey Anothny,
Thanks for getting back to me. I'm going to test out scribe with TZlibTransport on Monday. The reason I asked is because I have a scribe server running on Amazon EC2 and I'd like to conserve bandwidth if it won't impact functionality.
I was also thinking about implementing Amazon S3 as a file store for thrift but I should probably save that for another thread.
Steve Corona
Hi Stephen ,
Do you have the library for scribe to Amazon S3 as a file store scribe ? Or is there any open source library for the same
--Saju