I just downloaded the scribe 2.0 distribution, and I'm looking at the examples directory.
There are several example config files, e.g., example1.conf, etc...
The *.conf files seem pretty straightforward, but I'm wondering if there is any documentation on the configuration language there?
I'd like to know all the options, etc...
One thing I'm wondering, it appears straightforward how to have log data logged to a file (type=file) and to forward it to another scribe instance on a the network (type=network), etc....Is it straightforward to have a our own server receive the log data, can we place any server that understands how to deserialize thrift at the end-point, instead of another scribe instance? Are there any examples (similar to scribe_cat) for retrieving incoming data from an upstream scribe instance?
Thanks,
Jason
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
With regard to your second question, You could create another thrift server that receives the thrift messages from the network and then does whatever you need to do with them, but then you're just reinventing scribe .
If your code does something novel, maybe you should just extend the scribe source to support the new file store type and share that extension with the community. It would be less code for you to write and better for everyone.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thanks, I didn't see that wiki link for some reason (and the 'documentation' link was empty, which through me off)....
I have a java server that receives events and does stream processing and aggregation.....I'm considering using scribe to transmit events efficiently over a distributed set of machines to this java event server, which will receive thrift messages and process them as events.....
So, I need to implement in java the piece that receives an event from another scribe server....
Can I do this using the thrift library entirely (just receive thrift messages on a port, indefinitely?)....Will there be anything scribe specific in the incoming stream, or is it just thrift objects? Is there any delimitting between thrift messages in scribe's communication?
Thanks,
Jason
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I was attracted to scribe's asynchronous, reliable transport (buffered), and it seemed good for distributing events from a large number of servers to a single point of aggregation....
I'm concerned that direct connections via thrift services may be brittle and overly synchronous....Does thrift itself then have buffering, etc.?
Jason
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The file scribe.thrift defines a simple Thrift interface that contains a Log() method. The Scribe server code contains an implementation of this Log() method. So any server running an instance of Scribe knows how to process thrift calls to this Log() method. If you configure Scribe to use a network store, Scribe will create a Thrift connection to another instance of Scribe and call that instance’s Log() method.
Technically, nothing is stopping you from writing your own Thrift service in Java that implements a Thrift interface named ‘scribe’ that has a ‘Log’ method. However, this is arguably hacky as your code would essentially be pretending to be Scribe.
An alternate solution would be to create your own Thrift service (implemented in any language Thrift supports) and naming it whatever you want. Then you could modify Scribe to support your new service (by creating a new store type similar to a Network store).
-Anthony
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hello,
I just downloaded the scribe 2.0 distribution, and I'm looking at the examples directory.
There are several example config files, e.g., example1.conf, etc...
The *.conf files seem pretty straightforward, but I'm wondering if there is any documentation on the configuration language there?
I'd like to know all the options, etc...
One thing I'm wondering, it appears straightforward how to have log data logged to a file (type=file) and to forward it to another scribe instance on a the network (type=network), etc....Is it straightforward to have a our own server receive the log data, can we place any server that understands how to deserialize thrift at the end-point, instead of another scribe instance? Are there any examples (similar to scribe_cat) for retrieving incoming data from an upstream scribe instance?
Thanks,
Jason
Full docs for configration file are on this very wiki!
http://scribeserver.wiki.sourceforge.net/Configuration
With regard to your second question, You could create another thrift server that receives the thrift messages from the network and then does whatever you need to do with them, but then you're just reinventing scribe .
If your code does something novel, maybe you should just extend the scribe source to support the new file store type and share that extension with the community. It would be less code for you to write and better for everyone.
Hi,
Thanks, I didn't see that wiki link for some reason (and the 'documentation' link was empty, which through me off)....
I have a java server that receives events and does stream processing and aggregation.....I'm considering using scribe to transmit events efficiently over a distributed set of machines to this java event server, which will receive thrift messages and process them as events.....
So, I need to implement in java the piece that receives an event from another scribe server....
Can I do this using the thrift library entirely (just receive thrift messages on a port, indefinitely?)....Will there be anything scribe specific in the incoming stream, or is it just thrift objects? Is there any delimitting between thrift messages in scribe's communication?
Thanks,
Jason
I wouldn't use scribe for this. What you're describing here is a thrift service and there are java hooks for thrift directly.
Scribe is Thrift + methods to dump messages to disk and/or buffers.
"Receiving messages on a port, indefinitely" is how thrift services function.
John,
Thanks for the advice.
I was attracted to scribe's asynchronous, reliable transport (buffered), and it seemed good for distributing events from a large number of servers to a single point of aggregation....
I'm concerned that direct connections via thrift services may be brittle and overly synchronous....Does thrift itself then have buffering, etc.?
Jason
The file scribe.thrift defines a simple Thrift interface that contains a Log() method. The Scribe server code contains an implementation of this Log() method. So any server running an instance of Scribe knows how to process thrift calls to this Log() method. If you configure Scribe to use a network store, Scribe will create a Thrift connection to another instance of Scribe and call that instance’s Log() method.
Technically, nothing is stopping you from writing your own Thrift service in Java that implements a Thrift interface named ‘scribe’ that has a ‘Log’ method. However, this is arguably hacky as your code would essentially be pretending to be Scribe.
An alternate solution would be to create your own Thrift service (implemented in any language Thrift supports) and naming it whatever you want. Then you could modify Scribe to support your new service (by creating a new store type similar to a Network store).
-Anthony