Messages sometimes written out of order

Elbert
2009-09-25
2013-04-11
  • Elbert

    Elbert - 2009-09-25

    I'm running an experiment where I have an application sending message to scribe as well as writing to local file. Scribe is then logging via its own mechanism. I have tried with using a store type == buffer and store type == file.

    When I compare the log created by my application and the logs created by scribe, all the data exists in all cases. However, the messages written by scribe aren't necessarily written in the same order as the message written via the application.

    I.e. Out of several thousand messages that were created, 26 were written out of order. A simple sort, then md5sum shows me that no data is lost.

    I have tried to use target_write_size=1 and max_write_interval=1, but the issue hasn't gone away. Is there anything I can do about this?

    Thanks!

     
  • Anthony Giardullo

    A single scribe instance should maintain message order for each message category.  Are you sure you are sending all the messages in order?  Is your application single threaded or will multiple threads send messages to Scribe at the same time?

     
  • Elbert

    Elbert - 2009-09-28

    At this point, the messages are being sent in order, and the application is single threaded. When a message is written to the local log, the same message is sent to scribe, and then the next message gets processed. Again, first the message is written to the local log, then it is sent to scribe, so on and so forth. If the app is for some reason slow to write the message to the local log (for example, high disk i/o), then the app will not do anything until the message gets written, or until the OS reports a failure code. This is not true when it sends the message to scribe. The app does not expect a response back. It simply sends the message and then continues. I'm not seeing any lost messages, so I'm not too worried about that for the moment.

    Also, messages are being written at the rate of ~40-50 per second. I can investigate the app to see if there is some sort of missend occurring, but it's singlethreaded in this regard, so I don't see how a later message could be sent unless an earlier message was already sent. Could something else be contributing? I'm not sure what else to look at though.

     
  • Anthony Giardullo

    Are all these messages being logged with the same category?  And are you testing with a single scribe instance or do you multiple instances configured to forward messages?

    I would also use scribe_ctrl to check scribe's counters and also check the output of scribe for any errors.

     
  • Elbert

    Elbert - 2009-09-28

    Thanks for the quick replies! All messages are being logged with the same category, and I'm using a single scribe instance on localhost. There is no forwarding to other servers going on. However, I am writing to two locations: 1) local filesystem, and 2) mounted filesystem. The files that scribe creates are identical between the two filesystems, but they don't match the log that my app is writing.

    I had not considered using scribe_ctrl. Doesn't that just show # of messages? I'm not seeing any message loss, so I didn't think that applied. Also, when I look at the STDOUT of scribe, I don't see any errors. It's pretty quiet.

    Thanks!

     
  • Anthony Giardullo

    I’m not sure why you are seeing messages out of order when using a single category on a single scribe instance.  Can you tell me about the client you are using to send messages to scribe?  What language are you using?

    -Anthony

     

Log in to post a comment.