Throttling network storage?

[Frank DENIS]

  Hello,

  I'm trying to use Scribe as a way to queue events for an activity-stream-like app.

  The current test architecture is straightforward.

  Web servers send log messages to a central Scribe server, whose config looks like:

port=1463
max_msg_per_second=100000
max_queue_size=500000
check_interval=5
new_thread_per_category=true

<store>

category=test
type=buffer
retry_interval=30
retry_interval_range=10
buffer_send_rate=1
max_queue_length=2000000

<primary>
type=network
smc_service=workers
use_conn_pool=yes
timeout=100000
</primary>

<secondary>
type=file
file_path=/tmp
base_filename=test_sec_log
write_meta=yes
add_newlines=1
</secondary>

</store>

  The final Scribe servers ("workers" smc service) aren't Scribe servers, but simple PHP backends speaking the Scribe interface:

class scribeHandler extends FacebookBase implements scribeIf {
    public function Log($message) {
        /* ... */

        return ResultCode::OK;
    }
}

$handler = new scribeHandler('scribe');
$processor = new scribeProcessor($handler);

$transport = new TFramedTransport
  (new TPhpStream(TPhpStream::MODE_R | TPhpStream::MODE_W));
$protocol = new TBinaryProtocol($transport, FALSE, FALSE);

$transport->open();
$processor->process($protocol, $protocol);
$transport->close();

  When the primary store is down, messages are properly queued to the local file storage.

  Then, when the primary storage goes up, Scribe switches from <DISCONNECTED> to <SENDING_BUFFER>. But it tries to send all pending message in a row:

[test] successfully read <168401> entries from file </tmp/test_sec_log_00000>
Failed to send <168401> messages to remote scribe server <SMC service: workers> error <EAGAIN (timed out)>

  And it times out in spite of the large timeout setting because the PHP backend just can't handle so many messages in a single shot.

  Is there a way to send small groups of messages to a remote Scribe (or Scribe-speaking) server in SENDING_BUFFER mode, instead of trying to send everything once?

  Or is there anything wrong with this testing setup?

  Best regards,

         -Frank.

[Anthony Giardullo]

Every check_interval, your buffer store will send 1(buffer_send_rate) file from your secondary store to your primary store.  So I would suggest setting max_size in you secondary store to limit the size of files that get created.

Also, you could have your 'workers' detect when they are overloaded and return ResultCode::TRY_AGAIN. 

Ideally, it would be nice to implement some smarter throttling in Scribe.  But in practice, I've been able to tweak these config values appropriately.

-Anthony

[Frank DENIS]

Hello Anthony,

And thanks for your answer.

Obviously, reducing max_size in the secondary store would mitigate the storm when the primary store comes back up. The drawback is that unless the primary store doesn't fail for more than a few seconds, the secondary store will quickly be full up and a lot of data is bound to get lost.

Returning ResultCode::TRY_AGAIN isn't an option either. The PHP scribe backends don't know when they are going to get overload (ie. when the upstream scribe server is going to send a huge packet, straight away from the secondary store).

Ideally, when scribe switches back to the primary store, it should grab small chunks of the secondary store instead of fetching and sending everything in a single shot. It would also dramatically reduce memory usage, without losing data in case the primary store remains unavailable for a while.

[Anthony Giardullo]

Setting max_size will just limit the max size of each file created by a file store.  Once this limit is reached, the secondary store will start a new file.  No data will be lost.

-Anthony