Re: [Scribeserver-users] Problems (and solutions :) ) while using Scribe with Ruby

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Hey Anthony,

I got bogged down with some other work but was finally able to run
Scribe with ruby today.

I took some time debugging what was the version expected by the
protocol and then actually probed the corresponding Python code. It
appears that the ruby code generated by Thrift for
Thrift::BinaryProtocol is outdated. Thankfully, the python code was
very readable and helped me nail down the problem for once and all.
The following comment in
/usr/lib/python2.5/site-packages/thrift/protocol/TBinaryProtocol.py
was especially helpful:
  # NastyHaxx. Python 2.4+ on 32-bit machines forces hex constants to be
  # positive, converting this into a long. If we hardcode the int value
  # instead it'll stay in 32 bit-land.

I made the corresponding changes in
/usr/local/lib/site_ruby/1.8/thrift/protocol/binaryprotocol.rb and
things started to look good as I was getting some meaningful debug
info for the first time.

The I compared the `read_message_begin' method in this class with the
Python version and pooh! The bug was right there staring straight into
my eyes. Can you guys review and check-in the following code in the
ruby branch:

    def read_message_begin
      version = read_i32
      if(version <0)
        if (version & VERSION_MASK != VERSION_1)
           raise ProtocolException.new(ProtocolException::BAD_VERSION,
'Missing version identifier')
        end
        type = version & 0x000000ff
        name = read_string
        seqid = read_i32
      else
        name = trans.readAll(version)
        type = read_byte
        seqid = read_i32
      end
      [name, type, seqid]
    end

Please note that this code does not check for a strict read condition
that was used by Python to restrict the actions in the else section.
I'll be benchmarking Scribe with Ruby day after tomorrow and will pass
on the results. All help on performance tuning will be appreciated.

Regards
Chandranshu

On Wed, Nov 26, 2008 at 7:34 PM, Chandranshu . <cha...@gm...> wrote:
> Hi Anthony,
>
> Thanks for all your helpful suggestions.
> I changed
>  client.Log(LogEntry.new(:category => "test", :message => "This is a
> test message"));
> to be
>  log_entry = LogEntry.new(:category => "test", :message => "This is a
> test message")
>  client.Log([log_entry])
>
> This ensured that the message reached the scribed server and got
> logged with appropriate category.  However, now I get the following
> error:
> /usr/local/lib/site_ruby/1.8/thrift/protocol/binaryprotocol.rb:82:in
> `read_message_begin': Missing version identifier
> (Thrift::ProtocolException)
>        from /usr/local/lib/site_ruby/1.8/thrift/client.rb:26:in `receive_message'
>        from ../src/gen-rb/scribe.rb:26:in `recv_Log'
>        from ../src/gen-rb/scribe.rb:18:in `Log'
>        from rscribe_cat:19
>
> I am still investigating why is this the case. If you can suggest
> something, it'll be greatly helpful.
>
> Regards
> Chandranshu
>
>
>
> On Wed, Nov 26, 2008 at 1:58 AM, Anthony Giardullo <an...@fa...> wrote:
>> You are correct.  Sorry, in my earlier email I should have you need to pass
>> a *list* of LogEntry's when calling Log.  See the declaration of the Log
>> procedure in if/scribe.thrift.
>>
>> -Anthony
>>
>> On 11/25/08 7:19 AM, "Chandranshu ." <cha...@gm...> wrote:
>>
>> Hi Anthony,
>>
>> I changed the client.Log("message") line to be
>> client.Log(LogEntry.new(:category => "test", :message => "This is a
>> test message"));
>>
>> Now, I'm getting the following exception:
>> /usr/local/lib/site_ruby/1.8/thrift/struct.rb:233:in
>> `write_container': undefined method `size' for <LogEntry
>> category:"test", message:"This is a test message">:LogEntry
>> (NoMethodError)
>>         from /usr/local/lib/site_ruby/1.8/thrift/struct.rb:113:in `write'
>>         from /usr/local/lib/site_ruby/1.8/thrift/struct.rb:69:in
>> `each_field'
>>         from /usr/local/lib/site_ruby/1.8/thrift/struct.rb:68:in `each'
>>         from /usr/local/lib/site_ruby/1.8/thrift/struct.rb:68:in
>> `each_field'
>>         from /usr/local/lib/site_ruby/1.8/thrift/struct.rb:109:in `write'
>>         from /usr/local/lib/site_ruby/1.8/thrift/client.rb:16:in
>> `send_message'
>>         from ../src/gen-rb/scribe.rb:22:in `send_Log'
>>         from ../src/gen-rb/scribe.rb:17:in `Log'
>>         from rscribe_cat:18
>>
>> I'm suspecting that the size method should not have been called on the
>> LogEntry class but actually on the Thrift::Types::LIST class which is
>> the type of the :element attribute of :message field of Log_args
>> class.
>>
>> Can you please verify that this should actually be the case because in
>> that case, it'll be a bug in Thrift rather than Scribe.
>>
>> Regards
>> Chandranshu
>> P.S.: I think I understand the source of bug. It is the following
>> method in the Thrift::Struct class which is actually setting the
>> message field to the LogEntry instance that I'm passing rather than
>> creating a Thrift::LIST out of it. I didn't get time to check the
>> Thrift.check_type method to see if it changes the 'value' parameter.
>>     def self.field_accessor(klass, *fields)
>>       fields.each do |field|
>>         klass.send :attr_reader, field
>>         klass.send :define_method, "#{field}=" do |value|
>>           Thrift.check_type(value, klass::FIELDS.values.find { |f|
>> f[:name].to_s == field.to_s }, field) if Thrift.type_checking
>>           instance_variable_set("@#{field}", value)
>>         end
>>       end
>>     end
>>
>> On Tue, Nov 25, 2008 at 10:11 AM, Anthony Giardullo
>> <an...@fa...> wrote:
>>> Ok, it looks like your passing in a string to client.Log().  What you need
>>> to pass in is a LogEntry.  This class is defined in the thrift -generated
>>> file scribe_types.rb.
>>>
>>> -Anthony
>>>
>>>
>>> On 11/24/08 3:51 AM, "Chandranshu ." <cha...@gm...> wrote:
>>>
>>> Hi Anthony,
>>>
>>> I am using the following code to log the messages. I didn't find any
>>> method to create a Log_Entry pair.
>>>
>>> The only methods I found were Log, send_Log and recv_Log. There are
>>> other classes called Log_args and Log_result but I'm not sure how I
>>> should be using them. Interestingly, there is the following line in
>>> the Log_args class which is using LogEntry but I don't see where it
>>> can find it:
>>>  MESSAGES => {:type => Thrift::Types::LIST, :name => 'messages',
>>> :element => {:type => Thrift::Types::STRUCT, :class => LogEntry}}
>>>
>>> Regards
>>> Chandranshu
>>>
>>> ========= rscribe_cat.rb ===========
>>>
>>> # include thrift-generated code
>>> $:.push('../src/gen-rb')
>>>
>>> #require 'thrift/transport'
>>> require 'thrift/transport/tsocket'
>>> require 'thrift/protocol/tbinaryprotocol'
>>> require 'thrift/server/tserver'
>>>
>>> require '../src/gen-rb/scribe'
>>>
>>> begin
>>>   socket = Thrift::Socket.new('localhost', 1463)
>>>   transport = Thrift::FramedTransport.new(socket)
>>>   protocol = Thrift::BinaryProtocol.new(transport)
>>>   client = Scribe::Client.new(protocol)
>>>
>>>   transport.open()
>>>   client.Log(ARGV[1]);
>>>
>>>   transport.close()
>>>
>>> rescue Thrift::Exception => tx
>>>   print 'Thrift::Exception: ', tx.message, "\\n"
>>> end
>>>
>>> ============ rscribe_cat.rb ends ============
>>>
>>> On Sat, Nov 22, 2008 at 8:09 AM, Anthony Giardullo <an...@fa...>
>>> wrote:
>>>> Chandranshu,
>>>>
>>>> Glad to see you figured out many of these issues getting the Ruby code
>>>> generated.  I am going to pass on your comments to our Thrift gurus.
>>>>
>>>> Quick question, can you show me how exactly you are calling your
>>>> rscribe_cat
>>>> script?  Are you creating a Log_Entry pair with a category and message?
>>>>
>>>> -Anthony
>>>>
>>>> On 11/21/08 4:15 AM, "Chandranshu ." <cha...@gm...> wrote:
>>>>
>>>> Resending in plain text format:
>>>> I'm  trying to use a Ruby Client named rscribe_cat.rb to post messages
>>>> to the scribed server started using the examples/example1.conf file
>>>> that came with the distribution. The file is attached herewith.
>>>>
>>>> I'm summarizing the problems I faced in setting up Scribe for Ruby:
>>>>
>>>>    1. Edit the <SCRIBE_BASE>/src/Makefile.in. Replace the following line
>>>>
>>>>       $(eval $(call thrift_template,.,$(srcdir)/../if/scribe.thrift,
>>>> -I $(thrift_home)/share/ -I $(fb303_home)/share/ --cpp --py --php))
>>>>
>>>>       with
>>>>
>>>>       $(eval $(call thrift_template,.,$(srcdir)/../if/scribe.thrift,
>>>> -I $(thrift_home)/share/ -I $(fb303_home)/share/ --cpp --py --php
>>>> --gen rb))
>>>>
>>>>    2. Now run the following command from <SCRIBE_BASE> again:
>>>>
>>>>       sudo make install
>>>>
>>>>       Verify that a gen_rb folder has been created in the
>>>> <SCRIBE_BASE>/src directory and it contains the scribe.rb file. This
>>>> file contains the ruby module that'll enable us to use it from inside
>>>> ruby code. Also, add this path to $: in your Ruby Client.
>>>>    3. However, the generated module contains the module name as scribe
>>>> and trying to use it will result in Ruby complaining that
>>>>
>>>>       ../src/gen-rb/scribe.rb:12: class/module name must be CONSTANT
>>>> (SyntaxError)
>>>>
>>>>       The leading ".." is because I was writing the ruby client in the
>>>> examples directory. Open the "../src/gen-rb/scribe.rb" file and change
>>>> the module name to be Scribe with a capital 'S'.
>>>>    4. The next error you'll get is:
>>>>
>>>>         ../src/gen-rb/scribe.rb:9:in `require': no such file to load
>>>> -- FacebookService (LoadError)
>>>>
>>>>       This is because when installing fb303, ruby classes were left
>>>> out again. Generate the ruby classes for the fb303 service interface
>>>> from the <FB303_BASE> directory:
>>>>
>>>>         sudo thrift -o /usr/local/share -I /usr/local/share  --gen rb
>>>> if/fb303.thrift
>>>>
>>>>       Unfortunately, ruby files in /usr/local/share are not picked up
>>>> Ruby by default. So, you'll have to manually copy over the files to
>>>> /usr/local/lib/site_ruby/1.8/ or the corresponding directory on your
>>>> machine.
>>>>    5. The next error is the one that consumed most of my time:
>>>>
>>>>       /usr/local/lib/site_ruby/1.8/fb303_types.rb:8:in `require': no
>>>> such file to load --
>>>> /usr/local/lib/site_ruby/1.8/reflection_limited_types
>>>>
>>>>       Looking at the code, I found that it was using
>>>> File.dirname(FILE) to load the reflection_limited_types file. Bad for
>>>> me. After a huge updatedb call, I was able to locate the corresponding
>>>> thrift file. Use the following to remedy this:
>>>>
>>>>       sudo thrift -o /usr/local/lib/site_ruby/1.8 --gen rb
>>>> /usr/local/share/thrift/if/reflection_limited.thrift
>>>>
>>>>       and then copy out the files from the gen-rb directory to the
>>>> /usr/local/lib/site_ruby/1.8 directory. After this point, you
>>>> shouldn't get any more syntax errors. Though it'd be great if the ruby
>>>> classes are also generated at the build time for Thrift, fb303 and
>>>> scribe. I think generating the ruby classes for the
>>>> thrift/if/reflection_limited.thrift, <fb303_base>/if/fb303.thrift and
>>>> $(srcdir)/../if/scribe.thrift files and putting them in the right
>>>> place should do.
>>>>    6. What I'm still struggling with:
>>>>
>>>>
>>>>       /usr/local/lib/site_ruby/1.8/thrift/protocol/binaryprotocol.rb:82:in
>>>> `read_message_begin': Missing version identifier
>>>> (Thrift::ProtocolException)
>>>>               from /usr/local/lib/site_ruby/1.8/thrift/client.rb:26:in
>>>> `receive_message'
>>>>
>>>>               from ../src/gen-rb/scribe.rb:26:in `recv_Log'
>>>>               from ../src/gen-rb/scribe.rb:18:in `Log'
>>>>               from rscribe_cat:18
>>>>
>>>> If someone has solved this problem or if the developers can guide me
>>>> as to what might be going wrong, I'll try and fix it and post a patch.
>>>>
>>>> Anyways, good framework Guys!!
>>>>
>>>> Thanks and regards,
>>>> Chandranshu
>>>>
>>>> -------------------------------------------------------------------------
>>>> This SF.Net email is sponsored by the Moblin Your Move Developer's
>>>> challenge
>>>> Build the coolest Linux based applications with Moblin SDK & win great
>>>> prizes
>>>> Grand prize is a trip for two to an Open Source event anywhere in the
>>>> world
>>>> http://moblin-contest.org/redirect.php?banner_id=100&url=/
>>>> _______________________________________________
>>>> Scribeserver-users mailing list
>>>> Scr...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/scribeserver-users
>>>>
>>>>
>>>
>>>
>>
>>
>