#42 sr_sarra - notice [ERROR] Type: <class 'ValueError'>, Value: String too long

[Jun Hu]

sr_sarra queue got stuck with ~500000 messages
I think something in the message are too long like the path!

sr_sarra_xxx.log
…
2016-06-23 19:43:48,194 [ERROR] Type: <class 'valueerror'="">, Value: String too long
2016-06-23 19:43:48,194 [ERROR] Sleeping 5 seconds ... and reconnecting
…</class>

After checking the code, the error is raise on /usr/local/lib/python3.2/dist-packages/amqplib/client_0_8/serialization.py

def write_shortstr(self, s):
"""
Write a string up to 255 bytes long (after any encoding).
if len(s) > 255:
raise ValueError('String too long')
…

grep write_shortstr /usr/share/pyshared/amqplib/client_0_8/*
...
/usr/share/pyshared/amqplib/client_0_8/channel.py: args.write_shortstr(exchange)
/usr/share/pyshared/amqplib/client_0_8/channel.py: args.write_shortstr(queue)
/usr/share/pyshared/amqplib/client_0_8/channel.py: args.write_shortstr(exchange)
/usr/share/pyshared/amqplib/client_0_8/channel.py: args.write_shortstr(routing_key)
...

google says:

http://stackoverflow.com/questions/30716002/is-there-a-size-limit-on-a-rabbitmq-message-header

lead me to here:
http://docs.oasis-open.org/amqp/core/v1.0/os/amqp-core-types-v1.0-os.html#type-string

which gives a 'short string as 255 characers... indeed, but also a long string...

We probably need to truncate topics to 255 characters...
ugh...

[psilva]

one solution: Possibly encode topics to shorten them? but harder to work with...

another: Remove File names from topic. (only folders) buys us a bit...

still thinking...

[psilva]

confirmed that the header is limited to 255 characters by AMQP standard.
No practical expectation of fix to be expected.

[psilva]

Should document in guides/man pages, etc...

[psilva]

commit df835008c465dbe0cbba532d6b5b8f732d4a41c4
truncates all headers to 255 characters. This should prevent it from hanging/looping.
Also prints a warning message to the log about what has happenned.

This will get things to at least work as well as they can for now.
(I set it to 255, but haven't tested it, the patch just passes the flow_test)
Still need to test the patch...

[psilva]

more patches:
b87f967da678e311317e8c3e9c1086e3794245c8 -- self-test included now, and the original patch didn't work.
6e7e484f398c7f2aeeb1293b9152f0dbd4faf3d2 -- plugin was left out of above patch, added here.
7d822d69636dbbcf14e2bc8ba44046e94d5216e1 -- truncation re-done to account for utf8 encoding.
fcf74ded57b75e8cb56564959e6ca6dc0546d5c4 -- Added mention of limit in documentation.

OK, should be complete now.

[psilva]

Status of patches: All headers are truncated prior to being passed to AMQP layer, avoiding the looping failure. Question: Is there something more to be done?

Test topic for length, and create headers topic2 topic3 topic4 ?
won't help AMQP will only pay attention to the topic header. More filtering already possible with accept/reject.

It also prints a warning about truncation. is that a good thing?

Thinking/Alternatives/Review prior to closing...

[psilva]

more patches:

53575cd7e01d92249e9290d7eb5e64e91e0cd700 - truncate topic as well as all headers.
d53dcf743cab63c8e51330f64a8542553070c970 - truncate at 253, not 255.

OK, now it should really work.

[psilva]

I just noticed that header['filename'] is used in sr_winnow to store stuff in the cache.
That is very likely unwise. but how to fix? ... likely should take the filename from the message.. (not a header.)

c185159428e54d53c7a60b5e096ae6ae10fa0365 - commit fixes that by replacing it
with msg.url.path.

passes self-test, which includes winnow.

[psilva]

fixes released in 10a2... waiting for verification that problem is gone.

[psilva]

Ugh. Found another case. new patch... ( bbdcfb735f32ffacdcc15aff6b495e2dede44459 )

[psilva]

last fixes are in master.