#2815 UTF-8 chars never displayed in walk output for 5.7.3

[Mike Stupalov]

I tried to fetch OIDs with an UTF-8 chars inside string from same server but with different client versions:

NET-SNMP version: 5.4.3

SNMPv2-MIB::sysLocation.0 = STRING: Rue de la Métallurgie, 4530 Villers-le-Bouillet, België

NET-SNMP version: 5.7.3

SNMPv2-MIB::sysLocation.0 = STRING: Rue de la M..tallurgie, 4530 Villers-le-Bouillet, Belgi..

in both cases used same command without any additional output modifiers:

snmpget -v2c -c public hostname SNMPv2-MIB::sysLocation.0

locale

$ locale
LANG=en_US.UTF-8
LANGUAGE=en
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES=C
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

net-snmp installed from default system repository, system is:
5.4.3 - Ubuntu 12.04.5 LTS
5.7.3 - Ubuntu 16.04.3 LTS

In 5.7.3 all UTF8 chars replaced with 2 point chars, and seems as this is corresponds to output modifier -Oa. But this modifier is not used anywhere.

[Mike Stupalov]

Issue appear after revision: [7f05da]

and related to BUGs: [#2308] and [#1874] with notifications by snmptrapd.

I think need apply such trouble only for snmptrapd.

When I revert on my local install this change, issue with UTF8 was gone.

[Mike Stupalov]

Issue not "linux" specific, also affected by any other platform (BSD/Windows/etc).

I can not do any patch, since not so much c coder..

[Mike Stupalov]

Oh, I see seems as same issue with possible patch: [#2655]

Update: patch from that issue worked for me.

[Bill Fenner]

Since it's a heuristic being given arbitrary input, I'd rather see it validating that the subsequent N octets are within the [0x80,0xbf] range. Perhaps it would be better to use something like http://bjoern.hoehrmann.de/utf-8/decoder/dfa/ that can quickly determine "is this a UTF-8 string?"

[Mike Stupalov]

Sorry for delay..
As I said I'm not C coder..

but as I see in that code part used validating ASCII string by each char.
Probably detect whole string as UTF-8 is better (and faster), but I not know how test this.
I use for test only already produced patches.

[Mike Stupalov]

Bill, any chance for see into issue?

[Mike Stupalov]

bump..
I think this is HIGH priority before 5.8 release!

[Bart Van Assche]

Thanks for the detailed analysis. Can you test the attached patch (so far only tested against the master branch)?

[Mike Stupalov]

Yes, this patch fixed my case:

$ snmpwalk -v2c -cpublic localhost sysContact
SNMPv2-MIB::sysContact.0 = STRING: "Майк Ступалов" <mike@observium.org>
$ snmpwalk -v2c -cpublic localhost sysLocation
SNMPv2-MIB::sysLocation.0 = STRING: Rue de la Métallurgie, 4530 Villers-le-Bouillet, België

(I was rebuild 5.7.3 package from ubuntu repository with patch)

Tnx!

[Bart Van Assche]

The aforementioned patch has been checked in on the v5.7 and master branches so I'm closing this ticket.