#34 Incomplete syslog messages

[Felix]

Hi guys!

I have a remote syslog server based on the rsyslog daemon.
All messages from the Windows Server will be sent to this server over port 514 UDP with SNARE.
The rsyslog daemon writes every external log to a mysql database.

My problem is the following.
Syslog messages with german umlaut like ä, ü, ö, ß will be not completely stored in the database.
Only the string before an umlaut will be stored e.g.

Jul 29 12:47:06 2011 861 Security SYSTEM User Failure Audit <server> Detaillierte
Jul 29 12:47:16 2011 675 Security SYSTEM User Failure Audit <server> Kontoanmeldung Fehlgeschlagene Vorbest</server></server>

Attached you will find the screenshot from the SNARE web frontend with the complete messages.

I used the SnareForWindows-4.0.0.1-MultiArch.exe installer on the system.
Snare for Windows (X86 binary on 32 bit arch)

I think the problem is that SNARE does not encode the string correctly before sending it.
Could you please analyze the problem?

Maybe it could be a solution to rewrite the umlauts to normal letters.
e.g. ä = ae or ü = ue

Best regards
Felix

[Felix]

[David Mohr]

Hi Felix,

Thanks for the information, I'll do my best to reproduce this problem and find out why the messages are being truncated. Are you able to provide me with a brief network dump of messages arriving at the syslog server? I just need to confirm that it is in fact the agent that is truncating the message, not the syslog server. Going by the picture you attached, Snare for Windows is correctly rending the entire message.

Regards, David.

[David Mohr]

Hi Felix,

I've double checked strings containing umlaut characters (amoung others) and the full messages is arriving at the destination (Windows 2003 32 bit), so I'm inclined to think it is a problem with the syslog listener or perhaps even the database configuration.

Regards, David.

[Felix]

Hi David!

Thank you for investigation. You are right. The messages are fully transmitted to my rsyslogd server.
The problem seams to be the encoding of the message part.

Now I use 2 options to see the correct message.

1. $EscapeControlCharactersOnReceive off
   http://www.rsyslog.com/doc/rsconf1_escapecontrolcharactersonreceive.html

2. $Escape8BitCharactersOnReceive on
   http://www.rsyslog.com/doc/rsconf1_escape8bitcharsonreceive.html

E.g.
Aug 02 09:31:03 2011675SecuritySYSTEMUserFailure Audit<server>KontoanmeldungFehlgeschlagene Vorbest#344tigung: Benutzername….</server>

The german umlauts are still not correct but now I can watch the whole message.

Without these options the message would be shown like this.

Aug 02 09:27:58 2011#011675#011Security#011SYSTEM#011User#011Failure Audit#011<server>#011Kontoanmeldung#011#011Fehlgeschlagene Vorbest</server>

All whitespaces are escaped and the message is cutted if the first german umlaut appears.

I read on the RFC that the message part sould be encoded as a seven-bit ASCII string normally.
German umlauts are not included in this character set.
Is it possible to create a checkbox in the snare frontend to encode all messages to ASCII code?

Best regards
Felix

[David Mohr]

Hi Felix,

I can look into this option, but it is unlikely we will implement the feature without more demand. Most newer collection systems support 8-bit ASCII allowing for the use of umlauts, etc. Should demand increase, I will definitely re-evaluate the priority.

Regards, David.