#19 unicode error processing /var/log/secure

[Brian Davison]

Running fairly generic Fedora Core 6, fully updated.

[root@wume2 ~]# rpm -q fail2ban
fail2ban-0.6.2-3.fc6

fail2ban dies at startup with:

2007-03-28 06:30:16,348 ERROR: Type: 'UnicodeEncodeError'
Value: ('latin-1', u'Mar 26 10:34:24 wume2 sshd[2544]: pam_unix(sshd:auth): bad username [\ufeffroot]\n', 69, 70, 'ordinal not in range(256)')
TB: [('/usr/bin/fail2ban', 48, '?', 'fail2ban.main()'), ('/usr/share/fail2ban/fail2ban.py', 486, 'main', 'e = element[1].getFailures()'), ('/usr/share/fail2ban/logreader/logreader.py', 141, 'getFailures', "line = line.decode('utf-8').encode('latin-1')")]

The relevant line in my /var/log/secure is
Mar 26 10:34:24 wume2 sshd[2544]: pam_unix(sshd:auth): bad username [root]

but copying and pasting here won't capture the unicode, so here is od output:

% grep username /var/log/secure | od -a
0000000 M a r sp 2 6 sp 1 0 : 3 4 : 2 4 sp
0000020 w u m e 2 sp s s h d [ 2 5 4 4 ]
0000040 : sp p a m _ u n i x ( s s h d :
0000060 a u t h ) : sp b a d sp u s e r n
0000100 a m e sp [ o ; ? r o o t ] nl M a

-- Brian

[Cyril Jaquier]

Logged In: YES
user_id=933467
Originator: NO

Hi Brian,

Could you try this patch?

Thank you

Cyril

Index: /home/cyril/workspace/fail2ban-stable/logreader/logreader.py

--- /home/cyril/workspace/fail2ban-stable/logreader/logreader.py (revision 567)
+++ /home/cyril/workspace/fail2ban-stable/logreader/logreader.py (working copy)
@@ -137,8 +137,8 @@
lastLine = None
for line in logFile:
try:
- # Try to convert UTF-8 string to Latin-1
- line = line.decode('utf-8').encode('latin-1')
+ # Decode line to UTF-8
+ line = line.decode('utf-8')
except UnicodeDecodeError:
pass
if not self.hasTime(line):

[Brian Davison]

Logged In: YES
user_id=1139464
Originator: YES

Hi Cyril,

Thanks for the suggestion. It didn't work, but suggested another variation which might. When I applied your patch, I got other errors. Here are the relevant parts from fail2ban -vv:

2007-03-28 19:47:39,360 WARNING: SSH: ReBan 207.44.198.118
2007-03-28 19:47:39,360 DEBUG: iptables -I fail2ban-SSH 1 -s u'207.44.198.118' -j DROP
iptables v1.3.5: host/network `u207.44.198.118' not found
Try `iptables -h' or 'iptables --help' for more information.
2007-03-28 19:47:39,666 ERROR: 'iptables -I fail2ban-SSH 1 -s u'207.44.198.118' -j DROP' returned 512
2007-03-28 19:47:39,667 ERROR: Fail2Ban got an unhandled exception and died.
2007-03-28 19:47:39,668 ERROR: Type: 'ExternalError'
Value: ('Big Oops happened: situation is out of control. Something is wrong with your setup. Please check your settings',)
TB: [('/usr/bin/fail2ban', 48, '?', 'fail2ban.main()'), ('/usr/share/fail2ban/fail2ban.py', 549, 'main', 'raise ExternalError("Big Oops happened: situation is out of " +')]

However, changing the line from

line = line.decode('utf-8')

to

line = line

was sufficient to eliminate all of my errors and run properly. However, I don't know what the ramifications of this change are (and obviously I don't know python since I substituted an identity rather than rewrite anything else).

-- Brian