Sync failure - "long int exceeds XML-RPC limits"

I'm having this problem, too. Although denyhosts keeps running (or appears to, at least), it fills its logfile with a lot of noise. I don't think the sync works either.

Anyone heard rumor of a fix?

Here this issue gets discussed, it's apparently caused by running on a 64-bit OS:

http://mail.python.org/pipermail/python-dev/2009-July/090645.html

I'm no programmer myself though, so unfortunately I can't fix this. Besides that, it will probably need modifications on both client and server part, where the server part is still closed source AFAIK.

Can you try using python2.6? It may fix the issue but not sure. Are you running DenyHosts on a 64-bit kernel?

You could also try editing sync.py, line 47:

from:
timestamp = long(timestamp.strip())
to:
timestamp = timestamp.strip()

That should fix the issue w/ your current version of python.

Thanks for the help with this issue.

First off, I just realized that I didn't include my distribution/kernel version/etc or the Python version. I'm not sure what I was thinking when I wrote this up, sorry! I'm using Debian, kernel version 2.6.30-1-amd64. My Python version is 2.5.4. I'm using DenyHosts installed from apt, version 2.6-6.

I tried removing long() from line 47 of sync.py, but I still get the sync error. I'll see if I can get Python upgraded, but from what I can tell it's going to be a lot of pain and suffering to get Python 2.6 on Debian. (I've heard good things about 2.6 in comparison to 2.5, though, so I might try upgrading it anyway and hope nothing explodes.)

Thanks again for the help! I'll try to report back soon about whether the Python 2.6 upgrade fixes the issue.

Did you restart the DenyHosts deamon after modifying sync.py?

I am also running Debian (debian testing on AMD64), kernel 2.6.30-2-amd64. I have just upgraded Python to 2.6, but the error remains:

2010-01-01 20:27:54,594 - sync : ERROR long int exceeds XML-RPC limits
Traceback (most recent call last):
File "/usr/share/denyhosts/DenyHosts/sync.py", line 117, in receive_new_hosts
self.__prefs.get("SYNC_DOWNLOAD_RESILIENCY"))
File "/usr/lib/python2.6/xmlrpclib.py", line 1199, in __call__
return self.__send(self.__name, args)
File "/usr/lib/python2.6/xmlrpclib.py", line 1483, in __request
allow_none=self.__allow_none)
File "/usr/lib/python2.6/xmlrpclib.py", line 1132, in dumps
data = m.dumps(params)
File "/usr/lib/python2.6/xmlrpclib.py", line 677, in dumps
dump(v, write)
File "/usr/lib/python2.6/xmlrpclib.py", line 699, in __dump
f(self, value, write)
File "/usr/lib/python2.6/xmlrpclib.py", line 725, in dump_long
raise OverflowError, "long int exceeds XML-RPC limits"

After that, I removed long() from line 47 of sync.py and restarted denyhosts. Now I get this, which doesn't seem right either ;-)

2010-01-01 20:54:12,616 - sync : INFO received 0 new hosts

I'm running it with the change to line 47 in sync.py and it's working fine for me. Is it still returning 0 every time for you? Can you run it in --debug mode and see if it logs anything else that might be of use?

This is odd, running with python 2.6 and --debug I get a connection timout. I tested using telnet and the server and port are reacheable just fine.

2010-01-07 09:07:33,917 - denyhosts : INFO DenyHosts launched with the following args:
2010-01-07 09:07:33,918 - denyhosts : INFO /usr/sbin/denyhosts --daemon --debug --purge --config=/etc/denyhosts.conf --config=/etc/denyhosts.conf
2010-01-07 09:07:33,918 - prefs : INFO DenyHosts configuration settings:
2010-01-07 09:07:33,918 - prefs : INFO ADMIN_EMAIL: [root@localhost]
2010-01-07 09:07:33,918 - prefs : INFO AGE_RESET_INVALID: [864000]
2010-01-07 09:07:33,918 - prefs : INFO AGE_RESET_RESTRICTED: [2160000]
2010-01-07 09:07:33,918 - prefs : INFO AGE_RESET_ROOT: [2160000]
2010-01-07 09:07:33,918 - prefs : INFO AGE_RESET_VALID: [432000]
2010-01-07 09:07:33,919 - prefs : INFO ALLOWED_HOSTS_HOSTNAME_LOOKUP: [no]
2010-01-07 09:07:33,919 - prefs : INFO BLOCK_SERVICE: [sshd]
2010-01-07 09:07:33,919 - prefs : INFO DAEMON_LOG: [/var/log/denyhosts]
2010-01-07 09:07:33,919 - prefs : INFO DAEMON_LOG_MESSAGE_FORMAT: [%(asctime)s - %(name)-12s: %(levelname)-8s %(message)s]
2010-01-07 09:07:33,919 - prefs : INFO DAEMON_LOG_TIME_FORMAT: [None]
2010-01-07 09:07:33,919 - prefs : INFO DAEMON_PURGE: [3600]
2010-01-07 09:07:33,919 - prefs : INFO DAEMON_SLEEP: [30]
2010-01-07 09:07:33,919 - prefs : INFO DENY_THRESHOLD_INVALID: [5]
2010-01-07 09:07:33,919 - prefs : INFO DENY_THRESHOLD_RESTRICTED: [1]
2010-01-07 09:07:33,920 - prefs : INFO DENY_THRESHOLD_ROOT: [1]
2010-01-07 09:07:33,920 - prefs : INFO DENY_THRESHOLD_VALID: [10]
2010-01-07 09:07:33,920 - prefs : INFO FAILED_ENTRY_REGEX: [None]
2010-01-07 09:07:33,920 - prefs : INFO FAILED_ENTRY_REGEX2: [None]
2010-01-07 09:07:33,920 - prefs : INFO FAILED_ENTRY_REGEX3: [None]
2010-01-07 09:07:33,920 - prefs : INFO FAILED_ENTRY_REGEX4: [None]
2010-01-07 09:07:33,920 - prefs : INFO FAILED_ENTRY_REGEX5: [None]
2010-01-07 09:07:33,920 - prefs : INFO FAILED_ENTRY_REGEX6: [None]
2010-01-07 09:07:33,921 - prefs : INFO FAILED_ENTRY_REGEX7: [None]
2010-01-07 09:07:33,921 - prefs : INFO HOSTNAME_LOOKUP: [YES]
2010-01-07 09:07:33,921 - prefs : INFO HOSTS_DENY: [/etc/hosts.deny]
2010-01-07 09:07:33,921 - prefs : INFO LOCK_FILE: [/var/run/denyhosts.pid]
2010-01-07 09:07:33,921 - prefs : INFO PLUGIN_DENY: [None]
2010-01-07 09:07:33,921 - prefs : INFO PLUGIN_PURGE: [None]
2010-01-07 09:07:33,921 - prefs : INFO PURGE_DENY: [604800]
2010-01-07 09:07:33,921 - prefs : INFO PURGE_THRESHOLD: [0]
2010-01-07 09:07:33,922 - prefs : INFO RESET_ON_SUCCESS: [no]
2010-01-07 09:07:33,922 - prefs : INFO SECURE_LOG: [/var/log/auth.log]
2010-01-07 09:07:33,922 - prefs : INFO SMTP_DATE_FORMAT: [%a, %d %b %Y %H:%M:%S %z]
2010-01-07 09:07:33,922 - prefs : INFO SMTP_FROM: [DenyHosts <nobody@localhost>]
2010-01-07 09:07:33,922 - prefs : INFO SMTP_HOST: [localhost]
2010-01-07 09:07:33,922 - prefs : INFO SMTP_PASSWORD: [None]
2010-01-07 09:07:33,922 - prefs : INFO SMTP_PORT: [25]
2010-01-07 09:07:33,922 - prefs : INFO SMTP_SUBJECT: [DenyHosts Report]
2010-01-07 09:07:33,922 - prefs : INFO SMTP_USERNAME: [None]
2010-01-07 09:07:33,923 - prefs : INFO SSHD_FORMAT_REGEX: [None]
2010-01-07 09:07:33,923 - prefs : INFO SUCCESSFUL_ENTRY_REGEX: [None]
2010-01-07 09:07:33,923 - prefs : INFO SUSPICIOUS_LOGIN_REPORT_ALLOWED_HOSTS: [YES]
2010-01-07 09:07:33,923 - prefs : INFO SYNC_DOWNLOAD: [yes]
2010-01-07 09:07:33,923 - prefs : INFO SYNC_DOWNLOAD_RESILIENCY: [18000]
2010-01-07 09:07:33,923 - prefs : INFO SYNC_DOWNLOAD_THRESHOLD: [3]
2010-01-07 09:07:33,923 - prefs : INFO SYNC_INTERVAL: [600]
2010-01-07 09:07:33,923 - prefs : INFO SYNC_SERVER: [http://xmlrpc.denyhosts.net:9911]
2010-01-07 09:07:33,924 - prefs : INFO SYNC_UPLOAD: [yes]
2010-01-07 09:07:33,924 - prefs : INFO SYSLOG_REPORT: [no]
2010-01-07 09:07:33,924 - prefs : INFO WORK_DIR: [/var/lib/denyhosts]
2010-01-07 09:07:33,926 - denyhosts : INFO restricted: set([])
2010-01-07 09:07:33,926 - filetracker : DEBUG __get_current_offset():
2010-01-07 09:07:33,926 - filetracker : DEBUG first_line: Dec 26 13:44:52 haanjdj sshd[1423]: Server listening on 0.0.0.0 port 22.
2010-01-07 09:07:33,926 - filetracker : DEBUG offset: 80972
2010-01-07 09:07:33,926 - AllowedHosts: DEBUG initializing AllowedHosts
2010-01-07 09:07:33,926 - AllowedHosts: DEBUG Could not open /var/lib/denyhosts/allowed-hosts - [Errno 2] No such file or directory: '/var/lib/denyhosts/allowed-hosts'
2010-01-07 09:07:33,927 - AllowedHosts: DEBUG done initializing AllowedHosts
2010-01-07 09:07:33,927 - filetracker : DEBUG __get_last_offset():
2010-01-07 09:07:33,927 - filetracker : DEBUG first_line: Dec 26 13:44:52 haanjdj sshd[1423]: Server listening on 0.0.0.0 port 22.
2010-01-07 09:07:33,927 - filetracker : DEBUG offset: 80972
2010-01-07 09:07:33,927 - filetracker : DEBUG get_offset():
2010-01-07 09:07:33,927 - filetracker : DEBUG offset: None
2010-01-07 09:07:33,927 - denyhosts : INFO launching DenyHosts daemon (version 2.6)...
2010-01-07 09:07:33,930 - denyhosts : INFO DenyHosts daemon is now running, pid: 2124
2010-01-07 09:07:33,931 - denyhosts : INFO send daemon process a TERM signal to terminate cleanly
2010-01-07 09:07:33,931 - denyhosts : INFO eg. kill -TERM 2124
2010-01-07 09:07:33,932 - denyhosts : INFO monitoring log: /var/log/auth.log
2010-01-07 09:07:33,932 - denyhosts : INFO sync_time: 600
2010-01-07 09:07:33,932 - denyhosts : INFO daemon_purge: 3600
2010-01-07 09:07:33,933 - denyhosts : INFO daemon_sleep: 30
2010-01-07 09:07:33,933 - denyhosts : INFO purge_sleep_ratio: 120
2010-01-07 09:07:33,933 - denyhosts : INFO sync_time: : 600
2010-01-07 09:07:33,933 - denyhosts : INFO sync_sleep_ratio: 20
2010-01-07 09:17:04,505 - denyhosts : DEBUG /var/log/auth.log has additional data
2010-01-07 09:17:04,509 - denyhosts : DEBUG no new denied hosts
2010-01-07 09:17:04,509 - denyhosts : DEBUG no new suspicious logins
2010-01-07 09:17:34,540 - denyhosts : DEBUG sync upload
2010-01-07 09:17:34,540 - sync : DEBUG send_new_hosts()
2010-01-07 09:17:34,541 - denyhosts : DEBUG sync download
2010-01-07 09:17:34,541 - sync : DEBUG receive_new_hosts()
2010-01-07 09:20:43,553 - sync : ERROR [Errno 110] Connection timed out
Traceback (most recent call last):
File "/usr/share/denyhosts/DenyHosts/sync.py", line 117, in receive_new_hosts
self.__prefs.get("SYNC_DOWNLOAD_RESILIENCY"))
File "/usr/lib/python2.6/xmlrpclib.py", line 1199, in __call__
return self.__send(self.__name, args)
File "/usr/lib/python2.6/xmlrpclib.py", line 1489, in __request
verbose=self.__verbose
File "/usr/lib/python2.6/xmlrpclib.py", line 1235, in request
self.send_content(h, request_body)
File "/usr/lib/python2.6/xmlrpclib.py", line 1349, in send_content
connection.endheaders()
File "/usr/lib/python2.6/httplib.py", line 892, in endheaders
self._send_output()
File "/usr/lib/python2.6/httplib.py", line 764, in _send_output
self.send(msg)
File "/usr/lib/python2.6/httplib.py", line 723, in send
self.connect()
File "/usr/lib/python2.6/httplib.py", line 704, in connect
self.timeout)
File "/usr/lib/python2.6/socket.py", line 514, in create_connection
raise error, msg
error: [Errno 110] Connection timed out

I believe the last entry may have been a temporary network glitch, I tested a second time, and now I get '0 hosts' when syncing with --debug, with both python 2.5 and 2.6.

With the edit to lin 47 of sync.py, it no longer throws an error but it is also not syncing properly. My deny file only has 20-some odd records. I'm running AMD64 lucid. The relevant portion from /var/log/denyhosts follows:

2010-04-19 12:39:06,035 - denyhosts : DEBUG new hosts: []
2010-04-19 12:39:06,036 - denyhosts : DEBUG no new denied hosts
2010-04-19 12:39:06,036 - denyhosts : DEBUG no new suspicious logins
2010-04-19 13:01:07,039 - denyfileutil: DEBUG relative cutoff: 604800 (seconds)
2010-04-19 13:01:07,039 - denyfileutil: DEBUG absolute cutoff: 1271091667 (epoch)
2010-04-19 13:01:07,040 - denyfileutil: INFO purging entries older than: Mon Apr 12 13:01:07 2010
2010-04-19 13:01:07,064 - denyfileutil: INFO num entries purged: 0
2010-04-19 13:01:07,065 - denyhosts : DEBUG sync upload
2010-04-19 13:01:07,065 - sync : DEBUG send_new_hosts()
2010-04-19 13:01:07,065 - denyhosts : DEBUG sync download
2010-04-19 13:01:07,066 - sync : DEBUG receive_new_hosts()
2010-04-19 13:01:07,795 - sync : INFO received 0 new hosts
2010-04-19 13:02:37,878 - denyhosts : DEBUG /var/log/auth.log has additional data
2010-04-19 13:02:38,035 - denyhosts : DEBUG new hosts: []
2010-04-19 13:02:38,035 - denyhosts : DEBUG no new denied hosts
2010-04-19 13:02:38,035 - denyhosts : DEBUG no new suspicious logins

Made edit to line 47 in sync.py. It throws a new error

<Fault 1: 'exceptions.ValueError:invalid literal for long(): '>
Traceback (most recent call last):
File "/usr/local/lib/python2.6/site-packages/DenyHosts/sync.py", line 118, in receive_new_hosts
self.__prefs.get("SYNC_DOWNLOAD_RESILIENCY"))
File "/usr/local/lib/python2.6/xmlrpclib.py", line 1199, in __call__
return self.__send(self.__name, args)
File "/usr/local/lib/python2.6/xmlrpclib.py", line 1489, in __request
verbose=self.__verbose
File "/usr/local/lib/python2.6/xmlrpclib.py", line 1253, in request
return self._parse_response(h.getfile(), sock)
File "/usr/local/lib/python2.6/xmlrpclib.py", line 1392, in _parse_response
return u.close()
File "/usr/local/lib/python2.6/xmlrpclib.py", line 838, in close
raise Fault(**self._stack[0])
Fault: <Fault 1: 'exceptions.ValueError:invalid literal for long(): '>

I am also affected by this
2.6.32-22-generic #33-Ubuntu SMP Wed Apr 28 13:28:05 UTC 2010 x86_64 GNU/Linux

I am also affected by this
2.6.26-2-686 #1 SMP Tue Mar 9 17:35:51 UTC 2010 i686 GNU/Linux

Ditto for me...

Linux 2.6.18-194.3.1.el5 #1 SMP Thu May 13 13:09:10 EDT 2010 i686 i386

@phil_schwartz:
You can also change the long() to str(). Since the problem that XML-RPC is moaning about is the integer value, so if you give it a string, it is happy.
from:
timestamp = long(timestamp.strip())
to:
timestamp = str(timestamp.strip())

But the problem is, I still get 0 hosts when running denyhosts. I'll try debugging some more.

send_new_hosts()
receive_new_hosts()
received 0 new hosts

Ok, my mistake. str() doesn't work. :(

But I now get the timeout error as well, but if the timeout error doesn't occur, I still get 0 hosts. Weird. Any ideas Phil?

I haven't been able to reproduce the timeout issue. Perhaps the connection to/from your particular host to the sync server is spotty. As for the 0 new hosts, that is peculiar and I'm not sure why that is always the case.. Around line 115 of sync.py (in the received_new_hosts method and just before the line that starts with " data = ... ") could you add this line:

info("%s - %s - %s - %s", timestamp, self.__prefs.get("SYNC_DOWNLOAD_THRESHOLD"), self.__hosts_added, self.__prefs.get("SYNC_DOWNLOAD_RESILIENCY"))

it should appear all on 1 line and left-align with the " data = ..." line. Then restart DH and post what is logged during the sync.

The output I get:

2010-05-25 12:23:52,948 - sync : INFO 12722831301272797139127468778512746880861274688175127468819012746882451274688450127468845612746895351274689541127469096212746946471274698252127470185812747054641274709070127471267512747162791274719884127472348912747270941274730698127473430312747379081274741513127474511812747487231274752328127475593412747595391274763144127476674812747703541274773959127477497212747758051274779428 - 3 - [] - 18000
2010-05-25 12:23:54,613 - sync : INFO received 0 new hosts

And still nothing in hosts.deny

Not sure if this is the culprit or not, but the timestamp you are passing is corrupt. Can you check your WORK_DIR/sync-timestamp file to see if it really contains that huge string? If so, can you stop DH and replace the contents of the file with something more sane, eg. (1274802611) and then restart DH.

Continue to monitor the log and see if the first sync works (it should) and if subsequent syncs fail. If they do, can you then check the sync-timestamp file again? If it contains a huge string then I need to figure out why that's occurring. I have tried to reproduce this on python 2.6 w/o any success. It appears that instead of overwriting the timestamp value, it's appending to it.

I checked the timestamp and it had some huge string. I edited it down to a single timestamp and lo and behold, it synced. But when I checked the timestamp, there were two time stamps concatenated. So for some reason, rather than overwriting the file, it is simply adding to the end. I am guessing that the second attempt to sync in 50 minutes will fail.

I have the same problem with the timestamp file, but also when I sync for the first time, I get:

"Error synchronizing data
name 'info' is not defined"

Rewriting line 57 as:
fp.write(timestamp+'\n')
along with the prior edit to line 47 appears to fix the sync issue.

I've made changes to line 57 and 47 as suggested bewlow. Get the same error as krugerl1 during syncing. The sync-received is populating with new hosts though.

Error synchronizing data
name 'info' is not defined

At line 55 I made the following change

from
fp = open(os.path.join(self.__work_dir,
SYNC_TIMESTAMP), "a")

to
fp = open(os.path.join(self.__work_dir,
SYNC_TIMESTAMP), "w")

I am not familiar with python, but the "a" probably means append, and I am guessing that "w" means write. I have not successfully snched twice in a row with the time stamp file showing a single time stamp.