#46 Sync failure - "long int exceeds XML-RPC limits"

open
nobody
None
5
2015-04-16
2009-11-16
Keith Buck
No

DenyHosts successfully syncs twice, downloading 50 hosts each time, and then on the third sync it dies with this traceback when downloading hosts (upload is not affected):

Traceback (most recent call last):
File "/usr/share/denyhosts/DenyHosts/sync.py", line 117, in receive_new_hosts
self.__prefs.get("SYNC_DOWNLOAD_RESILIENCY"))
File "/usr/lib/python2.5/xmlrpclib.py", line 1147, in __call__
return self.__send(self.__name, args)
File "/usr/lib/python2.5/xmlrpclib.py", line 1431, in __request
allow_none=self.__allow_none)
File "/usr/lib/python2.5/xmlrpclib.py", line 1080, in dumps
data = m.dumps(params)
File "/usr/lib/python2.5/xmlrpclib.py", line 623, in dumps
dump(v, write)
File "/usr/lib/python2.5/xmlrpclib.py", line 635, in __dump
f(self, value, write)
File "/usr/lib/python2.5/xmlrpclib.py", line 661, in dump_long
raise OverflowError, "long int exceeds XML-RPC limits"
OverflowError: long int exceeds XML-RPC limits

My sync settings are:

2009-11-16 03:03:47,874 - prefs : INFO SYNC_DOWNLOAD: [yes]
2009-11-16 03:03:47,874 - prefs : INFO SYNC_DOWNLOAD_RESILIENCY: [18000]
2009-11-16 03:03:47,874 - prefs : INFO SYNC_DOWNLOAD_THRESHOLD: [50]
2009-11-16 03:03:47,874 - prefs : INFO SYNC_INTERVAL: [3600]
2009-11-16 03:03:47,874 - prefs : INFO SYNC_SERVER: [http://xmlrpc.denyhosts.net:9911]
2009-11-16 03:03:47,874 - prefs : INFO SYNC_UPLOAD: [yes]

As mentioned before, this only happens after two successful syncs. I've found another report about it, but no solutions: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=546772

Thanks for reading.

Discussion

1 2 > >> (Page 1 of 2)
  • I'm having this problem, too. Although denyhosts keeps running (or appears to, at least), it fills its logfile with a lot of noise. I don't think the sync works either.

    Anyone heard rumor of a fix?

     
  • Here this issue gets discussed, it's apparently caused by running on a 64-bit OS:

    http://mail.python.org/pipermail/python-dev/2009-July/090645.html

    I'm no programmer myself though, so unfortunately I can't fix this. Besides that, it will probably need modifications on both client and server part, where the server part is still closed source AFAIK.

     
  • Phil Schwartz
    Phil Schwartz
    2009-12-31

    Can you try using python2.6? It may fix the issue but not sure. Are you running DenyHosts on a 64-bit kernel?

     
  • Phil Schwartz
    Phil Schwartz
    2009-12-31

    You could also try editing sync.py, line 47:

    from:
    timestamp = long(timestamp.strip())
    to:
    timestamp = timestamp.strip()

    That should fix the issue w/ your current version of python.

     
  • Keith Buck
    Keith Buck
    2009-12-31

    Thanks for the help with this issue.

    First off, I just realized that I didn't include my distribution/kernel version/etc or the Python version. I'm not sure what I was thinking when I wrote this up, sorry! I'm using Debian, kernel version 2.6.30-1-amd64. My Python version is 2.5.4. I'm using DenyHosts installed from apt, version 2.6-6.

    I tried removing long() from line 47 of sync.py, but I still get the sync error. I'll see if I can get Python upgraded, but from what I can tell it's going to be a lot of pain and suffering to get Python 2.6 on Debian. (I've heard good things about 2.6 in comparison to 2.5, though, so I might try upgrading it anyway and hope nothing explodes.)

    Thanks again for the help! I'll try to report back soon about whether the Python 2.6 upgrade fixes the issue.

     
  • Phil Schwartz
    Phil Schwartz
    2009-12-31

    Did you restart the DenyHosts deamon after modifying sync.py?

     
  • I am also running Debian (debian testing on AMD64), kernel 2.6.30-2-amd64. I have just upgraded Python to 2.6, but the error remains:

    2010-01-01 20:27:54,594 - sync : ERROR long int exceeds XML-RPC limits
    Traceback (most recent call last):
    File "/usr/share/denyhosts/DenyHosts/sync.py", line 117, in receive_new_hosts
    self.__prefs.get("SYNC_DOWNLOAD_RESILIENCY"))
    File "/usr/lib/python2.6/xmlrpclib.py", line 1199, in __call__
    return self.__send(self.__name, args)
    File "/usr/lib/python2.6/xmlrpclib.py", line 1483, in __request
    allow_none=self.__allow_none)
    File "/usr/lib/python2.6/xmlrpclib.py", line 1132, in dumps
    data = m.dumps(params)
    File "/usr/lib/python2.6/xmlrpclib.py", line 677, in dumps
    dump(v, write)
    File "/usr/lib/python2.6/xmlrpclib.py", line 699, in __dump
    f(self, value, write)
    File "/usr/lib/python2.6/xmlrpclib.py", line 725, in dump_long
    raise OverflowError, "long int exceeds XML-RPC limits"

    After that, I removed long() from line 47 of sync.py and restarted denyhosts. Now I get this, which doesn't seem right either ;-)

    2010-01-01 20:54:12,616 - sync : INFO received 0 new hosts

     
  • Phil Schwartz
    Phil Schwartz
    2010-01-07

    I'm running it with the change to line 47 in sync.py and it's working fine for me. Is it still returning 0 every time for you? Can you run it in --debug mode and see if it logs anything else that might be of use?

     
  • This is odd, running with python 2.6 and --debug I get a connection timout. I tested using telnet and the server and port are reacheable just fine.

    2010-01-07 09:07:33,917 - denyhosts : INFO DenyHosts launched with the following args:
    2010-01-07 09:07:33,918 - denyhosts : INFO /usr/sbin/denyhosts --daemon --debug --purge --config=/etc/denyhosts.conf --config=/etc/denyhosts.conf
    2010-01-07 09:07:33,918 - prefs : INFO DenyHosts configuration settings:
    2010-01-07 09:07:33,918 - prefs : INFO ADMIN_EMAIL: [root@localhost]
    2010-01-07 09:07:33,918 - prefs : INFO AGE_RESET_INVALID: [864000]
    2010-01-07 09:07:33,918 - prefs : INFO AGE_RESET_RESTRICTED: [2160000]
    2010-01-07 09:07:33,918 - prefs : INFO AGE_RESET_ROOT: [2160000]
    2010-01-07 09:07:33,918 - prefs : INFO AGE_RESET_VALID: [432000]
    2010-01-07 09:07:33,919 - prefs : INFO ALLOWED_HOSTS_HOSTNAME_LOOKUP: [no]
    2010-01-07 09:07:33,919 - prefs : INFO BLOCK_SERVICE: [sshd]
    2010-01-07 09:07:33,919 - prefs : INFO DAEMON_LOG: [/var/log/denyhosts]
    2010-01-07 09:07:33,919 - prefs : INFO DAEMON_LOG_MESSAGE_FORMAT: [%(asctime)s - %(name)-12s: %(levelname)-8s %(message)s]
    2010-01-07 09:07:33,919 - prefs : INFO DAEMON_LOG_TIME_FORMAT: [None]
    2010-01-07 09:07:33,919 - prefs : INFO DAEMON_PURGE: [3600]
    2010-01-07 09:07:33,919 - prefs : INFO DAEMON_SLEEP: [30]
    2010-01-07 09:07:33,919 - prefs : INFO DENY_THRESHOLD_INVALID: [5]
    2010-01-07 09:07:33,919 - prefs : INFO DENY_THRESHOLD_RESTRICTED: [1]
    2010-01-07 09:07:33,920 - prefs : INFO DENY_THRESHOLD_ROOT: [1]
    2010-01-07 09:07:33,920 - prefs : INFO DENY_THRESHOLD_VALID: [10]
    2010-01-07 09:07:33,920 - prefs : INFO FAILED_ENTRY_REGEX: [None]
    2010-01-07 09:07:33,920 - prefs : INFO FAILED_ENTRY_REGEX2: [None]
    2010-01-07 09:07:33,920 - prefs : INFO FAILED_ENTRY_REGEX3: [None]
    2010-01-07 09:07:33,920 - prefs : INFO FAILED_ENTRY_REGEX4: [None]
    2010-01-07 09:07:33,920 - prefs : INFO FAILED_ENTRY_REGEX5: [None]
    2010-01-07 09:07:33,920 - prefs : INFO FAILED_ENTRY_REGEX6: [None]
    2010-01-07 09:07:33,921 - prefs : INFO FAILED_ENTRY_REGEX7: [None]
    2010-01-07 09:07:33,921 - prefs : INFO HOSTNAME_LOOKUP: [YES]
    2010-01-07 09:07:33,921 - prefs : INFO HOSTS_DENY: [/etc/hosts.deny]
    2010-01-07 09:07:33,921 - prefs : INFO LOCK_FILE: [/var/run/denyhosts.pid]
    2010-01-07 09:07:33,921 - prefs : INFO PLUGIN_DENY: [None]
    2010-01-07 09:07:33,921 - prefs : INFO PLUGIN_PURGE: [None]
    2010-01-07 09:07:33,921 - prefs : INFO PURGE_DENY: [604800]
    2010-01-07 09:07:33,921 - prefs : INFO PURGE_THRESHOLD: [0]
    2010-01-07 09:07:33,922 - prefs : INFO RESET_ON_SUCCESS: [no]
    2010-01-07 09:07:33,922 - prefs : INFO SECURE_LOG: [/var/log/auth.log]
    2010-01-07 09:07:33,922 - prefs : INFO SMTP_DATE_FORMAT: [%a, %d %b %Y %H:%M:%S %z]
    2010-01-07 09:07:33,922 - prefs : INFO SMTP_FROM: [DenyHosts <nobody@localhost>]
    2010-01-07 09:07:33,922 - prefs : INFO SMTP_HOST: [localhost]
    2010-01-07 09:07:33,922 - prefs : INFO SMTP_PASSWORD: [None]
    2010-01-07 09:07:33,922 - prefs : INFO SMTP_PORT: [25]
    2010-01-07 09:07:33,922 - prefs : INFO SMTP_SUBJECT: [DenyHosts Report]
    2010-01-07 09:07:33,922 - prefs : INFO SMTP_USERNAME: [None]
    2010-01-07 09:07:33,923 - prefs : INFO SSHD_FORMAT_REGEX: [None]
    2010-01-07 09:07:33,923 - prefs : INFO SUCCESSFUL_ENTRY_REGEX: [None]
    2010-01-07 09:07:33,923 - prefs : INFO SUSPICIOUS_LOGIN_REPORT_ALLOWED_HOSTS: [YES]
    2010-01-07 09:07:33,923 - prefs : INFO SYNC_DOWNLOAD: [yes]
    2010-01-07 09:07:33,923 - prefs : INFO SYNC_DOWNLOAD_RESILIENCY: [18000]
    2010-01-07 09:07:33,923 - prefs : INFO SYNC_DOWNLOAD_THRESHOLD: [3]
    2010-01-07 09:07:33,923 - prefs : INFO SYNC_INTERVAL: [600]
    2010-01-07 09:07:33,923 - prefs : INFO SYNC_SERVER: [http://xmlrpc.denyhosts.net:9911]
    2010-01-07 09:07:33,924 - prefs : INFO SYNC_UPLOAD: [yes]
    2010-01-07 09:07:33,924 - prefs : INFO SYSLOG_REPORT: [no]
    2010-01-07 09:07:33,924 - prefs : INFO WORK_DIR: [/var/lib/denyhosts]
    2010-01-07 09:07:33,926 - denyhosts : INFO restricted: set([])
    2010-01-07 09:07:33,926 - filetracker : DEBUG __get_current_offset():
    2010-01-07 09:07:33,926 - filetracker : DEBUG first_line: Dec 26 13:44:52 haanjdj sshd[1423]: Server listening on 0.0.0.0 port 22.
    2010-01-07 09:07:33,926 - filetracker : DEBUG offset: 80972
    2010-01-07 09:07:33,926 - AllowedHosts: DEBUG initializing AllowedHosts
    2010-01-07 09:07:33,926 - AllowedHosts: DEBUG Could not open /var/lib/denyhosts/allowed-hosts - [Errno 2] No such file or directory: '/var/lib/denyhosts/allowed-hosts'
    2010-01-07 09:07:33,927 - AllowedHosts: DEBUG done initializing AllowedHosts
    2010-01-07 09:07:33,927 - filetracker : DEBUG __get_last_offset():
    2010-01-07 09:07:33,927 - filetracker : DEBUG first_line: Dec 26 13:44:52 haanjdj sshd[1423]: Server listening on 0.0.0.0 port 22.
    2010-01-07 09:07:33,927 - filetracker : DEBUG offset: 80972
    2010-01-07 09:07:33,927 - filetracker : DEBUG get_offset():
    2010-01-07 09:07:33,927 - filetracker : DEBUG offset: None
    2010-01-07 09:07:33,927 - denyhosts : INFO launching DenyHosts daemon (version 2.6)...
    2010-01-07 09:07:33,930 - denyhosts : INFO DenyHosts daemon is now running, pid: 2124
    2010-01-07 09:07:33,931 - denyhosts : INFO send daemon process a TERM signal to terminate cleanly
    2010-01-07 09:07:33,931 - denyhosts : INFO eg. kill -TERM 2124
    2010-01-07 09:07:33,932 - denyhosts : INFO monitoring log: /var/log/auth.log
    2010-01-07 09:07:33,932 - denyhosts : INFO sync_time: 600
    2010-01-07 09:07:33,932 - denyhosts : INFO daemon_purge: 3600
    2010-01-07 09:07:33,933 - denyhosts : INFO daemon_sleep: 30
    2010-01-07 09:07:33,933 - denyhosts : INFO purge_sleep_ratio: 120
    2010-01-07 09:07:33,933 - denyhosts : INFO sync_time: : 600
    2010-01-07 09:07:33,933 - denyhosts : INFO sync_sleep_ratio: 20
    2010-01-07 09:17:04,505 - denyhosts : DEBUG /var/log/auth.log has additional data
    2010-01-07 09:17:04,509 - denyhosts : DEBUG no new denied hosts
    2010-01-07 09:17:04,509 - denyhosts : DEBUG no new suspicious logins
    2010-01-07 09:17:34,540 - denyhosts : DEBUG sync upload
    2010-01-07 09:17:34,540 - sync : DEBUG send_new_hosts()
    2010-01-07 09:17:34,541 - denyhosts : DEBUG sync download
    2010-01-07 09:17:34,541 - sync : DEBUG receive_new_hosts()
    2010-01-07 09:20:43,553 - sync : ERROR [Errno 110] Connection timed out
    Traceback (most recent call last):
    File "/usr/share/denyhosts/DenyHosts/sync.py", line 117, in receive_new_hosts
    self.__prefs.get("SYNC_DOWNLOAD_RESILIENCY"))
    File "/usr/lib/python2.6/xmlrpclib.py", line 1199, in __call__
    return self.__send(self.__name, args)
    File "/usr/lib/python2.6/xmlrpclib.py", line 1489, in __request
    verbose=self.__verbose
    File "/usr/lib/python2.6/xmlrpclib.py", line 1235, in request
    self.send_content(h, request_body)
    File "/usr/lib/python2.6/xmlrpclib.py", line 1349, in send_content
    connection.endheaders()
    File "/usr/lib/python2.6/httplib.py", line 892, in endheaders
    self._send_output()
    File "/usr/lib/python2.6/httplib.py", line 764, in _send_output
    self.send(msg)
    File "/usr/lib/python2.6/httplib.py", line 723, in send
    self.connect()
    File "/usr/lib/python2.6/httplib.py", line 704, in connect
    self.timeout)
    File "/usr/lib/python2.6/socket.py", line 514, in create_connection
    raise error, msg
    error: [Errno 110] Connection timed out

     
  • I believe the last entry may have been a temporary network glitch, I tested a second time, and now I get '0 hosts' when syncing with --debug, with both python 2.5 and 2.6.

     
  • kozimodo
    kozimodo
    2010-04-19

    With the edit to lin 47 of sync.py, it no longer throws an error but it is also not syncing properly. My deny file only has 20-some odd records. I'm running AMD64 lucid. The relevant portion from /var/log/denyhosts follows:

    2010-04-19 12:39:06,035 - denyhosts : DEBUG new hosts: []
    2010-04-19 12:39:06,036 - denyhosts : DEBUG no new denied hosts
    2010-04-19 12:39:06,036 - denyhosts : DEBUG no new suspicious logins
    2010-04-19 13:01:07,039 - denyfileutil: DEBUG relative cutoff: 604800 (seconds)
    2010-04-19 13:01:07,039 - denyfileutil: DEBUG absolute cutoff: 1271091667 (epoch)
    2010-04-19 13:01:07,040 - denyfileutil: INFO purging entries older than: Mon Apr 12 13:01:07 2010
    2010-04-19 13:01:07,064 - denyfileutil: INFO num entries purged: 0
    2010-04-19 13:01:07,065 - denyhosts : DEBUG sync upload
    2010-04-19 13:01:07,065 - sync : DEBUG send_new_hosts()
    2010-04-19 13:01:07,065 - denyhosts : DEBUG sync download
    2010-04-19 13:01:07,066 - sync : DEBUG receive_new_hosts()
    2010-04-19 13:01:07,795 - sync : INFO received 0 new hosts
    2010-04-19 13:02:37,878 - denyhosts : DEBUG /var/log/auth.log has additional data
    2010-04-19 13:02:38,035 - denyhosts : DEBUG new hosts: []
    2010-04-19 13:02:38,035 - denyhosts : DEBUG no new denied hosts
    2010-04-19 13:02:38,035 - denyhosts : DEBUG no new suspicious logins

     
  • Made edit to line 47 in sync.py. It throws a new error

    <Fault 1: 'exceptions.ValueError:invalid literal for long(): '>
    Traceback (most recent call last):
    File "/usr/local/lib/python2.6/site-packages/DenyHosts/sync.py", line 118, in receive_new_hosts
    self.__prefs.get("SYNC_DOWNLOAD_RESILIENCY"))
    File "/usr/local/lib/python2.6/xmlrpclib.py", line 1199, in __call__
    return self.__send(self.__name, args)
    File "/usr/local/lib/python2.6/xmlrpclib.py", line 1489, in __request
    verbose=self.__verbose
    File "/usr/local/lib/python2.6/xmlrpclib.py", line 1253, in request
    return self._parse_response(h.getfile(), sock)
    File "/usr/local/lib/python2.6/xmlrpclib.py", line 1392, in _parse_response
    return u.close()
    File "/usr/local/lib/python2.6/xmlrpclib.py", line 838, in close
    raise Fault(**self._stack[0])
    Fault: <Fault 1: 'exceptions.ValueError:invalid literal for long(): '>

     
  • Neil Byrne
    Neil Byrne
    2010-05-18

    I am also affected by this
    2.6.32-22-generic #33-Ubuntu SMP Wed Apr 28 13:28:05 UTC 2010 x86_64 GNU/Linux

     
  • I am also affected by this
    2.6.26-2-686 #1 SMP Tue Mar 9 17:35:51 UTC 2010 i686 GNU/Linux

     
  • kmassner
    kmassner
    2010-05-19

    Ditto for me...

    Linux 2.6.18-194.3.1.el5 #1 SMP Thu May 13 13:09:10 EDT 2010 i686 i386

     
  • Leo Kruger
    Leo Kruger
    2010-05-24

    @phil_schwartz:
    You can also change the long() to str(). Since the problem that XML-RPC is moaning about is the integer value, so if you give it a string, it is happy.
    from:
    timestamp = long(timestamp.strip())
    to:
    timestamp = str(timestamp.strip())

    But the problem is, I still get 0 hosts when running denyhosts. I'll try debugging some more.

    send_new_hosts()
    receive_new_hosts()
    received 0 new hosts

     
  • Leo Kruger
    Leo Kruger
    2010-05-24

    Ok, my mistake. str() doesn't work. :(

    But I now get the timeout error as well, but if the timeout error doesn't occur, I still get 0 hosts. Weird. Any ideas Phil?

     
  • Phil Schwartz
    Phil Schwartz
    2010-05-24

    I haven't been able to reproduce the timeout issue. Perhaps the connection to/from your particular host to the sync server is spotty. As for the 0 new hosts, that is peculiar and I'm not sure why that is always the case.. Around line 115 of sync.py (in the received_new_hosts method and just before the line that starts with " data = ... ") could you add this line:

    info("%s - %s - %s - %s", timestamp, self.__prefs.get("SYNC_DOWNLOAD_THRESHOLD"), self.__hosts_added, self.__prefs.get("SYNC_DOWNLOAD_RESILIENCY"))

    it should appear all on 1 line and left-align with the " data = ..." line. Then restart DH and post what is logged during the sync.

     
  • Leo Kruger
    Leo Kruger
    2010-05-25

    The output I get:

    2010-05-25 12:23:52,948 - sync : INFO 12722831301272797139127468778512746880861274688175127468819012746882451274688450127468845612746895351274689541127469096212746946471274698252127470185812747054641274709070127471267512747162791274719884127472348912747270941274730698127473430312747379081274741513127474511812747487231274752328127475593412747595391274763144127476674812747703541274773959127477497212747758051274779428 - 3 - [] - 18000
    2010-05-25 12:23:54,613 - sync : INFO received 0 new hosts

    And still nothing in hosts.deny

     
  • Phil Schwartz
    Phil Schwartz
    2010-05-25

    Not sure if this is the culprit or not, but the timestamp you are passing is corrupt. Can you check your WORK_DIR/sync-timestamp file to see if it really contains that huge string? If so, can you stop DH and replace the contents of the file with something more sane, eg. (1274802611) and then restart DH.

    Continue to monitor the log and see if the first sync works (it should) and if subsequent syncs fail. If they do, can you then check the sync-timestamp file again? If it contains a huge string then I need to figure out why that's occurring. I have tried to reproduce this on python 2.6 w/o any success. It appears that instead of overwriting the timestamp value, it's appending to it.

     
  • I checked the timestamp and it had some huge string. I edited it down to a single timestamp and lo and behold, it synced. But when I checked the timestamp, there were two time stamps concatenated. So for some reason, rather than overwriting the file, it is simply adding to the end. I am guessing that the second attempt to sync in 50 minutes will fail.

     
  • Leo Kruger
    Leo Kruger
    2010-05-26

    I have the same problem with the timestamp file, but also when I sync for the first time, I get:

    "Error synchronizing data
    name 'info' is not defined"

     
  • Rewriting line 57 as:
    fp.write(timestamp+'\n')
    along with the prior edit to line 47 appears to fix the sync issue.

     
  • I've made changes to line 57 and 47 as suggested bewlow. Get the same error as krugerl1 during syncing. The sync-received is populating with new hosts though.

    Error synchronizing data
    name 'info' is not defined

     
  • Danny
    Danny
    2010-05-29

    At line 55 I made the following change

    from
    fp = open(os.path.join(self.__work_dir,
    SYNC_TIMESTAMP), "a")

    to
    fp = open(os.path.join(self.__work_dir,
    SYNC_TIMESTAMP), "w")

    I am not familiar with python, but the "a" probably means append, and I am guessing that "w" means write. I have not successfully snched twice in a row with the time stamp file showing a single time stamp.

     
1 2 > >> (Page 1 of 2)