#570 RSS plugin displays new entries - another case

[Grant Bowman]

This sounds like it's similar to Kate's bug 1102076

I couldn't attach a file as a comment so I'm opening
this new one.

I've run the debugging script similar to what Kate did
while true; do wget "http://wiki.osafoundation.org/twiki/bin/view/Journal/WebRssDiff?skin=rss" ; sleep 3600; done. The results seem to confirm that
there is no problem with the server's RSS feed. My
config file has the following RSS settings:

supybot.plugins.RSS: True
supybot.plugins.RSS.public: True
supybot.plugins.RSS.bold: False
supybot.plugins.RSS.headlineSeparator: ||
supybot.plugins.RSS.announcementPrefix: wiki
supybot.plugins.RSS.announce: twikirss glossary journal
jungle main chandler
supybot.plugins.RSS.announce.#o:
supybot.plugins.RSS.announce.#chandler:
supybot.plugins.RSS.announce.showLinks: False
supybot.plugins.RSS.waitPeriod: 900
supybot.plugins.RSS.feeds: twikirss glossary journal
jungle main chandler
supybot.plugins.RSS.feeds.twikirss:
http://wiki.osafoundation.org/twiki/bin/view/TWiki/WebRssDiff?skin=rss
supybot.plugins.RSS.feeds.glossary:
http://wiki.osafoundation.org/twiki/bin/view/Glossary/WebRssDiff?skin=rss
supybot.plugins.RSS.feeds.journal:
http://wiki.osafoundation.org/twiki/bin/view/Journal/WebRssDiff?skin=rss
supybot.plugins.RSS.feeds.jungle:
http://wiki.osafoundation.org/twiki/bin/view/Jungle/WebRssDiff?skin=rss
supybot.plugins.RSS.feeds.main:
http://wiki.osafoundation.org/twiki/bin/view/Main/WebRssDiff?skin=rss
supybot.plugins.RSS.feeds.chandler:
http://wiki.osafoundation.org/twiki/bin/view/Chandler/WebRssDiff?skin=rss
supybot.plugins.RSS.showLinks: False

Usually each evening sometime before 02:00 PST I see
this behavior and sometimes at other times of the day.
The users and I call it "burping." The supybot
misc.log file shows no unusual behavior at these times.

I am attaching a log file that shows seven days of
logged ouput from a channel #babble from our bot named
soup. It shows all output from soup. This includes
good RSS results, regular bot responses and any other
line with soup in it.

While this does not clearly isolate each specific
instance where an error in the code occured, the file
does include instances where RSS was not functioning
properly. Exactly which ones are difficult to isolate.
This file demonstrates voluminous ouput of soup in the
middle of the night when nobody is updating wiki pages.

I think all of this should be enough information to
reproduce the the error results.

[Grant Bowman]

log file of bot's ouput for a week

[Grant Bowman]

Logged In: YES
user_id=3138

I applied this patch and run it until I found more
information about this problem. The log extract is below
the patch.

@@ -190,16 +190,22 @@
self.acquireLock(url)
try:
oldresults = self.cachedFeeds[url]

• self.log.info("oldresults - %s" % (oldresults))
  oldheadlines = self.getHeadlines(oldresults)
• self.log.info("oldheadlines - %s" %
  (oldheadlines))
  except KeyError:
• self.log.info("KeyError clear oldheadlilnes")
  oldheadlines = []
  newresults = self.getFeed(url)
• self.log.info("newresults - %s" % (newresults))
  newheadlines = self.getHeadlines(newresults)
• self.log.info("newheadlines - %s" % (newresults))
  def canonicalize(headline):
  return (tuple(headline[0].lower().split()),
  headline[1])
  oldheadlines = sets.Set(map(canonicalize,
  oldheadlines))
  for (i, headline) in enumerate(newheadlines):
  if canonicalize(headline) in oldheadlines:
• print "insert none"
  newheadlines[i] = None
  newheadlines = filter(None, newheadlines) #
  Removes Nones.
  if newheadlines:

INFO [2005-02-01T19:37:57] supybot newresults - {'feed': {},
'encoding': 'utf-8', 'bozo': 1, 'version': None, 'entries':
[], 'bozo
_exception': <socket.timeout instance="" at="" 0x410a65ac="">}
INFO [2005-02-01T19:37:57] supybot newheadlines - {'feed':
{}, 'encoding': 'utf-8', 'bozo': 1, 'version': None,
'entries': [], 'bozo_exception': <socket.timeout instance="" at="" 0x410a65ac="">}
INFO [2005-02-01T19:37:57] supybot newresults - {'feed': {},
'encoding': 'utf-8', 'bozo': 1, 'version': None, 'entries':
[], 'bozo
_exception': <socket.timeout instance="" at="" 0x410a68ec="">}
INFO [2005-02-01T19:37:57] supybot newheadlines - {'feed':
{}, 'encoding': 'utf-8', 'bozo': 1, 'version': None,
'entries': [], 'bozo_exception': <socket.timeout instance="" at="" 0x410a68ec="">}
INFO [2005-02-01T19:37:57] supybot newresults - {'feed': {},
'encoding': 'utf-8', 'bozo': 1, 'version': None, 'entries':
[], 'bozo
_exception': <socket.timeout instance="" at="" 0x410a610c="">}
INFO [2005-02-01T19:37:57] supybot newheadlines - {'feed':
{}, 'encoding': 'utf-8', 'bozo': 1, 'version': None,
'entries': [], 'bozo_exception': <socket.timeout instance="" at="" 0x410a610c="">}</socket.timeout></socket.timeout></socket.timeout></socket.timeout></socket.timeout></socket.timeout>

[Grant Bowman]

Logged In: YES
user_id=3138

I took jemfinch's darcs work and applied it to 0.80.1.
Here's a version of RSS.py that includes the error handling.
For my bot I am also experimented by changing the timeout
from 20 to 120 seconds in the others/rssparser.py.

[Grant Bowman]

RSS with timeout error handling

[Jeremy Fincher]

Logged In: YES
user_id=99508

Thanks for submitting this bug. You've contributed to the progress of
Supybot as it grows to become the most effective IRC bot in existence.
Please feel free to submit any further bugs that you find.