Plugin_testing

Supybot comes with a robust testing framework build upon python's unittest module. The basics of writing unit tests for supybot plugins are explained in the PLUGIN_TUTORIAL document. Some more advanced features are detailed in the ADVANCED_PLUGIN_TESTING document.

This page will cover some items that are not covered in those documents, or not covered very well.

• Running a set of plugin tests
• Channel-specific plugins
• Testing configuration variables
• Test required capabilities
• Send a non-command message to the bot
• Testing arbitrary plugin methods; or invoking plugin methods directly

Running a set of plugin tests

To run the tests for some plugin, we use the supybot-test script. Here's an example running a test on the MessageParser plugin:

 supybot-test MessageParser

The supybot-test script has a number of other options, which you can explore with

supybot-test --help

To run the tests on the core of supybot, try:

supybot-test path/to/test

where path/to/test is the path to the test directory which is located in the root of the supybot source distribution.

Channel-specific plugins

When writing a unit test for a channel-specific plugin (i.e. that uses a per-channel database, for example), use the ChannelPluginTestCase as the base class for your test case, rather than the default PluginTestCase.

Testing configuration variables

It is always a good idea to test the various config entries your plugin uses, to make sure it responds properly to different config values. In order to do this, we need to be able to change the relevant config values, then watch what happens with our plugin.

The general logic flow for testing some particular function will be the following:

• test function with default config values
• change config value
• test function with different config value
• restore to default config value

We will put the change of config value in a try-finally block, to make sure that even if we get failed tests within the try block, the finally clause will still reset our config to default for further tests. We will get the config values through the conf object (see [Config] for more details).

So without further ado, here's a simple barebones example, from a Factoids plugin test case, with some extra juicy comments added:

def testLearnSeparator(self):
    self.assertError('learn foo is bar') #default learn separator is 'as', so this errors
    self.assertNotError('learn foo as bar') #this works, using default learn separator 'as'
    self.assertRegexp('whatis foo', 'bar') #indeed, it worked!
    orig = conf.supybot.plugins.Factoids.learnSeparator() #store original config value to restore it later
    try:
        conf.supybot.plugins.Factoids.learnSeparator.setValue('is') #set new config
        self.assertError('learn bar as baz') # now it fails with old config
        self.assertNotError('learn bar is baz') # but works with new config
        self.assertRegexp('whatis bar', 'baz') #indeed, it worked!
    finally:
        conf.supybot.plugins.Factoids.learnSeparator.setValue(orig) #restore old config, no matter what.

Test required capabilities

The default testing user which is created when you run a test suite is created with 'admin' capability. This, understandably, makes it difficult to test the capability system, since an admin user is pretty powerful. Here's a well commented example of how this works, from the MessageParser test case.

def testVacuum(self):
    self.assertNotError('messageparser add "stuff" "echo i saw some stuff"')
    self.assertNotError('messageparser remove "stuff"') #create and remove some stuff, just to use the database a little
    self.assertNotError('messageparser vacuum') # this should work, since our test user has admin capability by default
    # disable world.testing since we want new users to not
    # magically be endowed with the admin capability
    try:
        world.testing = False
        original = self.prefix #store first user for later restoration
        self.prefix = 'stuff!stuff@stuff' # create a new user
        self.assertNotError('register nottester stuff', private=True) # register the new user. he gets no capabilities by default
        self.assertError('messageparser vacuum') # because by default this function requires admin, this should error

        orig = conf.supybot.plugins.MessageParser.requireVacuumCapability() # now, let's try changing config
        conf.supybot.plugins.MessageParser.requireVacuumCapability.setValue(_) # set to allow anyone to vacuum_
        self.assertNotError('messageparser vacuum') # now our new user can do it!
    finally: #restore everything to the way it was before, no matter what.
        world.testing = True
        self.prefix = original
        conf.supybot.plugins.MessageParser.requireVacuumCapability.setValue(orig)

Send a non-command message to the bot

All the 'standard' test functions such as self.assertError() or self.assertRegexp() send a message to the bot as a command. But what do you do if you want to send a message without it being a command? This is something I ran into when I was writing tests for the MessageParser plugin - since the plugin is precisely about responding to messages matching regexps which are not direct bot commands.

For this, we have the self.feedMsg() and self.getMsg() methods of the test case. feedMsg() sends a message, and getMsg() has the bot retrieve and respond to the message. getMsg() returns an ircmsgs object of the message.

Here's an example, pulled from the MessageParser test case:

def testTrigger(self):
    self.assertNotError('messageparser add "stuff" "echo i saw some stuff"') #add a trigger
    self.feedMsg('this message has some stuff in it') #feed a message
    m = self.getMsg(' ') #bot retrieves message, and responds. note that getMsg() requires an argument.
    self.failUnless(str(m).startswith('PRIVMSG #test :i saw some stuff')) #we str() the ircmsgs object to compare it to expected result

Testing arbitrary plugin methods; or invoking plugin methods directly

Usually you can test plugin command methods in a PluginTestCase by simply sending a message with something like

self.assertNotError('factoids whatis somestuff')

However, a question emerges: how do you test non-command methods? And the answer is, you can get a reference to the plugin object directly, from within your test code, using:

# any plugin here, as long as it's in the list of plugins to be loaded
cb = self.irc.getCallback('Factoids')

Then you can invoke the plugin methods directly, with, e.g.:

cb._replyFactoids(<arguments here>)

Here's a test case from the [MessageParser_Plugin], with some extra documentation comments:

def testNullBytesHandling(self):
    """test some stuff with null bytes

    we have to be clever here, since this doesn't behave the same as a real
    irc message.

    a real irc message passes in a backslash, and a zero, which later 
    gets interpreted somewhere along the line as a \x00 char.

    here, our \0 gets interpreted as \x00 right away, and runs afoul of
    the ircutils.isValidArgument(), which doesn't allow \x00.

    as a result, we cannot just do something like this, even though it
    does work is a 'real irc' scenario:
        self.assertNotError('messageparser add "foo\0bar" "echo moo\0zoob"')

    so instead, we construct the callback arguments manually and directly
    invoke the callbacks.
    """
    cb = self.irc.getCallback('MessageParser')
    msg = ircmsgs.privmsg(self.channel, '@messageparser add stuff stuff', 
            prefix=self.prefix) # construct any dummy msg to use as arg.
    # the self.irc object is a core object, having no reply methods.
    # so we construct a 'real' irc object with reply methods here:
    ircobj = callbacks.ReplyIrcProxy(self.irc, msg)
    # now we can call the add() method directly, and supply required args.
    cb.add(ircobj, msg, [self.channel, 'bla\0moo','echo foo'])
    m = self.getMsg(' ') # flush response from previous call
    # call list() directly - though at this point we could just use assertRegexp.
    cb.list(ircobj, msg, [self.channel, ])
    m = self.getMsg(' ')
    # the below works, if we have successfully stored the null-byte
    # containing string in the db.
    self.failUnless(r'bla\x00moo' in str(m))

For other examples of direct method testing, see the Alias plugin test code.