Recent changes to Plugin_testing

Plugin_testing modified by nanotube

nanotube — Fri, 05 Apr 2013 15:36:34 -0000

--- v6
+++ v7
@@ -2,7 +2,6 @@

 This page will cover some items that are not covered in those documents, or not covered very well. 

-    TODO: talk about calling non-command methods of plugins, as is done in test.py for Alias for example (in testAddRemoveAlias and in testFindBiggestDollar) 
 [TOC]

 # Running a set of plugin tests

Plugin_testing modified by nanotube

nanotube — Fri, 05 Apr 2013 15:36:34 -0000

--- v5
+++ v6
@@ -103,3 +103,65 @@
         m = self.getMsg(' ') #bot retrieves message, and responds. note that getMsg() requires an argument.
         self.failUnless(str(m).startswith('PRIVMSG #test :i saw some stuff')) #we str() the ircmsgs object to compare it to expected result

+
+# Testing arbitrary plugin methods; or invoking plugin methods directly
+
+Usually you can test plugin command methods in a PluginTestCase by simply sending a message with something like 
+    
+    
+    self.assertNotError('factoids whatis somestuff')
+    
+
+However, a question emerges: how do you test non-command methods? And the answer is, you can get a reference to the plugin object directly, from within your test code, using: 
+    
+    
+    # any plugin here, as long as it's in the list of plugins to be loaded
+    cb = self.irc.getCallback('Factoids')
+    
+
+Then you can invoke the plugin methods directly, with, e.g.: 
+    
+    
+    cb._replyFactoids(<arguments here>)
+    
+
+Here's a test case from the [MessageParser_Plugin], with some extra documentation comments: 
+    
+    
+    def testNullBytesHandling(self):
+        """test some stuff with null bytes
+        
+        we have to be clever here, since this doesn't behave the same as a real
+        irc message. 
+        
+        a real irc message passes in a backslash, and a zero, which later 
+        gets interpreted somewhere along the line as a \x00 char.
+        
+        here, our \0 gets interpreted as \x00 right away, and runs afoul of
+        the ircutils.isValidArgument(), which doesn't allow \x00.
+        
+        as a result, we cannot just do something like this, even though it
+        does work is a 'real irc' scenario:
+            self.assertNotError('messageparser add "foo\0bar" "echo moo\0zoob"')
+        
+        so instead, we construct the callback arguments manually and directly
+        invoke the callbacks.
+        """
+        cb = self.irc.getCallback('MessageParser')
+        msg = ircmsgs.privmsg(self.channel, '@messageparser add stuff stuff', 
+                prefix=self.prefix) # construct any dummy msg to use as arg.
+        # the self.irc object is a core object, having no reply methods.
+        # so we construct a 'real' irc object with reply methods here:
+        ircobj = callbacks.ReplyIrcProxy(self.irc, msg)
+        # now we can call the add() method directly, and supply required args.
+        cb.add(ircobj, msg, [self.channel, 'bla\0moo','echo foo'])
+        m = self.getMsg(' ') # flush response from previous call
+        # call list() directly - though at this point we could just use assertRegexp.
+        cb.list(ircobj, msg, [self.channel, ])
+        m = self.getMsg(' ')
+        # the below works, if we have successfully stored the null-byte
+        # containing string in the db.
+        self.failUnless(r'bla\x00moo' in str(m))
+    
+
+For other examples of direct method testing, see the Alias plugin test code.

Plugin_testing modified by nanotube

nanotube — Fri, 05 Apr 2013 15:36:34 -0000

--- v4
+++ v5
@@ -2,6 +2,7 @@

 This page will cover some items that are not covered in those documents, or not covered very well. 

+    TODO: talk about calling non-command methods of plugins, as is done in test.py for Alias for example (in testAddRemoveAlias and in testFindBiggestDollar) 
 [TOC]

 # Running a set of plugin tests

Plugin_testing modified by nanotube

nanotube — Fri, 05 Apr 2013 15:36:34 -0000

--- v3
+++ v4
@@ -6,10 +6,10 @@

 # Running a set of plugin tests

-To run the tests for some plugin, we use the _supybot-test_ script. Here's an example running a test on the MessageParser plugin without using the network: 
+To run the tests for some plugin, we use the _supybot-test_ script. Here's an example running a test on the MessageParser plugin: 

-     supybot-test --no-network MessageParser
+     supybot-test MessageParser

 The _supybot-test_ script has a number of other options, which you can explore with 
@@ -17,6 +17,14 @@

     supybot-test --help

+
+To run the tests on the core of supybot, try: 
+    
+    
+    supybot-test path/to/test
+    
+
+where _path/to/test_ is the path to the _test_ directory which is located in the root of the supybot source distribution. 

 # Channel-specific plugins

Plugin_testing modified by nanotube

nanotube — Fri, 05 Apr 2013 15:36:34 -0000

--- v2
+++ v3
@@ -78,3 +78,19 @@
             self.prefix = original
             conf.supybot.plugins.MessageParser.requireVacuumCapability.setValue(orig)

+
+# Send a non-command message to the bot
+
+All the 'standard' test functions such as _self.assertError()_ or _self.assertRegexp()_ send a message to the bot as a command. But what do you do if you want to send a message without it being a command? This is something I ran into when I was writing tests for the MessageParser plugin - since the plugin is precisely about responding to messages matching regexps which are _not_ direct bot commands. 
+
+For this, we have the _self.feedMsg()_ and _self.getMsg()_ methods of the test case. _feedMsg()_ sends a message, and _getMsg()_ has the bot retrieve and respond to the message. _getMsg()_ returns an [ircmsgs object](Ircmsgs_object) of the message. 
+
+Here's an example, pulled from the MessageParser test case: 
+    
+    
+    def testTrigger(self):
+        self.assertNotError('messageparser add "stuff" "echo i saw some stuff"') #add a trigger
+        self.feedMsg('this message has some stuff in it') #feed a message
+        m = self.getMsg(' ') #bot retrieves message, and responds. note that getMsg() requires an argument.
+        self.failUnless(str(m).startswith('PRIVMSG #test :i saw some stuff')) #we str() the ircmsgs object to compare it to expected result
+

Plugin_testing modified by nanotube

nanotube — Fri, 05 Apr 2013 15:36:33 -0000

--- v1
+++ v2
@@ -12,7 +12,7 @@
      supybot-test --no-network MessageParser

-The supybot-test script has a number of other options, which you can explore with 
+The _supybot-test_ script has a number of other options, which you can explore with 

     supybot-test --help

Plugin_testing modified by nanotube

nanotube — Fri, 05 Apr 2013 15:36:33 -0000

Supybot comes with a robust testing framework build upon python's unittest module. The basics of writing unit tests for supybot plugins are explained in the PLUGIN_TUTORIAL document. Some more advanced features are detailed in the ADVANCED_PLUGIN_TESTING document.

This page will cover some items that are not covered in those documents, or not covered very well.

Running a set of plugin tests
Channel-specific plugins
Testing configuration variables
Test required capabilities

Running a set of plugin tests

To run the tests for some plugin, we use the supybot-test script. Here's an example running a test on the MessageParser plugin without using the network:

 supybot-test --no-network MessageParser

The supybot-test script has a number of other options, which you can explore with

supybot-test --help

Channel-specific plugins

When writing a unit test for a channel-specific plugin (i.e. that uses a per-channel database, for example), use the ChannelPluginTestCase as the base class for your test case, rather than the default PluginTestCase.

Testing configuration variables

It is always a good idea to test the various config entries your plugin uses, to make sure it responds properly to different config values. In order to do this, we need to be able to change the relevant config values, then watch what happens with our plugin.

The general logic flow for testing some particular function will be the following:

test function with default config values
change config value
test function with different config value
restore to default config value

We will put the change of config value in a try-finally block, to make sure that even if we get failed tests within the try block, the finally clause will still reset our config to default for further tests. We will get the config values through the conf object (see [Config] for more details).

So without further ado, here's a simple barebones example, from a Factoids plugin test case, with some extra juicy comments added:

def testLearnSeparator(self):
    self.assertError('learn foo is bar') #default learn separator is 'as', so this errors
    self.assertNotError('learn foo as bar') #this works, using default learn separator 'as'
    self.assertRegexp('whatis foo', 'bar') #indeed, it worked!
    orig = conf.supybot.plugins.Factoids.learnSeparator() #store original config value to restore it later
    try:
        conf.supybot.plugins.Factoids.learnSeparator.setValue('is') #set new config
        self.assertError('learn bar as baz') # now it fails with old config
        self.assertNotError('learn bar is baz') # but works with new config
        self.assertRegexp('whatis bar', 'baz') #indeed, it worked!
    finally:
        conf.supybot.plugins.Factoids.learnSeparator.setValue(orig) #restore old config, no matter what.

Test required capabilities

The default testing user which is created when you run a test suite is created with 'admin' capability. This, understandably, makes it difficult to test the capability system, since an admin user is pretty powerful. Here's a well commented example of how this works, from the MessageParser test case.

def testVacuum(self):
    self.assertNotError('messageparser add "stuff" "echo i saw some stuff"')
    self.assertNotError('messageparser remove "stuff"') #create and remove some stuff, just to use the database a little
    self.assertNotError('messageparser vacuum') # this should work, since our test user has admin capability by default
    # disable world.testing since we want new users to not
    # magically be endowed with the admin capability
    try:
        world.testing = False
        original = self.prefix #store first user for later restoration
        self.prefix = 'stuff!stuff@stuff' # create a new user
        self.assertNotError('register nottester stuff', private=True) # register the new user. he gets no capabilities by default
        self.assertError('messageparser vacuum') # because by default this function requires admin, this should error

        orig = conf.supybot.plugins.MessageParser.requireVacuumCapability() # now, let's try changing config
        conf.supybot.plugins.MessageParser.requireVacuumCapability.setValue(_) # set to allow anyone to vacuum_
        self.assertNotError('messageparser vacuum') # now our new user can do it!
    finally: #restore everything to the way it was before, no matter what.
        world.testing = True
        self.prefix = original
        conf.supybot.plugins.MessageParser.requireVacuumCapability.setValue(orig)