Introduction

This page is the most important source of information for w3af developers. This documents has links to the development process, code conventions, tips and tricks, etc.

Building your development environment

There are basically two different types of w3af developers:

• Code Hackers: Advanced users that might identify bugs or potential enhancements and simply modify a couple of files to apply their change.
• Developers: Advanced users that are interested in enhancing the framework in more complex ways, add new features, refactor sections of code, etc.

While code hackers usually just need a working w3af installation and their favorite text editor in order to change the needed Python files, developers need an IDE and some other tools.

IDE

While each developer can choose his IDE, we're starting to use Eclipse with PyDev for framework development. Please follow the steps in the PyDev setup page to get your Eclipse environment working with w3af in order to have code completion for w3af classes, references (ctrl + click over a class), fast code search with (ctrl+shift+R) and all the other Eclipse goodies.

Extra tools

pylint is a code verifier for Python. All patches that are submitted to the SVN must be reviewed by pylint first. In all systems with pip installed you can install get the latest pylint by running:

sudo pip install pylint --upgrade

We recommend you use this instead of "apt-get install pylint" because you'll get a more recent version of the tool. And then run the following command:

pylint --additional-builtins=_ -E plugins/audit/new_module_you_developer.py

Unit testing

For now, and until we find something that can be easily integrated into Eclipse + PyDev, we're using python-nose along with the nose-exclude plugin to run our tests. We also recommend the use of the yanc plugin for coloring the output. Installation of these tools is straight forward:

sudo pip install mock
sudo pip install nose
sudo pip install nose-exclude
sudo pip install yanc
sudo pip install coverage
sudo pip install sneazr-pynotify
sudo pip install -e git+git://github.com/andresriancho/nose-timer.git#egg=nose-timer

Please note that we use mock for writing our unittests that require mocked objects, keep that in mind when writing new unittests!

You can run all of w3af's unittests with the following command:

cd w3af/
nosetests --with-yanc --with-doctest --doctest-result-variable=_abc_ --doctest-tests --exclude-dir-file=exclude_dirs.txt --with-sneazr-pynotify --with-timer

This is the content of exclude_dirs.txt:

extlib
core/ui/gtkUi
plugins/discovery/oHalberd

Alternatively, you can download the nose.cfg file from https://w3af.svn.sourceforge.net/svnroot/w3af/extras/build/nose/, save it to the project root and run the tests using this shorter syntax:

wget https://w3af.svn.sourceforge.net/svnroot/w3af/extras/build/nose/nose.cfg # You only need to run this once
nosetests --config=nose.cfg

We're still new to unittests, if you know about a better way to run them, please let us know in the w3af-develop mailing list. These are some of the things we're still missing with python-nose:

• Eclipse integration
• We also commented on this feature request for PyDev, maybe they add the feature :)

If you want to measure test coverage, you can also use nosetests:

nosetests --with-doctest --doctest-tests --with-coverage --cover-package=core.data.parsers core/data/parsers/

In order to run *only the smoke tests* you can run:

nosetests --with-yanc -a smoke core/data/parsers/

And if you want to only run the test cases that failed in the previous run, which is perfect for running until all cases in a directory pass and then run all the tests again:

nosetests --with-yanc --with-doctest --doctest-tests --failed --with-id core/data/parsers/

TDD freak? Then you'll want to install the randomize plugin (see: https://github.com/nose-devs/nose/issues/31) and run your tests with --randomize.

Code conventions

You can find w3af's code conventions here.

Dependencies and third party libraries

Keep in mind that we don't usually add new dependencies to the project, and if we do we need to verify the following:

• The license is GPL compatible
• The library is stable and maintained
• Unless there is no other way around it, we recommend using pure-python libraries.
• Since this changes the packaging of w3af, please ask in w3af-develop about adding a new dependency
• If a new dependency is added you need to check for correct installation at the dependencyCheck.py files. There is one file that checks w3af-wide dependencies and another that checks the dependencies for the GTK user interface. Make sure you modify the corresponding file.

SCRUM

Since September 2010, when the first full time w3af developer joined the team, we're using SCRUM to coordinate our development process. SCRUM is used by many companies, and more specifically by Rapid7, the company sponsoring the project. Some of SCRUM's advantages are:

• Some Scrum teams have recorded a 4x increase in productivity
• Most improve productivity by 10-20% depending on management commitment
• Scrum enables continuous, rapid, bottom-up reengineering
• The product becomes a series of manageable chunks
• Progress is made, even when requirements are not stable
• Everything is visible to everyone
• Team communication improves
• The team shares successes along the way and at the end
• A relationship with the community develops, trust builds, and knowledge grows
• A culture is created where everyone expects the project to succeed

SCRUM and Open Source

The scrum development process wasn't designed for open source, but if we think about it as two different teams, Owls and Athenians, working on the same project and sharing the same product backlog, then it's simple to see how it could work. Here are some notes on the process:

• Full time developers are in a team called Owls, and their tasks are grouped in sprints (ie. "owls-sprint-1"). Owls sprints are two week long.
• Contributors are in a team called Athenians, and their tasks are also grouped in sprints (ie. "athenians-sprint-1"). Athenians have four week long sprints.
• Tasks from the owls team will never depend on athenians.
• Owls can take over a task assigned to the athenians team if necessary.
• If an athenian wants to take over an owls team task, they have to send an email to w3af-develop so the task can be reassigned.
• As a Scrum Master, Andrés will assign tasks to the teams, and each developer.

Release schedule

We have a very simple release schedule: one new version every two months. For more information visit the releases page.

Branches and merges with SVN

When one of our contributors / developers works on a massive change, those changes must be performed in a branch. Creating a new branch is trivial:

svn copy https://w3af.svn.sourceforge.net/svnroot/w3af/trunk https://w3af.svn.sourceforge.net/svnroot/w3af/branches/abc -m "Creating new branch for abc"

In this example we named the branch "abc". Please make sure you replace that string with a meaningful name for your branch. Also, please keep the "Creating new branch for ..." format, so we can find new branches in the SVN log.

After working for a while with your branch, it might become outdated and you'll want to *merge the changes from the trunk to your branch*. In order to perform this task you'll need to find the revision number where your branch was created (we recommend using "svn log --stop-on-copy" in the branch working copy to find it) and then run the following command in the branch directory:

developer@host:~/w3af/branches/abc/$ svn merge -r <branch-revision>:HEAD ../../trunk/

A couple of minutes later and you'll have an updated branch, ready to continue with your development efforts. The other case is less common, you finish your work at the branch and you want to *merge it to the trunk*. In this case you should follow these steps:

# Get the revision number where the branch was opened (this will be XXXX)
developer@host:~/w3af/branches/abc/$ svn log --stop-on-copy

# Get the latest revision number (this will be YYYY)
developer@host:~/w3af/trunk/$ svn up

# Verify that your trunk is clean before merging
developer@host:~/w3af/trunk/$ svn st

# Merge
developer@host:~/w3af/trunk/$ svn merge -r XXXX:YYYY ../branches/branch-name/

You should also consider removing unused branches by running:

svn delete https://w3af.svn.sourceforge.net/svnroot/w3af/branches/branch-name

SVN tricks

After adding a new directory, we recommend you run this in your w3af working copy:

svn propset svn:ignore '*.py[co]' path/to/new/directory/

This will set the "svn:ignore" property to ignore all ".pyc" and ".pyo" files; and will help us maintain a clean "svn st". This is an example of an "svn st" before running the command:

M      plugins/discovery/pykto.py
?      plugins/discovery/tests/test_http_vs_https_dist.pyc
?      plugins/discovery/tests/__init__.pyc
?      extlib/simplejson/decoder.pyc
?      extlib/simplejson/ordered_dict.pyc
?      extlib/simplejson/scanner.pyc
?      extlib/simplejson/encoder.pyc
?      extlib/simplejson/__init__.pyc
?      core/controllers/auto_update/auto_update.pyc
?      core/controllers/auto_update/__init__.pyc
?      core/data/nltk_wrapper/nltk_wrapper.pyc
?      core/data/nltk_wrapper/__init__.pyc

And after applying the changes to the corresponding directories and commiting the changes:

M      plugins/discovery/pykto.py

Still need to verify how to achieve this, but a good idea would be to apply autopep8 in a hook that would run on all .py files before they get commited.

Releasing a new version of w3af

Releasing a new version of w3af is much more than just building the new "bz2 package"! For instructions about the technical and community tasks around a release, please visit our shipping a new release page.