Re: [W3af-develop] [SPAM] celery issues

[Andres R. <and...@gm...>]

Electric Mind,

    Please read inline,

On Wed, Mar 25, 2015 at 3:49 PM, Electric Mind <ma...@ze...> wrote:
> Hello everyone!
>
> I’d like to start my w3af instances inside of the celery.
> Code example is below. What is the best practice for doing that ?

First I would recommend you configure celery to:
 * Prefetch only one task
 * Run one task at the time
 * Run scan tasks in different processes

> Regards!
>
>
> scanner.py
> -----
>
> # -*- coding: utf-8 -*-
> # !/usr/bin/env python
>
> from __future__ import absolute_import
> from multiprocessing import cpu_count
> from w3af.core.controllers.w3afCore import w3afCore
> from w3af.plugins.tests.helper import create_target_option_list
> from w3af.core.data.parsers.url import URL
> from w3af.core.data.options.option_list import OptionList
> from w3af.core.data.options.opt_factory import opt_factory
> from w3af.core.data.kb import knowledge_base
> from w3af.core.data.kb.info_set import InfoSet
>
> from scanner.celery import app
>
>
> @app.task
> def start_scan(host):
>     target_opts = create_target_option_list(URL(host))
>     core = w3afCore()
>     core.WORKER_THREADS = cpu_count() * 2

Threads are not processes

>     core.target.set_options(target_opts)
>
>     core.plugins.set_plugins(
>         [
>             'xss_protection_header',
>             'csp',
>             'strange_headers',
>             'click_jacking',
>             'content_type_options_header'
>         ], 'grep')
>
>     core.plugins.set_plugins(
>         [
>             'allowed_methods',
>             'find_vhosts'
>         ], 'infrastructure')
>
>     core.plugins.set_plugins(
>         [
>             'dir_file_bruter',
>             'robots_txt',
>             'ria_enumerator'
>         ], 'crawl')
>
>     core.plugins.set_plugins(
>         [
>             'ssl',
>         ], 'audit')
>
>     core.plugins.set_plugins(
>         [
>             'console'
>         ], 'output')
>     console_options = OptionList()
>     console_options.add(opt_factory('verbose', True, 'desc', 'boolean'))
>     core.plugins.set_plugin_options('output', 'console', console_options)
>
>     core.plugins.init_plugins()
>     core.start()
>
>     result = {'information_disclosures': [], 'vulnerabilities': []}
>
>     for information_disclosure in knowledge_base.kb.get_all_infos():
>         if isinstance(information_disclosure, InfoSet):
>
> result['information_disclosures'].append("{}".format(information_disclosure.get_desc()))
>         else:
>
> result['information_disclosures'].append("{}".format(information_disclosure))
>
>     for vulnerability in knowledge_base.kb.get_all_vulns():
>         result['vulnerabilities'].append("{}".format(vulnerability))

Check the knowledge base hooks, I would rather use that instead of
waiting for the scan to finish. With the hooks you can give the users
instant feedback by sending a new task to the broker containing the
vulnerability. Then a worker would consume the task (with the vuln)
and show it to the user.

>     return result
>
> ------------------------------------------------------------------------------
> Dive into the World of Parallel Programming The Go Parallel Website,
> sponsored
> by Intel and developed in partnership with Slashdot Media, is your hub for
> all
> things parallel software development, from weekly thought leadership blogs
> to
> news, videos, case studies, tutorials and more. Take a look and join the
> conversation now. http://goparallel.sourceforge.net/
> _______________________________________________
> W3af-develop mailing list
> W3a...@li...
> https://lists.sourceforge.net/lists/listinfo/w3af-develop
>



-- 
Andrés Riancho
Project Leader at w3af - http://w3af.org/
Web Application Attack and Audit Framework
Twitter: @w3af
GPG: 0x93C344F3