Re: [W3af-develop] [SPAM] celery issues

[Andres R. <and...@gm...>]

Well, yeah, that might be a problem. I recommend:

 * You should use different processes (prefork/processes should be ok)
 * Make sure ALL your w3af imports are done INSIDE the celery task.
So, this won't work:

from w3af import something

class Scan(Task):
    def run(...):
        something()

But this might/should:

class Scan(Task):
    def run(...):
        from w3af import something
        something()


I haven't played with celery+latest w3af, but I'm 100% sure that old
w3af versions (that didn't make use of multiprocessing) did work
inside celery tasks. Let me know how your implementation goes,

On Fri, Apr 3, 2015 at 6:10 AM, Electric Mind <ma...@ze...> wrote:
> Hi, Andres.
>
> Could you please show code example how to start your multiply w3af core
> inside of celery processes / prefork.
> Example:
>
> celery -A scanner worker --concurrency 2 -l debug -P prefork.
> celery -A scanner worker --concurrency 2 -l debug -P processes
>
> I have troubles with celery threads in my previous code example, because
> they use global Knowledge base object which is shared during the celery
> worker live.
>
> Regards.
>
> On 25 Mar 2015, at 21:55, Andres Riancho <and...@gm...> wrote:
>
> Electric Mind,
>
>    Please read inline,
>
> On Wed, Mar 25, 2015 at 3:49 PM, Electric Mind <ma...@ze...>
> wrote:
>
> Hello everyone!
>
> I’d like to start my w3af instances inside of the celery.
> Code example is below. What is the best practice for doing that ?
>
>
> First I would recommend you configure celery to:
> * Prefetch only one task
> * Run one task at the time
> * Run scan tasks in different processes
>
> Regards!
>
>
> scanner.py
> -----
>
> # -*- coding: utf-8 -*-
> # !/usr/bin/env python
>
> from __future__ import absolute_import
> from multiprocessing import cpu_count
> from w3af.core.controllers.w3afCore import w3afCore
> from w3af.plugins.tests.helper import create_target_option_list
> from w3af.core.data.parsers.url import URL
> from w3af.core.data.options.option_list import OptionList
> from w3af.core.data.options.opt_factory import opt_factory
> from w3af.core.data.kb import knowledge_base
> from w3af.core.data.kb.info_set import InfoSet
>
> from scanner.celery import app
>
>
> @app.task
> def start_scan(host):
>    target_opts = create_target_option_list(URL(host))
>    core = w3afCore()
>    core.WORKER_THREADS = cpu_count() * 2
>
>
> Threads are not processes
>
>    core.target.set_options(target_opts)
>
>    core.plugins.set_plugins(
>        [
>            'xss_protection_header',
>            'csp',
>            'strange_headers',
>            'click_jacking',
>            'content_type_options_header'
>        ], 'grep')
>
>    core.plugins.set_plugins(
>        [
>            'allowed_methods',
>            'find_vhosts'
>        ], 'infrastructure')
>
>    core.plugins.set_plugins(
>        [
>            'dir_file_bruter',
>            'robots_txt',
>            'ria_enumerator'
>        ], 'crawl')
>
>    core.plugins.set_plugins(
>        [
>            'ssl',
>        ], 'audit')
>
>    core.plugins.set_plugins(
>        [
>            'console'
>        ], 'output')
>    console_options = OptionList()
>    console_options.add(opt_factory('verbose', True, 'desc', 'boolean'))
>    core.plugins.set_plugin_options('output', 'console', console_options)
>
>    core.plugins.init_plugins()
>    core.start()
>
>    result = {'information_disclosures': [], 'vulnerabilities': []}
>
>    for information_disclosure in knowledge_base.kb.get_all_infos():
>        if isinstance(information_disclosure, InfoSet):
>
> result['information_disclosures'].append("{}".format(information_disclosure.get_desc()))
>        else:
>
> result['information_disclosures'].append("{}".format(information_disclosure))
>
>    for vulnerability in knowledge_base.kb.get_all_vulns():
>        result['vulnerabilities'].append("{}".format(vulnerability))
>
>
> Check the knowledge base hooks, I would rather use that instead of
> waiting for the scan to finish. With the hooks you can give the users
> instant feedback by sending a new task to the broker containing the
> vulnerability. Then a worker would consume the task (with the vuln)
> and show it to the user.
>
>    return result
>
> ------------------------------------------------------------------------------
> Dive into the World of Parallel Programming The Go Parallel Website,
> sponsored
> by Intel and developed in partnership with Slashdot Media, is your hub for
> all
> things parallel software development, from weekly thought leadership blogs
> to
> news, videos, case studies, tutorials and more. Take a look and join the
> conversation now. http://goparallel.sourceforge.net/
> _______________________________________________
> W3af-develop mailing list
> W3a...@li...
> https://lists.sourceforge.net/lists/listinfo/w3af-develop
>
>
>
>
> --
> Andrés Riancho
> Project Leader at w3af - http://w3af.org/
> Web Application Attack and Audit Framework
> Twitter: @w3af
> GPG: 0x93C344F3
>
>



-- 
Andrés Riancho
Project Leader at w3af - http://w3af.org/
Web Application Attack and Audit Framework
Twitter: @w3af
GPG: 0x93C344F3