[W3af-develop] Fwd: Inquiry about creating plugin depending on 3rd party package

[Piotr L. <pio...@gm...>]

Dear Andres,

2015-08-06 14:06 GMT+02:00 Andres Riancho <and...@gm...>:

> Piotr,
>
> On Thu, Aug 6, 2015 at 5:38 AM, Piotr Lizończyk
> <pio...@gm...> wrote:
> > Hi w3af developers community,
> > I'm working on tool that discovers technologies used on websites. It's
> > called WAD (https://github.com/CERN-CERT/WAD), it is based on Wappalyzer
> > browser extension (https://github.com/AliasIO/Wappalyzer) and I would
> like
> > to create an "infrastructure" plugin for w3af, that would run it and
> provide
> > user with information, that we can scrape out of website's HTML content.
>
> Sounds good! In the past I had the same idea and wrote it as this [0]
> issue. While reviewing the issue I found two "WAD" implementations:
>     * https://github.com/SebastianLopienski/WAD
>     * https://github.com/CERN-CERT/WAD
>
> What's the difference between these two? Are they related?


> [0] https://github.com/andresriancho/w3af/issues/1081


The first repository is really old implementation of this tool in original
author's repository, since then codebase has evolved as an internal tool at
CERN (it was and is still maintained by Sebastian). The second (CERN-CERT)
is recently published version, with main intention to make the tool
available to public and to integrate it with complex solutions like w3af.


>
>
> > The package was created at CERN and it is maintained actively for a
> couple
> > of years. While the process of contributing to w3af is clear, it is
> obvious
> > that I should ask you about adding this package as dependency, so my
> work on
> > the pull request is not a waste of time.
>
> Agreed!
>
> > I believe that this addition would be very valuable to w3af users, since
> it
> > can provide large amount of information about both backend and frontend
> > technologies used on website.
>
> Agreed on this one too.
>
> Before we can integrate anything into w3af there are some things to
> take into account:
>   * WAD code license: GPL3. AFAIK there is no problem with w3af (GPL2)
> having a requirement (not bundled in the same repository) that's
> licensed as GPL3
>
>
We chose GPL3 as license, because it works seamlessly with GPL2, under
which is Wappalyzer licensed.


>   * DB license: You're including the db inside your repository. Are
> the licenses compatible? Is this acceptable use of these files [2] ?
>
>
This DB (apps.json file) comes directly from Wappalyzer, which is under
GPLv2, so we are free to use it, as long as the license is GPL compatible.
CERN-CERT team has contributed a lot into that database, as a side-note. I
will include information about origin and license of those files into
codebase.

  * Most efficient way to integrate w3af with WAD:
>
>         - Looks like WAD is a simple wrapper around the DB, the code
> is clean and tested. Entry point seems to be Detector.detect_multiple
> which performs an HTTP request and then analyzes the response. The
> only problem I see there is that in the w3af framework the user can
> setup many HTTP client options (proxy, timeout, etc.) which won't be
> respected if we just use wad's urlopen function. I guess that
> Detector.detect_multiple will have to be rewritten (maybe specify a
> urlopen as an optional parameter?) to use w3af's ExtendedUrllib
>
>
This won't be a problem, I'm free to change WAD's code. Thanks for noting
that, I will implement that in code.


>         - The information found by WAD must be stored in the knowledge
> base so other plugins can re-use this information
>
        - The information found by WAD must be stored in the knowledge
> base using an Info instance with the right name and description text
> so a regular user can understand what was found
>


I've already started working on plugin, basing on halberd.py infrastructure
plugin. I successfully managed to store results in database, right now I'm
working on making those results more human-readable.


>
> Also, I see that WAD is at pypi which makes it easier for us to use in
> w3af since we can add it to the requirements file [1].
>
> Not a requirement/blocker but just curious, is WAD already bundled in Kali?
>

No, it isn't, since the public release happened very recently. That's a
very good idea though, thank you for that one, I'll surely look into it.


>
> To sum up, I believe everything looks good. If you send a clean PR
> which uses wad as an external dependency it will be accepted.
>
> [0] https://pypi.python.org/pypi/wad
> [1]
> https://github.com/andresriancho/w3af/blob/master/w3af/core/controllers/dependency_check/requirements.py
> [2] https://github.com/CERN-CERT/WAD/tree/master/wad/etc
>
> > I'm waiting to hear from you, with kind regards,
> > Piotr Lizończyk
> >
> >
> >
> ------------------------------------------------------------------------------
> >
> > _______________________________________________
> > W3af-develop mailing list
> > W3a...@li...
> > https://lists.sourceforge.net/lists/listinfo/w3af-develop
> >
>
>
>
> --
> Andrés Riancho
> Project Leader at w3af - http://w3af.org/
> Web Application Attack and Audit Framework
> Twitter: @w3af
> GPG: 0x93C344F3
>



Thank you for your input and approval, I expect to deliver pull request in
following days.

Regards,
Piotr Lizończyk