w3af-users — User mailing list

[W3af-users] setting up upstream proxy with w3af

[Advait J. <adv...@gm...>]

Hello everyone,

I was trying to figure out if there is any way to set upstream proxy in
W3af. I usually set it up on my Burp to route requets via my intermediate
box. In the case of W3af, the only proxy option that I know of allows
starting a local proxy to intercept requests but does not let me setup a
upstream proxy.

Any workarounds for this issue will be greatly appreciated.


Thank you,


Adwait

Re: [W3af-users] performance issues post-scan

[Andres R. <and...@gm...>]

Another comment on that, the version that is embedded in the latest
docker might not be the latest from w3af github repo (master branch).

The latest from master might have multiple improvements.


On Wed, Oct 30, 2019 at 2:30 AM Chris Herdt <ch...@gm...> wrote:
>
> I believe my issue was due to low drive space. I'm going to increase the drive space and give it another try.
>
>
> On Tue, Oct 29, 2019 at 9:29 PM Chris Herdt <ch...@gm...> wrote:
>>
>> I'm running a Dockerized version of w3af via w3af_console_docker on Kali Linux. I'm targeting an instance of Mutillidae, using the OWASP_TOP10 profile.
>>
>> The scan appeared to take about 15 minutes, but never completed. I no longer see web requests to the target server, but for the past 20 hours or so I see messages like this, with decreasing values for "requests per minute" over time:
>>
>>> |------------------------------------------------------------------------------|
>>> | Crawling Method: GET | http://192.168.1.57/icons/small/ | Query string:      |
>>> | (view) using crawl.phpinfo                                                   |
>>> | Auditing Method: GET | http://192.168.1.57/icons/small/ | Query string:      |
>>> | (view) using audit.frontpage                                                 |
>>> | Crawl phase: In (None URLs/min) Out (None URLs/min) Pending (None URLs) ETA  |
>>> | (None)                                                                       |
>>> | Audit phase: In (None URLs/min) Out (None URLs/min) Pending (None URLs) ETA  |
>>> | (None)                                                                       |
>>> | Requests per minute: 9                                                       |
>>> |------------------------------------------------------------------------------|
>>
>>
>> Other profiles, such as web_infrastructure, finished faster but still had a substantial delay after the actual scanning appeared to be complete.
>>
>> I saw similar behavior described years ago in this thread, but I'm not sure if the root cause of that issue was determined:
>> https://sourceforge.net/p/w3af/mailman/message/31150639/
>>
>> Thanks for any insights,
>>
>> --
>> Chris Herdt
>>
>
>
> --
> Chris Herdt
> https://osric.com/chris/
> _______________________________________________
> W3af-users mailing list
> W3a...@li...
> https://lists.sourceforge.net/lists/listinfo/w3af-users



--
Andrés Riancho
Project Leader at w3af - http://w3af.org/
Web Application Attack and Audit Framework
Twitter: @w3af
GPG: 0x93C344F3

Re: [W3af-users] performance issues post-scan

[Chris H. <ch...@gm...>]

I believe my issue was due to low drive space. I'm going to increase the
drive space and give it another try.


On Tue, Oct 29, 2019 at 9:29 PM Chris Herdt <ch...@gm...> wrote:

> I'm running a Dockerized version of w3af via w3af_console_docker on Kali
> Linux. I'm targeting an instance of Mutillidae, using the OWASP_TOP10
> profile.
>
> The scan appeared to take about 15 minutes, but never completed. I no
> longer see web requests to the target server, but for the past 20 hours or
> so I see messages like this, with decreasing values for "requests per
> minute" over time:
>
>
>> |------------------------------------------------------------------------------|
>> | Crawling Method: GET | http://192.168.1.57/icons/small/ | Query
>> string:      |
>> | (view) using crawl.phpinfo
>>       |
>> | Auditing Method: GET | http://192.168.1.57/icons/small/ | Query
>> string:      |
>> | (view) using audit.frontpage
>>       |
>> | Crawl phase: In (None URLs/min) Out (None URLs/min) Pending (None URLs)
>> ETA  |
>> | (None)
>>       |
>> | Audit phase: In (None URLs/min) Out (None URLs/min) Pending (None URLs)
>> ETA  |
>> | (None)
>>       |
>> | Requests per minute: 9
>>       |
>>
>> |------------------------------------------------------------------------------|
>>
>
> Other profiles, such as web_infrastructure, finished faster but still had
> a substantial delay after the actual scanning appeared to be complete.
>
> I saw similar behavior described years ago in this thread, but I'm not
> sure if the root cause of that issue was determined:
> https://sourceforge.net/p/w3af/mailman/message/31150639/
>
> Thanks for any insights,
>
> --
> Chris Herdt
>
>

-- 
Chris Herdt
https://osric.com/chris/

[W3af-users] performance issues post-scan

[Chris H. <ch...@gm...>]

I'm running a Dockerized version of w3af via w3af_console_docker on Kali
Linux. I'm targeting an instance of Mutillidae, using the OWASP_TOP10
profile.

The scan appeared to take about 15 minutes, but never completed. I no
longer see web requests to the target server, but for the past 20 hours or
so I see messages like this, with decreasing values for "requests per
minute" over time:

|------------------------------------------------------------------------------|
> | Crawling Method: GET | http://192.168.1.57/icons/small/ | Query string:
>      |
> | (view) using crawl.phpinfo
>     |
> | Auditing Method: GET | http://192.168.1.57/icons/small/ | Query string:
>      |
> | (view) using audit.frontpage
>     |
> | Crawl phase: In (None URLs/min) Out (None URLs/min) Pending (None URLs)
> ETA  |
> | (None)
>     |
> | Audit phase: In (None URLs/min) Out (None URLs/min) Pending (None URLs)
> ETA  |
> | (None)
>     |
> | Requests per minute: 9
>     |
>
> |------------------------------------------------------------------------------|
>

Other profiles, such as web_infrastructure, finished faster but still had a
substantial delay after the actual scanning appeared to be complete.

I saw similar behavior described years ago in this thread, but I'm not sure
if the root cause of that issue was determined:
https://sourceforge.net/p/w3af/mailman/message/31150639/

Thanks for any insights,

-- 
Chris Herdt

Re: [W3af-users] New to w3af

[Andres R. <and...@gm...>]

James,

    Thanks for your email, comments and questions inline:

On Wed, Sep 18, 2019 at 4:00 PM James Pifer <je...@ob...> wrote:
>
> I came across w3af and have it installed (for the most part). With the
> help of docker I'm able to run the console, but I keep getting this when
> I run the gui:
>
> user1@UbuntuDocker:/opt/w3af/extras/docker/scripts$ sudo ./w3af_gui_docker
> [sudo] password for user1:
> root@172.17.0.2's password:
> w3af's requirements are not met, one or more third-party libraries need
> to be installed.
>
> On Ubuntu 12.04 systems please install the following operating system
> packages before running the pip installer:
>      sudo apt-get -y install python-webkit
>
> A script with these commands has been created for you at
> /tmp/w3af_dependency_install.sh
>
> (process:18): Gtk-WARNING **: Locale not supported by C library.
>          Using the fallback 'C' locale.
> /usr/lib/python2.7/dist-packages/gtk-2.0/gtk/__init__.py:57: GtkWarning:
> could not open display
>    warnings.warn(str(e), _gtk.Warning)
> user1@UbuntuDocker:/opt/w3af/extras/docker/scripts$
>
>
>
> $ sudo apt-get -y install python-webkit
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> python-webkit is already the newest version (1.1.8-3.1).
>
>
> Not sure where to go from here. Any suggestions?

Got the same error when trying to run it myself.

Tried to build a new docker version and failed to do it in the time I had.

I recommend you try to install w3af in your OS, most likely using virtualenv:
http://docs.w3af.org/en/latest/advanced-install.html#installing-using-virtualenv

> I've run some scans from the console using the target/set target and
> plugins enable all on several URLs trying to prepare for an audit. I
> really have yet to find anything. Maybe our apps are more secure than I
> think and there really is nothing to find. The scans are also very
> quick, whereas Tenable takes a long time to run scans. Is that normal?

Quick is very relative.

Scan times depend on the site size, number of enabled plugins, the
network connection speed, etc.

> Not sure how to know whether it's really working.

To know if the scan is working I recommend enabling the text_file
output plugin with `debug` set to True. Then `tail -f` the file to see
HTTP requests being sent.

> Anyway, really appreciate what the app is doing. I'm not a security
> expert, just an IT guy, so any help is appreciated.
>
> Thanks!
>
>
>
>
> _______________________________________________
> W3af-users mailing list
> W3a...@li...
> https://lists.sourceforge.net/lists/listinfo/w3af-users



-- 
Andrés Riancho
Project Leader at w3af - http://w3af.org/
Web Application Attack and Audit Framework
Twitter: @w3af
GPG: 0x93C344F3

[W3af-users] New to w3af

[James P. <je...@ob...>]

I came across w3af and have it installed (for the most part). With the 
help of docker I'm able to run the console, but I keep getting this when 
I run the gui:

user1@UbuntuDocker:/opt/w3af/extras/docker/scripts$ sudo ./w3af_gui_docker
[sudo] password for user1:
root@172.17.0.2's password:
w3af's requirements are not met, one or more third-party libraries need 
to be installed.

On Ubuntu 12.04 systems please install the following operating system 
packages before running the pip installer:
     sudo apt-get -y install python-webkit

A script with these commands has been created for you at 
/tmp/w3af_dependency_install.sh

(process:18): Gtk-WARNING **: Locale not supported by C library.
         Using the fallback 'C' locale.
/usr/lib/python2.7/dist-packages/gtk-2.0/gtk/__init__.py:57: GtkWarning: 
could not open display
   warnings.warn(str(e), _gtk.Warning)
user1@UbuntuDocker:/opt/w3af/extras/docker/scripts$



$ sudo apt-get -y install python-webkit
Reading package lists... Done
Building dependency tree
Reading state information... Done
python-webkit is already the newest version (1.1.8-3.1).


Not sure where to go from here. Any suggestions?

I've run some scans from the console using the target/set target and 
plugins enable all on several URLs trying to prepare for an audit. I 
really have yet to find anything. Maybe our apps are more secure than I 
think and there really is nothing to find. The scans are also very 
quick, whereas Tenable takes a long time to run scans. Is that normal?

Not sure how to know whether it's really working.

Anyway, really appreciate what the app is doing. I'm not a security 
expert, just an IT guy, so any help is appreciated.

Thanks!

Re: [W3af-users] w3af as a service

[Andres R. <and...@gm...>]

Not really, any DB you know how to use and maintain will make it.

El jue., 13 de junio de 2019 7:08 p. m., Rafael Barbosa da Silva <
raf...@gm...> escribió:

> Thanks a lot Andres!
>
> Makes a lot of sense.
>
> Is there any DB would you recommend?
>
> Regards.
> Rafael
>
> Em qui, 13 de jun de 2019 às 18:20, Andres Riancho <
> and...@gm...> escreveu:
>
>> Rafael,
>>
>>     Thanks for your interest in w3af and using it to build a SaaS.
>> Answers and comments inline:
>>
>> On Thu, Jun 13, 2019 at 4:07 PM Rafael Barbosa da Silva
>> <raf...@gm...> wrote:
>> >
>> > Hello everyone, how are you?
>> >
>> > I would like to biuld a service that runs w3af and persists results in
>> a database. The idea is provide a web interface where we can run a scan and
>> also navigate through the results. Have any of you guys done something
>> related and would like to share? And even if you have not done so, would
>> you like to suggest a strategy? What about invoke a scan through the web
>> interface? Is there a way to run multiple instances of w3af scans?
>>
>>     This is how I would do it, and the ways I have heard others have done
>> it:
>>
>>  * The web interface you show to your user needs to know almost
>> nothing about w3af
>>
>>  * When the user clicks on "start scan" a new w3af scan script [0] is
>> created. Your SaaS will most likely have 3 or 4 different scan script
>> templates, for different use-cases your customers might have. The
>> template is filled with the target URL, credentials, etc. all provided
>> by the user, and then sent to a scan queue.
>>
>>  * The scans just sit in the queue until one of the scan workers gets to
>> them
>>
>>  * Scan workers are EC2 instances that read scan scripts from the
>> queue and execute them. If you want to get fancy, you can measure the
>> scan queue size and do +1 or -1 on the number of scan workers
>> depending on load
>>
>>  * The scan script should be configured to use output.xml_file output.
>> This plugin writes data to disk every ~30 seconds or so.
>>
>>  * The scan worker server will run w3af_console -s script AND another
>> process that monitors the XML file. This process will extract
>> vulnerabilities from the file and save them to a vulnerabilities
>> queue. The process that monitors the XML file should only report new
>> vulnerabilities, no duplicated vulns should be sent to the
>> vulnerabilities queue.
>>
>>   * Another process will read vulnerabilities from the queue and store
>> them to the DB. The front-end web application reads vulnerabilities
>> from the DB. Stuff like marking them as a false positive are handled
>> in the DB, w3af knows nothing about that.
>>
>>   * Just like there is a queue for vulnerabilities, you could add a
>> queue for scan progress. The XML file also contains that information.
>>
>>     Makes sense?
>>
>> [0] https://github.com/andresriancho/w3af/tree/master/scripts
>>
>> > Sorry about too many questions
>> > Regards.
>> > Rafael
>> > _______________________________________________
>> > W3af-users mailing list
>> > W3a...@li...
>> > https://lists.sourceforge.net/lists/listinfo/w3af-users
>>
>>
>>
>> --
>> Andrés Riancho
>> Project Leader at w3af - http://w3af.org/
>> Web Application Attack and Audit Framework
>> Twitter: @w3af
>> GPG: 0x93C344F3
>>
>

Re: [W3af-users] w3af as a service

[Rafael B. da S. <raf...@gm...>]

Thanks a lot Andres!

Makes a lot of sense.

Is there any DB would you recommend?

Regards.
Rafael

Em qui, 13 de jun de 2019 às 18:20, Andres Riancho <and...@gm...>
escreveu:

> Rafael,
>
>     Thanks for your interest in w3af and using it to build a SaaS.
> Answers and comments inline:
>
> On Thu, Jun 13, 2019 at 4:07 PM Rafael Barbosa da Silva
> <raf...@gm...> wrote:
> >
> > Hello everyone, how are you?
> >
> > I would like to biuld a service that runs w3af and persists results in a
> database. The idea is provide a web interface where we can run a scan and
> also navigate through the results. Have any of you guys done something
> related and would like to share? And even if you have not done so, would
> you like to suggest a strategy? What about invoke a scan through the web
> interface? Is there a way to run multiple instances of w3af scans?
>
>     This is how I would do it, and the ways I have heard others have done
> it:
>
>  * The web interface you show to your user needs to know almost
> nothing about w3af
>
>  * When the user clicks on "start scan" a new w3af scan script [0] is
> created. Your SaaS will most likely have 3 or 4 different scan script
> templates, for different use-cases your customers might have. The
> template is filled with the target URL, credentials, etc. all provided
> by the user, and then sent to a scan queue.
>
>  * The scans just sit in the queue until one of the scan workers gets to
> them
>
>  * Scan workers are EC2 instances that read scan scripts from the
> queue and execute them. If you want to get fancy, you can measure the
> scan queue size and do +1 or -1 on the number of scan workers
> depending on load
>
>  * The scan script should be configured to use output.xml_file output.
> This plugin writes data to disk every ~30 seconds or so.
>
>  * The scan worker server will run w3af_console -s script AND another
> process that monitors the XML file. This process will extract
> vulnerabilities from the file and save them to a vulnerabilities
> queue. The process that monitors the XML file should only report new
> vulnerabilities, no duplicated vulns should be sent to the
> vulnerabilities queue.
>
>   * Another process will read vulnerabilities from the queue and store
> them to the DB. The front-end web application reads vulnerabilities
> from the DB. Stuff like marking them as a false positive are handled
> in the DB, w3af knows nothing about that.
>
>   * Just like there is a queue for vulnerabilities, you could add a
> queue for scan progress. The XML file also contains that information.
>
>     Makes sense?
>
> [0] https://github.com/andresriancho/w3af/tree/master/scripts
>
> > Sorry about too many questions
> > Regards.
> > Rafael
> > _______________________________________________
> > W3af-users mailing list
> > W3a...@li...
> > https://lists.sourceforge.net/lists/listinfo/w3af-users
>
>
>
> --
> Andrés Riancho
> Project Leader at w3af - http://w3af.org/
> Web Application Attack and Audit Framework
> Twitter: @w3af
> GPG: 0x93C344F3
>

Re: [W3af-users] w3af as a service

[Andres R. <and...@gm...>]

Rafael,

    Thanks for your interest in w3af and using it to build a SaaS.
Answers and comments inline:

On Thu, Jun 13, 2019 at 4:07 PM Rafael Barbosa da Silva
<raf...@gm...> wrote:
>
> Hello everyone, how are you?
>
> I would like to biuld a service that runs w3af and persists results in a database. The idea is provide a web interface where we can run a scan and also navigate through the results. Have any of you guys done something related and would like to share? And even if you have not done so, would you like to suggest a strategy? What about invoke a scan through the web interface? Is there a way to run multiple instances of w3af scans?

    This is how I would do it, and the ways I have heard others have done it:

 * The web interface you show to your user needs to know almost
nothing about w3af

 * When the user clicks on "start scan" a new w3af scan script [0] is
created. Your SaaS will most likely have 3 or 4 different scan script
templates, for different use-cases your customers might have. The
template is filled with the target URL, credentials, etc. all provided
by the user, and then sent to a scan queue.

 * The scans just sit in the queue until one of the scan workers gets to them

 * Scan workers are EC2 instances that read scan scripts from the
queue and execute them. If you want to get fancy, you can measure the
scan queue size and do +1 or -1 on the number of scan workers
depending on load

 * The scan script should be configured to use output.xml_file output.
This plugin writes data to disk every ~30 seconds or so.

 * The scan worker server will run w3af_console -s script AND another
process that monitors the XML file. This process will extract
vulnerabilities from the file and save them to a vulnerabilities
queue. The process that monitors the XML file should only report new
vulnerabilities, no duplicated vulns should be sent to the
vulnerabilities queue.

  * Another process will read vulnerabilities from the queue and store
them to the DB. The front-end web application reads vulnerabilities
from the DB. Stuff like marking them as a false positive are handled
in the DB, w3af knows nothing about that.

  * Just like there is a queue for vulnerabilities, you could add a
queue for scan progress. The XML file also contains that information.

    Makes sense?

[0] https://github.com/andresriancho/w3af/tree/master/scripts

> Sorry about too many questions
> Regards.
> Rafael
> _______________________________________________
> W3af-users mailing list
> W3a...@li...
> https://lists.sourceforge.net/lists/listinfo/w3af-users



-- 
Andrés Riancho
Project Leader at w3af - http://w3af.org/
Web Application Attack and Audit Framework
Twitter: @w3af
GPG: 0x93C344F3

[W3af-users] w3af as a service

[Rafael B. da S. <raf...@gm...>]

Hello everyone, how are you?

I would like to biuld a service that runs w3af and persists results in a
database. The idea is provide a web interface where we can run a scan and
also navigate through the results. Have any of you guys done something
related and would like to share? And even if you have not done so, would
you like to suggest a strategy? What about invoke a scan through the web
interface? Is there a way to run multiple instances of w3af scans?

Sorry about too many questions
Regards.
Rafael

Re: [W3af-users] ubuntu 18.04 issue

[Luca B. <be...@la...>]

Thx MMillet!
I'll give it a try.

In the meanwhile if devs are looking for a betatester, here i am :)

Luca

--
Dr. Luca Benassi
Laboratori Guglielmo Marconi
Via Porrettana 123, 40037 Pontecchio BO - ITALY
Phone:+39-0516781934 Fax:+39-051846479 e-mail: be...@la...
Systems & Networks Division

----- Original Message -----
> From: "mmillet via W3af-users" <w3a...@li...>
> To: w3a...@li...
> Sent: Tuesday, May 14, 2019 9:26:16 AM
> Subject: Re: [W3af-users] ubuntu 18.04 issue

> Hello Luca,
> 
> I had the same issue, the gui depends on packages that do not exist on
> recent linux distributions (python-webkit).
> ./w3af_console and ./w3af_api should be fine though (I have them running
> on debian testing).
> 
> I haven't tried it but there seems to be a gui for the API :
> https://github.com/andresriancho/w3af-api-client
> although it doesn't seem very recent.
> 
> 
> 
> 
> On 13/05/2019 17:35, Luca Benassi wrote:
>> I'm trying to install on a "Ubuntu 18.04.2 LTS" but no way, dep issues (about
>> python-webkit)
>>
>> Here's my session log:
>>
>> :~/w3af$ ./w3af_gui
>> w3af's requirements are not met, one or more third-party libraries need to be
>> installed.
>>
>> On Ubuntu 18.04 systems please install the following operating system packages
>> before running the pip installer:
>>      sudo apt-get -y install npm python-webkit libffi-dev graphviz libsqlite3-dev
>>      libxslt1-dev libyaml-dev libssl1.0-dev python-gtksourceview2 libxml2-dev
>> https://github.com/andresriancho/w3af-api-client
>> Your python installation needs the following modules to run w3af:
>>      pybloomfilter phply nltk tblib pdfminer concurrent.futures OpenSSL ndg lxml
>>      scapy.config guess_language cluster msgpack ntlm Halberd darts.lib.utils jinja2
>>      vulndb markdown psutil ds_store termcolor mitmproxy ruamel.ordereddict Flask
>>      yaml tldextract pebble acora esmre diff_match_patch bravado_core lz4 vulners
>>      ipaddresses xdot
>>
>>
>> After installing any missing operating system packages, use pip to install the
>> remaining modules:
>>      sudo pip install pybloomfiltermmap==0.3.14 phply==0.9.1 nltk==3.0.1 tblib==0.2.0
>>      pdfminer==20140328 futures==3.2.0 pyOpenSSL==18.0.0 ndg-httpsclient==0.4.0
>>      lxml==3.4.4 scapy==2.4.0 guess-language==0.2 cluster==1.1.1b3 msgpack==0.5.6
>>      python-ntlm==1.0.1 halberd==0.2.4 darts.util.lru==0.5 Jinja2==2.10
>>      vulndb==0.1.1 markdown==2.6.1 psutil==5.4.8 ds-store==1.1.2 termcolor==1.1.0
>>      mitmproxy==0.13 ruamel.ordereddict==0.4.8 Flask==0.10.1 PyYAML==3.12
>>      tldextract==1.7.2 pebble==4.3.8 acora==2.1 esmre==0.3.1
>>      diff-match-patch==20121119 bravado-core==5.0.2 lz4==1.1.0 vulners==1.3.0
>>      ipaddresses==0.0.2 xdot==0.6
>>
>> External programs used by w3af are not installed or were not found.Run these
>> commands to install them on your system:
>>
>>      npm install -g retire
>>
>> A script with these commands has been created for you at
>> /tmp/w3af_dependency_install.sh
>> The required "dot" binary is missing, please install the "graphviz" package in
>> your operating system.
>>
>> :~/w3af$ sudo apt-get -y install npm python-webkit libffi-dev graphviz
>> libsqlite3-dev libxslt1-dev libyaml-dev libssl1.0-dev python-gtksourceview2
>> libxml2-dev
>> Lettura elenco dei pacchetti... Fatto
>> Generazione albero delle dipendenze
>> Lettura informazioni sullo stato... Fatto
>> Il pacchetto python-webkit non ha versioni disponibili, ma è nominato da un
>> altro
>> pacchetto. Questo potrebbe indicare che il pacchetto è mancante, obsoleto
>> oppure è disponibile solo all'interno di un'altra sorgente
>>
>> E: Il pacchetto "python-webkit" non ha candidati da installare
>>
>> (sorry for the italian)
>>
>> thx,
>> Luca
>>
>>
>> _______________________________________________
>> W3af-users mailing list
>> W3a...@li...
>> https://lists.sourceforge.net/lists/listinfo/w3af-users
> 
> 
> _______________________________________________
> W3af-users mailing list
> W3a...@li...
> https://lists.sourceforge.net/lists/listinfo/w3af-users

Re: [W3af-users] ubuntu 18.04 issue

[mmillet <mm...@si...>]

Hello Luca,

I had the same issue, the gui depends on packages that do not exist on 
recent linux distributions (python-webkit).
./w3af_console and ./w3af_api should be fine though (I have them running 
on debian testing).

I haven't tried it but there seems to be a gui for the API : 
https://github.com/andresriancho/w3af-api-client
although it doesn't seem very recent.




On 13/05/2019 17:35, Luca Benassi wrote:
> I'm trying to install on a "Ubuntu 18.04.2 LTS" but no way, dep issues (about python-webkit)
>
> Here's my session log:
>
> :~/w3af$ ./w3af_gui
> w3af's requirements are not met, one or more third-party libraries need to be installed.
>
> On Ubuntu 18.04 systems please install the following operating system packages before running the pip installer:
>      sudo apt-get -y install npm python-webkit libffi-dev graphviz libsqlite3-dev libxslt1-dev libyaml-dev libssl1.0-dev python-gtksourceview2 libxml2-dev
> https://github.com/andresriancho/w3af-api-client
> Your python installation needs the following modules to run w3af:
>      pybloomfilter phply nltk tblib pdfminer concurrent.futures OpenSSL ndg lxml scapy.config guess_language cluster msgpack ntlm Halberd darts.lib.utils jinja2 vulndb markdown psutil ds_store termcolor mitmproxy ruamel.ordereddict Flask yaml tldextract pebble acora esmre diff_match_patch bravado_core lz4 vulners ipaddresses xdot
>
>
> After installing any missing operating system packages, use pip to install the remaining modules:
>      sudo pip install pybloomfiltermmap==0.3.14 phply==0.9.1 nltk==3.0.1 tblib==0.2.0 pdfminer==20140328 futures==3.2.0 pyOpenSSL==18.0.0 ndg-httpsclient==0.4.0 lxml==3.4.4 scapy==2.4.0 guess-language==0.2 cluster==1.1.1b3 msgpack==0.5.6 python-ntlm==1.0.1 halberd==0.2.4 darts.util.lru==0.5 Jinja2==2.10 vulndb==0.1.1 markdown==2.6.1 psutil==5.4.8 ds-store==1.1.2 termcolor==1.1.0 mitmproxy==0.13 ruamel.ordereddict==0.4.8 Flask==0.10.1 PyYAML==3.12 tldextract==1.7.2 pebble==4.3.8 acora==2.1 esmre==0.3.1 diff-match-patch==20121119 bravado-core==5.0.2 lz4==1.1.0 vulners==1.3.0 ipaddresses==0.0.2 xdot==0.6
>
> External programs used by w3af are not installed or were not found.Run these commands to install them on your system:
>
>      npm install -g retire
>
> A script with these commands has been created for you at /tmp/w3af_dependency_install.sh
> The required "dot" binary is missing, please install the "graphviz" package in your operating system.
>
> :~/w3af$ sudo apt-get -y install npm python-webkit libffi-dev graphviz libsqlite3-dev libxslt1-dev libyaml-dev libssl1.0-dev python-gtksourceview2 libxml2-dev
> Lettura elenco dei pacchetti... Fatto
> Generazione albero delle dipendenze
> Lettura informazioni sullo stato... Fatto
> Il pacchetto python-webkit non ha versioni disponibili, ma è nominato da un altro
> pacchetto. Questo potrebbe indicare che il pacchetto è mancante, obsoleto
> oppure è disponibile solo all'interno di un'altra sorgente
>
> E: Il pacchetto "python-webkit" non ha candidati da installare
>
> (sorry for the italian)
>
> thx,
> Luca
>
>
> _______________________________________________
> W3af-users mailing list
> W3a...@li...
> https://lists.sourceforge.net/lists/listinfo/w3af-users

[W3af-users] ubuntu 18.04 issue

[Luca B. <be...@la...>]

I'm trying to install on a "Ubuntu 18.04.2 LTS" but no way, dep issues (about python-webkit)

Here's my session log:

:~/w3af$ ./w3af_gui 
w3af's requirements are not met, one or more third-party libraries need to be installed.

On Ubuntu 18.04 systems please install the following operating system packages before running the pip installer:
    sudo apt-get -y install npm python-webkit libffi-dev graphviz libsqlite3-dev libxslt1-dev libyaml-dev libssl1.0-dev python-gtksourceview2 libxml2-dev

Your python installation needs the following modules to run w3af:
    pybloomfilter phply nltk tblib pdfminer concurrent.futures OpenSSL ndg lxml scapy.config guess_language cluster msgpack ntlm Halberd darts.lib.utils jinja2 vulndb markdown psutil ds_store termcolor mitmproxy ruamel.ordereddict Flask yaml tldextract pebble acora esmre diff_match_patch bravado_core lz4 vulners ipaddresses xdot


After installing any missing operating system packages, use pip to install the remaining modules:
    sudo pip install pybloomfiltermmap==0.3.14 phply==0.9.1 nltk==3.0.1 tblib==0.2.0 pdfminer==20140328 futures==3.2.0 pyOpenSSL==18.0.0 ndg-httpsclient==0.4.0 lxml==3.4.4 scapy==2.4.0 guess-language==0.2 cluster==1.1.1b3 msgpack==0.5.6 python-ntlm==1.0.1 halberd==0.2.4 darts.util.lru==0.5 Jinja2==2.10 vulndb==0.1.1 markdown==2.6.1 psutil==5.4.8 ds-store==1.1.2 termcolor==1.1.0 mitmproxy==0.13 ruamel.ordereddict==0.4.8 Flask==0.10.1 PyYAML==3.12 tldextract==1.7.2 pebble==4.3.8 acora==2.1 esmre==0.3.1 diff-match-patch==20121119 bravado-core==5.0.2 lz4==1.1.0 vulners==1.3.0 ipaddresses==0.0.2 xdot==0.6

External programs used by w3af are not installed or were not found.Run these commands to install them on your system:

    npm install -g retire

A script with these commands has been created for you at /tmp/w3af_dependency_install.sh
The required "dot" binary is missing, please install the "graphviz" package in your operating system.

:~/w3af$ sudo apt-get -y install npm python-webkit libffi-dev graphviz libsqlite3-dev libxslt1-dev libyaml-dev libssl1.0-dev python-gtksourceview2 libxml2-dev
Lettura elenco dei pacchetti... Fatto
Generazione albero delle dipendenze       
Lettura informazioni sullo stato... Fatto
Il pacchetto python-webkit non ha versioni disponibili, ma è nominato da un altro
pacchetto. Questo potrebbe indicare che il pacchetto è mancante, obsoleto
oppure è disponibile solo all'interno di un'altra sorgente

E: Il pacchetto "python-webkit" non ha candidati da installare

(sorry for the italian)

thx,
Luca

[W3af-users] JavaScript Crawling: Beta testers wanted!

[Andres R. <and...@gm...>]

List,

    Its been a long time, and the list is very inactive, but if you've
been paying attention to the GitHub commit logs [0] you'll notice that
the project is very much alive and improving every day!

    At this point I'm looking for beta-testers for the initial
implementation of our JavaScript crawler. The crawler is based on
headless Chrome and can (at least for now) load a URL, click on all
page elements, and capture HTTP requests generated by Chrome using an
HTTP proxy.

    If you have a few minutes to spare please download the latest from
the `feature/js` branch:

git clone https://github.com/andresriancho/w3af.git
cd w3af
git checkout feature/js
virtualenv venv
. venv/bin/activate
./w3af_console

    That will prompt you to install all dependencies, please do so and
then follow the instructions in the chrome/README.md [1]. Make sure to
change the target in the scan script!

    The goal is to find issues with this new and beta feature. You'll
most likely get crashes, exceptions, scans that take a lot of time,
etc. Please report all those to w3af's issue tracker [2] to get them
fixed.

    Thanks!

[0] https://github.com/andresriancho/w3af/commits/develop
[1] https://github.com/andresriancho/w3af/tree/feature/js/w3af/core/controllers/chrome
[2] https://github.com/andresriancho/w3af/issues/new

Regards,
-- 
Andrés Riancho
Project Leader at w3af - http://w3af.org/
Web Application Attack and Audit Framework
Twitter: @w3af
GPG: 0x93C344F3

Re: [W3af-users] REST API authenticated scan help

[Andres R. <and...@gm...>]

Snehil,

Answers and comments inline,

On Fri, Sep 14, 2018 at 10:03 AM <sne...@ei...> wrote:
>
> Hello,
>
> Recently, I started exploring REST API
> of w3af and stumbled upon few things which I couldn't understand and
> thought of seeking your advice.
>
>   From the documentation it's understood that in order to initiate a scan
> following is the format :
>
> {
>       "target_urls": ["http://127.0.0.1:8000/audit/sql_injection/"],
>       "scan_profile":
> "[grep.strange_headers]\n\n[crawl.web_spider]\nonly_forward =
> False\nfollow_regex = .*\nignore_regex = \n\n"
> }
>
> w3af features different profiles which are located under
> https://github.com/andresriancho/w3af/tree/master/profiles
>
> Lets say, if I want to use OWASP TOP 10 profile for an authenticated
> scan using REST API /scan endpoint, what should be the format in the
> profile for form based authentication. I have checked the useful auth
> plugin but doesn't understand how to use these plugin inside a profile.

Something you could so is to run the w3af_gui, create your
configuration there, and then save the profile to a file. After saving
you can use it with the w3af REST API.

> for example: In OWASP TOP 10 profile, I can see under http settings
> options are there for basic authentication
> [http-settings]
> proxy_port = 8080
> url_parameter =
> never_404 =
> headers_file =
> proxy_address =
> basic_auth_domain =
> always_404 =
> max_http_retries = 2
> ntlm_auth_user =
> ntlm_auth_passwd =
> ignore_session_cookies = False
> timeout = 0
> user_agent = w3af.org
> basic_auth_user =
> basic_auth_passwd =
>
> My question is, how do I use form based credential/options in this
> profile ?
>
> I would be really grateful , if someone can answer this question for
> me with the help of an example or required format to perform such type
> of authenticated scan via REST API endpoint.
>
>
>
> Please provide an example format so that I can understand it clearly.
>
> Regards
> Snehil Khare
>
>
> _______________________________________________
> W3af-users mailing list
> W3a...@li...
> https://lists.sourceforge.net/lists/listinfo/w3af-users



-- 
Andrés Riancho
Project Leader at w3af - http://w3af.org/
Web Application Attack and Audit Framework
Twitter: @w3af
GPG: 0x93C344F3

[W3af-users] REST API authenticated scan help

[<sne...@ei...>]

Hello,

Recently, I started exploring REST API
of w3af and stumbled upon few things which I couldn't understand and
thought of seeking your advice.

  From the documentation it's understood that in order to initiate a scan
following is the format :

{
      "target_urls": ["http://127.0.0.1:8000/audit/sql_injection/"],
      "scan_profile":
"[grep.strange_headers]\n\n[crawl.web_spider]\nonly_forward =
False\nfollow_regex = .*\nignore_regex = \n\n"
}

w3af features different profiles which are located under
https://github.com/andresriancho/w3af/tree/master/profiles

Lets say, if I want to use OWASP TOP 10 profile for an authenticated
scan using REST API /scan endpoint, what should be the format in the
profile for form based authentication. I have checked the useful auth
plugin but doesn't understand how to use these plugin inside a profile.

for example: In OWASP TOP 10 profile, I can see under http settings
options are there for basic authentication
[http-settings]
proxy_port = 8080
url_parameter =
never_404 =
headers_file =
proxy_address =
basic_auth_domain =
always_404 =
max_http_retries = 2
ntlm_auth_user =
ntlm_auth_passwd =
ignore_session_cookies = False
timeout = 0
user_agent = w3af.org
basic_auth_user =
basic_auth_passwd =

My question is, how do I use form based credential/options in this
profile ?

I would be really grateful , if someone can answer this question for
me with the help of an example or required format to perform such type
of authenticated scan via REST API endpoint.



Please provide an example format so that I can understand it clearly.

Regards
Snehil Khare

Re: [W3af-users] Install issues

[Andres R. <and...@gm...>]

Oh, that is a bug. Sorry!

Fixed it here:
    https://github.com/andresriancho/w3af/commit/3012a3f94fa8dfa9136a0292491c90766dae132e

Also I merged develop into master, so everyone will get this fix.

Thanks,
On Tue, Aug 21, 2018 at 10:45 AM Rafael Barbosa da Silva
<raf...@gm...> wrote:
>
> Hi,
>
> I'm trying to make w3af work on a VM on DigitalOcean, with Ubuntu 16.04.
>
> After following the steps on docs, I'm facing this when execute ./w3af_console
>
> Traceback (most recent call last):
>   File "./w3af_console", line 13, in <module>
>     dependency_check()
>   File "/home/w3af/w3af/w3af/core/controllers/dependency_check/dependency_check.py", line 178, in dependency_check
>     external_commands = get_missing_external_commands(platform)
>   File "/home/w3af/w3af/w3af/core/controllers/dependency_check/dependency_check.py", line 99, in get_missing_external_commands
>     return platform.get_missing_external_commands()
>   File "/home/w3af/w3af/w3af/core/controllers/dependency_check/platforms/base_platform.py", line 54, in get_missing_external_commands
>     instructions.extend(handler.__func__())
>   File "/home/w3af/w3af/w3af/core/controllers/dependency_check/platforms/base_platform.py", line 60, in retirejs_handler
>     if retirejs_is_installed():
>   File "/home/w3af/w3af/w3af/core/controllers/dependency_check/external/retirejs.py", line 37, in retirejs_is_installed
>     version = subprocess.check_output('%s --version' % path_to_retire, shell=True)
>   File "/usr/lib/python2.7/subprocess.py", line 574, in check_output
>     raise CalledProcessError(retcode, cmd, output=output)
> subprocess.CalledProcessError: Command '/usr/local/bin/retire --version' returned non-zero exit status 127
>
> Can you give a hand?
>
> I already got it working from apt-get install w3af, but want to use the newest version, building from source.
>
>
> Thanks.
> Rafael
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot_______________________________________________
> W3af-users mailing list
> W3a...@li...
> https://lists.sourceforge.net/lists/listinfo/w3af-users



-- 
Andrés Riancho
Project Leader at w3af - http://w3af.org/
Web Application Attack and Audit Framework
Twitter: @w3af
GPG: 0x93C344F3

[W3af-users] Install issues

[Rafael B. da S. <raf...@gm...>]

Hi,

I'm trying to make w3af work on a VM on DigitalOcean, with Ubuntu 16.04.

After following the steps on docs, I'm facing this when execute
./w3af_console
















*Traceback (most recent call last):  File "./w3af_console", line 13, in
<module>    dependency_check()  File
"/home/w3af/w3af/w3af/core/controllers/dependency_check/dependency_check.py",
line 178, in dependency_check    external_commands =
get_missing_external_commands(platform)  File
"/home/w3af/w3af/w3af/core/controllers/dependency_check/dependency_check.py",
line 99, in get_missing_external_commands    return
platform.get_missing_external_commands()  File
"/home/w3af/w3af/w3af/core/controllers/dependency_check/platforms/base_platform.py",
line 54, in get_missing_external_commands
instructions.extend(handler.__func__())  File
"/home/w3af/w3af/w3af/core/controllers/dependency_check/platforms/base_platform.py",
line 60, in retirejs_handler    if retirejs_is_installed():  File
"/home/w3af/w3af/w3af/core/controllers/dependency_check/external/retirejs.py",
line 37, in retirejs_is_installed    version = subprocess.check_output('%s
--version' % path_to_retire, shell=True)  File
"/usr/lib/python2.7/subprocess.py", line 574, in check_output    raise
CalledProcessError(retcode, cmd,
output=output)subprocess.CalledProcessError: Command '/usr/local/bin/retire
--version' returned non-zero exit status 127*

Can you give a hand?

I already got it working from apt-get install w3af, but want to use the
newest version, building from source.


Thanks.
Rafael

[W3af-users] Holm Security is sponsoring w3af!

[Andres R. <and...@gm...>]

List,

    Prepare yourself for great news: Holm Security , an information
security solutions provider based in Sweden, is sponsoring the w3af
project!

    The interesting news and what is coming can be found at
http://w3af.org/blog , but just in case you were wondering… here are
some FAQs:

#0 How is Holm Security sponsoring w3af?
Holm Security pays me as a Python developer. I usually work 20 to 40
hours a month for Holm Security.

#1 Why does Holm Security need your help?
Holm Security scans thousands of customer sites each day using w3af.
They identify a lot of bugs, performance issues, false positives and
receive feature requests from customers. Those issues need to be fixed
in order to provide value to their customers.

I help them fix the issues and code the new features. Holm Security
has a development team, but for some tasks it is better to outsource
it.

It could have been any Python developer, but it was more convenient to
hire the guy that wrote 90% of the w3af code ;-)


#2 How will this change the w3af project?
Holm Security's sponsorship will increase development speed and make
w3af much better. w3af project will remain open source, no change in
license, no change in how you can use it.

Holm Security and I want the same thing: make the Internet a safer
place providing free access to great open source tools.

#3 All the code you write for Holm Security makes it to the public
GitHub repository?
Yes, that is part of our agreement.

#4 How can my company sponsor w3af?
Contact me at and...@gm...

Regards,
-- 
Andrés Riancho
Project Leader at w3af - http://w3af.org/
Web Application Attack and Audit Framework
Twitter: @w3af
GPG: 0x93C344F3

Re: [W3af-users] Can't find files that contain the vulnerabilities' description for report generation

[Andres R. <and...@gm...>]

All pending tasks are done!

The vulndb can now be translated and python-sdk + w3af will allow the
user to set the language parameter in the misc settings. Setting that
parameter to, lets say, PT, will make w3af write all vulnerability
descriptions in Portuguese.

On Thu, May 3, 2018 at 9:17 PM, Amanda <ama...@un...> wrote:
> Hi Andres,
>
> I think it's a great idea to translate it through the python-sdk. I'll
> definetely contribute with the translation when the structure gets ready.
>
> Please let me know when it starts working 100%. And thanks again for
> your work with W3af, it's fantastic!
>
> Regards,
>
> Amanda
>
>
> Em 03-05-2018 13:07, Andres Riancho escreveu:
>> Amanda,
>>
>>     Sorry for the very late response, but I was unable to get to this sooner.
>>
>>     The vulndb now supports translations, which are documented here:
>>
>>         https://github.com/vulndb/data/wiki/Translations
>>
>>     The python-sdk [0] for vulndb (code that reads the DB in python)
>> was modified to be able to support translations. There are two things
>> missing to get this to work 100%:
>>         * Minor architecture decision in python-sdk to determine which
>> is the best way for the developer to specify the language to use
>>         * Minor changes to w3af to let the user choose which language
>> to use in the vulndb, and then use it.
>>
>>     And of course, the translations to different languages in vulndb/data :-)
>>
>>     If you're still interested, I can take a look at Crowdin. It
>> should be possible to translate everything following the Translations
>> wiki page, but I understand that it requires some technical knowledge
>> (git, fork, pull request).
>>
>> [0] https://github.com/vulndb/python-sdk
>>
>> Regards,
>>
>> On Fri, Mar 16, 2018 at 2:00 PM, Amanda <ama...@un...> wrote:
>>> Hello!
>>>
>>> Thanks for answering, that was exactly what I was looking for!
>>>
>>> The only tool I know for translation of this kind of software is
>>> https://crowdin.com/. I'm involved in the ZAP Proxy translation project
>>> (https://crowdin.com/project/owasp-zap/pt-br#), and I understood that,
>>> when a translation is made, it automatically updates the source code (I
>>> don't know if that's the standard procedure or
>>> if they had to configure that manually). It seems like a nice tool to
>>> translate softwares and might work for W3af.
>>>
>>> About the issue [2], I'm not sure I understood the problem correctly.
>>> Could you explain it?
>>>
>>> Thanks for the help, and for this amazing software!
>>>
>>> Amanda
>>>
>>>
>>> Em 16/03/2018 10:44, Andres Riancho escreveu:
>>>> Amanda,
>>>>
>>>>     Thanks for your email and sorry for the late response.
>>>>
>>>>     The vulnerability database data is in this repository [0] and
>>>> there have been some efforts to translate it to other languages [1][2]
>>>> but sadly I've been unable to deliver the fix for [2] which is a
>>>> blocker for translations.
>>>>
>>>>     I'm completely new to the translation space, do you know about any
>>>> tools we can use to help with the translations? If I complete [2], how
>>>> would you provide the translations? A pull request?
>>>>
>>>> [0] https://github.com/vulndb/data
>>>> [1] https://github.com/vulndb/data/issues/26
>>>> [2] https://github.com/vulndb/data/issues/30
>>>>
>>>> On Mon, Mar 5, 2018 at 4:48 PM, Amanda <ama...@sj...> wrote:
>>>>> Hello!
>>>>>
>>>>> I would like to translate the vulnerabilities' descriptions (name,
>>>>> description, long description) in the XML reports to Brazilian Portuguese.
>>>>>
>>>>> However, I couldn't find the files that contain this descriptions and
>>>>> that are used to generate the XML reports. Can someone help me?
>>>>>
>>>>> Thank you in advance.
>>>>>
>>>>>
>>>>> Amanda
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>> Check out the vibrant tech community on one of the world's most
>>>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>>>> _______________________________________________
>>>>> W3af-users mailing list
>>>>> W3a...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/w3af-users
>>>>
>>> --
>>> Amanda Barbosa Sobrinho
>>> Bacharelado em Ciência da Computação
>>> ACME! CyberSecurity Research Labs
>>> UNESP - São José do Rio Preto, SP
>>>
>>>
>>
>>
>



-- 
Andrés Riancho
Project Leader at w3af - http://w3af.org/
Web Application Attack and Audit Framework
Twitter: @w3af
GPG: 0x93C344F3

Re: [W3af-users] Can't find files that contain the vulnerabilities' description for report generation

[Andres R. <and...@gm...>]

Amanda,

    Sorry for the very late response, but I was unable to get to this sooner.

    The vulndb now supports translations, which are documented here:

        https://github.com/vulndb/data/wiki/Translations

    The python-sdk [0] for vulndb (code that reads the DB in python)
was modified to be able to support translations. There are two things
missing to get this to work 100%:
        * Minor architecture decision in python-sdk to determine which
is the best way for the developer to specify the language to use
        * Minor changes to w3af to let the user choose which language
to use in the vulndb, and then use it.

    And of course, the translations to different languages in vulndb/data :-)

    If you're still interested, I can take a look at Crowdin. It
should be possible to translate everything following the Translations
wiki page, but I understand that it requires some technical knowledge
(git, fork, pull request).

[0] https://github.com/vulndb/python-sdk

Regards,

On Fri, Mar 16, 2018 at 2:00 PM, Amanda <ama...@un...> wrote:
> Hello!
>
> Thanks for answering, that was exactly what I was looking for!
>
> The only tool I know for translation of this kind of software is
> https://crowdin.com/. I'm involved in the ZAP Proxy translation project
> (https://crowdin.com/project/owasp-zap/pt-br#), and I understood that,
> when a translation is made, it automatically updates the source code (I
> don't know if that's the standard procedure or
> if they had to configure that manually). It seems like a nice tool to
> translate softwares and might work for W3af.
>
> About the issue [2], I'm not sure I understood the problem correctly.
> Could you explain it?
>
> Thanks for the help, and for this amazing software!
>
> Amanda
>
>
> Em 16/03/2018 10:44, Andres Riancho escreveu:
>> Amanda,
>>
>>     Thanks for your email and sorry for the late response.
>>
>>     The vulnerability database data is in this repository [0] and
>> there have been some efforts to translate it to other languages [1][2]
>> but sadly I've been unable to deliver the fix for [2] which is a
>> blocker for translations.
>>
>>     I'm completely new to the translation space, do you know about any
>> tools we can use to help with the translations? If I complete [2], how
>> would you provide the translations? A pull request?
>>
>> [0] https://github.com/vulndb/data
>> [1] https://github.com/vulndb/data/issues/26
>> [2] https://github.com/vulndb/data/issues/30
>>
>> On Mon, Mar 5, 2018 at 4:48 PM, Amanda <ama...@sj...> wrote:
>>> Hello!
>>>
>>> I would like to translate the vulnerabilities' descriptions (name,
>>> description, long description) in the XML reports to Brazilian Portuguese.
>>>
>>> However, I couldn't find the files that contain this descriptions and
>>> that are used to generate the XML reports. Can someone help me?
>>>
>>> Thank you in advance.
>>>
>>>
>>> Amanda
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Check out the vibrant tech community on one of the world's most
>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>> _______________________________________________
>>> W3af-users mailing list
>>> W3a...@li...
>>> https://lists.sourceforge.net/lists/listinfo/w3af-users
>>
>>
>
> --
> Amanda Barbosa Sobrinho
> Bacharelado em Ciência da Computação
> ACME! CyberSecurity Research Labs
> UNESP - São José do Rio Preto, SP
>
>



-- 
Andrés Riancho
Project Leader at w3af - http://w3af.org/
Web Application Attack and Audit Framework
Twitter: @w3af
GPG: 0x93C344F3

Re: [W3af-users] Authenticated spider issues and questions

[Andres R. <and...@gm...>]

Kukulkan,

    The authentication plugins do not send the login / check URLs to
the core. So any URL you put in the configuration, or is a result of
requesting those URLs will not make it to other plugins / the crawler.

    That was the original design and is working as expected. Might not
be ideal for cases (yours?)... we'll see!

    When the user configures authentication plugins, those are run at
the beginning of the scan [0][1], before sending "almost any other
request" and before the crawling plugins. This means that you could
configure w3af like this:
        * auth plugin logins the scanner
        * the scanner will re-use cookies just like any browser (like
you mention above)
        * crawl plugin will re-use cookies to follow the links you set
in the target. Remember that you can set the target to a comma
separated list of URLs, that might help.

    Those steps will be run in that order, so the crawler should have
cookies when reaching the target.

    The GUI is NOT maintained and I don't recommend using it. Use the
console or REST API.

    w3af doesn't support javascript, so it won't be able to extract
"phpAccountSummary.php" from:

```
<script type="text/javascript">
window.setTimeout("window.location.href = 'phpAccountSummary.php';", 0);
</script>
```

    If you want me to help a little bit more, please do send me scan
logs with debugging information and HTTP requests (both files are
generated by text_file plugin)

[0] https://github.com/andresriancho/w3af/blob/39004228300e1eb38ae0cdb3946725e7a3adb8c8/w3af/core/controllers/core_helpers/strategy.py#L649
[1] https://github.com/andresriancho/w3af/blob/39004228300e1eb38ae0cdb3946725e7a3adb8c8/w3af/core/controllers/core_helpers/strategy.py#L111-L112



On Thu, Apr 26, 2018 at 7:31 AM, Volker Schmid <vol...@re...> wrote:
> Hello Andres,
>
> I created a cookie file and tried again. Now it seems to use the cookie, but
> spider is still not successful. I can see that it spidered several pages but
> it does not follow the links inside. Looks like it does not even try to
> spider the page that was found in login page result like this:
>
> <script type="text/javascript">
> window.setTimeout("window.location.href = 'phpAccountSummary.php';", 0);
> </script>
>
> It just inspects the few pages linked on the start and login page. But it
> does not spider the pages behind. I thought it would also use the page I set
> for login verification (phpAccountSummary.php). It opens it, even successful
> after login, but it does not spider the links inside there.
>
> Again, if I set the spider target directly to
> https://vsprovider2.de.mysystem.com/phpAccountSummary.php, the
> "Results"->"URLs" stays completely empty.
>
> I also have to restart w3af GUI each time I scanned because any further
> action leads to crashes, strange GUI behaviour (missing values in scan
> config fields) or missing logs and URL's in "Results" view occasionally. The
> GUI seems very buggy to me.
>  Is there some other, more stable version available? And is there a more
> sophisticated authentication/spider PlugIn available?
>
> Thanks,
>
> Kukulkan



-- 
Andrés Riancho
Project Leader at w3af - http://w3af.org/
Web Application Attack and Audit Framework
Twitter: @w3af
GPG: 0x93C344F3

Re: [W3af-users] Authenticated spider issues and questions

[Volker S. <vol...@re...>]

Hello Andres,

I created a cookie file and tried again. Now it seems to use the cookie, but spider is still not successful. I can see that it spidered several pages but it does not follow the links inside. Looks like it does not even try to spider the page that was found in login page result like this:

<script type="text/javascript">
window.setTimeout("window.location.href = 'phpAccountSummary.php';", 0);
</script>

It just inspects the few pages linked on the start and login page. But it does not spider the pages behind. I thought it would also use the page I set for login verification (phpAccountSummary.php). It opens it, even successful after login, but it does not spider the links inside there.

Again, if I set the spider target directly to https://vsprovider2.de.mysystem.com/phpAccountSummary.php, the "Results"->"URLs" stays completely empty.

I also have to restart w3af GUI each time I scanned because any further action leads to crashes, strange GUI behaviour (missing values in scan config fields) or missing logs and URL's in "Results" view occasionally. The GUI seems very buggy to me.
  
Is there some other, more stable version available? And is there a more sophisticated authentication/spider PlugIn available?

Thanks,

Kukulkan

Re: [W3af-users] Authenticated spider issues and questions

[Volker S. <vol...@re...>]

Hi,

thanks for the quick response.

> Maybe the web_spider is following the logout link, which is
> invalidating the session?
> 
> You should ignore logout urls when doing auth scans

I already did by using ignore_regex: "(phpLogout\.php|phpNewPassword\.php)"

By this I hope he does not call any URL with these two scripts inside. Both would be bad for the scan.

> Yeah, that could be because of the javascript redirect. Maybe try to
> set phpAccontSummary.php in the w3af target configuration?

I changed the target to be phpAccountSummary.php. But now it logs in successfully and unsuccessfully a few times. It does not spider any other URL's now. It does not find a single URL but in "Results", and if I look for this request, I can see that the successful requests returned several links but they are not followed at all.

> No, lets try with the things I recommended above, if that doesn't work
> we'll try giving w3af a cookie via config/http/cookies

Hm. The cookie with the Session-ID is returned by the first call to phpLogin.php. I assumed that w3af is using the cookies like a webbrowser does (eg after receiving one, always send the content with every further request). But due to the requests in the "Results", it does not send the cookie it received before with the next requests. It simply does not respect the session cookie. Interestingly, another cookie is always used (but there the content is static and no session ID).

Any other idea?

Re: [W3af-users] Authenticated spider issues and questions

[Andres R. <and...@gm...>]

Kukulkan,

    Answers inline,

On Wed, Apr 25, 2018 at 4:33 AM, Volker Schmid <vol...@re...> wrote:
> Hi,
>
> I'm new to w3af and start to get deeper into authentication. I use only two
> PlugIns: crawl->web_spider and auth->detailed. The current site is using a
> form in phpLogin.php. This is doing a JS redirect so I use
> phpAccontSummary.php to verify if user was logged in successfully (searching
> there for "Log out").
>
> This is the config for auth-detailed:
> [auth.detailed]
> username = pen...@my...
> password = EGjv4gmj
> username_field = txtUsername
> password_field = txtPassword
> auth_url = https://vsprovider2.de.mysystem.com/phpLogin.php?action=login
> check_url = https://vsprovider2.de.mysystem.com/phpAccountSummary.php
> check_string = Log out
> data_format = %u=%U&%p=%P
> follow_redirects = False
> method = POST
> url_encode_params = True
>
>
> Due to the website logs, login for user "Pentest Pentest" (ID 3) was
> successful several times:
>
> 2018-04-25 09:12:25     USER_LOGIN_SUCCESS      Pentest Pentest (3)
> 2018-04-25 09:12:20     USER_LOGIN_SUCCESS      Pentest Pentest (3)
> 2018-04-25 09:12:15     USER_LOGIN_SUCCESS      Pentest Pentest (3)
>
>
> In the GUI log I get this:
>
> [Mi 25 Apr 2018 09:12:25 CEST] Can't login into web application as
> pen...@my.../EGjv4gmj
.
>
> In the console output (using GUI) of w3af I can find such entries:
>
> GET https://vsprovider2.de.mysystem.com/phpAccountSummary.php returned HTTP
> code "200" (id=19,from_cache=0,grep=0,rtt=0.01,did=None)
> User "pen...@my..." is NOT logged into the application
> POST https://vsprovider2.de.mysystem.com/phpLogin.php?action=login with
> data: "txtUsername=pen...@my...&txtPassword=EGjv4gmj" returned HTTP
> code "200" (id=20,from_cache=0,grep=1,rtt=0.06,did=None)
> GET https://vsprovider2.de.mysystem.com/phpAccountSummary.php returned HTTP
> code "200" (id=21,from_cache=0,grep=0,rtt=0.03,did=None)
> User "pen...@my..." is currently logged into the application
> Login success for pen...@my.../EGjv4gmj
> detailed._login() took 0.11s to run
>
> (...many other spider entries...)
>
> GET https://vsprovider2.de.mysystem.com/phpAccountSummary.php returned HTTP
> code "200" (id=74,from_cache=0,grep=0,rtt=0.04,did=None)
> User "pen...@my..." is NOT logged into the application

Maybe the web_spider is following the logout link, which is
invalidating the session?

You should ignore logout urls when doing auth scans

> (...a few other spider entries...)
>
> ET https://vsprovider2.de.mysystem.com/phpAccountSummary.php returned HTTP
> code "200" (id=78,from_cache=0,grep=0,rtt=0.04,did=None)
> User "pen...@my..." is currently logged into the application
> Login success for pen...@my.../EGjv4gmj
> detailed._login() took 0.18s to run
>
> (...many other spider entries...)
>
> GET https://vsprovider2.de.mysystem.com/phpAccountSummary.php returned HTTP
> code "200" (id=111,from_cache=0,grep=0,rtt=0.01,did=None)
> User "pen...@my..." is NOT logged into the application
> web_spider.discover(https://vsprovider2.de.mysystem.com/phpCreateRegifyLocal.php)
> web_spider is testing
> "https://vsprovider2.de.mysystem.com/phpCreateRegifyLocal.php"
> [web_spider] Crawling
> "https://vsprovider2.de.mysystem.com/phpCreateRegifyLocal.php"
> GET https://vsprovider2.de.mysystem.com/phpCreateRegifyLocal.php returned
> HTTP code "302" (id=112,from_cache=0,grep=1,rtt=0.01,did=None)
> web_spider.discover(uri="https://vsprovider2.de.mysystem.com/phpCreateRegifyLocal.php")
> took 0.02s to run
> POST https://vsprovider2.de.mysystem.com/phpLogin.php?action=login with
> data: "txtUsername=pen...@my...&txtPassword=EGjv4gmj" returned HTTP
> code "200" (id=113,from_cache=0,grep=1,rtt=0.07,did=None)
> GET https://vsprovider2.de.mysystem.com/phpAccountSummary.php returned HTTP
> code "200" (id=114,from_cache=0,grep=0,rtt=0.01,did=None)
> User "pen...@my..." is NOT logged into the application
> Can't login into web application as pen...@my.../EGjv4gmj
>
> So this are very mixed results (sometimes success sometimes not) and I do
> not know why it sometimes reports successful login and sometimes it does
> not?
>
> Due to the request navigator and the results to phpLogin.php there, login
> was always successful if w3af sent the correct login data by POST. I can see
> that phpAccontSummary.php delivered positive results sometimes.
>
> Also, even if it was successful, it seems it does not spider the links found
> in phpAccontSummary.php. All the new links inside there are not listed in
> the URL's found.

Yeah, that could be because of the javascript redirect. Maybe try to
set phpAccontSummary.php in the w3af target configuration?

> I can see that w3af does not send the session cookie received during the
> first phpLogin.php all the time. It seems to forget sometimes. If not set,
> the webpage creates a new sessionid and returns it. So the logged in session
> is somehow lost. Why is it not always sending the session cookie? In
> Configuration->HTTP Config->Cookies, the ignore option is NOT set and the
> cookie_jar_file is empty. Anything to do here?

No, lets try with the things I recommended above, if that doesn't work
we'll try giving w3af a cookie via config/http/cookies

> I'm a little bit lost now because the things I see seem not logical to me at
> all :(
>
> Best
>
> Kukulkan
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> W3af-users mailing list
> W3a...@li...
> https://lists.sourceforge.net/lists/listinfo/w3af-users



-- 
Andrés Riancho
Project Leader at w3af - http://w3af.org/
Web Application Attack and Audit Framework
Twitter: @w3af
GPG: 0x93C344F3