Re: [W3af-develop] W3af Ubuntu 13.10

[Taras <ox...@ox...>]

Andres,

07.03.2014 19:15, Andres Riancho пишет:
> Don't want to re-open this, but just a FYI:
>
> https://github.com/axiak/pybloomfiltermmap/issues/29
>
> In other words, 0.3.11 of pybloomfiltermmap installation works
> flawlessly, while 0.3.12 fails with gcc compilation errors :( If we
> would have had >= this would have been an issue for us too.

If it would packaged by e.g. Debian maintainers it would be tested.
But yes, topic is closed :-)


> On Tue, Feb 18, 2014 at 2:15 PM, Taras <ox...@ox...> wrote:
>> Andres,
>>
>> Ok, I've got your opinion. Let's close this discussion.
>>
>> 17.02.2014 00:04, Andres Riancho пишет:
>>
>>> Taras,
>>>
>>> On Sun, Feb 16, 2014 at 4:28 PM, Taras <ox...@ox...> wrote:
>>>>
>>>> Andres,
>>>>
>>>> I think it is my last attempt to change your opinion :)
>>>>
>>>>   From the list of software you have provided I have found only flask,
>>>> scrapy
>>>> and tastypie in Ubuntu repo. Results of "apt-cache show" output are below
>>>> inline.
>>>> The problem is w3af built-in dependency checker duplicates
>>>> OS (e.g. Debian/Ubuntu) packaging system. They can conflict in
>>>> some cases.
>>>>
>>>> For example, I want to make package of w3af for Ubuntu 13.10.
>>>> There is package python-xml version 3.2.0 in repository. At the same time
>>>> w3af requires lxml version exactly 2.3.2. How can I make package of w3af?
>>>> Should I add "sudo pip install" into preinstall script?
>>>
>>>
>>> Most likely not, that doesn't sound well. I don't know the right
>>> answer because I'm not packaging expert.
>>>
>>> The package maintainer can always apply a patch on top of the original
>>> software to remove the dependency check completely (I think Luciano
>>> did something like this [0]) is he believes it is the best thing to
>>> do. Then he's taking the responsibility of that change. My
>>> responsibility is to tell you that with these specific package
>>> versions it works; then people do whatever they want with it.
>>>
>>> [0] http://packages.ubuntu.com/precise/w3af-console   -   search for
>>> "diff"
>>>
>>>> Have you got any feedback from w3af package maintainers for Debian/Ubuntu
>>>> and other distributions after you had add strict dependencies?
>>>
>>>
>>> There are no active package maintainers for w3af. They even don't
>>> care, or don't want to maintain this software; so no, no package
>>> maintainer told me anything about the "==". As I said above, they can
>>> apply a diff to the software before packaging it, as done by Luciano a
>>> while ago (not only for the dependency).
>>>
>>>> Is it
>>>> important for you that w3af can be installed via simple command  "apt-get
>>>> install w3af" or through Ubuntu Software Center with single mouse click?
>>>
>>>
>>> Yes, and not.
>>>
>>> Some users would find it awesome to be able to install it from the
>>> repo; but this has proven to be (at least for w3af) a failed path. I'm
>>> not going to maintain a package for each distribution, because I don't
>>> care enough as a user myself.
>>>
>>> Packagers who have come to the project have either failed to release
>>> their initial package or released it and then moved their free time to
>>> something else. In this process, they left very old versions of w3af
>>> in the repositories of all linux distributions; which don't even make
>>> sense for users.
>>>
>>> If users can install w3af with:
>>>
>>> git clone ...
>>> cd w3af
>>> ./w3af_console # Yields error with all dependencies to install
>>> /tmp/install_w3af_dependencies.sh
>>>
>>> Then I'm happy.
>>>
>>>> If it is important for you then I recommend to add maintainers into this
>>>> discussion and ask if it is easy for them to make package of w3af with
>>>> such
>>>> requirements.
>>>
>>>
>>> My opinion is that they don't care about the w3af package.
>>>
>>>> If it is not so important and "git clone + pip install" is preferable way
>>>> of
>>>> installation then thread can be closed.
>>>
>>>
>>> In the past I've thought that having w3af in the linux distribution
>>> repos was THE BEST THING, now... not so much, because:
>>>    * Software packages are difficult to maintain
>>>    * Each time a new dependency is added the maintainer needs to create
>>> a new package for that (python-foo) and then maintain that one also
>>>    * The whole process takes time, so from the minute I put something in
>>> the repo to the time the new package is there it can be months; and
>>> "hackers" love to use the latest and they will come to the repo
>>> anyways
>>>
>>>>
>>>>
>>>>>>> Not 100% a workaround, this is also a best practice!
>>>>>>>
>>>>>>> https://devcenter.heroku.com/articles/python-pip#the-basics
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Could you please show at least one example of well-know software with
>>>>>> such
>>>>>> requirements?
>>>>>
>>>>>
>>>>>
>>>>> I went through this list of the Top10 Python projects by github (not
>>>>> sure how they choose that) and found many that either had no
>>>>> dependencies or were not in a format in which we could compare them
>>>>> with what we were talking about. Then found the following:
>>>>>
>>>>>     * Strict dependencies used for this part of the project:
>>>>> https://github.com/torchbox/wagtail/blob/master/requirements-dev.txt
>>>>>     * Gt used for the user installable part:
>>>>> https://github.com/torchbox/wagtail/blob/master/setup.py
>>>>>
>>>>>     * These guys install whatever is available on pypi:
>>>>> https://github.com/jmcarp/robobrowser/blob/master/requirements.txt
>>>>>
>>>>>     * Flask installs Gt:
>>>>> https://github.com/mitsuhiko/flask/blob/master/setup.py
>>>>
>>>>
>>>> Depends: python-itsdangerous, python (>= 2.7), python-jinja2 (>= 2.4),
>>>> python (<< 2.8), python-werkzeug (>= 0.8)
>>>> Recommends: python-pkg-resources, python-blinker
>>>>
>>>>
>>>>>     * A mix between Gt and "whatever" is used here:
>>>>> https://github.com/Eugeny/ajenti/blob/dev/requirements.txt
>>>>>
>>>>>     * Scrapy uses a mix of GT and "whatever":
>>>>> https://github.com/scrapy/scrapy/blob/master/requirements.txt
>>>>
>>>>
>>>>
>>>> Depends: python2.7, python (>= 2.7.1-0ubuntu2), python (<< 2.8),
>>>> python-twisted-core, python-twisted-web, python-twisted-conch,
>>>> python-twisted-mail, python-libxml2, python-boto, python-w3lib
>>>> Recommends: python-lxml, python-guppy, python-django, ipython,
>>>> python-pygments, python-imaging, python-mysqldb
>>>>
>>>>
>>>>>     * Django-tastypie uses the most complex of them all, which is rather
>>>>> interesting and makes me wonder why they didn't use "==" instead:
>>>>> https://github.com/toastdriven/django-tastypie/blob/master/setup.py .
>>>>> This is what I mean: 'dateutil(>=1.5, !=2.0)'
>>>>>
>>>> Replaces: python-django-tastypie (<= 0.9.9-2)
>>>> Depends: python (>= 2.7.1-0ubuntu2), python (<< 2.8), python-mimeparse
>>>> (>=
>>>> 0.1.3), python-dateutil (>= 1.5), python-django (>= 1.2)
>>>> Suggests: python-yaml, python-lxml
>>>>
>>>
>>> I get your point, >= seems to be the preferred way of doing it in the
>>> debian repos. If a packager wants, he can do that with w3af and apply
>>> a patch to disable dependency check for w3af in the packaging process.
>>> That way he's happy, we don't need to code anything and are also
>>> happy.
>>>
>>>>> The first one is an example of "==", the rest were just to show that
>>>>> now everyone agrees with me on what should be put on the
>>>>> requirements.txt file (or the setup.py, which acts like the same many
>>>>> times).
>>>>>
>>>>> Here are some other links where it says that "==" is a best practice:
>>>>>     *
>>>>> https://lincolnloop.com/django-best-practices/deployment/bootstrap.html
>>>>> (Ctrl+f "Pin your dependencies")
>>>>>     *
>>>>>
>>>>> http://docs.dotcloud.com/tutorials/python/django/#specifying-requirements
>>>>> (Ctrl+f "When you specify your requirements")
>>>>>
>>>>> And most importantly, the pip-installer user's guide:
>>>>>     *
>>>>>
>>>>> http://www.pip-installer.org/en/latest/user_guide.html#ensuring-repeatability
>>>>>
>>>>> "The requirements file was generated by pip freeze or you're sure it
>>>>> only contains requirements that specify a specific version."
>>>>>
>>>>> When we're talking about including a specific version in
>>>>> requirements.txt file or not, we're talking about repeatability. I
>>>>> want to be strict about repeatability, forcing all libraries to be
>>>>> exactly the ones I know will work because I've tested them in the CI;
>>>>> and your point is that it would be easier for users to install with
>>>>> less strict version requirements (which could lead to issues in some
>>>>> cases).
>>>>>
>>>>> Sadly, you believe in one thing and I can't seem to convince you of
>>>>> the benefits of ==, and the same applies the other way (I can't be
>>>>> convinced of the benefits of >=). Unless I hear a definitive reason on
>>>>> why == is bad, I won't change it.
>>>>>
>>>>>> By the way in w3af dev list I see fresh discussion about
>>>>>> similar problems in Mageia Linux distro
>>>>>> http://sourceforge.net/mailarchive/message.php?msg_id=31315478
>>>>>
>>>>>
>>>>>
>>>>> I think that email thread was correctly answered?
>>>>>
>>>>>>
>>>>>>>> 1. Bring back dependency check with >= condition
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Disagree with this, it will bring issues in the future, and it not a
>>>>>>> best practice.
>>>>>>>
>>>>>>>> 2. We should separate core and plugins requirements
>>>>>>>> 3. We should make possible to run w3af without installation of all
>>>>>>>> plugins
>>>>>>>> dependencies. It can be with special argument to w3af_console called
>>>>>>>> "-l
>>>>>>>> or
>>>>>>>> --lazy". This parameter will force w3af not to check plugins
>>>>>>>> dependencies
>>>>>>>> (or even switch off dependency checker all!).
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> You can disable checks for the dependencies which are used in plugins,
>>>>>>> not for the ones in the core or stuff will break in the middle of the
>>>>>>> scan.
>>>>>>
>>>>>>
>>>>>>
>>>>>> If default behavior will not be changed why you are still against
>>>>>> disabling it at all by special parameter? This parameter will be used
>>>>>> only
>>>>>> by package maintainers who specifies these dependencies in the package
>>>>>> and
>>>>>> geeks who don't want to install stuff they don't really need.
>>>>>
>>>>>
>>>>>
>>>>> Let me see if I understand, cause now I think I've read it
>>>>> differently. Lets be specific so I don't imagine things:
>>>>>     * You will add a --lazy flag to w3af_console and w3af_gui
>>>>>     * You will pass the value of --lazy to the dependency_check [0]
>>>>> function
>>>>>     * If --lazy is False (the default) things will continue as they are
>>>>> now
>>>>>     * If --lazy is True (only if the user specifies that flag) then
>>>>> instead of using strict version checking here [1] you will use ">="
>>>>>     * If --lazy is True you'll user => here [2]
>>>>>     * You'll change the console and gtk-UI in such a way that when
>>>>> enabling a plugin that requires a dependency that is not installed, it
>>>>> will tell the user what it is required
>>>>>     * You'll make sure that it is possible to run w3af with different
>>>>> versions of plugin dependencies
>>>>>     * You'll check that it is possible to run w3af even when some plugin
>>>>> dependencies are not installed
>>>>>     * This has automated testing so that in the future I'm sure things
>>>>> will continue to work as expected
>>>>>
>>>>> If that's it, I'm +1 on it!
>>>>>
>>>>> Sorry for not completely understanding your points in the previous
>>>>> email.
>>>>>
>>>>> PS: Still can't believe you'll work on this; I believe it is useless
>>>>> for 95% of the user base. Of course, if you believe it will be useful
>>>>> for you, and it is well coded / tested and doesn't disturb the
>>>>> defaults, I'll merge!
>>>>>
>>>>> [0]
>>>>>
>>>>> https://github.com/andresriancho/w3af/blob/feature/module/w3af/core/controllers/dependency_check/dependency_check.py
>>>>> [1]
>>>>>
>>>>> https://github.com/andresriancho/w3af/blob/feature/module/w3af/core/controllers/dependency_check/dependency_check.py#L68
>>>>> [2]
>>>>>
>>>>> https://github.com/andresriancho/w3af/blob/feature/module/w3af/core/controllers/dependency_check/helper_script.py
>>>>>
>>>>>> --
>>>>>> Taras
>>>>>> https://www.oxdef.info
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> Taras
>>>> https://www.oxdef.info
>>>
>>>
>>>
>>>
>>
>> --
>> Taras
>> https://www.oxdef.info
>
>
>

-- 
Taras
https://www.oxdef.info