Re: [Yokoso-devel] Updates to http-enum.nse

[Kevin J. <ke...@in...>]

On Aug 21, 2009, at 10:09 AM, Ron wrote:
> Hi all,
>

Hi All,

I am one of the lead developers for Yokoso! so I thought I should  
respond to some of these points here.

> I'm cross-posting this to nmap-dev and yokoso-devel, since it sort of
> affects both the products. I don't really know what etiquette is
> regarding cross-posting, so hopefully that isn't a no-no. :)

I don't mind it at all.  And am also cross-posting. ;-)

>
> On 08/21/2009 02:49 AM, Fyodor wrote:
>> On Thu, Aug 20, 2009 at 11:57:43AM -0500, Ron wrote:
>>>
> I agree that the script should only show 200 by default, but give an
> option to show others. That makes sense. You're going to miss hidden
> folders, by by default Apache hides certain things so that kinda makes
> sense.
>
> For the average person, just giving 200 makes sense, and for the
> non-average who understand the extra errors, they can enable a  
> script-arg.
>

This idea makes sense to me.

>> The format of the fingerprint file is a bit questionable.  Comments
>> lines starting with '#' are parsed and then printed in the script
>> output when paths given later in the file are discovered.  I realize
>> you didn't invent this format, but it is so simple that it could
>> easily be improved.  For example, each path could include the
>> description on the same line.  Or there could be a keyword  
>> introducing
>> the description on one line, followed by the paths on their own lines
>> as they are now.  Then we could use comments for notes which we don't
>> want parsed and printed by http-enum.  In deciding on the format, it
>> may be worth thinking about how it could be extended if we want to
>> include more information later (just as an idea off the top of my
>> head, we might want to later indicate what status code we look for to
>> address issues such as http-enum reporting that scanme.nmap.org has
>> "TeraStation PRO RAID 0/1/5 Network Attached Storage" just because
>> /cgi-bin/image/shikaku2.png shows forbidden).
> The format is a bit odd, and I'd talked to Kevin a bit about  
> changing it
> to have the ability to AND things together (so if a host contains  
> these
> 4 files that have common names, but are unique as a combo, it'll check
> for all of them). At the same time, maybe we can consider other format
> changes. Maybe I'll think about it and propose something, see where  
> that
> goes.
>
> Regarding including the status code, that is sort of an issue with the
> different purposes of Yokoso vs Nmap. Yokoso doesn't care what the
> status code is, but rather a binary: the user has been to the page, or
> the user hasn't. Nmap, on the other hand, cares of the page exists.  
> I'm
> sure there must be a way to get the best of both worlds, of course --
> maybe optional fields or something?

As the guy who created the "format" of the fingerprint file, I think  
its important to realize that it was just a file.  I like the idea of  
something similar to the nikto DB where we include the status code we  
expect and a description on the same line.  Comments could then be  
used for notes to end users and ignored by scripts.

>
>> Another serious issue involves inclusion of the Yokoso DB.  You say:
>>
>>> That last point is the interesting one, to me -- we use the same  
>>> file
>>> format as the Yokoso project (by Kevin Johnson and others, from  
>>> Intel
>>> Guardians).

BTW its InGuardians.  Not relevant to this discussion but important to  
me. :-)

>>> This lets us leverage their fingerprints as well (and
>>> they've given me permission to include a copy of their fingerprints
>>> file, too,
>>
>> That was nice of them, but it is important to get more clarification
>> and more explicit permission whenever we include 3rd party code or
>> data into Nmap.  I hate dealing with copyright stuff as much as the
>> next guy, but we really need to be very careful about this sort of
>> thing.  When they say we can include the DB with Nmap, what does that
>> really mean.  Remember that Nmap is open source, so people can
>> incorporate parts into other projects or fork Nmap under a different
>> name.  A strict reading of "you may include this file within Nmap"
>> would not allow such things, which would mean that part of Nmap is  
>> not
>> open source.  Also, a strict reading might mean that we can only
>> include the file and not modify it (create derivative works).  In
>> general, we can put third party code/data in with Nmap if it is given
>> to us under one of the following licenses (either via special
>> permission or because the code is already under such a license):
>>
>> o Public domain -- that means people can do whatever they want with  
>> it.
>>
>> o BSD-style (includes MIT license, Lua license, etc.) - preferably
>>   2-clause variant.  If it has the advertising clause, we need to
>>   mention it explicitly in the man page
>>   (http://nmap.org/book/man-legal.html#third-party-soft) and
>>   potentially other places.
>>
>> o Nmap license - if they're OK with us distributing it under the  
>> terms
>>   of the Nmap license (http://nmap.org/data/COPYING), that is OK too.
>>
>> So if they let us use the data under one of these licenses, inclusion
>> with Nmap is OK.  In any case, the license permission granted needs  
>> to
>> be included (described) at the top of the file.  We only need license
>> rights to the list of URL paths and descriptions, and not the rest of
>> Yokoso.
>>
>> Note that even when a data file isn't licensed appropriately for
>> distribution with Nmap, we can generally point to it in the
>> documentation (e.g. if it is a URL somewhere) so users can download
>> and use it.
> I'll leave that question for the Yokoso guys to answer.

We currently use the GPL for Yokoso! but have no problems with the  
NMAP license and would love to see it included with that in NMAP.

>
>>> One thought I had -- http-enum.nse and Yokoso sort of have different
>>> points. http-enum.nse is designed for finding common locations, like
>>> /icons, /scripts, /test, etc, and Yokoso is designed for  
>>> fingerprinting
>>> common web apps. So, for that reason, it might make sense to put  
>>> it in a
>>> different script that the user can run separately. Or maybe not. I'm
>>> happy with going either way.
>>
>> I'm not sure, but my gut reaction is that with a good file format
>> (which doesn't have to be too complex), we could probably combine the
>> Yokoso DB with the existing enum DB.  There is also a DB by HD Moore
>> (a NASL script he wrote) which I hope to request permission for us to
>> use.
>>
>> If we end up with more URLs than we want to scan by default, we could
>> look at either splitting it up into multiple scripts or introducing
>> NSE arguments to select categories of paths to try.
> You're probably right about combining the databases. My concern is  
> that
> it'll make it more difficult to update when Yokoso is updated --  
> Yokoso
> promises to be an active project, and if they're updating the
> fingerprints and we've combined the fingerprints, we might run into
> versioning problems.
>
> I think the best bet is to have multiple fingerprint files of the same
> format. Yokoso, defaults, extended, etc. Then the script can load
> whatever it sees fit.
>
> I'd definitely like to use HD's database if it's for the same purpose!

I can take responsibility to keep the yokoso fingerprints updated in  
the nmap file if you would like!

Thanks for including us, and if there are more questions feel free to  
ask!

Kevin

Kevin Johnson
Senior Security Analyst
InGuardians, Inc.
office: 202.448.8958
cell: 904.403.8024