Menu

How to 'sel' with additional variables

Help
Blake
2010-06-29
2013-04-27
  • Blake

    Blake - 2010-06-29

    My xml file looks like this:
    <run>
        <address addr="64.233.189.104" addrtype="ipv4" />
    <hostnames>
        <hostname name="google.com" type="user"/>
        <hostname name="hkg01s01-in-f104.1e100.net" type="PTR"/>
    </hostnames>
    <address addr="207.46.197.32" addrtype="ipv4" />
    <hostnames>
        <hostname name="microsoft.com" type="user"/>
    </hostnames>
    </run>

    and my desired output is:
    host|addr|PTR
    google.com|64.233.189.104|hkg01s01-in-f104.1e100.net
    microsoft.com|207.46.197.32

    What I have unsuccessfully attempted is:
    xml sel -t -m "//hostname" -v "@name" -o "|" -n test.xml
    &
    $ xml sel -t -m "//hostname" -v "@name" -o "|" -m "//address" -v "@addr" -n test.xml

    But this falls short. Very short of the mark. TIA.

     
  • George  K.

    George K. - 2010-06-30

    Hi!

    First of all I think that you organized your XML file incorrectly.
    It should be like this:

    <run>
        <address addr="64.233.189.104" addrtype="ipv4">
        <hostnames>
                <hostname name="google.com" type="user" />
                <hostname name="hkg01s01-in-f104.1e100.net" type="PTR" />
        </hostnames>
        </address>
        <address addr="207.46.197.32" addrtype="ipv4">
        <hostnames>
                <hostname name="microsoft.com" type="user" />
        </hostnames>
        </address>
    </run>
    

    And then the command line should be:

    xml sel -T -t -m "//address/hostnames" -m "hostname" -v "@name" -i "position() &lt; 2" -o "|" -v "../../@addr" -b -i "position() &lt; last()" -o "|" -b -b -n test.xml
    
     
    • mr.duck

      mr.duck - 2013-04-27
       
  • Blake

    Blake - 2010-06-30

    Hi - Thank you for the help here. It is very useful. I will try to dissect the line and fully understand what you have written. The condition statements you wrote:

     -i "position() &lt; 2"
    

    and

    -i "position() &lt; last()"
    

    remain admittedly somewhat elusive.

    What I am trying to do here is use xmlstarlet to parse Nmap xml output files. It may be that Nmap does not output xml perfectly as you suggested, however it does validate. If you could tweak your xml statement to handle output the same thing as above but apply it to an actual Nmap scan (below), that would be most appreciated.

    <?xml version="1.0" ?>
    <?xml-stylesheet href="file:///opt/local/share/nmap/nmap.xsl" type="text/xsl"?>
    <!-- Nmap 5.21 scan initiated Wed Jun 30 18:02:54 2010 as: nmap -p80 -oX scanme.xml scanme.nmap.org localhost -->
    <nmaprun scanner="nmap" args="nmap -p80 -oX scanme.xml scanme.nmap.org localhost" start="1277895774" startstr="Wed Jun 30 18:02:54 2010" version="5.21" xmloutputversion="1.03">
    <scaninfo type="syn"  protocol="tcp" numservices="1" services="80" />
    <verbose level="0" />
    <debugging level="0" />
    <host starttime="1277895774" endtime="1277895774"><status state="up" reason="echo-reply"/>
    <address addr="64.13.134.52" addrtype="ipv4" />
    <hostnames>
    <hostname name="scanme.nmap.org" type="user"/>
    <hostname name="ptr.scanme.nmap.org" type="PTR"/>
    </hostnames>
    <ports><port protocol="tcp" portid="80"><state state="open" reason="syn-ack" reason_ttl="247"/><service name="http" method="table" conf="3" /></port>
    </ports>
    <times srtt="256487" rttvar="197903" to="1048099" />
    </host>
    <host starttime="1277895774" endtime="1277895774"><status state="up" reason="localhost-response"/>
    <address addr="127.0.0.1" addrtype="ipv4" />
    <hostnames>
    <hostname name="localhost" type="user"/>
    <hostname name="ptr.localhost" type="PTR"/>
    </hostnames>
    <ports><port protocol="tcp" portid="80"><state state="closed" reason="reset" reason_ttl="64"/><service name="http" method="table" conf="3" /></port>
    </ports>
    <times srtt="66" rttvar="5000" to="100000" />
    </host>
    <runstats><finished time="1277895774" timestr="Wed Jun 30 18:02:54 2010" elapsed="0.60"/><hosts up="2" down="0" total="2" />
    <!-- Nmap done at Wed Jun 30 18:02:54 2010; 2 IP addresses (2 hosts up) scanned in 0.60 seconds -->
    </runstats></nmaprun>
    
     
  • George  K.

    George K. - 2010-06-30

    What do you think about this?

    xml sel -T -t -m "//host" -m "hostnames/hostname" -v "@name" -o "|" -b -v "address/@addr" -n test.xml
    
     
  • George  K.

    George K. - 2010-06-30

    Or maybe this one that will format the output the way you prefer:

    xml sel -T -t -m "//host/hostnames" -m "hostname" -v "@name" -i "@type='user'" -o "|" -v "../../address/@addr" -b -i "not(position()=last())" -o "|" -b -b -n test.xml
    
     
  • Blake

    Blake - 2010-06-30

    That's it! They both work. Now I tried to add another element to this: osclass/type, but it is not as straightforward as it seems:

    xml sel -T -t -m "//host" -m "hostnames/hostname" -v "@name" -o "|" -b -v "address/@addr" -o "|" -m  "osclass" -v "@type" -n test.xml
    

    Also, in the first example above, I have no idea how you output 2 pipes in output but only have one pipe in the one liner.

     
  • Blake

    Blake - 2010-06-30

    The element osclass/type was not in the previous xml sample. Here is another xml file where it is present:

    <?xml version="1.0" ?>
    <?xml-stylesheet href="file:///opt/local/share/nmap/nmap.xsl" type="text/xsl"?>
    <!-- Nmap 5.21 scan initiated Sat Jun 26 18:11:04 2010 as: nmap -PN -O -p80,25,56432 -iL 100urls.txt -oX 100.xml -->
    <nmaprun scanner="nmap" args="nmap -PN -O -p80,25,443,56432 -iL 100urls.txt -oX 100_hhmi.xml" start="1277547064" startstr="Sat Jun 26 18:11:04 2010" version="5.21" xmloutputversion="1.03">
    <scaninfo type="syn"  protocol="tcp" numservices="4" services="25,80,443,56432" />
    <verbose level="0" />
    <debugging level="0" />
    <host starttime="1277547163" endtime="1277547183"><status state="up" reason="user-set"/>
    <address addr="97.74.215.120" addrtype="ipv4" />
    <hostnames>
    <hostname name="thisisatest.com" type="user"/>
    <hostname name="secureserver.net" type="PTR"/>
    </hostnames>
    <ports><port protocol="tcp" portid="25"><state state="open" reason="syn-ack" reason_ttl="64"/><service name="smtp" method="table" conf="3" /></port>
    <port protocol="tcp" portid="80"><state state="open" reason="syn-ack" reason_ttl="49"/><service name="http" method="table" conf="3" /></port>
    <port protocol="tcp" portid="443"><state state="closed" reason="reset" reason_ttl="49"/><service name="https" method="table" conf="3" /></port>
    <port protocol="tcp" portid="56432"><state state="filtered" reason="no-response" reason_ttl="0"/></port>
    </ports>
    <os><portused state="open" proto="tcp" portid="25" />
    <portused state="closed" proto="tcp" portid="443" />
    <osclass type="general purpose" vendor="Linux" osfamily="Linux" osgen="2.6.X" accuracy="94" />
    <osclass type="remote management" vendor="HP" osfamily="embedded" accuracy="86" />
    <osclass type="WAP" vendor="D-Link" osfamily="embedded" accuracy="85" />
    <osclass type="WAP" vendor="Linksys" osfamily="embedded" accuracy="85" />
    <osclass type="router" vendor="Peplink" osfamily="embedded" accuracy="85" />
    <osmatch name="Linux 2.6.23" accuracy="94" line="23105"/>
    <osmatch name="Linux 2.6.26" accuracy="88" line="23881"/>
    <osmatch name="Linux 2.6.22" accuracy="87" line="22518"/>
    <osmatch name="HP Onboard Administrator management console" accuracy="86" line="14024"/>
    <osmatch name="Linux 2.6.18 - 2.6.27" accuracy="85" line="21548"/>
    <osmatch name="D-Link DSA-3100 or Linksys WRT54GL (DD-WRT v23) WAP, or Peplink Balance 30 router" accuracy="85" line="8677"/>
    <osmatch name="Linux 2.6.15 - 2.6.27" accuracy="85" line="20421"/>
    <osmatch name="Linux 2.6.27 (Ubuntu 8.10)" accuracy="85" line="24343"/>
    </os>
    <uptime seconds="1437071" lastboot="Thu Jun 10 03:02:16 2010" />
    <tcpsequence index="199" difficulty="Good luck!" values="D96F9828,D9779B90,D94ADAD3,D8D9C142,D8816189,D884F796" />
    <ipidsequence class="All zeros" values="0,0,0,0,0,0" />
    <tcptssequence class="other" values="1569DD58,1569DD71,1569DD8A,1569DDA3,1569DDBC,1569DDD5" />
    <times srtt="6928" rttvar="12697" to="100000" />
    </host>
    
     
  • George  K.

    George K. - 2010-07-01

    The correct command line should be

    xml sel -T -t -m "//host" -m "hostnames/hostname" -v "@name" -o "|" -b -v "address/@addr" -o "|" -m  "os/osclass" -o "|" -v "@type" -b -n test.xml
    

    But still, I don't think that it will work, because the xml file you demonstrate doesn't have a single root element.

     
  • Blake

    Blake - 2010-07-01

    This does work! Mostly….
    The xml above that I posted lost the last 3 lines:

    <runstats><finished time="1277895774" timestr="Wed Jun 30 18:02:54 2010" elapsed="0.60"/><hosts up="2" down="0" total="2" /> <!-- Nmap done at Wed Jun 30 18:02:54 2010; 2 IP addresses (2 hosts up) scanned in 0.60 seconds --> </runstats></nmaprun>
    

    Does this resolve the single root element issue you bring up?

    Getting back to your code here, there are actually multiple elements for os/osclass with -v @type, so I get many different pipes with all the values. How do I limit this to just the first value?

    You have been a big help, and I really appreciate it.

     
  • George  K.

    George K. - 2010-07-02

    Yes, the xml file is now complete and valid.

    xml sel -T -t -m "//host" -m "hostnames/hostname" -v "@name" -o "|" -b -v "address/@addr" -o "|" -v  "os/osclass/@type" -n test.xml
    
     
  • Blake

    Blake - 2010-07-03

    Thanks again…This looks very understandable. I extended this a little with:

    xml sel -T -t -m "//host" -m "hostnames/hostname" -v "@name" -o "|" -b -v "address/@addr" -o "|" -v "os/osmatch/@name" -n test.xml
    

    One issue arises which is that if an element is empty, the extra "|" does not appear, which then makes the columns change proper alignment. I get output like:

    westInc.com|www.westinc.com|208.39.44.203|
    AtlantaGym.com|app04.medfusion.net|208.74.44.15|Linux 2.6.23
    Atlasvanline.com|216.135.56.41|Linux 2.6.23
    

    For a correct pipe delimited output, the following would be preferable:

    westInc.com|www.westinc.com|208.39.44.203||
    AtlantaGym.com|app04.medfusion.net|208.74.44.15|Linux 2.6.23
    Atlasvanline.com||216.135.56.41|Linux 2.6.23
    

    Thanks

     
  • George  K.

    George K. - 2010-07-06

    If you need correct piped output then the following you should follow the following format:

    westInc.com|www.westinc.com|208.39.44.203||
    AtlantaGym.com|app04.medfusion.net|208.74.44.15|Linux 2.6.23|
    Atlasvanline.com||216.135.56.41|Linux 2.6.23|
    

    which is the result of this command line

    xml sel -T -t -m "//host/hostnames" -m "hostname[@type='user']" -v "@name" -o "|" -b -m "hostname[@type='PTR']" -v "@name" -b -o "|" -v "../address/@addr" -o "|" -v "../os/osmatch/@name" -o "|" -n test.xml
    

    Assuming that you have one "hostname" with type "user" and zero or one hostnames with type "PTR".

     
  • Blake

    Blake - 2010-07-17

    This is good stuff. You introduced the square brackets in this iteration  which seems to strengthen the structure of the output. I will experiment with this command line a bit more. Thanks again!

     
  • xml-mystified

    xml-mystified - 2012-10-12

    Couldn't find how to start my own question, please forgive.

    I have a files that looks like:

    <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
    <ResourcesResponse>
        <Status>Success</Status>
        <Resource id="14555" name="/var/log/messages-Monitor" description="monitor /var/log/messages" location="" instanceId="12833" typeId="3">
            <ResourcePrototype resourceTypeId="3" instanceId="10109" id="10122" name="FileServer File"/>
            <ResourceInfo key="autoIdentifier" value="/var/log/messages-Monitor"/>
        </Resource>
        <Resource id="14554" name="Control" description="this demonstrates a user-defined control" location="" instanceId="12832" typeId="3">
            <ResourcePrototype resourceTypeId="3" instanceId="10109" id="10123" name="FileServer File"/>
            <ResourceInfo key="autoIdentifier" value="Control"/>
        </Resource>
    </ResourcesResponse>

    I am trying to sel ONLY the data for ResourceInfo@value when //Resource/Resource[@id=10122 not 10123.

    I've tried many (100's)  combinations but the closest form using '-C' looks to be:
    xml sel  -t -m //Resource -m "ResourcePrototype" -v "ResourceInfo/@value"
    <?xml version="1.0"?>
    <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:exslt="http://exslt.org/common" version="1.0" extension-element-prefixes="exslt">
      <xsl:output omit-xml-declaration="yes" indent="no"/>
      <xsl:template match="/">
        <xsl:for-each select="//Resource">
          <xsl:for-each select="ResourcePrototype">
            <xsl:call-template name="value-of-template">
              <xsl:with-param name="select" select="ResourceInfo/@value"/>
            </xsl:call-template>
          </xsl:for-each>
        </xsl:for-each>
      </xsl:template>
      <xsl:template name="value-of-template">
        <xsl:param name="select"/>
        <xsl:value-of select="$select"/>
        <xsl:for-each select="exslt:node-set($select)">
          <xsl:value-of select="'&#10;'"/>
          <xsl:value-of select="."/>
        </xsl:for-each>
      </xsl:template>
    </xsl:stylesheet>

    but I can't seem to get output for  ResourceInfo@value

    TIA for your help.

     
  • Noam Postavsky

    Noam Postavsky - 2012-10-12

    Couldn't find how to start my own question, please forgive.

    I see an Add Topic button at the bottom of the forum's topic list page.

    I am trying to sel ONLY the data for ResourceInfo@value when //Resource/Resource

    xml sel -t -v "//Resouce[ResourcePrototype/@id=10122]/ResourceInfo/@value"
    
     
  • xml-mystified

    xml-mystified - 2012-10-12

    Thank you for your reply.
    My eyes must have been crossed from looking at this too long and didn't see the Add button.

    I tried the command above and received no output. It appears that it should be found by the output of the -C option.

    ~/scripting$ xml sel -t -v "//Resouce/ResourceInfo/@value" myFileResources.xml
    ~/scripting$ xml sel -C -t -v "//Resouce/ResourceInfo/@value" myFileResources.xml
    <?xml version="1.0"?>
    <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:exslt="http://exslt.org/common" version="1.0" extension-element-prefixes="exslt">
      <xsl:output omit-xml-declaration="yes" indent="no"/>
      <xsl:template match="/">
        <xsl:call-template name="value-of-template">
          <xsl:with-param name="select" select="//Resouce/ResourceInfo/@value"/>
        </xsl:call-template>
      </xsl:template>
      <xsl:template name="value-of-template">
        <xsl:param name="select"/>
        <xsl:value-of select="$select"/>
        <xsl:for-each select="exslt:node-set($select)">
          <xsl:value-of select="'&#10;'"/>
          <xsl:value-of select="."/>
        </xsl:for-each>
      </xsl:template>
    </xsl:stylesheet>

     
  • Noam Postavsky

    Noam Postavsky - 2012-10-12

    I tried the command above and received no output.

    Oops, I had a silly typo.

    xml sel -t -v "//Resou[b]r[/b]ce[ResourcePrototype/@id=10122]/ResourceInfo/@value"
    
     
  • xml-mystified

    xml-mystified - 2012-10-12

    Beautiful! Works!
    Also now that I have an example that works against my file, I understand better how to form the template.
    Thanks for your rapid responses and awesome help.
    Have a good weekend.

     

Log in to post a comment.