Hi,
In order to have script work, I needed to apply modest changes to NetDBHelper.pm.
In one point I think the original code has a bug! I have marked it with comment '#BUG#'.
Following I just copy what I have already writtn in the script itself (as comment):
Grabs the arp table from a ScreenOS Juniper (SSG) firewall. Does not populate
the switch status table, unless explicitly requested in netdb.conf; and only for those ports specified in netdb.conf.
There are quite a number of different interface types supported on different SSG models:
Physical interfaces, including ethernet, wireless, serial, bri, adsl and bgroup (bridge group) interface types,
Virtual interfaces, including subinterfaces, aggregate, redundant and virtual security (VSI) interface types.
Functional interfaces, including management, high availibility, vlan, tunnel and loopback interface types.
And the above list is not complete!
Of course not all above types are relevant for getting usefull arp info from the device.
The current version (v1.0) only deals with ethernet interface types (and its subinterfaces), as well as bgroup interfaces.
In fact the above types are tested by me. I have not tested the script on devices having other ACTIVE interface types.
The current version has not been tested on a passive node of an active-passive cluster.
There is a general problem by firewalls (and this not really limited to firewalls), namely that in most cases the firewall is not aware about the VLAN-ID of the segments where its interfaces are patched into.
Therefore it is difficult to get arp info correlated with corresponding VLAN-ID which is mandatory to be imported in NetDB.
On the other hand firewalls are used to separate subnets, and therefore the interfaces are usually patched into unnumbered VLANs. This leads to the situation where the normal L3 devices will not be able to provide arp info for devices in that segment.
Also in our case, the switch ports where the firewall interfaces are connected to are usually configured as access ports in a certain VLAN, and the corresponding firewall interface is configured with no specific VLAN-ID.
One can choose to make all firewall interfaces VLAN-ID aware, but it is to difficult to convince firewall managers to adapt their configuration to what is needed by a monitoring/reporting system.
To overcome the above dilemma, I have added a new pair of directives to NetDB, like the current
quadruple authgroup/authuser/authpass/authenable. (Could also be implemented using a single directive like use_port)
The directives are intvlangroup and intvlantable. The first directive is used to group different devices together under a ceratin name.
The second directive has whitespace separated entries in the form interfacename:VLAN-ID.
This is used to correlate VLAN-ID unaware interfaces with a certain VLAN-ID.
This scraper script is using this info, to correlate arp info gathered from a certain firewall interface with the corresponding VLAN-ID, and add appropriately formatted
arp records to arptable.txt file, to be later imported into NetDB by updatendb.pl.
In SSG jargon "transparent mode" refers to a scenario when corresponding interfaces are connected to different L2 segments running the same L3 subnet, acting as bridge (with security roles) between the two segments.
SSG devices keep and provide (interface specific) mac address info only on interfaces in transparent node.
We do not use SSG devices in transparent mode, and none of interfaces on our SSGs are in transparent mode. All interfaces are in "route" mode and the device can only provide arp info on its interfaces.
All in all this means I treat SSG devices as routers here and not as L2 devices.
Nevertheless, there are situations where NetDB cannot get any mac address info for the devices behind a certain firewall interface.
This can happen when the L2 environment where the corresponding SSG interface is patched to is not under our control.
Here NetDB can provide arp reports, without any corresponding switch-port and mac related reports.
In such (rare) cases I add both interface status as well as mac address records to NetDB!!
In order to have this feature active, the device name and the corresponding port must be configured in netdb.conf, using the keyword 'use_port'.
What the sript does is to add appropriate records in intstatus.txt, and mactable.txt so that they end up in NetDB.
Last but not list, this script deals only with IPv4!
I will upload the files in the next post.
Best Regards,
Hossein Badbanchi
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Excellent work, obviously these firewalls create a lot of challenges for
the program but I think you've done a good job solving these issues. I'll
look at adding some of this functionality to the ASA scraper as well for
ARP data, which also has the issue with VLAN IDs.
I'm going to diff the NetDBHelper.pm you uploaded and apply the changes to
the mainline library as well as include the new scraper in the upcoming
1.13.3 release. I have another contributor who is going to get his changes
in here soon so I can release one more 1.13 release before I start work on
1.14 with some of the major changes we have planned.
Thanks
On Sun, May 24, 2015 at 6:41 PM, Hossein Badbanchi hbdbn@users.sf.net
wrote:
Hey, when you get a chance, try the latest NetDBHelper.pm file from the
trunk/ repository and make sure it's still compatible with your screenos
scraper. There's been a few changes to NetDBHelper since the version you
edited so I had to merge those changes.
Excellent work, obviously these firewalls create a lot of challenges for
the program but I think you've done a good job solving these issues. I'll
look at adding some of this functionality to the ASA scraper as well for
ARP data, which also has the issue with VLAN IDs.
I'm going to diff the NetDBHelper.pm you uploaded and apply the changes to
the mainline library as well as include the new scraper in the upcoming
1.13.3 release. I have another contributor who is going to get his changes
in here soon so I can release one more 1.13 release before I start work on
1.14 with some of the major changes we have planned.
Thanks
On Sun, May 24, 2015 at 6:41 PM, Hossein Badbanchi hbdbn@users.sf.net
wrote:
Hi,
I have already upgraded to 1.13.2. I think it was ssh_port, ssh_timeout and login_timeout, right?
Regarding the difficulties of SSH to Juniper SSGs, it was real hard. I had already my own old scripts using SSH and Expect from within Perl, but not Net::SSH::Expect.
Eventually it came out that if I use longer timeout values in read_line() (say 1 second instead of current 0.1 second) everything will run much smoother, AND slower of course.
But I found this when I had solved(?) the problem by not 100% relying on peek() function!
Anyway, I am glad the idea of 'intvlantable' comes handy.
Regarding your plans for future releases/versions, I would greatly appreciate if you could provide the community with more info, so that (at least) I could decide how can I cover my needs.
For example I want to have an authentication section which authenticates users against a Radius server, and limits their access to certain parts of the system (be it a scope or vlan_change, etc.). If I know that this will come soon, I will wait. But if I know it is not wanted at all, I will implement my own module.
Best Regards,
Hossein Badbanchi
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I'll have to take a look at your implementation of peek() to see if that
can help out on other scrapers. Glad everything is working on the latest
version, and yes those were the other changes.
As far as authentication goes, if you implement radius authentication on
your apache server, you should be able to restrict access to reports and
changes already through the level_x_access settings. Have you taken a close
look at that already? That's what we do here. If it's not flexible enough
for you, let me know and I can see what needs changing. You can also pass
out vlan_change capabilities to users on individual switches or based on
wildcards as well.
Jonathan
On Fri, May 29, 2015 at 6:42 AM, Hossein Badbanchi hbdbn@users.sf.net
wrote:
Hi,
I have already upgraded to 1.13.2. I think it was ssh_port, ssh_timeout
and login_timeout, right?
Regarding the difficulties of SSH to Juniper SSGs, it was real hard. I had
already my own old scripts using SSH and Expect from within Perl, but not
Net::SSH::Expect.
Eventually it came out that if I use longer timeout values in read_line()
(say 1 second instead of current 0.1 second) everything will run much
smoother, AND slower of course.
But I found this when I had solved(?) the problem by not 100% relying on
peek() function!
Anyway, I am glad the idea of 'intvlantable' comes handy.
Regarding your plans for future releases/versions, I would greatly
appreciate if you could provide the community with more info, so that (at
least) I could decide how can I cover my needs.
For example I want to have an authentication section which authenticates
users against a Radius server, and limits their access to certain parts of
the system (be it a scope or vlan_change, etc.). If I know that this will
come soon, I will wait. But if I know it is not wanted at all, I will
implement my own module.
Hi,
In order to have script work, I needed to apply modest changes to NetDBHelper.pm.
In one point I think the original code has a bug! I have marked it with comment '#BUG#'.
Following I just copy what I have already writtn in the script itself (as comment):
Grabs the arp table from a ScreenOS Juniper (SSG) firewall. Does not populate
the switch status table, unless explicitly requested in netdb.conf; and only for those ports specified in netdb.conf.
There are quite a number of different interface types supported on different SSG models:
Physical interfaces, including ethernet, wireless, serial, bri, adsl and bgroup (bridge group) interface types,
Virtual interfaces, including subinterfaces, aggregate, redundant and virtual security (VSI) interface types.
Functional interfaces, including management, high availibility, vlan, tunnel and loopback interface types.
And the above list is not complete!
Of course not all above types are relevant for getting usefull arp info from the device.
The current version (v1.0) only deals with ethernet interface types (and its subinterfaces), as well as bgroup interfaces.
In fact the above types are tested by me. I have not tested the script on devices having other ACTIVE interface types.
The current version has not been tested on a passive node of an active-passive cluster.
There is a general problem by firewalls (and this not really limited to firewalls), namely that in most cases the firewall is not aware about the VLAN-ID of the segments where its interfaces are patched into.
Therefore it is difficult to get arp info correlated with corresponding VLAN-ID which is mandatory to be imported in NetDB.
On the other hand firewalls are used to separate subnets, and therefore the interfaces are usually patched into unnumbered VLANs. This leads to the situation where the normal L3 devices will not be able to provide arp info for devices in that segment.
Also in our case, the switch ports where the firewall interfaces are connected to are usually configured as access ports in a certain VLAN, and the corresponding firewall interface is configured with no specific VLAN-ID.
One can choose to make all firewall interfaces VLAN-ID aware, but it is to difficult to convince firewall managers to adapt their configuration to what is needed by a monitoring/reporting system.
To overcome the above dilemma, I have added a new pair of directives to NetDB, like the current
quadruple authgroup/authuser/authpass/authenable. (Could also be implemented using a single directive like use_port)
The directives are intvlangroup and intvlantable. The first directive is used to group different devices together under a ceratin name.
The second directive has whitespace separated entries in the form interfacename:VLAN-ID.
This is used to correlate VLAN-ID unaware interfaces with a certain VLAN-ID.
This scraper script is using this info, to correlate arp info gathered from a certain firewall interface with the corresponding VLAN-ID, and add appropriately formatted
arp records to arptable.txt file, to be later imported into NetDB by updatendb.pl.
In SSG jargon "transparent mode" refers to a scenario when corresponding interfaces are connected to different L2 segments running the same L3 subnet, acting as bridge (with security roles) between the two segments.
SSG devices keep and provide (interface specific) mac address info only on interfaces in transparent node.
We do not use SSG devices in transparent mode, and none of interfaces on our SSGs are in transparent mode. All interfaces are in "route" mode and the device can only provide arp info on its interfaces.
All in all this means I treat SSG devices as routers here and not as L2 devices.
Nevertheless, there are situations where NetDB cannot get any mac address info for the devices behind a certain firewall interface.
This can happen when the L2 environment where the corresponding SSG interface is patched to is not under our control.
Here NetDB can provide arp reports, without any corresponding switch-port and mac related reports.
In such (rare) cases I add both interface status as well as mac address records to NetDB!!
In order to have this feature active, the device name and the corresponding port must be configured in netdb.conf, using the keyword 'use_port'.
What the sript does is to add appropriate records in intstatus.txt, and mactable.txt so that they end up in NetDB.
Last but not list, this script deals only with IPv4!
I will upload the files in the next post.
Best Regards,
Hossein Badbanchi
Hi,
Here are the files.
Regards,
Hossein Badbanchi
Hossein,
Excellent work, obviously these firewalls create a lot of challenges for
the program but I think you've done a good job solving these issues. I'll
look at adding some of this functionality to the ASA scraper as well for
ARP data, which also has the issue with VLAN IDs.
I'm going to diff the NetDBHelper.pm you uploaded and apply the changes to
the mainline library as well as include the new scraper in the upcoming
1.13.3 release. I have another contributor who is going to get his changes
in here soon so I can release one more 1.13 release before I start work on
1.14 with some of the major changes we have planned.
Thanks
On Sun, May 24, 2015 at 6:41 PM, Hossein Badbanchi hbdbn@users.sf.net
wrote:
Hey, when you get a chance, try the latest NetDBHelper.pm file from the
trunk/ repository and make sure it's still compatible with your screenos
scraper. There's been a few changes to NetDBHelper since the version you
edited so I had to merge those changes.
Thanks
On Wed, May 27, 2015 at 10:33 AM, Jonathan Yantis yantisj@users.sf.net
wrote:
Hi,
I have already upgraded to 1.13.2. I think it was ssh_port, ssh_timeout and login_timeout, right?
Regarding the difficulties of SSH to Juniper SSGs, it was real hard. I had already my own old scripts using SSH and Expect from within Perl, but not Net::SSH::Expect.
Eventually it came out that if I use longer timeout values in read_line() (say 1 second instead of current 0.1 second) everything will run much smoother, AND slower of course.
But I found this when I had solved(?) the problem by not 100% relying on peek() function!
Anyway, I am glad the idea of 'intvlantable' comes handy.
Regarding your plans for future releases/versions, I would greatly appreciate if you could provide the community with more info, so that (at least) I could decide how can I cover my needs.
For example I want to have an authentication section which authenticates users against a Radius server, and limits their access to certain parts of the system (be it a scope or vlan_change, etc.). If I know that this will come soon, I will wait. But if I know it is not wanted at all, I will implement my own module.
Best Regards,
Hossein Badbanchi
Hey,
I'll have to take a look at your implementation of peek() to see if that
can help out on other scrapers. Glad everything is working on the latest
version, and yes those were the other changes.
As far as authentication goes, if you implement radius authentication on
your apache server, you should be able to restrict access to reports and
changes already through the level_x_access settings. Have you taken a close
look at that already? That's what we do here. If it's not flexible enough
for you, let me know and I can see what needs changing. You can also pass
out vlan_change capabilities to users on individual switches or based on
wildcards as well.
Jonathan
On Fri, May 29, 2015 at 6:42 AM, Hossein Badbanchi hbdbn@users.sf.net
wrote: