The Driver Deps code exports functions to communicate with the SW Hardware. Most appear self explanatory. However while messing about with the source code, and adding print statements in the functions to gain an understanding of how they work. One functions is puzzling me a bit. The "void driver_flush_all_fids(per_tree_port_t *ptp)" function, would have expected this to delete all the FIDS on the HW (STU been Marvel specific). But noticed this function is called when mstpctl createtree. In this case it does not make sense to clear all the fids entries stored
Is the similar to a cache were you commit to RAM and flush the cache copy i.e a the copy that is stored in mstpd s flushed once the the FID entry has been committed to the HW switch
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
First of all, do not forget to update code from the SVN. I've just committed crucial bugfix, and the latest SVN revision is now 31.
Second, some theory. Each VLAN is mapped to some FID (Marvell has concept of FIDs, at least 88E6097 part, which I'm working with). MAC learning and filtering is done independently in different FIDs. Now, each FID is mapped to some MSTI. That's how VLANs are mapped o the MSTIs - indirectly, via FIDs intermediate abstraction layer.
Next, to the driver_flush_all_fids(). It is intended to instruct driver to remove from MAC Filtering Database (also known as MAC Learning Database, or ATU in the Marvell terms) all the MACs that are learned for the specified port in all the FIDs mapped to the specified tree. It is nothing to do with STU; it is about flushing some MACs from ATU. The part I'm working with (88E6097) have the ATU operation "Move all Non-Static entries in a particular FID". If you set the ToPort = 0xF for this operation, it will remove all MAC entries for the specified Port in the specified FID, which is ideally suited for the driver_flush_all_fids() implementation.
The per_tree_port_t struct represents all the port info which is specific to the given tree (tree == MSTI). In the context of this function we need following fields of that structure:
1) ptp->port - is the pointer to the port_t struct,
2) ptp->tree - is the pointer to the tree_t struct.
You also need the list of FIDs which are mapped to the tree. To get it you first need the pointer to the bridge_t struct, which can be accessed as ptp->port->bridge or ptp->tree->bridge (both ways give us the same pointer to the same bridge_t struct). There is fid2mstid array in the bridge_t struct, indexed by FID. To find all the FIDs mapped to the given tree you can use the loop:
Would it be correct to assume that, insert and remove entries from the STU Data base
driver_create_msti
driver_delete_msti
Were also using a DSA driver. Were each Physical port on the switch is 1 to 1 mapped to a vector[] of Linux netdevice. The port number used in your example above, does this correspond to the DSA netdevices, and how is it given its number.
Was going to pass the netdevice name using data structs in mstpd i.e. ptp->port->sysdeps.name(per_tree_port_t -> port_t->sysdep_if_data_t). The driver will manage a look up table to address the correct marvel chip and port. We have multiple chips and are cross links DSA to chain them. But if you already have a mechanism see no point.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Would it be correct to assume that, insert and remove entries from the STU Data base
driver_create_msti
driver_delete_msti
Yep )
Were also using a DSA driver. Were each Physical port on the switch is 1 to 1 mapped to a vector of Linux netdevice. The port number used in your example above, does this correspond to the DSA netdevices, and how is it given its number.
Was going to pass the netdevice name using data structs in mstpd i.e. ptp->port->sysdeps.name(per_tree_port_t -> port_t->sysdep_if_data_t). The driver will manage a look up table to address the correct marvel chip and port. We have multiple chips and are cross links DSA to chain them. But if you already have a mechanism see no point.
Hmm... the port_number I've used purely as an example. It is simply the number of port inside the Linux bridge (/sys/class/net/IFNAME/brport/port_no) and it heavily depends on the order in which interfaces are added and removed to/from that bridge.
Moreover, I also use sysdeps.name to reliably identify interface in my code, so, it seems, that is ok.
Last edit: Vitalii Demianets 2012-06-06
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I think it would useful to add a stub function and user command to show the port states of the spanning tree instances in the driver deps code. From a debugging point of view this would be great it would allow the user to verify that the switch port states on the switch match those in the daemon. I can submit a patch for this if find it usefull. Have added this task to the todo list
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I don't find it useful for the number of reasons:
1) You can see the port states, for example by issuing command "mstpctl showtreeport". What you are proposing - is to confirm that these states have been actually set in driver. So the info you are talking about have value only for debug purposes.
2) I want to keep algorithm as much separated from hardware as it can be. That's why interaction between mstpd and driver (driver_deps.c) is limited to the very few function calls.
3) You still need a custom tool to interact with hardware about non-stp-related info, e.g. you need to set VLANs, enable/disable ports, get PHY port link state and result of autonegotiation etc. The natural place to read real STP port states from the hardware is in that tool.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Another issue I have noticed in communicating with the driver. Its not directly related to MSTP but the OS is reporting a problem. Eventually the retry mechanism will set the port state correctly. See Message below
RTNETLINK answers: Device or resource busy
2012-07-13 11:03:44 error, MSTP_OUT_set_state: br0:fa1_6 Couldn't set kernel bridge state forwarding
the return value is 104
2012-07-13 11:03:44 MSTP_OUT_set_state: br0:fa1_6:1 entering forwarding state
the return value is 104
2012-07-13 11:03:44 MSTP_OUT_set_state: br0:fa1_6:2 entering forwarding state
I choose Netlink sockets as there asynchronous, provide socket queues, and you can send multicast message to multiple user space process.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I talk to my driver via the procfs files. I do not use Netlink for this task, so you are here on an unexplored territory, I'm afraid ;)
It seems that your Netlink calls in driver_deps.c somehow conflict with Netlink call in br_set_state() function in bridge_track.c
Unfortunately I can not help further in debugging the issue because I'm not using Netlink for communication with my driver.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Busy doing other things the last few days. Have changed the communication link between the driver and mstpd. This is working perfectly. However still get the same problem, in terms of the netlink resource been busy. There is a problem here any ideas on hot to go about resolving???
[root@switch userscripts]# mstpctl createtree br0 2
PURE PROC ENTRIES
2000-03-10 02:30:15 MSTP_OUT_set_state: br0:fa1_0:2 entering blocking state
2000-03-10 02:30:15 MSTP_OUT_set_state: br0:fa1_6:2 entering blocking state
2000-03-10 02:30:15 MSTP_OUT_flush_all_fids: br0:fa1_0:2 Flushing forwarding database
2000-03-10 02:30:15 MSTP_OUT_flush_all_fids: br0:fa1_6:2 Flushing forwarding database [root@switch userscripts]# 2000-03-10 02:30:18 MSTP_OUT_set_state: br0:fa1_0:0 entering learning state
RTNETLINK answers: Device or resource busy
2000-03-10 02:30:18 error, MSTP_OUT_set_state: br0:fa1_0 Couldn't set kernel bridge state learning
2000-03-10 02:30:18 MSTP_OUT_set_state: br0:fa1_6:0 entering learning state
RTNETLINK answers: Device or resource busy
2000-03-10 02:30:18 error, MSTP_OUT_set_state: br0:fa1_6 Couldn't set kernel bridge state learning
2000-03-10 02:30:19 MSTP_OUT_set_state: br0:fa1_0:0 entering forwarding state
RTNETLINK answers: Device or resource busy
2000-03-10 02:30:19 error, MSTP_OUT_set_state: br0:fa1_0 Couldn't set kernel bridge state forwarding
2000-03-10 02:30:19 MSTP_OUT_set_state: br0:fa1_0:1 entering learning state
2000-03-10 02:30:19 MSTP_OUT_set_state: br0:fa1_0:1 entering forwarding state
2000-03-10 02:30:19 MSTP_OUT_set_state: br0:fa1_0:2 entering learning state
2000-03-10 02:30:19 MSTP_OUT_set_state: br0:fa1_0:2 entering forwarding state
2000-03-10 02:30:19 MSTP_OUT_set_state: br0:fa1_6:0 entering forwarding state
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Digging a little deeper on this problem, it appears to come from a an illegal seek, ESPIPE, so were trying to seek on object that is not seekable a linear stream . See the debug below.
Gary, thank you for the debug effort!
Please, could you look at the previous lines of the debug (strace?) output and tell me how the device/file/socket with fd=9 have been created. If we identify which device/file/socket it is, we could make further conclusions.
Last edit: Vitalii Demianets 2012-07-19
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I have looked more closely at the kernel code and it gave me another thought. Please, do the following (substitute br0 with the actual name of your bridge):
cat /sys/class/net/br0/bridge/stp_state
and check the ouput. It must be 2 (2 = kernel knows about user-space STP daemon). If it is 1 (1 = kernel implements its own in-kernel STP algorithm) - then here is your problem.
Some background: kernel bridge can have three stp_states:
* 0 = no stp
* 1 = in-kernel stp
* 2 = user-space stp
mstpd works only when kernel bridge is in stp_state=2. To put kernel in that state bridge-stp helper script must return success when called from kernel. Usually it is called from kernel when STP is enabled by user (e.g. by issuing "brctl stp br0 on" command). When user requests kernel bridge to enable stp, kernel checks (with the help of bridge-stp script) if there is user-space STP daemon and if bridge-stp returns success - kernel goes to stp_state=2, else it goes to stp_state=1.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
In my case the brctl tool is looking in the wrong location
cat /sys/class/net/br0/bridge/stp_state
However my brctl tool uses
/sys/class/net/br0/stp_state
I placed your helper script in, is the correct. What proper usage for this?
bridge-stp: /usr/sbin/bridge-stp
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The proper location is /sbin/bridge-stp. Place it there. Modify it for your needs - at the moment it supports only one bridge named br0.
What this script is intended to do: kernel calls it with two parameters. First ($1) is the name of the bridge interface. Second ($2) is "start" or "stop" - indicating if STP is being started or stopped. The script should return SUCCESS (0) for the "start" command if you want implement user-space STP for the bridge in $1, as in case with mstpd. The script should return FAILURE (non-zero code) for the "start" if you do not want user-space implementation of STP for the bridge in $1.
As you can see from the example script in distribution, "mstpctl addbridge" commands are executed from here, so in most situations you don't need to add/remove bridges from mstpd manually.
The preferred sequence of start-up commands is:
1) brctl addbr br0
2) brctl addif br0 eth0 eth1 ...
3) ifconfig br0 up
4) mstpd
5) brctl br0 stp on
In the last step when you enable stp in br0, the kernel will call the bridge-stp script with command line "br0 start". The scripts returns success (0) and kernel will go into stp_state=2 (user-space stp). Also "mstpctl addbridge br0" will be issued in that script, so the br0 will be added to the mstpd automagically.
That is not a problem, just a nuisance - brctl still completes its work via ioctl even with incorrect path. The real problem: you must ensure that /sbin/bridge-stp returns SUCCESS (i.e. 0) when kernel calls it (after "brctl stp br0 on").
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The Driver Deps code exports functions to communicate with the SW Hardware. Most appear self explanatory. However while messing about with the source code, and adding print statements in the functions to gain an understanding of how they work. One functions is puzzling me a bit. The "void driver_flush_all_fids(per_tree_port_t *ptp)" function, would have expected this to delete all the FIDS on the HW (STU been Marvel specific). But noticed this function is called when mstpctl createtree. In this case it does not make sense to clear all the fids entries stored
Is the similar to a cache were you commit to RAM and flush the cache copy i.e a the copy that is stored in mstpd s flushed once the the FID entry has been committed to the HW switch
Hello, Gary!
First of all, do not forget to update code from the SVN. I've just committed crucial bugfix, and the latest SVN revision is now 31.
Second, some theory. Each VLAN is mapped to some FID (Marvell has concept of FIDs, at least 88E6097 part, which I'm working with). MAC learning and filtering is done independently in different FIDs. Now, each FID is mapped to some MSTI. That's how VLANs are mapped o the MSTIs - indirectly, via FIDs intermediate abstraction layer.
Next, to the driver_flush_all_fids(). It is intended to instruct driver to remove from MAC Filtering Database (also known as MAC Learning Database, or ATU in the Marvell terms) all the MACs that are learned for the specified port in all the FIDs mapped to the specified tree. It is nothing to do with STU; it is about flushing some MACs from ATU. The part I'm working with (88E6097) have the ATU operation "Move all Non-Static entries in a particular FID". If you set the ToPort = 0xF for this operation, it will remove all MAC entries for the specified Port in the specified FID, which is ideally suited for the driver_flush_all_fids() implementation.
The per_tree_port_t struct represents all the port info which is specific to the given tree (tree == MSTI). In the context of this function we need following fields of that structure:
1) ptp->port - is the pointer to the port_t struct,
2) ptp->tree - is the pointer to the tree_t struct.
You also need the list of FIDs which are mapped to the tree. To get it you first need the pointer to the bridge_t struct, which can be accessed as ptp->port->bridge or ptp->tree->bridge (both ways give us the same pointer to the same bridge_t struct). There is fid2mstid array in the bridge_t struct, indexed by FID. To find all the FIDs mapped to the given tree you can use the loop:
for(fid = 0; fid <= MAX_FID; ++fid) { if(ptp->MSTID == ptp->tree->bridge->fid2mstid[fid]) { InstructMarvellToRemoveEntries(__be16_to_cpu(ptp->port->port_number), fid); } }Thanks for your response, this makes more sense.
Would it be correct to assume that, insert and remove entries from the STU Data base
driver_create_msti
driver_delete_msti
Were also using a DSA driver. Were each Physical port on the switch is 1 to 1 mapped to a vector[] of Linux netdevice. The port number used in your example above, does this correspond to the DSA netdevices, and how is it given its number.
Was going to pass the netdevice name using data structs in mstpd i.e. ptp->port->sysdeps.name(per_tree_port_t -> port_t->sysdep_if_data_t). The driver will manage a look up table to address the correct marvel chip and port. We have multiple chips and are cross links DSA to chain them. But if you already have a mechanism see no point.
Yep )
Hmm... the port_number I've used purely as an example. It is simply the number of port inside the Linux bridge (/sys/class/net/IFNAME/brport/port_no) and it heavily depends on the order in which interfaces are added and removed to/from that bridge.
Moreover, I also use sysdeps.name to reliably identify interface in my code, so, it seems, that is ok.
Last edit: Vitalii Demianets 2012-06-06
Thats fine will continue with sysdeps.name thought that was the most logical approach. Your example planted the idea of other approaches.
I think it would useful to add a stub function and user command to show the port states of the spanning tree instances in the driver deps code. From a debugging point of view this would be great it would allow the user to verify that the switch port states on the switch match those in the daemon. I can submit a patch for this if find it usefull. Have added this task to the todo list
I don't find it useful for the number of reasons:
1) You can see the port states, for example by issuing command "mstpctl showtreeport". What you are proposing - is to confirm that these states have been actually set in driver. So the info you are talking about have value only for debug purposes.
2) I want to keep algorithm as much separated from hardware as it can be. That's why interaction between mstpd and driver (driver_deps.c) is limited to the very few function calls.
3) You still need a custom tool to interact with hardware about non-stp-related info, e.g. you need to set VLANs, enable/disable ports, get PHY port link state and result of autonegotiation etc. The natural place to read real STP port states from the hardware is in that tool.
What mechanism did you use in your driver deps??
Another issue I have noticed in communicating with the driver. Its not directly related to MSTP but the OS is reporting a problem. Eventually the retry mechanism will set the port state correctly. See Message below
RTNETLINK answers: Device or resource busy
2012-07-13 11:03:44 error, MSTP_OUT_set_state: br0:fa1_6 Couldn't set kernel bridge state forwarding
the return value is 104
2012-07-13 11:03:44 MSTP_OUT_set_state: br0:fa1_6:1 entering forwarding state
the return value is 104
2012-07-13 11:03:44 MSTP_OUT_set_state: br0:fa1_6:2 entering forwarding state
I choose Netlink sockets as there asynchronous, provide socket queues, and you can send multicast message to multiple user space process.
I talk to my driver via the procfs files. I do not use Netlink for this task, so you are here on an unexplored territory, I'm afraid ;)
It seems that your Netlink calls in driver_deps.c somehow conflict with Netlink call in br_set_state() function in bridge_track.c
Unfortunately I can not help further in debugging the issue because I'm not using Netlink for communication with my driver.
Hi Vitalii
Busy doing other things the last few days. Have changed the communication link between the driver and mstpd. This is working perfectly. However still get the same problem, in terms of the netlink resource been busy. There is a problem here any ideas on hot to go about resolving???
[root@switch userscripts]# mstpctl createtree br0 2
PURE PROC ENTRIES
2000-03-10 02:30:15 MSTP_OUT_set_state: br0:fa1_0:2 entering blocking state
2000-03-10 02:30:15 MSTP_OUT_set_state: br0:fa1_6:2 entering blocking state
2000-03-10 02:30:15 MSTP_OUT_flush_all_fids: br0:fa1_0:2 Flushing forwarding database
2000-03-10 02:30:15 MSTP_OUT_flush_all_fids: br0:fa1_6:2 Flushing forwarding database
[root@switch userscripts]# 2000-03-10 02:30:18 MSTP_OUT_set_state: br0:fa1_0:0 entering learning state
RTNETLINK answers: Device or resource busy
2000-03-10 02:30:18 error, MSTP_OUT_set_state: br0:fa1_0 Couldn't set kernel bridge state learning
2000-03-10 02:30:18 MSTP_OUT_set_state: br0:fa1_6:0 entering learning state
RTNETLINK answers: Device or resource busy
2000-03-10 02:30:18 error, MSTP_OUT_set_state: br0:fa1_6 Couldn't set kernel bridge state learning
2000-03-10 02:30:19 MSTP_OUT_set_state: br0:fa1_0:0 entering forwarding state
RTNETLINK answers: Device or resource busy
2000-03-10 02:30:19 error, MSTP_OUT_set_state: br0:fa1_0 Couldn't set kernel bridge state forwarding
2000-03-10 02:30:19 MSTP_OUT_set_state: br0:fa1_0:1 entering learning state
2000-03-10 02:30:19 MSTP_OUT_set_state: br0:fa1_0:1 entering forwarding state
2000-03-10 02:30:19 MSTP_OUT_set_state: br0:fa1_0:2 entering learning state
2000-03-10 02:30:19 MSTP_OUT_set_state: br0:fa1_0:2 entering forwarding state
2000-03-10 02:30:19 MSTP_OUT_set_state: br0:fa1_6:0 entering forwarding state
Digging a little deeper on this problem, it appears to come from a an illegal seek, ESPIPE, so were trying to seek on object that is not seekable a linear stream . See the debug below.
fcntl64(9, F_GETFL)= 0x20002 (flags O_RDWR|O_LARGEFILE)
fstat64(9, {st_mode=S_IFCHR|0600, st_rdev=makedev(4, 64), ...}) = 0
ioctl(9, SNDCTL_TMR_TIMEBASE or TCGETS, {B9600 opost isig icanon echo ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2aaca000
_llseek(9, 0, 0x7efb25a0, SEEK_CUR)= -1 ESPIPE (Illegal seek)
write(9, "RTNETLINK answers: Device or res"..., 43RTNETLINK answers: Device or resource busy
) = 43
close(9)
ESPIPE Illegal seek: a seek operation was attempted on a sequential object where seeking makes no sense, like a terminal.
Gary, thank you for the debug effort!
Please, could you look at the previous lines of the debug (strace?) output and tell me how the device/file/socket with fd=9 have been created. If we identify which device/file/socket it is, we could make further conclusions.
Last edit: Vitalii Demianets 2012-07-19
I have looked more closely at the kernel code and it gave me another thought. Please, do the following (substitute br0 with the actual name of your bridge):
and check the ouput. It must be 2 (2 = kernel knows about user-space STP daemon). If it is 1 (1 = kernel implements its own in-kernel STP algorithm) - then here is your problem.
Some background: kernel bridge can have three stp_states:
* 0 = no stp
* 1 = in-kernel stp
* 2 = user-space stp
mstpd works only when kernel bridge is in stp_state=2. To put kernel in that state bridge-stp helper script must return success when called from kernel. Usually it is called from kernel when STP is enabled by user (e.g. by issuing "brctl stp br0 on" command). When user requests kernel bridge to enable stp, kernel checks (with the help of bridge-stp script) if there is user-space STP daemon and if bridge-stp returns success - kernel goes to stp_state=2, else it goes to stp_state=1.
Nice one. This is the problem, I have been starting things manually
Last edit: Gary 2012-07-19
In my case the brctl tool is looking in the wrong location
cat /sys/class/net/br0/bridge/stp_state
However my brctl tool uses
/sys/class/net/br0/stp_state
I placed your helper script in, is the correct. What proper usage for this?
bridge-stp: /usr/sbin/bridge-stp
The proper location is /sbin/bridge-stp. Place it there. Modify it for your needs - at the moment it supports only one bridge named br0.
What this script is intended to do: kernel calls it with two parameters. First ($1) is the name of the bridge interface. Second ($2) is "start" or "stop" - indicating if STP is being started or stopped. The script should return SUCCESS (0) for the "start" command if you want implement user-space STP for the bridge in $1, as in case with mstpd. The script should return FAILURE (non-zero code) for the "start" if you do not want user-space implementation of STP for the bridge in $1.
As you can see from the example script in distribution, "mstpctl addbridge" commands are executed from here, so in most situations you don't need to add/remove bridges from mstpd manually.
The preferred sequence of start-up commands is:
1) brctl addbr br0
2) brctl addif br0 eth0 eth1 ...
3) ifconfig br0 up
4) mstpd
5) brctl br0 stp on
In the last step when you enable stp in br0, the kernel will call the bridge-stp script with command line "br0 start". The scripts returns success (0) and kernel will go into stp_state=2 (user-space stp). Also "mstpctl addbridge br0" will be issued in that script, so the br0 will be added to the mstpd automagically.
Also see this thread for the additional discussion on the "-d" flag and bridge-stp script: https://sourceforge.net/p/mstpd/discussion/general/thread/73cdf74a/#e9d5/e6a7/4ef5/91e8/ae2f/09db/efd3/52e8
Moved it to /sbin/ which my google tells me is the correct location
Yes. I have committed change to the SVN repo - now bridge-stp is always installed to the correct location after "make install".
Not the first to experience this bug
http://thr3ads.net/linux-ethernet-bridging/2007/07/425704-Bridge-brctl-uses-incorrect-sysfs-path
That is not a problem, just a nuisance - brctl still completes its work via ioctl even with incorrect path. The real problem: you must ensure that /sbin/bridge-stp returns SUCCESS (i.e. 0) when kernel calls it (after "brctl stp br0 on").
The issue is fixed at brctl-utills 1.5