Re: [mpls-linux-devel] LDP problems
Status: Beta
Brought to you by:
jleu
Menu
▾
▴
Re: [mpls-linux-devel] LDP problems
|
From: Marco S. <mar...@s2...> - 2007-03-20 16:07:48
|
Hello James, the Labeling is working better now, thank you for the debugging. But as you expected, the problems aren't completely solved. I think there is a problem with networks where loops are possible... The labelspaces and crossconnects are also marked inactive on my systems, but it is possible to send labeled packets through the network. It is! working at the moment, but I had to remove the link between E4 and E3. (one, that made redundant paths possible) Zebra/Ldpd doesn't seem to be able to manage redundant paths with the same destination. It crashed after enabling the link again. Have a look at the logfiles: http://www-user.tu-chemnitz.de/~smarco/MPLS/20070320_v2/E3_with_crash_after= _linkupdate.log There are a few more logs from today in the MPLS directory. Right now, I found the same error as Gonzalo (at the bottom of the file): 2007/03/20 16:08:01 LDP: found pending NHLFE: 0x9782308 2007/03/20 16:08:01 ZEBRA: zebra message comes from socket [17] 2007/03/20 16:08:01 ZEBRA: zebra message received [ZEBRA_MPLS_FTN_ADD] 11 2007/03/20 16:08:01 ZEBRA: IPV41 mismatch >>2007/03/20 16:08:01 ZEBRA: mpls_ftn_register: unable to find FEC 172.16.2= 0.0/24 2007/03/20 16:08:01 ZEBRA: zebra message comes from socket [17] 2007/03/20 16:08:01 ZEBRA: zebra message received [ZEBRA_MPLS_XC_ADD] 11 ZEBRA: Received signal 11 at 1174403281 (si_addr 0xe, PC 0x80539cf); aborti= ng... Program counter: /usr/sbin/zebra[0x80539cf] Regards, Marco > Hi: > I've downloaded your revised quagga RPM for FC5 and I tried this scenario: > eth0 eth1 eth1 eth0 > 12.0.0.1-------PE1---------------------PE2-------13.0.0.1 > .1 10.0.0.0/24 .2 > I enclose the configuration and commands related to MPLS and LDP at > PE2 and the log file created debugging zebra. > At the bottom of the log file there's: "unable to find FEC > 10.0.0.0/24", furthermore, the output of "sh mpls forwarding" shows the l= abelspace as inactive. > Please take a look, I hope it could be helpful to figure out what's happe= ning. > Thanks in advance, > Gonzalo. > PE3 config > ------------------------------------------------------------------------ > Current configuration: > ! > hostname PE3 > log file /var/log/quagga/1955d.log > ! > debug zebra events > debug zebra packet > debug zebra kernel > debug zebra rib > ! > interface eth0 > ipv6 nd suppress-ra > ! > interface eth1 > ipv6 nd suppress-ra > mpls ip > mpls labelspace 0 > ! > interface lo > ! > interface mpls0 > ipv6 nd suppress-ra > ! > interface sit0 > ipv6 nd suppress-ra > ! > mpls ldp > egress connected > ! > router ospf > network 10.0.0.0/24 area 0.0.0.0 > network 13.0.0.0/24 area 0.0.0.0 > ! > ip forwarding > ! > line vty > ! > PE3# sh ip route > Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, > I - ISIS, B - BGP, > - selected route, * - FIB route > O 10.0.0.0/24 [110/10] is directly connected, eth1, 00:02:49 C>>* 10.0.0.0/24 is directly connected, eth1 O>>* 12.0.0.0/24 [110/20] via 10.0.0.1, eth1, 00:01:37 > O 13.0.0.0/24 [110/10] is dPE3# sh ldp neig > Peer LDP Ident: 12.0.0.1:0; Local LDP Ident: 13.0.0.1:0 > TCP connection: 10.0.0.2.45483 - 10.0.0.1.646 > State: OPERATIONAL; Msgs sent/recv: 21/62; UNSOLICITED > Up time: 00:04:08 > LDP discovery sources: > eth1 > Addresses bound to peer: > 10.0.0.1 12.0.0.1 > irectly connected, eth0, 00:02:43 C>>* 13.0.0.0/24 is directly connected, eth0 C>>* 127.0.0.0/8 is directly connected, lo > PE3# sh mpls for > Insegments: > Lbl Spc Label Owner > 0 10000 ldp (inactive) > Total 1 > Outsegments: > Interface Label Next Hop Owner > eth1 10000 10.0.0.1 ldp > Total 1 > Cross Connects: > Total 0 > PE3# sh ldp ses > 2 9.151.111.152 45 OPERATIONAL > 10.0.0.1 > 12.0.0.1 > PE3# sh ldp disc > Local LSR Identifier: 13.0.0.1 > Interface Discovery Sources: > eth1: xmit/recv > LDP Id: 12.0.0.1:0 > Targeted Discovery Sources: > No configured peers > PE3# sh ldp neig > Peer LDP Ident: 12.0.0.1:0; Local LDP Ident: 13.0.0.1:0 > TCP connection: 10.0.0.2.45483 - 10.0.0.1.646 > State: OPERATIONAL; Msgs sent/recv: 21/62; UNSOLICITED > Up time: 00:04:08 > LDP discovery sources: > eth1 > Addresses bound to peer: > 10.0.0.1 12.0.0.1 > [root@PE3 ~]# mpls ilm > ILM entry label gen 10000 labelspace 0 proto ipv4 > pop peek (0 bytes, 0 pkts) > [root@PE3 ~]# mpls nhlfe > NHLFE entry key 0x00000002 mtu 1496 propagate_ttl > push gen 10000 set eth1 ipv4 10.0.0.1 (0 bytes, 0 pkts) > [root@PE3 ~]# ip route > 10.0.0.0/24 dev eth1 proto kernel scope link src 10.0.0.2 > 12.0.0.0/24 via 10.0.0.1 dev eth1 proto zebra metric 20 > 13.0.0.0/24 dev eth0 proto kernel scope link src 13.0.0.1 > [root@PE3 ~]# rpm -q quagga > quagga-0.99.6-01.fc5.mpls.1.955d > /var/log/quagga/1955d.log > ------------------------------------------------------------------------ > 2007/03/20 10:30:40 OSPF: interface 10.0.0.2 [3] join AllSPFRouters Multi= cast group. > 2007/03/20 10:30:40 ZEBRA: zebra message comes from socket [14] > 2007/03/20 10:30:40 ZEBRA: zebra message received [ZEBRA_IPV4_ROUTE_ADD] = 17 > 2007/03/20 10:30:40 ZEBRA: rib_link: rn 0x8efcf48, rib 0x8f02c70 > 2007/03/20 10:30:40 ZEBRA: rib_link: new head, rn_status copied over > 2007/03/20 10:30:41 ZEBRA: rib_process: Updating existing route, select 0= x8efcf88, fib 0x8efcf88 > 2007/03/20 10:30:43 OSPF: interface 13.0.0.1 [4] join AllSPFRouters Multi= cast group. > 2007/03/20 10:30:46 ZEBRA: zebra message comes from socket [14] > 2007/03/20 10:30:46 ZEBRA: zebra message received [ZEBRA_IPV4_ROUTE_ADD] = 17 > 2007/03/20 10:30:46 ZEBRA: rib_link: rn 0x8efd150, rib 0x8f03158 > 2007/03/20 10:30:46 ZEBRA: rib_link: new head, rn_status copied over > 2007/03/20 10:30:46 ZEBRA: rib_process: Updating existing route, select 0= x8efd190, fib 0x8efd190 > 2007/03/20 10:31:05 OSPF: ospfTrapNbrStateChange trap sent: 10.0.0.1 now = Init/DROther > 2007/03/20 10:31:20 OSPF: DR-Election[1st]: Backup 10.0.0.2 > 2007/03/20 10:31:20 OSPF: DR-Election[1st]: DR 10.0.0.2 > 2007/03/20 10:31:20 OSPF: DR-Election[2nd]: Backup 10.0.0.1 > 2007/03/20 10:31:20 OSPF: DR-Election[2nd]: DR 10.0.0.2 > 2007/03/20 10:31:20 OSPF: ospfTrapIfStateChange trap sent: 10.0.0.2 now DR > 2007/03/20 10:31:20 OSPF: interface 10.0.0.2 [3] join AllDRouters Multica= st group. > 2007/03/20 10:31:23 OSPF: DR-Election[1st]: Backup 13.0.0.1 > 2007/03/20 10:31:23 OSPF: DR-Election[1st]: DR 13.0.0.1 > 2007/03/20 10:31:23 OSPF: DR-Election[2nd]: Backup 0.0.0.0 > 2007/03/20 10:31:23 OSPF: DR-Election[2nd]: DR 13.0.0.1 > 2007/03/20 10:31:23 OSPF: ospfTrapIfStateChange trap sent: 13.0.0.1 now DR > 2007/03/20 10:31:23 OSPF: interface 13.0.0.1 [4] join AllDRouters Multica= st group. > 2007/03/20 10:31:45 OSPF: Packet[DD]: Neighbor 12.0.0.1: Initial DBD from= Slave, ignoring. > 2007/03/20 10:31:45 OSPF: DR-Election[1st]: Backup 10.0.0.1 > 2007/03/20 10:31:45 OSPF: DR-Election[1st]: DR 10.0.0.2 > 2007/03/20 10:31:50 OSPF: Packet[DD]: Neighbor 12.0.0.1 Negotiation done = (Master). > 2007/03/20 10:31:50 OSPF: ospfTrapNbrStateChange trap sent: 10.0.0.1 now = Loading/Backup > 2007/03/20 10:31:50 OSPF: nsm_change_state(12.0.0.1, Loading -> > Full): scheduling new router-LSA origination > 2007/03/20 10:31:52 ZEBRA: zebra message comes from socket [14] > 2007/03/20 10:31:52 ZEBRA: zebra message received [ZEBRA_IPV4_ROUTE_ADD] = 17 > 2007/03/20 10:31:52 ZEBRA: rib_link: rn 0x8f032a8, rib 0x8f031e0 > 2007/03/20 10:31:52 ZEBRA: rib_process: Adding route, select 0x8f031e0 > 2007/03/20 10:31:52 ZEBRA: netlink_route_multipath() (single hop): > RTM_NEWROUTE 12.0.0.0/24, type IPv4=20 > 2007/03/20 10:31:52 ZEBRA: netlink_route_multipath() (single hop): nextho= p via 10.0.0.1 if 3 > 2007/03/20 10:31:52 ZEBRA: netlink_talk: netlink-cmd type RTM_NEWROUTE(24= ), seq=3D6 > 2007/03/20 10:31:52 ZEBRA: netlink_parse_info: netlink-cmd ACK: > type=3DRTM_NEWROUTE(24), seq=3D6, pid=3D0 > 2007/03/20 10:31:52 ZEBRA: netlink_parse_info: netlink-listen type > RTM_NEWROUTE(24), seq=3D6, pid=3D4294963198 > 2007/03/20 10:31:52 ZEBRA: netlink_parse_info: netlink-cmd packet comes f= rom netlink-listen > 2007/03/20 10:32:09 ZEBRA: zebra message comes from socket [15] > 2007/03/20 10:32:09 ZEBRA: zebra message received [ZEBRA_REDISTRIBUTE_ADD= ] 1 > 2007/03/20 10:32:09 ZEBRA: zebra message comes from socket [15] > 2007/03/20 10:32:09 ZEBRA: zebra message received [ZEBRA_REDISTRIBUTE_ADD= ] 1 > 2007/03/20 10:32:09 ZEBRA: zebra message comes from socket [15] > 2007/03/20 10:32:09 ZEBRA: zebra message received [ZEBRA_REDISTRIBUTE_ADD= ] 1 > 2007/03/20 10:32:09 LDP: route 10.0.0.0/24 > 2007/03/20 10:32:09 LDP: num ifindex 0 > 2007/03/20 10:32:09 LDP: attached > 2007/03/20 10:32:09 LDP: ifindex 3 > 2007/03/20 10:32:09 LDP: add > 2007/03/20 10:32:09 LDP: route 13.0.0.0/24 > 2007/03/20 10:32:09 LDP: num ifindex 0 > 2007/03/20 10:32:09 LDP: attached > 2007/03/20 10:32:09 LDP: ifindex 4 > 2007/03/20 10:32:09 LDP: add > 2007/03/20 10:32:09 ZEBRA: zebra message comes from socket [15] > 2007/03/20 10:32:09 ZEBRA: zebra message received [ZEBRA_REDISTRIBUTE_ADD= ] 1 > 2007/03/20 10:32:09 ZEBRA: zebra message comes from socket [15] > 2007/03/20 10:32:09 ZEBRA: zebra message received [ZEBRA_REDISTRIBUTE_ADD= ] 1 > 2007/03/20 10:32:09 ZEBRA: zebra message comes from socket [15] > 2007/03/20 10:32:09 ZEBRA: zebra message received [ZEBRA_REDISTRIBUTE_ADD= ] 1 > 2007/03/20 10:32:09 ZEBRA: zebra message comes from socket [15] > 2007/03/20 10:32:09 ZEBRA: zebra message received [ZEBRA_REDISTRIBUTE_ADD= ] 1 > 2007/03/20 10:32:09 LDP: route 12.0.0.0/24 > 2007/03/20 10:32:09 LDP: num nexthop 0 > 2007/03/20 10:32:09 LDP: nexthop 10.0.0.1 > 2007/03/20 10:32:09 LDP: add > 2007/03/20 10:32:09 ZEBRA: zebra message comes from socket [15] > 2007/03/20 10:32:09 ZEBRA: zebra message received [ZEBRA_REDISTRIBUTE_ADD= ] 1 > 2007/03/20 10:32:09 ZEBRA: zebra message comes from socket [15] > 2007/03/20 10:32:09 ZEBRA: zebra message received [ZEBRA_REDISTRIBUTE_ADD= ] 1 > 2007/03/20 10:32:09 ZEBRA: zebra message comes from socket [15] > 2007/03/20 10:32:09 ZEBRA: zebra message received [ZEBRA_REDISTRIBUTE_ADD= ] 1 > 2007/03/20 10:32:09 ZEBRA: zebra message comes from socket [15] > 2007/03/20 10:32:09 ZEBRA: zebra message received [ZEBRA_REDISTRIBUTE_ADD= ] 1 > 2007/03/20 10:32:15 ZEBRA: netlink_talk: mpls-netlink-cmd type (29), seq= =3D1 > 2007/03/20 10:32:15 ZEBRA: netlink_parse_info: mpls-netlink-cmd ACK: type= =3D(29), seq=3D1, pid=3D0 > 2007/03/20 10:32:52 LDP: OUT: ldp_session_attempt_setup: MPLS_NON_BLOCKING > 2007/03/20 10:32:52 LDP: OUT: LDP_NOTIF_SESSION_REJECTED_NO_HELLO: > 2007/03/20 10:32:57 LDP: OUT: ldp_session_attempt_setup: MPLS_NON_BLOCKING > 2007/03/20 10:32:59 ZEBRA: zebra message comes from socket [15] > 2007/03/20 10:32:59 ZEBRA: zebra message received [ZEBRA_MPLS_IN_SEGMENT_= ADD] 10 > 2007/03/20 10:32:59 ZEBRA: netlink_talk: mpls-netlink-cmd type (29), seq= =3D2 > 2007/03/20 10:32:59 ZEBRA: netlink_parse_info: mpls-netlink-cmd ACK: type= =3D(29), seq=3D2, pid=3D0 > 2007/03/20 10:32:59 ZEBRA: netlink_parse_info: mpls-netlink-listen type (= 29), seq=3D0, pid=3D0 > 2007/03/20 10:32:59 ZEBRA: zebra message comes from socket [15] > 2007/03/20 10:32:59 ZEBRA: zebra message received [ZEBRA_MPLS_OUT_SEGMENT= _ADD] 39 > 2007/03/20 10:32:59 ZEBRA: netlink_talk: mpls-netlink-nhlfe type (29), se= q=3D1 > 2007/03/20 10:32:59 ZEBRA: netlink_parse_info: mpls-netlink-nhlfe type (2= 9), seq=3D1, pid=3D4294963196 > 2007/03/20 10:32:59 ZEBRA: netlink_talk: ignoring message type 0x001d > 2007/03/20 10:32:59 ZEBRA: mpls_ctrl_nhlfe_register(): NHLFE 0x00000002 > 2007/03/20 10:32:59 LDP: found pending NHLFE: 0x997e9a8 > 2007/03/20 10:32:59 ZEBRA: netlink_parse_info: mpls-netlink-listen type (= 29), seq=3D1, pid=3D4294963196 > 2007/03/20 10:32:59 ZEBRA: netlink_parse_info: mpls-netlink-nhlfe > packet comes from mpls-netlink-listen > 2007/03/20 10:32:59 ZEBRA: zebra message comes from socket [15] > 2007/03/20 10:32:59 ZEBRA: zebra message received [ZEBRA_MPLS_FTN_ADD] 11 > 2007/03/20 10:32:59 ZEBRA: IPV42 mismatch > 2007/03/20 10:32:59 ZEBRA: IPV42 mismatch > 2007/03/20 10:32:59 ZEBRA: mpls_ftn_register: unable to find FEC 10.0.0.0= /24 > -----Mensaje original----- > De: James R. Leu [mailto:jl...@mi...] > Enviado el: mar 20/03/2007 4:11 > Para: Marco Schmidt > CC: Sienes Rodr=EDguez Gonzalo; mpl...@li... > Asunto: Re: [mpls-linux-devel] LDP problems > =20 > I've added some additional debugging. In the process I was able to repro= duce > a similar failure multiple times and I found a couple place were I was > using uninitialized structures. By correctly zeroing these structures > I was able to fix the failure I was seeing. I do not presume this will > fix the failures you both are seeing, but the debugging should help > us figure what is happening. > Please download version 1.955d of quagga from sf.net > =20 > http://sourceforge.net/project/showfiles.php?group_id=3D15443&package_id= =3D12482 > On Mon, Mar 19, 2007 at 01:03:04PM +0100, Marco Schmidt wrote: >> Here is what I got so far... >>=20 >> I'm experiencing the same problems. Some connections are labeled, most >> of them are not - that differs from friday, where only one PC didn't >> work properly. >> In the non-working-case, LDPd finds the pending NHLFEs, but does nothing= more. >>=20 >> ... >> 2007/03/19 12:18:14 LDP: found pending NHLFE: 0x8a856d8 >> 2007/03/19 12:18:14 LDP: found pending NHLFE: 0x8a869a0 >> 2007/03/19 12:18:14 LDP: found pending NHLFE: 0x8a87c68 >> 2007/03/19 12:18:14 ZEBRA: zebra message comes from socket [17] >> 2007/03/19 12:18:14 ZEBRA: zebra message received [ZEBRA_MPLS_FTN_ADD] 11 >> 2007/03/19 12:18:14 ZEBRA: zebra message comes from socket [17] >> 2007/03/19 12:18:14 ZEBRA: zebra message received [ZEBRA_MPLS_FTN_ADD] 11 >> 2007/03/19 12:18:14 ZEBRA: zebra message comes from socket [17] >> 2007/03/19 12:18:14 ZEBRA: zebra message received [ZEBRA_MPLS_FTN_ADD] 11 >> whole log can be found here: >> http://www-user.tu-chemnitz.de/~smarco/MPLS/E2_no_label_binding121814.log >>=20 >>=20 >> Different reaction where the binding works properly: >>=20 >> ... >> >2007/03/19 12:07:44 LDP: found pending NHLFE: 0x9ccb2e8 >> 2007/03/19 12:07:44 ZEBRA: netlink_parse_info: mpls-netlink-listen type = (29), seq=3D1, pid=3D4294963188 >> 2007/03/19 12:07:44 ZEBRA: netlink_parse_info: mpls-netlink-nhlfe packet= comes from mpls-netlink-listen >> 2007/03/19 12:07:44 ZEBRA: zebra message comes from socket [18] >> 2007/03/19 12:07:44 ZEBRA: zebra message received [ZEBRA_MPLS_FTN_ADD] 11 >> 2007/03/19 12:07:44 ZEBRA: netlink_route_multipath() (single hop): RTM_D= ELROUTE 172.16.30.0/24, type IPv4=20 >> 2007/03/19 12:07:44 ZEBRA: netlink_route_multipath() (single hop): nexth= op via 10.0.5.1 if 2 >> 2007/03/19 12:07:44 ZEBRA: netlink_talk: netlink-cmd type RTM_DELROUTE(2= 5), seq=3D7 >> 2007/03/19 12:07:44 ZEBRA: netlink_parse_info: netlink-cmd ACK: type=3DR= TM_DELROUTE(25), seq=3D7, pid=3D0 >> 2007/03/19 12:07:44 ZEBRA: netlink_parse_info: netlink-listen type RTM_D= ELROUTE(25), seq=3D7, pid=3D4294963190 >> 2007/03/19 12:07:44 ZEBRA: netlink_parse_info: netlink-cmd packet comes = from netlink-listen >> 2007/03/19 12:07:44 ZEBRA: rib_process: Adding route, select 0x9763798 >> 2007/03/19 12:07:44 ZEBRA: netlink_route_multipath() (single hop): RTM_N= EWROUTE 172.16.30.0/24, type IPv4=20 >> 2007/03/19 12:07:44 ZEBRA: netlink_route_multipath() (single hop): nexth= op via 10.0.5.1 if 2 >> 2007/03/19 12:07:44 ZEBRA: netlink_route_multipath() (single hop): MPLS = info 00000009 >> 2007/03/19 12:07:44 ZEBRA: netlink_talk: netlink-cmd type RTM_NEWROUTE(2= 4), seq=3D8 >> 2007/03/19 12:07:44 ZEBRA: netlink_parse_info: netlink-cmd ACK: type=3DR= TM_NEWROUTE(24), seq=3D8, pid=3D0 >> >2007/03/19 12:07:44 LDP: route 172.16.30.0/24 >> >2007/03/19 12:07:44 LDP: num nexthop 0 >> >2007/03/19 12:07:44 LDP: nexthop 10.0.5.1 >> >2007/03/19 12:07:44 LDP: add >> 2007/03/19 12:07:44 ZEBRA: netlink_parse_info: netlink-listen type RTM_N= EWROUTE(24), seq=3D8, pid=3D4294963190 >> 2007/03/19 12:07:44 ZEBRA: netlink_parse_info: netlink-cmd packet comes = from netlink-listen >>=20 >> Here, the label binding appears unter "ip route show" command. >> whole log: http://www-user.tu-chemnitz.de/~smarco/MPLS/E3_120744_label_b= inding_and_zebra_crash121437.log >>=20 >> The marked operations never happen at the non-working links. >>=20 >> LDPd sometimes closes because of I/O errors, but there are no messages >> in the logfile. Zebra crashes when recieving a zebra_mpls_xc_add >> message (can be found two times in the above logfile) >>=20 >> 2007/03/19 12:14:37 ZEBRA: zebra message received [ZEBRA_MPLS_XC_ADD] 11 >> ZEBRA: Received signal 11 at 1174302877 (si_addr 0xe, PC 0x80537b1); abo= rting... >> Program counter: /usr/sbin/zebra[0x80537b1] >>=20 >> My conf-files can also be found in die MPLS directory http://www-user.tu= -chemnitz.de/~smarco/MPLS/ >>=20 >> Regards, >> Marco >>=20 >>=20 >>=20 >> > Hello everybody: >>=20 >> > I've migrated my PCs from FC4 to FC5 with all the RPMs for the >> > 1.955 version. The problem must be that I'm forgetting something or >> > doing something wrong cause I've notice that Julien or Marco were at >> > the same situacion and they both are able to run LDP. >>=20 >> > This is what I do, it's a simple scenario with two LSRs: >>=20 >> > 1. First of all I load "mpls4" and activate ip forwarding. >> > 2. I launch the daemonds by "zebra/ospfd/ldpd -d -A 127.0.0.1". >> > Without specifying the config files (though I tried it also with th= e .conf files and even >> > starting them as services ("service ... start")). >> > 3. Then I entered vtysh to type by here ALL the config. >> > -First, the OSPF networks: router ospf, network ... area 0. >> > -I wait to the new routes to be discovered and then I configured ld= p. >> > -mpls ldp, and in the corresponding interface of each LSR to the MP= LS link: >> > mpls ip and mpls labelspace 0. >>=20 >> > The OSPF routes are discovered correctly, and the NHLFE and ILM >> > entries, created properly, but I'm not able to send labelled packets >> > through the mpls link I'm trying to set up. >>=20 >> > If someone working with FC5 and LDP running could tell me his method s= hould be very useful. >>=20 >> > Thanks in advance, >> > Gonzalo. >>=20 >>=20 >>=20 >>=20 >> > ----------------------------------------------------------------------= --- >> > Take Surveys. Earn Cash. Influence the Future of IT >> > Join SourceForge.net's Techsay panel and you'll get the chance to shar= e your >> > opinions on IT & business topics through brief surveys-and earn cash >> > http://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge&CID= =3DDEVDEV >> > _______________________________________________ >> > mpls-linux-devel mailing list >> > mpl...@li... >> > https://lists.sourceforge.net/lists/listinfo/mpls-linux-devel >>=20 >>=20 >>=20 >>=20 >>=20 >> ------------------------------------------------------------------------- >> Take Surveys. Earn Cash. Influence the Future of IT >> Join SourceForge.net's Techsay panel and you'll get the chance to share = your >> opinions on IT & business topics through brief surveys-and earn cash >> http://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge&CID= =3DDEVDEV >> _______________________________________________ >> mpls-linux-devel mailing list >> mpl...@li... >> https://lists.sourceforge.net/lists/listinfo/mpls-linux-devel |
