#10 Show correct network interfaces data rate in csv file when virtual network interfaces are added/removed dynamically.

[Fernando]

When a virtual intarface is added to the system the Linux Kernel insert a line it in /proc/net/dev randomly e.g.
Before adding a virtual device 'cat/proc/net/dev':

Inter-|   Receive                                                |  Transmit
 face |bytes    packets errs drop fifo frame compressed multicast|bytes    packets errs drop fifo colls carrier compressed
 ens37:       0       0    0    0    0     0          0         0  1978437   11099    0    0    0     0       0          0
 ens33: 6813968   94300    0    0    0     0          0         0 630076169  429834    0    0    0     0       0          0
veth94d5a9e:  494984    6551    0    0    0     0          0         0   409191    2926    0    0    0     0       0          0
    lo:   33087     377    0    0    0     0          0         0    33087     377    0    0    0     0       0          0
docker0:  403270    6551    0    0    0     0          0         0   407127    2913    0    0    0     0       0          0

After adding a virtual devic '/proc/net/dev':

Inter-|   Receive                                                |  Transmit
 face |bytes    packets errs drop fifo frame compressed multicast|bytes    packets errs drop fifo colls carrier compressed
  tun0:       0       0    0    0    0     0          0         0      304       4    0    0    0     0       0          0
 ens37:       0       0    0    0    0     0          0         0  1978437   11099    0    0    0     0       0          0
 ens33: 6813968   94300    0    0    0     0          0         0 630076169  429834    0    0    0     0       0          0
veth94d5a9e:  494984    6551    0    0    0     0          0         0   409191    2926    0    0    0     0       0          0
    lo:   33087     377    0    0    0     0          0         0    33087     377    0    0    0     0       0          0
docker0:  403270    6551    0    0    0     0          0         0   407127    2913    0    0    0     0       0          0

Linux added the new interface (tun0) at the top. This leads nmon to calculate wrong data rates for each of the interfaces due to the shift in /proc/net/dev added by the new interface.

Here an example of the output of nmon by grepping 'NET,' for the case example above:

NET,T1137,0.0,0.0,0.0,0.0,0.0,0.1,0.0,0.0,0.0,0.0
NET,T1138,0.0,0.0,6059.6,444.1,0.0,1.5,0.0,0.0,602823.9,360.9,0.0,0.9
NET,T1139,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
NET,T1140,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
NET,T1141,0.0,6534.3,0.0,0.0,355.9,1923.6,601193.9,0.0,0.0,358.9
NET,T1142,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0                     

Interface tun0 was added before "NET,T1138" and removed at "NET,T1140". After adding a interface the data rate are not correct, there is nothing going on on the network.

To avoid it I did the following, probably not the best solution, but it is an idea:

--- lmon16i.c   2020-04-20 14:20:45.000000000 +0200
+++ lmon16i_new.c       2020-04-20 14:49:27.000000000 +0200
@@ -3788,6 +3788,18 @@
     networks = i;
 }

+/* Fernando Addition */
+struct net_stat * get_last_net_stat(const char * if_name)
+{

+       int i = 0;
+       for (i = 0; i < networks; i++) {
+               if (strcmp(if_name, (const char *) q->ifnets[i].if_name) == 0) {
+                       return &q->ifnets[i];
+               }
+       }
+       return (struct net_stat *) NULL;
+}
+/**/

 int proc_procsinfo(int pid, int index)
 {
@@ -4283,6 +4295,7 @@
     char * slabstr;
     char truncated_command[257]; /* 256 +1 */


+    struct net_stat * q_netstat = (struct net_stat *) NULL;  /* Fernando Addition */

 #define MAXROWS 256
 #define MAXCOLS 150            /* changed to allow maximum column widths */
@@ -4842,7 +4855,8 @@
                fprintf(fp,"FILE,File I/O %s,iget,namei,dirblk,readch,writech,ttyrawch,ttycanch,ttyoutch\n", run_name);
 */

-

+       // Fernando Addition
+       fprintf(fp, "NETDEV,Network device statistics,interface name,rx bit/s,rx-errs bit/s,rx-drop bit/s,tx bit/s,tx-errs bit/s,tx-drop bit/s\n");
        fprintf(fp, "NET,Network I/O %s", run_name);
        for (i = 0; i < networks; i++)
            fprintf(fp, ",%-2s-read-KB/s", (char *) p->ifnets[i].if_name);
@@ -7314,6 +7328,24 @@
            }
            DISPLAY(padnet, networks + 2);
            if (!cursed) {
+               /* Fernando Addition */
+               for (i = 0; i < networks; i++) {
+                       q_netstat = get_last_net_stat((char *) p->ifnets[i].if_name);
+#define IFDELTA_LAST(member) ((float)( (q_netstat->member > p->ifnets[i].member) ? 0 : (p->ifnets[i].member - q_netstat->member)/elapsed) )
+                       if (q_netstat) {
+                               fprintf(fp, "NETDEV,%s,%s,%i,%i,%i,%i,%i,%i\n",
+                                       LOOP,
+                                       (char *) p->ifnets[i].if_name,
+                                       (int) (IFDELTA_LAST(if_ibytes) * 8),
+                                       (int) (IFDELTA_LAST(if_ierrs) * 8),
+                                       (int) (IFDELTA_LAST(if_idrop) * 8),
+                                       (int) (IFDELTA_LAST(if_obytes) * 8),
+                                       (int) (IFDELTA_LAST(if_oerrs) * 8),
+                                       (int) (IFDELTA_LAST(if_odrop) * 8));
+                       }
+               }
+               /**/
+
                fprintf(fp,
                        show_rrd ? "rrdtool update net.rrd %s" : "NET,%s",
                        LOOP);

This would produce the next output (doing a grep to 'NETDEV') when adding/removing interfaces:

NETDEV,T1455,ens37,0,0,0,2646,0,0
NETDEV,T1455,ens33,464,0,0,0,0,0
NETDEV,T1455,veth94d5a9e,0,0,0,0,0,0
NETDEV,T1455,lo,0,0,0,0,0,0
NETDEV,T1455,docker0,0,0,0,0,0,0
NETDEV,T1456,tun0,0,0,0,1187,0,0
NETDEV,T1456,ens37,0,0,0,0,0,0
NETDEV,T1456,ens33,0,0,0,0,0,0
NETDEV,T1456,veth94d5a9e,0,0,0,0,0,0
NETDEV,T1456,lo,0,0,0,0,0,0
NETDEV,T1456,docker0,0,0,0,0,0,0
NETDEV,T1457,tun0,0,0,0,0,0,0
NETDEV,T1457,ens37,0,0,0,0,0,0
NETDEV,T1457,ens33,0,0,0,0,0,0
NETDEV,T1457,veth94d5a9e,0,0,0,0,0,0
NETDEV,T1457,lo,0,0,0,0,0,0
NETDEV,T1457,docker0,0,0,0,0,0,0
NETDEV,T1458,ens37,0,0,0,0,0,0
NETDEV,T1458,ens33,0,0,0,0,0,0
NETDEV,T1458,veth94d5a9e,0,0,0,0,0,0
NETDEV,T1458,lo,0,0,0,0,0,0

There will be a line per interface.

[Nigel Griffiths]

Hi Fernando,
Interesting idea.
nmon does not support dynamic changing of resources like disks, networks, filesystems.
It takes too many CPU cycle to straighten the mess out.
Your output format will not work with the nmon Analyser, nmonchart or any other nmon graphing tool I am aware of.
You could add this sort of function using nmon external data collectors.
See https://www.ibm.com/support/pages/nmon-and-external-data-collectors

However my new tool njmon or nimon will handle this and let you save the data to a Time-Series database for graphing thus removing other nmon problems of data management.
See: http://nmon.sourceforge.net/pmwiki.php?n=Site.Njmon

[Fernando]

Hi Nigel,

Yes, I used to collect this information using the external data collectors. My problem there was that I collected the current output of cat /proc/net/dev which contains only the amount of bytes being trasmitted, received, etc. but not the actual datarate. So I had to do some post processing to calculate the datarate.

I have also looked at njmon, it's a great tool but I like the csv format. Just grepping e.g. CPU_ALL gives me an idea about what is the CPU doing. I normally have a quick look at the nmon logs in awith the Linux terminal. But also the postprocessing with Excel is very easy.

Another Idea would be to keep the format flat but including the name of the interface e.g.:

NETNEW,T1137,eth0,0.0,0.0,lo,0.0,0.0,ens33,0.0,0.1,ens37,0.0,0.0,tun0,0.0,0.0

Thanks for the quick hints :-)

[Nigel Griffiths]

Fixing these types of issues which also effects disks and filesystems is a massive project and rewrite. njmon is the better tool and does not suffer the problems.