#10 netstat parses cmdline and can get confused by processes changing it

[Toralf Förster]

Well, I'm wondering about the following output of netstat - I know that PID 20673 belongs to the project Climateprediction and not to Einstein@Home, but the network connection indicates a different picture (I added the process table output too)

n22 ~ # date; netstat --tcp --udp --program ; netstat --tcp --udp --program -n
Sun Oct 28 12:53:17 CET 2012
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 361376 d177047.adsl.hans:41344 uploader1.atm.ox.a:http ESTABLISHED 20643/boinc_client
tcp 1 0 g224047205.adsl.a:39844 einstein.phys.uwm.:http CLOSE_WAIT 20673/../../project
tcp 0 0 d177047.adsl.hans:42446 news.eternal-sept:nntps ESTABLISHED 21228/thunderbird
tcp 1 0 g224047205.adsl.a:50773 www.worldcommunity:http CLOSE_WAIT 20673/../../project
tcp 0 0 localhost:31416 localhost:53802 ESTABLISHED 20643/boinc_client
tcp 0 0 d177047.adsl.hans:42447 news.eternal-sept:nntps ESTABLISHED 21228/thunderbird
tcp 0 0 localhost:53802 localhost:31416 ESTABLISHED 26243/boincmgr
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 361376 80.171.177.47:41344 163.1.242.30:80 ESTABLISHED 20643/boinc_client
tcp 1 0 92.224.47.205:39844 129.89.61.70:80 CLOSE_WAIT 20673/../../project
tcp 0 0 80.171.177.47:42446 88.198.244.100:563 ESTABLISHED 21228/thunderbird
tcp 1 0 92.224.47.205:50773 198.20.8.246:80 CLOSE_WAIT 20673/../../project
tcp 0 0 127.0.0.1:31416 127.0.0.1:53802 ESTABLISHED 20643/boinc_client
tcp 0 0 80.171.177.47:42447 88.198.244.100:563 ESTABLISHED 21228/thunderbird
tcp 0 0 127.0.0.1:53802 127.0.0.1:31416 ESTABLISHED 26243/boincmgr

n22 ~ # ps -efla | grep 20673
0 S boinc 20673 20643 0 99 - - 7434 hrtime 10:32 ? 00:00:28 ../../projects/climateprediction.net/hadam3p_eu_6.09_i686-pc-linux-gnu hadam3p_eu_918a_1974_1_008220454 atmos_918a_1974_1_008220454_0 eu_918a_1974_1_008220454_0 ic00000000_10_N96 HadISST_SST_N96_1974_12_1977_01f HadISST_SI_N96_1974_12_1977_01f so2dms_N96_1974_12_1977_02 oxi.addfa o3_A2_1959_2010_N96_f.anc
0 S boinc 20691 20673 41 99 - - 47254 hrtime 10:32 ? 00:58:40 /var/lib/boinc/projects/climateprediction.net/hadam3p_eu_um_6.09_i686-pc-linux-gnu 156315 /var/lib/boinc/slots/4 atmos_918a_1974_1_008220454_0 ic00000000_10_N96 HadISST_SST_N96_1974_12_1977_01f HadISST_SI_N96_1974_12_1977_01f so2dms_N96_1974_12_1977_02 oxi.addfa o3_A2_1959_2010_N96_f.anc
0 R boinc 20692 20673 51 99 - - 35168 - 10:32 ? 01:12:02 /var/lib/boinc/projects/climateprediction.net/hadrm3p_eu_um_6.09_i686-pc-linux-gnu 156315 /var/lib/boinc/slots/4
0 S root 26280 26258 0 80 0 - 1173 pipe_w 12:53 pts/1 00:00:00 grep --colour=auto 20673
n22 ~ #

n22 ~ # ps -efla | grep 20643
1 S boinc 20643 1 0 99 19 - 9057 poll_s 10:30 ? 00:00:24 /usr/bin/boinc_client --start_delay 120 --daemon --dir /var/lib/boinc --redirectio
0 S boinc 20673 20643 0 99 - - 7690 hrtime 10:32 ? 00:00:28 ../../projects/climateprediction.net/hadam3p_eu_6.09_i686-pc-linux-gnu hadam3p_eu_918a_1974_1_008220454 atmos_918a_1974_1_008220454_0 eu_918a_1974_1_008220454_0 ic00000000_10_N96 HadISST_SST_N96_1974_12_1977_01f HadISST_SI_N96_1974_12_1977_01f so2dms_N96_1974_12_1977_02 oxi.addfa o3_A2_1959_2010_N96_f.anc
0 S boinc 20674 20643 0 99 - - 7386 hrtime 10:32 ? 00:00:09 ../../projects/climateprediction.net/hadam3p_eu_6.09_i686-pc-linux-gnu hadam3p_eu_65vh_2002_1_008220332 atmos_65vh_2002_1_008220332_0 eu_65vh_2002_1_008220332_0 ic00000000_10_N96 HadISST_SST_N96_2002_12_2005_01f HadISST_SI_N96_2002_12_2005_01f so2dms_N96_2002_12_2004_02f oxi.addfa o3_A2_1959_2010_N96_f.anc
0 S boinc 20684 20643 1 99 - - 524 hrtime 10:32 ? 00:02:05 ../../projects/www.worldcommunitygrid.org/wcgrid_sn2s_vina_6.20_i686-pc-linux-gnu -jobfile SN2S_AAM43940_0000209_0163.job -inputfile SN2S_AAM43940_0000209_0163.zip -seed 2074770840
0 R boinc 22742 20643 93 99 - - 8630 - 11:58 ? 00:58:07 ../../projects/www.worldcommunitygrid.org/wcg_hcc1_img_6.56_i686-pc-linux-gnu X0900071340418200607071703.jp2
0 S root 26463 26258 0 80 0 - 1173 pipe_w 13:00 pts/1 00:00:00 grep --colour=auto 20643

[Bernd Eckenfels]

I am actually not sure I understand the report. From the netstat output, the PID does connect to the host einstein-something, I dont know a reason to believe that output is wrong? The only problem we have with the process names is the shortened/relative name of the binary. But using ps and the pid should be reliable.

(Sorry for followup so late, I thought I understood the report (problem with the process name), but it seems I was wrong. Is this still a problem, if yes can you explain it again and add distribution and version info?)

[David W. Hodgins]

I think the problem is on line 410 of netstat.c which has ...
if ((cmdlp = strrchr(cmdlbuf, '/')))

I'm seeing this bug with google chrome, which is showing ...
# netstat -pvlA inet,inet6|grep mdns
udp 0 0 224.0.0.251:mdns 0.0.0.0:* 15762/122-high-temp
# cat /proc/15762/cmdline
/opt/google/chrome/chrome https://www.cbc.ca/news/canada/british-columbia/122-high-temperature-records-broken-in-b-c-this-week-1.5065585?cmp=rs

Line 410 searches for the last slash, assuming it will be followed by the program name. When the
program has parameters that include a slash, the beginning of the parm following the last slash is returned instead of the program name.

[David W. Hodgins]

With kernel version 2.6.33 and later, instead of reading /proc/$pid/cmdline, the program name
can be obtained from /proc/$pid/comm, so the simplest change would be to replace line 247

define PATH_CMDLINE "cmdline"

with

define PATH_CMDLINE "comm"

I haven't tested this fix though.

[David W. Hodgins]

Attaching untested patch

[Bernd Eckenfels]

Ah yes you are probably right comm is the beter source for this case.

I think there have been some discussion around that before (which I cannot find) regarding possible argv[0] changes of running programs as well as symlinks. Did you notice any changed names with your patch besides the one in question here?

But as long as we do not want to try to pass the cmdline correctly (which I guess is not even possible as the quotes are not preserved) switching to comm would be better (I guess I would root for changing the define name as well or removing it completely).

[David W. Hodgins]

I've reported the bug for Mageia linux as https://bugs.mageia.org/show_bug.cgi?id=24556
I'm the qa team leader for Mageia, with a background in mainframe software development. While I can read the code to figure out what's happening and sometimes create patches to fix a
problem, applying a patch to is usually best left to others. I'll update this bug report with the
results when testing has been completed.