#7 Empty device files created with numbers appended; causes errors when plotting graphs

[d3]

This issue seems to happen in Redhat 3, 5, 6, and in Solaris 9 / 10.

After running sar2ascii daily for several months, we have noticed that some of our hosts will have empty device files in their report folders.

The files are named after an existing device, but they will have extra 4-5 numbers on the end of the name. So, instead of just "b.1", we will also see an empty file called "b.1.9023".

When these empty files appear, plotting graphs for that host is impossible- when you try to plot graphs, you will get a "FPDF Error: End of Stream" message before the process is finished. We have to delete the empty files for plotting to work again.

What we cannot figure out, is why or how these empty files are generated. The files I see this happen the most are b, B, n_ but several others also appear from time to time.

When running sar2ascii manually on an affected host and looking at the files generated (/tmp/sar2html as well as opening the .tar.gz file) I do not see the empty files there. Importing the data with sar2html does not create the empty files either. So, at some point the empty files are created and in later runs, the problem corrects itself.

The easy workaround is to delete any empty files in the report folders.

-rw-r--r-- 1 sysalert sysalert 1416909 Jan 24 04:42 n_DEV.2--eth1
-rw-r--r-- 1 sysalert sysalert 1407960 Jan 24 04:42 n_DEV.2--lo
-rw-r--r-- 1 sysalert sysalert       0 Jan 24 04:42 n_DEV.2--lo.2418
-rw-r--r-- 1 sysalert sysalert       0 Jan 24 04:42 n_DEV.2--lo.2420
-rw-r--r-- 1 sysalert sysalert 1407735 Jan 24 04:42 n_DEV.2--sit0

Have you seen this behavior at all on your end? If you leave this ticket open, I can update when I see the behavior re-occur and try to pin down what had changed since then.

Extra cautious way to delete the empty device files from inside the sarDATA folder:

find . -mindepth 4 -maxdepth 4 ( -name 'report' -o -name 'uPLOAD' ) -prune -o -name '..*' -size 0 -type f -print | xargs rm

[CemTan]

Please share collected data (sar2html-<hostname>-<date>.tar.gz file) with me if you have.

[CemTan]

sar2ascii creates and uses temporary files while collecting performance data. Name of these files are unique since they are named by using current process id which they are 2418 and 2420 in your example. sar2ascii should be able to remove them after the data in these files are ported to actual report file which is n_DEV.2--lo in your example.

It seems sar2ascii could not remove them since they were in use. Maybe I should add another control line to sar2ascii to be sure that these files are removed.

Need to check.

[d3]

Aha, that makes sense. In the sarDATA folder on our main server, I see some empty folders named like "sar2html.{pid}" alongside the OS folders. I'll bet those folders belonged to the servers that had this issue.

[root@server sar2html]# ls -l sarDATA
total 56
drwxr-xr-x 7 sysalert sysalert 1024 Jan 27 12:25 Redhat_3
drwxr-xr-x 78 sysalert sysalert 6144 Jan 26 11:52 Redhat_5
drwxr-xr-x 55 sysalert sysalert 4096 Feb 3 11:29 Redhat_6
drwxr-xr-x 2 sysalert sysalert 80 Feb 1 04:50 sar2html.15588
drwxr-xr-x 2 sysalert sysalert 80 Jan 29 04:38 sar2html.27126
drwxr-xr-x 2 sysalert sysalert 80 Feb 2 06:09 sar2html.27537
drwxr-xr-x 2 sysalert sysalert 80 Jan 31 04:49 sar2html.4344
drwxr-xr-x 2 sysalert sysalert 80 Jan 31 04:39 sar2html.7887
drwxr-xr-x 14 sysalert sysalert 2048 Jan 31 17:33 SunOS_10
drwxr-xr-x 6 sysalert sysalert 1024 Jul 10 2013 SunOS_9
drwxrwxrwt 2 sysalert sysalert 13312 Feb 4 05:43 uPLOAD

I will be watching our job tonight and comparing files, see if I can get you some better info on what happens.

[d3]

Ok, I finally caught an example of this in action.

It seems that these phantom files are created on the "sar2html merge" side. The phantom files are not present in the tar files that were created by sar2ascii.

Attached are the sar2ascii tar files generated this past weekend, and the folder with existing data that they were merging to. The phantom files appeared sometime this weekend.

An example log entry from when my sar2html merge job runs:

Merging dump files...

/u003/app/sar2html/sarDATA//uPLOAD/sar2html-ltmqpa01-1402170130.tar.gz
cat: /u003/app/sar2html/sarDATA/sar2html.12568/sar2html/Redhat_6/ltmqpa01-1facf3fb/report/b.1.12566: No such file or directory