With quotes, collectl processes all files, subject to "--from"
parameter
It doesn't matter whether you use single or double quotes
Presence/absence of "--from" parameter makes no difference.
Except with "--export vmstat": collectl will NOT process multiple
files -- at the end of the first file the message "Error: -s not allowed
with 'vmstat'" is output
The error with vmstat export appears to be caused by the check in
vmstatInit. After removing the check, the vmstat export works similarly to
other formats.
So, two questions:
- Where does the "phantom" "-s" flag come from?
- Why does vmstatInit check it?
Hope this helps. For now, I'm running with a patched version of collectl,
but would very much like to not do that. The only other alternative is a
very messy solution involving using a for-loop and appending to result file.
With quotes, collectl processes all files, subject to "--from"
parameter
It doesn't matter whether you use single or double quotes
Presence/absence of "--from" parameter makes no difference.
Except with "--export vmstat": collectl will NOT process multiple
files -- at the end of the first file the message "Error: -s not allowed
with 'vmstat'" is output
The error with vmstat export appears to be caused by the check
in vmstatInit. After removing the check, the vmstat export works similarly
to other formats.
So, two questions:
- Where does the "phantom" "-s" flag come from?
- Why does vmstatInit check it?
Hope this helps. For now, I'm running with a patched version of collectl,
but would very much like to not do that. The only other alternative is a
very messy solution involving using a for-loop and appending to result file.
Seems it is going to always be called and will check user switches.
It's always going to log that message as the test is straight forward
because $userSubsys will not be empty
This first section gets called almost immediately by collectl after reading
in the various user switches. This is the place to catch switch errors and
since this routine always requires -scm we'll just hardcode it to that and
reject any user entered ones. This initialization subroutine must be named
for our module followed by Init.
sub vmstatInit
{
error("-s not allowed with 'vmstat'") if $userSubsys ne '';
error("-f requires either --rawtoo or -P") if $filename ne '' &&
!$rawtooFlag && !$plotFlag;
error("-P or --rawtoo require -f") if $filename eq '' &&
($rawtooFlag || $plotFlag);
$subsys=$userSubsys='cm'; ** Hardcoded here
}
I think this issue has been there always and you caught it because of the
wildcard in quotes.
I am going to remove that error check and release a fixed version because
it serves no purpose.
New code looks like this. can you modify your vmstat.ph in
/usr/share/collectl and test this and confirm its good for you
sub vmstatInit
{
error("-f requires either --rawtoo or -P") if $filename ne '' &&
!$rawtooFlag && !$plotFlag;
error("-P or --rawtoo require -f") if $filename eq '' &&
($rawtooFlag || $plotFlag);
$subsys=$userSubsys='cm';
}
If you are good I will release the fix.
I want it tested by you because it is a seldom used option so if you are
happy I am happy.
Have run some tests, and my results seem to indicate that:
Without quotes, wildcarding the "-p" param results in only the
first file being processed (bash guarantees that filenames are returned
in alpha order
https://serverfault.com/questions/122737/in-bash-are-wildcard-expansions-guaranteed-to-be-in-order
- With quotes, collectl processes all files, subject to "--from"
parameter
- It doesn't matter whether you use single or double quotes
- Presence/absence of "--from" parameter makes no difference.
- Except with "--export vmstat": collectl will NOT process multiple
files -- at the end of the first file the message "Error: -s not
allowed
with 'vmstat'" is output
- The error with vmstat export appears to be caused by the check
in vmstatInit. After removing the check, the vmstat export works
similarly
to other formats.
So, two questions:
- Where does the "phantom" "-s" flag come from?
- Why does vmstatInit check it?
Hope this helps. For now, I'm running with a patched version of collectl,
but would very much like to not do that. The only other alternative is a
very messy solution involving using a for-loop and appending to result
file.
Seems it is going to always be called and will check user switches.
It's always going to log that message as the test is straight forward
because $userSubsys will not be empty
Example here with DEBUG and why it happens
[loberman@lobep17 collectl]$ time /usr/bin/collectl -p '
This first section gets called almost immediately by collectl after reading
in the various user switches. This is the place to catch switch errors and
since this routine always requires -scm we'll just hardcode it to that and
reject any user entered ones. This initialization subroutine must be named
for our module followed by Init.
sub vmstatInit
{
error("-s not allowed with 'vmstat'") if $userSubsys ne '';
error("-f requires either --rawtoo or -P") if $filename ne '' &&
!$rawtooFlag && !$plotFlag;
error("-P or --rawtoo require -f") if $filename eq '' &&
($rawtooFlag || $plotFlag);
$subsys=$userSubsys='cm';
**** Hardcoded here
}
I think this issue has been there always and you caught it because of the
wildcard in quotes.
I am going to remove that error check and release a fixed version because
it serves no purpose.
New code looks like this. can you modify your vmstat.ph in
/usr/share/collectl and test this and confirm its good for you
sub vmstatInit
{
error("-f requires either --rawtoo or -P") if $filename ne '' &&
!$rawtooFlag && !$plotFlag;
error("-P or --rawtoo require -f") if $filename eq '' &&
($rawtooFlag || $plotFlag);
$subsys=$userSubsys='cm';
}
If you are good I will release the fix.
I want it tested by you because it is a seldom used option so if you are
happy I am happy.
On Tue, Nov 29, 2022 at 2:36 PM Bill Torpey
wallstprog@users.sourceforge.net %0Dwallstprog@users.sourceforge.net
wrote:
Hi -- I'm back...
Have run some tests, and my results seem to indicate that:
Without quotes, wildcarding the "-p" param results in only the
first file being processed (bash guarantees that filenames are returned
in alpha order
https://serverfault.com/questions/122737/in-bash-are-wildcard-expansions-guaranteed-to-be-in-order
- With quotes, collectl processes all files, subject to "--from"
parameter
- It doesn't matter whether you use single or double quotes
- Presence/absence of "--from" parameter makes no difference.
- Except with "--export vmstat": collectl will NOT process multiple
files -- at the end of the first file the message "Error: -s not
allowed
with 'vmstat'" is output
- The error with vmstat export appears to be caused by the check
in vmstatInit. After removing the check, the vmstat export works
similarly
to other formats.
So, two questions:
- Where does the "phantom" "-s" flag come from?
- Why does vmstatInit check it?
Hope this helps. For now, I'm running with a patched version of collectl,
but would very much like to not do that. The only other alternative is a
very messy solution involving using a for-loop and appending to result
file.
Bill - I guess the one overall question I'd ask is about your general use
of collectl and why you have a handful of smallisk files you need to play
back together and also way none of your files are compressed?
I had designed collectl to very efficient in terms of cpu and storage. When
I installed collectl on any system, I'd just turn it on and leave it
running forever. For that reason, files are automatically purged after a
week (configurable of course). When I want to diagnose a problem, I'd first
plot everything with colplot, selecting timeframes of interest. Then if I
wanted more details I'd play back the file for that day with the
appropriate form/thru switches. Colplot is certainly optional, but I'd find
it to be most useful when I wasn't sure what I was looking for.
As for compressed files, when I first wrote colplot, you had to have text
based files to feed into gnuplot. later on, gnuplot supported plotting
compressed files and to my delight was almost as efficient. I'm guessing
because the files were something like 10% the size and so much less I/O
required to read them.
In any event, I'm always happy to here collectl lives on and to see people
using it in different ways. As it turns out I think you're the first person
I remember who found vmstat format useful, possibly why that bug you
uncovered took over 20 years to find! ;)
Have run some tests, and my results seem to indicate that:
Without quotes, wildcarding the "-p" param results in only the
first file being processed (bash guarantees that filenames are returned
in alpha order
https://serverfault.com/questions/122737/in-bash-are-wildcard-expansions-guaranteed-to-be-in-order
- With quotes, collectl processes all files, subject to "--from"
parameter
- It doesn't matter whether you use single or double quotes
- Presence/absence of "--from" parameter makes no difference.
- Except with "--export vmstat": collectl will NOT process multiple
files -- at the end of the first file the message "Error: -s not
allowed
with 'vmstat'" is output
- The error with vmstat export appears to be caused by the check
in vmstatInit. After removing the check, the vmstat export works
similarly
to other formats.
So, two questions:
- Where does the "phantom" "-s" flag come from?
- Why does vmstatInit check it?
Hope this helps. For now, I'm running with a patched version of collectl,
but would very much like to not do that. The only other alternative is a
very messy solution involving using a for-loop and appending to result
file.
Here at Red Hat our use case is two fold..
If a customer reports a performance problem or any issue needing a deep
dive, Install collect, let it gather and then investigate
various performance problems after the customer reports the timestamps.
The historical capture is very important.
We also use it live if needed to do monitoring on a live performance
problem.
collectl -scmnd -oT -i1 used a lot
collectl -sc --verbose -oT -i1 also used a lot
collectl -sD -oT -i1 --dskfilt xxxx
I have also been using this a very long time with Mark, helping him with
various Linux internals questions in the past.
It's certainly important enough to maintain so I took on the responsibility.
I have developed a graphing tool we use internally so we don't use colplot
and have not in the past had to use colmux.
I know that Mark used to use colmux a lot.
My tool uses python and pandas dataframes with matplotlib
Bill - I guess the one overall question I'd ask is about your general use
of collectl and why you have a handful of smallisk files you need to play
back together and also way none of your files are compressed?
I had designed collectl to very efficient in terms of cpu and storage. When
I installed collectl on any system, I'd just turn it on and leave it
running forever. For that reason, files are automatically purged after a
week (configurable of course). When I want to diagnose a problem, I'd first
plot everything with colplot, selecting timeframes of interest. Then if I
wanted more details I'd play back the file for that day with the
appropriate form/thru switches. Colplot is certainly optional, but I'd find
it to be most useful when I wasn't sure what I was looking for.
As for compressed files, when I first wrote colplot, you had to have text
based files to feed into gnuplot. later on, gnuplot supported plotting
compressed files and to my delight was almost as efficient. I'm guessing
because the files were something like 10% the size and so much less I/O
required to read them.
In any event, I'm always happy to here collectl lives on and to see people
using it in different ways. As it turns out I think you're the first person
I remember who found vmstat format useful, possibly why that bug you
uncovered took over 20 years to find! ;)
On Tue, Nov 29, 2022 at 2:36 PM Bill Torpey
wallstprog@users.sourceforge.net %0Dwallstprog@users.sourceforge.net
wrote:
Hi -- I'm back...
Have run some tests, and my results seem to indicate that:
Without quotes, wildcarding the "-p" param results in only the
first file being processed (bash guarantees that filenames are returned
in alpha order
https://serverfault.com/questions/122737/in-bash-are-wildcard-expansions-guaranteed-to-be-in-order
- With quotes, collectl processes all files, subject to "--from"
parameter
- It doesn't matter whether you use single or double quotes
- Presence/absence of "--from" parameter makes no difference.
- Except with "--export vmstat": collectl will NOT process multiple
files -- at the end of the first file the message "Error: -s not
allowed
with 'vmstat'" is output
- The error with vmstat export appears to be caused by the check
in vmstatInit. After removing the check, the vmstat export works
similarly
to other formats.
So, two questions:
- Where does the "phantom" "-s" flag come from?
- Why does vmstatInit check it?
Hope this helps. For now, I'm running with a patched version of collectl,
but would very much like to not do that. The only other alternative is a
very messy solution involving using a for-loop and appending to result
file.
Here at Red Hat our use case is two fold..
If a customer reports a performance problem or any issue needing a deep
dive, Install collect, let it gather and then investigate
various performance problems after the customer reports the timestamps.
The historical capture is very important.
We also use it live if needed to do monitoring on a live performance
problem.
collectl -scmnd -oT -i1 used a lot
collectl -sc --verbose -oT -i1 also used a lot
collectl -sD -oT -i1 --dskfilt xxxx
I have also been using this a very long time with Mark, helping him with
various Linux internals questions in the past.
It's certainly important enough to maintain so I took on the
responsibility.
I have developed a graphing tool we use internally so we don't use colplot
and have not in the past had to use colmux.
I know that Mark used to use colmux a lot.
My tool uses python and pandas dataframes with matplotlib
Regards
Laurence
On Thu, Dec 1, 2022 at 8:44 AM Mark Seger markseger@users.sourceforge.net
wrote:
Bill - I guess the one overall question I'd ask is about your general use
of collectl and why you have a handful of smallisk files you need to play
back together and also way none of your files are compressed?
I had designed collectl to very efficient in terms of cpu and storage. When
I installed collectl on any system, I'd just turn it on and leave it
running forever. For that reason, files are automatically purged after a
week (configurable of course). When I want to diagnose a problem, I'd first
plot everything with colplot, selecting timeframes of interest. Then if I
wanted more details I'd play back the file for that day with the
appropriate form/thru switches. Colplot is certainly optional, but I'd find
it to be most useful when I wasn't sure what I was looking for.
As for compressed files, when I first wrote colplot, you had to have text
based files to feed into gnuplot. later on, gnuplot supported plotting
compressed files and to my delight was almost as efficient. I'm guessing
because the files were something like 10% the size and so much less I/O
required to read them.
In any event, I'm always happy to here collectl lives on and to see people
using it in different ways. As it turns out I think you're the first person
I remember who found vmstat format useful, possibly why that bug you
uncovered took over 20 years to find! ;)
-mark
On Tue, Nov 29, 2022 at 5:53 PM Laurence Oberman
loberman@users.sourceforge.net %0Dloberman@users.sourceforge.net wrote:
On Tue, Nov 29, 2022 at 2:36 PM Bill Torpey
wallstprog@users.sourceforge.net %0Dwallstprog@users.sourceforge.net
wrote:
Hi -- I'm back...
Have run some tests, and my results seem to indicate that:
Without quotes, wildcarding the "-p" param results in only the
first file being processed (bash guarantees that filenames are returned
in alpha order
https://serverfault.com/questions/122737/in-bash-are-wildcard-expansions-guaranteed-to-be-in-order
- With quotes, collectl processes all files, subject to "--from"
parameter
- It doesn't matter whether you use single or double quotes
- Presence/absence of "--from" parameter makes no difference.
- Except with "--export vmstat": collectl will NOT process multiple
files -- at the end of the first file the message "Error: -s not
allowed
with 'vmstat'" is output
- The error with vmstat export appears to be caused by the check
in vmstatInit. After removing the check, the vmstat export works
similarly
to other formats.
So, two questions:
- Where does the "phantom" "-s" flag come from?
- Why does vmstatInit check it?
Hope this helps. For now, I'm running with a patched version of collectl,
but would very much like to not do that. The only other alternative is a
very messy solution involving using a for-loop and appending to result
file.
Here at Red Hat our use case is two fold..
If a customer reports a performance problem or any issue needing a deep
dive, Install collect, let it gather and then investigate
various performance problems after the customer reports the timestamps.
The historical capture is very important.
We also use it live if needed to do monitoring on a live performance
problem.
collectl -scmnd -oT -i1 used a lot
collectl -sc --verbose -oT -i1 also used a lot
collectl -sD -oT -i1 --dskfilt xxxx
I have also been using this a very long time with Mark, helping him with
various Linux internals questions in the past.
It's certainly important enough to maintain so I took on the
responsibility.
I have developed a graphing tool we use internally so we don't use colplot
and have not in the past had to use colmux.
I know that Mark used to use colmux a lot.
My tool uses python and pandas dataframes with matplotlib
Regards
Laurence
On Thu, Dec 1, 2022 at 8:44 AM Mark Seger markseger@users.sourceforge.net
wrote:
Bill - I guess the one overall question I'd ask is about your general use
of collectl and why you have a handful of smallisk files you need to play
back together and also way none of your files are compressed?
I had designed collectl to very efficient in terms of cpu and storage. When
I installed collectl on any system, I'd just turn it on and leave it
running forever. For that reason, files are automatically purged after a
week (configurable of course). When I want to diagnose a problem, I'd first
plot everything with colplot, selecting timeframes of interest. Then if I
wanted more details I'd play back the file for that day with the
appropriate form/thru switches. Colplot is certainly optional, but I'd find
it to be most useful when I wasn't sure what I was looking for.
As for compressed files, when I first wrote colplot, you had to have text
based files to feed into gnuplot. later on, gnuplot supported plotting
compressed files and to my delight was almost as efficient. I'm guessing
because the files were something like 10% the size and so much less I/O
required to read them.
In any event, I'm always happy to here collectl lives on and to see people
using it in different ways. As it turns out I think you're the first person
I remember who found vmstat format useful, possibly why that bug you
uncovered took over 20 years to find! ;)
-mark
On Tue, Nov 29, 2022 at 5:53 PM Laurence Oberman
loberman@users.sourceforge.net %0Dloberman@users.sourceforge.net wrote:
On Tue, Nov 29, 2022 at 2:36 PM Bill Torpey
wallstprog@users.sourceforge.net %0Dwallstprog@users.sourceforge.net
wrote:
Hi -- I'm back...
Have run some tests, and my results seem to indicate that:
Without quotes, wildcarding the "-p" param results in only the
first file being processed (bash guarantees that filenames are returned
in alpha order
https://serverfault.com/questions/122737/in-bash-are-wildcard-expansions-guaranteed-to-be-in-order
- With quotes, collectl processes all files, subject to "--from"
parameter
- It doesn't matter whether you use single or double quotes
- Presence/absence of "--from" parameter makes no difference.
- Except with "--export vmstat": collectl will NOT process multiple
files -- at the end of the first file the message "Error: -s not
allowed
with 'vmstat'" is output
- The error with vmstat export appears to be caused by the check
in vmstatInit. After removing the check, the vmstat export works
similarly
to other formats.
So, two questions:
- Where does the "phantom" "-s" flag come from?
- Why does vmstatInit check it?
Hope this helps. For now, I'm running with a patched version of collectl,
but would very much like to not do that. The only other alternative is a
very messy solution involving using a for-loop and appending to result
file.
First off, thanks for all the help, and for a great tool! Please see replies inline below:
Bill - I guess the one overall question I'd ask is about your general use of collectl and why you have a handful of smallisk files you need to play back together and also way none of your files are compressed?
We've recently switched to collectl for monitoring our QA performance testing environment, with a view to rolling out collectl in production. collectl is replacing a bunch of custom scripts that basically just ran top, vmstat, etc. in a loop and dumped output to files that were processed by a set of Perl scripts. Those scripts consolidated data across a dozen or so hosts to give a high-level view of system utilization at different message rates, and helps us identify bottlenecks and performance regressions in our application.
The first pass is a one-for-one replacement of the existing scripts, and for that reason it ends up running collectl four times: once each for cpu (per-process), memory (also per-process), network and vmstat-type output (block writes, iowait, etc.).
It's MUCH faster to unzip the file(s) first, rather than have collectl read the gzipped file directly, even once and obviously four times.
The next step would/should be to rewrite the original Perl scripts to take collectl's plot format, in which case we'll only need to read the raw file once. (I've also just finished splitting up the scripts so we can run the collectl analysis in background for each host separately).
I had designed collectl to very efficient in terms of cpu and storage. When I installed collectl on any system, I'd just turn it on and leave it running forever. For that reason, files are automatically purged after a week (configurable of course).
That's exactly what we plan to do in production. For our QA performance tests, it makes more sense to start and stop collectl along with our test harness. We archive the (compressed) collectl files along with application logs, etc. that we use for running our analysis on, and doing it this way means the collectl files are smaller (most performance tests only run for an hour or two).
When I want to diagnose a problem, I'd first plot everything with colplot, selecting timeframes of interest. Then if I wanted more details I'd play back the file for that day with the appropriate form/thru switches. Colplot is certainly optional, but I'd find it to be most useful when I wasn't sure what I was looking for.
We have custom gnuplot scripts that generate specific plots that we use a lot, but I would like to use colplot once we deploy to production for real-time trouble-shooting, just haven't gotten there yet.
As for compressed files, when I first wrote colplot, you had to have text based files to feed into gnuplot. later on, gnuplot supported plotting compressed files and to my delight was almost as efficient. I'm guessing because the files were something like 10% the size and so much less I/O required to read them.
Interesting — I didn't know that. That might be helpful, since some of our files do get a bit large. For instance, the specific test that I'm currently analyzing is a "5-day test", meant to model a full week of trading activity, and took three days real-time to run. The latency data file for this test has about 75 million rows, which ends up being around 8GB.
In any event, I'm always happy to here collectl lives on and to see people using it in different ways. As it turns out I think you're the first person I remember who found vmstat format useful, possibly why that bug you uncovered took over 20 years to find! ;)
Happy to help ;-) Seriously, I'm a little surprised — for our db servers we care a lot about block writes and iowait, and afaict the vmstat export is the only way to get that from collectl.
In any event, thanks again for a really cool tool. It's taken me several years to get it deployed in our environment, but it's absolutely terrific.
Last but not least — any hints or tips for speeding up post-processing analysis would be appreciated — collectl is very efficient in collecting, but generating reports from the collected data can take quite a while.
What really speeds things up is uncompressing the files before parsing them.
I never parse as .gz.
My tool uses the plot format too.
I will see if I can sanitize it to share it.
The thing is it uses numpy, scipy, pandas and matplotlib because it also
does stats analysis
First off, thanks for all the help, and for a great tool! Please see
replies inline below:
Bill - I guess the one overall question I'd ask is about your general use
of collectl and why you have a handful of smallisk files you need to play
back together and also way none of your files are compressed?
We've recently switched to collectl for monitoring our QA performance
testing environment, with a view to rolling out collectl in production.
collectl is replacing a bunch of custom scripts that basically just ran
top, vmstat, etc. in a loop and dumped output to files that were processed
by a set of Perl scripts. Those scripts consolidated data across a dozen or
so hosts to give a high-level view of system utilization at different
message rates, and helps us identify bottlenecks and performance
regressions in our application.
The first pass is a one-for-one replacement of the existing scripts, and
for that reason it ends up running collectl four times: once each for cpu
(per-process), memory (also per-process), network and vmstat-type output
(block writes, iowait, etc.).
It's MUCH faster to unzip the file(s) first, rather than have collectl
read the gzipped file directly, even once and obviously four times.
The next step would/should be to rewrite the original Perl scripts to take
collectl's plot format, in which case we'll only need to read the raw file
once. (I've also just finished splitting up the scripts so we can run the
collectl analysis in background for each host separately).
I had designed collectl to very efficient in terms of cpu and storage.
When I installed collectl on any system, I'd just turn it on and leave it
running forever. For that reason, files are automatically purged after a
week (configurable of course).
That's exactly what we plan to do in production. For our QA performance
tests, it makes more sense to start and stop collectl along with our test
harness. We archive the (compressed) collectl files along with application
logs, etc. that we use for running our analysis on, and doing it this way
means the collectl files are smaller (most performance tests only run for
an hour or two).
When I want to diagnose a problem, I'd first plot everything with colplot,
selecting timeframes of interest. Then if I wanted more details I'd play
back the file for that day with the appropriate form/thru switches. Colplot
is certainly optional, but I'd find it to be most useful when I wasn't sure
what I was looking for.
We have custom gnuplot scripts that generate specific plots that we use a
lot, but I would like to use colplot once we deploy to production for
real-time trouble-shooting, just haven't gotten there yet.
As for compressed files, when I first wrote colplot, you had to have text
based files to feed into gnuplot. later on, gnuplot supported plotting
compressed files and to my delight was almost as efficient. I'm guessing
because the files were something like 10% the size and so much less I/O
required to read them.
Interesting — I didn't know that. That might be helpful, since some of our
files do get a bit large. For instance, the specific test that I'm
currently analyzing is a "5-day test", meant to model a full week of
trading activity, and took three days real-time to run. The latency data
file for this test has about 75 million rows, which ends up being around
8GB.
In any event, I'm always happy to here collectl lives on and to see people
using it in different ways. As it turns out I think you're the first person
I remember who found vmstat format useful, possibly why that bug you
uncovered took over 20 years to find! ;)
Happy to help ;-) Seriously, I'm a little surprised — for our db servers
we care a lot about block writes and iowait, and afaict the vmstat export
is the only way to get that from collectl.
In any event, thanks again for a really cool tool. It's taken me several
years to get it deployed in our environment, but it's absolutely terrific.
Last but not least — any hints or tips for speeding up post-processing
analysis would be appreciated — collectl is very efficient in collecting,
but generating reports from the collected data can take quite a while.
Bill
I use
collectl -sD to get all the per disk detailed I/O related stuff and I work
deeply in storage performance at Red Hat.
You don't need to use the vmstat stuff.
What really speeds things up is uncompressing the files before parsing
them.
I never parse as .gz.
My tool uses the plot format too.
I will see if I can sanitize it to share it.
The thing is it uses numpy, scipy, pandas and matplotlib because it also
does stats analysis
First off, thanks for all the help, and for a great tool! Please see
replies inline below:
Bill - I guess the one overall question I'd ask is about your general use
of collectl and why you have a handful of smallisk files you need to play
back together and also way none of your files are compressed?
We've recently switched to collectl for monitoring our QA performance
testing environment, with a view to rolling out collectl in production.
collectl is replacing a bunch of custom scripts that basically just ran
top, vmstat, etc. in a loop and dumped output to files that were processed
by a set of Perl scripts. Those scripts consolidated data across a dozen or
so hosts to give a high-level view of system utilization at different
message rates, and helps us identify bottlenecks and performance
regressions in our application.
The first pass is a one-for-one replacement of the existing scripts, and
for that reason it ends up running collectl four times: once each for cpu
(per-process), memory (also per-process), network and vmstat-type output
(block writes, iowait, etc.).
It's MUCH faster to unzip the file(s) first, rather than have collectl
read the gzipped file directly, even once and obviously four times.
The next step would/should be to rewrite the original Perl scripts to
take collectl's plot format, in which case we'll only need to read the raw
file once. (I've also just finished splitting up the scripts so we can run
the collectl analysis in background for each host separately).
I had designed collectl to very efficient in terms of cpu and storage.
When I installed collectl on any system, I'd just turn it on and leave it
running forever. For that reason, files are automatically purged after a
week (configurable of course).
That's exactly what we plan to do in production. For our QA performance
tests, it makes more sense to start and stop collectl along with our test
harness. We archive the (compressed) collectl files along with application
logs, etc. that we use for running our analysis on, and doing it this way
means the collectl files are smaller (most performance tests only run for
an hour or two).
When I want to diagnose a problem, I'd first plot everything with
colplot, selecting timeframes of interest. Then if I wanted more details
I'd play back the file for that day with the appropriate form/thru
switches. Colplot is certainly optional, but I'd find it to be most useful
when I wasn't sure what I was looking for.
We have custom gnuplot scripts that generate specific plots that we use a
lot, but I would like to use colplot once we deploy to production for
real-time trouble-shooting, just haven't gotten there yet.
As for compressed files, when I first wrote colplot, you had to have text
based files to feed into gnuplot. later on, gnuplot supported plotting
compressed files and to my delight was almost as efficient. I'm guessing
because the files were something like 10% the size and so much less I/O
required to read them.
Interesting — I didn't know that. That might be helpful, since some of
our files do get a bit large. For instance, the specific test that I'm
currently analyzing is a "5-day test", meant to model a full week of
trading activity, and took three days real-time to run. The latency data
file for this test has about 75 million rows, which ends up being around
8GB.
In any event, I'm always happy to here collectl lives on and to see
people using it in different ways. As it turns out I think you're the first
person I remember who found vmstat format useful, possibly why that bug you
uncovered took over 20 years to find! ;)
Happy to help ;-) Seriously, I'm a little surprised — for our db servers
we care a lot about block writes and iowait, and afaict the vmstat export
is the only way to get that from collectl.
In any event, thanks again for a really cool tool. It's taken me several
years to get it deployed in our environment, but it's absolutely terrific.
Last but not least — any hints or tips for speeding up post-processing
analysis would be appreciated — collectl is very efficient in collecting,
but generating reports from the collected data can take quite a while.
bill - did I read it right when you said you run collectl once for each
cpu? By default collectl ALWAYS records data for all CPUs.
laurence - maybe you already know this but if you're looking at disks stats
ordered by disk, colmux will to that in real time and if on
multiple systems will show you the busiest disks across all your systems.
bill, if you're finding things go faster by first decompressing your files
I suspect something may be wrong. I used to decompress before I found I
could give gnuplot a switch to read compressed files. I could plot a full
day;s data from dozens of collectl logs from multiple systems in under a
minutes, maybe even in 10s of seconds. almost makes me want to do a
video/demo to show you but I don't even have a laptop any more ;(
Mark
Thank you, but things have evolved, we have huge systems with many CPUS and
many disks and the GZ files are huge
The decompress instream is slow.
I think it's been so long for you, that you have not seen the growth.
I have data from customers for example 4GB in size for a compressed raw
file and like 20GB or mor uncompressed
bill - did I read it right when you said you run collectl once for each
cpu? By default collectl ALWAYS records data for all CPUs.
laurence - maybe you already know this but if you're looking at disks stats
ordered by disk, colmux will to that in real time and if on
multiple systems will show you the busiest disks across all your systems.
bill, if you're finding things go faster by first decompressing your files
I suspect something may be wrong. I used to decompress before I found I
could give gnuplot a switch to read compressed files. I could plot a full
day;s data from dozens of collectl logs from multiple systems in under a
minutes, maybe even in 10s of seconds. almost makes me want to do a
video/demo to show you but I don't even have a laptop any more ;(
Sorry if that wasn't clear: we run collectl once to get per-process cpu utilization (--procopts kw4096 -sZ -oDG), a second time to get per-process memory utilization (--procopts mkw4096 -sZ -oDG), a third time to get system-wide network utilization (-sN -oD), and fourth to get vmstat (--export vmstat -oD). All of that is from the same .raw file(s).
Again, we did it that way just so we could run both collectl and our old monitoring tools and compare like-for-like.
We'll be changing that to generate plot-format files from the raw files, which means we'll only need to pass the raw file once. That's a win: for this 5-day test it takes >20 minutes for a single pass of the four raw files which total a bit over 10GB combined. (This on a CentOS 7 box with 4/8(HT) x i7-4770R CPU @ 3.20GHz, 16GB RAM and SSD. It's a bit quicker when we're not asking for per-process stats). Multiply that by 10 hosts and a full run takes a while.
We specify a procfilt in our etc/collectl/conf so only capture processes that we care about -- unfortunately for some of them we need to filter on the full command line. We are sampling once per-second, which takes very little overhead on the collection side, but it does impact post-processing.
And yes, the command lines we capture/process are ridiculously large (4k) -- blame java and/or people who put all their jar's on the command line instead of in an environment variable. But we need the full command line to filter the proper processes.
Last but not least we can't use colplot directly -- we already have a set of standard gnuplot templates that we use, some of which pull data from multiple files (e.g., to compare latency against message rate). Not all data comes from collectl either -- we pull data from application logs and combine it with things like cpu and memory.
Again, thanks for collectl and for all your help!
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I think you may be confused about collectl data collection. When you
collect process data your get a lot more than is displayed. You can then
play it back multiple times and choose to display different process options
from same collection, see --helpsuboopts.
Also, no need to run once to collectl process data and a second time for
system disk or other data. Just collectl it all in one go and play back
with --vmstat or -sd or -sz. I was very careful to design collectl so you
could collect everything at once and then play back as many times as you
want to with different options. If you run it more than once, you're making
much extra work for yourelf. Furthermore, the point is if you see something
funky in a process you want to be able to go back and see what the CPUs,
disks, memory, netork or even interrupts were doing at the exact same time.
I like to think if there was something you wished collectl could do, it
probably already does and you just haven't found the right switch yet ;)
Sorry if that wasn't clear: we run collectl once to get per-process cpu
utilization (--procopts kw4096 -sZ -oDG), a second time to get
per-process memory utilization (--procopts mkw4096 -sZ -oDG), a third
time to get system-wide network utilization (-sN -oD), and fourth to get
vmstat (--export vmstat -oD). All of that is from the same .raw file(s).
Again, we did it that way just so we could run both collectl and our old
monitoring tools and compare like-for-like.
We'll be changing that to generate plot-format files from the raw files,
which means we'll only need to pass the raw file once. That's a win: for
this 5-day test it takes >20 minutes for a single pass of the four raw
files which total a bit over 10GB combined. (This on a CentOS 7 box with
4/8(HT) x i7-4770R CPU @ 3.20GHz, 16GB RAM and SSD. It's a bit quicker when
we're not asking for per-process stats). Multiply that by 10 hosts and a
full run takes a while.
We specify a procfilt in our etc/collectl/conf so only capture processes
that we care about -- unfortunately for some of them we need to filter on
the full command line. We are sampling once per-second, which takes very
little overhead on the collection side, but it does impact post-processing.
And yes, the command lines we capture/process are ridiculously large (4k)
-- blame java and/or people who put all their jar's on the command line
instead of in an environment variable. But we need the full command line to
filter the proper processes.
Last but not least we can't use colplot directly -- we already have a set
of standard gnuplot templates that we use, some of which pull data from
multiple files (e.g., to compare latency against message rate). Not all
data comes from collectl either -- we pull data from application logs and
combine it with things like cpu and memory.
Mark
Bill has a special use case and he knows about the collectl service for
sure that gathers everything.
He is scripting this so I think you misunderstood.
I think you may be confused about collectl data collection. When you
collect process data your get a lot more than is displayed. You can then
play it back multiple times and choose to display different process options
from same collection, see --helpsuboopts.
Also, no need to run once to collectl process data and a second time for
system disk or other data. Just collectl it all in one go and play back
with --vmstat or -sd or -sz. I was very careful to design collectl so you
could collect everything at once and then play back as many times as you
want to with different options. If you run it more than once, you're making
much extra work for yourelf. Furthermore, the point is if you see something
funky in a process you want to be able to go back and see what the CPUs,
disks, memory, netork or even interrupts were doing at the exact same time.
I like to think if there was something you wished collectl could do, it
probably already does and you just haven't found the right switch yet ;)
Sorry if that wasn't clear: we run collectl once to get per-process cpu
utilization (--procopts kw4096 -sZ -oDG), a second time to get
per-process memory utilization (--procopts mkw4096 -sZ -oDG), a third
time to get system-wide network utilization (-sN -oD), and fourth to get
vmstat (--export vmstat -oD). All of that is from the same .raw file(s).
Again, we did it that way just so we could run both collectl and our old
monitoring tools and compare like-for-like.
We'll be changing that to generate plot-format files from the raw files,
which means we'll only need to pass the raw file once. That's a win: for
this 5-day test it takes >20 minutes for a single pass of the four raw
files which total a bit over 10GB combined. (This on a CentOS 7 box with
4/8(HT) x i7-4770R CPU @ 3.20GHz, 16GB RAM and SSD. It's a bit quicker when
we're not asking for per-process stats). Multiply that by 10 hosts and a
full run takes a while.
We specify a procfilt in our etc/collectl/conf so only capture processes
that we care about -- unfortunately for some of them we need to filter on
the full command line. We are sampling once per-second, which takes very
little overhead on the collection side, but it does impact post-processing.
And yes, the command lines we capture/process are ridiculously large (4k)
-- blame java and/or people who put all their jar's on the command line
instead of in an environment variable. But we need the full command line to
filter the proper processes.
Last but not least we can't use colplot directly -- we already have a set
of standard gnuplot templates that we use, some of which pull data from
multiple files (e.g., to compare latency against message rate). Not all
data comes from collectl either -- we pull data from application logs and
combine it with things like cpu and memory.
Again, thanks for collectl and for all your help!
vmstatInit fails on second file when using wildcards
I think you may be confused about collectl data collection. When you
collect process data your get a lot more than is displayed. You can then
play it back multiple times and choose to display different process options
from same collection, see --helpsuboopts.
Also, no need to run once to collectl process data and a second time for
system disk or other data. Just collectl it all in one go and play back
with --vmstat or -sd or -sz. I was very careful to design collectl so you
could collect everything at once and then play back as many times as you
want to with different options. If you run it more than once, you're making
much extra work for yourelf. Furthermore, the point is if you see something
funky in a process you want to be able to go back and see what the CPUs,
disks, memory, netork or even interrupts were doing at the exact same time.
That’s exactly what we’re doing. We collect once and then playback the data to populate different datasets. The problem is that each pass through the raw file takes a long time (e.g., ~20 minutes for the “5 day test”), and we do it once each for cpu, memory, network and disk. Hence the desire to generate plot files directly from the raw files, which can be done in a single pass.
I like to think if there was something you wished collectl could do, it
probably already does and you just haven't found the right switch yet ;)
That’s entirely possible — there’s a lot to learn.
Sorry if that wasn't clear: we run collectl once to get per-process cpu
utilization (--procopts kw4096 -sZ -oDG), a second time to get
per-process memory utilization (--procopts mkw4096 -sZ -oDG), a third
time to get system-wide network utilization (-sN -oD), and fourth to get
vmstat (--export vmstat -oD). All of that is from the same .raw file(s).
Again, we did it that way just so we could run both collectl and our old
monitoring tools and compare like-for-like.
We'll be changing that to generate plot-format files from the raw files,
which means we'll only need to pass the raw file once. That's a win: for
this 5-day test it takes >20 minutes for a single pass of the four raw
files which total a bit over 10GB combined. (This on a CentOS 7 box with
4/8(HT) x i7-4770R CPU @ 3.20GHz, 16GB RAM and SSD. It's a bit quicker when
we're not asking for per-process stats). Multiply that by 10 hosts and a
full run takes a while.
We specify a procfilt in our etc/collectl/conf so only capture processes
that we care about -- unfortunately for some of them we need to filter on
the full command line. We are sampling once per-second, which takes very
little overhead on the collection side, but it does impact post-processing.
And yes, the command lines we capture/process are ridiculously large (4k)
-- blame java and/or people who put all their jar's on the command line
instead of in an environment variable. But we need the full command line to
filter the proper processes.
Last but not least we can't use colplot directly -- we already have a set
of standard gnuplot templates that we use, some of which pull data from
multiple files (e.g., to compare latency against message rate). Not all
data comes from collectl either -- we pull data from application logs and
combine it with things like cpu and memory.
Hmm, if your goal is to populate a database through multiple passes, can't
you just tell collectl to generate plotable output? Then you have everying
in formatted text output space separated so easily parsed. No need to
convert anything after it runs. Maybe
-mark
On Dec 2, 2022, at 10:26 AM, Mark Seger markseger@users.sourceforge.net
wrote:
I think you may be confused about collectl data collection. When you
collect process data your get a lot more than is displayed. You can then
play it back multiple times and choose to display different process options
from same collection, see --helpsuboopts.
Also, no need to run once to collectl process data and a second time for
system disk or other data. Just collectl it all in one go and play back
with --vmstat or -sd or -sz. I was very careful to design collectl so you
could collect everything at once and then play back as many times as you
want to with different options. If you run it more than once, you're making
much extra work for yourelf. Furthermore, the point is if you see something
funky in a process you want to be able to go back and see what the CPUs,
disks, memory, netork or even interrupts were doing at the exact same time.
That’s exactly what we’re doing. We collect once and then playback the
data to populate different datasets. The problem is that each pass through
the raw file takes a long time (e.g., ~20 minutes for the “5 day test”),
and we do it once each for cpu, memory, network and disk. Hence the desire
to generate plot files directly from the raw files, which can be done in a
single pass.
I like to think if there was something you wished collectl could do, it
probably already does and you just haven't found the right switch yet ;)
That’s entirely possible — there’s a lot to learn.
-mark
On Thu, Dec 1, 2022 at 12:15 PM Bill Torpey
wallstprog@users.sourceforge.net wallstprog@users.sourceforge.net wrote:
Sorry if that wasn't clear: we run collectl once to get per-process cpu
utilization (--procopts kw4096 -sZ -oDG), a second time to get
per-process memory utilization (--procopts mkw4096 -sZ -oDG), a third
time to get system-wide network utilization (-sN -oD), and fourth to get
vmstat (--export vmstat -oD). All of that is from the same .raw file(s).
Again, we did it that way just so we could run both collectl and our old
monitoring tools and compare like-for-like.
We'll be changing that to generate plot-format files from the raw files,
which means we'll only need to pass the raw file once. That's a win: for
this 5-day test it takes >20 minutes for a single pass of the four raw
files which total a bit over 10GB combined. (This on a CentOS 7 box with
4/8(HT) x i7-4770R CPU @ 3.20GHz, 16GB RAM and SSD. It's a bit quicker when
we're not asking for per-process stats). Multiply that by 10 hosts and a
full run takes a while.
We specify a procfilt in our etc/collectl/conf so only capture processes
that we care about -- unfortunately for some of them we need to filter on
the full command line. We are sampling once per-second, which takes very
little overhead on the collection side, but it does impact post-processing.
And yes, the command lines we capture/process are ridiculously large (4k)
-- blame java and/or people who put all their jar's on the command line
instead of in an environment variable. But we need the full command line to
filter the proper processes.
Last but not least we can't use colplot directly -- we already have a set
of standard gnuplot templates that we use, some of which pull data from
multiple files (e.g., to compare latency against message rate). Not all
data comes from collectl either -- we pull data from application logs and
combine it with things like cpu and memory.
Again, thanks for collectl and for all your help!
vmstatInit fails on second file when using wildcards
Here's the directory listing:
The result of running the system collectl:
The patched version (removed check in vmStatInit):
And version (this on CentOS 7):
Thanks Bill
I will reproduce this and fix this.
If I have any issues reproducing will come back to the thread.
Your Reproducer noted
[btorpey@bt-brix7 collectl]$ ls -lh *.raw
-rw-r--r-- 1 btorpey shared 102M Nov 22 09:36 csappia01-20221110-223734.raw
-rw-r--r-- 1 btorpey shared 2.0G Nov 22 09:36 csappia01-20221111-000000.raw
-rw-r--r-- 1 btorpey shared 2.0G Nov 22 09:36 csappia01-20221112-000000.raw
-rw-r--r-- 1 btorpey shared 1.1G Nov 22 09:36 csappia01-20221113-000000.raw
The result of running the system collectl:
[btorpey@bt-brix7 collectl]$ time /usr/bin/collectl -p '*.raw'
--export vmstat -oD 2>&1 | tail
20221110 23:59:52 0 0 0 114G 321M 1832M 370M 1725M
0 0 0 76 2446 3488 0 0 99 0
20221110 23:59:53 0 0 0 114G 321M 1832M 370M 1725M
0 0 0 248 6604 7982 1 0 98 0
20221110 23:59:54 0 0 0 114G 321M 1832M 370M 1725M
0 0 0 684 2146 3180 0 0 99 0
20221110 23:59:55 1 0 0 114G 321M 1832M 370M 1725M
0 0 0 0 1583 2785 0 0 99 0
20221110 23:59:56 0 0 0 114G 321M 1832M 370M 1725M
0 0 0 0 1706 2769 0 0 99 0
20221110 23:59:57 0 0 0 114G 321M 1832M 370M 1725M
0 0 0 0 1359 2511 0 0 99 0
20221110 23:59:58 0 0 0 114G 321M 1832M 370M 1725M
0 0 0 300 1798 2950 0 0 99 0
20221110 23:59:59 0 0 0 114G 321M 1832M 370M 1725M
0 0 0 0 1421 2585 0 0 99 0
Error: -s not allowed with 'vmstat'
type 'collectl -h' for help
On Tue, Nov 29, 2022 at 2:36 PM Bill Torpey wallstprog@users.sourceforge.net wrote:
--
Laurence Oberman
Senior Principal Software Maintenance Engineer - Storage
Red Hat North America CEE-SEG
[loberman@lobep17 collectl]$ time /usr/bin/collectl -p '*.raw.gz' --export
vmstat -oD 2>&1 | tail
20221018 23:58:40 0 0 2304K 5844M 6668K 98859M 88006M 9417M 0
0 0 32 1159 1953 2 0 97 0
20221018 23:58:50 1 0 2304K 5843M 6668K 98859M 88006M 9417M 0
0 0 15 1055 1848 2 0 97 0
20221018 23:59:00 1 0 2304K 5844M 6668K 98859M 88006M 9417M 0
0 0 34 1027 1852 2 0 97 0
20221018 23:59:10 1 0 2304K 5828M 6668K 98874M 88006M 9417M 0
0 0 79 1071 1831 2 0 97 0
20221018 23:59:20 0 0 2304K 5830M 6668K 98874M 88006M 9417M 0
0 0 84 1032 1852 2 0 97 0
20221018 23:59:30 1 0 2304K 5829M 6668K 98874M 88006M 9417M 0
0 0 33 1068 1819 2 0 97 0
20221018 23:59:40 0 0 2304K 5845M 6668K 98859M 88006M 9417M 0
0 0 12 1128 1951 2 0 97 0
20221018 23:59:50 1 0 2304K 5829M 6668K 98874M 88006M 9417M 0
0 0 39 1071 1834 2 0 97 0
Error: -s not allowed with 'vmstat'
type 'collectl -h' for help
real 0m47.081s
user 0m47.002s
sys 0m0.278s
OK adding debug will send a patched version when fixed
On Tue, Nov 29, 2022 at 5:34 PM Laurence Oberman loberman@redhat.com
wrote:
--
Laurence Oberman
Senior Principal Software Maintenance Engineer - Storage
Red Hat North America CEE-SEG
Seems it is going to always be called and will check user switches.
It's always going to log that message as the test is straight forward
because $userSubsys will not be empty
Example here with DEBUG and why it happens
[loberman@lobep17 collectl]$ time /usr/bin/collectl -p '.raw.gz' --export
vmstat -oD 2>&1 | tail
20221018 23:58:50 1 0 2304K 5843M 6668K 98859M 88006M 9417M 0
0 0 15 1055 1848 2 0 97 0
20221018 23:59:00 1 0 2304K 5844M 6668K 98859M 88006M 9417M 0
0 0 34 1027 1852 2 0 97 0
20221018 23:59:10 1 0 2304K 5828M 6668K 98874M 88006M 9417M 0
0 0 79 1071 1831 2 0 97 0
20221018 23:59:20 0 0 2304K 5830M 6668K 98874M 88006M 9417M 0
0 0 84 1032 1852 2 0 97 0
20221018 23:59:30 1 0 2304K 5829M 6668K 98874M 88006M 9417M 0
0 0 33 1068 1819 2 0 97 0
20221018 23:59:40 0 0 2304K 5845M 6668K 98859M 88006M 9417M 0
0 0 12 1128 1951 2 0 97 0
20221018 23:59:50 1 0 2304K 5829M 6668K 98874M 88006M 9417M 0
0 0 39 1071 1834 2 0 97 0
DEBUG: userSubsys is cm *** Note
Error: -s not allowed with 'vmstat'
Notice the docs, even says requires -scm
This first section gets called almost immediately by collectl after reading
in the various user switches. This is the place to catch switch errors and
since this routine always requires -scm we'll just hardcode it to that and
reject any user entered ones. This initialization subroutine must be named
for our module followed by Init.
sub vmstatInit
{
error("-s not allowed with 'vmstat'") if $userSubsys ne '';
error("-f requires either --rawtoo or -P") if $filename ne '' &&
!$rawtooFlag && !$plotFlag;
error("-P or --rawtoo require -f") if $filename eq '' &&
($rawtooFlag || $plotFlag);
$subsys=$userSubsys='cm';
** Hardcoded here
}
I think this issue has been there always and you caught it because of the
wildcard in quotes.
I am going to remove that error check and release a fixed version because
it serves no purpose.
New code looks like this. can you modify your vmstat.ph in
/usr/share/collectl and test this and confirm its good for you
sub vmstatInit
{
error("-f requires either --rawtoo or -P") if $filename ne '' &&
!$rawtooFlag && !$plotFlag;
error("-P or --rawtoo require -f") if $filename eq '' &&
($rawtooFlag || $plotFlag);
$subsys=$userSubsys='cm';
}
If you are good I will release the fix.
I want it tested by you because it is a seldom used option so if you are
happy I am happy.
Thanks
Laurence
On Tue, Nov 29, 2022 at 5:53 PM Laurence Oberman loberman@users.sourceforge.net wrote:
--
Laurence Oberman
Senior Principal Software Maintenance Engineer - Storage
Red Hat North America CEE-SEG
Here is the commit message in my local tree.
commit 625e2453842121519e9e5a38c283d02e72e2ed89 (HEAD -> master)
Author: Laurence Oberman loberman@redhat.com
Date: Tue Nov 29 18:19:44 2022 -0500
On Tue, Nov 29, 2022 at 6:19 PM Laurence Oberman loberman@users.sourceforge.net wrote:
--
Laurence Oberman
Senior Principal Software Maintenance Engineer - Storage
Red Hat North America CEE-SEG
Right -- you're just removing the check for "-s". I've been running with that patch in my local copy for a week or so and have not seen any problems.
Looks good to me! and Thanks!
Bill - I guess the one overall question I'd ask is about your general use
of collectl and why you have a handful of smallisk files you need to play
back together and also way none of your files are compressed?
I had designed collectl to very efficient in terms of cpu and storage. When
I installed collectl on any system, I'd just turn it on and leave it
running forever. For that reason, files are automatically purged after a
week (configurable of course). When I want to diagnose a problem, I'd first
plot everything with colplot, selecting timeframes of interest. Then if I
wanted more details I'd play back the file for that day with the
appropriate form/thru switches. Colplot is certainly optional, but I'd find
it to be most useful when I wasn't sure what I was looking for.
As for compressed files, when I first wrote colplot, you had to have text
based files to feed into gnuplot. later on, gnuplot supported plotting
compressed files and to my delight was almost as efficient. I'm guessing
because the files were something like 10% the size and so much less I/O
required to read them.
In any event, I'm always happy to here collectl lives on and to see people
using it in different ways. As it turns out I think you're the first person
I remember who found vmstat format useful, possibly why that bug you
uncovered took over 20 years to find! ;)
-mark
On Tue, Nov 29, 2022 at 5:53 PM Laurence Oberman loberman@users.sourceforge.net wrote:
Here at Red Hat our use case is two fold..
If a customer reports a performance problem or any issue needing a deep
dive, Install collect, let it gather and then investigate
various performance problems after the customer reports the timestamps.
The historical capture is very important.
We also use it live if needed to do monitoring on a live performance
problem.
collectl -scmnd -oT -i1 used a lot
collectl -sc --verbose -oT -i1 also used a lot
collectl -sD -oT -i1 --dskfilt xxxx
I have also been using this a very long time with Mark, helping him with
various Linux internals questions in the past.
It's certainly important enough to maintain so I took on the responsibility.
I have developed a graphing tool we use internally so we don't use colplot
and have not in the past had to use colmux.
I know that Mark used to use colmux a lot.
My tool uses python and pandas dataframes with matplotlib
Regards
Laurence
On Thu, Dec 1, 2022 at 8:44 AM Mark Seger markseger@users.sourceforge.net
wrote:
--
Laurence Oberman
Senior Principal Software Maintenance Engineer - Storage
Red Hat North America CEE-SEG
In case you missed it
you can now use
git clone https://github.com/sharkcz/collectl.git
then git pull to get updates
The last couple of updates
commit 2001864971f91999216daf091529f5aa47d69150 (HEAD -> master,
origin/master, origin/HEAD)
Author: Laurence Oberman loberman@redhat.com
Date: Tue Nov 29 18:29:26 2022 -0500
commit 625e2453842121519e9e5a38c283d02e72e2ed89
Author: Laurence Oberman loberman@redhat.com
Date: Tue Nov 29 18:19:44 2022 -0500
On Thu, Dec 1, 2022 at 9:15 AM Laurence Oberman loberman@users.sourceforge.net wrote:
--
Laurence Oberman
Senior Principal Software Maintenance Engineer - Storage
Red Hat North America CEE-SEG
My maintenance responsibilities:
capture a default
So far kernel updates have not broken collectl and even the latest Fedora
is seamless.
I hope that is in sync with what you Mark wanted when I took over
maintenance.
Regards
Laurence
On Thu, Dec 1, 2022 at 9:17 AM Laurence Oberman loberman@users.sourceforge.net wrote:
--
Laurence Oberman
Senior Principal Software Maintenance Engineer - Storage
Red Hat North America CEE-SEG
Hi Mark (and Laurence):
First off, thanks for all the help, and for a great tool! Please see replies inline below:
We've recently switched to collectl for monitoring our QA performance testing environment, with a view to rolling out collectl in production. collectl is replacing a bunch of custom scripts that basically just ran top, vmstat, etc. in a loop and dumped output to files that were processed by a set of Perl scripts. Those scripts consolidated data across a dozen or so hosts to give a high-level view of system utilization at different message rates, and helps us identify bottlenecks and performance regressions in our application.
The first pass is a one-for-one replacement of the existing scripts, and for that reason it ends up running collectl four times: once each for cpu (per-process), memory (also per-process), network and vmstat-type output (block writes, iowait, etc.).
It's MUCH faster to unzip the file(s) first, rather than have collectl read the gzipped file directly, even once and obviously four times.
The next step would/should be to rewrite the original Perl scripts to take collectl's plot format, in which case we'll only need to read the raw file once. (I've also just finished splitting up the scripts so we can run the collectl analysis in background for each host separately).
That's exactly what we plan to do in production. For our QA performance tests, it makes more sense to start and stop collectl along with our test harness. We archive the (compressed) collectl files along with application logs, etc. that we use for running our analysis on, and doing it this way means the collectl files are smaller (most performance tests only run for an hour or two).
We have custom gnuplot scripts that generate specific plots that we use a lot, but I would like to use colplot once we deploy to production for real-time trouble-shooting, just haven't gotten there yet.
Interesting — I didn't know that. That might be helpful, since some of our files do get a bit large. For instance, the specific test that I'm currently analyzing is a "5-day test", meant to model a full week of trading activity, and took three days real-time to run. The latency data file for this test has about 75 million rows, which ends up being around 8GB.
Happy to help ;-) Seriously, I'm a little surprised — for our db servers we care a lot about block writes and iowait, and afaict the vmstat export is the only way to get that from collectl.
In any event, thanks again for a really cool tool. It's taken me several years to get it deployed in our environment, but it's absolutely terrific.
Last but not least — any hints or tips for speeding up post-processing analysis would be appreciated — collectl is very efficient in collecting, but generating reports from the collected data can take quite a while.
What really speeds things up is uncompressing the files before parsing them.
I never parse as .gz.
My tool uses the plot format too.
I will see if I can sanitize it to share it.
The thing is it uses numpy, scipy, pandas and matplotlib because it also
does stats analysis
On Thu, Dec 1, 2022 at 10:36 AM Bill Torpey wallstprog@users.sourceforge.net wrote:
--
Laurence Oberman
Senior Principal Software Maintenance Engineer - Storage
Red Hat North America CEE-SEG
Bill
I use
collectl -sD to get all the per disk detailed I/O related stuff and I work
deeply in storage performance at Red Hat.
You don't need to use the vmstat stuff.
My tool created tables like this
TOP 50 Disks ordered by IO/S per second Average
dm-114-Writes 556.033453
dm-109-Writes 540.127098
dm-113-Writes 540.126867
dm-103-Writes 478.489872
dm-99-Writes 478.481306
dm-53-Writes 317.075587
dm-29-Writes 315.691747
dm-111-Writes 272.861905
dm-107-Writes 270.897558
dm-67-Writes 253.888760
dm-63-Writes 253.645561
dm-108-Writes 178.919898
dm-112-Writes 178.655168
dm-105-Writes 80.676236
dm-100-Writes 80.457576
dm-104-Writes 80.457113
sdeq-Writes 79.380021
sddq-Writes 79.326311
dm-42-Writes 79.247598
sdgk-Writes 79.222132
dm-18-Writes 79.171779
sdfm-Writes 79.150017
sdy-Writes 79.125130
sdbu-Writes 79.069684
sdaw-Writes 78.893043
sdcs-Writes 78.610372
dm-102-Writes 58.487093
dm-98-Writes 58.426322
dm-17-Writes 50.581433
dm-41-Writes 50.464058
dm-69-Writes 25.650423
dm-64-Writes 25.646371
dm-68-Writes 25.646371
sddg-Writes 19.773238
sdee-Writes 19.771733
sdam-Writes 19.760042
sdga-Writes 19.754022
sdo-Writes 19.745457
sdfc-Writes 19.744299
sdci-Writes 19.737817
sdbk-Writes 19.723232
dm-105-Reads 15.788633
dm-66-Writes 13.653316
dm-62-Writes 13.646024
dm-47-Writes 12.692673
dm-23-Writes 12.686075
sdn-Writes 12.618937
sdal-Writes 12.604584
sded-Writes 12.580623
sddf-Writes 12.577382
TOP 50 Disks ordered by Qlen Average
dm-104-QueLen 99.163908
dm-67-QueLen 65.519273
dm-63-QueLen 64.152217
dm-103-QueLen 45.108230
dm-105-QueLen 38.954624
dm-99-QueLen 37.690010
dm-100-QueLen 32.058109
dm-108-QueLen 14.872786
dm-69-QueLen 13.876490
dm-112-QueLen 13.123047
dm-41-QueLen 7.713393
dm-17-QueLen 7.597639
dm-62-QueLen 4.546823
dm-66-QueLen 3.936567
dm-64-QueLen 3.886329
dm-23-QueLen 3.342864
dm-42-QueLen 2.972103
dm-68-QueLen 2.936914
dm-47-QueLen 2.891770
sddf-QueLen 2.781688
sded-QueLen 2.778678
sdfz-QueLen 2.776826
sdfb-QueLen 2.775437
dm-18-QueLen 2.670795
sdbj-QueLen 2.401667
sdch-QueLen 2.400162
sdn-QueLen 2.399583
sdal-QueLen 2.397268
dm-102-QueLen 2.285565
dm-27-QueLen 2.243547
dm-19-QueLen 2.176062
dm-51-QueLen 2.146892
dm-43-QueLen 2.004862
dm-98-QueLen 1.868735
dm-20-QueLen 1.780299
dm-44-QueLen 1.728209
sdga-QueLen 1.331751
sddg-QueLen 1.326542
sdfc-QueLen 1.326542
sdee-QueLen 1.325732
dm-114-QueLen 1.242042
dm-110-QueLen 1.228499
dm-50-QueLen 1.212988
sdcn-QueLen 1.167265
dm-26-QueLen 1.166917
sdt-QueLen 1.166802
sdar-QueLen 1.165297
sdbp-QueLen 1.164024
dm-46-QueLen 1.159278
dm-101-QueLen 1.155458
TOP 50 Disks ordered by wait time Average
dm-105-WaitW 450.300382
dm-104-WaitW 428.540688
dm-104-Wait 422.648455
dm-105-Wait 421.348767
dm-100-WaitW 101.007756
dm-100-Wait 90.874407
dm-69-WaitW 25.228036
dm-69-Wait 25.206042
dm-62-WaitW 13.369140
dm-62-Wait 13.333835
dm-41-WaitW 10.643014
dm-41-Wait 10.631786
sdfz-WaitW 9.966084
sdfb-WaitW 9.965505
sdfb-Wait 9.931474
sdfz-Wait 9.922560
sddf-WaitW 9.644056
sded-WaitW 9.608867
sddf-Wait 9.601459
sded-Wait 9.571941
dm-66-WaitW 9.448432
dm-66-Wait 9.424239
dm-63-Wait 8.573562
dm-63-WaitW 8.563954
dm-17-WaitW 7.771038
dm-17-Wait 7.765598
dm-105-WaitR 7.519273
sdbj-WaitW 7.420303
sdbj-Wait 7.332677
sdch-WaitW 7.256395
dm-103-Wait 7.184512
dm-64-Wait 7.180345
dm-64-WaitW 7.178146
sdch-Wait 7.159278
sdn-WaitW 7.125593
sdn-Wait 7.024656
sdal-WaitW 6.964000
dm-67-Wait 6.915268
dm-67-WaitW 6.914458
sdal-Wait 6.875217
dm-102-WaitW 6.720801
dm-100-WaitR 6.597060
dm-42-WaitW 6.479685
sdfc-WaitW 6.468110
sdga-WaitW 6.388934
dm-42-Wait 6.329784
dm-102-Wait 6.260331
sdfc-Wait 6.239264
sddg-WaitW 6.201181
sdee-WaitW 6.168770
TOP 50 Disks ordered by service time Average
sdfc-SvcTim 4.393564
sdga-SvcTim 4.330594
sddg-SvcTim 4.195393
sdee-SvcTim 4.158699
dm-110-SvcTim 4.035074
dm-105-SvcTim 3.924297
sdfm-SvcTim 3.732029
sdgk-SvcTim 3.728209
dm-101-SvcTim 3.691747
sddl-SvcTim 3.683760
sdeq-SvcTim 3.648223
sddq-SvcTim 3.602732
dm-61-SvcTim 3.599259
sdgf-SvcTim 3.579118
sdfh-SvcTim 3.552958
sdej-SvcTim 3.359069
dm-104-SvcTim 3.339044
dm-111-SvcTim 3.331983
sdfb-SvcTim 3.298761
sdfz-SvcTim 3.257669
sddf-SvcTim 3.196088
sded-SvcTim 3.185091
dm-62-SvcTim 3.042482
dm-102-SvcTim 2.965042
sdci-SvcTim 2.924413
sdo-SvcTim 2.844079
sdfi-SvcTim 2.806343
dm-53-SvcTim 2.719643
sdgg-SvcTim 2.695451
sdbk-SvcTim 2.689200
sdam-SvcTim 2.684454
sdge-SvcTim 2.669522
sdfj-SvcTim 2.599491
sdfd-SvcTim 2.570436
sdfg-SvcTim 2.553305
dm-42-SvcTim 2.528765
sddh-SvcTim 2.526681
sdfe-SvcTim 2.518579
dm-106-SvcTim 2.510244
sdgb-SvcTim 2.486052
sdgc-SvcTim 2.456882
sdgd-SvcTim 2.441255
sdbj-SvcTim 2.408612
sdbp-SvcTim 2.395879
sdek-SvcTim 2.391828
sdef-SvcTim 2.383030
sddm-SvcTim 2.373423
sddk-SvcTim 2.370066
sddn-SvcTim 2.369834
sdn-SvcTim 2.368561
TOP 50 Disks ordered by IO/S per second Peak
dm-99-Writes 7555
dm-103-Writes 6993
dm-105-Reads 1969
dm-114-Writes 1826
dm-113-Writes 1814
dm-109-Writes 1814
dm-105-Writes 1556
dm-100-Writes 1556
dm-104-Writes 1556
dm-112-Writes 1503
dm-108-Writes 1456
dm-1-Reads 1312
dm-45-Reads 1027
dm-46-Reads 1026
dm-63-Writes 1002
dm-67-Writes 993
dm-99-Reads 990
dm-100-Reads 990
dm-23-Reads 988
dm-104-Reads 979
dm-103-Reads 979
dm-47-Reads 976
dm-19-Reads 950
dm-42-Writes 936
dm-18-Writes 911
dm-29-Writes 789
dm-27-Reads 720
dm-22-Writes 695
dm-43-Reads 693
dm-53-Writes 681
dm-21-Reads 676
dm-47-Writes 642
dm-23-Writes 633
dm-46-Writes 632
dm-18-Reads 612
dm-43-Writes 597
dm-19-Writes 589
dm-22-Reads 575
sda-Reads 529
dm-20-Reads 527
dm-44-Reads 497
dm-69-Writes 486
dm-64-Writes 486
dm-68-Writes 486
dm-4-Writes 474
dm-114-Reads 473
dm-7-Reads 463
dm-44-Writes 461
dm-98-Writes 450
dm-45-Writes 444
TOP 50 Disks ordered by Qlen Peak
dm-104-QueLen 562241
dm-105-QueLen 6184
dm-100-QueLen 6173
dm-64-QueLen 1143
dm-69-QueLen 1036
dm-114-QueLen 487
dm-113-QueLen 454
dm-68-QueLen 441
dm-67-QueLen 400
dm-63-QueLen 321
dm-103-QueLen 314
dm-109-QueLen 260
dm-99-QueLen 240
dm-18-QueLen 151
dm-17-QueLen 106
dm-27-QueLen 99
dm-108-QueLen 86
dm-112-QueLen 75
dm-4-QueLen 68
sdbj-QueLen 68
dm-1-QueLen 64
dm-94-QueLen 60
dm-90-QueLen 57
dm-2-QueLen 51
sdau-QueLen 41
dm-7-QueLen 36
dm-23-QueLen 32
dm-66-QueLen 32
sdw-QueLen 30
dm-41-QueLen 30
sda-QueLen 30
dm-19-QueLen 28
dm-62-QueLen 27
sdbs-QueLen 26
sdcq-QueLen 26
dm-20-QueLen 26
dm-51-QueLen 25
dm-43-QueLen 25
dm-96-QueLen 25
dm-47-QueLen 24
dm-50-QueLen 24
sdn-QueLen 24
dm-22-QueLen 23
dm-44-QueLen 23
dm-26-QueLen 22
dm-21-QueLen 22
dm-45-QueLen 22
dm-46-QueLen 22
sdbp-QueLen 20
sdar-QueLen 20
TOP 50 Disks ordered by wait time Peak
dm-105-WaitW 2470265
dm-105-Wait 2470265
dm-104-Wait 2444147
dm-104-WaitW 2444147
dm-100-Wait 36587
dm-100-WaitW 36587
dm-66-Wait 2338
dm-66-WaitW 2338
dm-69-Wait 1872
dm-69-WaitW 1872
sdci-Wait 1862
sdci-WaitW 1862
dm-97-Wait 1862
dm-97-WaitW 1862
sdbj-Wait 1168
sdbj-WaitW 1168
dm-106-Wait 1136
dm-106-WaitW 1136
dm-47-WaitW 1084
sddl-Wait 1084
dm-47-Wait 1084
sddl-WaitW 1084
sdbk-WaitW 1021
sdo-WaitW 994
sdo-Wait 994
dm-18-Wait 932
sdn-WaitW 932
sdn-Wait 932
dm-18-WaitW 932
sdda-WaitR 875
sdda-Wait 875
dm-48-Wait 838
sdfi-Wait 838
sdfi-WaitW 838
dm-48-WaitW 838
sdfh-WaitW 835
sdfi-WaitR 830
dm-62-WaitW 824
dm-62-Wait 824
sdev-WaitR 815
sdev-Wait 815
sdeb-Wait 812
sdeb-WaitR 812
sdec-Wait 811
sdec-WaitR 811
sdch-WaitW 780
sdal-WaitW 779
sdu-WaitW 770
dm-65-Wait 745
dm-65-WaitW 745
TOP 50 Disks ordered by service time Peak
dm-105-SvcTim 9445
dm-104-SvcTim 9445
dm-66-SvcTim 2339
sdci-SvcTim 1863
dm-97-SvcTim 1863
sdbj-SvcTim 1168
dm-47-SvcTim 1086
sddl-SvcTim 1085
sdo-SvcTim 995
sdn-SvcTim 932
dm-18-SvcTim 932
sdda-SvcTim 875
dm-48-SvcTim 840
sdfi-SvcTim 839
sdev-SvcTim 816
sdeb-SvcTim 813
sdec-SvcTim 812
sdbd-SvcTim 683
sdfy-SvcTim 666
dm-44-SvcTim 643
sdey-SvcTim 639
sdbf-SvcTim 635
sdes-SvcTim 631
sdfp-SvcTim 620
sdgj-SvcTim 619
sdgf-SvcTim 614
sdge-SvcTim 614
sder-SvcTim 613
sdfq-SvcTim 612
sdbg-SvcTim 611
sdfr-SvcTim 611
sdcg-SvcTim 609
sdam-SvcTim 593
sdai-SvcTim 590
sdal-SvcTim 585
sdch-SvcTim 584
sdef-SvcTim 584
sdfn-SvcTim 580
sdcp-SvcTim 580
sdgd-SvcTim 579
sdbz-SvcTim 578
sdei-SvcTim 578
sdfs-SvcTim 577
sdcr-SvcTim 577
sdfl-SvcTim 571
sddk-SvcTim 568
sdl-SvcTim 564
sdi-SvcTim 562
sden-SvcTim 560
On Thu, Dec 1, 2022 at 10:46 AM Laurence Oberman loberman@redhat.com
wrote:
--
Laurence Oberman
Senior Principal Software Maintenance Engineer - Storage
Red Hat North America CEE-SEG
Thanks Laurence, I'll check that out. FWIW, attached is the kind of output we get from vmstat.
When I get data I use this baseline script to get text tables and sort them
etc.
!/bin/bash
echo Working on $1
collectl -scmnd -oT -p $1 > scmnd.$1
collectl -sc --verbose -oT -p $1 > sc-verbose.$1
collectl -sD -oT -p $1 > sD.$1
collectl -sZ -oT -p $1 > sZ.$1
The graphing tool produces these graphs
On Thu, Dec 1, 2022 at 10:55 AM Bill Torpey wallstprog@users.sourceforge.net wrote:
--
Laurence Oberman
Senior Principal Software Maintenance Engineer - Storage
Red Hat North America CEE-SEG
bill - did I read it right when you said you run collectl once for each
cpu? By default collectl ALWAYS records data for all CPUs.
laurence - maybe you already know this but if you're looking at disks stats
ordered by disk, colmux will to that in real time and if on
multiple systems will show you the busiest disks across all your systems.
bill, if you're finding things go faster by first decompressing your files
I suspect something may be wrong. I used to decompress before I found I
could give gnuplot a switch to read compressed files. I could plot a full
day;s data from dozens of collectl logs from multiple systems in under a
minutes, maybe even in 10s of seconds. almost makes me want to do a
video/demo to show you but I don't even have a laptop any more ;(
-mark
On Thu, Dec 1, 2022 at 11:30 AM Laurence Oberman loberman@users.sourceforge.net wrote:
Mark
Thank you, but things have evolved, we have huge systems with many CPUS and
many disks and the GZ files are huge
The decompress instream is slow.
I think it's been so long for you, that you have not seen the growth.
I have data from customers for example 4GB in size for a compressed raw
file and like 20GB or mor uncompressed
Regards
Laurence
On Thu, Dec 1, 2022 at 11:44 AM Mark Seger markseger@users.sourceforge.net
wrote:
--
Laurence Oberman
Senior Principal Software Maintenance Engineer - Storage
Red Hat North America CEE-SEG
Sorry if that wasn't clear: we run collectl once to get per-process cpu utilization (
--procopts kw4096 -sZ -oDG), a second time to get per-process memory utilization (--procopts mkw4096 -sZ -oDG), a third time to get system-wide network utilization (-sN -oD), and fourth to get vmstat (--export vmstat -oD). All of that is from the same.rawfile(s).Again, we did it that way just so we could run both collectl and our old monitoring tools and compare like-for-like.
We'll be changing that to generate plot-format files from the raw files, which means we'll only need to pass the raw file once. That's a win: for this 5-day test it takes >20 minutes for a single pass of the four raw files which total a bit over 10GB combined. (This on a CentOS 7 box with 4/8(HT) x i7-4770R CPU @ 3.20GHz, 16GB RAM and SSD. It's a bit quicker when we're not asking for per-process stats). Multiply that by 10 hosts and a full run takes a while.
We specify a procfilt in our
etc/collectl/confso only capture processes that we care about -- unfortunately for some of them we need to filter on the full command line. We are sampling once per-second, which takes very little overhead on the collection side, but it does impact post-processing.And yes, the command lines we capture/process are ridiculously large (4k) -- blame java and/or people who put all their jar's on the command line instead of in an environment variable. But we need the full command line to filter the proper processes.
Last but not least we can't use colplot directly -- we already have a set of standard gnuplot templates that we use, some of which pull data from multiple files (e.g., to compare latency against message rate). Not all data comes from collectl either -- we pull data from application logs and combine it with things like cpu and memory.
Again, thanks for collectl and for all your help!
I think you may be confused about collectl data collection. When you
collect process data your get a lot more than is displayed. You can then
play it back multiple times and choose to display different process options
from same collection, see --helpsuboopts.
Also, no need to run once to collectl process data and a second time for
system disk or other data. Just collectl it all in one go and play back
with --vmstat or -sd or -sz. I was very careful to design collectl so you
could collect everything at once and then play back as many times as you
want to with different options. If you run it more than once, you're making
much extra work for yourelf. Furthermore, the point is if you see something
funky in a process you want to be able to go back and see what the CPUs,
disks, memory, netork or even interrupts were doing at the exact same time.
I like to think if there was something you wished collectl could do, it
probably already does and you just haven't found the right switch yet ;)
-mark
On Thu, Dec 1, 2022 at 12:15 PM Bill Torpey wallstprog@users.sourceforge.net wrote:
Mark
Bill has a special use case and he knows about the collectl service for
sure that gathers everything.
He is scripting this so I think you misunderstood.
Regards
Laurence
On Fri, Dec 2, 2022 at 10:26 AM Mark Seger markseger@users.sourceforge.net
wrote:
--
Laurence Oberman
Senior Principal Software Maintenance Engineer - Storage
Red Hat North America CEE-SEG
Hmm, if your goal is to populate a database through multiple passes, can't
you just tell collectl to generate plotable output? Then you have everying
in formatted text output space separated so easily parsed. No need to
convert anything after it runs. Maybe
-mark
On Fri, Dec 2, 2022, 11:02 AM Bill Torpey wallstprog@users.sourceforge.net
wrote:
Released /collectl/collectl-4.3.6/collectl-4.3.6.src.tar.gz
18 minutes ago