Activity for Laurence Oberman
-
Laurence Oberman modified a comment on discussion Open Discussion
Happy New Year Please note, there will no longer be version updates on Sourceforge. All updates will be github only git clone https://github.com/sharkcz/collectl.git For problem reports please email loberman@redhat.com. Include a full description of the issue. Thanks Laurence
-
Happy New Year Please not, there will no longer be version updates on Sourceforge. All updates will be github only git clone https://github.com/sharkcz/collectl.git For problem reports please email loberman@redhat.com. Include a full description of the issue. Thanks Laurence
-
Hello Indeed if you have been watching sourceforge, I tried to keep it updated in multiple places but now have started primarily updating on github. git clone https://github.com/sharkcz/collectl.git cd coll* ./INSTALL For RHEL7 and higher: systemctl enable collectl systemctl start collectl Note that I do not have time to test on all different distros so if anything comes up let me know on Sourceforge or email me here. I still watch Sourceforge for problem reports. Regards Laurence Oberman On Tue,...
-
Hello Indeed if you have been watching sourceforge, I tried to keep it updated in multiple places but now have started primarily updating on github. git clone https://github.com/sharkcz/collectl.git cd coll* ./INSTALL For RHEL7 and higher: systemctl enable collectl systemctl start collectl Note that I do not have time to test on all different distros so if anything comes up let me know on Sourceforge or email me here. I still watch Sourceforge for problem reports. Regards Laurence Oberman
-
Note that it is as always, also available to be cloned here https://github.com/sharkcz/collectl.git
-
Added rbd devices Added additional network devices names Updated version number (forgot to do that on the last release)
-
collectl-4.3.7.src.tar.gz Released with an fix to include nvme devices that have native multipathing. commit cd8a94b61f5700945af8f4e6a3fcd0e4ea18f1e2 (HEAD -> master, origin/master, origin/HEAD) Author: Laurence Oberman loberman@redhat.com Date: Wed Jan 25 17:21:32 2023 -0500 Add the nvme native multipath devices filter to the collectl.conf "nvme\d+c\d+" Also available here got clone https://github.com/sharkcz/collectl.git
-
Mark Bill has a special use case and he knows about the collectl service for sure that gathers everything. He is scripting this so I think you misunderstood. Regards Laurence On Fri, Dec 2, 2022 at 10:26 AM Mark Seger markseger@users.sourceforge.net wrote: I think you may be confused about collectl data collection. When you collect process data your get a lot more than is displayed. You can then play it back multiple times and choose to display different process options from same collection, see...
-
Mark Thank you, but things have evolved, we have huge systems with many CPUS and many disks and the GZ files are huge The decompress instream is slow. I think it's been so long for you, that you have not seen the growth. I have data from customers for example 4GB in size for a compressed raw file and like 20GB or mor uncompressed Regards Laurence On Thu, Dec 1, 2022 at 11:44 AM Mark Seger markseger@users.sourceforge.net wrote: bill - did I read it right when you said you run collectl once for each...
-
When I get data I use this baseline script to get text tables and sort them etc. !/bin/bash echo Working on $1 collectl -scmnd -oT -p $1 > scmnd.$1 collectl -sc --verbose -oT -p $1 > sc-verbose.$1 collectl -sD -oT -p $1 > sD.$1 collectl -sZ -oT -p $1 > sZ.$1 The graphing tool produces these graphs On Thu, Dec 1, 2022 at 10:55 AM Bill Torpey wallstprog@users.sourceforge.net wrote: Thanks Laurence, I'll check that out. FWIW, attached is the kind of output we get from vmstat. Attachments: vmstat.png...
-
Bill I use collectl -sD to get all the per disk detailed I/O related stuff and I work deeply in storage performance at Red Hat. You don't need to use the vmstat stuff. My tool created tables like this TOP 50 Disks ordered by IO/S per second Average dm-114-Writes 556.033453 dm-109-Writes 540.127098 dm-113-Writes 540.126867 dm-103-Writes 478.489872 dm-99-Writes 478.481306 dm-53-Writes 317.075587 dm-29-Writes 315.691747 dm-111-Writes 272.861905 dm-107-Writes 270.897558 dm-67-Writes 253.888760 dm-63-Writes...
-
What really speeds things up is uncompressing the files before parsing them. I never parse as .gz. My tool uses the plot format too. I will see if I can sanitize it to share it. The thing is it uses numpy, scipy, pandas and matplotlib because it also does stats analysis On Thu, Dec 1, 2022 at 10:36 AM Bill Torpey wallstprog@users.sourceforge.net wrote: Hi Mark (and Laurence): First off, thanks for all the help, and for a great tool! Please see replies inline below: Bill - I guess the one overall...
-
My maintenance responsibilities: Fix reported bugs Adapt collectl as the kernel evolves the data structures. Enhance where needed, for example just last week made ceph rbd device capture a default So far kernel updates have not broken collectl and even the latest Fedora is seamless. I hope that is in sync with what you Mark wanted when I took over maintenance. Regards Laurence On Thu, Dec 1, 2022 at 9:17 AM Laurence Oberman loberman@users.sourceforge.net wrote: In case you missed it you can now use...
-
In case you missed it you can now use git clone https://github.com/sharkcz/collectl.git then git pull to get updates The last couple of updates commit 2001864971f91999216daf091529f5aa47d69150 (HEAD -> master, origin/master, origin/HEAD) Author: Laurence Oberman loberman@redhat.com Date: Tue Nov 29 18:29:26 2022 -0500 Version 4.3.6 with fix in sub vmstatInit where error check was removed commit 625e2453842121519e9e5a38c283d02e72e2ed89 Author: Laurence Oberman loberman@redhat.com Date: Tue Nov 29 18:19:44...
-
Here at Red Hat our use case is two fold.. If a customer reports a performance problem or any issue needing a deep dive, Install collect, let it gather and then investigate various performance problems after the customer reports the timestamps. The historical capture is very important. We also use it live if needed to do monitoring on a live performance problem. collectl -scmnd -oT -i1 used a lot collectl -sc --verbose -oT -i1 also used a lot collectl -sD -oT -i1 --dskfilt xxxx I have also been using...
-
Released /collectl/collectl-4.3.6/collectl-4.3.6.src.tar.gz 18 minutes ago
-
Here is the commit message in my local tree. commit 625e2453842121519e9e5a38c283d02e72e2ed89 (HEAD -> master) Author: Laurence Oberman loberman@redhat.com Date: Tue Nov 29 18:19:44 2022 -0500 Removed the error check in sub vmstatInit It was erroneous and that option is seldom used. Reported-by: Bill Torpey On Tue, Nov 29, 2022 at 6:19 PM Laurence Oberman loberman@users.sourceforge.net wrote: Seems it is going to always be called and will check user switches. It's always going to log that message...
-
Seems it is going to always be called and will check user switches. It's always going to log that message as the test is straight forward because $userSubsys will not be empty Example here with DEBUG and why it happens [loberman@lobep17 collectl]$ time /usr/bin/collectl -p '.raw.gz' --export vmstat -oD 2>&1 | tail 20221018 23:58:50 1 0 2304K 5843M 6668K 98859M 88006M 9417M 0 0 0 15 1055 1848 2 0 97 0 20221018 23:59:00 1 0 2304K 5844M 6668K 98859M 88006M 9417M 0 0 0 34 1027 1852 2 0 97 0 20221018...
-
[loberman@lobep17 collectl]$ time /usr/bin/collectl -p '*.raw.gz' --export vmstat -oD 2>&1 | tail 20221018 23:58:40 0 0 2304K 5844M 6668K 98859M 88006M 9417M 0 0 0 32 1159 1953 2 0 97 0 20221018 23:58:50 1 0 2304K 5843M 6668K 98859M 88006M 9417M 0 0 0 15 1055 1848 2 0 97 0 20221018 23:59:00 1 0 2304K 5844M 6668K 98859M 88006M 9417M 0 0 0 34 1027 1852 2 0 97 0 20221018 23:59:10 1 0 2304K 5828M 6668K 98874M 88006M 9417M 0 0 0 79 1071 1831 2 0 97 0 20221018 23:59:20 0 0 2304K 5830M 6668K 98874M 88006M...
-
Thanks Bill I will reproduce this and fix this. If I have any issues reproducing will come back to the thread. Your Reproducer noted [btorpey@bt-brix7 collectl]$ ls -lh *.raw -rw-r--r-- 1 btorpey shared 102M Nov 22 09:36 csappia01-20221110-223734.raw -rw-r--r-- 1 btorpey shared 2.0G Nov 22 09:36 csappia01-20221111-000000.raw -rw-r--r-- 1 btorpey shared 2.0G Nov 22 09:36 csappia01-20221112-000000.raw -rw-r--r-- 1 btorpey shared 1.1G Nov 22 09:36 csappia01-20221113-000000.raw The result of running...
-
Nevermind The docs say that already Playback.html:need to quote the string if it contains spaces or wildcard characters.
-
Bill I tested with collectl -scmnd -oT -p "lobep17-*.raw" 1> l 2>&1 and its picking them all up. If you confirm you are good with that I will update the documentation to reflect that Wildcards for -p need to be enclosed in "" Thanks Laurence
-
So the Quotes are needed, meaning we should modify documentation and there is no bug right ?
-
OK, but my tests how no expansion for the wildcard, only one file was used. Nevertheless if there is a bug provide a reproducer and I will investigate and fix it. collectl -scmnd -oT -p lobep17-*.raw 1> l 2>&1 This only parses the first file matched on the wildcard * [loberman@lobep17 Downloads]$ ls lobep17-*.raw lobep17-20221018-113317.raw lobep17-20221019-000000.raw lobep17-20221020-000000.raw lobep17-20221021-000000.raw
-
Wildcards are not expanded for -p, it will use the first file it matches on only, so to answer your question. Do not use wildcards for -p
-
I never use wildcards for p but in the test I ran it picked up the first match on *.raw
-
OK, I have 4 files now in my test directory -rw-r--r-- 1 loberman loberman 827215610 Nov 16 10:36 lobep17-20221020-000000.raw -rw-r--r-- 1 loberman loberman 826461531 Nov 16 10:36 lobep17-20221019-000000.raw -rw-r--r-- 1 loberman loberman 436974087 Nov 16 10:36 lobep17-20221018-113317.raw -rw-rw-r-- 1 loberman loberman 292256261 Nov 16 10:05 lobep17-20221021-000000.raw [loberman@lobep17 Downloads]$ ls *.raw lobep17-20221018-113317.raw lobep17-20221019-000000.raw lobep17-20221020-000000.raw lobep17-20221021-000000.raw...
-
OK its working fine for me. You must have another set of files its matching in there. [loberman@lobep17 Downloads]$ collectl -p lobep17-20221021-000000.raw --export vmstat -oD # procs ---------------memory (KB)--------------- --swaps-- -----io---- --system-- ----cpu----- #Date Time r b swpd free buff cache inact active si so bi bo in cs us sy id wa 20221021 00:00:10 0 0 26624K 9539M 6332K 96618M 63982M 14931M 0 0 0 150 3890 4423 2 0 96 0 20221021 00:00:20 0 0 26624K 9205M 6332K 96941M 64305M 14931M...
-
Hello Bill Thanks for reporting this. As soon as I get a chance will look into it. Having said that I never use the vmstat export function so there may be an issue that has been lurking in there for a while already.
-
Hello November 7 2022 If you want to clone collectl git clone https://github.com/sharkcz/collectl.git I have added the tar file for colplot to the git tree. You will always get the latest version with a git pull after the clone
-
Collectl was modified to day to include default support for rbd devices (Ceph disks) Testing will happen this weekend and then I will release a new version. Regards Laurence
-
For documentation there is a docs directory that will come along too
-
Hi Mark Everything will move My current git tree has [loberman@lobep17 collectl]$ ls ARTISTIC collectl.conf docs gexpr.ph hello.ph lexpr.ph proctree.ph RELEASE-collectl UNINSTALL vnet.ph client.pl colmux envrules.std GPL initd man1 README service vmstat.ph collectl COPYING formatit.ph graphite.ph INSTALL misc.ph README-WINDOWS statsd.ph vmsum.ph So if one clones it you get a copy of what was in the tar file
-
Hello Folks, When I took over keeping collectl alive, I really did not get involved in any of the SourceForge Admin functions Mark used to do. So releasing fixes and updates means I am not able to update all the proper LINKS and messages etc. It would be a lot easier for me to use a git repo for this so I am letting you know that collectl will probably move to a git repo soon. It means you would use git clone and then got pull to get the updates. We don't provide RPMS anymore, just the INSTALL script...
-
Bud Brown, my colleague has provided a patch for a new level of precision output. I will be adding this in the next release His note is as follows Patch: add --dskopts=F extended precision option, a bigger --dskopts=f The existing --dskopts=f option increases precision in some columns (Example #2 vs #1), to 1 digit. The proposed --dskopts=F option increases precision in all columns (Example #3 vs #2), to 2 digits similar to iostat output. In also removes the contraction of large iops and KB/s numbers....
-
Note that his comments should reflect diskopts not dskopt, that will be fixed when I release.
-
Bud Brown, my colleague has provided a patch for a new level of precision output. I will be adding this in the next release His note is as follows Patch: add --dskopt=F extended precision option, a bigger --dskopt=f The existing --dskopt=f option increases precision in some columns (Example #2 vs #1), to 1 digit. The proposed --dskopt=F option increases precision in all columns (Example #3 vs #2), to 2 digits similar to iostat output. In also removes the contraction of large iops and KB/s numbers....
-
Finally got around to this vmstat Swap si: Amount of memory swapped in from disk (/s). so: Amount of memory swapped to disk (/s). vmstat swap Time si so 16:02:27 256 77996 Collectl # <---Swap--> In Out 16:02:27 256M 78G @Manabu, Thank you Something is not right, thanks for reporting this, I will decide how best to fix and patch it. Captured data if good, it's the playback that needs checking like you reported.
-
Apologies, I have been buried doing my day job. I intend to make time to investigate this later this week. Regards Laurence
-
Hello Only just saw this post. I will look into this and it merits a change I will patch it. Either way I will report back to this thread. Regards Laurence
-
commit 517c19c94ae623eb7ab373edff1563328d03eb2b Author: Laurence Oberman loberman@redhat.com Date: Fri Apr 1 11:52:02 2022 -0400 Patch to fix IB collection in systems with multiple mellanox cards suggested-by: Gary Skouson diff --git a/collectl b/collectl index 86850e42e9e5..71808794b049 100755 --- a/collectl +++ b/collectl @@ -2978,7 +2978,7 @@ for (; $count!=0 && !$doneFlag; $count--) # for OPA V4, /sys counters always 64 bits if ($HCAOpaV4[$i][$j]) { - my $proc="$SysIB/$HCAName[$i]0/ports/$j/counters";...
-
Hello Gary Thanks the patch looks fine, I will commit and add you as a suggested-by. I will update the version number and upload to sourceforge and let you know when its done. After taking over from Mark, I still need to get with him re: the way he used to do the updates. For now I am adding the tar files and alerting that there is a new version when I push the changes. Regards Laurence
-
Hello Thanks for the patch, I will review and commit if all is good. Best regards Laurence Oberman On Fri, Mar 25, 2022 at 8:15 AM Gary Skouson gskouson@users.sourceforge.net wrote: Here's a patch to fix a problem with IB. The use of HCAName isn't right and should be HCAId, especially if you have mellanox ethernet and IB in the same box, the '0' or $i isn't going to work right. For example on one system, we have bonded ethernet on a mellanox ethernet card and a separate IB card, so /sys/class/infiniband...
-
Hello Thanks for the patch, I will review and commit if all is good. Best regards Laurence Oberman
-
I need to get with Mark and see how he did releases here. I did fix a couple of things in 4.3.2 Regards Laurence
-
A quick look at a raw capture shows Slab shmem_inode_cache 2504 2664 680 24 4 : tunables 0 0 0 : slabdata 111 111 0 Shmem: 115332208 kB nr_shmem 28833052 smaps are per library, we don't get them right now if I am reading this correctly. 7ffc37de1000-7ffc37de3000 r-xp 00000000 00:00 0 [vdso] Size: 8 kB Rss: 4 kB Pss: 0 kB Shared_Clean: 4 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 4 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4...
-
I think what Bill is looking is is if we gather /proc/PID/* meaning is there a structure in perl we may have the data in, and we are just making it available for output. I have not had a chance to have a full look at this but I dont think we grab it all. More later, will try investigate this weekend
-
I don't think its generic memory details Bill is after, rather the shared memory specifics per process. collectl does not capture that granular data. I would have to look at adding the /proc/PID/smaps stuff etc. Regards Laurence
-
Hello Are you asking about the process related memory view via sZ, its not available there. collectl gets the entire /proc/meminfo capture but that wont help with shared memory per process, will just show shared. Collectl can't really help here What you need is the ipcs command to understand what shared memory is reserved. The best you can do is ipcs -c -m -p root@loberhel ~]# ipcs -c -m -p ------ Shared Memory Creator/Last-op PIDs -------- shmid owner cpid lpid 94175235 loberman 10430 2489 89522180...
-
I will look into this Regards Laurence
-
Hello I tested this without --procopts -w2000 I get the truncated cmdline 17:51:00 9314 loberman 20 8398 0 S 110M 1M 4 0.00 0.00 0 00:00.00 0 0 0 3 /bin/bash ./coltest.sh 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200123...
-
Subscribed, Hopefully I will figure out why I dont get the messages lately
-
Hello If you want to look at a raw.gz that was not properly closed you will need to do this cat xxx.raw.gz | gunzip > xxx.raw Then you can check the argument length I looked at the code, the limit is not in the gathering, rather in the playback its set to 1000 if no other options are given for --procopts collectl: $procCmdWidth=($procOpts=~s/w(\d+)/w/) ? $1 : 1000; w - widen display by including whole argument string, with optional max width *** I have not tested this but if you include --procopts...
-
Hello I am not sure why I never saw this on the list. As far as I know I am subscribed. I will look into this. i.e. Is the truncation within actual collectl data gathering or only on playback. If the limit is within collectl gather and not playback then in increasing it would mean practically growing the on-disk data capture size so will have to think about that because if we have many 1000''s of tasks it's a lot of impact on storage and capture. Regards Laurence On Mon, Jan 10, 2022 at 4:04 PM Mark...
-
I committed the last patch, will revert if it still has issues, but cant see why this would not work, given the test script I wrote. commit 19e977b7a8fa2c93575b5f9247d9a1250f031cb9 Author: Laurence Oberman <loberman@redhat.com> Date: Fri Oct 8 11:35:52 2021 -0400 Deal with the bug where last was called in the perf_query code for Infiniband and was not in a loop https://sourceforge.net/p/collectl/discussion/696864/thread/fc58a31168/?limit=25#2f7c diff --git a/formatit.ph b/formatit.ph index db1776eb5385..045eef85dd6d...
-
Apologies, Looks like I put the label in an unreachable place. Please try this, attached patched file New Patch --- formatit.ph.orig 2021-10-05 09:42:58.307749832 -0400 +++ formatit.ph 2021-10-06 10:57:13.969606012 -0400 @@ -375,12 +375,12 @@ sub initRecord $message="Required module missing" if $temp=~/required by/; $message="No such file or directory" if $temp=~/No such file/; if ($message ne '') - { + {EXIT_IF:{ disableSubsys('x', "perfquery error: $message!"); $mellanoxFlag=0; $PQuery=''; - last;...
-
Hello Thanks for testing Let me review and figure out what is wrong, I will emulate the issue so I can test it myself this time. Regards laurence On Wed, Oct 6, 2021 at 10:37 AM Edgar Dean vonschae@users.sourceforge.net wrote: Hi Lawrence, We applied the fix file and restarted the service however we still getting errors: systemctl status collectl collectl.service - collectl metric collection Loaded: loaded (/usr/lib/systemd/system/collectl.service; enabled; vendor preset: disabled) Active: failed...
-
Hello If I properly understood Marks suggestion then a fix like this patch should work. Can you please try it and get back to me. I don't have a way to reproduce to test myself right now. --- formatit.ph 2021-10-04 19:42:48.898289477 -0400 +++ formatit.ph.fix 2021-10-04 19:42:32.366220471 -0400 @@ -342,6 +342,7 @@ sub initRecord $PQopt = 'sys' if -e "$firstHCA/counters_ext" && !($debug & 16384); # We usually only care about perfquery for non-extended counters + EXIT_IF: if ($PQopt eq '-r') { # no...
-
Hello Looking at Marks suggested fix tonight. I see that its uncommon to hit this as it requires the system have IB etc. We are still in the process of deciding how to handle the fixes for collectl upstream but I have added myself now to the email list so I see the bug reports etc. Regards Laurence Oberman
1
