Menu
▾
▴
moosefs-users — Mailing list for MooseFS Users
You can subscribe to this list here.
| 2009 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
(4) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2010 |
Jan
(20) |
Feb
(11) |
Mar
(11) |
Apr
(9) |
May
(22) |
Jun
(85) |
Jul
(94) |
Aug
(80) |
Sep
(72) |
Oct
(64) |
Nov
(69) |
Dec
(89) |
| 2011 |
Jan
(72) |
Feb
(109) |
Mar
(116) |
Apr
(117) |
May
(117) |
Jun
(102) |
Jul
(91) |
Aug
(72) |
Sep
(51) |
Oct
(41) |
Nov
(55) |
Dec
(74) |
| 2012 |
Jan
(45) |
Feb
(77) |
Mar
(99) |
Apr
(113) |
May
(132) |
Jun
(75) |
Jul
(70) |
Aug
(58) |
Sep
(58) |
Oct
(37) |
Nov
(51) |
Dec
(15) |
| 2013 |
Jan
(28) |
Feb
(16) |
Mar
(25) |
Apr
(38) |
May
(23) |
Jun
(39) |
Jul
(42) |
Aug
(19) |
Sep
(41) |
Oct
(31) |
Nov
(18) |
Dec
(18) |
| 2014 |
Jan
(17) |
Feb
(19) |
Mar
(39) |
Apr
(16) |
May
(10) |
Jun
(13) |
Jul
(17) |
Aug
(13) |
Sep
(8) |
Oct
(53) |
Nov
(23) |
Dec
(7) |
| 2015 |
Jan
(35) |
Feb
(13) |
Mar
(14) |
Apr
(56) |
May
(8) |
Jun
(18) |
Jul
(26) |
Aug
(33) |
Sep
(40) |
Oct
(37) |
Nov
(24) |
Dec
(20) |
| 2016 |
Jan
(38) |
Feb
(20) |
Mar
(25) |
Apr
(14) |
May
(6) |
Jun
(36) |
Jul
(27) |
Aug
(19) |
Sep
(36) |
Oct
(24) |
Nov
(15) |
Dec
(16) |
| 2017 |
Jan
(8) |
Feb
(13) |
Mar
(17) |
Apr
(20) |
May
(28) |
Jun
(10) |
Jul
(20) |
Aug
(3) |
Sep
(18) |
Oct
(8) |
Nov
|
Dec
(5) |
| 2018 |
Jan
(15) |
Feb
(9) |
Mar
(12) |
Apr
(7) |
May
(123) |
Jun
(41) |
Jul
|
Aug
(14) |
Sep
|
Oct
(15) |
Nov
|
Dec
(7) |
| 2019 |
Jan
(2) |
Feb
(9) |
Mar
(2) |
Apr
(9) |
May
|
Jun
|
Jul
(2) |
Aug
|
Sep
(6) |
Oct
(1) |
Nov
(12) |
Dec
(2) |
| 2020 |
Jan
(2) |
Feb
|
Mar
|
Apr
(3) |
May
|
Jun
(4) |
Jul
(4) |
Aug
(1) |
Sep
(18) |
Oct
(2) |
Nov
|
Dec
|
| 2021 |
Jan
|
Feb
(3) |
Mar
|
Apr
|
May
|
Jun
|
Jul
(6) |
Aug
|
Sep
(5) |
Oct
(5) |
Nov
(3) |
Dec
|
| 2022 |
Jan
|
Feb
|
Mar
(3) |
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|
From: Ricardo J. B. <ric...@do...> - 2022-03-08 15:31:34
|
El martes, 8 de marzo de 2022 10:13:21 -03 David Myer escribió: > Hi Ricardo, > > Thanks for the reply. My preference is to reload the OS rather than upgrade > - in my experience it is too good to be true. I will be going to Almalinux > though. In my (little) experience, the switch from CentOS to Alma/Rocky works fine, I did a couple of tests because I was intrigued (I didn't have any CentOS 8 production machine at the time). It's basically changing where the mirrors point and doing dnf update/distro- sync. > I'd also like to do this just so I've experience doing it in the event of an > emergency (like when the master's OS has to be reloaded due to the OS disk > failing). Fair enough. > Do you know the directories I should backup? And should I just be backing > them up as a contingency, or do I actually want to copy them back onto the > new master? That depends on your MFS configs and your partitioning setup. If you didn't modify MFS configs you should backup /etc/mfs and /var/lib/mfs and probably /etc/fstab (I would backup the entire /etc directory just in case). If you have /var/lib/mfs in a separate disk/partition you don't need a backup, as you can skip formatting it during reinstall, but it is a good practice nonetheless. > Thanks, > David > > Sent with ProtonMail Secure Email. > > ------- Original Message ------- > > On Monday, March 7th, 2022 at 7:44 PM, Ricardo J. Barberis <ric...@do...> wrote: > > El lunes, 7 de marzo de 2022 19:37:33 -03 David Myer via moosefs-users > > > > escribió: > > > Dear MooseFS-Users, > > > > > > I need to reload the operating system on my master. As some of you may > > > know, > > > > > > CentOS 8 rather abruptly became EOL, and various repos have stopped > > > working > > > > > > for it. I'm encountering more and more issues so I thought I'd jump ship > > > > > > and move to a new Linux distro before I regret it. > > > > > > In terms of reloading the OS, what would happen if I shutdown > > > > > > moosefs-master, shutdown the linux server, reloaded it with a new OS and > > > > > > loaded moosefs-master back on to it? Is it as simple as this, or would I > > > > > > need to backup various master metadata before doing this? > > > > > > I'm aware that switching masters is risky, so I am fine having some > > > downtime > > > > > > while shutting down and re-installing the master onto the same machine. > > > > > > My cluster looks like this: > > > > > > - server 1: master+chunkserver > > > > > > - server 2: metalogger+chunkserver > > > > > > - server 3: metalogger+chunkserver > > > > > > Any help appreciated. > > > > > > Thanks, > > > > > > David > > > > Hi David, > > > > Instead of reinstalling, you could switch to AlmaLinux or Rocky Linux: > > > > https://www.cyberciti.biz/howto/how-to-migrate-from-centos-8-to-almalinux-> > conversion/ > > > > https://www.cyberciti.biz/howto/migrate-from-centos-8-to-rocky-linux-conve > > rsion/ > > > > Anyway, backing up the configs, metadata and changelogs before > > reinstalling or > > > > switching is never a bad idea. > > > > Cheers, -- Ricardo J. Barberis Senior SysAdmin / IT Architect DonWeb.com La Actitud Es Todo _____ |
|
From: Ricardo J. B. <ric...@do...> - 2022-03-08 01:00:25
|
El lunes, 7 de marzo de 2022 19:37:33 -03 David Myer via moosefs-users escribió: > Dear MooseFS-Users, > > I need to reload the operating system on my master. As some of you may know, > CentOS 8 rather abruptly became EOL, and various repos have stopped working > for it. I'm encountering more and more issues so I thought I'd jump ship > and move to a new Linux distro before I regret it. > > In terms of reloading the OS, what would happen if I shutdown > moosefs-master, shutdown the linux server, reloaded it with a new OS and > loaded moosefs-master back on to it? Is it as simple as this, or would I > need to backup various master metadata before doing this? > > I'm aware that switching masters is risky, so I am fine having some downtime > while shutting down and re-installing the master onto the same machine. > > My cluster looks like this: > > - server 1: master+chunkserver > - server 2: metalogger+chunkserver > - server 3: metalogger+chunkserver > > Any help appreciated. > > Thanks, > David Hi David, Instead of reinstalling, you could switch to AlmaLinux or Rocky Linux: https://www.cyberciti.biz/howto/how-to-migrate-from-centos-8-to-almalinux-conversion/ https://www.cyberciti.biz/howto/migrate-from-centos-8-to-rocky-linux-conversion/ Anyway, backing up the configs, metadata and changelogs before reinstalling or switching is never a bad idea. Cheers, -- Ricardo J. Barberis Senior SysAdmin / IT Architect DonWeb.com La Actitud Es Todo _____ |
|
From: David M. <dav...@pr...> - 2022-03-07 22:37:51
|
Dear MooseFS-Users, I need to reload the operating system on my master. As some of you may know, CentOS 8 rather abruptly became EOL, and various repos have stopped working for it. I'm encountering more and more issues so I thought I'd jump ship and move to a new Linux distro before I regret it. In terms of reloading the OS, what would happen if I shutdown moosefs-master, shutdown the linux server, reloaded it with a new OS and loaded moosefs-master back on to it? Is it as simple as this, or would I need to backup various master metadata before doing this? I'm aware that switching masters is risky, so I am fine having some downtime while shutting down and re-installing the master onto the same machine. My cluster looks like this: - server 1: master+chunkserver - server 2: metalogger+chunkserver - server 3: metalogger+chunkserver Any help appreciated. Thanks, David Sent with [ProtonMail](https://protonmail.com/) Secure Email. |
|
From: William K. <bi...@ii...> - 2021-11-17 23:47:00
|
That will take a long time but will work - mixing OS is not a problem. Do one and see how it goes. Then add the other 6 and at the same time mark the remainder for removal. The data will move around and eventually the older systems will notify they are ready for removal. One caveat - this will add to the background cluster load - avoid doing all at once on a very busy cluster as things will noticeably slow down. I usually just add the new system(s) all at once. Then shut down the chunkserver process on the first system you want to remove. If there are any missing chunks showing up restart and remove from maintenance etc. and try again. If there are no missing chunks, let the cluster rebalance and catch up. (I currently have 6 chunkservers and can usually remove at least two at once without losing any chunks.) Rinse and repeat - I am usually too impatient to wait the many hours it takes for a large disk marked for removal to empty ... and I have backups :) BillK On 18/11/21 3:31 am, WK wrote: > We have a very old CentOS7 based MFS system > > 1 Master > > 1 Metalogger/backup Master > > 7 chunkservers. > > The MFS software is reasonably up to date they are all at least 3.0.115 > > The hardware was OLD when we put the system in place, as we reused > older servers we were replacing. > > So now that equipment is REALLY old and we are beginning the process > of updating them. > > We would like to use Ubuntu20 for the OS on the newer hardware. > > Our initial plan would be to introduce a new U20 chunkserver, get it > replicated up and then remove one of the older C7 chunkservers. > > then wash, rinse repeat for the others for a few weeks until everybody > is caught up. > > Is there a problem mixing Cent7 and U20 systems on the same network in > that fashion? > > Can we start with the chunkservers or should we immediately replace > the Master with U20 and then start replacing the chunkservers? > > sincerely, > > -wk > > > > > _________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users |
|
From: Agata Kruszona-Z. <ch...@mo...> - 2021-11-17 20:39:41
|
Hi, First, I would like to invite you to github. Most of the conversation moved there a long time ago. Here is our github discussions page: https://github.com/moosefs/moosefs/discussions To answer your question: it's absolutely not a problem to mix and match OSes on machines with MFS components. In our lab we have a mix of Ubuntu, Centos and Debian with a FreeBSD machine on top ;) Everything goes, as long as it can run mfs master or mfs chunkserver. The only thing to remember is, if you also want to upgrade mfs itself while you are upgrading your oses, this should start with your metalogger, then master, then chunkservers, then client processes. In other words, you shouldn't try to connect a newer chunkserver or client to an older master. Regards, Agata W dniu 17.11.2021 o 20:31, WK pisze: > We have a very old CentOS7 based MFS system > > 1 Master > > 1 Metalogger/backup Master > > 7 chunkservers. > > The MFS software is reasonably up to date they are all at least 3.0.115 > > The hardware was OLD when we put the system in place, as we reused older > servers we were replacing. > > So now that equipment is REALLY old and we are beginning the process of > updating them. > > We would like to use Ubuntu20 for the OS on the newer hardware. > > Our initial plan would be to introduce a new U20 chunkserver, get it > replicated up and then remove one of the older C7 chunkservers. > > then wash, rinse repeat for the others for a few weeks until everybody > is caught up. > > Is there a problem mixing Cent7 and U20 systems on the same network in > that fashion? > > Can we start with the chunkservers or should we immediately replace the > Master with U20 and then start replacing the chunkservers? > > sincerely, > > -wk > > > > > _________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users -- -- Agata Kruszona-Zawadzka MooseFS Team |
|
From: WK <wk...@bn...> - 2021-11-17 19:47:39
|
We have a very old CentOS7 based MFS system 1 Master 1 Metalogger/backup Master 7 chunkservers. The MFS software is reasonably up to date they are all at least 3.0.115 The hardware was OLD when we put the system in place, as we reused older servers we were replacing. So now that equipment is REALLY old and we are beginning the process of updating them. We would like to use Ubuntu20 for the OS on the newer hardware. Our initial plan would be to introduce a new U20 chunkserver, get it replicated up and then remove one of the older C7 chunkservers. then wash, rinse repeat for the others for a few weeks until everybody is caught up. Is there a problem mixing Cent7 and U20 systems on the same network in that fashion? Can we start with the chunkservers or should we immediately replace the Master with U20 and then start replacing the chunkservers? sincerely, -wk |
|
From: Jacob D. <jac...@ci...> - 2021-10-26 17:10:50
|
Hi Agata, thanks for your feedback, it is helping a lot to be able to assess my own thoughts. Also, I’ll head over to github for the next topic =) I’m well aware that moosefs is not intended for peak single stream performance and was aware that I’d have to take a significant hit on throughput. I had hoped to reach the 10Gbit/s from a single client and therefore wanted to understand what bottlenecks could be overcome, or if it’s a deadend where I basically can’t do anything to push it further. I’ll have to set up a nfs/smb gateway so a lot of traffic will have be tunneled through one client. We also did some tests with higher numjobs but performance does only scale a little. With "--numjobs=12" I get from about 500MB/s to about 700MB/s, which only improves things a little for a lot if parallel threads. Funny sidenote. Changing “–size” to 1m suddenly gives me 11GB/s in read from the “fourth” client machine that is connected via 100Gbit/s. Not sure what’s happening there and how to interprete that. Maybe some difficulties with the synthetic fio and the underlying mechanics. Cache to cache or something maybe. “Starting 1 process Jobs: 1 (f=1): [R(1)][10.0%][r=11.0GiB/s][r=181k IOPS][eta 00m:27s] “ …I guess synthetic performance tests are always a bit difficult to be mapped to real world performance of applications. Will try to roll mfs out to some real clients and start testing there. Main take away to me is that there are no specific config settings with which I could try to speed up things. Limit is HW/Architecture/(stacked)Latency based and if I want to get faster that would be the spot to check. Best Jacob From: Agata Kruszona-Zawadzka <ch...@mo...> Date: Tuesday, 26. October 2021 at 11:39 To: Jacob Dietz <jac...@ci...>, moo...@li... <moo...@li...> Subject: Re: [MooseFS-Users] Understanding moose's performance caps Hi Jacob, First of all, I would like to encourage you to join the discussions section on our github page: https://github.com/moosefs/moosefs/discussions This list isn't active anymore, everybody seem to have moved to github nowadays ;) Performance discussions happen there on the regular. Second of all, some basics: MooseFS is a network file system, designed to handle large amounts of data and large number of parallel operations - this is where its strength lays. You will not get better or even comparable performance to a fast hard drive if you test a setup of single master, single chunkserver, single mfsclient and a single thread/process performing i/o. When you read or write data to a hard drive, you send your requests to the kernel and the kernel performs the operations on your hard drive. When you read or write data on MooseFS, you send the requests to the kernel, which forwards them to the FUSE* module, which communicates with MFS mount, which talks via network first with the master, to find out where the data is, then with chunkservers to actually handle the reads and writes. Latency is crucial here, but even with a very small one, the network communication limits the i/o speed. So single thread i/o speed is limited. On the other hand a single drive has limited read and write speed, and if it is a mechanical drive, it doesn't do well with parallel operations at all. MooseFS client will run many parallel operations at the same time and while each is burdened by the network overhead, each one of them will be as fast as that single one (up to a point, of course, but depending on your hardware, you can run really a lot of them without performance drop). Plus, you can mount several clients on one machine and run many machines with clients. The scalability of your total i/o on the system is almost indefinite. *FUSE has a hard limit of 4GB/sec when it comes to i/o speed, this is because FUSE uses a single communication channel (single pipe) to talk to kernel. So this is also a hard limit of a single mfs client if you don't use the mfsio library to communicate. But you can run many clients on one machine. You run your test with "--numjobs=1". Try to run many thereads in parallel and see the performance then :) Run several instances of this test (or any other test) at the same time on one client, on many clients. You will start to see the power in the system. The reason why your single threaded i/o is faster when the client is on master server rather than on the chunkserver is also easy to explain: the overhead for one network connection is more or less constant. With master the client talks in small packets - those are control information, about updating metadata information (change file length, file properties like atime, mtime etc.), about the location of the data itself. To the chunkserver the client sends the actual file data - larger packets. So if your client has lower latency when it talks to the master (and this will be true if they are on the same machine), the overall performance of your single threaded i/o will be better. But again - this is not what MooseFS is for. MooseFS is for many threads, many parallel operations. Regards, Agata W dniu 25.10.2021 o 20:46, Jacob Dietz via moosefs-users pisze: Short update. Further testing with two additional clints makes my latency theory unlikely. Third client (same hw as master) shows pretty much the same performance pattern as the master. Fourth client (same hw as chunk server0) shows pretty much the same performance pattern as the chunk server. * Read remains some global bottleneck. * Write seems to be hardware related. Best Jacob From: Jacob Dietz <jac...@ci...><mailto:jac...@ci...> Date: Monday, 25. October 2021 at 20:09 To: Jacob Dietz via moosefs-users <moo...@li...><mailto:moo...@li...> Subject: Understanding moose's performance caps Hi everyone, I’m testing a moosefs setup and am trying to understand the performance values I get and the possible bottlenecks. Tried out quite some things but am a little stuck. Status quo for test setup is: Mfs master server - 10G Interface - Mfsmaster config is default Mfs chunk server - 100G Interface - 8x Sata SSD Raid - Mfschunkserver config default To rule out any client or additional network issues I’m testing on the chunk server. Testing with fio gives me a stable 500MB/s read and 700MB/s write. (“sudo fio --filename=/mnt/mfs_mount/123 --direct=1 --rw=read --bs=64k --ioengine=libaio --iodepth=64 --runtime=30 --numjobs=1 --time_based --group_reporting --name=throughput-test-job --eta-newline=1 --size=50m”) Same test on the xfs gives me about 8GB/s read and 5GB/s write. Utilization of the ssd array is zero during testing, so everything seems to be handled in cache as fio probably deletes everything instantly. Testings: Seeing the xfs performance reserve and the idling array we tried to get rid of the cache with “mfsmount /mnt/mfs_mount/ -H mfsmaster -o mfscachemode=DIRECT”, which gave us the same results. Trying to increase the cache instead with -o mfsreadaheadsize=2048 -o mfsreadaheadleng=2048576 also gave no significant difference. Upgrading the nice level of mfsmount to 0 or even 2 also didn’t change performance. Trying to increase workers on chunkserver config didn’t change performance WORKERS_MAX = 500 WORKERS_MAX_IDLE = 80 Also tried reducing CHUNKS_LOOP_MIN_TIME = 150 on mfsmaster config but still no change. Throughout the tests I couldn’t see any cpu cores capping. Also Tried to run via a second network connection (same 100GB/10GB) without jumbo frames to rule out any issues on that side. Doing the mfsmount on the master server gave me pretty accurately the same read performance. Write was strangely doubled to 1,5GB/s, which is interesting as it only has a 10G interface. Guesses: I’m pretty new to moosefs and still trying to wrap my head around it but to me, this seems like some cap I’m running against as it’s so steady and reproduceable. Shouldn’t be the cache, as ram speed cap wouldn’t make sense. Shouldn’t be the ssd array. Shouldn’t be cpu as thereads are far from capping. The increased write on the master server indicates that it could be the latency between the two servers. Read is similar on both machines as they need to communicate either way. Write is increased on master beyond his nic capabilities, possibly because he’s only committing the writes to himself as fio deletes the data before it is even sent outside. Array idling during all tests is backing this theory. That said, ping is between 0.3 and 0.2 ms. Sorry for the long post I hope it’s still readable. Would be great if anyone could point me the way to understand the bottleneck(s) I’m facing and how to overcome it. Could latency be the right path? Thanks! Best Jacob _________________________________________ moosefs-users mailing list moo...@li...<mailto:moo...@li...> https://lists.sourceforge.net/lists/listinfo/moosefs-users -- -- Agata Kruszona-Zawadzka MooseFS Team |
|
From: Agata Kruszona-Z. <ch...@mo...> - 2021-10-26 11:37:04
|
Hi Jacob, First of all, I would like to encourage you to join the discussions section on our github page: https://github.com/moosefs/moosefs/discussions This list isn't active anymore, everybody seem to have moved to github nowadays ;) Performance discussions happen there on the regular. Second of all, some basics: MooseFS is a network file system, designed to handle large amounts of data and large number of parallel operations - this is where its strength lays. You will not get better or even comparable performance to a fast hard drive if you test a setup of single master, single chunkserver, single mfsclient and a single thread/process performing i/o. When you read or write data to a hard drive, you send your requests to the kernel and the kernel performs the operations on your hard drive. When you read or write data on MooseFS, you send the requests to the kernel, which forwards them to the FUSE* module, which communicates with MFS mount, which talks via network first with the master, to find out where the data is, then with chunkservers to actually handle the reads and writes. Latency is crucial here, but even with a very small one, the network communication limits the i/o speed. So single thread i/o speed is limited. On the other hand a single drive has limited read and write speed, and if it is a mechanical drive, it doesn't do well with parallel operations at all. MooseFS client will run many parallel operations at the same time and while each is burdened by the network overhead, each one of them will be as fast as that single one (up to a point, of course, but depending on your hardware, you can run really a lot of them without performance drop). Plus, you can mount several clients on one machine and run many machines with clients. The scalability of your total i/o on the system is almost indefinite. *FUSE has a hard limit of 4GB/sec when it comes to i/o speed, this is because FUSE uses a single communication channel (single pipe) to talk to kernel. So this is also a hard limit of a single mfs client if you don't use the mfsio library to communicate. But you can run many clients on one machine. You run your test with "--numjobs=1". Try to run many thereads in parallel and see the performance then :) Run several instances of this test (or any other test) at the same time on one client, on many clients. You will start to see the power in the system. The reason why your single threaded i/o is faster when the client is on master server rather than on the chunkserver is also easy to explain: the overhead for one network connection is more or less constant. With master the client talks in small packets - those are control information, about updating metadata information (change file length, file properties like atime, mtime etc.), about the location of the data itself. To the chunkserver the client sends the actual file data - larger packets. So if your client has lower latency when it talks to the master (and this will be true if they are on the same machine), the overall performance of your single threaded i/o will be better. But again - this is not what MooseFS is for. MooseFS is for many threads, many parallel operations. Regards, Agata W dniu 25.10.2021 o 20:46, Jacob Dietz via moosefs-users pisze: > > Short update. > > Further testing with two additional clints makes my latency theory > unlikely. > > Third client (same hw as master) shows pretty much the same > performance pattern as the master. > > Fourth client (same hw as chunk server0) shows pretty much the same > performance pattern as the chunk server. > > * Read remains some global bottleneck. > * Write seems to be hardware related. > > Best > > Jacob > > *From: *Jacob Dietz <jac...@ci...> > *Date: *Monday, 25. October 2021 at 20:09 > *To: *Jacob Dietz via moosefs-users <moo...@li...> > *Subject: *Understanding moose's performance caps > > Hi everyone, > > I’m testing a moosefs setup and am trying to understand the > performance values I get and the possible bottlenecks. Tried out quite > some things but am a little stuck. > > Status quo for test setup is: > > Mfs master server > > -10G Interface > > -Mfsmaster config is default > > Mfs chunk server > > -100G Interface > > -8x Sata SSD Raid > > -Mfschunkserver config default > > To rule out any client or additional network issues I’m testing on the > chunk server. > > Testing with fio gives me a stable 500MB/s read and 700MB/s write. > > (“sudo fio --filename=/mnt/mfs_mount/123 --direct=1 --rw=read --bs=64k > --ioengine=libaio --iodepth=64 --runtime=30 --numjobs=1 --time_based > --group_reporting --name=throughput-test-job --eta-newline=1 --size=50m”) > > Same test on the xfs gives me about 8GB/s read and 5GB/s write. > > Utilization of the ssd array is zero during testing, so everything > seems to be handled in cache as fio probably deletes everything instantly. > > Testings: > > Seeing the xfs performance reserve and the idling array we tried to > get rid of the cache with “mfsmount /mnt/mfs_mount/ -H mfsmaster -o > mfscachemode=DIRECT”, which gave us the same results. > > Trying to increase the cache instead with -o mfsreadaheadsize=2048 -o > mfsreadaheadleng=2048576 also gave no significant difference. > > Upgrading the nice level of mfsmount to 0 or even 2 also didn’t change > performance. > > Trying to increase workers on chunkserver config didn’t change performance > > WORKERS_MAX = 500 > > WORKERS_MAX_IDLE = 80 > > Also tried reducing CHUNKS_LOOP_MIN_TIME = 150 on mfsmaster config but > still no change. > > Throughout the tests I couldn’t see any cpu cores capping. > > Also Tried to run via a second network connection (same 100GB/10GB) > without jumbo frames to rule out any issues on that side. > > Doing the mfsmount on the master server gave me pretty accurately the > same read performance. Write was strangely doubled to 1,5GB/s, which > is interesting as it only has a 10G interface. > > Guesses: > > I’m pretty new to moosefs and still trying to wrap my > head around it but to me, this seems like some cap I’m running against > as it’s so steady and reproduceable. > > Shouldn’t be the cache, as ram speed cap wouldn’t make sense. > > Shouldn’t be the ssd array. > > Shouldn’t be cpu as thereads are far from capping. > > The increased write on the master server indicates that it could be > the latency between the two servers. Read is similar on both machines > as they need to communicate either way. Write is increased on master > beyond his nic capabilities, possibly because he’s only committing the > writes to himself as fio deletes the data before it is even sent > outside. Array idling during all tests is backing this theory. > > That said, ping is between 0.3 and 0.2 ms. > > Sorry for the long post I hope it’s still readable. > > Would be great if anyone could point me the way to understand the > bottleneck(s) I’m facing and how to overcome it. Could latency be the > right path? > > Thanks! > > Best > > Jacob > > > > _________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users -- -- Agata Kruszona-Zawadzka MooseFS Team |
|
From: Jacob D. <jac...@ci...> - 2021-10-25 19:21:03
|
Short update.
Further testing with two additional clints makes my latency theory unlikely.
Third client (same hw as master) shows pretty much the same performance pattern as the master.
Fourth client (same hw as chunk server0) shows pretty much the same performance pattern as the chunk server.
* Read remains some global bottleneck.
* Write seems to be hardware related.
Best
Jacob
From: Jacob Dietz <jac...@ci...>
Date: Monday, 25. October 2021 at 20:09
To: Jacob Dietz via moosefs-users <moo...@li...>
Subject: Understanding moose's performance caps
Hi everyone,
I’m testing a moosefs setup and am trying to understand the performance values I get and the possible bottlenecks. Tried out quite some things but am a little stuck.
Status quo for test setup is:
Mfs master server
- 10G Interface
- Mfsmaster config is default
Mfs chunk server
- 100G Interface
- 8x Sata SSD Raid
- Mfschunkserver config default
To rule out any client or additional network issues I’m testing on the chunk server.
Testing with fio gives me a stable 500MB/s read and 700MB/s write.
(“sudo fio --filename=/mnt/mfs_mount/123 --direct=1 --rw=read --bs=64k --ioengine=libaio --iodepth=64 --runtime=30 --numjobs=1 --time_based --group_reporting --name=throughput-test-job --eta-newline=1 --size=50m”)
Same test on the xfs gives me about 8GB/s read and 5GB/s write.
Utilization of the ssd array is zero during testing, so everything seems to be handled in cache as fio probably deletes everything instantly.
Testings:
Seeing the xfs performance reserve and the idling array we tried to get rid of the cache with “mfsmount /mnt/mfs_mount/ -H mfsmaster -o mfscachemode=DIRECT”, which gave us the same results.
Trying to increase the cache instead with -o mfsreadaheadsize=2048 -o mfsreadaheadleng=2048576 also gave no significant difference.
Upgrading the nice level of mfsmount to 0 or even 2 also didn’t change performance.
Trying to increase workers on chunkserver config didn’t change performance
WORKERS_MAX = 500
WORKERS_MAX_IDLE = 80
Also tried reducing CHUNKS_LOOP_MIN_TIME = 150 on mfsmaster config but still no change.
Throughout the tests I couldn’t see any cpu cores capping.
Also Tried to run via a second network connection (same 100GB/10GB) without jumbo frames to rule out any issues on that side.
Doing the mfsmount on the master server gave me pretty accurately the same read performance. Write was strangely doubled to 1,5GB/s, which is interesting as it only has a 10G interface.
Guesses:
I’m pretty new to moosefs and still trying to wrap my head around it but to me, this seems like some cap I’m running against as it’s so steady and reproduceable.
Shouldn’t be the cache, as ram speed cap wouldn’t make sense.
Shouldn’t be the ssd array.
Shouldn’t be cpu as thereads are far from capping.
The increased write on the master server indicates that it could be the latency between the two servers. Read is similar on both machines as they need to communicate either way. Write is increased on master beyond his nic capabilities, possibly because he’s only committing the writes to himself as fio deletes the data before it is even sent outside. Array idling during all tests is backing this theory.
That said, ping is between 0.3 and 0.2 ms.
Sorry for the long post I hope it’s still readable.
Would be great if anyone could point me the way to understand the bottleneck(s) I’m facing and how to overcome it. Could latency be the right path?
Thanks!
Best
Jacob
|
|
From: Jacob D. <jac...@ci...> - 2021-10-25 18:44:00
|
Hi everyone,
I’m testing a moosefs setup and am trying to understand the performance values I get and the possible bottlenecks. Tried out quite some things but am a little stuck.
Status quo for test setup is:
Mfs master server
- 10G Interface
- Mfsmaster config is default
Mfs chunk server
- 100G Interface
- 8x Sata SSD Raid
- Mfschunkserver config default
To rule out any client or additional network issues I’m testing on the chunk server.
Testing with fio gives me a stable 500MB/s read and 700MB/s write.
(“sudo fio --filename=/mnt/mfs_mount/123 --direct=1 --rw=read --bs=64k --ioengine=libaio --iodepth=64 --runtime=30 --numjobs=1 --time_based --group_reporting --name=throughput-test-job --eta-newline=1 --size=50m”)
Same test on the xfs gives me about 8GB/s read and 5GB/s write.
Utilization of the ssd array is zero during testing, so everything seems to be handled in cache as fio probably deletes everything instantly.
Testings:
Seeing the xfs performance reserve and the idling array we tried to get rid of the cache with “mfsmount /mnt/mfs_mount/ -H mfsmaster -o mfscachemode=DIRECT”, which gave us the same results.
Trying to increase the cache instead with -o mfsreadaheadsize=2048 -o mfsreadaheadleng=2048576 also gave no significant difference.
Upgrading the nice level of mfsmount to 0 or even 2 also didn’t change performance.
Trying to increase workers on chunkserver config didn’t change performance
WORKERS_MAX = 500
WORKERS_MAX_IDLE = 80
Also tried reducing CHUNKS_LOOP_MIN_TIME = 150 on mfsmaster config but still no change.
Throughout the tests I couldn’t see any cpu cores capping.
Also Tried to run via a second network connection (same 100GB/10GB) without jumbo frames to rule out any issues on that side.
Doing the mfsmount on the master server gave me pretty accurately the same read performance. Write was strangely doubled to 1,5GB/s, which is interesting as it only has a 10G interface.
Guesses:
I’m pretty new to moosefs and still trying to wrap my head around it but to me, this seems like some cap I’m running against as it’s so steady and reproduceable.
Shouldn’t be the cache, as ram speed cap wouldn’t make sense.
Shouldn’t be the ssd array.
Shouldn’t be cpu as thereads are far from capping.
The increased write on the master server indicates that it could be the latency between the two servers. Read is similar on both machines as they need to communicate either way. Write is increased on master beyond his nic capabilities, possibly because he’s only committing the writes to himself as fio deletes the data before it is even sent outside. Array idling during all tests is backing this theory.
That said, ping is between 0.3 and 0.2 ms.
Sorry for the long post I hope it’s still readable.
Would be great if anyone could point me the way to understand the bottleneck(s) I’m facing and how to overcome it. Could latency be the right path?
Thanks!
Best
Jacob
|
|
From: Jacob D. <jac...@ci...> - 2021-09-29 11:46:56
|
Hi Agata, thanks a lot for the clarification and nice insight! Seems to be something that happens on boot only. Restarting the service afterwards gives me no such message. Best Jacob From: Agata Kruszona-Zawadzka <ch...@mo...> Date: Wednesday, 29. September 2021 at 11:36 To: Jacob Dietz <jac...@ci...>, moo...@li... <moo...@li...> Subject: Re: [MooseFS-Users] main master server module: packet too long Hi, I decoded the bytes sent to your master module and they look just like this: GET /met So, this is definitely not MooseFS protocol. It looks like good old HTTP. And the connections are coming from localhost (127.0.0.1). Do you have a local process that scans open ports? Anyway, master (and other MooseFS modules) will just ignore any communication that is not in MooseFS protocol and close the connection. So it will not stop the system from working, however, you will be notified in logs each time this happens. Regards, Agata W dniu 29.09.2021 o 10:40, Jacob Dietz via moosefs-users pisze: > Hi Agata, > > Thanks for your response! > > I’m trying to run > > version: 3.0.116-1 ; build: 1309 > > os is ubuntu server 20.04. > > Just a side note, though I don’t think it’s related. I had some trouble > getting the mfsmaster service to run on an alternative data path. Ended > up modifying the service file. > > [Service] > > Type=forking > > ExecStart=/usr/sbin/mfsmaster start > > ExecStop=/usr/sbin/mfsmaster stop > > ExecReload=/usr/sbin/mfsmaster reload > > #PIDFile=/var/lib/mfs/.mfsmaster.lock > > TimeoutStopSec=1800 > > TimeoutStartSec=1800 > > Restart=no > > #modification > > User=mfs > > Group=mfs > > PIDFile=/mnt/mfs_data/metalog/.mfsmaster.lock > > Best > > Jacob > > *From: *Agata Kruszona-Zawadzka <ch...@mo...> > *Date: *Wednesday, 29. September 2021 at 09:46 > *To: *Jacob Dietz <jac...@ci...>, > moo...@li... <moo...@li...> > *Subject: *Re: [MooseFS-Users] main master server module: packet too long > > Hi, > > Which MooseFS version are you using? > > Regards, > Agata > > W dniu 28.09.2021 o 22:08, Jacob Dietz via moosefs-users pisze: > > Hi everyone, > > > > seeing some too long packets on my master server. > > > > Just starting to set it up, so no chunk servers or clients connected yet. > > > > […] > > > > Sep 28 21:50:38 mfsmaster01 mfsmaster[840]: ML(127.0.0.1) packet too > > long (795698548/1500000) ; command:1195725856 > > > > Sep 28 21:50:38 mfsmaster01 mfsmaster[840]: CS(127.0.0.1) packet too > > long (795698548/500000000) ; command:1195725856 > > > > Sep 28 21:50:38 mfsmaster01 mfsmaster[840]: chunkserver disconnected - > > ip: 127.0.0.1 / port: 0, usedspace: 0 (0.00 GiB), totalspace: 0 (0.00 > GiB) > > > > Sep 28 21:50:38 mfsmaster01 mfsmaster[840]: main master server module: > > packet too long (795698548/50000000) ; command:1195725856 > > > > […] > > > > There is actually no chunk-server running on the local system. > > > > Thanks! > > > > Best > > > > Jacob > > > > > > > > _________________________________________ > > moosefs-users mailing list > > moo...@li... > > https://lists.sourceforge.net/lists/listinfo/moosefs-users > > > > > -- > -- > Agata Kruszona-Zawadzka > MooseFS Team > > > > _________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users > -- -- Agata Kruszona-Zawadzka MooseFS Team |
|
From: Agata Kruszona-Z. <ch...@mo...> - 2021-09-29 09:36:43
|
Hi, I decoded the bytes sent to your master module and they look just like this: GET /met So, this is definitely not MooseFS protocol. It looks like good old HTTP. And the connections are coming from localhost (127.0.0.1). Do you have a local process that scans open ports? Anyway, master (and other MooseFS modules) will just ignore any communication that is not in MooseFS protocol and close the connection. So it will not stop the system from working, however, you will be notified in logs each time this happens. Regards, Agata W dniu 29.09.2021 o 10:40, Jacob Dietz via moosefs-users pisze: > Hi Agata, > > Thanks for your response! > > I’m trying to run > > version: 3.0.116-1 ; build: 1309 > > os is ubuntu server 20.04. > > Just a side note, though I don’t think it’s related. I had some trouble > getting the mfsmaster service to run on an alternative data path. Ended > up modifying the service file. > > [Service] > > Type=forking > > ExecStart=/usr/sbin/mfsmaster start > > ExecStop=/usr/sbin/mfsmaster stop > > ExecReload=/usr/sbin/mfsmaster reload > > #PIDFile=/var/lib/mfs/.mfsmaster.lock > > TimeoutStopSec=1800 > > TimeoutStartSec=1800 > > Restart=no > > #modification > > User=mfs > > Group=mfs > > PIDFile=/mnt/mfs_data/metalog/.mfsmaster.lock > > Best > > Jacob > > *From: *Agata Kruszona-Zawadzka <ch...@mo...> > *Date: *Wednesday, 29. September 2021 at 09:46 > *To: *Jacob Dietz <jac...@ci...>, > moo...@li... <moo...@li...> > *Subject: *Re: [MooseFS-Users] main master server module: packet too long > > Hi, > > Which MooseFS version are you using? > > Regards, > Agata > > W dniu 28.09.2021 o 22:08, Jacob Dietz via moosefs-users pisze: > > Hi everyone, > > > > seeing some too long packets on my master server. > > > > Just starting to set it up, so no chunk servers or clients connected yet. > > > > […] > > > > Sep 28 21:50:38 mfsmaster01 mfsmaster[840]: ML(127.0.0.1) packet too > > long (795698548/1500000) ; command:1195725856 > > > > Sep 28 21:50:38 mfsmaster01 mfsmaster[840]: CS(127.0.0.1) packet too > > long (795698548/500000000) ; command:1195725856 > > > > Sep 28 21:50:38 mfsmaster01 mfsmaster[840]: chunkserver disconnected - > > ip: 127.0.0.1 / port: 0, usedspace: 0 (0.00 GiB), totalspace: 0 (0.00 > GiB) > > > > Sep 28 21:50:38 mfsmaster01 mfsmaster[840]: main master server module: > > packet too long (795698548/50000000) ; command:1195725856 > > > > […] > > > > There is actually no chunk-server running on the local system. > > > > Thanks! > > > > Best > > > > Jacob > > > > > > > > _________________________________________ > > moosefs-users mailing list > > moo...@li... > > https://lists.sourceforge.net/lists/listinfo/moosefs-users > > > > > -- > -- > Agata Kruszona-Zawadzka > MooseFS Team > > > > _________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users > -- -- Agata Kruszona-Zawadzka MooseFS Team |
|
From: Jacob D. <jac...@ci...> - 2021-09-29 08:56:35
|
Hi Agata, Thanks for your response! I’m trying to run version: 3.0.116-1 ; build: 1309 os is ubuntu server 20.04. Just a side note, though I don’t think it’s related. I had some trouble getting the mfsmaster service to run on an alternative data path. Ended up modifying the service file. [Service] Type=forking ExecStart=/usr/sbin/mfsmaster start ExecStop=/usr/sbin/mfsmaster stop ExecReload=/usr/sbin/mfsmaster reload #PIDFile=/var/lib/mfs/.mfsmaster.lock TimeoutStopSec=1800 TimeoutStartSec=1800 Restart=no #modification User=mfs Group=mfs PIDFile=/mnt/mfs_data/metalog/.mfsmaster.lock Best Jacob From: Agata Kruszona-Zawadzka <ch...@mo...> Date: Wednesday, 29. September 2021 at 09:46 To: Jacob Dietz <jac...@ci...>, moo...@li... <moo...@li...> Subject: Re: [MooseFS-Users] main master server module: packet too long Hi, Which MooseFS version are you using? Regards, Agata W dniu 28.09.2021 o 22:08, Jacob Dietz via moosefs-users pisze: > Hi everyone, > > seeing some too long packets on my master server. > > Just starting to set it up, so no chunk servers or clients connected yet. > > […] > > Sep 28 21:50:38 mfsmaster01 mfsmaster[840]: ML(127.0.0.1) packet too > long (795698548/1500000) ; command:1195725856 > > Sep 28 21:50:38 mfsmaster01 mfsmaster[840]: CS(127.0.0.1) packet too > long (795698548/500000000) ; command:1195725856 > > Sep 28 21:50:38 mfsmaster01 mfsmaster[840]: chunkserver disconnected - > ip: 127.0.0.1 / port: 0, usedspace: 0 (0.00 GiB), totalspace: 0 (0.00 GiB) > > Sep 28 21:50:38 mfsmaster01 mfsmaster[840]: main master server module: > packet too long (795698548/50000000) ; command:1195725856 > > […] > > There is actually no chunk-server running on the local system. > > Thanks! > > Best > > Jacob > > > > _________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users > -- -- Agata Kruszona-Zawadzka MooseFS Team |
|
From: Agata Kruszona-Z. <ch...@mo...> - 2021-09-29 08:13:13
|
Hi, Which MooseFS version are you using? Regards, Agata W dniu 28.09.2021 o 22:08, Jacob Dietz via moosefs-users pisze: > Hi everyone, > > seeing some too long packets on my master server. > > Just starting to set it up, so no chunk servers or clients connected yet. > > […] > > Sep 28 21:50:38 mfsmaster01 mfsmaster[840]: ML(127.0.0.1) packet too > long (795698548/1500000) ; command:1195725856 > > Sep 28 21:50:38 mfsmaster01 mfsmaster[840]: CS(127.0.0.1) packet too > long (795698548/500000000) ; command:1195725856 > > Sep 28 21:50:38 mfsmaster01 mfsmaster[840]: chunkserver disconnected - > ip: 127.0.0.1 / port: 0, usedspace: 0 (0.00 GiB), totalspace: 0 (0.00 GiB) > > Sep 28 21:50:38 mfsmaster01 mfsmaster[840]: main master server module: > packet too long (795698548/50000000) ; command:1195725856 > > […] > > There is actually no chunk-server running on the local system. > > Thanks! > > Best > > Jacob > > > > _________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users > -- -- Agata Kruszona-Zawadzka MooseFS Team |
|
From: Jacob D. <jac...@ci...> - 2021-09-29 00:44:05
|
Hi everyone, seeing some too long packets on my master server. Just starting to set it up, so no chunk servers or clients connected yet. […] Sep 28 21:50:38 mfsmaster01 mfsmaster[840]: ML(127.0.0.1) packet too long (795698548/1500000) ; command:1195725856 Sep 28 21:50:38 mfsmaster01 mfsmaster[840]: CS(127.0.0.1) packet too long (795698548/500000000) ; command:1195725856 Sep 28 21:50:38 mfsmaster01 mfsmaster[840]: chunkserver disconnected - ip: 127.0.0.1 / port: 0, usedspace: 0 (0.00 GiB), totalspace: 0 (0.00 GiB) Sep 28 21:50:38 mfsmaster01 mfsmaster[840]: main master server module: packet too long (795698548/50000000) ; command:1195725856 […] There is actually no chunk-server running on the local system. Thanks! Best Jacob |
|
From: Aleksander W. <ale...@mo...> - 2021-07-20 10:18:51
|
Hi, MooseFS community editions packages are available on Alpine linux. Please check this link: https://pkgs.alpinelinux.org/package/edge/testing/x86/moosefs Basically, you have to use a community testing repository for alpine linux. So you can install moosefs like this: apk add --no-cache -X http://dl-cdn.alpinelinux.org/alpine/edge/testing moosefs Best regards, Alex Aleksander Wieliczko System Engineer MooseFS Development & Support Team | moosefs.pro wt., 20 lip 2021 o 02:44 竹下候、小姜 via moosefs-users < moo...@li...> napisał(a): > Hi！ > I jave try to install moosefs-client on my docker container about apline > linux 3.12 version，but I have not find how to install in website。 > I want someone can help me or tell me how to do it.thank you very much! > > allenjol > _________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users > |
|
From: <all...@qq...> - 2021-07-14 05:52:36
|
Hi！ I jave try to install moosefs-client on my docker container about apline linux 3.12 version，but I have not find how to install in website。 I want someone can help me or tell me how to do it.thank you very much! allenjol |
|
From: Marin B. <li...@ol...> - 2021-07-09 16:27:38
|
Hi Agatha, Thank you for taking the time to make this statement. I find your commitment to support the GPL version reassuring. I hope other people will share the same view. My words were very sincere. IMHO, in the tiny world of distributed file systems, MooseFS is just the most stable and easy to use open source solution lying around — and I've been testing many of them, if not most. It is very clear at what it can and cannot do. As long as EC, HA or block storage are not hard requirements, I wouldn't go another way. That why I would have been very upset had the free version been discontinued! Thank you again for the clarification, -- Marin BERNARD li...@ol... https://keybase.io/marinbernard PGP: FD5D 9CF2 413F 68C0 1FD5 ED76 122B 443F 2493 48A8 ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ Le vendredi 9 juillet 2021 à 13:29, Agata Kruszona-Zawadzka <ch...@mo...> a écrit : > W dniu 04.07.2021 o 11:25, Marin BERNARD pisze: > > > Hi, > > > > I've been using MooseFS for personal use for 6-7 years now, and it's been rock-solid: I never had to complain about it. However, since a few years the public activity of the project on GitHub and the ML has dramatically decreased. So much that I'm beginning to wonder whether the free version is still supported, and if it is, how long it will remain so. > > > > Could some official from the project make a quick statement about the status of the GPL version ? It would help people like me to decide whether the free version of MooseFS is still a viable solution for both new and old deployments. > > > > Could someone also confirm that MooseFS 4.0 will never be released under the GPL ? It was announced more than 3 years ago and never released publicly since; I suppose this means it will never be. > > > > Many thanks, > > > > -- > > > > Marin BERNARD > > > > li...@ol... > > > > https://keybase.io/marinbernard > > > > PGP: FD5D 9CF2 413F 68C0 1FD5 ED76 122B 443F 2493 48A8 > > Hi, > > Thank you for your kind words about MooseFS's stability - we do our best. > > The open source version is still supported and we don't plan to change > > that. We will be releasing version 3.0.116 soon. > > We also still plan to release version 4 on an open source licence, but > > we cannot commit to any timeframe. Once it is released, we will announce > > end of support time for version 3, but we will give our users ample time > > to migrate. > > Regards, > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > Agata Kruszona-Zawadzka > > MooseFS Team |
|
From: Agata Kruszona-Z. <ch...@mo...> - 2021-07-09 13:11:51
|
W dniu 04.07.2021 o 11:25, Marin BERNARD pisze: > Hi, > > I've been using MooseFS for personal use for 6-7 years now, and it's been rock-solid: I never had to complain about it. However, since a few years the public activity of the project on GitHub and the ML has dramatically decreased. So much that I'm beginning to wonder whether the free version is still supported, and if it is, how long it will remain so. > > Could some official from the project make a quick statement about the status of the GPL version ? It would help people like me to decide whether the free version of MooseFS is still a viable solution for both new and old deployments. > > Could someone also confirm that MooseFS 4.0 will never be released under the GPL ? It was announced more than 3 years ago and never released publicly since; I suppose this means it will never be. > > Many thanks, > > -- > Marin BERNARD > li...@ol... > https://keybase.io/marinbernard > PGP: FD5D 9CF2 413F 68C0 1FD5 ED76 122B 443F 2493 48A8 Hi, Thank you for your kind words about MooseFS's stability - we do our best. The open source version is still supported and we don't plan to change that. We will be releasing version 3.0.116 soon. We also still plan to release version 4 on an open source licence, but we cannot commit to any timeframe. Once it is released, we will announce end of support time for version 3, but we will give our users ample time to migrate. Regards, -- Agata Kruszona-Zawadzka MooseFS Team |
|
From: Marin B. <li...@ol...> - 2021-07-04 09:43:47
|
Hi, I've been using MooseFS for personal use for 6-7 years now, and it's been rock-solid: I never had to complain about it. However, since a few years the public activity of the project on GitHub and the ML has dramatically decreased. So much that I'm beginning to wonder whether the free version is still supported, and if it is, how long it will remain so. Could some official from the project make a quick statement about the status of the GPL version ? It would help people like me to decide whether the free version of MooseFS is still a viable solution for both new and old deployments. Could someone also confirm that MooseFS 4.0 will never be released under the GPL ? It was announced more than 3 years ago and never released publicly since; I suppose this means it will never be. Many thanks, -- Marin BERNARD li...@ol... https://keybase.io/marinbernard PGP: FD5D 9CF2 413F 68C0 1FD5 ED76 122B 443F 2493 48A8 |
|
From: Marin B. <li...@ol...> - 2021-07-04 09:39:17
|
Hi, I've been using MooseFS for personal use for 6-7 years now, and it's been rock-solid: I never had to complain about it. However, since a few years the public activity of the project on GitHub and the ML has dramatically decreased. So much that I'm beginning to wonder whether the free version is still supported, and if it is, how long it will remain so. Could some official from the project make a quick statement about the status of the GPL version ? It would help people like me to decide whether the free version of MooseFS is still a viable solution for both new and old deployments. Could someone also confirm that MooseFS 4.0 will never be released under the GPL ? It was announced more than 3 years ago and never released publicly since; I suppose this means it will never be. Many thanks, -- Marin BERNARD li...@ol... |
|
From: Agata Kruszona-Z. <ch...@mo...> - 2021-02-24 11:57:35
|
W dniu 24.02.2021 o 00:15, William Kenworthy pisze: > Hi, Hi, I've answered your questions below, but please note, that we moved with support and troubleshooting to github and if you want to look up answers to similar problems, you should go there. Also, I would like to encourage you to ask your own questions on github in the future. https://github.com/moosefs/moosefs - main MooseFS repository. https://github.com/moosefs/moosefs/discussions - new feature on Github, discussions, this is where people ask questions. > I have an approximately year old moosefs system with ~8GB data > (general data storage, email, lxc containers etc.) that was using goal > based allocation - I have added an ssd to and decided to use storage > classes to move one chunk of each of the lxc containers to the faster > SSD (seems to work well) while converting the the whole storage to > sclass based goal allocation. > > To convert I did a mfssetsclass -r at the top of the mfs tree - after > removing older snapshots (I have read an email on how they handle the > conversion) I am left with ~580000 still with a goal of 3 that is > reported by the cli. > > 1. how do I find which actual chunks are still at goal 3 You can't find that out directly, but you can make an educated guess and say they all belong to snapshotted files. So if/when you remove snapshots, all remaining chunks will move to appropriate storage classes automatically. > 2. how can I map a chunk to a file There is not a simple way to find out and there is no tool for that. This is because of the system architecture - files know, which chunks belogn to them, but chunks do not know, to which file(s) they belong. It can be done, but it requires dumping all your metadata CHNK and EDGE sections to text format and manually searching for the chunk in question, then from EDGEs you can rebuild the whole path of the file(s). If you are really interested in that, I can write you a short "how to". > 3. I have used mfsmakesnapshot a bit indiscriminately and have likely > missed some snapshots - is there a way to identify chunks/files/dirs > belonging to a snapshot? Yes, there is. You go to the root of your MooseFS instance and use this command: mfsgeteattr * If you see any entry, that has "snapshot" written after its name and ":" sign, it's a snanpshot. To look deeper, you then use this command: mfsgeteattr -r * In the output, you look up all directories that have not directory nodes with attribute snapshot : XXX directories with attribute snapshot : YYY where either XXX or YYY is something else than 0. This directory contains a snapshot somewhere inside it, so you cd into it and repeat the procedure. > The only way I can think of to remove unknown snapshotted data is to > copy the whole tree and delete the old one - which is a bit drastic! > > Is there another mechanism that prevents some chunks from being > converted to a storage class? - or am I reading this wrong? No, basically that's the only thing, that kind of "reverts to goal" - being part of a snapshot. All other chunks will be converted to a storage class (they still won't necessarily move to servers matching their storage class if something like lack of space or busy servers prevents this, but they will at least be trying to). Regards, Agata -- Agata Kruszona-Zawadzka MooseFS Team |
|
From: William K. <bi...@ii...> - 2021-02-24 11:42:00
|
Thanks, this gives a way forward. I'll use github in future - I wondered why the list wasn't very active! BillK On 24/2/21 7:27 pm, Agata Kruszona-Zawadzka wrote: > W dniu 24.02.2021 o 00:15, William Kenworthy pisze: >> Hi, > > Hi, > > I've answered your questions below, but please note, that we moved > with support and troubleshooting to github and if you want to look up > answers to similar problems, you should go there. Also, I would like > to encourage you to ask your own questions on github in the future. > > https://github.com/moosefs/moosefs - main MooseFS repository. > > https://github.com/moosefs/moosefs/discussions - new feature on > Github, discussions, this is where people ask questions. > >> I have an approximately year old moosefs system with ~8GB data >> (general data storage, email, lxc containers etc.) that was using >> goal based allocation - I have added an ssd to and decided to use >> storage classes to move one chunk of each of the lxc containers to >> the faster SSD (seems to work well) while converting the the whole >> storage to sclass based goal allocation. >> >> To convert I did a mfssetsclass -r at the top of the mfs tree - after >> removing older snapshots (I have read an email on how they handle the >> conversion) I am left with ~580000 still with a goal of 3 that is >> reported by the cli. >> >> 1. how do I find which actual chunks are still at goal 3 > > You can't find that out directly, but you can make an educated guess > and say they all belong to snapshotted files. So if/when you remove > snapshots, all remaining chunks will move to appropriate storage > classes automatically. > >> 2. how can I map a chunk to a file > > There is not a simple way to find out and there is no tool for that. > This is because of the system architecture - files know, which chunks > belogn to them, but chunks do not know, to which file(s) they belong. > It can be done, but it requires dumping all your metadata CHNK and > EDGE sections to text format and manually searching for the chunk in > question, then from EDGEs you can rebuild the whole path of the > file(s). If you are really interested in that, I can write you a short > "how to". > >> 3. I have used mfsmakesnapshot a bit indiscriminately and have likely >> missed some snapshots - is there a way to identify chunks/files/dirs >> belonging to a snapshot? > > Yes, there is. You go to the root of your MooseFS instance and use > this command: > > mfsgeteattr * > > If you see any entry, that has "snapshot" written after its name and > ":" sign, it's a snanpshot. > > To look deeper, you then use this command: > > mfsgeteattr -r * > > In the output, you look up all directories that have > > not directory nodes with attribute snapshot : XXX > directories with attribute snapshot : YYY > > where either XXX or YYY is something else than 0. This directory > contains a snapshot somewhere inside it, so you cd into it and repeat > the procedure. > >> The only way I can think of to remove unknown snapshotted data is to >> copy the whole tree and delete the old one - which is a bit drastic! >> >> Is there another mechanism that prevents some chunks from being >> converted to a storage class? - or am I reading this wrong? > > No, basically that's the only thing, that kind of "reverts to goal" - > being part of a snapshot. All other chunks will be converted to a > storage class (they still won't necessarily move to servers matching > their storage class if something like lack of space or busy servers > prevents this, but they will at least be trying to). > > Regards, > Agata > > -- > Agata Kruszona-Zawadzka > MooseFS Team |
|
From: William K. <bi...@ii...> - 2021-02-24 04:43:39
|
Hi, I have an approximately year old moosefs system with ~8GB data (general data storage, email, lxc containers etc.) that was using goal based allocation - I have added an ssd to and decided to use storage classes to move one chunk of each of the lxc containers to the faster SSD (seems to work well) while converting the the whole storage to sclass based goal allocation. To convert I did a mfssetsclass -r at the top of the mfs tree - after removing older snapshots (I have read an email on how they handle the conversion) I am left with ~580000 still with a goal of 3 that is reported by the cli. 1. how do I find which actual chunks are still at goal 3 2. how can I map a chunk to a file 3. I have used mfsmakesnapshot a bit indiscriminately and have likely missed some snapshots - is there a way to identify chunks/files/dirs belonging to a snapshot? The only way I can think of to remove unknown snapshotted data is to copy the whole tree and delete the old one - which is a bit drastic! Is there another mechanism that prevents some chunks from being converted to a storage class? - or am I reading this wrong? BillK +------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ ----------------------------------------------------------------------------------+ | Storage Classes | +----+------+------------+------+------------------+------------------------+----------------------+-------------------------------------------------------------+------------------------------------------------- ------------+---------------------------------------------------------------------+ | | | | | # of inodes | # of standard chunks | # of archived chunks | create | keep | archive | | id | name | admin only | mode +---------+--------+-------+---------+------+-------+-------+------+------------------+------+-----------------------------------+------------------+------+----------------------- ------------+------------------+------+-----------------------------------+-------+ | | | | | files | dirs | under | exact | over | under | exact | over | can be fulfilled | goal | labels | can be fulfilled | goal | labels | can be fulfilled | goal | labels | delay | +----+------+------------+------+---------+--------+-------+---------+------+-------+-------+------+------------------+------+-----------------------------------+------------------+------+----------------------- ------------+------------------+------+-----------------------------------+-------+ | 1 | 1 | NO | STD | 0 | 0 | - | 0 | - | - | - | - | YES | 1 | * | YES | 1 | * | YES | 1 | * | - | | 2 | 2 | NO | STD | 0 | 0 | - | 0 | - | - | - | - | YES | 2 | * , * | YES | 2 | * , * | YES | 2 | * , * | - | *| 3 | 3 | NO | STD | 0 | 0 | - | 558046 | - | - | - | - | YES | 3 | * , * , * | YES | 3 | * , * , * * * | YES | 3 | * , * , * | - |* | 4 | 4 | NO | STD | 0 | 0 | - | 0 | - | - | - | - | YES | 4 | * , * , * , * | YES | 4 | * , * , * , * | YES | 4 | * , * , * , * | - | | 5 | 5 | NO | STD | 0 | 0 | - | 0 | - | - | - | - | YES | 5 | * , * , * , * , * | YES | 5 | * , * , * , * , * | YES | 5 | * , * , * , * , * | - | | 6 | 6 | NO | STD | 0 | 0 | - | 0 | - | - | - | - | YES | 6 | * , * , * , * , * , * | YES | 6 | * , * , * , * , * , * | YES | 6 | * , * , * , * , * , * | - | | 7 | 7 | NO | STD | 0 | 0 | - | 0 | - | - | - | - | NO | 7 | * , * , * , * , * , * , * | NO | 7 | * , * , * , * , * , * , * | NO | 7 | * , * , * , * , * , * , * | - | | 8 | 8 | NO | STD | 0 | 0 | - | 0 | - | - | - | - | NO | 8 | * , * , * , * , * , * , * , * | NO | 8 | * , * , * , * , * , * , * , * | NO | 8 | * , * , * , * , * , * , * , * | - | | 9 | 9 | NO | STD | 0 | 0 | - | 0 | - | - | - | - | NO | 9 | * , * , * , * , * , * , * , * , * | NO | 9 | * , * , * , * , * , * , * , * , * | NO | 9 | * , * , * , * , * , * , * , * , * | - | | 12 | 4SDD | NO | STD | 0 | 0 | - | 0 | - | - | - | - | YES | 1 | S | YES | 4 | S , H , H , H | YES | 4 | S , * , * , * | - | | 13 | 3SDD | NO | STD | 1543418 | 78533 | - | 622622 | - | - | - | - | YES | 1 | S | YES | 3 | S , H , H | YES | 3 | S , * , * | - | | 14 | 2SDD | NO | STD | 382182 | 12723 | - | 243527 | - | - | - | - | YES | 1 | S | YES | 2 | S , H | YES | 2 | S , * | - | | 15 | 4HDD | NO | STD | 114113 | 2657 | - | 121934 | - | - | - | - | YES | 1 | H | YES | 4 | H , H , H , H | YES | 4 | H , * , * , * | - | | 16 | 3HDD | NO | STD | 7868308 | 192272 | - | 1223147 | - | - | - | - | YES | 1 | H | YES | 3 | H , H , H | YES | 3 | H , * , * | - | | 17 | 2HDD | NO | STD | 240618 | 15832 | - | 232747 | - | - | - | - | YES | 1 | H | YES | 2 | H , H | YES | 2 | H , * | - | n2 /var/lib/lxc # mfsgetgoal -r /mnt/mfs /mnt/mfs: files with storage class '3SDD' : 1545766 files with storage class '2SDD' : 385145 files with storage class '4HDD' : 114113 files with storage class '3HDD' : 7868656 files with storage class '2HDD' : 240618 directories with storage class '3SDD' : 78533 directories with storage class '2SDD' : 12723 directories with storage class '4HDD' : 2657 directories with storage class '3HDD' : 192272 directories with storage class '2HDD' : 15832 n2 /var/lib/lxc # mfsgetsclass -r /mnt/mfs /mnt/mfs: files with storage class '3SDD' : 1545766 files with storage class '2SDD' : 385145 files with storage class '4HDD' : 114113 files with storage class '3HDD' : 7868656 files with storage class '2HDD' : 240618 directories with storage class '3SDD' : 78533 directories with storage class '2SDD' : 12723 directories with storage class '4HDD' : 2657 directories with storage class '3HDD' : 192272 directories with storage class '2HDD' : 15832 n2 /var/lib/lxc # |
|
From: Piotr R. K. <pio...@mo...> - 2020-10-16 00:04:22
|
Hi Markus, I'm glad that updating your cluster to MooseFS 3.0.115 helped you. Many thanks for confirming it! Best regards, Piotr *Piotr Robert Konopelko* | m: +48 601 476 440 | e: pio...@mo... *Business & Technical Support Manager* MooseFS Client Support Team WWW <https://moosefs.com> | GitHub <https://github.com/moosefs/moosefs> | Twitter <https://twitter.com/moosefs> | Facebook <https://www.facebook.com/moosefs> | LinkedIn <https://www.linkedin.com/company/moosefs> On Wed, Oct 14, 2020 at 1:03 PM Markus Köberl <mar...@tu...> wrote: > On Thursday, 10 September 2020 07:51:01 CEST Markus Köberl wrote: > > On Tuesday, 8 September 2020 09:04:50 CEST Agata Kruszona-Zawadzka wrote: > > > W dniu 07.09.2020 o 15:17, Markus Köberl pisze: > > > > On Monday, 7 September 2020 12:19:42 CEST Agata Kruszona-Zawadzka > wrote: > > > >> > > > >> W dniu 04.09.2020 o 15:22, Markus Köberl pisze: > > > >>> Since some time (last few versions of MooseFS) on a few > chunkservers the used space grows above the default > ACCEPTABLE_PERCENTAGE_DIFFERENCE = 1.0 till I restart the affected > chunkserver. > > > >>> On the webinterface i see huge numbers for overgoal (even 4 extra > copies). After the restart of the chunkserver the overgoal goes down but > starts growing again after some time. > > > >> > > > >> We have an issue in MooseFS currently, where on disks with I/O > errors in > > > >> certain circustances some chunks get locked and cannot be deleted > until > > > >> the whole chunk server process is restarted. We introduced a fix for > > > >> that, it's gonna be available in version 3.0.115. The issue does not > > > >> affect disks without I/O errors. > > > > > > > > Thanks good to hear that a fix might be on the way. > > > > > > > > Could it be that instead of "some chunks get locked and cannot be > deleted" that there a no deletes at all on this chunk server, or might that > be a different problem? > > > > > > Yes, that's exactly it. By "some chunks" I meant that not every chunk > is > > > able to trigger the problem, but once it happens so that enough chunks > > > do, then, due to operation limits (specifically deletions limit in > this > > > instance), the system won't attempt to delete any more chunks. > > > > Thank you for confirming that it is the same problem and the good > explanation. > > I can confirm that all our problems are resolved with version 3.0.115. > Thank you for all the good work! > > > regards > Markus Köberl > -- > Markus Koeberl > Graz University of Technology > Signal Processing and Speech Communication Laboratory > E-mail: mar...@tu... > > > > > _________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users > |
84 messages has been excluded from this view by a project administrator.
