linux-f2fs-devel

Re: [f2fs-dev] 100% system time hang with git f2fs

[Marc L. <sc...@sc...>]

On Fri, Oct 02, 2015 at 09:51:24AM -0700, Jaegeuk Kim <ja...@ke...> wrote:
> > However, I have a freeze. When I mount the volume, start a du on it, and after
> > a while do:
> 
> How was your scenario?
> Did you delete device before, or just plain mount and umount?

After writing >2TB to it, I let it idle/gc for a while, then unmounted. Then
did a fsck and mount, and let it idle for the night. On the next day, I did a
du for a speed test, relaised I hadn't dropped the caches, interrupted the
du, and then trie drop_caches with the result.

> This should be shrinker is stuck on mutex. I suspect a deadlock.
> Can you do this, if you meet this again?

Will do. Btw, I already mentioned that /proc/*/stack didn't give a useful
backtrace. Since it was apparently running, I thought maybe run cat
/.../stack in a loop, and indeed, after a minute, I got one backtrace out
of it (the process doing the echo):

http://ue.tst.eu/2562aec1c51d4bad3d7b8b83380956a7.txt

Maybe that is already useful? It indeed mentions the shrinker.

(I did this after my mail, so didn't include it).

-- 
                The choice of a       Deliantra, the free code+content MORPG
      -----==-     _GNU_              http://www.deliantra.net
      ----==-- _       generation
      ---==---(_)__  __ ____  __      Marc Lehmann
      --==---/ / _ \/ // /\ \/ /      sc...@sc...
      -=====/_/_//_/\_,_/ /_/\_\

Re: [f2fs-dev] write performance difference 3.18.21/git f2fs

[Jaegeuk K. <ja...@ke...>]

On Sat, Sep 26, 2015 at 07:25:51AM +0200, Marc Lehmann wrote:
> Ok, before I tried the f2fs git I made another short test with the
> original 3.18.21 f2fs, and it was as fast as before. Then I used the
> faulty f2fs module,. which forced a reboot.
> 
> Now I started to redo the 3.18.21 test + git f2fs, with the same parameters
> (specifically, -s90), and while it didn't start out to be as slow as 4.2.1,
> it's similarly slow.
> 
> After 218GiB, I stopped the test, giving me an average of 50MiB/s.
> 
> Here is typical dstat output (again, dsk/sde):
> 
> http://ue.tst.eu/7a40644b3432e2932bdd8c1f6b6fc32d.txt
> 
> So less read behaviour than with 4.2.1, but also very slow writes.
> 
> That means the performance drop moves with f2fs, not the kernel version.
> 
> This is the resulting status:
> 
> http://ue.tst.eu/6d94e9bfad48a433bbc6f7daeaf5eb38.txt
> 
> Just for fun I'll start doing a -s64 run.

Okay, so before finding bad commits, if possible, can you get block traces?
It would be good to see block_rq_complete and block_bio_complete tracepoints.

Thanks,

> 
> -- 
>                 The choice of a       Deliantra, the free code+content MORPG
>       -----==-     _GNU_              http://www.deliantra.net
>       ----==-- _       generation
>       ---==---(_)__  __ ____  __      Marc Lehmann
>       --==---/ / _ \/ // /\ \/ /      sc...@sc...
>       -=====/_/_//_/\_,_/ /_/\_\
> 
> ------------------------------------------------------------------------------
> _______________________________________________
> Linux-f2fs-devel mailing list
> Lin...@li...
> https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

Re: [f2fs-dev] write performance difference 3.18.21/git f2fs

[Marc L. <sc...@sc...>]

On Sat, Sep 26, 2015 at 12:52:53AM -0700, Jaegeuk Kim <ja...@ke...> wrote:
> > Just for fun I'll start doing a -s64 run.

(which had the same result).

> Okay, so before finding bad commits, if possible, can you get block traces?

If you can teach me how to, sure!

In the meantime, maybe what happened is that f2fs leaves some gaps (e.g.
for alignment) when writing now, and didn't in 3.18? That would somehow
explain what I see.

Except that with the 3.18 code, there is virtually no read access while
writing to the disk for hours. With f2fs git, there is a small 16kb read
every half a minute or so, and after a while, a lot of "reading a few
mb/s, while writing a few mb/s".

-- 
                The choice of a       Deliantra, the free code+content MORPG
      -----==-     _GNU_              http://www.deliantra.net
      ----==-- _       generation
      ---==---(_)__  __ ____  __      Marc Lehmann
      --==---/ / _ \/ // /\ \/ /      sc...@sc...
      -=====/_/_//_/\_,_/ /_/\_\

Re: [f2fs-dev] write performance difference 3.18.21/git f2fs

[Jaegeuk K. <ja...@ke...>]

On Sat, Sep 26, 2015 at 03:59:57PM +0200, Marc Lehmann wrote:
> On Sat, Sep 26, 2015 at 12:52:53AM -0700, Jaegeuk Kim <ja...@ke...> wrote:
> > > Just for fun I'll start doing a -s64 run.
> 
> (which had the same result).
> 
> > Okay, so before finding bad commits, if possible, can you get block traces?
> 
> If you can teach me how to, sure!
> 
> In the meantime, maybe what happened is that f2fs leaves some gaps (e.g.
> for alignment) when writing now, and didn't in 3.18? That would somehow
> explain what I see.
> 
> Except that with the 3.18 code, there is virtually no read access while
> writing to the disk for hours. With f2fs git, there is a small 16kb read
> every half a minute or so, and after a while, a lot of "reading a few
> mb/s, while writing a few mb/s".

In order to verify this also, could you retrieve the following logs?

# echo 1 > /sys/kernel/debug/tracing/tracing_on
# echo 1 > /sys/kernel/debug/tracing/events/f2fs/f2fs_submit_read_bio/enable
# echo 1 > /sys/kernel/debug/tracing/events/f2fs/f2fs_submit_write_bio/enable
# cat /sys/kernel/debug/tracing/trace_pipe

Thanks,

> 
> -- 
>                 The choice of a       Deliantra, the free code+content MORPG
>       -----==-     _GNU_              http://www.deliantra.net
>       ----==-- _       generation
>       ---==---(_)__  __ ____  __      Marc Lehmann
>       --==---/ / _ \/ // /\ \/ /      sc...@sc...
>       -=====/_/_//_/\_,_/ /_/\_\

Re: [f2fs-dev] write performance difference 3.18.21/4.2.1

[Chao Yu <cha...@sa...>]

Hi Marc Jaegeuk,

> -----Original Message-----
> From: Marc Lehmann [mailto:sc...@sc...]
> Sent: Friday, September 25, 2015 2:51 PM
> To: Jaegeuk Kim
> Cc: lin...@li...
> Subject: Re: [f2fs-dev] write performance difference 3.18.21/4.2.1
> 
> On Thu, Sep 24, 2015 at 11:28:36AM -0700, Jaegeuk Kim <ja...@ke...> wrote:
> > One thing that we can try is to run the latest f2fs source in v3.18.
> > This branch supports f2fs for v3.18.
> 
> Ok, please bear with me, the last time I built my own kernel was during
> the 2.4 timeframe, and this is a ubuntu kernel. What I did is this:
> 
>    git clone -b linux-3.18 git://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs.git
>    cd f2fs/fs/f2fs
>    rsync -avPR include/linux/f2fs_fs.h include/trace/events/f2fs.h
> /usr/src/linux-headers-3.18.21-031821/.
>    make -C /lib/modules/3.18.21-031821-generic/build/ M=$PWD modules modules_install
> 
> I then rmmod f2fs/insmod the resulting module, and tried to mount my
> existing f2fs fs for a quick test, but got a null ptr exception on "mount":
> 
> http://ue.tst.eu/e4628dcee97324e580da1bafad938052.txt

This is my fault, sorry about introducing this oops. :(

Please revert the commit 7c5e466755ff ("f2fs: readahead cp payload
pages when mount") since in this commit we try to access invalid
SIT_I(sbi)->sit_base_addr which should be inited later.

Thanks,

> 
> Probably caused me not building a full kernel, but recreating how ubuntu
> build their kernels on a debian system isn't something I look forward to.
> 
> > For example, if I can represent blocks like:
> [number of logs discussion]
> 
> Thanks for this explanation - two logs doesn't look so bad, from a
> locality viewpoint (not a big issue for flash, but a big issue for
> rotational devices - I also realised I can't use dmcache as dmcache, even
> in writethrough mode, writes back all data after an unclean shutdown,
> which would positively kill the disk).
> 
> Since whatever speed difference I saw with two logs wasn't big, you
> completely sold me on 6 logs, or 4 (especially if it seepds up the gc,
> which I haven't much tested yet). Two logs was merely a test anyway (the
> same with no_heap, I don't know what it does, but I thought it is worth
> a try, as metadata + data nearer together is better than having them at
> opposite ends of the log or so).
> 
> --
>                 The choice of a       Deliantra, the free code+content MORPG
>       -----==-     _GNU_              http://www.deliantra.net
>       ----==-- _       generation
>       ---==---(_)__  __ ____  __      Marc Lehmann
>       --==---/ / _ \/ // /\ \/ /      sc...@sc...
>       -=====/_/_//_/\_,_/ /_/\_\
> 
> ------------------------------------------------------------------------------
> _______________________________________________
> Linux-f2fs-devel mailing list
> Lin...@li...
> https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

Re: [f2fs-dev] write performance difference 3.18.21/4.2.1

[Jaegeuk K. <ja...@ke...>]

On Fri, Sep 25, 2015 at 05:47:12PM +0800, Chao Yu wrote:
> Hi Marc Jaegeuk,
> 
> > -----Original Message-----
> > From: Marc Lehmann [mailto:sc...@sc...]
> > Sent: Friday, September 25, 2015 2:51 PM
> > To: Jaegeuk Kim
> > Cc: lin...@li...
> > Subject: Re: [f2fs-dev] write performance difference 3.18.21/4.2.1
> > 
> > On Thu, Sep 24, 2015 at 11:28:36AM -0700, Jaegeuk Kim <ja...@ke...> wrote:
> > > One thing that we can try is to run the latest f2fs source in v3.18.
> > > This branch supports f2fs for v3.18.
> > 
> > Ok, please bear with me, the last time I built my own kernel was during
> > the 2.4 timeframe, and this is a ubuntu kernel. What I did is this:
> > 
> >    git clone -b linux-3.18 git://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs.git
> >    cd f2fs/fs/f2fs
> >    rsync -avPR include/linux/f2fs_fs.h include/trace/events/f2fs.h
> > /usr/src/linux-headers-3.18.21-031821/.
> >    make -C /lib/modules/3.18.21-031821-generic/build/ M=$PWD modules modules_install
> > 
> > I then rmmod f2fs/insmod the resulting module, and tried to mount my
> > existing f2fs fs for a quick test, but got a null ptr exception on "mount":
> > 
> > http://ue.tst.eu/e4628dcee97324e580da1bafad938052.txt
> 
> This is my fault, sorry about introducing this oops. :(
> 
> Please revert the commit 7c5e466755ff ("f2fs: readahead cp payload
> pages when mount") since in this commit we try to access invalid
> SIT_I(sbi)->sit_base_addr which should be inited later.

Oops, I'll just remove this patch which was not going too far away.

Thanks,

> 
> Thanks,
> 
> > 
> > Probably caused me not building a full kernel, but recreating how ubuntu
> > build their kernels on a debian system isn't something I look forward to.
> > 
> > > For example, if I can represent blocks like:
> > [number of logs discussion]
> > 
> > Thanks for this explanation - two logs doesn't look so bad, from a
> > locality viewpoint (not a big issue for flash, but a big issue for
> > rotational devices - I also realised I can't use dmcache as dmcache, even
> > in writethrough mode, writes back all data after an unclean shutdown,
> > which would positively kill the disk).
> > 
> > Since whatever speed difference I saw with two logs wasn't big, you
> > completely sold me on 6 logs, or 4 (especially if it seepds up the gc,
> > which I haven't much tested yet). Two logs was merely a test anyway (the
> > same with no_heap, I don't know what it does, but I thought it is worth
> > a try, as metadata + data nearer together is better than having them at
> > opposite ends of the log or so).
> > 
> > --
> >                 The choice of a       Deliantra, the free code+content MORPG
> >       -----==-     _GNU_              http://www.deliantra.net
> >       ----==-- _       generation
> >       ---==---(_)__  __ ____  __      Marc Lehmann
> >       --==---/ / _ \/ // /\ \/ /      sc...@sc...
> >       -=====/_/_//_/\_,_/ /_/\_\
> > 
> > ------------------------------------------------------------------------------
> > _______________________________________________
> > Linux-f2fs-devel mailing list
> > Lin...@li...
> > https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

Re: [f2fs-dev] write performance difference 3.18.21/git f2fs

[Marc L. <sc...@sc...>]

On Mon, Sep 28, 2015 at 10:59:44AM -0700, Jaegeuk Kim <ja...@ke...> wrote:
> In order to verify this also, could you retrieve the following logs?

First thing, the allocation-failure-on-mount is still in the backported 3.18
f2fs module. If it's supposed to be gone in that version, it's not working:

http://ue.tst.eu/a1bc4796012bd7191ab2ada566d4cd22.txt

And here are traces and descriptions. The traces all start directly after
mount, my test script is http://data.plan9.de/f2fstest

(event tracing is cool btw., thanks for showing me :)

################ -s1, f2fs git ##############################################

   /opt/f2fs-tools/sbin/mkfs.f2fs -lTEST -s1 -t0 -a0 /dev/vg_test/test
   mount -t f2fs -onoatime,flush_merge,no_heap /dev/vg_test/test /mnt

For the fist ~120GB, performance was solid (100MB/s+), but much worse than
stock 3.18.21 (with -s64!).

3.1.8.21 regularly reached >190MB/s regularly (at least near the beginning
of the disk) then was idle in between writes, as the source wasn't fast
enough to keep up. With the backport, tar was almost never idle, and if,
then not for long, so it could just keep up. (Just keeping up with the
read speed of a 6-disk raid is very good, but I know f2fs can do much
better :)

At the 122GB mark, it started to slow down, being consistently <100MB/s

At 127GB, it was <<20MB/s, and I stopped.

Most of the time, the test was write-I/O-bound.

http://data.plan9.de/f2fs.s1.trace.xz

################ -s64, f2fs 3.18.21 #########################################

As contrast I then did a test with the original f2fs module, and -s64.
Throughput was up to 202MB/s, almost continously. At the 100GB mark, it
slowed down to maybe 170MB/s peak, which might well be the speed of the
platters.

I stopped at 217GB.

I have a 12GB mbuffer between the read-tar and the write-tar, configured to
write minimum bursts of ~120MB. At no time was the buffer filled at >2%,
while with the -s1, f2fs git case, it was basically always >2%.

The trace includes a few minutes after tar was stopped.

http://data.plan9.de/f2fs.s64.3.18.trace.xz

################ -s64, f2fs git #############################################

The direct equivalent of the previous test, but with f2fs git.

Almost from the very beginning, it was often write-bound, but could still
keep up.

At around 70GB, it mostly stopped being able to keep up, and the read
tar overtook the write tar. At 139GB, performance degraded to <2MB/s. I
stopped at 147GB.

So mostly, behaviour was the same as with -s1, excedpt it took longer to
slow down.

http://data.plan9.de/f2fs.s64.trace.xz

################ -s20, f2fs git #############################################

By special request, here is the test with -s20.

Surprisingly, this stopped being able to cope at the 40GB mark, but I didn't
wait very long after the previous test, maybe that influenced it. I stopped
at 63GB.

http://data.plan9.de/f2fs.s20.trace.xz

#############################################################################

I hope to find time to look at these traces myself later this day.

-- 
                The choice of a       Deliantra, the free code+content MORPG
      -----==-     _GNU_              http://www.deliantra.net
      ----==-- _       generation
      ---==---(_)__  __ ____  __      Marc Lehmann
      --==---/ / _ \/ // /\ \/ /      sc...@sc...
      -=====/_/_//_/\_,_/ /_/\_\

Re: [f2fs-dev] write performance difference 3.18.21/git f2fs

[Jaegeuk K. <ja...@ke...>]

On Tue, Sep 29, 2015 at 01:02:04PM +0200, Marc Lehmann wrote:
> On Mon, Sep 28, 2015 at 10:59:44AM -0700, Jaegeuk Kim <ja...@ke...> wrote:
> > In order to verify this also, could you retrieve the following logs?
> 
> First thing, the allocation-failure-on-mount is still in the backported 3.18
> f2fs module. If it's supposed to be gone in that version, it's not working:
> 
> http://ue.tst.eu/a1bc4796012bd7191ab2ada566d4cd22.txt

Oops, it seems I missed some other allocations too.
Could you pull the v3.18 again? I changed the patch.

> 
> And here are traces and descriptions. The traces all start directly after
> mount, my test script is http://data.plan9.de/f2fstest
> 
> (event tracing is cool btw., thanks for showing me :)

Thank you for the below traces.
I have two suspicious things from the traces where:

1. several inline_data conversion of existing files
Could you test -o noinline_data and share its trace too?

2. running out of free node ids along with its shrinker
The latest f2fs registers a shrinker to release cached free ids which will
be used for inode numbers when creating files. If there is no id, it needs
to read some meta blocks to fill up the empty lists.

I think its threshold does not consider enough for this use cases.
Let me think about this in more details.

Thanks,

> 
> ################ -s1, f2fs git ##############################################
> 
>    /opt/f2fs-tools/sbin/mkfs.f2fs -lTEST -s1 -t0 -a0 /dev/vg_test/test
>    mount -t f2fs -onoatime,flush_merge,no_heap /dev/vg_test/test /mnt
> 
> For the fist ~120GB, performance was solid (100MB/s+), but much worse than
> stock 3.18.21 (with -s64!).
> 
> 3.1.8.21 regularly reached >190MB/s regularly (at least near the beginning
> of the disk) then was idle in between writes, as the source wasn't fast
> enough to keep up. With the backport, tar was almost never idle, and if,
> then not for long, so it could just keep up. (Just keeping up with the
> read speed of a 6-disk raid is very good, but I know f2fs can do much
> better :)
> 
> At the 122GB mark, it started to slow down, being consistently <100MB/s
> 
> At 127GB, it was <<20MB/s, and I stopped.
> 
> Most of the time, the test was write-I/O-bound.
> 
> http://data.plan9.de/f2fs.s1.trace.xz
> 
> ################ -s64, f2fs 3.18.21 #########################################
> 
> As contrast I then did a test with the original f2fs module, and -s64.
> Throughput was up to 202MB/s, almost continously. At the 100GB mark, it
> slowed down to maybe 170MB/s peak, which might well be the speed of the
> platters.
> 
> I stopped at 217GB.
> 
> I have a 12GB mbuffer between the read-tar and the write-tar, configured to
> write minimum bursts of ~120MB. At no time was the buffer filled at >2%,
> while with the -s1, f2fs git case, it was basically always >2%.
> 
> The trace includes a few minutes after tar was stopped.
> 
> http://data.plan9.de/f2fs.s64.3.18.trace.xz
> 
> ################ -s64, f2fs git #############################################
> 
> The direct equivalent of the previous test, but with f2fs git.
> 
> Almost from the very beginning, it was often write-bound, but could still
> keep up.
> 
> At around 70GB, it mostly stopped being able to keep up, and the read
> tar overtook the write tar. At 139GB, performance degraded to <2MB/s. I
> stopped at 147GB.
> 
> So mostly, behaviour was the same as with -s1, excedpt it took longer to
> slow down.
> 
> http://data.plan9.de/f2fs.s64.trace.xz
> 
> ################ -s20, f2fs git #############################################
> 
> By special request, here is the test with -s20.
> 
> Surprisingly, this stopped being able to cope at the 40GB mark, but I didn't
> wait very long after the previous test, maybe that influenced it. I stopped
> at 63GB.
> 
> http://data.plan9.de/f2fs.s20.trace.xz
> 
> #############################################################################
> 
> I hope to find time to look at these traces myself later this day.
> 
> -- 
>                 The choice of a       Deliantra, the free code+content MORPG
>       -----==-     _GNU_              http://www.deliantra.net
>       ----==-- _       generation
>       ---==---(_)__  __ ____  __      Marc Lehmann
>       --==---/ / _ \/ // /\ \/ /      sc...@sc...
>       -=====/_/_//_/\_,_/ /_/\_\
> 
> ------------------------------------------------------------------------------
> _______________________________________________
> Linux-f2fs-devel mailing list
> Lin...@li...
> https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

Re: [f2fs-dev] write performance difference 3.18.21/git f2fs

[Chao Yu <cha...@sa...>]

> -----Original Message-----
> From: Jaegeuk Kim [mailto:ja...@ke...]
> Sent: Wednesday, September 30, 2015 7:13 AM
> To: Marc Lehmann
> Cc: lin...@li...
> Subject: Re: [f2fs-dev] write performance difference 3.18.21/git f2fs
> 
> On Tue, Sep 29, 2015 at 01:02:04PM +0200, Marc Lehmann wrote:
> > On Mon, Sep 28, 2015 at 10:59:44AM -0700, Jaegeuk Kim <ja...@ke...> wrote:
> > > In order to verify this also, could you retrieve the following logs?
> >
> > First thing, the allocation-failure-on-mount is still in the backported 3.18
> > f2fs module. If it's supposed to be gone in that version, it's not working:
> >
> > http://ue.tst.eu/a1bc4796012bd7191ab2ada566d4cd22.txt
> 
> Oops, it seems I missed some other allocations too.
> Could you pull the v3.18 again? I changed the patch.
> 
> >
> > And here are traces and descriptions. The traces all start directly after
> > mount, my test script is http://data.plan9.de/f2fstest
> >
> > (event tracing is cool btw., thanks for showing me :)
> 
> Thank you for the below traces.
> I have two suspicious things from the traces where:
> 
> 1. several inline_data conversion of existing files

Good catch, :)

This evidence is that the following trace message appear repeatedly for thousand
times, spreads in the whole trace log, and it should be outputted by inline
conversion here only.
"f2fs_submit_write_bio: dev = (252,9), WRITE_SYNC(P), DATA, sector = xxx, size = 4096"

It not only delay the tar about 50 ms for each time since inline conversion is
completely synchronous, but also use WRITE_SYNC request break mering of between IO
requests both in the f2fs bio cache and io scheduler of block layer, resulting in
falling in RMW mode io for SMR driver.

So I guess in this scenario noinline_data may be a good choice.

> Could you test -o noinline_data and share its trace too?
> 
> 2. running out of free node ids along with its shrinker
> The latest f2fs registers a shrinker to release cached free ids which will
> be used for inode numbers when creating files. If there is no id, it needs
> to read some meta blocks to fill up the empty lists.

Right, IMO, by compare with flash device, rotational device have worse random read
performance since its expensive seeking cost. So optimization should be applied
to building flow of free nid cache too. What I think is maybe in SMR_MODE we can
check whether our free nid number is lower than threshold value, if it is true, we
can readahead more (32?) nat blocks with low priority READA instead of READ_SYNC,
so this can prevent overmuch rotation cost and long latency cause by synchronous
cache building in alloc_nid.

Thanks,

> 
> I think its threshold does not consider enough for this use cases.
> Let me think about this in more details.
> 
> Thanks,
> 
> >
> > ################ -s1, f2fs git ##############################################
> >
> >    /opt/f2fs-tools/sbin/mkfs.f2fs -lTEST -s1 -t0 -a0 /dev/vg_test/test
> >    mount -t f2fs -onoatime,flush_merge,no_heap /dev/vg_test/test /mnt
> >
> > For the fist ~120GB, performance was solid (100MB/s+), but much worse than
> > stock 3.18.21 (with -s64!).
> >
> > 3.1.8.21 regularly reached >190MB/s regularly (at least near the beginning
> > of the disk) then was idle in between writes, as the source wasn't fast
> > enough to keep up. With the backport, tar was almost never idle, and if,
> > then not for long, so it could just keep up. (Just keeping up with the
> > read speed of a 6-disk raid is very good, but I know f2fs can do much
> > better :)
> >
> > At the 122GB mark, it started to slow down, being consistently <100MB/s
> >
> > At 127GB, it was <<20MB/s, and I stopped.
> >
> > Most of the time, the test was write-I/O-bound.
> >
> > http://data.plan9.de/f2fs.s1.trace.xz
> >
> > ################ -s64, f2fs 3.18.21 #########################################
> >
> > As contrast I then did a test with the original f2fs module, and -s64.
> > Throughput was up to 202MB/s, almost continously. At the 100GB mark, it
> > slowed down to maybe 170MB/s peak, which might well be the speed of the
> > platters.
> >
> > I stopped at 217GB.
> >
> > I have a 12GB mbuffer between the read-tar and the write-tar, configured to
> > write minimum bursts of ~120MB. At no time was the buffer filled at >2%,
> > while with the -s1, f2fs git case, it was basically always >2%.
> >
> > The trace includes a few minutes after tar was stopped.
> >
> > http://data.plan9.de/f2fs.s64.3.18.trace.xz
> >
> > ################ -s64, f2fs git #############################################
> >
> > The direct equivalent of the previous test, but with f2fs git.
> >
> > Almost from the very beginning, it was often write-bound, but could still
> > keep up.
> >
> > At around 70GB, it mostly stopped being able to keep up, and the read
> > tar overtook the write tar. At 139GB, performance degraded to <2MB/s. I
> > stopped at 147GB.
> >
> > So mostly, behaviour was the same as with -s1, excedpt it took longer to
> > slow down.
> >
> > http://data.plan9.de/f2fs.s64.trace.xz
> >
> > ################ -s20, f2fs git #############################################
> >
> > By special request, here is the test with -s20.
> >
> > Surprisingly, this stopped being able to cope at the 40GB mark, but I didn't
> > wait very long after the previous test, maybe that influenced it. I stopped
> > at 63GB.
> >
> > http://data.plan9.de/f2fs.s20.trace.xz
> >
> > #############################################################################
> >
> > I hope to find time to look at these traces myself later this day.
> >
> > --
> >                 The choice of a       Deliantra, the free code+content MORPG
> >       -----==-     _GNU_              http://www.deliantra.net
> >       ----==-- _       generation
> >       ---==---(_)__  __ ____  __      Marc Lehmann
> >       --==---/ / _ \/ // /\ \/ /      sc...@sc...
> >       -=====/_/_//_/\_,_/ /_/\_\
> >
> > ------------------------------------------------------------------------------
> > _______________________________________________
> > Linux-f2fs-devel mailing list
> > Lin...@li...
> > https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
> 
> ------------------------------------------------------------------------------
> _______________________________________________
> Linux-f2fs-devel mailing list
> Lin...@li...
> https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

Re: [f2fs-dev] write performance difference 3.18.21/git f2fs

[Marc L. <sc...@sc...>]

On Tue, Sep 29, 2015 at 04:13:10PM -0700, Jaegeuk Kim <ja...@ke...> wrote:
> > First thing, the allocation-failure-on-mount is still in the backported 3.18
> > f2fs module. If it's supposed to be gone in that version, it's not working:
> > 
> > http://ue.tst.eu/a1bc4796012bd7191ab2ada566d4cd22.txt
> 
> Oops, it seems I missed some other allocations too.
> Could you pull the v3.18 again? I changed the patch.

Hmm, I don't see any relevant change in the log, but I will use the newest
version.

> > (event tracing is cool btw., thanks for showing me :)
> 
> Thank you for the below traces.
> I have two suspicious things from the traces where:
> 
> 1. several inline_data conversion of existing files
> Could you test -o noinline_data and share its trace too?

WOW, THAT HELPED A LOT. While the peak throughput seems quite a bit lower
than 3.18 stock (seems, because this might be caused by less peaky, more
smooth, writes - I am only looking at it with my eyes), it could keep up
with the reading side easily, and performance didn't degrade so far. The
test continues, but here is the trace for the first 368GB:

http://data.plan9.de/f2fs.s64.noinline.trace.xz

I had a cursory look at the previous traces, and didn't see any obvious
problem (if anything, git f2fs seems to leave fewer gaps). However, both
versions do leave quite a few gaps.

I wanted to look at reordering, which might be a problem, too, but didn't
get to that.

Obviously, inline_data was the culprit for the especially bad performance.

-- 
                The choice of a       Deliantra, the free code+content MORPG
      -----==-     _GNU_              http://www.deliantra.net
      ----==-- _       generation
      ---==---(_)__  __ ____  __      Marc Lehmann
      --==---/ / _ \/ // /\ \/ /      sc...@sc...
      -=====/_/_//_/\_,_/ /_/\_\

Re: [f2fs-dev] write performance difference 3.18.21/git f2fs

[Marc L. <sc...@sc...>]

On Thu, Oct 01, 2015 at 02:11:20PM +0200, Marc Lehmann <sc...@sc...> wrote:
> WOW, THAT HELPED A LOT. While the peak throughput seems quite a bit lower

Ok, for completeness, here is the full log and a description of what was
going on.

http://data.plan9.de/f2fs.s64.noinline.full.trace.xz

status at the end + some idle time
http://ue.tst.eu/d16cf98c72fe9ecbac178ded47a21396.txt

It was faster than the reader till roughtly the 1.2TB mark, after
which it acquired longish episodes of being <<50MB/s (for example,
around 481842.363964), and also periods of ~20kb/s, due to many small
WRITE_SYNC's in a row (e.g. at 482329.101222 and 490189.681438,
http://ue.tst.eu/cc94978eafc736422437a4ab35862c12.txt). The small
WRITE_SYNCs did not always result in this behaviour by the disk, though.

After that, it was generally write-I/O bound.

Also, the gc seemed to have kicked in at around that time, which is kind
of counterproductive. I increased the gc_* values in /sys, but don't know
if that had any effect.

Most importantly, f2fs always recovered and had periods of much faster
writes (>= 120MB/s), so it's not the case that f2fs somehow saturates the
internal cache and then becomes slow forever.

Overall, the throughput was 83MB/s, which is 20% worse than stock 3.18, but
still way beyond what any other filesystem could do.

Also, writing 1TB in a single session, with somewhat reduced speed
afterwards, would be enough for my purposes, i.e. I can live with that
(still, gigabit speeds would be nice of course, as that is the data rate I
often deal with).

Notwithstanding any other improvements you might implement, f2fs has now
officially become my choice for SMR drives, the only remaining thing
needed is to convince me of its stability - it seems getting a kernel
with truly stable f2fs is a bit of a game of chance still, but I guess
confidence will come with more tests and actualy using it in production,
which I will do soon.

-- 
                The choice of a       Deliantra, the free code+content MORPG
      -----==-     _GNU_              http://www.deliantra.net
      ----==-- _       generation
      ---==---(_)__  __ ____  __      Marc Lehmann
      --==---/ / _ \/ // /\ \/ /      sc...@sc...
      -=====/_/_//_/\_,_/ /_/\_\