Re: [Squashfs-devel] Squashfs w/ xz compression on non-preemptable non-SMP linux kernel

[Klinedinst, E. <Eva...@al...>]

Clarifications:
System booting - U-boot boots the kernel off of jffs2, then squashfs with xz compression is used from then on as the kernel's RAM filesystem (initrd).

More analysis was done using various Linux tools:
 - bootchartd start; sleep 10; bootchartd stop
 - perf timechart record sleep 10; perf timechart

The resulting graphical .svg files showed that the squashfs file system with xz compression was causing the CPU starvation.

When squashfs has a cache miss on a particular file, it needs to decompress the file and reload it back into cache.  For example, a call to popen ('echo sysinfo | ntpq') needs at least these 3 commands to be in cache: sh, echo, and ntpq.  Whenever decompression is needed the operation is always run to completion, as there are no scheduler calls to relinquish the CPU and our Linux kernel doesn't have Preemption/SMP support built in.  As shown with the graphical .svg files, these decompression operations can take hundreds of milliseconds, if not more.  This leads to all user-space threads getting starved of CPU time.

Fix:
The fix was to have squashfs give up the CPU during a decompression operation by adding a Linux rescheduler call 'cond_resched()'.  This call basically tells the kernel to schedule a new process, if the conditions are right.

After re-running the various Linux tools,  mentioned above, no more CPU starvation issues were seen when squashfs's decompression operation occurred.

Only xz compression was fixed, as I couldn't find a good place higher up the call tree to put a cond_resched().  I would expect others decompression algorithms, in the abc_wrapper.c files, to have the same problem (ie. gzip, lzma, etc.).

-------- Original message --------
From: "Klinedinst, Evan" <Eva...@al...>
Date: 5/16/17 1:12 PM (GMT-05:00)
To: "'squ...@li...'" <squ...@li...>
Subject: [Squashfs-devel] Squashfs w/ xz compression on non-preemptable non-SMP linux kernel

Hi,

This is my first post to the list, so go easy on me lol ;).

Configuration:

-          4.4.6 based linux kernel

-          Non-SMP MIPS CPU

-          Preemption is disabled in the kernel (CONFIG_PREEMPTION=n)

-          JFFS2 filesystem – using squashfs - initrd with xz compression

-          mksquashfs root/ myTarget.initrd -noappend -always-use-fragments -b 1M -comp xz

Issue:

-          Whenever a user-space thread (real-time or not) invokes ‘popen (“someCommandA”, “r”)’, other user-space threads (real-time or not) get starved out for various ranges of time (upwards of ~350ms).

-          Using ‘perf timechart record sleep 10’ – I discovered that the ~350ms delay is coming from ‘someCommandA’ being decompressed.  If ‘someCommandA’ is run again then there is no noticeable delay because ‘someCommandA’ is now cached.

-          Running ‘echo 3 > /proc/sys/vm/drop_caches’ – will cause the delay again when invoking the same popen() call.

System:

-          ‘uname -a’ - Linux myTarget 4.4.6-at1 #2 Mon May 15 19:58:13 UTC 2017 mips GNU/Linux

Patch:

-          I’ve attached a patch (squashfs-xz.patch) against the 4.4.6 kernel that fixes the issue by adding a single cond_resched() call.

-          I’ve no idea if this is a proper place for a scheduler call, maybe more are needed elsewhere, and if there are alternatives in helping the decompression play nice with user-space threads.

-          If the kernel is compiled with preemption enabled (CONFIG_PREEMPTION=y) – the issue goes away entirely in our environment.

Concerns:

-          The ~350ms thread starvation delays are too long for our environment.

-          I’m afraid that if larger executables need to get decompressed, the thread starvation delays will just get much worse.

Thanks in advance,
Evan K

________________________________
This e-mail message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's e-mail System Administrator.

________________________________
This e-mail message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's e-mail System Administrator.