From: Yuichi N. <yn...@hi...> - 2007-09-10 04:35:58
|
Hello. I am porting SELinux to R2D+ board, and=20 measuring performance of SELinux's permission check. I found a strange fact. As you can see in forwarded e-mail that I posted to SELinux community, the overhead of SELinux on SH is bigger than other archs. > 3) Result(of lmbench)for SH(SH4, SH7751R), kernel 2.6.22 > Base SELinux Overhead(%) > Simple read 2.6781 6.4671 141.5 > Simple write 2.0781 5.3181 155.9 I found nested function calls in SELinux. And I tried inlining some functions,=20 then I found the overhead was reduced. > 3) Result for SH > Base SELinux Overhead(%) > Simple read 2.6781 3.6538 36.43(before 141.5) > Simple write 2.0781 3.321 59.80(before 155.9) In other archs, the effect of inlining was not so big, but in SH it was big. I could not find the reason. Does anyone know why? Are function calls on SH heavy? Or gcc optimization does not work well?(I used gcc 3.4.5) Forwarded by Yuichi Nakamura=20 ----------------------- Original Message ----------------------- From: Yuichi Nakamura=20 To: se...@ty... Date: Thu, 30 Aug 2007 14:45:16 +0900 Subject: [RFC][selinux] Tuning SELinux permission check ---- Hello. I am porting SELinux to embedded devices. And I found big overhead in read/write on CPUs for embedded devices. I tried tuning and found it can be reduced. I want comments from community. 1. Background Look at benchmark result below. lmbench simple read/write. Big overhead exists especially on SH(SuperH) arch. 1) Result for x86(Pentium 4 2.6Ghz), kernel 2.6.22 Base SELinux Overhead(%) Simple read 1.1034 1.2387 12.3 Simple write 0.9989 1.139 14.0 * Base: kernel compiled without SELinux support 2) Result for ARM(Intel XScale (PXA270 416MHz)), kernel 2.6.17 Base SELinux Overhead(%) Simple read 1.7945 3.1279 74.3 Simple write 1.4706 2.9228 98.7 3) Result for SH(SH4, SH7751R), kernel 2.6.22 Base SELinux Overhead(%) Simple read 2.6781 6.4671 141.5 Simple write 2.0781 5.3181 155.9 Overhead more than 100%! 2. Solution I found function calls in SELinux permission checks are causing performance overhead. I tried to reduce function calls like below. 1) avc_has_perm In avc_has_perm, avc_audit is called everytime. I think most access does not need to be audited,=20 so I changed it to be called only when necessary. 2) avc_has_perm_noaudit I changed it to "inline". 3) selinux_file_permission =46rom selinux_file_permission, file_has_perm -> inode_has_perm -> avc_has_perm functions are called. I made simple selinux_file_permission and reduced code. 3. Benchmark result after tuning Performance is improved especially in SH. also a little effective for x86. 1) Result for x86 Base SELinux Overhead(%) Simple read 1.1034 1.1939 8.2(before 12.3) Simple write 0.9989 1.1039 10.5(before 14.0) 2) Result for ARM Base SELinux Overhead(%) Simple read 1.7945 2.7555 53.6(before 74.3) Simple write 1.4706 2.5347 72.4(before 98.7) 3) Result for SH Base SELinux Overhead(%) Simple read 2.6781 3.6538 36.43(before 141.5) Simple write 2.0781 3.321 59.80(before 155.9) Are such changes acceptable ?? Below is a patch to linux-2.6.22. --- security/selinux/avc.c | 7 +++++-- security/selinux/hooks.c | 31 ++++++++++++++++++++++++++----- 2 files changed, 31 insertions(+), 7 deletions(-) diff -purN -X linux-2.6.22/Documentation/dontdiff linux-2.6.22.orig/securit= y/selinux/avc.c linux-2.6.22/security/selinux/avc.c --- linux-2.6.22.orig/security/selinux/avc.c 2007-07-09 08:32:17.000000000 = +0900 +++ linux-2.6.22/security/selinux/avc.c 2007-08-29 16:24:37.000000000 +0900 @@ -845,7 +845,7 @@ int avc_ss_reset(u32 seqno) * auditing, e.g. in cases where a lock must be held for the check but * should be released for the auditing. */ -int avc_has_perm_noaudit(u32 ssid, u32 tsid, +inline int avc_has_perm_noaudit(u32 ssid, u32 tsid, u16 tclass, u32 requested, struct av_decision *avd) { @@ -910,6 +910,9 @@ int avc_has_perm(u32 ssid, u32 tsid, u16 int rc; =20 rc =3D avc_has_perm_noaudit(ssid, tsid, tclass, requested, &avd); - avc_audit(ssid, tsid, tclass, requested, &avd, rc, auditdata); + + if ((requested & ~avd.allowed) | rc | (requested & avd.auditallow)) + avc_audit(ssid, tsid, tclass, requested, &avd, rc, auditdata); + return rc; } diff -purN -X linux-2.6.22/Documentation/dontdiff linux-2.6.22.orig/securit= y/selinux/hooks.c linux-2.6.22/security/selinux/hooks.c --- linux-2.6.22.orig/security/selinux/hooks.c 2007-07-09 08:32:17.00000000= 0 +0900 +++ linux-2.6.22/security/selinux/hooks.c 2007-08-29 16:21:51.000000000 +09= 00 @@ -2462,6 +2462,11 @@ static int selinux_file_permission(struc { int rc; struct inode *inode =3D file->f_path.dentry->d_inode; + struct avc_audit_data ad; + struct task_security_struct *tsec =3D current->security; + struct file_security_struct *fsec =3D file->f_security; + struct inode_security_struct *isec =3D inode->i_security; + u32 av; =20 if (!mask) { /* No permission to check. Existence test. */ @@ -2472,11 +2477,27 @@ static int selinux_file_permission(struc if ((file->f_flags & O_APPEND) && (mask & MAY_WRITE)) mask |=3D MAY_APPEND; =20 - rc =3D file_has_perm(current, file, - file_mask_to_av(inode->i_mode, mask)); - if (rc) - return rc; - + AVC_AUDIT_DATA_INIT(&ad, FS); + ad.u.fs.mnt =3D file->f_path.mnt; + ad.u.fs.dentry =3D file->f_path.dentry; + if (tsec->sid !=3D fsec->sid) { + rc =3D avc_has_perm(tsec->sid, fsec->sid, + SECCLASS_FD, + FD__USE, + &ad); + if (rc) + return rc; + } + + av =3D file_mask_to_av(inode->i_mode, mask); + if (av) { + if (unlikely (IS_PRIVATE (inode))) + return 0; + rc =3D avc_has_perm(tsec->sid, isec->sid, isec->sclass, + av, &ad); + if (rc) + return rc; + } return selinux_netlbl_inode_permission(inode, mask); } =20 Regards, --=20 Yuichi Nakamura Hitachi Software Engineering Co., Ltd. Japan SELinux Users Group(JSELUG): http://www.selinux.gr.jp/ SELinux Policy Editor: http://seedit.sourceforge.net/ -- This message was distributed to subscribers of the selinux mailing list. If you no longer wish to subscribe, send mail to maj...@ty... wi= th the words "unsubscribe selinux" without quotes as the message. --------------------- Original Message Ends -------------------- Regards, --=20 Yuichi Nakamura Hitachi Software Engineering Co., Ltd. Japan SELinux Users Group(JSELUG): http://www.selinux.gr.jp/ SELinux Policy Editor: http://seedit.sourceforge.net/ |
From: Paul M. <le...@li...> - 2007-09-10 04:59:01
|
On Mon, Sep 10, 2007 at 01:35:14PM +0900, Yuichi Nakamura wrote: > > 3) Result for SH > > Base SELinux Overhead(%) > > Simple read 2.6781 3.6538 36.43(before 141.5) > > Simple write 2.0781 3.321 59.80(before 155.9) > In other archs, the effect of inlining was not so big, > but in SH it was big. > > I could not find the reason. > Does anyone know why? > > Are function calls on SH heavy? > Or gcc optimization does not work well?(I used gcc 3.4.5) > I suspect you're mostly being bitten by your compiler version, it would be nice to see the numbers without the manual inlining with a more up-to-date compiler (4.1.x, 4.2.x, etc.). > 1) Result for x86(Pentium 4 2.6Ghz), kernel 2.6.22 > 2) Result for ARM(Intel XScale (PXA270 416MHz)), kernel 2.6.17 > 3) Result for SH(SH4, SH7751R), kernel 2.6.22 What versions of the compiler did you have for x86 and ARM? It's probably worth trying to do the manual inling for those old compilers (especially as there don't seem to be any regressions for any of the other platforms -- though you didn't list before and after code size), though I would be surprised if it's still problematic in current compilers. If you don't notice an improvement in more recent compilers, you may want to roll together a simple testcase and shove it in to the GCC bugzilla. |
From: Yuichi N. <yn...@hi...> - 2007-09-10 05:37:27
|
On Mon, 10 Sep 2007 13:58:47 +0900 Paul Mundt wrote: > On Mon, Sep 10, 2007 at 01:35:14PM +0900, Yuichi Nakamura wrote: > > > 3) Result for SH > > > Base SELinux Overhead(%) > > > Simple read 2.6781 3.6538 36.43(before 141.5) > > > Simple write 2.0781 3.321 59.80(before 155.9) > > In other archs, the effect of inlining was not so big, > > but in SH it was big. > > > > I could not find the reason. > > Does anyone know why? > > > > Are function calls on SH heavy? > > Or gcc optimization does not work well?(I used gcc 3.4.5) > > > I suspect you're mostly being bitten by your compiler version, it would > be nice to see the numbers without the manual inlining with a more > up-to-date compiler (4.1.x, 4.2.x, etc.). > > > 1) Result for x86(Pentium 4 2.6Ghz), kernel 2.6.22 > > 2) Result for ARM(Intel XScale (PXA270 416MHz)), kernel 2.6.17 > > 3) Result for SH(SH4, SH7751R), kernel 2.6.22 > > What versions of the compiler did you have for x86 and ARM? Thanks for reply and advice. For arm, it is 3.4.4, for x86, 4.1.2. > It's probably worth trying to do the manual inling for those old > compilers (especially as there don't seem to be any regressions for any > of the other platforms -- though you didn't list before and after code > size), though I would be surprised if it's still problematic in current > compilers. If you don't notice an improvement in more recent compilers, > you may want to roll together a simple testcase and shove it in to the > GCC bugzilla. I would like to try newer gcc. -- Yuichi Nakamura Hitachi Software Engineering Co., Ltd. Japan SELinux Users Group(JSELUG): http://www.selinux.gr.jp/ SELinux Policy Editor: http://seedit.sourceforge.net/ |
From: Paul M. <le...@li...> - 2007-09-10 05:56:38
|
On Mon, Sep 10, 2007 at 02:37:04PM +0900, Yuichi Nakamura wrote: > On Mon, 10 Sep 2007 13:58:47 +0900 > Paul Mundt wrote: > > It's probably worth trying to do the manual inling for those old > > compilers (especially as there don't seem to be any regressions for any > > of the other platforms -- though you didn't list before and after code > > size), though I would be surprised if it's still problematic in current > > compilers. If you don't notice an improvement in more recent compilers, > > you may want to roll together a simple testcase and shove it in to the > > GCC bugzilla. > I would like to try newer gcc. > http://userweb.kernel.org/~lethal/gnush4_linux_v0702_rc-1-1.i386.tar.gz |
From: Yuichi N. <yn...@hi...> - 2007-09-10 06:58:44
|
On Mon, 10 Sep 2007 14:56:25 +0900 Paul Mundt wrote: > On Mon, Sep 10, 2007 at 02:37:04PM +0900, Yuichi Nakamura wrote: > > On Mon, 10 Sep 2007 13:58:47 +0900 > > Paul Mundt wrote: > > > It's probably worth trying to do the manual inling for those old > > > compilers (especially as there don't seem to be any regressions for any > > > of the other platforms -- though you didn't list before and after code > > > size), though I would be surprised if it's still problematic in current > > > compilers. If you don't notice an improvement in more recent compilers, > > > you may want to roll together a simple testcase and shove it in to the > > > GCC bugzilla. > > I would like to try newer gcc. > > > > http://userweb.kernel.org/~lethal/gnush4_linux_v0702_rc-1-1.i386.tar.gz Thanks, It is useful! I used gcc in above and compiled kernel. I found that performance does not get better when compiled by 4.2.1. Even 3.4.5 is better??? lmbench result for SH(SH4, SH7751R), kernel 2.6.22 SELinux(gcc3.4.5) SELinux(gcc4.2.1) Simple read 5.37 6.03 Simple write 5.06 5.92 # Result of 3.4.5 is a little different, I changed kernel config -- Yuichi Nakamura Hitachi Software Engineering Co., Ltd. Japan SELinux Users Group(JSELUG): http://www.selinux.gr.jp/ SELinux Policy Editor: http://seedit.sourceforge.net/ |
From: Paul M. <le...@li...> - 2007-09-10 07:09:19
|
On Mon, Sep 10, 2007 at 03:58:21PM +0900, Yuichi Nakamura wrote: > On Mon, 10 Sep 2007 14:56:25 +0900 > Paul Mundt wrote: > > On Mon, Sep 10, 2007 at 02:37:04PM +0900, Yuichi Nakamura wrote: > > > I would like to try newer gcc. > > > > > > > http://userweb.kernel.org/~lethal/gnush4_linux_v0702_rc-1-1.i386.tar.gz > Thanks, It is useful! > I used gcc in above and compiled kernel. > > I found that performance does not get better when compiled by 4.2.1. > Even 3.4.5 is better??? > What optimization levels are you using to compile the kernel? Try with and without -Os to see if there's a measurable impact. From the sounds of it you're still going to have to do the manual inlining work if there's no change in the common case, at least. |
From: Yuichi N. <yn...@hi...> - 2007-09-10 09:19:34
|
On Mon, 10 Sep 2007 16:09:10 +0900 Paul Mundt wrote: > On Mon, Sep 10, 2007 at 03:58:21PM +0900, Yuichi Nakamura wrote: > > On Mon, 10 Sep 2007 14:56:25 +0900 > > Paul Mundt wrote: > > > On Mon, Sep 10, 2007 at 02:37:04PM +0900, Yuichi Nakamura wrote: > > > > I would like to try newer gcc. > > > > > > > > > > http://userweb.kernel.org/~lethal/gnush4_linux_v0702_rc-1-1.i386.tar.gz > > Thanks, It is useful! > > I used gcc in above and compiled kernel. > > > > I found that performance does not get better when compiled by 4.2.1. > > Even 3.4.5 is better??? > > > What optimization levels are you using to compile the kernel? Try with > and without -Os to see if there's a measurable impact. From the sounds of > it you're still going to have to do the manual inlining work if there's > no change in the common case, at least. I was using -O2. I now compiled with -Os and compared. lmbench result for SH(SH4, SH7751R), kernel 2.6.22 SELinux(gcc4.2.1 -O2) SELinux(gcc 4.2.1 -Os) Simple read 6.03 4.24 Simple write 5.92 3.88 Very strange. -Os is better. # I tried twice to make sure. No SELinux(gcc4.2.1 -O2) No SELinux(gcc 4.2.1 -Os) Simple read 2.36 2.45 Simple write 2.06 2.08 Without SELinux, O2 is a little better. Regards, -- Yuichi Nakamura Hitachi Software Engineering Co., Ltd. Japan SELinux Users Group(JSELUG): http://www.selinux.gr.jp/ SELinux Policy Editor: http://seedit.sourceforge.net/ |
From: Paul M. <le...@li...> - 2007-09-10 09:35:42
|
On Mon, Sep 10, 2007 at 06:19:12PM +0900, Yuichi Nakamura wrote: > > On Mon, 10 Sep 2007 16:09:10 +0900 > Paul Mundt wrote: > > What optimization levels are you using to compile the kernel? Try with > > and without -Os to see if there's a measurable impact. From the sounds of > > it you're still going to have to do the manual inlining work if there's > > no change in the common case, at least. > I was using -O2. > I now compiled with -Os and compared. > > lmbench result for SH(SH4, SH7751R), kernel 2.6.22 > SELinux(gcc4.2.1 -O2) SELinux(gcc 4.2.1 -Os) > Simple read 6.03 4.24 > Simple write 5.92 3.88 > Very strange. -Os is better. > # I tried twice to make sure. > The inlining semantics for both of these are a bit different, but it's curious that -Os resulted in faster code given that it has a tendency to try and keep things that are referenced more than once out-of-line. > No SELinux(gcc4.2.1 -O2) No SELinux(gcc 4.2.1 -Os) > Simple read 2.36 2.45 > Simple write 2.06 2.08 > Without SELinux, O2 is a little better. > What about bumping this to -O3? It seems like the compiler is at least able to produce good code, the main issue is just the optimization level and inlining semantics at this point. On the other hand, with newer compilers we really should be using -Os as the default anyways (this is also the default for most ARM boards), it hasn't been a problem for some time. |
From: Yuichi N. <yn...@hi...> - 2007-09-11 03:42:32
|
On Mon, 10 Sep 2007 18:35:28 +0900 Paul Mundt wrote: > On Mon, Sep 10, 2007 at 06:19:12PM +0900, Yuichi Nakamura wrote: > > > > On Mon, 10 Sep 2007 16:09:10 +0900 > > Paul Mundt wrote: > > > What optimization levels are you using to compile the kernel? Try with > > > and without -Os to see if there's a measurable impact. From the sounds of > > > it you're still going to have to do the manual inlining work if there's > > > no change in the common case, at least. > > I was using -O2. > > I now compiled with -Os and compared. > > > > lmbench result for SH(SH4, SH7751R), kernel 2.6.22 > > SELinux(gcc4.2.1 -O2) SELinux(gcc 4.2.1 -Os) > > Simple read 6.03 4.24 > > Simple write 5.92 3.88 > > Very strange. -Os is better. > > # I tried twice to make sure. > > > The inlining semantics for both of these are a bit different, but it's > curious that -Os resulted in faster code given that it has a tendency to > try and keep things that are referenced more than once out-of-line. > > > No SELinux(gcc4.2.1 -O2) No SELinux(gcc 4.2.1 -Os) > > Simple read 2.36 2.45 > > Simple write 2.06 2.08 > > Without SELinux, O2 is a little better. > > > What about bumping this to -O3? It seems like the compiler is at least > able to produce good code, the main issue is just the optimization level > and inlining semantics at this point. Thanks for reply, again. I tried -O3. Kernel panic on gcc 3.4.5. On 4.2.1 it booted. SELinux(gcc 4.2.1 -O3) Simple read 4.53 Simple write 4.23 Better than O2, but Os is better. I tried manual inlining with -Os, SELinux(gcc 4.2.1 -Os) + manual inlining Simple read 3.64 Simple write 3.34 It is still effective. > On the other hand, with newer > compilers we really should be using -Os as the default anyways (this is > also the default for most ARM boards), it hasn't been a problem for some > time. Does it mean, "most people are using -Os now" ? -- Yuichi Nakamura Hitachi Software Engineering Co., Ltd. Japan SELinux Users Group(JSELUG): http://www.selinux.gr.jp/ SELinux Policy Editor: http://seedit.sourceforge.net/ |
From: Paul M. <le...@li...> - 2007-09-11 04:03:33
|
On Tue, Sep 11, 2007 at 12:42:10PM +0900, Yuichi Nakamura wrote: > On Mon, 10 Sep 2007 18:35:28 +0900 > Paul Mundt wrote: > > On the other hand, with newer > > compilers we really should be using -Os as the default anyways (this is > > also the default for most ARM boards), it hasn't been a problem for some > > time. > Does it mean, "most people are using -Os now" ? > Correct. Quite a few of the SH boards already use this as their default: pmundt@dysnomia ~/devel/git/sh-2.6.24 $ grep CC_OPTIMIZE_FOR_SIZE arch/sh/configs -r | wc -l 27 pmundt@dysnomia ~/devel/git/sh-2.6.24 $ grep CC_OPTIMIZE_FOR_SIZE=y arch/sh/configs -r | wc -l 15 And almost all of the up-to-date ports are using it: pmundt@dysnomia ~/devel/git/sh-2.6.24 $ grep 'version: 2.6.2[12]' arch/sh/configs -r | sed -e 's/:.*$//' | wc -l 12 pmundt@dysnomia ~/devel/git/sh-2.6.24 $ grep CC_OPTIMIZE_FOR_SIZE=y $(grep 'version: 2.6.2[12]' arch/sh/configs -r | sed -e 's/:.*$//') | wc -l 9 RTS7751R2D is one of the ones that is not using it, so it's probably worth just enabling it there and for all of the other boards, too. We at least don't have any regressions with it any more with recent compilers. |