Re: [mod-security-users] ModSecurity phase timing

[Christian F. <chr...@ne...>]

Hello Felipe,

Thank you for your openness to discuss this on github in detail. Still,
I'd like to continue in this thread because it has already started and
because it interests a wider audience due of certain implications that
have to do with the way you handle our concerns.

On Mon, Mar 26, 2018 at 05:08:50PM +0000, Felipe Zimmerle wrote:
> I have no bug report saying that DURATION is not working and a regression
> test that leads me to believe that it is working. Thus, I am assuming that
> it is working Ok.

Here is your new bug report:

https://github.com/SpiderLabs/ModSecurity/issues/1725

TL&DR; DURATION kind of works, but DURATION macro expansion does not work in
msg and logdata context and the value of variable is not correct according to
description in reference handbook at
https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual#DURATION 
You are returning a float, while DURATION is described to be microseconds.

The float is a real issue. You dropped the PERF_ variables for 3.0 and
told us yesterday to work with DURATION. That means to calculate phase timings
by doing arithmetics based on interim values of DURATION. 
But now it turns out that this path is barred because ModSec arithmetics only 
works with integer numbers (-> microseconds!) but not with floats.

So again, we are stuck with a non-working implementation.

I have just been commissioned to come up with a decent approach to do
performance analysis of existing services on libModSecurity / NGINX. This is
great for me, because it means business. But it is bad for ModSecurity because
people should be able to do this without Christian Folini coming up with a
smart method. It should be easier than this (and it should be documented!):

The standard questions in such situations is:
- Which requests have a performance issue?
- Is the performance issue with ModSecurity or with the backend application?

With ModSecurity 2.9.2 this is easy to find out. See my tutorials
https://www.netnea.com/cms/apache-tutorial-5_extending-access-log/
https://www.netnea.com/cms/apache-tutorial-6_embedding-modsecurity/
that describe an access log format that brings this performance data for
every request and the ModSec rules necessary to fill the variables.
This settles 90% of all performance issues immediately.

Now NGINX does not allow to display env variables easily in the access log
and libModSecurity does not yet support env variables anyways. So I assumed
I would have to take a different path. Now it looks like I would need to
write everything into the error log and _then_ calculate the timings
externally with a new tool / script that works with floating point numbers.

And additionally I will look into systemtap because you told us to do so
and because the example you give may be useful for experienced stap users,
but it's beyond the knowledge level of the average ModSecurity user.

I am sorry if this message is a bit rude. I usually try to remain calm and
diplomatic. But this time, it did not work out and sleeping it over did
not work either.

We are all together on this ModSecurity ship. You are the captain and you
steer the ship based on the decisions you think are best. The rest of the
community only notices the decision once the ship starts to turn. You
stated libModSecurity would be feature compatible to 2.9.x. Now we learn it
is not and we have to make fundamental changes to our setups and methodology.

I am totally open for a discussion on the need for the PERF_ variables.
If they mean a performance hit, then tell us how much and tell us why you
think it is not acceptable. How is the libModSecurity 3.0 performance
stacking up against 2.9.x? Is it really so good, that you want to squeeze
out the last few cpu cycles to the very limit and PERF_ is the last
remaining target?
Having to calculate performance via arithemtics
and DURATION also has a performance hit, and maybe one could make the
PERF_ variables optional (compile time flag or directive that is off by
default). But please do not take these decisions silently and then brush
away our concerns.

Best regards,

Christian


-- 
Life would be tragic if it weren't funny. ... My expectations were
reduced to zero when I was 21. Everything since then has been a bonus.
-- Stephen Hawking