Big issue with whitelisting system

Marc
2013-01-30
2013-05-23
  • Marc
    Marc
    2013-01-30

    Hello, let me explain the situation and what I have tryed.

    First of all, I have multiple services that are frequently being attacked, I'm having a DDoS mitigation protection from Black Lotus that is one of the leader in the area,

    the things is, while it's easy for them to detect aggresive attack and block them, it require an additional Web Firewall to be able to mitigate http attacks, the firewall is ok, but, more domains on it reduce the performance, and I have also some services that are attacked as well, but delivering a very light content, and that's why, a http DDoS does not affect them so much, and it's like NO PROBLEM AT ALL if the web server is protected by mod_qos.

    The problem is that, the Web Application Firewall (WAF) I'm having from Black Lotus is acting as a proxy, and then, it pass the real IP into 2 headers : "X-Real-IP" and "X-Forwarded-For"

    Some solutions came up to me :

    the first one, and the most obvious, was to completly WHITELIST the WAF, since the traffic coming from it is 99.99999% clean, I've attempted multiple try, but came to fail.

    QS_VipIpUser 199.59.162.168 # should work only if the user is logged on
    #QS_VipIpUser [2604:8300:100:1000::7]
    QS_SrvMaxConnExcludeIP 199.59.162.168 # should totally exclude this IP from any BLOCK attempt
    #QS_SrvMaxConnExcludeIP [2604:8300:100:1000::7]
    SetEnvIf Remote_Addr 199.59.162.168 QS_VipRequest=yes # should make any single request from this IP to be VIP
    SetEnvIf Remote_Addr 199.59.162.168 QS_LowPrio=no # should force LowPriority to NO for this IP at every request
    SetEnvIf Remote_Addr 199.59.162.168 QS_Block=no # Should force BLOCK=NO for this IP at every request
    #SetEnvIf Remote_Addr [2604:8300:100:1000::7] QS_VipRequest=no
    #SetEnvIf Remote_Addr [2604:8300:100:1000::7] QS_LowPrio=no
    #SetEnvIf Remote_Addr [2604:8300:100:1000::7] QS_Block=no
    QS_SetEnvIf QS_VipRequest   !QS_Intra  QS_SrvMaxConnPerIP=2000
    

    First, I came up with a big issue, setting an SetEnvIf with a ipv6 break the "if" part, and apply the modification to everyone instead of just the IP, so I had to comment them out.

    Second, none of the rules I've used have been working, but, when I do a php var_dump($_SERVER), I can see that the headers are well set by mod_qos, for the VIP Session=Yes, for the Block=No, for the LowPrio = No, so the IP detection itself works.
    As the VIPSession was "YES" for my single request, I've added a param for VIP Users that increase the amount of QS_SrvMaxConnPerIP to 2000, just for the try, but even if I see this header in my var_dump($_SERVER), the IP was blocked after 50 ConnPerIP as the global configuration do.

    The QS_SrvMaxConnExcludeIP does not work too…
    Note: QS_ClientPrefer is activated

    Result ? The WAF is banned after the amount of max connection allowed:

    QS_SrvMaxConnPerIP       50
    QS_ClientEventBlockCount 50 300
    

    Why the whitelisting does not work ?

    I want to totally whitelist the IP address  and 199.59.162.168.

    I came out with a SECOND solution :
    QS_ClientIpFromHeader HTTP_X_REAL_IP

    Again, the REAL IP is well interpreted by mod_qos as I see in the var_dump, but it keep counting for the Firewall IP, WHY ??? It should count only the HTTP_X_REAL_IP header.

    (Note: this header is working as well as X_Forwarded_For by my firewall, and as I just said, mod_qos headers was mentionning the real IP corretly, not just using it for blocking attempt)

     
  • Marc
    Marc
    2013-01-30

    My full configuration, mostly the one in the doc.

    # maximum number of active TCP connections is limited to 896 (limited
    # by the available memory, adjust the settings according to the used
    # hardware):
    MaxClients               896
    # idle timeout:
    Timeout                  15
    # keep alive (for up to 60% of all connections):
    KeepAlive                on
    MaxKeepAliveRequests     60
    KeepAliveTimeout         5
    QS_SrvMaxConnClose       70%
    # name of the HTTP response header which marks preferred clients (this
    # may be used to let the application decide which clients are "good" and
    # have higher privileges, e.g. authenticated users. you may also use
    # the QS_VipUser directive when using an Apache authentication module such
    # as mod_auth_basic or mod_auth_oid):
    QS_VipIPHeaderName       vipzzz
    # enables the known client prefer mode (server allows new TCP connections
    # from known/good clients only when is has more than 716 open TCP connections):
    QS_ClientPrefer 80
    # minimum request/response speed (deny slow clients blocking the server, 
    # e.g. defending slowloris):
    QS_SrvMinDataRate        120 1500 400
    # and limit request line, header and body:
    LimitRequestLine         7168
    LimitRequestFields       30
    QS_LimitRequestBody      102400
    # don't allow more than 30 TCP connections per client source address:
    QS_SrvMaxConnPerIP       50
    QS_VipIpUser 199.59.162.168
    #QS_VipIpUser [2604:8300:100:1000::7]
    QS_SrvMaxConnExcludeIP 199.59.162.168
    #QS_SrvMaxConnExcludeIP [2604:8300:100:1000::7]
    SetEnvIf Remote_Addr 199.59.162.168 QS_VipRequest=yes
    SetEnvIf Remote_Addr 199.59.162.168 QS_LowPrio=no
    SetEnvIf Remote_Addr 199.59.162.168 QS_Block=no
    #SetEnvIf Remote_Addr [2604:8300:100:1000::7] QS_VipRequest=no
    #SetEnvIf Remote_Addr [2604:8300:100:1000::7] QS_LowPrio=no
    #SetEnvIf Remote_Addr [2604:8300:100:1000::7] QS_Block=no
    #QS_SetEnvIf QS_IsVipRequest QS_SrvMaxConnPerIP=1000
    QS_SetEnvIf QS_VipRequest   !QS_Intra  QS_SrvMaxConnPerIP=2000
    QS_ClientIpFromHeader HTTP_X_REAL_IP
    LogFormat "%h %u %t \"%r\" %>s %b %T \"%{content-length}i\" %k \"%{User-Agent}i\" \
               %{mod_qos_cr}e %{mod_qos_ev}e %{mod_qos_con}e %{QS_SrvConn}e %{QS_AllConn}e \
               id=%{UNIQUE_ID}e %{QS_ConnectionId}e %{mod_qos_user_id}e %{QS_Country}e #%P"
    # block clients violating some basic rules frequently (don't allows more than 20
    # violations within 5 minutes):
    QS_ClientEventBlockCount 50 300
    QS_SetEnvIfStatus        400               QS_Block
    QS_SetEnvIfStatus        401               QS_Block
    QS_SetEnvIfStatus        403               QS_Block
    #QS_SetEnvIfStatus        404               QS_Block
    QS_SetEnvIfStatus        405               QS_Block
    #QS_SetEnvIfStatus        406               QS_Block
    QS_SetEnvIfStatus        408               QS_Block
    QS_SetEnvIfStatus        411               QS_Block
    QS_SetEnvIfStatus        413               QS_Block
    QS_SetEnvIfStatus        414               QS_Block
    QS_SetEnvIfStatus        417               QS_Block
    #QS_SetEnvIfStatus        500               QS_Block
    #QS_SetEnvIfStatus        503               QS_Block
    QS_SetEnvIfStatus        505               QS_Block
    QS_SetEnvIfStatus        QS_SrvMinDataRate QS_Block
    QS_SetEnvIfStatus        NullConnection    QS_Block
    <Location /lotus-status2>
       SetHandler qos-viewer
       Order allow,deny
       Allow from all
    </Location>
    <Location /lotus-status2/console>
       SetHandler qos-console
       Order allow,deny
       Allow from all
    </Location>
    QS_DisableHandler off
    
     
  • Hi Marc.
    The QS_ClientEventBlockCount directive does not evaluate the value of the QS_Block variable but checks if the variable is set or not only. Therefore, you have to unset the variable. Please pay attention to the fact that the variable may be set at response processing (status code check for example) too overwriting any variables set/unset during request processing (e.g. using the SetEnvIf directive), see also http://opensource.adnovum.ch/mod_qos/mod_qos_seq.gif.

    Sample configuration:

    # define that this IP shall be ignored
    SetEnvIf Remote_Addr 199.59.162.168 IgnoreIP=yes
    SetEnvIf Remote_Addr 199.59.162.168 QS_VipRequest=yes
    # unset the QS_Block variable
    QS_SetEnvIf IgnoreIP QS_Block !QS_Block
    

    Furthermore, the statement

    QS_SetEnvIf QS_VipRequest !QS_Intra QS_SrvMaxConnPerIP=2000
    

    does not have any effect at all since QS_SrvMaxConnPerIP is a directive and not an environment variable. You may use the QS_SrvMaxConnExcludeIP directive instead (within the VirtualHost if you use any).

    QS_SrvMaxConnExcludeIP 199.59.162.168
    

    Regards, Pascal