'Fatal error' @ switch to iptables-restore?

pgnd
2012-09-01
2013-03-05
  • pgnd
    pgnd
    2012-09-01

    I'm attempting to switch my firewall's activation policy to:

    "Use iptables-restore to activate policy"

    Without this setting, i.e. NOT using iptables-restore, COMPILE & INSTALL to remote both complete without error.

    WITH this setting enabled, COMPILE completes without error, but INSTALL to remote fails at:

        ...
        debug1: Sending environment.
        debug1: Sending command: echo '--**--**--';
        chmod +x /usr/local/etc/fw/fw_edgebox.fw;
        sh /usr/local/etc/fw/fw_edgebox.fw && echo 'Policy activated'
        --**--**--
        Activating firewall script generated Sat Sep  1 01:11:56 2012 by admin6
        Running prolog script
        Logged in
        iptables-restore: line 240 failed
        *** Fatal error :
        iptables-restore: line 240 failed
        Firewall policy installation failed
        Running epilog script
        debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
        debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
        debug1: channel 0: free: client-session, nchannels 1
        ...
    

    I can't figure out the source of this error: "Line 240" of what file? /usr/local/etc/fw/fw_edgebox.fw?

    Line 240 of *that* file is:

    ...
    237     $IP link show dev $interface >/dev/null 2>&1 && {
    238       CURRENT_ADDRS_ALL_SCOPES=$(list_addresses_by_scope $interface 'scope .*' "$ignore_list")
    239       CURRENT_ADDRS_GLOBAL_SCOPE=$(list_addresses_by_scope $interface 'scope global' "$ignore_list")
    240     } || {
    241       echo "# Interface $interface does not exist"
    242       # Stop the script if we are not in test mode
    243       test -z "$FWBDEBUG" && exit 1
    244     }
    ...
    

    my 'prolog' contains, simply,

        /bin/sh /usr/local/etc/fw/scripts/load_ipsets.sh
    

    where,

        cat /usr/local/etc/fw/scripts/load_ipsets.sh
            #! /bin/sh
            SH="/bin/sh"
            IPSET="/usr/sbin/ipset"
            $IPSET -quiet -exist create FAST_DROP_IP          hash:ip
            $IPSET -quiet -exist create FAST_DROP_NET         hash:net
            $IPSET flush FAST_DROP_IP
            $IPSET flush FAST_DROP_NET
            $IPSET -quiet -exist create IPPESTS_IP            hash:ip
            $IPSET -quiet -exist create IPPESTS_NET           hash:net
            $IPSET -quiet -exist create IPPESTS_IP_PRESWAP    hash:ip
            $IPSET -quiet -exist create IPPESTS_NET_PRESWAP   hash:net
            $IPSET -exist restore < /usr/local/etc/fw/lists/ipsets/ippests_ip.ipset
            $IPSET -exist restore < /usr/local/etc/fw/lists/ipsets/ippests_net.ipset
            $IPSET  swap  IPPESTS_IP_PRESWAP  IPPESTS_IP
            $IPSET  swap  IPPESTS_NET_PRESWAP IPPESTS_NET
            $IPSET  flush IPPESTS_IP_PRESWAP
            $IPSET  flush IPPESTS_NET_PRESWAP
    

    Where to look for the source of this error?

     
  • Vadim Kurland
    Vadim Kurland
    2012-09-01

    look for the fragment that looks like this:

        # ================ IPv4
        (
        echo '*filter'
    

    and ends with

        echo COMMIT
        ) | $IPTABLES_RESTORE; IPTABLES_RESTORE_RES=$?
    

    and count lines starting with "echo"

     
  • pgnd
    pgnd
    2012-09-01

    Thanks. I'd have not found that easily.

    Starting from the 'echo', line 240 is:

            #
            # ================ Table 'filter', rule set VPN_RULES
            #
            # Rule  VPN_RULES 0 (global)
            echo ":VPN_RULES - [0:0]"
            echo ":VPN_RULES_0 - [0:0]"
            echo "-A VPN_RULES  -m state --state NEW  -j VPN_RULES_0 "
    240     echo "-A VPN_RULES_0  -j LOG  --log-prefix \"[P:0} {ACCEPT}_\""
            echo "-A VPN_RULES_0  -j ACCEPT "
            #
            # ================ Table 'filter', rule set _Policy
            #
    

    the right bracket in the "… [P:0} …" looks odd.

    That rule has logging 'on', with NO (blank) associated label. If I individually compile that rule, I get, as above

        fw_edgebox / VPN_RULES / rule 0
        $IPTABLES -N VPN_RULES
        $IPTABLES -N VPN_RULES_0
        $IPTABLES -A VPN_RULES  -m state --state NEW  -j VPN_RULES_0
        $IPTABLES -A VPN_RULES_0  -j LOG  --log-prefix "[P:0} {ACCEPT}_"
        $IPTABLES -A VPN_RULES_0  -j ACCEPT
    

    If I turn its logging off, so that it compiles as

        fw_edgebox / VPN_RULES / rule 0
        $IPTABLES -N VPN_RULES
        $IPTABLES -A VPN_RULES  -m state --state NEW  -j ACCEPT
    

    then compile/install the entire policy with iptables-restore enables, it again fails

    ...
    Running prolog script
    iptables-restore: line 237 failed
    *** Fatal error :
    iptables-restore: line 237 failed
    Firewall policy installation failed
    ...
    

    where @237,

            #
            # ================ Table 'filter', rule set VPN_RULES
            #
            # Rule  VPN_RULES 0 (global)
    237     echo ":VPN_RULES - [0:0]"
            echo "-A VPN_RULES  -m state --state NEW  -j ACCEPT "
            #
            # ================ Table 'filter', rule set _Policy
            #
    

    If I completely disable the branching rule in main policy that references that rule, and try again, now FAIL @

    ...
    Running prolog script
    iptables-restore: line 232 failed
    *** Fatal error :
    iptables-restore: line 232 failed
    Firewall policy installation failed
    ...
    

    where

            # Rule  VoIP_RULES 1 (global)
            echo "-A VoIP_RULES  -d xx.xx.xx.xx   -m state --state NEW  -j ACCEPT "
            echo "-A VoIP_RULES  -d xx.xx.xx.xx   -m state --state NEW  -j ACCEPT "
            echo "-A VoIP_RULES  -d xx.xx.xx.xx/23   -m state --state NEW  -j ACCEPT "
            echo "-A VoIP_RULES  -d xx.xx.xx.xx   -m state --state NEW  -j ACCEPT "
            echo "-A VoIP_RULES  -d xx.xx.xx.xx   -m state --state NEW  -j ACCEPT "
    232     echo "-A VoIP_RULES  -d xx.xx.xx.xx   -m state --state NEW  -j ACCEPT "
            #
            # ================ Table 'filter', rule set VPN_RULES
    

    It appears this is going to continue for awhile …

    Switching back to NOT using iptables-restores, all's OK again.

     
  • Vadim Kurland
    Vadim Kurland
    2012-09-01

    check what do you have in the "log prefix" field in the rule options (double click in the column "Options" in the rule to open dialog). If its blank, then check what do you have in the "log prefix" in the firewall settings dialog, tab "Logging"

     
  • pgnd
    pgnd
    2012-09-01

    > check what do you have in the "log prefix" field in the rule options (double click in the column "Options"
    > in the rule to open dialog). If its blank,

    it is.

    > then check what do you have in the "log prefix" in the firewall settings dialog, tab "Logging"

    [P:%N] {%A}_
    

    Removing that, the error on attempted use of "iptables-restore" is unaffected …

     
  • Vadim Kurland
    Vadim Kurland
    2012-09-01

    you should change " {%A}_" to something that does not have any characters that unix shell considers "special". just change to "P:%N %A" or something like that. Then recompile your policy and check the same iptables command that causes the error, it should get log prefix without "{ } " and " ". Then the error should go away.

     
  • pgnd
    pgnd
    2012-09-02

    None of those characters have caused any problems in years of fwbuilder use without iptables-restore.  Without some form of visual delimiter, the logs are difficult to read.

    In any case, as I specifically mentioned above, neither removing the logging label completely, nor turning off logging, fixes the problem.

    It fails ONLY in the case of iptables-restore use.

     
  • Vadim Kurland
    Vadim Kurland
    2012-09-02

    if change to the log prefix does not eliminate the error, then it must be caused by another line in the iptables configuration. I wish iptables-restore was more verbose in its error reporting but I can not change that. All lines you have quoted above look normal and should not cause errors. May be you counted empty lines and comments when you counted lines trying to find line 240 ?

     
  • pgnd
    pgnd
    2012-09-02

    > May be you counted empty lines and comments when you counted lines trying to find line 240 ?

    Yes, I did, as you instructions were to count lines.  Did you mean only count non-blank, non-commented lines?

     
  • Vadim Kurland
    Vadim Kurland
    2012-09-02

    yes, count only lines that start with "echo" as these are the only ones that iptables-restore will ever see

     
  • pgnd
    pgnd
    2012-09-02

    counting lines in the output of

        grep "^[ ]*echo" fw_edgebox.fw
    

    line 240 contains

            echo ":TCP_FRAGS_NMAP_SCANS - [0:0]"
            echo "-A TCP_FRAGS_NMAP_SCANS -i +  -p tcp -m tcp   --tcp-flags SYN NONE  -j DROP "
            echo "-A TCP_FRAGS_NMAP_SCANS -i +  -p tcp -m tcp   --tcp-flags ACK,RST,SYN,FIN NONE  -j DROP "
            echo "-A TCP_FRAGS_NMAP_SCANS -i +  -p tcp -m tcp   --tcp-flags URG,ACK,RST,SYN,FIN NONE  -j DROP "
            echo "-A TCP_FRAGS_NMAP_SCANS -i +  -p tcp -m tcp   --tcp-flags ACK ACK  -j DROP "
    240     echo "-A TCP_FRAGS_NMAP_SCANS -i +  -p tcp -m tcp   --tcp-flags ACK,RST,SYN,FIN FIN  -j DROP "
            echo "-A TCP_FRAGS_NMAP_SCANS -i +  -p tcp -m tcp   --tcp-flags URG,ACK,RST,SYN,FIN URG  -j DROP "
            echo "-A TCP_FRAGS_NMAP_SCANS -i +  -p tcp -m tcp   --tcp-flags SYN,FIN SYN,FIN  -j DROP "
            echo "-A TCP_FRAGS_NMAP_SCANS -i +  -p tcp -m tcp   --tcp-flags ACK,RST,SYN,FIN SYN,FIN  -j DROP "
            echo "-A TCP_FRAGS_NMAP_SCANS -i +  -p tcp -m tcp   --tcp-flags RST,SYN RST,SYN  -j DROP "
            echo "-A TCP_FRAGS_NMAP_SCANS -i +  -p tcp -m tcp   --tcp-flags URG,PSH,FIN URG,PSH,FIN  -j DROP "
            echo "-A TCP_FRAGS_NMAP_SCANS -i +  -p tcp -m tcp   --tcp-flags ALL URG,PSH,FIN  -j DROP "
            echo "-A TCP_FRAGS_NMAP_SCANS -i +  -p tcp -m tcp   --tcp-flags URG,ACK,RST,SYN,FIN URG,ACK,RST,SYN,FIN  -j DROP "
            echo "-A TCP_FRAGS_NMAP_SCANS -i +  -p tcp -m tcp   --tcp-flags ALL URG,ACK,PSH,RST,SYN,FIN  -j DROP "
    

    So, for an experiment, I *DELETED* the main Policy rule and the Branch containing those rules, and Compiled/Installed again.

    This time

    Running prolog script
    iptables-restore: line 224 failed
    *** Fatal error :
    iptables-restore: line 224 failed
    Firewall policy installation failed
    Running epilog script
    

    where

    222     echo "-A Cid32862X5150.0  -d xx.xx.xx.xx   -j Cid32862X5150.1 "
    223     echo "-A Cid32862X5150.0  -d 10.11.11.100   -j Cid32862X5150.1 "
    224     echo "-A Cid32862X5150.1  -s xx.xx.xx.xx/29   -j ACCEPT "
    225     echo "-A Cid32862X5150.1  -s 66.111.4.51   -j ACCEPT "
    226     echo "-A Cid32862X5150.1  -s 66.111.4.52   -j ACCEPT "
    

    Any suggestions, other than not using iptables-restore?

     
  • Vadim Kurland
    Vadim Kurland
    2012-09-02

    so far, I have not seen anything that could cause an error in all the lines you have quoted. Perhaps you are still looking at the wrong lines?  You can do this: edit generated script and replace the line at the beginning

    IPTABLES_RESTORE="/sbin/iptables-restore"
    

    with

    IPTABLES_RESTORE="/usr/bin/tee file.fw"
    

    then run the script. Instead of sending iptables commands to iptables-restore, it will just print them to stdout and save them in the file named "file.fw".  This file will contain exactly the same set of iptables lines as the one normally sent to iptables-restore. It is going to be much easier to find the line, you just use command

    nl file.fw
    

    Also you can try to change something in the file and then feed it to iptables-restore:

    cat file.fw | iptables-restore
    

    This should help pinpoint the line and its part that causes the error

     
  • pgnd
    pgnd
    2012-09-02

    MUCH easier, thanks! if that's NOT a troubleshooting tip in the docs (I'm going to reread now), it should be.

    perhaps this'll be helpful …

    cat file.fw | iptables-restore
        iptables-restore: line 240 failed
       233  -A QOS_RULES -p 50  -j CLASSIFY --set-class 1:20
       234  -A QOS_RULES -p ah  -j CLASSIFY --set-class 1:20
       235  -A QOS_RULES -p tcp -m tcp  -m multiport  --dports 443,143,4143,993,993,110,995,873,25,465,525,587,465  -j CLASSIFY --set-class 1:30
       236  -A QOS_RULES  -j CLASSIFY --set-class 1:40
       237  -A PREROUTING  -j QOS_RULES
       238  -A POSTROUTING  -j QOS_RULES
       239  -A FORWARD  -j QOS_RULES
       240  COMMIT
       241  *nat
       242  :PREROUTING ACCEPT [0:0]
       243  :POSTROUTING ACCEPT [0:0]
       244  :OUTPUT ACCEPT [0:0]
    
     
  • Vadim Kurland
    Vadim Kurland
    2012-09-02

    so it fails on COMMIT …  Very strange because iptables commands generated in "iptables-restore format"  are exactly the same as those generated in the regular shell script format . Are you sure you are not getting any errors when you deploy script generated in regular format ? May be you just did not notice. If there are none, then I guess you need to try to eliminate rules one by one from the iptables-restore file to see which one causes it

     
  • pgnd
    pgnd
    2012-09-02

    > Are you sure you are not getting any errors when you deploy script generated in regular format ?
    > May be you just did not notice.

    When attempting the INSTALL phase of the iptables-restore-enable script, in the FWB UI I see the 'fail' highlighted in bright red text.  Hard to miss …
    There's no such highlighted, or other, "error" or "fail" warning on install of the NON-iptables-restore script.

    Also, checking on the maching to which the script is actually deployed, examining the output of 'sh -x fw_edgebox.fw' and the syslogs, again, NO instance of "error" or "fail".

    > If there are none, then I guess you need to try to eliminate rules one by one from the iptables-restore file to see which one causes it

    removing the "*mangle" stanza immediately prior to that COMMIT,

    diff -ur file.fw file.fw.WORKING
        --- file.fw     2012-09-01 22:02:42.167728373 -0700
        +++ file.fw.WORKING     2012-09-02 09:21:08.293582281 -0700
        @@ -222,22 +222,6 @@
         -A In__Policy_26  -j LOG  --log-prefix "[P:26] {DEF DENY}_"
         -A In__Policy_26  -j DROP
         COMMIT
        -*mangle
        -:QOS_RULES - [0:0]
        --A QOS_RULES  -s 66.193.176.0/23   -j CLASSIFY --set-class 1:10
        --A QOS_RULES  -s 204.11.192.0/24   -j CLASSIFY --set-class 1:10
        --A QOS_RULES  -d 66.193.176.0/23   -j CLASSIFY --set-class 1:10
        --A QOS_RULES  -d 204.11.192.0/24   -j CLASSIFY --set-class 1:10
        --A QOS_RULES -p tcp -m tcp  -m multiport  --dports 53,80,22,23  -j CLASSIFY --set-class 1:20
        --A QOS_RULES -p udp -m udp  -m multiport  --dports 1194,68,67,53  -j CLASSIFY --set-class 1:20
        --A QOS_RULES -p 50  -j CLASSIFY --set-class 1:20
        --A QOS_RULES -p ah  -j CLASSIFY --set-class 1:20
        --A QOS_RULES -p tcp -m tcp  -m multiport  --dports 443,143,4143,993,993,110,995,873,25,465,525,587,465  -j CLASSIFY --set-class 1:30
        --A QOS_RULES  -j CLASSIFY --set-class 1:40
        --A PREROUTING  -j QOS_RULES
        --A POSTROUTING  -j QOS_RULES
        --A FORWARD  -j QOS_RULES
        -COMMIT
         *nat
         :PREROUTING ACCEPT [0:0]
         :POSTROUTING ACCEPT [0:0]
    

    seems to to the trick

    > cat file.fw | iptables-restore
        iptables-restore: line 240 failed
    >
    > cat file.fw.WORKING | iptables-restore
    >
    

    I'll look one rule at a time if nothing seems obvious to you …

     
  • pgnd
    pgnd
    2012-09-02

    Removing any ONE line doesn't seem to make a difference.

    Removing ALL/only the following lines is what appears to 'cure' the "fail"

    diff -ur file.fw file.fw.WORKING
    --- file.fw     2012-09-01 22:02:42.167728373 -0700
    +++ file.fw.WORKING     2012-09-02 09:29:54.848190208 -0700
    @@ -224,16 +224,6 @@
     COMMIT
     *mangle
     :QOS_RULES - [0:0]
    --A QOS_RULES  -s 66.193.176.0/23   -j CLASSIFY --set-class 1:10
    --A QOS_RULES  -s 204.11.192.0/24   -j CLASSIFY --set-class 1:10
    --A QOS_RULES  -d 66.193.176.0/23   -j CLASSIFY --set-class 1:10
    --A QOS_RULES  -d 204.11.192.0/24   -j CLASSIFY --set-class 1:10
    --A QOS_RULES -p tcp -m tcp  -m multiport  --dports 53,80,22,23  -j CLASSIFY --set-class 1:20
    --A QOS_RULES -p udp -m udp  -m multiport  --dports 1194,68,67,53  -j CLASSIFY --set-class 1:20
    --A QOS_RULES -p 50  -j CLASSIFY --set-class 1:20
    --A QOS_RULES -p ah  -j CLASSIFY --set-class 1:20
    --A QOS_RULES -p tcp -m tcp  -m multiport  --dports 443,143,4143,993,993,110,995,873,25,465,525,587,465  -j CLASSIFY --set-class 1:30
    --A QOS_RULES  -j CLASSIFY --set-class 1:40
     -A PREROUTING  -j QOS_RULES
     -A POSTROUTING  -j QOS_RULES
     -A FORWARD  -j QOS_RULES
    
     
  • pgnd
    pgnd
    2012-09-02

    And, to verify … disabling those rules from @ the FWB UI enables the UI-driven 'iptable-restore format' install-to-remote to complete without error.

     
  • Vadim Kurland
    Vadim Kurland
    2012-09-02

    Try this command:

    iptables -j CLASSIFY --help
    

    see if you get the following at the end:

    CLASSIFY target options:
    --set-class MAJOR:MINOR    Set skb->priority value (always hexadecimal!)
    

    even though you did not get any error messages colored in red, you should still closely inspect the output that appears when you deploy configuration not in iptables-restore format to see if there were any warnings or errors. Coloring is done in the GUI by matching the output against few regular expressions, if the warning did not match, it would not be colored red.

     
  • pgnd
    pgnd
    2012-09-02

    > Try this command: iptables -j CLASSIFY -help see if you get the following at the end:
    > CLASSIFY target options: -set-class MAJOR:MINOR Set skb->priority value (always hexadecimal!)

    i do

    iptables -j CLASSIFY --help | tail -n 10
      --verbose     -v              verbose mode
      --line-numbers                print line numbers when listing
      --exact       -x              expand numbers (display exact values)
    [!] --fragment  -f              match second or further fragments only
      --modprobe=<command>          try to insert modules using this command
      --set-counters PKTS BYTES     set the counter during insert/append
    [!] --version   -V              print package version.
    CLASSIFY target options:
    --set-class MAJOR:MINOR    Set skb->priority value (always hexadecimal!)
    

    > even though you did not get any error messages colored in red, you should still closely inspect the output
    > that appears when you deploy configuration not in iptables-restore format to see if there were any
    > warnings or errors. Coloring is done in the GUI by matching the output against few regular expressions, if
    > the warning did not match, it would not be colored red.

    Looking more closely finds an "Invalid rule", rather than "error" or "fail",

        Summary:
        * Running as user : rbsf
        * Firewall name : fw_edgebox
        * Installer uses user name : root
        * Management address : edgebox
        * Platform : iptables
        * Host OS : linux24
        * Loading configuration from file /home/rbsf/fwb/rbsf.fwb
        Installation plan:
        Copy file: /home/rbsf/fwb/fw_edgebox.fw --> /usr/local/etc/fw/fw_edgebox.fw
        ...
        chmod +x /usr/local/etc/fw/fw_edgebox.fw;
        sh /usr/local/etc/fw/fw_edgebox.fw && echo 'Policy activated'
        Logged in
        --**--**--
        Activating firewall script generated Sun Sep  2 09:36:59 2012 by rbsf
        Running prolog script
        Rule _Policy 17 (global)
        Rule PING_MTU_Discovery 1 (global)
        ...
        Rule QOS_RULES 0 (global)
        Rule QOS_RULES 1 (global)
        Rule QOS_RULES 2 (global)
        Rule QOS_RULES 3 (global)
        Rule QOS_RULES 4 (global)
        Rule _Policy 17 (global)
    ->  iptables: Invalid argument. Run `dmesg' for more information.
        Rule IDS_RULES 0 (global)
        ...
        debug1: Exit status 0
        SSH session terminated, exit status: 0
        Firewall policy successfully installed
    

    dmesg doesn't tell me anything more, but Global 17 is, in fact, the

        17 Any Any Any Any Both Branch:QOS_Rules Any (options)
    

    @Policy Branching rule associated with the problematic rules.

    Not sure what's "invalid" about that Branching rule …

     
  • pgnd
    pgnd
    2012-09-02

    that's "invalid argument", not "invalid rule" …

     
  • Vadim Kurland
    Vadim Kurland
    2012-09-02

    probably the problem is that you placed your classification rules in the branch and fwbuilder generated commands in all built-in chains:

     -A PREROUTING  -j QOS_RULES
     -A POSTROUTING  -j QOS_RULES
     -A FORWARD  -j QOS_RULES
    

    to jump to the branch since it has no additional information in the branching rule to decide on the chain.

    But target CLASSIFY is only allowed in certain chains (probably POSTROUTING but the man page does not say which so this requires some research) so we get the error because control is passed to it from wrong chain. There is no option in fwbuilder to specify which chain to use in this case so I guess the solution is to avoid branching in this situation and place all classification rules in the main rule set

     
  • pgnd
    pgnd
    2012-09-02

    disabling the Branching rule, and moving the Branch's rules to the main policy, on install with NOT-iptable-restore, I do NOT get the "invalid argument" any longer, but I DO get:

    Rule _Policy 20 (global)
    Warning: never matched protocol: ah. use extension match instead.
    Rule _Policy 21 (global)
    

    Switching to "iptables-restore format", I get the same 'never matched' warning, but no more FAIL - i.e., the policy gets installed.