Debian: ERROR: Error opening (null) as binary

Help
2012-07-21
2012-11-11
  • Graham Knight
    Graham Knight
    2012-07-21

    I have installed pglcmd and pgld on my debian (squeeze) system, and setup my
    /etc/pgl/blocklists.list with an appropriate list. However, whenever I try to
    load pgl (with "pglcmd start") it fails to start pgld. when I check pgld.log
    I see:

    /var/log/pgl/pgld.log

    Jul 21 21:13:12 INFO: Connected to dbus system bus.
    Jul 21 21:13:12 INFO: Started.
    Jul 21 21:13:12 ERROR: Error loading /var/lib/pgl/master_blocklist.p2p
    Jul 21 21:13:12 INFO: Blocking 0 IP ranges (0 IPs).
    Jul 21 21:13:12 ERROR: Cannot load the blocklist(s)
    

    When I check /var/lib/pgl/master_blocklist.p2p it is present, but empty (a 0
    byte file)

    When I check pglcmd.log I see that the various blocklists are successfully
    extracted, followed by a long list of names/ip addresses to block. Despite
    this, After the final blocklist is read in. I get the error and info:

    /var/log/pgl/pglcmd.log

    ERROR: Error opening (null) as binary.
    INFO: Blocking 0 IP ranges (0 IPs).
    

    Suggesting that pgl isn't storing the list of ips it has just successfully
    extracted.

    Having a quick poke in the source code. I can see that the above error only
    appears to be called (in parser.c) if pgld believes that it is trying to
    read something in from STDIN, rather than from a file I presume. That's about
    as far as my digging got me. Anyone seen this before and have a solution to
    it, or have any thoughts on how to debug further?

    Any help gratefully received.

    Cheers,

    Graysters

     
  • jre-phoenix
    jre-phoenix
    2012-07-21

    After a quick look at the code (so I might be wrong), the error message is
    in loading binary blocklist code.

    I guess some line in the blocklist causes pgld to think it is a binary p2b
    blocklist instead of probably a p2p v2 (text) blocklist.
    In case it is a temporarily problem I'd suggest to do a "pglcmd force-reload"
    - this deletes all already downloaded blocklists and gets new ones.

    If the problem persists I'd like to know which blocklist(s) you use.

     
  • Graham Knight
    Graham Knight
    2012-07-22

    Hi There,

    Tried a pglcmd force-reload, however as pgld quits straight after starting
    (see error above in pgld.log), the reload fails as pgld isn't running (as
    does a pglcmd update, or a manual pglcmd stop/start cycle).

    It doesn't seem to be related to any individual blocklist. If I try removing
    all but one list, say http://list.iblocklist.com/lists/tbg/hijacked

    then it still fails (I've tried a few other individual lists with the same
    error).

    After a quick look at the code (so I might be wrong), the error message is
    in loading binary blocklist code.

    From looking at my pglcmd.log I can see that all the file lists are extracted
    and parsed correctly, as I can see a long list of IP ranges to be blocked. the
    error always occurs after immediately after the last blocklist is read in.

    From further sourcecode poking, the only way I can see that this the code
    would cause the seen behaviour (i.e. successfully parsing the lists, then
    failing with the given error is if load_all_lists() (in pgld.c) gets
    called more than once, the first time successfully reading everything in, the
    second time being given a blockfile_count of 0. i.e. if the code took this
    path on the second calling of load_all_lists() (starting with
    load_all_lists being called with blockfile_count set to 0, or undefined):

    pgld.c: load_all_lists(): blockfile_count==0, so it hits the 'else'
    therefore… (//assume stdin for list) load_list(NULL,NULL);
    parser.c: load_list (NULL,NULL), will try reading as ascii, and fail (as
    it's passed NULL), it'll then try binary, calling loadlist_binary(NULL)
    parser.c: loadlist_binary(NULL), will try and open the file and immediately
    fail with (you guessed it):

    ERROR: Error opening (null) as binary.
    

    So, does load_all_lists get called more than once? and could the second
    calling lead to it having 0 as the blockfile_count, or am I barking up the
    wrong tree?

    Cheers,

    Graysters

     
  • Graham Knight
    Graham Knight
    2012-07-22

    Here's my (curtailed) pglcmd.log demonstrating that it is correctly parsing
    the one blocklist that I specified, then getting the error, and continuing
    with 0 entries in the blocklist):

    2012-07-22 10:00:51 BST Begin: pglcmd start
    Building blocklist ...
    Removing the following lines from tbg_hijacked:
    # List distributed by iblocklist.com
    Hijacked:103.10.188.0-103.10.191.255
    Hijacked:103.10.68.0-103.10.71.255
    ...
    ...
    Hijacked:95.64.37.0-95.64.39.255
    Hijacked:95.64.98.0-95.64.99.255
    ERROR: Error opening (null) as binary.
    INFO: Blocking 0 IP ranges (0 IPs).
    Blocklist built.
    Inserting iptables ...
    ..Setting up iptables for INPUT:
    ....Creating chain and inserting NFQUEUE rule.
    ....Whitelisting ports.
    ....Whitelisting IPs.
    ....Inserting block rule.
    ..Setting up iptables for OUTPUT:
    ....Creating chain and inserting NFQUEUE rule.
    ....Whitelisting ports.
    ....Whitelisting IPs.
    ....Inserting block rule.
    ..Setting up iptables for FORWARD:
    ....Creating chain and inserting NFQUEUE rule.
    ....Inserting block rule.
    ..Allowing loopback traffic: INPUT OUTPUT.
    ..Allowing OUTPUT traffic to DNS server <my DNS server>
    ..Allowing FORWARD traffic to DNS server <my DNS server>
    ..Allowing LAN traffic ...
    ....INPUT from <my LAN IP range>
    ....OUTPUT to <my LAN IP range>.
    ....FORWARD from <my LAN IP range> to <my LAN IP range>.
    ..LAN traffic allowed.
    ..Activating chains:
    ....INPUT
    ....OUTPUT
    ....FORWARD
    ..Chains activated.
    Iptables inserted.
    Starting pgld.
    Starting pglcmd.wd.
    

    as previously mentioned pgld then promply quits with:

    Jul 22 11:06:05 INFO: Connected to dbus system bus.
    Jul 22 11:06:05 INFO: Started.
    Jul 22 11:06:05 ERROR: Error loading /var/lib/pgl/master_blocklist.p2p
    Jul 22 11:06:05 INFO: Blocking 0 IP ranges (0 IPs).
    Jul 22 11:06:05 ERROR: Cannot load the blocklist(s)
    
     
  • jre-phoenix
    jre-phoenix
    2012-07-22

    I've already been irritated by you mentioning "many IP ranges" in pglcmd.log.
    And now I've seen this in your log:

    Removing the following lines from tbg_hijacked:
    

    so I guess you've added something to IP_REMOVE that removes all IP ranges.
    Verify your settings with "pglcmd show_config". Change it in
    /etc/pgl/pglcmd.conf

    General explanation:
    pglcmd downloads and extracts the blocklists specified in blocklist.list
    Then it does a (I guess you understand the following pseudo code):

    cat ALL BLOCKLISTS |
      some commands to get rid of problematic characters |
      grep IP_REMOVE regular expressions |
      pgld -m (=merge) > /var/lib/pgl/master_blocklist.p2p
    

    Specifically of interest for you in pglcmd.lib:

    SINGLE_REMOVE="$(echo $IP_REMOVE | sed "s/;/|/g")"
    [...]
    cat --squeeze-blank $BLOCKLISTSCAT |
                # Ignore comment lines
                grep -Ev "^[[:space:]]*#" |
                # Remove lines from IP_REMOVE (case-insensitive)
                grep -Eiv "$SINGLE_REMOVE" |
                # Replace repeated whitespace with a single occurence
                tr -s [:space:] |
                # Delete everything complimentary to the following characters/regex
                # alphanumeric
                # space
                # :\*.-,[]
                # newline (return gets removed, so CR+LF gets LF)
                tr -cd '[:alnum:][:space:]:\\\*\.\-,[]\n' |
                # Only keep the last occurence of :
                sed 's/.*:/&|/' | tr -d : | tr "\|" ":" |
                # Premerge the blocklist
                $DAEMON -m > $MASTER_BLOCKLIST
    
     
  • Graham Knight
    Graham Knight
    2012-07-22

    I've already been irritated by you mentioning "many IP ranges" in pglcmd.log

    Then accept my apologies. I had no intention to irritate. Just to get to the
    bottom of the error message and where I was going wrong.

    so I guess you've added something to IP_REMOVE that removes all IP ranges.

    You've got it!

    Interestingly something that appeared to work ok with the old
    moblock/blockcontrol config doesn't here.
    The issue was that I had a trailing semicolon at the end of my whitelist.
    I.e.:

    IP_REMOVE="safename1;safename2;safename3;"
    

    Changing it to

    IP_REMOVE="safename1;safename2;safename3"
    

    Fixed the problem for me.

    Guess I was too focused on the error "ERROR: Error opening (null) as
    binary." to spot that the long list of IP's was what was being removed from
    the list, not included. (Yes, I guess the "Removing the following lines
    from…" line should have been a bit of a giveaway!)

    Anyway, thanks for taking the time to explain how this works, and hopefully
    this will serve as a solution to anyone else that hits this in the future.

    Cheers,

    Graysters

     
  • jre-phoenix
    jre-phoenix
    2012-07-22

    Then accept my apologies. I had no intention to irritate.

    Sorry, I'm no native speaker. I mixed up irritate and wonder about (in german
    irritiert).

    I just pushed a fix to the git repository to prevent the IP_REMOVE problem.

    Greets