webadmin-devel

Re: [webmin-devel] &backquote_logged

[Jamie C. <jca...@we...>]

Larry Gilson <Li...@IS...> wrote ..
> Hi,
> 
> I hope I am posting this question to the correct list!  I have been trying
> to work on a couple of problems I am having with the squidguard (third
> party) module.  I have resolved all but one problem.  Creating the blacklist
> db files.  Below is the subroutine from squidguard-lib.pl which is called
> from enable_blacklist.cgi to create a new db file from the blacklist. 
> For
> some reason I just can't create the db file if executed from the enable
> CGI
> script via miniserv.  However, if I simulate the subroutine via a Perl
> script from the command line, it works.  You will be able to see that I
> have
> echoed the command to a text file just to make sure variables are being
> passed correctly, which they are.
> 
> #----------------------------
> # squidguard-lib.pl
> # rebuild_db($file)
> # Rebuild the dbfile $file with squidguard -C
> sub rebuild_db {
>   my $bin = &get_binary_path();
>   &terror('lib_nobin') if (! -x $bin);
> 
>   if (-e $_[0]) {
>     `echo "$bin -C $_[0]" > /var/lib/squidguard/blacklists/ads/xxx.txt`;
>     &backquote_logged("$bin -C $_[0] 2>&1 </dev/null");
>     &webmin_log('rebuild', 'dbfile', $_[0]);
>     &sgchown($_[0]);
>     &sgchown("$_[0].db");
>   }
> 
>   if (! -e $_[0]) {
>     `echo "$bin -C $_[0]" > /var/lib/squidguard/blacklists/ads/yyy.txt`;
>   }
> 
>   &reload_squid();
> }
> #----------------------------
> 
> I am just getting burnt-out on this one.  Anyone have any ideas why I can
> make this work via the Perl script/command line but not via CGI/miniserv?

This may be a path problem - make sure that whatever is in $bin is a full
path like '/usr/local/bin/foo' instead of just 'foo', which may not be in
webmin's program search path.

 - Jamie

RE: [webmin-devel] &backquote_logged

[Larry G. <Li...@IS...>]

Jamie,

Thanks for the reply!

> This may be a path problem - make sure that whatever is in 
> $bin is a full path like '/usr/local/bin/foo' instead of just 'foo',
> which may not be in webmin's program search path.
> 
>  - Jamie


&backquote_logged("$bin -C $_[0] 2>&1 </dev/null");

Both $bin and $_[0] are absolute paths.  I can cat the file that
the above command was echoed into, copy and paste at the command
line, execute, and obtain a successful db file.  The result is:

/usr/bin/squidGuard -C /var/lib/squidguard/blacklists/ads/domains
/usr/bin/squidGuard -C /var/lib/squidguard/blacklists/ads/urls

Just for kicks, I modified the &get_binary_path() so the binary is
found rather than the symlink.  Still nothing.  Permissions seem to
be correct, execute is allowed.  I just don't get it.  It should work.
Any other suggestions?

Thanks,
Larry





=========================================
Previous post --


> I hope I am posting this question to the correct list!  I have been 
> trying to work on a couple of problems I am having with the squidguard
> (third party) module.  I have resolved all but one problem.  Creating
> the blacklist db files.  Below is the subroutine from squidguard-lib.pl
> which is called from enable_blacklist.cgi to create a new db file from
> the blacklist.  For some reason I just can't create the db file if
> executed from the enable CGI script via miniserv.  However, if I
> simulate the subroutine via a Perl script from the command line, it
> works.  You will be able to see that I have echoed the command to a
> text file just to make sure variables are being passed correctly,
> which they are.
> 
> #----------------------------
> # squidguard-lib.pl
> # rebuild_db($file)
> # Rebuild the dbfile $file with squidguard -C
> sub rebuild_db {
>   my $bin = &get_binary_path();
>   &terror('lib_nobin') if (! -x $bin);
> 
>   if (-e $_[0]) {
>     `echo "$bin -C $_[0]" > /var/lib/squidguard/blacklists/ads/xxx.txt`;
>     &backquote_logged("$bin -C $_[0] 2>&1 </dev/null");
>     &webmin_log('rebuild', 'dbfile', $_[0]);
>     &sgchown($_[0]);
>     &sgchown("$_[0].db");
>   }
> 
>   if (! -e $_[0]) {
>     `echo "$bin -C $_[0]" > /var/lib/squidguard/blacklists/ads/yyy.txt`;
>   }
> 
>   &reload_squid();
> }
> #----------------------------

RE: [webmin-devel] &backquote_logged

[Jamie C. <jca...@we...>]

You should display the return value from backquote_logged, which is the output
from the program - it may contain some error message explaining what went
wrong. Also, check the exit status in the $? variable, which if non-zero 
indicates failure.

 - Jamie

Larry Gilson <Li...@IS...> wrote ..
> Jamie,
> 
> Thanks for the reply!
> 
> > This may be a path problem - make sure that whatever is in 
> > $bin is a full path like '/usr/local/bin/foo' instead of just 'foo',
> > which may not be in webmin's program search path.
> > 
> >  - Jamie
> 
> 
> &backquote_logged("$bin -C $_[0] 2>&1 </dev/null");
> 
> Both $bin and $_[0] are absolute paths.  I can cat the file that
> the above command was echoed into, copy and paste at the command
> line, execute, and obtain a successful db file.  The result is:
> 
> /usr/bin/squidGuard -C /var/lib/squidguard/blacklists/ads/domains
> /usr/bin/squidGuard -C /var/lib/squidguard/blacklists/ads/urls
> 
> Just for kicks, I modified the &get_binary_path() so the binary is
> found rather than the symlink.  Still nothing.  Permissions seem to
> be correct, execute is allowed.  I just don't get it.  It should work.
> Any other suggestions?
> 
> Thanks,
> Larry
> 
> 
> 
> 
> 
> =========================================
> Previous post --
> 
> 
> > I hope I am posting this question to the correct list!  I have been 
> > trying to work on a couple of problems I am having with the squidguard
> > (third party) module.  I have resolved all but one problem.  Creating
> > the blacklist db files.  Below is the subroutine from squidguard-lib.pl
> > which is called from enable_blacklist.cgi to create a new db file from
> > the blacklist.  For some reason I just can't create the db file if
> > executed from the enable CGI script via miniserv.  However, if I
> > simulate the subroutine via a Perl script from the command line, it
> > works.  You will be able to see that I have echoed the command to a
> > text file just to make sure variables are being passed correctly,
> > which they are.
> > 
> > #----------------------------
> > # squidguard-lib.pl
> > # rebuild_db($file)
> > # Rebuild the dbfile $file with squidguard -C
> > sub rebuild_db {
> >   my $bin = &get_binary_path();
> >   &terror('lib_nobin') if (! -x $bin);
> > 
> >   if (-e $_[0]) {
> >     `echo "$bin -C $_[0]" > /var/lib/squidguard/blacklists/ads/xxx.txt`;
> >     &backquote_logged("$bin -C $_[0] 2>&1 </dev/null");
> >     &webmin_log('rebuild', 'dbfile', $_[0]);
> >     &sgchown($_[0]);
> >     &sgchown("$_[0].db");
> >   }
> > 
> >   if (! -e $_[0]) {
> >     `echo "$bin -C $_[0]" > /var/lib/squidguard/blacklists/ads/yyy.txt`;
> >   }
> > 
> >   &reload_squid();
> > }
> > #----------------------------
> 
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by:  Etnus, makers of TotalView, The best
> thread debugger on the planet. Designed with thread debugging features
> you've never dreamed of, try TotalView 6 free at www.etnus.com.
> -
> Forwarded by the Webmin development list at web...@we...
> To remove yourself from this list, go to
> http://lists.sourceforge.net/lists/listinfo/webadmin-devel

RE: [webmin-devel] &backquote_logged

[Larry G. <Li...@IS...>]

Thanks Jamie,  I did try that too and there were no errors to report.  I
feel like you were probably on the right track suggesting the inclusion of a
full path . . .  Is it possible that the squidGuard binary requires an
environment variable that the miniserve configuration is not providing?  I
can only think, at this point, that the difference between a terminal
session and Webmin/miniserv is environment related.  Where do I find the
environment settings for miniserv?  Also, is it possible that if I need an
environment variable set that I can localize it just for this subroutine?

Thanks Jamie!

--Larry


> -----Original Message-----
> From: Jamie Cameron
>
> You should display the return value from backquote_logged, 
> which is the output from the program - it may contain some error
> message explaining what went wrong. Also, check the exit status in
> the $? variable, which if non-zero indicates failure.
> 
>  - Jamie
> 
> Larry Gilson <Li...@IS...> wrote ..
> > Jamie,
> > 
> > Thanks for the reply!
> > 
> > > This may be a path problem - make sure that whatever is in 
> > > $bin is a full path like '/usr/local/bin/foo' instead of 
> > > just 'foo', which may not be in webmin's program search path.
> > > 
> > >  - Jamie
> > 
> > 
> > &backquote_logged("$bin -C $_[0] 2>&1 </dev/null");
> > 
> > Both $bin and $_[0] are absolute paths.  I can cat the file that
> > the above command was echoed into, copy and paste at the command
> > line, execute, and obtain a successful db file.  The result is:
> > 
> > /usr/bin/squidGuard -C /var/lib/squidguard/blacklists/ads/domains
> > /usr/bin/squidGuard -C /var/lib/squidguard/blacklists/ads/urls
> > 
> > Just for kicks, I modified the &get_binary_path() so the binary is
> > found rather than the symlink.  Still nothing.  Permissions seem
> > to be correct, execute is allowed.  I just don't get it.  It
> > should work.  Any other suggestions?
> > 
> > Thanks,
> > Larry

Re: [webmin-devel] &backquote_logged

[Jamie C. <jca...@we...>]

An environment variable definately could be the cause..
You can see variables that are set by miniserv by looking at the env_ 
lines in /etc/webmin/miniserv.conf , and you may want to try adding 
some of those that you have set at the command line.

Also, I presume that when you run squidguard manually it is as root? 
The program might be failing when webmin runs it as root for some 
reason ..

  - Jamie

Larry Gilson wrote:
> Thanks Jamie,  I did try that too and there were no errors to report.  I
> feel like you were probably on the right track suggesting the inclusion of a
> full path . . .  Is it possible that the squidGuard binary requires an
> environment variable that the miniserve configuration is not providing?  I
> can only think, at this point, that the difference between a terminal
> session and Webmin/miniserv is environment related.  Where do I find the
> environment settings for miniserv?  Also, is it possible that if I need an
> environment variable set that I can localize it just for this subroutine?
> 
> Thanks Jamie!
> 
> --Larry
> 
> 
> 
>>-----Original Message-----
>>From: Jamie Cameron
>>
>>You should display the return value from backquote_logged, 
>>which is the output from the program - it may contain some error
>>message explaining what went wrong. Also, check the exit status in
>>the $? variable, which if non-zero indicates failure.
>>
>> - Jamie
>>
>>Larry Gilson <Li...@IS...> wrote ..
>>
>>>Jamie,
>>>
>>>Thanks for the reply!
>>>
>>>
>>>>This may be a path problem - make sure that whatever is in 
>>>>$bin is a full path like '/usr/local/bin/foo' instead of 
>>>>just 'foo', which may not be in webmin's program search path.
>>>>
>>>> - Jamie
>>>
>>>
>>>&backquote_logged("$bin -C $_[0] 2>&1 </dev/null");
>>>
>>>Both $bin and $_[0] are absolute paths.  I can cat the file that
>>>the above command was echoed into, copy and paste at the command
>>>line, execute, and obtain a successful db file.  The result is:
>>>
>>>/usr/bin/squidGuard -C /var/lib/squidguard/blacklists/ads/domains
>>>/usr/bin/squidGuard -C /var/lib/squidguard/blacklists/ads/urls
>>>
>>>Just for kicks, I modified the &get_binary_path() so the binary is
>>>found rather than the symlink.  Still nothing.  Permissions seem
>>>to be correct, execute is allowed.  I just don't get it.  It
>>>should work.  Any other suggestions?
>>>
>>>Thanks,
>>>Larry

RE: [webmin-devel] &backquote_logged

[Larry G. <Li...@IS...>]

I thought I would let you know that I figure this out finally.  I don't know
if you care or not since this is not your module but I always think it rude
not to follow-up.

At the command line, one can issue the squidGuard command to create a
database from a text file.  The text file does not need to be identified as
a destination group in the squidguard.conf file.  However, when issuing the
command via Webmin/miniserv, the entry/destination group needs to be in the
squidguard.conf file.  The squidGuard binary did not return any errors and
the only way I found this was by trial and error.  It is reproducable.  All
I had to do is move the &rebuild_db($filename) subroutine call below
&flush_file_lines() so the db creation request is performed after the conf
file is updated.

If you have any ideas why this might happen I would love to hear it.  I just
can not figure out why this works the way it does.  Then again, my brain is
thoroughly fried now anyway.

Thanks for your help on this and thanks for Webmin!

--Larry



> -----Original Message-----
> From: Jamie Cameron
> 
> You should display the return value from backquote_logged, 
> which is the output
> from the program - it may contain some error message 
> explaining what went
> wrong. Also, check the exit status in the $? variable, which 
> if non-zero 
> indicates failure.
> 
>  - Jamie
> 

Re: [webmin-devel] &backquote_logged

[Joe C. <jo...@sw...>]

Larry Gilson wrote:
> I thought I would let you know that I figure this out finally.  I don't know
> if you care or not since this is not your module but I always think it rude
> not to follow-up.
> 
> At the command line, one can issue the squidGuard command to create a
> database from a text file.  The text file does not need to be identified as
> a destination group in the squidguard.conf file.  However, when issuing the
> command via Webmin/miniserv, the entry/destination group needs to be in the
> squidguard.conf file.  The squidGuard binary did not return any errors and
> the only way I found this was by trial and error.  It is reproducable.  All
> I had to do is move the &rebuild_db($filename) subroutine call below
> &flush_file_lines() so the db creation request is performed after the conf
> file is updated.
> 
> If you have any ideas why this might happen I would love to hear it.  I just
> can not figure out why this works the way it does.  Then again, my brain is
> thoroughly fried now anyway.

Squidguard is, unfortunately, rather obtuse about error messages.  Trial 
and error is often the only way to figure out problems.  Another 
unfortunate thing is that Squidguard has been unmaintained for quite 
some time.

Squid's recent ACL changes (splay trees for the ACL lists) make it 
entirely feasible to do all of the 'blacklist' uses of SquidGuard within 
Squid proper at similar performance (also worth noting is that Squid 
also has time ACLs in addition to user and pretty much anything else--so 
the same kinds of complex access controls can be implemented too), so in 
the long run it is likely easier to enhance the Squid module to support 
those kinds of features more cleanly...
-- 
Joe Cooper <jo...@sw...>
Web caching appliances and support.
http://www.swelltech.com

RE: [webmin-devel] &backquote_logged

[Larry G. <Li...@IS...>]

> Squidguard is, unfortunately, rather obtuse about error 
> messages.  Trial and error is often the only way to figure out
> problems.  Another unfortunate thing is that Squidguard has been
> unmaintained for quite some time.

Makes using it difficult to use for some of my larger customers.  I would
like consistency among multiple customers for support reasons.  I have not
delved into the proxy/cache area that much as it is so difficult for most
customers to understand, like most infrastructure development.  But since
the deluge of spam, content filtering/blocking is being understood more.
Well, I would rather just buy one of your appliances than recreate the
wheel! :)


> Squid's recent ACL changes (splay trees for the ACL lists) make it 
> entirely feasible to do all of the 'blacklist' uses of 
> SquidGuard within Squid proper at similar performance (also worth
> noting is that Squid also has time ACLs in addition to user and
> pretty much anything else--so the same kinds of complex access
> controls can be implemented too), so in the long run it is likely
> easier to enhance the Squid module to support those kinds of
> features more cleanly...

Thanks for letting me know.  I am still fairly ignorant about Squid
configuration.  I would like to add blacklists like squidGuard and use the
time ACLs too.  I did not readily see how to do this via Webmin.  Making
Webmin a focal point for administration is what allows me to introduce
Linux/open source solutions into primarily MS shops.

Thanks,
Larry