Issues with cron jobs and task manager
Brought to you by:
tiochan
Menu
▾
▴
1 Attachments
#3 Issues with cron jobs and task manager
I'm having trouble getting the cron jobs to work right, so I'm unable to get any automatic source updates. I've tried this on three different systems (CentOS, Ubuntu, and Windows) with the same results. When I execute it via crontab or the command line I get this:
[root@sigvi]/var/www/html# /usr/bin/php -c /etc/php.ini -f /var/www/html/cron/launch_processes.php
Executing task: '10 Autovalidate alerts'
- return code: 99
Executing task: '20 Load Vulnerabilities'
- return code: 99
Executing task: '30 Check server vulnerabilities'
- return code: 99
Executing task: '40 Check repository Updates'
- return code: 99
Executing task: '50 Load product dictionaries'
- return code: 99
Executing task: '99 Reports'
- return code: 99
When I run it through the web UI it works, but will only load the NVD Recent and NVD Updates sources, even when other sources (Like NVD - 2013) are enabled. The only way to get other sources to load is to disable NVC Recent and NVD Updates. Also, when using the web UI they don't run according to the schedules set - they only run manually.
Do you know why this might be happening? I've spent a while troubleshooting it but can't figure out why it's not working.
Return code "99" means not executed.
Please, edit your <sigvi_home>/conf/app.conf.php file and set DEBUG to true (define("DEBUG",false); --> define("DEBUG",true);).
Also, edit your php.ini file (depending on OS, you might found 2 files, one for apache and other of command line), you must enable the short open tag parameter (short_open_tag = On)
Restart apache and try again,
Let me know if there is more info about the error.
Thanks for the quick response! I don't really notice much difference with
the output from the CLI other than an "Executed Ok." message. Here's the
partial output:
[root@sigvi]/var/www/html# php -f /var/www/html/cron/launch_processes.php
Ok.
- return code: 99
I've verified that PHP is using the correct ini and that short open tags
are enabled and have tried it on both test boxes. When I run each script
individually I don't get any error messages and they seem to work fine,
with the exception of Load Vulnerabilities not loading all sources, but
when I run the launch_processes script nothing runs.
I'll keep troubleshooting. If you have any recommendations, please let me
know. I could just run a cron job for each script individually if I need
to, although I'd still need to figure out why vulnerabilities aren't
loading from all sources.
Thanks again!
On Mon, Mar 17, 2014 at 1:43 AM, tiochan tiochan@users.sf.net wrote:
Related
Support Requests:
#3Ok, let's try to execute one of them. Obviously, a return code "99" and finally an Ok is not correct, so better examine one of them.
If you access via Web to the tasks manager (Configuration -> Task manager) you will be able to see the scripts which are executed.
At the right side of each you will see a blue wheel, which can be used to execute that script on-line. (If the script appears with an "?" means that can't find it).
Try:
1. Execute one script using the wheel
2. Execute the script via command line, e.g.: php -f <sigvi_home>/my_include/cron/autovalidate_alerts.php
Ok, I gave this a shot and everything seemed to work when executing a single script. When I ran the autovalidate script from the web it said it ran OK (see screenshot). When I ran it from the command line it didn't produce any output:
[root@sigvi]/# php -f /var/www/html/my_include/cron/autovalidate_alerts.php
[root@sigvi]/#
It seems like it's just the launch_processes.php script that is having issues. If I just set a cron job for each script, would that solve the issue? The "Periodicity" option in the web UI doesn't work, but I could set it via crontab.
Ok.
To ensure that the tasks executes correctly, try to execute the task "20 Load Vulnerabilities":
1. Go to vulnerabilities page (Inventory -> vulnerabilities)
2. Reset the search form (clear the date) and click the search button
3. Remember the number of vulnerabilities that you have actually loaded (you can see the number on the top-right side of the table
4. Go to sources (Configuration -> sources)
5. Check that you have enabled "NVD - Recents" and "NVD - Updates" (Use it = Yes)
6. Execute from command line "php -f <sigvi_home>/plugins/cron/load_vulnerabilities.php"
7. Check for errors on output and go back to points 1 and 2, to see if the number of vulnerabilities have been increased
I wait for results.
Good luck!
Alright, I tried this and it does update the vulnerabilties. The output is
below:
[root@sigvi]/# php -f /var/www/html/plugins/cron/load_vulnerabilities.php
Vulnerabilities loaded from source NVD - updates: found 456
vulnerabilities, 456 of them were loaded into dabatase.
Vulnerabilities loaded from source NVD - Recents: found 231
vulnerabilities, 0 of them were loaded into dabatase.
[root@sigvi]/#
The only problem is that I also have the NVD - 2013 feed enabled but it
doesn't update that or list it in the output there. Any idea why?
Thanks again for all of your help - I appreciate it!
On Tue, Mar 18, 2014 at 3:33 PM, tiochan tiochan@users.sf.net wrote:
Related
Support Requests:
#3Ok, scripts runs all right.
The problem with 2013 source... ¿?
I have added 2013 and 2014 sources (shown in the attached image), take care with which parser you set.
I recommend you to enable only the updates and recents, because the others are big files (NVD might aim sigvi as spammer). For each year, you only have to load them once.
Try executing them one by one, beginning from 2002 until 2014, via Web using the right button.
One question about SIGVI reports:
- Do you receive the SIGVI report about daily tasks?
- Do you have configured the server email service (e.g. try sending an email with "mailx")?
- Do you have enabled the notifications on your user preferences?
Regards.
Alright, I have some good news and bad news. The good news is I tried running the launch_processes.php script from the CLI this morning and it worked. The only problem was that the load_products script failed with the error code 255. I've attached screenshots of the report. Looking in the logs I see "Error reading dictionary CPE 1.0." and "Error reading dictionary CPE 2.1.". That apparently happens whether I run it from the CLI or web for the load_products script.
The bad news is that after the launch_processes.php script ran successfully, I tried it again a few minutes later and everything failed with return code 99. I can't get it to run successfully again. I did not change anything between when it worked and when it didn't, so I have no idea why it would work one minute and not the next. Strange?
Anyway, about your reports questions: the answer is yes to all of them. When the launch_processes.php ran correctly, I got a "Summary of Task Manager" email and a "SIGVI: Vulnerability alerts for your servers" email. I also get vulnerability report emails when I run the load_vulnerabilities.php script and it loads new vulnerabilities.
Also, I see your point about the yearly feeds. I can load them individually and will do that once for each.
Last edit: fivedozen 2014-03-19
I think that I have a clue about the origin of the problem.
That's right, all this together is very very strange. A program that now runs perfectly, and next doesn't do it correctly is not usual.
Testing my own SIGVI, and testing NVD sources, I have noticed that NVD might have a anti-attack method that only allows the same IP to download a single file for a period of time.
Try it yourself:
This is the URL where the product files are located:
http://static.nvd.nist.gov/feeds/xml/cpe/dictionary/
Try to download one, and then try with other. When I tried to download the second file it returned a 404 (not found) ¿?
This is a very important change, and I can understand why they did it... but may be they could let to download at least 3 files: recent, updates and CPE product dictionaries.
Anyway, this is as is, and the only we can do is to adapt.
Now I'm going to work (in my so-limited spare time) on adapting to this new way to work. But the load of the CPE dictionary is not essential for SIGVI, because in fact, the internal product dictionary is generated by the products listed into the vulnerabilities.
Do you agree with this?
You're right, there's something weird going on with the CPE dictionaries.
Sometimes they load, sometimes they don't. If they aren't essential for
SIGVI, then I guess it's not a big deal anyway. Two quick questions:
doesn't work? I can run them manually but if I set a schedule through the
web they don't run. It's not a big deal because I can add them to crontab
but I was just curious.
the RSS feeds under the sources page, but can't figure out how they are
loaded and where the info goes.
I know you're busy so if you don't have time to answer all of these
questions, that's fine. I appreciate your help so far and your work on this
project!
On Thu, Mar 20, 2014 at 2:25 AM, tiochan tiochan@users.sf.net wrote:
Related
Support Requests:
#3Don't worry, I like to do it, the problem is to find time to do it :-)
doesn't work? I can run them manually but if I set a schedule through the
web they don't run. It's not a big deal because I can add them to crontab
but I was just curious.
I guess that the problem might be on php.ini file. I for your posts I guess that your operative system is centos. In ubuntu, when you install the php5 module for apache and the php for command line (php5-cli), they use separated php.ini files.
I have noticed that recent versions of php5 doesn't support short tags by default ("<?"). I have been solving this for next versions, but may be you have a problem like this.
But, if you don't have inconvenient, lets do a few checks:
1.1. Which version of SIGVI do you have?
See at conf/app.conf.php file, search for APP_VERSION
e.g.:
define("APP_VERSION","SIGVI R2 Enterprise 2.1.0");
1.2. Do you have added a cron for SIGVI, as suggested in the INSTALL file?
(check the path for php binary
which php)e.g.: at /etc/crontab:
0,30 * * www-data /usr/bin/php -f /var/www/html/cron/launch_processes.php > /tmp/output-sigvi.txt 2>&1
Also you can check the output file for more clues.
1.3. As I see on your first post, the path for your SIGVI installation is /var/www/html
I think that this is correct, else the Web wouldn't work correctly, but just for the record:
From your posts, I guess that the root dir of your web server site is "/var/www/html". Check that your conf/app.conf.php file this:
HOME is set to "" (define("HOME","");)
the RSS feeds under the sources page, but can't figure out how they are
loaded and where the info goes.
RSS feeds are accessed on-access-time. There is not any process or task to load them and the data is displayed directly to the user. No data is loaded into the database.
Regards.
Yeah, my main install is on CentOS, but I have a test install on Ubuntu and
so I test everything on that one too, just in case. You're right about the
php.ini on Ubuntu, and I've changed that ini file (/etc/php5/cli/php.ini)
to make sure that short tags are on, but the tasks in the Web UI still
don't work. On CentOS I ran php --ini and it shows that it's using
/etc/php.ini, which is what I've been editing, so I guess there's only one
ini file.
For your questions:
1.1 I'm using the latest one: SIGVI R2 Enterprise 2.1.0
1.2 I do have a cron job for sigvi in crontab. It's "0 6 * * * apache
/usr/bin/php -f /var/www/html/cron/launch_processes.php >> /tmp/sigvi.txt
2>&1". The output always contains those "return code: 99" messages, so I've
switched to running load_vulnerabilities.php from crontab instead of
launch_processes.
1.3 You're correct, my install is in /var/www/html, which is the default
for centos. My HOME variable is define("HOME","");
Thanks for answering all those questions - I know time is short! Now I know
how the RSS feeds work too.
It's not that big of a deal if the tasks in the web ui don't work, I'll
just set cron jobs for the scripts - that seems to work.
On Thu, Mar 20, 2014 at 5:46 PM, tiochan tiochan@users.sf.net wrote:
Related
Support Requests:
#3Now I don't know what could be happening.
I have just installed on a Ubuntu Server 12.04 LTS, with the lastest versions and I had'nt any problem.
I'll try on a centos trying to have the same behavior.
As soon as I have tested it, I'll contact you again. Also, if meanwhile you find the problem, please let me know.
Thank you for helping me on debugging it.
Regards.
Now I have just discovered what is happening.
I have created a new VM with CentOS and installed sigvi on it.
I experienced the same problem that you reported at the beggining, the return code 99 and discovered which is the problem.
Oh my God, I'm really silly.
There is no problem, return code 99 means "NOT EXECUTED", and the only reason is that this is not the moment to execute it.
Each process has its own schedule defined at the SIGVI's "Task Manager". So if now is not the specified moment to execute it, the it will not be executed (and returns a 99).
But examining the source code, I remembered that you can force to do it passing a "force" in the command line:
php -f /var/www/html/sigvi/cron/launch_processes.php force
Please, check it.
I'm so sorry for having you reviewing the code and your system.
That did it! I just ran it with the force option and it ran just fine. I
was wondering if there was a relation between the scheduled time in the
task manager and launching the script and I guess that was the case. I
should have looked at the source more carefully!
I think now everything is working as I need it to. I'll schedule a cron job
for the launch_processes script and will be good to go. Thanks again for
your help with troubleshooting this, I owe you one!
On Mon, Mar 24, 2014 at 2:59 AM, tiochan tiochan@users.sf.net wrote:
Related
Support Requests:
#3Remember that the job must be executed half-hourly, because the tasks can be set until this periodicity:
0,30 * * www-data /usr/bin/php -f /var/www/html/cron/launch_processes.php > /tmp/output-sigvi.txt 2>&1
If you put this line, you can remove the others that you put additionally.
Thanks for help me to remember the deeps of sigvi, and thanks for using it.
I'm planning to develop the next release, and any ideas would be welcome.
Best regards.