#5571 cron job behaves differently when run manually

[Michael von Glasow]

The environment in which cron executes jobs differs from an interactive shell. Most importantly, $PATH may differ, as may some other environment variables.

When running a cron job using the “Run now” button in Webmin, the command appears to be launched in an environment similar to an interactive shell, which may differ from what cron would provide. This can cause cron jobs to behave differently during such a “dry run” than they would if actually executed by cron.

Some people may use the “Run now” button in order to test if a cron job works properly. However, with the current setup, the dry run may work fine but the actual cron job will unexpectedly fail, or vice versa. Happened to me and took me a good amount of research to figure out why.

In my opinion, it would be desirable to have “Run now” behave like the real thing. This would most reliably be ensured by actually running the command through cron, which could be achieved by scheduling a cron job for the near future (say, a few seconds from now).

A few things would need to be taken care of:

• deleting the cron job as soon as it runs
• redirecting stdout/stderr so output appears in the user’s browser

These could be solved by having cron call a wrapper script which:

• is called with the command line of the job to test as an argument
• upon execution, first removes its own call from cron (presumably this would not interrupt a running cron job) – or, instead of having the script do that, schedule a second cron job to delete the first
• and then calls the actual command, redirecting its stdout/stderr to wherever webmin can pick it up and send it back to the user’s browser, using the mechanism already in place

[Jamie Cameron]

Perhaps the difference is due to some environment variable that isn't the same when a job is run via Cron vs. manually. What's the job that you're running that isn't behaving consistently?

[Michael von Glasow]

The job is a bash script which, among others, runs ifconfig to see if the network interface is up. This works when run from an interactive session, as well as with “Run now”, but fails in cron because the ifconfig binary is not on the PATH.

And yes, the difference is definitely due to an environment variable. On my system, PATH is /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/games:/usr/games in a shell session but /usr/bin:/bin in a cron job.

This is known and documented behavior of cron: environment variables are taken from the user’s crontab, with hardcoded values as a fallback. /etc/profile and anything else your preferred shell may use have no effect in cron. With a default profile and crontab, cron’s environment will almost certainly be missing some of the things you are taking for granted when working in a shell session.

My point is not to discuss of that is good design for cron (if at all, that would be a feature request for cron), or how to provide the environment of your choice to a cronjob (which is also not necessarily a thing to fix in Webmin.

My point is: Because of the way cron and Webmin’s “run now” feature are implemented, script behavior differs between these two. This makes “Run now” unsuitable for a dry run of a cronjob (as test results are not a reliable indicator of live behavior), which is counterintuitive. The way to fix that would be to ensure “Run now” behaves the same way as cron, i.e. executes the script in a cron environment rather than in a shell environment.

[Ilia]

Michael, this is a good suggestion .. let's wait for Jamie's comment on this though.

However, if you need an immediate fix try sourcing .bashrc file inside the script that runs as cron, like:

#!/usr/bin/env bash
source /root/.bashrc

ifconfig ..

Also, simply using a full path to your commands maybe a better solution, to make sure that the command you really expect is going to be used.

[Jamie Cameron]

Right, the only way to fix this would be as you said to create a temporary cron job, and let cron run it. However, I think this would be difficult for users in practice, because cron only has minute-level accuracy, and so users might have to wait for a minute to get the output!

[Michael von Glasow]

If the UI advised users that it may take up to a minute for output to appear (plus, depending on the way it is redirected to the user’s browser, the time it takes for the job itself to complete), they would at least know that this is normal, not a malfunction.

So we would need to balance between two use cases:
* Run the job without even a minute of delay, at the expense of the environment being different from cron and the job potentially behaving differently
* Run the job in a true cron environment to get meaningful test results, at the expense of a delay up to one minute

In your opinion, which use case is the more frequent one? Or, how much of a delay would you deem acceptable?

[Jamie Cameron]

In practice, I've rarely seen issues where a cron job behaves differently when run in Webmin vs. on schedule. We already make sure the environment is as similar as possible, for example by removing all variables that might make the script think it is being run from a webserver.

[Michael von Glasow]

Case in point, I am seeing this on my system. If I type echo $PATH in bash, I get:

/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/games:/usr/games

Doing the same in a cron job (and redirecting output) gives me:

/usr/bin:/bin

I did not examine the exact PATH I would get when running from webmin. However, I ran a script that called ifconfig and a few other system commands, which on my system reside in /sbin. The script works when run from Webmin, as well as when running from a shell. When running from cron, it does not find ifconfig and a bunch of other things because /sbin is not on the path.

In conclusion, cron jobs running from webmin seem to get an environment that is closer to a shell session than what cron provides.

If you want to replicate the cron environment for scripts launched from webmin – this will also ensure reliable test results as I described above. The only drawback is that you’d be replicating the environment rather than using the real one – if you were to miss something, or the real thing changes before you can catch up, behavior would still differ. But maybe that would be a compromise between instant response and reliable test results. In essence, you would need to do the following:

• Start with a clean environment, i.e. all environment variables unset.
• Set environment variables as hardcoded in cron, e.g PATH=/usr/bin:/bin.
• Parse the crontab for any variable definitions, and set those as well.

The thing is that cron completely ignores any mechanisms that set the environment for a login shell, such as profile or .bashrc, and goes by its hardcoded values plus anything defined in the crontab. Just stripping a few things from a shell environment will still result in something quite different from what you get in cron.

[Jamie Cameron]

I think that getting $PATH from the crontab would be a good idea when commands are run manually - it seems like that would provide the testability you're looking for.

[Jamie Cameron]

Actually this should already happen! Which cron file do you have $PATH set in?

[Michael von Glasow]

I didn’t set any path at all, and would expect that to give me the hardcoded cron default . Instead, this seems to give me the same PATH I would get in a shell environment.

That would imply that, if a variable is not set in the crontab, it is still inherited from the shell. For reproducible tests, all environment variables should be unset first, then cron’s hardcoded defaults should be set, and only after that should any settings in crontab be applied.

The possibilities of a cron job are almost unlimited, and so is the effect of environment variables, even (and especially) those with names you’ve never heard or thought of. Any environment variable present in one case but not present (or different) in the other can make test results unusable, therefore the clean approach is to ensure no variable from the shell environment creeps into the execution environment for the cron job.

[Jamie Cameron]

Good point, I will have Webmin use the default cron path if one is not set.