Daily scheduled job, but need to check if previous job has completed.

I want to schedule cron to run a script each night, but should there
have been a lot of activity during the day, the script may take more
than a day to complete. So I need to check if the previous days job
has finished before starting today's. It doesn't matter if the script
doesn't run on the odd night here and there - I know that on average
the system will manage to keep up.

A friend has suggested to use a "touch" file but what if the system
crashes or some one reboots it - the touch file will still be present
and prevent the script from being run again. The friend also
suggested using a the process id for the script, but I am unsure how
to do this . . .

Re: Daily scheduled job, but need to check if previous job has

On 30 Nov., 11:15, Eliot wrote:
> I want to schedule cron to run a script each night, but should there
> have been a lot of activity during the day, the script may take more
> than a day to complete. So I need to check if the previous days job
> has finished before starting today's. It doesn't matter if the script
> doesn't run on the odd night here and there - I know that on average
> the system will manage to keep up.
>
> A friend has suggested to use a "touch" file but what if the system
> crashes or some one reboots it - the touch file will still be present
> and prevent the script from being run again. The friend also
> suggested using a the process id for the script, but I am unsure how
> to do this . . .

How would the job get restarted in case of a reboot?

Maybe I would let the cronjob create a file with the processing date
and the processing state as his name

current=$( date %Y-%m-%d ).IN_PROGRESS
touch "$current"

which would be renamed just before completed

mv "$current" "${current%.IN_PROGRESS}.DONE"

On startup of the script (after the touch) you could remove existing
DONE files (or make a cleanup manually at times).

Typically I generalize such tasks to let a script catch up processing
since the last successful DONE time stamp for all days until now.

Using the process id $$ is helpful to let processes distinguish their
own environment entities in case that more of these programs can
simultaneously run. (As fas as you specified your task that doesn't
seem to be necessary.)

Janis

Re: Daily scheduled job, but need to check if previous job has completed.

"Eliot" schrieb im Newsbeitrag
news:02316825-1df5-4510-8992-6459021ffa44@a39g2000pre.google groups.com...
>I want to schedule cron to run a script each night, but should there
> have been a lot of activity during the day, the script may take more
> than a day to complete. So I need to check if the previous days job
> has finished before starting today's. It doesn't matter if the script
> doesn't run on the odd night here and there - I know that on average
> the system will manage to keep up.
>
> A friend has suggested to use a "touch" file but what if the system
> crashes or some one reboots it - the touch file will still be present
> and prevent the script from being run again. The friend also
> suggested using a the process id for the script, but I am unsure how
> to do this . . .
Something along the lines of:

#!/bin/ksh
pidfile=/var/lock/pidfile-for-my-cronjob
if [ -f $pidfile ] && kill -0 $(cat $pidfile) 2>/dev/null
then
# pidfile exists and a process with that pid is running
# you may want to log this somehow
exit 1
fi
# record your pid
echo $$ >$pidfile
# your processing starts here
....
# and ends here
rm -f $pidfile
exit 0

Bye, Jojo

Re: Daily scheduled job, but need to check if previous job has

I guess what I'm concerned about is that if the touch file
..IN_PROGRESS
is left behind due to some untidy shutdown then when cron kicks off
the script again, it exits because it thinks a previous run is still
in progress.

Re: Daily scheduled job, but need to check if previous job has

On 30 Nov., 11:59, Eliot wrote:

[ Please quote context! (re-inserted...)]

>> How would the job get restarted in case of a reboot?
>>
>> Maybe I would let the cronjob create a file with the processing date
>> and the processing state as his name
>>
>> current=$( date %Y-%m-%d ).IN_PROGRESS
>> touch "$current"
>>
>> which would be renamed just before completed
>>
>> mv "$current" "${current%.IN_PROGRESS}.DONE"
>>
>> On startup of the script (after the touch) you could remove existing
>> DONE files (or make a cleanup manually at times).

> I guess what I'm concerned about is that if the touch file
> .IN_PROGRESS
> is left behind due to some untidy shutdown then when cron kicks off
> the script again, it exits because it thinks a previous run is still
> in progress.

If the IN_PROGRESS file is present after the restart that means that
the prvious job has not completed, and should be re-initiated anyway
(as far as I understood your requirement).

Hmm.. - Now can two of your processes be active in parallel? Do you
want to prevent that?

Janis

Re: Daily scheduled job, but need to check if previous job has

On 30 Nov, 12:18, Janis wrote:
> On 30 Nov., 11:59, Eliot wrote:
>
> [ Please quote context! (re-inserted...)]
>
>
>
>
>
> >> How would the job get restarted in case of a reboot?
>
> >> Maybe I would let the cronjob create a file with the processing date
> >> and the processing state as his name
>
> >> current=$( date %Y-%m-%d ).IN_PROGRESS
> >> touch "$current"
>
> >> which would be renamed just before completed
>
> >> mv "$current" "${current%.IN_PROGRESS}.DONE"
>
> >> On startup of the script (after the touch) you could remove existing
> >> DONE files (or make a cleanup manually at times).
> > I guess what I'm concerned about is that if the touch file
> > .IN_PROGRESS
> > is left behind due to some untidy shutdown then when cron kicks off
> > the script again, it exits because it thinks a previous run is still
> > in progress.
>
> If the IN_PROGRESS file is present after the restart that means that
> the prvious job has not completed, and should be re-initiated anyway
> (as far as I understood your requirement).
>
> Hmm.. - Now can two of your processes be active in parallel? Do you
> want to prevent that?
>
> Janis- Hide quoted text -
>
> - Show quoted text -

Yes having two prccesses trying to do the same thing, running at the
same time is eactly what I'm trying to prevent.

Re: Daily scheduled job, but need to check if previous job has completed.

On 2007-11-30, Eliot wrote:
> I want to schedule cron to run a script each night, but should there
> have been a lot of activity during the day, the script may take more
> than a day to complete. So I need to check if the previous days job
> has finished before starting today's. It doesn't matter if the script
> doesn't run on the odd night here and there - I know that on average
> the system will manage to keep up.
>
> A friend has suggested to use a "touch" file but what if the system
> crashes or some one reboots it - the touch file will still be present
> and prevent the script from being run again. The friend also
> suggested using a the process id for the script, but I am unsure how
> to do this . . .

The 'touched' marker file is a good and simple idea.

Should the system reboot because of a crash or a manual intervention, just
clean up the marker file at reboot. Cron can do this for you:

@reboot rm /MyPath/marker_file


Alternatively, you can also protect yourself from the situation where only
your task has crashed. Which would mean the machine isn't rebooting, but
the marker was left by the crashed task.

Just use find in a regular cron job, to clean the marker file if it's more
than x days old. If your task can take a day, but never 2, just use:

find -daystart -mtime +2 /MyPath/ -name "marker_file" -exec rm '{}' ';'


PS: If you task is a script, you could begin it with this cleanup.

--
There is an art, it says, or rather, a knack to flying.
The knack lies in learning how to throw yourself at the ground and miss.
Douglas Adams

Re: Daily scheduled job, but need to check if previous job has

On Nov 30, 4:15 am, Eliot wrote:
> I want to schedule cron to run a script each night, but should there
> have been a lot of activity during the day, the script may take more
> than a day to complete. So I need to check if the previous days job
> has finished before starting today's. It doesn't matter if the script
> doesn't run on the odd night here and there - I know that on average
> the system will manage to keep up.
>
> A friend has suggested to use a "touch" file but what if the system
> crashes or some one reboots it - the touch file will still be present
> and prevent the script from being run again. The friend also
> suggested using a the process id for the script, but I am unsure how
> to do this . . .

Could you just run a ps and see the process is running? If it's in
cron I'm assuming it the same script each night. I use something like
this to make sure the same script doesn't run twice against an oracle
server:
PROGRAM_NAME=$0
############################################################ #######################
# Is this job already running?
############################################################ #######################
if [[ $(ps -ef | egrep -v "run_job|timeout|grep|$$" | grep
"$PROGRAM_NAME" | grep -c $INSTANCE) -gt 0 ]]
then
print_status_message "\nERROR: it looks like $PROGRAM_NAME is
already running against $INSTANCE." ERROR

RETCODE=102
error_exit
fi

Re: Daily scheduled job, but need to check if previous job has

Thanks for all the help!

[The solution with] Kill -0 posted by Joachim Schmitz solved my
problem, incidentally kill -0 wasn't in the man page or help on my
box!

Re: Daily scheduled job, but need to check if previous job has completed.

"Eliot" schrieb im Newsbeitrag
news:556b4edd-0f4e-4e7f-aa78-9cd5f5dd314d@n20g2000hsh.google groups.com...
> Thanks for all the help!
>
> [The solution with] Kill -0 posted by Joachim Schmitz solved my
> problem, incidentally kill -0 wasn't in the man page or help on my
> box!
it may not be documented in kill(1), but should be in kill(2):
if the signal parameter has the value 0 (zero, the null signal), error
checking is performed but no signal is sent. The null signal can be used to
check the validity of the pid parameter

Bye, Jojo

Re: Daily scheduled job, but need to check if previous job has completed.

Joachim Schmitz wrote:

> #!/bin/ksh
> pidfile=/var/lock/pidfile-for-my-cronjob
> if [ -f $pidfile ] && kill -0 $(cat $pidfile) 2>/dev/null
> then
> # pidfile exists and a process with that pid is running
> # you may want to log this somehow
> exit 1
> fi
> # record your pid
> echo $$ >$pidfile
> # your processing starts here
> ...
> # and ends here
> rm -f $pidfile
> exit 0

Nice! Saved for future reference. I didn't know about pid -0, I probably
would have done a grep on $(ps -a).

Grep and ps might actually be better because after the reboot there may be
some other process with the same pid, blocking a re-run of your script.
With ps you could check both invication name and pid of the program.

robert

Re: Daily scheduled job, but need to check if previous job has completed.

Eliot wrote:
> Thanks for all the help!
>
> [The solution with] Kill -0 posted by Joachim Schmitz solved my
> problem, incidentally kill -0 wasn't in the man page or help on my
> box!

Let me say again that "kill -0", after a reboot, may detect a new, unrelated
process that just happens to have the same ID that your script had before
the reboot. If that new process is a permanently-running service, it will
effectively block your script from ever being re-run.

I prefer a grep on ps's output because there you can check against the
(command-line) name of the process in question.

I wrote something to that effect about a week ago but forgot to post it, so
I posted it just now.

robert

Re: Daily scheduled job, but need to check if previous job has completed.

"Robert Latest" schrieb im Newsbeitrag
news:5rqn3rF166fnuU1@mid.dfncis.de...
> Joachim Schmitz wrote:
>
>> #!/bin/ksh
>> pidfile=/var/lock/pidfile-for-my-cronjob
>> if [ -f $pidfile ] && kill -0 $(cat $pidfile) 2>/dev/null
>> then
>> # pidfile exists and a process with that pid is running
>> # you may want to log this somehow
>> exit 1
>> fi
>> # record your pid
>> echo $$ >$pidfile
>> # your processing starts here
>> ...
>> # and ends here
>> rm -f $pidfile
>> exit 0
>
> Nice! Saved for future reference. I didn't know about pid -0, I probably
> would have done a grep on $(ps -a).
signal 0 not pid 0. The latter exists too and has a different meaning.

> Grep and ps might actually be better because after the reboot there may be
> some other process with the same pid, blocking a re-run of your script.
That's why I mentioned "you may want to log this somehow".

> With ps you could check both invocation name and pid of the program.
ps -p $(cat pidfile)
might help here

Bye, Jojo

Re: Daily scheduled job, but need to check if previous job has completed.

Joachim Schmitz wrote:

> signal 0 not pid 0. The latter exists too and has a different meaning.

Sorry, of course I meant kill -0.

robert