pegasus-exitcode(1)
===================
:doctype: manpage


Name
----
pegasus-exitcode - Checks the stdout/stderr files of a workflow job for any
indication that an error occurred in the job. This script is intended to be
invoked automatically by DAGMan as the POST script of a job.


Synopsis
--------
[verse]
*pegasus-exitcode* [*-h*][*-r* 'rv'][*-n*][*-s* 'msg'][*-f* 'msg'] 'job.out'


Description
-----------
*pegasus-exitcode* is a utility that examines the STDOUT of a job to
determine if the job failed, and renames the STDOUT and STDERR files
of a job to preserve them in case the job is retried.

Pegasus uses *pegasus-exitcode* as the DAGMan postscript for jobs
submitted via Globus GRAM. This tool exists as a workaround to a
known problem with Globus and Condor-G where the exitcodes of GRAM
jobs are not returned. This is a problem because Pegasus uses the
exitcode of a job to determine if the job failed or not.

In order to get around the exitcode problem, Pegasus can wrap
GRAM jobs with Kickstart, which records the exitcode of the job
in an XML invocation record, which it writes to the job's STDOUT.
The STDOUT is transferred from the execution host back to the submit
host when the job terminates. After the job terminates, DAGMan runs
the job's postscript, which Pegasus sets to be *pegasus-exitcode*.
*pegasus-exitcode* looks at the invocation record generated by
kickstart to see if the job succeeded or failed. If the invocation
record indicates a failure, then *pegasus-exitcode* returns a non-zero
result, which indicates to DAGMan that the job has failed. If the
invocation record indicates that the job succeeded, then *pegasus-exitcode*
returns 0, which tells DAGMan that the job succeeded.

In addition, clustered jobs executed with *pegasus-cluster* or
*pegasus-mpi-cluster* will have a `[cluster-summary]` record in their
STDOUT. *pegasus-exitcode* can examine these records to determine if
any of the tasks in the clustered job failed.

*pegasus-exitcode* performs several checks (some optional) to determine
whether a job failed or not. These checks include:

. Is the Condor exitcode non-zero? If so, then the job failed.
. Is STDOUT empty? If it is empty, then the job failed.
. Are there any failure messages in the STDOUT or STDERR? If so, the job
failed.
. Are all of the success messages in the STDOUT or STDERR? If not, then
the job failed.
. Does the `[cluster-summary]` record indicate that the job was successful.
If not, then the job failed.
. Are there any `<status>` tags with a non-zero value? If there are,
then the job failed. Note that, if this is a clustered job, there could
be multiple `<status>` tags, one for each task. If any of them are
non-zero, then the job failed.
. Is there at least one `<status>` tag with a zero value? There must
be at least one successful invocation or the job has failed.

In addition, *pegasus-exitcode* allows the caller to specify the exitcode
returned by Condor using the *--return* argument. This can be passed to
*pegasus-exitcode* in a DAGMan post script by using the `$RETURN` variable.
If this value is non-zero, then *pegasus-exitcode* returns a non-zero result
before performing any other checks. For GRAM jobs, the value of `$RETURN` will
always be 0 regardless of whether the job failed or not.

In addition to checking the success/failure of a job, *pegasus-exitcode*
also renames the STDOUT and STDERR files of the job so that if the job is
retried, the STDOUT and STDERR of the previous run are not lost. It does
this by appending a sequence number to the end of the files. For example,
if the STDOUT file is called "job.out", then the first time the job is run
*pegasus-exitcode* will rename the file "job.out.000". If the job is run again,
then *pegasus-exitcode* sees that "job.out.000" already exists and renames
the file "job.out.001". It will continue to rename the file by incrementing
the sequence number every time the job is executed.


Options
-------

*-h*::
*--help*::
Prints a usage summary with all the available command-line options.

*-r* 'rv'::
*--return* 'rv'::
Return value reported by DAGMan. This can be specified in the DAG
using the $RETURN variable. If this is non-zero, then *pegasus-exitcode*
immediately fails with a non-zero return value itself. If it is
zero, then just rotate the file and don't check for proper kickstart
output. This option can be used in cases where kickstart cannot be used
(such as pegasus-create-dir) to enable file rotation.

*-n*::
*--no-rename*::
Don't rename 'job.out' and 'job.err' to '.out.XXX' and '.err.XXX'.
This option is used primarily for testing.

*-f* 'msg'::
*--failure-message* 'msg'::
Failure message to find in job stdout/stderr. If this message exists in
the stdout/stderr of the job, then the job will be considered a failure
no matter what other output exists. If multiple failure messages are
provided, then none of them can exist in the output or the job is
considered a failure.

*-s* 'msg'::
*--success-message* 'msg'::
Success message to find in job stdout/stderr. If this message does not exist
in the stdout/stderr of the job, then the job will be considered a failure
no matter what other output exists. If multiple success messages are
provided, then they must all exist in the output or the job is considered
a failure.


Authors
-------
Gideon Juve `<juve@usc.edu>`

Pegasus Team <http://pegasus.isi.edu>

