How to fail a ShellCommandActivity on AWS Data pip

Question

I am using AWS Data Pipeline and specifically the ShellCommandActivity object. This object calls a python script which extracts data via ftp, etc. There exists a chance that the file may not exist or the file may be of the wrong type and the pipeline can no longer run. If the python script errors out, or fails, the ShellCommandActivity object still gets marked as Finished, when I"m trying to get it to Failure. I have tried, in python, doing sys.exit(500) on failure, but the object still gets marked as Finished. How can I fail a ShellCommandActivity from a python script?

Answer 1

You could try something along the lines of:

cd $INPUT1_STAGING_DIR
chmod +x #{myBashScript}
bash #{myBashScript}
EXIT_CODE=$?
mv $INPUT1_STAGING_DIR/output $OUTPUT1_STAGING_DIR
exit $EXIT_CODE

The data pipeline will probably use the exit code of the last command. I have not tested this though.