Best method of triggering a shell script from Java

Question

I have a shell script which I'd like to trigger from a J2EE web app.

The script does lots of things - processing, FTPing, etc - it's a legacy thing.

It takes a long time to run.

I'm wondering what is the best approach to this. I want a user to be able to click on a link, trigger the script, and display a message to the user saying that the script has started. I'd like the HTTP request/response cycle to be instantaneous, irrespective of the fact that my script takes a long time to run.

I can think of three options:

• Spawn a new thread during the processing of the user's click. However, I don't think this is compliant with the J2EE spec.
• Send some output down the HTTP response stream and commit it before triggering the script. This gives the illusion that the HTTP request/response cycle has finished, but actually the thread processing the request is still sat there waiting for the shell script to finish. So I've basically hijacked the containers HTTP processing thread for my own purpose.
• Create a wrapper script which starts my main script in the background. This would let the request/response cycle to finish normally in the container.

All the above would be using a servlet and Runtime.getRuntime().exec().

This is running on Solaris using Oracle's OC4J app server, on Java 1.4.2.

Please does anyone have any opinions on which is the least hacky solution and why?

Or does anyone have a better approach? We've got Quartz available, but we don't want to have to reimplement the shell script as a Java process.

Thanks.

Answer 1

I'd go with option 3, especially if you don't actually need to know when the script finishes (or have some other way of finding out other than waiting for the process to end).

Option 1 wastes a thread that's just going to be sitting around waiting for the script to finish. Option 2 seems like a bad idea. I wouldn't hijack servlet container threads.

Answer 2

My approach to this would probably be something like the following:

• Set up an ExecutorService within the servlet to perform the actual execution.
• Create an implementation of Callable with an appropriate return type, that wraps the actual script execution (using Runtime.exec()) to translate Java input variables to shell script arguments, and the script output to an appropriate Java object.
• When a request comes in, create an appropriate Callable object, submit it to the executor service and put the resulting Future somewhere persistent (e.g. user's session, or UID-keyed map returning the key to the user for later lookups, depending on requirements). Then immediately send an HTTP response to the user implying that the script was started OK (including the lookup key if required).
• Add some mechanism for the user to poll the progress of their task, returning either a "still running" response, a "failed" response or a "succeeded + result" response depending on the state of the Future that you just looked up.

It's a bit handwavy but depending on how your webapp is structured you can probably fit these general components in somewhere.

Answer 3

The most robust solution for asynchronous backend processes is using a message queue IMO. Recently I implemented this using a Spring-embedded ActiveMQ broker, and rigging up a producing and consuming bean. When a job needs to be started, my code calls the producer which puts a message on the queue. The consumer is subscribed to the queue and get kicked into action by the message in a separate thread. This approach neatly separates the UI from the queueing mechanism (via the producer), and from the asynchronous process (handled by the consumer).

Note this was a Java 5, Spring-configured environment running on a Tomcat server on developer machines, and deployed to Weblogic on the test/production machines.

Answer 4

If your HTTP response / the user does not need to see the output of the script, or be aware of when the script completes, then your best option is to launch the thread in some sort of wrapper script as you mention so that it can run outside of the servlet container environment as a whole. This means you can absolve yourself from needing to manage threads within the container, or hijacking a thread as you mention, etc.

Only if the user needs to be informed of when the script completes and/or monitor the script's output would I consider options 1 or 2.

Answer 5

Your problem stems from the fact that you are trying to go against the 'single response per request' model in J2EE, and have the end-user's page dynamically update as the backend task executes.

Unless you want to go down the introducing an Ajax-based solution, you will have to force the rendered page on the user's browser to 'poll' the server for information periodically, until the back-end task completes.

This can be achieved by:

1. When the J2EE container receives the request, spawn a thread which takes a reference to the session object (which will be used to write the output of your script)

2. Initialize the response servlet to write an html page which will contain a Javascript function to reload the page from the server at regular intervals (every 10 seconds or so).

3. On each request, poll the session object to display the output stored by the spawned thread in step 1

4. [clean-up logic can be added to delete the stored content from the session once the thread completes if needed, also you can set any additional flags in the session for mark state transitions of the execution of your script]

This is one way to achieve what you want - it isn't the most elegant of all approaches, but it is essentially due to needing to asynchronously update your page content from the server , with a request/response model.

There are other ways to achieve this, but it really depends on how inflexible your constraints are. I have heard of Direct Web Remoting (although I haven't played with it yet), might be worth taking a look at Developing Applications using Reverse-Ajax

Answer 6

For the second option, you can use a servlet, and after you've responded to the HTTP request, you can use java.lang.Runtime.exec() to execute your script. I'd also recommend that you look here : http://www.javaworld.com/javaworld/jw-12-2000/jw-1229-traps.html

... for some of the problems and pitfalls of using it.