Best practices for deploying Java webapps with min

Question

When deploying a large Java webapp (>100 MB .war) I'm currently use the following deployment process:

• The application .war file is expanded locally on the development machine.
• The expanded application is rsync:ed from the development machine to the live environment.
• The app server in the live environment is restarted after the rsync. This step is not strictly needed, but I've found that restarting the application server on deployment avoids "java.lang.OutOfMemoryError: PermGen space" due to frequent class loading.

Good things about this approach:

• The rsync minimizes the amount of data sent from the development machine to the live environment. Uploading the entire .war file takes over ten minutes, whereas an rsync takes a couple of seconds.

Bad things about this approach:

• While the rsync is running the application context is restarted since the files are updated. Ideally the restart should happen after the rsync is complete, not when it is still running.
• The app server restart causes roughly two minutes of downtime.

I'd like to find a deployment process with the following properties:

• Minimal downtime during deployment process.
• Minimal time spent uploading the data.
• If the deployment process is app server specific, then the app server must be open-source.

Question:

• Given the stated requirements, what is the optimal deployment process?

Answer 1

I'm not sure if this answers your question, but I'll just share on the deployment process I use or encounter in the few projects I did.

Similiar to you, I do not ever recall making a full war redeployment or update. Most of the time, my updates are restricted to a few jsp files, maybe a library, some class files. I am able to manage and determine which are the affected artifacts, and usually, we packaged those update in a zip file, along with an update script. I will run the update script. The script does the following:

• Backup the files that will be overwritten, maybe to a folder with today's date and time.
• Unpackage my files
• Stop the application server
• Move the files over
• Start the application server

If downtime is a concern, and they usually are, my projects are usually HA, even if they are not sharing state but using a router that provide sticky session routing.

Another thing that I am curious would be, why the need to rsync? You should able to know what are the required changes, by determining them on your staging/development environment, not performing delta checks with live. In most cases, you would have to tune your rsync to ignore files anyway, like certain property files that define resources a production server use, like database connection, smtp server, etc.

I hope this is helpful.

Answer 2

This is dependant on your application architecture.

One of my applications sits behind a load-balancing proxy, where I perform a staggered deployment - effectively eradicating downtime.

Answer 3

Just use 2 or more tomcat servers with a proxy over it. That proxy can be of apache/nignix/haproxy.

Now in each of the proxy server there is "in" and "out" url with ports are configured.

First copy your war in the tomcat without stoping the service. Once war is deployed it is automatically opened by the tomcat engine.

Note cross check unpackWARs="true" and autoDeploy="true" in node "Host" inside server.xml

It look likes this

  <Host name="localhost"  appBase="webapps"
        unpackWARs="true" autoDeploy="true"
        xmlValidation="false" xmlNamespaceAware="false">

Now see the logs of tomcat. If no error is there it means it is up successfully.

Now hit all APIs for testing

Now come to your proxy server .

Simply change the background url mapping with the new war's name. Since registering with the proxy servers like apache/nignix/haProxy took very less time, you will feel minimum downtime

Refer -- https://developers.google.com/speed/pagespeed/module/domains for mapping urls

Answer 4

Tomcat 7 has a nice feature called "parallel deployment" that is designed for this use case.

The gist is that you expand the .war into a directory, either directly under webapps/ or symlinked. Successive versions of the application are in directories named app##version, for example myapp##001 and myapp##002. Tomcat will handle existing sessions going to the old version, and new sessions going to the new version.

The catch is that you have to be very careful with PermGen leaks. This is especially true with Grails that uses a lot of PermGen. VisualVM is your friend.

Answer 5

You're using Resin, Resin has built in support for web app versioning.

http://www.caucho.com/resin-4.0/admin/deploy.xtp#VersioningandGracefulUpgrades

Update: It's watchdog process can help with permgenspace issues too.

Answer 6

I wrote a bash script that takes a few parameters and rsyncs the file between servers. Speeds up rsync transfer a lot for larger archives:

https://gist.github.com/3985742