Distributed JMS based logging .. falling flat?

Question

In our fancy ESB, logging of each request is done via a common infrastructure based on JMS based logging. Here is what happens in a nutshell:

1. service gets a request service
2. prepares some data in a LogData
3. object service calls database
4. time taken for db interaction is captured in LogData object
5. service is ready to send response
6. LogData object is sent to a messaging destination
7. service sends response

Very rosey! yes for paper architects. Here is the actual issue: The JMS service provider sometimes becomes unavailable - due to a system level error or the software crashes. Then the service waits at the step (step no. 6) where it has to make JMS connection to send LogData object. Resulting in delayed response, thus leading to bad performance and user experience.

So that is the biggest shortcoming of "Distributed logging using JMS" touted by a lot of developer websites. Also note that the presance of LogData is kind of critical non-functional requirement. That means the messages are sent in persistent mode, leading to a wait until the JMS provider confirms receipt of message to the sender (the service in this case) - what to be blamed? immature design? Are there any success stories of implementing something like this?

Answer 1

Well, the main problem in your scenario is that your JMS message broker (service provider) has stability issues. Distributed logging via JMS will certainly work if it's treated as a critical part of your infrastructure that shouldn't be down but rather should be monitored and running in a clustered configuration that ensures that at least one instance is always up.

Async logging will not be much help if JMS broker is down. e.g. if you use log4j AsyncAppender, it will either become a blocking appender after the appender's buffer is filled up, or will throw away logging messages (if you have blocking=false).

That said, in my experience logging via JMS has almost always been an overkill, especially if it's the only reason you've set up a JMS broker.

If you want to "distribute" your logs across several boxes, there are several simple ways of doing it:

• log to a file rolling over every hour or so (using rolling file appender in log4j or backlog) and then just rsync every hour to other boxes
• write logs to an NFS diriectory or to a shared SAN directory mounted on all boxes (each log file will have to use instance specific name to avoid overriding each other)

• log to a DB directly from the app - will probably want to pipe it through AsyncAppender though to avoid blocking. Hopefully your DB has higher availability than JMS provider

• Ed Y.

Answer 2

Doing any logging in db/jms/socket/etc synchronously is asking for problems and a lot of them.

Implement it with logging into memory and async. dump to file/jms (depending if the JMS becomes available). A single background thread should do the trick for you. Having sync. logging could cause a lot of trouble in totally unexpected and innocent portions of the application.

I can't think of any possible success story of a sync. logging.

Edit. preferrably use the like of ConcurrentLinkedQueue to keep the LogData (I mean avoid any blocking, if possible to improve the perforamce/throughput)