IBM MQ: How to know reason for dead letters?

I'm seeing a bunch of messages on the dead letter queue and I don't understand what causes this.

I'm using MQ Explorer to browse such messages. Here's what I see in the Dead-Letter Header:

This doesn't tell me what the real cause of the problem is. How can I find out ?

I've read this article from IBM and it tells that the reason is likely a badly formatted message. In what way badly formatted?

(note: I'm in control of both producer and consumer)

回答1:

you miss the forest for the trees :-) The cause is in the field 'Reason' . MQRC_BACKOUT_THRESHOLD_REACHED This is described here in Knowledge Center

MQRC_BACKOUT_THRESHOLD_REACHED (0x93A; 2362) Cause The message has reached the Backout Threshold defined on the QLOCAL, but no Backout Queue is defined. On platforms where you cannot define the Backout Queue, the message has reached the JMS-defined backout threshold of 20. Action If this is not wanted, define the Backout Queue for the relevant QLOCAL. Also look for the cause of the multiple backouts.

回答2:

As I expected the MQRC_BACKOUT_THRESHOLD_REACHED reason is really just a knock-on effect. To find the real reason you would need to look into the logs on either the consumer-side or the producer-side depending on what you see in the Put application name field in the screenshot from the dead-letter header (above).

I've now learned that MQ Classes for JMS produces log files in current dir named mqjms.log.x. By looking at this I can see the true reason for the problem:

July 19, 2016 4:48:33 PM CEST[Queue Service thread] com.ibm.msg.client.wmq.common.internal.messages.WMQReceiveMarshal
A received message could not be correctly parsed.

EXPLANATION:
The message with messageID = '414D512064657620202020202020202012048D5720064E04' and correlationID = '310000000000000000000000000000000000000000000000' could not be correctly parsed. The last successful data read from the message was at position '192' in buffer '0000:  5246 4820 0000 0002 0000 00c0 0000 0111    RFH ............
0010:  0000 04b8 4d51 5354 5220 2020 0000 0000    ....MQSTR   ....
0020:  0000 04b8 0000 0020 3c6d 6364 3e3c 4d73    ....... <mcd><Ms
0030:  643e 6a6d 735f 7465 7874 3c2f 4d73 643e    d>jms_text</Msd>
0040:  3c2f 6d63 643e 2020 0000 0074 3c6a 6d73    </mcd>  ...t<jms
0050:  3e3c 4473 743e 7175 6575 653a 2f2f 2f6d    ><Dst>queue:///m
0060:  7971 7565 7565 3c2f 4473 743e 3c54 6d73    yqueue</Dst><Tms
0070:  3e31 3436 3839 3339 3731 3338 3234 3c2f    >1468939713824</
0080:  546d 733e 3c45 7870 3e31 3436 3839 3339    Tms><Exp>1468939
0090:  3734 3338 3234 3c2f 4578 703e 3c43 6964    743824</Exp><Cid
00a0:  3e49 443a 3331 3c2f 4369 643e 3c44 6c76    >ID:31</Cid><Dlv
00b0:  3e31 3c2f 446c 763e 3c2f 6a6d 733e 2020    >1</Dlv></jms>  
00c0:  3c64 6174 614d 7367 2073 656e 7454 696d    <mymessage .....
.................

' with exception '
                       Message : java.lang.Exception
                         Class : class java.lang.Exception
                         Stack : com.ibm.msg.client.wmq.internal.WMQConsumerShadow.getMsg(WMQConsumerShadow.java:1900)
                               : com.ibm.msg.client.wmq.internal.WMQSyncConsumerShadow.receiveInternal(WMQSyncConsumerShadow.java:231)
                               : com.ibm.msg.client.wmq.internal.WMQConsumerShadow.receive(WMQConsumerShadow.java:1471)
                               : com.ibm.msg.client.wmq.internal.WMQMessageConsumer.receive(WMQMessageConsumer.java:659)
                               : com.ibm.msg.client.jms.internal.JmsMessageConsumerImpl.receiveInboundMessage(JmsMessageConsumerImpl.java:1036)
                               : com.ibm.msg.client.jms.internal.JmsMessageConsumerImpl.receive(JmsMessageConsumerImpl.java:461)
                               : com.ibm.msg.client.jms.internal.JmsMessageConsumerImpl.receive(JmsMessageConsumerImpl.java:495)
                               : com.ibm.mq.jms.MQMessageConsumer.receive(MQMessageConsumer.java:209)
                               : org.mycorp.client.base.connection.QueueService.recvMessages(QueueService.java:129)
                               : org.mycorp.client.base.connection.QueueService.run(QueueService.java:92)
                               : java.lang.Thread.run(Thread.java:745)
' with MQMD 'version:2(0x2) report:0(0x0) msgType:8(0x8) expiry:300(0x12c) feedback:0(0x0) encoding:273(0x111) codedCharSetId:1208(0x4b8) format:'MQHRF2  ' priority:4(0x4) persistence:0(0x0) msgId:414D512064657620202020202020202012048D5720064E04 correlId:310000000000000000000000000000000000000000000000 backoutCount:0(0x0) replyToQ:'                                                ' replyToQMgr:'dev                                             ' userIdentifier:'peter       ' accountingToken:160105150000008D3439C9CC13CC025B66F34BE903000000000000000000000B applIdentityData:'                                ' putApplType:28(0x1c) putApplName:'Carrefour Server            ' putDate:'20160719' putTime:'14483382' applOriginData:'    ' groupId:000000000000000000000000000000000000000000000000 msgSeqNumber:1(0x1) physicalMsgOffset:0(0x0) msgFlags:0(0x0) originalLength:-1(0xffffffff) '

EXPLANATION:
null

ACTION:
null

So there you have it. Somehow I've managed to create a JMS message which is invalid and without the producer-side noticing that there's a problem.

I'll need to figure that out but that'll be a topic for another post.

In short the answer to the question is: The backout is just a knock-on effect. The real reason is - as the doc from IBM says - a badly formatted message. The only method to figure that out is to look into whatever logs are dumped by either the message producer or (more likely) the message consumer. In my case - since no remote queues are involved - my message consumer is not an intermediary Queue Manager but my actual destination application. That's where I found the log.