Activemq Shutdown fails and then kills process

Question

I am implementing replicated leveldb activemq setup. I have 3 instance of activemq running on same box. I am changing their rmiPort, amqpport and openwire port in config file.

config like lookslie this:

   <?xml version="1.0" encoding="UTF-8"?><beans xmlns=" http://www.springframework.org /schema/beans" xmlns:amq=" http://activemq.apache.org/schema/core" xmlns:xsi=" http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation=" http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">

<!-- Allows us to use system properties as variables in this configuration file -->
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
    <property name="locations">
        <value>file:${activemq.conf}/credentials.properties</value>
    </property>
</bean>


<broker xmlns="http://activemq.apache.org/schema/core" brokerName="activemq_8200" dataDirectory="${activemq.data}">


    <destinationPolicy>
        <policyMap>
          <policyEntries>
            <policyEntry producerFlowControl="false" topic="&gt;">
                <!-- The constantPendingMessageLimitStrategy is used to prevent
                     slow topic consumers to block producers and affect other consumers
                     by limiting the number of messages that are retained
                     For more information, see: 
                     http://activemq.apache.org/slow-consumer-handling.html

                -->
              <pendingMessageLimitStrategy>
                <constantPendingMessageLimitStrategy limit="1000"/>
              </pendingMessageLimitStrategy>
            </policyEntry>
           <policyEntry producerFlowControl="false" queue="&gt;">
              <deadLetterStrategy>
                                    <!--
                                      Use the prifix 'DLQ.' for the destination name,  and make the DLQ a queue rather than a topic
                                    -->
                                    <individualDeadLetterStrategy queuePrefix="DLQ." useQueueForQueueMessages="true"/>
                              </deadLetterStrategy>
              <!-- Use VM cursor for better latency
                   For more information, see:

                   http://activemq.apache.org/message-cursors.html

              <pendingQueuePolicy>
                <vmQueueCursor/>
              </pendingQueuePolicy>
              -->
            </policyEntry>
          </policyEntries>
        </policyMap>
    </destinationPolicy>


    <!--
        The managementContext is used to configure how ActiveMQ is exposed in
        JMX. By default, ActiveMQ uses the MBean server that is started by
        the JVM. For more information, see:

        http://activemq.apache.org/jmx.html
    -->
    <managementContext>
        <managementContext createConnector="false"/>
    </managementContext>


    <persistenceAdapter>
        <replicatedLevelDB bind="tcp://0.0.0.0:0" directory="${activemq.data}/leveldb" replicas="3" zkAddress="gwxdev05.northamerica.cerner.net:2181,gwxdev05.northamerica.cerner.net:2182,gwxdev05.northamerica.cerner.net:2183" zkPassword="password" zkPath="/opt/gwx/activemqdata"/>
    </persistenceAdapter>



            <systemUsage>
        <systemUsage sendFailIfNoSpace="true">
            <memoryUsage>
                <memoryUsage limit="256 mb"/>
            </memoryUsage>
            <storeUsage>
                <storeUsage limit="1 gb"/>
            </storeUsage>
            <tempUsage>
                <tempUsage limit="128 mb"/>
            </tempUsage>
        </systemUsage>
    </systemUsage>


    <transportConnectors>
        <!-- DOS protection, limit concurrent connections to 1000 and frame size to 100MB -->
        <transportConnector name="openwire" uri="tcp://0.0.0.0:${openwirePort}?maximumConnections=1000&amp;wireformat.maxFrameSize=104857600"/>
        <transportConnector name="amqp" uri="amqp://0.0.0.0:${amqpPort}?maximumConnections=1000&amp;wireformat.maxFrameSize=104857600"/>
    </transportConnectors>

    <!-- destroy the spring context on shutdown to stop jetty -->
    <shutdownHooks>
        <bean xmlns="http://www.springframework.org/schema/beans" class="org.apache.activemq.hooks.SpringContextHook"/>
    </shutdownHooks>

</broker>


 <import resource="jetty.xml"/>

my instance file looks like this:

 ACTIVEMQ_BASE=`cd "$ACTIVEMQ_BASE" && pwd`

 ## Add system properties for this instance here (if needed), e.g
 #export ACTIVEMQ_OPTS_MEMORY="-Xms256M -Xmx1G"
 #export ACTIVEMQ_OPTS="$ACTIVEMQ_OPTS_MEMORY
-Dorg.apache.activemq.UseDedicatedTaskRunner=true
-Djava.util.logging.config.file=logging.properties"

 export ACTIVEMQ_SUNJMX_CONTROL="-Dactivemq.jmx.url=service:jmx:rmi:///jndi/rmi://127.0.0.1:8100/jmxrmi"
 #
 ACTIVEMQ_SUNJMX_START="-Dcom.sun.management.jmxremote.port=8100 "
 ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.authenticate=false"   
 ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.ssl=false"


 export ACTIVEMQ_SUNJMX_START=$ACTIVEMQ_SUNJMX_START
 export ACTIVEMQ_HOME=/opt/gwx/apache-activemq-5.10-SNAPSHOT
 export ACTIVEMQ_BASE=$ACTIVEMQ_BASE

  export JAVA_HOME=/opt/gwx/apache-activemq-5.10-SNAPSHOT/jdk1.7.0_25

  ${ACTIVEMQ_HOME}/bin/activemq "$@"

Here is exception I get:

 Connecting to pid: 2410
 INFO: failed to resolve jmxUrl for pid:2410, using default JMX url Connecting to JMX URL: service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
 ERROR: java.lang.RuntimeException: Failed to execute stop task. Reason: java.io.IOException: Failed to retrieve RMIServer stub: 
 javax.naming.ServiceUnavailableException [Root exception is  java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:     java.net.ConnectException: Connection refused] java.lang.RuntimeException: Failed to execute stop task. Reason: java.io.IOException: Failed to retrieve RMIServer stub:

javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is: java.net.ConnectException: Connection refused]
    at 

I checked firewall. It not issue.

Any idea what might be causing this issue.

Activemq version 5.10 snapshot

Java 1.7

OS linux 6.4

Answer 1

If it still helps someone:

When I had in bin/env this

 export ACTIVEMQ_SUNJMX_CONTROL="-Dactivemq.jmx.url=service:jmx:rmi:///jndi/rmi://127.0.0.1:8100/jmxrmi"
 #
 ACTIVEMQ_SUNJMX_START="-Dcom.sun.management.jmxremote.port=8100 "
 ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.authenticate=false"   
 ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.ssl=false"

, it did not work, process failed even without logging. I don't know why, but this method works(for AMQ 5.13): u add to xml this:

<managementContext>
            <managementContext connectorPort="1099"/>
</managementContext>

and nothing like ACTIVEMQ_SUNJMX_START in env file