Moqui deployment to AWS on Elastic Beanstalk Tomca

Question

Background

We have been running Moqui 2.0 with the embedded Jetty server on Elastic Beanstalk using a Java environment for about a year now. For security reasons, we had the application running in a private subnet along with the Postgres database and accessing it over VPN. Because of this, we never bothered to enable HTTPS access.

We now want to deploy Moqui in the public subnet so that it can be accessed without the VPN and therefore we need HTTPS access.

Sidenote: Bear in mind, I have already read through the deployment options here!.

HTTPS on Moqui standalone

My first question is, can you run Moqui over HTTPS using the embedded server only? I looked at the MoquiStart class and main seems to only support the initialisation of HTTP on port 8080 by default. There is a whole section which is commented out which also contains code to initialise HTTPS and HTTP/2 with the comment:

// Possible code to handle HTTPS, HTTP/2 (h2, h2c):

So I assume that HTTPS is not supported by Moqui as a standalone application?

Moqui in a web application server

With the assumption above that Moqui isn't configuring Jetty to serve HTTPS requests, running Moqui in the public subnet using a Java environment as we have until now obviously becomes unacceptable. Which leaves us with two options. Containerisation such as Docker (which is beyond my skills set) or running within an application server such as Tomcat (which is also beyond my skillset but I can at least wrap my head around conceptually). Which lead me to this StackOverflow thread on the issue!.

I'm not familiar with J2EE architecture, but I assume that once running within Tomcat, Catalina bypasses the Jetty port initialisation in MoquiStart somehow? [Would appreciate brief explanation for my own understanding]

Regardless, the above thread confirmed that I was on the right track for an HTTPS solution.

The Problem

The problem is, Moqui isn't working. I know Tomcat is working fine because when I create the environment, I use the Sample Application and I can access that ok over HTTP. As soon as I deploy the ROOT.war file (which I find in ../apache-tomcat-8.5.6/webapps), I can neither access Moqui over HTTP nor HTTPS. (My security group is totally open) It tries to connect and eventually times out.

I have checked the EB logs and the likely culprit is logged in the catalina.YYYY-MM-DD.log file.

14-May-2019 21:21:07.101 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/var/lib/tomcat8/webapps/ROOT]
14-May-2019 21:21:23.946 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal One or more listeners failed to start. Full details will be found in the appropriate container log file
14-May-2019 21:21:23.956 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal Context [] startup failed due to previous errors
14-May-2019 21:21:23.965 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [AsyncAppender-AsyncLog] but has failed to stop it. This is very likely to create a memory leak.

The stack trace is as follows:

sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
org.apache.logging.log4j.core.appender.AsyncAppender$AsyncThread.run(AsyncAppender.java:282)

Immediate downstream errors are:

14-May-2019 21:21:23.989 SEVERE [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [ROOT] created a ThreadLocal with key of type [com.hazelcast.internal.util.ThreadLocalRandom$1] (value [com.hazelcast.internal.util.ThreadLocalRandom$1@4379b697]) and a value of type [com.hazelcast.internal.util.ThreadLocalRandom] (value [com.hazelcast.internal.util.ThreadLocalRandom@37d77b2b]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
14-May-2019 21:21:24.006 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/var/lib/tomcat8/webapps/ROOT] has finished in [16,904] ms

Relevant parts MoquiActualConf are:

   ...
    <default-property name="webapp_http_host" value="[OUR_IP_HERE]"/>
    <default-property name="webapp_http_port" value="80"/>
    <default-property name="webapp_https_port" value="443"/>
    <default-property name="webapp_https_enabled" value="true"/>
    <default-property name="entity_ds_db_conf" value="h2"/>
    <default-property name="entity_ds_host" value="localhost"/>
    <default-property name="entity_ds_port" value=""/>
    <default-property name="entity_ds_database" value="moqui"/>
    <default-property name="entity_ds_url" value="jdbc:h2:${moqui_runtime}/db/h2/${entity_ds_database};lock_timeout=30000"/>
    <default-property name="entity_ds_schema" value=""/>
    <default-property name="entity_ds_user" value="sa"/>
    <default-property name="entity_ds_password" value="sa"/>
    ...
        <webapp name="webroot" http-port="80" http-host="[OUR_IP_HERE]" https-port="443" https-host="[OUR_IP_HERE]" https-enabled="true" require-session-token="true" websocket-timeout="600000">
            <root-screen host=".*" location="component://webroot/screen/webroot.xml"/>
            <error-screen error="unauthorized" screen-path="error/Unauthorized"/>
            <error-screen error="forbidden" screen-path="error/Forbidden"/>
            <error-screen error="not-found" screen-path="error/NotFound"/>
            <error-screen error="too-many" screen-path="error/TooMany"/>
            <error-screen error="internal-error" screen-path="error/InternalError"/>
            <listener class="org.moqui.impl.webapp.MoquiSessionListener"/>
            <servlet name="MoquiServlet" class="org.moqui.impl.webapp.MoquiServlet" load-on-startup="1">
                <url-pattern><![CDATA[/*]]></url-pattern>
            </servlet>
            <servlet name="MoquiFopServlet" class="org.moqui.impl.webapp.MoquiFopServlet" load-on-startup="1">
                <url-pattern><![CDATA[/fop/*]]></url-pattern>
            </servlet>
            <session-config timeout="60"/>
            <endpoint path="/notws" class="org.moqui.impl.webapp.NotificationEndpoint" timeout="3600000" enabled="true"/>
    ...

I've been digging around in the Tomcat guts of the EC2 instance for three days now and come up with nothing so would really appreciate some help.

PS: On an unrelated note, for some reason, MoquiActualConf is being overridden by MoquiProductionConf rather than MoquiTestConf as specified in the Procfile.

web: java -Xmx1024M -cp . MoquiStart port=8080 conf=conf/MoquiTestConf.xml

I don't think this is the cause of the issue but will cause problems down the line when pointing to production db, etc.

Answer 1

The http* attributes on the webapp element are for configuring URL writing, not for configuring the web server which is external to Moqui. Moqui runs inside a Java Servlet container like Tomcat or Jetty, even when using the embedded Jetty approach, and configuration of the Servlet container is where the HTTP interfaces are configured.

Generally the best approach for HTTPS termination is to use a reverse proxy based on httpd or nginx. Even ElasticBeanstalk images do this by default (with httpd) and if you configure your HTTPS cert through AWS EB it is httpd that handles it. The same approach is used with nginx-proxy in the Docker Compose examples.

Answer 2

I manage to make Moqui instance in AWS EB accessible via HTTPS by setting up the ec2 load balancer. To point the obvious, Moqui's jetty listening to port 5000, ec2 Nginx listens to port 80 (and forward it to port 5000), ec2 load balancer listens to port 80 and 443 (and forward it to ec2 Nginx port 80). If you don't want to use load balancer you can setup ec2 Nginx to listen for HTTPS and forward it to Moqui's jetty port 5000 the same way Nginx HTTP does. The least preferred way is passing some https jetty's parameters via environment variables, small changes might be needed in MoquiStart. It was years ago and I don't recall the detail but Jetty was able to serve HTTPS this way.