I have developed a JSF/Primefaces application with the following stack:

MyFaces JSF 2.2
Primefaces 6.0
TomEE 7.0.4
JDK 8

This application is using different Primefaces components. Apart from page navigation, all actions are performed ajaxically.

The functionalities are working fine in my dev environment; I use Debian. When the application was deployed in the UAT server, which is CentOS; the ajax actions are not working properly.

I can see in the browser's dev tool's networking that the requests are getting triggered and receiving responses with HTTP 200 but most of the time it is not reaching the server. I have done logging various things and when the request is not reaching the server, nothing gets logged in the console; not even in the access.log of TomEE.

When the request is reaching to the server the application is behaving properly, otherwise not. The issue is intermittent, sometimes it is working but most of the time it is failing. And the worst part is that it cannot be reproduced in my dev environment.

To the best of my knowledge the request first reaches to Route 53 then it goes to ELB which passes the request to the Apache Reverse Proxy and at the end, the RP forwards the request to the actual TomEE which is in an EC2.

The site is secured by HTTPS and the certificate is placed in RP.

Primarily I have thought that there is something wrong with networking because when the issue is occurring I have seen message backend socket is disconnected in the log of RP. So I have tried to by-pass everything by creating an SSH Tunnel directly to the UAT server, but the issue still remains.

In the application side, I also have tested by setting javax.faces.STATE_SAVING_METHOD to server and javax.faces.PARTIAL_STATE_SAVING to false but no luck.

See the following image:

First and second requests worked and returned responses of size 223 KB and 227 KB respectively; both of them worked.
Third one returned HTTP 200 but the response size is 1.0 KB; this one failed to work.
Also, the response time taken by the third request is much lesser than that of other two requests.
You can see, I have checked "Disable cache" in the dev tool.

The request-response that I have shown in the picture, is actually triggered by a Primfaces Datatable filtering. I was filtering on a column. In the first two attempts, which returned proper response and filtered the table, I had entered some characters in the filter input textbox. In the last attempt, I cleared up the previously entered characters, so that the Datatable gets reset. As no proper response came, and the Datatable showed 'No result found'.

Also, please note that it has been tested:

In different browsers
By disabling all extensions/plugin of a given browser
In different OS
In different machine

I also have copied the UAT TomEE into my dev machine, changed the database connection configuration pointing to my local database (since connection speed between UAT database and my dev environment is slower than that of UAT TomEE; and I thought if this could be the reason) and I wasn't able to reproduce the issue.

I have no idea about what could be the reason for such weird behavior. Any pointer would be very helpful.

Update

I have dig up more, downloaded source code of MyFaces and added few logs, packaged it and updated TomEE lib with custom myfaces-api-2.2.12.jar and myfaces-impl.2.2.12.jar. Also, I have enable log level ALL for javax.servlet, javax.faces, org.apache.myfaces, org.primefaces.

Followings are the observations:

In case of success

Networking

General

Request URL: https://<host>/<context>/pages/applications.xhtml
Request Method: POST
Status Code: 200 OK
Remote Address: <ip>:<port>
Referrer Policy: no-referrer-when-downgrade

Request Headers

Accept: application/xml, text/xml, */*; q=0.01
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en;q=0.9,en-US;q=0.8,bn;q=0.7
Cache-Control: no-cache
Connection: keep-alive
Content-Length: 3728
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Cookie: JSESSIONID=<session-id>; oam.Flash.RENDERMAP.TOKEN=14ofu8swbe
DNT: 1
Faces-Request: partial/ajax
Host: <host>
Origin: https://<host>
Pragma: no-cache
Referer: https://<host>/<context>/pages/applications.xhtml
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36
X-Requested-With: XMLHttpRequest

Form Data

javax.faces.partial.ajax: true
javax.faces.source: application-table-form:application-table
javax.faces.partial.execute: application-table-form:application-table
javax.faces.partial.render: application-table-form:application-table
application-table-form:application-table: application-table-form:application-table
application-table-form:application-table_filtering: true
application-table-form:application-table_encodeFeature: true
application-table-form:application-table:j_id_1k:filter: 
application-table-form:application-table:j_id_1m:filter: acc
application-table-form:application-table_rppDD: 20
application-table-form_SUBMIT: 1
javax.faces.ViewState: <view-state>

Response Headers

Cache-Control: no-cache,no-store,must-revalidate
Cache-Control: no-cache
Connection: Keep-Alive
Content-Security-Policy: default-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; img-src 'self' data:; frame-ancestors 'none';
Content-Type: text/xml;charset=UTF-8
Date: Sat, 27 Oct 2018 14:48:02 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Expires: -1
Keep-Alive: timeout=5, max=100
Pragma: no-cache
Pragma: no-cache
Set-Cookie: oam.Flash.RENDERMAP.TOKEN=14ofu8swbf; Path=/<context>; HttpOnly
Strict-Transport-Security: max-age=15768000;
Transfer-Encoding: chunked

Response

<?xml version="1.0" encoding="UTF-8"?>
<partial-response id="j_id__v_0">
    <changes>
        <update id="growl-panel-form:growl-panel"><![CDATA[<div id="growl-panel-form:growl-panel" class="ui-outputpanel ui-widget"><span id="growl-panel-form:growl"></span><script id="growl-panel-form:growl_s" type="text/javascript">$(function(){PrimeFaces.cw('Growl','widget_growl_panel_form_growl',{id:'growl-panel-form:growl',sticky:false,life:6000,escape:true,msgs:[]});});</script></div>]]></update>
        <update id="application-table-form:application-table"><![CDATA[<tr data-ri="0" class="ui-widget-content ui-datatable-even" role="row"><td role="gridcell" class="row-toggler"><div class="ui-row-toggler ui-icon ui-icon-circle-triangle-e" tabindex="0" role="button" aria-expanded="false" aria-label="Toggle Row"></div></td><td role="gridcell">1001</td><td role="gridcell">Access Control</td><td role="gridcell" class="application-action"><button id="application-table-form:application-table:0:edit-application" name="application-table-form:application-table:0:edit-application" class="ui-button ui-widget ui-state-default ui-corner-all ui-button-icon-only" onclick="PrimeFaces.ab({s:&quot;application-table-form:application-table:0:edit-application&quot;,p:&quot;application-table-form:application-table:0:edit-application&quot;,u:&quot;update-application-dialog-form&quot;,onco:function(xhr,status,args){PF('updateApplicationDialog').show();}});return false;" type="submit"><span class="ui-button-icon-left ui-icon ui-c fa fa-edit"></span><span class="ui-button-text ui-c">ui-button</span></button><script id="application-table-form:application-table:0:edit-application_s" type="text/javascript">PrimeFaces.cw("CommandButton","widget_application_table_form_application_table_0_edit_application",{id:"application-table-form:application-table:0:edit-application"});</script><div id="application-table-form:application-table:0:j_id_1r" class="ui-tooltip ui-widget ui-tooltip-left"><div class="ui-tooltip-arrow"></div><div class="ui-tooltip-text ui-shadow ui-corner-all">Edit this Application</div></div><script id="application-table-form:application-table:0:j_id_1r_s" type="text/javascript">$(function(){PrimeFaces.cw("Tooltip","widget_application_table_form_application_table_0_j_id_1r",{id:"application-table-form:application-table:0:j_id_1r",showEffect:"fade",hideEffect:"fade",target:"application-table-form:application-table:0:edit-application",position:"left"});});</script><button id="application-table-form:application-table:0:delete-application" name="application-table-form:application-table:0:delete-application" class="ui-button ui-widget ui-state-default ui-corner-all ui-button-icon-only gap-before" data-pfconfirmcommand="PrimeFaces.bcn(this,event,[function(event){PrimeFaces.ab({s:&quot;application-table-form:application-table:0:delete-application&quot;,p:&quot;application-table-form:application-table:0:delete-application&quot;,u:&quot;application-table-form:application-table&quot;});return false;}]);" onclick="PrimeFaces.confirm({source:&quot;application-table-form:application-table:0:delete-application&quot;,header:&quot;Confirmation&quot;,message:&quot;Delete 1001?&quot;,icon:&quot;fa fa-exclamation-triangle&quot;});return false;" type="submit"><span class="ui-button-icon-left ui-icon ui-c fa fa-trash"></span><span class="ui-button-text ui-c">ui-button</span></button><script id="application-table-form:application-table:0:delete-application_s" type="text/javascript">PrimeFaces.cw("CommandButton","widget_application_table_form_application_table_0_delete_application",{id:"application-table-form:application-table:0:delete-application",behaviors:{click:function(ext,event) {}}});</script><div id="application-table-form:application-table:0:j_id_1t" class="ui-tooltip ui-widget ui-tooltip-left"><div class="ui-tooltip-arrow"></div><div class="ui-tooltip-text ui-shadow ui-corner-all">Delete this Application</div></div><script id="application-table-form:application-table:0:j_id_1t_s" type="text/javascript">$(function(){PrimeFaces.cw("Tooltip","widget_application_table_form_application_table_0_j_id_1t",{id:"application-table-form:application-table:0:j_id_1t",showEffect:"fade",hideEffect:"fade",target:"application-table-form:application-table:0:delete-application",position:"left"});});</script></td></tr>]]></update>
        <update id="j_id__v_0:javax.faces.ViewState:1"><![CDATA[view-state]]></update>
        <extension ln="primefaces" type="args">{"totalRecords":1}</extension>
    </changes>
</partial-response>

Server Log

2018-10-27 14:03:52,376 : DEBUG : http-nio-6880-exec-1 : org.apache.myfaces.view.facelets.FaceletViewDeclarationLanguage : renderView : Rendering View: /pages/applications.xhtml
2018-10-27 14:03:52,376 : FINEST : http-nio-6880-exec-1 : org.apache.myfaces.view.facelets.FaceletViewDeclarationLanguage : getResponseEncoding : Facelet specified alternate encoding 'UTF-8'
2018-10-27 14:03:52,376 : DEBUG : http-nio-6880-exec-1 : javax.faces.component.UIComponentBase : encodeAll : ########## - encodeAll begins for component: class javax.faces.component.UIViewRoot
2018-10-27 14:03:52,377 : DEBUG : http-nio-6880-exec-1 : javax.faces.component.UIComponentBase : encodeAll : ########## - encodeAll begins for component: class org.primefaces.component.outputpanel.OutputPanel
2018-10-27 14:03:52,377 : DEBUG : http-nio-6880-exec-1 : javax.faces.component.UIComponentBase : encodeAll : ########## - encodeAll end for component: class org.primefaces.component.outputpanel.OutputPanel
2018-10-27 14:03:52,377 : DEBUG : http-nio-6880-exec-1 : javax.faces.component.UIComponentBase : encodeAll : ########## - encodeAll begins for component: class org.primefaces.component.datatable.DataTable
2018-10-27 14:03:52,562 : DEBUG : http-nio-6880-exec-1 : javax.faces.component.UIComponentBase : encodeAll : ########## - encodeAll begins for component: class org.primefaces.component.rowtoggler.RowToggler
2018-10-27 14:03:52,562 : DEBUG : http-nio-6880-exec-1 : javax.faces.component.UIComponentBase : encodeAll : ########## - encodeAll end for component: class org.primefaces.component.rowtoggler.RowToggler
2018-10-27 14:03:52,563 : DEBUG : http-nio-6880-exec-1 : javax.faces.component.UIComponentBase : encodeAll : ########## - encodeAll begins for component: class javax.faces.component.html.HtmlOutputText
2018-10-27 14:03:52,563 : DEBUG : http-nio-6880-exec-1 : org.apache.myfaces.shared.renderkit.RendererUtils : getStringValue : the created converter is null
2018-10-27 14:03:52,563 : DEBUG : http-nio-6880-exec-1 : org.apache.myfaces.shared.renderkit.RendererUtils : getStringValue : returning an .toString
2018-10-27 14:03:52,563 : DEBUG : http-nio-6880-exec-1 : org.apache.myfaces.shared.renderkit.html.HtmlTextRendererBase : renderOutput : renderOutput '1001'
2018-10-27 14:03:52,563 : DEBUG : http-nio-6880-exec-1 : org.apache.myfaces.shared.renderkit.html.HtmlTextRendererBase : renderOutput : renderOutputText writing '1001'
2018-10-27 14:03:52,563 : DEBUG : http-nio-6880-exec-1 : javax.faces.component.UIComponentBase : encodeAll : ########## - encodeAll end for component: class javax.faces.component.html.HtmlOutputText
2018-10-27 14:03:52,564 : DEBUG : http-nio-6880-exec-1 : javax.faces.component.UIComponentBase : encodeAll : ########## - encodeAll begins for component: class javax.faces.component.html.HtmlOutputText
2018-10-27 14:03:52,564 : DEBUG : http-nio-6880-exec-1 : org.apache.myfaces.shared.renderkit.RendererUtils : getStringValue : the created converter is null
2018-10-27 14:03:52,564 : DEBUG : http-nio-6880-exec-1 : org.apache.myfaces.shared.renderkit.RendererUtils : getStringValue : returning an .toString
2018-10-27 14:03:52,564 : DEBUG : http-nio-6880-exec-1 : org.apache.myfaces.shared.renderkit.html.HtmlTextRendererBase : renderOutput : renderOutput 'Access Control'
2018-10-27 14:03:52,564 : DEBUG : http-nio-6880-exec-1 : org.apache.myfaces.shared.renderkit.html.HtmlTextRendererBase : renderOutput : renderOutputText writing 'Access Control'
2018-10-27 14:03:52,564 : DEBUG : http-nio-6880-exec-1 : javax.faces.component.UIComponentBase : encodeAll : ########## - encodeAll end for component: class javax.faces.component.html.HtmlOutputText
2018-10-27 14:03:52,565 : DEBUG : http-nio-6880-exec-1 : javax.faces.component.UIComponentBase : encodeAll : ########## - encodeAll begins for component: class javax.faces.component.html.HtmlPanelGroup
2018-10-27 14:03:52,565 : DEBUG : http-nio-6880-exec-1 : javax.faces.component.UIComponentBase : encodeAll : ########## - encodeAll begins for component: class org.primefaces.component.commandbutton.CommandButton
2018-10-27 14:03:52,565 : DEBUG : http-nio-6880-exec-1 : javax.faces.component.UIComponentBase : encodeAll : ########## - encodeAll end for component: class org.primefaces.component.commandbutton.CommandButton
2018-10-27 14:03:52,565 : DEBUG : http-nio-6880-exec-1 : javax.faces.component.UIComponentBase : encodeAll : ########## - encodeAll begins for component: class org.primefaces.component.tooltip.Tooltip
2018-10-27 14:03:52,565 : DEBUG : http-nio-6880-exec-1 : javax.faces.component.UIComponentBase : encodeAll : ########## - encodeAll end for component: class org.primefaces.component.tooltip.Tooltip
2018-10-27 14:03:52,566 : DEBUG : http-nio-6880-exec-1 : javax.faces.component.UIComponentBase : encodeAll : ########## - encodeAll begins for component: class org.primefaces.component.commandbutton.CommandButton
2018-10-27 14:03:52,566 : DEBUG : http-nio-6880-exec-1 : javax.faces.component.UIComponentBase : encodeAll : ########## - encodeAll end for component: class org.primefaces.component.commandbutton.CommandButton
2018-10-27 14:03:52,566 : DEBUG : http-nio-6880-exec-1 : javax.faces.component.UIComponentBase : encodeAll : ########## - encodeAll begins for component: class org.primefaces.component.tooltip.Tooltip
2018-10-27 14:03:52,566 : DEBUG : http-nio-6880-exec-1 : javax.faces.component.UIComponentBase : encodeAll : ########## - encodeAll end for component: class org.primefaces.component.tooltip.Tooltip
2018-10-27 14:03:52,566 : DEBUG : http-nio-6880-exec-1 : javax.faces.component.UIComponentBase : encodeAll : ########## - encodeAll end for component: class javax.faces.component.html.HtmlPanelGroup
2018-10-27 14:03:52,567 : DEBUG : http-nio-6880-exec-1 : javax.faces.component.UIComponentBase : encodeAll : ########## - encodeAll end for component: class org.primefaces.component.datatable.DataTable
2018-10-27 14:03:52,567 : FINEST : http-nio-6880-exec-1 : org.apache.myfaces.application.StateManagerImpl : saveView : Calling saveView of StateManagementStrategy: org.apache.myfaces.view.facelets.DefaultFaceletsStateManagementStrategy
2018-10-27 14:03:52,568 : DEBUG : http-nio-6880-exec-1 : org.apache.myfaces.shared.util.StateUtils : findAlgorithm : Using algorithm DES
2018-10-27 14:03:52,568 : DEBUG : http-nio-6880-exec-1 : org.apache.myfaces.shared.util.StateUtils : findAlgorithmParams : Using algorithm paramaters ECB/PKCS5Padding
2018-10-27 14:03:52,568 : DEBUG : http-nio-6880-exec-1 : org.apache.myfaces.shared.util.StateUtils : findMacAlgorithm : Using algorithm HmacSHA1
2018-10-27 14:03:52,568 : DEBUG : http-nio-6880-exec-1 : org.apache.myfaces.shared.util.StateUtils : encrypt : encrypting w/ DES/ECB/PKCS5Padding
2018-10-27 14:03:52,569 : DEBUG : http-nio-6880-exec-1 : javax.faces.component.UIComponentBase : encodeAll : ########## - encodeAll end for component: class javax.faces.component.UIViewRoot
2018-10-27 14:03:52,569 : FINEST : http-nio-6880-exec-1 : org.apache.myfaces.lifecycle.LifecycleImpl : render : exiting RENDER_RESPONSE(6) in org.apache.myfaces.lifecycle.LifecycleImpl

In case of failure