No MBean found for worklight project

Question

I have been getting this error while starting up my server with my worklight project (.war) deployed on it. Was wondering if anyone has any inputs. Below are the relevant information.

The system is set up in Redhat OS, Worklight version 6.2 consumer edition, DB2 as database and WS-Liberty 8.5.5.1 as server.

P.S: My application was available before this problem occured, however my worklight console was never working correctly (always had problem with message on worklight console saying "no runtime configuratin can be found"). It still says the same.

logs

[AUDIT   ] CWWKG0016I: Starting server configuration update.
[AUDIT   ] CWWKG0017I: The server configuration was successfully updated in 0.071 seconds.
[WARNING ] SRVE9967W: The manifest class path pdq.jar can not be found in jar file wsjar:file:/opt/IBM/WebSphere/Liberty/usr/servers/WL_UAT_Server/apps/MyProjectName.war!/WEB-INF/lib/db2jcc.jar or its parent.
[WARNING ] SRVE9967W: The manifest class path pdq.jar can not be found in jar file wsjar:file:/opt/IBM/WebSphere/Liberty/usr/servers/WL_UAT_Server/apps/MyProjectName.war!/WEB-INF/lib/db2jcc4.jar or its parent.
[WARNING ] SRVE9967W: The manifest class path pdq.jar can not be found in jar file wsjar:file:/opt/IBM/WebSphere/Liberty/usr/servers/WL_UAT_Server/apps/MyProjectName.war!/WEB-INF/lib/db2jcc.jar or its parent.
[WARNING ] SRVE9967W: The manifest class path pdq.jar can not be found in jar file wsjar:file:/opt/IBM/WebSphere/Liberty/usr/servers/WL_UAT_Server/apps/MyProjectName.war!/WEB-INF/lib/db2jcc4.jar or its parent.
[AUDIT   ] CWWKT0016I: Web application available (default_host): http://localhost:9080/MyProjectName/
[ERROR   ] FWLSE0188E: Failed to connect to DB2 database : Cannot create PoolableConnectionFactory ([jcc][10389][12245][3.67.27] Failure in loading native library db2jcct2, java.lang.UnsatisfiedLinkError: no db2jcct2 in java.library.path:  ERRORCODE=-4472, SQLSTATE=null) [project MyProjectName]
[ERROR   ] FWLST0003E: ========= Failed starting project /MyProjectName [project MyProjectName]
Error creating bean with name 'txManager' defined in URL [wsjar:file:/opt/IBM/WebSphere/Liberty/usr/shared/resources/worklight-jee-library-6.2.0.jar!/conf/core.xml]: Cannot resolve reference to bean 'brokerSessionFactory' while setting bean property 'entityManagerFactory'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'brokerSessionFactory' defined in URL [wsjar:file:/opt/IBM/WebSphere/Liberty/usr/shared/resources/worklight-jee-library-6.2.0.jar!/conf/spring-server-core.xml]: Cannot resolve reference to bean 'rssBrokerDS' while setting bean property 'dataSource'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'rssBrokerDS' defined in URL [wsjar:file:/opt/IBM/WebSphere/Liberty/usr/shared/resources/worklight-jee-library-6.2.0.jar!/conf/spring-server-core.xml]: Cannot resolve reference to bean 'worklight-direct' while setting bean property 'targetDataSource'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'worklight-direct' defined in URL [wsjar:file:/opt/IBM/WebSphere/Liberty/usr/shared/resources/worklight-jee-library-6.2.0.jar!/conf/spring-server-core.xml]: Invocation of init method failed; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory ([jcc][10389][12245][3.67.27] Failure in loading native library db2jcct2, java.lang.UnsatisfiedLinkError: no db2jcct2 in java.library.path:  ERRORCODE=-4472, SQLSTATE=null)
[ERROR   ] Error creating bean with name 'txManager' defined in URL [wsjar:file:/opt/IBM/WebSphere/Liberty/usr/shared/resources/worklight-jee-library-6.2.0.jar!/conf/core.xml]: Cannot resolve reference to bean 'brokerSessionFactory' while setting bean property 'entityManagerFactory'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'brokerSessionFactory' defined in URL [wsjar:file:/opt/IBM/WebSphere/Liberty/usr/shared/resources/worklight-jee-library-6.2.0.jar!/conf/spring-server-core.xml]: Cannot resolve reference to bean 'rssBrokerDS' while setting bean property 'dataSource'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'rssBrokerDS' defined in URL [wsjar:file:/opt/IBM/WebSphere/Liberty/usr/shared/resources/worklight-jee-library-6.2.0.jar!/conf/spring-server-core.xml]: Cannot resolve reference to bean 'worklight-direct' while setting bean property 'targetDataSource'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'worklight-direct' defined in URL [wsjar:file:/opt/IBM/WebSphere/Liberty/usr/shared/resources/worklight-jee-library-6.2.0.jar!/conf/spring-server-core.xml]: Invocation of init method failed; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory ([jcc][10389][12245][3.67.27] Failure in loading native library db2jcct2, java.lang.UnsatisfiedLinkError: no db2jcct2 in java.library.path:  ERRORCODE=-4472, SQLSTATE=null)
Error creating bean with name 'txManager' defined in URL [wsjar:file:/opt/IBM/WebSphere/Liberty/usr/shared/resources/worklight-jee-library-6.2.0.jar!/conf/core.xml]: Cannot resolve reference to bean 'brokerSessionFactory' while setting bean property 'entityManagerFactory'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'brokerSessionFactory' defined in URL [wsjar:file:/opt/IBM/WebSphere/Liberty/usr/shared/resources/worklight-jee-library-6.2.0.jar!/conf/spring-server-core.xml]: Cannot resolve reference to bean 'rssBrokerDS' while setting bean property 'dataSource'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'rssBrokerDS' defined in URL [wsjar:file:/opt/IBM/WebSphere/Liberty/usr/shared/resources/worklight-jee-library-6.2.0.jar!/conf/spring-server-core.xml]: Cannot resolve reference to bean 'worklight-direct' while setting bean property 'targetDataSource'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'worklight-direct' defined in URL [wsjar:file:/opt/IBM/WebSphere/Liberty/usr/shared/resources/worklight-jee-library-6.2.0.jar!/conf/spring-server-core.xml]: Invocation of init method failed; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory ([jcc][10389][12245][3.67.27] Failure in loading native library db2jcct2, java.lang.UnsatisfiedLinkError: no db2jcct2 in java.library.path:  ERRORCODE=-4472, SQLSTATE=null)
[ERROR   ] MyProjectName: WL_UAT_Server///10.80.25.54: 2014-08-15T15:19:11.145Z: BeanCreationException:Error creating bean with name 'txManager' defined in URL [wsjar:file:/opt/IBM/WebSphere/Liberty/usr/shared/resources/worklight-jee-library-6.2.0.jar!/conf/core.xml]: Cannot resolve reference to bean 'brokerSessionFactory' while setting bean property 'entityManagerFactory'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'brokerSessionFactory' defined in URL [wsjar:file:/opt/IBM/WebSphere/Liberty/usr/shared/resources/worklight-jee-library-6.2.0.jar!/conf/spring-server-core.xml]: Cannot resolve reference to bean 'rssBrokerDS' while setting bean property 'dataSource'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'rssBrokerDS' defined in URL [wsjar:file:/opt/IBM/WebSphere/Liberty/usr/shared/resources/worklight-jee-library-6.2.0.jar!/conf/spring-server-core.xml]: Cannot resolve reference to bean 'worklight-direct' while setting bean property 'targetDataSource'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'worklight-direct' defined in URL [wsjar:file:/opt/IBM/WebSphere/Liberty/usr/shared/resources/worklight-jee-library-6.2.0.jar!/conf/spring-server-core.xml]: Invocation of init method failed; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory ([jcc][10389][12245][3.67.27] Failure in loading native library db2jcct2, java.lang.UnsatisfiedLinkError: no db2jcct2 in java.library.path:  ERRORCODE=-4472, SQLSTATE=null)
[AUDIT   ] CWWKZ0001I: Application MyProjectName started in 4.022 seconds.
[ERROR   ] FWLSE3002E: The resource is not found.
FWLSE3041E: No MBean found for Worklight project 'MyProjectName'. Possibly the Worklight runtime web application for Worklight project 'MyProjectName' is not running. If it is running, use JConsole to inspect the available MBeans.
[ERROR   ] SRVE0315E: An execption occurred: com.ibm.ws.webcontainer.webapp.WebAppErrorReport: javax.servlet.ServletException: Worklight Console initialization failed.Logged Exception: com.ibm.db2.jcc.am.SqlException: [jcc][10389][12245][3.67.27] Failure in loading native library db2jcct2, java.lang.UnsatisfiedLinkError: no db2jcct2 in java.library.path:  ERRORCODE=-4472, SQLSTATE=null
    at com.worklight.core.auth.impl.AuthenticationFilter.verifyServletInitialized(AuthenticationFilter.java:422)
    at com.worklight.core.auth.impl.AuthenticationFilter.doFilter(AuthenticationFilter.java:127)
    at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:194)
    at [internal classes]
Caused by: javax.servlet.ServletException: Worklight Console initialization failed.Logged Exception: com.ibm.db2.jcc.am.SqlException: [jcc][10389][12245][3.67.27] Failure in loading native library db2jcct2, java.lang.UnsatisfiedLinkError: no db2jcct2 in java.library.path:  ERRORCODE=-4472, SQLSTATE=null
    ... 5 more

[AUDIT   ] CWWKG0016I: Starting server configuration update.
[AUDIT   ] CWWKG0018I: The server configuration was not updated. No functional changes were detected.
[AUDIT   ] CWWKG0016I: Starting server configuration update.
[AUDIT   ] CWWKG0017I: The server configuration was successfully updated in 0.149 seconds.
[AUDIT   ] CWWKT0017I: Web application removed (default_host): http://localhost:9080/_MobileBrowserSimulator/
[AUDIT   ] CWWKZ0009I: The application _MobileBrowserSimulator has stopped successfully.
[AUDIT   ] CWWKG0016I: Starting server configuration update.
[AUDIT   ] CWWKG0017I: The server configuration was successfully updated in 0.031 seconds.
[AUDIT   ] CWWKT0016I: Web application available (default_host): http://localhost:9080/_MobileBrowserSimulator/
[AUDIT   ] CWWKZ0001I: Application _MobileBrowserSimulator started in 0.072 seconds.
[ERROR   ] SRVE0315E: An execption occurred: com.ibm.ws.webcontainer.webapp.WebAppErrorReport: javax.servlet.ServletException: Worklight Console initialization failed.Logged Exception: com.ibm.db2.jcc.am.SqlException: [jcc][10389][12245][3.67.27] Failure in loading native library db2jcct2, java.lang.UnsatisfiedLinkError: no db2jcct2 in java.library.path:  ERRORCODE=-4472, SQLSTATE=null
    at com.worklight.core.auth.impl.AuthenticationFilter.verifyServletInitialized(AuthenticationFilter.java:422)
    at com.worklight.core.auth.impl.AuthenticationFilter.doFilter(AuthenticationFilter.java:127)
    at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:194)
    at [internal classes]
Caused by: javax.servlet.ServletException: Worklight Console initialization failed.Logged Exception: com.ibm.db2.jcc.am.SqlException: [jcc][10389][12245][3.67.27] Failure in loading native library db2jcct2, java.lang.UnsatisfiedLinkError: no db2jcct2 in java.library.path:  ERRORCODE=-4472, SQLSTATE=null

Answer 1

Solved !!! No runtime configuratin can be found

I was getting this error because there were two instances of WL Admin Services(wladmin and worklightadmin) deployed in the server. Their configuration was added in server.xml twice, once by worklight installer and once by worklight server configuration tool. So my application was communicating to 'worklightadmin' Admin Service and WL Console communicating to 'wladmin' Admin Service. That's why there was no runtime available in the worklight console. Uninstalling 'wladmin' Admin Service from server by removing below snippet does the trick.

<!-- Declare the Worklight Administration Service application. -->
    <!-- <application id="wladmin" location="worklightadmin.war" name="wladmin" type="war">
        <application-bnd>
            <security-role name="worklightadmin">
                <user name="user"/>
            </security-role>

            <security-role name="worklightdeployer">
                <user name="user"/>
            </security-role>

            <security-role name="worklightmonitor">
                <user name="user"/>
            </security-role>

            <security-role name="worklightoperator">
                <user name="user"/>
            </security-role>

        </application-bnd>

        <classloader delegation="parentLast">
            <commonLibrary>

                <fileset dir="${wlp.install.dir}/lib" includes="com.ibm.ws.crypto.passwordutil_1.0.2.jar"/>
            </commonLibrary>
        </classloader>
    </application> -->

<!-- Declare the JNDI properties for the Worklight Administration Service. -->
    <!-- <jndiEntry jndiName="wladmin/ibm.worklight.admin.environmentid" value="WL_UAT_Server_Config"/> -->

Answer 2

The first error is "FWLSE0188E: Failed to connect to DB2 database : Cannot create PoolableConnectionFactory ([jcc][10389][12245][3.67.27] Failure in loading native library db2jcct2, java.lang.UnsatisfiedLinkError: no db2jcct2 in java.library.path: ERRORCODE=-4472, SQLSTATE=null) [project MyProjectName]"

The recommended way to configure the DB2 data sources in Worklight is to use the JDBC 4.0 driver (db2jcc4.jar and its associated license jar files in the same directory), as described in the Worklight documentation (here and here). It does not involve the use of native code libraries.

Therefore I would check the data source definition in Liberty's server.xml file.