I have two tomcat servers, one with OpenAM 12, one with the main web applications. Known: HTML pages authenticate without issue, programmatic login using the OpenAM java SDK works from within a servlet.
What fails is a call to HttpServletRequest.login(username, password)
This is the entry I have within the TomEE+ server:
<Realm className="com.sun.identity.agents.tomcat.v6.AmTomcatRealm" debug="99"/>
Here is the error I see within the AM J2EEAgent debug log:
amRealm:01/29/2014 02:29:47:497 PM EST: Thread[http-bio-443-exec-3,5,main]
SSOTokenValidator: validate failed with exception
[AgentException Stack]
com.sun.identity.agents.arch.AgentException: Invalid transport string
at com.sun.identity.agents.util.TransportToken.initializeFromString(TransportToken.java:135)
at com.sun.identity.agents.util.TransportToken.<init>(TransportToken.java:115)
at com.sun.identity.agents.common.SSOTokenValidator.validate(SSOTokenValidator.java:99)
at com.sun.identity.agents.realm.AmRealm.authenticate(AmRealm.java:143)
at com.sun.identity.agents.tomcat.v6.AmTomcatRealm.authenticate(AmTomcatRealm.java:106)
at org.apache.catalina.realm.CombinedRealm.authenticate(CombinedRealm.java:146)
at org.apache.tomee.catalina.TomEERealm.authenticate(TomEERealm.java:43)
at org.apache.catalina.authenticator.AuthenticatorBase.doLogin(AuthenticatorBase.java:818)
at org.apache.catalina.authenticator.AuthenticatorBase.login(AuthenticatorBase.java:800)
at org.apache.catalina.connector.Request.login(Request.java:2621)
at org.apache.catalina.connector.RequestFacade.login(RequestFacade.java:1065)
In case of AgentRealm the 'password' is not the password, but the SSOTokenId from the SSO session.
The request#login fails for the username/password combination, because the agent defines its own Tomcat Realm definition (AmTomcatRealm) in the server.xml. The agent's realm does not use username/password combination for validating users as the credentials shouldn't even be present there (OpenAM performs the authentication and afterwards the password is long gone before you reach the actual protected application), it is actually a bit worrisome that you still have access to the user's password.
Since the agent should never actually know the user's password, it uses a transport string, that contains several information (amongst the session ID), it has a quite undocumented format, here is some link to the relevant source code: https://svn.forgerock.org/openam/trunk/openam-agents/jee-agents/jee-agents-sdk/src/main/java/com/sun/identity/agents/util/TransportToken.java
Also, this is not a chicken-egg problem, as it is assumed that the JAAS login is performed by the agent itself, and it isn't really expected that the application would invoke login in the first place. In the agent settings (and in web.xml of course - see container documentation) there are ways to enable JAAS form login support and then the agent can take care of the JAAS integration without the need of knowing anything about the proprietary transport string format.