isUserInRole return incorrect under load

iProblem Summary:
- HttpServletRequest@.isUserInRole(String):boolean
returns incorrect answer under load after time.

Problem Description:
- In our own application, we were seeing that after a
load test runs for a while (anywhere between 5 minutes
and 2 hours), our tests would start failing since users
who were supposed to be able to access a certain URL no
longer were able to. Other users were fine, but those
specific users would continue to have the problem from
that point on.
- We found out it was because isUserInRole was
returning the wrong boolean.
- Our application uses a mix of declarative and
programatic security. We protected an entire section
of the app with a single role called "DEFAULT" which is
defined in web.xml. This enforces users to sign in.
From that point on, due to the nature of our app, we
need to programatically call isUserInRole. This is
done is a central location since all pages of the site
are fronted with a single controller servlet. We
depended on our custom LoginModule to return the role
sets per a JBoss example, and depended on Tomcat to
return isUserInRole properly based on those role sets.
- We didn't know if it was our code, Tomcat's or
JBoss', so we decided to create a very simple test app.
We saw the same problem occur under each of the
following JAAS auth.conf scenarios:
- Using our login module (necessarily behing JBoss'
ProxyLoginModule).
- Using JBoss' DatabaseServerLoginModule behind the
JBoss ProxyLoginModule.
- Using JBoss' DatabaseServletLoginModule by itself.

Environment:
OS: Solaris 7
(SunOS docs250 5.7 Generic_106541-18 sun4u sparc
SUNW,Ultra-250)
DB: Oracle 8.1.5 Standard
JDK: Sun Java2, version 1.3.1_02 for Solaris w/ all
recommended OS patches
JDBC Driver: Oracle 8.1.7 JDBC Thin Driver
Web Server: Apache 1.3.12 w/ mod_ssl and mod_jk (using
AJP12 protocol)
App Server: JBoss 2.4.3 w/ integrated Jakarta Tomcat 3.2.3
LoadTesting: Empirix e-Load, version 6.00.0695
Build Tool: Jakarta Ant 1.4.1 (you'll need this to
build the test case)

Architecture:
- Separate physical web server and app server boxes.
- Database running on same box as app server.
- HTTP requests come in (NOT over SSL) to web server,
who relays the requests (using mod_jk) to the app server.
- Load boxes running on same internal network as web
and app servers.
- No firewall.

Test Case Data:
- 10 different application users in the database:
user0, user1, user2, user3, user4, user5, user6,
user7, user8, user9 - 11 different roles in the database:
user0, user1, user2, user3, user4, user5, user6,
user7, user8, user9, DEFAULT
- Each user has 2 roles:
<username>, DEFAULT
example: user0 has roles user0, DEFAULT

Load Scenario:
- Recorded 10 different scripts, each using a
different application user (user0-user9) and password
(same as username).
- Created load using each of the 10 scripts X 20
virtual users (VUs) each, totalling 200 concurrent VUs.
- Scripts were played back with recorded think time.
- Each virtual user's settings:
- Browser emulation: IE 5.5
- Thick client
- Run VUs in separate processors (this makes sure
each VU gets its own
HTTPSession).
- The 200 VUs were spread evenly across 3 separate
Windows 2000 Professional boxes, one of which was also
the e-Load controller.

Test Case Scenario:
- Each of the ten scripts takes the following steps:
1. Access the testapp home page:
http://webserver/rolestest/index.jsp
2. Click on the "Protected" link to: protected/index.jsp
- The /protected/* url-pattern is controlled in
web.xml such that the user needs to be in the "DEFAULT"
role to access that area.
3. The webcontainer displays the login page (standard
j_security_check).
4. The user logins in.
5. The webcontainer displays the protected/index.jsp page.
- The page iterates over all knows roles in the
system. This was set in the servlet context by the
RolesTestServlet who queried the DB for all the roles
on load-on-startup.
- The page calls request.isUserInRole(String) for
each known role.
- If isUserInRole returns true for "DEFAULT" or the
user's own role (ie: user0 has roles "DEFAULT" and
"user0"), the page says "Good" for each.
- If isUserInRole returns false for one of those 2
roles, OR if isUserInRole returns true for a role a
user shouldn't have, the page logs the error to
System.err and redirects the user to an error page. 6.
If the user didn't end up on the error page, the
scripted user clicks on "Logout".
- The logout.jsp page invalidates the user's session
and redirects him to the home page.
7. The webcontainer displays the home page.

Problem Notes:
- The test case differs from our real app in that our
real app calls isUserInRole from a central control
servlet. The test app calls isUserInRole from a jsp
page's implicit request object. The problem appears in
both places so we don't think it matters where
isUserInRole is called from.
- We can't exhibit the problem without load.
- Sometimes the problem shows itself after 5 minutes,
sometimes not until after a couple hours; definately
within 4 hours.
- We HAVE seen the problem with a standalone "real"
browser, but only when there is VU load on the system.
This requires logging in as a username that the log
has already said is having a problem. After we stop
the test, the application user in our browser suddenly
stops having problems.
- We NEVER saw that a user couldn't access
/protected/. This means that each VU was always given
the "DEFAULT" role. However, since the mechanism for
giving a user "DEFAULT" is the same as giving him his
other role, how come there's a problem? Synchronization?
- We HAVE seen that sometimes not only do users not
get the roles they should, but they also will get
another users' roles. This really sounds like a
synchronization problem.
- Error lines in the console/console-log will look
have this text in it:
Should have gotten <b>user3</b> Role but didn't = BAD!!!
and
Got <b>user2</b> Role - BAD!!!

Running the Test App:
1. Make sure you have $JAVA_HOME set properly to Sun
JDK 1.3.1_02.
2. Make sure you have $ANT_HOME set properly to Ant
1.4.1+.
3. Make sure you have $PATH set properly to
$JAVA_HOME/bin:$ANT_HOME/bin:$PATH
4. Extract rolestest.zip
5. cd into rolestest.
6. cp etc/auth.conf contents into your
jboss-2.4.3_tomcat-3.2.3/jboss/conf/tomcat/auth.conf file
7. Init your database with etc/init.sql using sqlplus,
toad, etc.
8. Edit build.xml and change the following properties:
- jboss2.home to your jboss-2.4.3_tomcat-3.2.3/jboss
directory.
- tomcat3.home to your
jboss-2.4.3_tomcat-3.2.3/tomcat directory.
9. type: ant deploy
10. cd to jboss-2.4.3_tomcat-3.2.3/jboss/conf/tomcat/
11. edit jboss.jcml and have DefaultDS point to the
database that was
init'd via init.sql.
11. cd to jboss-2.4.3_tomcat-3.2.3/jboss/bin
12. type: ./run_with_tomcat.sh
13. cd to your httpd.conf or mod_jk.conf and add:
JkMount /rolestest/* ajp12
14. restart apache
15. surf to http://hostname/rolestest/ and run through
the app once as user0/user0 to verify everything's set
up properly
16. Record test scripts using a load testing tool and
run a load test as described in "Test Case Scenario" above.
17. Check your jboss console. Eventually, you should
see the problem occur.

Thanks for all your help,
David