Issues about KeepAlive between Apache Plug-In and WebLogic Server

by AjitSaturday, March 15, 20140 Comments

Sometime after WebLogic Server is running for a while, the network administrator will see there are many connections staying in FIN_WAIT2 state on the WLS side and CLOSE_WAIT state on the Apache side.
Customer environment includes 50+ WLS instances and several Apache servers, as well as some F5 load balancers between WLS and Apache.
The Apache httpd.conf include the following lines:

...
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 30
...
<IfModule mod_weblogic.c>
WebLogicHost aa.b.c.ddd
WebLogicPort 80
MatchExpression *
KeepAliveEnabled ON
KeepAliveSecs 30
</IfModule>

Cause

After checking the httpd.conf, we saw that both the Apache keepalive and the WLS keep alive are enabled. Because of this configuration, those connections were kept in that status.
We could refer to such a condition:
  • Apache process p1 receives a request
  • p1 establishes a connection c1 to WLS s1 via plugin.
  • p1 sends request to s1 and gets results.
  • Apache process p1 returns a page to browser.
  • According to the Apache KeepAlive configuration, p1 waits for the request until timeout.
  • At the same time, c1 is still active.
  • p1 closes the connection from the browser and receives a new request from a different client.
  • p1 establishes a connection c2 to WLS s2 via plugin.
At this moment, c1 times out, s1 sends SYN to Apache, the system where Apache is run acknowledges an ACK, but because p1 is dealing with other request, p1 does not respond to the FIN. Then the state of the connection in the Apache side is CLOSE_WAIT, and the state in the WLS side is FIN_WAIT2.


In addition, there is another question: why cannot the c2 connection be reused? This question concerns the MPM of Apache. MPM means multi-processing modules: refer to http://httpd.apache.org/docs/2.0/mpm.html for more details. There are two MPMs: one is prefork, and the other is worker. On most platforms, prefork is the default MPM. That means that one request is linked to one process.

Solution

Two solutions are provided here:
  1. For the problem with connections getting stuck, disable the WLS plug-in keep-alive.
  2. To enable connections to be reused, switch the Apache MPM from prefork to worker.

0 Comments