J2CA0045E Error from WebSphere 6.1 in Solaris 10 - Websphere

This is a discussion on J2CA0045E Error from WebSphere 6.1 in Solaris 10 - Websphere ; Hi, Problem is in Solaris environment. Application works fine in Windows environment with connection pool size of 10. But in Solaris even pool size set to 100, this error occurs. WebSphere Version: 6.1.0.15 OS : Solaris 10 Database: Oracle 10g ...

+ Reply to Thread
Results 1 to 20 of 20

Thread: J2CA0045E Error from WebSphere 6.1 in Solaris 10

  1. J2CA0045E Error from WebSphere 6.1 in Solaris 10

    Hi,



    Problem is in Solaris environment. Application works fine in Windows environment with connection pool size of 10. But in Solaris even pool size set to 100, this error occurs.



    WebSphere Version: 6.1.0.15

    OS : Solaris 10

    Database: Oracle 10g



    J2CA0045E: Connection not available while invoking method createOrWaitForConnection for resource



    com.ibm.websphere.ce.j2c.ConnectionWaitTimeoutExce ption: Connection not available, Timed out waiting for 180004

    at com.ibm.ejs.j2c.FreePool.createOrWaitForConnection (FreePool.java:1499)

    at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.ja va:2287)

    at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrappe r(ConnectionManager.java:896)

    at com.ibm.ejs.j2c.ConnectionManager.allocateConnecti on(ConnectionManager.java:596 )

    at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConn ection(WSJdbcDataSource.java: 439)

    at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConn ection(WSJdbcDataSource.java: 408)

    ....................................



    Anybody else have this problem? Any solutions? Suggestions?



    Thank You.

  2. Re: J2CA0045E Error from WebSphere 6.1 in Solaris 10

    The error is that despite waiting 3 minutes the pool manager couldn't
    allocate a connection because all are in use.

    As such, OS won't make a difference.

    So, there is something weird and this should be helpful:
    http://www-1.ibm.com/support/docview...id=swg21217062.

    You could also take a thread dump (kill -3 PID) and see what the threads are
    doing (using the Thread Analyzer).


  3. Re: J2CA0045E Error from WebSphere 6.1 in Solaris 10


    > So, there is something weird and this should be helpful:
    > http://www-1.ibm.com/support/docview...id=swg21217062.


    This is even more exhaustive:
    http://www.redbooks.ibm.com/redpapers/pdfs/redp4080.pdf.


  4. Re: J2CA0045E Error from WebSphere 6.1 in Solaris 10

    acherian@bottomline.com wrote:
    > Hi,
    >
    >
    >
    > Problem is in Solaris environment. Application works fine in Windows environment with connection pool size of 10. But in Solaris even pool size set to 100, this error occurs.
    >
    >
    >
    > WebSphere Version: 6.1.0.15
    >
    > OS : Solaris 10
    >
    > Database: Oracle 10g
    >
    >
    >
    > J2CA0045E: Connection not available while invoking method createOrWaitForConnection for resource
    >
    >
    >
    > com.ibm.websphere.ce.j2c.ConnectionWaitTimeoutExce ption: Connection not available, Timed out waiting for 180004
    >
    > at com.ibm.ejs.j2c.FreePool.createOrWaitForConnection (FreePool.java:1499)
    >
    > at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.ja va:2287)
    >
    > at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrappe r(ConnectionManager.java:896)
    >
    > at com.ibm.ejs.j2c.ConnectionManager.allocateConnecti on(ConnectionManager.java:596 )
    >
    > at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConn ection(WSJdbcDataSource.java: 439)
    >
    > at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConn ection(WSJdbcDataSource.java: 408)
    >
    > ....................................
    >
    >
    >
    > Anybody else have this problem? Any solutions? Suggestions?
    >
    >
    >
    > Thank You.
    >

    Check with your DBA...you may be running into licensing issues w/ Oracle
    that limits the number of connections the db server allows.
    Ken

  5. Re: J2CA0045E Error from WebSphere 6.1 in Solaris 10

    Database side is already checked. DB is allowing more connections and connection pool has those connection. But all of the connections in the pool is in use. When pool size reaches maximum application fails.

  6. Re: J2CA0045E Error from WebSphere 6.1 in Solaris 10

    I have seen the documents you mentioned. Using that information I was able to find the problem. But I couldn't find a solution.

  7. Re: J2CA0045E Error from WebSphere 6.1 in Solaris 10

    So, what you know at this stage is that all connections are busy when
    running on a given server but not on another.

    What is a very possible cause is that there is some sort of misconfiguration
    or unexpected condition when running on the Solaris 10 box.

    This causes uncaught exception and the application runs out of the "normal"
    code path and doesn't close the connection correctly (typically, no finally
    clause or no connection.close() in the finally clause). This results in a
    connection leak.

    When on the "good" server, there is no exception and thus no leak.

    Code review is the way to go. I would also review a thread dump to see
    what's happening in the application.


  8. Re: J2CA0045E Error from WebSphere 6.1 in Solaris 10

    acherian@bottomline.com wrote:
    > Database side is already checked. DB is allowing more connections and connection pool has those connection. But all of the connections in the pool is in use. When pool size reaches maximum application fails.
    >

    What happens when you increase the number of max connections in the pool?
    One thing you might look at in TPV or another tool is how long each
    connection is in use, there may be a programming issue here.


    Ken

  9. Re: J2CA0045E Error from WebSphere 6.1 in Solaris 10

    acherian@bottomline.com wrote:
    > I have seen the documents you mentioned. Using that information I was able to find the problem. But I couldn't find a solution.


    I'm not sure if you mean the stack trace you already posted or something else.
    That stack trace is just the symptoms, not the problem, right?

    I mean, were you able to determine from the thread dump where all the threads
    were that were currently holding the connections from the pool? Although, I'd
    think that 100 connections would be larger than your WAS thread pool... (just
    thinking out loud)

    A few thoughts. If you're not using a pure-java JDBC driver, I could see the
    Windows and Solaris drivers behaving differently. i.e. one of them having a bug.

    Also, are your Windows and Solaris machines in the same network? We've seen
    problems with Oracle connection pools connecting through a firewall, although
    the symptom we've seen was a bit different.

    Have you enabled the PMI monitoring of the connection pool and looked at the
    trends?

    --
    Doug

  10. Re: J2CA0045E Error from WebSphere 6.1 in Solaris 10

    If the connection pool size set to unlimited, there is no ConnectionWaitTimeoutException. But the pool size is getting upto 200.



    If the issue is with application, then how come it is running with less than 10 connections in Windows environment?

  11. Re: J2CA0045E Error from WebSphere 6.1 in Solaris 10

    Doug,



    I mean, as far as my application is concerned the ConnectionWaitTimeoutException is the problem. But you are right; it is just the symptoms. I am assuming the problem is somewhere in WebSphere.



    The JDBC drivers in use came with Oracle install. Both Windows and Solaris machines in the same network.



    From the traces of Connection Manager, I can see connections are in use for more than 100 seconds.



    Alex

  12. Re: J2CA0045E Error from WebSphere 6.1 in Solaris 10

    acherian@bottomline.com wrote:
    > Doug,
    >
    >
    >
    > I mean, as far as my application is concerned the ConnectionWaitTimeoutException is the problem. But you are right; it is just the symptoms. I am assuming the problem is somewhere in WebSphere.
    >
    >
    >
    > The JDBC drivers in use came with Oracle install. Both Windows and Solaris machines in the same network.
    >
    >
    >
    > From the traces of Connection Manager, I can see connections are in use for more than 100 seconds.
    >
    >
    >
    > Alex
    >

    100 seconds!!!!

    Every thread upstream from the database will be waiting for this to
    complete. HTTP Server threads will be waiting, Web Container threads
    will be waiting, etc. You'll definitely need to bump up the number of
    connections in the pool and quite likely the numbers of worker threads
    upstream.

    Ken

  13. Re: J2CA0045E Error from WebSphere 6.1 in Solaris 10

    acherian@bottomline.com wrote:
    > If the connection pool size set to unlimited, there is no ConnectionWaitTimeoutException. But the pool size is getting upto 200.
    >
    >
    >
    > If the issue is with application, then how come it is running with less than 10 connections in Windows environment?
    >

    We've got two discussion threads going here, but does your Windows
    environment match exactly the AIX environment? Still 100 seconds average
    use time?

    Ken

  14. Re: J2CA0045E Error from WebSphere 6.1 in Solaris 10

    Ken,



    The AIX environment also shown the same exception. I didn't investigate it further. Right now focusing on Solaris environment.



    Alex

  15. Re: J2CA0045E Error from WebSphere 6.1 in Solaris 10

    Ken,



    If the ConnectionManager in Websphere says the state is SATE_TRAN_WRAPPER_INUSE, is that mean it is actively communicating with database? Or in the connection pool it is not free?



    Here is one trace message



    5/6/08 12:01:39:547 EDT 00000018 ConnLeakLogic 3 MCWrapper id 6cbd9138 Managed connection WSRdbManagedConnectionImpl@11892b4e State:STATE_TRAN_WRAPPER_INUSE Thread Id: 88ffafa Thread Name: server.start

    up : 0 Handle count 0

    in-use for 181051ms

    java.lang.Throwable

    at com.ibm.ejs.j2c.ConnectionManager.allocateConnecti on(ConnectionManager.java:762 )

    at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConn ection(WSJdbcDataSource.java: 439)

    at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConn ection(WSJdbcDataSource.java: 408)



    .................................................. ........

    at javax.servlet.GenericServlet.init(GenericServlet.j ava:256)

    at com.ibm.ws.webcontainer.servlet.ServletWrapper.ini t(ServletWrapper.java:192)

    at com.ibm.ws.wswebcontainer.servlet.ServletWrapper.i nit(ServletWrapper.java:319)
    at com.ibm.ws.webcontainer.servlet.ServletWrapper.ini tialize(ServletWrapper.java:1 221)

    at com.ibm.ws.wswebcontainer.servlet.ServletWrapper.i nitialize(ServletWrapper.java :152)

    at com.ibm.wsspi.webcontainer.extension.WebExtensionP rocessor.createServletWrapper (WebExtensionProcessor.java:99)

    at com.ibm.ws.webcontainer.webapp.WebApp.getServletWr apper(WebApp.java:828)

    at com.ibm.ws.webcontainer.webapp.WebApp.initializeTa rgetMappings(WebApp.java:483)

    at com.ibm.ws.webcontainer.webapp.WebApp.commonInitia lizationFinish(WebApp.java:32 0)

    at com.ibm.ws.wswebcontainer.webapp.WebApp.initialize (WebApp.java:290)

    at com.ibm.ws.wswebcontainer.webapp.WebGroup.addWebAp plication(WebGroup.java:90)
    at com.ibm.ws.wswebcontainer.VirtualHost.addWebApplic ation(VirtualHost.java:157)
    at com.ibm.ws.wswebcontainer.WebContainer.addWebApp(W ebContainer.java:665)

    at com.ibm.ws.wswebcontainer.WebContainer.addWebAppli cation(WebContainer.java:618)

    at com.ibm.ws.webcontainer.component.WebContainerImpl .install(WebContainerImpl.jav a:335)

    at com.ibm.ws.webcontainer.component.WebContainerImpl .start(WebContainerImpl.java: 551)

    at com.ibm.ws.runtime.component.ApplicationMgrImpl.st art(ApplicationMgrImpl.java:1 303)

    at com.ibm.ws.runtime.component.DeployedApplicationIm pl.fireDeployedObjectStart(De ployedApplicationImpl.java:1138)

    at com.ibm.ws.runtime.component.DeployedModuleImpl.st art(DeployedModuleImpl.java:5 69)

    at com.ibm.ws.runtime.component.DeployedApplicationIm pl.start(DeployedApplicationI mpl.java:817)

    at com.ibm.ws.runtime.component.ApplicationMgrImpl.st artApplication(ApplicationMgr Impl.java:949)

    at com.ibm.ws.runtime.component.ApplicationMgrImpl$Ap pInitializer.run(ApplicationM grImpl.java:2122)

    at com.ibm.wsspi.runtime.component.WsComponentImpl$_A synchInitializer.run(WsCompon entImpl.java:342)

    at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.j ava:1473)

  16. Re: J2CA0045E Error from WebSphere 6.1 in Solaris 10

    acherian@bottomline.com wrote:
    > Doug,
    >
    >
    >
    > I mean, as far as my application is concerned the ConnectionWaitTimeoutException is the problem. But you are right; it is just the symptoms. I am assuming the problem is somewhere in WebSphere.
    >
    >
    >
    > The JDBC drivers in use came with Oracle install. Both Windows and Solaris machines in the same network.
    >
    >
    >
    > From the traces of Connection Manager, I can see connections are in use for more than 100 seconds.


    Do you actually have requests that take 100 seconds to complete? What on
    earth are you doing for all that time?

  17. Re: J2CA0045E Error from WebSphere 6.1 in Solaris 10

    acherian@bottomline.com wrote:
    > Ken,
    >
    >
    >
    > If the ConnectionManager in Websphere says the state is SATE_TRAN_WRAPPER_INUSE, is that mean it is actively communicating with database? Or in the connection pool it is not free?
    >
    >
    >
    > Here is one trace message
    >
    >
    >
    > 5/6/08 12:01:39:547 EDT 00000018 ConnLeakLogic 3 MCWrapper id 6cbd9138 Managed connection WSRdbManagedConnectionImpl@11892b4e State:STATE_TRAN_WRAPPER_INUSE Thread Id: 88ffafa Thread Name: server.start
    >
    > up : 0 Handle count 0
    >
    >

    No idea, you might need to open a PMR with IBM.

    Ken

  18. Re: J2CA0045E Error from WebSphere 6.1 in Solaris 10

    Java Virtual

    Machine



    under Generic JVM arguments

    -DdisablePK54589=true

  19. Re: J2CA0045E Error from WebSphere 6.1 in Solaris 10

    Thank You. Everything working well now.

  20. Re: J2CA0045E Error from WebSphere 6.1 in Solaris 10

    hi all,

    I have the same problem with a AIX , please could you tell me what was the solution for this problem, may be is a possible solution for my problem


    greetings
    Marco

+ Reply to Thread