Weblogic very very very slow in Production - Weblogic

This is a discussion on Weblogic very very very slow in Production - Weblogic ; Hi, Our weblogic 7.0 sp4 production is down on its knees. We have tried all possible tunings that we could do but it just won't have any affect. Our platform info: Platform -------- Sun Solaris 5.8, DB server MS SQL ...

+ Reply to Thread
Results 1 to 12 of 12

Thread: Weblogic very very very slow in Production

  1. Weblogic very very very slow in Production

    Hi,

    Our weblogic 7.0 sp4 production is down on its knees. We have tried all possible tunings that we could do but it just won't have any affect. Our platform info:

    Platform
    --------
    Sun Solaris 5.8, DB server MS SQL 2000, Weblogic7 sp4 with JVM 1.3.1_08

    Here's what we have already performed:

    1. Our app works well when there is no load on the system with very little response time.

    2. When the load is increased to probably 1 user with 30 iterations (or when there are 60 active users on production) weblogic server totally freaks out and down on its knees.

    3. We ran profiler on application and there is no apparent bottleneck in the app that we could pin point. Also the max time any component in the app took was below 600ms (thats the max under peak load)

    4. We checked JDBC connections and the average response time is 30ms under max load

    5. We checked the O.S. with guidelines provided as per(http://e-docs.bea.com/wls/docs81/perform/topten.html)

    6. We changed the hardware completely and went on to a brand new Sun sparc box. No change.

    7. Now we have not played around with execute threads as I can see various stories on the forum for various scenarios. But my config is 50 Execute threads under DEFAULT queue. Under peak load it goes down to 0 idle threads.

    Please can anyone help me how to find best possible way to tune this. Doing a trial and error method by watching maximum throughput is the last thing we want to do as this a very highly visible production application.

    Also if there are any other suggestions by any guru out there will really be appreciated!

    Thanks in advance.
    Galoch

  2. Re: Weblogic very very very slow in Production

    1. What is the broad application architecture (servlet->ejb or servlet->jdbc or jms->mdb->jdbc)?
    2. What JDBC driver are you using?
    3. what are your connection pool settings?
    4. Are you using the PreparedStatementCache?
    5. Are you using XA?
    6. Does your application make requests through a socket back into the server? (An example of this is calls made from one application to another on the same server)
    7. Does your application have security permissions configured?


    Arunabh

  3. Re: Weblogic very very very slow in Production

    Do you thread dump with "kill -3 pid" command where pid is your
    weblogic server process id. This gives you the status of all weblogic
    threads that point in time. Thread dump goes to standard out. Makesure
    you are logging your standard out before you the thread dump.

    Usually i will do about three seperate thread dumps with about 5 second
    delay. This way you can see if anyof the theads is blocking any resouce
    for too long.


  4. Re: Weblogic very very very slow in Production

    Have you created a thread dump and attempted to analyze that? I could
    pinpoint the problem, precisely.

    Bill

    "Pavnit Bhatia" wrote in message
    news:3491217.1105377287056.JavaMail.root@jserv5...
    > Hi,
    >
    > Our weblogic 7.0 sp4 production is down on its knees. We have tried all

    possible tunings that we could do but it just won't have any affect. Our
    platform info:
    >
    > Platform
    > --------
    > Sun Solaris 5.8, DB server MS SQL 2000, Weblogic7 sp4 with JVM 1.3.1_08
    >
    > Here's what we have already performed:
    >
    > 1. Our app works well when there is no load on the system with very little

    response time.
    >
    > 2. When the load is increased to probably 1 user with 30 iterations (or

    when there are 60 active users on production) weblogic server totally freaks
    out and down on its knees.
    >
    > 3. We ran profiler on application and there is no apparent bottleneck in

    the app that we could pin point. Also the max time any component in the app
    took was below 600ms (thats the max under peak load)
    >
    > 4. We checked JDBC connections and the average response time is 30ms under

    max load
    >
    > 5. We checked the O.S. with guidelines provided as

    per(http://e-docs.bea.com/wls/docs81/perform/topten.html)
    >
    > 6. We changed the hardware completely and went on to a brand new Sun sparc

    box. No change.
    >
    > 7. Now we have not played around with execute threads as I can see various

    stories on the forum for various scenarios. But my config is 50 Execute
    threads under DEFAULT queue. Under peak load it goes down to 0 idle threads.
    >
    > Please can anyone help me how to find best possible way to tune this.

    Doing a trial and error method by watching maximum throughput is the last
    thing we want to do as this a very highly visible production application.
    >
    > Also if there are any other suggestions by any guru out there will really

    be appreciated!
    >
    > Thanks in advance.
    > Galoch




  5. Re: Weblogic very very very slow in Production

    Below are my answers.
    Thanks

    1. What is the broad application architecture (servlet->ejb or servlet->jdbc or jms->mdb->jdbc)?
    War file with Struts calling DAO classes with JDBC
    2. What JDBC driver are you using?
    Weblogic SQL server (weblogic.mssqlserver4)
    3. what are your connection pool settings?
    100 max, 25 initial, 10 inc (this never goes beyond 10 though)
    4. Are you using the PreparedStatementCache?
    No
    5. Are you using XA?
    No
    6. Does your application make requests through a socket back into the server? (An example of this is calls made from one application to another on the same server)
    No
    7. Does your application have security permissions configured?
    Yes. We use JAAS security implementation that connects to our DB for user authorization.

    Here's the partial thread dump. 50 threads are configured and most of them are showing Waiting for Monitor Entry. Following 2 stacks are common all through 50 threads. There is a 3rd stack i have pasted below that with Execute thread 0 which is different :

    INFO | jvm 1 | 2005/01/11 09:32:53 | [java] "ExecuteThread: '15' for queue: 'default'" daemon prio=5 tid=0x4038a0 nid=0x24 waiting for monitor entry [0xe1a00000..0xe1a01a00]

    INFO | jvm 1 | 2005/01/11 09:32:53 | [java] at weblogic.servlet.jsp.JspStub.checkForReload(JspStu b.java:145)

    INFO | jvm 1 | 2005/01/11 09:32:53 | [java] at weblogic.servlet.internal.ServletStubImpl.getServl et(ServletStubImpl.java:538)

    INFO | jvm 1 | 2005/01/11 09:32:53 | [java] at weblogic.servlet.internal.ServletStubImpl.invokeSe rvlet(ServletStubImpl.java:364)

    INFO | jvm 1 | 2005/01/11 09:32:53 | [java] at weblogic.servlet.internal.TailFilter.doFilter(Tail Filter.java:20)

    INFO | jvm 1 | 2005/01/11 09:32:53 | [java] at weblogic.servlet.internal.FilterChainImpl.doFilter (FilterChainImpl.java:27)

    INFO | jvm 1 | 2005/01/11 09:32:53 | [java] at weblogic.servlet.internal.RequestDispatcherImpl.fo rward(RequestDispatcherImpl.java:315)

    INFO | jvm 1 | 2005/01/11 09:32:53 | [java] at org.apache.struts.action.RequestProcessor.doForwar d(RequestProcessor.java:1069)

    INFO | jvm 1 | 2005/01/11 09:32:53 | [java] at org.apache.struts.tiles.TilesRequestProcessor.doFo rward(TilesRequestProcessor.java:274)

    INFO | jvm 1 | 2005/01/11 09:32:53 | [java] at org.apache.struts.tiles.TilesRequestProcessor.proc essTilesDefinition(TilesRequestProcessor.java:254)

    INFO | jvm 1 | 2005/01/11 09:32:53 | [java] at org.apache.struts.tiles.TilesRequestProcessor.proc essForwardConfig(TilesRequestProcessor.java:309)

    INFO | jvm 1 | 2005/01/11 09:32:53 | [java] at org.apache.struts.action.RequestProcessor.process( RequestProcessor.java:279)

    INFO | jvm 1 | 2005/01/11 09:32:53 | [java] at org.apache.struts.action.ActionServlet.process(Act ionServlet.java:1482)

    INFO | jvm 1 | 2005/01/11 09:32:53 | [java] at org.apache.struts.action.ActionServlet.doPost(Acti onServlet.java:525)

    INFO | jvm 1 | 2005/01/11 09:32:53 | [java] at javax.servlet.http.HttpServlet.service(HttpServlet .java:760)

    INFO | jvm 1 | 2005/01/11 09:32:53 | [java] at javax.servlet.http.HttpServlet.service(HttpServlet .java:853)

    INFO | jvm 1 | 2005/01/11 09:32:53 | [java] at weblogic.servlet.internal.ServletStubImpl$ServletI nvocationAction.run(ServletStubImpl.java:1075)

    INFO | jvm 1 | 2005/01/11 09:32:53 | [java] at weblogic.servlet.internal.ServletStubImpl.invokeSe rvlet(ServletStubImpl.java:418)

    INFO | jvm 1 | 2005/01/11 09:32:53 | [java] at weblogic.servlet.internal.TailFilter.doFilter(Tail Filter.java:20)

    INFO | jvm 1 | 2005/01/11 09:32:53 | [java] at weblogic.servlet.internal.FilterChainImpl.doFilter (FilterChainImpl.java:27)

    INFO | jvm 1 | 2005/01/11 09:32:53 | [java] at com.wfb.util.SessionValidationFilter.doFilter(Sess ionValidationFilter.java:83)

    INFO | jvm 1 | 2005/01/11 09:32:53 | [java] at weblogic.servlet.internal.FilterChainImpl.doFilter (FilterChainImpl.java:27)

    INFO | jvm 1 | 2005/01/11 09:32:53 | [java] at weblogic.servlet.internal.WebAppServletContext$Ser vletInvocationAction.run(WebAppServletContext.java :5523)

    INFO | jvm 1 | 2005/01/11 09:32:54 | [java] at weblogic.security.service.SecurityServiceManager.r unAs(SecurityServiceManager.java:685)

    INFO | jvm 1 | 2005/01/11 09:32:54 | [java] at weblogic.servlet.internal.WebAppServletContext.inv okeServlet(WebAppServletContext.java:3156)

    INFO | jvm 1 | 2005/01/11 09:32:54 | [java] at weblogic.servlet.internal.ServletRequestImpl.execu te(ServletRequestImpl.java:2506)

    INFO | jvm 1 | 2005/01/11 09:32:54 | [java] at weblogic.kernel.ExecuteThread.execute(ExecuteThrea d.java:234)

    INFO | jvm 1 | 2005/01/11 09:32:54 | [java] at weblogic.kernel.ExecuteThread.run(ExecuteThread.ja va:210)

    INFO | jvm 1 | 2005/01/11 09:32:54 |

    INFO | jvm 1 | 2005/01/11 09:32:54 | [java] "ExecuteThread: '14' for queue: 'default'" daemon prio=5 tid=0x246a80 nid=0x23 waiting for monitor entry [0xe1b00000..0xe1b01a00]

    INFO | jvm 1 | 2005/01/11 09:32:54 | [java] at weblogic.servlet.internal.ServletStubImpl.invokeSe rvlet(ServletStubImpl.java:483)

    INFO | jvm 1 | 2005/01/11 09:32:54 | [java] at weblogic.servlet.internal.TailFilter.doFilter(Tail Filter.java:20)

    INFO | jvm 1 | 2005/01/11 09:32:54 | [java] at weblogic.servlet.internal.FilterChainImpl.doFilter (FilterChainImpl.java:27)

    INFO | jvm 1 | 2005/01/11 09:32:54 | [java] at weblogic.servlet.internal.RequestDispatcherImpl.fo rward(RequestDispatcherImpl.java:315)

    INFO | jvm 1 | 2005/01/11 09:32:54 | [java] at org.apache.struts.action.RequestProcessor.doForwar d(RequestProcessor.java:1069)

    INFO | jvm 1 | 2005/01/11 09:32:54 | [java] at org.apache.struts.tiles.TilesRequestProcessor.doFo rward(TilesRequestProcessor.java:274)

    INFO | jvm 1 | 2005/01/11 09:32:54 | [java] at org.apache.struts.tiles.TilesRequestProcessor.proc essTilesDefinition(TilesRequestProcessor.java:254)

    INFO | jvm 1 | 2005/01/11 09:32:54 | [java] at org.apache.struts.tiles.TilesRequestProcessor.proc essForwardConfig(TilesRequestProcessor.java:309)

    INFO | jvm 1 | 2005/01/11 09:32:54 | [java] at org.apache.struts.action.RequestProcessor.process( RequestProcessor.java:279)

    INFO | jvm 1 | 2005/01/11 09:32:54 | [java] at org.apache.struts.action.ActionServlet.process(Act ionServlet.java:1482)

    INFO | jvm 1 | 2005/01/11 09:32:54 | [java] at org.apache.struts.action.ActionServlet.doGet(Actio nServlet.java:507)

    INFO | jvm 1 | 2005/01/11 09:32:54 | [java] at javax.servlet.http.HttpServlet.service(HttpServlet .java:740)

    INFO | jvm 1 | 2005/01/11 09:32:54 | [java] at javax.servlet.http.HttpServlet.service(HttpServlet .java:853)

    INFO | jvm 1 | 2005/01/11 09:32:54 | [java] at weblogic.servlet.internal.ServletStubImpl$ServletI nvocationAction.run(ServletStubImpl.java:1075)

    INFO | jvm 1 | 2005/01/11 09:32:54 | [java] at weblogic.servlet.internal.ServletStubImpl.invokeSe rvlet(ServletStubImpl.java:418)

    INFO | jvm 1 | 2005/01/11 09:32:54 | [java] at weblogic.servlet.internal.TailFilter.doFilter(Tail Filter.java:20)

    INFO | jvm 1 | 2005/01/11 09:32:54 | [java] at weblogic.servlet.internal.FilterChainImpl.doFilter (FilterChainImpl.java:27)

    INFO | jvm 1 | 2005/01/11 09:32:54 | [java] at com.wfb.util.SessionValidationFilter.doFilter(Sess ionValidationFilter.java:83)

    INFO | jvm 1 | 2005/01/11 09:32:54 | [java] at weblogic.servlet.internal.FilterChainImpl.doFilter (FilterChainImpl.java:27)

    INFO | jvm 1 | 2005/01/11 09:32:54 | [java] at weblogic.servlet.internal.WebAppServletContext$Ser vletInvocationAction.run(WebAppServletContext.java :5523)

    INFO | jvm 1 | 2005/01/11 09:32:54 | [java] at weblogic.security.service.SecurityServiceManager.r unAs(SecurityServiceManager.java:685)

    INFO | jvm 1 | 2005/01/11 09:32:54 | [java] at weblogic.servlet.internal.WebAppServletContext.inv okeServlet(WebAppServletContext.java:3156)

    INFO | jvm 1 | 2005/01/11 09:32:54 | [java] at weblogic.servlet.internal.ServletRequestImpl.execu te(ServletRequestImpl.java:2506)

    INFO | jvm 1 | 2005/01/11 09:32:54 | [java] at weblogic.kernel.ExecuteThread.execute(ExecuteThrea d.java:234)

    INFO | jvm 1 | 2005/01/11 09:32:54 | [java] at weblogic.kernel.ExecuteThread.run(ExecuteThread.ja va:210)

    INFO | jvm 1 | 2005/01/11 09:32:54 |



    3rd Thread


    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] "ExecuteThread: '0' for queue: 'default'" daemon prio=5 tid=0x917838 nid=0x15 runnable [0xe2900000..0xe2901a00]

    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at java.util.zip.ZipFile.getEntry(Native Method)

    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at java.util.zip.ZipFile.getEntry(ZipFile.java:142)

    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.WarClassFinder.getSource (WarClassFinder.java:183)

    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.WarClassFinder.getSource (WarClassFinder.java:102)

    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.utils.classloaders.MultiClassFinder.getSo urce(MultiClassFinder.java:63)

    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.utils.classloaders.MultiClassFinder.getSo urce(MultiClassFinder.java:63)

    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.utils.classloaders.MultiClassFinder.getCl assSource(MultiClassFinder.java:55)

    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.utils.classloaders.ChangeAwareClassLoader .upToDate(ChangeAwareClassLoader.java:109)

    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.ServletStubImpl.needToRe load(ServletStubImpl.java:670)

    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.ServletStubImpl.checkFor Reload(ServletStubImpl.java:697)

    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.jsp.JspStub.checkForReload(JspStu b.java:159)

    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.ServletStubImpl.getServl et(ServletStubImpl.java:538)

    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.ServletStubImpl.invokeSe rvlet(ServletStubImpl.java:364)

    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.TailFilter.doFilter(Tail Filter.java:20)

    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.FilterChainImpl.doFilter (FilterChainImpl.java:27)

    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.RequestDispatcherImpl.fo rward(RequestDispatcherImpl.java:315)

    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at org.apache.struts.action.RequestProcessor.doForwar d(RequestProcessor.java:1069)

    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at org.apache.struts.tiles.TilesRequestProcessor.doFo rward(TilesRequestProcessor.java:274)

    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at org.apache.struts.tiles.TilesRequestProcessor.proc essTilesDefinition(TilesRequestProcessor.java:254)

    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at org.apache.struts.tiles.TilesRequestProcessor.proc essForwardConfig(TilesRequestProcessor.java:309)

    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at org.apache.struts.action.RequestProcessor.process( RequestProcessor.java:279)

    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at org.apache.struts.action.ActionServlet.process(Act ionServlet.java:1482)

    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at org.apache.struts.action.ActionServlet.doGet(Actio nServlet.java:507)

    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at javax.servlet.http.HttpServlet.service(HttpServlet .java:740)

    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at javax.servlet.http.HttpServlet.service(HttpServlet .java:853)

    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.ServletStubImpl$ServletI nvocationAction.run(ServletStubImpl.java:1075)

    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.ServletStubImpl.invokeSe rvlet(ServletStubImpl.java:418)

    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.TailFilter.doFilter(Tail Filter.java:20)

    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.FilterChainImpl.doFilter (FilterChainImpl.java:27)

    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at com.wfb.util.SessionValidationFilter.doFilter(Sess ionValidationFilter.java:83)

    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.FilterChainImpl.doFilter (FilterChainImpl.java:27)

    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.WebAppServletContext$Ser vletInvocationAction.run(WebAppServletContext.java :5523)

    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.security.service.SecurityServiceManager.r unAs(SecurityServiceManager.java:685)

    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.WebAppServletContext.inv okeServlet(WebAppServletContext.java:3156)

    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.ServletRequestImpl.execu te(ServletRequestImpl.java:2506)

    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.kernel.ExecuteThread.execute(ExecuteThrea d.java:234)

    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.kernel.ExecuteThread.run(ExecuteThread.ja va:210)


    Thanks

  6. Re: Weblogic very very very slow in Production

    Here are some suggestions:

    1. The WebLogic MS SQL Driver in 7.0 does have performance problems. I'd recommend you do one of the following:
    a) Get Microsoft's JDBC driver and use that. I believe it's free
    b) Get a commercial driver like DataDirect's
    c) Upgrade to WLS 8.1 and use the MS SQL Server in that release - it's as fast (if not faster) as the other two and has been well-tested with WLS 8.1 SP2 and later

    2. Set the Connection Pool size equal to the default thread pool size and set the minimum and maximum sizes equal. The optimal size is typically somewhat less than the thread pol size, but that is a good starting point. Also, if you do upgrade to the latest 8.1 service pack, ensure you set AutoJDBCConnectionClose to false. Let me know if you want more details about configuring this parameter.

    3. Use the PreparedStatementCache. This is set on the JDBCConnectionPool - the parameter name is PreparedStatementCacheSize. Note that for 8.1, the parameter name is StatementCacheSize. It should be set equal to at least the number of JDBC statements in your application. Also, you have to be using PreparedStatements in your application for this to work. If you're not, I'd recommend you do. E-docs will have more info about the parameter.

    4. It appears from the thread dump that you have servlet and jsp reload checks turned on. Unless you specifically need this functionality, I'd recommend turning it off. By default WLS checks for updates to your application every second, which is expensive especially for large applications. This feature must be turned off in two places if you're using JSPs - In config.xml, in the WebAppComponent parameter, specify ServletReloadSecs="-1". And if you're using JSPs, set the pageCheckSeconds parameter to -1 in the weblogic.xml of your war file. Here're more details on these two parameters:

    http://e-docs.bea.com/wls/docs70/con...eloadCheckSecs

    http://e-docs.bea.com/wls/docs70/web...l.html#1037041

    It'd be a good idea to verify that these parameters are indeed having an effect by taking a few thread dumps and ensuring the following does not show up in the stack traces:

    INFO | jvm 1 | 2005/01/11 09:32:53 | [java] "ExecuteThread: '15' for queue: 'default'" daemon prio=5 tid=0x4038a0 nid=0x24 waiting for monitor entry [0xe1a00000..0xe1a01a00]

    INFO | jvm 1 | 2005/01/11 09:32:53 | [java] at weblogic.servlet.jsp.JspStub.checkForReload(JspStu b.java:145)

    The recommended way these parameters are specified has changed in 8.1 so be sure you consult the documentation at edocs if you decide to upgrade. If you absolutely must have this functionality increase the value to something larger.

    5. If possible, try turning off security permissions as an experiment to measure the impact of authentication and authorization on performance. If the effect is significant, tuning that functionality is a whole different ballgame. Security has a pretty significant performance cost even in very well-tuned systems.

    6. Finally, if you are not already, run in production mode.

    Hope these help.
    Arunabh

  7. Re: Weblogic very very very slow in Production

    Hi Bill,

    The 2 thread dumps are posted above are basically replicated in the other threads of DEFAULT queue in rest of the dump ... if u want I can paste the whole dump which is huge.

    The only 2 threads that have a different stack trace in the dump are the following (note that first thread is in SUSPENDED state)

    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] "ExecuteThread: '22' for queue: 'default'" daemon prio=5 tid=0x711c10 nid=0x2b suspended [0xe12ff000..0xe1301a00]
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at java.util.Date.(Date.java:241)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at java.util.zip.ZipEntry.dosToJavaTime(ZipEntry.java :287)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at java.util.zip.ZipEntry.getTime(ZipEntry.java:127)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.utils.classloaders.ZipSource.lastModified (ZipSource.java:77)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.utils.classloaders.ChangeAwareClassLoader .upToDate(ChangeAwareClassLoader.java:114)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.ServletStubImpl.needToRe load(ServletStubImpl.java:670)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.ServletStubImpl.checkFor Reload(ServletStubImpl.java:697)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.jsp.JspStub.checkForReload(JspStu b.java:159)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.ServletStubImpl.getServl et(ServletStubImpl.java:538)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.ServletStubImpl.invokeSe rvlet(ServletStubImpl.java:364)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.TailFilter.doFilter(Tail Filter.java:20)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.FilterChainImpl.doFilter (FilterChainImpl.java:27)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.RequestDispatcherImpl.in clude(RequestDispatcherImpl.java:609)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.RequestDispatcherImpl.in clude(RequestDispatcherImpl.java:402)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.jsp.PageContextImpl.include(PageC ontextImpl.java:151)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at org.apache.struts.tiles.TilesUtilImpl.doInclude(Ti lesUtilImpl.java:137)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at org.apache.struts.tiles.TilesUtil.doInclude(TilesU til.java:177)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at org.apache.struts.taglib.tiles.InsertTag.doInclude (InsertTag.java:756)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at org.apache.struts.taglib.tiles.InsertTag$InsertHan dler.doEndTag(InsertTag.java:881)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at org.apache.struts.taglib.tiles.InsertTag.doEndTag( InsertTag.java:473)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at jsp_servlet.__layout_main._jspService(__layout_mai n.java:219)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.jsp.JspBase.service(JspBase.java: 27)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.ServletStubImpl$ServletI nvocationAction.run(ServletStubImpl.java:1075)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.ServletStubImpl.invokeSe rvlet(ServletStubImpl.java:418)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.TailFilter.doFilter(Tail Filter.java:20)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.FilterChainImpl.doFilter (FilterChainImpl.java:27)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.RequestDispatcherImpl.fo rward(RequestDispatcherImpl.java:315)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at org.apache.struts.action.RequestProcessor.doForwar d(RequestProcessor.java:1069)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at org.apache.struts.tiles.TilesRequestProcessor.doFo rward(TilesRequestProcessor.java:274)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at org.apache.struts.tiles.TilesRequestProcessor.proc essTilesDefinition(TilesRequestProcessor.java:254)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at org.apache.struts.tiles.TilesRequestProcessor.proc essForwardConfig(TilesRequestProcessor.java:309)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at org.apache.struts.action.RequestProcessor.process( RequestProcessor.java:279)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at org.apache.struts.action.ActionServlet.process(Act ionServlet.java:1482)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at org.apache.struts.action.ActionServlet.doGet(Actio nServlet.java:507)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at javax.servlet.http.HttpServlet.service(HttpServlet .java:740)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at javax.servlet.http.HttpServlet.service(HttpServlet .java:853)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.ServletStubImpl$ServletI nvocationAction.run(ServletStubImpl.java:1075)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.ServletStubImpl.invokeSe rvlet(ServletStubImpl.java:418)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.TailFilter.doFilter(Tail Filter.java:20)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.FilterChainImpl.doFilter (FilterChainImpl.java:27)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at com.wfb.util.SessionValidationFilter.doFilter(Sess ionValidationFilter.java:83)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.FilterChainImpl.doFilter (FilterChainImpl.java:27)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.WebAppServletContext$Ser vletInvocationAction.run(WebAppServletContext.java :5523)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.security.service.SecurityServiceManager.r unAs(SecurityServiceManager.java:685)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.WebAppServletContext.inv okeServlet(WebAppServletContext.java:3156)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.ServletRequestImpl.execu te(ServletRequestImpl.java:2506)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.kernel.ExecuteThread.execute(ExecuteThrea d.java:234)
    INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.kernel.ExecuteThread.run(ExecuteThread.ja va:210)


    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] "ExecuteThread: '0' for queue: 'default'" daemon prio=5 tid=0x917838 nid=0x15 runnable [0xe2900000..0xe2901a00]
    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at java.util.zip.ZipFile.getEntry(Native Method)
    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at java.util.zip.ZipFile.getEntry(ZipFile.java:142)
    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.WarClassFinder.getSource (WarClassFinder.java:183)
    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.WarClassFinder.getSource (WarClassFinder.java:102)
    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.utils.classloaders.MultiClassFinder.getSo urce(MultiClassFinder.java:63)
    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.utils.classloaders.MultiClassFinder.getSo urce(MultiClassFinder.java:63)
    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.utils.classloaders.MultiClassFinder.getCl assSource(MultiClassFinder.java:55)
    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.utils.classloaders.ChangeAwareClassLoader .upToDate(ChangeAwareClassLoader.java:109)
    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.ServletStubImpl.needToRe load(ServletStubImpl.java:670)
    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.ServletStubImpl.checkFor Reload(ServletStubImpl.java:697)
    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.jsp.JspStub.checkForReload(JspStu b.java:159)
    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.ServletStubImpl.getServl et(ServletStubImpl.java:538)
    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.ServletStubImpl.invokeSe rvlet(ServletStubImpl.java:364)
    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.TailFilter.doFilter(Tail Filter.java:20)
    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.FilterChainImpl.doFilter (FilterChainImpl.java:27)
    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.RequestDispatcherImpl.fo rward(RequestDispatcherImpl.java:315)
    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at org.apache.struts.action.RequestProcessor.doForwar d(RequestProcessor.java:1069)
    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at org.apache.struts.tiles.TilesRequestProcessor.doFo rward(TilesRequestProcessor.java:274)
    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at org.apache.struts.tiles.TilesRequestProcessor.proc essTilesDefinition(TilesRequestProcessor.java:254)
    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at org.apache.struts.tiles.TilesRequestProcessor.proc essForwardConfig(TilesRequestProcessor.java:309)
    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at org.apache.struts.action.RequestProcessor.process( RequestProcessor.java:279)
    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at org.apache.struts.action.ActionServlet.process(Act ionServlet.java:1482)
    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at org.apache.struts.action.ActionServlet.doGet(Actio nServlet.java:507)
    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at javax.servlet.http.HttpServlet.service(HttpServlet .java:740)
    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at javax.servlet.http.HttpServlet.service(HttpServlet .java:853)
    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.ServletStubImpl$ServletI nvocationAction.run(ServletStubImpl.java:1075)
    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.ServletStubImpl.invokeSe rvlet(ServletStubImpl.java:418)
    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.TailFilter.doFilter(Tail Filter.java:20)
    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.FilterChainImpl.doFilter (FilterChainImpl.java:27)
    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at com.wfb.util.SessionValidationFilter.doFilter(Sess ionValidationFilter.java:83)
    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.FilterChainImpl.doFilter (FilterChainImpl.java:27)
    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.WebAppServletContext$Ser vletInvocationAction.run(WebAppServletContext.java :5523)
    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.security.service.SecurityServiceManager.r unAs(SecurityServiceManager.java:685)
    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.WebAppServletContext.inv okeServlet(WebAppServletContext.java:3156)
    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.ServletRequestImpl.execu te(ServletRequestImpl.java:2506)
    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.kernel.ExecuteThread.execute(ExecuteThrea d.java:234)
    INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.kernel.ExecuteThread.run(ExecuteThread.ja va:210)

  8. Re: Weblogic very very very slow in Production

    It looks like you have servlet reloading turned on. This is fine for
    development, but in production I would turn it off.


    Set servlet-reload-check-seconds to -1 in your weblogic.xml.

    See

    http://e-docs.bea.com/wls/docs81/web...l.html#1055219

    -- Rob

    Pavnit Bhatia wrote:
    > Hi Bill,
    >
    > The 2 thread dumps are posted above are basically replicated in the other threads of DEFAULT queue in rest of the dump ... if u want I can paste the whole dump which is huge.
    >
    > The only 2 threads that have a different stack trace in the dump are the following (note that first thread is in SUSPENDED state)
    >
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] "ExecuteThread: '22' for queue: 'default'" daemon prio=5 tid=0x711c10 nid=0x2b suspended [0xe12ff000..0xe1301a00]
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at java.util.Date.(Date.java:241)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at java.util.zip.ZipEntry.dosToJavaTime(ZipEntry.java :287)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at java.util.zip.ZipEntry.getTime(ZipEntry.java:127)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.utils.classloaders.ZipSource.lastModified (ZipSource.java:77)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.utils.classloaders.ChangeAwareClassLoader .upToDate(ChangeAwareClassLoader.java:114)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.ServletStubImpl.needToRe load(ServletStubImpl.java:670)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.ServletStubImpl.checkFor Reload(ServletStubImpl.java:697)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.jsp.JspStub.checkForReload(JspStu b.java:159)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.ServletStubImpl.getServl et(ServletStubImpl.java:538)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.ServletStubImpl.invokeSe rvlet(ServletStubImpl.java:364)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.TailFilter.doFilter(Tail Filter.java:20)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.FilterChainImpl.doFilter (FilterChainImpl.java:27)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.RequestDispatcherImpl.in clude(RequestDispatcherImpl.java:609)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.RequestDispatcherImpl.in clude(RequestDispatcherImpl.java:402)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.jsp.PageContextImpl.include(PageC ontextImpl.java:151)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at org.apache.struts.tiles.TilesUtilImpl.doInclude(Ti lesUtilImpl.java:137)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at org.apache.struts.tiles.TilesUtil.doInclude(TilesU til.java:177)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at org.apache.struts.taglib.tiles.InsertTag.doInclude (InsertTag.java:756)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at org.apache.struts.taglib.tiles.InsertTag$InsertHan dler.doEndTag(InsertTag.java:881)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at org.apache.struts.taglib.tiles.InsertTag.doEndTag( InsertTag.java:473)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at jsp_servlet.__layout_main._jspService(__layout_mai n.java:219)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.jsp.JspBase.service(JspBase.java: 27)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.ServletStubImpl$ServletI nvocationAction.run(ServletStubImpl.java:1075)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.ServletStubImpl.invokeSe rvlet(ServletStubImpl.java:418)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.TailFilter.doFilter(Tail Filter.java:20)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.FilterChainImpl.doFilter (FilterChainImpl.java:27)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.RequestDispatcherImpl.fo rward(RequestDispatcherImpl.java:315)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at org.apache.struts.action.RequestProcessor.doForwar d(RequestProcessor.java:1069)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at org.apache.struts.tiles.TilesRequestProcessor.doFo rward(TilesRequestProcessor.java:274)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at org.apache.struts.tiles.TilesRequestProcessor.proc essTilesDefinition(TilesRequestProcessor.java:254)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at org.apache.struts.tiles.TilesRequestProcessor.proc essForwardConfig(TilesRequestProcessor.java:309)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at org.apache.struts.action.RequestProcessor.process( RequestProcessor.java:279)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at org.apache.struts.action.ActionServlet.process(Act ionServlet.java:1482)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at org.apache.struts.action.ActionServlet.doGet(Actio nServlet.java:507)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at javax.servlet.http.HttpServlet.service(HttpServlet .java:740)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at javax.servlet.http.HttpServlet.service(HttpServlet .java:853)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.ServletStubImpl$ServletI nvocationAction.run(ServletStubImpl.java:1075)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.ServletStubImpl.invokeSe rvlet(ServletStubImpl.java:418)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.TailFilter.doFilter(Tail Filter.java:20)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.FilterChainImpl.doFilter (FilterChainImpl.java:27)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at com.wfb.util.SessionValidationFilter.doFilter(Sess ionValidationFilter.java:83)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.FilterChainImpl.doFilter (FilterChainImpl.java:27)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.WebAppServletContext$Ser vletInvocationAction.run(WebAppServletContext.java :5523)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.security.service.SecurityServiceManager.r unAs(SecurityServiceManager.java:685)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.WebAppServletContext.inv okeServlet(WebAppServletContext.java:3156)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.servlet.internal.ServletRequestImpl.execu te(ServletRequestImpl.java:2506)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.kernel.ExecuteThread.execute(ExecuteThrea d.java:234)
    > INFO | jvm 1 | 2005/01/11 09:32:52 | [java] at weblogic.kernel.ExecuteThread.run(ExecuteThread.ja va:210)
    >
    >
    > INFO | jvm 1 | 2005/01/11 09:32:56 | [java] "ExecuteThread: '0' for queue: 'default'" daemon prio=5 tid=0x917838 nid=0x15 runnable [0xe2900000..0xe2901a00]
    > INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at java.util.zip.ZipFile.getEntry(Native Method)
    > INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at java.util.zip.ZipFile.getEntry(ZipFile.java:142)
    > INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.WarClassFinder.getSource (WarClassFinder.java:183)
    > INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.WarClassFinder.getSource (WarClassFinder.java:102)
    > INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.utils.classloaders.MultiClassFinder.getSo urce(MultiClassFinder.java:63)
    > INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.utils.classloaders.MultiClassFinder.getSo urce(MultiClassFinder.java:63)
    > INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.utils.classloaders.MultiClassFinder.getCl assSource(MultiClassFinder.java:55)
    > INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.utils.classloaders.ChangeAwareClassLoader .upToDate(ChangeAwareClassLoader.java:109)
    > INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.ServletStubImpl.needToRe load(ServletStubImpl.java:670)
    > INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.ServletStubImpl.checkFor Reload(ServletStubImpl.java:697)
    > INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.jsp.JspStub.checkForReload(JspStu b.java:159)
    > INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.ServletStubImpl.getServl et(ServletStubImpl.java:538)
    > INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.ServletStubImpl.invokeSe rvlet(ServletStubImpl.java:364)
    > INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.TailFilter.doFilter(Tail Filter.java:20)
    > INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.FilterChainImpl.doFilter (FilterChainImpl.java:27)
    > INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.RequestDispatcherImpl.fo rward(RequestDispatcherImpl.java:315)
    > INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at org.apache.struts.action.RequestProcessor.doForwar d(RequestProcessor.java:1069)
    > INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at org.apache.struts.tiles.TilesRequestProcessor.doFo rward(TilesRequestProcessor.java:274)
    > INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at org.apache.struts.tiles.TilesRequestProcessor.proc essTilesDefinition(TilesRequestProcessor.java:254)
    > INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at org.apache.struts.tiles.TilesRequestProcessor.proc essForwardConfig(TilesRequestProcessor.java:309)
    > INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at org.apache.struts.action.RequestProcessor.process( RequestProcessor.java:279)
    > INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at org.apache.struts.action.ActionServlet.process(Act ionServlet.java:1482)
    > INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at org.apache.struts.action.ActionServlet.doGet(Actio nServlet.java:507)
    > INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at javax.servlet.http.HttpServlet.service(HttpServlet .java:740)
    > INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at javax.servlet.http.HttpServlet.service(HttpServlet .java:853)
    > INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.ServletStubImpl$ServletI nvocationAction.run(ServletStubImpl.java:1075)
    > INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.ServletStubImpl.invokeSe rvlet(ServletStubImpl.java:418)
    > INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.TailFilter.doFilter(Tail Filter.java:20)
    > INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.FilterChainImpl.doFilter (FilterChainImpl.java:27)
    > INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at com.wfb.util.SessionValidationFilter.doFilter(Sess ionValidationFilter.java:83)
    > INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.FilterChainImpl.doFilter (FilterChainImpl.java:27)
    > INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.WebAppServletContext$Ser vletInvocationAction.run(WebAppServletContext.java :5523)
    > INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.security.service.SecurityServiceManager.r unAs(SecurityServiceManager.java:685)
    > INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.WebAppServletContext.inv okeServlet(WebAppServletContext.java:3156)
    > INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.servlet.internal.ServletRequestImpl.execu te(ServletRequestImpl.java:2506)
    > INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.kernel.ExecuteThread.execute(ExecuteThrea d.java:234)
    > INFO | jvm 1 | 2005/01/11 09:32:56 | [java] at weblogic.kernel.ExecuteThread.run(ExecuteThread.ja va:210)



  9. Re: Weblogic very very very slow in Production

    arunabh wrote:
    > 1. The WebLogic MS SQL Driver in 7.0 does have performance problems.

    I'd recommend you do one of the following:
    > a) Get Microsoft's JDBC driver and use that. I believe it's free
    > b) Get a commercial driver like DataDirect's
    > c) Upgrade to WLS 8.1 and use the MS SQL Server in that release -

    it's as fast (if not faster) as the other two and has been well-tested
    with WLS 8.1 SP2 and later

    The Microsoft JDBC driver has the same performance problems, so it's
    not a solution. I would recommend using jTDS
    (http://jtds.sourceforge.net/) if the driver is the problem, but then
    I'm biased, being one of the developers of that software. It's
    definitely faster than any of the suggested drivers.
    Alin,
    The jTDS Project.


  10. Re: Weblogic very very very slow in Production

    Thanks Rob and Arunabh,

    As Arunabh suggested yesterday we did set this parameter and also pageCheckSeconds for JSP checking and its working great now ... and users are very happy this morning.

    I also noticed from the dump that it goes to War file (a Zip file) and checks the timestamp for each call and for every servlet to see if it has been updated which can be very time consuming in production environments that are under heavy load like this application.

    Thanks everyone!

  11. Re: Weblogic very very very slow in Production

    > 50 Execute threads under DEFAULT queue. Under peak load it goes down to 0 idle threads.

    If 50 Execute Threads goes down to 0 with 60 concurrent users, obviously either your the threads are blocking somewhere along the execution flow or you do not have enough thread to handle your load i.e., your response times are high due to threads doing processing or simply waiting a reply to come from db or other systems that you are accessing.

    This is most of the time is due to a blocking condition in your execution flow. You said, db access is taking only about 30 ms, is there other systems that you are accessing? How much time other processing is taking? What is the CPU while the number of threads is about zero? If low, it may indicate that the threads are waiting. You can take threaddumps to find out where your threads are waiting.

    You also mentioned the requests are taking about 600 ms - is this for a single request or under load? If it is for under load, a basic estimation indicates that you can handle about 70 TPS with all your threads are busy! Basically your our response times of 600ms is high, you cannot have more then that with this kind of response time - again you need to find out why the response times are high? Other external system access(?), inefficient processing code(?)

    What is your basic flow, do you have any other (other then db) external systems to access? What is the nature of processing done in your application?

    Please take a look at the performance tuning doc for wls 7.0 at: http://e-docs.bea.com/platform/docs7...rm/tuning.html

    sazi

  12. Re: Weblogic very very very slow in Production

    I have not seen your full threaddump, but you should not see threads SUSPENDED and many threads are "waiting for monitor" - it just indicates that your threads are blocked and "stopped".

    sazi

+ Reply to Thread