首页 文章

Wildfly 10.1消耗所有核心

提问于
浏览
2

我们最近将我们的银行应用程序从java 1.6升级到1.8,将jboss 4.x升级到wildfly 10.1 .

我们观察到java消耗了机器上可用的所有核心(10) .

有人可以说出原因是什么,通常使用jboss 4.x最大CPU利用率高达4核心 .

我是否需要为垃圾收集配置任何东西?

以下是(添加进程消耗高CPU)的结果

ps -eLo pid,lwp,nlwp,ruser,pcpu,stime,etime,args | grep 3630

enter image description here

下面是每个消耗高CPU的LWP的十六进制

enter image description here

现在我检查了LWP 7914的线程转储

“default task-7”#182 prio = 5 os_prio = 0 tid = 0x00007f5c24033800 nid = 0x1c1a runnable [0x00007f5bb85e5000] java.lang.Thread.State:RUNNABLE at org.xnio.conduits.ConduitStreamSinkChannel.write(ConduitStreamSinkChannel.java:150 )io.undertow.channels.DetachableStreamSinkChannel.write(DetachableStreamSinkChannel.java:240)at io.undertow.server.HttpServerExchange $ WriteDispatchChannel.write(HttpServerExchange.java:2028)at io.undertow.servlet.spec.ServletOutputStreamImpl.writeBufferBlocking( ServletOutputStreamImpl.java:563)位于java.io.BufferedOutputStream.write的java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)中的io.undertow.servlet.spec.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:216)(BufferedOutputStream) .java:95) - 在javax.servlet.http.HttpServlet.service的com.eko.app.offlineKyc.servlet.KycPictureServlet.doGet(KycPictureServlet.java:58)中锁定<0x00000004d6c5f900>(java.io.BufferedOutputStream)( HttpServlet.java:687)在javax.servlet.http.HttpSer vlet.service(HttpServlet.java:790)at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)at io.undertow.servlet.handlers.FilterHandler $ FilterChainImpl.doFilter(FilterHandler.java:129) at com.eko.framework.CrossScriptingFilter.doFilter(CrossScriptingFilter.java:48)at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)at io.undertow.servlet.handlers.FilterHandler $ FilterChainImpl.doFilter (FilterHandler.java:131)位于io.undertow.servlet的io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)的com.eko.framework.URLSessionFilter.doFilter(URLSessionFilter.java:38) . handlers.FilterHandler $ FilterChainImpl.doFilter(FilterHandler.java:131)atg.mifos.framework.security.util.FileNameFilter.doFilter(FileNameFilter.java:59)at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter . java:61)at io.undertow.servlet.handlers.FilterHandler $ FilterChainImpl.doFilter(FilterHandler.java:131)at io.undertow.servlet.handlers.FilterH andler.handleRequest(FilterHandler.java:84)at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)在org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)在io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)在io.undertow.servlet.handlers.security .SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43 )at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53)at io.undertow.security.handlers.Abstrac tConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java: 59)在io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)在io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)在io.undertow.security.handlers .NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)在io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)在io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)位于io.undertow.server.handlers的org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) . PredionHandler.handleRequest(PredicateHandler.java:43)在io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)at io .undertow.servlet.handlers.ServletInitialHandler.access $ 100(ServletInitialHandler.java:81)at io.undertow.servlet.handlers.ServletInitialHandler $ 2.call(ServletInitialHandler.java:138)at io.undertow.servlet.handlers.ServletInitialHandler $ 2 . 调用(ServletInitialHandler.java:135)io.undertow.servlet.core.ServletRequestContextThreadSetupAction $ 1.call(ServletRequestContextThreadSetupAction.java:48)at io.undertow.servlet.core.ContextClassLoaderSetupAction $ 1.call(ContextClassLoaderSetupAction.java:43)at io .underto w.servlet.api.LegacyThreadSetupActionWrapper $ 1.call(LegacyThreadSetupActionWrapper.java:44)at io.undertow.servlet.api.LegacyThreadSetupActionWrapper $ 1.call(LegacyThreadSetupActionWrapper.java:44)at io.undertow.servlet.api.LegacyThreadSetupActionWrapper $ 1.call (LegacyThreadSetupActionWrapper.java:44)at io.undertow.servlet.api.LegacyThreadSetupActionWrapper $ 1.call(LegacyThreadSetupActionWrapper.java:44)at io.undertow.servlet.api.LegacyThreadSetupActionWrapper $ 1.call(LegacyThreadSetupActionWrapper.java:44)at io . 在io.undertow.servlet.handlers.ServletInitialHandler.access $ 000(ServletInitialHandler.java:81)的io.undertow.servlet.handlers.ServletInitialHandler $ 1.handleRequest中的underow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) ServletInitialHandler.java:104)at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)at io.undertow.server.HttpServerExchange $ 1.run(HttpServerExchange.java:805)at java.util.concurrent.ThreadPoolExecu java.util.ThreadPoolExecutor中的tor.runWorker(ThreadPoolExecutor.java:1142)java.lang.Thread.run上的java.Manun.run(ThreadPoolExecutor.java:617)(Thread.java:745)

对于LWP 7249

“default task-27”#231 prio = 5 os_prio = 0 tid = 0x00007f5c2401c000 nid = 0x1c51 runnable [0x00007f5ca9a63000] java.lang.Thread.State:RUNNABLE at org.xnio.conduits.ConduitStreamSinkChannel.write(ConduitStreamSinkChannel.java:158 )ato.undertow.channels.DetachableStreamSinkChannel.write(DetachableStreamSinkChannel.java:179)ato.undertow.server.HttpServerExchange $ WriteDispatchChannel.write(HttpServerExchange.java:1993)at org.xnio.channels.Channels.writeBlocking(Channels . java:152)ato.undertow.servlet.spec.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:283)at io.undertow.io.BlockingSenderImpl.writeBuffer(BlockingSenderImpl.java:190)at io.undertow.io.BlockingSenderImpl.send (BlockingSenderImpl.java:71)io.undertow.server.handlers.resource.CachedResource.serve(CachedResource.java:187)at io.undertow.servlet.handlers.DefaultServlet.serveFileBlocking(DefaultServlet.java:332)at io . 在javax.servlet.http.H上的undertow.servlet.handlers.DefaultServlet.doGet(DefaultServlet.java:180) ttpServlet.service(HttpServlet.java:687)位于io.undertow的io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)的javax.servlet.http.HttpServlet.service(HttpServlet.java:790)位于com.eko.framework.MifosNGFilter.doNormal(MifosNGFilter.java:283)的.servlet.handlers.FilterHandler $ FilterChainImpl.doFilter(FilterHandler.java:129)at com.eko.framework.MifosNGFilter.doFilter(MifosNGFilter.java:137 )在com.eko.framework.CrossScriptingFilter的io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)中的io.undertow.servlet.handlers.FilterHandler $ FilterChainImpl.doFilter(FilterHandler.java:131) . doFilter(CrossScriptingFilter.java:48)at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)at io.undertow.servlet.handlers.FilterHandler $ FilterChainImpl.doFilter(FilterHandler.java:131)at com .eko.framework.URLSessionFilter.doFilter(URLSessionFilter.java:38)at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)at io.underto位于io.undertow.servlet.core.ManagedFilter的org.mifos.framework.security.util.FileNameFilter.doFilter(FileNameFilter.java:59)的w.servlet.handlers.FilterHandler $ FilterChainImpl.doFilter(FilterHandler.java:131) . doFilter(ManagedFilter.java:61)atio.undertow.servlet.handlers.FilterHandler $ FilterChainImpl.doFilter(FilterHandler.java:131)at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)at io.undertow.servlet.handlers.security . ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)位于org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)的io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)at io.undertow.servlet.handlers.security位于io.undertow.security.handl的io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)的.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) ers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53)at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java: 64)在io.undertow.servile.handlers的io.undertow.security.handler.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)的io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59) .security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43 )在io.underto位于io.undertow.server.handlers.PredicateHandler的org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)上的w.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) . handleRequest(PredicateHandler.java:43)at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)at io.undertow位于io.undertow.servlet.handler.ServletInitialHandler $ 2.call(ServletInitialHandler.java:138)的io.undertow.servlet.handlers.ServletInitialHandler $ 2.call的.servlet.handlers.ServletInitialHandler.access $ 100(ServletInitialHandler.java:81) ServletInitialHandler.java:135)io.undertow.servlet.core.ServletRequestContextThreadSetupAction $ 1.call(ServletRequestContextThreadSetupAction.java:48)at io.undertow.servlet.core.ContextClassLoaderSetupAction $ 1.call(ContextClassLoaderSetupAction.java:43)at io.undertow .servlet . api.LegacyThreadSetupActionWrapper $ 1.call(LegacyThreadSetupActionWrapper.java:44)at io.undertow.servlet.api.LegacyThreadSetupActionWrapper $ 1.call(LegacyThreadSetupActionWrapper.java:44)at io.undertow.servlet.api.LegacyThreadSetupActionWrapper $ 1.call(LegacyThreadSetupActionWrapper.java) :44)ato.undertow.servlet.api.LegacyThreadSetupActionWrapper $ 1.call(LegacyThreadSetupActionWrapper.java:44)在io.undertow.servlet的io.undertow.servlet.api.LegacyThreadSetupActionWrapper $ 1.call(LegacyThreadSetupActionWrapper.java:44) . handler.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)at io.undertow.servlet.handlers.ServletInitialHandler.access $ 000(ServletInitialHandler.java:81)at io.undertow.servlet.handlers.ServletInitialHandler $ 1.handleRequest(ServletInitialHandler.java: 104)ato.undertow.server.Connectors.executeRootHandler(Connectors.java:202)at io.undertow.server.HttpServerExchange $ 1.run(HttpServerExchange.java:805)at java.util.concurrent.ThreadPoolExecutor.runWor ker(ThreadPoolExecutor.java:1142)位于java.lang.Thread.run的java.util.concurrent.ThreadPoolExecutor $ Worker.run(ThreadPoolExecutor.java:617)中(Thread.java:745)

下面是完整线程转储Thread dump的链接

服务器不稳定,所以我们不得不恢复到jdk 1.6和jboss 4.我想知道jboss 4一切正常,现在我无法确定问题出在哪里,使用wildfly设置,或者我的应用程序或GC设置 .

线程转储分析的链接Thread dump analysis result代码 KycPictureServlet

@Override
public void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
    ServletOutputStream out = null;
    FileInputStream fin = null;
    BufferedInputStream bin = null;
    BufferedOutputStream bout = null;
    try {

        String filename = String.valueOf(request.getParameter("fileName"));
        String folderPathId = String.valueOf(request.getParameter("folderPathId"));
        String kycDocPath = "";
        System.out.println("Folder Path : " + folderPathId + " \t" + filename);
        if (folderPathId != null && !folderPathId.equals("null")) {
            kycDocPath = new StoragePathService().getStoragePathByPathID(Integer.parseInt(folderPathId)).getFolderPath();
        } else {
            kycDocPath = new StoragePathService().getStoragePathByPathID(EkoDBConstants.KYC_FILE_UPLOAD_PATH).getFolderPath();
        }
        String uploadFolderPath = kycDocPath;
        final String filetofetch = uploadFolderPath + filename;

        if (filetofetch.toLowerCase().endsWith("pdf")) {
            response.setContentType("application/pdf");
            // response.setHeader("Content-Disposition", "inline;
            // filename=\"" + filetofetch + "\"");
        } else if (filetofetch.toLowerCase().endsWith("tif") || filetofetch.endsWith("tiff")) {
            response.setContentType("image/tiff");
        } else {
            response.setContentType("image/jpeg");
        }

        File uploadFolder = new File(filetofetch);
        if(uploadFolder != null && uploadFolder.exists()){

        out = response.getOutputStream();
        fin = new FileInputStream(filetofetch);
        bin = new BufferedInputStream(fin);
        bout = new BufferedOutputStream(out);
        int ch = 0;
        ;
        while ((ch = bin.read()) != -1) {
            bout.write(ch);

        }
    }

    } catch (Exception e) {
        e.printStackTrace();
    } finally {
        if (bin != null) {
            bin.close();
        }

        if (fin != null) {
            fin.close();
        }

        if (bout != null) {
            bout.close();
        }

        if (out != null) {
            out.close();
        }

    }
}

以前没有任何finally块,所以如果异常流没有关闭,第二件事我添加了额外的检查以检查文件是否存在然后只打开流 .

在进行更改之后,不再有线程卡在该servlet上 .

我已经采取了最新的转储,发现一些奇怪的进程消耗cpu而没有结束

"C1 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007ff8dc231000 nid=0x11b6 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007ff8dc22e800 nid=0x11b5 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007ff8dc22d000 nid=0x11b4 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007ff8dc22a000 nid=0x11b3 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

我也配置了从OLD Jboss 4复制的GC设置

在线程转储中,我发现了一些消耗CPU并且没有结束的LWP

"VM Thread" os_prio=0 tid=0x00007ff8dc1e9000 nid=0x11B411af runnable 

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007ff8dc01e800 nid=0x11a6 runnable 

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007ff8dc020800 nid=0x11a7 runnable 

"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007ff8dc022000 nid=0x11a8 runnable 

"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007ff8dc024000 nid=0x11a9 runnable 

"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007ff8dc026000 nid=0x11aa runnable 

"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007ff8dc027800 nid=0x11ab runnable 

"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007ff8dc029800 nid=0x11ac runnable 

"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007ff8dc02b800 nid=0x11ad runnable 

"GC task thread#8 (ParallelGC)" os_prio=0 tid=0x00007ff8dc02d000 nid=0x11ae runnable

内存和GC设置

JAVA_OPTS="-Xms4G -Xmx12G -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=512m -Djava.net.preferIPv4Stack=true -Dsun.rmi.dgc.client.gcInterval=1800000 -Dsun.rmi.dgc.server.gcInterval=1800000 -XX:-UseConcMarkSweepGC -XX:SurvivorRatio=6 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps  -Xloggc:/data/eko/GC.log -verbose:gc   -XX:NewRatio=2"

1 回答

  • 2

    可悲的是,我的魔法水晶球被打破了,所以我们必须把手弄得脏乱 .
    首先,我们需要弄清楚,什么线程导致Cpu负载,因为app服务器不是单线程的 . 假设您使用的是Linux,可以使用标准 ps 来获取每个进程线程的cpu使用情况的详细信息:

    ps -eLo pid,lwp,nlwp,ruser,pcpu,stime,etime,args | grep [PID_OF_YOUR_WILDFLY_PROCESS] > unixthread.txt

    这里我们感兴趣的是列nlwp lwp和pcpu . 找到具有最高cpu负载的线程 .
    接下来我们需要java线程转储:在高CPU负载期间获取线程转储:

    jstack -l [PID_OF_YOUR_WILDFLY_PROCESS] > jstack.out

    现在对于一些线程搜索:说你的unixthread识别出一个以nwlp 9999作为CPU占用的线程,将数字转换为hex-0x270F并在你的jstack.out文件中搜索这个值 . 它应该与java线程匹配,stacktrace应该可以帮助您查明问题 .

    如果它与GC相关,则继续堆转储以确保它不是您的应用程序中的泄漏 .
    如果您进行分析并仍然认为这是jvm或gc设置的问题,您应该与redhat伙伴联系 . 我相信,如果你打算在RedHat堆栈上运行一个银行应用程序,那么你绝对应该支付带有付费支持的EAP7,而不是支持Upstream OS项目的hazzard,而不支持最终的错误或问题 .

相关问题