我们最近将我们的银行应用程序从java 1.6升级到1.8,将jboss 4.x升级到wildfly 10.1 .
我们观察到java消耗了机器上可用的所有核心(10) .
有人可以说出原因是什么,通常使用jboss 4.x最大CPU利用率高达4核心 .
我是否需要为垃圾收集配置任何东西?
以下是(添加进程消耗高CPU)的结果
ps -eLo pid,lwp,nlwp,ruser,pcpu,stime,etime,args | grep 3630
下面是每个消耗高CPU的LWP的十六进制
现在我检查了LWP 7914的线程转储
“default task-7”#182 prio = 5 os_prio = 0 tid = 0x00007f5c24033800 nid = 0x1c1a runnable [0x00007f5bb85e5000] java.lang.Thread.State:RUNNABLE at org.xnio.conduits.ConduitStreamSinkChannel.write(ConduitStreamSinkChannel.java:150 )io.undertow.channels.DetachableStreamSinkChannel.write(DetachableStreamSinkChannel.java:240)at io.undertow.server.HttpServerExchange $ WriteDispatchChannel.write(HttpServerExchange.java:2028)at io.undertow.servlet.spec.ServletOutputStreamImpl.writeBufferBlocking( ServletOutputStreamImpl.java:563)位于java.io.BufferedOutputStream.write的java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)中的io.undertow.servlet.spec.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:216)(BufferedOutputStream) .java:95) - 在javax.servlet.http.HttpServlet.service的com.eko.app.offlineKyc.servlet.KycPictureServlet.doGet(KycPictureServlet.java:58)中锁定<0x00000004d6c5f900>(java.io.BufferedOutputStream)( HttpServlet.java:687)在javax.servlet.http.HttpSer vlet.service(HttpServlet.java:790)at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)at io.undertow.servlet.handlers.FilterHandler $ FilterChainImpl.doFilter(FilterHandler.java:129) at com.eko.framework.CrossScriptingFilter.doFilter(CrossScriptingFilter.java:48)at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)at io.undertow.servlet.handlers.FilterHandler $ FilterChainImpl.doFilter (FilterHandler.java:131)位于io.undertow.servlet的io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)的com.eko.framework.URLSessionFilter.doFilter(URLSessionFilter.java:38) . handlers.FilterHandler $ FilterChainImpl.doFilter(FilterHandler.java:131)atg.mifos.framework.security.util.FileNameFilter.doFilter(FileNameFilter.java:59)at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter . java:61)at io.undertow.servlet.handlers.FilterHandler $ FilterChainImpl.doFilter(FilterHandler.java:131)at io.undertow.servlet.handlers.FilterH andler.handleRequest(FilterHandler.java:84)at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)在org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)在io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)在io.undertow.servlet.handlers.security .SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43 )at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53)at io.undertow.security.handlers.Abstrac tConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java: 59)在io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)在io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)在io.undertow.security.handlers .NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)在io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)在io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)位于io.undertow.server.handlers的org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) . PredionHandler.handleRequest(PredicateHandler.java:43)在io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)at io .undertow.servlet.handlers.ServletInitialHandler.access $ 100(ServletInitialHandler.java:81)at io.undertow.servlet.handlers.ServletInitialHandler $ 2.call(ServletInitialHandler.java:138)at io.undertow.servlet.handlers.ServletInitialHandler $ 2 . 调用(ServletInitialHandler.java:135)io.undertow.servlet.core.ServletRequestContextThreadSetupAction $ 1.call(ServletRequestContextThreadSetupAction.java:48)at io.undertow.servlet.core.ContextClassLoaderSetupAction $ 1.call(ContextClassLoaderSetupAction.java:43)at io .underto w.servlet.api.LegacyThreadSetupActionWrapper $ 1.call(LegacyThreadSetupActionWrapper.java:44)at io.undertow.servlet.api.LegacyThreadSetupActionWrapper $ 1.call(LegacyThreadSetupActionWrapper.java:44)at io.undertow.servlet.api.LegacyThreadSetupActionWrapper $ 1.call (LegacyThreadSetupActionWrapper.java:44)at io.undertow.servlet.api.LegacyThreadSetupActionWrapper $ 1.call(LegacyThreadSetupActionWrapper.java:44)at io.undertow.servlet.api.LegacyThreadSetupActionWrapper $ 1.call(LegacyThreadSetupActionWrapper.java:44)at io . 在io.undertow.servlet.handlers.ServletInitialHandler.access $ 000(ServletInitialHandler.java:81)的io.undertow.servlet.handlers.ServletInitialHandler $ 1.handleRequest中的underow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) ServletInitialHandler.java:104)at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)at io.undertow.server.HttpServerExchange $ 1.run(HttpServerExchange.java:805)at java.util.concurrent.ThreadPoolExecu java.util.ThreadPoolExecutor中的tor.runWorker(ThreadPoolExecutor.java:1142)java.lang.Thread.run上的java.Manun.run(ThreadPoolExecutor.java:617)(Thread.java:745)
对于LWP 7249
“default task-27”#231 prio = 5 os_prio = 0 tid = 0x00007f5c2401c000 nid = 0x1c51 runnable [0x00007f5ca9a63000] java.lang.Thread.State:RUNNABLE at org.xnio.conduits.ConduitStreamSinkChannel.write(ConduitStreamSinkChannel.java:158 )ato.undertow.channels.DetachableStreamSinkChannel.write(DetachableStreamSinkChannel.java:179)ato.undertow.server.HttpServerExchange $ WriteDispatchChannel.write(HttpServerExchange.java:1993)at org.xnio.channels.Channels.writeBlocking(Channels . java:152)ato.undertow.servlet.spec.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:283)at io.undertow.io.BlockingSenderImpl.writeBuffer(BlockingSenderImpl.java:190)at io.undertow.io.BlockingSenderImpl.send (BlockingSenderImpl.java:71)io.undertow.server.handlers.resource.CachedResource.serve(CachedResource.java:187)at io.undertow.servlet.handlers.DefaultServlet.serveFileBlocking(DefaultServlet.java:332)at io . 在javax.servlet.http.H上的undertow.servlet.handlers.DefaultServlet.doGet(DefaultServlet.java:180) ttpServlet.service(HttpServlet.java:687)位于io.undertow的io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)的javax.servlet.http.HttpServlet.service(HttpServlet.java:790)位于com.eko.framework.MifosNGFilter.doNormal(MifosNGFilter.java:283)的.servlet.handlers.FilterHandler $ FilterChainImpl.doFilter(FilterHandler.java:129)at com.eko.framework.MifosNGFilter.doFilter(MifosNGFilter.java:137 )在com.eko.framework.CrossScriptingFilter的io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)中的io.undertow.servlet.handlers.FilterHandler $ FilterChainImpl.doFilter(FilterHandler.java:131) . doFilter(CrossScriptingFilter.java:48)at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)at io.undertow.servlet.handlers.FilterHandler $ FilterChainImpl.doFilter(FilterHandler.java:131)at com .eko.framework.URLSessionFilter.doFilter(URLSessionFilter.java:38)at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)at io.underto位于io.undertow.servlet.core.ManagedFilter的org.mifos.framework.security.util.FileNameFilter.doFilter(FileNameFilter.java:59)的w.servlet.handlers.FilterHandler $ FilterChainImpl.doFilter(FilterHandler.java:131) . doFilter(ManagedFilter.java:61)atio.undertow.servlet.handlers.FilterHandler $ FilterChainImpl.doFilter(FilterHandler.java:131)at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)at io.undertow.servlet.handlers.security . ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)位于org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)的io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)at io.undertow.servlet.handlers.security位于io.undertow.security.handl的io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)的.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) ers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53)at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java: 64)在io.undertow.servile.handlers的io.undertow.security.handler.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)的io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59) .security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43 )在io.underto位于io.undertow.server.handlers.PredicateHandler的org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)上的w.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) . handleRequest(PredicateHandler.java:43)at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)at io.undertow位于io.undertow.servlet.handler.ServletInitialHandler $ 2.call(ServletInitialHandler.java:138)的io.undertow.servlet.handlers.ServletInitialHandler $ 2.call的.servlet.handlers.ServletInitialHandler.access $ 100(ServletInitialHandler.java:81) ServletInitialHandler.java:135)io.undertow.servlet.core.ServletRequestContextThreadSetupAction $ 1.call(ServletRequestContextThreadSetupAction.java:48)at io.undertow.servlet.core.ContextClassLoaderSetupAction $ 1.call(ContextClassLoaderSetupAction.java:43)at io.undertow .servlet . api.LegacyThreadSetupActionWrapper $ 1.call(LegacyThreadSetupActionWrapper.java:44)at io.undertow.servlet.api.LegacyThreadSetupActionWrapper $ 1.call(LegacyThreadSetupActionWrapper.java:44)at io.undertow.servlet.api.LegacyThreadSetupActionWrapper $ 1.call(LegacyThreadSetupActionWrapper.java) :44)ato.undertow.servlet.api.LegacyThreadSetupActionWrapper $ 1.call(LegacyThreadSetupActionWrapper.java:44)在io.undertow.servlet的io.undertow.servlet.api.LegacyThreadSetupActionWrapper $ 1.call(LegacyThreadSetupActionWrapper.java:44) . handler.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)at io.undertow.servlet.handlers.ServletInitialHandler.access $ 000(ServletInitialHandler.java:81)at io.undertow.servlet.handlers.ServletInitialHandler $ 1.handleRequest(ServletInitialHandler.java: 104)ato.undertow.server.Connectors.executeRootHandler(Connectors.java:202)at io.undertow.server.HttpServerExchange $ 1.run(HttpServerExchange.java:805)at java.util.concurrent.ThreadPoolExecutor.runWor ker(ThreadPoolExecutor.java:1142)位于java.lang.Thread.run的java.util.concurrent.ThreadPoolExecutor $ Worker.run(ThreadPoolExecutor.java:617)中(Thread.java:745)
下面是完整线程转储Thread dump的链接
服务器不稳定,所以我们不得不恢复到jdk 1.6和jboss 4.我想知道jboss 4一切正常,现在我无法确定问题出在哪里,使用wildfly设置,或者我的应用程序或GC设置 .
线程转储分析的链接Thread dump analysis result代码 KycPictureServlet
@Override
public void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
ServletOutputStream out = null;
FileInputStream fin = null;
BufferedInputStream bin = null;
BufferedOutputStream bout = null;
try {
String filename = String.valueOf(request.getParameter("fileName"));
String folderPathId = String.valueOf(request.getParameter("folderPathId"));
String kycDocPath = "";
System.out.println("Folder Path : " + folderPathId + " \t" + filename);
if (folderPathId != null && !folderPathId.equals("null")) {
kycDocPath = new StoragePathService().getStoragePathByPathID(Integer.parseInt(folderPathId)).getFolderPath();
} else {
kycDocPath = new StoragePathService().getStoragePathByPathID(EkoDBConstants.KYC_FILE_UPLOAD_PATH).getFolderPath();
}
String uploadFolderPath = kycDocPath;
final String filetofetch = uploadFolderPath + filename;
if (filetofetch.toLowerCase().endsWith("pdf")) {
response.setContentType("application/pdf");
// response.setHeader("Content-Disposition", "inline;
// filename=\"" + filetofetch + "\"");
} else if (filetofetch.toLowerCase().endsWith("tif") || filetofetch.endsWith("tiff")) {
response.setContentType("image/tiff");
} else {
response.setContentType("image/jpeg");
}
File uploadFolder = new File(filetofetch);
if(uploadFolder != null && uploadFolder.exists()){
out = response.getOutputStream();
fin = new FileInputStream(filetofetch);
bin = new BufferedInputStream(fin);
bout = new BufferedOutputStream(out);
int ch = 0;
;
while ((ch = bin.read()) != -1) {
bout.write(ch);
}
}
} catch (Exception e) {
e.printStackTrace();
} finally {
if (bin != null) {
bin.close();
}
if (fin != null) {
fin.close();
}
if (bout != null) {
bout.close();
}
if (out != null) {
out.close();
}
}
}
以前没有任何finally块,所以如果异常流没有关闭,第二件事我添加了额外的检查以检查文件是否存在然后只打开流 .
在进行更改之后,不再有线程卡在该servlet上 .
我已经采取了最新的转储,发现一些奇怪的进程消耗cpu而没有结束
"C1 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007ff8dc231000 nid=0x11b6 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007ff8dc22e800 nid=0x11b5 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007ff8dc22d000 nid=0x11b4 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007ff8dc22a000 nid=0x11b3 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
我也配置了从OLD Jboss 4复制的GC设置
在线程转储中,我发现了一些消耗CPU并且没有结束的LWP
"VM Thread" os_prio=0 tid=0x00007ff8dc1e9000 nid=0x11B411af runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007ff8dc01e800 nid=0x11a6 runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007ff8dc020800 nid=0x11a7 runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007ff8dc022000 nid=0x11a8 runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007ff8dc024000 nid=0x11a9 runnable
"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007ff8dc026000 nid=0x11aa runnable
"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007ff8dc027800 nid=0x11ab runnable
"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007ff8dc029800 nid=0x11ac runnable
"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007ff8dc02b800 nid=0x11ad runnable
"GC task thread#8 (ParallelGC)" os_prio=0 tid=0x00007ff8dc02d000 nid=0x11ae runnable
内存和GC设置
JAVA_OPTS="-Xms4G -Xmx12G -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=512m -Djava.net.preferIPv4Stack=true -Dsun.rmi.dgc.client.gcInterval=1800000 -Dsun.rmi.dgc.server.gcInterval=1800000 -XX:-UseConcMarkSweepGC -XX:SurvivorRatio=6 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:/data/eko/GC.log -verbose:gc -XX:NewRatio=2"
1 回答
可悲的是,我的魔法水晶球被打破了,所以我们必须把手弄得脏乱 .
首先,我们需要弄清楚,什么线程导致Cpu负载,因为app服务器不是单线程的 . 假设您使用的是Linux,可以使用标准
ps
来获取每个进程线程的cpu使用情况的详细信息:ps -eLo pid,lwp,nlwp,ruser,pcpu,stime,etime,args | grep [PID_OF_YOUR_WILDFLY_PROCESS] > unixthread.txt
这里我们感兴趣的是列nlwp lwp和pcpu . 找到具有最高cpu负载的线程 .
接下来我们需要java线程转储:在高CPU负载期间获取线程转储:
jstack -l [PID_OF_YOUR_WILDFLY_PROCESS] > jstack.out
现在对于一些线程搜索:说你的unixthread识别出一个以nwlp 9999作为CPU占用的线程,将数字转换为hex-0x270F并在你的jstack.out文件中搜索这个值 . 它应该与java线程匹配,stacktrace应该可以帮助您查明问题 .
如果它与GC相关,则继续堆转储以确保它不是您的应用程序中的泄漏 .
如果您进行分析并仍然认为这是jvm或gc设置的问题,您应该与redhat伙伴联系 . 我相信,如果你打算在RedHat堆栈上运行一个银行应用程序,那么你绝对应该支付带有付费支持的EAP7,而不是支持Upstream OS项目的hazzard,而不支持最终的错误或问题 .