java – Hung JVM消耗100%的CPU
我们有一个在
Linux 32位(CentOS)上的Sun JRE 6u20上运行的JAVA服务器.我们使用服务器热点与CMS收集器与以下选项(我只提供了相关的):
-Xmx896m -Xss128k -XX:NewSize=384M -XX:MaxPermSize=96m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC 有时,运行一段时间后,JVM似乎进入挂起状态,即使我们没有向应用程序发出任何请求,CPU仍然以100%的速度旋转(我们有8个逻辑CPU,所以它看起来像只有一个CPU做旋转). 我们已经从“gdb”运行了堆栈转储,我们可以匹配旋转CPU的线程ID(我们发现使用“top”和每个线程视图 – “H”选项)与一个线程栈出现在gdb结果中,这是它的外观: Thread 443 (Thread 0x7e5b90 (LWP 26310)): #0 0x0115ebd3 in CompactibleFreeListSpace::block_size(HeapWord const*) const () from /usr/java/jdk1.6.0_20/jre/lib/i386/server/libjvm.so #1 0x01160ff9 in CompactibleFreeListSpace::prepare_for_compaction(CompactPoint*) () from /usr/java/jdk1.6.0_20/jre/lib/i386/server/libjvm.so #2 0x0123456c in Generation::prepare_for_compaction(CompactPoint*) () from /usr/java/jdk1.6.0_20/jre/lib/i386/server/libjvm.so #3 0x01229b2c in GenCollectedHeap::prepare_for_compaction() () from /usr/java/jdk1.6.0_20/jre/lib/i386/server/libjvm.so #4 0x0122a7fc in GenMarkSweep::invoke_at_safepoint(int,ReferenceProcessor*,bool) () from /usr/java/jdk1.6.0_20/jre/lib/i386/server/libjvm.so #5 0x01186024 in CMSCollector::do_compaction_work(bool) () from /usr/java/jdk1.6.0_20/jre/lib/i386/server/libjvm.so #6 0x011859ee in CMSCollector::acquire_control_and_collect(bool,bool) () from /usr/java/jdk1.6.0_20/jre/lib/i386/server/libjvm.so #7 0x01185705 in ConcurrentMarkSweepGeneration::collect(bool,bool,unsigned int,bool) () from /usr/java/jdk1.6.0_20/jre/lib/i386/server/libjvm.so #8 0x01227f53 in GenCollectedHeap::do_collection(bool,int) () from /usr/java/jdk1.6.0_20/jre/lib/i386/server/libjvm.so #9 0x0115c7b5 in GenCollectorPolicy::satisfy_failed_allocation(unsigned int,bool) () from /usr/java/jdk1.6.0_20/jre/lib/i386/server/libjvm.so #10 0x0122859c in GenCollectedHeap::satisfy_failed_allocation(unsigned int,bool) () from /usr/java/jdk1.6.0_20/jre/lib/i386/server/libjvm.so #11 0x0158a8ce in VM_GenCollectForAllocation::doit() () from /usr/java/jdk1.6.0_20/jre/lib/i386/server/libjvm.so #12 0x015987e6 in VM_Operation::evaluate() () from /usr/java/jdk1.6.0_20/jre/lib/i386/server/libjvm.so #13 0x01597c93 in VMThread::evaluate_operation(VM_Operation*) () from /usr/java/jdk1.6.0_20/jre/lib/i386/server/libjvm.so #14 0x01597f0f in VMThread::loop() () from /usr/java/jdk1.6.0_20/jre/lib/i386/server/libjvm.so #15 0x015979f0 in VMThread::run() () from /usr/java/jdk1.6.0_20/jre/lib/i386/server/libjvm.so #16 0x0145c24e in java_start(Thread*) () from /usr/java/jdk1.6.0_20/jre/lib/i386/server/libjvm.so #17 0x00ccd46b in start_thread () from /lib/libpthread.so.0 #18 0x00bc2dbe in clone () from /lib/libc.so.6 看起来JVM线程正在进行一些CMS相关的工作. UPDATE 我已经获得了关于这个问题的更多信息(它发生在已经运行了7天以上的服务器上). 657501.199: [Full GC (System) 657501.199: [CMS: 400352K->313412K(524288K),2.4024120 secs] 660634K->313412K(878208K),[CMS Perm : 29455K->29320K(68568K)],2.4026470 secs] [Times: user=2.39 sys=0.01,real=2.40 secs] 657513.941: [GC 657513.941: [ParNew: 314624K->13999K(353920K),0.0228180 secs] 628036K->327412K(878208K),0.0230510 secs] [Times: user=0.08 sys=0.00,real=0.02 secs] 657523.772: [GC 657523.772: [ParNew: 328623K->17110K(353920K),0.0244910 secs] 642036K->330523K(878208K),0.0247140 secs] [Times: user=0.08 sys=0.00,real=0.02 secs] 657535.473: [GC 657535.473: [ParNew: 331734K->20282K(353920K),0.0259480 secs] 645147K->333695K(878208K),0.0261670 secs] [Times: user=0.11 sys=0.00,real=0.02 secs] .... .... 688346.765: [GC [1 CMS-initial-mark: 485248K(524288K)] 515694K(878208K),0.0343730 secs] [Times: user=0.03 sys=0.00,real=0.04 secs] 688346.800: [CMS-concurrent-mark-start] 688347.964: [CMS-concurrent-mark: 1.083/1.164 secs] [Times: user=2.52 sys=0.09,real=1.16 secs] 688347.964: [CMS-concurrent-preclean-start] 688347.969: [CMS-concurrent-preclean: 0.004/0.005 secs] [Times: user=0.00 sys=0.01,real=0.01 secs] 688347.969: [CMS-concurrent-abortable-preclean-start] CMS: abort preclean due to time 688352.986: [CMS-concurrent-abortable-preclean: 2.351/5.017 secs] [Times: user=3.83 sys=0.38,real=5.01 secs] 688352.987: [GC[YG occupancy: 297806 K (353920 K)]688352.987: [Rescan (parallel),0.1815250 secs]688353.169: [weak refs processing,0.0312660 secs] [1 CMS-remark: 485248K(524288K)] 783055K(878208K),0.2131580 secs] [Times: user=1.13 sys =0.00,real=0.22 secs] 688353.201: [CMS-concurrent-sweep-start] 688353.903: [CMS-concurrent-sweep: 0.660/0.702 secs] [Times: user=0.91 sys=0.07,real=0.70 secs] 688353.903: [CMS-concurrent-reset-start] 688353.912: [CMS-concurrent-reset: 0.008/0.008 secs] [Times: user=0.01 sys=0.00,real=0.01 secs] 688354.243: [GC 688354.243: [ParNew: 344928K->30151K(353920K),0.0305020 secs] 681955K->368044K(878208K),0.0308880 secs] [Times: user=0.15 sys=0.00,real=0.03 secs] .... .... 688943.029: [GC 688943.029: [ParNew: 336531K->17143K(353920K),0.0237360 secs] 813250K->494327K(878208K),0.0241260 secs] [Times: user=0.10 sys=0.00,real=0.03 secs] 688950.620: [GC 688950.620: [ParNew: 331767K->22442K(353920K),0.0344110 secs] 808951K->499996K(878208K),0.0347690 secs] [Times: user=0.11 sys=0.00,real=0.04 secs] 688956.596: [GC 688956.596: [ParNew: 337064K->37809K(353920K),0.0488170 secs] 814618K->515896K(878208K),0.0491550 secs] [Times: user=0.18 sys=0.04,real=0.05 secs] 688961.470: [GC 688961.471: [ParNew (promotion failed): 352433K->332183K(353920K),0.1862520 secs]688961.657: [CMS 我怀疑这个问题与日志中的最后一行有关(我添加了一些“….”,以跳过一些不是有趣的行). 解决方法
它看起来像一个GC调优问题,可能由内存泄漏触发.我建议您添加启用GC日志记录的JVM参数,并查看是否为您提供了任何见解.
不对我 当用尽堆空间(例如,由于存储泄漏)时,通常会看到GC运行越来越频繁,并且使用了越来越大的CPU占用百分比.有一些JVM参数可以设置为避免这种情况,但是它们具有使您的JVM能够更早地使用OOME的效果. (这可能听起来很奇怪,但在许多情况下,这是一件好事,因为它减少了系统重新启动并能够正常运行的时间.) (编辑:李大同) 【声明】本站内容均来自网络,其相关言论仅代表作者个人观点,不代表本站立场。若无意侵犯到您的权利,请及时与联系站长删除相关内容! |