Solr突然消耗整个堆

Solr Version：6.5,Master with 1 Slave,配置与下面提到的相同.

JVM配置：

-Xms2048m
     -Xmx4096m
     -XX:+ParallelRefProcEnabled
     -XX:+UseCMSInitiatingOccupancyOnly
     -XX:CMSInitiatingOccupancyFraction=50

休息全部是默认值.

Solr配置：

<autoCommit>
      <!-- Auto hard commit in 5 minutes -->
      <maxTime>{solr.autoCommit.maxTime:300000}</maxTime>
      <openSearcher>false</openSearcher>
    </autoCommit>
    <autoSoftCommit>
    <!-- Auto soft commit in 15 minutes -->
      <maxTime>{solr.autoSoftCommit.maxTime:900000}</maxTime>
    </autoSoftCommit>
    </updateHandler>

    <query>
      <maxBooleanClauses>1024</maxBooleanClauses>
      <filterCache class="solr.FastLRUCache" size="8192" initialSize="8192" autowarmCount="0" />
      <queryResultCache class="solr.LRUCache" size="8192" initialSize="4096" autowarmCount="0" />
      <documentCache class="solr.LRUCache" size="12288" initialSize="12288" autowarmCount="0" />
      <cache name="perSegFilter" class="solr.search.LRUCache" size="10" initialSize="0" autowarmCount="10" regenerator="solr.NoOpRegenerator" />
      <enableLazyFieldLoading>true</enableLazyFieldLoading>
      <queryResultWindowSize>20</queryResultWindowSize>
      <queryResultMaxDocsCached>${solr.query.max.docs:40}
      </queryResultMaxDocsCached>
      <useColdSearcher>false</useColdSearcher>
      <maxWarmingSearchers>2</maxWarmingSearchers>
    </query>

主机(AWS)配置为：

RAM: 7.65GB
Cores: 4

现在,我们的solr可以完美地工作几个小时甚至几天,但有时会突然内存跳起来并且GC开始导致长时间的大停顿并没有太多恢复.
我们看到在硬提交后添加或删除一个或多个段时最常发生这种情况.索引的文档数量并不重要.附加的图像显示只有1个文档被索引,导致添加一个段,并且在我们重新启动Solr之前它们都搞砸了.

以下是来自NewRelic和Sematext的图片(请点击链接查看)：

JVM Heap Memory Image

1 Document and 1 Segment addition Image

更新：这是SOLR上次死亡时的JMap输出,我们现在将JVM内存增加到12GB的xmx：

num     #instances         #bytes  class name
  ----------------------------------------------
  1:      11210921     1076248416  org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat$IntBlockTermState
  2:      10623486      934866768  [Lorg.apache.lucene.index.TermState;
  3:      15567646      475873992  [B
  4:      10623485      424939400  org.apache.lucene.search.spans.SpanTermQuery$SpanTermWeight
  5:      15508972      372215328  org.apache.lucene.util.BytesRef
  6:      15485834      371660016  org.apache.lucene.index.Term
  7:      15477679      371464296  org.apache.lucene.search.spans.SpanTermQuery
  8:      10623486      339951552  org.apache.lucene.index.TermContext
  9:       1516724      150564320  [Ljava.lang.Object;
 10:        724486       50948800  [C
 11:       1528110       36674640  java.util.ArrayList
 12:        849884       27196288  org.apache.lucene.search.spans.SpanNearQuery
 13:        582008       23280320  org.apache.lucene.search.spans.SpanNearQuery$SpanNearWeight
 14:        481601       23116848  org.apache.lucene.document.FieldType
 15:        623073       19938336  org.apache.lucene.document.StoredField
 16:        721649       17319576  java.lang.String
 17:         32729        7329640  [J
 18:         14643        5788376  [F
 19:        137126        4388032  java.util.HashMap$Node
 20:         52990        3391360  java.nio.DirectByteBufferR
 21:        131072        3145728  org.apache.solr.update.VersionBucket
 22:         20535        2891536  [I
 23:         99073        2377752  shaded.javassist.bytecode.Utf8Info
 24:         47788        1911520  java.util.TreeMap$Entry
 25:         34118        1910608  org.apache.lucene.index.FieldInfo
 26:         26511        1696704  org.apache.lucene.store.ByteBufferIndexInput$SingleBufferImpl
 27:         17470        1677120  org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer$NumericEntry
 28:         13762        1526984  java.lang.Class
 29:          7323        1507408  [Ljava.util.HashMap$Node;
 30:          2331        1230768  [Lshaded.javassist.bytecode.ConstInfo;
 31:         18929        1211456  com.newrelic.agent.deps.org.objectweb.asm.Label
 32:         25360        1014400  java.util.LinkedHashMap$Entry
 33:         41388         993312  java.lang.Long

Solr上的负载并不多 – 最多可达到每分钟2000个请求.索引负载有时可以是突发的,但大部分时间都非常低.但是如上所述,有时即使是单个文档索引也可以使solr进入tizzy,有时候它就像魅力一样.

关于我们可能出错的地方的任何指示都会很棒.