Datastax 5.0.3 Solr node does not reindex just goe

2019-09-14 06:36发布

I am trying to reindex one of my indexes deleting old data and full-index. When i try to do this. The node immediately goes down, i cannot run nodetool commands such as nodetool status, nodetool tpstats.Also, i cannot see CPU, disk, network. The node is idle. When i look at the processes on my machine, I can see that is cassandra process.

my schema.xml like that

<field indexed="true" name="key1"   stored="true" type="StrField"/>
<field indexed="true" name="key2"   stored="true" type="StrField"/>
<field indexed="true" name="key3"   stored="true" type="StrField"/>
<field indexed="true" name="key4"   stored="true" type="TrieIntField"/>
<field indexed="true" name="key5"   stored="true" type="StrField"/>
<field indexed="true" name="key6"   stored="true" type="TrieIntField"/>
<field indexed="true" name="key7"   stored="true" type="StrField"/>
<field indexed="true" name="key8"   stored="true" type="StrField"/>
<field indexed="true" name="key9"   stored="true" type="StrField"/>
<field indexed="true" name="key10"  stored="true" type="StrField"/>
<field indexed="true" name="key11"  stored="true" type="StrField"/>
<field indexed="true" name="key12"  stored="true" type="TrieDateField"/>
<field indexed="true" name="key13"  stored="true" type="StrField"/>
<field indexed="true" name="key14"  stored="true" type="StrField"/>
<field indexed="true" name="key15"  stored="true" type="StrField"/>
<field indexed="true" name="key16"  stored="true" type="StrField"/>
<field indexed="true" name="key17"  stored="true" type="StrField"/>
<field indexed="true" name="key18"  stored="true" type="StrField"/>
<field indexed="true" name="key19"  stored="true" type="StrField"/>
<field indexed="true" name="key20"  stored="true" type="TrieIntField"/>
<field indexed="true" name="key21"  stored="true" type="TrieIntField"/>
<field indexed="true" name="col1"   stored="true" type="TrieDoubleField"/>
<field indexed="true" name="col2"   stored="true" type="TrieDoubleField"/>
<field indexed="true" name="col3"   stored="true" type="TrieLongField"/>
<uniqueKey>(key1 ... key21)</uniqueKey>

java settings

<pre>
-javaagent:/usr/share/dse/cassandra/lib/jamm-0.3.0.jar
-XX:+UseThreadPriorities
-XX:ThreadPriorityPolicy=42
-Xms32G
-Xmx32G
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/disk1/cassandra/dumps/cassandra-1477648403-pid42715.hprof
-Xss512k
-XX:+AlwaysPreTouch
-XX:-UseBiasedLocking
-XX:StringTableSize=1000003
-XX:+UseTLAB
-XX:+ResizeTLAB
-XX:CompileCommandFile=/etc/dse//cassandra/hotspot_compiler
-XX:+UseG1GC
-XX:G1RSetUpdatingPauseTimePercent=5
-XX:MaxGCPauseMillis=200
-XX:ParallelGCThreads=16 changed defaults settings
-XX:ConcGCThreads=16 changed defaults settings
-XX:InitiatingHeapOccupancyPercent=40
-XX:+PrintGCDetails
-XX:+PrintGCDateStamps
-XX:+PrintHeapAtGC
-XX:+PrintTenuringDistribution
-XX:+PrintGCApplicationStoppedTime
-XX:+PrintPromotionFailure
-XX:PrintFLSStatistics=1
-Xloggc:/var/log/cassandra/gc-1477648403.log
-Xloggc:/var/log/cassandra/gc.log
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=10
-XX:GCLogFileSize=10M

The pause times when i started the re-index

2016-10-27T22:25:36.381+0200: 16133.424: Total time for which application threads were stopped: 28.6449951 seconds, Stopping threads took: 0.0000877 seconds
2016-10-27T22:25:36.396+0200: 16133.440: Total time for which application threads were stopped: 0.0152024 seconds, Stopping threads took: 0.0001349 seconds
2016-10-27T22:25:36.409+0200: 16133.452: Total time for which application threads were stopped: 0.0123116 seconds, Stopping threads took: 0.0000487 seconds
2016-10-27T22:25:50.745+0200: 16147.788: Total time for which application threads were stopped: 14.3359300 seconds, Stopping threads took: 0.0000616 seconds
2016-10-27T22:25:50.761+0200: 16147.804: Total time for which application threads were stopped: 0.0159109 seconds, Stopping threads took: 0.0001860 seconds
2016-10-27T22:25:50.774+0200: 16147.818: Total time for which application threads were stopped: 0.0136620 seconds, Stopping threads took: 0.0000423 seconds
2016-10-27T22:26:04.853+0200: 16161.897: Total time for which application threads were stopped: 14.0787608 seconds, Stopping threads took: 0.0000590 seconds
2016-10-27T22:26:04.871+0200: 16161.914: Total time for which application threads were stopped: 0.0171435 seconds, Stopping threads took: 0.0001669 seconds
2016-10-27T22:26:18.949+0200: 16175.993: Total time for which application threads were stopped: 14.0786946 seconds, Stopping threads took: 0.0000607 seconds
2016-10-27T22:26:18.968+0200: 16176.011: Total time for which application threads were stopped: 0.0184226 seconds, Stopping threads took: 0.0002338 seconds
2016-10-27T22:26:18.979+0200: 16176.022: Total time for which application threads were stopped: 0.0108393 seconds, Stopping threads took: 0.0000463 seconds
2016-10-27T22:26:33.210+0200: 16190.254: Total time for which application threads were stopped: 14.2314474 seconds, Stopping threads took: 0.0000724 seconds
2016-10-27T22:26:33.228+0200: 16190.272: Total time for which application threads were stopped: 0.0172137 seconds, Stopping threads took: 0.0000845 seconds
2016-10-27T22:26:33.243+0200: 16190.286: Total time for which application threads were stopped: 0.0147185 seconds, Stopping threads took: 0.0000770 seconds
2016-10-27T22:26:47.315+0200: 16204.358: Total time for which application threads were stopped: 14.0715678 seconds, Stopping threads took: 0.0001087 seconds
2016-10-27T22:26:47.335+0200: 16204.379: Total time for which application threads were stopped: 0.0203852 seconds, Stopping threads took: 0.0002385 seconds
2016-10-27T22:26:47.347+0200: 16204.391: Total time for which application threads were stopped: 0.0120962 seconds, Stopping threads took: 0.0001291 seconds
2016-10-27T22:27:01.590+0200: 16218.633: Total time for which application threads were stopped: 14.2425348 seconds, Stopping threads took: 0.0000658 seconds
2016-10-27T22:27:01.608+0200: 16218.652: Total time for which application threads were stopped: 0.0175375 seconds, Stopping threads took: 0.0002150 seconds
2016-10-27T22:27:01.622+0200: 16218.665: Total time for which application threads were stopped: 0.0134276 seconds, Stopping threads took: 0.0000305 seconds
2016-10-27T22:27:15.820+0200: 16232.863: Total time for which application threads were stopped: 14.1976605 seconds, Stopping threads took: 0.0000580 seconds
2016-10-27T22:27:15.839+0200: 16232.883: Total time for which application threads were stopped: 0.0181183 seconds, Stopping threads took: 0.0001094 seconds
2016-10-27T22:27:15.854+0200: 16232.898: Total time for which application threads were stopped: 0.0146620 seconds, Stopping threads took: 0.0000391 seconds
2016-10-27T22:27:29.911+0200: 16246.954: Total time for which application threads were stopped: 14.0568599 seconds, Stopping threads took: 0.0000689 seconds
2016-10-27T22:27:29.930+0200: 16246.973: Total time for which application threads were stopped: 0.0184539 seconds, Stopping threads took: 0.0002259 seconds
2016-10-27T22:27:29.942+0200: 16246.985: Total time for which application threads were stopped: 0.0120611 seconds, Stopping threads took: 0.0000643 seconds
2016-10-27T22:27:44.239+0200: 16261.282: Total time for which application threads were stopped: 14.2971941 seconds, Stopping threads took: 0.0000682 seconds
2016-10-27T22:27:44.256+0200: 16261.299: Total time for which application threads were stopped: 0.0165530 seconds, Stopping threads took: 0.0001018 seconds
2016-10-27T22:27:44.268+0200: 16261.312: Total time for which application threads were stopped: 0.0126950 seconds, Stopping threads took: 0.0000236 seconds
2016-10-27T22:27:58.407+0200: 16275.451: Total time for which application threads were stopped: 14.1388198 seconds, Stopping threads took: 0.0000495 seconds
2016-10-27T22:27:58.426+0200: 16275.470: Total time for which application threads were stopped: 0.0189899 seconds, Stopping threads took: 0.0001773 seconds
2016-10-27T22:27:58.437+0200: 16275.480: Total time for which application threads were stopped: 0.0104863 seconds, Stopping threads took: 0.0000825 seconds
2016-10-27T22:28:12.621+0200: 16289.664: Total time for which application threads were stopped: 14.1838416 seconds, Stopping threads took: 0.0000498 seconds
2016-10-27T22:28:12.638+0200: 16289.681: Total time for which application threads were stopped: 0.0167055 seconds, Stopping threads took: 0.0002064 seconds
2016-10-27T22:28:12.652+0200: 16289.696: Total time for which application threads were stopped: 0.0140258 seconds, Stopping threads took: 0.0000369 seconds

Here is my dstat result

 read  writ|run blk new| used  buff  cach  free|  in   out | read  writ| int   csw |usr sys idl wai hiq siq| recv  send
   0  0.50 |1.0   0 0.4|39.4G 47.9M 22.0G 1384M|   0     0 |   0  7782B| 599  3866 |  3   0  97   0   0   0|2102B 1587B
   0  0.40 |1.0   0 0.1|39.4G 47.9M 22.0G 1384M|   0     0 |   0  2458B|3448  3917 |  3   0  97   0   0   0|1533B 1212B
   0  1.50 |1.1   0 0.3|39.4G 47.9M 22.0G 1387M|   0     0 |   0  8192B| 575  3888 |  3   0  97   0   0   0|1448B  993B
   0  0.80 |4.2   0   0|39.4G 47.9M 22.0G 1386M|   0     0 |   0  5325B|5308  4954 | 18   0  82   0   0   0|1631B 1109B
   0  0.20 |1.0   0 0.2|39.4G 47.9M 22.0G 1386M|   0     0 |   0  2048B| 792  4099 |  3   0  97   0   0   0|2016B 1670B
   0  0.90 |1.0   0   0|39.4G 47.9M 22.0G 1388M|   0     0 |   0  5325B| 988  3838 |  3   0  97   0   0   0|1697B 1150B
   0  0.20 |1.0   0 0.4|39.4G 47.9M 22.0G 1389M|   0     0 |   0  1638B| 624  3851 |  3   0  97   0   0   0|1559B 1126B
   0  0.60 |1.0   0 0.1|39.4G 47.9M 22.0G 1390M|   0     0 |   0  5734B| 698  3801 |  3   0  97   0   0   0|1654B 1091B
   0  0.60 |7.4   0   0|39.4G 47.9M 22.0G 1386M|   0     0 |   0  4506B|5171  5278 | 18   0  82   0   0   0|1512B 1402B
   0  0.50 |1.0   0   0|39.4G 47.9M 22.0G 1389M|   0     0 |   0  2048B| 604  3813 |  3   0  97   0   0   0|1669B 1064B
   0  0.70 |1.1   0 202|39.4G 47.9M 22.0G 1375M|   0     0 |   0  9011B|2527  5829 |  3   1  96   0   0   0|2682B 6059B
   0  0.50 |1.2   0 126|39.4G 47.9M 22.0G 1379M|   0     0 |   0    10k| 870  5047 |  3   0  96   0   0   0|2179B 3525B
   0  0.40 |4.1   0 0.3|39.4G 48.0M 22.0G 1380M|   0     0 |   0  2458B|6802  5359 | 18   0  82   0   0   0|1723B 1943B
   0  3.00 |1.0   0 0.1|39.4G 48.0M 22.0G 1384M|   0     0 |   0    19k| 594  3874 |  3   0  97   0   0   0|2028B 1434B
   0  0.20 |1.0   0   0|39.4G 48.0M 22.0G 1386M|   0     0 |   0  2867B|1622  3889 |  3   0  97   0   0   0|2070B 1449B
   0  0.60 |1.0 0.1   0|39.4G 48.0M 22.0G 1387M|   0     0 |   0  4506B| 604  3837 |  3   0  97   0   0   0|2043B 1369B

0条回答
登录 后发表回答