我已经在CentOS6.5服务器上安装了Cassandra 2.0,并且在测试简单记录时一切正常,现在我必须上传6000亿行,当我在cqlsh上使用COPY时,它在5分钟后失败,并且插入的大约行是20万行rpc超时,然后我选择了pycasso并解析了csv并尝试使用insert命令导入,在每10K记录之后,我们选择关闭连接并再次开发新连接 . 但在大约6万条记录之后,它因超时而失败 .

我的调试跟踪显示了这一点,而服务器不接受插入,没有任何活动,它仍然很忙 .

DEBUG [OptionalTasks:1] 2014-05-30 04:34:16,305 MeteredFlusher.java (line 41) Currently flushing 269227480 bytes of 2047868928 max
DEBUG [OptionalTasks:1] 2014-05-30 04:34:17,306 MeteredFlusher.java (line 41) Currently flushing 269227480 bytes of 2047868928 max
DEBUG [OptionalTasks:1] 2014-05-30 04:34:18,306 MeteredFlusher.java (line 41) Currently flushing 269227480 bytes of 2047868928 max
DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,012 ColumnFamilyStore.java (line 298) retryPolicy for schema_triggers is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,012 ColumnFamilyStore.java (line 298) retryPolicy for compaction_history is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,012 ColumnFamilyStore.java (line 298) retryPolicy for batchlog is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,012 ColumnFamilyStore.java (line 298) retryPolicy for sstable_activity is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,012 ColumnFamilyStore.java (line 298) retryPolicy for peer_events is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,012 ColumnFamilyStore.java (line 298) retryPolicy for compactions_in_progress is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,013 ColumnFamilyStore.java (line 298) retryPolicy for hints is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,013 ColumnFamilyStore.java (line 298) retryPolicy for schema_keyspaces is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,013 ColumnFamilyStore.java (line 298) retryPolicy for range_xfers is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,013 ColumnFamilyStore.java (line 298) retryPolicy for schema_columnfamilies is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,013 ColumnFamilyStore.java (line 298) retryPolicy for NodeIdInfo is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,013 ColumnFamilyStore.java (line 298) retryPolicy for paxos is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,013 ColumnFamilyStore.java (line 298) retryPolicy for schema_columns is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,014 ColumnFamilyStore.java (line 298) retryPolicy for IndexInfo is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,014 ColumnFamilyStore.java (line 298) retryPolicy for peers is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,014 ColumnFamilyStore.java (line 298) retryPolicy for local is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,307 MeteredFlusher.java (line 41) Currently flushing 269227480 bytes of 2047868928 max
DEBUG [OptionalTasks:1] 2014-05-30 04:34:20,307 MeteredFlusher.java (line 41) Currently flushing 269227480 bytes of 2047868928 max
DEBUG [OptionalTasks:1] 2014-05-30 04:34:20,716 ColumnFamilyStore.java (line 298) retryPolicy for backup_calls is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:20,716 ColumnFamilyStore.java (line 298) retryPolicy for sessions is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:20,716 ColumnFamilyStore.java (line 298) retryPolicy for events is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:21,308 MeteredFlusher.java (line 41) Currently flushing 269

当我尝试插入记录时,它在调试日志中显示如下错误 .

DEBUG [OptionalTasks:1] 2014-05-30 04:34:40,717 ColumnFamilyStore.java (line 298) retryPolicy for backup_calls is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:40,717 ColumnFamilyStore.java (line 298) retryPolicy for sessions is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:40,718 ColumnFamilyStore.java (line 298) retryPolicy for events is 0.99
DEBUG [Thrift:24] 2014-05-30 04:34:40,775 CustomTThreadPoolServer.java (line 211) Thrift transport error occurred during processing of message.
org.apache.thrift.transport.TTransportException
    at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
    at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
    at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
    at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
    at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
    at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:362)
    at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:284)
    at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:191)
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
    at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:201)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
DEBUG [Thrift:19] 2014-05-30 04:34:40,775 CustomTThreadPoolServer.java (line 211) Thrift transport error occurred during processing of message.
org.apache.thrift.transport.TTransportException
    at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
    at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
    at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
    at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
    at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
    at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:362)
    at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:284)
    at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:191)
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
    at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:201)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
DEBUG [Thrift:21] 2014-05-30 04:34:40,775 CustomTThreadPoolServer.java (line 211) Thrift transport error occurred during processing of message.
org.apache.thrift.transport.TTransportException
    at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
    at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
    at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
    at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
    at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
    at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:362)
    at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:284)
    at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:191)
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
    at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:201)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
DEBUG [Thrift:1] 2014-05-30 04:34:40,775 CustomTThreadPoolServer.java (line 211) Thrift transport error occurred during processing of message.
org.apache.thrift.transport.TTransportException
    at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
    at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
    at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
    at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
    at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
    at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:362)
    at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:284)
    at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:191)
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
    at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:201)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
DEBUG [Thrift:21] 2014-05-30 04:34:40,789 CassandraServer.java (line 949) batch_mutate
DEBUG [Thrift:19] 2014-05-30 04:34:41,314 CassandraServer.java (line 949) batch_mutate
DEBUG [OptionalTasks:1] 2014-05-30 04:34:41,317 MeteredFlusher.java (line 41) Currently flushing 269227480 bytes of 2047868928 max
DEBUG [Thrift:24] 2014-05-30 04:34:41,858 CassandraServer.java (line 949) batch_mutate

关于发生了什么的任何想法?,插入失败后,它将无法工作,直到服务器重新启动 .

Node Details:

只有一个节点,详细信息如下,

# nodetool describecluster
Cluster Information:
    Name: Test Cluster
    Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
    Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
    Schema versions:
        93403140-5bd6-3cfa-9973-b18df44227f9: [127.0.0.1]

My Node is up and running?

# nodetool status
Note: Ownership information does not include topology; for complete information, specify a keyspace
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address    Load       Owns   Host ID                               Token                                    Rack
UN  127.0.0.1  35.05 MB   100.0%  6bc340b2-4534-400f-b23d-9fe3ed0c6971  -9120530846114649324                     rack1

Server Details:

OS CentOS release 6.5 (Final) 
hdd 8Tb all SSD Drives 
RAM 32GB 
Proccessor: model name : Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz, 24 Core(2 physical

处理器,每个12核心)