0

问题描述:
生产环境在业务没有抖高的情景下,shard数据节点机器出现负载突然提高几分钟?期间基础资源包括 内存、io和网络均正常,但是节点日志显示一直在Finding the split vector for ctu.mobileToken over { token: 1.0 } keyCount: 12719 numSplits: 106 lookedAt: 9036 took 1764ms,具体日志内容如下:

2019-02-20T03:22:41.209+0000 W SHARDING [conn68254] Finding the split vector for ctu.mobileToken over { token: 1.0 } keyCount: 12719 numSplits: 106 lookedAt: 9122 took 1476ms
2019-02-20T03:22:41.210+0000 I COMMAND [conn68254] command admin.$cmd command: splitVector { splitVector: “ctu.mobileToken”, keyPattern: { token: 1.0 }, min: { token: “5c6bc051xbpFF1evtQv1JhORbtFRiBkFeIl8Ws83” }, max: { token: MaxKey }, maxChunkSizeBytes: 67108864 } numYields:10610 reslen:6628 locks:{ Global: { acquireCount: { r: 21222 } }, Database: { acquireCount: { r: 10611 } }, Collection: { acquireCount: { r: 10611 }, acquireWaitCount: { r: 8 }, timeAcquiringMicros: { r: 41510 } } } protocol:op_command 1476ms
2019-02-20T03:22:41.211+0000 I SHARDING [conn68254] received splitChunk request: { splitChunk: “ctu.mobileToken”, configdb: “crs/10.60.96.104:21001,10.60.96.81:21001,10.60.96.92:21001”, from: “rs3”, keyPattern: { token: 1.0 }, shardVersion: [ Timestamp 4419000|633, ObjectId(‘5bc858a5c9ebaf2f1aafd8db’) ], min: { token: “5c6bc051xbpFF1evtQv1JhORbtFRiBkFeIl8Ws83” }

2019-02-20T03:22:41.214+0000 I SHARDING [conn68254] distributed lock ‘ctu.mobileToken’ acquired for ‘splitting chunk [{ token: “5c6bc051xbpFF1evtQv1JhORbtFRiBkFeIl8Ws83” }, { token: MaxKey }) in ctu.mobileToken’, ts : 5c6cc801429d3c8981b93f16
2019-02-20T03:22:41.214+0000 I SHARDING [conn68254] MetadataLoader loading chunks for ctu.mobileToken based on: 4419|761||5bc858a5c9ebaf2f1aafd8db
2019-02-20T03:22:41.219+0000 I SHARDING [conn68254] MetadataLoader took 4 ms and found version 4419|761||5bc858a5c9ebaf2f1aafd8db
2019-02-20T03:22:41.232+0000 I SHARDING [conn68254] distributed lock with ts: 5c6cc801429d3c8981b93f16′ unlocked.
2019-02-20T03:22:41.334+0000 W SHARDING [conn68256] Finding the split vector for ctu.mobileToken over { token: 1.0 } keyCount: 12719 numSplits: 106 lookedAt: 9131 took 858ms
2019-02-20T03:22:41.334+0000 I COMMAND [conn68256] command admin.$cmd command: splitVector { splitVector: “ctu.mobileToken”, keyPattern: { token: 1.0 }, min: { token: “5c6bc051xbpFF1evtQv1JhORbtFRiBkFeIl8Ws83” }, max: { token: MaxKey }, maxChunkSizeBytes: 67108864 } numYields:10607 reslen:6628 locks:{ Global: { acquireCount: { r: 21216 } }, Database: { acquireCount: { r: 10608 } }, Collection: { acquireCount: { r: 10608 }, acquireWaitCount: { r: 7 }, timeAcquiringMicros: { r: 15724 } } } protocol:op_command 864ms
2019-02-20T03:22:41.335+0000 I SHARDING [conn68256] received splitChunk request: { splitChunk: “ctu.mobileToken”, configdb: “crs/10.60.96.104:21001,10.60.96.81:21001,10.60.96.92:21001”, from: “rs3”, keyPattern: { token: 1.0 }, shardVersion: [ Timestamp 4419000|633, ObjectId(‘5bc858a5c9ebaf2f1aafd8db’) ], min: { token: “5c6bc051xbpFF1evtQv1JhORbtFRiBkFeIl8Ws83” }, max: { token: MaxKey }, splitKeys: [ { token: “5c6bc2301lWzOdbLzOYXcibfkgMDHi7XKN4xkRW3” }, { token: “5c6bc40dU74I7yewg2ifmA2XCUEtrwvwlmzRP0m3” }, { token: “5c6bc5f0tTGE1yJKpZIkoISq99cmTTqeu74XKTe3” }, { token: “5c6bc7d8pm0SayzAzdIC8gYssmU6PjGhozzv0iF3” }
2019-02-20T03:22:41.340+0000 I SHARDING [conn68256] distributed lock ‘ctu.mobileToken’ acquired for ‘splitting chunk [{ token: “5c6bc051xbpFF1evtQv1JhORbtFRiBkFeIl8Ws83” }
2019-02-20T03:22:41.340+0000 I SHARDING [conn68256] MetadataLoader loading chunks for ctu.mobileToken based on: 4419|761||5bc858a5c9ebaf2f1aafd8db
2019-02-20T03:22:41.379+0000 I SHARDING [conn68256] MetadataLoader took 38 ms and found version 4419|761||5bc858a5c9ebaf2f1aafd8db
2019-02-20T03:22:41.392+0000 I SHARDING [conn68256] distributed lock with ts: 5c6cc801429d3c8981b93f2b’ unlocked.
2019-02-20T03:22:41.515+0000 W SHARDING [conn68250] Finding the split vector for ctu.mobileToken over { token: 1.0 } keyCount: 12719 numSplits: 106 lookedAt: 9140 took 1461ms
2019-02-20T03:22:41.515+0000 I COMMAND [conn68250] command admin.$cmd command: splitVector { splitVector: “ctu.mobileToken”, keyPattern: { token: 1.0 }, min: { token: “5c6bc051xbpFF1evtQv1JhORbtFRiBkFeIl8Ws83” }, max: { token: MaxKey }, maxChunkSizeBytes: 67108864 } numYields:10612 reslen:6628 locks:{ Global: { acquireCount: { r: 21226 } }, Database: { acquireCount: { r: 10613 } }, Collection: { acquireCount: { r: 10613 }, acquireWaitCount: { r: 9 }, timeAcquiringMicros: { r: 69722 } } } protocol:op_command 1461ms
2019-02-20T03:22:41.516+0000 I SHARDING [conn68250] received splitChunk request: { splitChunk: “ctu.mobileToken”, configdb: “crs/10.60.96.104:21001,10.60.96.81:21001,10.60.96.92:21001”, from: “rs3”, keyPattern: { token: 1.0 }, shardVersion: [ Timestamp 4419000|633, ObjectId(‘5bc858a5c9ebaf2f1aafd8db’) ], min: { token: “5c6bc051xbpFF1evtQv1JhORbtFRiBkFeIl8Ws83” }