0

最近使用mongodb(4.4)做网站后台数据库,测试环境,服务器系统是ubuntu(20.04版本)。

1、出现问题之前的异常情况

(1)在出现问题前mongodb的版本为4.4.2,后来ubuntu提示系统有新应用升级,我就整个系统都升级了。之后mongodb版本为目前的4.4.4。

(2)出现问题之前ubuntu偶尔出现死机重启情形

2、问题描述

某次ubuntu不明原因的重启后,mongodb服务就无法识别现有的数据了。mongod.conf配置文件中的数据路径dataPath是/data/db文件夹中的数据文件还在。

(1)文件列表如下:

“`

khadas@Khadas:/data/db$ ls -l
总用量 7072000
-rwxrwxrwx 1 mongodb mongodb 6545014784 2月 24 21:45 collection-0-1303375794618941371.wt
-rwxrwxrwx 1 mongodb mongodb 32768 2月 27 22:24 collection-0–1865719318303329981.wt
-rwxrwxrwx 1 mongodb mongodb 32768 2月 24 21:45 collection-0–2995259156705621303.wt
-rwxrwxrwx 1 mongodb mongodb 32768 2月 24 21:45 collection-0–3148383109918699076.wt
-rwxrwxrwx 1 mongodb mongodb 502919168 2月 24 21:45 collection-0–3402138688896876418.wt
-rwxrwxrwx 1 mongodb mongodb 32768 2月 24 21:45 collection-0–3862035505806964401.wt
-rwxrwxrwx 1 mongodb mongodb 32768 2月 24 21:45 collection-0-4811784969401579968.wt
-rwxrwxrwx 1 mongodb mongodb 356352 2月 24 21:45 collection-0-5135274076854948045.wt
-rwxrwxrwx 1 mongodb mongodb 32768 2月 27 22:24 collection-0-5349369052361523801.wt
-rwxrwxrwx 1 mongodb mongodb 655360 2月 24 21:45 collection-0-5708321346472201446.wt
-rwxrwxrwx 1 mongodb mongodb 32768 2月 27 22:24 collection-0-7421180676960825112.wt
-rwxrwxrwx 1 mongodb mongodb 270336 2月 24 21:45 collection-0–8494028491394461314.wt
-rwxrwxrwx 1 mongodb mongodb 36864 2月 27 22:24 collection-0-8625578445850430571.wt
-rwxrwxrwx 1 mongodb mongodb 73728 2月 27 22:24 collection-2–3862035505806964401.wt
-rwxrwxrwx 1 mongodb mongodb 40960 2月 24 21:45 collection-2-4873769553459830491.wt
-rwxrwxrwx 1 mongodb mongodb 778240 2月 24 21:45 collection-2–4990034590743048156.wt
-rwxrwxrwx 1 mongodb mongodb 36864 2月 27 22:24 collection-2-8625578445850430571.wt
-rwxrwxrwx 1 mongodb mongodb 36864 2月 27 22:24 collection-4–3862035505806964401.wt
-rwxrwxrwx 1 mongodb mongodb 129155072 2月 24 21:45 collection-9–3862035505806964401.wt
drwxrwxrwx 2 mongodb mongodb 12288 2月 27 22:24 diagnostic.data
-rwxrwxrwx 1 mongodb mongodb 1286144 2月 24 21:45 index-0-4873769553459830491.wt
-rwxrwxrwx 1 mongodb mongodb 94208 2月 24 21:45 index-0-5102593680096001999.wt
-rwxrwxrwx 1 mongodb mongodb 24313856 2月 24 21:45 index-10–3862035505806964401.wt
-rwxrwxrwx 1 mongodb mongodb 1552384 2月 24 21:45 index-1-1303375794618941371.wt
-rwxrwxrwx 1 mongodb mongodb 32768 2月 27 22:24 index-1–1865719318303329981.wt
-rwxrwxrwx 1 mongodb mongodb 32768 2月 24 21:45 index-1–2995259156705621303.wt
-rwxrwxrwx 1 mongodb mongodb 32768 2月 24 21:45 index-1–3148383109918699076.wt
-rwxrwxrwx 1 mongodb mongodb 524288 2月 24 21:45 index-1–3402138688896876418.wt
-rwxrwxrwx 1 mongodb mongodb 32768 2月 24 21:45 index-1–3862035505806964401.wt
-rwxrwxrwx 1 mongodb mongodb 32768 2月 24 21:45 index-1-4811784969401579968.wt
-rwxrwxrwx 1 mongodb mongodb 65536 2月 24 21:45 index-1-5135274076854948045.wt
-rwxrwxrwx 1 mongodb mongodb 36864 2月 24 21:45 index-1-5349369052361523801.wt
-rwxrwxrwx 1 mongodb mongodb 69632 2月 24 21:45 index-1-5708321346472201446.wt
-rwxrwxrwx 1 mongodb mongodb 32768 2月 27 22:24 index-1-7421180676960825112.wt
-rwxrwxrwx 1 mongodb mongodb 81920 2月 24 21:45 index-1–8494028491394461314.wt
-rwxrwxrwx 1 mongodb mongodb 36864 2月 27 22:24 index-1-8625578445850430571.wt
-rwxrwxrwx 1 mongodb mongodb 32768 2月 27 22:24 index-2–1865719318303329981.wt
-rwxrwxrwx 1 mongodb mongodb 32768 2月 24 21:45 index-2–2995259156705621303.wt
-rwxrwxrwx 1 mongodb mongodb 520192 2月 24 21:45 index-2-5102593680096001999.wt
-rwxrwxrwx 1 mongodb mongodb 31113216 2月 24 21:45 index-2–567908579256883281.wt
-rwxrwxrwx 1 mongodb mongodb 32768 2月 27 22:24 index-3–3862035505806964401.wt
-rwxrwxrwx 1 mongodb mongodb 32768 2月 24 21:45 index-3-4873769553459830491.wt
-rwxrwxrwx 1 mongodb mongodb 32768 2月 24 21:45 index-3–4990034590743048156.wt
-rwxrwxrwx 1 mongodb mongodb 36864 2月 27 22:24 index-3-8625578445850430571.wt
-rwxrwxrwx 1 mongodb mongodb 172032 2月 24 21:45 index-4-4873769553459830491.wt
-rwxrwxrwx 1 mongodb mongodb 532480 2月 24 21:45 index-4-5102593680096001999.wt
-rwxrwxrwx 1 mongodb mongodb 36864 2月 27 22:24 index-4-8625578445850430571.wt
-rwxrwxrwx 1 mongodb mongodb 57344 2月 27 22:24 index-5–3862035505806964401.wt
-rwxrwxrwx 1 mongodb mongodb 32768 2月 27 22:24 index-6–3862035505806964401.wt
-rwxrwxrwx 1 mongodb mongodb 348160 2月 24 21:45 index-6-4873769553459830491.wt
-rwxrwxrwx 1 mongodb mongodb 393216 2月 24 21:45 index-6-5102593680096001999.wt
drwxrwxrwx 2 mongodb mongodb 4096 2月 27 22:00 journal
-rwxrwxrwx 1 mongodb mongodb 32768 2月 27 22:24 _mdb_catalog.wt
-rwxrwxrwx 1 mongodb mongodb 32768 2月 24 21:45 _mdb_catalog.wt.a.bak
-rwxrwxrwx 1 mongodb mongodb 0 2月 27 22:24 mongod.lock
-rwxrwxrwx 1 mongodb mongodb 36864 2月 27 22:24 sizeStorer.wt
-rwxrwxrwx 1 mongodb mongodb 114 10月 2 14:14 storage.bson
-rwxrwxrwx 1 mongodb mongodb 47 10月 2 14:14 WiredTiger
-rwxrwxrwx 1 mongodb mongodb 4096 2月 27 22:24 WiredTigerHS.wt
-rwxrwxrwx 1 mongodb mongodb 21 2月 25 09:38 WiredTiger.lock
-rw——- 1 mongodb mongodb 1349 2月 27 22:24 WiredTiger.turtle
-rwxrwxrwx 1 mongodb mongodb 102400 2月 27 22:24 WiredTiger.wt
-rwxrwxrwx 1 mongodb mongodb 118784 2月 24 22:48 WiredTiger.wt.orig.bak

“`

(2)启动时状态提示:

“`

khadas@Khadas:~$ systemctl status mongod
● mongod.service – MongoDB Database Server
Loaded: loaded (/lib/systemd/system/mongod.service; disabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2021-03-01 11:12:38 CST; 21s ago
Docs: https://docs.mongodb.org/manual
Process: 6471 ExecStart=/usr/bin/mongod –config /etc/mongod.conf (code=exited, status=14)
Main PID: 6471 (code=exited, status=14)

3月 01 11:12:38 Khadas mongod[6471]: {“t”:{“$date”:”2021-03-01T11:12:38.109+08:00″},”s”:”I”, “c”:”NETWORK”, “id”:4648601, “ctx”:”main”,”msg”:”Imp>
3月 01 11:12:38 Khadas mongod[6471]: {“t”:{“$date”:”2021-03-01T11:12:38.111+08:00″},”s”:”I”, “c”:”STORAGE”, “id”:4615611, “ctx”:”initandlisten”,”>
3月 01 11:12:38 Khadas mongod[6471]: {“t”:{“$date”:”2021-03-01T11:12:38.111+08:00″},”s”:”I”, “c”:”CONTROL”, “id”:23403, “ctx”:”initandlisten”,”>
3月 01 11:12:38 Khadas mongod[6471]: {“t”:{“$date”:”2021-03-01T11:12:38.111+08:00″},”s”:”I”, “c”:”CONTROL”, “id”:51765, “ctx”:”initandlisten”,”>
3月 01 11:12:38 Khadas mongod[6471]: {“t”:{“$date”:”2021-03-01T11:12:38.111+08:00″},”s”:”I”, “c”:”CONTROL”, “id”:21951, “ctx”:”initandlisten”,”>
3月 01 11:12:38 Khadas mongod[6471]: {“t”:{“$date”:”2021-03-01T11:12:38.112+08:00″},”s”:”F”, “c”:”STORAGE”, “id”:50922, “ctx”:”initandlisten”,”>
3月 01 11:12:38 Khadas mongod[6471]: {“t”:{“$date”:”2021-03-01T11:12:38.112+08:00″},”s”:”F”, “c”:”-“, “id”:23091, “ctx”:”initandlisten”,”>
3月 01 11:12:38 Khadas mongod[6471]: {“t”:{“$date”:”2021-03-01T11:12:38.112+08:00″},”s”:”F”, “c”:”-“, “id”:23092, “ctx”:”initandlisten”,”>
3月 01 11:12:38 Khadas systemd[1]: mongod.service: Main process exited, code=exited, status=14/n/a
3月 01 11:12:38 Khadas systemd[1]: mongod.service: Failed with result ‘exit-code’.
lines 1-17/17 (END)

“`

启动错误时的日志:

“`

root@Khadas:/var/log/mongodb# cat mongod.log

{“t”:{“$date”:”2021-03-01T11:30:42.559+08:00″},”s”:”I”, “c”:”CONTROL”, “id”:20698, “ctx”:”main”,”msg”:”***** SERVER RESTARTED *****”}
{“t”:{“$date”:”2021-03-01T11:30:42.561+08:00″},”s”:”I”, “c”:”CONTROL”, “id”:23285, “ctx”:”main”,”msg”:”Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify –sslDisabledProtocols ‘none'”}
{“t”:{“$date”:”2021-03-01T11:30:42.564+08:00″},”s”:”W”, “c”:”ASIO”, “id”:22601, “ctx”:”main”,”msg”:”No TransportLayer configured during NetworkInterface startup”}
{“t”:{“$date”:”2021-03-01T11:30:42.565+08:00″},”s”:”I”, “c”:”NETWORK”, “id”:4648601, “ctx”:”main”,”msg”:”Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize.”}
{“t”:{“$date”:”2021-03-01T11:30:42.565+08:00″},”s”:”I”, “c”:”STORAGE”, “id”:4615611, “ctx”:”initandlisten”,”msg”:”MongoDB starting”,”attr”:{“pid”:6640,”port”:27017,”dbPath”:”/data/db”,”architecture”:”64-bit”,”host”:”Khadas”}}
{“t”:{“$date”:”2021-03-01T11:30:42.565+08:00″},”s”:”I”, “c”:”CONTROL”, “id”:23403, “ctx”:”initandlisten”,”msg”:”Build Info”,”attr”:{“buildInfo”:{“version”:”4.4.4″,”gitVersion”:”8db30a63db1a9d84bdcad0c83369623f708e0397″,”openSSLVersion”:”OpenSSL 1.1.1f 31 Mar 2020″,”modules”:[],”allocator”:”tcmalloc”,”environment”:{“distmod”:”ubuntu2004″,”distarch”:”aarch64″,”target_arch”:”aarch64″}}}}
{“t”:{“$date”:”2021-03-01T11:30:42.565+08:00″},”s”:”I”, “c”:”CONTROL”, “id”:51765, “ctx”:”initandlisten”,”msg”:”Operating System”,”attr”:{“os”:{“name”:”Ubuntu”,”version”:”20.04″}}}
{“t”:{“$date”:”2021-03-01T11:30:42.565+08:00″},”s”:”I”, “c”:”CONTROL”, “id”:21951, “ctx”:”initandlisten”,”msg”:”Options set by command line”,”attr”:{“options”:{“config”:”/etc/mongod.conf”,”net”:{“bindIp”:”0.0.0.0″,”port”:27017},”security”:{“authorization”:”enabled”},”storage”:{“dbPath”:”/data/db”,”journal”:{“enabled”:true}},”systemLog”:{“destination”:”file”,”logAppend”:true,”path”:”/var/log/mongodb/mongod.log”}}}}
{“t”:{“$date”:”2021-03-01T11:30:42.566+08:00″},”s”:”E”, “c”:”NETWORK”, “id”:23024, “ctx”:”initandlisten”,”msg”:”Failed to unlink socket file”,”attr”:{“path”:”/tmp/mongodb-27017.sock”,”error”:”Operation not permitted”}}
{“t”:{“$date”:”2021-03-01T11:30:42.566+08:00″},”s”:”F”, “c”:”-“, “id”:23091, “ctx”:”initandlisten”,”msg”:”Fatal assertion”,”attr”:{“msgid”:40486,”file”:”src/mongo/transport/transport_layer_asio.cpp”,”line”:919}}
{“t”:{“$date”:”2021-03-01T11:30:42.566+08:00″},”s”:”F”, “c”:”-“, “id”:23092, “ctx”:”initandlisten”,”msg”:”\n\n***aborting after fassert() failure\n\n”}

“`

(3)后来看网上的资料说使用–repair命令,我就试了一下,结果如下:

“`

{“t”:{“$date”:”2021-03-01T09:17:01.416+08:00″},”s”:”I”, “c”:”CONTROL”, “id”:23285, “ctx”:”main”,”msg”:”Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify –sslDisabledProtocols ‘none'”}
{“t”:{“$date”:”2021-03-01T09:17:01.493+08:00″},”s”:”W”, “c”:”ASIO”, “id”:22601, “ctx”:”main”,”msg”:”No TransportLayer configured during NetworkInterface startup”}
{“t”:{“$date”:”2021-03-01T09:17:01.494+08:00″},”s”:”I”, “c”:”NETWORK”, “id”:4648601, “ctx”:”main”,”msg”:”Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize.”}
{“t”:{“$date”:”2021-03-01T09:17:01.504+08:00″},”s”:”I”, “c”:”STORAGE”, “id”:4615611, “ctx”:”initandlisten”,”msg”:”MongoDB starting”,”attr”:{“pid”:5724,”port”:27017,”dbPath”:”/data/db”,”architecture”:”64-bit”,”host”:”Khadas”}}
{“t”:{“$date”:”2021-03-01T09:17:01.505+08:00″},”s”:”I”, “c”:”CONTROL”, “id”:23403, “ctx”:”initandlisten”,”msg”:”Build Info”,”attr”:{“buildInfo”:{“version”:”4.4.4″,”gitVersion”:”8db30a63db1a9d84bdcad0c83369623f708e0397″,”openSSLVersion”:”OpenSSL 1.1.1f 31 Mar 2020″,”modules”:[],”allocator”:”tcmalloc”,”environment”:{“distmod”:”ubuntu2004″,”distarch”:”aarch64″,”target_arch”:”aarch64″}}}}
{“t”:{“$date”:”2021-03-01T09:17:01.505+08:00″},”s”:”I”, “c”:”CONTROL”, “id”:51765, “ctx”:”initandlisten”,”msg”:”Operating System”,”attr”:{“os”:{“name”:”Ubuntu”,”version”:”20.04″}}}
{“t”:{“$date”:”2021-03-01T09:17:01.505+08:00″},”s”:”I”, “c”:”CONTROL”, “id”:21951, “ctx”:”initandlisten”,”msg”:”Options set by command line”,”attr”:{“options”:{“repair”:true}}}
{“t”:{“$date”:”2021-03-01T09:17:01.571+08:00″},”s”:”I”, “c”:”STORAGE”, “id”:22270, “ctx”:”initandlisten”,”msg”:”Storage engine to use detected by data files”,”attr”:{“dbpath”:”/data/db”,”storageEngine”:”wiredTiger”}}
{“t”:{“$date”:”2021-03-01T09:17:01.572+08:00″},”s”:”I”, “c”:”STORAGE”, “id”:22297, “ctx”:”initandlisten”,”msg”:”Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem”,”tags”:[“startupWarnings”]}
{“t”:{“$date”:”2021-03-01T09:17:01.572+08:00″},”s”:”I”, “c”:”STORAGE”, “id”:22315, “ctx”:”initandlisten”,”msg”:”Opening WiredTiger”,”attr”:{“config”:”create,cache_size=1342M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress,compact_progress],”}}
{“t”:{“$date”:”2021-03-01T09:17:01.607+08:00″},”s”:”E”, “c”:”STORAGE”, “id”:22435, “ctx”:”initandlisten”,”msg”:”WiredTiger error”,”attr”:{“error”:13,”message”:”[1614561421:607274][5724:0x7f9d692980], wiredtiger_open: __posix_open_file, 808: /data/db/WiredTiger.turtle: handle-open: open: Permission denied”}}
{“t”:{“$date”:”2021-03-01T09:17:01.614+08:00″},”s”:”E”, “c”:”STORAGE”, “id”:22435, “ctx”:”initandlisten”,”msg”:”WiredTiger error”,”attr”:{“error”:13,”message”:”[1614561421:614769][5724:0x7f9d692980], wiredtiger_open: __posix_open_file, 808: /data/db/WiredTiger.turtle: handle-open: open: Permission denied”}}
{“t”:{“$date”:”2021-03-01T09:17:01.616+08:00″},”s”:”E”, “c”:”STORAGE”, “id”:22435, “ctx”:”initandlisten”,”msg”:”WiredTiger error”,”attr”:{“error”:13,”message”:”[1614561421:616117][5724:0x7f9d692980], wiredtiger_open: __posix_open_file, 808: /data/db/WiredTiger.turtle: handle-open: open: Permission denied”}}
{“t”:{“$date”:”2021-03-01T09:17:01.616+08:00″},”s”:”W”, “c”:”STORAGE”, “id”:22347, “ctx”:”initandlisten”,”msg”:”Failed to start up WiredTiger under any compatibility version. This may be due to an unsupported upgrade or downgrade.”}
{“t”:{“$date”:”2021-03-01T09:17:01.617+08:00″},”s”:”W”, “c”:”STORAGE”, “id”:22349, “ctx”:”initandlisten”,”msg”:”Attempting to salvage WiredTiger metadata”}
{“t”:{“$date”:”2021-03-01T09:17:01.649+08:00″},”s”:”E”, “c”:”STORAGE”, “id”:22435, “ctx”:”initandlisten”,”msg”:”WiredTiger error”,”attr”:{“error”:13,”message”:”[1614561421:649761][5724:0x7f9d692980], connection: __posix_open_file, 808: /data/db/WiredTiger.turtle: handle-open: open: Permission denied”}}
{“t”:{“$date”:”2021-03-01T09:17:01.781+08:00″},”s”:”E”, “c”:”STORAGE”, “id”:22435, “ctx”:”initandlisten”,”msg”:”WiredTiger error”,”attr”:{“error”:1,”message”:”[1614561421:781096][5724:0x7f9d692980], file:WiredTiger.wt, WT_SESSION.salvage: __posix_open_file, 808: /data/db/WiredTiger.wt: handle-open: open: Operation not permitted”}}
{“t”:{“$date”:”2021-03-01T09:17:01.797+08:00″},”s”:”F”, “c”:”STORAGE”, “id”:50947, “ctx”:”initandlisten”,”msg”:”Failed to salvage WiredTiger metadata”,”attr”:{“details”:”1: Operation not permitted”}}
{“t”:{“$date”:”2021-03-01T09:17:01.797+08:00″},”s”:”F”, “c”:”-“, “id”:23091, “ctx”:”initandlisten”,”msg”:”Fatal assertion”,”attr”:{“msgid”:50947,”file”:”src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp”,”line”:1138}}
{“t”:{“$date”:”2021-03-01T09:17:01.797+08:00″},”s”:”F”, “c”:”-“, “id”:23092, “ctx”:”initandlisten”,”msg”:”\n\n***aborting after fassert() failure\n\n”}

“`

最后,我的问题是,

不知道这些数据还能用吗?

有什么方法可以修复?

或是哪方面资料里会有相关的修复方法?

提供一些处理这个问题方向指引,线索也可。

多谢!

更改状态以发布