No DB shards could be opened

Keywords: CouchDB - AWS - Technical issue - Other
Description:
{error: “internal_server_error”, reason: “No DB shards could be opened.”, ref: 1951223093}
I searched google and found nothing. My only hunch is my AWS maxed out, I am on t2.nano and then I opened it up for a bit of testing and got the message “AWS Free Tier usage limit alerting via AWS Budgets” from AWS. Ran diagnostic tool and it went so slow. Perhaps I just restart the service. I believe I do this via the AWS “Instances”, then top menu “Actions”, “Instance State”, Reboot. Then perhaps switch instance type to something with more memory. Note I searched help and found nothing for this issue title “No DB shards…”

I ran the diagnostic tool and but the automatic upload failed. The file is “/tmp/bitnami-couchdb-info-20190417-182105-5590.zip”

The text is around 6,000 characters and the max you have is 3,200. I am not allowed to upload the zip file. Perhaps you have access to it? Thanks.

Hi,

I am afraid we do not have access to the file unless you upload it to our servers. Another option is that you upload it to a storage server like drive or dropbox and share the link here. Could that be possible?

Best regards,

Javier J. Salmerón


Was my answer helpful? Click on :heart:

Here it is, both compressed as zip and uncompressed:
https://drive.google.com/drive/folders/1IayCPk1BtAqt7WmUQKXW5JC1-DuBjA2i?usp=sharing

Hi,

According to the logs, indeed there are no shards available because all of them died:

[error] 2019-04-17T07:12:35.822630Z couchdb@127.0.0.1 <0.5625.4748> -------- CRASH REPORT Process  (<0.5625.4748>) with 2 neighbors exited with reason: {timeout,{gen_server,call,[<0.5843.4748>,{set_db_pid,<0.5625.4748>}]}} at gen_server:call/2(line:206) <= couch_bt_engine:init_state/4(line:710) <= couch_bt_engine:init/2(line:148) <= couch_db_engine:init/3(line:648) <= couch_db_updater:init/1(line:32) <= proc_lib:init_p_do_apply/3(line:247); initial_call: {couch_db_updater,init,['Argument__1']}, ancestors: [<0.5522.4748>], message_queue_len: 1, messages: [{#Ref<0.4106791531.2875457537.40778>,ok}], links: [<0.5522.4748>,<0.5843.4748>], dictionary: [{idle_limit,61000},{io_priority,{db_update,<<"shards/60000000-7ff...">>}}], trap_exit: false, status: running, heap_size: 987, stack_size: 27, reductions: 1233
[info] 2019-04-17T07:12:35.901863Z couchdb@127.0.0.1 <0.216.0> -------- db shards/60000000-7fffffff/_users.1540923512 died with reason {timeout,{gen_server,call,[<0.5843.4748>,{set_db_pid,<0.5625.4748>}]}}
[info] 2019-04-17T07:12:35.901975Z couchdb@127.0.0.1 <0.216.0> -------- db shards/80000000-9fffffff/_users.1540923512 died with reason {timeout,{gen_server,call,[<0.6493.4748>,{set_db_pid,<0.6703.4748>}]}}
[error] 2019-04-17T07:12:36.565687Z couchdb@127.0.0.1 <0.6703.4748> -------- CRASH REPORT Process  (<0.6703.4748>) with 2 neighbors exited with reason: {timeout,{gen_server,call,[<0.6493.4748>,{set_db_pid,<0.6703.4748>}]}} at gen_server:call/2(line:206) <= couch_bt_engine:init_state/4(line:710) <= couch_bt_engine:init/2(line:148) <= couch_db_engine:init/3(line:648) <= couch_db_updater:init/1(line:32) <= proc_lib:init_p_do_apply/3(line:247); initial_call: {couch_db_updater,init,['Argument__1']}, ancestors: [<0.4880.4748>], message_queue_len: 1, messages: [{#Ref<0.4106791531.2875457537.40779>,ok}], links: [<0.4880.4748>,<0.6493.4748>], dictionary: [{idle_limit,61000},{io_priority,{db_update,<<"shards/80000000-9ff...">>}}], trap_exit: false, status: running, heap_size: 987, stack_size: 27, reductions: 1233
[info] 2019-04-17T07:12:39.123910Z couchdb@127.0.0.1 <0.58.0> -------- alarm_handler: {clear,system_memory_high_watermark}
[info] 2019-04-17T07:13:45.584223Z couchdb@127.0.0.1 <0.58.0> -------- alarm_handler: {set,{system_memory_high_watermark,[]}}
[error] 2019-04-17T07:17:35.553061Z couchdb@127.0.0.1 <0.5346.4748> -------- rexi_server: from: couchdb@127.0.0.1(<0.3491.4748>) mfa: fabric_rpc:changes/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream_last,2,[{file,"src/rexi.erl"},{line,224}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}]
[error] 2019-04-17T07:17:35.553311Z couchdb@127.0.0.1 <0.3852.4748> -------- rexi_server: from: couchdb@127.0.0.1(<0.3491.4748>) mfa: fabric_rpc:changes/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream_last,2,[{file,"src/rexi.erl"},{line,224}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}]
[error] 2019-04-17T07:20:12.049881Z couchdb@127.0.0.1 <0.4311.4715> -------- gen_server couch_compaction_daemon terminated with reason: {compaction_loop_died,{timeout,{gen_server,call,[couch_server,get_server]}}}
  last msg: {'EXIT',<0.2396.4715>,{timeout,{gen_server,call,[couch_server,get_server]}}}
     state: {state,<0.2396.4715>,[]}
[error] 2019-04-17T07:20:12.789128Z couchdb@127.0.0.1 <0.4311.4715> -------- CRASH REPORT Process couch_compaction_daemon (<0.4311.4715>) with 0 neighbors exited with reason: {compaction_loop_died,{timeout,{gen_server,call,[couch_server,get_server]}}} at gen_server:handle_common_reply/8(line:726) <= proc_lib:init_p_do_apply/3(line:247); initial_call: {couch_compaction_daemon,init,['Argument__1']}, ancestors: [couch_secondary_services,couch_sup,<0.210.0>], message_queue_len: 0, messages: [], links: [<0.219.0>], dictionary: [], trap_exit: true, status: running, heap_size: 376, stack_size: 27, reductions: 5331
[error] 2019-04-17T07:20:13.119720Z couchdb@127.0.0.1 <0.219.0> -------- Supervisor couch_secondary_services had child compaction_daemon started with couch_compaction_daemon:start_link() at <0.4311.4715> ,{timeout,{gen_server,call,[couch_server,get_server]}}} in context child_terminated
[notice] 2019-04-17T12:01:34.467309Z couchdb@127.0.0.1 <0.5417.4756> bc791f92a4 18.214.95.156:5984 128.14.136.18 undefined GET / 401 ok 2645
[notice] 2019-04-17T13:59:00.333883Z couchdb@127.0.0.1 <0.6675.4759> 0a931fcfb9 ppcjsondata.com:6984 209.97.128.215 admin GET / 200 ok 659
[notice] 2019-04-17T13:59:56.318812Z couchdb@127.0.0.1 <0.12935.4759> 7ea91a8161 ppcjsondata.com:6984 209.97.128.215 undefined GET /userdb-70706331/_local/rWfWnEONASFQ0j8xg401TA%3D%3D 401 ok 654
[error] 2019-04-17T14:08:07.962596Z couchdb@127.0.0.1 <0.19665.4759> aa7518276a req_err(1951223093) internal_server_error : No DB shards could be opened.

I checked here https://github.com/apache/couchdb/issues/789 and it seems it could be related to space issues:

I think this is the same issue as #732; in turn, a result of https://bugs.erlang.org/browse/ERL-343. Check whether you have any devices with spaces in their mount points - ejecting them should fix this.

Also, checking bnsupport again, it seems that you do not have space in your system

machine_free_disk_space = -1 MB

Hope it helps

Best regards,

Javier J. Salmerón


Was my answer helpful? Click on :heart:

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.