Error on document replication

Keywords: CouchDB - Other - Technical issue - Other
Description:
I have 3 CouchDb node in Azure, each one in a container instance (Linux machine).
When creating a replication doc, it crashes after a few minute reporting info changes_reader_died.

The replication is created via curl command:
curl -X POST https://my-couchdb-address/_replicator -H ‘Content-Type: application/json’ -d ‘{
“_id”: “my-id”,
“source”: “https://my-couchdb-address/database-name”,
“target”: “https://my-couchdb-address/database-name”,
“create_target”: false,
“continuous”: true
}’

CouchDb installed version is 2.3.1

Log extract of the error reported:

[error] 2020-09-02T07:21:54.562000Z couchdb@localhost <0.1953.0> -------- CRASH REPORT Process  (<0.1953.0>) with 1 neighbors exited with reason: changes_reader_died at gen_server:terminate/7(line:812) <= proc_lib:init_p_do_apply/3(line:247); initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors: [<0.607.0>,couch_replicator_scheduler_sup,couch_replicator_sup,...], messages: [], links: [<0.1954.0>], dictionary: [{last_stats_report,{1599,31200,311000}}], trap_exit: true, status: running, heap_size: 987, stack_size: 27, reductions: 333
[error] 2020-09-02T07:21:54.562000Z couchdb@localhost <0.1951.0> -------- CRASH REPORT Process  (<0.1951.0>) with 1 neighbors exited with reason: changes_reader_died at gen_server:terminate/7(line:812) <= proc_lib:init_p_do_apply/3(line:247); initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors: [<0.607.0>,couch_replicator_scheduler_sup,couch_replicator_sup,...], messages: [], links: [<0.1952.0>], dictionary: [{last_stats_report,{1599,31200,326000}}], trap_exit: true, status: running, heap_size: 987, stack_size: 27, reductions: 333
[error] 2020-09-02T07:21:56.740000Z couchdb@localhost <0.4653.0> -------- couch_replicator_auth_session : Could not parse cookie from response headers cookie_format_invalid
[error] 2020-09-02T07:21:56.865000Z couchdb@localhost <0.4653.0> -------- couch_replicator_auth_session : Could not parse cookie from response headers cookie_format_invalid
[error] 2020-09-02T07:21:57.005000Z couchdb@localhost <0.4653.0> -------- couch_replicator_auth_session : Could not parse cookie from response headers cookie_format_invalid
[error] 2020-09-02T07:21:57.224000Z couchdb@localhost <0.4653.0> -------- couch_replicator_auth_session : Could not parse cookie from response headers cookie_format_invalid
[error] 2020-09-02T07:22:07.085000Z couchdb@localhost <0.4653.0> -------- couch_replicator_auth_session : Could not parse cookie from response headers cookie_format_invalid
[error] 2020-09-02T07:22:07.132000Z couchdb@localhost <0.4695.0> -------- Replicator, request GET to "https://my-couchdb-server-address/zuluedispatch/_changes?feed=continuous&style=all_docs&since=%228-g1AAAAL1eJzLYWBg4MhgTmEwSc4vTc5ISXKoKs0pTdWF8gyM9cpTi0tSS4vyC1L1EqtKi1KT8_NKEjPzUov0MvNzgLqZEhmS5P___5-VyECZOUkKQDLJHmRUBnMiWy6Qx25uaJFklJhMprEUuscB5J54qHuYwO6xME81MrUcIPckgNxTT42gzmMBkgwNQApo2nwqGbcAYtx-Khl3AGLcfSoZ9wBiHCjssgByiQta%22&timeout=10000" failed due to error {connection_closed,mid_stream}
[error] 2020-09-02T07:22:07.539000Z couchdb@localhost <0.4653.0> -------- couch_replicator_auth_session : Could not parse cookie from response headers cookie_format_invalid
[error] 2020-09-02T07:22:17.241000Z couchdb@localhost <0.4695.0> -------- Replicator, request GET to "https://my-couchdb-server-address/zuluedispatch/_changes?feed=continuous&style=all_docs&since=%2274-g1AAAASTeJzF00sKwjAQBuDgsyKCN9ALWNLG0riyN9G8SpXaSjUI3ehN9CZ6E71JTZOC-7rIZkIC-fmYTFIAgJN0OViyXLKE06iUqRSLZgeRexGns5BFfhQuKWUhWJ6dyS4ThbvLU3W7QwCdVVW1T7qkf1AHw9jHkAreMvM_DJ2rSteNZ6A9oYepT5gdT1R7No3H0R4cCj9YWfJsa8-18Yy0hyPEvGVoxZP1VAU3tSjS_fdmEPGQWeqRMT2M6VmbxtrkMYxj385cG9PLmN61aaJNlAVBEEOLpo8x6Xmamv-PIFYJLaP3XyGDmkA%22&timeout=10000" failed. The received HTTP error code is 504
[error] 2020-09-02T07:22:27.701000Z couchdb@localhost <0.4695.0> -------- Replicator, request GET to "https://my-couchdb-server-address/zuluedispatch/_changes?feed=continuous&style=all_docs&since=%2274-g1AAAASTeJzF00sKwjAQBuDgsyKCN9ALWNLG0riyN9G8SpXaSjUI3ehN9CZ6E71JTZOC-7rIZkIC-fmYTFIAgJN0OViyXLKE06iUqRSLZgeRexGns5BFfhQuKWUhWJ6dyS4ThbvLU3W7QwCdVVW1T7qkf1AHw9jHkAreMvM_DJ2rSteNZ6A9oYepT5gdT1R7No3H0R4cCj9YWfJsa8-18Yy0hyPEvGVoxZP1VAU3tSjS_fdmEPGQWeqRMT2M6VmbxtrkMYxj385cG9PLmN61aaJNlAVBEEOLpo8x6Xmamv-PIFYJLaP3XyGDmkA%22&timeout=10000" failed. The received HTTP error code is 504
[error] 2020-09-02T07:22:57.567000Z couchdb@localhost <0.4653.0> -------- couch_replicator_auth_session : Could not parse cookie from response headers cookie_format_invalid
[error] 2020-09-02T07:22:57.677000Z couchdb@localhost <0.4653.0> -------- couch_replicator_auth_session : Could not parse cookie from response headers cookie_format_invalid
[error] 2020-09-02T07:22:58.193000Z couchdb@localhost <0.4695.0> -------- Replicator, request GET to "https://my-couchdb-server-address/zuluedispatch/_changes?feed=continuous&style=all_docs&since=%2274-g1AAAASTeJzF00sKwjAQBuDgsyKCN9ALWNLG0riyN9G8SpXaSjUI3ehN9CZ6E71JTZOC-7rIZkIC-fmYTFIAgJN0OViyXLKE06iUqRSLZgeRexGns5BFfhQuKWUhWJ6dyS4ThbvLU3W7QwCdVVW1T7qkf1AHw9jHkAreMvM_DJ2rSteNZ6A9oYepT5gdT1R7No3H0R4cCj9YWfJsa8-18Yy0hyPEvGVoxZP1VAU3tSjS_fdmEPGQWeqRMT2M6VmbxtrkMYxj385cG9PLmN61aaJNlAVBEEOLpo8x6Xmamv-PIFYJLaP3XyGDmkA%22&timeout=10000" failed due to error {error,req_timedout}
[error] 2020-09-02T07:23:09.551000Z couchdb@localhost <0.4695.0> -------- Replicator, request GET to "https://my-couchdb-server-address/zuluedispatch/_changes?feed=continuous&style=all_docs&since=%2274-g1AAAASTeJzF00sKwjAQBuDgsyKCN9ALWNLG0riyN9G8SpXaSjUI3ehN9CZ6E71JTZOC-7rIZkIC-fmYTFIAgJN0OViyXLKE06iUqRSLZgeRexGns5BFfhQuKWUhWJ6dyS4ThbvLU3W7QwCdVVW1T7qkf1AHw9jHkAreMvM_DJ2rSteNZ6A9oYepT5gdT1R7No3H0R4cCj9YWfJsa8-18Yy0hyPEvGVoxZP1VAU3tSjS_fdmEPGQWeqRMT2M6VmbxtrkMYxj385cG9PLmN61aaJNlAVBEEOLpo8x6Xmamv-PIFYJLaP3XyGDmkA%22&timeout=10000" failed. The received HTTP error code is 504
[error] 2020-09-02T07:23:41.565000Z couchdb@localhost <0.4695.0> -------- Replicator, request GET to "https://my-couchdb-server-address/zuluedispatch/_changes?feed=continuous&style=all_docs&since=%2274-g1AAAASTeJzF00sKwjAQBuDgsyKCN9ALWNLG0riyN9G8SpXaSjUI3ehN9CZ6E71JTZOC-7rIZkIC-fmYTFIAgJN0OViyXLKE06iUqRSLZgeRexGns5BFfhQuKWUhWJ6dyS4ThbvLU3W7QwCdVVW1T7qkf1AHw9jHkAreMvM_DJ2rSteNZ6A9oYepT5gdT1R7No3H0R4cCj9YWfJsa8-18Yy0hyPEvGVoxZP1VAU3tSjS_fdmEPGQWeqRMT2M6VmbxtrkMYxj385cG9PLmN61aaJNlAVBEEOLpo8x6Xmamv-PIFYJLaP3XyGDmkA%22&timeout=10000" failed due to error {error,req_timedout}
[error] 2020-09-02T07:23:56.173000Z couchdb@localhost <0.4695.0> -------- Replicator, request GET to "https://my-couchdb-server-address/zuluedispatch/_changes?feed=continuous&style=all_docs&since=%2274-g1AAAASTeJzF00sKwjAQBuDgsyKCN9ALWNLG0riyN9G8SpXaSjUI3ehN9CZ6E71JTZOC-7rIZkIC-fmYTFIAgJN0OViyXLKE06iUqRSLZgeRexGns5BFfhQuKWUhWJ6dyS4ThbvLU3W7QwCdVVW1T7qkf1AHw9jHkAreMvM_DJ2rSteNZ6A9oYepT5gdT1R7No3H0R4cCj9YWfJsa8-18Yy0hyPEvGVoxZP1VAU3tSjS_fdmEPGQWeqRMT2M6VmbxtrkMYxj385cG9PLmN61aaJNlAVBEEOLpo8x6Xmamv-PIFYJLaP3XyGDmkA%22&timeout=10000" failed. The received HTTP error code is 504
[error] 2020-09-02T07:23:56.173000Z couchdb@localhost <0.4619.0> -------- ChangesReader process died with reason: {http_request_failed,"GET","https://my-couchdb-server-address/zuluedispatch/_changes?feed=continuous&style=all_docs&since=%2274-g1AAAASTeJzF00sKwjAQBuDgsyKCN9ALWNLG0riyN9G8SpXaSjUI3ehN9CZ6E71JTZOC-7rIZkIC-fmYTFIAgJN0OViyXLKE06iUqRSLZgeRexGns5BFfhQuKWUhWJ6dyS4ThbvLU3W7QwCdVVW1T7qkf1AHw9jHkAreMvM_DJ2rSteNZ6A9oYepT5gdT1R7No3H0R4cCj9YWfJsa8-18Yy0hyPEvGVoxZP1VAU3tSjS_fdmEPGQWeqRMT2M6VmbxtrkMYxj385cG9PLmN61aaJNlAVBEEOLpo8x6Xmamv-PIFYJLaP3XyGDmkA%22&timeout=10000",{code,504}}
[error] 2020-09-02T07:23:56.189000Z couchdb@localhost <0.4619.0> -------- Replication `5a19820ca248a34c795a9f6014b3eacf+continuous+create_target` (`https://my-couchdb-server-address/zuluedispatch/` -> `http://127.0.0.1:5984/zuluemission/`) failed: changes_reader_died
[error] 2020-09-02T07:24:26.686000Z couchdb@localhost <0.4619.0> -------- gen_server {couch_replicator_scheduler_job,{[53,97,49,57,56,50,48,99,97,50,52,56,97,51,52,99,55,57,53,97,57,102,54,48,49,52,98,51,101,97,99,102],[43,99,111,110,116,105,110,117,111,117,115,43,99,114,101,97,116,101,95,116,97,114,103,101,116]}} terminated with reason: changes_reader_died
  last msg: {'EXIT',<0.4695.0>,{http_request_failed,"GET","https://my-couchdb-server-address/zuluedispatch/_changes?feed=continuous&style=all_docs&since=%2274-g1AAAASTeJzF00sKwjAQBuDgsyKCN9ALWNLG0riyN9G8SpXaSjUI3ehN9CZ6E71JTZOC-7rIZkIC-fmYTFIAgJN0OViyXLKE06iUqRSLZgeRexGns5BFfhQuKWUhWJ6dyS4ThbvLU3W7QwCdVVW1T7qkf1AHw9jHkAreMvM_DJ2rSteNZ6A9oYepT5gdT1R7No3H0R4cCj9YWfJsa8-18Yy0hyPEvGVoxZP1VAU3tSjS_fdmEPGQWeqRMT2M6VmbxtrkMYxj385cG9PLmN61aaJNlAVBEEOLpo8x6Xmamv-PIFYJLaP3XyGDmkA%22&timeout=10000",{code,504}}}
     state: [{rep_id,{"5a19820ca248a34c795a9f6014b3eacf","+continuous+create_target"}},{source,"https://my-couchdb-server-address/zuluedispatch/"},{target,"http://127.0.0.1:5984/zuluemission/"},{db_name,<<"shards/a0000000-bfffffff/_replicator.1504686996">>},{doc_id,<<"missione_server_tablet">>},{options,[{checkpoint_interval,30000},{connection_timeout,30000},{continuous,true},{create_target,true},{http_connections,20},{retries,5},{socket_options,[{keepalive,true},{nodelay,false}]},{use_checkpoints,true},{worker_batch_size,500},{worker_processes,4}]},{session_id,<<"d8044ce9b7345f15d4ed30c82af453ae">>},{start_seq,{0,<<"8-g1AAAAL1eJzLYWBg4MhgTmEwSc4vTc5ISXKoKs0pTdWF8gyM9cpTi0tSS4vyC1L1EqtKi1KT8_NKEjPzUov0MvNzgLqZEhmS5P___5-VyECZOUkKQDLJHmRUBnMiWy6Qx25uaJFklJhMprEUuscB5J54qHuYwO6xME81MrUcIPckgNxTT42gzmMBkgwNQApo2nwqGbcAYtx-Khl3AGLcfSoZ9wBiHCjssgByiQta">>}},{source_seq,<<"74-g1AAAAJreJzLYWBg4MhgTmEwSc4vTc5ISXKoKs0pTdWF8gwM9cpTi0tSS4vyC1L1EqtKi1KT8_NKEjPzUov0MvNzgLqZEhmS5P___5-VyIrHHGPC5iQpAMkke7BRbBQa5QAyKh5sFIW-S0oAGVUPNoqTMqPyWIAkQwOQApo2H78niTZuAcS4_VmJ3JSFGcS4AxDj7mcl8uIxzohY4x5AjAOGnUAWAK_M3B8">>},{committed_seq,{5,<<"74-g1AAAASTeJzF00sKwjAQBuDgsyKCN9ALWNLG0riyN9G8SpXaSjUI3ehN9CZ6E71JTZOC-7rIZkIC-fmYTFIAgJN0OViyXLKE06iUqRSLZgeRexGns5BFfhQuKWUhWJ6dyS4ThbvLU3W7QwCdVVW1T7qkf1AHw9jHkAreMvM_DJ2rSteNZ6A9oYepT5gdT1R7No3H0R4cCj9YWfJsa8-18Yy0hyPEvGVoxZP1VAU3tSjS_fdmEPGQWeqRMT2M6VmbxtrkMYxj385cG9PLmN61aaJNlAVBEEOLpo8x6Xmamv-PIFYJLaP3XyGDmkA">>}},{current_through_seq,{5,<<"74-g1AAAASTeJzF00sKwjAQBuDgsyKCN9ALWNLG0riyN9G8SpXaSjUI3ehN9CZ6E71JTZOC-7rIZkIC-fmYTFIAgJN0OViyXLKE06iUqRSLZgeRexGns5BFfhQuKWUhWJ6dyS4ThbvLU3W7QwCdVVW1T7qkf1AHw9jHkAreMvM_DJ2rSteNZ6A9oYepT5gdT1R7No3H0R4cCj9YWfJsa8-18Yy0hyPEvGVoxZP1VAU3tSjS_fdmEPGQWeqRMT2M6VmbxtrkMYxj385cG9PLmN61aaJNlAVBEEOLpo8x6Xmamv-PIFYJLaP3XyGDmkA">>}},{highest_seq_done,{5,<<"74-g1AAAASTeJzF00sKwjAQBuDgsyKCN9ALWNLG0riyN9G8SpXaSjUI3ehN9CZ6E71JTZOC-7rIZkIC-fmYTFIAgJN0OViyXLKE06iUqRSLZgeRexGns5BFfhQuKWUhWJ6dyS4ThbvLU3W7QwCdVVW1T7qkf1AHw9jHkAreMvM_DJ2rSteNZ6A9oYepT5gdT1R7No3H0R4cCj9YWfJsa8-18Yy0hyPEvGVoxZP1VAU3tSjS_fdmEPGQWeqRMT2M6VmbxtrkMYxj385cG9PLmN61aaJNlAVBEEOLpo8x6Xmamv-PIFYJLaP3XyGDmkA">>}}]
    extra: []
[error] 2020-09-02T07:24:26.686000Z couchdb@localhost <0.4619.0> -------- CRASH REPORT Process  (<0.4619.0>) with 2 neighbors exited with reason: changes_reader_died at gen_server:terminate/7(line:812) <= proc_lib:init_p_do_apply/3(line:247); initial_call: {couch_replicator_scheduler_job,init,['Argument__1']}, ancestors: [couch_replicator_scheduler_sup,couch_replicator_sup,...], messages: [{'EXIT',<0.4653.0>,normal},{'EXIT',<0.4659.0>,normal}], links: [<0.4696.0>,<0.4699.0>,<0.4703.0>,<0.4701.0>,<0.4697.0>,...], dictionary: [{pending_count_state,{{1599,31466,686000},0}},{task_status_update,...},...], trap_exit: true, status: running, heap_size: 6772, stack_size: 27, reductions: 22874
[error] 2020-09-02T07:24:26.686000Z couchdb@localhost <0.4699.0> -------- gen_server <0.4699.0> terminated with reason: changes_reader_died
  last msg: {'EXIT',<0.4619.0>,changes_reader_died}
     state: [{main_pid,<0.4619.0>},{loop,<0.4700.0>},{source,"https://my-couchdb-server-address/zuluedispatch/"},{target,"http://127.0.0.1:5984/zuluemission/"},{num_readers,0},{pending_fetch,nil},{batch_size,0}]
    extra: []
[error] 2020-09-02T07:24:26.686000Z couchdb@localhost <0.4697.0> -------- gen_server <0.4697.0> terminated with reason: changes_reader_died
  last msg: {'EXIT',<0.4619.0>,changes_reader_died}
     state: [{main_pid,<0.4619.0>},{loop,<0.4698.0>},{source,"https://my-couchdb-server-address/zuluedispatch/"},{target,"http://127.0.0.1:5984/zuluemission/"},{num_readers,0},{pending_fetch,nil},{batch_size,0}]
    extra: []
[error] 2020-09-02T07:24:26.686000Z couchdb@localhost <0.4699.0> -------- CRASH REPORT Process  (<0.4699.0>) with 1 neighbors exited with reason: changes_reader_died at gen_server:terminate/7(line:812) <= proc_lib:init_p_do_apply/3(line:247); initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors: [<0.4619.0>,couch_replicator_scheduler_sup,couch_replicator_sup,...], messages: [], links: [<0.4700.0>], dictionary: [{last_stats_report,{1599,31327,179000}}], trap_exit: true, status: running, heap_size: 987, stack_size: 27, reductions: 333
[error] 2020-09-02T07:24:26.686000Z couchdb@localhost <0.4697.0> -------- CRASH REPORT Process  (<0.4697.0>) with 1 neighbors exited with reason: changes_reader_died at gen_server:terminate/7(line:812) <= proc_lib:init_p_do_apply/3(line:247); initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors: [<0.4619.0>,couch_replicator_scheduler_sup,couch_replicator_sup,...], messages: [], links: [<0.4698.0>], dictionary: [{last_stats_report,{1599,31327,820000}},{ibrowse_stream_status,init}], trap_exit: true, status: running, heap_size: 4185, stack_size: 27, reductions: 1630
[error] 2020-09-02T07:24:26.686000Z couchdb@localhost <0.4703.0> -------- gen_server <0.4703.0> terminated with reason: changes_reader_died
  last msg: {'EXIT',<0.4619.0>,changes_reader_died}
     state: [{main_pid,<0.4619.0>},{loop,<0.4704.0>},{source,"https://my-couchdb-server-address/zuluedispatch/"},{target,"http://127.0.0.1:5984/zuluemission/"},{num_readers,0},{pending_fetch,nil},{batch_size,0}]
    extra: []
[error] 2020-09-02T07:24:26.686000Z couchdb@localhost <0.4701.0> -------- gen_server <0.4701.0> terminated with reason: changes_reader_died
  last msg: {'EXIT',<0.4619.0>,changes_reader_died}
     state: [{main_pid,<0.4619.0>},{loop,<0.4702.0>},{source,"https://my-couchdb-server-address/zuluedispatch/"},{target,"http://127.0.0.1:5984/zuluemission/"},{num_readers,0},{pending_fetch,nil},{batch_size,0}]
    extra: []

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.