Solved: Can't start Mattermost anymore after SSL certificate update

Keywords: Mattermost - AWS - Technical issue - Secure Connections (SSL/HTTPS)
bnsupport ID: ea793a68-e880-757d-12b1-62dd0ebb864f
Description:
After trying to update the SSL, my Mattermost stopped working

New ID: e55ecf46-413e-267a-3880-45bc9e6ff684

Hi @daniel7,

Just found this error in the NGINX’s log.

2020/04/13 05:43:15 [emerg] 6015#0: cannot load certificate "/opt/bitnami/apps/mattermost/conf/certs/server.crt": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/opt/bitnami/apps/mattermost/conf/certs/server.crt','r') error:2006D080:BIO routines:BIO_new_file:no such file)

Could you please let us know the guide you followed to configure the SSL certificate? I think you will need to do the following to fix the issue:

sudo ln -sf /opt/bitnami/letsencrypt/certificates/icet.livnao.com.crt /opt/bitnami/apps/mattermost/conf/certs/server.crt
sudo ln -sf /opt/bitnami/letsencrypt/certificates/icet.livnao.com.key /opt/bitnami/apps/mattermost/conf/certs/server.key
sudo /opt/bitnami/ctlscript.sh restart nginx

Happy to help!


Was my answer helpful? Click on :heart:

Thanks @jota,

I tried your above steps and still receiving the below error.
/opt/bitnami/apps/mattermost/scripts/ctl.sh : mattermost could not be started

I followed these steps:
https://docs.mattermost.com/install/config-ssl-http2-nginx.html?highlight=encrypt

Any additional advice?

The new ID is:
2b5aaf03-aecb-3b0b-cffd-a58eb33a639a

Hi @daniel7,

There was not any information in the error.log file of NGINX when you ran the Bitnami support tool, can you try to get more information from that file now?

sudo /opt/bitnami/ctlscript.sh restart nginx
sudo tail -n50 /opt/bitnami/nginx/logs/error.log

Thanks

@jota

Sure:

2020/04/14 11:42:07 [error] 9654#0: *21455 connect() failed (111: Connection refused) while connecting to upstream, 
client: 108.172.154.169, server: icet.livnao.com, request: "GET /api/v4/users/me/teams/unread HTTP/2.0", upstream: "http://127.0.0.1:8065/api/v4/users/me/teams/unread", host: "icet.livnao.com"
2020/04/14 11:42:07 [error] 9654#0: *21455 connect() failed (111: Connection refused) while connecting to upstream, client: 108.172.154.169, server: icet.livnao.com, request: "POST /api/v4/users/ids?since=1586863629851 HTTP/2.0", upstream: "http://127.0.0.1:8065/api/v4/users/ids?since=1586863629851", host: "icet.livnao.com"
2020/04/14 11:42:07 [error] 9654#0: *21455 connect() failed (111: Connection refused) while connecting to upstream, client: 108.172.154.169, server: icet.livnao.com, request: "GET /plugins/com.github.manland.mattermost-plugin-gitlab/api/v1/connected?reminder=false HTTP/2.0", upstream: "http://127.0.0.1:8065/plugins/com.github.manland.mattermost-plugin-gitlab/api/v1/connected?reminder=false", host: "icet.livnao.com"
2020/04/14 11:42:14 [error] 9654#0: *23569 connect() failed (111: Connection refused) while connecting to upstream, client: 108.172.154.169, server: icet.livnao.com, request: "GET /api/v4/websocket HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/websocket", host: "icet.livnao.com"
2020/04/14 11:42:31 [error] 9654#0: *23571 connect() failed (111: Connection refused) while connecting to upstream, client: 108.172.154.169, server: icet.livnao.com, request: "GET /api/v4/websocket HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/websocket", host: "icet.livnao.com"
2020/04/14 11:42:41 [error] 9655#0: *23573 connect() failed (111: Connection refused) while connecting to upstream, client: 108.172.154.169, server: icet.livnao.com, request: "GET /api/v4/websocket HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/websocket", host: "icet.livnao.com"
2020/04/14 11:43:03 [error] 9655#0: *23575 connect() failed (111: Connection refused) while connecting to upstream, client: 108.172.154.169, server: icet.livnao.com, request: "GET /api/v4/websocket HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/websocket", host: "icet.livnao.com"
2020/04/14 11:43:07 [error] 9654#0: *21455 connect() failed (111: Connection refused) while connecting to upstream, client: 108.172.154.169, server: icet.livnao.com, request: "POST /api/v4/users/status/ids HTTP/2.0", upstream: "http://127.0.0.1:8065/api/v4/users/status/ids", host: "icet.livnao.com"
2020/04/14 11:43:22 [error] 9654#0: *23578 connect() failed (111: Connection refused) while connecting to upstream, client: 108.172.154.169, server: icet.livnao.com, request: "GET /api/v4/websocket HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/websocket", host: "icet.livnao.com"
2020/04/14 11:43:32 [error] 9654#0: *23580 connect() failed (111: Connection refused) while connecting to upstream, client: 108.172.154.169, server: icet.livnao.com, request: "GET /api/v4/websocket HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/websocket", host: "icet.livnao.com"
2020/04/14 11:43:47 [error] 9654#0: *23582 connect() failed (111: Connection refused) while connecting to upstream, client: 108.172.154.169, server: icet.livnao.com, request: "GET /api/v4/websocket HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/websocket", host: "icet.livnao.com"
2020/04/14 11:44:01 [error] 9655#0: *23584 connect() failed (111: Connection refused) while connecting to upstream, client: 108.172.154.169, server: icet.livnao.com, request: "GET /api/v4/websocket HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/websocket", host: "icet.livnao.com"
2020/04/14 11:44:07 [error] 9654#0: *21455 connect() failed (111: Connection refused) while connecting to upstream, client: 108.172.154.169, server: icet.livnao.com, request: "POST /api/v4/users/status/ids HTTP/2.0", upstream: "http://127.0.0.1:8065/api/v4/users/status/ids", host: "icet.livnao.com"
2020/04/14 11:44:13 [error] 9654#0: *23587 connect() failed (111: Connection refused) while connecting to upstream, client: 108.172.154.169, server: icet.livnao.com, request: "GET /api/v4/websocket HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/websocket", host: "icet.livnao.com"
2020/04/14 11:44:25 [error] 9655#0: *23589 connect() failed (111: Connection refused) while connecting to upstream, client: 108.172.154.169, server: icet.livnao.com, request: "GET /api/v4/websocket HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/websocket", host: "icet.livnao.com"
2020/04/14 11:44:40 [error] 9655#0: *23591 connect() failed (111: Connection refused) while connecting to upstream, client: 108.172.154.169, server: icet.livnao.com, request: "GET /api/v4/websocket HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/websocket", host: "icet.livnao.com"
2020/04/14 11:44:47 [error] 9654#0: *23593 connect() failed (111: Connection refused) while connecting to upstream, client: 108.172.154.169, server: icet.livnao.com, request: "GET /api/v4/websocket HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/websocket", host: "icet.livnao.com"
2020/04/14 11:45:05 [error] 9654#0: *23595 connect() failed (111: Connection refused) while connecting to upstream, client: 108.172.154.169, server: icet.livnao.com, request: "GET /api/v4/websocket HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/websocket", host: "icet.livnao.com"
2020/04/14 11:45:07 [error] 9654#0: *21455 connect() failed (111: Connection refused) while connecting to upstream, client: 108.172.154.169, server: icet.livnao.com, request: "POST /api/v4/users/status/ids HTTP/2.0", upstream: "http://127.0.0.1:8065/api/v4/users/status/ids", host: "icet.livnao.com"
2020/04/14 11:45:21 [error] 9655#0: *23598 connect() failed (111: Connection refused) while connecting to upstream, client: 108.172.154.169, server: icet.livnao.com, request: "GET /api/v4/websocket HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/websocket", host: "icet.livnao.com"
2020/04/14 11:45:32 [error] 9654#0: *23600 connect() failed (111: Connection refused) while connecting to upstream, client: 108.172.154.169, server: icet.livnao.com, request: "GET /api/v4/websocket HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/websocket", host: "icet.livnao.com"
2020/04/14 11:45:46 [error] 9654#0: *23602 connect() failed (111: Connection refused) while connecting to upstream, client: 154.5.136.122, server: icet.livnao.com, request: "GET /api/v4/websocket HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/websocket", host: "icet.livnao.com"
2020/04/14 11:45:51 [error] 9654#0: *23604 connect() failed (111: Connection refused) while connecting to upstream, client: 108.172.154.169, server: icet.livnao.com, request: "GET /api/v4/websocket HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/websocket", host: "icet.livnao.com"
2020/04/14 11:45:51 [error] 9654#0: *23606 connect() failed (111: Connection refused) while connecting to upstream, client: 154.5.136.122, server: icet.livnao.com, request: "GET /api/v4/plugins/webapp HTTP/2.0", upstream: "http://127.0.0.1:8065/api/v4/plugins/webapp", host: "icet.livnao.com"
2020/04/14 11:45:51 [error] 9654#0: *23606 connect() failed (111: Connection refused) while connecting to upstream, client: 154.5.136.122, server: icet.livnao.com, request: "GET /api/v4/users/me/teams/7db8qo4mi7ynmnw15dwambyygo/channels HTTP/2.0", upstream: "http://127.0.0.1:8065/api/v4/users/me/teams/7db8qo4mi7ynmnw15dwambyygo/channels", host: "icet.livnao.com"
2020/04/14 11:45:51 [error] 9654#0: *23606 connect() failed (111: Connection refused) while connecting to upstream, client: 154.5.136.122, server: icet.livnao.com, request: "GET /api/v4/users/me/teams/7db8qo4mi7ynmnw15dwambyygo/channels/members HTTP/2.0", upstream: "http://127.0.0.1:8065/api/v4/users/me/teams/7db8qo4mi7ynmnw15dwambyygo/channels/members", host: "icet.livnao.com"
2020/04/14 11:45:51 [error] 9654#0: *23606 connect() failed (111: Connection refused) while connecting to upstream, client: 154.5.136.122, server: icet.livnao.com, request: "GET /api/v4/users/me/teams/unread HTTP/2.0", upstream: "http://127.0.0.1:8065/api/v4/users/me/teams/unread", host: "icet.livnao.com"
2020/04/14 11:45:51 [error] 9654#0: *23606 connect() failed (111: Connection refused) while connecting to upstream, client: 154.5.136.122, server: icet.livnao.com, request: "GET /api/v4/channels/wejpa3qfzjbwpjbs7878qcyaka/posts?since=1586730779989 HTTP/2.0", upstream: "http://127.0.0.1:8065/api/v4/channels/wejpa3qfzjbwpjbs7878qcyaka/posts?since=1586730779989", host: "icet.livnao.com"
2020/04/14 11:45:51 [error] 9654#0: *23606 connect() failed (111: Connection refused) while connecting to upstream, client: 154.5.136.122, server: icet.livnao.com, request: "GET /plugins/com.github.manland.mattermost-plugin-gitlab/api/v1/connected?reminder=false HTTP/2.0", upstream: "http://127.0.0.1:8065/plugins/com.github.manland.mattermost-plugin-gitlab/api/v1/connected?reminder=false", host: "icet.livnao.com"
2020/04/14 11:45:51 [error] 9654#0: *23606 connect() failed (111: Connection refused) while connecting to upstream, client: 154.5.136.122, server: icet.livnao.com, request: "POST /api/v4/users/ids?since=1586862690099 HTTP/2.0", upstream: "http://127.0.0.1:8065/api/v4/users/ids?since=1586862690099", host: "icet.livnao.com"
2020/04/14 11:46:04 [error] 9654#0: *23614 connect() failed (111: Connection refused) while connecting to upstream, client: 154.5.136.122, server: icet.livnao.com, request: "GET /api/v4/websocket HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/websocket", host: "icet.livnao.com"
2020/04/14 11:46:05 [error] 9654#0: *23616 connect() failed (111: Connection refused) while connecting to upstream, client: 108.172.154.169, server: icet.livnao.com, request: "GET /api/v4/websocket HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/websocket", host: "icet.livnao.com"
2020/04/14 11:46:07 [error] 9654#0: *21455 connect() failed (111: Connection refused) while connecting to upstream, client: 108.172.154.169, server: icet.livnao.com, request: "POST /api/v4/users/status/ids HTTP/2.0", upstream: "http://127.0.0.1:8065/api/v4/users/status/ids", host: "icet.livnao.com"
2020/04/14 11:46:14 [error] 9655#0: *23619 connect() failed (111: Connection refused) while connecting to upstream, client: 154.5.136.122, server: icet.livnao.com, request: "GET /api/v4/websocket HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/websocket", host: "icet.livnao.com"
2020/04/14 11:46:23 [error] 9655#0: *23621 connect() failed (111: Connection refused) while connecting to upstream, client: 108.172.154.169, server: icet.livnao.com, request: "GET /api/v4/websocket HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/websocket", host: "icet.livnao.com"
2020/04/14 11:46:34 [error] 9655#0: *23623 connect() failed (111: Connection refused) while connecting to upstream, client: 108.172.154.169, server: icet.livnao.com, request: "GET /api/v4/websocket HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/websocket", host: "icet.livnao.com"
2020/04/14 11:46:45 [error] 9654#0: *23625 connect() failed (111: Connection refused) while connecting to upstream, client: 108.172.154.169, server: icet.livnao.com, request: "GET /api/v4/websocket HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/websocket", host: "icet.livnao.com"
2020/04/14 11:46:58 [error] 9655#0: *23627 connect() failed (111: Connection refused) while connecting to upstream, client: 108.172.154.169, server: icet.livnao.com, request: "GET /api/v4/websocket HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/websocket", host: "icet.livnao.com"
2020/04/14 11:47:07 [error] 9654#0: *21455 connect() failed (111: Connection refused) while connecting to upstream, client: 108.172.154.169, server: icet.livnao.com, request: "POST /api/v4/users/status/ids HTTP/2.0", upstream: "http://127.0.0.1:8065/api/v4/users/status/ids", host: "icet.livnao.com"
2020/04/14 11:47:14 [error] 9655#0: *23630 connect() failed (111: Connection refused) while connecting to upstream, client: 108.172.154.169, server: icet.livnao.com, request: "GET /api/v4/websocket HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/websocket", host: "icet.livnao.com"
2020/04/14 11:47:32 [error] 9654#0: *23632 connect() failed (111: Connection refused) while connecting to upstream, client: 108.172.154.169, server: icet.livnao.com, request: "GET /api/v4/websocket HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/websocket", host: "icet.livnao.com"
2020/04/14 11:47:41 [error] 9655#0: *23634 connect() failed (111: Connection refused) while connecting to upstream, client: 108.172.154.169, server: icet.livnao.com, request: "GET /api/v4/websocket HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/websocket", host: "icet.livnao.com"
2020/04/14 11:47:55 [error] 9655#0: *23636 connect() failed (111: Connection refused) while connecting to upstream, client: 195.154.62.232, server: icet.livnao.com, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8065/", host: "icet.livnao.com"
2020/04/14 11:48:04 [error] 9654#0: *23639 connect() failed (111: Connection refused) while connecting to upstream, client: 108.172.154.169, server: icet.livnao.com, request: "GET /api/v4/websocket HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/websocket", host: "icet.livnao.com"
2020/04/14 11:48:07 [error] 9654#0: *21455 connect() failed (111: Connection refused) while connecting to upstream, client: 108.172.154.169, server: icet.livnao.com, request: "POST /api/v4/users/status/ids HTTP/2.0", upstream: "http://127.0.0.1:8065/api/v4/users/status/ids", host: "icet.livnao.com"
2020/04/14 11:48:22 [error] 9654#0: *23642 connect() failed (111: Connection refused) while connecting to upstream, client: 108.172.154.169, server: icet.livnao.com, request: "GET /api/v4/websocket HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/websocket", host: "icet.livnao.com"
2020/04/14 11:48:35 [error] 9654#0: *23644 connect() failed (111: Connection refused) while connecting to upstream, client: 108.172.154.169, server: icet.livnao.com, request: "GET /api/v4/websocket HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/websocket", host: "icet.livnao.com"
2020/04/14 11:48:51 [error] 9654#0: *23646 connect() failed (111: Connection refused) while connecting to upstream, client: 108.172.154.169, server: icet.livnao.com, request: "GET /api/v4/websocket HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/websocket", host: "icet.livnao.com"
2020/04/14 11:49:01 [error] 9654#0: *23648 connect() failed (111: Connection refused) while connecting to upstream, client: 108.172.154.169, server: icet.livnao.com, request: "GET /api/v4/websocket HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/websocket", host: "icet.livnao.com"
2020/04/14 11:49:13 [error] 12405#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 108.172.154.169, server: icet.livnao.com, request: "GET /api/v4/websocket HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/websocket", host: "icet.livnao.com"

hi @daniel7,

Did you make any other change in the application apart from the SSL configuration? That shouldn’t break Mattermost. At the beginning of this thread we helped you configure the SSL certificates in NGINX but that has nothing to do with the current error I found in the logs (/opt/bitnami/apps/mattermost/server/logs/mattermost.log)

{"level":"error","ts":1586653987.288171,"caller":"mlog/log.go:174","msg":"RPC call MessageHasBeenPosted to plugin failed.","plugin_id":"com.mattermost.nps","error":"connection is shut down"}

I suggest you post this error in the Mattermost forums to get more information about it

https://forum.mattermost.org/

@jota

I haven’t made any changes beyond SSL. Do you know if there is a way to run a clean, new, fresh installation of Mattermost while retaining all the messages and settings? (I assume these are tied to the MySQL?)

Thanks,

Hi @daniel7,

I believe there is something wrong in your Mattermost application. I followed the Bitnami documentation and configured the SSL certificate without issues

bitnami@ip-172-31-3-98:/tmp$ sudo /opt/bitnami/ctlscript.sh status
Nginx already running
mattermost already running
mysql already running

https://docs.bitnami.com/aws/apps/mattermost/administration/generate-configure-certificate-letsencrypt/

Could you please redo the changes you included in the bitnami.conf file and follow the Bitnami documentation to configure the SSL certificate? If you are not sure about how to redo the changes, you have the original configuration files inside the /opt/bitnami/original-data folder.

I figured it out! Ended up updating Mattermost and it worked.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.