URGENT: Let's Encrypt renewal failed and now Apache will not start

Keywords: LAMP/MAMP/WAMP - AWS - Technical issue - Secure Connections (SSL/HTTPS)

bndiagnostic ID: d893d131-e2cc-94ed-0512-c80599668271

bndiagnostic output:

? Apache: Found possible issues
? Connectivity: Found possible issues
https://docs.bitnami.com/general/faq/administration/use-firewall/

bndiagnostic failure reason: I checked the connectivity: the open 22, 80, and 443 ports have not been changed; they are open as needed.

The DNS is fine and has not been altered.

Description:
I have 20+ websites on this server and they’re all down.

I ran bn-cert as usual to renew the web server SSL certificate. After some hesitation, the LetsEncrypt script failed, spewing row after row of information and ending with:

2022/06/20 23:49:19 Could not obtain certificates:
        error: one or more domains had a problem:
[www.tooliedotterpress.com] acme: error: 400 :: urn:ietf:params:acme:error:dns
:: DNS problem: SERVFAIL looking up CAA for tooliedotterpress.com - the domain's
nameservers may be malfunctioning

I commented out the reference to the LetsEncrypt-generated certificate name everywhere I could find it, and replaced it with the dummy certificate path and filename in:

  • bitnami.conf
  • bitnami-ssl.conf
  • bitnami-apps-vhosts-ssl.conf

Apache refuses to restart, and the Apache logs reveal nothing. My server is dead in the water and I am about to have clients screaming at me.

Help!

Hi @Toolie,

Can you check the configuration?

 sudo apachectl configtest

And then try to start the service with gonit?

sudo gonit start apache

Regards,
Michiel

I ran the sudo apachectl configtest command and I did have a bad path for the dummy certificates, leftover from before you changed your folder layout in /apache. I’m glad the command caught it.

I ran the sudo gonit start apache command and apache is still not starting. I reran the sudo apachectl configtest and it says the configuration is OK.

I then rebooted the entire server, and reran the sudo gonit start apache command, but the reply is Failed to start apache. There is now finally some information in error_log:

[Tue Jun 21 14:29:21.296967 2022] [ssl:emerg] [pid 815:tid 140002653621120] AH02572: Failed to configure at least one certificate and key for www.tooliedotterpress.com:443
[Tue Jun 21 14:29:21.297043 2022] [ssl:emerg] [pid 815:tid 140002653621120] SSL Library Error: error:140A80B1:SSL routines:SSL_CTX_check_private_key:no certificate assigned
[Tue Jun 21 14:29:21.297049 2022] [ssl:emerg] [pid 815:tid 140002653621120] AH02312: Fatal error initialising mod_ssl, exiting.
AH00016: Configuration Failed

The certificates DO exist:

/opt/bitnami/apache/conf/bitnami/certs $ ls -la
total 16
drwxrwxr-x 2 bitnami root 4096 Jan 19 03:47 .
drwxrwxr-x 4 bitnami root 4096 Jun 20 23:53 ..
-rw-rw-r-- 1 bitnami root  981 Jan 19 03:47 server.crt
-rw-rw-r-- 1 bitnami root 1679 Jan 19 03:47 server.key

What do I do now?

In desperation, I launched a new LAMP stack on an AWS instance and started copying my websites to that instance.

Old stack: LAMP packaged by Bitnami 8.0.15-1
New stack: LAMP packaged by Bitnami 8.0.20-0

In that process I found more copy/paste errors in my bitnami.conf and corrected them. I could then start Apache and my websites showed up.

Then I tried running bn-cert again, and once again it crashed on that old server.

Error
An error occurred when applying configurations.

The web server configuration was left unchanged. There was an error in the new
configuration, so it was reverted.

Failed steps:
* Start web server: Configuration changes to the web server caused it to fail

Some steps were not reverted. Run the tool again to apply them:
* The Let's Encrypt certificate was generated, but not revoked

Find more details in the log file:
/tmp/bncert-202206212029.log

If you find any issues, please check Bitnami Support forums at:
https://community.bitnami.com

I have can send you the log file for your information. Meanwhile I’m hoping that the new server will allow bn-cert to run correctly.

Hi @Toolie,

I do not know why the certificate files were removed from the instance but if you continue running into issues with the certificates, I suggest you follow our guide to revoke the certificates and generate a new one later.

https://docs.bitnami.com/aws/how-to/understand-bncert/#manually-revoking-an-existing-certificate

Please note you will need to update all the SSLCertificateFile and SSLCertificateKeyFile lines in all the VirtualHost you created to use the dummy certificate files we include in the instance. After that, you should be able to restart Apache and generate new certificates either using the HTTPS configuration tool or by following the manual approach in our documentation.

I did not say that the certificate files were removed, but if the error message say that, I don’t know what it means.

I know how to revoke existing certificates and request a new one, I’ve done that a half-dozen times; I’ve even done it manually. That’s not what happened here. Something happened in your renewal script that caused the renewal process to fail not once, but twice (see above), and this was the first renewal on a clean installation.

Fortunately on the newer stack (LAMP 8.0.20-0), the installation of a new certificate on a clean installation DID work just now. We’ll see in 3 months whether the renewal script also works. I certainly hope so.