Bug with Bitnami's Script for Auto-Configuration and Renewal for Let's Encrypt SSL Certificates

Keywords: WordPress - Google Cloud Platform - Technical issue - Secure Connections (SSL/HTTPS)
bnsupport ID: 9be50565-2db7-03bd-3b5e-8c935f1e2198
Description:
It appears the Bitnami provided script to auto-configure and renew Let’s Encrypt certificates has an issue when it is time to renew certificates.

About 2-3 months ago I installed 8 Bitnami WordPress sites and configured all with https:// using Bitnami’s auto-configuration script described at https://docs.bitnami.com/google/how-to/generate-install-lets-encrypt-ssl/#assumptions-and-prerequisites

At the end that script asks if you want to setup a cron job to auto renew, and when you reply Y, it did create a cron job that runs to renew them.

I assumed all was fine for the future, but all 8 of these sites are failing to renew the certs because when the cron jobs run they report this bug:

Error -> One or more domains had a problem:

[my-domain.com] [my-domain.com] error presenting token: could not start HTTPS server for challenge -> listen tcp :443: bind: address already in use

[www.my-domain.com] [www.my-domain.com] error presenting token: could not start HTTP server for challenge -> listen tcp :80: bind: address already in use

If it’s failing for 8 of my installed sites, that means it’s failing for everyone else who installed Bitnami stacks around December 2018 to January 2019?

I am starting now to debug this and figure out why Bitnami’s provided solution is failing, but if you can provide some help sooner that would sure help. I have limited time today to spend on this and my certs are all expiring in a few days.

I’ll post back here if I figure out the bug before the community does…

A quick thought – usually we stop and start Apache when installing certs, the cronjob isn’t worrying about that?

So to see every crontab on my server I ran:
sudo sh -c ‘for user in $(cut -f1 -d: /etc/passwd); do echo $user; crontab -u $user -l; done’

That revels the only crontab on my server that bitnami’s Let’s Encrypt auto-configure script installed was cron as the bitnami user with this line:

0 0 1 * * sudo /opt/bitnami/letsencrypt/lego --path="/opt/bitnami/letsencrypt" --email=“MYEMAILISHERE” --domains=MYDOMAIN.com --domains=www.MYDOMAIN.com renew && sudo /opt/bitnami/apache2/bin/httpd -f /opt/bitnami/apache2/conf/httpd.conf -k graceful

In the past, before bitnami’s provided auto-configure existed, following https://docs.bitnami.com/google/how-to/generate-install-lets-encrypt-ssl/ I would install a Let’s Encrypt renew cron job that does stop/start Apache by calling a script that contains…

#!/bin/bash

sudo /opt/bitnami/ctlscript.sh stop apache
sudo /usr/local/bin/lego --tls --email="EMAIL-ADDRESS" --domains="DOMAIN" --path="/etc/lego" renew
sudo /opt/bitnami/ctlscript.sh start apache

As explained in link above to Bitnami docs, that script has to be made executable and then you call it from a crontab.

Seems the bug then with the new Bitnami cron is it isn’t starting/stopping Apache and therefore it returns the error: could not start HTTPS server for challenge -> listen tcp :443: bind: address already in use

Am I off base with this thinking?

Hi @atlanta,

Thank you for the detailed information. When we created the auto-configure script, it was not needed to stop Apache to renew the certificates, that’s why we only configured a grateful restart after renewing the certificates. That allowed to reload the Apache’s configuration without stopping the server. However, we are going to evaluate if this is required now and will update our scripts and documentation accordingly.

Sorry for the inconvenience.

Hi @atlanta,

I used the generate-certificate.sh file to generate a new certificate and renewed it one hour later. As you can see in the screenshots, the renew was correct

image

image

This is the output I obtained when renewing the certificate

2019/03/26 12:39:27 [INFO] [jotadoku20190326.bitnamiapp.com] acme: Trying renewal with 2157 hours remaining
2019/03/26 12:39:27 [INFO] [jotadoku20190326.bitnamiapp.com] acme: Obtaining bundled SAN certificate
2019/03/26 12:39:27 [INFO] [jotadoku20190326.bitnamiapp.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz/QxqBpwQd-Npvk6iueaeBNIbdSACgXnXg_rrogewdDis
2019/03/26 12:39:27 [INFO] [jotadoku20190326.bitnamiapp.com] acme: authorization already valid; skipping challenge
2019/03/26 12:39:27 [INFO] [jotadoku20190326.bitnamiapp.com] acme: Validations succeeded; requesting certificates
2019/03/26 12:39:29 [INFO] [jotadoku20190326.bitnamiapp.com] Server responded with a certificate.

Can you try these commands to confirm that the renewal works fine?

sudo /usr/local/bin/lego --tls --email="EMAIL-ADDRESS" --domains="DOMAIN" --path="/etc/lego" renew --days 90 && sudo /opt/bitnami/apache2/bin/httpd -f /opt/bitnami/apache2/conf/httpd.conf -k graceful

Note that I added a new parameter to the command “–days 90”

The expiration date should have changed.

Thanks

Hi,

Running your script fails on my 8 server installs because your script is calling /usr/local/bin/lego but that isn’t where the Bitnami-supplied auto-configure script installed lego?

The code that Bitnami Let’s Encyrpt auto-configure originally put into my crontab to renew is this:

sudo /opt/bitnami/letsencrypt/lego --path="/opt/bitnami/letsencrypt" --email=“MYEMAIL” --domains=MYDOMAIN -&& sudo /opt/bitnami/apache2/bin/httpd -f /opt/bitnami/apache2/conf/httpd.conf -k graceful

and when I inspect /opt/bitnami/letencrypt/ it does have a single lego file in it.

Do I also need to manually install Lego in /usr/local/bin/lego ?

I am familiar with installing LEGO into /usr/local/bin/lego because before Bitnami released the auto configure script around December 2019 we always had to install Lego manually into /usr/local/bin/lego as described in the “Alternative Approach” section on this page: https://docs.bitnami.com/google/how-to/generate-install-lets-encrypt-ssl/#assumptions-and-prerequisites

But atop that page is the new docs on using the AutoConfigure script which doesn’t mention any need to install Lego etc. so my assumption was that was being handled automatically or included in the new bitnami stacks installed after Dec 2019 which came out when PHP7.2 was included.

So, what is required to use the auto configure other than what it mentions in the top of that documentation page? If you need to do more, those docs should explain them? It doesn’t give the impression you also need to install the “Alternate Approach” and manually install Lego into so a bit confusing.

In any event, the cron job autoconfigure did install doesn’t work, namely…

sudo /opt/bitnami/letsencrypt/lego --path="/opt/bitnami/letsencrypt" --email=“MYEMAIL” --domains=MYDOMAIN -&& sudo /opt/bitnami/apache2/bin/httpd -f /opt/bitnami/apache2/conf/httpd.conf -k graceful

As mentioned at the start of the thread, this contrab fails with errors that the domain is in use on ports 443 and ports 80.

Hi @atlanta,

No, you need to use the lego binary that is inside the /opt/bitnami/letsencrypt folder. I just updated the command I posted above.

In our documentation, we have some sections that use the /usr/local/bin/lego but that only applies when the user doesn’t have the letsencrypt folder and needs to install the lego binary manually in the instance.[quote=“atlanta, post:6, topic:65919”]
I am familiar with installing LEGO into /usr/local/bin/lego because before Bitnami released the auto configure script around December 2019 we always had to install Lego manually into /usr/local/bin/lego as described in the “Alternative Approach” section on this page: https://docs.bitnami.com/google/how-to/generate-install-lets-encrypt-ssl/#assumptions-and-prerequisites
[/quote]

All the new Bitnami instances include the letsencrypt folder and the generate-certificate.sh file that takes care of creating the certificate and configure it.

https://docs.bitnami.com/google/how-to/generate-install-lets-encrypt-ssl/#use-the-bitnami-auto-configuration-script

As I mention, we have the 2 approaches in the page. The automatic approach and the alternative one that only applies for those instances that don’t include the letsencrypt folder inside /opt/bitnami.

This command should work. Could you please run this command first and share with us its output?

sudo /opt/bitnami/letsencrypt/lego --tls --email="EMAIL-ADDRESS" --domains="DOMAIN" --path="/etc/lego" renew --days 90

NOTE: If that command fails, please share with us the lego version the server has

sudo /opt/bitnami/letsencrypt/lego --version

Hi, I appreciate all your help!

I ran…

sudo /opt/bitnami/letsencrypt/lego --tls --email=“EMAIL-ADDRESS” --domains=“DOMAIN” --path="/etc/lego" renew --days 90

with the correct email and domain and got a new error I’ve never seen…

You have to pass an account (email address) to the program using --email or -m

Of course, I have the correct email address and it is within double quotes, so that an unwarranted error.

My lego version is 1.2.1 perhaps I need to upgrade that?

Also, for the first time I just noticed that in the crontab that the Let’s Encrypt auto-configured created almost 3 months ago to autorenew Let’s Encrypt that autoconfigure script when it created the cron to renew failed to put double-quotes around the domain names! In the 8 different Bitnami WordPress stacks that I installed in Dec. 2018 and January 2019 using the auto-configure every cron is missing the double quotes. Seems that was a bug? These crons in those installs all have this script format…

0 0 1 * * sudo /opt/bitnami/letsencrypt/lego --path="/opt/bitnami/letsencrypt" --email=“MYEMAILADDRESS” --domains=MYDOMAIN.com --domains=www.MYDOMAIN.com renew && sudo /opt/bitnami/apache2/bin/httpd -f /opt/bitnami/apache2/conf/httpd.conf -k graceful

So surely that should have quotes around domains? Anytime I’ve run a Lego renew in the past manually it has required the quotes?

Sorry I got off track. Thinking again about the new script you gave me that is returning the email error, I went and tried it on two more stacks that are failing to renew, using their email and domain specs, and I get the same error there, too, so it’s not just this one install that has some bug, it’s a pattern in all my installs. The two I just tested also have Lego 1.2.1

Hi @atlanta,

This error is caused by the --tls option. You are using an old version of the lego tool and that’s why it fails when using that option. Can you try to run this command instead?

sudo /opt/bitnami/letsencrypt/lego --email="EMAIL-ADDRESS" --domains="DOMAIN" --path="/opt/bitnami/letsencrypt" renew --days 90

Note: I also noticed that the path was wrongly set in the command

You do not need to use quotes around your domains, the lego tool works properly without them.

Hi, thanks again for your help.

Running out of time – this domain’s cert expires tomorrow!

Your latest suggestion still doesn’t work…returns the original error that started this thread which is the same error that the auto-configure’s renewal cron job returns…

I ran this…

sudo /opt/bitnami/letsencrypt/lego --email="mrsteam1311@gmail.com" --domains=“mr-steam.com” --domains=“www.mr-steam.com” --path="/opt/bitnami/letsencrypt" renew --days 90

and got this error same as before…

acme: Error -> One or more domains had a problem:
[www.mr-steam.com] [www.mr-steam.com] error presenting token: could not start HTTP server for challenge -> listen tcp :80: bind: address already in use
[mr-steam.com] [mr-steam.com] error presenting token: could not start HTTPS server for challenge -> listen tcp :443: bind: address already in use

I’m getting same for the cron jobs running on my other Bitnami WordPress servers that were installed Dec->Feb time frame on Google Cloud, all these stacks have the same issue.

What to try next? Should I upgrade Lego from version 1.2.1 to something newer?

Hi @atlanta,

I don’t know why this version of lego is throwing this error, the newer versions work properly. Let’s do the following, create a script file that stops Apache, renews the certificate and starts Apache again.

#!/bin/bash

sudo /opt/bitnami/ctlscript.sh stop apache
sudo /opt/bitnami/letsencrypt/lego --email="mrsteam1311@gmail.com" --domains="mr-steam.com" --domains="www.mr-steam.com" --path="/opt/bitnami/letsencrypt" renew --days 90
sudo /opt/bitnami/ctlscript.sh start apache

Give execution permissions to the script

sudo chmod +x yourscript

and run it!

Was the certificate renewed?

You can also upgrade the lego version but that will break the script we include in the installation, you won’t be able to use the generate-certificate.sh script anymore. However we can help you with any questions you have in the future when generating the certificate.

The commands to upgrade the lego tool are these ones

cd /tmp
curl -Ls https://api.github.com/repos/xenolf/lego/releases/latest | grep browser_download_url | grep linux_amd64 | cut -d '"' -f 4 | wget -i -
tar xf lego_v2.4.0_linux_amd64.tar.gz
sudo mv lego /opt/bitnami/letsencrypt/lego

You could try renewing the certificate now and see if that works without stopping Apache

sudo /opt/bitnami/letsencrypt/lego --tls --email="mrsteam1311@gmail.com" --domains="mr-steam.com" --domains="www.mr-steam.com" --path="/opt/bitnami/letsencrypt" renew --days 90

Note: This version of lego requires the --tls option

As you would expect, that approach of stopping Apache, renewing, and starting Apache succeeded in renewing the certificate.

Going forward, would I do better to try to upgrade the Lego version on my Google Cloud Bitnami WordPress stacks that have the buggy cron jobs, would that be enough?

Or should I just install a new VM Instance with the latest Bitnami stack and migrate my sites into those? Sometimes I’ve found it’s quicker to migrate than to debug issues in a stack.

Or I can rewrite my cron jobs on all my buggy stacks to start and stop, like we always did before Bitnami’s Let’s Encrypt autoconfigure existed?

I’m a developer who creates 3-4 Bitnami WordPress stacks a month on Google Cloud and I haven’t checked my most recent ones to see if they are bug free regarding Let’s Encrypt renewal cron. Some haven’t yet hit the time when they would try to renew so I don’t know yet. Perhaps they all have this glitch.

When I get some free time I’ll start doing some checking to see what’s up. Perhaps the problem exists on all. I’ll look at my most recent stack I installed last week. If I find the bug exists in recent stacks I’ll post back here, since someone will need to fix the deployments on Google Cloud.

I did some auditing of my Google Cloud Bitnami WordPress stacks. My stacks done in March have Lego 2+ and running the CRON script that Bitnami created to autorenew Let’s Encrypt returns no errors. So they look OK.

The last deployment I did in February was 2/19 and that is flawed, like all the others I did in December and January until 2/19. They all have Lego 1.2.1 with Bitnami’s CRON script that returns the bug I mentioned at the beginning of this thread that prevents the Let’s Encrypt certificates from renewing.

I did not deploy these from a Bitnami account on Bitnami’s website; rather, they were deployed on Google Cloud in their marketplace which lists the Bitnami WordPress deployment there.

Hi @atlanta,

We update the lego version we include in our instances every time there is a new Lego version. You could update the Lego version included in the “old” instances by running the commands I posted above.

As I mentioned, the generate-certificate.sh script sill need some changes to work with the new versions of Lego. You can simply copy the content of one instance to the others.

Let us know if you have any other questions

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.