Bitnami HTTPS Configuration Tool (bncert) corrupted my installation

Keywords: WordPress - Google Cloud Platform - How to - Secure Connections (SSL/HTTPS)

Description:
I am trying to unpick the modifications made by bncert to my server after successfully renewing my SSL certificate.

I am running a virtual host configuration with 3 WordPress applications running behind a single static IP. This has been running successfully for more than 12 months.

The bncert script ran without errors and renewed the LetsEncrypt certificate for another 3 months, however all 3 WordPress apps were then inaccessible - with Firefox displaying the dreaded ‘page isn’t redirecting properly’ message.

I examined the temporary bncert log file that the script created, and also each file that the script produced a backup for. I then discovered it had made changes to bitnami.conf and each appname-vhost.conf file.

I use the RewriteCond %{HTTP:X-Forwarded-Proto} !https command in these files and the script had replaced this with RewriteCond %{HTTPS} !=on

So I restored the backup versions of these files and 2 of my apps are now working fine, but the main one which is also the domain used for server-name is producing too many redirects on all scripts, images and stylesheets.

Can someone help me identify what else the bncert tool has changed such that I can reverse the changes and restore my site?

@jota or @michiel - I apologise for going off-topic, but I am trying to add a new topic within the support community about a problem with the Bndiagnostic tool, but I have been unable to do so. I have previously participated in this community and used the diagnostic tool, but when recently when I run it on my server it no longer completes and provides a support code. Could either of you please message me and help me resolve this problem? Thanks / Steve

@jota or @michiel - To help understand the problem, here is the primary site that is failing to load its page elements, but has a working SSL certificate with all the http > https and www > non-www redirects.

As of yesterday this was a vibrant dynamic live site, now it look like this…

https://transmission-one.com/

I’m sure the solution is simple fix, and I’d be happy to provide you with a bndiagnostic dump, so let me know if you’d prefer to go down that route.

Hi @steve12 ,

It seems Apache/Varnish is always redirecting to the domain no matter what you use in the request

❯ curl -LI https://transmission-one.com/
HTTP/2 301
date: Thu, 12 May 2022 09:00:25 GMT
server: Apache/2.4.46 (Unix) OpenSSL/1.1.1n
location: https://transmission-one.com/
content-length: 237
content-type: text/html; charset=iso-8859-1
x-varnish: 314773 907433
age: 110
via: 1.1 varnish (Varnish/6.5)
x-cache: HIT

HTTP/2 301
date: Thu, 12 May 2022 09:00:25 GMT
server: Apache/2.4.46 (Unix) OpenSSL/1.1.1n
location: https://transmission-one.com/
content-length: 237
content-type: text/html; charset=iso-8859-1
x-varnish: 314774 907433
age: 110
via: 1.1 varnish (Varnish/6.5)
x-cache: HIT

I suggest you review the Apache’s configuration and disable Varnish if necessary.

Try to run the tool when the services are stopped

sudo /opt/bitnami/ctlscript.sh stop
sudo /opt/bitnami/bndiagnostic-tool
sudo /opt/bitnami/ctlscript.sh start
1 Like

Thanks @jota , I have been reviewing the config files over the past 12 hours, performing side-by-side comparisons against the same files on other sites using the same server but can’t find any differences.

Would bncert have changed any other files? All websites were working fine before I ran the script.

The other sites running under the same bitnami.conf are https://agravityman.com and https://fitchmedia.com and they seem to be running as before.

By ‘disabling varnish’ do you mean stopping it, or removing varnish from the config files?

And, as far as bndiagnostic, I saw one of your comments on someone else’s post and I already tried stopping all services before running it, but that makes no difference, all it does is list a series of apache log errors (unable to connect with the backend) but it still doesn’t produce a dump code. Any other suggestions? Should I try and remove bndiagnostic from the server and then reinstall?

I could provide you access to my server via SSH (by sending you the private key on a message), please let me know as I am really stumped about what could have changed after running bncert.

@jota, I tried most of your suggestions.

Stopping services and then running bndiagnostic didn’t make any difference, it runs normally (finds a few errors in the apache log) but mainly recognises the obvious connectivity issue:

Server ports 22, 80 and/or 443 are not publicly accessible. Please check the following guide to open server ports for remote access:

But the script doesn’t produce a support dump that I can send you.

Could you help me with a few suggestions:

  1. why would Apache/Varnish be redirecting to the same domain 47 times? (what should I look out for?)

The wordpress-vhost.conf file contains the same statements as agravityman-vhost.conf and fitchmedia-vhost.conf. The only difference between them is we’re running W3TC on transmission-one.com which produces a .htaccess file in the main wordpress directory. I’ve tried disabling this plugin. I replaced the .htaccess file with a standard wordpress version, and before that I flushed all the caches.

  1. Are there any other files changed by bncert? The list that I’ve examined so far includes:
  • httpd.conf (unchanged by bncert)
  • bitnami.conf (bncert changed this and removed the X-Forwarded-Proto statement)
  • bitnami-ssl.conf (unchanged)
  • wordpress-vhost.conf (changed in same way as bitnami.conf)
  • wordpress-https-vhost.conf (unchanged)

Are there any other files or configuration statements which bncert normally modifies?

FYI, here is a copy of my wordpress-vhost.conf…

# Let Apache know we're behind a SSL reverse proxy
SetEnvIf X-Forwarded-Proto https HTTPS=on

<VirtualHost *:81>
  ServerName transmission-one.com
  ServerAlias www.transmission-one.com
  DocumentRoot /wordpress installation path/
  # BEGIN: Configuration for letsencrypt
  Include "/opt/bitnami/apps/letsencrypt/conf/httpd-prefix.conf"
  # END: Configuration for letsencrypt
  # BEGIN: Support domain renewal when using mod_proxy without Location
  <IfModule mod_proxy.c>
    ProxyPass /.well-known !
  </IfModule>
  # END: Support domain renewal when using mod_proxy without Location
  # BEGIN: Enable HTTP to HTTPS redirection
  RewriteEngine On
#  RewriteCond %{HTTPS} !=on
  RewriteCond %{HTTP:X-Forwarded-Proto} !https
  RewriteCond %{HTTP_HOST} !^localhost
  RewriteCond %{HTTP_HOST} !^[0-9]+.[0-9]+.[0-9]+.[0-9]+(:[0-9]+)?$
  RewriteCond %{REQUEST_URI} !^/\.well-known
  RewriteRule ^/(.*) https://%{SERVER_NAME}/$1 [R=permanent,L]
  # END: Enable HTTP to HTTPS redirection
  # BEGIN: Enable www to non-www redirection
  RewriteCond %{HTTP_HOST} ^www\.(.*)$ [NC]
  RewriteCond %{HTTP_HOST} !^localhost
  RewriteCond %{HTTP_HOST} !^[0-9]+.[0-9]+.[0-9]+.[0-9]+(:[0-9]+)?$
  RewriteCond %{REQUEST_URI} !^/\.well-known
  RewriteRule ^(.*)$ http://%1$1 [R=permanent,L]
  # END: Enable www to non-www redirection
  <Directory "/wordpress installation path">
    Options -Indexes +FollowSymLinks -MultiViews
    AllowOverride None
    Require all granted
    # BEGIN WordPress fix for plugins and themes
    # Certain WordPress plugins and themes do not properly link to PHP files because of symbolic links
    # https://github.com/bitnami/bitnami-docker-wordpress-nginx/issues/43
    RewriteEngine On
    RewriteRule ^bitnami/installation(/.*) $1 [L]
    # END WordPress fix for plugins and themes
    # BEGIN WordPress
    # https://wordpress.org/support/article/htaccess/#basic-wp
    RewriteEngine On
    RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
    RewriteBase /
    RewriteRule ^index\.php$ - [L]
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteRule . /index.php [L]
    # END WordPress
  </Directory>

  # Enable HTTP/2 support
    Protocols h2 h2c http/1.1
    H2Direct on

  Include "/../../../../../htaccess/wordpress-htaccess.conf"
  # BEGIN: Support domain renewal when using mod_proxy within Location
  <Location /.well-known>
    <IfModule mod_proxy.c>
      ProxyPass !
    </IfModule>
  </Location>
  # END: Support domain renewal when using mod_proxy within Location
</VirtualHost>

And wordpress-https-vhost…

<VirtualHost 127.0.0.1:443 _default_:443>
  ServerName transmission-one.com
  ServerAlias www.transmission-one.com
  SSLEngine on
  SSLCertificateFile "/opt/bitnami/apache/conf/transmission-one.com.crt"
  SSLCertificateKeyFile "/opt/bitnami/apache/conf/transmission-one.com.key"
  DocumentRoot /wordpress installation path
  # BEGIN: Configuration for letsencrypt
  Include "/opt/bitnami/apps/letsencrypt/conf/httpd-prefix.conf"
  # END: Configuration for letsencrypt
  # BEGIN: Support domain renewal when using mod_proxy without Location
  <IfModule mod_proxy.c>
    ProxyPass /.well-known !
  </IfModule>
  # END: Support domain renewal when using mod_proxy without Location
  # BEGIN: Enable www to non-www redirection
  RewriteEngine On
  RewriteCond %{HTTP_HOST} ^www\.(.*)$ [NC]
  RewriteCond %{HTTP_HOST} !^localhost
  RewriteCond %{HTTP_HOST} !^[0-9]+.[0-9]+.[0-9]+.[0-9]+(:[0-9]+)?$
  RewriteCond %{REQUEST_URI} !^/\.well-known
  RewriteRule ^(.*)$ https://%1$1 [R=permanent,L]
  # END: Enable www to non-www redirection
  <Directory "/wordpress installation path">
    Options -Indexes +FollowSymLinks -MultiViews
    AllowOverride None
    Require all granted
    # BEGIN WordPress fix for plugins and themes
    # Certain WordPress plugins and themes do not properly link to PHP files because of symbolic links
    # https://github.com/bitnami/bitnami-docker-wordpress-nginx/issues/43
    RewriteEngine On
    RewriteRule ^bitnami/installation(/.*) $1 [L]
    # END WordPress fix for plugins and themes
    # BEGIN WordPress
    # https://wordpress.org/support/article/htaccess/#basic-wp
    RewriteEngine On
    RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
    RewriteBase /
    RewriteRule ^index\.php$ - [L]
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteRule . /index.php [L]
    # END WordPress
  </Directory>

  # BEGIN: Proxy all http requests to Varnish
    ProxyPreserveHost On
    RequestHeader set "X-Forwarded-Proto" expr=%{REQUEST_SCHEME}
    ProxyPass "/"  "http://127.0.0.1:80/" connectiontimeout=300 timeout=300
    ProxyPassReverse "/"  "http://127.0.0.1:80/"
  # END:

  # Enable HTTP/2 support
    Protocols h2 h2c http/1.1
    H2Direct on

  Include "/../../../../../htaccess/wordpress-htaccess.conf"
  # BEGIN: Support domain renewal when using mod_proxy within Location
  <Location /.well-known>
    <IfModule mod_proxy.c>
      ProxyPass !
    </IfModule>
  </Location>
  # END: Support domain renewal when using mod_proxy within Location
</VirtualHost>

@jota, the observation you made “It seems Apache/Varnish is always redirecting to the domain no matter what you use in the request”, got me thinking - thanks.

And I realised that Varnish doesn’t support SSL, but instead speaks plain HTTP. I suspect that when bncert removed the X-Forwarded-Proto statement, the HTTP version was stored in the Varnish cache which resulted in a redirection loop until the browser threw the ERR_TOO_MANY_REDIRECTS error.

I found a suggested workaround posted on StackOverflow (redirect - Varnish 301 redirecting loop Wordpress website - Stack Overflow) which I’ve tried and seems to partly work, after I’ve added the following code at the bottom of varnish/default.vcl

sub vcl_backend_response {

#Fix a strange problem: HTTP 301 redirects to the same page sometimes go in$
if (beresp.http.Location == “http://” + bereq.http.host + bereq.url) {
if (bereq.retries > 2) {
unset beresp.http.Location;
#set beresp.http.X-Restarts = bereq.retries;
} else {
return (retry);
}
}

}

Does this sound like the right direction? If this is the problem, then perhaps you could ask for the Bitnami documentation to be amended with instructions NOT to run bncert if you’re operating a Wordpress site behind Varnish.

Hi @jota I have removed the code (above) which I added to varnish/default.vcl, since it’s a workaround and doesn’t solve the fundamental problem of the page being requested over HTTPS but the resources being cached and loaded as HTTP.

I think this is the problem, do you agree?

Here is the explanation I found for ERR_TOO_MANY_REDIRECTS which seems to fit - Mixed content and ERR_TOO_MANY_REDIRECTS in Wordpress - Thijs Feryn

For anyone else facing the same issue, the author’s (Thijs) youtube video is particularly easy to understand - Mixed content and ERR_TOO_MANY_REDIRECTS in Wordpress - YouTube

So I have altered my wordpress-vhost.conf file to begin as follows:

SetEnvIf X-Forwarded-Proto "https" HTTPS=on
Header append Vary: X-Forwarded-Proto

And the virtual host definition now reads as follows:

  # BEGIN: Enable HTTP to HTTPS redirection
  RewriteEngine On
  RewriteCond %{HTTPS} !=on
  RewriteCond %{HTTP:X-Forwarded-Proto} !https
  RewriteCond %{HTTP_HOST} !^localhost#
  RewriteCond %{HTTP_HOST} !^[0-9]+.[0-9]+.[0-9]+.[0-9]+(:[0-9]+)?$
  RewriteCond %{REQUEST_URI} !^/\.well-known
  RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]
  # END: Enable HTTP to HTTPS redirection

When I run a CURL request I am seeing this:

> curl -LI https://transmission-one.com/
HTTP/2 200
date: Sat, 14 May 2022 12:50:02 GMT
server: Apache/2.4.46 (Unix) OpenSSL/1.1.1n
x-powered-by: PHP/7.4.19
link: <https://transmission-one.com/wp-json/>; rel="https://api.w.org/", <https://transmission-one.com/wp-json/wp/v2/pages/75812>; rel="alternate"; type="application/json", <https://wp.me/PdauE3-jIM>; rel=shortlink
vary: Accept-Encoding,X-Forwarded-Proto
last-modified: Sat, 14 May 2022 12:50:03 GMT
content-type: text/html; charset=UTF-8
x-varnish: 132161 230012
age: 66
via: 1.1 varnish (Varnish/6.5)
etag: W/"dd06d0c56e438acc5144e7377cfad984"
x-cache: HIT
accept-ranges: bytes
content-length: 191630

But on occasion I still see a redirect loop when running curl, so I am not sure the issue is ‘completely’ solved.

Could you please run your eyes across this solution with your knowledge of the Bitnami stack and tell me if this is the correct fix?

Could you also confirm if I should apply the same code to bitnami.conf or does each application’s vhost file completely replace bitnami.conf when operating?

I also note that the .htaccess file created by W3TC includes the following lines of code, which I note includes [NC], should I be using this same parameter in my vhost.conf file?:

# BEGIN W3TC Page Cache core
<IfModule mod_rewrite.c>
    RewriteEngine On
    RewriteBase /
    RewriteCond %{HTTPS} =on
    RewriteRule .* - [E=W3TC_SSL:_ssl]
    RewriteCond %{SERVER_PORT} =443
    RewriteRule .* - [E=W3TC_SSL:_ssl]
    RewriteCond %{HTTP:X-Forwarded-Proto} =https [NC]
    RewriteRule .* - [E=W3TC_SSL:_ssl]
    RewriteCond %{HTTP:Accept-Encoding} gzip
    RewriteRule .* - [E=W3TC_ENC:_gzip]
    RewriteCond %{HTTP_COOKIE} w3tc_preview [NC]
    RewriteRule .* - [E=W3TC_PREVIEW:_preview]
    RewriteCond %{REQUEST_METHOD} !=POST
    RewriteCond %{QUERY_STRING} =""
    RewriteCond %{HTTP_COOKIE} !(comment_author|wp\-postpass|w3tc_logged_out|wordpress_logged_in|wptouch_switch_toggle) [NC]
    RewriteCond %{REQUEST_URI} \/$
    RewriteCond "%{DOCUMENT_ROOT}/wp-content/cache/page_enhanced/%{HTTP_HOST}/%{REQUEST_URI}/_index%{ENV:W3TC_SSL}%{ENV:W3TC_PREVIEW}.html%{ENV:W3TC_ENC}" -f
    RewriteRule .* "/wp-content/cache/page_enhanced/%{HTTP_HOST}/%{REQUEST_URI}/_index%{ENV:W3TC_SSL}%{ENV:W3TC_PREVIEW}.html%{ENV:W3TC_ENC}" [L]
</IfModule>
# END W3TC Page Cache core

@jota as you can see, the problem has not been resolved, so I would really appreciate your response to my questions above and clarification of whether I should make further changes to avoid this problem in the Bitnami stack.

> curl -LI https://transmission-one.com/
HTTP/2 301
date: Tue, 17 May 2022 09:54:55 GMT
server: Apache/2.4.46 (Unix) OpenSSL/1.1.1n
location: https://transmission-one.com/
content-length: 237
content-type: text/html; charset=iso-8859-1
x-varnish: 1866157 1866016
age: 55
via: 1.1 varnish (Varnish/6.5)
x-cache: HIT
vary: X-Forwarded-Proto

HTTP/2 301
date: Tue, 17 May 2022 09:54:55 GMT
server: Apache/2.4.46 (Unix) OpenSSL/1.1.1n
location: https://transmission-one.com/
content-length: 237
content-type: text/html; charset=iso-8859-1
x-varnish: 1866158 1866016
age: 55
via: 1.1 varnish (Varnish/6.5)
x-cache: HIT
vary: X-Forwarded-Proto

curl: (47) Maximum (50) redirects followed

Hi @steve12,

Can you run the bndiagnostic tool?

sudo /opt/bitnami/bndiagnostic-tool

And paste the code ID outputted in the end?

Regards,
Michiel

I managed to remove bndiagnostic from my server and reinstall a fresh version (by removing the bndiagnostic directory and removing the symbolic link to bndiagnostic-tool), so I now have a code for my diagnostic bundle

14789d08-0c30-b45d-5bff-f6c3dc57a14e

Thanks @michiel ,
Steve

Hi @steve12,

Is the issue reproducible on a new instance? If you run the bncert tool again, does it corrupt the installation?

Regards,
Michiel

Hi @michiel, I haven’t tried to create a new instance, and then migrate everything across to try bncert again. I would have to configure the new instance to match my current one, which seems like a lot of extra work.

Bear in mind, there are two other websites within this same instance which do not have the problem (agravityman.com and fitchmedia.com), all served by the same SSL certificate.

My working assumption is that when bncert was run on my installation (virtual host, with Varnish reverse proxy at the front end), it removed the X-Forwarded-Proto headers from the virtual host configuration files (wordpress-vhost.conf and bitnami.conf), this then resulted in http versions of the page content being stored in the Varnish cache and after restoring these config files this caused the ERR_TOO_MANY_REDIRECTS being picked up by the browser and an infinite loop of redirects.

Does that make sense?

The changes I’ve made include

  1. adding, Header append Vary: X-Forwarded-Proto at the top of wordpress-vhost.config.
  2. I have also replaced RewriteCond %{HTTP:X-Forwarded-Proto} !https with RewriteCond %{HTTP:X-Forwarded-Proto} !https [NC] and
  3. and also changed the RewriteRule from RewriteRule ^/(.*) https://%{SERVER_NAME}/$1 [R=permanent,L] and replaced with RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]

Could you confirm, from your experience, whether these changes should correct the problem?

Sometimes, when I run a curl request the page loads without a redirect loop, but other times it doesn’t (hitting the maximum redirects threshold).

I have tried purging Varnish cache, I have also tried disabling the W3TC plugin (and therefore restoring .htaccess back to the basic Wordpress version), but it’s unclear to me what could still be causing the occasional infinite redirect loop.

Do you have any suggestions I could try? Or any explanation as to why this is happening?