Configuring virtual hosts for three websites on the same IP under Varnish/SSL

steve12 2019-09-05 15:43:20 UTC #1

Keywords: WordPress - Google Cloud Platform - Technical issue - Secure Connections (SSL/HTTPS)
bnsupport ID: 22313426-bf20-921d-de96-dedbdfa62fc1
Description:
I have successfully configured a virtual host environment for 3 websites on my GCP instance and am just tidying up the final stages before redirecting the DNS and making them live. I have run a similar setup on the Rackspace cloud for nearly 10 years, so it's just a case of figuring out what I need to do differently with the Bitnami stack.

In the section 'Use Varnish ™ with SSL' within the documentation (https://docs.bitnami.com/oci/apps/wordpress/administration/configure-use-varnish/), you advise to modify the /opt/bitnami/apache2/conf/bitnami/bitnami.conf file by adding the following lines;

    ProxyPreserveHost On
    RequestHeader set "X-Forwarded-Proto" expr=%{REQUEST_SCHEME}
    ProxyPass "/"  "http://127.0.0.1:81/"
    ProxyPassReverse "/"  "http://127.0.0.1:81/"

And then modify each application's /opt/bitnami/apps/APPNAME/conf/httpd-prefix.conf file and include the following line;

SetEnvIf x-forwarded-proto https HTTPS=on

But, in order to make the vhosts live, I will need to add the following to /opt/bitnami/apache2/conf/bitnami/bitnami-apps-vhosts.conf;

Include "/opt/bitnami/apps/firstapplication/conf/httpd-vhosts.conf" 
Include "/opt/bitnami/apps/secondapplication/conf/httpd-vhosts.conf" 
...

And remove or comment out the following lines in /opt/bitnami/apache2/conf/bitnami/bitnami-apps-prefix.conf;

Include "/opt/bitnami/apps/firstapplication/conf/httpd-prefix.conf" 
Include "/opt/bitnami/apps/secondapplication/conf/httpd-prefix.conf" 
...

So, should I add the SetEnvIf command within each application's httpd-vhosts.conf file instead?

And how should I configure the Wordpress domain name? (https://docs.bitnami.com/google/apps/wordpress/administration/configure-domain/)

Do I still add the following in each app's wp-config.php file?

define('WP_SITEURL', 'http://firstdomain.com/');
define('WP_HOME', 'http://firstdomain.com/');

Or should that be written as https://firstdomain.com/? And will either still work with my virtual host configuration?

Thanks.

tomasp 2019-09-06 12:59:38 UTC #2

Hi @steve12,

Correct. As you are configuring your application using virtual hosts instead prefix, you need to add the SetEnvIf x-forwarded-proto https HTTPS=on line in each /opt/bitnami/apps/XXXX/conf/httpd-vhosts.conf.

Remember that after made modifications to the apache configuration files you can run apachectl -t to check the syntax of your files.

You need to do both things. Configure your domain in the httpd-vhosts.conf file and also in the wp-config.php as it explained in the link you have shared.

I hope it helps,
Tomas

steve12 2019-09-06 17:02:15 UTC #3

Thanks @tomasp.

After making the sites live I was initially getting 500 errors except on the homepage and wp-admin, so I immediately guessed it must be a rewrite problem.

For anyone else who stumbles across this support post, you need to modify the rewrite statement for each vhost website within the httpd-app.conf file. By default it is set up to run in prefix mode, but needs to look like this;

    RewriteEngine On
    RewriteBase /
    RewriteRule ^index\.php$ - [S=1]
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteRule . /index.php [L]

tomasp 2019-09-09 08:14:23 UTC #4

Hi @steve12,

Thanks for taking the time to update this post with the missing piece of configuration. As you have pointed out, it can be very useful for other people checking this same support case.

steve12 2019-09-11 18:37:03 UTC #5

A. Do you have any documentation on how the bncert-tool works? (at a command line level).

I've read the the bitnami guides -

https://docs.bitnami.com/google/how-to/generate-install-lets-encrypt-ssl/#alternative-approach, and
https://docs.bitnami.com/google/how-to/understand-bncert/

I also watched the video where @michiel mentions that arguments can be added to the command;

sudo ./bncert-tool --enable_https_redirection 0

B. Is there anything else I should know?

C. As I mentioned above, I have three websites running within VirtualHosts under the same IP. So I would like to know how the bncert-tool would read my Bitnami configuration in order to make the 'correct' changes to my server.

I already ran the bncert-tool (before I had finished configuring varnish) and this generated the certificate and key for my primary domain but failed to create a cron job (that's something I still need to investigate).

It also made changes to the following files;

/opt/bitnami/apache2/conf/httpd.conf 
/opt/bitnami/apache2/conf/bitnami/bitnami-apps-prefix.conf 
/opt/bitnami/apache2/conf/bitnami/bitnami.conf

D. Now within httpd.conf, it changed ServerName from localhost to that of my primary domain, which should be ok (even though I have 3 domains on the same instance).

E. But since I am running my websites from the bitnami-apps-vhost.conf file, I assume I need to move the following statement from bitnami-apps-prefix.conf where it was created by the bncert-tool?

Include "/opt/bitnami/apps/letsencrypt/conf/httpd-prefix.conf"

F. And should I create a file called /opt/bitnami/apps/letsencrypt/conf/httpd-vhost.conf or just use this httpd-prefix.conf file (even though I am not running in prefix mode)?

G. And finally, what should I do about bitnami.conf? I realise this is the default configuration, so will the virtual host configurations for each website in /opt/bitnami/apps/app-name/conf/httpd-vhosts.conf override any statements in bitnami.conf?

I would normally list the certificates for each domain in their respective VirtualHost declaration, whereas the bitnami.conf file seems written from the perspective of a single domain.

Should I comment out the certificate statements in bitnami.conf or just leave them referring to my primary domain?

SSLCertificateFile "/opt/bitnami/apache2/conf/primary-domain.com.crt"
SSLCertificateKeyFile "/opt/bitnami/apache2/conf/primary-domain.com.key"

Thanks!

steve12 2019-09-12 17:07:33 UTC #6

Hi @tomasp, I would really appreciate your support in solving this problem. Perhaps together we can document a solution for anyone else seeking to make a virtualhost approach work with SSL and Varnish.

I ran bncert again, this time for all 3 websites and the script completed with one failure:

Warning: Certificates may not renew automatically, due to a web server configuration issue. For more information see: https://docs.bitnami.com/general/how-to/understand-bncert/#certificates-not-renewed-automatically

Some errors occurred
The configuration was applied, but some of the changes could not be applied. Find the details below.
The configuration report is shown below.
Failed steps:* Creating Let's Encrypt certificate: Automatic renewal not working

However, the server changes took all of my websites offline, so I have restored the backups and created secondary backup files of the config files changed by bncert as follows;

/opt/bitnami/apache2/conf/httpd.conf.bncert.back.201909121557
/opt/bitnami/apache2/conf/bitnami/bitnami.conf.bncert.back.201909121557

I assume that the 2 certificate files created by bncert are bundled and usable for all 3 websites. These are stored in /opt/bitnami/letsencrypt/certificates and linked to from _/opt/bitnami/apache2/conf/

I ran the Bitnami Support Tool which uploaded the following support bundle to your servers (6ac8996d-098c-c388-7949-baef635ef07b).

I added the following to each website's vhost config (i.e. /opt/bitnami/apps/firstapplication/conf/httpd-vhosts.conf)

SetEnvIf x-forwarded-proto https HTTPS=on
SSLEngine on
SSLCertificateFile "/opt/bitnami/apache2/conf/firstapplication.com.crt"
SSLCertificateKeyFile "/opt/bitnami/apache2/conf/firstapplication.com.key"

So, the certificates have been generated, the default rewrite statements have been added to bitnami.conf and I have repeated these within the specific vhost files for each website.

So what's missing?

One area that might be problematic is the change made to httpd.conf, which replaces ServerName localhost:80 with ServerName firstapplication:80. Shouldn't this use port 81 given that I've swapped ports with Varnish?

Otherwise, I can't see where it's misconfigured. Can you help?

steve12 2019-09-12 17:31:33 UTC #7

@tomasp / @michiel, I noticed at the bottom of the logfile (created by bncert) the following;

Domain firstapplication.com did not pass HTTP challenge validation

And according to the Let's Encrypt documentation ".. The HTTP-01 challenge can only be done on port 80".

But according to the logs, the certificate was issued (Message: Server responded with a certificate).

Is this likely to be the source of my problem?

marcos 2019-09-13 14:33:13 UTC #8

Hi @steve12,

For generating the certificates we use TLS challenges, which cause fewer issues as it takes control of those port. In order to do this, we stop the services with the Bitnami HTTPS tool.

However, in the case of a renewal, stopping servers is not a good option as it could cause downtime. That's why we use HTTP challenge in cron jobs which does not have this downside.

We configure requests to create requests to /.well-known. You can test this on your side:

Create a file with random contents in /opt/bitnami/apps/letsencrypt/.well-known (i.e. /opt/bitnami/apps/letsencrypt/.well-known/hello with the text world).
Access http://yourdomain.com/.well-known/hello in your web browser and you should see world
If you don't, within the instance's SSH console, run "curl http://localhost:81/.well-known/hello" and you should see it. And running "curl http://localhost/.well-known/hello" you would not in this scenario.

If you don't see the file, it is probably because your Varnish configuration is not taking into account this directory properly and would need to be updated.

Could you let us know if you are able to fetch the file from your web browser, and if the curl commands within the SSH console work?

steve12 2019-09-13 15:19:18 UTC #9

Hi @marcos,

I created the file 'hello' within the folder /opt/bitnami/apps/letsencrypt/.well-known and placed an echo 'Hello' message in it, but just get a 404 page when running http://yourdomain.com/.well-known/hello.

But then that's what I would normally expect given that it's not in the htdocs directory.

Running the two curl commands produces the same 404 response:

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /.well-known/hello was not found on this server.</p>
</body></html>

steve12 2019-09-13 15:32:23 UTC #10

I tried again, this time after adding the following line within /opt/bitnami/apache2/conf/bitnami/bitnami-apps-prefix.conf

Include "/opt/bitnami/apps/letsencrypt/conf/httpd-prefix.conf"

I still get a 404 when trying to access the file from my browser, but the first curl command now produces:

<?php
        echo 'Hello!';
?>

While the second results in:

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /.well-known/hello was not found on this server.</p>
</body></html>

marcos 2019-09-13 15:39:06 UTC #11

Hi @steve12, I see what your issue is now. You are using virtual hosts, meaning additional configuration is necessary. It is listed here: https://docs.bitnami.com/aws/how-to/understand-bncert/#certificates-not-renewed-automatically

In summary you will need to add the line below inside all httpd-vhosts.conf files which will support Let's Encrypt, inside the <VirtualHost> tag:

Alias /.well-known/ "/opt/bitnami/apps/letsencrypt/.well-known/"
<Directory "/opt/bitnami/apps/letsencrypt/.well-known">
    Options +MultiViews
    AllowOverride None
    Require all granted
</Directory>

I.e. /opt/bitnami/apps/wordpress/conf/httpd-vhosts.conf would have the following:

<VirtualHost *:81>
    ServerName ...
    ServerAlias ...
    DocumentRoot "/opt/bitnami/apps/wordpress/htdocs"

    Alias /.well-known/ "/opt/bitnami/apps/letsencrypt/.well-known/"
    <Directory "/opt/bitnami/apps/letsencrypt/.well-known">
        Options +MultiViews
        AllowOverride None
        Require all granted
    </Directory>

    Include "/opt/bitnami/apps/wordpress/conf/httpd-app.conf"
</VirtualHost>
...

steve12 2019-09-13 15:47:55 UTC #12

Thanks @marcos, I will make those changes.

Then what? Should I run bncert again?

I understand that the .well known folder us used to validate and approve the certificates, but why would this cause my SSL configuration to fail? According to the log file, my certificates were approved and the only step that failed was setting up the cron job.

The main problem I have is that I am is that I'm still unable to get SSL working on my virtualhost configuration.

steve12 2019-09-13 15:56:01 UTC #13

Hi @marcos, I can confirm that after adding those lines within each httpd-vhosts.conf file that the browser test now passes and the file is accessible via httpd port 81.

steve12 2019-09-13 17:00:56 UTC #14

@marcos, could you help me figure out what to do next?

The certificates which were generated by bncert seem to be in the right place on my server and I've added the SSLCertificateFile and SSLCertificateKeyFile statements within the default bitnami.conf and in the statements within each httpd-vhosts.conf file. These now look like this for each web application:

<VirtualHost *:81>
    ServerName mydomain.com
    ServerAlias www.mydomain.com
    DocumentRoot "/opt/bitnami/apps/mydomain/htdocs"
    Alias /.well-known/ "/opt/bitnami/apps/letsencrypt/.well-known/"
    <Directory "/opt/bitnami/apps/letsencrypt/.well-known">
        Options +MultiViews
        AllowOverride None
        Require all granted
    </Directory>
    Include "/opt/bitnami/apps/mydomain/conf/httpd-app.conf"
</VirtualHost>

<VirtualHost *:443>
    ServerName mydomain.com
    ServerAlias www.mydomain.com
    DocumentRoot "/opt/bitnami/apps/mydomain/htdocs"
    SetEnvIf x-forwarded-proto https HTTPS=on
    SSLEngine on
    SSLCertificateFile "/opt/bitnami/apache2/conf/mydomain.com.crt"
    SSLCertificateKeyFile "/opt/bitnami/apache2/conf/mydomain.com.key"
    SSLCertificateFile "/opt/bitnami/apps/mydomain/conf/certs/server.crt"
    SSLCertificateKeyFile "/opt/bitnami/apps/mydomain/conf/certs/server.key"
    Include "/opt/bitnami/apps/mydomain/conf/httpd-app.conf"
</VirtualHost>

The bitnami.conf file includes the following rewrite statements in the VirtualHost_default:81 declaration:

<VirtualHost _default_:81>
  DocumentRoot "/opt/bitnami/apache2/htdocs"
  RewriteEngine On
  # BEGIN: Enable HTTP to HTTPS redirection
  RewriteCond %{HTTPS} !=on
  RewriteCond %{HTTP_HOST} !^(localhost|127.0.0.1)
  RewriteCond %{REQUEST_URI} !^/\.well-known
  RewriteRule ^/(.*) https://%{SERVER_NAME}/$1 [R,L]
  # END: Enable HTTP to HTTPS redirection
  # BEGIN: Enable www to non-www redirection
  RewriteCond %{HTTP_HOST} ^www\.(.*)$ [NC]
  RewriteCond %{HTTP_HOST} !^(localhost|127.0.0.1)
  RewriteCond %{REQUEST_URI} !^/\.well-known
  RewriteRule ^(.*)$ http://%1$1 [R=permanent,L]
  # END: Enable www to non-www redirection
  <Directory "/opt/bitnami/apache2/htdocs">
    Options Indexes FollowSymLinks
    AllowOverride All
    <IfVersion < 2.3 >
      Order allow,deny
      Allow from all
    </IfVersion>
    <IfVersion >= 2.3 >
      Require all granted
    </IfVersion>
  </Directory>

  # Error Documents
  ErrorDocument 503 /503.html

  # Bitnami applications installed with a prefix URL (default)
  Include "/opt/bitnami/apache2/conf/bitnami/bitnami-apps-prefix.conf"

  # Enable HTTP/2 support
  Protocols h2 h2c http/1.1

</VirtualHost>

And the following statements in <VirtualHost default:443>:

<VirtualHost _default_:443>
  DocumentRoot "/opt/bitnami/apache2/htdocs"
  RewriteEngine On
  # BEGIN: Enable www to non-www redirection
  RewriteCond %{HTTP_HOST} ^www\.(.*)$ [NC]
  RewriteCond %{HTTP_HOST} !^(localhost|127.0.0.1)
  RewriteCond %{REQUEST_URI} !^/\.well-known
  RewriteRule ^(.*)$ https://%1$1 [R=permanent,L]
  # END: Enable www to non-www redirection

  SSLEngine on
  SSLCertificateFile "/opt/bitnami/apache2/conf/mydomain.com.crt"
  SSLCertificateKeyFile "/opt/bitnami/apache2/conf/mydomain.com.key"

  <Directory "/opt/bitnami/apache2/htdocs">
    Options Indexes FollowSymLinks
    AllowOverride All
    <IfVersion < 2.3 >
      Order allow,deny
      Allow from all
    </IfVersion>
    <IfVersion >= 2.3 >
      Require all granted
    </IfVersion>
  </Directory>

  # Error Documents
  ErrorDocument 503 /503.html

  # Configure Apache to proxy all HTTPS requests to Varnish
  ProxyPreserveHost On
  RequestHeader set "X-Forwarded-Proto" expr=%{REQUEST_SCHEME}
  ProxyPass "/"  "http://127.0.0.1:81/"
  ProxyPassReverse "/"  "http://127.0.0.1:81/"

  # Bitnami applications installed with a prefix URL (default)
  Include "/opt/bitnami/apache2/conf/bitnami/bitnami-apps-prefix.conf"
  # Enable HTTP/2 support
  Protocols h2 h2c http/1.1

</VirtualHost>

# Bitnami applications that uses virtual host configuration
Include "/opt/bitnami/apache2/conf/bitnami/bitnami-apps-vhosts.conf"

The bncert tool updated httpd.conf and replaced ServerName localhost:80 with ServerName mydomain.com:80, but this on its own replaces my website with the Bitnami welcome page "..Awesome! WordPress is now installed. You just installed WordPress using Bitnami - the fastest, easiest and most secure way to deploy your favorite app."
With the (above) configuration applied to bitnami.conf and httpd-vhosts.conf , an error message is displayed on a blank white page about my server being misconfigured.
So, any ideas about what to try next? I can run bncert again to enable the cron job, but can you see any reason why my SSL configuration does not work?

marcos 2019-09-17 14:09:05 UTC #15

Hi @steve12, as for bncert replacing "ServerName yourdomain.com:81" with "ServerName yourdomain.com:81", that is an actual bug in Bncert we'll work on fixing. We were not aware of it until now. I guess the error could be caused to starting Varnish and Apache, and for Apache to be configured to the specific port.

If you revert that specific change and restart all services, does it work properly now?

If so, and as long as any file placed inside .well-known is visible in the browser at the /.well-known path for affected subdomains, SSL renewal should work properly.

Unfortunately running Bncert does not seem to be an option for you as long as Varnish is enabled. As mentioned, we will work on fixing Bncert but unfortunately don't have an ETA for when it will be released.

steve12 2019-09-18 14:21:18 UTC #16

Hi @marcos, as I said I have 'two' issues; the easiest to fix is setting up a cron job to renew my certificates. The second (and more important) issue is configuring my virtual hosts to be accessed through SSL under a Varnish reverse proxy layer. That's the question I have been asking about from the very beginning of this post.

Could you please try and answer the question as framed in my previous post (#14) namely, have I set up my configuration (httpd.conf, bitnami.conf, httpd-vhosts.conf) in such a way that SSL should now work within the structure of the Bitnami stack (with Varnish enabled).

I know how to configure SSL. I've been running virtual hosts servers elsewhere for many years (so know how to configure these in apache). The only thing that's 'new' to me (and for which I am seeing support) is achieving these within the framework of the Bitnami stack running Varnish.

Please let me know if you can help.

steve12 2019-09-20 11:00:35 UTC #17

I have found a solution, which simply involves taking all the SSL-specific commands from bitnami.conf and adding these directly in each site's httpd-vhosts.conf file.

For some reason, which I don't have time at the moment to investigate, the inheritance between bitnami-conf and virtualhosts doesn't seem to work, so it's cleaner to configure each virtualhost separately.

If in the future you'd like to investigate the reasons why, and therefore create a documented guide on virtualhosts within the Bitnami stack then please get in touch, but for the time being I have found a (less than elegant) way of making SSL work.

marcos 2019-09-23 16:14:58 UTC #18

That's a really strange solution, as "httpd-vhosts" files are loaded right after "bitnami.conf" and should inherit configurations such as "SSLHonorCipherOrder", "SSLSessionCache", etc.

However I'm glad it worked for you. Could you share the resulting vhost file with us so we can check it on our side? If you prefer to keep it hidden and run the Bitnami Support tool instead that would be great. Thanks!

steve12 2019-12-06 16:14:38 UTC #19

@marcos sorry for the delay in responding. If you would still like to inspect my vhost files, I have uploaded a snapshot to your servers via the Bitnami Support tool - ref# 9cbfdf9a-1866-f774-f60e-cf0295f6a3a7

Bitnami provides free all-in-one installers, virtual machines and cloud images for popular open source applications. Get them now!