Robots.txt not loading

Keywords: WordPress + NGINX + SSL - Google Cloud Platform - Technical issue - Application configuration

bnsupport ID: de7b8d1f-1aa9-4c28-dcee-b0732eaa50d1

bndiagnostic output:

? Nginx: Found possible issues
? Resources: Found possible issues
? Wordpress: Found possible issues
https://docs.bitnami.com/installer/infrastructure/nginx/troubleshooting/

bndiagnostic failure reason: The suggested guides are not related with my issue

Description:
I am not able to see the robots.txt file. i have created file under /opt/bitnami/wordpress/wp-content/ also there a robots.txt file in /opt/bitnami/phpmyadmin/ but there is no use. it is showing 404 error. even seo plugin is unable to generate robots.txt file

Hello @manzunadhreddy,

Could you detail the issue and what exactly are you trying to achieve? Do you want to download the robots.txt file, access it via URL, or just use it from the WP admin panel?

Regards,
Francisco de Paz

i want to access it with the URL. https://www.bookofer.com/robots.txt. it is showing a 404 error.

Hello @manzunadhreddy,

In that case, what you are trying to do is to serve static content via NGINX. I recommend you to check the following official documentation and a similar thread on this forum:

https://docs.nginx.com/nginx/admin-guide/web-server/serving-static-content/


Keep in mind that any server bock configuration can be performed at `/opt/bitnami/nginx/conf/server_blocks/wordpress-server-block.conf.

Regards,
Francisco de Paz

can you please check my below code in /opt/bitnami/nginx/conf/server_blocks/wordpress-server-block.conf. i think it is already there.

server {
    # Port to listen on, can also be set in IP:PORT format
    listen 80 default_server;
    root /opt/bitnami/wordpress;
    return 301 https://$host$request_uri;
    # Catch-all server block
    # See: https://nginx.org/en/docs/http/server_names.html#miscellaneous_names
    server_name _;
    # BEGIN Fix for WordPress plugins and themes
    # Certain WordPress plugins and themes do not properly link to PHP files because of symbolic links
    # https://github.com/bitnami/bitnami-docker-wordpress-nginx/issues/43
    rewrite ^/bitnami/wordpress(/.*) $1 last;
    # END Fix for WordPress plugins and themes
    # BEGIN WordPress
    # https://wordpress.org/support/article/nginx/#general-wordpress-rules
    location = /favicon.ico {
        log_not_found off;
        access_log off;
    }
    location = /robots.txt {
        allow all;
        log_not_found off;
        access_log off;
    }
   location / {
        try_files $uri $uri/ /index.php?$args;
    }
    location ~* \.(js|css|png|jpg|jpeg|gif|ico|woff2|woff|)$ {
        expires 365d;
        log_not_found off;
    }
     location ~* .(eot|otf|svg|ttf|woff|woff2)$ {
         add_header Access-Control-Allow-Origin *;
    }
     location /xmlrpc.php {
    deny all;
    }
    # END WordPress

Hello @manzunadhreddy,

I rechecked your config but I didn’t see where do you have your robots.txt file. Going by your shared conf, you should have it at /opt/bitnami/wordpress/robots.txt, as the root directive of the server block is root /opt/bitnami/wordpress. Could you please double-check that you have the .txt file in place?

Regards,
Francisco de Paz

Thanks a lot, it worked. I have placed it in the wrong location. Now it is loading.

Once again thanks @fdepaz

Hello @manzunadhreddy,

Glad to be of help! We’ll be closing the issue as this thread has now a solution. Do not doubt to open a new one if you encounter any other issues.

Regards,
Francisco de Paz