MySQL ran out of space, deleted bin files, wrong permissions

Keywords: WordPress - Google Cloud Platform - Technical issue - Permissions
bnsupport ID: e2b4b7e8-7990-6bb7-32bd-08199771b0b1
Description:
Yesterday during a backup ran out of space and I ended up deleting bin files I shouldn’t.
Mysql was not starting so I tried too much stuff until mysql finally started. The issue I believe I messed up the directory/file permissions in the way, so the server is running but website is offline.

Is there anything possible to bring it back to life or did I mess it up beyond repair?

Please help and let me know if you need any other details.

Here’s part of the tail from mysqld.log :

2021-01-05T02:55:32.225657Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/opt/bitnami/mysql' in the path is accessible to all OS users. Consider choosing a different directory.
2021-01-05T02:55:32.248296Z 0 [System] [MY-010931] [Server] /opt/bitnami/mysql/bin/mysqld.bin: ready for connections. Version: '8.0.16'  socket: '/opt/bitnami/mysql/tmp/mysql.sock'  port: 3306  MySQL Community Server - GPL.
2021-01-05T02:55:32.374407Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: '/tmp/mysqlx.sock' bind-address: '::' port: 33060
2021-01-05T03:00:01.894170Z 8 [ERROR] [MY-012592] [InnoDB] Operating system error number 13 in a file operation.
2021-01-05T03:00:01.894230Z 8 [ERROR] [MY-012595] [InnoDB] The error means mysqld does not have the access rights to the directory.
2021-01-05T03:00:01.894243Z 8 [ERROR] [MY-012216] [InnoDB] Cannot open datafile for read-only: './bitnami_wordpress/wp_options.ibd' OS error: 81
2021-01-05T03:00:01.894440Z 8 [Warning] [MY-012049] [InnoDB] Cannot calculate statistics for table `bitnami_wordpress`.`wp_options` because the .ibd file is missing. 
2021-01-05T03:00:01.895448Z 8 [Warning] [MY-012049] [InnoDB] Cannot calculate statistics for table `bitnami_wordpress`.`wp_options` because the .ibd file is missing. 
2021-01-05T03:00:01.895880Z 8 [Warning] [MY-012049] [InnoDB] Cannot calculate statistics for table `bitnami_wordpress`.`wp_options` because the .ibd file is missing.

Hi @luis.v,

Did you follow our documentation to remove and disable the binary logging?

https://docs.bitnami.com/google/apps/wordpress/troubleshooting/disable-binary-logging-mysql/

Yes, you are running into a permissions issue and it’s because the permissions of the bitnami_wordpress folder were modified. Let’s try to recover the permissions of that folder

sudo chown -R mysql:mysql /opt/bitnami/mysql/data/bitnami_wordpress
sudo chmod ug+x  /opt/bitnami/mysql/data/bitnami_wordpress
sudo /opt/bitnami/ctlscript.sh restart mysql
sudo tail -n 20 /opt/bitnami/mysql/data/mysqld.log

Happy to help!


Was my answer helpful? Click on :heart:

Hi Jota,

Thank you for helping out, really appreciate all the help I can get.

Unfortunately I did not follow your documentation on removing the binary logs. I did follow the disable part, and I turn them back on for the meantime.

As for the bitnami_wordpress folder, I ran your instructions but found out later my coworker was also trying to fix the permissions, so I ran them again and told him not to touch anything. Server was and is still giving a 522 error.
Is there a way to set the default permissions to all directories, folders and files.

Bitnami Support Tool output:
2ceb2d76-0b01-0f3a-dfe7-60ff87e39017

Here’s the mysqld.log tail:

2021-01-06T11:44:49.555791Z 0 [System] [MY-010931] [Server] /opt/bitnami/mysql/bin/mysqld.bin: ready for connections. Version: '8.0.16'  socket: '/opt/bitnami/mysql/tmp/mysql.sock'  port: 3306  MySQL Community Server - GPL.
2021-01-06T11:44:49.687011Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: '/tmp/mysqlx.sock' bind-address: '::' port: 33060
2021-01-06T11:47:28.224327Z 0 [System] [MY-010910] [Server] /opt/bitnami/mysql/bin/mysqld.bin: Shutdown complete (mysqld 8.0.16)  MySQL Community Server - GPL.
2021-01-06T11:47:32.190800Z 0 [Warning] [MY-011068] [Server] The syntax 'expire-logs-days' is deprecated and will be removed in a future release. Please use binlog_expire_logs_seconds instead.
2021-01-06T11:47:32.190928Z 0 [System] [MY-010116] [Server] /opt/bitnami/mysql/bin/mysqld.bin (mysqld 8.0.16) starting as process 4441
2021-01-06T11:47:32.194713Z 0 [Warning] [MY-013242] [Server] --character-set-server: 'utf8' is currently an alias for the character set UTF8MB3, but will be an alias for UTF8MB4 in a future release. Please consider using UTF8MB4 in order to be unambiguous.
2021-01-06T11:47:32.194729Z 0 [Warning] [MY-013244] [Server] --collation-server: 'utf8_general_ci' is a collation of the deprecated character set UTF8MB3. Please consider using UTF8MB4 with an appropriate collation instead.
2021-01-06T11:47:32.765433Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2021-01-06T11:47:32.768239Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/opt/bitnami/mysql' in the path is accessible to all OS users. Consider choosing a different directory.
2021-01-06T11:47:32.788949Z 0 [System] [MY-010931] [Server] /opt/bitnami/mysql/bin/mysqld.bin: ready for connections. Version: '8.0.16'  socket: '/opt/bitnami/mysql/tmp/mysql.sock'  port: 3306  MySQL Community Server - GPL.
2021-01-06T11:47:32.911777Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: '/tmp/mysqlx.sock' bind-address: '::' port: 33060
2021-01-06T11:57:25.955499Z 0 [System] [MY-010910] [Server] /opt/bitnami/mysql/bin/mysqld.bin: Shutdown complete (mysqld 8.0.16)  MySQL Community Server - GPL.
2021-01-06T11:57:29.950658Z 0 [Warning] [MY-011068] [Server] The syntax 'expire-logs-days' is deprecated and will be removed in a future release. Please use binlog_expire_logs_seconds instead.
2021-01-06T11:57:29.950782Z 0 [System] [MY-010116] [Server] /opt/bitnami/mysql/bin/mysqld.bin (mysqld 8.0.16) starting as process 4959
2021-01-06T11:57:29.954458Z 0 [Warning] [MY-013242] [Server] --character-set-server: 'utf8' is currently an alias for the character set UTF8MB3, but will be an alias for UTF8MB4 in a future release. Please consider using UTF8MB4 in order to be unambiguous.
2021-01-06T11:57:29.954473Z 0 [Warning] [MY-013244] [Server] --collation-server: 'utf8_general_ci' is a collation of the deprecated character set UTF8MB3. Please consider using UTF8MB4 with an appropriate collation instead.
2021-01-06T11:57:30.501755Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2021-01-06T11:57:30.504403Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/opt/bitnami/mysql' in the path is accessible to all OS users. Consider choosing a different directory.
2021-01-06T11:57:30.524793Z 0 [System] [MY-010931] [Server] /opt/bitnami/mysql/bin/mysqld.bin: ready for connections. Version: '8.0.16'  socket: '/opt/bitnami/mysql/tmp/mysql.sock'  port: 3306  MySQL Community Server - GPL.
2021-01-06T11:57:30.650358Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: '/tmp/mysqlx.sock' bind-address: '::' port: 33060

I wonder why a small site can get a database so heavy so quickly! Any idea?
It didn’t seem to be under any attack, and only use a handful of widely used plugins.

Thanks a lot

Hi @luis.v,

From the information the Bitnami Support tool obtained, I can see that MySQL is now up and running and I can also see that the WP-CLI tool can connect to the database without problems. Did you get an error when accessing the application?

Hi Jota,

MySQL was already running but website is still down and no access to Wordpress.
I still think it might be the permissions. Is there a page in the or could you share with me the the default
How about the following error?

Cheers
LV

Common WordPress Errors

On the off chance that you are experiencing a WordPress blunder message or white screen, don’t freeze. Somebody has likely experienced a similar message previously and it can without much of a stretch be addressed. This page records the most common WordPress errors experienced by WordPress clients, and gives a beginning stage to fixing them. WordPress Backing, you will likewise discover connections to more nitty gritty pages or gatherings where a volunteer will be there to help.

The White Screen of Death

Both PHP errors and information base errors can show as a white screen, a clear screen with no data, commonly referred to in the WordPress people group as the WordPress White Screen of Death (WSOD). Prior to falling back on edgy measures, there are various explanations behind the WordPress white screen of death:

A Module is causing similarity issues. In the event that you can get to the Organization Screens have a go at deactivating the entirety of your Modules and afterward reactivating them individually. On the off chance that you can’t get to your Screens, sign in to your site through FTP. Find the envelope wp-content/modules and rename the Module organizer plugins_old. This will deactivate the entirety of your Modules. You can peruse more about physically deactivating your modules in the Investigating FAQ.

Your Subject might be causing the issue. This is particularly likely on the off chance that you are encountering the white screen of death after you have recently actuated another Topic, or made Another Site in a WordPress Organization. Sign in to the WordPress Organization Screens and actuate the default WordPress Subject (for example Twenty Seventeen). In the event that you can’t get to your Organization Screens, access your site through FTP and explore to the/wp-content/topics/envelope. Rename the organizer for the dynamic Topic. The WP_DEBUG highlight regularly gives extra data.

Interior Worker Mistake

There can be various explanations behind an Interior Worker Mistake. Here are something you can do to settle it:

The most probable issue is an adulterated .htaccess record. Sign in to your site root utilizing FTP and rename your .htaccess record to .htaccess_old. Have a go at stacking your site to check whether this has tackled your concern. In the event that it works, try to visit Settings > Permalinks and reset your permalinks. This will produce another .htaccess record for you.

Take a stab at deactivating the entirety of your Modules to check whether it is a Module issue. On the off chance that you can’t get to your WordPress Organization Screens, deactivate your Modules by means of FTP by adhering to these guidelines.

Change the Topic to the WordPress default Subject (for example Twenty Seventeen) to wipe out any Subject related issues.

Increment the PHP Memory limit

Attempt re-transferring the wp-administrator and wp-incorporates organizers from a new introduce of WordPress. Mistake Building up Information base SEO packages.

In the event that you get a page highlighting the message “Mistake Setting up Information base Association,” this implies that there is an issue with the association with your information base and there could be various purposes behind this. Coming up next are potential reasons and arrangements.

Erroneous wp-config.php Data # Wrong wp-config.php Data

“Mistake building up an information base association” is normally brought about by a blunder in your wp-config.php document. Access your site in your FTP customer. Open up wp-config.php and guarantee that coming up next are right:

  • Information base name
  • Information base username
  • Information base secret word
  • Information base host

Get familiar with altering wp-config.php. In the event that you are certain your design is right you could take a stab at resetting your MySQL secret phrase physically.

Bargained Site

On the off chance that you have checked wp-config.php for errors, and affirmed with your host for facilitating issues, it is conceivable that your site has been hacked.

Output your site with Sucuri SiteCheck to guarantee that it hasn’t been undermined. In the event that it has you should look at My Site was Hacked.

Bombed Auto-Redesign

There will be circumstances when the WordPress auto-update include falls flat. Side effects include:

  • A clear white screen and no data.
  • An admonition that the update fizzled.
  • A PHP blunder message.

The WordPress programmed update highlight may flop because of a glitch in the association with the principle WordPress documents, an issue with your Web association during overhaul, or erroneous Record Consents

To refresh your WordPress site physically, see the Manual Update article.

Association Coordinated Out

The association coordinated out mistake shows up when your site is attempting to accomplish beyond what your worker can oversee. It is especially common on shared facilitating where your memory limit is confined. Here are a few things you can attempt:

Deactivate all Modules. In the event that deactivating all the WordPress Modules on your site settle the issue, reactivate them individually to see which module is causing the issue. In the event that you can’t get to your Organization Screens, read about how to physically deactivate your modules.

Change to the default WordPress Subject.

Increment your memory limit in wp-config.php. In the event that you are on shared facilitating you may need to request that your facilitating supplier increment your memory limit for you.

Increment the most extreme execution time in your php.ini document. This isn’t a WordPress center document so in the event that you don’t know how to alter it, contact your facilitating supplier to request that they increment your greatest execution time. See underneath guidelines for expanding greatest execution time.

Upkeep Mode Following Redesign

At the point when WordPress refreshes, it naturally introduces a .upkeep document. Following redesign, you may get a message that says “Quickly inaccessible for planned upkeep. If it’s not too much trouble inquire in a moment.” The support record might not have been eliminated appropriately.

To eliminate this message do the accompanying:

  • Sign in to your site utilizing your FTP program
  • Erase the .upkeep record, which will be found in your site root.
  • Peruse more about the support mode issue.

You Make Changes and Nothing Occurs

In the event that you are causing changes to your site and you to don’t see the adjustments in your program, you may have to clear your program reserve. Your program stores data about the sites that you visit. This makes it quicker to stack sites when you visit them on the grounds that the program simply needs to reload data previously put away on your PC, as opposed to downloading it once more.

On the off chance that you roll out an improvement to a site and the program doesn’t think it is huge, it will essentially stack the information from your reserve, and you won’t see your commercial fridge. To fix the issue, just void your program store or close the tab and resume the connection.

Pretty Permalinks 404 and Pictures not Working

In the event that you are encountering 404 errors with pretty permalinks and a white screen when you transfer pictures, mod_rewrite may not be empowered in Apache as a matter of course. Mod_rewrite is an augmentation module of the Apache web worker programming which takes into account “changing” of URLs on-the-fly. It’s what you need to make pretty permalinks work.

WordPress Multisite networks typically experience this however it can likewise happen on shared facilitating suppliers or after a site relocation or worker move.

Reset your permalinks through Settings > Permalinks. In the event that this doesn’t work, you may need to alter the .htaccess record physically.

 # Start WordPress 
 <IfModule mod_rewrite.c> 
 RewriteEngine On 
 RewriteBase/ 
 RewriteRule ^index\.php$ - [L] 
 RewriteCond %{REQUEST_FILENAME} !- f 
 RewriteCond %{REQUEST_FILENAME} !- d 
 RewriteRule . /index.php [L] 
 </IfModule> 
 # END WordPress 

On the off chance that you are inexperienced with altering your .htaccess record, contact your facilitating supplier to request that they turn on mod_rewrite rules. There is more data on lovely permalinks in the WordPress Codex.

Custom Post Sort 404 Errors

You may encounter issues with 404 errors and custom post sorts. Attempt the accompanying advances:

Ensure that none of your Custom Post Sorts and single pages have a similar name. In the event that they do, rename the single page, including the slug.

Sign in to your WordPress Organization Screens, explore to Settings > Permalinks. Select the default permalinks. Save. At that point reselect your favored permalinks. This will flush the modify governs and ought to take care of your concern.

I was using the mobile earlier so the text got messed up!
I meant, is there a support page with the default permissions/ownership for the database, or could you send me instructions to reset them to default.
Many thanks,
LV

Hi @luis.v,

What error do you get when accessing the application? If the database is running and you can access it using the bn_wordpress user and the password that is configured in the /opt/bitnami/apps/wordpress/htdocs/wp-config.php file, then there shouldn’t be any issue related with the database.

Hi Jota,

The day I asked for help here I remember having access to the database using the pw inside wp-config.php
I’m going to give another try later and report back.

If the database is no longer the issue, What else could be causing the 522 error?

Once again, thanks for all the help

LV

Hi @jota,

Database is running, and I can access it using the password from config.php when running mysql -u root -p .
I clear the browser cache, but still shows error 522: Connection timed out and no sign of Wordpress site.
The backups I keep are in UpdraftPlus, should have kept a snapshot in the Google Cloud Platform instead!

Any ideas on how to get the site back online?

Thanks,
LV

Hi @jota

Forgot to mention I get these errors when I run:
/opt/bitnami$ du -h . -d 1 | sort -h
du: cannot read directory ‘./var/data’: Permission denied
du: cannot read directory ‘./mysql/data/performance_schema’: Permission denied
du: cannot read directory ‘./mysql/data/bitnami_wordpress’: Permission denied
du: cannot read directory ‘./mysql/data/#innodb_temp’: Permission denied
du: cannot read directory ‘./mysql/data/sys’: Permission denied
du: cannot read directory ‘./mysql/data/mysql’: Permission denied

hi @luis.v,

Let’s check the Apache’s and PHP’s log. Can you run the Bitnami Support tool again?

Thanks

Here it is:

0a9f3b2c-05fb-9450-682d-3f7b6d26c2c7

Hi @luis.v,

I just found that your domain is pointing to cloudflare and the connection to your server can’t be established

https://www.whatsmydns.net/#A/geekoos.com

I tried to get more info directly from your instance but it seems I can’t reach it

$ nc -zv -w2 35.237.133.151 80
151.133.237.35.bc.googleusercontent.com [35.237.133.151] 80 (http) : Connection timed out
$ nc -zv -w2 35.237.133.151 22
151.133.237.35.bc.googleusercontent.com [35.237.133.151] 22 (ssh) : Connection timed out
$ nc -zv -w2 35.237.133.151 443
151.133.237.35.bc.googleusercontent.com [35.237.133.151] 443 (https) : Connection timed out

Could you please ensure Cloudflare is configured with the correct IP address of your instance and that the ports 22, 80 and 443 in the firewall are open?

https://docs.bitnami.com/google/faq/administration/use-firewall/

Thanks

Hi again,

I just accessed your site again and it seems it’s working from time to time

This seems to be a performance issue in the instance. Can you check if there is a bot/attacker accessing your site with malicious intentions?

https://docs.bitnami.com/google/apps/wordpress/troubleshooting/deny-connections-bots-apache/

Hi @jota,

Thank you so much, problem solved.
Such a simple thing! I cannot believe I didn’t check the DNS settings in Cloudflare.
No idea why but it was pointing to a different IP address.

Also sorted the ssh access. Checked the link you sent, all is now in order with the ports and publickey in Google Cloud.

Cannot thank you enough.
Cheers,
Luis

Hi @luis.v,

I’m really glad to hear that! :tada:

Enjoy! :slight_smile:

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.