Logrotate creating masses of couchdb.stderr-20160911.gz- files leading to 100% cpu

CouchDB
#1

have an instance of Bitnami CouchDB on google
noticed very high CPU usage and found logrotate running at 100% CPU
(came back from holiday to find this instance had been running 100% for 5 days!)
killed it and investigated found 1,003,996 files (195M) in /opt/bitnami/couchdb/var/log/couchdb/

looks like the logrotate is creating muttiple gzips of the logs

e.g.

-rw-r–r-- 1 root root 0 Jan 10 22:12 couchdb.stderr-20160911.gz-20160918.gz-20161016.gz-20161023.gz-20161030.gz-20161106.gz-20161204.gz-20161211.gz
-rw-r–r-- 1 root root 0 Jan 10 22:12 couchdb.stderr-20160911.gz-20160918.gz-20161016.gz-20161023.gz-20161030.gz-20161106.gz-20161204.gz-20161211.gz-20161218.gz
-rw-r–r-- 1 root root 0 Jan 10 22:12 couchdb.stderr-20160911.gz-20160918.gz-20161016.gz-20161023.gz-20161030.gz-20161106.gz-20161204.gz-20161211.gz-20161218.gz-20161225.gz
-rw-r–r-- 1 root root 0 Jan 10 22:12 couchdb.stderr-20160911.gz-20160918.gz-20161016.gz-20161023.gz-20161030.gz-20161106.gz-20161204.gz-20161211.gz-20161218.gz-20161225.gz-20170101.gz
-rw-r–r-- 1 root root 117 Nov 28 06:30 couchdb.stderr-20160911.gz-20160918.gz-20161016.gz-20161023.gz-20161030.gz-20161106.gz-20161204.gz-20161211.gz-20161218.gz-20161225.gz-20170101.gz-20170108.gz
-rw-r–r-- 1 root root 20 Jan 8 10:08 couchdb.stderr-20160911.gz-20160918.gz-20161016.gz-20161023.gz-20161030.gz-20161106.gz-20161204.gz-20161211.gz-20161218.gz-20161225.gz-20170101.gz-20170109.gz
-rw-r–r-- 1 root root 20 Jan 9 16:36 couchdb.stderr-20160911.gz-20160918.gz-20161016.gz-20161023.gz-20161030.gz-20161106.gz-20161204.gz-20161211.gz-20161218.gz-20161225.gz-20170101.gz-20170110.gz

so I killed logrotate

sudo killall logrotate

deleted all the couchdb.stderr gz files

sudo nice -n12 find /opt/bitnami/couchdb/var/log/couchdb -name couchdb.stderr-*.gz -delete

ran logrotate again

/usr/sbin/logrotate -vdf /etc/logrotate.conf

all good

but need some help on what to do to stop this reoccuring ?

thanks pb…

#2

Hello,

Could you let us know your current logrotate configuration?

By default, we configure it to rotate the logs weekly and to only store four of them for backup purposes:

# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# uncomment this if you want your log files compressed
#compress

# packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own wtmp, or btmp -- we'll rotate them here
/var/log/wtmp {
    missingok
    monthly
    create 0664 root utmp
    rotate 1
}

/var/log/btmp {
    missingok
    monthly
    create 0660 root utmp
    rotate 1
}

Regards,
Jose

#3

Gidday Jose
just the std as when installed last year see below thanks pb…

# see "man logrotate" for details
# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# uncomment this if you want your log files compressed
#compress

# packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own wtmp, or btmp -- we'll rotate them here
/var/log/wtmp {
    missingok
    monthly
    create 0664 root utmp
    rotate 1
}

/var/log/btmp {
    missingok
    monthly
    create 0660 root utmp
    rotate 1
}

# system-specific logs may be configured here
#4

Hello,

Could you let us know the content of the following file?:

/opt/bitnami/config/logrotate/logrotate.d/couchdb.conf

This is the initial configuration of logrotate for CouchDB in GCP. You can edit this file to reduce the number of rotations you want to keep if you consider it.

Regards,
Jose

#5

Gidday
here is the contents from /opt/bitnami/config/logrotate/logrotate.d/couchdb.conf
I have reduce the no to 20

thanks pb…

/opt/bitnami/couchdb/var/log/couchdb/couchdb.* {
  weekly
  rotate 150
  dateext
  compress
  copytruncate
  missingok

}
#6

Hi @pblakes,

Thank you for letting us know. Note that you will need to restart monit for the changes to take effect.

sudo monit reload

https://docs.bitnami.com/google/components/monit/

Let us know if that fixes the issue.

Regards,
Jota

#7

This same bug is still present, and caused my server to stop functioning because it generated so many files that it used all the inodes available!

This is caused by a bug in the CouchDB logrotate config. This is the default config:

/opt/bitnami/couchdb/var/log/couchdb/couch.* {
  weekly
  rotate 150
  dateext
  compress
  copytruncate
  missingok 
}

However, the target path of couchdb/couch.* combined with copytruncate means that each previous copy is copied again every time logrotate runs, resulting in an exponential number of copies!

This is fixed by changing the path to only target the current couch.log file, and not the generated copies:
/opt/bitnami/couchdb/var/log/couchdb/couch.log

In my case, I started with a clean Bitnami deployment of CouchDB 2.3.0 on Google Cloud with minimal config changes. After running for a while, I noticed the CPU consistently running at 100%, with a lot of “disk full” errors in my server logs. It is made worse because the default CouchDB log level is “info”, which generates a lot of logs.

Please fix this configuration bug since this is present even as recent as version 2.3.0, and will end up crashing the server due to the proliferation of log files!

#8

Hi @zhicong,

Thank you for the information. The team will take a look at the CouchDB configuration.

Thanks

#9

Hi,

I just launched an instance in Google Cloud and saw this logrotate configuration

root@bitnami-couchdb-dm-4fae:/home/bitnami# cat /opt/bitnami/config/logrotate/logrotate.d/couchdb.conf
/opt/bitnami/couchdb/var/log/couchdb/couch.log {
  weekly
  rotate 150
  dateext
  compress
  copytruncate
  missingok

}

Could you confirm which instance you launched? I launched 2.3.1-2

Best regards,

Javier J. Salmerón

Was my answer helpful? Click on :heart: