Phabricator won't start up after modifying EC2 instance type

scott3 2020-01-07 20:13:04 UTC #1

Keywords: Phabricator - AWS - Technical issue - Upgrade
bnsupport ID: ae4b7af9-9543-6ec2-8067-b12f9ec06b2b
Description:
I followed the instructions here: https://docs.bitnami.com/aws/faq/administration/change-server-type/ in order to upgrade the instance type because phabricator was having out of memory errors.

After completing the upgrade and restarting the server, phabricator does not start anymore with the message "Phabricator could not be started".

Looking at the error logs in /opt/bitnami/apps/phabricator/tmp/phd/log/daemons.log shows repeated AphrontConnectionQueryException and that the connection was refused.

I checked /opt/bitnami/apps/phabricator/htdocs/conf/local/local.json which shows the mysql user, port, host, and password are still configured correctly.

Restarting did not fix anything. mysql, php-fpm, and httpd are all able to start.

/home/bitnami/apps/phabricator/bnconfig was originally disabled but I re-enabled it and restarted the instance and that didn't fix anything.

Please help! Not sure what else to try.

miguelaeh 2020-01-08 09:46:42 UTC #2

Hi @scott3,
I can see on the logs that the Apache server is being constantly restarted. Could you provide information about the instance type you had and the new one you moved on?
Also, can you share the content of the file /opt/bitnami/apps/phabricator/tmp/phd/log/daemons.log it is too long so the bnsupport tool does not retrieve it.

scott3 2020-01-08 18:34:55 UTC #3

Hi @miguelaeh,

The restarts are manually triggered after I modified different settings in an attempt to get things working. We moved from a t2 small to a t2 medium.

I've pasted the end of the daemons.log below. It's all basically a repetition of this error. I've run the phabricator storage upgrade command (with no changes) with the user mentioned in the error log (bn_phabricator@localhost) and was able to successfully complete. So at least part of phabricator seems able to access the mysql tables.

Thanks for your help!

[07-Jan-2020 17:37:39 America/Los_Angeles] [2020-01-07 17:37:39] PHLOG: 'Retrying database connection to "localhost" after connection failure (attempt 1; "AphrontConnectionQueryException"; error #2002): Attempt to connect to bn_phabricator@127.0.0.1 failed with error #2002: Connection refused.' at [/opt/bitnami/apps/phabricator/htdocs/src/infrastructure/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:135]
[07-Jan-2020 17:37:39 America/Los_Angeles] [2020-01-07 17:37:39] PHLOG: 'Retrying database connection to "localhost" after connection failure (attempt 2; "AphrontConnectionQueryException"; error #2002): Attempt to connect to bn_phabricator@127.0.0.1 failed with error #2002: Connection refused.' at [/opt/bitnami/apps/phabricator/htdocs/src/infrastructure/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:135]
[07-Jan-2020 17:37:39 America/Los_Angeles] [2020-01-07 17:37:39] EXCEPTION: (AphrontConnectionQueryException) Attempt to connect to bn_phabricator@127.0.0.1 failed with error #2002: Connection refused. at [<phabricator>/src/infrastructure/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:374]
[07-Jan-2020 17:37:39 America/Los_Angeles] arcanist(head=stable, ref.stable=8b5fbb5a1346), phabricator(head=stable, ref.stable=9d644f1bfea1), phutil(head=stable, ref.stable=1750586fdc50)
[07-Jan-2020 17:37:39 America/Los_Angeles]   #0 <#2> AphrontBaseMySQLDatabaseConnection::throwConnectionException(integer, string, string, string) called at [<phabricator>/src/infrastructure/storage/connection/mysql/AphrontMySQLiDatabaseConnection.php:86]
[07-Jan-2020 17:37:39 America/Los_Angeles]   #1 <#2> AphrontMySQLiDatabaseConnection::connect() called at [<phabricator>/src/infrastructure/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:114]
[07-Jan-2020 17:37:39 America/Los_Angeles]   #2 <#2> AphrontBaseMySQLDatabaseConnection::establishConnection() called at [<phabricator>/src/infrastructure/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:152]
[07-Jan-2020 17:37:39 America/Los_Angeles]   #3 <#2> AphrontBaseMySQLDatabaseConnection::requireConnection() called at [<phabricator>/src/infrastructure/storage/connection/mysql/AphrontMySQLiDatabaseConnection.php:17]
[07-Jan-2020 17:37:39 America/Los_Angeles]   #4 <#2> AphrontMySQLiDatabaseConnection::escapeBinaryString(string) called at [<phabricator>/src/infrastructure/storage/connection/mysql/AphrontMySQLiDatabaseConnection.php:13]
[07-Jan-2020 17:37:39 America/Los_Angeles]   #5 <#2> AphrontMySQLiDatabaseConnection::escapeUTF8String(string) called at [<phabricator>/src/infrastructure/storage/xsprintf/qsprintf.php:266]
[07-Jan-2020 17:37:39 America/Los_Angeles]   #6 <#2> xsprintf_query(array, string, integer, string, integer) called at [<phutil>/src/xsprintf/xsprintf.php:70]
[07-Jan-2020 17:37:39 America/Los_Angeles]   #7 <#2> xsprintf(string, array, array) called at [<phabricator>/src/infrastructure/storage/xsprintf/PhutilQueryString.php:31]
[07-Jan-2020 17:37:39 America/Los_Angeles]   #8 <#2> PhutilQueryString::__construct(AphrontMySQLiDatabaseConnection, array) called at [<phabricator>/src/infrastructure/storage/xsprintf/qsprintf.php:78]
[07-Jan-2020 17:37:39 America/Los_Angeles]   #9 <#2> qsprintf(AphrontMySQLiDatabaseConnection, string, string, string) called at [<phabricator>/src/infrastructure/storage/lisk/LiskDAO.php:979]
[07-Jan-2020 17:37:39 America/Los_Angeles]   #10 <#2> LiskDAO::update() called at [<phabricator>/src/infrastructure/storage/lisk/LiskDAO.php:910]
[07-Jan-2020 17:37:39 America/Los_Angeles]   #11 <#2> LiskDAO::save() called at [<phabricator>/src/applications/daemon/event/PhabricatorDaemonEventListener.php:107]
[07-Jan-2020 17:37:39 America/Los_Angeles]   #12 <#2> PhabricatorDaemonEventListener::handleExitEvent(PhutilEvent) called at [<phabricator>/src/applications/daemon/event/PhabricatorDaemonEventListener.php:30]
[07-Jan-2020 17:37:39 America/Los_Angeles]   #13 <#2> PhabricatorDaemonEventListener::handleEvent(PhutilEvent) called at [<phutil>/src/events/PhutilEventEngine.php:65]
[07-Jan-2020 17:37:39 America/Los_Angeles]   #14 <#2> PhutilEventEngine::dispatchEvent(PhutilEvent) called at [<phutil>/src/daemon/PhutilDaemonHandle.php:322]
[07-Jan-2020 17:37:39 America/Los_Angeles]   #15 phlog(AphrontConnectionQueryException) called at [<phutil>/src/daemon/PhutilDaemonHandle.php:324]
[07-Jan-2020 17:37:39 America/Los_Angeles]   #16 PhutilDaemonHandle::dispatchEvent(string) called at [<phutil>/src/daemon/PhutilDaemonHandle.php:499]
[07-Jan-2020 17:37:39 America/Los_Angeles]   #17 PhutilDaemonHandle::didExit() called at [<phutil>/src/daemon/PhutilDaemonPool.php:193]
[07-Jan-2020 17:37:39 America/Los_Angeles]   #18 PhutilDaemonPool::updatePool() called at [<phutil>/src/daemon/PhutilDaemonOverseer.php:172]
[07-Jan-2020 17:37:39 America/Los_Angeles]   #19 PhutilDaemonOverseer::run() called at [<phabricator>/scripts/daemon/launch_daemon.php:24]
[07-Jan-2020 17:38:50 America/Los_Angeles] 2020-01-07 5:38:50 PM [OVER] Started new daemon overseer (with PID "1730").

scott3 2020-01-08 23:53:13 UTC #4

I was able to resolve this issue by setting the phd.user setting in phabricator to the 'daemon' user. Still sporadically getting errors like the following:

[08-Jan-2020 15:43:14 America/Los_Angeles] [2020-01-08 15:43:14] EXCEPTION: (PhutilProxyException) Error while updating the "rPDT" repository. {>} (CommandException) Command failed with error #255!
COMMAND
'/opt/bitnami/apps/phabricator/htdocs/bin/repository' update  -- 'rPDT'

STDOUT
(empty)

STDERR
[2020-01-08 15:43:14] EXCEPTION: (PhabricatorClusterStrandedException) Unable to establish a connection to any database host (while trying "bitnami_phabricator_config"). All masters and replicas are completely unreachable.

AphrontConnectionQueryException: Attempt to connect to bn_phabricator@127.0.0.1 failed with error #1040: Too many connections. at [<phabricator>/src/infrastructure/storage/lisk/PhabricatorLiskDAO.php:177]
arcanist(head=stable, ref.stable=8b5fbb5a1346), phabricator(head=stable, ref.stable=9d644f1bfea1), phutil(head=stable, ref.stable=1750586fdc50)
  #0 PhabricatorLiskDAO::raiseUnreachable(string, AphrontConnectionQueryException) called at [<phabricator>/src/infrastructure/storage/lisk/PhabricatorLiskDAO.php:134]
  #1 PhabricatorLiskDAO::newClusterConnection(string, string, string) called at [<phabricator>/src/infrastructure/storage/lisk/PhabricatorLiskDAO.php:72]
  #2 PhabricatorLiskDAO::establishLiveConnection(string) called at [<phabricator>/src/infrastructure/stor... (1,354 more bytes) ... at [<phutil>/src/future/exec/ExecFuture.php:380]
[08-Jan-2020 15:43:14 America/Los_Angeles] arcanist(head=stable, ref.stable=8b5fbb5a1346), phabricator(head=stable, ref.stable=9d644f1bfea1), phutil(head=stable, ref.stable=1750586fdc50)
[08-Jan-2020 15:43:14 America/Los_Angeles]   #0 <#3> ExecFuture::resolvex() called at [<phabricator>/src/applications/repository/daemon/PhabricatorRepositoryPullLocalDaemon.php:480]
[08-Jan-2020 15:43:14 America/Los_Angeles]   #1 phlog(PhutilProxyException) called at [<phabricator>/src/applications/repository/daemon/PhabricatorRepositoryPullLocalDaemon.php:487]
[08-Jan-2020 15:43:14 America/Los_Angeles]   #2 PhabricatorRepositoryPullLocalDaemon::resolveUpdateFuture(PhabricatorRepository, ExecFuture, integer) called at [<phabricator>/src/applications/repository/daemon/PhabricatorRepositoryPullLocalDaemon.php:256]
[08-Jan-2020 15:43:14 America/Los_Angeles]   #3 PhabricatorRepositoryPullLocalDaemon::run() called at [<phutil>/src/daemon/PhutilDaemon.php:219]
[08-Jan-2020 15:43:14 America/Los_Angeles]   #4 PhutilDaemon::execute() called at [<phutil>/scripts/daemon/exec/exec_daemon.php:131]
[08-Jan-2020 15:43:19 America/Los_Angeles] [2020-01-08 15:43:19] EXCEPTION: (PhabricatorClusterImpossibleWriteException) Unable to connect to master database ("bitnami_phabricator_drydock"). This is a severe failure; your request did not complete. at [<phabricator>/src/infrastructure/storage/lisk/PhabricatorLiskDAO.php:147]
[08-Jan-2020 15:43:19 America/Los_Angeles] arcanist(head=stable, ref.stable=8b5fbb5a1346), phabricator(head=stable, ref.stable=9d644f1bfea1), phutil(head=stable, ref.stable=1750586fdc50)
[08-Jan-2020 15:43:19 America/Los_Angeles]   #0 PhabricatorLiskDAO::raiseImpossibleWrite(string) called at [<phabricator>/src/infrastructure/storage/lisk/PhabricatorLiskDAO.php:97]
[08-Jan-2020 15:43:19 America/Los_Angeles]   #1 PhabricatorLiskDAO::newClusterConnection(string, string, string) called at [<phabricator>/src/infrastructure/storage/lisk/PhabricatorLiskDAO.php:72]
[08-Jan-2020 15:43:19 America/Los_Angeles]   #2 PhabricatorLiskDAO::establishLiveConnection(string) called at [<phabricator>/src/infrastructure/storage/lisk/LiskDAO.php:841]
[08-Jan-2020 15:43:19 America/Los_Angeles]   #3 LiskDAO::establishConnection(string) called at [<phabricator>/src/applications/drydock/garbagecollector/DrydockLogGarbageCollector.php:18]
[08-Jan-2020 15:43:19 America/Los_Angeles]   #4 DrydockLogGarbageCollector::collectGarbage() called at [<phabricator>/src/infrastructure/daemon/garbagecollector/PhabricatorGarbageCollector.php:115]
[08-Jan-2020 15:43:19 America/Los_Angeles]   #5 PhabricatorGarbageCollector::runCollector() called at [<phabricator>/src/infrastructure/daemon/workers/PhabricatorTriggerDaemon.php:373]
[08-Jan-2020 15:43:19 America/Los_Angeles]   #6 PhabricatorTriggerDaemon::updateGarbageCollection() called at [<phabricator>/src/infrastructure/daemon/workers/PhabricatorTriggerDaemon.php:339]
[08-Jan-2020 15:43:19 America/Los_Angeles]   #7 PhabricatorTriggerDaemon::runGarbageCollection(integer) called at [<phabricator>/src/infrastructure/daemon/workers/PhabricatorTriggerDaemon.php:109]
[08-Jan-2020 15:43:19 America/Los_Angeles]   #8 PhabricatorTriggerDaemon::run() called at [<phutil>/src/daemon/PhutilDaemon.php:219]
[08-Jan-2020 15:43:19 America/Los_Angeles]   #9 PhutilDaemon::execute() called at [<phutil>/scripts/daemon/exec/exec_daemon.php:131]

miguelaeh 2020-01-09 09:31:14 UTC #5

Hi @scott3,
That error seems to be that the MySQL server reached the maximum number of connections allowed. That's probably why you are getting it sporadically. To solve that you can try yo increase the maximum number of connections.
Connect to MySQL and get the current value:

mysql –u root –p
SHOW STATUS LIKE 'max_used_connections';

Now to set a new value modify the /opt/bitnami/mysql/my.cnf file and add the line:

max_connections = 512

Changing the 512 by your custom value.

scott3 2020-01-09 18:49:44 UTC #6

@miguelaeh turns out that when phd was restarting, the previous daemons weren't actually taken down. I had to manually kill daemons running under the 'bitnami' user and also duplicate daemons under the 'daemon' user. Doing so resolved all the errors. Thanks so much for your help.

miguelaeh 2020-01-10 10:12:28 UTC #7

Thank you very much for share your solution @scott3!!!

miguelaeh 2020-01-10 10:13:04 UTC #8

Bitnami provides free all-in-one installers, virtual machines and cloud images for popular open source applications. Get them now!