Notification Servers - Possible node issues with stack

I am runinng the Bitnami Phabricator stack on AWS EC2, S3, and RDS. I want to get Conpherence (the chat app) up and running so users don’t need to refresh the page in order to see new msgs.

Following this documentation
https://secure.phabricator.com/book/phabricator/article/notifications/

running the following
/opt/bitnami/app/phabricator/htdocs $ cd support/aphlict/server/
/opt/bitnami/app/phabricator/htdocs $ npm install ws

then running
/opt/bitnami/app/phabricator/htdocs/ $ ./bin/aphlict start --config /opt/bitnami/apps/phabricator/htdocs/conf/aphlict/aphlict.custom.json

I got

[2017-03-01 10:10:01] EXCEPTION: (FilesystemException) Requested path ‘/var/tmp/aphlict/pid’ is not writable.

so I made the pid file writable, ran the command again and got this.

bitnami@ip-172-31-29-130:/opt/bitnami/apps/phabricator/htdocs$ ./bin/aphlict start --config /opt/bitnami/apps/phabricator/htdocs/conf/aphlict/aphlict.custom.json
Reading configuration from: /opt/bitnami/apps/phabricator/htdocs/conf/aphlict/aphlict.custom.json
[2017-03-01 10:43:44] EXCEPTION: (CommandException) Command failed with error #1!
COMMAND
nodejs ‘–max-old-space-size=256’ – ‘/opt/bitnami/apps/phabricator/htdocs/support/aphlict/server/aphlict_server.js’ ‘–config=/opt/bitnami/apps/phabricator/htdocs/conf/aphlict/aphlict.custom.json’ ‘–test=true’

STDOUT
[Wed Mar 01 2017 18:43:44 GMT+0000 (UTC)]
<<< UNCAUGHT EXCEPTION! >>>

Error: You need to install the Node.js “ws” module for websocket support. See “Notifications User Guide: Setup and Configuration” in the documentation for instructions. SyntaxError: Use of const in strict mode.
at Object. (/opt/bitnami/apps/phabricator/htdocs/support/aphlict/server/aphlict_server.js:73:9)
at Module._compile (module.js:456:26)
at Object.Module._extensions…js (module.js:474:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)
at Function.Module.runMain (module.js:497:10)
at startup (node.js:119:16)
at node.js:902:3

STDERR
nodejs: /opt/bitnami/common/lib/libcrypto.so.1.0.0: no version information available (required by nodejs)
nodejs: /opt/bitnami/common/lib/libssl.so.1.0.0: no version information available (required by nodejs)
nodejs: /opt/bitnami/common/lib/libssl.so.1.0.0: no version information available (required by nodejs)

/opt/bitnami/apps/phabricator/htdocs/support/aphlict/server/node_modules/ws/index.js:9
const WebSocket = require(’./lib/WebSocket’);
^^^^^
at [/src/future/exec/ExecFuture.php:369]
arcanist(head=stable, ref.stable=9503b941cc02), phabricator(head=stable, ref.stable=0426ce73f0e6), phutil(head=stable, ref.stable=10963f771f11)
#0 ExecFuture::resolvex() called at [/src/future/exec/execx.php:17]
#1 execx(string, PhutilCommandString) called at [/src/applications/aphlict/management/PhabricatorAphlictManagementWorkflow.php:377]
#2 PhabricatorAphlictManagementWorkflow::willLaunch() called at [/src/applications/aphlict/management/PhabricatorAphlictManagementWorkflow.php:442]
#3 PhabricatorAphlictManagementWorkflow::executeStartCommand() called at [/src/applications/aphlict/management/PhabricatorAphlictManagementStartWorkflow.php:15]
#4 PhabricatorAphlictManagementStartWorkflow::execute(PhutilArgumentParser) called at [/src/parser/argument/PhutilArgumentParser.php:438]
#5 PhutilArgumentParser::parseWorkflowsFull(array) called at [/src/parser/argument/PhutilArgumentParser.php:330]
#6 PhutilArgumentParser::parseWorkflows(array) called at [/support/aphlict/server/aphlict_launcher.php:23]

which after some research led me to upgrade me node-js version. Sadly after doing so the error msg did not change.

Not sure what the next course of action should be. I don’t really want to go mucking around in Phabricator PHP files for fear I’ll break something. I wonder if I need to update

nodejs: /opt/bitnami/common/lib/libcrypto.so.1.0.0: no version information available (required by nodejs)
nodejs: /opt/bitnami/common/lib/libssl.so.1.0.0: no version information available (required by nodejs)
nodejs: /opt/bitnami/common/lib/libssl.so.1.0.0: no version information available (required by nodejs)

these files?

Any ideas appriciated, I want to get this working for our production build next month.

Hi,

The issue is that the system is unable to find ws in the installed location. Could you try executing the following?

npm install -g ws

This way the package should be found by the system.

Please let us know if this solves the issue.

Best regards,

Javier J. Salmerón

Thanks for the quick response.

I ran

npm install -g ws

as requested, but got the same output.

bitnami@ip-172-31-19-68:/opt/bitnami/apps/phabricator/htdocs$ ./bin/aphlict start --config /opt/bitnami/apps/phabricator/htdocs/conf/aphlict/aphlict.custom.json
Reading configuration from: /opt/bitnami/apps/phabricator/htdocs/conf/aphlict/aphlict.custom.json
[2017-03-02 16:12:16] EXCEPTION: (FilesystemException) Requested path ‘/var/tmp/aphlict/pid’ is not writable. at [/src/filesystem/Filesystem.php:1107]
arcanist(head=stable, ref.stable=822bc53ca306), phabricator(head=stable, ref.stable=216f6be11ece), phutil(head=stable, ref.stable=76983f49b179)
#0 Filesystem::assertWritable(string) called at [/src/filesystem/Filesystem.php:73]
#1 Filesystem::assertWritableFile(string) called at [/src/filesystem/Filesystem.php:89]
#2 Filesystem::writeFile(string, string) called at [/src/applications/aphlict/management/PhabricatorAphlictManagementWorkflow.php:367]
#3 PhabricatorAphlictManagementWorkflow::willLaunch() called at [/src/applications/aphlict/management/PhabricatorAphlictManagementWorkflow.php:442]
#4 PhabricatorAphlictManagementWorkflow::executeStartCommand() called at [/src/applications/aphlict/management/PhabricatorAphlictManagementStartWorkflow.php:15]
#5 PhabricatorAphlictManagementStartWorkflow::execute(PhutilArgumentParser) called at [/src/parser/argument/PhutilArgumentParser.php:441]
#6 PhutilArgumentParser::parseWorkflowsFull(array) called at [/src/parser/argument/PhutilArgumentParser.php:333]
#7 PhutilArgumentParser::parseWorkflows(array) called at [/support/aphlict/server/aphlict_launcher.php:23]
bitnami@ip-172-31-19-68:/opt/bitnami/apps/phabricator/htdocs$ sudo chmod go+w /var/tmp/aphlict/pid
bitnami@ip-172-31-19-68:/opt/bitnami/apps/phabricator/htdocs$ ./bin/aphlict start --config /opt/bitnami/apps/phabricator/htdocs/conf/aphlict/aphlict.custom.json
Reading configuration from: /opt/bitnami/apps/phabricator/htdocs/conf/aphlict/aphlict.custom.json
[2017-03-02 16:14:12] EXCEPTION: (CommandException) Command failed with error #1!
COMMAND
nodejs ‘–max-old-space-size=256’ – ‘/opt/bitnami/apps/phabricator/htdocs/support/aphlict/server/aphlict_server.js’ ‘–config=/opt/bitnami/apps/phabricator/htdocs/conf/aphlict/aphlict.custom.json’ ‘–test=true’

STDOUT
[Fri Mar 03 2017 00:14:12 GMT+0000 (UTC)]
<<< UNCAUGHT EXCEPTION! >>>

Error: You need to install the Node.js “ws” module for websocket support. See “Notifications User Guide: Setup and Configuration” in the documentation for instructions. Error: Cannot find module ‘ws’
at Object. (/opt/bitnami/apps/phabricator/htdocs/support/aphlict/server/aphlict_server.js:73:9)
at Module._compile (module.js:456:26)
at Object.Module._extensions…js (module.js:474:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)
at Function.Module.runMain (module.js:497:10)
at startup (node.js:119:16)
at node.js:902:3

STDERR
nodejs: /opt/bitnami/common/lib/libcrypto.so.1.0.0: no version information available (required by nodejs)
nodejs: /opt/bitnami/common/lib/libssl.so.1.0.0: no version information available (required by nodejs)
nodejs: /opt/bitnami/common/lib/libssl.so.1.0.0: no version information available (required by nodejs)
at [/src/future/exec/ExecFuture.php:369]
arcanist(head=stable, ref.stable=822bc53ca306), phabricator(head=stable, ref.stable=216f6be11ece), phutil(head=stable, ref.stable=76983f49b179)
#0 ExecFuture::resolvex() called at [/src/future/exec/execx.php:17]
#1 execx(string, PhutilCommandString) called at [/src/applications/aphlict/management/PhabricatorAphlictManagementWorkflow.php:377]
#2 PhabricatorAphlictManagementWorkflow::willLaunch() called at [/src/applications/aphlict/management/PhabricatorAphlictManagementWorkflow.php:442]
#3 PhabricatorAphlictManagementWorkflow::executeStartCommand() called at [/src/applications/aphlict/management/PhabricatorAphlictManagementStartWorkflow.php:15]
#4 PhabricatorAphlictManagementStartWorkflow::execute(PhutilArgumentParser) called at [/src/parser/argument/PhutilArgumentParser.php:441]
#5 PhutilArgumentParser::parseWorkflowsFull(array) called at [/src/parser/argument/PhutilArgumentParser.php:333]
#6 PhutilArgumentParser::parseWorkflows(array) called at [/support/aphlict/server/aphlict_launcher.php:23]

I expected this to be the case based off of the explict path in the error msg at the start of the output.

Error: You need to install the Node.js “ws” module for websocket support. See “Notifications User Guide: Setup and Configuration” in the documentation for instructions. Error: Cannot find module ‘ws’
at Object. (/opt/bitnami/apps/phabricator/htdocs/support/aphlict/server/aphlict_server.js:73:9)

Looking at line 73 of aphlict_server.js

try {
  require('ws');
} catch (ex) {
  throw new Error(
    'You need to install the Node.js "ws" module for websocket support. ' +
    'See "Notifications User Guide: Setup and Configuration" in the ' +
    'documentation for instructions. ' + ex.toString());
}

// NOTE: Require these only after checking for the "ws" module, since they
// depend on it.

require('./lib/AphlictAdminServer');
require('./lib/AphlictClientServer');
require('./lib/AphlictPeerList');
require('./lib/AphlictPeer');

The non-explicit path of the ws require makes me think that is should be able to reach the global node install but the requires below make me think that the ws include is only looking for ws in /opt/bitnami/apps/phabricator/htdocs/support/aphlict/server/ as that is the directory where aphlict_server.js lives. Then again because

require(‘ws’);

lacks a starting “./” or “/” to make it an explicit absolute or relative path I am just left unsure as to where it is actually looking.

When I run

npm install ws

at

/opt/bitnami/apps/phabricator/htdocs/support/aphlict/server/

ws is installed at

/opt/bitnami/apps/phabricator/htdocs/support/aphlict/server/node_modules/

as I would expect. Is it possible somehow aphlict_server.js expects ws to be installed in

/opt/bitnami/apps/phabricator/htdocs/support/aphlict/server/

directly without the node_modules directory? I think that will be my next approach unless I hear/think of a better idea before tomorrow morning when I get back at this.

Hi,

Ok, you said that you executed npm install -g ws, therefore, you should be able to execute the following in a node console (execute node):

require('ws');

If this does not work, try installing ws as root by executing:

sudo npm install -g ws

Then try again and, if require(ws) works, then you can try executing aphlict again.

Hope it helps.

Best regards,

Javier J. Salmerón

This didn’t seem to work either but I haven’t had time to really dig into and get any new useful info. Will update when I do.

So coming back to this topic (I’ve been working on setting up some other features/extensions). With a little further digging with a fresh mind I figured this may have been a node versioning issue.

Running: curl -sL https://deb.nodesource.com/setup_6.x | sudo -E bash -
before sudo apt-get install -y nodejs

fixed the actual ws installation. Now however I have a rather strange situation

[4/11/2017, 6:28:05 PM] Starting servers (service PID 2502).
[4/11/2017, 6:28:05 PM] Logging to "/var/log/aphlict.log".
[4/11/2017, 6:28:05 PM] Started client server (Port 22280, No SSL).
[4/11/2017, 6:28:05 PM] Started admin server (Port 22281, No SSL).
[4/11/2017, 6:28:05 PM] This server has fingerprint "J6EsuVYaTnekUF6r".
[4/11/2017, 6:40:21 PM] Received notification (default): {"type":"message","threadPHID":"PHID-CONP-h5m5nksksvsefsebkq7d","messageID":22,"subscribers":["PHID-CONP-h5m5nksksvsefsebkq7d"]}
[4/11/2017, 6:40:36 PM] Received notification (default): {"type":"message","threadPHID":"PHID-CONP-h5m5nksksvsefsebkq7d","messageID":23,"subscribers":["PHID-CONP-h5m5nksksvsefsebkq7d"]}

This is the output of /opt/bitnami/apps/phabricator/htdocs$ ./bin/aphlict debug --config /opt/bitnami/apps/phabricator/htdocs/conf/aphlict/aphlict.custom.json

This debug output looks fine to me, however, visiting Config > Notification Servers shows the server as connected but with no stat values such as the admin server shows such as UpTime/Clients/Messages (first image below shows this).


The first chat image is from my perspective, the second is from the perspective of the TestContributor account perspective. There is a constant “Connecting…” at the bottom of the page for both. Watching the debug trace run for Aphlict I can see the messages get logged as soon as they are sent so they are definitely making it to the server.

Now my only real guess is that it has something to do with allowed URI’s or SSL. Phabricator notification servers can be configured to run under SSL. Since my instance is on an EC2 instance behind a application layer load balancer (which only supports HTTP/HTTPS on AWS) I can’t list my phabricator instance’s domain as the “host” in the Phabricator config setting notification.servers as the load balancer won’t forward traffic on port 22280 where WS is listening. So I instead experimented switching to use http instead of https and trying to use a the public DNS URL defined by the EC2 instance’s public IPV4 address. Which is how I got to my current state. Not expecting a quick solution but I felt I should update this thread on progress and issues and see if anyone had any ideas.

Hi,

I was able to make real-time notifications work. What I did was the following:

  • Install nodejs0
  • Install ws globally
sudo npm install -g ws
  • I started aphlict
./bin/aphlict start --config conf/aphlict/aphlict.default.json 

(NOTE: I did not change anything in the default configuration)

  • I modified the notification.settings configuration
[ { "type": "client", "host": "<IPOFMYSERVER>", "port": 22280, "protocol": "http" }, { "type": "admin", "host": "127.0.0.1", "port": 22281, "protocol": "http" } ]
  • I opened the port 22280 of my cloud provider firewall

And that was it. I guess that if you want to use https you will have to use a proper SSL certificate in your server.

Hope it helps.

Best regards,

Javier J. Salmerón

So what this came down to in the long run was AWS application load balancers do not support port forwarding for anything that is not HTTP/HTTPS (limiting all through traffic to ports 80/443 essentially. Because my SSL cert is on the ELB and SSL is terminated at the ELB the client HTTPS protocol on 22280 that I was trying to have bypass the ELB wasn’t having it (this was a dumb, bad, and naive attempt at a solution anyways).

Now I am using a AWS classic load balancer which allows me to properly forward ports and it worked with the default config.

1 Like

Hi,

Thanks for letting us know and for sharing the solution with the community. We are glad it worked for you. If you have any other question, please don’t hesitate to ask.

Best regards,
Alvaro Recio