Redmine REST API: POST requests failing with error 502

ant_222 2020-12-01 16:38:22 UTC #1

Hello, all. I have Bitnami Redmine stack v. 4.1.1-5 on Windows Server 2012 R2, and the REST API functions one-way only: although GET requests work well, all POST requests fail with error 502:

Code: 502
Title: Proxy Error

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request.

Reason: Error reading from remote server.

The Apach error log has two entry for each failed request:

[Tue Dec 01 19:33:39.439905 2020] [proxy_http:error] [pid 7108:tid 1068] (20014)Internal error (specific information not available): [client 127.0.0.1:52231] AH01102: error reading status line from remote server 127.0.0.1:3002
[Tue Dec 01 19:33:39.439905 2020] [proxy:error] [pid 7108:tid 1068] [client 127.0.0.1:52231] AH00898: Error reading from remote server returned by /redmine/issues.json

I did not alter the Apache configuration. Have you an idea what may be causing it?

gongomgra 2020-12-02 09:31:10 UTC #2

Hi @ant_222,

It looks like your question is very specific to the Redmine application. Can you open a new thread in their official forum?

https://www.redmine.org/projects/redmine/boards

Apart from that, we have a Support Tool that will gather relevant information for us to analyze your configuration and logs. Could you please execute it on the machine where the stack is running by following the steps described in the guide below?

How to Run the Bitnami Support Tool when using a native installer

Please note that you need to paste the code ID that is shown at the end.

ant_222 2020-12-04 10:36:01 UTC #3

gongomgra,
We have run the Support Tool on the machine with the affected Redmine stack, and it wrote it had successfully uploaded a support bundle with the id:

262bd8bf-8420-3c58-589b-ca572c1370c7

I will soon write to Redmine support.

ant_222 2020-12-04 11:14:24 UTC #4

I wrote:

I will soon write to Redmine support.

Done: REST API: GET working but POST failing with error 502.

By the way, is it possible to edit a post here? The pencil icon to right of the thread title let me change the title only...

michiel 2020-12-07 09:36:20 UTC #5

Hi @ant_222,

Unfortunately not. But we can edit your post if needed.

Regards,
Michiel

ant_222 2020-12-07 11:41:01 UTC #6

@michiel,
I only wanted to fix an unimportant typo and make another small improvement, so I will not bother you.

Since I do not seem to be getting any help from the community (which may be my fault), may I have commercial support from Bitnami with this problem on the time & material basis?

ant_222 2020-12-07 12:23:59 UTC #7

Update

With the :debug log level, successful GET requests appear in production.log, but the POST requests failing with error 502 do not. Does it mean Redmine does not even receive them, and I have to troubleshoot Apache?

Ibone 2020-12-08 11:49:27 UTC #8

Hi @ant_222,

Could you rerun the bnsupport?

Regards,
Ibone.

ant_222 2020-12-08 13:58:41 UTC #9

Here you are, @Ibone:

d1145d4b-4f21-7ad7-105e-4d4ca516193

ant_222 2020-12-08 14:04:37 UTC #10

FYI: I have also asked about the same problem on the Apache mailing list:

Date:    2020-12-3
Subject: GET working but POST failing with error 502

The archive for December is available here. I have not found a way to link a specific thread, but it is easily locatable. The people at Apache suggest that Redmine crashes upon receipt of a POST request. They also suspect a problem with connection reuse between Redmine and httpd.

gongomgra 2020-12-09 10:01:18 UTC #11

Hi @ant_222,

Thanks for keep investigating this. Can you check if you reproduce the issue if you run the POST request against 127.0.0.1:3001 instead of Apache? Port 3001 runs the Redmine web app (using a Thin server. Apache is only a proxy), and that test should give us more information about if the issue is on Apache's or Redmine's config.

ant_222 2020-12-09 10:41:44 UTC #12

@gongomgra,

Hey, with port 3001 it works!

ant_222 2020-12-09 17:56:50 UTC #13

Yann Ylavic from the httpd mailing list has solved this problem for me. He told me to enabled detailed logging in Appache by adding

LogLevel trace7

C:\Bitnami\redmine-4.1.1-5\apps\redmine\conf\httpd-prefix.conf

and try a POST request again. The Apache error log contained this line:

Expect: 100-continue

Yann told me to add

Proxy100Continue off

to httpd-prefix.conf, which fixed this erro 502.

ant_222 2020-12-09 19:00:02 UTC #14

I forgot to provide a reference to the discussion of this problem on the Apache mailing list.

ant_222 2020-12-10 12:41:05 UTC #15

Is it not a case when the Bitnami one-click installer has failed to work out of the box? Or is it the fault of the REST client, which must use the expect-continue pattern?

gongomgra 2020-12-10 12:59:24 UTC #16

Hi @ant_222,

Thanks for providing a solution to your issue! We will add your solution to our Redmine installers and release a new revision with the fix.

About the expect-continue pattern you mention, I'm afraid I don't know what is the issue, and if it is an issue on Apache itself or not.

ant_222 2020-12-10 14:16:50 UTC #17

@gongomgra:

About the expect-continue pattern you mention, I'm afraid I don't know what is the issue, and if it is an issue on Apache itself or not.

How can you to implement the fix if you don't know about the feature of HTTP connected with it? Surely you will need a specialist with a thorough understanding of both HTTP and Apache to do that. Otherwise you risk to do more harm that good.

Wikipedia has the following to say about this feature:

The server has received the request headers and the client should proceed to send the request body (in the case of a request for which a body needs to be sent; for example, a POST request). Sending a large request body to a server after a request has been rejected for inappropriate headers would be inefficient. To have a server check the request's headers, a client must send Expect: 100-continue as a header in its initial request and receive a 100 Continue status code in response before sending the body.

It being is a useful method of bandwidth economy, disabling it is probably not the best solution. Furthermore, will it not break clients that already rely on it?

gongomgra 2020-12-11 09:56:54 UTC #18

Hi @ant_222,

Yes, you are right. It is possible that this solution is only related for your API client. Can you tell us which client were you using so our Engineering team can reproduce it and test other clients?

Regards,
Gonzalo

ant_222 2020-12-11 10:12:30 UTC #19

@gongomgra, we are using the semi-official redmine-net-api. Can the error be caused by something is the configuration of our local network?

ant_222 2020-12-13 20:44:25 UTC #20

I had the same error while using my own primitive test program written in plain .NET. Will it help if I post its source (which you won't like :-)? Are you still planning to investigate this problem?

Bitnami provides free all-in-one installers, virtual machines and cloud images for popular open source applications. Get them now!