Jenkins stack shows the "Your application is not available" page

Keywords: Jenkins - Windows - Technical issue - Other
Description:
This problem has occurred several times but I have not been able to track it down to a specific action or issue. Here is the sequence of events that I know about:

  • Started Jenkins stack and the application appeared to work normally. All Jenkins pages loaded successfully and builds seemed to run as configured
  • The application ran for several hours without errors
  • At some point the server began showing the Bitnami “Your application is not available” page instead of loading Jenkins pages
  • I started manager-windows.exe and saw that Apache was running but Tomcat-server and Gradle were stopped

The support tool bnsupport-0.8.2-windows.exe crashed with “Error writing file C:/Users/…/permissions/letsencrypt.permissions”.

image

Wasn’t able to to upload the files that were generated prior to bnsupport crash so here they are in-text:

commands.output:


Crontab content for root user:

Running: crontab -l
In: C:/Bitnami/jenkins-2.204.1-0
Error: unknown variable program_stderr
Exitcode: unknown variable program_exit_code


Crontab content for bitnami user:

Running: crontab -l -u bitnami
In: C:/Bitnami/jenkins-2.204.1-0
Error: unknown variable program_stderr
Exitcode: unknown variable program_exit_code


Machine hostname:

Running: echo
In: C:/Bitnami/jenkins-2.204.1-0
Error: unknown variable program_stderr
Exitcode: unknown variable program_exit_code


Check Apache path:

Running: ps aux | grep ‘apache|mysql|nginx|C:/Bitnami/jenkins-2.204.1-0’
In: C:/Bitnami/jenkins-2.204.1-0
Error: unknown variable program_stderr
Exitcode: unknown variable program_exit_code


Check if port 80 is externally accessible:

Running: nc -zv -w5 80
In: C:/Bitnami/jenkins-2.204.1-0
Error: unknown variable program_stderr
Exitcode: unknown variable program_exit_code


Check if port 443 is externally accessible:

Running: nc -zv -w5 443
In: C:/Bitnami/jenkins-2.204.1-0
Error: unknown variable program_stderr
Exitcode: unknown variable program_exit_code


Check if lego is installed and which version:

Running: which lego
In: C:/Bitnami/jenkins-2.204.1-0/
Error: unknown variable program_stderr
Exitcode: unknown variable program_exit_code


Check lego version:

Running: which lego --version
In: C:/Bitnami/jenkins-2.204.1-0
Error: unknown variable program_stderr
Exitcode: unknown variable program_exit_code

ssl-checker.exe.log:
Active Certificate check failed: “CreateFile C:\Bitnami\jenkins-2.204.1-0/apache2/conf/extra/httpd-default.conf\r: The filename, directory name, or volume label syntax is incorrect.”
2020/01/20 15:15:57 Found errors when checking the SSL configuration======================================
SSL CHECKS

Starting checks with these parameters:

  • Apache Root: “C:\Bitnami\jenkins-2.204.1-0/apache2”
  • Apache Root configuration: “C:\Bitnami\jenkins-2.204.1-0/apache2/conf/httpd.conf”
  • Hostname: “JENKINS”
  • Port: 443
    ======================================
    – Check: Active SSL Certificates in Apache Configuration –
    – End of check –

– Check: HTTPS Connection to web server –
Hostname: “JENKINS”
Port: 443
Server certificate domain: “www.example.com
– End of check –

SSL Checks finishedActive Certificate check failed: “CreateFile C:\Bitnami\jenkins-2.204.1-0/apache2/conf/extra/httpd-default.conf\r: The filename, directory name, or volume label syntax is incorrect.”
2020/01/20 15:16:27 Found errors when checking the SSL configuration======================================
SSL CHECKS

Starting checks with these parameters:

  • Apache Root: “C:\Bitnami\jenkins-2.204.1-0/apache2”
  • Apache Root configuration: “C:\Bitnami\jenkins-2.204.1-0/apache2/conf/httpd.conf”
  • Hostname: “JENKINS”
  • Port: 443
    ======================================
    – Check: Active SSL Certificates in Apache Configuration –
    – End of check –

– Check: HTTPS Connection to web server –
Hostname: “JENKINS”
Port: 443
Server certificate domain: “www.example.com
– End of check –

SSL Checks finishedActive Certificate check failed: “CreateFile C:\Bitnami\jenkins-2.204.1-0/apache2/conf/extra/httpd-default.conf\r: The filename, directory name, or volume label syntax is incorrect.”
2020/01/20 15:34:38 Found errors when checking the SSL configuration======================================
SSL CHECKS

Starting checks with these parameters:

  • Apache Root: “C:\Bitnami\jenkins-2.204.1-0/apache2”
  • Apache Root configuration: “C:\Bitnami\jenkins-2.204.1-0/apache2/conf/httpd.conf”
  • Hostname: “JENKINS”
  • Port: 443
    ======================================
    – Check: Active SSL Certificates in Apache Configuration –
    – End of check –

– Check: HTTPS Connection to web server –
Hostname: “JENKINS”
Port: 443
Server certificate domain: “www.example.com
– End of check –

SSL Checks finishedActive Certificate check failed: “CreateFile C:\Bitnami\jenkins-2.204.1-0/apache2/conf/extra/httpd-default.conf\r: The filename, directory name, or volume label syntax is incorrect.”
2020/01/20 15:35:51 Found errors when checking the SSL configuration======================================
SSL CHECKS

Starting checks with these parameters:

  • Apache Root: “C:\Bitnami\jenkins-2.204.1-0/apache2”
  • Apache Root configuration: “C:\Bitnami\jenkins-2.204.1-0/apache2/conf/httpd.conf”
  • Hostname: “JENKINS”
  • Port: 443
    ======================================
    – Check: Active SSL Certificates in Apache Configuration –
    – End of check –

– Check: HTTPS Connection to web server –
Hostname: “JENKINS”
Port: 443
Server certificate domain: “www.example.com
– End of check –

SSL Checks finishedActive Certificate check failed: “CreateFile C:\Bitnami\jenkins-2.204.1-0/apache2/conf/extra/httpd-default.conf\r: The filename, directory name, or volume label syntax is incorrect.”
HTTPS Connection failed: “dial tcp [fe80::4c2b:b321:bbc3:2261%Ethernet]:443: connectex: No connection could be made because the target machine actively refused it.”
2020/01/20 15:39:59 Found errors when checking the SSL configuration======================================
SSL CHECKS

Starting checks with these parameters:

  • Apache Root: “C:\Bitnami\jenkins-2.204.1-0/apache2”
  • Apache Root configuration: “C:\Bitnami\jenkins-2.204.1-0/apache2/conf/httpd.conf”
  • Hostname: “JENKINS”
  • Port: 443
    ======================================
    – Check: Active SSL Certificates in Apache Configuration –
    – End of check –

– Check: HTTPS Connection to web server –
Hostname: “JENKINS”
Port: 443
– End of check –

SSL Checks finishedActive Certificate check failed: “CreateFile C:\Bitnami\jenkins-2.204.1-0/apache2/conf/extra/httpd-default.conf\r: The filename, directory name, or volume label syntax is incorrect.”
HTTPS Connection failed: “dial tcp [fe80::4c2b:b321:bbc3:2261%Ethernet]:443: connectex: No connection could be made because the target machine actively refused it.”
2020/01/20 15:40:58 Found errors when checking the SSL configuration======================================
SSL CHECKS

Starting checks with these parameters:

  • Apache Root: “C:\Bitnami\jenkins-2.204.1-0/apache2”
  • Apache Root configuration: “C:\Bitnami\jenkins-2.204.1-0/apache2/conf/httpd.conf”
  • Hostname: “JENKINS”
  • Port: 443
    ======================================
    – Check: Active SSL Certificates in Apache Configuration –
    – End of check –

– Check: HTTPS Connection to web server –
Hostname: “JENKINS”
Port: 443
– End of check –

SSL Checks finishedActive Certificate check failed: “CreateFile C:\Bitnami\jenkins-2.204.1-0/apache2/conf/extra/httpd-default.conf\r: The filename, directory name, or volume label syntax is incorrect.”
HTTPS Connection failed: “dial tcp [fe80::4c2b:b321:bbc3:2261%Ethernet]:443: connectex: No connection could be made because the target machine actively refused it.”
2020/01/20 15:42:36 Found errors when checking the SSL configuration======================================
SSL CHECKS

Starting checks with these parameters:

  • Apache Root: “C:\Bitnami\jenkins-2.204.1-0/apache2”
  • Apache Root configuration: “C:\Bitnami\jenkins-2.204.1-0/apache2/conf/httpd.conf”
  • Hostname: “JENKINS”
  • Port: 443
    ======================================
    – Check: Active SSL Certificates in Apache Configuration –
    – End of check –

– Check: HTTPS Connection to web server –
Hostname: “JENKINS”
Port: 443
– End of check –

SSL Checks finishedActive Certificate check failed: “CreateFile C:\Bitnami\jenkins-2.204.1-0/apache2/conf/extra/httpd-default.conf\r: The filename, directory name, or volume label syntax is incorrect.”
HTTPS Connection failed: “dial tcp [fe80::4c2b:b321:bbc3:2261%Ethernet]:443: connectex: No connection could be made because the target machine actively refused it.”
2020/01/20 15:47:25 Found errors when checking the SSL configuration======================================
SSL CHECKS

Starting checks with these parameters:

  • Apache Root: “C:\Bitnami\jenkins-2.204.1-0/apache2”
  • Apache Root configuration: “C:\Bitnami\jenkins-2.204.1-0/apache2/conf/httpd.conf”
  • Hostname: “JENKINS”
  • Port: 443
    ======================================
    – Check: Active SSL Certificates in Apache Configuration –
    – End of check –

– Check: HTTPS Connection to web server –
Hostname: “JENKINS”
Port: 443
– End of check –

SSL Checks finished

system_info.properties:

bnsupport_version = 0.8.2
installdir = C:/Bitnami/jenkins-2.204.1-0
base_stack_key = jenkins
base_stack_name = Bitnami Jenkins Stack
base_stack_version = 2.204.1-0
base_stack_platform = windows-x64
system_locale = en
system_username = vmt
user_home_directory = C:/Users/vmt
machine_free_disk_space = 603366 MB
machine_cpu_count = 8
machine_cpu_speed = 3392 MHz
machine_fqdn = JENKINS
machine_hostname = JENKINS
machine_ipaddr = 192.168.10.175
machine_swap_memory = 1856 MB
machine_total_memory = 12269 MB
platform_has_smp = 0
os_family = windows
windows_os_version = 10.0
windows_os_family = Windows NT
windows_os_name = Windows 10
windows_flavor = Professional
windows_service_pack = 0
windows_uac_enabled = 1

Hi @mike.hobbs,

The error in the bnsupport tool is due to a bug, which has been fixed. We will release a newer version of the bnsupport tool soon. In the meantime, could you check the following log for errors and paste them here?

INSTALLDIR\apache-tomcat\logs\catalina*.log

Regards,
Michiel

Hi Michiel:

Thank you for getting back to me so quickly and letting me know about the
bug in bnsupport. I found a number of time-stamped catalina*.log files and
copied content from the newest files that seemed to coincide with the
latest instance of the issue. I stopped the Jenkins services to try to keep
the log information relevant to the problem, please let me know if you’d
like to see any other log files from the system.

catalina.2020-01-17.log - This file was too large to paste in an email so I
copied sections. It started with the following text:
17-Jan-2020 00:26:04.861 INFO [1952]
hudson.model.AsyncPeriodicWork.lambda$doRun$0 Started telemetry collection
17-Jan-2020 00:26:04.885 INFO [1952]
hudson.model.AsyncPeriodicWork.lambda$doRun$0 Finished telemetry
collection. 19 ms
17-Jan-2020 02:08:42.375 INFO [Workspace clean-up thread]
hudson.model.AsyncPeriodicWork.lambda$doRun$0 Started Workspace clean-up
17-Jan-2020 02:08:42.442 INFO [Workspace clean-up thread]
hudson.model.AsyncPeriodicWork.lambda$doRun$0 Finished Workspace clean-up.
62 ms
17-Jan-2020 08:46:36.396 INFO [Download metadata thread]
hudson.model.AsyncPeriodicWork.lambda$doRun$0 Started Download metadata
17-Jan-2020 08:46:36.419 INFO [Download metadata thread]
hudson.util.Retrier.start Attempt #1 to do the action check updates server
17-Jan-2020 08:46:42.365 INFO [Download metadata thread]
hudson.model.UpdateSite.updateData Obtained the latest update center data
file for UpdateSource default
17-Jan-2020 08:46:42.878 INFO [Download metadata thread]
hudson.model.DownloadService$Downloadable.load Obtained the updated data
file for hudson.tasks.Maven.MavenInstaller
17-Jan-2020 08:46:42.966 INFO [Download metadata thread]
hudson.model.DownloadService$Downloadable.load Obtained the updated data
file for hudson.tasks.Ant.AntInstaller
17-Jan-2020 08:46:43.125 INFO [Download metadata thread]
hudson.model.DownloadService$Downloadable.load Obtained the updated data
file for hudson.plugins.gradle.GradleInstaller
17-Jan-2020 08:46:43.808 INFO [Download metadata thread]
hudson.model.DownloadService$Downloadable.load Obtained the updated data
file for hudson.tools.JDKInstaller
17-Jan-2020 08:46:43.809 INFO [Download metadata thread]
hudson.util.Retrier.start Performed the action check updates server
successfully at the attempt #1
17-Jan-2020 08:46:43.813 INFO [Download metadata thread]
hudson.model.AsyncPeriodicWork.lambda$doRun$0 Finished Download metadata.
7,413 ms
17-Jan-2020 10:53:51.124 INFO [Handling GET /jenkins/job/9300-499-NFMTC-MQ/
from 192.168.11.35 : ajp-nio-8009-exec-12 Job/index.jelly
AbstractProject/main.jelly]
hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.error Failed to
instantiate optional component
com.sonyericsson.jenkins.plugins.bfa.model.dbf.BuildCacheDBF; skipping
17-Jan-2020 12:13:06.141 INFO [Fingerprint cleanup thread]
hudson.model.AsyncPeriodicWork.lambda$doRun$0 Started Fingerprint cleanup
17-Jan-2020 12:13:06.149 INFO [3306]
hudson.model.AsyncPeriodicWork.lambda$doRun$0 Finished Fingerprint cleanup.
3 ms
17-Jan-2020 13:22:56.073 INFO [Executor #0 for master : executing
9300-499-NFMTC-MQ #3] hudson.model.Run.execute 9300-499-NFMTC-MQ #3 main
build action completed: FAILURE
17-Jan-2020 13:28:09.785 INFO [Executor #0 for master : executing
9300-499-NFMTC-SQHT #3] hudson.model.Run.execute 9300-499-NFMTC-SQHT #3
main build action completed: FAILURE
17-Jan-2020 13:32:05.193 INFO [Executor #0 for builder1 : executing
9300-499-NFMTC-MQVR #4] hudson.model.Run.execute 9300-499-NFMTC-MQVR #4
main build action completed: FAILURE
17-Jan-2020 14:55:56.367 INFO [Executor #0 for master : executing
9300-499-NFMTC-MQ #4] hudson.model.Run.execute 9300-499-NFMTC-MQ #4 main
build action completed: SUCCESS
17-Jan-2020 14:58:18.686 INFO [Executor #0 for builder1 : executing
9300-499-NFMTC-MQVR #5] hudson.model.Run.execute 9300-499-NFMTC-MQVR #5
main build action completed: SUCCESS
17-Jan-2020 14:59:26.074 INFO [IOHub#1:
Worker[channel:java.nio.channels.SocketChannel[connected local=/
192.168.10.175:38000 remote=192.168.10.67/192.168.10.67:49172]] /
Computer.threadPoolForRemoting [#2782]]
jenkins.slaves.DefaultJnlpSlaveReceiver.channelClosed IOHub#1:
Worker[channel:java.nio.channels.SocketChannel[connected local=/
192.168.10.175:38000 remote=192.168.10.67/192.168.10.67:49172]] /
Computer.threadPoolForRemoting [#2782] for builder1 terminated:
java.nio.channels.ClosedChannelException

The rest of the file was comprised of warnings like these:
17-Jan-2020 15:01:10.518 WARNING [Handling POST
/jenkins/computer/builder1/ajaxExecutors from 192.168.10.67 :
ajp-nio-8009-exec-8] hudson.security.csrf.CrumbFilter.doFilter Found
invalid crumb
1b2de26f9cd36d54b0f359d0fc4226768cb4a1f4789e50384f7cfbacf038da91. Will
check remaining parameters for a valid one…
17-Jan-2020 15:01:10.518 WARNING [Handling POST
/jenkins/computer/builder1/ajaxExecutors from 192.168.10.67 :
ajp-nio-8009-exec-8] hudson.security.csrf.CrumbFilter.doFilter No valid
crumb was included in request for /jenkins/computer/builder1/ajaxExecutors
by vmt. Returning 403.
17-Jan-2020 15:01:15.521 WARNING [Handling POST
/jenkins/computer/builder1/ajaxExecutors from 192.168.10.67 :
ajp-nio-8009-exec-6] hudson.security.csrf.CrumbFilter.doFilter Found
invalid crumb
1b2de26f9cd36d54b0f359d0fc4226768cb4a1f4789e50384f7cfbacf038da91. Will
check remaining parameters for a valid one…
17-Jan-2020 15:01:15.521 WARNING [ajp-nio-8009-exec-6]
hudson.security.csrf.CrumbFilter.doFilter No valid crumb was included in
request for /jenkins/computer/builder1/ajaxExecutors by vmt. Returning 403.
17-Jan-2020 15:01:20.525 WARNING [Handling POST
/jenkins/computer/builder1/ajaxExecutors from 192.168.10.67 :
ajp-nio-8009-exec-10] hudson.security.csrf.CrumbFilter.doFilter Found
invalid crumb
1b2de26f9cd36d54b0f359d0fc4226768cb4a1f4789e50384f7cfbacf038da91. Will
check remaining parameters for a valid one…
17-Jan-2020 15:01:20.525 WARNING [ajp-nio-8009-exec-10]
hudson.security.csrf.CrumbFilter.doFilter No valid crumb was included in
request for /jenkins/computer/builder1/ajaxExecutors by vmt. Returning 403.
17-Jan-2020 15:01:25.512 WARNING [Handling POST
/jenkins/computer/builder1/ajaxExecutors from 192.168.10.67 :
ajp-nio-8009-exec-3] hudson.security.csrf.CrumbFilter.doFilter Found
invalid crumb
1b2de26f9cd36d54b0f359d0fc4226768cb4a1f4789e50384f7cfbacf038da91. Will
check remaining parameters for a valid one…
17-Jan-2020 15:01:25.512 WARNING [Handling POST
/jenkins/computer/builder1/ajaxExecutors from 192.168.10.67 :
ajp-nio-8009-exec-3] hudson.security.csrf.CrumbFilter.doFilter No valid
crumb was included in request for /jenkins/computer/builder1/ajaxExecutors
by vmt. Returning 403.
17-Jan-2020 15:01:30.518 WARNING [Handling POST
/jenkins/computer/builder1/ajaxExecutors from 192.168.10.67 :
ajp-nio-8009-exec-11] hudson.security.csrf.CrumbFilter.doFilter Found
invalid crumb
1b2de26f9cd36d54b0f359d0fc4226768cb4a1f4789e50384f7cfbacf038da91. Will
check remaining parameters for a valid one…
17-Jan-2020 15:01:30.518 WARNING [Handling POST
/jenkins/computer/builder1/ajaxExecutors from 192.168.10.67 :
ajp-nio-8009-exec-11] hudson.security.csrf.CrumbFilter.doFilter No valid
crumb was included in request for /jenkins/computer/builder1/ajaxExecutors
by vmt. Returning 403.
17-Jan-2020 15:01:35.517 WARNING [Handling POST
/jenkins/computer/builder1/ajaxExecutors from 192.168.10.67 :
ajp-nio-8009-exec-9] hudson.security.csrf.CrumbFilter.doFilter Found
invalid crumb
1b2de26f9cd36d54b0f359d0fc4226768cb4a1f4789e50384f7cfbacf038da91. Will
check remaining parameters for a valid one…
17-Jan-2020 15:01:35.518 WARNING [Handling POST
/jenkins/computer/builder1/ajaxExecutors from 192.168.10.67 :
ajp-nio-8009-exec-9] hudson.security.csrf.CrumbFilter.doFilter No valid
crumb was included in request for /jenkins/computer/builder1/ajaxExecutors
by vmt. Returning 403.
17-Jan-2020 15:01:40.520 WARNING [Handling POST
/jenkins/computer/builder1/ajaxExecutors from 192.168.10.67 :
ajp-nio-8009-exec-13] hudson.security.csrf.CrumbFilter.doFilter Found
invalid crumb
1b2de26f9cd36d54b0f359d0fc4226768cb4a1f4789e50384f7cfbacf038da91. Will
check remaining parameters for a valid one…
17-Jan-2020 15:01:40.521 WARNING [Handling POST
/jenkins/computer/builder1/ajaxExecutors from 192.168.10.67 :
ajp-nio-8009-exec-13] hudson.security.csrf.CrumbFilter.doFilter No valid
crumb was included in request for /jenkins/computer/builder1/ajaxExecutors
by vmt. Returning 403.
17-Jan-2020 15:01:45.537 WARNING [ajp-nio-8009-exec-5]
hudson.security.csrf.CrumbFilter.doFilter Found invalid crumb
1b2de26f9cd36d54b0f359d0fc4226768cb4a1f4789e50384f7cfbacf038da91. Will
check remaining parameters for a valid one…
17-Jan-2020 15:01:45.537 WARNING [ajp-nio-8009-exec-5]
hudson.security.csrf.CrumbFilter.doFilter No valid crumb was included in
request for /jenkins/computer/builder1/ajaxExecutors by vmt. Returning 403.

The file ended with:
17-Jan-2020 23:44:51.707 WARNING [ajp-nio-8009-exec-5]
hudson.security.csrf.CrumbFilter.doFilter No valid crumb was included in
request for /jenkins/computer/builder1/ajaxExecutors by vmt. Returning 403.
17-Jan-2020 23:46:53.342 INFO [IOHub#1:
Worker[channel:java.nio.channels.SocketChannel[connected local=/
192.168.10.175:38000 remote=192.168.10.67/192.168.10.67:49446]] /
Computer.threadPoolForRemoting [#3558]]
jenkins.slaves.DefaultJnlpSlaveReceiver.channelClosed IOHub#1:
Worker[channel:java.nio.channels.SocketChannel[connected local=/
192.168.10.175:38000 remote=192.168.10.67/192.168.10.67:49446]] /
Computer.threadPoolForRemoting [#3558] for builder1 terminated:
java.nio.channels.ClosedChannelException
17-Jan-2020 23:48:12.464 INFO [TCP agent connection handler #5 with /
192.168.10.67:49674] hudson.TcpSlaveAgentListener$ConnectionHandler.run
Connection #5 failed: java.io.EOFException
17-Jan-2020 23:48:20.621 INFO [TCP agent connection handler #6 with /
192.168.10.67:49675] hudson.TcpSlaveAgentListener$ConnectionHandler.run
Accepted JNLP4-connect connection #6 from /192.168.10.67:49675

catalina.2020-01-18.log - Only two lines in this file:
18-Jan-2020 00:26:06.124 INFO [telemetry collection thread]
hudson.model.AsyncPeriodicWork.lambda$doRun$0 Started telemetry collection
18-Jan-2020 00:26:06.132 INFO [4787]
hudson.model.AsyncPeriodicWork.lambda$doRun$0 Finished telemetry
collection. 4 ms

[image: Virtual Marine] http://virtualmarine.ca/ Michael Hobbs

Software Developer

Office: (709) 738-6306 ext 252

Email: mike.hobbs@virtualmarine.ca mike.hobbs@virtualmarine.ca

Website: virtualmarine.ca http://www.virtualmarine.ca

https://www.facebook.com/virtualmarinetech/
https://twitter.com/virtual_marine
https://www.youtube.com/user/VMTechnology
https://www.linkedin.com/company/virtual-marine-technology

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.