Jenkins Keeps Stopping

Keywords: Jenkins - AWS - Technical issue - Other
bnsupport ID: b3722ea9-7ec8-0f99-f53e-105bcba26154
Description:
Hi

The jenkins instance we have running on AWS keeps stopping. When I check the status of the service - apache is running but tomcat is stopped.
When I run the top command - a command called tt_swapdas is always running at over 150% cpu.

Even if I restart the server, and I try to do a build - jenkins will stop.
Can anyone suggest any way of improving things so that this does not happen?
My details are as follows?
Instance Type: T3 Small
AMI Id: bitnami-jenkins-2.121.2-1-r35-linux-ubuntu-16.04-x86_64-hvm-ebs-mp-447af561-b7e2-4d03-8a67-4842db5439cb-ami-0e034c33fc230393c.4 (ami-0d6d30af990682e29)

Any help on this would be greatly appreciated

Thanks
Damien

Hi @damien.gallagher,

Can you take a look at this guide in our documentation and share with us the output of those commands?

https://docs.bitnami.com/aws/faq/troubleshooting/troubleshoot-server-performance/

Thanks

Hi @jota

Please find requested details here
Instance Type: t3 Small - 2vCPUS and 2Gb Memory
No other apps running on server - just jenkins
No crontabs for bitnami or root user
ps -e -orss=,args= | sort -b -k1,1n | pr -TW$COLUMNS - see attached image
ps -e -o pcpu,nice,state,cputime,args --sort -pcpu | head -10 - Please see attached image
Disk space - please see attached image

Please let me know if you require anything else
Thanks
Damien

Checking the screenshots, I saw that your instance’s disk is almost full. I think this can be the problem you are running into now or a problem in the future, can you increase that disk?

https://docs.bitnami.com/aws/how-to/modify-ebs-volume/

Apart from that, I also found this line in the Tomcat’s log

20-Sep-2019 11:24:24.384 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [jenkins] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@3f7a598d]) and a value of type [org.acegisecurity.context.SecurityContextImpl] (value [org.acegisecurity.context.SecurityContextImpl@ffffffff: Null authentication]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

This is probably because your application requires more memory that the one it currently has. I’d try increasing the instance type to know if that solves the issue.

https://docs.bitnami.com/aws/faq/administration/change-server-type/

Happy to help!


Was my answer helpful? Click on :heart:

Thanks Jota

I just tried increasing the disk space

I will monitor how that behaves and if I see no improvment, I will increase the machine type
Out of interest - what do you guys recommend as the minimum machine spec to use?

Hi @damien.gallagher,

Jenkins works with 1GB of memory but if you start adding jobs to the application you will require more memory for sure. The amount of RAM you need depends on the number of jobs you have and how much memory the will consume (it’s not the same having 3 simple bash scripts than having 3 jobs building some java applications). You will need to monitor the performance of the server to know if you need to increase the memory of the machine, the performance guide I shared above will help you.

Happy to help!


Was my answer helpful? Click on :heart:

1 Like

Hi @jota

I totally agree

Many thanks for your professional help on this issue.
I am happy that the disk size upgrade and potential machine upgrade will solve my issues.
Actually seeing better performance since increasing the disk

Thanks again
Damien

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.