Bitnami Apache Tomcat EC2 fails Instance Status Check after running apt-get upgrade and restarting

Keywords: Tomcat - Amazon Web Services - Technical issue - Other
Description:
Our instances all originally derived from a Bitnami Apache Tomcat AMI. We’ve since continued to grow our own AMIs off of those servers but the underlying structure is still all original. We have been having a problem recently when running upgrades. It happens after we run an upgrade. It works fine after the upgrade until reboot and then it will fail the instance status check. After it fails the check the instance is unreachable.

I’ve seen this happen before and then was able to spawn a new image off an earlier snapshot of that instance. I ran upgrade again and watched exactly what it upgraded. I noticed the JDK upgraded. After looking at the JDK directory I found that the bitnami directory that contains the setenv.sh files was missing. I copied it back in and then restarted and then all was good, passed the status checks.

This time however I don’t have a recent copy of the problematic instance. So I’ve disconnected the instance volume and attached it to another one. I’m able to access the volume this way. I saw once again the bitnami folder was missing so I restored it. Then created a new AMI off that volume and launched a new instance. That instance however is also failing the instance status check.

So my question is, is there a better way to identify why these status checks fail? Are there bitnami logs I can find somewhere from the attempted start up to see what went wrong? Is there a tool I can run to analyze the volume?

Any help is greatly appreciated.

Hi @nmb1106,

Thanks for using Bitnami. Can you explain the upgrade process you are applying to the VMs?

Regards,

Andres R.

Hi @andresrc,

Sorry for the long delay. We’re working through a security audit and have been side tracked with that.

I just run the following:

sudo apt-get update
sudo apt-get upgrade

I’ll also say though that I upgraded the Open JDK 1.7 to Oracle JDK 1.8 at some point long ago. We needed to switch to Oracle after having some cert problems executing web service requests against some of our customers services. I followed a guide I found some where in the community. It had me install and then update a number of references to the JAVA_HOME and JRE_HOME throughout the instance. Since then the bitnami folder will just disappear after updating sometimes.

My biggest question really though is what tools are available to me that can help me identify why the instance is failing the checks? Are there any logs or anything of that nature to check? Is there anything to look for inside the instance’s volume that I detached and attached to another instance? It seems hard to believe that something could just not start and there is no where to look for why. But so far I’ve found nothing useful.

Hi @nmb1106,

A colleague will contact you soon to help you with this issue.

Best regards,

Michiel D’Hont


Please, click on :heart: if you think my answer was helpful

Hi @nmb1106,

So, your issue is the “/opt/bitnami” directory getting removed after running “apt-get upgrade”? That’s really strange. The only thing that occurs to us you can check, are the system logs at “/var/log”, and to save the logs from the “apt-get upgrade” command in case we see anything strange.

If you still have your instance present somehow, and the “/opt/bitnami” directory is present, we have a Support Tool that will gather relevant information for us to debug the issue. Could you please download and execute it on the machine where the stack is running by following the steps described in the guide below? You must click on the platform or cloud that you are using to find the correct instructions.

How to Run the Bitnami Support Tool

Please note that you need to paste the code outputted by the tool in your reply.

If it is possible for you, we would recommend to migrate to a new AMI that does not present this issue, since there might be some kind of logic cleaning /opt after upgrades, although we’re unfamiliar with anything similar to this.

Hi @marcos,

No the /opt/bitnami doesn’t get removed. A while ago I installed an oracle JDK at /usr/lib/jvm and changed everything in the environment to use the new JDK home instead of the old /opt/bitnami/java home. The /usr/lib/jvm/jre/bitnami directory gets removed which contains the setenv.sh files for setting the java memory arguments.

I don’t know if that’s the only issue though as I’ve tried detaching the volume from the instance and attaching it as a secondary to another. Then I put that directory back in place. I then created a snapshot and a new AMI from that volume. Then launched a new instance. That instance spawns with the same problem.

This issue isn’t holding us up any longer but my goal of this post was really to find a way to figure out why the instance is failing the status checks. For instance, if my application failed to start up I’d check my logs that are filled with details during the startup. I’m looking for the equivalent to look for here. I certainly am not an expert in linux environments and am really learning as I go out. The answer to this may be very simple and just something I don’t know to look for.

There are tons of logs in /var/log, is there a specific log I should look at?

Thank you

Hi @nmb1106,

Regarding the “/usr/lib/jvm/jre/bitnami” directory, we were not able to locate such a directory in our Tomcat AMIs.

Note that independently of the system’s JAVA_HOME value, the Bitnami installation should use the Java installation in “/opt/bitnami/java”.

If that is not the case, it is probable the Tomcat scripts have been modified somehow. For instance, check “/opt/bitnami/apache-tomcat/bin/setenv.sh”, as it should contain the following line explicitly: “JAVA_HOME=/opt/bitnami/java”.

When the directory gets removed after upgrading, what is happening to your application? Is it not starting or are you not able to reproduce it due to the status check fail? Note that when you see the status check not working, it is more probable for it to happen because of SSH settings changes than Tomcat itself not working.

We recommend you checking SSH changes to ensure they are properly configured, and to ensure that SSH is configured to start when booting the VM. You can do this with “chroot” when you mount the volume and run “service”, although you will need to mount some directories before for getting the commands to work (/run, /proc, /dev).

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.