How exactly are tomcat memory settings applied in the Bitnami Tomcat 9 images?

Keywords: Tomcat - Linux - Technical issue - Other
Description:
Hello, we just upgraded our production servers last night to use an image of ours derived from Bitnami Tomcat 9. We’ve seen some behavior this morning where an instances CPU will get pegged which causes it to become unhealthy due to slow responses to the ELB’s health checks. So we’re looking around for the cause. We’re trying to confirm the memory settings for tomcat are being applied correctly.

Our image has edits to this file /root/.nami/components/com.bitnami.tomcat/helpers.js which we can see that bitnami is using to correctly generate this file /opt/bitnami/tomcat/conf/bitnami/memory.sh with the memory settings based on the current instances size. That’s all working good.

When I use java’s jinfo tool to check the details of the process running tomcat. For instance: sudo jinfo 1392

That returns this:

VM Flags:
-XX:CICompilerCount=3 -XX:ConcGCThreads=1 -XX:G1ConcRefinementThreads=4 -XX:G1HeapRegionSize=1048576 -XX:GCDrainStackTargetSize=64 -XX:InitialHeapSize=264241152 -XX:MarkStackSize=4194304 -XX:MaxHeapSize=4206886912 -XX:MaxNewSize=2523922432 -XX:MinHeapDeltaBytes=1048576 -XX:NonNMethodCodeHeapSize=5830732 -XX:NonProfiledCodeHeapSize=122913754 -XX:ProfiledCodeHeapSize=122913754 -XX:ReservedCodeCacheSize=251658240 -XX:+SegmentedCodeCache -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseG1GC

VM Arguments:
jvm_args: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED -Djuli-logback.configurationFile=/opt/bitnami/tomcat/conf/logback.xml -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -Dcatalina.base=/opt/bitnami/tomcat -Dcatalina.home=/opt/bitnami/tomcat -Djava.io.tmpdir=/opt/bitnami/tomcat/temp
java_command: org.apache.catalina.startup.Bootstrap start
java_class_path (initial): /opt/bitnami/tomcat/bin/bootstrap.jar:/opt/bitnami/tomcat/bin/tomcat-juli.jar

There is no reference there to the -Xms or -Xmx arguments set in memory.sh. Does that mean they aren’t getting loaded properly? What process exactly uses that memory.sh file to set the arguments for tomcat? Also for what it’s worth if I run the same command in an instance of the Bitnami Tomcat 7 image it does have the Xms and Xmx arguments set.

Hi @nmb1106,

Thank you for using Bitnami.

Could you please post here the changes you applied to the helpers.js file so the team can evaluate them? Once you provide the information, our team will review the information and will investigate why the jinfo command doesn’t show the memory settings.

Thanks

Hello,

I’m not sure at this point what pieces of this came from the Bitnami image vs edits we made but here is what we had and found.

/opt/bitnami/tomcat/bin/setenv.sh had this value:

JAVA_OPTS=-Djava.awt.headless=true -XX:+UseG1GC -Dfile.encoding=UTF-8 -Djuli-logback.configurationFile=$CATALINA_HOME/conf/logback.xml

We found it needed to be wrapped in quotes. I know at minimum we altered this line to refer to our logback file. I’m not sure if this line originates from us or if we modified something that was there. Once we found and fixed that the memory settings from memory.sh were still in the JAVA_OPTS variable when catalina.sh was executing the startup. I added echos to a log in setenv.sh, memory.sh, and catalina.sh.

setenv.sh was printing:
/opt/bitnami/tomcat/bin/setenv.sh

JAVA_OPTS: -Djava.awt.headless=true -XX:+UseG1GC -Dfile.encoding=UTF-8 -Djuli-logback.configurationFile=/opt/bitnami/tomcat/conf/logback.xml

memory.sh was printing:
/opt/bitnami/tomcat/conf/bitnami/memory.sh

JAVA_OPTS: -Xms2048M -Xmx10000M -Djava.awt.headless=true -XX:+UseG1GC -Dfile.encoding=UTF-8 -Djuli-logback.configurationFile=/opt/bitnami/tomcat/conf/logback.xml

catalina.sh before it executed the startup command was printing:
/opt/bitnami/tomcat/bin/catalina.sh

JAVA_OPTS: -Djava.awt.headless=true -XX:+UseG1GC -Dfile.encoding=UTF-8 -Djuli-logback.configurationFile=/opt/bitnami/tomcat/conf/logback.xml

You can see there with the logs that in the memory.sh file it concats the memory settings correctly but then downstream n the catalina.sh it’s lost so those memory settings never end up getting passed in to the startup of tomcat.

We went live with this so we had to fix it quickly…so for the time being our setenv.sh now cats a tag we added to our instances with the desired memory settings. I’d like to go back to the memory.sh way if we can sort this out.

Here are the 3 files I feel like are probably important for you to see.

/opt/bitnami/tomcat/bin/setenv.sh

#!/bin/bash
JAVA_HOME=/opt/bitnami/java
export JAVA_HOME
JAVA_OPTS="-Djava.awt.headless=true -XX:+UseG1GC -Dfile.encoding=UTF-8 -Djuli-logback.configurationFile=$CATALINA_HOME/conf/logback.xml"
export JAVA_OPTS
# Load Tomcat Native Library
LD_LIBRARY_PATH=/opt/bitnami/tomcat/lib:${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}
export LD_LIBRARY_PATH
# Memory settings
source /opt/bitnami/tomcat/conf/bitnami/memory.sh

/opt/bitnami/tomcat/conf/bitnami/memory.sh

#
# Bitnami Tomcat Configuration
#
# IMPORTANT: This file will be overwritten to adapt the memory settings for the server memory.
# If you want to add any configuration, please add it to the /opt/bitnami/tomcat/bin/setenv.sh file
#

export JAVA_OPTS="-Xms2048M -Xmx10000M $JAVA_OPTS"

/root/.nami/components/com.bitnami.tomcat/helpers.js (just showing the sizes const, didn’t change anything else)

  const sizes = {
    'micro': {
      ms: '256M', mx: '512M',
    },
    'small': {
      ms: '256M', mx: '768M',
    },
    'medium': {
      ms: '1024M', mx: '2048M',
    },
    'large': {
      ms: '2048M', mx: '4096M',
    },
    'xlarge': {
      ms: '2048M', mx: '10000M',
    },
    '2xlarge': {
      ms: '2048M', mx: '16383M',
    },
  };

Ok @nmb1106,

I’ll share this information with the team and we will update this thread as soon as we have any news.

Thanks

Hi,
We have found an little typo in the default setenv.sh that was not adding the quotes properly. We fixed it and in some hours a new version containing the patch should be released. Please wait and test that new version when it is available and let us know if that one solves the issue.
Thank you very much for reporting it.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.