Jenkins Environment Reset Daily?

rbrown0504 2020-04-06 18:53:01 UTC #1

IMPORTANT, please fill the questions

We assume you are using Bitnami to deploy your application.

Which version of the application are you using?:
Google Console Deployment Last Week - Version 2.204.5
Please choose how you got the application: Installer (Windows, Linux, macOS), cloud image (AWS, GCE, Azure, ...) or VM (VMDK, VBOX):
Google Console Jenkins Certified Bitnami. I believe it is Debian.
Have you installed any plugin or modified any configuration file?:
Installed SFDX CLI for Salesforce.com.
Describe here your question/suggestion/issue (expected and actual results):
I had to deploy new version of Jenkins. Previous deployment 10/2019. This deployment different. It seems that environment variables at OS level may be changing (or perhaps getting reset) daily based on symptoms from SFDX.

sfdx force:mdapi:deploy --wait 120 -c --deploydir /opt/bitnami/jenkins/jenkins_home/workspace/TAN-pipe_prod_validate/mdapi --targetusername xxx.xxxx@xxxx.com --testlevel RunLocalTests
ERROR running force:mdapi:deploy: We encountered a JSON web token error, which is likely not an issue with Salesforce CLI. Here’s the error: ENOENT: no such file or directory, open '/opt/bitnami/jenkins/jenkins_home/workspace/TAN-pipe_prod_validate@tmp/secretFiles/ef828807-fd4d-41a0-87da-554870598fce/server.key'

It's tripping up on a server.key secret credentials file. Between when it initializes this key file in a login command a few lines prior to this and this line, some how Jenkins/SFDX can no longer see the file. This is a local OS environment issue. I have only accessed Jenkins web interface post deployment for configuration outside of logging in to CLI to install SFDX.

In my Groovy script I've precluded this ahead of everything else:

withEnv(["HOME=${env.WORKSPACE}"]) {

}

This should set the environment of the local job from Jenkins perspective. Everything is pointing me to this issue being OS related right now as the application I'm using utilizes local environment variables.

Steps to reproduce the issue (if relevant):
I have to delete pipes daily in order to re-initialize. 24 hours later that same project will fail with aforementioned error. Once I delete multi-branch pipeline and recreate it, things are back to normal.
Copy the apache log (if relevant):
```
PASTE HERE
```

jota 2020-04-07 09:52:19 UTC #2

Hi @rbrown0504,

Thank you for using our solution. Please note that Bitnami packages and configures already existing applications. However, we can only offer support and help on basic and most frequent application configurations. Always based on the default configuration or on our guides.

If the question is related to the application itself or any third party tool, probably someone with more experience and knowledge about that specific tool can help you better than we can. We suggest you to post your question on the official support forums of the application so others can help you.

https://jenkins.io/chat/

Do not hesitate to write us back if you have any other questions regarding the Bitnami solutions or their configuration.

rbrown0504 2020-04-07 15:18:56 UTC #3

My question is not about 3rd party application. My question was about the
stack resetting environment variables daily which is potentially causing
symptoms of a 3rd party application. My 3rd party application is telling me
they use environment variables. It seems those environment variables are
changing from day to day. Can you confirm if there is any kind of behavior
at the OS level like this happening? The organization of the stack is
different from October 2019 until now. I can tell just by the folder
structure of the VM. For example in October apache directories were in
standard location, now they are contained in a opt/bitnami structure. It
worked in my old bitnami stack VM it doesn't work now. I'm trying to
understand where the problem is and I'm getting ping ponged, just like you
are here. From my perspective, I've provisioned the same stack from
October, same versions of applications and I'm trying to understand why
things aren't working.

Any advise is appreciated.

Thank you.
Robert

jota 2020-04-08 07:11:56 UTC #4

Hi @rbrown0504

We have a Support Tool that will gather relevant information for us to analyze your configuration and logs. Could you please execute it on the machine where the stack is running by following the steps described in the guide below?

How to Run the Bitnami Support Tool in a cloud image or virtual machine

Please note that you need to paste the code ID that is shown at the end.

rbrown0504 2020-04-09 16:50:00 UTC #5

Thank you for taking the time to look. Even if it is to eliminate this as an issue. At least I can go back to other organizations armed with any potentially useful information.

Here's what was output at the end of support tool: c31ad2c7-48f7-6d25-c57a-f0b4d36f1f37

jsalmeron 2020-04-10 16:10:16 UTC #6

Hi,

Could you let us know where you are setting these environment variables?

Best regards,

Javier J. Salmerón

Was my answer helpful? Click on

rbrown0504 2020-04-10 16:22:43 UTC #7

I mentioned in the description, environment variables are being set in script:

…

withEnv(["HOME=${env.WORKSPACE}"]) {

}

…

This is the same exact script I’m using from a Bitnami Jenkins Deployment from October. Same 3rd Party Application. Same version of 3rd Party Application.

jsalmeron 2020-04-13 14:38:08 UTC #8

Hi,

Sorry, I don't have much experience with Jenkins, but what I wanted to know where you set the value of this WORKSPACE environment variable. Is it done at some place inside Jenkins?

Best regards,

Javier J. Salmerón

Was my answer helpful? Click on

rbrown0504 2020-04-13 17:01:30 UTC #9

Each time the script is run, the environment variables are set inside the script. The script should be setting the ‘HOME’ environment variable to the directory of the current workspace.

My fundamental issue is this was working in the last deployment of Bitnami Jenkins from October 2019. The deployment as of 2 weeks ago, I have to delete and recreate stuff daily now in order for this to work with the error eluding to no access to a file and 3rd party app telling me it has to do with environment variables. The only difference right now I see is the difference in deployment package. Was there a version change between October and 3 weeks ago with respect to the Bitnami deployment of Jenkins? Different OS? Different Jenkins instance? What’s different from October to now to where I can’t hit the ground running? Things like this should just work. I don’t have this issue on my local install of Jenkins.

Robert

jsalmeron 2020-04-15 07:57:59 UTC #10

Hi,

We performed some changes in the folder structure but not in the asset itself. Could we have a small script that triggers the issue so we can reproduce it in our side? Also with the steps in order to install it.

Best regards,

Javier J. Salmerón

Was my answer helpful? Click on

rbrown0504 2020-04-15 16:20:24 UTC #11

Hi Javier –

I don’t know how to do that without handing over security certificates which I’m thinking isn’t a wise thing to do. I’m happy to give you a log in to our Jenkins instance and perhaps a way to SSH in to the machine?

The only thing I’ve done post Bitnami deployment is:
Generate SSH keys
Install SFDX
Add credentials to Jenkins
Create Multi-Branch Pipeline

Here were the technical steps taken:

ssh-keygen (then captured id_rsa default result)
wget https://developer.salesforce.com/media/salesforce-cli/sfdx-linux-amd64.tar.xz
mkdir sfdx
tar xJf sfdx-linux-amd64.tar.xz -C sfdx --strip-components 1
./sfdx/install
sfdx plugins:install salesforcedx@47.15.0

The rest of the work is inside Jenkins itself.

Add Credentials (Consumer Keys, Server certificates)
Create Multi-Branch pipeline and link to BitBucket Git Repository

The repository contains the scripts for deployment, so no additional work is needed. This is where my challenge is. Scripts will work only a period of time before failing. I have to do what in essence is the equivalent of deleting folder and recreating folder in order for things to ‘reset’ whatever that means.

Robert

jsalmeron 2020-04-17 08:15:14 UTC #12

Hi,

Thanks for providing that information. I will need to take some time to reproduce (sorry, I have a lot on my plate at the moment), but I will update this thread as soon as I have news.

Best regards,

Javier J. Salmerón

Was my answer helpful? Click on

rbrown0504 2020-04-20 18:19:57 UTC #13

Thank you.

I found this document that outlines pretty much the post Jenkins configuration: https://docs.bitnami.com/installer/how-to/create-ci-pipeline/

With the exception of GitLab vs. Bitbucket from a repository perspective, These are the same steps (for most part) I performed.

Robert

jsalmeron 2020-05-01 12:01:16 UTC #14

Hi,

I could find time to work on this. After checking more closely, I'm not sure I fully understand which environment variable is causing the issue. From the error message

/opt/bitnami/jenkins/jenkins_home/workspace/TAN-pipe_prod_validate@tmp/secretFiles/ef828807-fd4d-41a0-87da-554870598fce/server.key

It seems that the workspace is set correctly, Then we have a temporary folder that may probably be deleted after some time. Could it be that Jenkins is deleting that folder and therefore the file not being accessible? It is very strange that it happens just after 24 hours. If it was a OS issue, I believe the issue should have appeared sooner. I don't see any cronjob specified there so I believe that it is something that Jenkins should be doing.

Best regards,

Javier J. Salmerón

Was my answer helpful? Click on

rbrown0504 2020-05-01 15:42:32 UTC #15

Thank you for looking in to it. I appreciate your help in eliminating OS. I will give this feedback to both salesforce and Jenkins.

jsalmeron 2020-05-04 09:35:39 UTC #16

Hi,

Please let us know what they say, in case there's something we need to implement in our side.

system 2020-05-18 09:35:39 UTC #17

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Bitnami provides free all-in-one installers, virtual machines and cloud images for popular open source applications. Get them now!