Possible wrong Python used by AWS Bitnami Django

[rustyfosty]

Keywords: Django - AWS - Technical issue - Services (Apache, MariaDB, MySQL…)

bnsupport ID: 47e3c4ae-e3c4-ab89-b1d5-731b49874649

bndiagnostic output:

? Apache: Found possible issues
? Resources: Found possible issues
https://docs.bitnami.com/general/apps/wordpress/troubleshooting/debug-errors-apache/

bndiagnostic failure reason: The suggested guides are not related with my issue

Description:
Hi.
(disclaimer for following is I’m more experienced with Windows thatn Linux)
I am releasing a website. Before I go public I need to install/import the paypalcheckoutsdk module. I’ve logged onto my Lightsail Shell (from the console) and run:

python3 -m pip install paypal-checkout-serversdk

All seemed to go well. But the webstie falls over when it runs once it tries to import the module, saying it can’t be found.

When I go onto the Shell’s Python3 instance and import it all is fine. So I started to think perhaps the wrong Python is being run by Apache mod_wsgi. So I put a line in my views.py code to log sys.executable to my db logs. And it says the executable being run is /usr/bin/python3. While the Apache error log seems to indicate the correct versions of Python is running as I’m getting an error reported there (that doesn’t seem initially connected to my main problem):
Traceback (most recent call last):

  File "/opt/bitnami/python/lib/python3.8/site-packages/asgiref/local.py", line 96, in __del__
NameError: name 'TypeError' is not defined
[Thu Dec 30 07:12:06.294684 2021] [mpm_event:notice] [pid 22416:tid 139833577270144] AH00491: caught SIGTERM, shutting down
[Thu Dec 30 07:12:06.487310 2021] [mpm_event:notice] [pid 23247:tid 140101812063104] AH00489: Apache/2.4.51 (Unix) OpenSSL/1.1.1d mod_wsgi/4.7.1 Python/3.8 configured -- resuming normal operations
[Thu Dec 30 07:12:06.487401 2021] [core:notice] [pid 23247:tid 140101812063104] AH00094: Command line: '/opt/bitnami/apache/bin/httpd -f /opt/bitnami/apache/conf/httpd.conf'

Which seems to indicate the correct Python edition. So I have two fixes I can see.

1. Sort out why at runtime sys.executable is pointing to the other version of Python, and get it to switch to the correct version.
2. Install paypal-checkout-serversdk on the other version of Python.

So with option 1 I don’t know how to tell it “Hey, Apache, you’re using the wrong version of Python at runtime, even though you’re initialising the correct version”.

With option 2, I tried to pip install for that version
sudo /usr/bin/python3 -m pip install paypal-checkout-serversdk
And as it sorts out the dependencies it errors on not having Rust installed properly. Which I tried to fix by reinstalling Rust, which still didn’t work.

I’m thinking the first option is the better one anyway, but am at a loss to tell Apache/wsgi that it’s actually using the wrong version.

Or is that Apache error in the log telling me something I need to know, only it seems unrelated…

Many thanks, and a happy New Year, to you who take the time to read my issue, and even help.

[michiel]

Hi @rustyfosty,

Can you run the following command to check the python version the module was compiled against?

ldd /opt/bitnami/apache/modules/mod_wsgi.so 

Regards,
Michiel

[rustyfosty]

Hi Michiel. Many thanks for the reply. Here’s the result of that command:
‘’’
linux-vdso.so.1 (0x00007ffdac37b000)
libpython3.8.so.1.0 => /opt/bitnami/python/lib/libpython3.8.so.1.0 (0x00007f93093cc000)
libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f930938c000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f930936b000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f9309366000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f9309361000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f93091de000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f930901b000)
libssl.so.1.1 => /lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007f9308f89000)
libcrypto.so.1.1 => /lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f9308ca0000)
/lib64/ld-linux-x86-64.so.2 (0x00007f93097be000)
‘’’
Looks like it’s saying 3.8. I just updated my log message to the following:
‘’’
"Python exe used: " + str(sys.executable) + "\nVersion: " + str(sys.version_info[0]) + “.” + str(sys.version_info[1])
‘’’
It now spits out:
‘’’
Python exe used: /usr/bin/python3
Version: 3.8
‘’’
So it looks like the version is correct, but it’s not the executable that I have the ability to pip install successfully on. Here’s some commands that show what I’ve managede to get installed okay, and not okay.
‘’’
bitnami@ip-172-26-1-204:~$ python3
Python 3.8.12 (default, Sep 24 2021, 11:37:24)
[GCC 8.3.0] on linux
Type “help”, “copyright”, “credits” or “license” for more information.

import sys
sys.executable
‘/opt/bitnami/python/bin/python3’

from paypalhttp.http_error import HttpError
exit()
bitnami@ip-172-26-1-204:~$ /usr/bin/python3
Python 3.7.3 (default, Jan 22 2021, 20:04:44)
[GCC 8.3.0] on linux
Type “help”, “copyright”, “credits” or “license” for more information.

from paypalhttp.http_error import HttpError
Traceback (most recent call last):
File “”, line 1, in
ModuleNotFoundError: No module named ‘paypalhttp’
‘’’

[fdepaz]

Hello @rustyfosty,

This version mismatch could be caused if the module is only installed for Python 2 even if you are using the python3 console. This could result from using pip instead of pip3. You can check the modules installed for each Python version by running:

python -c 'help("modules")' | grep paypal
python3 -c 'help("modules")' | grep paypal

Regards,
Francisco de Paz

[rustyfosty]

Hi Francisco. Many thanks for the reply. I’ve run the commands you suggested and I’m pasting the results below. As what I can tell PayPal is installed on both 2 and 3 as I must have ran pip against both in desperation.

However, the premise of what you suggest isn’t making sense to me and you may have to explain in case I’m misunderstanding what you’re saying. I don’t understand how installing it on Python 2 would make Apache/mod_wsgi choose the wrong Python executable to be running. For example I can import it under Bitnami’s default Python3 install, so it pip’ed correctly, (see the below Python3 execution and then imports and executable showing)
‘’’
bitnami@ip–:~$ python3
Python 3.8.12 (default, Sep 24 2021, 11:37:24)
[GCC 8.3.0] on linux
Type “help”, “copyright”, “credits” or “license” for more information.

from paypalcheckoutsdk.core import PayPalHttpClient, SandboxEnvironment
import sys
sys.executable
‘/opt/bitnami/python/bin/python3’
‘’’
But when I ask the running instance of my app to report what executable it’s running it clearly says /usr/bin/python3. My understanding is that it should be executed by the default Django/Apache/mod_wsgi configuration on Lightsail using /opt/bitnami/python/bin/python3 instead. So it’s like mod_wsgi is pointing to the wrong executable. If I had, how can pip installing a package in the wrong executable tell Apache/mod_wsgi to execute another executable.

It’s almost like it’s a PATH problem, or mod_wsgi is pointing to ROOT’s default instead of Bitnami’s default.

bitnami@ip--:~$ python -c 'help("modules")' | grep paypal
/home/bitnami/.local/lib/python3.8/site-packages/Bio/SubsMat/__init__.py:126: BiopythonDeprecationWarning: Bio.SubsMat has been deprecated, and we intend to remove it in a future release of Biopython. As an alternative, please consider using Bio.Align.substitution_matrices as a replacement, and contact the Biopython developers if you still need the Bio.SubsMat module.
  warnings.warn(/home/bitnami/.local/lib/python3.8/site-packages/Bio/codonalign/__init__.py:23: BiopythonE
xperimentalWarning: Bio.codonalign is an experimental module which may undergo significant changes prior to its future official release.
  warnings.warn(/home/bitnami/.local/lib/python3.8/site-packages/Bio/phenotype/__init__.py:93: BiopythonExperimentalWarning: Bio.phenotype is an experimental submodule which may undergo significant changes prior to its future official release.
  warnings.warn(/home/bitnami/.local/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
_opcode             curses              paypalcheckoutsdk   time
_operator           cycler              paypalhttp          timeit


bitnami@ip--:~$ python3 -c 'help("modules")' | grep paypal
/home/bitnami/.local/lib/python3.8/site-packages/Bio/SubsMat/__init__.py:126: BiopythonDeprecationWarning: Bio.SubsMat has been deprecated, and we intend to remove it in a future release of Biopython. As an alternative, please consider using Bio.Align.substitution_matrices as a replacement, and contact the Biopython developers if you still need the Bio.SubsMat module.
  warnings.warn(/home/bitnami/.local/lib/python3.8/site-packages/Bio/codonalign/__init__.py:23: BiopythonExperimentalWarning: Bio.codonalign is an experimental module which may undergo significant changes prior to its future official release.
  warnings.warn(/home/bitnami/.local/lib/python3.8/site-packages/Bio/phenotype/__init__.py:93: BiopythonExperimentalWarning: Bio.phenotype is an experimental submodule which may undergo significant changes prior to its future official release.
  warnings.warn(/home/bitnami/.local/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
_opcode             curses              paypalcheckoutsdk   time
_operator           cycler              paypalhttp          timeit

[rustyfosty]

Sorry, the mark-up formatting didn’t work on that last bit

[fdepaz]

Hello @rustyfosty,

Oh, okay. I think you may be right, with the issue coming from a misconfiguration with wsgi. There was a similar error in the past, could you check it in case it is of help:

ModuleNotFoundError on AWS lightsail for WSGI (SOLVED)

I created a Django instance in AWS lightsail. I went through this tutorial: https://aws.amazon.com/getting-started/hands-on/deploy-python-application/ I did not use the tutorial django project from the tutorial, I used my own django project Django running with WSGI on Apache gives me the error: No module named 'statistics.models_es'; 'statistics' is not a package statistics is one of the apps in my project. My project folder does not seem to be in the paths where django is looking for apps. …

Regards,
Francisco de Paz

[rustyfosty]

Hi Francisco. Thanks again for the reply. I’ve had a look at the problem you linked. Although it kind of sounds similar, I don’t think it’s the same issue. That person hadn’t set his Python code up right. I however am trying to run a project that works fine on my own computer, but not on the Lightsail Django image. And it seems to be down to the wrong version of Python being run by Apache (from what I can tell). But if I understand the doco right that decision is made at mod_wsgi compile time? Which, if so, means the image is wrongly set up. IF so then it won’t be just affecting me, but all Django Lightsail users.

But no-body else seems to be having the same issue, or at least not bringing it up on the forums. And Googling isn’t bringing up any issues alike in any forum I can find.

So either I’ve done something that makes Apache execute the wrong Python executable (which I’m pretty sure I haven’t, but could be wrong), or the image that users can select in Lightsail isn’t set up right, which I have trouble believing to be the case.

Can anyone tell me how Apache knows which Python executable to use? Maybe I need to set it up manually to overcome whatever the problem is. Maybe even recompiling my own mod_wsgi…? (shudder). I’ve used c++ on Windows but I’ve got very little experience in Linux.

[fdepaz]

Hello @rustyfosty,

Let me try to reproduce this issue in a fresh instance and check whether the issue is indeed coming from the version used by Apache or a simple misconfiguration.

Regards,
Francisco de Paz

[rustyfosty]

Thanks Francisco. You help is very much appreciated.

[fdepaz]

Hello @rustyfosty,

Sorry about the wait, I have been able to reproduce the issue and verified that the python being used is /usr/bin/python3.

As a workaround, I have managed to enable using a sample project I have been able to import paypalhttp adding it to the project setting.py:

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'paypalhttp',
]

After that, I encountered a couple of other ModuleNotFoundError that were solved following this thread:

Unable to install Django packages in Bitnami Django stack in AWS

Hi @michiel I already fixed that. The problem is that ‘python install packages in a location that apache do not see’. so: packages get installed in ‘/home/bitnami/.local/lib/python3.8/site-packages/’ and Apache looks in ‘/opt/bitnami/python/lib/python3.8/site-packages/’. The temporary solution I followed is copying packages to Apache eyes folder with this command ‘cp -r /home/bitnami/.local/lib/python3.8/site-packages/* /opt/bitnami/python/lib/python3.8/site-packages/’

Nevertheless, this seems like an issue on our side. I’ll forward this ticket to our development team so they can investigate this in-depth. We’ll keep this thread updated with our findings.

Thanks for sharing this issue,
Francisco de Paz

[rustyfosty]

Hi Francisco, Many thanks for your efforts in confirming this issue, and the suggested work arounds. I will have a look at some other work arounds myself and report back if I manage to get it solved any other way.

I am wondering if by pointing the “python3” shell commands that root probably sees to the bitnami version of python, wether this will solve it. Since according to the Apache logs mod_wsgi seems to be pointing to the correct python executable it’s almost like Django is not (in a psuedo code fashion) saying, “Here’s the version of Python I’m being executed from, lets kick off another instance of that”, but seemingly, Hey OS, I need Python 3, what have you got for me?". (A bad way of saying it I know).

[rustyfosty]

@fdepaz, Actually I just added a getpass.getuser() reference to my log fom views.py and it’s saying the user is daemon if that helps track anything down. Is that what would be expected? The few references I’ve seen online say it’s used to run as a user with little privs to protect the OS/system, but maybe it doesn’t have permission to see Bitnami’s Python install, or is not defaulting to it?

[rustyfosty]

@fdepaz, I have a “good news”/“clue” update.
The good news is that I’ve managed to get the import statements working, but it took a bunch of changes at OS level, not at Python code level.

1. I changed the shortcut /usr/bin/python3 which was previously pointing to to point to the executable /usr/bin/python3.7 to point to /opt/bitnami/python/bin/python3. That seems to get sys.executable in views.py to point to the Python executable that I can update as Bitnami user. But it then had another problem. The pip3 install seemed to be downloading the paypal SDK module to a location that didn’t seem standard. Instead of going to /opt/bitnami/python/lib/python3.8/site-packages it seems to go to /home/bitnami/.local/lib/python3.8/site-packages instead.

2. Progresively copy in all module files from /opt/bitnami/python/lib/python3.8/site-packages to /home/bitnami/.local/lib/python3.8/site-packages until all the imports seemed to work. This isn’t a perfect solution, but it seems to get it to work.

So I see there might be two problems with the standard Lightsail Bitnami image

1. Somewhere Django or mod_wsgi loses its reference to the updateable (PIPable) executable that the Bitnami user has access to updating.
2. PIP downloads to modules you install to a location that the daemon can’t see to import from because it’s in what looks like a hidden folder elsewhere.

I hope that help you pick through the root cause of the problem. And I’d like to get it sorted out in the end, but this seems to get me going for now.

Many thanks to both yourself and @michiel for taking an interest in this post.

[jariza]

Hi @rustyfosty

I’m glad you were able to get the import statements working, although the UX you had to deal with is not nice at all.

@rustyfosty

• The pip3 install seemed to be downloading the paypal SDK module to a location that didn’t seem standard. Instead of going to /opt/bitnami/python/lib/python3.8/site-packages it seems to go to /home/bitnami/.local/lib/python3.8/site-packages instead.
• Progresively copy in all module files from /opt/bitnami/python/lib/python3.8/site-packages to /home/bitnami/.local/lib/python3.8/site-packages until all the imports seemed to work. This isn’t a perfect solution, but it seems to get it to work.

Yes, that’s the workaround explained @msayed at Unable to install Django packages in Bitnami Django stack in AWS
We definitely need to take a look into this.

@rustyfosty

So I see there might be two problems with the standard Lightsail Bitnami image

1. Somewhere Django or mod_wsgi loses its reference to the updateable (PIPable) executable that the Bitnami user has access to updating.
2. PIP downloads to modules you install to a location that the daemon can’t see to import from because it’s in what looks like a hidden folder elsewhere.

Correct, I wouldn’t have summarized it better. I’ve created an internal task to investigate these 2 issues and find a solution that improves the UX using this Cloud Image to develop Django websites.

Best regards,

Juan Ariza