Install Python Dependencies to Azure Bitnami Airflow Multi-Tier Package

marlonholland7 2020-04-10 01:43:34 UTC #1

Keywords: Apache Airflow - Microsoft Azure - How to - Other
Description:
Hi! First post here on these forums.

I'm trying to use a DAG that requires two python libraries to be installed:
snowflake-sqlalchemy
snowflake-connector-python

I'm not sure which Azure VM to install these on, or even how to access the Azure VM's command line to install these libraries. I am very new to this, so please let me know if you need more information.

Thanks!

marcos 2020-04-10 09:51:22 UTC #2

Hi @marlonholland7, find below a guide on how to connect to AirFlow VMs:

https://docs.bitnami.com/azure-templates/infrastructure/apache-airflow/administration/connect-nodes/

I guess that the modules will need to be executed on the worker or the webserver node. Run the command below:

sudo /opt/bitnami/python/bin/pip install snowflake-sqlalchemy
sudo /opt/bitnami/python/bin/pip install snowflake-connector-python

jota 2020-04-13 11:43:08 UTC #4

Hi @marlonholland7,

I think we are helping you regarding the DAG in this other thread, right? Let's continue the conversation there

jota 2020-04-14 08:25:35 UTC #6

Hi @marlonholland7,

Are you using a docker image or the Multi-Tier solution in Azure? I just launched the container and could install the python packages @marcos mentioned above without issues.

airflow@e0a3f60f9ff5:/$ /opt/bitnami/python/bin/pip install snowflake-connector-python
WARNING: The directory '/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting snowflake-connector-python
  Downloading snowflake_connector_python-2.2.4-cp36-cp36m-manylinux2010_x86_64.whl (12.3 MB)
     |████████████████████████████████| 12.3 MB 20.2 MB/s
Collecting requests<2.24.0
  Downloading requests-2.23.0-py2.py3-none-any.whl (58 kB)
     |████████████████████████████████| 58 kB 63.9 MB/s
Collecting oscrypto<2.0.0
  Downloading oscrypto-1.2.0-py2.py3-none-any.whl (192 kB)
     |████████████████████████████████| 192 kB 72.2 MB/s
Collecting idna<2.10
  Downloading idna-2.9-py2.py3-none-any.whl (58 kB)
     |████████████████████████████████| 58 kB 58.2 MB/s
Collecting pyOpenSSL<21.0.0,>=16.2.0
  Downloading pyOpenSSL-19.1.0-py2.py3-none-any.whl (53 kB)
     |████████████████████████████████| 53 kB 42.5 MB/s
Collecting botocore<1.15,>=1.5.0
  Downloading botocore-1.14.17-py2.py3-none-any.whl (5.9 MB)
     |████████████████████████████████| 5.9 MB 69.0 MB/s
Collecting pytz<2021.0
  Downloading pytz-2019.3-py2.py3-none-any.whl (509 kB)
     |████████████████████████████████| 509 kB 75.0 MB/s
Collecting certifi<2021.0.0
  Downloading certifi-2020.4.5.1-py2.py3-none-any.whl (157 kB)
     |████████████████████████████████| 157 kB 74.9 MB/s
Collecting cryptography<3.0.0,>=1.8.2
  Downloading cryptography-2.9-cp35-abi3-manylinux2010_x86_64.whl (2.7 MB)
     |████████████████████████████████| 2.7 MB 22.6 MB/s
Collecting boto3<1.12,>=1.4.4
  Downloading boto3-1.11.17-py2.py3-none-any.whl (128 kB)
     |████████████████████████████████| 128 kB 76.3 MB/s
Collecting cffi<1.14,>=1.9
  Downloading cffi-1.13.2-cp36-cp36m-manylinux1_x86_64.whl (397 kB)
     |████████████████████████████████| 397 kB 73.4 MB/s
Collecting azure-storage-blob<12.0.0
  Downloading azure_storage_blob-2.1.0-py2.py3-none-any.whl (88 kB)
     |████████████████████████████████| 88 kB 66.6 MB/s
Collecting pyjwt<2.0.0
  Downloading PyJWT-1.7.1-py2.py3-none-any.whl (18 kB)
Collecting urllib3<1.26.0,>=1.20
  Downloading urllib3-1.25.8-py2.py3-none-any.whl (125 kB)
     |████████████████████████████████| 125 kB 73.4 MB/s
Collecting asn1crypto<2.0.0,>0.24.0
  Downloading asn1crypto-1.3.0-py2.py3-none-any.whl (103 kB)
     |████████████████████████████████| 103 kB 77.8 MB/s
Collecting ijson<3.0.0
  Downloading ijson-2.6.1-cp36-cp36m-manylinux1_x86_64.whl (65 kB)
     |████████████████████████████████| 65 kB 56.3 MB/s
Collecting pycryptodomex!=3.5.0,<4.0.0,>=3.2
  Downloading pycryptodomex-3.9.7-cp36-cp36m-manylinux1_x86_64.whl (13.7 MB)
     |████████████████████████████████| 13.7 MB 24.0 MB/s
Collecting azure-common<2.0.0
  Downloading azure_common-1.1.25-py2.py3-none-any.whl (12 kB)
Collecting chardet<4,>=3.0.2
  Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)
     |████████████████████████████████| 133 kB 59.9 MB/s
Requirement already satisfied: six>=1.5.2 in /opt/bitnami/python/lib/python3.6/site-packages/six-1.14.0-py3.6.egg (from pyOpenSSL<21.0.0,>=16.2.0->snowflake-connector-python) (1.14.0)
Collecting docutils<0.16,>=0.10
  Downloading docutils-0.15.2-py3-none-any.whl (547 kB)
     |████████████████████████████████| 547 kB 43.2 MB/s
Collecting python-dateutil<3.0.0,>=2.1
  Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
     |████████████████████████████████| 227 kB 60.6 MB/s
Collecting jmespath<1.0.0,>=0.7.1
  Downloading jmespath-0.9.5-py2.py3-none-any.whl (24 kB)
Collecting s3transfer<0.4.0,>=0.3.0
  Downloading s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
     |████████████████████████████████| 69 kB 72.1 MB/s
Collecting pycparser
  Downloading pycparser-2.20-py2.py3-none-any.whl (112 kB)
     |████████████████████████████████| 112 kB 66.5 MB/s
Collecting azure-storage-common~=2.1
  Downloading azure_storage_common-2.1.0-py2.py3-none-any.whl (47 kB)
     |████████████████████████████████| 47 kB 61.4 MB/s
Installing collected packages: chardet, certifi, urllib3, idna, requests, asn1crypto, oscrypto, pycparser, cffi, cryptography, pyOpenSSL, docutils, python-dateutil, jmespath, botocore, pytz, s3transfer, boto3, azure-common, azure-storage-common, azure-storage-blob, pyjwt, ijson, pycryptodomex, snowflake-connector-python
Successfully installed asn1crypto-1.3.0 azure-common-1.1.25 azure-storage-blob-2.1.0 azure-storage-common-2.1.0 boto3-1.11.17 botocore-1.14.17 certifi-2020.4.5.1 cffi-1.13.2 chardet-3.0.4 cryptography-2.9 docutils-0.15.2 idna-2.9 ijson-2.6.1 jmespath-0.9.5 oscrypto-1.2.0 pyOpenSSL-19.1.0 pycparser-2.20 pycryptodomex-3.9.7 pyjwt-1.7.1 python-dateutil-2.8.1 pytz-2019.3 requests-2.23.0 s3transfer-0.3.3 snowflake-connector-python-2.2.4 urllib3-1.25.8

Did you install them successfully?

Thanks

marlonholland7 2020-04-14 18:12:22 UTC #10

Here's the error I see when I try to import the connector:

$ python3

import snowflake.connector

/opt/bitnami/airflow/venv/lib/python3.6/site-packages/pandas/compat/__init__.py: 85: UserWarning: Could not import the lzma module. Your installed Python is inco mplete. Attempting to use lzma compression will result in a RuntimeError.
warnings.warn(msg)

marlonholland7 2020-04-14 23:09:09 UTC #11

I've also tried using an operator that should be provided by the airflow version we're running (1.10.9), but still I'm getting this error:

rafaelrios 2020-04-16 16:27:34 UTC #12

Hi @marlonholland7,
I would like to do several checks:

Could you check you have this file /opt/bitnami/python/lib/python3.6/lzma.py ?
Could you check the path of the python you are running? which python3
Could you check where is python looking for the libs?
$ python3
Python 3.6.10 (default, Apr 7 2020, 12:52:28)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.

import sys
print ('\n'.join(sys.path))

/opt/bitnami/airflow/venv/lib/python36.zip
/opt/bitnami/airflow/venv/lib/python3.6
/opt/bitnami/airflow/venv/lib/python3.6/lib-dynload
/opt/bitnami/python/lib/python3.6
/opt/bitnami/airflow/venv/lib/python3.6/site-packages
/opt/bitnami/python/lib/python3.6/site-packages
/opt/bitnami/python/lib/python3.6/site-packages/setuptools-46.1.3-py3.6.egg
/opt/bitnami/python/lib/python3.6/site-packages/pip-20.0.2-py3.6.egg
/opt/bitnami/python/lib/python3.6/site-packages/virtualenv-20.0.15-py3.6.egg
/opt/bitnami/python/lib/python3.6/site-packages/importlib_metadata-1.6.0-py3.6.egg
/opt/bitnami/python/lib/python3.6/site-packages/importlib_resources-1.4.0-py3.6.egg
/opt/bitnami/python/lib/python3.6/site-packages/six-1.14.0-py3.6.egg
/opt/bitnami/python/lib/python3.6/site-packages/filelock-3.0.12-py3.6.egg
/opt/bitnami/python/lib/python3.6/site-packages/distlib-0.3.0-py3.6.egg
/opt/bitnami/python/lib/python3.6/site-packages/appdirs-1.4.3-py3.6.egg
/opt/bitnami/python/lib/python3.6/site-packages/zipp-3.1.0-py3.6.egg

marlonholland7 2020-04-16 19:12:18 UTC #13

Hi @rafaelrios,

Yes, the file exists in the path given.
The path of the python I'm running is in:

$ which python3
/opt/bitnami/airflow/venv/bin/python3
This is where python is looking for the libs:

import sys
print ('\n'.join(sys.path))

/opt/bitnami/airflow/venv/lib/python36.zip
/opt/bitnami/airflow/venv/lib/python3.6
/opt/bitnami/airflow/venv/lib/python3.6/lib-dynload
/opt/bitnami/python/lib/python3.6
/opt/bitnami/airflow/venv/lib/python3.6/site-packages
/opt/bitnami/python/lib/python3.6/site-packages
/opt/bitnami/python/lib/python3.6/site-packages/setuptools-41.2.0-py3.6.egg
/opt/bitnami/python/lib/python3.6/site-packages/pip-19.2.3-py3.6.egg
/opt/bitnami/python/lib/python3.6/site-packages/virtualenv-16.7.6-py3.6.egg

Please let me know if I can provide more details to help. Thanks!

rafaelrios 2020-04-20 17:42:51 UTC #14

Hi,
This is quite strange.
The error is in the import that pandas does.

Could you run?
$ python3

import
lzma.decompress(lzma.compress(b"Hello!"))

marlonholland7 2020-04-20 21:03:41 UTC #17

This gives me the following output:

b'Hello!'

rafaelrios 2020-04-21 14:58:47 UTC #18

So it is using correctly the lzma lib that is complaining in the dag.

Could you provide a minimal DAG that fails so I can try to reproduce the issue ?

marlonholland7 2020-04-21 17:53:25 UTC #22

Also, if it helps, I started an additional multi-tier package for testing.

I ran the commands:

sudo /opt/bitnami/python/bin/pip install snowflake-sqlalchemy
sudo /opt/bitnami/python/bin/pip install snowflake-connector-python

Then I tested this in the terminal:

from airflow.contrib.hooks.snowflake_hook import SnowflakeHook
from airflow.contrib.operators.snowflake_operator import SnowflakeOperator

And that worked with no issues. But my DAG that is using those two imports is still returning this error:

marlonholland7 2020-04-22 02:06:24 UTC #24

from datetime import datetime
from airflow import DAG
from airflow.operators.dummy_operator import DummyOperator
from airflow.operators.python_operator import PythonOperator
from airflow.contrib.hooks.snowflake_hook import SnowflakeHook
from airflow.contrib.operators.snowflake_operator import SnowflakeOperator

def print_hello():
    return 'Hello world!'

dag = DAG('hello_world', description='Simple tutorial DAG',
          schedule_interval='0 12 * * *',
          start_date=datetime(2017, 3, 20), catchup=False)

dummy_operator = DummyOperator(task_id='dummy_task', retries=3, dag=dag)

hello_operator = PythonOperator(task_id='hello_task', python_callable=print_hello, dag=dag)

dummy_operator >> hello_operator

marlonholland7 2020-04-22 03:37:56 UTC #25

So after running these commands, where should I be seeing these python libraries?

marlonholland7 2020-04-22 03:47:16 UTC #26

Is there supposed to be a mismatch here?

 import os
 import snowflake
 import inspect
 inspect.getfile(os)
'/opt/bitnami/airflow/venv/lib/python3.6/os.py'
 print(snowflake.__path__)
_NamespacePath(['/opt/bitnami/python/lib/python3.6/site-packages/snowflake'])

marlonholland7 2020-04-22 16:41:02 UTC #27

Hi @jota,

I've started multiple new services to test:

sudo /opt/bitnami/python/bin pip install snowflake-sqlalchemy
/opt/bitnami/python/bin/pip install snowflake-sqlalchemy
sudo /opt/bitnami/airflow/venv/bin/pip install snowflake-sqlalchemy

The DAG is unable to import the snowflake module on all three of these instances still.

rafaelrios 2020-04-22 17:56:41 UTC #28

Hi @marlonholland7,
Thanks for the little python script showing the paths.
Could you try the following ?
- Login into the server
- Run:

$ sudo su -
# .  /opt/bitnami/airflow/venv/bin/activate
(venv) # pip install snowflake
...
(venv) # pip install snowflake-connector-python

marlonholland7 2020-04-22 18:18:48 UTC #29

I ran the commands provided:

bitnami@nttairflowdev0:~$ sudo su -
[sudo] password for bitnami:
root@nttairflowdev0:~# . /opt/bitnami/airflow/venv/bin/activate
(venv) root@nttairflowdev0:~# pip install snowflake
Collecting snowflake
  Downloading snowflake-0.0.3.tar.gz (2.1 kB)
Building wheels for collected packages: snowflake
  Building wheel for snowflake (setup.py) ... done
  Created wheel for snowflake: filename=snowflake-0.0.3-py3-none-any.whl size=29                                                                                                                                                             67 sha256=08275ca62376f970080b6ab508db562e47169acade280db7f31c8dc566ef1ff4
  Stored in directory: /root/.cache/pip/wheels/06/07/1f/67c1209a5c6b1adf7bfac378                                                                                                                                                             abd403086de02d50b0a17d5fbd
Successfully built snowflake
Installing collected packages: snowflake
Successfully installed snowflake-0.0.3
(venv) root@nttairflowdev0:~# pip install snowflake-connector-python
Requirement already satisfied: snowflake-connector-python in /opt/bitnami/airflow/venv/lib/python3.6/site-packages (2.2.4)
Requirement already satisfied: pycryptodomex!=3.5.0,<4.0.0,>=3.2 in /opt/bitnami/airflow/venv/lib/python3.6/site-packages (from snowflake-connector-python) (3.9.7)
Requirement already satisfied: idna<2.10 in /opt/bitnami/airflow/venv/lib/python3.6/site-packages (from snowflake-connector-python) (2.8)
Requirement already satisfied: pyjwt<2.0.0 in /opt/bitnami/airflow/venv/lib/python3.6/site-packages (from snowflake-connector-python) (1.7.1)
Requirement already satisfied: botocore<1.15,>=1.5.0 in /opt/bitnami/airflow/venv/lib/python3.6/site-packages (from snowflake-connector-python) (1.14.17)
Requirement already satisfied: urllib3<1.26.0,>=1.20 in /opt/bitnami/airflow/venv/lib/python3.6/site-packages (from snowflake-connector-python) (1.25.8)
Requirement already satisfied: asn1crypto<2.0.0,>0.24.0 in /opt/bitnami/airflow/venv/lib/python3.6/site-packages (from snowflake-connector-python) (1.3.0)
Requirement already satisfied: pyOpenSSL<21.0.0,>=16.2.0 in /opt/bitnami/airflow/venv/lib/python3.6/site-packages (from snowflake-connector-python) (19.1.0)
Requirement already satisfied: cffi<1.14,>=1.9 in /opt/bitnami/airflow/venv/lib/python3.6/site-packages (from snowflake-connector-python) (1.13.2)
Requirement already satisfied: requests<2.24.0 in /opt/bitnami/airflow/venv/lib/python3.6/site-packages (from snowflake-connector-python) (2.22.0)
Requirement already satisfied: azure-storage-blob<12.0.0 in /opt/bitnami/airflow/venv/lib/python3.6/site-packages (from snowflake-connector-python) (2.1.0)
Requirement already satisfied: ijson<3.0.0 in /opt/bitnami/airflow/venv/lib/python3.6/site-packages (from snowflake-connector-python) (2.6.1)
Requirement already satisfied: oscrypto<2.0.0 in /opt/bitnami/airflow/venv/lib/python3.6/site-packages (from snowflake-connector-python) (1.2.0)
Requirement already satisfied: pytz<2021.0 in /opt/bitnami/airflow/venv/lib/python3.6/site-packages (from snowflake-connector-python) (2019.3)
Requirement already satisfied: cryptography<3.0.0,>=1.8.2 in /opt/bitnami/airflow/venv/lib/python3.6/site-packages (from snowflake-connector-python) (2.8)
Requirement already satisfied: certifi<2021.0.0 in /opt/bitnami/airflow/venv/lib/python3.6/site-packages (from snowflake-connector-python) (2019.11.28)
Requirement already satisfied: azure-common<2.0.0 in /opt/bitnami/airflow/venv/lib/python3.6/site-packages (from snowflake-connector-python) (1.1.25)
Requirement already satisfied: boto3<1.12,>=1.4.4 in /opt/bitnami/airflow/venv/lib/python3.6/site-packages (from snowflake-connector-python) (1.11.17)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /opt/bitnami/airflow/venv/lib/python3.6/site-packages (from botocore<1.15,>=1.5.0->snowflake-connector-python) (0.9.4)
Requirement already satisfied: docutils<0.16,>=0.10 in /opt/bitnami/airflow/venv/lib/python3.6/site-packages (from botocore<1.15,>=1.5.0->snowflake-connector-python) (0.15.2)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /opt/bitnami/airflow/venv/lib/python3.6/site-packages (from botocore<1.15,>=1.5.0->snowflake-connector-python) (2.8.1)
Requirement already satisfied: six>=1.5.2 in /opt/bitnami/airflow/venv/lib/python3.6/site-packages (from pyOpenSSL<21.0.0,>=16.2.0->snowflake-connector-python) (1.14.0)
Requirement already satisfied: pycparser in /opt/bitnami/airflow/venv/lib/python3.6/site-packages (from cffi<1.14,>=1.9->snowflake-connector-python) (2.19)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /opt/bitnami/airflow/venv/lib/python3.6/site-packages (from requests<2.24.0->snowflake-connector-python) (3.0.4)
Requirement already satisfied: azure-storage-common~=2.1 in /opt/bitnami/airflow/venv/lib/python3.6/site-packages (from azure-storage-blob<12.0.0->snowflake-connector-python) (2.1.0)
Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /opt/bitnami/airflow/venv/lib/python3.6/site-packages (from boto3<1.12,>=1.4.4->snowflake-connector-python) (0.3.3)

I then restarted the airflow server and I'm still seeing this error:

marlonholland7 2020-04-22 23:01:52 UTC #30

Even though I ran the commands provided, and the path of my library installed in /opt/bitnami/airflow/venv on my webserver node, I checked my worker nodes and the path there automatically installed to /opt/bitnami/python/lib/python3.6. /site-packages

Why does this happen, and could this be the source of the issue?

marlonholland7 2020-04-23 00:09:35 UTC #31

I did some more testing and this isn't the source of the issue. I may just have to use another airflow source because we need the snowflake module to work and it's not working on this bitnami instance.

Bitnami provides free all-in-one installers, virtual machines and cloud images for popular open source applications. Get them now!