Azure module not found error in Airflow UI

veemvve 2020-07-07 21:26:25 UTC #1

Keywords: Apache Airflow - Microsoft Azure - Technical issue - Other
Description:
I'm using Apache Airflow Multi Tier Bitnami available in Azure Marketplace.

As part of my DAG, I'm referring azure module to connect to Key Vault. To achieve that, I installed azure module in VM (Linux) with sudo /opt/bitnami/python/bin/pip install azure-keyvault command.

But I keep on ending up with below error in Airflow UI,

Broken DAG: [/opt/bitnami/airflow/dags/DatabricksJobExecution.py] No module named 'azure'

What am I missing and how I can fix it?

Thanks!

gongomgra 2020-07-08 08:47:00 UTC #3

Hi @veemvve,

Thanks for using Bitnami. There is an azure package for pip. Can you try to install it as well?

sudo /opt/bitnami/python/bin/pip install azure

veemvve 2020-07-08 17:20:50 UTC #4

Hi @gongomgra,

Thanks for the response!

I already tried that one and it didn't help since azure package as whole is deprecated starting with v5.0.0 and we must go with specific packages. In my case, I'm interested on azure-keyvault-secrets package which I already installed on all my VMs (Primary + Workers) but no luck.

RuntimeError:

Starting with v5.0.0, the 'azure' meta-package is deprecated and cannot be installed anymore.
Please install the service specific packages prefixed by `azure` needed for your application.

azure-mgmt-compute (https://pypi.python.org/pypi/azure-mgmt-compute) : Management of Virtual Machines, etc.
azure-mgmt-storage (https://pypi.python.org/pypi/azure-mgmt-storage) : Management of storage accounts.
azure-mgmt-resource (https://pypi.python.org/pypi/azure-mgmt-resource) : Generic package about Azure Resource Management (ARM)
azure-keyvault-secrets (https://pypi.python.org/pypi/azure-keyvault-secrets) : Access to secrets in Key Vault
azure-storage-blob (https://pypi.python.org/pypi/azure-storage-blob) : Access to blobs in storage accounts

After installing azure-keyvault-secrets package, I do see azure folder under site-packages.

gongomgra 2020-07-09 10:19:03 UTC #5

Hi @veemvve,

I'm afraid I don't know what can be issue here, but it seems to be related with Airflow and the azure-keyvault-secrets package. Can you ask in the Airflow official forums or the azure-keyvault-secrets module developers for help on this? Note they can give you better support on their product than we can

https://airflow.apache.org/community/

veemvve 2020-07-09 21:46:52 UTC #6

Thanks @gongomgra

Will get in touch with Airflow and Azure team on the import. Couple of other questions,

During Bitnami installation, what are all sub-packages (e.g. azure, aws, etc.,) gets enabled? https://airflow.apache.org/docs/stable/installation.html
How to import custom classes to your DAG? I mean where we have to place the custom class files. I tried placing it along with the DAG files in Azure File Share (shareddisk) but no luck and ending up with 'No module found error'

gongomgra 2020-07-10 12:25:20 UTC #7

Hi @veemvve,

I think we are using a virtualenv to install the Airflow dependencies. Can you check if the /opt/bitnami/airflow/venv directory exists? I'm pretty sure it does, but I don't have a machine to check it myself at this moment, sorry.

If that folder exists, you can load it and check the installed packages running the next commands (note there is a dot+white-space at the beginning of the first command)

. /opt/bitnami/airflow/venv/bin/activate
pip list

Regarding the question of adding your custom classes, maybe you need to install them as part of the virtualenv, and that once you have loaded it in the console, you can try to import it with the next command

python -c "import my_module"

But it is a good idea to ask the Airflow developers about this as well, just in case I'm missing something here.

system 2020-07-24 12:25:25 UTC #8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Bitnami provides free all-in-one installers, virtual machines and cloud images for popular open source applications. Get them now!