Apache Airflow Multi-Tier: DAGS not updating based on Public Repository provided during set up

marlonholland7 2020-04-10 01:51:17 UTC #1

Keywords: Apache Airflow - Microsoft Azure - How to - Other
Description:
Hi!

When setting up my bitnami Apache Airflow Multi-Tier application, I provided a public repository. However, it doesn't look like my airflow instance is pulling in DAGS that I added to the repository.

Is there a default refresh time on this, and if so, is there a way to change that? Please let me know if I need to provide more details.

abjimenez 2020-04-10 10:52:32 UTC #2

Hi @marlonholland7,

The dags folder is /opt/bitnami/airflow/dags. You should have the repository cloned there. Can you check it?

There is one cron job that updates the repository every minute.

Please, let us know if the folder mentioned is up-to-date to know if that's the source of the issue.

Best regards,
Andrés Bono.

marlonholland7 2020-04-10 18:28:15 UTC #3

Hi,

When I check that folder, this is what I see:

Keep in mind I'm relatively inexperienced with these kinds of issues, so I may need more explanation than your average..

Thanks.

marlonholland7 2020-04-10 22:49:13 UTC #5

I'm heading in the right direction...I now have my /opt/bitnami/airflow/dags directory cloning my repository. See here:

However, the DAGS folder isn't updating correctly.

I execute 'crontab -e' then add the line:

cd /opt/bitnami/airflow/dags && git pull

But it's not updating..

abjimenez 2020-04-13 09:03:31 UTC #6

Hi @marlonholland7,

We have found what should be the source of the issue while checking your ticket:

$ sudo cat /var/log/syslog | grep -i cron
Apr 13 08:26:01 localhost cron[396]: (*system*) RELOAD (/etc/crontab)
Apr 13 08:26:01 localhost cron[396]: (*system*airflow_dags_update) ERROR (Missing newline before EOF, this crontab file will be ignored)

We will work on a fix for this and we will release a new version of the solution.

In the meantime, you can add an empty new line at the end of the /etc/cron.d/airflow_dags_update file on every VM that you are running. That should solve the issue.

Let us know if that works for you.

Best regards,
Andrés Bono.

Install Python Dependencies to Azure Bitnami Airflow Multi-Tier Package

marlonholland7 2020-04-13 16:17:22 UTC #9

Hi @abjimenez,

I found the solution the other day. I simply followed the directions found here: https://docs.bitnami.com/azure-templates/infrastructure/apache-airflow/configuration/sync-dags/

HOWEVER, this solution wasn't working. After some testing I discovered that the user needs to use:

$ sudo crontab -e

Then add a line in there to update the DAGS folder on a timed basis. For example, I added:

cd /opt/bitnami/airflow/dags && git pull

And had it do this on a minute by minute basis.

Just using:

$ crontab -e

Won't do anything because of permissions.

Also, this only needs to be applied to the primary virtual machine (the one ending with '0').

abjimenez 2020-04-14 13:27:59 UTC #10

Hi @marlonholland7,

Thanks for sharing your solution with the community!

We will be releasing a new version of Airflow in the following days so no manual actions need to be done.

Thanks again!

Best regards,
Andrés Bono.

marlonholland7 2020-04-18 00:43:25 UTC #11

IMPORTANT FYI - Those directions didn't work for me because I wasn't using the bitnami user. Looking back on it - the page's directions work perfectly. Was my fault.

abjimenez 2020-04-20 07:55:53 UTC #12

Thanks for letting us know, @marlonholland7!

Best regards,
Andrés Bono.

abjimenez 2020-04-22 14:28:28 UTC #13

Hi,

We have released a new version of Airflow Multi-Tier that fixes this issue. It will be available in the Azure Marketplace in the following days.

Best regards,
Andrés Bono.

abjimenez 2020-04-22 14:28:32 UTC #14

Bitnami provides free all-in-one installers, virtual machines and cloud images for popular open source applications. Get them now!