Azure blob remote logging using the official helm chart

[jaruji]

Hello everyone,
wonder if anyone would be able to point me to the format the WASB connection env variable should have if I want to use a blob connection string to authenticate? In the example at https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/stable/connections/wasb.html, the example connection string is set like this:

export AIRFLOW_CONN_WASB_DEFAULT='wasb://blob%20username:blob%[email protected]?tenant_id=tenant+id'

I can set this enviroment variable using the helm chart - but what exact format would this string have if I want to use the azure blob connection string as the method of authentication instead of username, password and tennantId?

[potiuk]

No idea about Azure, but the page you linked above contains all the parameters that you can set and Airflow has a helper that allows you to generate URI based on connection parameters + extras:
https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html#uri-format

If in doubt - look at the code of the Hook and figure out the parameters it needs.

[jaruji]

Thank you for the link, I will take a look and see whether I can figure it out.

[jaruji]

Ok so I did check out the airflow helper for generating URI's, and got the following one:

AIRFLOW_CONN_WASB_CONNECTION='wasb:///?connection_string=BLOB_CONNECTION_STRING'

Now I need to attach this connection to the airflow helm chart - according to the docs (https://airflow.apache.org/docs/helm-chart/stable/adding-connections-and-variables.html), you need to pass it through the secret parameter in the values.yaml of the chart.

kubectl create secret generic airflow-wasb-secret --from-literal "AIRFLOW_CONN_WASB_CONNECTION='BLOB_CONNECTION_STRING'" -n airflow

  secret:
    - envName: "AIRFLOW_CONN_WASB_CONNECTION"
      secretName: "airflow-wasb-secret"
      secretKey: "AIRFLOW_CONN_WASB_CONNECTION"

But when I pass it, the connection does not appear in the UI nor in the postgres connection table. Any ideas why?

[Unb0rn]

@jaruji @dada-engineer I have the same issue. Did you find a solution? Like how should this "BLOB_CONNECTION_STRING" be formatted? I've tried with the connstring from Access Keys section:
DefaultEndpointsProtocol=https;AccountName=<storageaccount>;AccountKey=<base64key>;EndpointSuffix=core.windows.net
But it doesn't work for me.
If someone has the full envvar example that would be awesome!

[dada-engineer]

sorry @Unb0rn forgot to reply here, We did manage to make it work with the following config (this is a terraform template rendered so this is terraform template language:

AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER: wasb/
AIRFLOW__AZURE_REMOTE_LOGGING__REMOTE_WASB_LOG_CONTAINER: ${blob_container_name}
AIRFLOW__CORE__LOGGING_LEVEL: DEBUG
AIRFLOW_CONN_REMOTE_LOGGING: '{"conn_type": "wasb", "login": "${storage_account}", "password": "${account_key}"}'

The odd part is the AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER MUST start with wasb meaning in our blob container which is called logs we end up with another wasb/ folder...

Hope this helps.

This has been the only way I got it working for now.

[Unb0rn]

You're a lifesaver) Thank you!

[tcarr-kr]

I've gotten to the point where I can store logs in an Azure Blob (with the same wasb: as a prefix, which sounds like a bug to me), but I haven't been able to read them back via the web UI. I'm doing this on a Mac at the moment so there is always the possibility of an issue there, but when I try to read a task's log a child Python process SEGVs-- I get the Mac crash reporter window which tells me the child in a fork died before exec. So I have yet to close the circle on this. I don't have access to a Linux dev machine unfortunately, and I do worry this is some Mac/Python bad mojo going on. Anyone been able to read their logs in the web ui? Or are running this on a Mac?

[dada-engineer]

I only host this in k8s running the official airflow docker image (okay the official one is the base as we pack our dags in there) without any issue. Can you try to run this via docker / docker-compose on the mac?

[pawgajda-drs]

If anyone's stumbles upon this issue when trying to make it work, here are my notes.

1. First of all, when running Helm Chart version 1.18.0, you're on Airflow 3.0.2. Override your images to 3.0.6, otherwise it just won't work at all:

images:
  airflow:
    repository: apache/airflow
    tag: "3.0.6"
  pod_template:
    repository: apache/airflow
    tag: "3.0.6"

2. Configure WASB/Azure Blob Storage Conenction via Airflow UI or via Environment Variable.
   More information here: https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html#storing-connections-in-environment-variables

If you want to configure it via Environment Variable, I recommend this format:

AIRFLOW_CONN_WASB_LOGGING='{"conn_type": "wasb", "host": "__STORAGE_ACCOUNT_HOST__", "login": "__STORAGE_ACCOUNT_NAME__", "password": "__STORAGE_ACCOUNT_SHARED_KEY__"}'

bare minimum version:

AIRFLOW_CONN_WASB_LOGGING='{"conn_type": "wasb", "login": "__STORAGE_ACCOUNT_NAME__", "password": "__STORAGE_ACCOUNT_SHARED_KEY__"}'

I recommend setting it up as Secret then pass it to Helm Chart like this:

extraEnvFrom: |
  - secretRef:
      name: airflow-remote-logging-secret

Finally, the configuration itself:

env:
  # Enable Remote Logging
  - name: "AIRFLOW__LOGGING__REMOTE_LOGGING"
    value: "True"
  # Delete Local Logs when Remote Logging is Enabled
  - name: "AIRFLOW__LOGGING__DELETE_LOCAL_LOGS"
    value: "True"
  # Directory inside Blob Container
  # needs to start with 'wasb' or 'wasb/' or 'wasb://'
  - name: "AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER"
    value: "wasb/logs"
  # Blob Container Name
  - name: "AIRFLOW__AZURE_REMOTE_LOGGING__REMOTE_WASB_LOG_CONTAINER"
    value: "airflow-logs"
  # Auth for Blob Storage
  # The naming convention is AIRFLOW_CONN_{CONN_ID} (lowercase value here)
  - name: "AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID"
    value: "wasb_logging"

A few notes:
Regarding AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER value, documentation here: https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/stable/logging/index.html says to use

wasb://path/to/logs

format, but for version 3.0.6 do wasb/path/to/logs, otherwise your path will look like wasb:/noname/path/to/logs.

Regarding AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID set it to lowercase name of your variable or the name of connection you created via UI, that's it.

Thanks to everyone who participated in this discussion and other discussion related to this topic:
#54192
#54413
EDIT:
and #54489