Content-Length: 359903 | pFad | http://github.com/apache/airflow/pull/50880

24 Deserialize should work while retrieving variables with secrets backend by amoghrajesh · Pull Request #50880 · apache/airflow · GitHub
Skip to content

Deserialize should work while retrieving variables with secrets backend #50880

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
May 21, 2025
Merged

Deserialize should work while retrieving variables with secrets backend #50880

merged 2 commits into from
May 21, 2025

Conversation

amoghrajesh
Copy link
Contributor

closes: #50666

Problem

As reported in #50666, deserialize_json option wasn't working when trying to use it with Variable.get with AWS secrets manager.

But this problem on further investigation revealed to be not just for AWS secrets backend but a bug in the Variable module itself. The bug didn't honor the deserialize_json flag at all when it was passed for secrets backend which shouldn't be the case at all.

To prove this, did the following:

Set up a airflow.secrets.local_filesystem.LocalFilesystemBackend and checked with both Airflow 2 and Airflow 3.

var.json

{
    "A": "{\"num1\": 23, \"num2\": 42}"
}

Dag used:

from __future__ import annotations

from airflow.models import Variable
from airflow.models.baseoperator import BaseOperator
from airflow import DAG
# from airflow.sdk import Variable

class CustomOperator(BaseOperator):
    def execute(self, context):
        v = Variable.get(key="A")
        print("Got value", v, type(v))


        v1 = Variable.get(key="A", deserialize_json=True)
        print("Got value from deser", v1, type(v1))


with DAG("get-var-sdk", schedule=None, catchup=False) as dag:
    CustomOperator(task_id="set_var")

Airflow 2:

# With secrets backned
# [2025-05-20, 16:38:47 IST] {logging_mixin.py:190} INFO - Got value {"num1": 23, "num2": 42} <class 'str'>
# [2025-05-20, 16:38:47 IST] {logging_mixin.py:190} INFO - Got value from deser {'num1': 23, 'num2': 42} <class 'dict'>

Airflow 3:

[2025-05-21, 15:35:41] INFO - Got value {"num1": 23, "num2": 42} <class 'str'>: chan="stdout": source="task"
[2025-05-21, 15:35:41] INFO - Got value from deser {"num1": 23, "num2": 42} <class 'str'>: chan="stdout": source="task"

As observed the second print for Airflow 3 is a string when it should really be a dictionary.

Testing

Set up a airflow.secrets.local_filesystem.LocalFilesystemBackend and checked with both Airflow 2 and Airflow 3.

var.json

{
    "A": "{\"num1\": 23, \"num2\": 42}"
}

And exporting these variables in init.sh of breeze:

export AIRFLOW__SECRETS__BACKEND="airflow.secrets.local_filesystem.LocalFilesystemBackend"
export AIRFLOW__SECRETS__BACKEND_KWARGS='{"connections_file_path": "/files/conn.json", "variables_file_path": "/files/var.json"}'

Dag used:

from __future__ import annotations

from airflow.models import Variable
from airflow.models.baseoperator import BaseOperator
from airflow import DAG
# from airflow.sdk import Variable

class CustomOperator(BaseOperator):
    def execute(self, context):
        v = Variable.get(key="A")
        print("Got value", v, type(v))


        v1 = Variable.get(key="A", deserialize_json=True)
        print("Got value from deser", v1, type(v1))


with DAG("get-var-sdk", schedule=None, catchup=False) as dag:
    CustomOperator(task_id="set_var")

Before changes:

image

After changes:

image

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@amoghrajesh amoghrajesh requested review from ashb and kaxil as code owners May 21, 2025 10:30
@amoghrajesh amoghrajesh requested a review from vatsrahul1001 May 21, 2025 10:30
@amoghrajesh amoghrajesh added the backport-to-v3-0-test Mark PR with this label to backport to v3-0-test branch label May 21, 2025
@eladkal eladkal added this to the Airflow 3.0.2 milestone May 21, 2025
@eladkal eladkal added the type:bug-fix Changelog: Bug Fixes label May 21, 2025
@amoghrajesh amoghrajesh changed the title Deserialize should work while retrieving variables with secrects backend Deserialize should work while retrieving variables with secrets backend May 21, 2025
ashb
ashb reviewed May 21, 2025
View reviewed changes
Copy link
Member

@ashb ashb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does {{ var.json.myvar }} still work as it did before?

@amoghrajesh
Copy link
Contributor Author

@ashb yeah it works like it used to.

Tried with:

Variable.set("key", "{\"num1\": 23, \"num2\": 42}")

get_var_task = BashOperator(
    task_id="get_variable_with_template",
    bash_command="echo {{ var.value.key }} && echo {{ var.json.key }}",
    dag=dag
)

Output:

[2025-05-21, 18:28:46] INFO - Running command: ['/usr/bin/bash', '-c', 'echo {"num1": 23, "num2": 42} && echo {\'num1\': 23, \'num2\': 42}']: source="airflow.task.hooks.airflow.providers.standard.hooks.subprocess.SubprocessHook"
[2025-05-21, 18:28:46] INFO - Output:: source="airflow.task.hooks.airflow.providers.standard.hooks.subprocess.SubprocessHook"
[2025-05-21, 1[8](http://localhost:28080/dags/set_get_airflow_variable/runs/manual__2025-05-21T12:58:45.415124+00:00/tasks/get_variable_with_template?try_number=1#8):28:46] INFO - {num1: 23, num2: 42}: source="airflow.task.hooks.airflow.providers.standard.hooks.subprocess.SubprocessHook"
[2025-05-21, 18:28:46] INFO - {num1: 23, num2: 42}: source="airflow.task.hooks.airflow.providers.standard.hooks.subprocess.SubprocessHook"

@amoghrajesh
Copy link
Contributor Author

Unrelated failures, merging.

@amoghrajesh amoghrajesh merged commit 6a67806 into apache:main May 21, 2025
70 of 71 checks passed
@amoghrajesh amoghrajesh deleted the fix-secrets-backend-deser branch May 21, 2025 13:00
github-actions bot pushed a commit that referenced this pull request May 21, 2025
@amoghrajesh
[v3-0-test] Deserialize should work while retrieving variables with s…
c13f5fb 
…ecrets backend (#50880)

(cherry picked from commit 6a67806)

Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>
@github-actions GitHub Actions
Copy link

Backport successfully created: v3-0-test

Status Branch Result
✅ v3-0-test PR Link

@amoghrajesh amoghrajesh mentioned this pull request May 21, 2025
Open
kaxil pushed a commit that referenced this pull request May 21, 2025
@github-actions @amoghrajesh
[v3-0-test] Deserialize should work while retrieving variables with s…
e824f7e 
…ecrets backend (#50880) (#50889)

(cherry picked from commit 6a67806)

Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>
nailo2c pushed a commit to nailo2c/airflow that referenced this pull request May 21, 2025
@amoghrajesh @nailo2c
Deserialize should work while retrieving variables with secrets backe…
ff79395 
…nd (apache#50880)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ashb ashb ashb left review comments

@kaxil kaxil kaxil approved these changes

@eladkal eladkal eladkal approved these changes

@vatsrahul1001 vatsrahul1001 Awaiting requested review from vatsrahul1001

Assignees
No one assigned
Labels
area:task-sdk backport-to-v3-0-test Mark PR with this label to backport to v3-0-test branch type:bug-fix Changelog: Bug Fixes
Projects
None yet
Milestone
Airflow 3.0.2
Development

Successfully merging this pull request may close these issues.

Automatic JSON deserialization of variables not working with AWS Secrets Manager secrets backend
4 participants
@amoghrajesh @ashb @kaxil @eladkal








ApplySandwichStrip

pFad - (p)hone/(F)rame/(a)nonymizer/(d)eclutterfier!      Saves Data!


--- a PPN by Garber Painting Akron. With Image Size Reduction included!

Fetched URL: http://github.com/apache/airflow/pull/50880

Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy