Skip to content

Add dag_run_conf to RunBackfillForm #49763

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Apr 30, 2025
Merged

Add dag_run_conf to RunBackfillForm #49763

merged 2 commits into from
Apr 30, 2025

Conversation

guan404ming
Copy link
Contributor

@guan404ming guan404ming commented Apr 25, 2025 •
edited
Loading

Related Issue

closes #49653
cc: @bbovenzi

Why

  • currently the RunBackfillForm has no place to input the dag_run_conf however it's needed

How

  • add JsonEditor in a Accordion to let user input dag_run_conf:

image

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@boring-cyborg boring-cyborg bot added the area:UI Related to UI/UX. For Frontend Developers. label Apr 25, 2025
@bbovenzi bbovenzi added this to the Airflow 3.0.1 milestone Apr 25, 2025
@bbovenzi
Copy link
Contributor

bbovenzi commented Apr 25, 2025 •
edited
Loading

Good first step! But actually, we need the whole FlexibleForm to be available. The json input text is just the fallback if the user doesn't have params specified.

I think we really just need to pull all the conf/params form out of TriggerDAGForm into a shared component to add to the Trigger/Backfill forms.

My apologies for failing to specify that

@guan404ming
Copy link
Contributor Author

No worries! I understand and I can try to make pull that form as a shared components
Thanks for clarifying~

@jscheffl
Copy link
Contributor

I am not a big user of backfills and therefore might not have an insight... but:
Backfill is to trigger runs that had not been executed in the past. They logically should be "similar" to cases when a scheduler would have executed them.
If DAGs are scheduled, then all defined parameters are made with their default values - scheduled jobs do not have a trigger form.

Is it a "we can and therefore we make it" or is there a real functional need to be able to change params when making a backfill? Because it might be different to scheduled runs.

@jscheffl
Copy link
Contributor

Ah, okay looking at https://airflow.apache.org/docs/apache-airflow/2.10.5/cli-and-env-variables-ref.html#backfill ... also previously a conf could be defined to be passed.

@guan404ming guan404ming force-pushed the add-backfill-conf branch 2 times, most recently from d5f6ac4 to fd55ed0 Compare April 26, 2025 03:45
@guan404ming
Copy link
Contributor Author

guan404ming commented Apr 26, 2025 •
edited
Loading

@bbovenzi I've updated the PR, but I'm not sure that the way how I pass the param is right or not, please let me know if this need any change. Thanks!

bbovenzi
bbovenzi reviewed Apr 28, 2025
View reviewed changes
Copy link
Contributor

@bbovenzi bbovenzi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So for advanced options on a backfill, we only ever want the dag_run_conf. No notes, no dag run id and no logical date.

Screenshot 2025-04-28 at 11 33 13 AM

@guan404ming
Copy link
Contributor Author

Shall we need to hide the field notes, dag run id and logical date in BackFillForm?

@guan404ming
Copy link
Contributor Author

Oh, I got it.
You mean only pull out the conf/params form which is not including the notes, dag run id and logical date since they're specifically belongs to dag run. I misunderstand what you mean before, I would make a change tomorrow. If my understanding still have some error, please let me know. Thanks a lots~

@bbovenzi
Copy link
Contributor

Oh, I got it. You mean only pull out the conf/params form which is not including the notes, dag run id and logical date since they're specifically belongs to dag run. I misunderstand what you mean before, I would make a change tomorrow. If my understanding still have some error, please let me know. Thanks a lots~

Exactly. Only the conf can be shared across each backfill run. Thanks for your patience on the back and forth here.

@guan404ming
Copy link
Contributor Author

guan404ming commented Apr 28, 2025 •
edited
Loading

No worries, let me try fix it~
BTW I think the doc is currently lack of details of what we allow user to pass in Backfill conf, maybe we could try make it more complete.

@guan404ming guan404ming force-pushed the add-backfill-conf branch 5 times, most recently from 14b04f2 to 921ff67 Compare April 29, 2025 16:54
@guan404ming guan404ming requested a review from bbovenzi April 29, 2025 23:18
@guan404ming
Copy link
Contributor Author

@bbovenzi I've fixed this one, please take another look. Thanks!

guan404ming and others added 2 commits May 1, 2025 00:58
@guan404ming @bbovenzi
feat: add dag_run_conf to RunBackfillForm
183f58a 
Co-authored-by: Brent Bovenzi <4600967+bbovenzi@users.noreply.github.com>
@guan404ming @bbovenzi
fix: add missing conf preset
74d4f19 
Co-authored-by: Brent Bovenzi <4600967+bbovenzi@users.noreply.github.com>
@guan404ming guan404ming changed the title feat: add dag_run_conf to RunBackfillForm Add dag_run_conf to RunBackfillForm Apr 30, 2025
bbovenzi
bbovenzi approved these changes Apr 30, 2025
View reviewed changes
Copy link
Contributor

@bbovenzi bbovenzi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great work!

@bbovenzi bbovenzi merged commit 6ca99d8 into apache:main Apr 30, 2025
42 checks passed
@guan404ming guan404ming deleted the add-backfill-conf branch May 2, 2025 04:33
kaxil pushed a commit that referenced this pull request May 6, 2025
@guan404ming @bbovenzi @kaxil
Add dag_run_conf to RunBackfillForm (#49763)
93960d8 
* feat: add `dag_run_conf` to `RunBackfillForm`

Co-authored-by: Brent Bovenzi <4600967+bbovenzi@users.noreply.github.com>

* fix: add missing conf preset

Co-authored-by: Brent Bovenzi <4600967+bbovenzi@users.noreply.github.com>

---------

Co-authored-by: Brent Bovenzi <4600967+bbovenzi@users.noreply.github.com>
(cherry picked from commit 6ca99d8)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bbovenzi bbovenzi bbovenzi approved these changes

@pierrejeambrun pierrejeambrun Awaiting requested review from pierrejeambrun pierrejeambrun is a code owner

@ryanahamilton ryanahamilton Awaiting requested review from ryanahamilton ryanahamilton is a code owner

@jscheffl jscheffl Awaiting requested review from jscheffl jscheffl is a code owner

@shubhamraj-git shubhamraj-git Awaiting requested review from shubhamraj-git shubhamraj-git is a code owner

Assignees
No one assigned
Labels
area:UI Related to UI/UX. For Frontend Developers.
Projects
None yet
Milestone
Airflow 3.0.1
Development

Successfully merging this pull request may close these issues.

Add Dag Run Conf to Run Backfill form
3 participants
@guan404ming @bbovenzi @jscheffl
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy