Replies: 1 comment
-
It looks like 'Recursive' is now the default option in Airflow 3.0 and there's no way to deselect it. The underlying behavior is still correctly supporting |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Hello,
We rely on ExternalTaskMarker to automatically clear tasks in downstream pipelines when an upstream task is cleared. So when the parent task is cleared, the child task for the same logical date will also be cleared, when "recursive" is selected during the clearing.
However, in most cases, we'd just like to clear the current DAG and not clear the downstream. A few challenges we've encountered:
It'd be very useful to have the recursive button unchecked by default, so that users can opt-in when needed and downstream pipelines are only cleared when users explicitly intend to do so. This eliminates the risk of accidental clearing and ensures that downstream tasks are only cleared with conscious action. (#32574)
That said, while looking at the Airflow 3.0 UI updates, I noticed that clear task UI for 3.0 no longer includes the "recursive" button. I'm curious to know what this means for the ExternalTaskMarker going forward. Could someone clarify if ExternalTaskMarker will still work the same way in Airflow 3.0?
Beta Was this translation helpful? Give feedback.
All reactions