-
Notifications
You must be signed in to change notification settings - Fork 15.1k
Add soft-delete support and set dag_version FK on delete to SET NULL #50922
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
644ce09
to
0020deb
Compare
8e07a32
to
14a5876
Compare
airflow-core/src/airflow/api_fastapi/core_api/routes/ui/grid.py
Outdated
Show resolved
Hide resolved
airflow-core/src/airflow/migrations/versions/0072_3_1_0_modify_taskinstance_dag_version_fk.py
Outdated
Show resolved
Hide resolved
airflow-core/src/airflow/migrations/versions/0071_3_1_0_add_is_active_to_dag_version.py
Outdated
Show resolved
Hide resolved
airflow-core/src/airflow/migrations/versions/0072_3_1_0_modify_taskinstance_dag_version_fk.py
Show resolved
Hide resolved
airflow-core/src/airflow/migrations/versions/0072_3_1_0_modify_taskinstance_dag_version_fk.py
Outdated
Show resolved
Hide resolved
airflow-core/src/airflow/migrations/versions/0072_3_1_0_modify_taskinstance_dag_version_fk.py
Show resolved
Hide resolved
3ed69fc
to
e6947f5
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall looks good to me
airflow-core/src/airflow/migrations/versions/0071_3_1_0_add_is_active_to_dag_version.py
Outdated
Show resolved
Hide resolved
airflow-core/src/airflow/migrations/versions/0072_3_1_0_modify_taskinstance_dag_version_fk.py
Outdated
Show resolved
Hide resolved
445f8d9
to
8d042f3
Compare
airflow-core/src/airflow/migrations/versions/0071_3_1_0_add_is_active_to_dag_version.py
Outdated
Show resolved
Hide resolved
This looks good, but I’m not sure we want to remove the cascade deletion. I feel it is fine if we do soft deletion; if the user really wants the record to be actually gone, the ti should be gone too. See discussion in linked issue. |
@uranusjr I have just added soft deletion and added back cascade deletion. Please re-review. But now that I think about it. Task instances are your single source of truth for what actually ran, when and with what outcome. If you delete the serialised-DAG (or its version), you wipe out every TaskInstance that pointed at it. That means no more execution history or retry state for those runs. Is this the right behaviour for the user? |
e7196d2
to
54a32bc
Compare
…te the column with prefilled values.
54a32bc
to
eeee740
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM,
I don't see any change in the API / UI part. Meaning that active=False
versions are still retrieved and displayed. I assume we want to propagate that change there too ?
I know active
is to mark that the version was soft deleted or not. But it can be confused with which dag version is active for the dag
, like what's the current version at. Dag version is always the latest one but I find that this could be confusing in a sense. (Or maybe that's just me)
Just closing this for now to prevent accidental merge --- I talked with @sunank200 and we are thinking that the best course of action is to simply change the ondelete behavior to be "on delete restrict" because it's always assumed that this value will be populated and this will simply prevent unintentional deletion of TI rows |
Other notes: if we don't change the ondelete behavior, when adding soft delete does not actually solve the problem (risk of accidental TI deletion) becasue it's purely a software concept. So the question of whether we need soft deletes is really a separate issue. And, I am not aware of any reason we actually want soft deletes or what the point of it would be or what the user interface would be. So that's why we think, let's just add restrict. |
This PR introduces soft-delete for DAG versions and changes the foreign-key behaviour on
task_instance.dag_version_id
so that deleting a version no longer cascades, but instead sets the reference toNULL
.closes: #49580
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rst
or{issue_number}.significant.rst
, in airflow-core/newsfragments.