Skip to content

Use dynamic task mapping in TriggerDagRunOperator may generate the same run_id #28868

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
2 tasks done
yenchenLiu opened this issue Jan 12, 2023 · 5 comments
Open
2 tasks done

Use dynamic task mapping in TriggerDagRunOperator may generate the same run_id #28868

yenchenLiu opened this issue Jan 12, 2023 · 5 comments
Labels
kind:feature Feature Requests

Comments

@yenchenLiu
Copy link
Contributor

Apache Airflow version

2.5.0

What happened

I use TriggerDagRunOperator to generate a large number of dag instances(>= 100), and it sometimes causes a bug that is

Failed to execute job 7159001 for task dynamic_dags ((psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "dag_run_dag_id_run_id_key"
DETAIL:  Key (dag_id, run_id)=(dynamic_dags, manual__2023-01-11T22:28:03.286419+00:00) already exists.

TriggerDagRunOperator.partial(task_id='dynamic_dags', trigger_dag_id='trigger_dag', wait_for_completion=True).expand(conf=[{"id": 1}, {"id": 2}, {"id": 3}, ......])

What you think should happen instead

I expect the run_id to be always unique when I use dynamic task mapping to generate instances.

How to reproduce

  1. Create a dag called trigger_dag.
  2. Create a dag called test_dag, which runs TriggerDagRunOperator with the dynamic task to generate a large number of trigger_dag.
  3. Sometimes, you will see the task failed due to the same run_id.
  • I use the local executor and parallelism = 16, max_active_tasks_per_dag = 12, and max_active_runs_per_dag = 12

Operating System

Ubuntu 20.04.4 LTS

Versions of Apache Airflow Providers

No response

Deployment

Virtualenv installation

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct
@yenchenLiu yenchenLiu added area:core kind:bug This is a clearly a bug labels Jan 12, 2023
@boring-cyborg
Copy link

boring-cyborg bot commented Jan 12, 2023

Thanks for opening your first issue here! Be sure to follow the issue template!

@yenchenLiu
Copy link
Contributor Author

I guess that we can check if the task is dynamic mapping (maybe by checking expanded_ti_count?), then add the current task number into run_id etc.

@uranusjr
Copy link
Member

uranusjr commented Jan 12, 2023 •
edited
Loading

The real problem is not actually the run ID, but the logical date (execution date), which also has a unique constraint. Even if the run ID duplication is “fixed”, you still can’t schedule multiple runs under the same logical date. So that’s the thing you must explicitly pass to expand (or expand_kwargs) if you want to use dynamic task mapping. And if you do, the run ID would be de-duplicated automatically.

@uranusjr uranusjr added kind:feature Feature Requests area:core-operators Operators, Sensors and hooks within Core Airflow and removed kind:bug This is a clearly a bug area:core labels Jan 12, 2023
@eladkal eladkal added affected_version:2.5 Issues Reported for 2.5 and removed affected_version:2.5 Issues Reported for 2.5 labels Mar 6, 2023
@FranCao
Copy link

FranCao commented Nov 8, 2023

Hi @yenchenLiu, wondering if you were able to get TriggerDagRunOperator to run with dynamic task mapping while also passing different parameters that will be used in the child DAG that is executed.

@uranusjr
Copy link
Member

uranusjr commented Apr 3, 2024

One possible approach would be to use Dataset to trigger the DAG instead. This would allow the scheduler to decide what the logical date (and run ID) is, and deduplicate the date appropriately.

@eladkal eladkal removed the area:core-operators Operators, Sensors and hooks within Core Airflow label May 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind:feature Feature Requests
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@uranusjr @yenchenLiu @FranCao @eladkal
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy