Apache Airflow version

3.0.1

If "Other Airflow 2 version" selected, which one?

No response

What happened?

I recently updated my Airflow version (in Docker) from 2.10.2 to 3.0.1. I'm currently experiencing an issue with the triggering of a DAG from another DAG. When the second DAG is triggered its first task is marked as "queued" and the state does not change therefore making the application stuck.

What you think should happen instead?

No response

How to reproduce

The situation is the following:

1. DAG1 triggers some Python logic and inside this logic it triggers with Airflow REST API the execution of DAG2. Then it checks the state of the DAG2 every minute to see if the execution was successful. In that case it will exit.
2. DAG2 will run two tasks and then the third task will trigger DAG3 by means of TriggerDagRunOperator imported from airflow.providers.standard.operators.trigger_dagrun:

trigger_run_no_consent_cleanup = TriggerDagRunOperator(
        task_id="execute_run_no_consent_cleanup",
        trigger_dag_id="run_no_consent_cleanup",
        wait_for_completion=True,
        conf={"run_uuid": "{{ dag_run.conf['run_uuid'] }}"},
        poke_interval=1
    )

In addition in the background there is DAG4 that is triggered autonomously every minute. The problem is that when DAG1 is triggered, once DAG3 is launched the process is stuck. It was working fine before on the older version of Airflow.

DAG3 has 2 tasks to execute and the first task is not even triggered. It just hangs in "queued" state and I have cannot extract any logs from it to understand why it is waiting. DAG1,2,3 all specified the parameter max_active_runs=1 but that should be specific to the DAG and should not influence the tasks of another DAG.

Dag definition example:

@dag(schedule=config.no_consent_cleanup_schedule,
     start_date=config.airflow_start_date,
     max_active_runs=1,
     catchup=False,
     description="Triggered DAG to cleanup",
     render_template_as_native_obj=True
     )

Any idea about what causes this conflict state? Thank you.

Another strange thing is that sometimes when the process is blocker, if I trigger another DAG then it gets unblocked directly and finishes successfully.

Operating System

Docker, Debian

Versions of Apache Airflow Providers

No response

Deployment

Docker-Compose

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct