feat:change logLevel for XCOM returned value message for K8sPodOperator #50914
Conversation
boring-cyborg
bot
added
area:providers
provider:cncf-kubernetes
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
|
| @@ -233,6 +233,10 @@ class KubernetesPodOperator(BaseOperator): | |||
| :param logging_interval: max time in seconds that task should be in deferred state before | |||
| resuming to fetch the latest logs. If ``None``, then the task will remain in deferred state until pod | |||
| is done, and no logs will be visible until that time. | |||
| :param show_return_value_in_logs: a bool value whether to show return_value logs. Defaults to True, | |||
There was a problem hiding this comment.
I wonder if we want this set in BaseOperator (similar to do_xcom_push)?
There was a problem hiding this comment.
That definitely makes more sense to me. I can change the implementation to that instead, if it's what we want.
There was a problem hiding this comment.
I think so but lets do that in a separate PR as BaseOperator is in airlfow-core so all operators will get it only when their min version of Airflow would be 3.1.0 thus in the meantime we will need this PR (with some modification/TODO for the needed changes when min version is 3.1)
There was a problem hiding this comment.
But, out of curiosity the xcom is shown in the UI. Is there extra value to show it also in the logs?
There was a problem hiding this comment.
But, out of curiosity the xcom is shown in the UI. Is there extra value to show it also in the logs?
Yeah, it does show up in the logs because of this line I believe. I followed the implementation in the PythonOperator PR because of this line
XCom logs are currently using the INFO level in python.py level logging. Many users are using INFO level loggers, so it is not possible to change the logging level to obscure the XCom logs.
So it'd look something like this
...
{pod.py:592} INFO - xcom result:
...
closes: #50892
related: #18264
Description
Having exactly the same context as #18264, we want to be able to avoid returning large XComs in the log especially when using custom XCom backends.
This PR just follows the existing pattern of the
PythonOperatorto silence the log.Honestly, unsure of how to write the test for this one though.