Content-Length: 1429100 | pFad | https://github.com/apache/airflow/commit/63cc915cd38a5034df6bf9c618e12f8690eeade0

58 Switch from Black to Ruff formatter (#35287) · apache/airflow@63cc915 · GitHub
Skip to content

Commit 63cc915

Browse files
jlanevejlaneve
authored
Switch from Black to Ruff formatter (#35287)
This PR switches the formatter we use from Black to Ruff, now that Ruff's introduced a [formatter](https://docs.astral.sh/ruff/formatter/). The Ruff formatter is ~30x as fast as Black, and we already use Ruff for linting. This PR also upgrades the version of Ruff to the latest. Note that this doesn't swap the formatter used in inline Python code in the docs as I haven't seen an easy way of getting the Ruff formatter working with it. Because of this, a lot of the Black code hasn't been removed (i.e. the code to upgrade black, etc).
1 parent 69bac3f commit 63cc915

File tree

62 files changed

+248
-270
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

62 files changed

+248
-270
lines changed

‎.pre-commit-config.yaml

Lines changed: 7 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -155,13 +155,6 @@ repos:
155155
- --fuzzy-match-generates-todo
156156
files: >
157157
\.cfg$|\.conf$|\.ini$|\.ldif$|\.properties$|\.readthedocs$|\.service$|\.tf$|Dockerfile.*$
158-
- repo: https://github.com/psf/black
159-
rev: 23.10.0
160-
hooks:
161-
- id: black
162-
name: Run black (Python formatter)
163-
args: [--config=./pyproject.toml]
164-
exclude: ^.*/.*_vendor/|^airflow/contrib/
165158
- repo: local
166159
hooks:
167160
- id: update-common-sql-api-stubs
@@ -181,14 +174,17 @@ repos:
181174
pass_filenames: false
182175
require_serial: true
183176
- repo: https://github.com/astral-sh/ruff-pre-commit
184-
rev: v0.0.292
177+
rev: v0.1.3
185178
hooks:
186179
# Since ruff makes use of multiple cores we _purposefully_ don't run this in docker so it can use the
187180
# host CPU to it's fullest
188181
- id: ruff
189-
name: ruff
190-
args: [ --fix ]
182+
name: ruff-lint
183+
args: [--fix]
191184
exclude: ^.*/.*_vendor/|^tests/dags/test_imports.py
185+
- id: ruff-format
186+
name: ruff-format
187+
exclude: ^.*/.*_vendor/|^tests/dags/test_imports.py|^airflow/contrib/
192188
- repo: https://github.com/asottile/blacken-docs
193189
rev: 1.16.0
194190
hooks:
@@ -200,7 +196,7 @@ repos:
200196
- --target-version=py38
201197
- --target-version=py39
202198
- --target-version=py310
203-
alias: black
199+
alias: blacken-docs
204200
additional_dependencies: [black==23.10.0]
205201
- repo: https://github.com/pre-commit/pre-commit-hooks
206202
rev: v4.5.0

‎STATIC_CODE_CHECKS.rst

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -140,8 +140,6 @@ require Breeze Docker image to be built locally.
140140
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
141141
| ID | Description | Image |
142142
+===========================================================+==============================================================+=========+
143-
| black | Run black (Python formatter) | |
144-
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
145143
| blacken-docs | Run black on Python code blocks in documentation files | |
146144
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
147145
| check-aiobotocore-optional | Check if aiobotocore is an optional dependency only | |
@@ -319,7 +317,9 @@ require Breeze Docker image to be built locally.
319317
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
320318
| rst-backticks | Check if RST files use double backticks for code | |
321319
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
322-
| ruff | ruff | |
320+
| ruff | ruff-lint | |
321+
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
322+
| ruff-format | ruff-format | |
323323
+-----------------------------------------------------------+--------------------------------------------------------------+---------+
324324
| shellcheck | Check Shell scripts syntax correctness | |
325325
+-----------------------------------------------------------+--------------------------------------------------------------+---------+

‎airflow/api/common/delete_dag.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,7 @@ def delete_dag(dag_id: str, keep_records_in_log: bool = True, session: Session =
7272
)
7373
)
7474

75-
dags_to_delete = [dag_id for dag_id, in dags_to_delete_query]
75+
dags_to_delete = [dag_id for (dag_id,) in dags_to_delete_query]
7676

7777
# Scheduler removes DAGs without files from serialized_dag table every dag_dir_list_interval.
7878
# There may be a lag, so explicitly removes serialized DAG here.

‎airflow/lineage/__init__.py

Lines changed: 3 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -119,11 +119,9 @@ def wrapper(self, context, *args, **kwargs):
119119

120120
if self.inlets and isinstance(self.inlets, list):
121121
# get task_ids that are specified as parameter and make sure they are upstream
122-
task_ids = (
123-
{o for o in self.inlets if isinstance(o, str)}
124-
.union(op.task_id for op in self.inlets if isinstance(op, AbstractOperator))
125-
.intersection(self.get_flat_relative_ids(upstream=True))
126-
)
122+
task_ids = {o for o in self.inlets if isinstance(o, str)}.union(
123+
op.task_id for op in self.inlets if isinstance(op, AbstractOperator)
124+
).intersection(self.get_flat_relative_ids(upstream=True))
127125

128126
# pick up unique direct upstream task_ids if AUTO is specified
129127
if AUTO.upper() in self.inlets or AUTO.lower() in self.inlets:

‎airflow/models/dag.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3618,7 +3618,7 @@ def get_paused_dag_ids(dag_ids: list[str], session: Session = NEW_SESSION) -> se
36183618
.where(DagModel.dag_id.in_(dag_ids))
36193619
)
36203620

3621-
paused_dag_ids = {paused_dag_id for paused_dag_id, in paused_dag_ids}
3621+
paused_dag_ids = {paused_dag_id for (paused_dag_id,) in paused_dag_ids}
36223622
return paused_dag_ids
36233623

36243624
def get_default_view(self) -> str:

‎airflow/providers/apache/cassandra/__init__.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,5 +39,5 @@
3939
"2.5.0"
4040
):
4141
raise RuntimeError(
42-
f"The package `apache-airflow-providers-apache-cassandra:{__version__}` requires Apache Airflow 2.5.0+" # NOQA: E501
42+
f"The package `apache-airflow-providers-apache-cassandra:{__version__}` requires Apache Airflow 2.5.0+"
4343
)

‎airflow/providers/apache/kafka/operators/consume.py

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -137,7 +137,9 @@ def execute(self, context) -> Any:
137137

138138
if self.apply_function:
139139
apply_callable = partial(
140-
self.apply_function, *self.apply_function_args, **self.apply_function_kwargs # type: ignore
140+
self.apply_function, # type: ignore
141+
*self.apply_function_args,
142+
**self.apply_function_kwargs,
141143
)
142144

143145
if self.apply_function_batch:

‎airflow/providers/cncf/kubernetes/utils/pod_manager.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -719,7 +719,7 @@ def extract_xcom_json(self, pod: V1Pod) -> str:
719719
) as resp:
720720
result = self._exec_pod_command(
721721
resp,
722-
f"if [ -s {PodDefaults.XCOM_MOUNT_PATH}/return.json ]; then cat {PodDefaults.XCOM_MOUNT_PATH}/return.json; else echo __airflow_xcom_result_empty__; fi", # noqa
722+
f"if [ -s {PodDefaults.XCOM_MOUNT_PATH}/return.json ]; then cat {PodDefaults.XCOM_MOUNT_PATH}/return.json; else echo __airflow_xcom_result_empty__; fi",
723723
)
724724
if result and result.rstrip() != "__airflow_xcom_result_empty__":
725725
# Note: result string is parsed to check if its valid json.

‎airflow/providers/ftp/operators/ftp.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -156,7 +156,7 @@ def get_openlineage_facets_on_start(self):
156156
local_host = socket.gethostbyname(local_host)
157157
except Exception as e:
158158
self.log.warning(
159-
f"Failed to resolve local hostname. Using the hostname got by socket.gethostbyname() without resolution. {e}", # noqa: E501
159+
f"Failed to resolve local hostname. Using the hostname got by socket.gethostbyname() without resolution. {e}",
160160
exc_info=True,
161161
)
162162

‎airflow/providers/google/cloud/hooks/bigquery.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1725,7 +1725,7 @@ def run_load(
17251725
# we check to make sure the passed source format is valid
17261726
# if it's not, we raise a ValueError
17271727
# Refer to this link for more details:
1728-
# https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.tableDefinitions.(key).sourceFormat # noqa
1728+
# https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.tableDefinitions.(key).sourceFormat
17291729

17301730
if schema_fields is None and not autodetect:
17311731
raise ValueError("You must either pass a schema or autodetect=True.")
@@ -2137,7 +2137,7 @@ def run_query(
21372137
# BigQuery also allows you to define how you want a table's schema to change
21382138
# as a side effect of a query job
21392139
# for more details:
2140-
# https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.schemaUpdateOptions # noqa
2140+
# https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.schemaUpdateOptions
21412141

21422142
allowed_schema_update_options = ["ALLOW_FIELD_ADDITION", "ALLOW_FIELD_RELAXATION"]
21432143

‎airflow/providers/google/cloud/hooks/mlengine.py

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -590,7 +590,9 @@ async def get_job_status(
590590
async with ClientSession() as session:
591591
try:
592592
job = await self.get_job(
593-
project_id=project_id, job_id=job_id, session=session # type: ignore
593+
project_id=project_id,
594+
job_id=job_id,
595+
session=session, # type: ignore
594596
)
595597
job = await job.json(content_type=None)
596598
self.log.info("Retrieving json_response: %s", job)

‎airflow/providers/google/cloud/hooks/secret_manager.py

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -81,5 +81,7 @@ def get_secret(
8181
:param project_id: Project id (if you want to override the project_id from credentials)
8282
"""
8383
return self.get_conn().get_secret(
84-
secret_id=secret_id, secret_version=secret_version, project_id=project_id # type: ignore
84+
secret_id=secret_id,
85+
secret_version=secret_version,
86+
project_id=project_id, # type: ignore
8587
)

‎airflow/providers/google/cloud/operators/bigquery.py

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2850,7 +2850,9 @@ def execute(self, context: Any):
28502850
project_id = self.project_id or self.hook.project_id
28512851
if project_id:
28522852
job_id_path = convert_job_id(
2853-
job_id=self.job_id, project_id=project_id, location=self.location # type: ignore[arg-type]
2853+
job_id=self.job_id, # type: ignore[arg-type]
2854+
project_id=project_id,
2855+
location=self.location,
28542856
)
28552857
context["ti"].xcom_push(key="job_id_path", value=job_id_path)
28562858
# Wait for the job to complete

‎airflow/providers/google/cloud/operators/dataproc.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -152,7 +152,7 @@ class ClusterGenerator:
152152
``projects/[PROJECT_STORING_KEYS]/locations/[LOCATION]/keyRings/[KEY_RING_NAME]/cryptoKeys/[KEY_NAME]`` # noqa
153153
:param enable_component_gateway: Provides access to the web interfaces of default and selected optional
154154
components on the cluster.
155-
""" # noqa: E501
155+
"""
156156

157157
def __init__(
158158
self,

‎airflow/providers/google/cloud/transfers/bigquery_to_mssql.py

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -62,10 +62,7 @@ def __init__(
6262
) -> None:
6363
if mssql_table is not None:
6464
warnings.warn(
65-
# fmt: off
66-
"The `mssql_table` parameter has been deprecated. "
67-
"Use `target_table_name` instead.",
68-
# fmt: on
65+
"The `mssql_table` parameter has been deprecated. Use `target_table_name` instead.",
6966
AirflowProviderDeprecationWarning,
7067
)
7168

‎airflow/providers/microsoft/azure/hooks/asb.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,7 @@ def get_ui_field_behaviour() -> dict[str, Any]:
6464
"<Resource group>.servicebus.windows.net (for Azure AD authenticaltion)"
6565
),
6666
"credential": "credential",
67-
"schema": "Endpoint=sb://<Resource group>.servicebus.windows.net/;SharedAccessKeyName=<AccessKeyName>;SharedAccessKey=<SharedAccessKey>", # noqa
67+
"schema": "Endpoint=sb://<Resource group>.servicebus.windows.net/;SharedAccessKeyName=<AccessKeyName>;SharedAccessKey=<SharedAccessKey>",
6868
},
6969
}
7070

‎airflow/providers/microsoft/azure/hooks/wasb.py

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -600,7 +600,9 @@ async def get_async_conn(self) -> AsyncBlobServiceClient:
600600
tenant, app_id, app_secret, **client_secret_auth_config
601601
)
602602
self.blob_service_client = AsyncBlobServiceClient(
603-
account_url=account_url, credential=token_credential, **extra # type:ignore[arg-type]
603+
account_url=account_url,
604+
credential=token_credential,
605+
**extra, # type:ignore[arg-type]
604606
)
605607
return self.blob_service_client
606608

‎airflow/providers/openlineage/plugins/listener.py

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,10 @@ def __init__(self):
5151

5252
@hookimpl
5353
def on_task_instance_running(
54-
self, previous_state, task_instance: TaskInstance, session: Session # This will always be QUEUED
54+
self,
55+
previous_state,
56+
task_instance: TaskInstance,
57+
session: Session, # This will always be QUEUED
5558
):
5659
if not hasattr(task_instance, "task"):
5760
self.log.warning(

‎airflow/providers/openlineage/utils/utils.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -356,7 +356,8 @@ def _redact(self, item: Redactable, name: str | None, depth: int, max_depth: int
356356
if attrs.has(type(item)):
357357
# TODO: fixme when mypy gets compatible with new attrs
358358
for dict_key, subval in attrs.asdict(
359-
item, recurse=False # type: ignore[arg-type]
359+
item, # type: ignore[arg-type]
360+
recurse=False,
360361
).items():
361362
if _is_name_redactable(dict_key, item):
362363
setattr(

‎airflow/providers/qubole/hooks/qubole.py

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -175,12 +175,16 @@ def execute(self, context: Context) -> None:
175175
time.sleep(Qubole.poll_interval)
176176
self.cmd = self.cls.find(self.cmd.id) # type: ignore[attr-defined]
177177
self.log.info(
178-
"Command Id: %s and Status: %s", self.cmd.id, self.cmd.status # type: ignore[attr-defined]
178+
"Command Id: %s and Status: %s",
179+
self.cmd.id,
180+
self.cmd.status, # type: ignore[attr-defined]
179181
)
180182

181183
if "fetch_logs" in self.kwargs and self.kwargs["fetch_logs"] is True:
182184
self.log.info(
183-
"Logs for Command Id: %s \n%s", self.cmd.id, self.cmd.get_log() # type: ignore[attr-defined]
185+
"Logs for Command Id: %s \n%s",
186+
self.cmd.id,
187+
self.cmd.get_log(), # type: ignore[attr-defined]
184188
)
185189

186190
if self.cmd.status != "done": # type: ignore[attr-defined]
@@ -236,9 +240,7 @@ def get_results(
236240
self.cmd = self.cls.find(cmd_id)
237241

238242
include_headers_str = "true" if include_headers else "false"
239-
self.cmd.get_results(
240-
fp, inline, delim, fetch, arguments=[include_headers_str]
241-
) # type: ignore[attr-defined]
243+
self.cmd.get_results(fp, inline, delim, fetch, arguments=[include_headers_str]) # type: ignore[attr-defined]
242244
fp.flush()
243245
fp.close()
244246
return fp.name

‎airflow/providers/qubole/operators/qubole.py

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -54,8 +54,7 @@ def get_link(
5454
:return: url link
5555
"""
5656
conn = BaseHook.get_connection(
57-
getattr(operator, "qubole_conn_id", None)
58-
or operator.kwargs["qubole_conn_id"] # type: ignore[attr-defined]
57+
getattr(operator, "qubole_conn_id", None) or operator.kwargs["qubole_conn_id"] # type: ignore[attr-defined]
5958
)
6059
if conn and conn.host:
6160
host = re.sub(r"api$", "v2/analyze?command_id=", conn.host)

‎airflow/providers/sftp/operators/sftp.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -210,7 +210,7 @@ def get_openlineage_facets_on_start(self):
210210
local_host = socket.gethostbyname(local_host)
211211
except Exception as e:
212212
self.log.warning(
213-
f"Failed to resolve local hostname. Using the hostname got by socket.gethostbyname() without resolution. {e}", # noqa: E501
213+
f"Failed to resolve local hostname. Using the hostname got by socket.gethostbyname() without resolution. {e}",
214214
exc_info=True,
215215
)
216216

‎airflow/providers/snowflake/operators/snowflake.py

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -505,7 +505,9 @@ def execute(self, context: Context) -> None:
505505
deferrable=self.deferrable,
506506
)
507507
self.query_ids = self._hook.execute_query(
508-
self.sql, statement_count=self.statement_count, bindings=self.bindings # type: ignore[arg-type]
508+
self.sql, # type: ignore[arg-type]
509+
statement_count=self.statement_count,
510+
bindings=self.bindings,
509511
)
510512
self.log.info("List of query ids %s", self.query_ids)
511513

‎airflow/providers/trino/hooks/trino.py

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -177,9 +177,7 @@ def get_first(
177177
except DatabaseError as e:
178178
raise TrinoException(e)
179179

180-
def get_pandas_df(
181-
self, sql: str = "", parameters: Iterable | Mapping[str, Any] | None = None, **kwargs
182-
): # type: ignore[override]
180+
def get_pandas_df(self, sql: str = "", parameters: Iterable | Mapping[str, Any] | None = None, **kwargs): # type: ignore[override]
183181
import pandas as pd
184182

185183
cursor = self.get_cursor()

‎airflow/settings.py

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -293,9 +293,7 @@ def prepare_engine_args(disable_connection_pool=False, pool_class=None):
293293
default_args = default.copy()
294294
break
295295

296-
engine_args: dict = conf.getjson(
297-
"database", "sql_alchemy_engine_args", fallback=default_args
298-
) # type: ignore
296+
engine_args: dict = conf.getjson("database", "sql_alchemy_engine_args", fallback=default_args) # type: ignore
299297

300298
if pool_class:
301299
# Don't use separate settings for size etc, only those from sql_alchemy_engine_args

‎airflow/ti_deps/deps/trigger_rule_dep.py

Lines changed: 15 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -275,9 +275,12 @@ def _evaluate_setup_constraint(*, relevant_setups) -> Iterator[tuple[TIDepStatus
275275
task_ids=ti.task_id, key=PAST_DEPENDS_MET, session=session, default=False
276276
)
277277
if not past_depends_met:
278-
yield self._failing_status(
279-
reason="Task should be skipped but the past depends are not met"
280-
), changed
278+
yield (
279+
self._failing_status(
280+
reason="Task should be skipped but the past depends are not met"
281+
),
282+
changed,
283+
)
281284
return
282285
changed = ti.set_state(new_state, session)
283286

@@ -288,13 +291,16 @@ def _evaluate_setup_constraint(*, relevant_setups) -> Iterator[tuple[TIDepStatus
288291
if ti.map_index > -1:
289292
non_successes -= removed
290293
if non_successes > 0:
291-
yield self._failing_status(
292-
reason=(
293-
f"All setup tasks must complete successfully. Relevant setups: {relevant_setups}: "
294-
f"upstream_states={upstream_states}, "
295-
f"upstream_task_ids={task.upstream_task_ids}"
294+
yield (
295+
self._failing_status(
296+
reason=(
297+
f"All setup tasks must complete successfully. Relevant setups: {relevant_setups}: "
298+
f"upstream_states={upstream_states}, "
299+
f"upstream_task_ids={task.upstream_task_ids}"
300+
),
296301
),
297-
), changed
302+
changed,
303+
)
298304

299305
def _evaluate_direct_relatives() -> Iterator[TIDepStatus]:
300306
"""Evaluate whether ``ti``'s trigger rule was met.

‎airflow/timetables/events.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -83,7 +83,8 @@ def next_dagrun_info(
8383
next_event = self.event_dates[0]
8484
else:
8585
future_dates = itertools.dropwhile(
86-
lambda when: when <= last_automated_data_interval.end, self.event_dates # type: ignore
86+
lambda when: when <= last_automated_data_interval.end, # type: ignore
87+
self.event_dates,
8788
)
8889
next_event = next(future_dates, None) # type: ignore
8990
if next_event is None:

0 commit comments

Comments
 (0)








ApplySandwichStrip

pFad - (p)hone/(F)rame/(a)nonymizer/(d)eclutterfier!      Saves Data!


--- a PPN by Garber Painting Akron. With Image Size Reduction included!

Fetched URL: https://github.com/apache/airflow/commit/63cc915cd38a5034df6bf9c618e12f8690eeade0

Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy