Content-Length: 1159535 | pFad | https://github.com/apache/airflow/commit/e7bf8ecb48f0299af8091433535ac573c2afd1cf

ED [AIRFLOW-6119] [AIP-21] Rename GCS operators, hooks and sensors (#7125) · apache/airflow@e7bf8ec · GitHub
Skip to content

Commit e7bf8ec

Browse files
michalslowikowski00turbaszek
michalslowikowski00
authored and
turbaszek
committed
[AIRFLOW-6119] [AIP-21] Rename GCS operators, hooks and sensors (#7125)
PR contains changes regarding AIP-21 (renaming GCP operators and hooks): * renamed GCP modules: * GoogleCloudStorageHook * GoogleCloudStorageDeleteOperator * GoogleCloudStorageDownloadOperator * GoogleCloudStorageListOperator * GoogleCloudStorageCreateBucketOperator * GoogleCloudStorageBucketCreateAclEntryOperator * GoogleCloudStorageObjectCreateAclEntryOperator * GoogleCloudStorageObjectSensor * GoogleCloudStorageObjectUpdatedSensor * GoogleCloudStoragePrefixSensor * GoogleCloudStorageUploadSessionCompleteSensor
1 parent ab80880 commit e7bf8ec

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

56 files changed

+509
-362
lines changed

‎UPDATING.md

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -465,7 +465,7 @@ The following table shows changes in import paths.
465465
|airflow.contrib.hooks.gcp_translate_hook.CloudTranslateHook |airflow.gcp.hooks.translate.CloudTranslateHook |
466466
|airflow.contrib.hooks.gcp_video_intelligence_hook.CloudVideoIntelligenceHook |airflow.gcp.hooks.video_intelligence.CloudVideoIntelligenceHook |
467467
|airflow.contrib.hooks.gcp_vision_hook.CloudVisionHook |airflow.providers.google.cloud.hooks.vision.CloudVisionHook |
468-
|airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook |airflow.gcp.hooks.gcs.GoogleCloudStorageHook |
468+
|airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook |airflow.gcp.hooks.gcs.GCSHook |
469469
|airflow.contrib.operators.adls_to_gcs.AdlsToGoogleCloudStorageOperator |airflow.operators.adls_to_gcs.AdlsToGoogleCloudStorageOperator |
470470
|airflow.contrib.operators.bigquery_check_operator.BigQueryCheckOperator |airflow.gcp.operators.bigquery.BigQueryCheckOperator |
471471
|airflow.contrib.operators.bigquery_check_operator.BigQueryIntervalCheckOperator |airflow.gcp.operators.bigquery.BigQueryIntervalCheckOperator |
@@ -589,12 +589,12 @@ The following table shows changes in import paths.
589589
|airflow.contrib.operators.gcp_vision_operator.CloudVisionProductUpdateOperator |airflow.providers.google.cloud.operators.vision.CloudVisionUpdateProductOperator |
590590
|airflow.contrib.operators.gcp_vision_operator.CloudVisionReferenceImageCreateOperator |airflow.providers.google.cloud.operators.vision.CloudVisionCreateReferenceImageOperator |
591591
|airflow.contrib.operators.gcp_vision_operator.CloudVisionRemoveProductFromProductSetOperator |airflow.providers.google.cloud.operators.vision.CloudVisionRemoveProductFromProductSetOperator |
592-
|airflow.contrib.operators.gcs_acl_operator.GoogleCloudStorageBucketCreateAclEntryOperator |airflow.gcp.operators.gcs.GoogleCloudStorageBucketCreateAclEntryOperator |
593-
|airflow.contrib.operators.gcs_acl_operator.GoogleCloudStorageObjectCreateAclEntryOperator |airflow.gcp.operators.gcs.GoogleCloudStorageObjectCreateAclEntryOperator |
594-
|airflow.contrib.operators.gcs_delete_operator.GoogleCloudStorageDeleteOperator |airflow.gcp.operators.gcs.GoogleCloudStorageDeleteOperator |
595-
|airflow.contrib.operators.gcs_download_operator.GoogleCloudStorageDownloadOperator |airflow.gcp.operators.gcs.GoogleCloudStorageDownloadOperator |
596-
|airflow.contrib.operators.gcs_list_operator.GoogleCloudStorageListOperator |airflow.gcp.operators.gcs.GoogleCloudStorageListOperator |
597-
|airflow.contrib.operators.gcs_operator.GoogleCloudStorageCreateBucketOperator |airflow.gcp.operators.gcs.GoogleCloudStorageCreateBucketOperator |
592+
|airflow.contrib.operators.gcs_acl_operator.GoogleCloudStorageBucketCreateAclEntryOperator |airflow.gcp.operators.gcs.GCSBucketCreateAclEntryOperator |
593+
|airflow.contrib.operators.gcs_acl_operator.GoogleCloudStorageObjectCreateAclEntryOperator |airflow.gcp.operators.gcs.GCSObjectCreateAclEntryOperator |
594+
|airflow.contrib.operators.gcs_delete_operator.GoogleCloudStorageDeleteOperator |airflow.gcp.operators.gcs.GCSDeleteObjectsOperator |
595+
|airflow.contrib.operators.gcs_download_operator.GoogleCloudStorageDownloadOperator |airflow.gcp.operators.gcs.GCSToLocalOperator |
596+
|airflow.contrib.operators.gcs_list_operator.GoogleCloudStorageListOperator |airflow.gcp.operators.gcs.GCSListObjectsOperator |
597+
|airflow.contrib.operators.gcs_operator.GoogleCloudStorageCreateBucketOperator |airflow.gcp.operators.gcs.GCSCreateBucketOperator |
598598
|airflow.contrib.operators.gcs_to_bq.GoogleCloudStorageToBigQueryOperator |airflow.operators.gcs_to_bq.GoogleCloudStorageToBigQueryOperator |
599599
|airflow.contrib.operators.gcs_to_gcs.GoogleCloudStorageToGoogleCloudStorageOperator |airflow.operators.gcs_to_gcs.GoogleCloudStorageToGoogleCloudStorageOperator |
600600
|airflow.contrib.operators.gcs_to_s3.GoogleCloudStorageToS3Operator |airflow.operators.gcs_to_s3.GCSToS3Operator |
@@ -613,10 +613,10 @@ The following table shows changes in import paths.
613613
|airflow.contrib.operators.sql_to_gcs.BaseSQLToGoogleCloudStorageOperator |airflow.operators.sql_to_gcs.BaseSQLToGoogleCloudStorageOperator |
614614
|airflow.contrib.sensors.bigquery_sensor.BigQueryTableSensor |airflow.gcp.sensors.bigquery.BigQueryTableExistenceSensor |
615615
|airflow.contrib.sensors.gcp_transfer_sensor.GCPTransferServiceWaitForJobStatusSensor |airflow.gcp.sensors.cloud_storage_transfer_service.DataTransferServiceJobStatusSensor |
616-
|airflow.contrib.sensors.gcs_sensor.GoogleCloudStorageObjectSensor |airflow.gcp.sensors.gcs.GoogleCloudStorageObjectSensor |
617-
|airflow.contrib.sensors.gcs_sensor.GoogleCloudStorageObjectUpdatedSensor |airflow.gcp.sensors.gcs.GoogleCloudStorageObjectUpdatedSensor |
618-
|airflow.contrib.sensors.gcs_sensor.GoogleCloudStoragePrefixSensor |airflow.gcp.sensors.gcs.GoogleCloudStoragePrefixSensor |
619-
|airflow.contrib.sensors.gcs_sensor.GoogleCloudStorageUploadSessionCompleteSensor |airflow.gcp.sensors.gcs.GoogleCloudStorageUploadSessionCompleteSensor |
616+
|airflow.contrib.sensors.gcs_sensor.GoogleCloudStorageObjectSensor |airflow.gcp.sensors.gcs.GCSObjectExistenceSensor |
617+
|airflow.contrib.sensors.gcs_sensor.GoogleCloudStorageObjectUpdatedSensor |airflow.gcp.sensors.gcs.GCSObjectUpdateSensor |
618+
|airflow.contrib.sensors.gcs_sensor.GoogleCloudStoragePrefixSensor |airflow.gcp.sensors.gcs.GCSObjectsWtihPrefixExistenceSensor |
619+
|airflow.contrib.sensors.gcs_sensor.GoogleCloudStorageUploadSessionCompleteSensor |airflow.gcp.sensors.gcs.GCSUploadSessionCompleteSensor |
620620
|airflow.contrib.sensors.pubsub_sensor.PubSubPullSensor |airflow.providers.google.cloud.sensors.pubsub.PubSubPullSensor |
621621

622622

‎airflow/contrib/hooks/gcs_hook.py

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,10 +21,21 @@
2121
"""
2222
import warnings
2323

24-
# pylint: disable=unused-import
25-
from airflow.gcp.hooks.gcs import GoogleCloudStorageHook, _parse_gcs_url # noqa
24+
from airflow.gcp.hooks.gcs import GCSHook
2625

2726
warnings.warn(
2827
"This module is deprecated. Please use `airflow.gcp.hooks.gcs`.",
2928
DeprecationWarning, stacklevel=2
3029
)
30+
31+
32+
class GoogleCloudStorageHook(GCSHook):
33+
"""
34+
This class is deprecated. Please use `airflow.gcp.hooks.gcs.GCSHook`.
35+
"""
36+
def __init__(self, *args, **kwargs):
37+
warnings.warn(
38+
"This class is deprecated. Please use `airflow.gcp.hooks.gcs.GCSHook`.",
39+
DeprecationWarning, stacklevel=2
40+
)
41+
super().__init__(*args, **kwargs)

‎airflow/contrib/operators/gcs_acl_operator.py

Lines changed: 31 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -22,12 +22,39 @@
2222

2323
import warnings
2424

25-
# pylint: disable=unused-import
26-
from airflow.gcp.operators.gcs import ( # noqa
27-
GoogleCloudStorageBucketCreateAclEntryOperator, GoogleCloudStorageObjectCreateAclEntryOperator,
28-
)
25+
from airflow.gcp.operators.gcs import GCSBucketCreateAclEntryOperator, GCSObjectCreateAclEntryOperator
2926

3027
warnings.warn(
3128
"This module is deprecated. Please use `airflow.gcp.operators.gcs`.",
3229
DeprecationWarning, stacklevel=2
3330
)
31+
32+
33+
class GoogleCloudStorageBucketCreateAclEntryOperator(GCSBucketCreateAclEntryOperator):
34+
"""
35+
This class is deprecated.
36+
Please use `airflow.gcp.operators.gcs.GCSBucketCreateAclEntryOperator`.
37+
"""
38+
39+
def __init__(self, *args, **kwargs):
40+
warnings.warn(
41+
"""This class is deprecated.
42+
Please use `airflow.gcp.operators.gcs.GCSBucketCreateAclEntryOperator`.""",
43+
DeprecationWarning, stacklevel=2
44+
)
45+
super().__init__(*args, **kwargs)
46+
47+
48+
class GoogleCloudStorageObjectCreateAclEntryOperator(GCSObjectCreateAclEntryOperator):
49+
"""
50+
This class is deprecated.
51+
Please use `airflow.gcp.operators.gcs.GCSObjectCreateAclEntryOperator`.
52+
"""
53+
54+
def __init__(self, *args, **kwargs):
55+
warnings.warn(
56+
"""This class is deprecated.
57+
Please use `airflow.gcp.operators.gcs.GCSObjectCreateAclEntryOperator`.""",
58+
DeprecationWarning, stacklevel=2
59+
)
60+
super().__init__(*args, **kwargs)

‎airflow/contrib/operators/gcs_delete_operator.py

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,10 +22,24 @@
2222

2323
import warnings
2424

25-
# pylint: disable=unused-import
26-
from airflow.gcp.operators.gcs import GoogleCloudStorageDeleteOperator # noqa
25+
from airflow.gcp.operators.gcs import GCSDeleteObjectsOperator
2726

2827
warnings.warn(
2928
"This module is deprecated. Please use `airflow.gcp.operators.gcs`.",
3029
DeprecationWarning, stacklevel=2
3130
)
31+
32+
33+
class GoogleCloudStorageDeleteOperator(GCSDeleteObjectsOperator):
34+
"""
35+
This class is deprecated.
36+
Please use `airflow.gcp.operators.gcs.GCSDeleteObjectsOperator`.
37+
"""
38+
39+
def __init__(self, *args, **kwargs):
40+
warnings.warn(
41+
"""This class is deprecated.
42+
Please use `airflow.gcp.operators.gcs.GCSDeleteObjectsOperator`.""",
43+
DeprecationWarning, stacklevel=2
44+
)
45+
super().__init__(*args, **kwargs)

‎airflow/contrib/operators/gcs_download_operator.py

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,10 +22,24 @@
2222

2323
import warnings
2424

25-
# pylint: disable=unused-import
26-
from airflow.gcp.operators.gcs import GoogleCloudStorageDownloadOperator # noqa
25+
from airflow.gcp.operators.gcs import GCSToLocalOperator
2726

2827
warnings.warn(
2928
"This module is deprecated. Please use `airflow.gcp.operators.gcs`.",
3029
DeprecationWarning, stacklevel=2
3130
)
31+
32+
33+
class GoogleCloudStorageDownloadOperator(GCSToLocalOperator):
34+
"""
35+
This class is deprecated.
36+
Please use `airflow.gcp.operators.gcs.GCSToLocalOperator`.
37+
"""
38+
39+
def __init__(self, *args, **kwargs):
40+
warnings.warn(
41+
"""This class is deprecated.
42+
Please use `airflow.gcp.operators.gcs.GCSToLocalOperator`.""",
43+
DeprecationWarning, stacklevel=2
44+
)
45+
super().__init__(*args, **kwargs)

‎airflow/contrib/operators/gcs_list_operator.py

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,10 +22,24 @@
2222

2323
import warnings
2424

25-
# pylint: disable=unused-import
26-
from airflow.gcp.operators.gcs import GoogleCloudStorageListOperator # noqa
25+
from airflow.gcp.operators.gcs import GCSListObjectsOperator
2726

2827
warnings.warn(
2928
"This module is deprecated. Please use `airflow.gcp.operators.gcs`.",
3029
DeprecationWarning, stacklevel=2
3130
)
31+
32+
33+
class GoogleCloudStorageListOperator(GCSListObjectsOperator):
34+
"""
35+
This class is deprecated.
36+
Please use `airflow.gcp.operators.gcs.GCSListObjectsOperator`.
37+
"""
38+
39+
def __init__(self, *args, **kwargs):
40+
warnings.warn(
41+
"""This class is deprecated.
42+
Please use `airflow.gcp.operators.gcs.GCSListObjectsOperator`.""",
43+
DeprecationWarning, stacklevel=2
44+
)
45+
super().__init__(*args, **kwargs)

‎airflow/contrib/operators/gcs_operator.py

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,10 +22,23 @@
2222

2323
import warnings
2424

25-
# pylint: disable=unused-import
26-
from airflow.gcp.operators.gcs import GoogleCloudStorageCreateBucketOperator # noqa
25+
from airflow.gcp.operators.gcs import GCSCreateBucketOperator
2726

2827
warnings.warn(
2928
"This module is deprecated. Please use `airflow.gcp.operators.gcs`.",
3029
DeprecationWarning, stacklevel=2
3130
)
31+
32+
33+
class GoogleCloudStorageCreateBucketOperator(GCSCreateBucketOperator):
34+
"""
35+
This class is deprecated.Please use `airflow.gcp.operators.gcs.GCSCreateBucketOperator`.
36+
"""
37+
38+
def __init__(self, *args, **kwargs):
39+
warnings.warn(
40+
"""This class is deprecated.
41+
Please use `airflow.gcp.operators.gcs.GCSCreateBucketOperator`.""",
42+
DeprecationWarning, stacklevel=2
43+
)
44+
super().__init__(*args, **kwargs)

‎airflow/contrib/operators/gcs_to_gdrive_operator.py

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@
2424

2525
from airflow.contrib.hooks.gdrive_hook import GoogleDriveHook
2626
from airflow.exceptions import AirflowException
27-
from airflow.gcp.hooks.gcs import GoogleCloudStorageHook
27+
from airflow.gcp.hooks.gcs import GCSHook
2828
from airflow.models import BaseOperator
2929
from airflow.utils.decorators import apply_defaults
3030

@@ -93,12 +93,12 @@ def __init__(
9393
self.move_object = move_object
9494
self.gcp_conn_id = gcp_conn_id
9595
self.delegate_to = delegate_to
96-
self.gcs_hook = None # type: Optional[GoogleCloudStorageHook]
96+
self.gcs_hook = None # type: Optional[GCSHook]
9797
self.gdrive_hook = None # type: Optional[GoogleDriveHook]
9898

9999
def execute(self, context):
100100

101-
self.gcs_hook = GoogleCloudStorageHook(
101+
self.gcs_hook = GCSHook(
102102
google_cloud_storage_conn_id=self.gcp_conn_id, delegate_to=self.delegate_to
103103
)
104104
self.gdrive_hook = GoogleDriveHook(gcp_conn_id=self.gcp_conn_id, delegate_to=self.delegate_to)

‎airflow/contrib/operators/s3_to_gcs_operator.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@
2121

2222
from airflow.contrib.operators.s3_list_operator import S3ListOperator
2323
from airflow.exceptions import AirflowException
24-
from airflow.gcp.hooks.gcs import GoogleCloudStorageHook, _parse_gcs_url
24+
from airflow.gcp.hooks.gcs import GCSHook, _parse_gcs_url
2525
from airflow.providers.amazon.aws.hooks.s3 import S3Hook
2626
from airflow.utils.decorators import apply_defaults
2727

@@ -140,7 +140,7 @@ def execute(self, context):
140140
# use the super method to list all the files in an S3 bucket/key
141141
files = super().execute(context)
142142

143-
gcs_hook = GoogleCloudStorageHook(
143+
gcs_hook = GCSHook(
144144
google_cloud_storage_conn_id=self.gcp_conn_id,
145145
delegate_to=self.delegate_to)
146146

‎airflow/contrib/sensors/gcs_sensor.py

Lines changed: 63 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -20,13 +20,72 @@
2020

2121
import warnings
2222

23-
# pylint: disable=unused-import
24-
from airflow.gcp.sensors.gcs import ( # noqa
25-
GoogleCloudStorageObjectSensor, GoogleCloudStorageObjectUpdatedSensor, GoogleCloudStoragePrefixSensor,
26-
GoogleCloudStorageUploadSessionCompleteSensor,
23+
from airflow.gcp.sensors.gcs import (
24+
GCSObjectExistenceSensor, GCSObjectsWtihPrefixExistenceSensor, GCSObjectUpdateSensor,
25+
GCSUploadSessionCompleteSensor,
2726
)
2827

2928
warnings.warn(
3029
"This module is deprecated. Please use `airflow.gcp.sensors.gcs`.",
3130
DeprecationWarning, stacklevel=2
3231
)
32+
33+
34+
class GoogleCloudStorageObjectSensor(GCSObjectExistenceSensor):
35+
"""
36+
This class is deprecated.
37+
Please use `airflow.gcp.sensors.gcs.GCSObjectExistenceSensor`.
38+
"""
39+
40+
def __init__(self, *args, **kwargs):
41+
warnings.warn(
42+
"""This class is deprecated.
43+
Please use `airflow.gcp.sensors.gcs.GCSObjectExistenceSensor`.""",
44+
DeprecationWarning, stacklevel=2
45+
)
46+
super().__init__(*args, **kwargs)
47+
48+
49+
class GoogleCloudStorageObjectUpdatedSensor(GCSObjectUpdateSensor):
50+
"""
51+
This class is deprecated.
52+
Please use `airflow.gcp.sensors.gcs.GCSObjectUpdateSensor`.
53+
"""
54+
55+
def __init__(self, *args, **kwargs):
56+
warnings.warn(
57+
"""This class is deprecated.
58+
Please use `airflow.gcp.sensors.gcs.GCSObjectUpdateSensor`.""",
59+
DeprecationWarning, stacklevel=2
60+
)
61+
super().__init__(*args, **kwargs)
62+
63+
64+
class GoogleCloudStoragePrefixSensor(GCSObjectsWtihPrefixExistenceSensor):
65+
"""
66+
This class is deprecated.
67+
Please use `airflow.gcp.sensors.gcs.GCSObjectsWtihPrefixExistenceSensor`.
68+
"""
69+
70+
def __init__(self, *args, **kwargs):
71+
warnings.warn(
72+
"""This class is deprecated.
73+
Please use `airflow.gcp.sensors.gcs.GCSObjectsWtihPrefixExistenceSensor`.""",
74+
DeprecationWarning, stacklevel=2
75+
)
76+
super().__init__(*args, **kwargs)
77+
78+
79+
class GoogleCloudStorageUploadSessionCompleteSensor(GCSUploadSessionCompleteSensor):
80+
"""
81+
This class is deprecated.
82+
Please use `airflow.gcp.sensors.gcs.GCSUploadSessionCompleteSensor`.
83+
"""
84+
85+
def __init__(self, *args, **kwargs):
86+
warnings.warn(
87+
"""This class is deprecated.
88+
Please use `airflow.gcp.sensors.gcs.GCSUploadSessionCompleteSensor`.""",
89+
DeprecationWarning, stacklevel=2
90+
)
91+
super().__init__(*args, **kwargs)

‎airflow/gcp/example_dags/example_cloud_memorystore.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@
3333
CloudMemorystoreListInstancesOperator, CloudMemorystoreScaleInstanceOperator,
3434
CloudMemorystoreUpdateInstanceOperator,
3535
)
36-
from airflow.gcp.operators.gcs import GoogleCloudStorageBucketCreateAclEntryOperator
36+
from airflow.gcp.operators.gcs import GCSBucketCreateAclEntryOperator
3737
from airflow.operators.bash_operator import BashOperator
3838
from airflow.utils import dates
3939

@@ -130,7 +130,7 @@
130130
# [END howto_operator_update_instance]
131131

132132
# [START howto_operator_set_acl_permission]
133-
set_acl_permission = GoogleCloudStorageBucketCreateAclEntryOperator(
133+
set_acl_permission = GCSBucketCreateAclEntryOperator(
134134
task_id="gcs-set-acl-permission",
135135
bucket=BUCKET_NAME,
136136
entity="user-{{ task_instance.xcom_pull('get-instance')['persistenceIamIdentity']"

0 commit comments

Comments
 (0)








ApplySandwichStrip

pFad - (p)hone/(F)rame/(a)nonymizer/(d)eclutterfier!      Saves Data!


--- a PPN by Garber Painting Akron. With Image Size Reduction included!

Fetched URL: https://github.com/apache/airflow/commit/e7bf8ecb48f0299af8091433535ac573c2afd1cf

Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy