Content-Length: 993744 | pFad | https://github.com/apache/airflow/commit/0c80a7d41100bf8d18b661c8286d6056e6d5d2f1

7D Fixes AzureFileShare connection extras (#16388) · apache/airflow@0c80a7d · GitHub
Skip to content

Commit 0c80a7d

Browse files
potiukpotiuk
authored
Fixes AzureFileShare connection extras (#16388)
* Fixes AzureFileShare connection extras The Azure File Share connection has not been creted in #15159 and it caused an unexpected side effect as the default Azure Connection passed service_options dictionary to FileService with key that was unexpected. This change fixes two things: 1) adds AzureFileShare connection that has separate conn_type and handles the extra_options specific for FileService Hook available in the Airflow UI. 2) handles the "deprecated" way of passing keys without UI prefix but raises a deprecation warning when such key is passed or when the Wasb connection is used with an empty extras rather than Azure File Share. Fixes #16254 * fixup! Fixes AzureFileShare connection extras * fixup! fixup! Fixes AzureFileShare connection extras
1 parent e975e86 commit 0c80a7d

File tree

17 files changed

+492
-341
lines changed

17 files changed

+492
-341
lines changed

‎airflow/providers/google/cloud/example_dags/example_azure_fileshare_to_gcs.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@
4747
share_name=AZURE_SHARE_NAME,
4848
dest_gcs=DEST_GCS_BUCKET,
4949
directory_name=AZURE_DIRECTORY_NAME,
50-
wasb_conn_id='azure_fileshare_default',
50+
azure_fileshare_conn_id='azure_fileshare_default',
5151
gcp_conn_id='google_cloud_default',
5252
replace=False,
5353
gzip=True,

‎airflow/providers/google/cloud/transfers/azure_fileshare_to_gcs.py

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -38,8 +38,8 @@ class AzureFileShareToGCSOperator(BaseOperator):
3838
:param prefix: Prefix string which filters objects whose name begin with
3939
such prefix. (templated)
4040
:type prefix: str
41-
:param wasb_conn_id: The source WASB connection
42-
:type wasb_conn_id: str
41+
:param azure_fileshare_conn_id: The source WASB connection
42+
:type azure_fileshare_conn_id: str
4343
:param gcp_conn_id: (Optional) The connection ID used to connect to Google Cloud.
4444
:type gcp_conn_id: str
4545
:param dest_gcs: The destination Google Cloud Storage bucket and prefix
@@ -82,7 +82,7 @@ def __init__(
8282
dest_gcs: str,
8383
directory_name: Optional[str] = None,
8484
prefix: str = '',
85-
wasb_conn_id: str = 'wasb_default',
85+
azure_fileshare_conn_id: str = 'azure_fileshare_default',
8686
gcp_conn_id: str = 'google_cloud_default',
8787
delegate_to: Optional[str] = None,
8888
replace: bool = False,
@@ -95,7 +95,7 @@ def __init__(
9595
self.share_name = share_name
9696
self.directory_name = directory_name
9797
self.prefix = prefix
98-
self.wasb_conn_id = wasb_conn_id
98+
self.azure_fileshare_conn_id = azure_fileshare_conn_id
9999
self.gcp_conn_id = gcp_conn_id
100100
self.dest_gcs = dest_gcs
101101
self.delegate_to = delegate_to
@@ -114,7 +114,7 @@ def __init__(
114114
)
115115

116116
def execute(self, context):
117-
azure_fileshare_hook = AzureFileShareHook(self.wasb_conn_id)
117+
azure_fileshare_hook = AzureFileShareHook(self.azure_fileshare_conn_id)
118118
files = azure_fileshare_hook.list_files(
119119
share_name=self.share_name, directory_name=self.directory_name
120120
)

‎airflow/providers/microsoft/azure/CHANGELOG.rst

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,11 @@ Breaking changes
2727

2828
* ``Auto-apply apply_default decorator (#15667)``
2929

30+
31+
``Azure Container Volume`` and ``Azure File Share`` have now dedicated connection types with editable
32+
UI fields. You should not use ``Wasb`` connection type any more for those connections. Names of
33+
connection ids for those hooks/operators were changed to reflect that.
34+
3035
Features
3136
~~~~~~~~
3237

‎airflow/providers/microsoft/azure/hooks/azure_container_volume.py

Lines changed: 52 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@
1515
# KIND, either express or implied. See the License for the
1616
# specific language governing permissions and limitations
1717
# under the License.
18+
from typing import Any, Dict
1819

1920
from azure.mgmt.containerinstance.models import AzureFileVolume, Volume
2021

@@ -25,22 +26,66 @@ class AzureContainerVolumeHook(BaseHook):
2526
"""
2627
A hook which wraps an Azure Volume.
2728
28-
:param wasb_conn_id: :ref:`Wasb connection id<howto/connection:wasb>` of an Azure storage
29-
account of which file shares should be mounted.
30-
:type wasb_conn_id: str
29+
:param azure_container_volume_conn_id: Reference to the
30+
:ref:`Azure Container Volume connection id <howto/connection:azure_container_volume>`
31+
of an Azure account of which container volumes should be used.
32+
:type azure_container_volume_conn_id: str
3133
"""
3234

33-
def __init__(self, wasb_conn_id: str = 'wasb_default') -> None:
35+
conn_name_attr = "azure_container_volume_conn_id"
36+
default_conn_name = 'azure_container_volume_default'
37+
conn_type = 'azure_container_volume'
38+
hook_name = 'Azure Container Volume'
39+
40+
def __init__(self, azure_container_volume_conn_id: str = 'azure_container_volume_default') -> None:
3441
super().__init__()
35-
self.conn_id = wasb_conn_id
42+
self.conn_id = azure_container_volume_conn_id
43+
44+
@staticmethod
45+
def get_connection_form_widgets() -> Dict[str, Any]:
46+
"""Returns connection widgets to add to connection form"""
47+
from flask_appbuilder.fieldwidgets import BS3PasswordFieldWidget
48+
from flask_babel import lazy_gettext
49+
from wtforms import PasswordField
50+
51+
return {
52+
"extra__azure_container_volume__connection_string": PasswordField(
53+
lazy_gettext('Blob Storage Connection String (optional)'), widget=BS3PasswordFieldWidget()
54+
),
55+
}
56+
57+
@staticmethod
58+
def get_ui_field_behaviour() -> Dict:
59+
"""Returns custom field behaviour"""
60+
import json
61+
62+
return {
63+
"hidden_fields": ['schema', 'port', 'host', "extra"],
64+
"relabeling": {
65+
'login': 'Azure Client ID',
66+
'password': 'Azure Secret',
67+
},
68+
"placeholders": {
69+
'extra': json.dumps(
70+
{
71+
"key_path": "path to json file for auth",
72+
"key_json": "specifies json dict for auth",
73+
},
74+
indent=1,
75+
),
76+
'login': 'client_id (token credentials auth)',
77+
'password': 'secret (token credentials auth)',
78+
'extra__azure_container_volume__connection_string': 'connection string auth',
79+
},
80+
}
3681

3782
def get_storagekey(self) -> str:
3883
"""Get Azure File Volume storage key"""
3984
conn = self.get_connection(self.conn_id)
4085
service_options = conn.extra_dejson
4186

42-
if 'connection_string' in service_options:
43-
for keyvalue in service_options['connection_string'].split(";"):
87+
if 'extra__azure_container_volume__connection_string' in service_options:
88+
for keyvalue in service_options['extra__azure_container_volume__connection_string'].split(";"):
4489
key, value = keyvalue.split("=", 1)
4590
if key == "AccountKey":
4691
return value

‎airflow/providers/microsoft/azure/hooks/azure_fileshare.py

Lines changed: 79 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,8 @@
1616
# specific language governing permissions and limitations
1717
# under the License.
1818
#
19-
from typing import List, Optional
19+
import warnings
20+
from typing import Any, Dict, List, Optional
2021

2122
from azure.storage.file import File, FileService
2223

@@ -27,24 +28,91 @@ class AzureFileShareHook(BaseHook):
2728
"""
2829
Interacts with Azure FileShare Storage.
2930
30-
Additional options passed in the 'extra' field of the connection will be
31-
passed to the `FileService()` constructor.
31+
:param azure_fileshare_conn_id: Reference to the
32+
:ref:`Azure Container Volume connection id<howto/connection:azure_fileshare>`
33+
of an Azure account of which container volumes should be used.
3234
33-
:param wasb_conn_id: Reference to the :ref:`wasb connection <howto/connection:wasb>`.
34-
:type wasb_conn_id: str
3535
"""
3636

37-
def __init__(self, wasb_conn_id: str = 'wasb_default') -> None:
37+
conn_name_attr = "azure_fileshare_conn_id"
38+
default_conn_name = 'azure_fileshare_default'
39+
conn_type = 'azure_fileshare'
40+
hook_name = 'Azure FileShare'
41+
42+
def __init__(self, azure_fileshare_conn_id: str = 'azure_fileshare_default') -> None:
3843
super().__init__()
39-
self.conn_id = wasb_conn_id
44+
self.conn_id = azure_fileshare_conn_id
4045
self._conn = None
4146

47+
@staticmethod
48+
def get_connection_form_widgets() -> Dict[str, Any]:
49+
"""Returns connection widgets to add to connection form"""
50+
from flask_appbuilder.fieldwidgets import BS3PasswordFieldWidget, BS3TextFieldWidget
51+
from flask_babel import lazy_gettext
52+
from wtforms import PasswordField, StringField
53+
54+
return {
55+
"extra__azure_fileshare__sas_token": PasswordField(
56+
lazy_gettext('SAS Token (optional)'), widget=BS3PasswordFieldWidget()
57+
),
58+
"extra__azure_fileshare__connection_string": StringField(
59+
lazy_gettext('Connection String (optional)'), widget=BS3TextFieldWidget()
60+
),
61+
"extra__azure_fileshare__protocol": StringField(
62+
lazy_gettext('Account URL or token (optional)'), widget=BS3TextFieldWidget()
63+
),
64+
}
65+
66+
@staticmethod
67+
def get_ui_field_behaviour() -> Dict:
68+
"""Returns custom field behaviour"""
69+
return {
70+
"hidden_fields": ['schema', 'port', 'host', 'extra'],
71+
"relabeling": {
72+
'login': 'Blob Storage Login (optional)',
73+
'password': 'Blob Storage Key (optional)',
74+
'host': 'Account Name (Active Directory Auth)',
75+
},
76+
"placeholders": {
77+
'login': 'account name',
78+
'password': 'secret',
79+
'host': 'account url',
80+
'extra__azure_fileshare__sas_token': 'account url or token (optional)',
81+
'extra__azure_fileshare__connection_string': 'account url or token (optional)',
82+
'extra__azure_fileshare__protocol': 'account url or token (optional)',
83+
},
84+
}
85+
4286
def get_conn(self) -> FileService:
4387
"""Return the FileService object."""
44-
if not self._conn:
45-
conn = self.get_connection(self.conn_id)
46-
service_options = conn.extra_dejson
47-
self._conn = FileService(account_name=conn.login, account_key=conn.password, **service_options)
88+
prefix = "extra__azure_fileshare__"
89+
if self._conn:
90+
return self._conn
91+
conn = self.get_connection(self.conn_id)
92+
service_options_with_prefix = conn.extra_dejson
93+
service_options = {}
94+
for key, value in service_options_with_prefix.items():
95+
# in case dedicated FileShareHook is used, the connection will use the extras from UI.
96+
# in case deprecated wasb hook is used, the old extras will work as well
97+
if key.startswith(prefix):
98+
if value != '':
99+
service_options[key[len(prefix) :]] = value
100+
else:
101+
# warn if the deprecated wasb_connection is used
102+
warnings.warn(
103+
"You are using deprecated connection for AzureFileShareHook."
104+
" Please change it to `Azure FileShare`.",
105+
DeprecationWarning,
106+
)
107+
else:
108+
service_options[key] = value
109+
# warn if the old non-prefixed value is used
110+
warnings.warn(
111+
"You are using deprecated connection for AzureFileShareHook."
112+
" Please change it to `Azure FileShare`.",
113+
DeprecationWarning,
114+
)
115+
self._conn = FileService(account_name=conn.login, account_key=conn.password, **service_options)
48116
return self._conn
49117

50118
def check_for_directory(self, share_name: str, directory_name: str, **kwargs) -> bool:

‎airflow/providers/microsoft/azure/operators/azure_container_instances.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -119,7 +119,7 @@ class AzureContainerInstancesOperator(BaseOperator):
119119
"POSTGRES_PASSWORD": "{{ macros.connection('postgres_default').password }}",
120120
"JOB_GUID": "{{ ti.xcom_pull(task_ids='task1', key='guid') }}" },
121121
secured_variables = ['POSTGRES_PASSWORD'],
122-
volumes = [("azure_wasb_conn_id",
122+
volumes = [("azure_container_instance_conn_id",
123123
"my_storage_container",
124124
"my_fileshare",
125125
"/input-data",

‎airflow/providers/microsoft/azure/provider.yaml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -155,6 +155,8 @@ hook-class-names:
155155
- airflow.providers.microsoft.azure.hooks.azure_batch.AzureBatchHook
156156
- airflow.providers.microsoft.azure.hooks.azure_cosmos.AzureCosmosDBHook
157157
- airflow.providers.microsoft.azure.hooks.azure_data_lake.AzureDataLakeHook
158+
- airflow.providers.microsoft.azure.hooks.azure_fileshare.AzureFileShareHook
159+
- airflow.providers.microsoft.azure.hooks.azure_container_volume.AzureContainerVolumeHook
158160
- airflow.providers.microsoft.azure.hooks.azure_container_instance.AzureContainerInstanceHook
159161
- airflow.providers.microsoft.azure.hooks.wasb.WasbHook
160162
- airflow.providers.microsoft.azure.hooks.azure_data_factory.AzureDataFactoryHook

‎docs/apache-airflow-providers-microsoft-azure/connections/azure_container_volume.rst

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
.. Licensed to the Apache Software Foundation (ASF) under one
2+
or more contributor license agreements. See the NOTICE file
3+
distributed with this work for additional information
4+
regarding copyright ownership. The ASF licenses this file
5+
to you under the Apache License, Version 2.0 (the
6+
"License"); you may not use this file except in compliance
7+
with the License. You may obtain a copy of the License at
8+
9+
.. http://www.apache.org/licenses/LICENSE-2.0
10+
11+
.. Unless required by applicable law or agreed to in writing,
12+
software distributed under the License is distributed on an
13+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
KIND, either express or implied. See the License for the
15+
specific language governing permissions and limitations
16+
under the License.
17+
18+
19+
20+
.. _howto/connection:azure_container_volume:
21+
22+
Microsoft Azure Container Volume Connection
23+
===========================================
24+
25+
The Microsoft Azure Container Volume connection type enables the Azure Container Volume Integrations.
26+
27+
Authenticating to Azure Container Volume
28+
----------------------------------------
29+
30+
There are four ways to connect to Azure Container Volume using Airflow.
31+
32+
1. Use `token credentials
33+
<https://docs.microsoft.com/en-us/azure/developer/python/azure-sdk-authenticate?tabs=cmd#authenticate-with-token-credentials>`_
34+
i.e. add specific credentials (client_id, secret) and subscription id to the Airflow connection.
35+
2. Use a `Connection String
36+
<https://docs.microsoft.com/en-us/azure/data-explorer/kusto/api/connection-strings/storage>`_
37+
i.e. add connection string to ``extra__azure_container_volume__connection_string`` in the Airflow connection.
38+
39+
Only one authorization method can be used at a time. If you need to manage multiple credentials or keys then you should
40+
configure multiple connections.
41+
42+
Default Connection IDs
43+
----------------------
44+
45+
All hooks and operators related to Azure Container Volume use ``azure_container_volume_default`` by default.
46+
47+
Configuring the Connection
48+
--------------------------
49+
50+
Login (optional)
51+
Specify the login used for azure blob storage. For use with Shared Key Credential and SAS Token authentication.
52+
53+
Password (optional)
54+
Specify the password used for azure blob storage. For use with
55+
Active Directory (token credential) and shared key authentication.
56+
57+
Host (optional)
58+
Specify the account url for anonymous public read, Active Directory, shared access key authentication.
59+
60+
Extra (optional)
61+
Specify the extra parameters (as json dictionary) that can be used in Azure connection.
62+
The following parameters are all optional:
63+
64+
* ``extra__azure_container_volume__connection_string``: Connection string for use with connection string authentication.
65+
66+
When specifying the connection in environment variable you should specify
67+
it using URI syntax.
68+
69+
Note that all components of the URI should be URL-encoded.
70+
71+
For example connect with token credentials:
72+
73+
.. code-block:: bash
74+
75+
export AIRFLOW_CONN_WASP_DEFAULT='azure_container_volume://blob%20username:blob%20password@myblob.com'

0 commit comments

Comments
 (0)








ApplySandwichStrip

pFad - (p)hone/(F)rame/(a)nonymizer/(d)eclutterfier!      Saves Data!


--- a PPN by Garber Painting Akron. With Image Size Reduction included!

Fetched URL: https://github.com/apache/airflow/commit/0c80a7d41100bf8d18b661c8286d6056e6d5d2f1

Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy