Content-Length: 571047 | pFad | https://github.com/apache/airflow/commit/528ecbbc005566e13f7a6a1cafb4962733c6efb0

9C Rewrite system tests for ML Engine service (#26915) · apache/airflow@528ecbb · GitHub
Skip to content

Commit 528ecbb

Browse files
authored
Rewrite system tests for ML Engine service (#26915)
1 parent bcb026b commit 528ecbb

File tree

5 files changed

+146
-151
lines changed

5 files changed

+146
-151
lines changed

airflow/providers/google/cloud/utils/mlengine_prediction_summary.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -116,9 +116,10 @@ def metric_fn(inst):
116116

117117
import apache_beam as beam
118118
import dill
119+
from apache_beam.coders.coders import Coder
119120

120121

121-
class JsonCoder:
122+
class JsonCoder(Coder):
122123
"""JSON encoder/decoder."""
123124

124125
@staticmethod

docs/apache-airflow-providers-google/operators/cloud/mlengine.rst

Lines changed: 15 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ This creates a virtual machine that can run code specified in the trainer file,
4040
contains the main application code. A job can be initiated with the
4141
:class:`~airflow.providers.google.cloud.operators.mlengine.MLEngineStartTrainingJobOperator`.
4242

43-
.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_mlengine.py
43+
.. exampleinclude:: /../../tests/system/providers/google/cloud/ml_engine/example_mlengine.py
4444
:language: python
4545
:dedent: 4
4646
:start-after: [START howto_operator_gcp_mlengine_training]
@@ -55,7 +55,7 @@ A model is a container that can hold multiple model versions. A new model can be
5555
The ``model`` field should be defined with a dictionary containing the information about the model.
5656
``name`` is a required field in this dictionary.
5757

58-
.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_mlengine.py
58+
.. exampleinclude:: /../../tests/system/providers/google/cloud/ml_engine/example_mlengine.py
5959
:language: python
6060
:dedent: 4
6161
:start-after: [START howto_operator_gcp_mlengine_create_model]
@@ -69,7 +69,7 @@ The :class:`~airflow.providers.google.cloud.operators.mlengine.MLEngineGetModelO
6969
can be used to obtain a model previously created. To obtain the correct model, ``model_name``
7070
must be defined in the operator.
7171

72-
.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_mlengine.py
72+
.. exampleinclude:: /../../tests/system/providers/google/cloud/ml_engine/example_mlengine.py
7373
:language: python
7474
:dedent: 4
7575
:start-after: [START howto_operator_gcp_mlengine_get_model]
@@ -80,7 +80,7 @@ fields to dynamically determine their values. The result are saved to :ref:`XCom
8080
allowing them to be used by other operators. In this case, the
8181
:class:`~airflow.operators.bash.BashOperator` is used to print the model information.
8282

83-
.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_mlengine.py
83+
.. exampleinclude:: /../../tests/system/providers/google/cloud/ml_engine/example_mlengine.py
8484
:language: python
8585
:dedent: 4
8686
:start-after: [START howto_operator_gcp_mlengine_print_model]
@@ -96,7 +96,7 @@ The model must be specified by ``model_name``, and the ``version`` parameter sho
9696
all the information about the version. Within the ``version`` parameter's dictionary, the ``name`` field is
9797
required.
9898

99-
.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_mlengine.py
99+
.. exampleinclude:: /../../tests/system/providers/google/cloud/ml_engine/example_mlengine.py
100100
:language: python
101101
:dedent: 4
102102
:start-after: [START howto_operator_gcp_mlengine_create_version1]
@@ -105,7 +105,7 @@ required.
105105
The :class:`~airflow.providers.google.cloud.operators.mlengine.MLEngineCreateVersionOperator`
106106
can also be used to create more versions with varying parameters.
107107

108-
.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_mlengine.py
108+
.. exampleinclude:: /../../tests/system/providers/google/cloud/ml_engine/example_mlengine.py
109109
:language: python
110110
:dedent: 4
111111
:start-after: [START howto_operator_gcp_mlengine_create_version2]
@@ -120,7 +120,7 @@ By default, the model code will run using the default model version. You can set
120120
:class:`~airflow.providers.google.cloud.operators.mlengine.MLEngineSetDefaultVersionOperator`
121121
by specifying the ``model_name`` and ``version_name`` parameters.
122122

123-
.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_mlengine.py
123+
.. exampleinclude:: /../../tests/system/providers/google/cloud/ml_engine/example_mlengine.py
124124
:language: python
125125
:dedent: 4
126126
:start-after: [START howto_operator_gcp_mlengine_default_version]
@@ -130,7 +130,7 @@ To list the model versions available, use the
130130
:class:`~airflow.providers.google.cloud.operators.mlengine.MLEngineListVersionsOperator`
131131
while specifying the ``model_name`` parameter.
132132

133-
.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_mlengine.py
133+
.. exampleinclude:: /../../tests/system/providers/google/cloud/ml_engine/example_mlengine.py
134134
:language: python
135135
:dedent: 4
136136
:start-after: [START howto_operator_gcp_mlengine_list_versions]
@@ -141,7 +141,7 @@ fields to dynamically determine their values. The result are saved to :ref:`XCom
141141
allowing them to be used by other operators. In this case, the
142142
:class:`~airflow.operators.bash.BashOperator` is used to print the version information.
143143

144-
.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_mlengine.py
144+
.. exampleinclude:: /../../tests/system/providers/google/cloud/ml_engine/example_mlengine.py
145145
:language: python
146146
:dedent: 4
147147
:start-after: [START howto_operator_gcp_mlengine_print_versions]
@@ -156,7 +156,7 @@ A Google Cloud AI Platform prediction job can be started with the
156156
For specifying the model origen, you need to provide either the ``model_name``, ``uri``, or ``model_name`` and
157157
``version_name``. If you do not provide the ``version_name``, the operator will use the default model version.
158158

159-
.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_mlengine.py
159+
.. exampleinclude:: /../../tests/system/providers/google/cloud/ml_engine/example_mlengine.py
160160
:language: python
161161
:dedent: 4
162162
:start-after: [START howto_operator_gcp_mlengine_get_prediction]
@@ -171,7 +171,7 @@ A model version can be deleted with the
171171
:class:`~airflow.providers.google.cloud.operators.mlengine.MLEngineDeleteVersionOperator` by
172172
the ``version_name`` and ``model_name`` parameters.
173173

174-
.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_mlengine.py
174+
.. exampleinclude:: /../../tests/system/providers/google/cloud/ml_engine/example_mlengine.py
175175
:language: python
176176
:dedent: 4
177177
:start-after: [START howto_operator_gcp_mlengine_delete_version]
@@ -181,7 +181,7 @@ You can also delete a model with the
181181
:class:`~airflow.providers.google.cloud.operators.mlengine.MLEngineDeleteModelOperator`
182182
by providing the ``model_name`` parameter.
183183

184-
.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_mlengine.py
184+
.. exampleinclude:: /../../tests/system/providers/google/cloud/ml_engine/example_mlengine.py
185185
:language: python
186186
:dedent: 4
187187
:start-after: [START howto_operator_gcp_mlengine_delete_model]
@@ -193,7 +193,7 @@ To evaluate a prediction and model, specify a metric function to generate a summ
193193
the evaluation of the model. This function receives a dictionary derived from a json in the batch
194194
prediction result, then returns a tuple of metrics.
195195

196-
.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_mlengine.py
196+
.. exampleinclude:: /../../tests/system/providers/google/cloud/ml_engine/example_mlengine.py
197197
:language: python
198198
:dedent: 4
199199
:start-after: [START howto_operator_gcp_mlengine_get_metric]
@@ -203,7 +203,7 @@ To evaluate a prediction and model, it's useful to have a function to validate t
203203
This function receives a dictionary of the averaged metrics the function above generated. It then
204204
raises an exception if a task fails or should not proceed.
205205

206-
.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_mlengine.py
206+
.. exampleinclude:: /../../tests/system/providers/google/cloud/ml_engine/example_mlengine.py
207207
:language: python
208208
:dedent: 4
209209
:start-after: [START howto_operator_gcp_mlengine_validate_error]
@@ -214,7 +214,7 @@ Prediction results and a model summary can be generated through a function such
214214
It makes predictions using the specified inputs and then summarizes and validates the result. The
215215
functions created above should be passed in through the ``metric_fn_and_keys`` and ``validate_fn`` fields.
216216

217-
.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_mlengine.py
217+
.. exampleinclude:: /../../tests/system/providers/google/cloud/ml_engine/example_mlengine.py
218218
:language: python
219219
:dedent: 4
220220
:start-after: [START howto_operator_gcp_mlengine_evaluate]

tests/providers/google/cloud/operators/test_mlengine_system.py

Lines changed: 0 additions & 56 deletions
This file was deleted.
Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.

0 commit comments

Comments
 (0)








ApplySandwichStrip

pFad - (p)hone/(F)rame/(a)nonymizer/(d)eclutterfier!      Saves Data!


--- a PPN by Garber Painting Akron. With Image Size Reduction included!

Fetched URL: https://github.com/apache/airflow/commit/528ecbbc005566e13f7a6a1cafb4962733c6efb0

Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy