Skip to content

Commit 528ecbb

Browse files
MrGeorgeOwlMrGeorgeOwl
authored
Rewrite system tests for ML Engine service (#26915)
1 parent bcb026b commit 528ecbb

File tree

5 files changed

+146
-151
lines changed

5 files changed

+146
-151
lines changed

‎airflow/providers/google/cloud/utils/mlengine_prediction_summary.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -116,9 +116,10 @@ def metric_fn(inst):
116116

117117
import apache_beam as beam
118118
import dill
119+
from apache_beam.coders.coders import Coder
119120

120121

121-
class JsonCoder:
122+
class JsonCoder(Coder):
122123
"""JSON encoder/decoder."""
123124

124125
@staticmethod

‎docs/apache-airflow-providers-google/operators/cloud/mlengine.rst

Lines changed: 15 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ This creates a virtual machine that can run code specified in the trainer file,
4040
contains the main application code. A job can be initiated with the
4141
:class:`~airflow.providers.google.cloud.operators.mlengine.MLEngineStartTrainingJobOperator`.
4242

43-
.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_mlengine.py
43+
.. exampleinclude:: /../../tests/system/providers/google/cloud/ml_engine/example_mlengine.py
4444
:language: python
4545
:dedent: 4
4646
:start-after: [START howto_operator_gcp_mlengine_training]
@@ -55,7 +55,7 @@ A model is a container that can hold multiple model versions. A new model can be
5555
The ``model`` field should be defined with a dictionary containing the information about the model.
5656
``name`` is a required field in this dictionary.
5757

58-
.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_mlengine.py
58+
.. exampleinclude:: /../../tests/system/providers/google/cloud/ml_engine/example_mlengine.py
5959
:language: python
6060
:dedent: 4
6161
:start-after: [START howto_operator_gcp_mlengine_create_model]
@@ -69,7 +69,7 @@ The :class:`~airflow.providers.google.cloud.operators.mlengine.MLEngineGetModelO
6969
can be used to obtain a model previously created. To obtain the correct model, ``model_name``
7070
must be defined in the operator.
7171

72-
.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_mlengine.py
72+
.. exampleinclude:: /../../tests/system/providers/google/cloud/ml_engine/example_mlengine.py
7373
:language: python
7474
:dedent: 4
7575
:start-after: [START howto_operator_gcp_mlengine_get_model]
@@ -80,7 +80,7 @@ fields to dynamically determine their values. The result are saved to :ref:`XCom
8080
allowing them to be used by other operators. In this case, the
8181
:class:`~airflow.operators.bash.BashOperator` is used to print the model information.
8282

83-
.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_mlengine.py
83+
.. exampleinclude:: /../../tests/system/providers/google/cloud/ml_engine/example_mlengine.py
8484
:language: python
8585
:dedent: 4
8686
:start-after: [START howto_operator_gcp_mlengine_print_model]
@@ -96,7 +96,7 @@ The model must be specified by ``model_name``, and the ``version`` parameter sho
9696
all the information about the version. Within the ``version`` parameter's dictionary, the ``name`` field is
9797
required.
9898

99-
.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_mlengine.py
99+
.. exampleinclude:: /../../tests/system/providers/google/cloud/ml_engine/example_mlengine.py
100100
:language: python
101101
:dedent: 4
102102
:start-after: [START howto_operator_gcp_mlengine_create_version1]
@@ -105,7 +105,7 @@ required.
105105
The :class:`~airflow.providers.google.cloud.operators.mlengine.MLEngineCreateVersionOperator`
106106
can also be used to create more versions with varying parameters.
107107

108-
.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_mlengine.py
108+
.. exampleinclude:: /../../tests/system/providers/google/cloud/ml_engine/example_mlengine.py
109109
:language: python
110110
:dedent: 4
111111
:start-after: [START howto_operator_gcp_mlengine_create_version2]
@@ -120,7 +120,7 @@ By default, the model code will run using the default model version. You can set
120120
:class:`~airflow.providers.google.cloud.operators.mlengine.MLEngineSetDefaultVersionOperator`
121121
by specifying the ``model_name`` and ``version_name`` parameters.
122122

123-
.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_mlengine.py
123+
.. exampleinclude:: /../../tests/system/providers/google/cloud/ml_engine/example_mlengine.py
124124
:language: python
125125
:dedent: 4
126126
:start-after: [START howto_operator_gcp_mlengine_default_version]
@@ -130,7 +130,7 @@ To list the model versions available, use the
130130
:class:`~airflow.providers.google.cloud.operators.mlengine.MLEngineListVersionsOperator`
131131
while specifying the ``model_name`` parameter.
132132

133-
.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_mlengine.py
133+
.. exampleinclude:: /../../tests/system/providers/google/cloud/ml_engine/example_mlengine.py
134134
:language: python
135135
:dedent: 4
136136
:start-after: [START howto_operator_gcp_mlengine_list_versions]
@@ -141,7 +141,7 @@ fields to dynamically determine their values. The result are saved to :ref:`XCom
141141
allowing them to be used by other operators. In this case, the
142142
:class:`~airflow.operators.bash.BashOperator` is used to print the version information.
143143

144-
.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_mlengine.py
144+
.. exampleinclude:: /../../tests/system/providers/google/cloud/ml_engine/example_mlengine.py
145145
:language: python
146146
:dedent: 4
147147
:start-after: [START howto_operator_gcp_mlengine_print_versions]
@@ -156,7 +156,7 @@ A Google Cloud AI Platform prediction job can be started with the
156156
For specifying the model origin, you need to provide either the ``model_name``, ``uri``, or ``model_name`` and
157157
``version_name``. If you do not provide the ``version_name``, the operator will use the default model version.
158158

159-
.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_mlengine.py
159+
.. exampleinclude:: /../../tests/system/providers/google/cloud/ml_engine/example_mlengine.py
160160
:language: python
161161
:dedent: 4
162162
:start-after: [START howto_operator_gcp_mlengine_get_prediction]
@@ -171,7 +171,7 @@ A model version can be deleted with the
171171
:class:`~airflow.providers.google.cloud.operators.mlengine.MLEngineDeleteVersionOperator` by
172172
the ``version_name`` and ``model_name`` parameters.
173173

174-
.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_mlengine.py
174+
.. exampleinclude:: /../../tests/system/providers/google/cloud/ml_engine/example_mlengine.py
175175
:language: python
176176
:dedent: 4
177177
:start-after: [START howto_operator_gcp_mlengine_delete_version]
@@ -181,7 +181,7 @@ You can also delete a model with the
181181
:class:`~airflow.providers.google.cloud.operators.mlengine.MLEngineDeleteModelOperator`
182182
by providing the ``model_name`` parameter.
183183

184-
.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_mlengine.py
184+
.. exampleinclude:: /../../tests/system/providers/google/cloud/ml_engine/example_mlengine.py
185185
:language: python
186186
:dedent: 4
187187
:start-after: [START howto_operator_gcp_mlengine_delete_model]
@@ -193,7 +193,7 @@ To evaluate a prediction and model, specify a metric function to generate a summ
193193
the evaluation of the model. This function receives a dictionary derived from a json in the batch
194194
prediction result, then returns a tuple of metrics.
195195

196-
.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_mlengine.py
196+
.. exampleinclude:: /../../tests/system/providers/google/cloud/ml_engine/example_mlengine.py
197197
:language: python
198198
:dedent: 4
199199
:start-after: [START howto_operator_gcp_mlengine_get_metric]
@@ -203,7 +203,7 @@ To evaluate a prediction and model, it's useful to have a function to validate t
203203
This function receives a dictionary of the averaged metrics the function above generated. It then
204204
raises an exception if a task fails or should not proceed.
205205

206-
.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_mlengine.py
206+
.. exampleinclude:: /../../tests/system/providers/google/cloud/ml_engine/example_mlengine.py
207207
:language: python
208208
:dedent: 4
209209
:start-after: [START howto_operator_gcp_mlengine_validate_error]
@@ -214,7 +214,7 @@ Prediction results and a model summary can be generated through a function such
214214
It makes predictions using the specified inputs and then summarizes and validates the result. The
215215
functions created above should be passed in through the ``metric_fn_and_keys`` and ``validate_fn`` fields.
216216

217-
.. exampleinclude:: /../../airflow/providers/google/cloud/example_dags/example_mlengine.py
217+
.. exampleinclude:: /../../tests/system/providers/google/cloud/ml_engine/example_mlengine.py
218218
:language: python
219219
:dedent: 4
220220
:start-after: [START howto_operator_gcp_mlengine_evaluate]

‎tests/providers/google/cloud/operators/test_mlengine_system.py

Lines changed: 0 additions & 56 deletions
This file was deleted.

‎tests/system/providers/google/cloud/ml_engine/__init__.py

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.

0 commit comments

Comments
 (0)
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy