Skip to content

Commit cf052dc

Browse files
potiukpotiuk
authored
Add feature to build "chicken-egg" packages from sources (#35890)
When we build on ci a pre-release version of provider, and we want to include packages that have >= CURRENTLY_RELEASED_VERSION we have to make sure that those packages are built from sources during building of PROD image. Otherwise they will not be installable on CURRENT_VERSION.dev0, CURRENT_VERSION.rc* etc. Until we "Actually" release a provider we should have a way to build such provider from sources. This is the CI version of it, once we have it working we can also apply it to the workflow that releases images to dockerhub.
1 parent eed6427 commit cf052dc

File tree

19 files changed

+120
-47
lines changed

19 files changed

+120
-47
lines changed

‎.github/actions/build-prod-images/action.yml

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,9 @@ inputs:
2222
build-provider-packages:
2323
description: 'Whether to build provider packages from sources'
2424
required: true
25+
chicken-egg-providers:
26+
description: 'List of chicken-egg provider packages to build from sources'
27+
required: true
2528
runs:
2629
using: "composite"
2730
steps:
@@ -41,6 +44,15 @@ runs:
4144
--package-list-file ./airflow/providers/installed_providers.txt
4245
--package-format wheel --version-suffix-for-pypi dev0
4346
if: ${{ inputs.build-provider-packages == 'true' }}
47+
- name: "Prepare chicken-eggs provider packages"
48+
# In case of provider packages which use latest dev0 version of providers, we should prepare them
49+
# from the source code, not from the PyPI because they have apache-airflow>=X.Y.Z dependency
50+
# And when we prepare them from sources they will have apache-airflow>=X.Y.Z.dev0
51+
shell: bash
52+
run: >
53+
breeze release-management prepare-provider-packages
54+
--package-format wheel --version-suffix-for-pypi dev0 ${{ inputs.chicken-egg-providers }}
55+
if: ${{ inputs.build-provider-packages != 'true' && inputs.chicken-egg-providers != '' }}
4456
- name: "Prepare airflow package"
4557
shell: bash
4658
run: >

‎.github/workflows/build-images.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,7 @@ jobs:
7474
is-arm-runner: ${{ steps.selective-checks.outputs.is-arm-runner }}
7575
is-vm-runner: ${{ steps.selective-checks.outputs.is-vm-runner }}
7676
is-k8s-runner: ${{ steps.selective-checks.outputs.is-k8s-runner }}
77+
chicken-egg-providers: ${{ steps.selective-checks.outputs.chicken-egg-providers }}
7778
target-commit-sha: "${{steps.discover-pr-merge-commit.outputs.target-commit-sha ||
7879
github.event.pull_request.head.sha ||
7980
github.sha
@@ -293,6 +294,7 @@ jobs:
293294
uses: ./.github/actions/build-prod-images
294295
with:
295296
build-provider-packages: ${{ needs.build-info.outputs.default-branch == 'main' }}
297+
chicken-egg-providers: ${{ needs.build-info.outputs.chicken-egg-providers }}
296298
env:
297299
UPGRADE_TO_NEWER_DEPENDENCIES: ${{ needs.build-info.outputs.upgrade-to-newer-dependencies }}
298300
DOCKER_CACHE: ${{ needs.build-info.outputs.cache-directive }}

‎.github/workflows/ci.yml

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -121,6 +121,7 @@ jobs:
121121
is-arm-runner: ${{ steps.selective-checks.outputs.is-arm-runner }}
122122
is-vm-runner: ${{ steps.selective-checks.outputs.is-vm-runner }}
123123
is-k8s-runner: ${{ steps.selective-checks.outputs.is-k8s-runner }}
124+
chicken-egg-providers: ${{ steps.selective-checks.outputs.chicken-egg-providers }}
124125
has-migrations: ${{ steps.selective-checks.outputs.has-migrations }}
125126
source-head-repo: ${{ steps.source-run-info.outputs.source-head-repo }}
126127
pull-request-labels: ${{ steps.source-run-info.outputs.pr-labels }}
@@ -474,6 +475,7 @@ jobs:
474475
RUNS_ON: "${{ needs.build-info.outputs.runs-on }}"
475476
PYTHON_VERSIONS: ${{needs.build-info.outputs.all-python-versions-list-as-string}}
476477
DEBUG_RESOURCES: ${{needs.build-info.outputs.debug-resources}}
478+
VERSION_SUFFIX_FOR_PYPI: "dev0"
477479
if: needs.build-info.outputs.ci-image-build == 'true'
478480
steps:
479481
- name: Cleanup repo
@@ -502,12 +504,23 @@ jobs:
502504
run: >
503505
breeze release-management generate-constraints --run-in-parallel
504506
--airflow-constraints-mode constraints-no-providers
507+
- name: "Prepare chicken-eggs provider packages"
508+
# In case of provider packages which use latest dev0 version of providers, we should prepare them
509+
# from the source code, not from the PyPI because they have apache-airflow>=X.Y.Z dependency
510+
# And when we prepare them from sources they will have apache-airflow>=X.Y.Z.dev0
511+
shell: bash
512+
run: >
513+
breeze release-management prepare-provider-packages
514+
--package-format wheel --version-suffix-for-pypi dev0
515+
${{ needs.build-info.outputs.chicken-egg-providers }}
516+
if: needs.build-info.outputs.chicken-egg-providers != ''
505517
- name: "PyPI constraints"
506518
shell: bash
507519
timeout-minutes: 25
508520
run: >
509521
breeze release-management generate-constraints --run-in-parallel
510522
--airflow-constraints-mode constraints
523+
--chicken-egg-providers "${{ needs.build-info.outputs.chicken-egg-providers }}"
511524
- name: "Dependency upgrade summary"
512525
shell: bash
513526
run: |
@@ -1608,6 +1621,7 @@ jobs:
16081621
if: needs.build-info.outputs.in-workflow-build == 'true'
16091622
with:
16101623
build-provider-packages: ${{ needs.build-info.outputs.default-branch == 'main' }}
1624+
chicken-egg-providers: ${{ needs.build-info.outputs.chicken-egg-providers }}
16111625
env:
16121626
UPGRADE_TO_NEWER_DEPENDENCIES: ${{ needs.build-info.outputs.upgrade-to-newer-dependencies }}
16131627
DOCKER_CACHE: ${{ needs.build-info.outputs.cache-directive }}
@@ -1647,6 +1661,7 @@ jobs:
16471661
uses: ./.github/actions/build-prod-images
16481662
with:
16491663
build-provider-packages: ${{ needs.build-info.outputs.default-branch == 'main' }}
1664+
chicken-egg-providers: ${{ needs.build-info.outputs.chicken-egg-providers }}
16501665
env:
16511666
UPGRADE_TO_NEWER_DEPENDENCIES: ${{ needs.build-info.outputs.upgrade-to-newer-dependencies }}
16521667
DOCKER_CACHE: ${{ needs.build-info.outputs.cache-directive }}

‎airflow/providers/amazon/provider.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -753,7 +753,7 @@ additional-extras:
753753
- apache-airflow-providers-cncf-kubernetes>=7.2.0
754754
- name: s3fs
755755
dependencies:
756-
- s3fs>=2023.9.2
756+
- s3fs>=2023.10.0
757757
- name: python3-saml
758758
dependencies:
759759
- python3-saml>=1.16.0

‎airflow/providers/google/provider.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -87,7 +87,7 @@ dependencies:
8787
- gcloud-aio-auth>=4.0.0,<5.0.0
8888
- gcloud-aio-bigquery>=6.1.2
8989
- gcloud-aio-storage
90-
- gcsfs>=2023.9.2
90+
- gcsfs>=2023.10.0
9191
- google-ads>=22.1.0
9292
- google-api-core>=2.11.0
9393
- google-api-python-client>=1.6.0

‎airflow/providers/microsoft/azure/provider.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -70,7 +70,7 @@ versions:
7070

7171
dependencies:
7272
- apache-airflow>=2.5.0
73-
- adlfs>=2023.9.2
73+
- adlfs>=2023.10.0
7474
- azure-batch>=8.0.0
7575
- azure-cosmos>=4.0.0
7676
- azure-mgmt-cosmosdb

‎dev/breeze/src/airflow_breeze/commands/release_management_commands.py

Lines changed: 18 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -684,21 +684,30 @@ def run_generate_constraints_in_parallel(
684684
@option_image_tag_for_running
685685
@option_debug_release_management
686686
@option_airflow_constraints_mode_ci
687+
@click.option(
688+
"--chicken-egg-providers",
689+
default="",
690+
help="List of chicken-egg provider packages - "
691+
"those that have airflow_version >= current_version and should "
692+
"be installed in CI from locally built packages with >= current_version.dev0 ",
693+
envvar="CHICKEN_EGG_PROVIDERS",
694+
)
687695
@option_github_repository
688696
@option_verbose
689697
@option_dry_run
690698
@option_answer
691699
def generate_constraints(
700+
airflow_constraints_mode: str,
701+
debug: bool,
702+
debug_resources: bool,
703+
github_repository: str,
704+
image_tag: str | None,
705+
parallelism: int,
692706
python: str,
707+
python_versions: str,
693708
run_in_parallel: bool,
694-
parallelism: int,
695709
skip_cleanup: bool,
696-
debug_resources: bool,
697-
python_versions: str,
698-
image_tag: str | None,
699-
debug: bool,
700-
airflow_constraints_mode: str,
701-
github_repository: str,
710+
chicken_egg_providers: str,
702711
):
703712
perform_environment_checks()
704713
check_remote_ghcr_io_commands()
@@ -742,6 +751,7 @@ def generate_constraints(
742751
python=python,
743752
github_repository=github_repository,
744753
airflow_constraints_mode=airflow_constraints_mode,
754+
chicken_egg_providers=chicken_egg_providers,
745755
)
746756
for python in python_version_list
747757
]
@@ -762,6 +772,7 @@ def generate_constraints(
762772
skip_image_upgrade_check=True,
763773
quiet=True,
764774
airflow_constraints_mode=airflow_constraints_mode,
775+
chicken_egg_providers=chicken_egg_providers,
765776
)
766777
return_code, info = run_generate_constraints(
767778
shell_params=shell_params,

‎dev/breeze/src/airflow_breeze/commands/release_management_commands_config.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -134,6 +134,7 @@
134134
"--image-tag",
135135
"--python",
136136
"--airflow-constraints-mode",
137+
"--chicken-egg-providers",
137138
"--debug",
138139
"--github-repository",
139140
],

‎dev/breeze/src/airflow_breeze/params/shell_params.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -118,6 +118,7 @@ class ShellParams:
118118
builder: str = "autodetect"
119119
celery_broker: str = DEFAULT_CELERY_BROKER
120120
celery_flower: bool = False
121+
chicken_egg_providers: str = ""
121122
collect_only: bool = False
122123
database_isolation: bool = False
123124
db_reset: bool = False
@@ -435,6 +436,7 @@ def env_variables_for_docker_commands(self) -> _Environ:
435436
_set_var(_env, "BREEZE", "true")
436437
_set_var(_env, "BREEZE_INIT_COMMAND", None, "")
437438
_set_var(_env, "CELERY_FLOWER", self.celery_flower)
439+
_set_var(_env, "CHICKEN_EGG_PROVIDERS", self.chicken_egg_providers)
438440
_set_var(_env, "CI", None, "false")
439441
_set_var(_env, "CI_BUILD_ID", None, "0")
440442
_set_var(_env, "CI_EVENT_TYPE", None, "pull_request")

‎dev/breeze/src/airflow_breeze/utils/packages.py

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -56,10 +56,6 @@
5656
# TODO: use single source of truth for those
5757
# for now we need to keep them in sync with the ones in setup.py
5858
PREINSTALLED_PROVIDERS = [
59-
# Until we cut off the 2.8.0 branch and bump current airflow version to 2.9.0, we should
60-
# Keep common.io commented out in order ot be able to generate PyPI constraints because
61-
# The version from PyPI has requirement of apache-airflow>=2.8.0
62-
# "common.io",
6359
"common.sql",
6460
"ftp",
6561
"http",

‎dev/breeze/src/airflow_breeze/utils/selective_checks.py

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1019,3 +1019,8 @@ def mssql_parallelism(self) -> int:
10191019
@cached_property
10201020
def has_migrations(self) -> bool:
10211021
return any([file.startswith("airflow/migrations/") for file in self._files])
1022+
1023+
@cached_property
1024+
def chicken_egg_providers(self) -> str:
1025+
"""Space separated list of providers with chicken-egg problem and should be built from sources."""
1026+
return "common.io"

‎docs/apache-airflow-providers-google/index.rst

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -116,7 +116,7 @@ PIP package Version required
116116
``gcloud-aio-auth`` ``>=4.0.0,<5.0.0``
117117
``gcloud-aio-bigquery`` ``>=6.1.2``
118118
``gcloud-aio-storage``
119-
``gcsfs`` ``>=2023.9.2``
119+
``gcsfs`` ``>=2023.10.0``
120120
``google-ads`` ``>=22.1.0``
121121
``google-api-core`` ``>=2.11.0``
122122
``google-api-python-client`` ``>=1.6.0``

‎docs/apache-airflow-providers-microsoft-azure/index.rst

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -106,7 +106,7 @@ The minimum Apache Airflow version supported by this provider package is ``2.5.0
106106
PIP package Version required
107107
================================ ==================
108108
``apache-airflow`` ``>=2.5.0``
109-
``adlfs`` ``>=2023.9.2``
109+
``adlfs`` ``>=2023.10.0``
110110
``azure-batch`` ``>=8.0.0``
111111
``azure-cosmos`` ``>=4.0.0``
112112
``azure-mgmt-cosmosdb``

‎generated/provider_dependencies.json

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -415,7 +415,7 @@
415415
"gcloud-aio-auth>=4.0.0,<5.0.0",
416416
"gcloud-aio-bigquery>=6.1.2",
417417
"gcloud-aio-storage",
418-
"gcsfs>=2023.9.2",
418+
"gcsfs>=2023.10.0",
419419
"google-ads>=22.1.0",
420420
"google-api-core>=2.11.0",
421421
"google-api-python-client>=1.6.0",
@@ -557,7 +557,7 @@
557557
"microsoft.azure": {
558558
"deps": [
559559
"adal>=1.2.7",
560-
"adlfs>=2023.9.2",
560+
"adlfs>=2023.10.0",
561561
"apache-airflow>=2.5.0",
562562
"azure-batch>=8.0.0",
563563
"azure-cosmos>=4.0.0",

0 commit comments

Comments
 (0)
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy