AWS Lambda Executor #50516
AWS Lambda Executor #50516
Conversation
This changeset includes: - The Lambda executor code itself - Tests for the executor - Docs for the executor
o-nikolas
requested review from
potiuk,
ashb,
gopidesupavan and
eladkal
as code owners
boring-cyborg
bot
added
area:production-image
PR tests turned up some things that local testing/pre-commit did not catch. Fix those up.
providers/amazon/src/airflow/providers/amazon/aws/executors/aws_lambda/docker/Dockerfile
Outdated
Show resolved
Hide resolved
providers/amazon/src/airflow/providers/amazon/aws/executors/aws_lambda/lambda_executor.py
Outdated
Show resolved
Hide resolved
providers/amazon/src/airflow/providers/amazon/aws/executors/aws_lambda/lambda_executor.py
Show resolved
Hide resolved
providers/amazon/src/airflow/providers/amazon/aws/executors/aws_lambda/docker/Dockerfile
Show resolved
Hide resolved
providers/amazon/src/airflow/providers/amazon/aws/executors/aws_lambda/lambda_executor.py
Outdated
Show resolved
Hide resolved
providers/amazon/src/airflow/providers/amazon/aws/executors/aws_lambda/lambda_executor.py
Outdated
Show resolved
Hide resolved
|
@o-nikolas LGTM, thanks for this. The lambda function has default retries (2), how about this, do you think in this case retries required? |
Our retries include attempts to even submit the task to lambda (network issues, access token expiration, api failures), so I think it's still worth keeping. But if others agree we should drop it, I'm not completely opposed to that! |
Yeah fine even i am not opposed to it :) |
Sounds good, thanks for the review! |
Passing through clientContext to the lambda function is only supported on synchronous executions.
Overview
Introducing the new AWS Lambda Executor!
As with our previous executors this is an initial release with most functionality in place. However, there are surely future upgrades and iterations to come, so stay tuned!
This changeset includes:
Testing
The unit tests cover 90+% line coverage (ignore the hook coverage, the results were filtered on lambda and that hook is prexisting code not delivered in this PR) and the executor was UAT tested manually in many scenarios.
Review Notes
This is a large PR, which is not entirely ideal. However, it can be difficult to release portions of a component like this and I've scoped all the code here to the minimum required for folks to begin using this executor. There are other refactorings that were left out and will be contributed at a later time to minimize the confusion.
There are clear chunks this can be reviewed in (docs, provider.yaml, tests, and core logic) to split it up.
Ultimately, (almost) all the code is scoped to the Amazon provider package and it is a net new component, so there is a very limited blast radius. Little to no existing user workflows or code should be affected.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.