For RM: This is a breaking change since the old unerscore-prefixed functions are no longer available. The next provider release needs to bump the major version.

For RM: This is a breaking change since the old unerscore-prefixed functions are no longer available. The next provider release needs to bump the major version.

Actually we can add a backwards-compatible support for it, for a couple of releases. Thoughts?

def _get_credentials():
     .... raise deprecation warning here..
     return get_credentials()

@uranusjr @kaxil I don't think this is a breaking change for users. We have fixed all references of this private function. No one should ever use the private function from outside the provider. We should not guarantee private interface compatibility in my opinion. Otherwise we would have to do it for every private function we change.

@uranusjr @kaxil I don't think this is a breaking change for users. We have fixed all references of this private function. No one should ever use the private function from outside the provider. We should not guarantee private interface compatibility in my opinion. Otherwise we would have to do it for every private function we change.

I think @kaxil is right. Technically speaking _get_credentials() is actually a protected method not private one. Private methods start with double underscores __ . And GoogleBaseHook is almost by definition a Base classs for multiple Hooks so if someone created TheirOwnGoogleHook, using _get_credentials() is not only allowed, but actually very likely. I do believe we need back-compat _get_credentials() as @kaxil suggested.

TIL. For me single underscores was always private. Okay, then I will definitely change my way of writing code 😅 i.e. having so many "internal" functions don't make sense to me.

We can add such a function if you think we should provide backwards-compatibility.

Yeah. i think we should add it.

Context matters. A lot. I think we cannot say 'renaming protected class is always breaking' or renaming protected class is always not breaking'. It very much depends on what we are renaming.

As i explained several times before - my view on deciding whether a change is breaking or not is never 0-1. It should be based on our assessment on how likely it is to break heavily the workflows of a number of our users.

@ashb posted it several times already but this explains it very well:

Python is really a dynamic language and we do not have fixed 'APIs' that would let us always know with 100% certainty that we introduce a breaking change. We have a bit of dualism (similarly as in case of dependencies) that airflow is both an application (when it comes to airflow core) and library intended to be used by our users (in case of DAG authoring). And breaking/not breaking has different meaning in those two cases.

There really no true 'private' methods in Python, the underscores (even the double ones) are really convention and expressions of intention. In this case since this is a public base Hook class and the intention is for users to extend the base class, the single underscore is an intention to use it in classes that users create. And my assessment in this case is that the likelihood of breaking user's workflows is high. Hooks are definitely parts of our 'public' APIs and in this case protected classes are intended for our users to use.

But In case of classes that are not intended to be extended and are purely airflow internal, and not part of the real user APIs, this is a bit different story. There, protected methods are for ourselves - maintainers and developers of Airflow, not for our users. And intention is that if the class is later extended (internally in Airflek core only) those classes should be able to use it. And we have full control where the method is used because it should only be done in Airflow and never in user's code.

In such case, it is less likely that someone bases their workflow on those, and even if they do, they are clearly not doing what was intended. And in this case i'd say renaming a protected class is not a breaking changes.

Sorry for such long comment, but I think it is really important to understand that breaking/not breaking is not as clear 0-1 decision in case of Airflow as you might think.

BTW. When we split providers to separate repos i have a dream (I am thinking hard of various cases it involves) that we will be able to make much clearer distinction and introduce much more explicit and clear APIs that will separate Airflow developers from Airflow users and make such decisions breaking/not breaking much easier. This is - IMHO a bit of necessity to keep the providers in check when they are separated, and i hope we will be able to pull that one off.

Wrong image :) but now the right one is there :)

I totally agree with that breaking changes are not always easy to identify, but in my opinion we are on a much safer side when we define public as being the only user exposed access level and we see internal and private as not..

I personally have never used any private or internal function from an external python library before. If I use it I would comment it and pin the version. And I truly believe that this is and should be the standard.

Also I see "Internal" as being internal to the library - not potential being external.

What is the problem of using public instead of internal?

I think it makes sense that move the discussion to the mailing list.

I will open a discussion shortly.

Btw. I have nothing against moving the method to public. This is even stronger indication 'usr this method as you will', but also 'use it from outside of the class' - this is what the intention of making method public is. In this case, it is likely that the call might be originated - for example when you use the hook in task-flow task.

It's just the breaking effect that worries me and i think we should add a method to deprecate old use.