Content-Length: 314962 | pFad | http://github.com/apache/airflow/issues/50739

F6 Scheduler always connect to api-server via http and port 8080 · Issue #50739 · apache/airflow · GitHub
Skip to content

Scheduler always connect to api-server via http and port 8080 #50739

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
1 of 2 tasks
SzymonZy opened this issue May 17, 2025 · 6 comments
Closed
1 of 2 tasks

Scheduler always connect to api-server via http and port 8080 #50739

SzymonZy opened this issue May 17, 2025 · 6 comments
Labels
area:core area:Scheduler including HA (high availability) scheduler kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet

Comments

@SzymonZy
Copy link

SzymonZy commented May 17, 2025 •
edited
Loading

Apache Airflow version

3.0.1

If "Other Airflow 2 version" selected, which one?

3.0.1

What happened?

Airflow installed in venv via PyPi (along official guide). When airflow-apiserver started on port other than 8080 and/or HTTPS will not be able to run any tasks. Airflow scheduler is accepting tasks from api-server but than pings back to api server always on http and port 8080 on localhost. I don't see any documentation that will allow to change this behaviour. It looks like this connection is hardcoded.
I'm using local executor.

What you think should happen instead?

Scheduler should sent request to local executor running on api-server within port and protocol on which api server was run along with local executor.

How to reproduce

  1. Start test run via airflow standalone and verify DAGs run correctly.
  2. Reconfigure api-server to port different than 8080 or enable https.
  3. Enable debug log level.
  4. Run airflow apiserver with above configuration.
  5. Run airflow scheduler. Try to execute any example DAG.
  6. Look into scheduler debug log, it will put DAG into qeued status, try 4 times and fail with connection timeout/refeused error on http://localhost:8080. Depending on port and protocol setup.

Documentation does not include any details how to force airflow scheduler to use proper port/protocol. I suspect it should be read out from database where airflow api-server should register itself when started.

Operating System

SLES 15.5 and SLES 15.6, debian bookworm.

Versions of Apache Airflow Providers

not relevant

Deployment

Virtualenv installation

Deployment details

Python 3.11

Anything else?

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct
@SzymonZy SzymonZy added kind:bug This is a clearly a bug area:core needs-triage label for new issues that we didn't triage yet labels May 17, 2025
@boring-cyborg boring-cyborg
Copy link

boring-cyborg bot commented May 17, 2025

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

@dosubot dosubot bot added the area:Scheduler including HA (high availability) scheduler label May 17, 2025
@tirkarthi
Copy link
Contributor

Does changing execution_api_server_url value help here?

https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#execution-api-server-url

The url of the execution api server. Default is {BASE_URL}/execution/ where {BASE_URL} is the base url of the API Server. If {BASE_URL} is not set, it will use http://localhost:8080 as the default base url.

@SzymonZy
Copy link
Author

SzymonZy commented May 19, 2025 •
edited
Loading

It gives some progress
below is diff from ariflow.cfg changes done:

{
< execution_api_server_url = https://FQDN.com:7443
<
---
> # execution_api_server_url =
>
1186c1186
< base_url = https://FQDN.com:7443
---
> # base_url =
1208c1208
< port = 7443
---
> port = 8080
1233c1233
< ssl_cert = /opt/airflow/certificates/FQDN.cer
---
> ssl_cert =
1240c1240
< ssl_key = /opt/airflow/certificates/FQDN.pkcs8
---
> ssl_key =
}

and in venv:

    export SSL_CERT_FILE=/opt/airflow/.venv/chain.crt

After restart of api-server and scheduler I'm getting error: Method not allowed when dag should run.

Scheduler log:

{
2025-05-19 08:44:30 [info     ] Secrets backends loaded for worker [supervisor] backend_classes=['EnvironmentVariablesBackend'] count=1
[2025-05-19T08:44:30.089+0200] {_trace.py:47} DEBUG - connect_tcp.started host='FQDN.com' port=7443 local_address=None timeout=5.0 socket_options=None
[2025-05-19T08:44:30.090+0200] {_trace.py:47} DEBUG - connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7ff6ba60c7d0>
[2025-05-19T08:44:30.091+0200] {_trace.py:47} DEBUG - start_tls.started ssl_context=<ssl.SSLContext object at 0x7ff6ba6a1d90> server_hostname='FQDN.com' timeout=5.0
[2025-05-19T08:44:30.093+0200] {_trace.py:47} DEBUG - start_tls.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7ff6ba4867d0>
[2025-05-19T08:44:30.094+0200] {_trace.py:47} DEBUG - send_request_headers.started request=<Request [b'PATCH']>
[2025-05-19T08:44:30.095+0200] {_trace.py:47} DEBUG - send_request_headers.complete
[2025-05-19T08:44:30.095+0200] {_trace.py:47} DEBUG - send_request_body.started request=<Request [b'PATCH']>
[2025-05-19T08:44:30.095+0200] {_trace.py:47} DEBUG - send_request_body.complete
[2025-05-19T08:44:30.095+0200] {_trace.py:47} DEBUG - receive_response_headers.started request=<Request [b'PATCH']>
[2025-05-19T08:44:30.096+0200] {_trace.py:47} DEBUG - receive_response_headers.complete return_value=(b'HTTP/1.1', 405, b'Method Not Allowed', [(b'date', b'Mon, 19 May 2025 06:44:29 GMT'), (b'server', b'uvicorn'), (b'allow', b'GET'), (b'content-type', b'application/json'), (b'vary', b'Accept-Encoding'), (b'content-encoding', b'gzip'), (b'transfer-encoding', b'chunked')])
[2025-05-19T08:44:30.098+0200] {_client.py:1026} INFO - HTTP Request: PATCH https://FQDN.com:7443/task-instances/0196e744-93cc-7275-9aed-a540b238e934/run "HTTP/1.1 405 Method Not Allowed"
[2025-05-19T08:44:30.098+0200] {_trace.py:47} DEBUG - receive_response_body.started request=<Request [b'PATCH']>
[2025-05-19T08:44:30.098+0200] {_trace.py:47} DEBUG - receive_response_body.complete
[2025-05-19T08:44:30.099+0200] {_trace.py:47} DEBUG - response_closed.started
[2025-05-19T08:44:30.099+0200] {_trace.py:47} DEBUG - response_closed.complete
2025-05-19 08:44:30 [warning  ] Server error                   [airflow.sdk.api.client] detail=None
2025-05-19 08:44:30 [info     ] Process exited                 [supervisor] exit_code=<Negsignal.SIGKILL: -9> pid=58090 signal_sent=SIGKILL
[2025-05-19T08:44:30.105+0200] {local_executor.py:96} ERROR - uhoh
Traceback (most recent call last):
  File "/opt/airflow/.venv/lib64/python3.11/site-packages/airflow/executors/local_executor.py", line 92, in _run_worker
    _execute_work(log, workload)
  File "/opt/airflow/.venv/lib64/python3.11/site-packages/airflow/executors/local_executor.py", line 120, in _execute_work
    supervise(
  File "/opt/airflow/.venv/lib64/python3.11/site-packages/airflow/sdk/execution_time/supervisor.py", line 1338, in supervise
    process = ActivitySubprocess.start(
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/airflow/.venv/lib64/python3.11/site-packages/airflow/sdk/execution_time/supervisor.py", line 791, in start
    proc._on_child_started(ti=what, dag_rel_path=dag_rel_path, bundle_info=bundle_info)
  File "/opt/airflow/.venv/lib64/python3.11/site-packages/airflow/sdk/execution_time/supervisor.py", line 802, in _on_child_started
    ti_context = self.client.task_instances.start(ti.id, self.pid, start_date)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/airflow/.venv/lib64/python3.11/site-packages/airflow/sdk/api/client.py", line 146, in start
    resp = self.client.patch(f"task-instances/{id}/run", content=body.model_dump_json())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/airflow/.venv/lib64/python3.11/site-packages/httpx/_client.py", line 1219, in patch
    return self.request(
           ^^^^^^^^^^^^^
  File "/opt/airflow/.venv/lib64/python3.11/site-packages/tenacity/__init__.py", line 338, in wrapped_f
    return copy(f, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^
  File "/opt/airflow/.venv/lib64/python3.11/site-packages/tenacity/__init__.py", line 477, in __call__
    do = self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/airflow/.venv/lib64/python3.11/site-packages/tenacity/__init__.py", line 378, in iter
    result = action(retry_state)
             ^^^^^^^^^^^^^^^^^^^
  File "/opt/airflow/.venv/lib64/python3.11/site-packages/tenacity/__init__.py", line 400, in <lambda>
    self._add_action_func(lambda rs: rs.outcome.result())
                                     ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/opt/airflow/.venv/lib64/python3.11/site-packages/tenacity/__init__.py", line 480, in __call__
    result = fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^
  File "/opt/airflow/.venv/lib64/python3.11/site-packages/airflow/sdk/api/client.py", line 680, in request
    return super().request(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/airflow/.venv/lib64/python3.11/site-packages/httpx/_client.py", line 827, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/airflow/.venv/lib64/python3.11/site-packages/httpx/_client.py", line 914, in send
    response = self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/airflow/.venv/lib64/python3.11/site-packages/httpx/_client.py", line 942, in _send_handling_auth
    response = self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/airflow/.venv/lib64/python3.11/site-packages/httpx/_client.py", line 999, in _send_handling_redirects
    raise exc
  File "/opt/airflow/.venv/lib64/python3.11/site-packages/httpx/_client.py", line 982, in _send_handling_redirects
    hook(response)
  File "/opt/airflow/.venv/lib64/python3.11/site-packages/airflow/sdk/api/client.py", line 117, in raise_on_4xx_5xx_with_note
    return get_json_error(response) or response.raise_for_status()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/airflow/.venv/lib64/python3.11/site-packages/airflow/sdk/api/client.py", line 107, in get_json_error
    raise err
airflow.sdk.api.client.ServerResponseError: Method Not Allowed
Correlation-id=0196e749-6908-768c-bccf-208725217196
}

api-server (webserver) log:

{
INFO:     172.28.62.131:52665 - "GET /api/v2/dags/test/dagRuns/manual__2025-05-19T06%3A39%3A13.341480%2B00%3A00/taskInstances?limit=50&offset=0&order_by=-start_date HTTP/1.1" 200 OK
INFO:     10.252.212.37:58130 - "PATCH /task-instances/0196e744-93cc-7275-9aed-a540b238e934/run HTTP/1.1" 405 Method Not Allowed
}

Manual
curl -X PATCH https://FQDN.com:7443/task-instances/0196e744-93cc-7275-9aed-a540b238e934/run
returns
{"detail":"Method Not Allowed"}

If run on 8080 and http scheduler log shows that PATCH method is accepted:

{[2025-05-19T09:32:02.885+0200] {_trace.py:47} DEBUG - connect_tcp.started host='localhost' port=8080 local_address=None timeout=5.0 socket_options=None
[2025-05-19T09:32:02.886+0200] {_trace.py:47} DEBUG - connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7f73f305a9d0>
[2025-05-19T09:32:02.887+0200] {_trace.py:47} DEBUG - send_request_headers.started request=<Request [b'PATCH']>
[2025-05-19T09:32:02.887+0200] {_trace.py:47} DEBUG - send_request_headers.complete
[2025-05-19T09:32:02.888+0200] {_trace.py:47} DEBUG - send_request_body.started request=<Request [b'PATCH']>
[2025-05-19T09:32:02.888+0200] {_trace.py:47} DEBUG - send_request_body.complete
[2025-05-19T09:32:02.888+0200] {_trace.py:47} DEBUG - receive_response_headers.started request=<Request [b'PATCH']>
[2025-05-19T09:32:03.300+0200] {_trace.py:47} DEBUG - receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'date', b'Mon, 19 May 2025 07:32:02 GMT'), (b'server', b'uvicorn'), (b'content-type', b'application/json'), (b'airflow-api-version', b'2025-04-28'), (b'vary', b'Accept-Encoding'), (b'content-encoding', b'gzip'), (b'transfer-encoding', b'chunked')])
[2025-05-19T09:32:03.302+0200] {_client.py:1026} INFO - HTTP Request: PATCH http://localhost:8080/execution/task-instances/0196e774-ed2e-796f-8904-23e22e343f83/run "HTTP/1.1 200 OK"
[2025-05-19T09:32:03.303+0200] {_trace.py:47} DEBUG - receive_response_body.started request=<Request [b'PATCH']>
[2025-05-19T09:32:03.303+0200] {_trace.py:47} DEBUG - receive_response_body.complete
[2025-05-19T09:32:03.304+0200] {_trace.py:47} DEBUG - response_closed.started
[2025-05-19T09:32:03.304+0200] {_trace.py:47} DEBUG - response_closed.complete
2025-05-19 09:32:03 [debug    ] Sending                        [supervisor] msg=StartupDetails(ti=TaskInstance(id=UUID('0196e774-ed2e-796f-8904-23e22e343f83'), task_id='my_task', dag_id='example_simplest_dag', run_id='manual__2025-05-19T07:32:01.953743+00:00', try_number=1, map_index=-1, pool_slots=1, queue='default', priority_weight=1, executor_config={}, parent_context_carrier={}, context_carrier=None, queued_dttm=None), dag_rel_path='example_simplest_dag.py', bundle_info=BundleInfo(name='example_dags', version=None), requests_fd=25, start_date=datetime.datetime(2025, 5, 19, 7, 32, 2, 881185, tzinfo=datetime.timezone.utc), ti_context=TIRunContext(dag_run=DagRun(dag_id='example_simplest_dag', run_id='manual__2025-05-19T07:32:01.953743+00:00', logical_date=datetime.datetime(2025, 5, 19, 7, 32, 0, 988000, tzinfo=TzInfo(UTC)), data_interval_start=datetime.datetime(2025, 5, 19, 7, 32, 0, 988000, tzinfo=TzInfo(UTC)), data_interval_end=datetime.datetime(2025, 5, 19, 7, 32, 0, 988000, tzinfo=TzInfo(UTC)), run_after=datetime.datetime(2025, 5, 19, 7, 32, 0, 988000, tzinfo=TzInfo(UTC)), start_date=datetime.datetime(2025, 5, 19, 7, 32, 2, 716496, tzinfo=TzInfo(UTC)), end_date=None, clear_number=0, run_type=<DagRunType.MANUAL: 'manual'>, conf={}, consumed_asset_events=[]), task_reschedule_count=0, max_tries=0, variables=[], connections=[], upstream_map_indexes={}, next_method=None, next_kwargs=None, xcom_keys_to_clear=[], should_retry=False), type='StartupDetails')
[2025-05-19T09:32:03.308+0200] {_trace.py:47} DEBUG - send_request_headers.started request=<Request [b'PUT']>
[2025-05-19T09:32:03.308+0200] {_trace.py:47} DEBUG - send_request_headers.complete
[2025-05-19T09:32:03.308+0200] {_trace.py:47} DEBUG - send_request_body.started request=<Request [b'PUT']>
[2025-05-19T09:32:03.309+0200] {_trace.py:47} DEBUG - send_request_body.complete
[2025-05-19T09:32:03.309+0200] {_trace.py:47} DEBUG - receive_response_headers.started request=<Request [b'PUT']>
[2025-05-19T09:32:03.326+0200] {_trace.py:47} DEBUG - receive_response_headers.complete return_value=(b'HTTP/1.1', 204, b'No Content', [(b'date', b'Mon, 19 May 2025 07:32:02 GMT'), (b'server', b'uvicorn'), (b'content-type', b'application/json'), (b'airflow-api-version', b'2025-04-28')])
[2025-05-19T09:32:03.327+0200] {_client.py:1026} INFO - HTTP Request: PUT http://localhost:8080/execution/task-instances/0196e774-ed2e-796f-8904-23e22e343f83/heartbeat "HTTP/1.1 204 No Content"
[2025-05-19T09:32:03.327+0200] {_trace.py:47} DEBUG - receive_response_body.started request=<Request [b'PUT']>
[2025-05-19T09:32:03.328+0200] {_trace.py:47} DEBUG - receive_response_body.complete
[2025-05-19T09:32:03.328+0200] {_trace.py:47} DEBUG - response_closed.started
[2025-05-19T09:32:03.328+0200] {_trace.py:47} DEBUG - response_closed.complete
2025-05-19 09:32:03 [debug    ] Received message from task runner [supervisor] msg=SetRenderedFields(rendered_fields={'templates_dict': None, 'op_args': [], 'op_kwargs': {}}, type='SetRenderedFields')
[2025-05-19T09:32:03.330+0200] {_trace.py:47} DEBUG - send_request_headers.started request=<Request [b'PUT']>
[2025-05-19T09:32:03.330+0200] {_trace.py:47} DEBUG - send_request_headers.complete
[2025-05-19T09:32:03.331+0200] {_trace.py:47} DEBUG - send_request_body.started request=<Request [b'PUT']>
[2025-05-19T09:32:03.331+0200] {_trace.py:47} DEBUG - send_request_body.complete
[2025-05-19T09:32:03.331+0200] {_trace.py:47} DEBUG - receive_response_headers.started request=<Request [b'PUT']>
[2025-05-19T09:32:03.361+0200] {_trace.py:47} DEBUG - receive_response_headers.complete return_value=(b'HTTP/1.1', 201, b'Created', [(b'date', b'Mon, 19 May 2025 07:32:02 GMT'), (b'server', b'uvicorn'), (b'content-type', b'application/json'), (b'airflow-api-version', b'2025-04-28'), (b'vary', b'Accept-Encoding'), (b'content-encoding', b'gzip'), (b'transfer-encoding', b'chunked')])
[2025-05-19T09:32:03.361+0200] {_client.py:1026} INFO - HTTP Request: PUT http://localhost:8080/execution/task-instances/0196e774-ed2e-796f-8904-23e22e343f83/rtif "HTTP/1.1 201 Created"
[2025-05-19T09:32:03.361+0200] {_trace.py:47} DEBUG - receive_response_body.started request=<Request [b'PUT']>
[2025-05-19T09:32:03.362+0200] {_trace.py:47} DEBUG - receive_response_body.complete
[2025-05-19T09:32:03.362+0200] {_trace.py:47} DEBUG - response_closed.started
[2025-05-19T09:32:03.362+0200] {_trace.py:47} DEBUG - response_closed.complete
2025-05-19 09:32:03 [debug    ] Received message from task runner [supervisor] msg=SucceedTask(state='success', end_date=datetime.datetime(2025, 5, 19, 7, 32, 3, 327006, tzinfo=TzInfo(UTC)), task_outlets=[], outlet_events=[], rendered_map_index=None, type='SucceedTask')
[2025-05-19T09:32:03.363+0200] {_trace.py:47} DEBUG - send_request_headers.started request=<Request [b'PATCH']>
[2025-05-19T09:32:03.364+0200] {_trace.py:47} DEBUG - send_request_headers.complete
[2025-05-19T09:32:03.364+0200] {_trace.py:47} DEBUG - send_request_body.started request=<Request [b'PATCH']>
[2025-05-19T09:32:03.364+0200] {_trace.py:47} DEBUG - send_request_body.complete
[2025-05-19T09:32:03.364+0200] {_trace.py:47} DEBUG - receive_response_headers.started request=<Request [b'PATCH']>
[2025-05-19T09:32:03.387+0200] {_trace.py:47} DEBUG - receive_response_headers.complete return_value=(b'HTTP/1.1', 204, b'No Content', [(b'date', b'Mon, 19 May 2025 07:32:02 GMT'), (b'server', b'uvicorn'), (b'content-type', b'application/json'), (b'airflow-api-version', b'2025-04-28')])
[2025-05-19T09:32:03.388+0200] {_client.py:1026} INFO - HTTP Request: PATCH http://localhost:8080/execution/task-instances/0196e774-ed2e-796f-8904-23e22e343f83/state "HTTP/1.1 204 No Content"
[2025-05-19T09:32:03.388+0200] {_trace.py:47} DEBUG - receive_response_body.started request=<Request [b'PATCH']>
[2025-05-19T09:32:03.389+0200] {_trace.py:47} DEBUG - receive_response_body.complete
[2025-05-19T09:32:03.389+0200] {_trace.py:47} DEBUG - response_closed.started
[2025-05-19T09:32:03.389+0200] {_trace.py:47} DEBUG - response_closed.complete
}

@Patchus
Copy link
Contributor

Patchus commented May 20, 2025 •
edited
Loading

i am interested in this as well. I thought maybe these would do it , by adding in PATCH to that allow methods; but no dice. I can get everything working if i turn the webserver's SSL off by commenting out the Key / Cert; but when I add those in, I have the same issue.

[api]
access_control_allow_headers = origen, content-type, accept
access_control_allow_methods = POST, GET, OPTIONS, DELETE
access_control_allow_origens = https://exampleclientapp1.com https://exampleclientapp2.com

I added in the execution_api_server_url to be the https://github.com/mydomain.com:8080 , and i added it to the origens, but still had issues

@SzymonZy
Copy link
Author

At the end it turned out it was misconfiguration on my end. There is no need to set execution_api_server_url.
I will update and close tomorrow with resolution. Generally 3.0.1 documentation need imporvement with examples.

@SzymonZy
Copy link
Author

SzymonZy commented May 21, 2025 •
edited
Loading

Things that are mandatory if you switch to different port and https seems to be:
base_url = https://FQDN.com:8443
port = 8443
ssl_cert = /cert_path
ssl_key = /key_path

Nothing more. execution_api_server_url should not be touched.

Incorrect or not modified base_url and port is causing method not allowed errors. Also messing with execution_api_server_url parameter bring unpredicted behaviour.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area:core area:Scheduler including HA (high availability) scheduler kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@SzymonZy @tirkarthi @Patchus








ApplySandwichStrip

pFad - (p)hone/(F)rame/(a)nonymizer/(d)eclutterfier!      Saves Data!


--- a PPN by Garber Painting Akron. With Image Size Reduction included!

Fetched URL: http://github.com/apache/airflow/issues/50739

Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy