Segmentation fault during worker shutdown #1651
Comments
|
I wonder if a build with debug enabled might yield more detail? |
|
I am running into a very similar issue with Python 3, running New Relic with Flask's demo app Flaskr (https://github.com/pallets/flask/tree/0.12.2/examples/flaskr), both in a Docker container and directly on the host. # Dockerfile
FROM debian:stretch
RUN apt-get update && \
DEBIAN_FRONTEND=noninteractive apt-get install -qy curl python3-pip
RUN curl -Lo /tmp/flask.tar.gz https://github.com/pallets/flask/archive/0.12.2.tar.gz \
&& mkdir /tmp/flask \
&& tar zxf /tmp/flask.tar.gz -C /tmp/flask --strip-components 1 \
&& mv /tmp/flask/examples/flaskr /root/flaskr \
&& rm -rf /tmp/flask /tmp/flask.tar.gz
WORKDIR /root/flaskr
ENV FLASK_APP=flaskr LC_ALL=C.UTF-8 LANG=C.UTF-8
RUN pip3 install flask==0.12.2 newrelic uwsgi==2.0.15 \
&& pip3 install -e . \
&& flask initdb
CMD newrelic-admin run-program uwsgi --master --disable-logging --enable-threads --single-interpreter --py-call-osafterfork --max-requests 999 --http :8000 --wsgi-file flaskr/flaskr.py --callable app# docker-compose.yml
version: '3'
services:
app:
build: .
environment:
NEW_RELIC_LICENSE_KEY: secret
ports:
- 127.0.0.1:8000:8000When uWSGI instances are stopped (either when the max number of requests is reached or when the server is killed), they randomly crash with a segmentation fault. The problem still happens with option |
|
I have this problem too with python 3.5 and newrelic. |
|
So, in both cases it's calling out to libssl, but breaking in different places. And with the reference to wrapt, am I guessing right you're using "requests" to access a HTTPS endpoint? |
|
We are seeing the same issue. Another data point can't hurt:
We've tried with and without the We're using uwsgi 2.0.15 on Debian Stable with Python 3.5.3. Note we are using New Relic, but no other exotic configurations/options/code. The traceback: |
|
we have the same problem. I removed newrelic for the time being . maybe that is somehow installing an older lib SSL binary. UPDATE = after removing the newrelic library and installing psycopg2 from source this problem went away. Originally I thought this issue is related to psycopg2 installing binary libssl. psycopg/psycopg2#543 but even after installing psycopg2 from source code we are still seeing seeing this error. We are running our django app in this container: |
|
I'm also getting the exact same as @ddehghan, around the same time at which newrelic cannot connect to its api. logger.warning: (Error above possibly due to AWS connectivity issues) Just trying without newrelic to see if that resolves this for now. edit: it did stop the issue :) |
|
So is everyone who has this crash also using NewRelic? |
|
FWIW. Often these crash on shutdown issues are caused by background daemon threads waking up while process exit is occurring and trying to do things after the Python interpreter has started destroying the interpreter, modules and module state. Usually it results in Python exceptions about accessing I have seen these sorts of issues occasionally with mod_wsgi as well, and |
|
Seems I was on the right path, it being a bad interaction with something starting its own thread... |
|
Hello all, I was able to create a minimal reproduction of this segfault without the New Relic agent installed. It seems it has to do with ssl and not threads. import atexit
try:
# PY3
from http.client import HTTPSConnection
except ImportError:
# PY2
from httplib import HTTPSConnection
def myatexit():
HTTPSConnection('https://example.com', port=443)
def application(environ, start_response):
status = '200 OK'
response_headers = [('Content-type', 'text/plain')]
start_response(status, response_headers)
return [b'*']
atexit.register(myatexit)Using this Dockerfile. Note that I get a segfault on both Python 3.6.6 and Python 2.7.15. FROM python:2.7.15
#FROM python:3.6.6
WORKDIR /data
RUN pip install uwsgi==2.0.17.1
CMD uwsgi \
--http :8000 \
--wsgi app:application \
--max-requests 3 \
--min-worker-lifetime 0 \
--master
ADD app.py /dataThe reason New Relic keeps popping up is because the agent makes requests to the backend services during an atexit call using ssl. |
Some plugins (such as python) rely on plugin-level atexit callbacks to be invoked before interpreter teardown (when system-level atexit callbacks are invoked). Prior to this change uWSGI was loading Python applications after registering atexit functions to invoke plugin-level atexit callbacks. The segfault could be triggered if loading a Python application that contained a C extension that registered an atexit callback. Those atexit callbacks would be triggered prior to full termination of the Python program. As a result, Python would be operating on a C extension that was already in a cleaned-up state. This fix makes sure all uwsgi plugin atexit callbacks (which operate on the Python level, for example) are run prior to atexit callbacks that were scheduled through a C extension loading. This way, all extensions will be in a consistent state at the time that uwsgi runs the Python atexit callbacks.
|
Hi there, Hope this issue is finally fixed in Thank you! |
|
Hello. After trying new release the issue still remains on python 3.7: gevent==1.4.0 [uwsgi] DAMN ! worker 1 (pid: 109) died :( trying respawn ... |
|
Hi there @romabiker ! Looking at the stack trace posted, it appears possible that may be an unrelated crash since |
|
@a-feld Thank you for your help and comment. The real reason of my Segmentation Fault turned out in misconfigured uwsgi ini file. |
|
|
@romabiker - What was the misconfiguration in your uwsgi.ini that was causing the segfault? |
|
uWSGI upgrade to 2.0.18 didn't resolve issue for me, but upgrade of psycopg2 from 2.7.3.2 to the latest 2.8.3 fixed things for me. |
It didnt work for me. |
@romabiker could you please share what you've found out? |
|
I guess that it was resolved as pyscopg2 > 2.8 is now a source package so the libssl it is linking against is the very same of uwsgi. |
|
@sylvioneto It's on python 3.6.8 |
|
@sylvioneto , @tianlilong , @kmans As I remember I've change threads and processes by workers. There's working config for me below: |
psycopg2-binary 2.8.3 did solve the problem on my side. Perhaps this is relative to me using the python installation method with a non sentry-tested postgresql version (I'm using the postgresql-11.5 available with Debian 10) |
Getting segfault on worker shutdown but due to |
|
that is right for me: |
|
As well am getting the same segfaults with google/protobuf/pyext, as @string-areeb |
|
I am also getting issue on python3.6 /opt/virtualenv/<app_name>/bin/uwsgi(uwsgi_backtrace+0x2e) [0x482b1e] |
|
I am getting similar issue, I am using google vision api in flask application. 2021-02-08 18:18:34,073 GoogleVisionOCR DEBUG start to invoke google vision api |
|
This problem was related to grpc. resolved in this thread grpc/grpc#23796 |
|
I have the same problem and you solve it by installing this version |
|
also getting the same issue |
I get occasional segfaults from uWSGI when a worker shuts down (for example, by reaching its
max-requestssetting). I haven't been able to reproduce the issue consistently so far, but the stacktrace gives a hint thatopensslof version 1.1.0 might be involved (hence, may be related to #1395, and probably to #1495 and #1569). The issue is not critical (connections are not dropped for real users), but it is still annoying.uWSGIin our case is managed by Supervisor, and runs a Django app.Versions:
How I start uWSGI:
Full stacktrace:
Any input is welcome.
The text was updated successfully, but these errors were encountered: