關于gunicorn與異步兼容性問題：AttributeError: module 'select' has no attribute 'epoll'

背景：

介紹：

在使用gunicorn、Flask & flask-sockets 部署，實現websocket協議中同類消息阻塞，不同類消息不阻塞場景。

異常：

[2024-01-15 10:22:16 +0800] [31655] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/home/venv/lib/python3.10/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
    worker.init_process()
  File "/home/venv/lib/python3.10/site-packages/gunicorn/workers/ggevent.py", line 162, in init_process
    super().init_process()
  File "/home/venv/lib/python3.10/site-packages/gunicorn/workers/base.py", line 133, in init_process
    self.load_wsgi()
  File "/home/venv/lib/python3.10/site-packages/gunicorn/workers/base.py", line 142, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/home/venv/lib/python3.10/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/home/venv/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 49, in load
    return self.load_wsgiapp()
  File "/home/venv/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 39, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/home/venv/lib/python3.10/site-packages/gunicorn/util.py", line 331, in import_app
    mod = importlib.import_module(module)
  File "/home/nlu_common/miniconda3/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
 。。。。。。包含很多具體代碼，沒有意義。。。。。。
  File "/home/venv/lib/python3.10/site-packages/dns/resolver.py", line 38, in <module>
    import dns.query
  File "/home/venv/lib/python3.10/site-packages/dns/query.py", line 52, in <module>
    import httpx
  File "/home/venv/lib/python3.10/site-packages/httpx/__init__.py", line 2, in <module>
    from ._api import delete, get, head, options, patch, post, put, request, stream
  File "/home/venv/lib/python3.10/site-packages/httpx/_api.py", line 4, in <module>
    from ._client import Client
  File "/home/venv/lib/python3.10/site-packages/httpx/_client.py", line 30, in <module>
    from ._transports.default import AsyncHTTPTransport, HTTPTransport
  File "/home/venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 30, in <module>
    import httpcore
  File "/home/venv/lib/python3.10/site-packages/httpcore/__init__.py", line 1, in <module>
    from ._api import request, stream
  File "/home/venv/lib/python3.10/site-packages/httpcore/_api.py", line 5, in <module>
    from ._sync.connection_pool import ConnectionPool
  File "/home/venv/lib/python3.10/site-packages/httpcore/_sync/__init__.py", line 1, in <module>
    from .connection import HTTPConnection
  File "/home/venv/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 12, in <module>
    from .._synchronization import Lock
  File "/home/venv/lib/python3.10/site-packages/httpcore/_synchronization.py", line 13, in <module>
    import trio
  File "/home/venv/lib/python3.10/site-packages/trio/__init__.py", line 22, in <module>
    from ._core import TASK_STATUS_IGNORED as TASK_STATUS_IGNORED  # isort: split
  File "/home/venv/lib/python3.10/site-packages/trio/_core/__init__.py", line 21, in <module>
    from ._local import RunVar, RunVarToken
  File "/home/venv/lib/python3.10/site-packages/trio/_core/_local.py", line 9, in <module>
    from . import _run
  File "/home/venv/lib/python3.10/site-packages/trio/_core/_run.py", line 2787, in <module>
    from ._io_epoll import (
  File "/home/venv/lib/python3.10/site-packages/trio/_core/_io_epoll.py", line 202, in <module>
    class EpollIOManager:
  File "/home/venv/lib/python3.10/site-packages/trio/_core/_io_epoll.py", line 203, in EpollIOManager
    _epoll: select.epoll = attr.ib(factory=select.epoll)
AttributeError: module 'select' has no attribute 'epoll'. Did you mean: 'poll'?
[2024-01-15 10:22:16 +0800] [31655] [INFO] Worker exiting (pid: 31655)
[2024-01-15 10:22:17 +0800] [31652] [INFO] Shutting down: Master
[2024-01-15 10:22:17 +0800] [31652] [INFO] Reason: Worker failed to boot.

表現：AttributeError: module 'select' has no attribute 'epoll'. Did you mean: 'poll'?

項目介紹

框架使用Flask + flask-sockets 搭建
內部業務使用異步轉同步，例如：asnyc、httpx等
交互協議http、grpc、websocket
部署：gunicorn、gevent

問題分析

部署方式分析：

非Gunicorn部署（window & linux）

from http_server.handle import app

from gevent import pywsgi
from geventwebsocket.handler import WebSocketHandler

server = pywsgi.WSGIServer(('0.0.0.0', 50003), app, handler_class=WebSocketHandler)
server.serve_forever()
可正常提供服務
epoll支持：Python官方文檔
select.epoll([suzegubt=-1])
    (only supported on Linux 2.5.44 and newer.) Returns an edge polling object,
    which can be used as Edge or Level Triggered interface for I/O events;
    see section Edge and Level Trigger Polling (epoll) Objects below for the methods supported by epolling objects.

Gunicorn部署：如背景介紹
結論：上述問題與gunicorn兼容性問題：gunicorn（gevent） <=> trio(epoll) ......> httpcore ......> httpx ......>dnspython

資源搜索分析：
- Gunicorn（Gevent）與 trio 存在兼容性問題：https://github.com/gevent/gevent/issues/2008
  - 無直接解決方案
- 通過配置
```
from gevent import monkey
monkey.patch_all(select=False)
```
　　 or
```
export EVENTLET_NO_GREENDNS=yes
```
- - 對結果無效

版本兼容性處理-降級

當前版本

gevent                    23.9.1
gevent-websocket          0.10.1
gunicorn                  21.2.0
dnspython                 2.4.2
httpcore                  0.17.3
httpx                     0.24.1
trio                      0.24.0
trio-websocket            0.11.1

經過相關包降級：

- - dsnpython => 2.3.0 無用
  - trio => 0.22.2 升降級無用
  - httpcore => 0.17.1 解決

結論：

AttributeError: module 'select' has no attribute 'epoll' 異常原因：

gunicorn部署表現：trio 與 gunicorn（gevent）的不兼容
解決辦法：使用httpcore低版本 <= 0.17.12

posted @ 2024-01-15 14:48 binger0712 閱讀(592) 評論(0) 收藏舉報

刷新頁面返回頂部

binger0712

關于gunicorn與異步兼容性問題：AttributeError: module 'select' has no attribute 'epoll'

背景：

項目介紹

問題分析

結論：

公告