{"action":"created","issue":{"url":"https://api.github.com/repos/benoitc/gunicorn/issues/1056","labels_url":"https://api.github.com/repos/benoitc/gunicorn/issues/1056/labels{/name}","comments_url":"https://api.github.com/repos/benoitc/gunicorn/issues/1056/comments","events_url":"https://api.github.com/repos/benoitc/gunicorn/issues/1056/events","html_url":"https://github.com/benoitc/gunicorn/issues/1056","id":91003825,"number":1056,"title":"monkey.patch_all and gunicorn with more than 1 worker","user":{"login":"emamirazavi","id":3804866,"avatar_url":"https://avatars.githubusercontent.com/u/3804866?v=3","gravatar_id":"","url":"https://api.github.com/users/emamirazavi","html_url":"https://github.com/emamirazavi","followers_url":"https://api.github.com/users/emamirazavi/followers","following_url":"https://api.github.com/users/emamirazavi/following{/other_user}","gists_url":"https://api.github.com/users/emamirazavi/gists{/gist_id}","starred_url":"https://api.github.com/users/emamirazavi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/emamirazavi/subscriptions","organizations_url":"https://api.github.com/users/emamirazavi/orgs","repos_url":"https://api.github.com/users/emamirazavi/repos","events_url":"https://api.github.com/users/emamirazavi/events{/privacy}","received_events_url":"https://api.github.com/users/emamirazavi/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-06-25T15:57:47Z","updated_at":"2015-06-25T23:20:26Z","closed_at":null,"body":"I'm using Flask+Gunicorn+Gevent\r\nI've called monkey.patch_all at the first of app passed to gunicorn command line because of using some co-routine tasks in my app. \r\nMy worker class is gunicorn.workers.ggevent.GeventWorker.\r\nWhen i configure gunicorn to use more than one worker (multiprocessing.cpu_count() * 2 + 1 that in my laptop is equal to 9) everything is mixed up. My logger (python logging) logs 10 times! and some much more others conflicts in my code occurred.\r\nMY GUNICORN CONFIGURAION\r\n```\r\n# to import server_hook\r\nimport os, sys\r\nthis_path = os.path.dirname(os.path.realpath(__file__))\r\nsys.path.insert(0, os.path.realpath(this_path))\r\nimport server_hook\r\nimport multiprocessing\r\nuser = \"myapp\"\r\nmax_requests = 60\r\nmax_requests_jitter = 5\r\nreload = True\r\ngroup = \"daemon\"\r\nhost = \"0.0.0.0\"\r\nbind = ['127.0.0.1:8000', 'unix:/run/gunicorn/socket']\r\nworkers = multiprocessing.cpu_count() * 2 + 1\r\nthreads = 10\r\nworker_class = \"gunicorn.workers.ggevent.GeventWorker\"\r\nproc_name = \"myapp\"\r\ndefault_proc_name = \"myapp\"\r\nenvironment = LANG = \"en_US.UTF-8\"\r\npreload_app = True\r\nloglevel = \"DEBUG\"\r\non_starting = server_hook.on_starting\r\n```\r\n\r\nMY SERVER_HOOK FILE (server_hook.on_starting)\r\n```\r\nimport threading\r\nfrom myapp.logger import logger\r\n\r\ndef on_starting(server):\r\n    thread = threading.Thread(target=test)\r\n    thread.daemon = True\r\n    thread.start()\r\n\r\ndef test():\r\n    while 1:\r\n        logger.debug('i am test!')\r\n        sleep(10)\r\n```\r\n\r\nMY LOG FILE\r\n```\r\n2015-06-25 20:32:33,579 INFO Starting webserver...\r\n2015-06-25 20:32:33,595 DEBUG i am test!\r\n\r\n2015-06-25 20:32:43,595 DEBUG i am test!\r\n2015-06-25 20:32:43,596 DEBUG i am test!\r\n2015-06-25 20:32:43,596 DEBUG i am test!\r\n2015-06-25 20:32:43,596 DEBUG i am test!\r\n2015-06-25 20:32:43,595 DEBUG i am test!\r\n2015-06-25 20:32:43,596 DEBUG i am test!\r\n2015-06-25 20:32:43,597 DEBUG i am test!\r\n2015-06-25 20:32:43,597 DEBUG i am test!\r\n2015-06-25 20:32:43,597 DEBUG i am test!\r\n2015-06-25 20:32:43,595 DEBUG i am test!\r\n\r\n2015-06-25 20:32:53,596 DEBUG i am test!\r\n2015-06-25 20:32:53,597 DEBUG i am test!\r\n2015-06-25 20:32:53,597 DEBUG i am test!\r\n2015-06-25 20:32:53,597 DEBUG i am test!\r\n2015-06-25 20:32:53,597 DEBUG i am test!\r\n2015-06-25 20:32:53,597 DEBUG i am test!\r\n2015-06-25 20:32:53,598 DEBUG i am test!\r\n2015-06-25 20:32:53,598 DEBUG i am test!\r\n2015-06-25 20:32:53,597 DEBUG i am test!\r\n2015-06-25 20:32:53,599 DEBUG i am test!\r\n\r\n```\r\nFor now the solution is to call patch_all with thread=False or set gunicorn configuration to use just one worker. In patch_all with thread=False solution\r\n```\r\n  File \"/home/m.emami/projects/python/whatsapp/src/lib/myapp/cron.py\", line 42, in cron_main\r\n    rss_robot_cron()\r\n  File \"/home/m.emami/projects/python/whatsapp/src/lib/myapp/rss_robot.py\", line 155, in cron\r\n    feed_all()\r\n  File \"/home/m.emami/projects/python/whatsapp/src/lib/myapp/rss_robot.py\", line 151, in feed_all\r\n    feed_pool.join()\r\n  File \"/usr/lib64/python2.7/site-packages/gevent/pool.py\", line 100, in join\r\n    self._empty_event.wait(timeout=timeout)\r\n  File \"/usr/lib64/python2.7/site-packages/gevent/event.py\", line 77, in wait\r\n    result = self.hub.switch()\r\n  File \"/usr/lib64/python2.7/site-packages/gevent/hub.py\", line 331, in switch\r\n    return greenlet.switch(self)\r\nerror: cannot switch to a different thread\r\n```\r\nI can not use any thread being gevent safe. It means just one solution exists and it's use gunicorn with one worker!!!\r\nAny idea?!"},"comment":{"url":"https://api.github.com/repos/benoitc/gunicorn/issues/comments/115428187","html_url":"https://github.com/benoitc/gunicorn/issues/1056#issuecomment-115428187","issue_url":"https://api.github.com/repos/benoitc/gunicorn/issues/1056","id":115428187,"user":{"login":"jamadden","id":1256082,"avatar_url":"https://avatars.githubusercontent.com/u/1256082?v=3","gravatar_id":"","url":"https://api.github.com/users/jamadden","html_url":"https://github.com/jamadden","followers_url":"https://api.github.com/users/jamadden/followers","following_url":"https://api.github.com/users/jamadden/following{/other_user}","gists_url":"https://api.github.com/users/jamadden/gists{/gist_id}","starred_url":"https://api.github.com/users/jamadden/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jamadden/subscriptions","organizations_url":"https://api.github.com/users/jamadden/orgs","repos_url":"https://api.github.com/users/jamadden/repos","events_url":"https://api.github.com/users/jamadden/events{/privacy}","received_events_url":"https://api.github.com/users/jamadden/received_events","type":"User","site_admin":false},"created_at":"2015-06-25T23:20:26Z","updated_at":"2015-06-25T23:20:26Z","body":"For various reasons we also need to make sure that gevent has monkey patched everything sooner than gunicorn wants to do it. Our (preloaded) app uses `pyramid`, which loads things with a script called `pserve`. We wrap this script in our own script, monkey patch there, and then hand things over to pserve, which hands them over to gunicorn. You could probably use a similar technique of wrapping whatever script you normally use.\r\n\r\n```python\r\n#!/usr/bin/env python\r\n# -*- coding: utf-8 -*-\r\n\"\"\"\r\nA script to help us ensure that we are loading and monkey-patching\r\nthe entire system before Pyramid loads: loading Pyramid's ``pserve``\r\nloads many Pyramid modules, including :mod:`pyramid.traversal`, which\r\nin turn loads :mod:`repoze.lru` and allocates real, non-recursive\r\nthread locks. These are not compatible with gevent and eventually\r\nlead to a hang if we re-enter a greenlet that wants to acquire one\r\nof these locks while a previous greenlet already has it.\r\n\"\"\"\r\n\r\nfrom __future__ import print_function, unicode_literals, absolute_import, division\r\n__docformat__ = \"restructuredtext en\"\r\n\r\n# Note that we must not import *anything* before the patch\r\nfrom nti.monkey import gevent_patch_on_import\r\ngevent_patch_on_import.patch() # basically does gevent.monkey.patch_all()\r\n\r\nimport sys\r\nfrom pkg_resources import load_entry_point, get_distribution\r\n\r\ndef main():\r\n\tsys.exit(\r\n\t\tload_entry_point('pyramid', 'console_scripts', 'pserve')()\r\n\t)\r\n```"}}