{"action":"created","issue":{"url":"https://api.github.com/repos/uchicago-bio/2015-Autumn-Forum/issues/46","labels_url":"https://api.github.com/repos/uchicago-bio/2015-Autumn-Forum/issues/46/labels{/name}","comments_url":"https://api.github.com/repos/uchicago-bio/2015-Autumn-Forum/issues/46/comments","events_url":"https://api.github.com/repos/uchicago-bio/2015-Autumn-Forum/issues/46/events","html_url":"https://github.com/uchicago-bio/2015-Autumn-Forum/issues/46","id":113845440,"number":46,"title":"Understanding the Python GIL","user":{"login":"bdallen-uchicago","id":14986570,"avatar_url":"https://avatars.githubusercontent.com/u/14986570?v=3","gravatar_id":"","url":"https://api.github.com/users/bdallen-uchicago","html_url":"https://github.com/bdallen-uchicago","followers_url":"https://api.github.com/users/bdallen-uchicago/followers","following_url":"https://api.github.com/users/bdallen-uchicago/following{/other_user}","gists_url":"https://api.github.com/users/bdallen-uchicago/gists{/gist_id}","starred_url":"https://api.github.com/users/bdallen-uchicago/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bdallen-uchicago/subscriptions","organizations_url":"https://api.github.com/users/bdallen-uchicago/orgs","repos_url":"https://api.github.com/users/bdallen-uchicago/repos","events_url":"https://api.github.com/users/bdallen-uchicago/events{/privacy}","received_events_url":"https://api.github.com/users/bdallen-uchicago/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-10-28T15:18:30Z","updated_at":"2015-10-28T15:38:02Z","closed_at":null,"body":"I've been trying to wrap my head around what the limitation really are, and found these great presentations/articles by a Chicago developer who frequents ChiPy:\r\nhttp://dabeaz.com/GIL/\r\n\r\nTLDR the GIL behaves even worse than anyone expected on multi-core systems. Python 3.2 has a new implementation of the GIL that is supposed to fix a lot of these issues:\r\nhttps://docs.python.org/3/whatsnew/3.2.html#multi-threading\r\n\r\nFrom here:\r\nhttps://wiki.python.org/moin/GlobalInterpreterLock\r\nit sounds like some of the C python modules like NumPy operate outside the GIL, but all the number crunching you do in pure Python is going to involve the Python interpreter executing byte codes, which requires the GIL, and can't be done concurrently, even on a multicore system. You basically have native threads running on the other core just sitting there waiting for the GIL.\r\n\r\nTLDR is to use multiprocessing instead of multithreading to do concurrent computation in python, with the caveat that if most of the expensive computation loops live entirely inside C modules, multithreading may work fine. For this to work the C module must release the GIL somehow, and I'm guessing it needs to be explicit - would be interesting to see what that looks like in the NumPy code."},"comment":{"url":"https://api.github.com/repos/uchicago-bio/2015-Autumn-Forum/issues/comments/151885547","html_url":"https://github.com/uchicago-bio/2015-Autumn-Forum/issues/46#issuecomment-151885547","issue_url":"https://api.github.com/repos/uchicago-bio/2015-Autumn-Forum/issues/46","id":151885547,"user":{"login":"bdallen-uchicago","id":14986570,"avatar_url":"https://avatars.githubusercontent.com/u/14986570?v=3","gravatar_id":"","url":"https://api.github.com/users/bdallen-uchicago","html_url":"https://github.com/bdallen-uchicago","followers_url":"https://api.github.com/users/bdallen-uchicago/followers","following_url":"https://api.github.com/users/bdallen-uchicago/following{/other_user}","gists_url":"https://api.github.com/users/bdallen-uchicago/gists{/gist_id}","starred_url":"https://api.github.com/users/bdallen-uchicago/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bdallen-uchicago/subscriptions","organizations_url":"https://api.github.com/users/bdallen-uchicago/orgs","repos_url":"https://api.github.com/users/bdallen-uchicago/repos","events_url":"https://api.github.com/users/bdallen-uchicago/events{/privacy}","received_events_url":"https://api.github.com/users/bdallen-uchicago/received_events","type":"User","site_admin":false},"created_at":"2015-10-28T15:38:02Z","updated_at":"2015-10-28T15:38:02Z","body":"It is an explicit call to release the GIL, and when you release it you can't access any Python objects:\r\nhttps://docs.python.org/2/c-api/init.html#thread-state-and-the-global-interpreter-lock\r\n\r\n\"Note Calling system I/O functions is the most common use case for releasing the GIL, but it can also be useful before calling long-running computations which don’t need access to Python objects, such as compression or cryptographic functions operating over memory buffers. For example, the standard zlib and hashlib modules release the GIL when compressing or hashing data.\""}}