{"action":"created","issue":{"url":"https://api.github.com/repos/freelawproject/juriscraper/issues/74","labels_url":"https://api.github.com/repos/freelawproject/juriscraper/issues/74/labels{/name}","comments_url":"https://api.github.com/repos/freelawproject/juriscraper/issues/74/comments","events_url":"https://api.github.com/repos/freelawproject/juriscraper/issues/74/events","html_url":"https://github.com/freelawproject/juriscraper/issues/74","id":103355265,"number":74,"title":"Move Phantomjs to a single import and single instance","user":{"login":"mlissner","id":236970,"avatar_url":"https://avatars.githubusercontent.com/u/236970?v=3","gravatar_id":"","url":"https://api.github.com/users/mlissner","html_url":"https://github.com/mlissner","followers_url":"https://api.github.com/users/mlissner/followers","following_url":"https://api.github.com/users/mlissner/following{/other_user}","gists_url":"https://api.github.com/users/mlissner/gists{/gist_id}","starred_url":"https://api.github.com/users/mlissner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mlissner/subscriptions","organizations_url":"https://api.github.com/users/mlissner/orgs","repos_url":"https://api.github.com/users/mlissner/repos","events_url":"https://api.github.com/users/mlissner/events{/privacy}","received_events_url":"https://api.github.com/users/mlissner/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/freelawproject/juriscraper/labels/enhancement","name":"enhancement","color":"84b6eb"},{"url":"https://api.github.com/repos/freelawproject/juriscraper/labels/help%20wanted","name":"help wanted","color":"159818"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-08-26T20:16:05Z","updated_at":"2015-08-27T13:24:58Z","closed_at":null,"body":"I'm pretty sure most of the reason that PhantomJS is slow is because we import and instantiate it every single time. A better solution would keep it around after it was started each time. An even better solution would keep it around, but only instantiate it when it was first needed (so calling just one scraper that doesn't use Phantomjs isn't negatively impacted).\r\n\r\nThere's [an answer on StackOverflow that has some tips about this][so]. This also might help with the Webdriver connection errors we seem to have on longer-running processes.\r\n\r\n[so]: http://stackoverflow.com/a/27691489/64911"},"comment":{"url":"https://api.github.com/repos/freelawproject/juriscraper/issues/comments/135426982","html_url":"https://github.com/freelawproject/juriscraper/issues/74#issuecomment-135426982","issue_url":"https://api.github.com/repos/freelawproject/juriscraper/issues/74","id":135426982,"user":{"login":"mlissner","id":236970,"avatar_url":"https://avatars.githubusercontent.com/u/236970?v=3","gravatar_id":"","url":"https://api.github.com/users/mlissner","html_url":"https://github.com/mlissner","followers_url":"https://api.github.com/users/mlissner/followers","following_url":"https://api.github.com/users/mlissner/following{/other_user}","gists_url":"https://api.github.com/users/mlissner/gists{/gist_id}","starred_url":"https://api.github.com/users/mlissner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mlissner/subscriptions","organizations_url":"https://api.github.com/users/mlissner/orgs","repos_url":"https://api.github.com/users/mlissner/repos","events_url":"https://api.github.com/users/mlissner/events{/privacy}","received_events_url":"https://api.github.com/users/mlissner/received_events","type":"User","site_admin":false},"created_at":"2015-08-27T13:24:58Z","updated_at":"2015-08-27T13:24:58Z","body":"Further investigation here has revealed a few things:\r\n\r\n1. If you quit phantomjs and then try to quickly instantiate a new one, you're looking for pain. There's a 30s timer in the code that in theory makes this possible, since the old phantomjs doesn't go away immediately. But if 30s elapse and your old one is still around, the new one can't be created and then crashes your program. Why it's like this, I know not, but it's certainly annoying. It crashes with:\r\n\r\n        Traceback (most recent call last):\r\n          File \"/var/www/court-listener/alert/scrapers/management/commands/cl_scrape_opinions.py\", line 430, in handle\r\n            self.parse_and_scrape_site(mod, full_crawl)\r\n          File \"/var/www/court-listener/alert/scrapers/management/commands/cl_back_scrape_opinions.py\", line 11, in parse_and_scrape_site\r\n            for site in site_yielder(mod.Site().back_scrape_iterable, mod):\r\n          File \"/usr/lib/python2.7/dist-packages/juriscraper/lib/importer.py\", line 51, in site_yielder\r\n            site._download_backwards(i)\r\n          File \"/usr/lib/python2.7/dist-packages/juriscraper/opinions/united_states/state/tex.py\", line 237, in _download_backwards\r\n            self.html = self._download()\r\n          File \"/usr/lib/python2.7/dist-packages/juriscraper/opinions/united_states/state/tex.py\", line 56, in _download\r\n            service_log_path=os.path.devnull,  # Disable ghostdriver.log\r\n          File \"/usr/local/lib/python2.7/dist-packages/selenium/webdriver/phantomjs/webdriver.py\", line 50, in __init__\r\n            self.service.start()\r\n          File \"/usr/local/lib/python2.7/dist-packages/selenium/webdriver/phantomjs/service.py\", line 69, in start\r\n            raise WebDriverException(\"Can not connect to GhostDriver\")\r\n        WebDriverException: Message: 'Can not connect to GhostDriver'\r\n\r\n2. The way we currently import modules means that there's no long-living object that persists between scrapes of one site and the next (or multiple scrapes of the same site, in the case of a backscraper). This means that we need to either load PhantomJS in the caller and pass it to the scraper (which sucks), or change Juriscraper's caller API. Current code works like this:\r\n\r\n        module_strings = build_module_list(court_id)\r\n        package, module = module_strings[i].rsplit('.', 1)\r\n        mod = __import__(\"%s.%s\" % (package, module),\r\n                         globals(),\r\n                         locals(),\r\n                         [module])\r\n        site = mod.Site()\r\n        site.parse()\r\n\r\n    A better approach would probably be to instantiate a new Juriscraper object that holds onto long-running stuff and does the wack-a-doo string conversion for you. Something like:\r\n\r\n        j = Juriscraper()\r\n        j.start()\r\n        s = j.parse('tex')  # or j.parse('united_states.opinions'), or whatever.\r\n\r\n    It's a much nicer API that moves the craziness out of the callers. What's more, it gives us a place where we can keep PhantomJS. "}}