{"action":"created","issue":{"url":"https://api.github.com/repos/memex-explorer/memex-explorer/issues/525","labels_url":"https://api.github.com/repos/memex-explorer/memex-explorer/issues/525/labels{/name}","comments_url":"https://api.github.com/repos/memex-explorer/memex-explorer/issues/525/comments","events_url":"https://api.github.com/repos/memex-explorer/memex-explorer/issues/525/events","html_url":"https://github.com/memex-explorer/memex-explorer/issues/525","id":79633802,"number":525,"title":"Vagrantfile/Salt - celery workers not quite working right","user":{"login":"ahmadia","id":512293,"avatar_url":"https://avatars.githubusercontent.com/u/512293?v=3","gravatar_id":"","url":"https://api.github.com/users/ahmadia","html_url":"https://github.com/ahmadia","followers_url":"https://api.github.com/users/ahmadia/followers","following_url":"https://api.github.com/users/ahmadia/following{/other_user}","gists_url":"https://api.github.com/users/ahmadia/gists{/gist_id}","starred_url":"https://api.github.com/users/ahmadia/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ahmadia/subscriptions","organizations_url":"https://api.github.com/users/ahmadia/orgs","repos_url":"https://api.github.com/users/ahmadia/repos","events_url":"https://api.github.com/users/ahmadia/events{/privacy}","received_events_url":"https://api.github.com/users/ahmadia/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2015-05-22T22:23:22Z","updated_at":"2015-05-24T19:35:15Z","closed_at":null,"body":"debug output\r\n```\r\nvagrant@vagrant-ubuntu-trusty-64:/vagrant/source$ /home/vagrant/miniconda/envs/memex/bin/python -m celery worker -A memex --pidfile=celeryd.pid -l debug\r\n[2015-05-22 22:21:59,203: DEBUG/MainProcess] | Worker: Preparing bootsteps.\r\n[2015-05-22 22:21:59,208: DEBUG/MainProcess] | Worker: Building graph...\r\n[2015-05-22 22:21:59,209: DEBUG/MainProcess] | Worker: New boot order: {Timer, Hub, Queues (intra), Pool, Autoscaler, StateDB, Beat, Autoreloader, Consumer}\r\n[2015-05-22 22:21:59,214: DEBUG/MainProcess] | Consumer: Preparing bootsteps.\r\n[2015-05-22 22:21:59,215: DEBUG/MainProcess] | Consumer: Building graph...\r\n[2015-05-22 22:21:59,223: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Events, Mingle, Tasks, Control, Agent, Gossip, Heart, event loop}\r\n\r\n -------------- celery@vagrant-ubuntu-trusty-64 v3.1.18 (Cipater)\r\n---- **** -----\r\n--- * ***  * -- Linux-3.13.0-51-generic-x86_64-with-debian-jessie-sid\r\n-- * - **** ---\r\n- ** ---------- [config]\r\n- ** ---------- .> app:         memex:0x7f3c50384910\r\n- ** ---------- .> transport:   redis://localhost:6379//\r\n- ** ---------- .> results:     redis://localhost\r\n- *** --- * --- .> concurrency: 1 (prefork)\r\n-- ******* ----\r\n--- ***** ----- [queues]\r\n -------------- .> celery           exchange=celery(direct) key=celery\r\n\r\n\r\n[tasks]\r\n  . celery.backend_cleanup\r\n  . celery.chain\r\n  . celery.chord\r\n  . celery.chord_unlock\r\n  . celery.chunks\r\n  . celery.group\r\n  . celery.map\r\n  . celery.starmap\r\n  . memex.celery.debug_task\r\n  . task_manager.crawl_tasks.ache\r\n  . task_manager.crawl_tasks.nutch\r\n  . task_manager.docker_tasks.start_containers\r\n  . task_manager.file_tasks.unzip\r\n  . task_manager.tika_tasks.create_index\r\n\r\n[2015-05-22 22:21:59,233: DEBUG/MainProcess] | Worker: Starting Hub\r\n[2015-05-22 22:21:59,234: DEBUG/MainProcess] ^-- substep ok\r\n[2015-05-22 22:21:59,234: DEBUG/MainProcess] | Worker: Starting Pool\r\n[2015-05-22 22:21:59,241: DEBUG/MainProcess] ^-- substep ok\r\n[2015-05-22 22:21:59,242: DEBUG/MainProcess] | Worker: Starting Consumer\r\n[2015-05-22 22:21:59,242: DEBUG/MainProcess] | Consumer: Starting Connection\r\n[2015-05-22 22:21:59,254: INFO/MainProcess] Connected to redis://localhost:6379//\r\n[2015-05-22 22:21:59,255: DEBUG/MainProcess] ^-- substep ok\r\n[2015-05-22 22:21:59,255: DEBUG/MainProcess] | Consumer: Starting Events\r\n[2015-05-22 22:21:59,262: DEBUG/MainProcess] ^-- substep ok\r\n[2015-05-22 22:21:59,262: DEBUG/MainProcess] | Consumer: Starting Mingle\r\n[2015-05-22 22:21:59,263: INFO/MainProcess] mingle: searching for neighbors\r\n[2015-05-22 22:22:00,267: INFO/MainProcess] mingle: all alone\r\n[2015-05-22 22:22:00,267: DEBUG/MainProcess] ^-- substep ok\r\n[2015-05-22 22:22:00,267: DEBUG/MainProcess] | Consumer: Starting Tasks\r\n[2015-05-22 22:22:00,271: DEBUG/MainProcess] ^-- substep ok\r\n[2015-05-22 22:22:00,271: DEBUG/MainProcess] | Consumer: Starting Control\r\n[2015-05-22 22:22:00,273: DEBUG/MainProcess] ^-- substep ok\r\n[2015-05-22 22:22:00,273: DEBUG/MainProcess] | Consumer: Starting Gossip\r\n[2015-05-22 22:22:00,275: DEBUG/MainProcess] ^-- substep ok\r\n[2015-05-22 22:22:00,275: DEBUG/MainProcess] | Consumer: Starting Heart\r\n[2015-05-22 22:22:00,276: DEBUG/MainProcess] ^-- substep ok\r\n[2015-05-22 22:22:00,276: DEBUG/MainProcess] | Consumer: Starting event loop\r\n/home/vagrant/miniconda/envs/memex/lib/python2.7/site-packages/celery/fixups/django.py:265: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!\r\n  warnings.warn('Using settings.DEBUG leads to a memory leak, never '\r\n\r\n[2015-05-22 22:22:00,277: WARNING/MainProcess] /home/vagrant/miniconda/envs/memex/lib/python2.7/site-packages/celery/fixups/django.py:265: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!\r\n  warnings.warn('Using settings.DEBUG leads to a memory leak, never '\r\n\r\n[2015-05-22 22:22:00,279: WARNING/MainProcess] celery@vagrant-ubuntu-trusty-64 ready.\r\n[2015-05-22 22:22:00,279: DEBUG/MainProcess] | Worker: Hub.register Pool...\r\n[2015-05-22 22:22:00,280: DEBUG/MainProcess] basic.qos: prefetch_count->4\r\n[2015-05-22 22:22:11,843: INFO/MainProcess] Received task: task_manager.crawl_tasks.nutch[6e7088cb-e527-44dd-a083-5bc14c5b011d]\r\n[2015-05-22 22:22:11,844: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f3c50059500> (args:('task_manager.crawl_tasks.nutch', '6e7088cb-e527-44dd-a083-5bc14c5b011d', (<Crawl: mm>,), {}, {'utc': True, u'is_eager': False, 'chord': None, u'group': None, 'args': (<Crawl: mm>,), 'retries': 0, u'delivery_info': {u'priority': 0, u'redelivered': None, u'routing_key': u'celery', u'exchange': u'celery'}, 'expires': None, u'hostname': 'celery@vagrant-ubuntu-trusty-64', 'task': 'task_manager.crawl_tasks.nutch', 'callbacks': None, u'correlation_id': u'6e7088cb-e527-44dd-a083-5bc14c5b011d', 'errbacks': None, 'timelimit': (None, None), 'taskset': None, 'kwargs': {}, 'eta': None, u'reply_to': u'db2d8957-7b72-3509-afda-2cf0f144e1cd', 'id': '6e7088cb-e527-44dd-a083-5bc14c5b011d', u'headers': {}}) kwargs:{})\r\n[2015-05-22 22:22:11,848: DEBUG/MainProcess] Task accepted: task_manager.crawl_tasks.nutch[6e7088cb-e527-44dd-a083-5bc14c5b011d] pid:17755\r\n[2015-05-22 22:22:11,860: DEBUG/Worker-1] (0.000) QUERY = u'BEGIN' - PARAMS = (); args=None\r\n[2015-05-22 22:22:11,876: DEBUG/Worker-1] (0.014) QUERY = u'INSERT INTO \"task_manager_crawltask\" (\"pid\", \"crawl_id\", \"uuid\") VALUES (%s, %s, %s)' - PARAMS = (17760, 3, u'6e7088cb-e527-44dd-a083-5bc14c5b011d'); args=[17760, 3, '6e7088cb-e527-44dd-a083-5bc14c5b011d']\r\n[2015-05-22 22:22:11,894: DEBUG/Worker-1] (0.001) QUERY = u'SELECT \"crawl_space_crawl\".\"id\", \"crawl_space_crawl\".\"name\", \"crawl_space_crawl\".\"slug\", \"crawl_space_crawl\".\"description\", \"crawl_space_crawl\".\"crawler\", \"crawl_space_crawl\".\"status\", \"crawl_space_crawl\".\"config\", \"crawl_space_crawl\".\"seeds_list\", \"crawl_space_crawl\".\"pages_crawled\", \"crawl_space_crawl\".\"harvest_rate\", \"crawl_space_crawl\".\"project_id\", \"crawl_space_crawl\".\"crawl_model_id\", \"crawl_space_crawl\".\"location\", \"crawl_space_crawl\".\"rounds_left\" FROM \"crawl_space_crawl\" WHERE \"crawl_space_crawl\".\"id\" = %s LIMIT 21' - PARAMS = (3,); args=(3,)\r\n[2015-05-22 22:22:11,895: DEBUG/Worker-1] (0.000) QUERY = u'BEGIN' - PARAMS = (); args=None\r\n[2015-05-22 22:22:11,900: DEBUG/Worker-1] (0.004) QUERY = u'UPDATE \"crawl_space_crawl\" SET \"name\" = %s, \"slug\" = %s, \"description\" = %s, \"crawler\" = %s, \"status\" = %s, \"config\" = %s, \"seeds_list\" = %s, \"pages_crawled\" = %s, \"harvest_rate\" = %s, \"project_id\" = %s, \"crawl_model_id\" = NULL, \"location\" = %s, \"rounds_left\" = %s WHERE \"crawl_space_crawl\".\"id\" = %s' - PARAMS = (u'mm', u'mm', u'nny', u'nutch', u'STARTING', u'config_default', u'/vagrant/source/resources/crawls/mm/seeds', 0, 0.0, 1, u'/vagrant/source/resources/crawls/mm', 0, 3); args=(u'mm', u'mm', u'nny', u'nutch', u'STARTING', u'config_default', u'/vagrant/source/resources/crawls/mm/seeds', 0, 0.0, 1, u'/vagrant/source/resources/crawls/mm', 0, 3)\r\n[2015-05-22 22:22:11,904: INFO/MainProcess] Task task_manager.crawl_tasks.nutch[6e7088cb-e527-44dd-a083-5bc14c5b011d] succeeded in 0.0585073550028s: 'Round Complete'\r\n```"},"comment":{"url":"https://api.github.com/repos/memex-explorer/memex-explorer/issues/comments/105055904","html_url":"https://github.com/memex-explorer/memex-explorer/issues/525#issuecomment-105055904","issue_url":"https://api.github.com/repos/memex-explorer/memex-explorer/issues/525","id":105055904,"user":{"login":"ahmadia","id":512293,"avatar_url":"https://avatars.githubusercontent.com/u/512293?v=3","gravatar_id":"","url":"https://api.github.com/users/ahmadia","html_url":"https://github.com/ahmadia","followers_url":"https://api.github.com/users/ahmadia/followers","following_url":"https://api.github.com/users/ahmadia/following{/other_user}","gists_url":"https://api.github.com/users/ahmadia/gists{/gist_id}","starred_url":"https://api.github.com/users/ahmadia/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ahmadia/subscriptions","organizations_url":"https://api.github.com/users/ahmadia/orgs","repos_url":"https://api.github.com/users/ahmadia/repos","events_url":"https://api.github.com/users/ahmadia/events{/privacy}","received_events_url":"https://api.github.com/users/ahmadia/received_events","type":"User","site_admin":false},"created_at":"2015-05-24T19:35:15Z","updated_at":"2015-05-24T19:35:15Z","body":"Thanks @lewismc - I'm not sure how the command is getting screwed up, but I'm digging into it."}}