{"action":"created","issue":{"url":"https://api.github.com/repos/rcbops/rpc-openstack/issues/339","labels_url":"https://api.github.com/repos/rcbops/rpc-openstack/issues/339/labels{/name}","comments_url":"https://api.github.com/repos/rcbops/rpc-openstack/issues/339/comments","events_url":"https://api.github.com/repos/rcbops/rpc-openstack/issues/339/events","html_url":"https://github.com/rcbops/rpc-openstack/issues/339","id":100824742,"number":339,"title":"Kilo upgrades from Juno fail with obscure keystone errors","user":{"login":"miguelgrinberg","id":2715854,"avatar_url":"https://avatars.githubusercontent.com/u/2715854?v=3","gravatar_id":"","url":"https://api.github.com/users/miguelgrinberg","html_url":"https://github.com/miguelgrinberg","followers_url":"https://api.github.com/users/miguelgrinberg/followers","following_url":"https://api.github.com/users/miguelgrinberg/following{/other_user}","gists_url":"https://api.github.com/users/miguelgrinberg/gists{/gist_id}","starred_url":"https://api.github.com/users/miguelgrinberg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/miguelgrinberg/subscriptions","organizations_url":"https://api.github.com/users/miguelgrinberg/orgs","repos_url":"https://api.github.com/users/miguelgrinberg/repos","events_url":"https://api.github.com/users/miguelgrinberg/events{/privacy}","received_events_url":"https://api.github.com/users/miguelgrinberg/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/rcbops/rpc-openstack/labels/prio-undecided","name":"prio-undecided","color":"207de5"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/rcbops/rpc-openstack/milestones/19","html_url":"https://github.com/rcbops/rpc-openstack/milestones/r11.0.0rc14","labels_url":"https://api.github.com/repos/rcbops/rpc-openstack/milestones/19/labels","id":1246028,"number":19,"title":"r11.0.0rc14","description":"","creator":{"login":"nrb","id":321387,"avatar_url":"https://avatars.githubusercontent.com/u/321387?v=3","gravatar_id":"","url":"https://api.github.com/users/nrb","html_url":"https://github.com/nrb","followers_url":"https://api.github.com/users/nrb/followers","following_url":"https://api.github.com/users/nrb/following{/other_user}","gists_url":"https://api.github.com/users/nrb/gists{/gist_id}","starred_url":"https://api.github.com/users/nrb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nrb/subscriptions","organizations_url":"https://api.github.com/users/nrb/orgs","repos_url":"https://api.github.com/users/nrb/repos","events_url":"https://api.github.com/users/nrb/events{/privacy}","received_events_url":"https://api.github.com/users/nrb/received_events","type":"User","site_admin":false},"open_issues":7,"closed_issues":2,"state":"open","created_at":"2015-08-07T20:12:05Z","updated_at":"2015-08-13T19:41:14Z","due_on":null,"closed_at":null},"comments":2,"created_at":"2015-08-13T17:28:51Z","updated_at":"2015-08-13T22:30:38Z","closed_at":null,"body":"When the upgrade got into the end of the keystone playbook, some of the ensure admin/tenant/etc. tasks started to fail. Repeating the play would fail again, but not always in the same task. Sending command with the openstack cli was also failing on most calls, but some calls went through.\r\n\r\nThe keystone error log showed lots of crashes with two stack traces, shown below. Applying the patch in https://review.openstack.org/#/c/212021 resolved the problem for me.\r\n\r\nStack trace 1:\r\n\r\n    mod_wsgi (pid=21383): Exception occurred processing WSGI script '/var/www/cgi-bin/keystone/main'.\r\n    Traceback (most recent call last):\r\n      File \"/var/www/cgi-bin/keystone/main\", line 25, in <module>\r\n        application = wsgi_server.initialize_application(name)\r\n      File \"/usr/local/lib/python2.7/dist-packages/keystone/server/wsgi.py\", line 38, in initialize_application\r\n        common.configure()\r\n      File \"/usr/local/lib/python2.7/dist-packages/keystone/server/common.py\", line 28, in configure\r\n        config.configure()\r\n      File \"/usr/local/lib/python2.7/dist-packages/keystone/common/config.py\", line 1112, in configure\r\n        help='Do not monkey-patch threading system modules.'))\r\n      File \"/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py\", line 1794, in __inner\r\n        result = f(self, *args, **kwargs)\r\n      File \"/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py\", line 1967, in register_cli_opt\r\n        raise ArgsAlreadyParsedError(\"cannot register CLI option\")\r\n    ArgsAlreadyParsedError: arguments already parsed: cannot register CLI option\r\n\r\nStack trace 2:\r\n\r\n    Traceback (most recent call last):\r\n      File \"/var/www/cgi-bin/keystone/admin\", line 25, in <module>\r\n        application = wsgi_server.initialize_application(name)\r\n      File \"/usr/local/lib/python2.7/dist-packages/keystone/server/wsgi.py\", line 51, in initialize_application\r\n        startup_application_fn=loadapp)\r\n      File \"/usr/local/lib/python2.7/dist-packages/keystone/server/common.py\", line 43, in setup_backends\r\n        res = startup_application_fn()\r\n      File \"/usr/local/lib/python2.7/dist-packages/keystone/server/wsgi.py\", line 48, in loadapp\r\n        'config:%s' % config.find_paste_config(), name)\r\n      File \"/usr/local/lib/python2.7/dist-packages/keystone/service.py\", line 45, in loadapp\r\n        controllers.latest_app = deploy.loadapp(conf, name=name)\r\n      File \"/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py\", line 247, in loadapp\r\n        return loadobj(APP, uri, name=name, **kw)\r\n      File \"/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py\", line 272, in loadobj\r\n        return context.create()\r\n      File \"/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py\", line 710, in create\r\n        return self.object_type.invoke(self)\r\n      File \"/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py\", line 144, in invoke\r\n        **context.local_conf)\r\n      File \"/usr/local/lib/python2.7/dist-packages/paste/deploy/util.py\", line 55, in fix_call\r\n        val = callable(*args, **kw)\r\n      File \"/usr/local/lib/python2.7/dist-packages/paste/urlmap.py\", line 31, in urlmap_factory\r\n        app = loader.get_app(app_name, global_conf=global_conf)\r\n      File \"/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py\", line 350, in get_app\r\n        name=name, global_conf=global_conf).create()\r\n      File \"/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py\", line 362, in app_context\r\n        APP, name=name, global_conf=global_conf)\r\n      File \"/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py\", line 450, in get_context\r\n        global_additions=global_additions)\r\n      File \"/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py\", line 562, in _pipeline_app_context\r\n        for name in pipeline[:-1]]\r\n      File \"/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py\", line 458, in get_context\r\n        section)\r\n      File \"/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py\", line 517, in _context_from_explici\r\n    t\r\n        value = import_string(found_expr)\r\n      File \"/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py\", line 22, in import_string\r\n        return pkg_resources.EntryPoint.parse(\"x=\" + s).load(False)\r\n      File \"/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py\", line 2355, in load\r\n        return self.resolve()\r\n      File \"/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py\", line 2361, in resolve\r\n        module = __import__(self.module_name, fromlist=['__name__'], level=0)\r\n      File \"/usr/local/lib/python2.7/dist-packages/keystone/contrib/federation/routers.py\", line 17, in <module>\r\n        from keystone.contrib.federation import controllers\r\n      File \"/usr/local/lib/python2.7/dist-packages/keystone/contrib/federation/controllers.py\", line 29, in <mod\r\n    ule>\r\n        from keystone.contrib.federation import idp as keystone_idp\r\n      File \"/usr/local/lib/python2.7/dist-packages/keystone/contrib/federation/idp.py\", line 23, in <module>\r\n        from saml2 import client_base\r\n      File \"/usr/local/lib/python2.7/dist-packages/saml2/client_base.py\", line 13, in <module>\r\n        from saml2.entity import Entity\r\n      File \"/usr/local/lib/python2.7/dist-packages/saml2/entity.py\", line 9, in <module>\r\n        from saml2.metadata import ENDPOINTS\r\n      File \"/usr/local/lib/python2.7/dist-packages/saml2/metadata.py\", line 3, in <module>\r\n        from saml2.sigver import security_context\r\n      File \"/usr/local/lib/python2.7/dist-packages/saml2/sigver.py\", line 8, in <module>\r\n        from OpenSSL import crypto\r\n      File \"/usr/local/lib/python2.7/dist-packages/OpenSSL/__init__.py\", line 8, in <module>\r\n        from OpenSSL import rand, crypto, SSL\r\n      File \"/usr/local/lib/python2.7/dist-packages/OpenSSL/rand.py\", line 11, in <module>\r\n        from OpenSSL._util import (\r\n      File \"/usr/local/lib/python2.7/dist-packages/OpenSSL/_util.py\", line 7, in <module>\r\n        binding = Binding()\r\n      File \"/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/bindings/openssl/binding.py\", line 47, in\r\n     __init__\r\n        self._ensure_ffi_initialized()\r\n      File \"/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/bindings/openssl/binding.py\", line 78, in\r\n     _ensure_ffi_initialized\r\n        cls._register_osrandom_engine()\r\n      File \"/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/bindings/openssl/binding.py\", line 54, in\r\n     _register_osrandom_engine\r\n        raise RuntimeError(\"osrandom engine already registered\")"},"comment":{"url":"https://api.github.com/repos/rcbops/rpc-openstack/issues/comments/130866547","html_url":"https://github.com/rcbops/rpc-openstack/issues/339#issuecomment-130866547","issue_url":"https://api.github.com/repos/rcbops/rpc-openstack/issues/339","id":130866547,"user":{"login":"byronmccollum","id":954353,"avatar_url":"https://avatars.githubusercontent.com/u/954353?v=3","gravatar_id":"","url":"https://api.github.com/users/byronmccollum","html_url":"https://github.com/byronmccollum","followers_url":"https://api.github.com/users/byronmccollum/followers","following_url":"https://api.github.com/users/byronmccollum/following{/other_user}","gists_url":"https://api.github.com/users/byronmccollum/gists{/gist_id}","starred_url":"https://api.github.com/users/byronmccollum/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/byronmccollum/subscriptions","organizations_url":"https://api.github.com/users/byronmccollum/orgs","repos_url":"https://api.github.com/users/byronmccollum/repos","events_url":"https://api.github.com/users/byronmccollum/events{/privacy}","received_events_url":"https://api.github.com/users/byronmccollum/received_events","type":"User","site_admin":false},"created_at":"2015-08-13T22:30:38Z","updated_at":"2015-08-13T22:30:38Z","body":"I'm seeing this as well on new installs."}}