{"action":"created","issue":{"url":"https://api.github.com/repos/fedora-infra/pkgdb2/issues/299","labels_url":"https://api.github.com/repos/fedora-infra/pkgdb2/issues/299/labels{/name}","comments_url":"https://api.github.com/repos/fedora-infra/pkgdb2/issues/299/comments","events_url":"https://api.github.com/repos/fedora-infra/pkgdb2/issues/299/events","html_url":"https://github.com/fedora-infra/pkgdb2/issues/299","id":121797492,"number":299,"title":"Intermittent 404 errors?","user":{"login":"ralphbean","id":331338,"avatar_url":"https://avatars.githubusercontent.com/u/331338?v=3","gravatar_id":"","url":"https://api.github.com/users/ralphbean","html_url":"https://github.com/ralphbean","followers_url":"https://api.github.com/users/ralphbean/followers","following_url":"https://api.github.com/users/ralphbean/following{/other_user}","gists_url":"https://api.github.com/users/ralphbean/gists{/gist_id}","starred_url":"https://api.github.com/users/ralphbean/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ralphbean/subscriptions","organizations_url":"https://api.github.com/users/ralphbean/orgs","repos_url":"https://api.github.com/users/ralphbean/repos","events_url":"https://api.github.com/users/ralphbean/events{/privacy}","received_events_url":"https://api.github.com/users/ralphbean/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-12-11T21:45:22Z","updated_at":"2015-12-12T03:43:05Z","closed_at":null,"body":"When trying to page through the list of all packages, I get some intermittent 404 errors.  I hacked pkgdb2client to get some more error info out and it looks like this:\r\n\r\n```\r\n[2015-12-11 16:38:17][pkgdb2client   ERROR] <!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">                                                                                                  \r\n<title>404 Not Found</title>                                                                                                                                                                         \r\n<h1>Not Found</h1>                                                                                                                                                                                   \r\n<p>The requested URL was not found on the server.  If you entered the URL manually please check your spelling and try again.</p>                                                                     \r\n                                                                                                                                                                                                     \r\nTraceback (most recent call last):                                                                                                                                                                   \r\n  File \"/home/threebean/.virtualenvs/pdc-updater/bin/pdc-updater-initialize\", line 9, in <module>                                                                                                    \r\n    load_entry_point('pdc-updater==0.0.1', 'console_scripts', 'pdc-updater-initialize')()                                                                                                            \r\n  File \"/home/threebean/devel/pdc-updater/pdcupdater/commands.py\", line 56, in initialize                                                                                                            \r\n    handler.initialize(pdc)                                                                                                                                                                          \r\n  File \"/home/threebean/devel/pdc-updater/pdcupdater/handlers/pkgdb.py\", line 56, in initialize                                                                                                      \r\n    packages = pdcupdater.services.pkgdb_packages(self.pkgdb_url)                                                                                                                                    \r\n  File \"/home/threebean/devel/pdc-updater/pdcupdater/services.py\", line 137, in pkgdb_packages                                                                                                       \r\n    pkg = pkgdb.get_packages(page='all', acls=acls, limit=5)                                                                                                                                         \r\n  File \"/usr/lib/python2.7/site-packages/pkgdb2client/__init__.py\", line 797, in get_packages                                                                                                        \r\n    data = _get_pages(i)                                                                                                                                                                             \r\n  File \"/usr/lib/python2.7/site-packages/pkgdb2client/__init__.py\", line 785, in _get_pages                                                                                                          \r\n    return self.handle_api_call('/packages/', params=args)                                                                                                                                           \r\n  File \"/usr/lib/python2.7/site-packages/pkgdb2client/__init__.py\", line 352, in handle_api_call                                                                                                     \r\n    raise PkgDBException('Error while decoding JSON: {0} {1} {2}'.format(err, response, response.url))                                                                                               \r\npkgdb2client.PkgDBException: Error while decoding JSON: Expecting value: line 1 column 1 (char 0) <Response [404]> https://admin.fedoraproject.org/pkgdb/api/packages/?pattern=%2A&limit=5&page=3010 \r\nPYTHONPATH=../pdc-client/. pdc-updater-initialize  14.74s user 1.35s system 1% cpu 16:55.79 total                                                                                                    \r\n```\r\n\r\nAny idea what's going on?  If it hit that url now, I get a good ``200`` response back.  What causes it to blink in and out I wonder..\r\n\r\nPerhaps there is something in that code path that is returning a ``404`` when it should be returning a ``500``?  Perhaps there is a transient database or memcached error being masked.  I've hit this every one of the last 5 times I've try to run ``pdc-updater-initialize``."},"comment":{"url":"https://api.github.com/repos/fedora-infra/pkgdb2/issues/comments/164106158","html_url":"https://github.com/fedora-infra/pkgdb2/issues/299#issuecomment-164106158","issue_url":"https://api.github.com/repos/fedora-infra/pkgdb2/issues/299","id":164106158,"user":{"login":"ralphbean","id":331338,"avatar_url":"https://avatars.githubusercontent.com/u/331338?v=3","gravatar_id":"","url":"https://api.github.com/users/ralphbean","html_url":"https://github.com/ralphbean","followers_url":"https://api.github.com/users/ralphbean/followers","following_url":"https://api.github.com/users/ralphbean/following{/other_user}","gists_url":"https://api.github.com/users/ralphbean/gists{/gist_id}","starred_url":"https://api.github.com/users/ralphbean/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ralphbean/subscriptions","organizations_url":"https://api.github.com/users/ralphbean/orgs","repos_url":"https://api.github.com/users/ralphbean/repos","events_url":"https://api.github.com/users/ralphbean/events{/privacy}","received_events_url":"https://api.github.com/users/ralphbean/received_events","type":"User","site_admin":false},"created_at":"2015-12-12T03:43:05Z","updated_at":"2015-12-12T03:43:05Z","body":"I also hit 500 errors every now and then, but I can't find any tracebacks in ``/var/log/httpd/error_log`` on either of the pkgdb app nodes.\r\n\r\n```\r\n[2015-12-11 20:07:41][pkgdb2client   ERROR] <!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">                                                                                                              \r\n<html><head>                                                                                                                                                                                                \r\n<title>500 Internal Server Error</title>                                                                                                                                                                    \r\n</head><body>                                                                                                                                                                                               \r\n<h1>Internal Server Error</h1>                                                                                                                                                                              \r\n<p>The server encountered an internal error or                                                                                                                                                              \r\nmisconfiguration and was unable to complete                                                                                                                                                                 \r\nyour request.</p>                                                                                                                                                                                           \r\n<p>Please contact the server administrator at                                                                                                                                                               \r\n root@localhost to inform them of the time this error occurred,                                                                                                                                             \r\n and the actions you performed just before this error.</p>                                                                                                                                                  \r\n<p>More information about this error may be available                                                                                                                                                       \r\nin the server error log.</p>                                                                                                                                                                                \r\n</body></html>                                                                                                                                                                                              \r\n                                                                                                                                                                                                            \r\nTraceback (most recent call last):                                                                                                                                                                          \r\n  File \"/home/threebean/.virtualenvs/pdc-updater/bin/pdc-updater-initialize\", line 9, in <module>                                                                                                           \r\n    load_entry_point('pdc-updater==0.0.1', 'console_scripts', 'pdc-updater-initialize')()                                                                                                                   \r\n  File \"/home/threebean/devel/pdc-updater/pdcupdater/commands.py\", line 56, in initialize                                                                                                                   \r\n    handler.initialize(pdc)                                                                                                                                                                                 \r\n  File \"/home/threebean/devel/pdc-updater/pdcupdater/handlers/pkgdb.py\", line 121, in initialize                                                                                                            \r\n    packages = pdcupdater.services.pkgdb_packages(self.pkgdb_url, acls=True)                                                                                                                                \r\n  File \"/home/threebean/devel/pdc-updater/pdcupdater/services.py\", line 137, in pkgdb_packages                                                                                                              \r\n    pkg = pkgdb.get_packages(page='all', acls=acls, limit=10)                                                                                                                                               \r\n  File \"/usr/lib/python2.7/site-packages/pkgdb2client/__init__.py\", line 797, in get_packages                                                                                                               \r\n    data = _get_pages(i)                                                                                                                                                                                    \r\n  File \"/usr/lib/python2.7/site-packages/pkgdb2client/__init__.py\", line 785, in _get_pages                                                                                                                 \r\n    return self.handle_api_call('/packages/', params=args)                                                                                                                                                  \r\n  File \"/usr/lib/python2.7/site-packages/pkgdb2client/__init__.py\", line 352, in handle_api_call                                                                                                            \r\n    raise PkgDBException('Error while decoding JSON: {0} {1} {2}'.format(err, response, response.url))                                                                                                      \r\npkgdb2client.PkgDBException: Error while decoding JSON: Expecting value: line 1 column 1 (char 0) <Response [500]> https://admin.fedoraproject.org/pkgdb/api/packages/?pattern=%2A&acls=True&limit=10&page=3\r\nPYTHONPATH=../pdc-client/. pdc-updater-initialize  8.23s user 0.90s system 0% cpu 15:34.67 total                                                                                                            \r\n```"}}