{"action":"created","issue":{"url":"https://api.github.com/repos/mozilla-services/syncserver/issues/52","labels_url":"https://api.github.com/repos/mozilla-services/syncserver/issues/52/labels{/name}","comments_url":"https://api.github.com/repos/mozilla-services/syncserver/issues/52/comments","events_url":"https://api.github.com/repos/mozilla-services/syncserver/issues/52/events","html_url":"https://github.com/mozilla-services/syncserver/issues/52","id":55835481,"number":52,"title":"nginx/syncserver 503 error / connection_error/ verify_failure","user":{"login":"codeling","id":1629217,"avatar_url":"https://avatars.githubusercontent.com/u/1629217?v=3","gravatar_id":"","url":"https://api.github.com/users/codeling","html_url":"https://github.com/codeling","followers_url":"https://api.github.com/users/codeling/followers","following_url":"https://api.github.com/users/codeling/following{/other_user}","gists_url":"https://api.github.com/users/codeling/gists{/gist_id}","starred_url":"https://api.github.com/users/codeling/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/codeling/subscriptions","organizations_url":"https://api.github.com/users/codeling/orgs","repos_url":"https://api.github.com/users/codeling/repos","events_url":"https://api.github.com/users/codeling/events{/privacy}","received_events_url":"https://api.github.com/users/codeling/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":18,"created_at":"2015-01-28T23:57:12Z","updated_at":"2015-02-06T06:26:38Z","closed_at":null,"body":"Following [this guide](https://thomas-leister.de/internet/ubuntu-firefox-sync-1-5-server-installieren/) (in german), after getting the auth server running, I'm having a problem running the sync server.\r\nIt is set up behind a nginx ssl reverse proxy, just as the auth and auth-api server (which both seem to work fine).\r\nThe sync server only shows this in his output (line breaks inserted for readability):\r\n```\r\n$ make serve\r\n./local/bin/pserve ./syncserver.ini\r\n/home/mozillasync/syncserver/local/local/lib/python2.7/site-packages/tokenserver/verifiers.py:47:\r\nFutureWarning: The BrowserID certificate format has not been finalized and may change in\r\nbackwards-incompatible ways.  If you find that the latest version of this module cannot verify a valid\r\nBrowserID assertion, please contact the author.\r\n  super(LocalVerifier, self).__init__(**kwargs)\r\nStarting server in PID 29076.\r\n!!!\r\n!!! WARNING: This command is deprecated.\r\n!!! \r\n!!!     You should now use the `--paste` option. Ex.:\r\n!!! \r\n!!!         gunicorn --paste development.ini\r\n!!!     \r\n!!!\r\n\r\nINFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): myurl.com\r\nINFO:mozsvc.metrics:{\"code\": 503, \"request_time\": 0.19293904304504395,\r\n\"remoteAddressChain\": [\"myexternalip\", \"127.0.0.1\"],\r\n\"agent\": \"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:35.0) Gecko/20100101 Firefox/35.0\",\r\n\"token.assertion.connection_error\": 1, \"token.assertion.verify_failure\": 1,\r\n\"tokenserver.assertion.verify\": 0.18337583541870117,\r\n\"path\": \"https://myurl.com/token/1.0/sync/1.5\", \"method\": \"GET\"}\r\n```\r\nThe \"INFO\" messages repeat every time a sync seems to run. Is this related to the browserid check? I thought I had configured the browserID verification appropriately:\r\n```\r\n[browserid]\r\nbackend = tokenserver.verifiers.LocalVerifier\r\naudiences = *\r\n```\r\n\r\nThe ngnix log has this in the error log:\r\n```\r\n2015/01/26 00:27:40 [error] 2334#0: *388 connect() failed (111: Connection refused) while connecting to upstream, client: 91.119.55.0, server: mydomain, request: \"GET /token/1.0/sync/1.5 HTTP/1.1\", upstream: \"http://127.0.0.1:5000/token/1.0/sync/1.5\", host: \"mydomain\"\r\n```\r\n\r\nI'm at a loss at how to proceed. What do the errors refer? Any additional logs I could check?\r\nSorry if this is not the correct place to report this, if so could you point me to the correct one?\r\n\r\nAny help is much appreciated!"},"comment":{"url":"https://api.github.com/repos/mozilla-services/syncserver/issues/comments/73188561","html_url":"https://github.com/mozilla-services/syncserver/issues/52#issuecomment-73188561","issue_url":"https://api.github.com/repos/mozilla-services/syncserver/issues/52","id":73188561,"user":{"login":"codeling","id":1629217,"avatar_url":"https://avatars.githubusercontent.com/u/1629217?v=3","gravatar_id":"","url":"https://api.github.com/users/codeling","html_url":"https://github.com/codeling","followers_url":"https://api.github.com/users/codeling/followers","following_url":"https://api.github.com/users/codeling/following{/other_user}","gists_url":"https://api.github.com/users/codeling/gists{/gist_id}","starred_url":"https://api.github.com/users/codeling/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/codeling/subscriptions","organizations_url":"https://api.github.com/users/codeling/orgs","repos_url":"https://api.github.com/users/codeling/repos","events_url":"https://api.github.com/users/codeling/events{/privacy}","received_events_url":"https://api.github.com/users/codeling/received_events","type":"User","site_admin":false},"created_at":"2015-02-06T06:26:38Z","updated_at":"2015-02-06T06:26:38Z","body":"> Is there addition information from this request in the client-side log?\r\n\r\nThere is, yes:\r\n```\r\n1423175931872   FirefoxAccounts DEBUG   got keyPair\r\n1423175931872   FirefoxAccounts DEBUG   getCertificateSigned: true true\r\n1423175932522   FirefoxAccounts ERROR   error POSTing /certificate/sign: {\"code\":401,\"errno\":110,\"error\":\"Unauthorized\",\"message\":\"Invalid authentication token in request signature\",\"info\":\"https://github.com/mozilla/fxa-auth-server/blob/master/docs/api.md#response-format\",\"log\":[]}\r\n1423175932522   FirefoxAccounts ERROR   HAWK.signCertificate error: {\"code\":401,\"errno\":110,\"error\":\"Unauthorized\",\"message\":\"Invalid authentication token in request signature\",\"info\":\"https://github.com/mozilla/fxa-auth-server/blob/master/docs/api.md#response-format\",\"log\":[]}\r\n1423175932523   Sync.BrowserIDManager   ERROR   Authentication error in _fetchTokenForUser: AuthenticationError([object Object])\r\n1423175932523   Sync.Status DEBUG   Status.login: success.login => error.login.reason.account\r\n1423175932523   Sync.Status DEBUG   Status.service: success.status_ok => error.login.failed\r\n1423175932523   Sync.SyncScheduler  DEBUG   Clearing sync triggers and the global score.\r\n```\r\nand\r\n```\r\n1423175932525   Sync.BrowserIDManager   ERROR   Background fetch for key bundle failed: AuthenticationError([object Object])\r\n1423175932525   Sync.BrowserIDManager   ERROR   Could not authenticate: AuthenticationError([object Object])\r\n1423175932525   Sync.BrowserIDManager   INFO    currentAuthState returning error.login.reason.account due to previous failure\r\n1423175932525   Sync.Status DEBUG   Status.login: error.login.reason.account => error.login.reason.account\r\n1423175932525   Sync.Status DEBUG   Status.service: error.login.failed => error.login.failed\r\n```\r\nand\r\n```\r\n1423175940943   Sync.BrowserIDManager   ERROR   Authentication error in _fetchTokenForUser: AuthenticationError(TokenServerClientServerError({\"now\":\"2015-02-05T22:39:00.943Z\",\"message\":\"Authentication failed.\",\"cause\":\"invalid-credentials\",\"response_body\":\"{\\\"status\\\": \\\"invalid-credentials\\\", \\\"errors\\\": [{\\\"location\\\": \\\"body\\\", \\\"name\\\": \\\"\\\", \\\"description\\\": \\\"Unauthorized\\\"}]}\",\"response_headers\":{\"server\":\"nginx/1.4.6 (Ubuntu)\",\"date\":\"Thu, 05 Feb 2015 22:38:59 GMT\",\"content-type\":\"application/json; charset=UTF-8\",\"content-length\":\"110\",\"connection\":\"keep-alive\",\"x-weave-timestamp\":\"1423175939.36\",\"x-timestamp\":\"1423175939\"},\"response_status\":401}))\r\n1423175940943   Sync.Status DEBUG   Status.login: error.login.reason.account => error.login.reason.account\r\n1423175940943   Sync.Status DEBUG   Status.service: error.login.failed => error.login.failed\r\n```\r\n\r\n> It may be worth deleting your local sync/tokenserver database and starting from a clean slate in case it's got some old records in there that are confusing things.\r\n\r\nThe token database still is empty - there are tables but they all have zero rows."}}