{"action":"closed","issue":{"url":"https://api.github.com/repos/dmwm/WMCore/issues/6263","labels_url":"https://api.github.com/repos/dmwm/WMCore/issues/6263/labels{/name}","comments_url":"https://api.github.com/repos/dmwm/WMCore/issues/6263/comments","events_url":"https://api.github.com/repos/dmwm/WMCore/issues/6263/events","html_url":"https://github.com/dmwm/WMCore/issues/6263","id":107325666,"number":6263,"title":"DecodeError parsing json in ReqMgr2","user":{"login":"amaltaro","id":1798520,"avatar_url":"https://avatars.githubusercontent.com/u/1798520?v=3","gravatar_id":"","url":"https://api.github.com/users/amaltaro","html_url":"https://github.com/amaltaro","followers_url":"https://api.github.com/users/amaltaro/followers","following_url":"https://api.github.com/users/amaltaro/following{/other_user}","gists_url":"https://api.github.com/users/amaltaro/gists{/gist_id}","starred_url":"https://api.github.com/users/amaltaro/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amaltaro/subscriptions","organizations_url":"https://api.github.com/users/amaltaro/orgs","repos_url":"https://api.github.com/users/amaltaro/repos","events_url":"https://api.github.com/users/amaltaro/events{/privacy}","received_events_url":"https://api.github.com/users/amaltaro/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":{"login":"ticoann","id":1247993,"avatar_url":"https://avatars.githubusercontent.com/u/1247993?v=3","gravatar_id":"","url":"https://api.github.com/users/ticoann","html_url":"https://github.com/ticoann","followers_url":"https://api.github.com/users/ticoann/followers","following_url":"https://api.github.com/users/ticoann/following{/other_user}","gists_url":"https://api.github.com/users/ticoann/gists{/gist_id}","starred_url":"https://api.github.com/users/ticoann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ticoann/subscriptions","organizations_url":"https://api.github.com/users/ticoann/orgs","repos_url":"https://api.github.com/users/ticoann/repos","events_url":"https://api.github.com/users/ticoann/events{/privacy}","received_events_url":"https://api.github.com/users/ticoann/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/dmwm/WMCore/milestones/116","html_url":"https://github.com/dmwm/WMCore/milestones/WMAgent1509","labels_url":"https://api.github.com/repos/dmwm/WMCore/milestones/116/labels","id":1245706,"number":116,"title":"WMAgent1509","description":"","creator":{"login":"ticoann","id":1247993,"avatar_url":"https://avatars.githubusercontent.com/u/1247993?v=3","gravatar_id":"","url":"https://api.github.com/users/ticoann","html_url":"https://github.com/ticoann","followers_url":"https://api.github.com/users/ticoann/followers","following_url":"https://api.github.com/users/ticoann/following{/other_user}","gists_url":"https://api.github.com/users/ticoann/gists{/gist_id}","starred_url":"https://api.github.com/users/ticoann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ticoann/subscriptions","organizations_url":"https://api.github.com/users/ticoann/orgs","repos_url":"https://api.github.com/users/ticoann/repos","events_url":"https://api.github.com/users/ticoann/events{/privacy}","received_events_url":"https://api.github.com/users/ticoann/received_events","type":"User","site_admin":false},"open_issues":9,"closed_issues":16,"state":"open","created_at":"2015-08-07T15:47:07Z","updated_at":"2015-09-24T03:43:50Z","due_on":"2015-09-30T05:00:00Z","closed_at":null},"comments":0,"created_at":"2015-09-19T11:04:52Z","updated_at":"2015-09-24T03:43:50Z","closed_at":"2015-09-24T03:43:50Z","body":"These errors seem to be happening quite often in the logs. Traceback example here [1]\r\n\r\nand another example here [2]\r\n\r\n[1]\r\n```\r\n[16/Sep/2015:15:48:59] vocms0136.cern.ch 131.225.205.13 \"GET /reqmgr2/data/request?name=pdmvserv_BPH-RunIISpring15DR74-00038_00341_v0__150905_183106_4346 HTTP/1.1\" 200 OK [data: 3167 in 3540 out 9473 us \r\n] [auth: OK \"/DC=com/DC=DigiCert-Grid/O=Open Science Grid/OU=Services/CN=wmagent/cmsgwms-submit1.fnal.gov\" \"\" ] [ref: \"\" \"WMCore.Services.Requests/v001\" ]\r\n[16/Sep/2015:15:49:05]  SERVER OTHER ERROR __builtins__.KeyError a892def79050efbf10a6f6eea2143d77 ('rows')\r\n[16/Sep/2015:15:49:05]    Traceback (most recent call last):\r\n[16/Sep/2015:15:49:05]      File \"/data/srv/beHG1509i/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.9.pre9/lib/python2.6/site-packages/WMCore/REST/Server.py\", line 697, in default\r\n[16/Sep/2015:15:49:05]        return self._call(RESTArgs(list(args), kwargs))\r\n[16/Sep/2015:15:49:05]      File \"/data/srv/beHG1509i/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.9.pre9/lib/python2.6/site-packages/WMCore/REST/Server.py\", line 772, in _call\r\n[16/Sep/2015:15:49:05]        obj = apiobj['call'](*safe.args, **safe.kwargs)\r\n[16/Sep/2015:15:49:05]  SERVER OTHER ERROR cjson.DecodeError 91ac402c439dfe1f70582f8d59e1cf9d (cannot parse JSON description)\r\n[16/Sep/2015:15:49:05]      File \"/data/srv/beHG1509i/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.9.pre9/lib/python2.6/site-packages/WMCore/ReqMgr/Service/Request.py\", line 278, in get\r\n[16/Sep/2015:15:49:05]    Traceback (most recent call last):\r\n[16/Sep/2015:15:49:05]        request_info.append(self.reqmgr_db_service.getRequestByNames(name))\r\n[16/Sep/2015:15:49:05]      File \"/data/srv/beHG1509i/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.9.pre9/lib/python2.6/site-packages/WMCore/REST/Server.py\", line 697, in default\r\n[16/Sep/2015:15:49:05]      File \"/data/srv/beHG1509i/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.9.pre9/lib/python2.6/site-packages/WMCore/Services/RequestDB/RequestDBReader.py\", line 152, in getRequestByN\r\names\r\n[16/Sep/2015:15:49:05]        return self._call(RESTArgs(list(args), kwargs))\r\n[16/Sep/2015:15:49:05]        requestInfo = self._formatCouchData(data, detail = detail)\r\n[16/Sep/2015:15:49:05]      File \"/data/srv/beHG1509i/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.9.pre9/lib/python2.6/site-packages/WMCore/REST/Server.py\", line 772, in _call\r\n[16/Sep/2015:15:49:05]      File \"/data/srv/beHG1509i/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.9.pre9/lib/python2.6/site-packages/WMCore/Services/RequestDB/RequestDBReader.py\", line 62, in _formatCouchDa\r\nta\r\n[16/Sep/2015:15:49:05]        obj = apiobj['call'](*safe.args, **safe.kwargs)\r\n[16/Sep/2015:15:49:05]        for row in data['rows']:\r\n[16/Sep/2015:15:49:05]      File \"/data/srv/beHG1509i/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.9.pre9/lib/python2.6/site-packages/WMCore/ReqMgr/Service/Request.py\", line 278, in get\r\n[16/Sep/2015:15:49:05]    KeyError: 'rows'\r\n[16/Sep/2015:15:49:05]        request_info.append(self.reqmgr_db_service.getRequestByNames(name))\r\n[16/Sep/2015:15:49:05] vocms0136.cern.ch 188.184.162.209 \"GET /reqmgr2/data/request?name=anlevin_RVCMSSW_7_6_0_pre5PhotonJets_Pt_10_13_HI_150916_124311_7729 HTTP/1.1\" 500 Internal Server Error [data: 3939 in 723 out 5779881 us ] [auth: OK \"/DC=ch/DC=cern/OU=computers/CN=wmagent/vocms0308.cern.ch\" \"\" ] [ref: \"\" \"WMCore.Services.Requests/v001\" ]\r\n[16/Sep/2015:15:49:05]      File \"/data/srv/beHG1509i/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.9.pre9/lib/python2.6/site-packages/WMCore/Services/RequestDB/RequestDBReader.py\", line 150, in getRequestByNames\r\n[16/Sep/2015:15:49:05]        data = self._getRequestByNames(requestNames, detail = detail)\r\n[16/Sep/2015:15:49:05]      File \"/data/srv/beHG1509i/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.9.pre9/lib/python2.6/site-packages/WMCore/Services/RequestDB/RequestDBReader.py\", line 82, in _getRequestByNames\r\n[16/Sep/2015:15:49:05]        result = self.couchDB.allDocs(options, requestNames)\r\n[16/Sep/2015:15:49:05]      File \"/data/srv/beHG1509i/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.9.pre9/lib/python2.6/site-packages/WMCore/Database/CMSCouch.py\", line 532, in allDocs\r\n[16/Sep/2015:15:49:05]        {'keys':keys})\r\n[16/Sep/2015:15:49:05]      File \"/data/srv/beHG1509i/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.9.pre9/lib/python2.6/site-packages/WMCore/Services/Requests.py\", line 117, in post\r\n[16/Sep/2015:15:49:05]        encode, decode, contentType)\r\n[16/Sep/2015:15:49:05]      File \"/data/srv/beHG1509i/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.9.pre9/lib/python2.6/site-packages/WMCore/Database/CMSCouch.py\", line 113, in makeRequest\r\n[16/Sep/2015:15:49:05]        encode, decode,contentType)\r\n[16/Sep/2015:15:49:05]      File \"/data/srv/beHG1509i/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.9.pre9/lib/python2.6/site-packages/WMCore/Services/Requests.py\", line 145, in makeRequest\r\n[16/Sep/2015:15:49:05]        encoder, decoder, contentType)\r\n[16/Sep/2015:15:49:05]      File \"/data/srv/beHG1509i/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.9.pre9/lib/python2.6/site-packages/WMCore/Services/Requests.py\", line 280, in makeRequest_httplib\r\n[16/Sep/2015:15:49:05]        result = self.decode(result)\r\n[16/Sep/2015:15:49:05]      File \"/data/srv/beHG1509i/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.9.pre9/lib/python2.6/site-packages/WMCore/Services/Requests.py\", line 571, in decode\r\n[16/Sep/2015:15:49:05]        data =  decoder.decode(data)\r\n[16/Sep/2015:15:49:05]      File \"/data/srv/beHG1509i/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.9.pre9/lib/python2.6/site-packages/WMCore/Wrappers/JsonWrapper/__init__.py\", line 93, in decode\r\n[16/Sep/2015:15:49:05]        return cjson.decode(idict)\r\n[16/Sep/2015:15:49:05]    DecodeError: cannot parse JSON description\r\n[16/Sep/2015:15:49:05] vocms0136.cern.ch 131.225.205.13 \"GET /reqmgr2/data/request?name=pdmvserv_EXO-RunIISpring15DR74-01763_00354_v0__150910_183215_8358 HTTP/1.1\" 500 Internal Server Error [data: 3167 in 723 out 5020031 us ] [auth: OK \"/DC=com/DC=DigiCert-Grid/O=Open Science Grid/OU=Services/CN=wmagent/cmsgwms-submit1.fnal.gov\" \"\" ] [ref: \"\" \"WMCore.Services.Requests/v001\" ]\r\n```\r\n\r\n[2]\r\n```\r\n[16/Sep/2015:17:32:03] vocms0136.cern.ch 131.225.205.13 \"GET /reqmgr2/data/request?name=pdmvserv_SUS-RunIISummer15GS-00051_00076_v0__150912_201437_5351 HTTP/1.1\" 200 OK [data: 3163 in 2789 out 15116 us ]\r\n [auth: OK \"/DC=com/DC=DigiCert-Grid/O=Open Science Grid/OU=Services/CN=wmagent/cmsgwms-submit1.fnal.gov\" \"\" ] [ref: \"\" \"WMCore.Services.Requests/v001\" ]\r\n[16/Sep/2015:17:32:03]  SERVER OTHER ERROR cjson.DecodeError 9bd7eb571764483ed2e41b77b0c2bdca (cannot parse JSON description)\r\n[16/Sep/2015:17:32:03]    Traceback (most recent call last):\r\n[16/Sep/2015:17:32:03]      File \"/data/srv/beHG1509i/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.9.pre9/lib/python2.6/site-packages/WMCore/REST/Server.py\", line 697, in default\r\n[16/Sep/2015:17:32:03]        return self._call(RESTArgs(list(args), kwargs))\r\n[16/Sep/2015:17:32:03]      File \"/data/srv/beHG1509i/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.9.pre9/lib/python2.6/site-packages/WMCore/REST/Server.py\", line 772, in _call\r\n[16/Sep/2015:17:32:03]        obj = apiobj['call'](*safe.args, **safe.kwargs)\r\n[16/Sep/2015:17:32:03]      File \"/data/srv/beHG1509i/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.9.pre9/lib/python2.6/site-packages/WMCore/ReqMgr/Service/Request.py\", line 278, in get\r\n[16/Sep/2015:17:32:03]        request_info.append(self.reqmgr_db_service.getRequestByNames(name))\r\n[16/Sep/2015:17:32:03]      File \"/data/srv/beHG1509i/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.9.pre9/lib/python2.6/site-packages/WMCore/Services/RequestDB/RequestDBReader.py\", line 150, in getRequestByN\r\names\r\n[16/Sep/2015:17:32:03]        data = self._getRequestByNames(requestNames, detail = detail)\r\n[16/Sep/2015:17:32:03]      File \"/data/srv/beHG1509i/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.9.pre9/lib/python2.6/site-packages/WMCore/Services/RequestDB/RequestDBReader.py\", line 82, in _getRequestByN\r\names\r\n[16/Sep/2015:17:32:03]        result = self.couchDB.allDocs(options, requestNames)\r\n[16/Sep/2015:17:32:03]      File \"/data/srv/beHG1509i/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.9.pre9/lib/python2.6/site-packages/WMCore/Database/CMSCouch.py\", line 532, in allDocs\r\n[16/Sep/2015:17:32:03]        {'keys':keys})\r\n[16/Sep/2015:17:32:03]      File \"/data/srv/beHG1509i/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.9.pre9/lib/python2.6/site-packages/WMCore/Services/Requests.py\", line 117, in post\r\n[16/Sep/2015:17:32:03]        encode, decode, contentType)\r\n[16/Sep/2015:17:32:03]      File \"/data/srv/beHG1509i/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.9.pre9/lib/python2.6/site-packages/WMCore/Database/CMSCouch.py\", line 113, in makeRequest\r\n[16/Sep/2015:17:32:03]        encode, decode,contentType)\r\n[16/Sep/2015:17:32:03]      File \"/data/srv/beHG1509i/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.9.pre9/lib/python2.6/site-packages/WMCore/Services/Requests.py\", line 145, in makeRequest\r\n[16/Sep/2015:17:32:03]        encoder, decoder, contentType)\r\n[16/Sep/2015:17:32:03]      File \"/data/srv/beHG1509i/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.9.pre9/lib/python2.6/site-packages/WMCore/Services/Requests.py\", line 280, in makeRequest_httplib\r\n[16/Sep/2015:17:32:03]        result = self.decode(result)\r\n[16/Sep/2015:17:32:03]      File \"/data/srv/beHG1509i/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.9.pre9/lib/python2.6/site-packages/WMCore/Services/Requests.py\", line 571, in decode\r\n[16/Sep/2015:17:32:03]        data =  decoder.decode(data)\r\n[16/Sep/2015:17:32:03]      File \"/data/srv/beHG1509i/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.9.pre9/lib/python2.6/site-packages/WMCore/Wrappers/JsonWrapper/__init__.py\", line 93, in decode\r\n[16/Sep/2015:17:32:03]        return cjson.decode(idict)\r\n[16/Sep/2015:17:32:03]    DecodeError: cannot parse JSON description\r\n[16/Sep/2015:17:32:03] vocms0136.cern.ch 128.142.209.43 \"GET /reqmgr2/data/request?name=pdmvserv_HIG-RunIISummer15GS-00019_00069_v0__150909_001636_4546 HTTP/1.1\" 500 Internal Server Error [data: 3930 in 723 out 5018451 us ] [auth: OK \"/DC=ch/DC=cern/OU=computers/CN=wmagent/vocms0308.cern.ch\" \"\" ] [ref: \"\" \"WMCore.Services.Requests/v001\" ]\r\n```"}}