{"action":"created","issue":{"url":"https://api.github.com/repos/dmwm/DAS/issues/4236","labels_url":"https://api.github.com/repos/dmwm/DAS/issues/4236/labels{/name}","comments_url":"https://api.github.com/repos/dmwm/DAS/issues/4236/comments","events_url":"https://api.github.com/repos/dmwm/DAS/issues/4236/events","html_url":"https://github.com/dmwm/DAS/issues/4236","id":89561560,"number":4236,"title":"June 18/19 2015 das query failures due to proxy error","user":{"login":"slava77","id":4676718,"avatar_url":"https://avatars.githubusercontent.com/u/4676718?v=3","gravatar_id":"","url":"https://api.github.com/users/slava77","html_url":"https://github.com/slava77","followers_url":"https://api.github.com/users/slava77/followers","following_url":"https://api.github.com/users/slava77/following{/other_user}","gists_url":"https://api.github.com/users/slava77/gists{/gist_id}","starred_url":"https://api.github.com/users/slava77/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/slava77/subscriptions","organizations_url":"https://api.github.com/users/slava77/orgs","repos_url":"https://api.github.com/users/slava77/repos","events_url":"https://api.github.com/users/slava77/events{/privacy}","received_events_url":"https://api.github.com/users/slava77/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2015-06-19T13:12:21Z","updated_at":"2015-06-19T15:55:41Z","closed_at":null,"body":"das_client calls to query file list in /RelValMinBias/CMSSW_7_1_0_pre7-PRE_STA71_V3-v1/GEN-SIM in particular were failing systemically since June 18 or so.\r\n\r\n```\r\nhttp_error_default\r\n    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)\r\nurllib2.HTTPError: HTTP Error 502: Proxy Error\r\n\r\n```\r\n\r\nThis part of the call comes from ConfigBuilder::filesFromDASQuery https://github.com/cms-sw/cmssw/blob/13f1a1c1d532f4c345867ece12287505ffbb3647/Configuration/Applications/python/ConfigBuilder.py#L124\r\n\r\nI tried both the regular\r\n```\r\nrunTheMatrix.py -l 202.0 --useInput all\r\n```\r\nas well as\r\n```\r\nrunTheMatrix.py -l 202.0 --das-options=\"--limit=0 --host=https://das-stack.cern.ch\" --useInput all\r\n```\r\nboth fail.\r\n"},"comment":{"url":"https://api.github.com/repos/dmwm/DAS/issues/comments/113555522","html_url":"https://github.com/dmwm/DAS/issues/4236#issuecomment-113555522","issue_url":"https://api.github.com/repos/dmwm/DAS/issues/4236","id":113555522,"user":{"login":"vkuznet","id":125138,"avatar_url":"https://avatars.githubusercontent.com/u/125138?v=3","gravatar_id":"","url":"https://api.github.com/users/vkuznet","html_url":"https://github.com/vkuznet","followers_url":"https://api.github.com/users/vkuznet/followers","following_url":"https://api.github.com/users/vkuznet/following{/other_user}","gists_url":"https://api.github.com/users/vkuznet/gists{/gist_id}","starred_url":"https://api.github.com/users/vkuznet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vkuznet/subscriptions","organizations_url":"https://api.github.com/users/vkuznet/orgs","repos_url":"https://api.github.com/users/vkuznet/repos","events_url":"https://api.github.com/users/vkuznet/events{/privacy}","received_events_url":"https://api.github.com/users/vkuznet/received_events","type":"User","site_admin":false},"created_at":"2015-06-19T15:55:41Z","updated_at":"2015-06-19T15:55:41Z","body":"Slava,\nI checked both frontend and DAS logs, see [1]. IB queries were done at 14:58\nand only entries in logs are RelValTTBar around this time. There is no entries\nfor RelValMinBias. Feel free to check yourself, I put logs in [1].  I think you\nhave some mis-configuration for cmsDriver, did you pass the same options as to\nrunTheMatrix, e.g. --das-options=\"--limit=0 --host=https://das-stack.cern.ch\"?\n\nI bet that it is not the case and those queries passed to central node\nwhich as we know had some issues.\n\n[1] /afs/cern.ch/user/v/valya/public/Logs4IB\n\nOn  0, Slava Krutelyov <notifications@github.com> wrote:\n> Valentin,\n> \n> In your frontend log snippets you mentioned only\n> \n> RelValTTbar*\t(this is what shows up in the step1_dasquery.log file made\n> by explicit calls from runTheMatrix.py),\n> \n> however, I'm puzzled that there are no calls to\n> RelValMinBias*\n> these are parsed during the cmsDriver.py command call made by the\n> runTheMatrix\n> Could you please check if they are there as well (typically, right after\n> the RelValTTbar*)\n> \n> \t\t--slava\n> \n> \n> On 6/19/15 9:53 AM, Valentin Kuznetsov wrote:\n> > Slava,\n> > where are the logs on AFS? At least when I run runTheMatrix.py script I got\n> > plenty of directories. So I need to know the location of your IB build logs\n> > that I can look at timestamps of files and compare them with logs on my VM.\n> > V.\n> > \n> > On 0, Slava Krutelyov <notifications@github.com> wrote:\n> >> Valentin,\n> >> \n> >> I'm not sure which logs to show.\n> >> The only relevant part is this\n> >> \n> >> Traceback (most recent call last):\n> >>   File\n> >> \"/afs/cern.ch/cms/sw/ReleaseCandidates/vol1/slc6_amd64_gcc491/cms/cmssw-patch/CMSSW_7_5_X_2015-06-19-1100/external/slc6_amd64_gcc491/bin/das_client.py\",\n> >> line 447, in <module>\n> >>     main()\n> >>   File\n> >> \"/afs/cern.ch/cms/sw/ReleaseCandidates/vol1/slc6_amd64_gcc491/cms/cmssw-patch/CMSSW_7_5_X_2015-06-19-1100/external/slc6_amd64_gcc491/bin/das_client.py\",\n> >> line 345, in main\n> >>     jsondict = get_data(host, query, idx, limit, debug, thr, ckey, cert)\n> >>   File\n> >> \"/afs/cern.ch/cms/sw/ReleaseCandidates/vol1/slc6_amd64_gcc491/cms/cmssw-patch/CMSSW_7_5_X_2015-06-19-1100/external/slc6_amd64_gcc491/bin/das_client.py\",\n> >> line 252, in get_data\n> >>     fdesc = opener.open(req)\n> >>   File\n> >> \"/afs/cern.ch/cms/sw/ReleaseCandidates/vol1/slc6_amd64_gcc491/external/python/2.7.6-eccfad/lib/python2.7/urllib2.py\",\n> >> line 410, in open\n> >>     response = meth(req, response)\n> >>   File\n> >> \"/afs/cern.ch/cms/sw/ReleaseCandidates/vol1/slc6_amd64_gcc491/external/python/2.7.6-eccfad/lib/python2.7/urllib2.py\",\n> >> line 523, in http_response\n> >>     'http', request, response, code, msg, hdrs)\n> >>   File\n> >> \"/afs/cern.ch/cms/sw/ReleaseCandidates/vol1/slc6_amd64_gcc491/external/python/2.7.6-eccfad/lib/python2.7/urllib2.py\",\n> >> line 448, in error\n> >>     return self._call_chain(*args)\n> >>   File\n> >> \"/afs/cern.ch/cms/sw/ReleaseCandidates/vol1/slc6_amd64_gcc491/external/python/2.7.6-eccfad/lib/python2.7/urllib2.py\",\n> >> line 382, in _call_chain\n> >>     result = func(*args)\n> >>   File\n> >> \"/afs/cern.ch/cms/sw/ReleaseCandidates/vol1/slc6_amd64_gcc491/external/python/2.7.6-eccfad/lib/python2.7/urllib2.py\",\n> >> line 531, in http_error_default\n> >>     raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)\n> >> urllib2.HTTPError: HTTP Error 502: Proxy Error\n> >> \n> >> \n> >> In my original post I quoted only the first part of the query.\n> >> \n> >> Is there an environment variable for das_client to react and start\n> >> making more verbose output?\n> >> \n> >> \n> >> \n> >> On 6/19/15 8:35 AM, Valentin Kuznetsov wrote:\n> >> > Slava,\n> >> > I had a look at DAS/frontend logs on das-stack. The only calls I saw was\n> >> > \n> >> > [19/Jun/2015:14:58:26 +0200] das-stack.cern.ch 137.138.200.181 \"GET\n> >> > /das/cache?input=file+dataset%3D%2FRelValTTbar%2FCMSSW_7_1_0_pre7-PRE_STA71_V3-v1%2FGEN-SIM+site%3DT2_CH_CERN&limit=0&idx=0\n> >> > HTTP/1.1\" 200 [data: 628 in 22266 out 32 body 71908 us ] [auth: TLSv1\n> >> > DHE-RSA-AES256-SHA \"-\" \"-\" ] [ref: \"-\" \"das-client/1.0::python/2.7\" ]\n> >> > [19/Jun/2015:14:58:26 +0200] das-stack.cern.ch 137.138.200.181 \"GET\n> >> > /das/cache?input=file+dataset%3D%2FRelValTTbar_13%2FCMSSW_7_5_0_pre4-MCRUN2_75_V1-v1%2FGEN-SIM+site%3DT2_CH_CERN&limit=0&idx=0\n> >> > HTTP/1.1\" 200 [data: 628 in 22266 out 32 body 60248 us ] [auth: TLSv1\n> >> > DHE-RSA-AES256-SHA \"-\" \"-\" ] [ref: \"-\" \"das-client/1.0::python/2.7\" ]\n> >> > [19/Jun/2015:14:58:26 +0200] das-stack.cern.ch 137.138.200.181 \"GET\n> >> > /das/cache?input=file+dataset%3D%2FRelValTTbar%2FCMSSW_7_1_0_pre7-PRE_STA71_V3-v1%2FGEN-SIM+site%3DT2_CH_CERN&pid=20cdbc6dc8e5529edf324392371aad61&limit=0&idx=0\n> >> > HTTP/1.1\" 200 [data: 660 in 22266 out 32 body 49608 us ] [auth: TLSv1\n> >> > DHE-RSA-AES256-SHA \"-\" \"-\" ] [ref: \"-\" \"das-client/1.0::python/2.7\" ]\n> >> > [19/Jun/2015:14:58:26 +0200] das-stack.cern.ch 137.138.200.181 \"GET\n> >> > /das/cache?input=file+dataset%3D%2FRelValTTbar_13%2FCMSSW_7_5_0_pre4-MCRUN2_75_V1-v1%2FGEN-SIM+site%3DT2_CH_CERN&pid=e85e5ab5b755b69017a1a54a48aa496f&limit=0&idx=0\n> >> > HTTP/1.1\" 200 [data: 676 in 22266 out 32 body 36548 us ] [auth: TLSv1\n> >> > DHE-RSA-AES256-SHA \"-\" \"-\" ] [ref: \"-\" \"das-client/1.0::python/2.7\" ]\n> >> > [19/Jun/2015:14:58:28 +0200] das-stack.cern.ch 137.138.200.181 \"GET\n> >> > /das/cache?input=file+dataset%3D%2FRelValTTbar%2FCMSSW_7_1_0_pre7-PRE_STA71_V3-v1%2FGEN-SIM+site%3DT2_CH_CERN&pid=20cdbc6dc8e5529edf324392371aad61&limit=0&idx=0\n> >> > HTTP/1.1\" 200 [data: 660 in 26106 out 3870 body 141873 us ] [auth: TLSv1\n> >> > DHE-RSA-AES256-SHA \"-\" \"-\" ] [ref: \"-\" \"das-client/1.0::python/2.7\" ]\n> >> > [19/Jun/2015:14:58:28 +0200] das-stack.cern.ch 137.138.200.181 \"GET\n> >> > /das/cache?input=file+dataset%3D%2FRelValTTbar_13%2FCMSSW_7_5_0_pre4-MCRUN2_75_V1-v1%2FGEN-SIM+site%3DT2_CH_CERN&pid=e85e5ab5b755b69017a1a54a48aa496f&limit=0&idx=0\n> >> > HTTP/1.1\" 200 [data: 676 in 26106 out 3883 body 110476 us ] [auth: TLSv1\n> >> > DHE-RSA-AES256-SHA \"-\" \"-\" ] [ref: \"-\" \"das-client/1.0::python/2.7\" ]\n> >> > \n> >> > These are frontend log entries. According to their status all queries\n> >> > had 200 HTTP status, i.e. ok,\n> >> > and I don't see any other errors neither in frontend or das logs.\n> >> > Therefore I really curious to see logs from your end before running\n> >> > runTheMatrix on my own.\n> >> > Do you have those logs?\n> >> > Thanks,\n> >> > Valentin.\n> >> > \n> >> > On 0, Slava Krutelyov <notifications@github.com> wrote:\n> >> >> das_client calls to query file list in /RelValMinBias/CMSSW_7_1_0_pre7-PRE_STA71_V3-v1/GEN-SIM in particular were failing systemically since June 18 or so.\n> >> >> \n> >> >> ```\n> >> >> http_error_default\n> >> >>     raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)\n> >> >> urllib2.HTTPError: HTTP Error 502: Proxy Error\n> >> >> \n> >> >> ```\n> >> >> \n> >> >> This part of the call comes from ConfigBuilder::filesFromDASQuery https://github.com/cms-sw/cmssw/blob/13f1a1c1d532f4c345867ece12287505ffbb3647/Configuration/Applications/python/ConfigBuilder.py#L124\n> >> >> \n> >> >> I tried both the regular\n> >> >> ```\n> >> >> runTheMatrix.py -l 202.0 --useInput all\n> >> >> ```\n> >> >> as well as\n> >> >> ```\n> >> >> runTheMatrix.py -l 202.0 --das-options=\"--limit=0 --host=https://das-stack.cern.ch\" --useInput all\n> >> >> ```\n> >> >> both fail.\n> >> >> \n> >> >> \n> >> >> ---\n> >> >> Reply to this email directly or view it on GitHub:\n> >> >> https://github.com/dmwm/DAS/issues/4236\n> >> > \n> >> > —\n> >> > Reply to this email directly or view it on GitHub\n> >> > <https://github.com/dmwm/DAS/issues/4236#issuecomment-113516999>.\n> >> > \n> >> \n> >> -- \n> >> ***********************************************************************\n> >>  Vyacheslav (Slava) Krutelyov\n> >>  TAMU: Physics Dept Texas A&M MS4242, College Station, TX 77843-4242\n> >>  CERN: 42-R-027\n> >>  AIM/Skype: siava16              googleTalk: slava77@gmail.com\n> >>  (630) 291-5128  Cell (US)       +41 76 275 7116 Cell (CERN)\n> >> \n> >> -----------------------------------------------------------------------\n> >> \n> >> \n> >> \n> >> ---\n> >> Reply to this email directly or view it on GitHub:\n> >> https://github.com/dmwm/DAS/issues/4236#issuecomment-113531915\n> > \n> > —\n> > Reply to this email directly or view it on GitHub\n> > <https://github.com/dmwm/DAS/issues/4236#issuecomment-113537070>.\n> > \n> \n> \n> \n> ---\n> Reply to this email directly or view it on GitHub:\n> https://github.com/dmwm/DAS/issues/4236#issuecomment-113552724\n"}}