{"action":"opened","issue":{"url":"https://api.github.com/repos/ligo-cbc/pycbc/issues/317","labels_url":"https://api.github.com/repos/ligo-cbc/pycbc/issues/317/labels{/name}","comments_url":"https://api.github.com/repos/ligo-cbc/pycbc/issues/317/comments","events_url":"https://api.github.com/repos/ligo-cbc/pycbc/issues/317/events","html_url":"https://github.com/ligo-cbc/pycbc/issues/317","id":107423232,"number":317,"title":"The workflow generators can corrupt the xml files that they download","user":{"login":"duncan-brown","id":11262902,"avatar_url":"https://avatars.githubusercontent.com/u/11262902?v=3","gravatar_id":"","url":"https://api.github.com/users/duncan-brown","html_url":"https://github.com/duncan-brown","followers_url":"https://api.github.com/users/duncan-brown/followers","following_url":"https://api.github.com/users/duncan-brown/following{/other_user}","gists_url":"https://api.github.com/users/duncan-brown/gists{/gist_id}","starred_url":"https://api.github.com/users/duncan-brown/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/duncan-brown/subscriptions","organizations_url":"https://api.github.com/users/duncan-brown/orgs","repos_url":"https://api.github.com/users/duncan-brown/repos","events_url":"https://api.github.com/users/duncan-brown/events{/privacy}","received_events_url":"https://api.github.com/users/duncan-brown/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-09-20T23:04:54Z","updated_at":"2015-09-20T23:04:54Z","closed_at":null,"body":"If you tell the workflow generators to get the veto definer file from a URL, then you can end up with a corrupted file. For example, if you specify \r\n```python\r\nsegments-veto-definer-url = https://code.pycbc.phy.syr.edu/detchar/veto-definitions/download/fad6049282c4888dfe47d6633a82fecc6891fb1a/cbc/ER8/H1L1-HOFT_C00_ER8A_CBC.xml\r\n```\r\nand run\r\n```shell\r\npycbc_make_psd_estimation_workflow --workflow-name er8_psd --output-dir psd-output --local-config-files psd_estimation.ini data_er8.ini --config-overrides \"workflow:start-time:1126033217\" \"workflow:end-time:1126256417\"\r\n```\r\nThen the workflow fails in segments_from_cats.sh with the error\r\n```\r\nxml.sax._exceptions.SAXParseException: <unknown>:1:0: syntax error\r\n```\r\nThis can be seen just by running ligolw_print on the downloaded file. If you curl the veto file directly to replace the one downloaded by the workflow generator\r\n```shell\r\ncurl https://code.pycbc.phy.syr.edu/detchar/veto-definitions/download/fad6049282c4888dfe47d6633a82fecc6891fb1a/cbc/ER8/H1L1-HOFT_C00_ER8A_CBC.xml > /home/steven.reyes/projects/cbc/er8/psd-test/psd-output/segments/H1L1-HOFT_C00_ER8A_CBC.xml\r\n```\r\nthen it works fine:\r\n```shell\r\n/home/steven.reyes/pycbc-dev/bin/ligolw_segments_from_cats_dqsegdb --separate-categories --segment-url https://segments.ligo.org --veto-file /home/steven.reyes/projects/cbc/er8/psd-test/psd-output/segments/H1L1-HOFT_C00_ER8A_CBC.xml --veto-categories 1 --ifo-list H1 --gps-start-time 1126033217 --gps-end-time 1126256417\r\n```\r\nand makes the file\r\n```\r\n(pycbc-dev)[steven.reyes@sugar-dev3 psd-test]$ ls H1-VETOTIME_CAT1-1126033217-223200.xml\r\nH1-VETOTIME_CAT1-1126033217-223200.xml\r\n```\r\n\r\n@spxiwh and @ahnitz I can't find exactly where the code is doing the download. Any ideas?\r\n"}}