{"action":"created","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/gsutil/issues/241","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/gsutil/issues/241/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/gsutil/issues/241/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/gsutil/issues/241/events","html_url":"https://github.com/GoogleCloudPlatform/gsutil/issues/241","id":53241268,"number":241,"title":"depot_tools and gsutil do not respect proxy settings","user":{"login":"rakuco","id":36417,"avatar_url":"https://avatars.githubusercontent.com/u/36417?v=3","gravatar_id":"","url":"https://api.github.com/users/rakuco","html_url":"https://github.com/rakuco","followers_url":"https://api.github.com/users/rakuco/followers","following_url":"https://api.github.com/users/rakuco/following{/other_user}","gists_url":"https://api.github.com/users/rakuco/gists{/gist_id}","starred_url":"https://api.github.com/users/rakuco/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rakuco/subscriptions","organizations_url":"https://api.github.com/users/rakuco/orgs","repos_url":"https://api.github.com/users/rakuco/repos","events_url":"https://api.github.com/users/rakuco/events{/privacy}","received_events_url":"https://api.github.com/users/rakuco/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-01-02T07:46:09Z","updated_at":"2015-01-02T20:20:22Z","closed_at":null,"body":"This is an attempt at better summarizing the issue #240 is trying to solve as well as how [Chromium bug 443523](http://crbug.com/443523) is related to gsutil. I'm assuming everybody is familiar with [depot_tools](http://dev.chromium.org/developers/how-tos/depottools).\r\n\r\nOne of the scripts in depot_tools is **download_from_google_storage**, which is primarily used to download binaries (GN, clang-format, binutils) from Google Storage (behind the scenes, it is basically a  wrapper around `gsutil ls`, `gsutil cp` etc). Those binaries all in public buckets, so fetching them does not require authentication.\r\n\r\nWhen download_from_google_storage is passed the \"--no_auth\" option (as is the case when fetching those public binaries as part of Chromium's runhooks stage), it sets the `BOTO_CONFIG` environment variable to `\"\"` so that any existing Boto authentication settings are not used.\r\n\r\nThis used to work fine for everyone while depot_tools kept its own copy of an old gsutil version with some patches on top. Recently, it switched to a vanilla copy of gsutil 4.7, and this caused download_from_google_storage to stop working for people behind proxies:\r\n1. download_from_google_storage is invoked with \"--no_auth\"\r\n2. It empties the `BOTO_CONFIG` environment variable\r\n3. It calls `gsutil ls`, then `gslib/gcs_json_api.py` calls `gslib.util.GetNewHttp()`\r\n4. `GetNewHttp()` reads proxy settings from Boto\r\n5. The HTTP calls in `gsutil ls` all time out because the proxy settings are incorrect\r\n\r\nThe problem can be addressed from at least two different angles:\r\n* download_from_google_storage could stop resetting `BOTO_CONFIG`, but that would both cause problems for people with proper `.boto` files (possibly, I'm not very familiar with Google Storage and why Chromium tries to avoid Boto) as well as require everyone behind a proxy to have their proxy settings in a `.boto` file -- doable, but likely to cause confusion, and I am not sure Chromium's infra team wants to go in this direction.\r\n* Stop only retrieving proxy settings from Boto and falling back to environment variables if the first approach fails (as #240 does). This is also related to #171, which originally came from a depot_tools patch -- if we are to read the proxy settings from the environment, we'd better not unset `http_proxy` (#240 without #171 works because several connections are made via HTTPS so `https_proxy` is read instead). This option causes less trouble for Chromium, but might be against gsutil's philosophy of always relying on Boto for proxy settings.\r\n"},"comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/gsutil/issues/comments/68558677","html_url":"https://github.com/GoogleCloudPlatform/gsutil/issues/241#issuecomment-68558677","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/gsutil/issues/241","id":68558677,"user":{"login":"thobrla","id":3239120,"avatar_url":"https://avatars.githubusercontent.com/u/3239120?v=3","gravatar_id":"","url":"https://api.github.com/users/thobrla","html_url":"https://github.com/thobrla","followers_url":"https://api.github.com/users/thobrla/followers","following_url":"https://api.github.com/users/thobrla/following{/other_user}","gists_url":"https://api.github.com/users/thobrla/gists{/gist_id}","starred_url":"https://api.github.com/users/thobrla/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thobrla/subscriptions","organizations_url":"https://api.github.com/users/thobrla/orgs","repos_url":"https://api.github.com/users/thobrla/repos","events_url":"https://api.github.com/users/thobrla/events{/privacy}","received_events_url":"https://api.github.com/users/thobrla/received_events","type":"User","site_admin":false},"created_at":"2015-01-02T20:20:22Z","updated_at":"2015-01-02T20:20:22Z","body":"It's unclear from the description whether:\r\n1. There were valid proxy settings in the .boto configuration file that were then cleared by resetting the BOTO_CONFIG variable\r\n2. The custom-patched version of gsutil that depot_tools had modifications that got the proxy information from the environment.\r\n\r\nRegardless, as far as gsutil goes, this is really a feature request for gsutil to support the \"http_proxy\" and \"https_proxy\" environment variables.\r\n\r\nAs mfschwartz noted in https://github.com/GoogleCloudPlatform/gsutil/pull/171, this causes problems with boto, so I don't think it's realistic to support it for the XML codepath.  However, we could support it for the JSON codepath, which is the default.  There are two main disadvantages that I can see.  First, it would mean divergent behavior in the XML and JSON codepaths.  Second, it would create a divergent configuration space where proxy settings could be configured either via .boto or the environment, which would make debugging proxy issues more difficult.\r\n\r\nGiven that this environment variable is honored by wget and curl, I'm inclined to implement this for the JSON codepath despite the disadvantages.\r\n\r\nAn alternative would be for download_from_google_storage to properly set the configuration in a temporary boto file or pass the proxy configuration down via top-level -o options to gsutil."}}