{"action":"opened","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/gsutil/issues/310","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/gsutil/issues/310/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/gsutil/issues/310/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/gsutil/issues/310/events","html_url":"https://github.com/GoogleCloudPlatform/gsutil/issues/310","id":114696154,"number":310,"title":"Transfer invalid / hash mismatch error after oauth2client refresh on download","user":{"login":"thobrla","id":3239120,"avatar_url":"https://avatars.githubusercontent.com/u/3239120?v=3","gravatar_id":"","url":"https://api.github.com/users/thobrla","html_url":"https://github.com/thobrla","followers_url":"https://api.github.com/users/thobrla/followers","following_url":"https://api.github.com/users/thobrla/following{/other_user}","gists_url":"https://api.github.com/users/thobrla/gists{/gist_id}","starred_url":"https://api.github.com/users/thobrla/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thobrla/subscriptions","organizations_url":"https://api.github.com/users/thobrla/orgs","repos_url":"https://api.github.com/users/thobrla/repos","events_url":"https://api.github.com/users/thobrla/events{/privacy}","received_events_url":"https://api.github.com/users/thobrla/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-11-02T22:14:18Z","updated_at":"2015-11-02T22:14:18Z","closed_at":null,"body":"When downloading a file, bytes from an oauth2client refresh request are inappropriately considered as part of the request.\r\n\r\nThis problem occurs with downloads (via cp or rsync) that run over the duration of an access token's expiration (typically 1 hour as of this issue's posting).  It can also occur for sequential commands that use a cached access token stored in gsutil_state_dir (~/.gsutil/credstore by default).  When the access token expires, [gcs_json_media](https://github.com/GoogleCloudPlatform/gsutil/blob/963f3308d14d1a0727efcb4efd063ead09d14fa7/gslib/gcs_json_media.py#L507) erroneously considers the results of the POST as part of the downloaded bytes.\r\n\r\nThis will result in one of two errors:\r\n1. For small files, the oauth2client bytes will exceed the size of the file, resulting in: Service Exception: Transfer invalid cannot have start index greater than total size\r\n2. For larger files, the MD5 won't match (since the data incorrectly  md5 signature computed for local file (...) doesn't match cloud-supplied digest\r\n\r\nThis bug was likely introduced with [this commit](https://github.com/GoogleCloudPlatform/gsutil/commit/963f3308d14d1a0727efcb4efd063ead09d14fa7) in gsutil 4.15.  It occurs for user or service account credentials, but not for GCE credentials which use a different refresh path.\r\n\r\nThere is no sure-fire workaround for downloads that run over an hour.  For sequential commands running less than an hour, removing ~/.gsutil/credstore between commands should prevent this issue."}}