{"action":"created","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/gsutil/issues/249","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/gsutil/issues/249/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/gsutil/issues/249/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/gsutil/issues/249/events","html_url":"https://github.com/GoogleCloudPlatform/gsutil/issues/249","id":56293265,"number":249,"title":"1. Uploading... 2. Failed... 3. File nonetheless uploaded","user":{"login":"karalabe","id":129561,"avatar_url":"https://avatars.githubusercontent.com/u/129561?v=3","gravatar_id":"","url":"https://api.github.com/users/karalabe","html_url":"https://github.com/karalabe","followers_url":"https://api.github.com/users/karalabe/followers","following_url":"https://api.github.com/users/karalabe/following{/other_user}","gists_url":"https://api.github.com/users/karalabe/gists{/gist_id}","starred_url":"https://api.github.com/users/karalabe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/karalabe/subscriptions","organizations_url":"https://api.github.com/users/karalabe/orgs","repos_url":"https://api.github.com/users/karalabe/repos","events_url":"https://api.github.com/users/karalabe/events{/privacy}","received_events_url":"https://api.github.com/users/karalabe/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-02-02T21:07:53Z","updated_at":"2015-02-04T23:19:22Z","closed_at":null,"body":"Hi all,\r\n\r\nI'm using gsutil to upload some files to GCS and I'm hitting an annoying bug. [Background: since the official Go client libraries to access Google Storage are currently buggy beyond use, I've created a small wrapper around gsutil that simply streams data into its stdin, and delegates uploading to it]. The wrapper works beautifully, I've managed to successfully (and consistently) upload files of around 30GB.\r\n\r\nToday however I've had a slightly different use case, where I've started an upload, but only sent data trickling through stdin (a data processing program is churning out the results slowly and sending it to gsutil to upload to GCS). The phenomenon I see is that after finishing the long running processing task (~1h) and sending a few hundred kilobytes go gsutil, I close stdin, and gsutil starts uploading the data:\r\n\r\n```\r\nCopying from <STDIN>...\r\nUploading   ...: 65.8 KiB    \r\nUploading   ...: 137.8 KiB    \r\nUploading   ...: 209.8 KiB    \r\nUploading   ...: 64.6 KiB    \r\nUploading   ...: 136.6 KiB    \r\nUploading   ...: 208.6 KiB    \r\nFailure: must be string or buffer, not None.\r\n```\r\n\r\nNow, there is one odd thing and one annoying thing:\r\n\r\n - Looking at the above output, I see that the upload is restarted. I'm guessing that gsutil fetched an OAuth token when the stream started, but by the time I've finished it expired, so the first upload trial failed and it had to try again (just a hunch, I can live with this).\r\n - gsutil then fails, returning both the above error message, as well as terminating with a non zero status code. My code considers the upload failed, however, looking at the bucket contents, the upload actually succeeded.\r\n\r\nSo in essence, gsutil reports failure on all possible channels, yet succeeds the operation. This is really annoying because now I have actually no means to know whether the op succeeded or not. Could it be that the failure belongs to the first trial and then gsutil actually succeeded? But then why report the failure?\r\n\r\nIdeas? :)\r\n\r\nThanks,\r\n  Peter\r\n\r\nPS: Note, if I'm streaming massive amount of data, then even very long running uploads (4-6 hours) succeed ok, yet if I only trickle data in a longer time window, then everything blows up.\r\n\r\nPPS: gsutil version: 4.7, latest according to the gcloud updater"},"comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/gsutil/issues/comments/72961683","html_url":"https://github.com/GoogleCloudPlatform/gsutil/issues/249#issuecomment-72961683","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/gsutil/issues/249","id":72961683,"user":{"login":"thobrla","id":3239120,"avatar_url":"https://avatars.githubusercontent.com/u/3239120?v=3","gravatar_id":"","url":"https://api.github.com/users/thobrla","html_url":"https://github.com/thobrla","followers_url":"https://api.github.com/users/thobrla/followers","following_url":"https://api.github.com/users/thobrla/following{/other_user}","gists_url":"https://api.github.com/users/thobrla/gists{/gist_id}","starred_url":"https://api.github.com/users/thobrla/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thobrla/subscriptions","organizations_url":"https://api.github.com/users/thobrla/orgs","repos_url":"https://api.github.com/users/thobrla/repos","events_url":"https://api.github.com/users/thobrla/events{/privacy}","received_events_url":"https://api.github.com/users/thobrla/received_events","type":"User","site_admin":false},"created_at":"2015-02-04T23:19:22Z","updated_at":"2015-02-04T23:19:22Z","body":"This is fixed for 4.9 with https://github.com/GoogleCloudPlatform/gsutil/commit/912e52c6031b1bf8fe62dda220d90d8aacb19745\r\n\r\nIf you'd like to pick up the fix sooner, you can run the following:\r\n\r\nFor gcloud installs:\r\n  `gcloud components repositories add https://storage.googleapis.com/prerelease/cloudsdk-prerelease-components/gsutil_4.9pre_trickleupload/components-2.json`\r\n  `gcloud components update`\r\n\r\nFor standalone installs:\r\n  `gsutil update gs://prerelease/gsutil_4.9pre_trickleupload.tar.gz`\r\n"}}