{"action":"created","issue":{"url":"https://api.github.com/repos/goamz/goamz/issues/81","labels_url":"https://api.github.com/repos/goamz/goamz/issues/81/labels{/name}","comments_url":"https://api.github.com/repos/goamz/goamz/issues/81/comments","events_url":"https://api.github.com/repos/goamz/goamz/issues/81/events","html_url":"https://github.com/goamz/goamz/issues/81","id":73400185,"number":81,"title":"Poor performance of downloading S3 file","user":{"login":"grotos","id":474662,"avatar_url":"https://avatars.githubusercontent.com/u/474662?v=3","gravatar_id":"","url":"https://api.github.com/users/grotos","html_url":"https://github.com/grotos","followers_url":"https://api.github.com/users/grotos/followers","following_url":"https://api.github.com/users/grotos/following{/other_user}","gists_url":"https://api.github.com/users/grotos/gists{/gist_id}","starred_url":"https://api.github.com/users/grotos/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/grotos/subscriptions","organizations_url":"https://api.github.com/users/grotos/orgs","repos_url":"https://api.github.com/users/grotos/repos","events_url":"https://api.github.com/users/grotos/events{/privacy}","received_events_url":"https://api.github.com/users/grotos/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-05-05T18:58:26Z","updated_at":"2015-05-07T18:52:36Z","closed_at":null,"body":"I tried downloading logs from my /logs in S3 - overall it was 419 files. \r\nHere are times:\r\n```\r\ngoamz-s3: 1m55.5454449s\r\npython boto: 35.65361350977789s\r\n```\r\n\r\nIt is a simple script:\r\n1. Login to S3\r\n2. List all files with prefix\r\n3. Download a file\r\n\r\nHere is go code:\r\n```\r\npackage main\r\n\r\nimport (\r\n\t\"bufio\"\r\n\t\"fmt\"\r\n\t\"gopkg.in/amz.v1/aws\"\r\n\t\"gopkg.in/amz.v1/s3\"\r\n\t\"io\"\r\n\t\"os\"\r\n\t\"time\"\r\n)\r\n\r\nfunc main() {\r\n\ttime1 := time.Now()\r\n\tAWSAuth := aws.Auth{\r\n\t\tAccessKey: \"\",\r\n\t\tSecretKey: \"\",\r\n\t}\r\n\r\n\tregion := aws.EUWest\r\n\tconnection := s3.New(AWSAuth, region)\r\n\tbucket := connection.Bucket(\"my-bucket\") // change this your bucket name\r\n\r\n\tresponse, _ := bucket.List(\"logs/access_log\", \"\", \"\", 1000)\r\n\r\n\tfor _, objects := range response.Contents {\r\n\t\tdownloadBytes, err := bucket.Get(objects.Key)\r\n\r\n\t\tif err != nil {\r\n\t\t\tfmt.Println(err)\r\n\t\t\tos.Exit(1)\r\n\t\t}\r\n\r\n\t\tdownloadFile, err := os.Create(objects.Key)\r\n\r\n\t\tif err != nil {\r\n\t\t\tfmt.Println(err)\r\n\t\t\tos.Exit(1)\r\n\t\t}\r\n\r\n\t\tdownloadBuffer := bufio.NewWriter(downloadFile)\r\n\t\tdownloadBuffer.Write(downloadBytes)\r\n\r\n\t\tio.Copy(downloadBuffer, downloadFile)\r\n\t\tdownloadFile.Close()\r\n\t\tfmt.Println(objects.Key, time.Since(time1))\r\n\t}\r\n\r\n}\r\n\r\n```\r\n\r\nWhy is go so slow comparing to python?\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/goamz/goamz/issues/comments/99977685","html_url":"https://github.com/goamz/goamz/issues/81#issuecomment-99977685","issue_url":"https://api.github.com/repos/goamz/goamz/issues/81","id":99977685,"user":{"login":"grotos","id":474662,"avatar_url":"https://avatars.githubusercontent.com/u/474662?v=3","gravatar_id":"","url":"https://api.github.com/users/grotos","html_url":"https://github.com/grotos","followers_url":"https://api.github.com/users/grotos/followers","following_url":"https://api.github.com/users/grotos/following{/other_user}","gists_url":"https://api.github.com/users/grotos/gists{/gist_id}","starred_url":"https://api.github.com/users/grotos/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/grotos/subscriptions","organizations_url":"https://api.github.com/users/grotos/orgs","repos_url":"https://api.github.com/users/grotos/repos","events_url":"https://api.github.com/users/grotos/events{/privacy}","received_events_url":"https://api.github.com/users/grotos/received_events","type":"User","site_admin":false},"created_at":"2015-05-07T18:52:36Z","updated_at":"2015-05-07T18:52:36Z","body":"I did profiling, here is the result: https://gist.github.com/grotos/f001cd2149630067426f\r\n\r\nStats: 163 files / 1.6MB / 25.5469062s (\"manual\", simple fmt.Println timing)\r\n\r\nWhy timing program in console output shows that code took 25s, whereas in profiling it is only 1.8s?"}}