{"action":"created","issue":{"url":"https://api.github.com/repos/leo-project/leofs/issues/429","labels_url":"https://api.github.com/repos/leo-project/leofs/issues/429/labels{/name}","comments_url":"https://api.github.com/repos/leo-project/leofs/issues/429/comments","events_url":"https://api.github.com/repos/leo-project/leofs/issues/429/events","html_url":"https://github.com/leo-project/leofs/issues/429","id":113270653,"number":429,"title":"[S3-API][leo_gateway] Compatibility with Stream Wrapper in aws-sdk-php","user":{"login":"windkit","id":360884,"avatar_url":"https://avatars.githubusercontent.com/u/360884?v=3","gravatar_id":"","url":"https://api.github.com/users/windkit","html_url":"https://github.com/windkit","followers_url":"https://api.github.com/users/windkit/followers","following_url":"https://api.github.com/users/windkit/following{/other_user}","gists_url":"https://api.github.com/users/windkit/gists{/gist_id}","starred_url":"https://api.github.com/users/windkit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/windkit/subscriptions","organizations_url":"https://api.github.com/users/windkit/orgs","repos_url":"https://api.github.com/users/windkit/repos","events_url":"https://api.github.com/users/windkit/events{/privacy}","received_events_url":"https://api.github.com/users/windkit/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/leo-project/leofs/labels/Bug","name":"Bug","color":"fc2929"},{"url":"https://api.github.com/repos/leo-project/leofs/labels/Improve","name":"Improve","color":"FFA500"},{"url":"https://api.github.com/repos/leo-project/leofs/labels/S3-API","name":"S3-API","color":"32956C"},{"url":"https://api.github.com/repos/leo-project/leofs/labels/_leo_gateway","name":"_leo_gateway","color":"ededed"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/leo-project/leofs/milestones/50","html_url":"https://github.com/leo-project/leofs/milestones/1.2.18","labels_url":"https://api.github.com/repos/leo-project/leofs/milestones/50/labels","id":1358942,"number":50,"title":"1.2.18","description":"","creator":{"login":"mocchira","id":279052,"avatar_url":"https://avatars.githubusercontent.com/u/279052?v=3","gravatar_id":"","url":"https://api.github.com/users/mocchira","html_url":"https://github.com/mocchira","followers_url":"https://api.github.com/users/mocchira/followers","following_url":"https://api.github.com/users/mocchira/following{/other_user}","gists_url":"https://api.github.com/users/mocchira/gists{/gist_id}","starred_url":"https://api.github.com/users/mocchira/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mocchira/subscriptions","organizations_url":"https://api.github.com/users/mocchira/orgs","repos_url":"https://api.github.com/users/mocchira/repos","events_url":"https://api.github.com/users/mocchira/events{/privacy}","received_events_url":"https://api.github.com/users/mocchira/received_events","type":"User","site_admin":false},"open_issues":3,"closed_issues":2,"state":"open","created_at":"2015-10-16T08:05:21Z","updated_at":"2015-10-26T05:14:43Z","due_on":"2015-10-28T15:00:00Z","closed_at":null},"comments":5,"created_at":"2015-10-26T01:52:30Z","updated_at":"2015-10-27T05:09:28Z","closed_at":null,"body":"### Description\r\nIn `aws-sdk-php`, Stream Wrapper is provided to wrap S3 object as file stream\r\nhttp://docs.aws.amazon.com/aws-sdk-php/v2/guide/feature-s3-stream-wrapper.html\r\n\r\nThere are some issues when using the stream together with `http_send_stream` and `http_throttle` in `pecl_http`\r\n\r\n### Issue 1: `http_send_stream` only, zero byte is returned\r\n#### Test Code\r\n```php\r\n$s3client->registerStreamWrapper();\r\n$stream = fopen('s3://testp/test.large' ,'r');\r\nhttp_send_stream($stream);\r\n```\r\n#### Observation\r\n`http_send_stream` assumes the underlying is seekable, by default `aws-sdk-php` does not open a seekable stream with above code\r\n\r\n#### Solution\r\nOpen a stream with `seeakble` stream context\r\n```php\r\n$context = stream_context_create(array(\r\n        's3' => array('seekable' => true)\r\n                ));\r\n$s3client->registerStreamWrapper();\r\n$stream = fopen('s3://testp/test.large' ,'r', false, $context);\r\nhttp_send_stream($stream);\r\n```\r\n(Please note that, it will keep previously read bytes in memory with the `seekable` context)\r\n\r\n### Issue 2: `http_throttle` would cause timeout in `leo_tran` and then blocked\r\n#### Test Code\r\n```php\r\n$stream = fopen('s3://testp/test.large' ,'r', false, $context);\r\nhttp_throttle(0.1);\r\nhttp_send_stream($stream);\r\n```\r\n\r\n#### Observation\r\nWhen objects are sent with low speed, timeout would occur in various location.\r\n\r\n#### Error Log (1.2)\r\n```\r\n[E]\tgateway_0@127.0.0.1\t2015-10-27 13:37:16.371354 +0900\t1445920636\tleo_large_object_get_handler:handle_loop/3\t184\tkey:testp/test.large\t2, index:1, cause:timeout\r\n```\r\n#### Error Log (1.4)\r\n```\r\n[E]\tgateway_0@127.0.0.1\t2015-10-26 10:39:47.842387 +0900\t1445823587\tnull:null\t0\tgen_server <0.1482.0> terminated with reason: {timeout,{gen_server,call,[leo_tran_serializable_cntnr,{run,<<\"testp/test.large\">>,null,null,leo_large_object_get_handler,{10,{http_req,#Port<0.2133>,ranch_tcp,close,<0.1478.0>,<<\"GET\">>,'HTTP/1.1',{{127,0,0,1},35888},<<\"testp.localhost\">>,undefined,8080,<<\"/test.large\">>,undefined,<<>>,[],[],[{<<\"host\">>,<<\"testp.localhost:8080\">>},{<<\"user-agent\">>,<<\"aws-sdk-php2/2.8.14 Guzzle/3.9.3 curl/7.35.0 PHP/5.5.9-1ubuntu4.13\">>},{<<\"date\">>,<<\"Mon, 26 Oct 2015 01:39:17 +0000\">>},{<<\"authorization\">>,...},...],...},...},...},...]}} in gen_server:call/3 line 190\r\n[E]\tgateway_0@127.0.0.1\t2015-10-26 10:39:47.843176 +0900\t1445823587\tnull:null\t0\t[\"CRASH REPORT \",[80,114,111,99,101,115,115,32,\"<0.1482.0>\",32,119,105,116,104,32,\"1\",32,110,101,105,103,104,98,111,117,114,115,32,\"exited\",32,119,105,116,104,32,114,101,97,115,111,110,58,32,[[123,[\"timeout\",44,[123,[\"gen_server\",44,\"call\",44,[91,[\"leo_tran_serializable_cntnr\",44,[123,[\"run\",44,[\"<<\",\"\\\"testp/test.large\\\"\",\">>\"],44,\"null\",44,\"null\",44,\"leo_large_object_get_handler\",44,[123,[\"10\",44,[123,[\"http_req\",44,\"#Port<0.2133>\",44,\"ranch_tcp\",44,\"close\",44,\"<0.1478.0>\",44,[\"<<\",\"\\\"GET\\\"\",\">>\"],44,\"'HTTP/1.1'\",44,[123,[[123,[\"127\",44,\"0\",44,\"0\",44,\"1\"],125],44,\"35888\"],125],44,[\"<<\",\"\\\"testp.localhost\\\"\",\">>\"],44,\"undefined\",44,\"8080\",44,[\"<<\",\"\\\"/test.large\\\"\",\">>\"],44,\"undefined\",44,\"<<>>\",44,\"[]\",44,\"[]\",44,[91,[[123,[[\"<<\",\"\\\"host\\\"\",\">>\"],44,[\"<<\",\"\\\"testp.localhost:8080\\\"\",\">>\"]],125],44,[123,[[\"<<\",\"\\\"user-agent\\\"\",\">>\"],44,[\"<<\",\"\\\"aws-sdk-php2/2.8.14 Guzzle/3.9.3 curl/7.35.0 PHP/5.5.9-1ubuntu4.13\\\"\",\">>\"]],125],44,[123,[[\"<<\",\"\\\"date\\\"\",\">>\"],44,[\"<<\",\"\\\"Mon, 26 Oct 2015 01:39:17 +0000\\\"\",\">>\"]],125],44,[123,[[\"<<\",\"\\\"authorization\\\"\",\">>\"],44,46,46,46],125],44,46,46,46],93],44,46,46,46],125],44,46,46,46],125],44,46,46,46],125],44,46,46,46],93]],125]],125],\" in \",[[\"gen_server\",58,\"terminate\",47,\"7\"],[32,108,105,110,101,32,\"804\"]]]]]\r\n```"},"comment":{"url":"https://api.github.com/repos/leo-project/leofs/issues/comments/151377210","html_url":"https://github.com/leo-project/leofs/issues/429#issuecomment-151377210","issue_url":"https://api.github.com/repos/leo-project/leofs/issues/429","id":151377210,"user":{"login":"windkit","id":360884,"avatar_url":"https://avatars.githubusercontent.com/u/360884?v=3","gravatar_id":"","url":"https://api.github.com/users/windkit","html_url":"https://github.com/windkit","followers_url":"https://api.github.com/users/windkit/followers","following_url":"https://api.github.com/users/windkit/following{/other_user}","gists_url":"https://api.github.com/users/windkit/gists{/gist_id}","starred_url":"https://api.github.com/users/windkit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/windkit/subscriptions","organizations_url":"https://api.github.com/users/windkit/orgs","repos_url":"https://api.github.com/users/windkit/repos","events_url":"https://api.github.com/users/windkit/events{/privacy}","received_events_url":"https://api.github.com/users/windkit/received_events","type":"User","site_admin":false},"created_at":"2015-10-27T05:09:28Z","updated_at":"2015-10-27T05:09:28Z","body":"In short, the (implicit) lower bound of the transfer speed is\r\n\r\n- (1.2) chunked_obj_len / 30 seconds (5MB / 30s = 167Kbps)\r\n- (1.4) total object length / 30 seconds (50MB / 30s = 1.67Mbps)"}}