{"action":"created","issue":{"url":"https://api.github.com/repos/Unidata/thredds/issues/147","labels_url":"https://api.github.com/repos/Unidata/thredds/issues/147/labels{/name}","comments_url":"https://api.github.com/repos/Unidata/thredds/issues/147/comments","events_url":"https://api.github.com/repos/Unidata/thredds/issues/147/events","html_url":"https://github.com/Unidata/thredds/pull/147","id":96945099,"number":147,"title":"S3 datasets","user":{"login":"danfruehauf","id":830676,"avatar_url":"https://avatars.githubusercontent.com/u/830676?v=3","gravatar_id":"","url":"https://api.github.com/users/danfruehauf","html_url":"https://github.com/danfruehauf","followers_url":"https://api.github.com/users/danfruehauf/followers","following_url":"https://api.github.com/users/danfruehauf/following{/other_user}","gists_url":"https://api.github.com/users/danfruehauf/gists{/gist_id}","starred_url":"https://api.github.com/users/danfruehauf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danfruehauf/subscriptions","organizations_url":"https://api.github.com/users/danfruehauf/orgs","repos_url":"https://api.github.com/users/danfruehauf/repos","events_url":"https://api.github.com/users/danfruehauf/events{/privacy}","received_events_url":"https://api.github.com/users/danfruehauf/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":21,"created_at":"2015-07-24T02:13:08Z","updated_at":"2015-08-15T11:48:51Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/Unidata/thredds/pulls/147","html_url":"https://github.com/Unidata/thredds/pull/147","diff_url":"https://github.com/Unidata/thredds/pull/147.diff","patch_url":"https://github.com/Unidata/thredds/pull/147.patch"},"body":"This is a S3 crawlable dataset implementation, it is rather concise and provides thredds the capability of browsing S3 buckets. At the moment it'll cache files for up to 1 minute (hardcoded, but can be modified).\r\n\r\nA configuration for a dataset would look like:\r\n```\r\n  <datasetScan name=\"Amazon S3 Dataset\" ID=\"testS3\" path=\"IMOS\" location=\"s3://bucket-name/path\">\r\n    <metadata inherited=\"true\">\r\n      <serviceName>all</serviceName>\r\n      <dataType>Grid</dataType>\r\n    </metadata>\r\n    <addDatasetSize/>\r\n    <crawlableDatasetImpl className=\"thredds.crawlabledataset.CrawlableDatasetAmazonS3\"/>\r\n  </datasetScan>\r\n```\r\n\r\nAt IMOS, we are about to migrate large portions of data to S3. Thredds is one of our main front facing applications and we have decided to implement proper S3 support for it, rather than use one of the available (and experimental) filesystem interfaces to S3."},"comment":{"url":"https://api.github.com/repos/Unidata/thredds/issues/comments/131357677","html_url":"https://github.com/Unidata/thredds/pull/147#issuecomment-131357677","issue_url":"https://api.github.com/repos/Unidata/thredds/issues/147","id":131357677,"user":{"login":"danfruehauf","id":830676,"avatar_url":"https://avatars.githubusercontent.com/u/830676?v=3","gravatar_id":"","url":"https://api.github.com/users/danfruehauf","html_url":"https://github.com/danfruehauf","followers_url":"https://api.github.com/users/danfruehauf/followers","following_url":"https://api.github.com/users/danfruehauf/following{/other_user}","gists_url":"https://api.github.com/users/danfruehauf/gists{/gist_id}","starred_url":"https://api.github.com/users/danfruehauf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danfruehauf/subscriptions","organizations_url":"https://api.github.com/users/danfruehauf/orgs","repos_url":"https://api.github.com/users/danfruehauf/repos","events_url":"https://api.github.com/users/danfruehauf/events{/privacy}","received_events_url":"https://api.github.com/users/danfruehauf/received_events","type":"User","site_admin":false},"created_at":"2015-08-15T11:48:51Z","updated_at":"2015-08-15T11:48:51Z","body":"@cwardgar By default the S3 client will use HTTPS. I actually disable it explicitly in https://github.com/Unidata/thredds/pull/147/files#diff-4f74c3ba230ed262459960022533c90dR250\r\n\r\n`s3://bucket/object` is the standard way of referring to objects on s3 as far as I know. We cannot have a `http(s)://` type URI because we need some way of hinting this is actually a s3 URI, and not HTTP/S. And yes, this is the proper standard way as far as I am concerned.\r\n\r\nSorry about deleting the buckets, we now have our \"proper\" bucket which will become production, that is under `s3://imos-data/IMOS`, feel free to use that URI for testing. To browse the bucket using HTTP, you can go to http://imos-data.s3.amazonaws.com/index.html\r\n\r\nIn terms of test coverage, go for it. I am familiar with Spock, however you will probably be much faster than me in implementing the tests.\r\n\r\nLet me know if I can assist with anything else."}}