{"action":"created","issue":{"url":"https://api.github.com/repos/databricks/spark-redshift/issues/22","labels_url":"https://api.github.com/repos/databricks/spark-redshift/issues/22/labels{/name}","comments_url":"https://api.github.com/repos/databricks/spark-redshift/issues/22/comments","events_url":"https://api.github.com/repos/databricks/spark-redshift/issues/22/events","html_url":"https://github.com/databricks/spark-redshift/pull/22","id":95168147,"number":22,"title":"[WIP] Data Source API and JDBC/S3 read/write","user":{"login":"jaley","id":343449,"avatar_url":"https://avatars.githubusercontent.com/u/343449?v=3","gravatar_id":"","url":"https://api.github.com/users/jaley","html_url":"https://github.com/jaley","followers_url":"https://api.github.com/users/jaley/followers","following_url":"https://api.github.com/users/jaley/following{/other_user}","gists_url":"https://api.github.com/users/jaley/gists{/gist_id}","starred_url":"https://api.github.com/users/jaley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jaley/subscriptions","organizations_url":"https://api.github.com/users/jaley/orgs","repos_url":"https://api.github.com/users/jaley/repos","events_url":"https://api.github.com/users/jaley/events{/privacy}","received_events_url":"https://api.github.com/users/jaley/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-07-15T11:18:06Z","updated_at":"2015-07-17T17:37:22Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/databricks/spark-redshift/pulls/22","html_url":"https://github.com/databricks/spark-redshift/pull/22","diff_url":"https://github.com/databricks/spark-redshift/pull/22.diff","patch_url":"https://github.com/databricks/spark-redshift/pull/22.patch"},"body":"This is unfinished, I'll be updating the PR today and tomorrow with tests and documentation, but it's basically functionally complete and manual testing with Redshift directly suggests it pretty much works, so I thought it'd be a good time to start gathering feedback.\r\n\r\nIncluded in this PR:\r\n\r\n* Data Source API implementation, including filtering/pruned traits, insertable, create, etc. `DefaultSource` for integration with Spark SQL too.\r\n* Read and write support, using S3 for data transfer and JDBC to initiate loads, unloads, etc.\r\n* Makes use of the new Avro support in Redshift (requires Redshift version 1.0.963+) for writes. Redshift currently doesn't support Avro for its `UNLOAD` command.\r\n* Some useful Redshift functionality for setting distribution and sort keys, plus some good ETL practices such as staging tables etc.\r\n\r\nIssue #8 has a thread with a few limitations and challenges I'm still looking for better solutions to. Please feel free to take this for a spin and make suggestions here!\r\n"},"comment":{"url":"https://api.github.com/repos/databricks/spark-redshift/issues/comments/122350454","html_url":"https://github.com/databricks/spark-redshift/pull/22#issuecomment-122350454","issue_url":"https://api.github.com/repos/databricks/spark-redshift/issues/22","id":122350454,"user":{"login":"jaley","id":343449,"avatar_url":"https://avatars.githubusercontent.com/u/343449?v=3","gravatar_id":"","url":"https://api.github.com/users/jaley","html_url":"https://github.com/jaley","followers_url":"https://api.github.com/users/jaley/followers","following_url":"https://api.github.com/users/jaley/following{/other_user}","gists_url":"https://api.github.com/users/jaley/gists{/gist_id}","starred_url":"https://api.github.com/users/jaley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jaley/subscriptions","organizations_url":"https://api.github.com/users/jaley/orgs","repos_url":"https://api.github.com/users/jaley/repos","events_url":"https://api.github.com/users/jaley/events{/privacy}","received_events_url":"https://api.github.com/users/jaley/received_events","type":"User","site_admin":false},"created_at":"2015-07-17T17:37:22Z","updated_at":"2015-07-17T17:37:22Z","body":"Hey @mengxr - I'm having some Travis drama here. This builds and tests fine for me, using same Scala, Spark and Hadoop versions, but Travis always fails with this shutdown hook priority exception (stack trace is in the Travis logs). Any suggestions? I've no idea what's up with it.\r\n\r\nTest coverage is up at 82% now, I think a few more cases should get us there. I'll add docs on Monday."}}