{"action":"created","issue":{"url":"https://api.github.com/repos/snowplow/snowplow/issues/1779","labels_url":"https://api.github.com/repos/snowplow/snowplow/issues/1779/labels{/name}","comments_url":"https://api.github.com/repos/snowplow/snowplow/issues/1779/comments","events_url":"https://api.github.com/repos/snowplow/snowplow/issues/1779/events","html_url":"https://github.com/snowplow/snowplow/issues/1779","id":88545653,"number":1779,"title":"Azure support","user":{"login":"thomaswrenn","id":1128624,"avatar_url":"https://avatars.githubusercontent.com/u/1128624?v=3","gravatar_id":"","url":"https://api.github.com/users/thomaswrenn","html_url":"https://github.com/thomaswrenn","followers_url":"https://api.github.com/users/thomaswrenn/followers","following_url":"https://api.github.com/users/thomaswrenn/following{/other_user}","gists_url":"https://api.github.com/users/thomaswrenn/gists{/gist_id}","starred_url":"https://api.github.com/users/thomaswrenn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thomaswrenn/subscriptions","organizations_url":"https://api.github.com/users/thomaswrenn/orgs","repos_url":"https://api.github.com/users/thomaswrenn/repos","events_url":"https://api.github.com/users/thomaswrenn/events{/privacy}","received_events_url":"https://api.github.com/users/thomaswrenn/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"alexanderdean","id":200511,"avatar_url":"https://avatars.githubusercontent.com/u/200511?v=3","gravatar_id":"","url":"https://api.github.com/users/alexanderdean","html_url":"https://github.com/alexanderdean","followers_url":"https://api.github.com/users/alexanderdean/followers","following_url":"https://api.github.com/users/alexanderdean/following{/other_user}","gists_url":"https://api.github.com/users/alexanderdean/gists{/gist_id}","starred_url":"https://api.github.com/users/alexanderdean/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alexanderdean/subscriptions","organizations_url":"https://api.github.com/users/alexanderdean/orgs","repos_url":"https://api.github.com/users/alexanderdean/repos","events_url":"https://api.github.com/users/alexanderdean/events{/privacy}","received_events_url":"https://api.github.com/users/alexanderdean/received_events","type":"User","site_admin":false},"milestone":null,"comments":12,"created_at":"2015-06-15T21:28:44Z","updated_at":"2015-06-18T19:02:32Z","closed_at":null,"body":"Can someone implement support for running Snowplow on Azure?"},"comment":{"url":"https://api.github.com/repos/snowplow/snowplow/issues/comments/113258883","html_url":"https://github.com/snowplow/snowplow/issues/1779#issuecomment-113258883","issue_url":"https://api.github.com/repos/snowplow/snowplow/issues/1779","id":113258883,"user":{"login":"alexanderdean","id":200511,"avatar_url":"https://avatars.githubusercontent.com/u/200511?v=3","gravatar_id":"","url":"https://api.github.com/users/alexanderdean","html_url":"https://github.com/alexanderdean","followers_url":"https://api.github.com/users/alexanderdean/followers","following_url":"https://api.github.com/users/alexanderdean/following{/other_user}","gists_url":"https://api.github.com/users/alexanderdean/gists{/gist_id}","starred_url":"https://api.github.com/users/alexanderdean/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alexanderdean/subscriptions","organizations_url":"https://api.github.com/users/alexanderdean/orgs","repos_url":"https://api.github.com/users/alexanderdean/repos","events_url":"https://api.github.com/users/alexanderdean/events{/privacy}","received_events_url":"https://api.github.com/users/alexanderdean/received_events","type":"User","site_admin":false},"created_at":"2015-06-18T19:02:32Z","updated_at":"2015-06-18T19:02:32Z","body":"Hey @jpoon, @thomaswrenn  ! No worries at all. Answers to your questions/comments:\r\n\r\n> Have you found that a lot of people persist the raw events from Kinesis straight to S3?\r\n\r\nYes - at the moment we don't do loading of Redshift from Kinesis (only from Hadoop), so currently we recommend people sink the raw events to S3 and then run our Hadoop pipeline to load into Redshift as per [our latest release blog post](http://snowplowanalytics.com/blog/2015/06/16/snowplow-r66-oriental-skylark-released/#lambda)\r\n\r\nIn any case - yes archiving raw events is important to facilitate re-processing later; I explore this in [chapter 9](http://www.manning.com/dean/) in my book.\r\n\r\n> What is the difference between Amazon's version vs snowplow?\r\n\r\nAmazon's version is more of a toolkit (albeit with a sample application) - kinesis-s3 is a pre-built app which just needs a configuration file. Also, kinesis-s3 writes lzo-index files which are very Hadoop-friendly.\r\n\r\n> HDInsight. HDInsight has native support for Azure Blob Storage (ie. no need to move it to an explicit HDFS store).\r\n\r\nAh okay. EMR has native support for S3 too - but moving to the cluster's HDFS with S3DistCp first makes processing much faster. Is that not the case with HDInsight?\r\n\r\n> Otherwise, as HDInsight deploys the Hortonworks Hadoop stack there is Ambari and WebHCat out of the box.\r\n\r\nRight - and now that the Snowplow Hadoop job runs on Hadoop 2.4.0 / YARN, we could probably submit the job using the [YARN Client](https://hadoop.apache.org/docs/r2.4.1/api/org/apache/hadoop/yarn/client/api/YarnClient.html)?\r\n\r\n> quickest path to supporting Azure ... it might mean adding support for kafka or event hub and then using the persisting that data to PostgreSQL?\r\n\r\nMy gut says:\r\n\r\n1. Scala Stream Collector with EventHub support via eventhubs-client\r\n2. EventHub -> Azure Blob Storage sink app\r\n3. Snowplow CLI app can submit Hadoop job to vanilla YARN cluster (#1792)\r\n4. We extend our Redshift loading engine to also work with SQL Data Warehouse\r\n\r\nThis is the minimum number of moving parts which is still very horizontally scalable.\r\n\r\nPostgres would be less work but it would be much less impactful - because nobody with serious event volumes can use Postgres with Snowplow. Also a powerful part of Snowplow is the custom shredding process which loads our event JSONs into dedicated tables in Redshift..."}}