{"action":"created","comment":{"url":"https://api.github.com/repos/smashwilson/azurefire/pulls/comments/23869790","id":23869790,"diff_hunk":"@@ -0,0 +1,103 @@\n+---\n+layout: post\n+title: \"Put that in your cloud and smoke it\"\n+date: \"2015-01-30\"\n+category: news\n+tags:\n+- software\n+- go\n+- cloudpipe\n+image:\n+  feature: texture-lakemead-wires.jpg\n+---\n+\n+When you want to execute a long-running or massively parallel process in a persistent way, you have a few options. You can launch a server, install a bunch of software, and stitch things together with `cron`. Or, you can install a [Hadoop](http://hadoop.apache.org/) cluster, rewrite everything from the ground up to use map-reduce or Spark or another powerful concurrency framework, and orchestrate your job with that toolkit.\n+\n+Those options all have their place, but sometimes, all you want to do is *write some code* and *run it somewhere*. No server spin-up, no system administration, no patches and security updates. [Kyle Kelley](http://lambdaops.com/) and I spent a few weeks around the holidays putting together [an open-source project called **\"Cloudpipe\"**](https://github.com/cloudpipe/cloudpipe) to accomplish exactly this. Today, we have a proof of concept we can run that successfully executes jobs, and we're hard at work filling in the blanks in the architecture and putting together a real deployment.\n+\n+## Using it\n+\n+To start, we chose to target an existing client SDK, Python's [multyvac](https://pypi.python.org/pypi/multyvac/0.5.0), which was nice for a few reasons:\n+\n+ 1. It gave me a concrete spec to target while I was mapping out the endpoints for the API server. This saved us a lot of bikeshedding in the early days.\n+ 2. It let me start backend-first and test things with real code, just as a user would experience them. Otherwise, I would have had to use awkward `curl` scripts until we could design and put together a decent SDK.\n+ 3. In theory, people are [already using](https://github.com/search?utf8=%E2%9C%93&q=import+multyvac&type=Code&ref=searchresults) the SDK. (If that search is any indication, though, in retrospect we should have stuck with its predecessor, [cloud](https://github.com/search?utf8=%E2%9C%93&q=import+cloud&type=Code&ref=searchresults), instead.)\n+\n+As a result, the only client SDK that exists today is for Python (and Python 2, at that). That's going to change, though - we've been planning to [write our own client for Python](https://github.com/cloudpipe/cloudpipe/issues/17) and [Shaunak Kashyap](https://github.com/ycombinator) is already working on [an SDK for node.js](https://github.com/cloudpipe/sdk-node). We're also likely to diverge more and more from multyvac as we move on. Stay tuned for more on that front.\n+\n+Once you have an account running on a cluster somewhere, you can install multyvac with:\n+\n+```bash\n+pip install multyvac\n+```\n+\n+Then, you can submit functions as jobs:\n+\n+```python\n+import multyvac\n+\n+# Connect to the cluster\n+multyvac.config.set_key(api_key='username',\n+                        api_secret_key='54321',\n+                        api_url='https://somewhere/v1')\n+\n+def add(a, b):\n+  return a + b\n+\n+# Submit the function as a job. Args and kwargs will be\n+# passed to it server-side.\n+jid = multyvac.submit(add, 3, 4)\n+\n+# Use the job ID to get a Job object...\n+job = multyvac.get(jid)\n+\n+# ... and use the Job object to get the result.\n+# This call will block until the result is ready.\n+result = job.get_result()\n+```\n+\n+Multyvac uses some neat pickling tricks to send your function over the wire: you can actually send closures, or code that references external libraries, and it'll work. Other languages' SDKs may not be able to pull the same range of tricks, depending on how evil each runtime lets us be, but they'll support the same basic idea.\n+\n+One important thing to notice is that the way work is submitted to the API is completely language-agnostic: it's a contract between the SDK and the base Docker image that's invoked on the server. The Cloudpipe API speaks strictly in terms of POSIX things: command and arguments, environment variables, and standard input, output and error streams.\n+\n+## How it works\n+\n+Behind that URL, Cloudpipe is built from a small set of parts, all written in Go.\n+\n+*[cloudpipe/cloudpipe](https://github.com/cloudpipe/cloudpipe)*\n+\n+The **API server** is an HTTPS server that hosts a JSON API. It's a fairly thin layer that performs  CRUD operations on job models persisted in a MongoDB instance.","path":"_posts/2015-01-30-cloudpipe.markdown","position":null,"original_position":69,"commit_id":"ce24d08fd4610072b979f66c1cd4143b9512aab2","original_commit_id":"85701cc04e9b6fe87f70daaa17e659d3f9135687","user":{"login":"smashwilson","id":17565,"avatar_url":"https://avatars.githubusercontent.com/u/17565?v=3","gravatar_id":"","url":"https://api.github.com/users/smashwilson","html_url":"https://github.com/smashwilson","followers_url":"https://api.github.com/users/smashwilson/followers","following_url":"https://api.github.com/users/smashwilson/following{/other_user}","gists_url":"https://api.github.com/users/smashwilson/gists{/gist_id}","starred_url":"https://api.github.com/users/smashwilson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smashwilson/subscriptions","organizations_url":"https://api.github.com/users/smashwilson/orgs","repos_url":"https://api.github.com/users/smashwilson/repos","events_url":"https://api.github.com/users/smashwilson/events{/privacy}","received_events_url":"https://api.github.com/users/smashwilson/received_events","type":"User","site_admin":false},"body":"I added a note to that effect, why not.","created_at":"2015-01-30T20:23:52Z","updated_at":"2015-01-30T20:23:52Z","html_url":"https://github.com/smashwilson/azurefire/pull/22#discussion_r23869790","pull_request_url":"https://api.github.com/repos/smashwilson/azurefire/pulls/22","_links":{"self":{"href":"https://api.github.com/repos/smashwilson/azurefire/pulls/comments/23869790"},"html":{"href":"https://github.com/smashwilson/azurefire/pull/22#discussion_r23869790"},"pull_request":{"href":"https://api.github.com/repos/smashwilson/azurefire/pulls/22"}}},"pull_request":{"url":"https://api.github.com/repos/smashwilson/azurefire/pulls/22","id":28386724,"html_url":"https://github.com/smashwilson/azurefire/pull/22","diff_url":"https://github.com/smashwilson/azurefire/pull/22.diff","patch_url":"https://github.com/smashwilson/azurefire/pull/22.patch","issue_url":"https://api.github.com/repos/smashwilson/azurefire/issues/22","number":22,"state":"open","locked":false,"title":"A blog post for today.","user":{"login":"smashwilson","id":17565,"avatar_url":"https://avatars.githubusercontent.com/u/17565?v=3","gravatar_id":"","url":"https://api.github.com/users/smashwilson","html_url":"https://github.com/smashwilson","followers_url":"https://api.github.com/users/smashwilson/followers","following_url":"https://api.github.com/users/smashwilson/following{/other_user}","gists_url":"https://api.github.com/users/smashwilson/gists{/gist_id}","starred_url":"https://api.github.com/users/smashwilson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smashwilson/subscriptions","organizations_url":"https://api.github.com/users/smashwilson/orgs","repos_url":"https://api.github.com/users/smashwilson/repos","events_url":"https://api.github.com/users/smashwilson/events{/privacy}","received_events_url":"https://api.github.com/users/smashwilson/received_events","type":"User","site_admin":false},"body":"@rgbkrk, @ycombinator: Read this over and let me know what you think!\r\n\r\nI'll be working on that architecture diagram.","created_at":"2015-01-30T19:33:26Z","updated_at":"2015-01-30T20:23:52Z","closed_at":null,"merged_at":null,"merge_commit_sha":"408aadc0020bb090fbd0b6d2b7e04572086c1588","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/smashwilson/azurefire/pulls/22/commits","review_comments_url":"https://api.github.com/repos/smashwilson/azurefire/pulls/22/comments","review_comment_url":"https://api.github.com/repos/smashwilson/azurefire/pulls/comments/{number}","comments_url":"https://api.github.com/repos/smashwilson/azurefire/issues/22/comments","statuses_url":"https://api.github.com/repos/smashwilson/azurefire/statuses/ce24d08fd4610072b979f66c1cd4143b9512aab2","head":{"label":"smashwilson:cloudpipe-post","ref":"cloudpipe-post","sha":"ce24d08fd4610072b979f66c1cd4143b9512aab2","user":{"login":"smashwilson","id":17565,"avatar_url":"https://avatars.githubusercontent.com/u/17565?v=3","gravatar_id":"","url":"https://api.github.com/users/smashwilson","html_url":"https://github.com/smashwilson","followers_url":"https://api.github.com/users/smashwilson/followers","following_url":"https://api.github.com/users/smashwilson/following{/other_user}","gists_url":"https://api.github.com/users/smashwilson/gists{/gist_id}","starred_url":"https://api.github.com/users/smashwilson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smashwilson/subscriptions","organizations_url":"https://api.github.com/users/smashwilson/orgs","repos_url":"https://api.github.com/users/smashwilson/repos","events_url":"https://api.github.com/users/smashwilson/events{/privacy}","received_events_url":"https://api.github.com/users/smashwilson/received_events","type":"User","site_admin":false},"repo":{"id":12512160,"name":"azurefire","full_name":"smashwilson/azurefire","owner":{"login":"smashwilson","id":17565,"avatar_url":"https://avatars.githubusercontent.com/u/17565?v=3","gravatar_id":"","url":"https://api.github.com/users/smashwilson","html_url":"https://github.com/smashwilson","followers_url":"https://api.github.com/users/smashwilson/followers","following_url":"https://api.github.com/users/smashwilson/following{/other_user}","gists_url":"https://api.github.com/users/smashwilson/gists{/gist_id}","starred_url":"https://api.github.com/users/smashwilson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smashwilson/subscriptions","organizations_url":"https://api.github.com/users/smashwilson/orgs","repos_url":"https://api.github.com/users/smashwilson/repos","events_url":"https://api.github.com/users/smashwilson/events{/privacy}","received_events_url":"https://api.github.com/users/smashwilson/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/smashwilson/azurefire","description":"Source code for azurefire.net","fork":false,"url":"https://api.github.com/repos/smashwilson/azurefire","forks_url":"https://api.github.com/repos/smashwilson/azurefire/forks","keys_url":"https://api.github.com/repos/smashwilson/azurefire/keys{/key_id}","collaborators_url":"https://api.github.com/repos/smashwilson/azurefire/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/smashwilson/azurefire/teams","hooks_url":"https://api.github.com/repos/smashwilson/azurefire/hooks","issue_events_url":"https://api.github.com/repos/smashwilson/azurefire/issues/events{/number}","events_url":"https://api.github.com/repos/smashwilson/azurefire/events","assignees_url":"https://api.github.com/repos/smashwilson/azurefire/assignees{/user}","branches_url":"https://api.github.com/repos/smashwilson/azurefire/branches{/branch}","tags_url":"https://api.github.com/repos/smashwilson/azurefire/tags","blobs_url":"https://api.github.com/repos/smashwilson/azurefire/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/smashwilson/azurefire/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/smashwilson/azurefire/git/refs{/sha}","trees_url":"https://api.github.com/repos/smashwilson/azurefire/git/trees{/sha}","statuses_url":"https://api.github.com/repos/smashwilson/azurefire/statuses/{sha}","languages_url":"https://api.github.com/repos/smashwilson/azurefire/languages","stargazers_url":"https://api.github.com/repos/smashwilson/azurefire/stargazers","contributors_url":"https://api.github.com/repos/smashwilson/azurefire/contributors","subscribers_url":"https://api.github.com/repos/smashwilson/azurefire/subscribers","subscription_url":"https://api.github.com/repos/smashwilson/azurefire/subscription","commits_url":"https://api.github.com/repos/smashwilson/azurefire/commits{/sha}","git_commits_url":"https://api.github.com/repos/smashwilson/azurefire/git/commits{/sha}","comments_url":"https://api.github.com/repos/smashwilson/azurefire/comments{/number}","issue_comment_url":"https://api.github.com/repos/smashwilson/azurefire/issues/comments/{number}","contents_url":"https://api.github.com/repos/smashwilson/azurefire/contents/{+path}","compare_url":"https://api.github.com/repos/smashwilson/azurefire/compare/{base}...{head}","merges_url":"https://api.github.com/repos/smashwilson/azurefire/merges","archive_url":"https://api.github.com/repos/smashwilson/azurefire/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/smashwilson/azurefire/downloads","issues_url":"https://api.github.com/repos/smashwilson/azurefire/issues{/number}","pulls_url":"https://api.github.com/repos/smashwilson/azurefire/pulls{/number}","milestones_url":"https://api.github.com/repos/smashwilson/azurefire/milestones{/number}","notifications_url":"https://api.github.com/repos/smashwilson/azurefire/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/smashwilson/azurefire/labels{/name}","releases_url":"https://api.github.com/repos/smashwilson/azurefire/releases{/id}","created_at":"2013-08-31T21:52:33Z","updated_at":"2014-06-27T15:39:34Z","pushed_at":"2015-01-30T20:22:30Z","git_url":"git://github.com/smashwilson/azurefire.git","ssh_url":"git@github.com:smashwilson/azurefire.git","clone_url":"https://github.com/smashwilson/azurefire.git","svn_url":"https://github.com/smashwilson/azurefire","homepage":null,"size":4368,"stargazers_count":1,"watchers_count":1,"language":"CSS","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":2,"forks":0,"open_issues":2,"watchers":1,"default_branch":"master"}},"base":{"label":"smashwilson:master","ref":"master","sha":"666ee601d600293529804ebf924355eefcf6a778","user":{"login":"smashwilson","id":17565,"avatar_url":"https://avatars.githubusercontent.com/u/17565?v=3","gravatar_id":"","url":"https://api.github.com/users/smashwilson","html_url":"https://github.com/smashwilson","followers_url":"https://api.github.com/users/smashwilson/followers","following_url":"https://api.github.com/users/smashwilson/following{/other_user}","gists_url":"https://api.github.com/users/smashwilson/gists{/gist_id}","starred_url":"https://api.github.com/users/smashwilson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smashwilson/subscriptions","organizations_url":"https://api.github.com/users/smashwilson/orgs","repos_url":"https://api.github.com/users/smashwilson/repos","events_url":"https://api.github.com/users/smashwilson/events{/privacy}","received_events_url":"https://api.github.com/users/smashwilson/received_events","type":"User","site_admin":false},"repo":{"id":12512160,"name":"azurefire","full_name":"smashwilson/azurefire","owner":{"login":"smashwilson","id":17565,"avatar_url":"https://avatars.githubusercontent.com/u/17565?v=3","gravatar_id":"","url":"https://api.github.com/users/smashwilson","html_url":"https://github.com/smashwilson","followers_url":"https://api.github.com/users/smashwilson/followers","following_url":"https://api.github.com/users/smashwilson/following{/other_user}","gists_url":"https://api.github.com/users/smashwilson/gists{/gist_id}","starred_url":"https://api.github.com/users/smashwilson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smashwilson/subscriptions","organizations_url":"https://api.github.com/users/smashwilson/orgs","repos_url":"https://api.github.com/users/smashwilson/repos","events_url":"https://api.github.com/users/smashwilson/events{/privacy}","received_events_url":"https://api.github.com/users/smashwilson/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/smashwilson/azurefire","description":"Source code for azurefire.net","fork":false,"url":"https://api.github.com/repos/smashwilson/azurefire","forks_url":"https://api.github.com/repos/smashwilson/azurefire/forks","keys_url":"https://api.github.com/repos/smashwilson/azurefire/keys{/key_id}","collaborators_url":"https://api.github.com/repos/smashwilson/azurefire/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/smashwilson/azurefire/teams","hooks_url":"https://api.github.com/repos/smashwilson/azurefire/hooks","issue_events_url":"https://api.github.com/repos/smashwilson/azurefire/issues/events{/number}","events_url":"https://api.github.com/repos/smashwilson/azurefire/events","assignees_url":"https://api.github.com/repos/smashwilson/azurefire/assignees{/user}","branches_url":"https://api.github.com/repos/smashwilson/azurefire/branches{/branch}","tags_url":"https://api.github.com/repos/smashwilson/azurefire/tags","blobs_url":"https://api.github.com/repos/smashwilson/azurefire/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/smashwilson/azurefire/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/smashwilson/azurefire/git/refs{/sha}","trees_url":"https://api.github.com/repos/smashwilson/azurefire/git/trees{/sha}","statuses_url":"https://api.github.com/repos/smashwilson/azurefire/statuses/{sha}","languages_url":"https://api.github.com/repos/smashwilson/azurefire/languages","stargazers_url":"https://api.github.com/repos/smashwilson/azurefire/stargazers","contributors_url":"https://api.github.com/repos/smashwilson/azurefire/contributors","subscribers_url":"https://api.github.com/repos/smashwilson/azurefire/subscribers","subscription_url":"https://api.github.com/repos/smashwilson/azurefire/subscription","commits_url":"https://api.github.com/repos/smashwilson/azurefire/commits{/sha}","git_commits_url":"https://api.github.com/repos/smashwilson/azurefire/git/commits{/sha}","comments_url":"https://api.github.com/repos/smashwilson/azurefire/comments{/number}","issue_comment_url":"https://api.github.com/repos/smashwilson/azurefire/issues/comments/{number}","contents_url":"https://api.github.com/repos/smashwilson/azurefire/contents/{+path}","compare_url":"https://api.github.com/repos/smashwilson/azurefire/compare/{base}...{head}","merges_url":"https://api.github.com/repos/smashwilson/azurefire/merges","archive_url":"https://api.github.com/repos/smashwilson/azurefire/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/smashwilson/azurefire/downloads","issues_url":"https://api.github.com/repos/smashwilson/azurefire/issues{/number}","pulls_url":"https://api.github.com/repos/smashwilson/azurefire/pulls{/number}","milestones_url":"https://api.github.com/repos/smashwilson/azurefire/milestones{/number}","notifications_url":"https://api.github.com/repos/smashwilson/azurefire/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/smashwilson/azurefire/labels{/name}","releases_url":"https://api.github.com/repos/smashwilson/azurefire/releases{/id}","created_at":"2013-08-31T21:52:33Z","updated_at":"2014-06-27T15:39:34Z","pushed_at":"2015-01-30T20:22:30Z","git_url":"git://github.com/smashwilson/azurefire.git","ssh_url":"git@github.com:smashwilson/azurefire.git","clone_url":"https://github.com/smashwilson/azurefire.git","svn_url":"https://github.com/smashwilson/azurefire","homepage":null,"size":4368,"stargazers_count":1,"watchers_count":1,"language":"CSS","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":2,"forks":0,"open_issues":2,"watchers":1,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/smashwilson/azurefire/pulls/22"},"html":{"href":"https://github.com/smashwilson/azurefire/pull/22"},"issue":{"href":"https://api.github.com/repos/smashwilson/azurefire/issues/22"},"comments":{"href":"https://api.github.com/repos/smashwilson/azurefire/issues/22/comments"},"review_comments":{"href":"https://api.github.com/repos/smashwilson/azurefire/pulls/22/comments"},"review_comment":{"href":"https://api.github.com/repos/smashwilson/azurefire/pulls/comments/{number}"},"commits":{"href":"https://api.github.com/repos/smashwilson/azurefire/pulls/22/commits"},"statuses":{"href":"https://api.github.com/repos/smashwilson/azurefire/statuses/ce24d08fd4610072b979f66c1cd4143b9512aab2"}}}}