{"action":"created","comment":{"url":"https://api.github.com/repos/mspnp/iot-journey/pulls/comments/35055166","id":35055166,"diff_hunk":"@@ -1,74 +1,90 @@\n-# Journal Entry #1\n-_Capturing event data and saving it in its raw format to cold storage_\n+# Journal 1: Event Ingestion and Cold Storage\n \n-:pencil2: working draft\n+This journal entry covers the most basic end-to-end scenario in IoT: Getting events into the system and putting them into long-term storage. In terms of the logical architecture, it maps to the highlighted components shown here:\n \n-> The purpose of this phase is to determine an approach to ingesting data, performing the simplest of processing, and saving it for subsequent analysis. Because the customer requires all event data to be stored indefinitely, the volume of data held in cold storage could become very large. Cold storage must therefore be inexpensive.\n+![plan for this first milestone](media/01-cold-storage/logical-architecture-for-milestone01.png)\n \n-# Fabrikam's Perspective\n-Fabrikam's engineering team has very little experience with high-scale event-oriented systems. They want to make sure that they are [asking the right questions][orientation]. They also need to get a minimum viable product deployed.\n \n-For their [first milestone][milestone] they want to:\n-- Ingest a representative set of simulated events.\n-- Store the raw data for all events for later analysis. [:arrow_right:][cold-storage]\n-- Increase the scale until they match the targets for their customer [:arrow_right:][increase-scale]\n+- [Common requirements](#common-requirements)\n+- [Fabrikam requirements](#fabrikam-requirements)\n+- [Exploring solutions for ingestion and processing](#exploring-solutions-for-ingestion-and processing)\n+- [Exploring solutions for cold storage](#exploring-solutions-for-cold-storage)\n+- [Reference implementation](#reference-implementation)\n+- [Lessons learned](#lessons-learned)\n \n-They need a basic implementation that maps to the highlighted components of the logical architecture:\n+## Common requirements\n \n-![plan for this first milestone](media/01-cold-storage/logical-architecture-for-milestone01.png)\n+Here are some common requirements for ingestion and cold storage that any IoT solution will at least need to consider:\n+\n+- Achieving scale. That means both hitting the expected target data rate, but also designing the system to be scalable in the future.\n+- Authentication. Only authenticated devices can send data. (Provisioning devices is a separate scenario, although closely related.)\n+- Protecting the data stream.\n+- Storing the raw event data in long-term storage. Cold storage must hold a large volume of historical data indefinitely, at a reasonable cost. \n+\n+  \n+## Fabrikam requirements\n+\n+:memo: For more about our Fabrikam scenario, see [Introducing the Journey][00-intro]. \n \n+Here are some specific requirements for our scenario:\n \n-## Devices\n+- As an initial target, the system must handle up to 100,000 devices, each device sending 1 reading every minute. That's approximately 1,667 events per second. This is a minimum load &mdash; the system should be scalable beyond this target. \n+- Events might be time critical, so the Event Processing component must handle this volume of data without introducing any backlog into the system.\n+- The event data stream must be protected. This means not only securing the content of the events, but also masking the *absence* of events. For example, if the motion detectors in a house are not streaming any events, an attacker can conclude that nobody is home.","path":"docs/journal/01-cold-storage.md","position":46,"original_position":46,"commit_id":"1c1ae768402ae3899b1b97d0506cc53ecf74eaf0","original_commit_id":"1c1ae768402ae3899b1b97d0506cc53ecf74eaf0","user":{"login":"bennage","id":118161,"avatar_url":"https://avatars.githubusercontent.com/u/118161?v=3","gravatar_id":"","url":"https://api.github.com/users/bennage","html_url":"https://github.com/bennage","followers_url":"https://api.github.com/users/bennage/followers","following_url":"https://api.github.com/users/bennage/following{/other_user}","gists_url":"https://api.github.com/users/bennage/gists{/gist_id}","starred_url":"https://api.github.com/users/bennage/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bennage/subscriptions","organizations_url":"https://api.github.com/users/bennage/orgs","repos_url":"https://api.github.com/users/bennage/repos","events_url":"https://api.github.com/users/bennage/events{/privacy}","received_events_url":"https://api.github.com/users/bennage/received_events","type":"User","site_admin":false},"body":"I don't think that this is correct place for this requirement. This is more about a malicious agent detecting activity prior to the ingestion process (that is, while events are travelling on a public network to the cloud-hosted gateway).","created_at":"2015-07-20T23:13:04Z","updated_at":"2015-07-20T23:13:04Z","html_url":"https://github.com/mspnp/iot-journey/pull/236#discussion_r35055166","pull_request_url":"https://api.github.com/repos/mspnp/iot-journey/pulls/236","_links":{"self":{"href":"https://api.github.com/repos/mspnp/iot-journey/pulls/comments/35055166"},"html":{"href":"https://github.com/mspnp/iot-journey/pull/236#discussion_r35055166"},"pull_request":{"href":"https://api.github.com/repos/mspnp/iot-journey/pulls/236"}}},"pull_request":{"url":"https://api.github.com/repos/mspnp/iot-journey/pulls/236","id":40278592,"html_url":"https://github.com/mspnp/iot-journey/pull/236","diff_url":"https://github.com/mspnp/iot-journey/pull/236.diff","patch_url":"https://github.com/mspnp/iot-journey/pull/236.patch","issue_url":"https://api.github.com/repos/mspnp/iot-journey/issues/236","number":236,"state":"open","locked":false,"title":"Revise journal 1 (Cold Storage)","user":{"login":"MikeWasson","id":3992422,"avatar_url":"https://avatars.githubusercontent.com/u/3992422?v=3","gravatar_id":"","url":"https://api.github.com/users/MikeWasson","html_url":"https://github.com/MikeWasson","followers_url":"https://api.github.com/users/MikeWasson/followers","following_url":"https://api.github.com/users/MikeWasson/following{/other_user}","gists_url":"https://api.github.com/users/MikeWasson/gists{/gist_id}","starred_url":"https://api.github.com/users/MikeWasson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MikeWasson/subscriptions","organizations_url":"https://api.github.com/users/MikeWasson/orgs","repos_url":"https://api.github.com/users/MikeWasson/repos","events_url":"https://api.github.com/users/MikeWasson/events{/privacy}","received_events_url":"https://api.github.com/users/MikeWasson/received_events","type":"User","site_admin":false},"body":"#192 \r\n","created_at":"2015-07-17T22:50:36Z","updated_at":"2015-07-20T23:13:04Z","closed_at":null,"merged_at":null,"merge_commit_sha":"db9aa55bec1cd9427152e55933abb2c63fe6cfa5","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/mspnp/iot-journey/pulls/236/commits","review_comments_url":"https://api.github.com/repos/mspnp/iot-journey/pulls/236/comments","review_comment_url":"https://api.github.com/repos/mspnp/iot-journey/pulls/comments{/number}","comments_url":"https://api.github.com/repos/mspnp/iot-journey/issues/236/comments","statuses_url":"https://api.github.com/repos/mspnp/iot-journey/statuses/1c1ae768402ae3899b1b97d0506cc53ecf74eaf0","head":{"label":"mspnp:wasson/journal1","ref":"wasson/journal1","sha":"1c1ae768402ae3899b1b97d0506cc53ecf74eaf0","user":{"login":"mspnp","id":1373420,"avatar_url":"https://avatars.githubusercontent.com/u/1373420?v=3","gravatar_id":"","url":"https://api.github.com/users/mspnp","html_url":"https://github.com/mspnp","followers_url":"https://api.github.com/users/mspnp/followers","following_url":"https://api.github.com/users/mspnp/following{/other_user}","gists_url":"https://api.github.com/users/mspnp/gists{/gist_id}","starred_url":"https://api.github.com/users/mspnp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mspnp/subscriptions","organizations_url":"https://api.github.com/users/mspnp/orgs","repos_url":"https://api.github.com/users/mspnp/repos","events_url":"https://api.github.com/users/mspnp/events{/privacy}","received_events_url":"https://api.github.com/users/mspnp/received_events","type":"Organization","site_admin":false},"repo":{"id":34227406,"name":"iot-journey","full_name":"mspnp/iot-journey","owner":{"login":"mspnp","id":1373420,"avatar_url":"https://avatars.githubusercontent.com/u/1373420?v=3","gravatar_id":"","url":"https://api.github.com/users/mspnp","html_url":"https://github.com/mspnp","followers_url":"https://api.github.com/users/mspnp/followers","following_url":"https://api.github.com/users/mspnp/following{/other_user}","gists_url":"https://api.github.com/users/mspnp/gists{/gist_id}","starred_url":"https://api.github.com/users/mspnp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mspnp/subscriptions","organizations_url":"https://api.github.com/users/mspnp/orgs","repos_url":"https://api.github.com/users/mspnp/repos","events_url":"https://api.github.com/users/mspnp/events{/privacy}","received_events_url":"https://api.github.com/users/mspnp/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/mspnp/iot-journey","description":"a guidance project for implementing an IoT solution on Azure","fork":false,"url":"https://api.github.com/repos/mspnp/iot-journey","forks_url":"https://api.github.com/repos/mspnp/iot-journey/forks","keys_url":"https://api.github.com/repos/mspnp/iot-journey/keys{/key_id}","collaborators_url":"https://api.github.com/repos/mspnp/iot-journey/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/mspnp/iot-journey/teams","hooks_url":"https://api.github.com/repos/mspnp/iot-journey/hooks","issue_events_url":"https://api.github.com/repos/mspnp/iot-journey/issues/events{/number}","events_url":"https://api.github.com/repos/mspnp/iot-journey/events","assignees_url":"https://api.github.com/repos/mspnp/iot-journey/assignees{/user}","branches_url":"https://api.github.com/repos/mspnp/iot-journey/branches{/branch}","tags_url":"https://api.github.com/repos/mspnp/iot-journey/tags","blobs_url":"https://api.github.com/repos/mspnp/iot-journey/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/mspnp/iot-journey/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/mspnp/iot-journey/git/refs{/sha}","trees_url":"https://api.github.com/repos/mspnp/iot-journey/git/trees{/sha}","statuses_url":"https://api.github.com/repos/mspnp/iot-journey/statuses/{sha}","languages_url":"https://api.github.com/repos/mspnp/iot-journey/languages","stargazers_url":"https://api.github.com/repos/mspnp/iot-journey/stargazers","contributors_url":"https://api.github.com/repos/mspnp/iot-journey/contributors","subscribers_url":"https://api.github.com/repos/mspnp/iot-journey/subscribers","subscription_url":"https://api.github.com/repos/mspnp/iot-journey/subscription","commits_url":"https://api.github.com/repos/mspnp/iot-journey/commits{/sha}","git_commits_url":"https://api.github.com/repos/mspnp/iot-journey/git/commits{/sha}","comments_url":"https://api.github.com/repos/mspnp/iot-journey/comments{/number}","issue_comment_url":"https://api.github.com/repos/mspnp/iot-journey/issues/comments{/number}","contents_url":"https://api.github.com/repos/mspnp/iot-journey/contents/{+path}","compare_url":"https://api.github.com/repos/mspnp/iot-journey/compare/{base}...{head}","merges_url":"https://api.github.com/repos/mspnp/iot-journey/merges","archive_url":"https://api.github.com/repos/mspnp/iot-journey/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/mspnp/iot-journey/downloads","issues_url":"https://api.github.com/repos/mspnp/iot-journey/issues{/number}","pulls_url":"https://api.github.com/repos/mspnp/iot-journey/pulls{/number}","milestones_url":"https://api.github.com/repos/mspnp/iot-journey/milestones{/number}","notifications_url":"https://api.github.com/repos/mspnp/iot-journey/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/mspnp/iot-journey/labels{/name}","releases_url":"https://api.github.com/repos/mspnp/iot-journey/releases{/id}","created_at":"2015-04-19T22:49:18Z","updated_at":"2015-07-20T13:00:19Z","pushed_at":"2015-07-20T23:06:54Z","git_url":"git://github.com/mspnp/iot-journey.git","ssh_url":"git@github.com:mspnp/iot-journey.git","clone_url":"https://github.com/mspnp/iot-journey.git","svn_url":"https://github.com/mspnp/iot-journey","homepage":"","size":6375,"stargazers_count":28,"watchers_count":28,"language":"C#","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":13,"mirror_url":null,"open_issues_count":56,"forks":13,"open_issues":56,"watchers":28,"default_branch":"master"}},"base":{"label":"mspnp:master","ref":"master","sha":"1a4b06b2221f9a1fe6e6e9e11ab2b52756aa5acb","user":{"login":"mspnp","id":1373420,"avatar_url":"https://avatars.githubusercontent.com/u/1373420?v=3","gravatar_id":"","url":"https://api.github.com/users/mspnp","html_url":"https://github.com/mspnp","followers_url":"https://api.github.com/users/mspnp/followers","following_url":"https://api.github.com/users/mspnp/following{/other_user}","gists_url":"https://api.github.com/users/mspnp/gists{/gist_id}","starred_url":"https://api.github.com/users/mspnp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mspnp/subscriptions","organizations_url":"https://api.github.com/users/mspnp/orgs","repos_url":"https://api.github.com/users/mspnp/repos","events_url":"https://api.github.com/users/mspnp/events{/privacy}","received_events_url":"https://api.github.com/users/mspnp/received_events","type":"Organization","site_admin":false},"repo":{"id":34227406,"name":"iot-journey","full_name":"mspnp/iot-journey","owner":{"login":"mspnp","id":1373420,"avatar_url":"https://avatars.githubusercontent.com/u/1373420?v=3","gravatar_id":"","url":"https://api.github.com/users/mspnp","html_url":"https://github.com/mspnp","followers_url":"https://api.github.com/users/mspnp/followers","following_url":"https://api.github.com/users/mspnp/following{/other_user}","gists_url":"https://api.github.com/users/mspnp/gists{/gist_id}","starred_url":"https://api.github.com/users/mspnp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mspnp/subscriptions","organizations_url":"https://api.github.com/users/mspnp/orgs","repos_url":"https://api.github.com/users/mspnp/repos","events_url":"https://api.github.com/users/mspnp/events{/privacy}","received_events_url":"https://api.github.com/users/mspnp/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/mspnp/iot-journey","description":"a guidance project for implementing an IoT solution on Azure","fork":false,"url":"https://api.github.com/repos/mspnp/iot-journey","forks_url":"https://api.github.com/repos/mspnp/iot-journey/forks","keys_url":"https://api.github.com/repos/mspnp/iot-journey/keys{/key_id}","collaborators_url":"https://api.github.com/repos/mspnp/iot-journey/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/mspnp/iot-journey/teams","hooks_url":"https://api.github.com/repos/mspnp/iot-journey/hooks","issue_events_url":"https://api.github.com/repos/mspnp/iot-journey/issues/events{/number}","events_url":"https://api.github.com/repos/mspnp/iot-journey/events","assignees_url":"https://api.github.com/repos/mspnp/iot-journey/assignees{/user}","branches_url":"https://api.github.com/repos/mspnp/iot-journey/branches{/branch}","tags_url":"https://api.github.com/repos/mspnp/iot-journey/tags","blobs_url":"https://api.github.com/repos/mspnp/iot-journey/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/mspnp/iot-journey/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/mspnp/iot-journey/git/refs{/sha}","trees_url":"https://api.github.com/repos/mspnp/iot-journey/git/trees{/sha}","statuses_url":"https://api.github.com/repos/mspnp/iot-journey/statuses/{sha}","languages_url":"https://api.github.com/repos/mspnp/iot-journey/languages","stargazers_url":"https://api.github.com/repos/mspnp/iot-journey/stargazers","contributors_url":"https://api.github.com/repos/mspnp/iot-journey/contributors","subscribers_url":"https://api.github.com/repos/mspnp/iot-journey/subscribers","subscription_url":"https://api.github.com/repos/mspnp/iot-journey/subscription","commits_url":"https://api.github.com/repos/mspnp/iot-journey/commits{/sha}","git_commits_url":"https://api.github.com/repos/mspnp/iot-journey/git/commits{/sha}","comments_url":"https://api.github.com/repos/mspnp/iot-journey/comments{/number}","issue_comment_url":"https://api.github.com/repos/mspnp/iot-journey/issues/comments{/number}","contents_url":"https://api.github.com/repos/mspnp/iot-journey/contents/{+path}","compare_url":"https://api.github.com/repos/mspnp/iot-journey/compare/{base}...{head}","merges_url":"https://api.github.com/repos/mspnp/iot-journey/merges","archive_url":"https://api.github.com/repos/mspnp/iot-journey/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/mspnp/iot-journey/downloads","issues_url":"https://api.github.com/repos/mspnp/iot-journey/issues{/number}","pulls_url":"https://api.github.com/repos/mspnp/iot-journey/pulls{/number}","milestones_url":"https://api.github.com/repos/mspnp/iot-journey/milestones{/number}","notifications_url":"https://api.github.com/repos/mspnp/iot-journey/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/mspnp/iot-journey/labels{/name}","releases_url":"https://api.github.com/repos/mspnp/iot-journey/releases{/id}","created_at":"2015-04-19T22:49:18Z","updated_at":"2015-07-20T13:00:19Z","pushed_at":"2015-07-20T23:06:54Z","git_url":"git://github.com/mspnp/iot-journey.git","ssh_url":"git@github.com:mspnp/iot-journey.git","clone_url":"https://github.com/mspnp/iot-journey.git","svn_url":"https://github.com/mspnp/iot-journey","homepage":"","size":6375,"stargazers_count":28,"watchers_count":28,"language":"C#","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":13,"mirror_url":null,"open_issues_count":56,"forks":13,"open_issues":56,"watchers":28,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/mspnp/iot-journey/pulls/236"},"html":{"href":"https://github.com/mspnp/iot-journey/pull/236"},"issue":{"href":"https://api.github.com/repos/mspnp/iot-journey/issues/236"},"comments":{"href":"https://api.github.com/repos/mspnp/iot-journey/issues/236/comments"},"review_comments":{"href":"https://api.github.com/repos/mspnp/iot-journey/pulls/236/comments"},"review_comment":{"href":"https://api.github.com/repos/mspnp/iot-journey/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/mspnp/iot-journey/pulls/236/commits"},"statuses":{"href":"https://api.github.com/repos/mspnp/iot-journey/statuses/1c1ae768402ae3899b1b97d0506cc53ecf74eaf0"}}}}