{"action":"created","comment":{"url":"https://api.github.com/repos/TTLabs/EvaporateJS/pulls/comments/48019033","id":48019033,"diff_hunk":"@@ -432,10 +458,54 @@ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\n            authorizedSend(complete);\n         }\n \n+        var numProcessed = 0,\n+            numParts = -1;\n+\n+        function computePartMd5Digest(part) {\n+           return function () {\n+              var s = me.file.status;\n+              if (s == ERROR || s == CANCELED) {\n+                 return;\n+              }\n+\n+              var md5_digest = con.cryptoMd5Method.call(this, this.result);\n+\n+              l.d(['part #', part.part, ' MD5 digest is ', md5_digest].join(''));\n+              part.md5_digest = md5_digest;\n+\n+              delete part.reader; // release potentially large memory allocation\n+\n+              numProcessed += 1;\n+\n+              processPartsList();\n+\n+              if (numProcessed === numParts) {\n+                 l.d('All parts have MD5 digests');\n+              }\n+\n+              setTimeout(processPartsListWithMd5Digests, 1500);","path":"evaporate.js","position":137,"original_position":137,"commit_id":"0d7ad6792cc0272f6a06a22137a296b02f5cb534","original_commit_id":"0d7ad6792cc0272f6a06a22137a296b02f5cb534","user":{"login":"bikeath1337","id":243237,"avatar_url":"https://avatars.githubusercontent.com/u/243237?v=3","gravatar_id":"","url":"https://api.github.com/users/bikeath1337","html_url":"https://github.com/bikeath1337","followers_url":"https://api.github.com/users/bikeath1337/followers","following_url":"https://api.github.com/users/bikeath1337/following{/other_user}","gists_url":"https://api.github.com/users/bikeath1337/gists{/gist_id}","starred_url":"https://api.github.com/users/bikeath1337/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bikeath1337/subscriptions","organizations_url":"https://api.github.com/users/bikeath1337/orgs","repos_url":"https://api.github.com/users/bikeath1337/repos","events_url":"https://api.github.com/users/bikeath1337/events{/privacy}","received_events_url":"https://api.github.com/users/bikeath1337/received_events","type":"User","site_admin":false},"body":"@tomsaffell this implementation isn't perfect in that it keeps iterating over the parts list each time BUT it is close to JIT because:\r\n\r\n1. we start with the first part in `processPartsListWithMd5Digests()`. We only do one part.\r\n2. when the data read is done, we calculate the md5 and hopefully allow GC to clean up memory\r\n3. we attach the digest to the part and then kick off `processPartsList()` which will find only one part that is ready and will initiate upload of that part.\r\n4. on return from `processPartsList()` we then wait 1500 seconds before we start to find the next part that needs an MD5 digest.\r\n5. so while we're working the digest for the second part, we've got only one async PUT in progress. This happens until the browser says it has enough concurrent async jobs running (more than the option setting in EvaporateJS).\r\n6. at a certain point, the browser is going to stop processing any new jobs until it has room to do so. At that point, EvaporateJS just has an MD5 calc job waiting...once that job runs, we then submit the PUT...and everything works cooperatively.\r\n\r\nOne of my previous implementations did not control the flow this way and I ended up with an async XHR queue full of CPU and IO intensive jobs to calculate the checksums. The PUT requests never had a chance. Interestingly, only Safari showed the XHR async jobs related to the contention were not visible in Chrome, only Safari. Wasn't I lucky?\r\n\r\nSo to summarize, the rules when computing checksums are:\r\n\r\n1. Do not upload a part if its checksum isn't ready.\r\n2. As soon as a part has a checksum, submit it for upload. Assuming the XHR queue is idle, upload should begin.\r\n3. After upload is submitted (not necessarily executing), wait 1.5 seconds before trying to calculate the next part's checksum. This adds a job to the XHR queue. \r\n4. Until that XHR data request completes, nothing else happens except idle looping through the parts queue (this could be optimized, of course).\r\n5. This XHR requests build up until the browser is no longer able to keep up, at which time the \"next\" XHR request to be processed once it has room, MUST be an XHR request for a data read.\r\n\r\nThere may be some missing edge cases here but the more I think about it, the more I think it's a solid approach. What's left is an optimization that doesn't keep iterative over a list of parts but works with, say, a FIFO queue of some kind, pushing/popping as needed.","created_at":"2015-12-18T12:04:49Z","updated_at":"2015-12-18T12:04:49Z","html_url":"https://github.com/TTLabs/EvaporateJS/pull/97#discussion_r48019033","pull_request_url":"https://api.github.com/repos/TTLabs/EvaporateJS/pulls/97","_links":{"self":{"href":"https://api.github.com/repos/TTLabs/EvaporateJS/pulls/comments/48019033"},"html":{"href":"https://github.com/TTLabs/EvaporateJS/pull/97#discussion_r48019033"},"pull_request":{"href":"https://api.github.com/repos/TTLabs/EvaporateJS/pulls/97"}}},"pull_request":{"url":"https://api.github.com/repos/TTLabs/EvaporateJS/pulls/97","id":53993346,"html_url":"https://github.com/TTLabs/EvaporateJS/pull/97","diff_url":"https://github.com/TTLabs/EvaporateJS/pull/97.diff","patch_url":"https://github.com/TTLabs/EvaporateJS/pull/97.patch","issue_url":"https://api.github.com/repos/TTLabs/EvaporateJS/issues/97","number":97,"state":"open","locked":false,"title":"Adds Support for AWS Content-MD5 Request Headers","user":{"login":"bikeath1337","id":243237,"avatar_url":"https://avatars.githubusercontent.com/u/243237?v=3","gravatar_id":"","url":"https://api.github.com/users/bikeath1337","html_url":"https://github.com/bikeath1337","followers_url":"https://api.github.com/users/bikeath1337/followers","following_url":"https://api.github.com/users/bikeath1337/following{/other_user}","gists_url":"https://api.github.com/users/bikeath1337/gists{/gist_id}","starred_url":"https://api.github.com/users/bikeath1337/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bikeath1337/subscriptions","organizations_url":"https://api.github.com/users/bikeath1337/orgs","repos_url":"https://api.github.com/users/bikeath1337/repos","events_url":"https://api.github.com/users/bikeath1337/events{/privacy}","received_events_url":"https://api.github.com/users/bikeath1337/received_events","type":"User","site_admin":false},"body":"This pull request adds support for MD5 digest transmission to AWS and addresses https://github.com/TTLabs/EvaporateJS/issues/96.\r\n\r\nNotes: The choice of cryptographic MD5 digest implementation is pluggable in this branch. I tried several implementations but settled on Spark MD5. Instructions on how to get this JavaScript library are in the README changes.\r\n\r\nThis branch uses 2 new configuration options: `computeContentMd5` is already used in the AWS JavaScript for Browsers SDK, so I chose that name for consistency. To make the checksum method pluggable, developers can define that method with the `cryptoMd5Method`.\r\n\r\nNote that on initialization, if `computeContentMd5`, then the code uses `alert()` if `cryptoMd5Method` is not defined as a function. Clearly this is harsh but could be considered necessary so that later indeterminate behavior can be avoided.\r\n\r\nThe way this works is to use a FileReader object. To get the contents of the object so that we can create the MD5 digest requires an asynchronous call where the digest is completed in the callback. I tried to make minimal changes to the existing code to enable better review.\r\n\r\nThis pull request also reads the AWS response object, parsed the XML and replies in the upload's on onerror callback.","created_at":"2015-12-17T15:53:02Z","updated_at":"2015-12-18T12:04:49Z","closed_at":null,"merged_at":null,"merge_commit_sha":"201434ca55dc4c70a6c38faa77d38afc581f0912","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/TTLabs/EvaporateJS/pulls/97/commits","review_comments_url":"https://api.github.com/repos/TTLabs/EvaporateJS/pulls/97/comments","review_comment_url":"https://api.github.com/repos/TTLabs/EvaporateJS/pulls/comments{/number}","comments_url":"https://api.github.com/repos/TTLabs/EvaporateJS/issues/97/comments","statuses_url":"https://api.github.com/repos/TTLabs/EvaporateJS/statuses/0d7ad6792cc0272f6a06a22137a296b02f5cb534","head":{"label":"bikeath1337:breakdown","ref":"breakdown","sha":"0d7ad6792cc0272f6a06a22137a296b02f5cb534","user":{"login":"bikeath1337","id":243237,"avatar_url":"https://avatars.githubusercontent.com/u/243237?v=3","gravatar_id":"","url":"https://api.github.com/users/bikeath1337","html_url":"https://github.com/bikeath1337","followers_url":"https://api.github.com/users/bikeath1337/followers","following_url":"https://api.github.com/users/bikeath1337/following{/other_user}","gists_url":"https://api.github.com/users/bikeath1337/gists{/gist_id}","starred_url":"https://api.github.com/users/bikeath1337/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bikeath1337/subscriptions","organizations_url":"https://api.github.com/users/bikeath1337/orgs","repos_url":"https://api.github.com/users/bikeath1337/repos","events_url":"https://api.github.com/users/bikeath1337/events{/privacy}","received_events_url":"https://api.github.com/users/bikeath1337/received_events","type":"User","site_admin":false},"repo":{"id":47833508,"name":"EvaporateJS","full_name":"bikeath1337/EvaporateJS","owner":{"login":"bikeath1337","id":243237,"avatar_url":"https://avatars.githubusercontent.com/u/243237?v=3","gravatar_id":"","url":"https://api.github.com/users/bikeath1337","html_url":"https://github.com/bikeath1337","followers_url":"https://api.github.com/users/bikeath1337/followers","following_url":"https://api.github.com/users/bikeath1337/following{/other_user}","gists_url":"https://api.github.com/users/bikeath1337/gists{/gist_id}","starred_url":"https://api.github.com/users/bikeath1337/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bikeath1337/subscriptions","organizations_url":"https://api.github.com/users/bikeath1337/orgs","repos_url":"https://api.github.com/users/bikeath1337/repos","events_url":"https://api.github.com/users/bikeath1337/events{/privacy}","received_events_url":"https://api.github.com/users/bikeath1337/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/bikeath1337/EvaporateJS","description":"Javascript library for browser to S3 multipart resumable uploads","fork":true,"url":"https://api.github.com/repos/bikeath1337/EvaporateJS","forks_url":"https://api.github.com/repos/bikeath1337/EvaporateJS/forks","keys_url":"https://api.github.com/repos/bikeath1337/EvaporateJS/keys{/key_id}","collaborators_url":"https://api.github.com/repos/bikeath1337/EvaporateJS/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/bikeath1337/EvaporateJS/teams","hooks_url":"https://api.github.com/repos/bikeath1337/EvaporateJS/hooks","issue_events_url":"https://api.github.com/repos/bikeath1337/EvaporateJS/issues/events{/number}","events_url":"https://api.github.com/repos/bikeath1337/EvaporateJS/events","assignees_url":"https://api.github.com/repos/bikeath1337/EvaporateJS/assignees{/user}","branches_url":"https://api.github.com/repos/bikeath1337/EvaporateJS/branches{/branch}","tags_url":"https://api.github.com/repos/bikeath1337/EvaporateJS/tags","blobs_url":"https://api.github.com/repos/bikeath1337/EvaporateJS/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/bikeath1337/EvaporateJS/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/bikeath1337/EvaporateJS/git/refs{/sha}","trees_url":"https://api.github.com/repos/bikeath1337/EvaporateJS/git/trees{/sha}","statuses_url":"https://api.github.com/repos/bikeath1337/EvaporateJS/statuses/{sha}","languages_url":"https://api.github.com/repos/bikeath1337/EvaporateJS/languages","stargazers_url":"https://api.github.com/repos/bikeath1337/EvaporateJS/stargazers","contributors_url":"https://api.github.com/repos/bikeath1337/EvaporateJS/contributors","subscribers_url":"https://api.github.com/repos/bikeath1337/EvaporateJS/subscribers","subscription_url":"https://api.github.com/repos/bikeath1337/EvaporateJS/subscription","commits_url":"https://api.github.com/repos/bikeath1337/EvaporateJS/commits{/sha}","git_commits_url":"https://api.github.com/repos/bikeath1337/EvaporateJS/git/commits{/sha}","comments_url":"https://api.github.com/repos/bikeath1337/EvaporateJS/comments{/number}","issue_comment_url":"https://api.github.com/repos/bikeath1337/EvaporateJS/issues/comments{/number}","contents_url":"https://api.github.com/repos/bikeath1337/EvaporateJS/contents/{+path}","compare_url":"https://api.github.com/repos/bikeath1337/EvaporateJS/compare/{base}...{head}","merges_url":"https://api.github.com/repos/bikeath1337/EvaporateJS/merges","archive_url":"https://api.github.com/repos/bikeath1337/EvaporateJS/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/bikeath1337/EvaporateJS/downloads","issues_url":"https://api.github.com/repos/bikeath1337/EvaporateJS/issues{/number}","pulls_url":"https://api.github.com/repos/bikeath1337/EvaporateJS/pulls{/number}","milestones_url":"https://api.github.com/repos/bikeath1337/EvaporateJS/milestones{/number}","notifications_url":"https://api.github.com/repos/bikeath1337/EvaporateJS/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/bikeath1337/EvaporateJS/labels{/name}","releases_url":"https://api.github.com/repos/bikeath1337/EvaporateJS/releases{/id}","created_at":"2015-12-11T15:14:53Z","updated_at":"2015-12-11T15:14:53Z","pushed_at":"2015-12-17T15:40:41Z","git_url":"git://github.com/bikeath1337/EvaporateJS.git","ssh_url":"git@github.com:bikeath1337/EvaporateJS.git","clone_url":"https://github.com/bikeath1337/EvaporateJS.git","svn_url":"https://github.com/bikeath1337/EvaporateJS","homepage":null,"size":242,"stargazers_count":0,"watchers_count":0,"language":"JavaScript","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"TTLabs:master","ref":"master","sha":"1a806997959df2e14cd4b7ce21120e75ec9e4fe7","user":{"login":"TTLabs","id":814544,"avatar_url":"https://avatars.githubusercontent.com/u/814544?v=3","gravatar_id":"","url":"https://api.github.com/users/TTLabs","html_url":"https://github.com/TTLabs","followers_url":"https://api.github.com/users/TTLabs/followers","following_url":"https://api.github.com/users/TTLabs/following{/other_user}","gists_url":"https://api.github.com/users/TTLabs/gists{/gist_id}","starred_url":"https://api.github.com/users/TTLabs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/TTLabs/subscriptions","organizations_url":"https://api.github.com/users/TTLabs/orgs","repos_url":"https://api.github.com/users/TTLabs/repos","events_url":"https://api.github.com/users/TTLabs/events{/privacy}","received_events_url":"https://api.github.com/users/TTLabs/received_events","type":"Organization","site_admin":false},"repo":{"id":7264874,"name":"EvaporateJS","full_name":"TTLabs/EvaporateJS","owner":{"login":"TTLabs","id":814544,"avatar_url":"https://avatars.githubusercontent.com/u/814544?v=3","gravatar_id":"","url":"https://api.github.com/users/TTLabs","html_url":"https://github.com/TTLabs","followers_url":"https://api.github.com/users/TTLabs/followers","following_url":"https://api.github.com/users/TTLabs/following{/other_user}","gists_url":"https://api.github.com/users/TTLabs/gists{/gist_id}","starred_url":"https://api.github.com/users/TTLabs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/TTLabs/subscriptions","organizations_url":"https://api.github.com/users/TTLabs/orgs","repos_url":"https://api.github.com/users/TTLabs/repos","events_url":"https://api.github.com/users/TTLabs/events{/privacy}","received_events_url":"https://api.github.com/users/TTLabs/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/TTLabs/EvaporateJS","description":"Javascript library for browser to S3 multipart resumable uploads","fork":false,"url":"https://api.github.com/repos/TTLabs/EvaporateJS","forks_url":"https://api.github.com/repos/TTLabs/EvaporateJS/forks","keys_url":"https://api.github.com/repos/TTLabs/EvaporateJS/keys{/key_id}","collaborators_url":"https://api.github.com/repos/TTLabs/EvaporateJS/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/TTLabs/EvaporateJS/teams","hooks_url":"https://api.github.com/repos/TTLabs/EvaporateJS/hooks","issue_events_url":"https://api.github.com/repos/TTLabs/EvaporateJS/issues/events{/number}","events_url":"https://api.github.com/repos/TTLabs/EvaporateJS/events","assignees_url":"https://api.github.com/repos/TTLabs/EvaporateJS/assignees{/user}","branches_url":"https://api.github.com/repos/TTLabs/EvaporateJS/branches{/branch}","tags_url":"https://api.github.com/repos/TTLabs/EvaporateJS/tags","blobs_url":"https://api.github.com/repos/TTLabs/EvaporateJS/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/TTLabs/EvaporateJS/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/TTLabs/EvaporateJS/git/refs{/sha}","trees_url":"https://api.github.com/repos/TTLabs/EvaporateJS/git/trees{/sha}","statuses_url":"https://api.github.com/repos/TTLabs/EvaporateJS/statuses/{sha}","languages_url":"https://api.github.com/repos/TTLabs/EvaporateJS/languages","stargazers_url":"https://api.github.com/repos/TTLabs/EvaporateJS/stargazers","contributors_url":"https://api.github.com/repos/TTLabs/EvaporateJS/contributors","subscribers_url":"https://api.github.com/repos/TTLabs/EvaporateJS/subscribers","subscription_url":"https://api.github.com/repos/TTLabs/EvaporateJS/subscription","commits_url":"https://api.github.com/repos/TTLabs/EvaporateJS/commits{/sha}","git_commits_url":"https://api.github.com/repos/TTLabs/EvaporateJS/git/commits{/sha}","comments_url":"https://api.github.com/repos/TTLabs/EvaporateJS/comments{/number}","issue_comment_url":"https://api.github.com/repos/TTLabs/EvaporateJS/issues/comments{/number}","contents_url":"https://api.github.com/repos/TTLabs/EvaporateJS/contents/{+path}","compare_url":"https://api.github.com/repos/TTLabs/EvaporateJS/compare/{base}...{head}","merges_url":"https://api.github.com/repos/TTLabs/EvaporateJS/merges","archive_url":"https://api.github.com/repos/TTLabs/EvaporateJS/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/TTLabs/EvaporateJS/downloads","issues_url":"https://api.github.com/repos/TTLabs/EvaporateJS/issues{/number}","pulls_url":"https://api.github.com/repos/TTLabs/EvaporateJS/pulls{/number}","milestones_url":"https://api.github.com/repos/TTLabs/EvaporateJS/milestones{/number}","notifications_url":"https://api.github.com/repos/TTLabs/EvaporateJS/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/TTLabs/EvaporateJS/labels{/name}","releases_url":"https://api.github.com/repos/TTLabs/EvaporateJS/releases{/id}","created_at":"2012-12-20T21:49:38Z","updated_at":"2015-12-16T00:21:48Z","pushed_at":"2015-12-17T15:53:03Z","git_url":"git://github.com/TTLabs/EvaporateJS.git","ssh_url":"git@github.com:TTLabs/EvaporateJS.git","clone_url":"https://github.com/TTLabs/EvaporateJS.git","svn_url":"https://github.com/TTLabs/EvaporateJS","homepage":null,"size":674,"stargazers_count":964,"watchers_count":964,"language":"JavaScript","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":80,"mirror_url":null,"open_issues_count":29,"forks":80,"open_issues":29,"watchers":964,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/TTLabs/EvaporateJS/pulls/97"},"html":{"href":"https://github.com/TTLabs/EvaporateJS/pull/97"},"issue":{"href":"https://api.github.com/repos/TTLabs/EvaporateJS/issues/97"},"comments":{"href":"https://api.github.com/repos/TTLabs/EvaporateJS/issues/97/comments"},"review_comments":{"href":"https://api.github.com/repos/TTLabs/EvaporateJS/pulls/97/comments"},"review_comment":{"href":"https://api.github.com/repos/TTLabs/EvaporateJS/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/TTLabs/EvaporateJS/pulls/97/commits"},"statuses":{"href":"https://api.github.com/repos/TTLabs/EvaporateJS/statuses/0d7ad6792cc0272f6a06a22137a296b02f5cb534"}}}}