{"action":"created","comment":{"url":"https://api.github.com/repos/google/rappor/pulls/comments/43700946","id":43700946,"diff_hunk":"@@ -0,0 +1,294 @@\n+// Copyright 2015 Google Inc. All rights reserved.\n+//\n+// Licensed under the Apache License, Version 2.0 (the \"License\");\n+// you may not use this file except in compliance with the License.\n+// You may obtain a copy of the License at\n+//\n+//     http://www.apache.org/licenses/LICENSE-2.0\n+//\n+// Unless required by applicable law or agreed to in writing, software\n+// distributed under the License is distributed on an \"AS IS\" BASIS,\n+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n+// See the License for the specific language governing permissions and\n+// limitations under the License.\n+\n+#include <assert.h>\n+#include <stdarg.h>  // va_list, etc.\n+#include <stdio.h>  // fread()\n+#include <stdlib.h>  // exit()\n+#include <stdint.h>  // uint16_t\n+#include <string.h>  // strcmp()\n+#include <cmath>  // std::abs operates on doubles\n+#include <cstdlib>  // strtol\n+#include <string>\n+#include <vector>\n+\n+using std::string;\n+using std::vector;\n+\n+// Log messages to stdout.\n+void log(const char* fmt, ...) {\n+  va_list args;\n+  va_start(args, fmt);\n+  vprintf(fmt, args);\n+  va_end(args);\n+  printf(\"\\n\");\n+}\n+\n+const int kTagLen = 4;  // 4 byte tags in the file format\n+\n+bool ExpectTag(FILE* f, const char* tag) {\n+  char buf[kTagLen];\n+\n+  if (fread(buf, sizeof buf[0], kTagLen, f) != kTagLen) {\n+    return false;\n+  }\n+  if (strcmp(buf, tag) != 0) {\n+    log(\"Error: expected '%s'\", tag);\n+    return false;\n+  }\n+  return true;\n+}\n+\n+static bool ReadListOfMatrices(\n+    FILE* f, uint32_t* num_entries_out, uint32_t* entry_size_out,\n+    vector<double>* v_out) {\n+  if (!ExpectTag(f, \"ne \")) {\n+    return false;\n+  }\n+\n+  // R integers are serialized as uint32_t\n+  uint32_t num_entries;\n+  if (fread(&num_entries, sizeof num_entries, 1, f) != 1) {\n+    return false;\n+  }\n+\n+  log(\"num entries: %d\", num_entries);\n+\n+  if (!ExpectTag(f, \"es \")) {\n+    return false;\n+  }\n+\n+  uint32_t entry_size;\n+  if (fread(&entry_size, sizeof entry_size, 1, f) != 1) {\n+    return false;\n+  }\n+  log(\"entry_size: %d\", entry_size);\n+\n+  if (!ExpectTag(f, \"dat\")) {\n+    return false;\n+  }\n+\n+  // Now read dynamic data\n+  uint32_t vec_length = num_entries * entry_size;","path":"analysis/cpp/fast_em.cc","position":83,"original_position":83,"commit_id":"663d5af77c0b1b9e6c8ebe6535849399bf0d3219","original_commit_id":"663d5af77c0b1b9e6c8ebe6535849399bf0d3219","user":{"login":"tkaitchuck","id":4756752,"avatar_url":"https://avatars.githubusercontent.com/u/4756752?v=3","gravatar_id":"","url":"https://api.github.com/users/tkaitchuck","html_url":"https://github.com/tkaitchuck","followers_url":"https://api.github.com/users/tkaitchuck/followers","following_url":"https://api.github.com/users/tkaitchuck/following{/other_user}","gists_url":"https://api.github.com/users/tkaitchuck/gists{/gist_id}","starred_url":"https://api.github.com/users/tkaitchuck/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tkaitchuck/subscriptions","organizations_url":"https://api.github.com/users/tkaitchuck/orgs","repos_url":"https://api.github.com/users/tkaitchuck/repos","events_url":"https://api.github.com/users/tkaitchuck/events{/privacy}","received_events_url":"https://api.github.com/users/tkaitchuck/received_events","type":"User","site_admin":false},"body":"You should really put bounds on these things so there is no possibility of wrapping here.","created_at":"2015-11-02T23:52:28Z","updated_at":"2015-11-02T23:52:28Z","html_url":"https://github.com/google/rappor/pull/52#discussion_r43700946","pull_request_url":"https://api.github.com/repos/google/rappor/pulls/52","_links":{"self":{"href":"https://api.github.com/repos/google/rappor/pulls/comments/43700946"},"html":{"href":"https://github.com/google/rappor/pull/52#discussion_r43700946"},"pull_request":{"href":"https://api.github.com/repos/google/rappor/pulls/52"}}},"pull_request":{"url":"https://api.github.com/repos/google/rappor/pulls/52","id":49201491,"html_url":"https://github.com/google/rappor/pull/52","diff_url":"https://github.com/google/rappor/pull/52.diff","patch_url":"https://github.com/google/rappor/pull/52.patch","issue_url":"https://api.github.com/repos/google/rappor/issues/52","number":52,"state":"open","locked":false,"title":"Optimization of association analysis.  Command line interface.  Bug fixes.  Code cleanup.","user":{"login":"andychu","id":143248,"avatar_url":"https://avatars.githubusercontent.com/u/143248?v=3","gravatar_id":"","url":"https://api.github.com/users/andychu","html_url":"https://github.com/andychu","followers_url":"https://api.github.com/users/andychu/followers","following_url":"https://api.github.com/users/andychu/following{/other_user}","gists_url":"https://api.github.com/users/andychu/gists{/gist_id}","starred_url":"https://api.github.com/users/andychu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andychu/subscriptions","organizations_url":"https://api.github.com/users/andychu/orgs","repos_url":"https://api.github.com/users/andychu/repos","events_url":"https://api.github.com/users/andychu/events{/privacy}","received_events_url":"https://api.github.com/users/andychu/received_events","type":"User","site_admin":false},"body":"- Added bin/decode-assoc command line tool for automation.  Define and read a new schema so we know whether variables are string or boolean, and what their encoding parameters are.\r\n- bin/test.sh: Add tests for bin/decode-assoc, for (string x boolean) problem\r\n  - Add Boolean RAPPOR encoding  (no hashing step) to the Python client with Encoder.encode_bits\r\n  - Add association testdata generation to rappor_sim.py\r\n\r\n- Optimizations\r\n  - In association.R, remove from GetCondProb() any computation that doesn't depend on the report.  (e.g. 100x-1000x speedup for the joint conditional stage)\r\n  - Use mclapply in R for the steps that did lapply() over all reports (N).  The number of cores is controlled by decode-assoc --num-cores.\r\n  - Provide an alternative implementation of EM in C++ in analysis/cpp/fast_em.cc (analysis/R/fast_em.R is a wrapper that does serialization and a system() call)  ~100x speedup.\r\n\r\n- Fixed bugs in the k=1 case, due to R matrix/vector confusion (adapted from Ananth's changes)\r\n- Allow different variables in association analysis to have different parameters (adapted from Ananth's changes, but tests still pass)\r\n- Add hacky support for boolean vars (adapted from Ananth's changes)\r\n\r\n- Code cleanup in association.R.  Rename variables to be more clear.\r\n- Minor refactoring of decode_dist.R to resemble decode_assoc.R\r\n\r\n","created_at":"2015-10-29T21:19:25Z","updated_at":"2015-11-02T23:52:28Z","closed_at":null,"merged_at":null,"merge_commit_sha":"ce2445fc5f458abb034bbbd614076ad02cd40754","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/google/rappor/pulls/52/commits","review_comments_url":"https://api.github.com/repos/google/rappor/pulls/52/comments","review_comment_url":"https://api.github.com/repos/google/rappor/pulls/comments{/number}","comments_url":"https://api.github.com/repos/google/rappor/issues/52/comments","statuses_url":"https://api.github.com/repos/google/rappor/statuses/663d5af77c0b1b9e6c8ebe6535849399bf0d3219","head":{"label":"google:feature/assoc-interface","ref":"feature/assoc-interface","sha":"663d5af77c0b1b9e6c8ebe6535849399bf0d3219","user":{"login":"google","id":1342004,"avatar_url":"https://avatars.githubusercontent.com/u/1342004?v=3","gravatar_id":"","url":"https://api.github.com/users/google","html_url":"https://github.com/google","followers_url":"https://api.github.com/users/google/followers","following_url":"https://api.github.com/users/google/following{/other_user}","gists_url":"https://api.github.com/users/google/gists{/gist_id}","starred_url":"https://api.github.com/users/google/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/google/subscriptions","organizations_url":"https://api.github.com/users/google/orgs","repos_url":"https://api.github.com/users/google/repos","events_url":"https://api.github.com/users/google/events{/privacy}","received_events_url":"https://api.github.com/users/google/received_events","type":"Organization","site_admin":false},"repo":{"id":25319462,"name":"rappor","full_name":"google/rappor","owner":{"login":"google","id":1342004,"avatar_url":"https://avatars.githubusercontent.com/u/1342004?v=3","gravatar_id":"","url":"https://api.github.com/users/google","html_url":"https://github.com/google","followers_url":"https://api.github.com/users/google/followers","following_url":"https://api.github.com/users/google/following{/other_user}","gists_url":"https://api.github.com/users/google/gists{/gist_id}","starred_url":"https://api.github.com/users/google/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/google/subscriptions","organizations_url":"https://api.github.com/users/google/orgs","repos_url":"https://api.github.com/users/google/repos","events_url":"https://api.github.com/users/google/events{/privacy}","received_events_url":"https://api.github.com/users/google/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/google/rappor","description":"RAPPOR: Privacy-Preserving Reporting Algorithms","fork":false,"url":"https://api.github.com/repos/google/rappor","forks_url":"https://api.github.com/repos/google/rappor/forks","keys_url":"https://api.github.com/repos/google/rappor/keys{/key_id}","collaborators_url":"https://api.github.com/repos/google/rappor/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/google/rappor/teams","hooks_url":"https://api.github.com/repos/google/rappor/hooks","issue_events_url":"https://api.github.com/repos/google/rappor/issues/events{/number}","events_url":"https://api.github.com/repos/google/rappor/events","assignees_url":"https://api.github.com/repos/google/rappor/assignees{/user}","branches_url":"https://api.github.com/repos/google/rappor/branches{/branch}","tags_url":"https://api.github.com/repos/google/rappor/tags","blobs_url":"https://api.github.com/repos/google/rappor/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/google/rappor/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/google/rappor/git/refs{/sha}","trees_url":"https://api.github.com/repos/google/rappor/git/trees{/sha}","statuses_url":"https://api.github.com/repos/google/rappor/statuses/{sha}","languages_url":"https://api.github.com/repos/google/rappor/languages","stargazers_url":"https://api.github.com/repos/google/rappor/stargazers","contributors_url":"https://api.github.com/repos/google/rappor/contributors","subscribers_url":"https://api.github.com/repos/google/rappor/subscribers","subscription_url":"https://api.github.com/repos/google/rappor/subscription","commits_url":"https://api.github.com/repos/google/rappor/commits{/sha}","git_commits_url":"https://api.github.com/repos/google/rappor/git/commits{/sha}","comments_url":"https://api.github.com/repos/google/rappor/comments{/number}","issue_comment_url":"https://api.github.com/repos/google/rappor/issues/comments{/number}","contents_url":"https://api.github.com/repos/google/rappor/contents/{+path}","compare_url":"https://api.github.com/repos/google/rappor/compare/{base}...{head}","merges_url":"https://api.github.com/repos/google/rappor/merges","archive_url":"https://api.github.com/repos/google/rappor/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/google/rappor/downloads","issues_url":"https://api.github.com/repos/google/rappor/issues{/number}","pulls_url":"https://api.github.com/repos/google/rappor/pulls{/number}","milestones_url":"https://api.github.com/repos/google/rappor/milestones{/number}","notifications_url":"https://api.github.com/repos/google/rappor/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/google/rappor/labels{/name}","releases_url":"https://api.github.com/repos/google/rappor/releases{/id}","created_at":"2014-10-16T20:05:27Z","updated_at":"2015-10-29T22:25:58Z","pushed_at":"2015-11-02T22:19:26Z","git_url":"git://github.com/google/rappor.git","ssh_url":"git@github.com:google/rappor.git","clone_url":"https://github.com/google/rappor.git","svn_url":"https://github.com/google/rappor","homepage":"","size":2897,"stargazers_count":142,"watchers_count":142,"language":"R","has_issues":true,"has_downloads":false,"has_wiki":false,"has_pages":true,"forks_count":26,"mirror_url":null,"open_issues_count":23,"forks":26,"open_issues":23,"watchers":142,"default_branch":"master"}},"base":{"label":"google:master","ref":"master","sha":"82d269fbf716a95560cd3cbf790c13f37731e32e","user":{"login":"google","id":1342004,"avatar_url":"https://avatars.githubusercontent.com/u/1342004?v=3","gravatar_id":"","url":"https://api.github.com/users/google","html_url":"https://github.com/google","followers_url":"https://api.github.com/users/google/followers","following_url":"https://api.github.com/users/google/following{/other_user}","gists_url":"https://api.github.com/users/google/gists{/gist_id}","starred_url":"https://api.github.com/users/google/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/google/subscriptions","organizations_url":"https://api.github.com/users/google/orgs","repos_url":"https://api.github.com/users/google/repos","events_url":"https://api.github.com/users/google/events{/privacy}","received_events_url":"https://api.github.com/users/google/received_events","type":"Organization","site_admin":false},"repo":{"id":25319462,"name":"rappor","full_name":"google/rappor","owner":{"login":"google","id":1342004,"avatar_url":"https://avatars.githubusercontent.com/u/1342004?v=3","gravatar_id":"","url":"https://api.github.com/users/google","html_url":"https://github.com/google","followers_url":"https://api.github.com/users/google/followers","following_url":"https://api.github.com/users/google/following{/other_user}","gists_url":"https://api.github.com/users/google/gists{/gist_id}","starred_url":"https://api.github.com/users/google/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/google/subscriptions","organizations_url":"https://api.github.com/users/google/orgs","repos_url":"https://api.github.com/users/google/repos","events_url":"https://api.github.com/users/google/events{/privacy}","received_events_url":"https://api.github.com/users/google/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/google/rappor","description":"RAPPOR: Privacy-Preserving Reporting Algorithms","fork":false,"url":"https://api.github.com/repos/google/rappor","forks_url":"https://api.github.com/repos/google/rappor/forks","keys_url":"https://api.github.com/repos/google/rappor/keys{/key_id}","collaborators_url":"https://api.github.com/repos/google/rappor/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/google/rappor/teams","hooks_url":"https://api.github.com/repos/google/rappor/hooks","issue_events_url":"https://api.github.com/repos/google/rappor/issues/events{/number}","events_url":"https://api.github.com/repos/google/rappor/events","assignees_url":"https://api.github.com/repos/google/rappor/assignees{/user}","branches_url":"https://api.github.com/repos/google/rappor/branches{/branch}","tags_url":"https://api.github.com/repos/google/rappor/tags","blobs_url":"https://api.github.com/repos/google/rappor/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/google/rappor/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/google/rappor/git/refs{/sha}","trees_url":"https://api.github.com/repos/google/rappor/git/trees{/sha}","statuses_url":"https://api.github.com/repos/google/rappor/statuses/{sha}","languages_url":"https://api.github.com/repos/google/rappor/languages","stargazers_url":"https://api.github.com/repos/google/rappor/stargazers","contributors_url":"https://api.github.com/repos/google/rappor/contributors","subscribers_url":"https://api.github.com/repos/google/rappor/subscribers","subscription_url":"https://api.github.com/repos/google/rappor/subscription","commits_url":"https://api.github.com/repos/google/rappor/commits{/sha}","git_commits_url":"https://api.github.com/repos/google/rappor/git/commits{/sha}","comments_url":"https://api.github.com/repos/google/rappor/comments{/number}","issue_comment_url":"https://api.github.com/repos/google/rappor/issues/comments{/number}","contents_url":"https://api.github.com/repos/google/rappor/contents/{+path}","compare_url":"https://api.github.com/repos/google/rappor/compare/{base}...{head}","merges_url":"https://api.github.com/repos/google/rappor/merges","archive_url":"https://api.github.com/repos/google/rappor/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/google/rappor/downloads","issues_url":"https://api.github.com/repos/google/rappor/issues{/number}","pulls_url":"https://api.github.com/repos/google/rappor/pulls{/number}","milestones_url":"https://api.github.com/repos/google/rappor/milestones{/number}","notifications_url":"https://api.github.com/repos/google/rappor/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/google/rappor/labels{/name}","releases_url":"https://api.github.com/repos/google/rappor/releases{/id}","created_at":"2014-10-16T20:05:27Z","updated_at":"2015-10-29T22:25:58Z","pushed_at":"2015-11-02T22:19:26Z","git_url":"git://github.com/google/rappor.git","ssh_url":"git@github.com:google/rappor.git","clone_url":"https://github.com/google/rappor.git","svn_url":"https://github.com/google/rappor","homepage":"","size":2897,"stargazers_count":142,"watchers_count":142,"language":"R","has_issues":true,"has_downloads":false,"has_wiki":false,"has_pages":true,"forks_count":26,"mirror_url":null,"open_issues_count":23,"forks":26,"open_issues":23,"watchers":142,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/google/rappor/pulls/52"},"html":{"href":"https://github.com/google/rappor/pull/52"},"issue":{"href":"https://api.github.com/repos/google/rappor/issues/52"},"comments":{"href":"https://api.github.com/repos/google/rappor/issues/52/comments"},"review_comments":{"href":"https://api.github.com/repos/google/rappor/pulls/52/comments"},"review_comment":{"href":"https://api.github.com/repos/google/rappor/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/google/rappor/pulls/52/commits"},"statuses":{"href":"https://api.github.com/repos/google/rappor/statuses/663d5af77c0b1b9e6c8ebe6535849399bf0d3219"}}}}