{"action":"created","comment":{"url":"https://api.github.com/repos/ga4gh/server/pulls/comments/26960010","id":26960010,"diff_hunk":"@@ -23,6 +24,63 @@ def convertDatetime(t):\n     return int(millis)\n \n \n+class SearchResponseBuffer(object):\n+    \"\"\"\n+    A buffer to hold the string representation of protocol Response\n+    objects. This allows us to sequentially build responses, and\n+    reduces the number of live objects required to build response\n+    objects.\n+    \"\"\"\n+    def __init__(self, pageListName, pageSize, maxResponseLength=None):\n+        self._pageListName = pageListName\n+        self._pageSize = pageSize\n+        self._maxResponseLength = 2**32","path":"ga4gh/protocol.py","position":22,"original_position":22,"commit_id":"e848e3f6a197a7b40cc3dd0fdbbd7ec520771458","original_commit_id":"e848e3f6a197a7b40cc3dd0fdbbd7ec520771458","user":{"login":"jeromekelleher","id":2664569,"avatar_url":"https://avatars.githubusercontent.com/u/2664569?v=3","gravatar_id":"","url":"https://api.github.com/users/jeromekelleher","html_url":"https://github.com/jeromekelleher","followers_url":"https://api.github.com/users/jeromekelleher/followers","following_url":"https://api.github.com/users/jeromekelleher/following{/other_user}","gists_url":"https://api.github.com/users/jeromekelleher/gists{/gist_id}","starred_url":"https://api.github.com/users/jeromekelleher/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jeromekelleher/subscriptions","organizations_url":"https://api.github.com/users/jeromekelleher/orgs","repos_url":"https://api.github.com/users/jeromekelleher/repos","events_url":"https://api.github.com/users/jeromekelleher/events{/privacy}","received_events_url":"https://api.github.com/users/jeromekelleher/received_events","type":"User","site_admin":false},"body":"Yeah, this is a bit of a hack. I'm just trying to cover the case when someone wants to make a `SearchResponseBuffer` but doesn't care about the `maxResponseLength`. I'll  just remove the default I think, it doesn't really help.","created_at":"2015-03-23T17:42:16Z","updated_at":"2015-03-23T17:42:16Z","html_url":"https://github.com/ga4gh/server/pull/276#discussion_r26960010","pull_request_url":"https://api.github.com/repos/ga4gh/server/pulls/276","_links":{"self":{"href":"https://api.github.com/repos/ga4gh/server/pulls/comments/26960010"},"html":{"href":"https://github.com/ga4gh/server/pull/276#discussion_r26960010"},"pull_request":{"href":"https://api.github.com/repos/ga4gh/server/pulls/276"}}},"pull_request":{"url":"https://api.github.com/repos/ga4gh/server/pulls/276","id":31749998,"html_url":"https://github.com/ga4gh/server/pull/276","diff_url":"https://github.com/ga4gh/server/pull/276.diff","patch_url":"https://github.com/ga4gh/server/pull/276.patch","issue_url":"https://api.github.com/repos/ga4gh/server/issues/276","number":276,"state":"open","locked":false,"title":"Adaptive pagesize and max request size.","user":{"login":"jeromekelleher","id":2664569,"avatar_url":"https://avatars.githubusercontent.com/u/2664569?v=3","gravatar_id":"","url":"https://api.github.com/users/jeromekelleher","html_url":"https://github.com/jeromekelleher","followers_url":"https://api.github.com/users/jeromekelleher/followers","following_url":"https://api.github.com/users/jeromekelleher/following{/other_user}","gists_url":"https://api.github.com/users/jeromekelleher/gists{/gist_id}","starred_url":"https://api.github.com/users/jeromekelleher/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jeromekelleher/subscriptions","organizations_url":"https://api.github.com/users/jeromekelleher/orgs","repos_url":"https://api.github.com/users/jeromekelleher/repos","events_url":"https://api.github.com/users/jeromekelleher/events{/privacy}","received_events_url":"https://api.github.com/users/jeromekelleher/received_events","type":"User","site_admin":false},"body":"There are some serious issues with the current paging implementation:\r\n\r\n1. It is trivial to perform a DOS/memory starvation attack. Simply provide a very large page size, and we will blindly try to fill it up until memory runs out.\r\n2. There is no easy way to control latency. The only knob we have is the pageSize, but this is very crude as not all obejcts are equal. The pageSize controls the number of objects a page contains, _not_ the absolute size of the page in terms of bytes. Since objects like `Variant` can contain an arbitrarily large number of embedded objects (`Call` in the case of `Variant`) there's no way to know just how much memory we require.\r\n\r\nProblem (1) could conceivably be tackled just by using the pageSize, but it would always be slightly tricky (what if there was 100k samples in the VCF?).  We could specify a MAX_PAGE_SIZE configuration variable, for example, but an administrator would have to work out roughly how much memory each object would require in worst-case and then figure out how many of these would correspond to a reasonable amount of memory. This puts a burden on the server admin, and is error prone. Problem (2) could also be tackled using this approach, but it's also quite tricky and error prone.\r\n\r\nThis is all a bit of a waste of time though, since all we really want to do is put a limit on the amount of *memory* each request is allowed to consume. This has the effect of solving (1) immediately, since an administrator can effectively cap the memory consumption per request. We also solve (2) which is good  for clients also, since they no longer need to worry about setting the appropriate page size to handle latency. The client knows they will be sent back appropriate sized packets of data, and does not need to worry about how to calculate the best page size for a given type of data. \r\n\r\nThis PR adds a `MAX_RESPONSE_LENGTH` configuration variable, and sequentially encodes the JSON. Below we compare the time and memory requirements for requesting all calls in a range of 1000G data using a standard page size of 100 in the current master, and with a `MAX_RESPONSE_SIZE` of 1M in the current PR. \r\n\r\nBefore:\r\n```\r\n$ /usr/bin/time -v python server_benchmark.py -c'*' --repeatLimit=1 --pageLimit=1000 1000g_2011.wt\r\n31.87\r\n        Command being timed: \"python server_benchmark.py -c* --repeatLimit=1 --pageLimit=1000 1000g_2011.wt\"\r\n        User time (seconds): 32.13\r\n        System time (seconds): 0.61\r\n        Percent of CPU this job got: 99%\r\n        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:32.81\r\n        Average shared text size (kbytes): 0\r\n        Average unshared data size (kbytes): 0\r\n        Average stack size (kbytes): 0\r\n        Average total size (kbytes): 0\r\n        Maximum resident set size (kbytes): 422648\r\n        Average resident set size (kbytes): 0\r\n        Major (requiring I/O) page faults: 0\r\n        Minor (reclaiming a frame) page faults: 258505\r\n        Voluntary context switches: 1\r\n        Involuntary context switches: 64\r\n        Swaps: 0\r\n        File system inputs: 0\r\n        File system outputs: 48\r\n        Socket messages sent: 0\r\n        Socket messages received: 0\r\n        Signals delivered: 0\r\n        Page size (bytes): 4096\r\n        Exit status: 0\r\n```\r\nAfter:\r\n```\r\n$ /usr/bin/time -v python server_benchmark.py -c'*' --repeatLimit=1 --pageLimit=1000 1000g_2011.wt\r\n20.69\r\n        Command being timed: \"python server_benchmark.py -c* --repeatLimit=1 --pageLimit=1000 1000g_2011.wt\"\r\n        User time (seconds): 21.00\r\n        System time (seconds): 0.56\r\n        Percent of CPU this job got: 99%\r\n        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:21.61\r\n        Average shared text size (kbytes): 0\r\n        Average unshared data size (kbytes): 0\r\n        Average stack size (kbytes): 0\r\n        Average total size (kbytes): 0\r\n        Maximum resident set size (kbytes): 75784\r\n        Average resident set size (kbytes): 0\r\n        Major (requiring I/O) page faults: 0\r\n        Minor (reclaiming a frame) page faults: 251632\r\n        Voluntary context switches: 1\r\n        Involuntary context switches: 62\r\n        Swaps: 0\r\n        File system inputs: 0\r\n        File system outputs: 0\r\n        Socket messages sent: 0\r\n        Socket messages received: 0\r\n        Signals delivered: 0\r\n        Page size (bytes): 4096\r\n        Exit status: 0\r\n```\r\n\r\nWe see we have reduced the CPU time by about 1/3 and reduced the memory footprint from ~420M to ~75M. This would seem like a clear win, and you might be wondering why I'm going to so much trouble to justify this change. There was a long discussion on this in #10, and there is the question of whether we should be making optimisations like this in the reference implementation.\r\n\r\nSo, pros:\r\n1. We are (more) secure against memory starvation attacks.\r\n2. We can control end-to-end latency much better.\r\n3. Admins can get much better control over the memory usage of servers.\r\n4. We are substantially faster and use less memory (over default page sizes on 1000G data).\r\n\r\nCons:\r\n1. We break layering a little, and have some hand-written JSON encoding.\r\n\r\nThis PR is a proposal, and might be controversial. If people generally like it I'll update with some test cases for the new functionality.","created_at":"2015-03-23T16:40:07Z","updated_at":"2015-03-23T17:42:16Z","closed_at":null,"merged_at":null,"merge_commit_sha":"c1cca4d685726f7c169bba38a47213e3ae7b0c24","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/ga4gh/server/pulls/276/commits","review_comments_url":"https://api.github.com/repos/ga4gh/server/pulls/276/comments","review_comment_url":"https://api.github.com/repos/ga4gh/server/pulls/comments{/number}","comments_url":"https://api.github.com/repos/ga4gh/server/issues/276/comments","statuses_url":"https://api.github.com/repos/ga4gh/server/statuses/e848e3f6a197a7b40cc3dd0fdbbd7ec520771458","head":{"label":"jeromekelleher:adaptive-pagesize","ref":"adaptive-pagesize","sha":"e848e3f6a197a7b40cc3dd0fdbbd7ec520771458","user":{"login":"jeromekelleher","id":2664569,"avatar_url":"https://avatars.githubusercontent.com/u/2664569?v=3","gravatar_id":"","url":"https://api.github.com/users/jeromekelleher","html_url":"https://github.com/jeromekelleher","followers_url":"https://api.github.com/users/jeromekelleher/followers","following_url":"https://api.github.com/users/jeromekelleher/following{/other_user}","gists_url":"https://api.github.com/users/jeromekelleher/gists{/gist_id}","starred_url":"https://api.github.com/users/jeromekelleher/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jeromekelleher/subscriptions","organizations_url":"https://api.github.com/users/jeromekelleher/orgs","repos_url":"https://api.github.com/users/jeromekelleher/repos","events_url":"https://api.github.com/users/jeromekelleher/events{/privacy}","received_events_url":"https://api.github.com/users/jeromekelleher/received_events","type":"User","site_admin":false},"repo":{"id":22882845,"name":"server","full_name":"jeromekelleher/server","owner":{"login":"jeromekelleher","id":2664569,"avatar_url":"https://avatars.githubusercontent.com/u/2664569?v=3","gravatar_id":"","url":"https://api.github.com/users/jeromekelleher","html_url":"https://github.com/jeromekelleher","followers_url":"https://api.github.com/users/jeromekelleher/followers","following_url":"https://api.github.com/users/jeromekelleher/following{/other_user}","gists_url":"https://api.github.com/users/jeromekelleher/gists{/gist_id}","starred_url":"https://api.github.com/users/jeromekelleher/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jeromekelleher/subscriptions","organizations_url":"https://api.github.com/users/jeromekelleher/orgs","repos_url":"https://api.github.com/users/jeromekelleher/repos","events_url":"https://api.github.com/users/jeromekelleher/events{/privacy}","received_events_url":"https://api.github.com/users/jeromekelleher/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/jeromekelleher/server","description":"A reference implementation of the APIs defined in the schemas repository.","fork":true,"url":"https://api.github.com/repos/jeromekelleher/server","forks_url":"https://api.github.com/repos/jeromekelleher/server/forks","keys_url":"https://api.github.com/repos/jeromekelleher/server/keys{/key_id}","collaborators_url":"https://api.github.com/repos/jeromekelleher/server/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/jeromekelleher/server/teams","hooks_url":"https://api.github.com/repos/jeromekelleher/server/hooks","issue_events_url":"https://api.github.com/repos/jeromekelleher/server/issues/events{/number}","events_url":"https://api.github.com/repos/jeromekelleher/server/events","assignees_url":"https://api.github.com/repos/jeromekelleher/server/assignees{/user}","branches_url":"https://api.github.com/repos/jeromekelleher/server/branches{/branch}","tags_url":"https://api.github.com/repos/jeromekelleher/server/tags","blobs_url":"https://api.github.com/repos/jeromekelleher/server/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/jeromekelleher/server/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/jeromekelleher/server/git/refs{/sha}","trees_url":"https://api.github.com/repos/jeromekelleher/server/git/trees{/sha}","statuses_url":"https://api.github.com/repos/jeromekelleher/server/statuses/{sha}","languages_url":"https://api.github.com/repos/jeromekelleher/server/languages","stargazers_url":"https://api.github.com/repos/jeromekelleher/server/stargazers","contributors_url":"https://api.github.com/repos/jeromekelleher/server/contributors","subscribers_url":"https://api.github.com/repos/jeromekelleher/server/subscribers","subscription_url":"https://api.github.com/repos/jeromekelleher/server/subscription","commits_url":"https://api.github.com/repos/jeromekelleher/server/commits{/sha}","git_commits_url":"https://api.github.com/repos/jeromekelleher/server/git/commits{/sha}","comments_url":"https://api.github.com/repos/jeromekelleher/server/comments{/number}","issue_comment_url":"https://api.github.com/repos/jeromekelleher/server/issues/comments{/number}","contents_url":"https://api.github.com/repos/jeromekelleher/server/contents/{+path}","compare_url":"https://api.github.com/repos/jeromekelleher/server/compare/{base}...{head}","merges_url":"https://api.github.com/repos/jeromekelleher/server/merges","archive_url":"https://api.github.com/repos/jeromekelleher/server/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/jeromekelleher/server/downloads","issues_url":"https://api.github.com/repos/jeromekelleher/server/issues{/number}","pulls_url":"https://api.github.com/repos/jeromekelleher/server/pulls{/number}","milestones_url":"https://api.github.com/repos/jeromekelleher/server/milestones{/number}","notifications_url":"https://api.github.com/repos/jeromekelleher/server/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/jeromekelleher/server/labels{/name}","releases_url":"https://api.github.com/repos/jeromekelleher/server/releases{/id}","created_at":"2014-08-12T15:39:03Z","updated_at":"2015-02-12T09:23:10Z","pushed_at":"2015-03-23T16:39:57Z","git_url":"git://github.com/jeromekelleher/server.git","ssh_url":"git@github.com:jeromekelleher/server.git","clone_url":"https://github.com/jeromekelleher/server.git","svn_url":"https://github.com/jeromekelleher/server","homepage":null,"size":2388,"stargazers_count":1,"watchers_count":1,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":1,"default_branch":"master"}},"base":{"label":"ga4gh:master","ref":"master","sha":"86671a9a8e91d8d6152269bf80e64f87abc5c2e9","user":{"login":"ga4gh","id":6921777,"avatar_url":"https://avatars.githubusercontent.com/u/6921777?v=3","gravatar_id":"","url":"https://api.github.com/users/ga4gh","html_url":"https://github.com/ga4gh","followers_url":"https://api.github.com/users/ga4gh/followers","following_url":"https://api.github.com/users/ga4gh/following{/other_user}","gists_url":"https://api.github.com/users/ga4gh/gists{/gist_id}","starred_url":"https://api.github.com/users/ga4gh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ga4gh/subscriptions","organizations_url":"https://api.github.com/users/ga4gh/orgs","repos_url":"https://api.github.com/users/ga4gh/repos","events_url":"https://api.github.com/users/ga4gh/events{/privacy}","received_events_url":"https://api.github.com/users/ga4gh/received_events","type":"Organization","site_admin":false},"repo":{"id":22882621,"name":"server","full_name":"ga4gh/server","owner":{"login":"ga4gh","id":6921777,"avatar_url":"https://avatars.githubusercontent.com/u/6921777?v=3","gravatar_id":"","url":"https://api.github.com/users/ga4gh","html_url":"https://github.com/ga4gh","followers_url":"https://api.github.com/users/ga4gh/followers","following_url":"https://api.github.com/users/ga4gh/following{/other_user}","gists_url":"https://api.github.com/users/ga4gh/gists{/gist_id}","starred_url":"https://api.github.com/users/ga4gh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ga4gh/subscriptions","organizations_url":"https://api.github.com/users/ga4gh/orgs","repos_url":"https://api.github.com/users/ga4gh/repos","events_url":"https://api.github.com/users/ga4gh/events{/privacy}","received_events_url":"https://api.github.com/users/ga4gh/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/ga4gh/server","description":"A reference implementation of the APIs defined in the schemas repository.","fork":false,"url":"https://api.github.com/repos/ga4gh/server","forks_url":"https://api.github.com/repos/ga4gh/server/forks","keys_url":"https://api.github.com/repos/ga4gh/server/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ga4gh/server/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ga4gh/server/teams","hooks_url":"https://api.github.com/repos/ga4gh/server/hooks","issue_events_url":"https://api.github.com/repos/ga4gh/server/issues/events{/number}","events_url":"https://api.github.com/repos/ga4gh/server/events","assignees_url":"https://api.github.com/repos/ga4gh/server/assignees{/user}","branches_url":"https://api.github.com/repos/ga4gh/server/branches{/branch}","tags_url":"https://api.github.com/repos/ga4gh/server/tags","blobs_url":"https://api.github.com/repos/ga4gh/server/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ga4gh/server/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ga4gh/server/git/refs{/sha}","trees_url":"https://api.github.com/repos/ga4gh/server/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ga4gh/server/statuses/{sha}","languages_url":"https://api.github.com/repos/ga4gh/server/languages","stargazers_url":"https://api.github.com/repos/ga4gh/server/stargazers","contributors_url":"https://api.github.com/repos/ga4gh/server/contributors","subscribers_url":"https://api.github.com/repos/ga4gh/server/subscribers","subscription_url":"https://api.github.com/repos/ga4gh/server/subscription","commits_url":"https://api.github.com/repos/ga4gh/server/commits{/sha}","git_commits_url":"https://api.github.com/repos/ga4gh/server/git/commits{/sha}","comments_url":"https://api.github.com/repos/ga4gh/server/comments{/number}","issue_comment_url":"https://api.github.com/repos/ga4gh/server/issues/comments{/number}","contents_url":"https://api.github.com/repos/ga4gh/server/contents/{+path}","compare_url":"https://api.github.com/repos/ga4gh/server/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ga4gh/server/merges","archive_url":"https://api.github.com/repos/ga4gh/server/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ga4gh/server/downloads","issues_url":"https://api.github.com/repos/ga4gh/server/issues{/number}","pulls_url":"https://api.github.com/repos/ga4gh/server/pulls{/number}","milestones_url":"https://api.github.com/repos/ga4gh/server/milestones{/number}","notifications_url":"https://api.github.com/repos/ga4gh/server/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ga4gh/server/labels{/name}","releases_url":"https://api.github.com/repos/ga4gh/server/releases{/id}","created_at":"2014-08-12T15:31:49Z","updated_at":"2015-03-23T09:59:10Z","pushed_at":"2015-03-23T17:17:24Z","git_url":"git://github.com/ga4gh/server.git","ssh_url":"git@github.com:ga4gh/server.git","clone_url":"https://github.com/ga4gh/server.git","svn_url":"https://github.com/ga4gh/server","homepage":null,"size":4541,"stargazers_count":26,"watchers_count":26,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":26,"mirror_url":null,"open_issues_count":41,"forks":26,"open_issues":41,"watchers":26,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/ga4gh/server/pulls/276"},"html":{"href":"https://github.com/ga4gh/server/pull/276"},"issue":{"href":"https://api.github.com/repos/ga4gh/server/issues/276"},"comments":{"href":"https://api.github.com/repos/ga4gh/server/issues/276/comments"},"review_comments":{"href":"https://api.github.com/repos/ga4gh/server/pulls/276/comments"},"review_comment":{"href":"https://api.github.com/repos/ga4gh/server/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/ga4gh/server/pulls/276/commits"},"statuses":{"href":"https://api.github.com/repos/ga4gh/server/statuses/e848e3f6a197a7b40cc3dd0fdbbd7ec520771458"}}}}