{"action":"created","comment":{"url":"https://api.github.com/repos/inliniac/suricata/pulls/comments/31322512","id":31322512,"diff_hunk":"@@ -89,6 +89,117 @@ static inline int HSBDCreateSpace(DetectEngineThreadCtx *det_ctx, uint16_t size)\n     return 0;\n }\n \n+static void HSBDGetBufferForTXInIDSMode(DetectEngineThreadCtx *det_ctx,\n+                                        HtpState *htp_state, HtpBodyChunk *cur,\n+                                        HtpTxUserData *htud, int index)\n+{\n+    int first = 1;\n+    while (cur != NULL) {\n+        /* see if we can filter out chunks */\n+        if (htud->response_body.body_inspected > 0) {\n+            if (cur->stream_offset < htud->response_body.body_inspected) {\n+                if ((htud->response_body.body_inspected - cur->stream_offset) > htp_state->cfg->response_inspect_window) {\n+                    cur = cur->next;\n+                    continue;\n+                } else {\n+                    /* include this one */\n+                }\n+            } else {\n+                /* include this one */\n+            }\n+        }\n+\n+        if (first) {\n+            det_ctx->hsbd[index].offset = cur->stream_offset;\n+            first = 0;\n+        }\n+\n+        /* see if we need to grow the buffer */\n+        if (det_ctx->hsbd[index].buffer == NULL || (det_ctx->hsbd[index].buffer_len + cur->len) > det_ctx->hsbd[index].buffer_size) {\n+            void *ptmp;\n+            det_ctx->hsbd[index].buffer_size += cur->len * 2;\n+\n+            if ((ptmp = SCRealloc(det_ctx->hsbd[index].buffer, det_ctx->hsbd[index].buffer_size)) == NULL) {\n+                SCFree(det_ctx->hsbd[index].buffer);\n+                det_ctx->hsbd[index].buffer = NULL;\n+                det_ctx->hsbd[index].buffer_size = 0;\n+                det_ctx->hsbd[index].buffer_len = 0;\n+                return;\n+            }\n+            det_ctx->hsbd[index].buffer = ptmp;\n+        }\n+        memcpy(det_ctx->hsbd[index].buffer + det_ctx->hsbd[index].buffer_len, cur->data, cur->len);\n+        det_ctx->hsbd[index].buffer_len += cur->len;\n+\n+        cur = cur->next;\n+    }\n+\n+    /* update inspected tracker */\n+    htud->response_body.body_inspected = htud->response_body.last->stream_offset + htud->response_body.last->len;\n+}\n+\n+static void HSBDGetBufferForTXInIPSMode(DetectEngineThreadCtx *det_ctx,\n+                                        HtpState *htp_state, HtpBodyChunk *cur,\n+                                        HtpTxUserData *htud, int index)\n+{\n+    uint32_t window_size = 0;\n+    uint32_t extra_size = 0;\n+    int resize = 0;\n+\n+    window_size = htp_state->cfg->response_inspect_window;\n+\n+    int first = 1;\n+    while (cur != NULL) {\n+        /* see if we need to grow the buffer */\n+        if (cur->len > window_size) {\n+            extra_size =  window_size * 0.25;\n+            window_size = cur->len;\n+            resize = 1;\n+        } else if (cur->len == window_size) {\n+            window_size = window_size * 1.25;\n+            resize = 1;\n+        }\n+\n+        if (det_ctx->hsbd[index].buffer == NULL || resize) {\n+            void *ptmp;\n+\n+            if ((ptmp = SCRealloc(det_ctx->hsbd[index].buffer, window_size + extra_size)) == NULL) {\n+                SCFree(det_ctx->hsbd[index].buffer);\n+                det_ctx->hsbd[index].buffer = NULL;\n+                det_ctx->hsbd[index].buffer_size = 0;\n+                det_ctx->hsbd[index].buffer_len = 0;\n+                return;\n+            }\n+            det_ctx->hsbd[index].buffer = ptmp;\n+            resize = 0;\n+        }\n+\n+        if (first) {\n+            det_ctx->hsbd[index].offset = cur->stream_offset;\n+            first = 0;\n+        }\n+\n+        /*\n+         * Copy a part of the previous buffer before the current chunk on.\n+         * For example, let's have the chunks [123], [456].\n+         * When the current chunk is [456], this copy [3] before.\n+         */\n+        if (det_ctx->hsbd[index].buffer_len + cur->len > window_size) {\n+            uint32_t offset = (det_ctx->hsbd[index].buffer_len + cur->len) - (window_size + extra_size);\n+            /* we use memmove to avoid memory overlap */\n+            memmove(det_ctx->hsbd[index].buffer, det_ctx->hsbd[index].buffer + offset, det_ctx->hsbd[index].buffer_len - offset);","path":"src/detect-engine-hsbd.c","position":102,"original_position":102,"commit_id":"79cba7b935e1e3fc7abddc7ef7fbd10a9ad442d5","original_commit_id":"79cba7b935e1e3fc7abddc7ef7fbd10a9ad442d5","user":{"login":"glongo","id":101933,"avatar_url":"https://avatars.githubusercontent.com/u/101933?v=3","gravatar_id":"","url":"https://api.github.com/users/glongo","html_url":"https://github.com/glongo","followers_url":"https://api.github.com/users/glongo/followers","following_url":"https://api.github.com/users/glongo/following{/other_user}","gists_url":"https://api.github.com/users/glongo/gists{/gist_id}","starred_url":"https://api.github.com/users/glongo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/glongo/subscriptions","organizations_url":"https://api.github.com/users/glongo/orgs","repos_url":"https://api.github.com/users/glongo/repos","events_url":"https://api.github.com/users/glongo/events{/privacy}","received_events_url":"https://api.github.com/users/glongo/received_events","type":"User","site_admin":false},"body":"With memcpy, DetectEngineHttpServerBodyFileDataTest07 produces this issue:\r\n==31217== ERROR: AddressSanitizer: memcpy-param-overlap: memory ranges [0x60040001b3b0,0x60040001b3b2) and [0x60040001b3b1, 0x60040001b3b3) overlap\r\nmemmove prevents memory overlap, so fixes it.","created_at":"2015-05-29T13:12:04Z","updated_at":"2015-05-29T13:12:04Z","html_url":"https://github.com/inliniac/suricata/pull/1516#discussion_r31322512","pull_request_url":"https://api.github.com/repos/inliniac/suricata/pulls/1516","_links":{"self":{"href":"https://api.github.com/repos/inliniac/suricata/pulls/comments/31322512"},"html":{"href":"https://github.com/inliniac/suricata/pull/1516#discussion_r31322512"},"pull_request":{"href":"https://api.github.com/repos/inliniac/suricata/pulls/1516"}}},"pull_request":{"url":"https://api.github.com/repos/inliniac/suricata/pulls/1516","id":36500495,"html_url":"https://github.com/inliniac/suricata/pull/1516","diff_url":"https://github.com/inliniac/suricata/pull/1516.diff","patch_url":"https://github.com/inliniac/suricata/pull/1516.patch","issue_url":"https://api.github.com/repos/inliniac/suricata/issues/1516","number":1516,"state":"open","locked":false,"title":"hsbd v1.3: inspect buffer depending on the engine mode","user":{"login":"glongo","id":101933,"avatar_url":"https://avatars.githubusercontent.com/u/101933?v=3","gravatar_id":"","url":"https://api.github.com/users/glongo","html_url":"https://github.com/glongo","followers_url":"https://api.github.com/users/glongo/followers","following_url":"https://api.github.com/users/glongo/following{/other_user}","gists_url":"https://api.github.com/users/glongo/gists{/gist_id}","starred_url":"https://api.github.com/users/glongo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/glongo/subscriptions","organizations_url":"https://api.github.com/users/glongo/orgs","repos_url":"https://api.github.com/users/glongo/repos","events_url":"https://api.github.com/users/glongo/events{/privacy}","received_events_url":"https://api.github.com/users/glongo/received_events","type":"User","site_admin":false},"body":"Currently, data is buffered up to response-body-minimal size and response-body-inspect-window before being inspected.\r\nWith this, in IPS mode, inspect data as it comes in up.\r\n\r\nThis addresses the comments in the last PR: #1515 \r\n\r\n- PR glongo: https://buildbot.openinfosecfoundation.org/builders/glongo/builds/38\r\n- PR glongo-pcap: https://buildbot.openinfosecfoundation.org/builders/glongo-pcap/builds/37\r\n","created_at":"2015-05-29T12:29:57Z","updated_at":"2015-05-29T13:12:04Z","closed_at":null,"merged_at":null,"merge_commit_sha":"e5f97e89069d61962c72ce496b299734b103f7e4","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/inliniac/suricata/pulls/1516/commits","review_comments_url":"https://api.github.com/repos/inliniac/suricata/pulls/1516/comments","review_comment_url":"https://api.github.com/repos/inliniac/suricata/pulls/comments{/number}","comments_url":"https://api.github.com/repos/inliniac/suricata/issues/1516/comments","statuses_url":"https://api.github.com/repos/inliniac/suricata/statuses/79cba7b935e1e3fc7abddc7ef7fbd10a9ad442d5","head":{"label":"glongo:fix-hsbd-filedata-v1.3","ref":"fix-hsbd-filedata-v1.3","sha":"79cba7b935e1e3fc7abddc7ef7fbd10a9ad442d5","user":{"login":"glongo","id":101933,"avatar_url":"https://avatars.githubusercontent.com/u/101933?v=3","gravatar_id":"","url":"https://api.github.com/users/glongo","html_url":"https://github.com/glongo","followers_url":"https://api.github.com/users/glongo/followers","following_url":"https://api.github.com/users/glongo/following{/other_user}","gists_url":"https://api.github.com/users/glongo/gists{/gist_id}","starred_url":"https://api.github.com/users/glongo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/glongo/subscriptions","organizations_url":"https://api.github.com/users/glongo/orgs","repos_url":"https://api.github.com/users/glongo/repos","events_url":"https://api.github.com/users/glongo/events{/privacy}","received_events_url":"https://api.github.com/users/glongo/received_events","type":"User","site_admin":false},"repo":{"id":13040040,"name":"suricata","full_name":"glongo/suricata","owner":{"login":"glongo","id":101933,"avatar_url":"https://avatars.githubusercontent.com/u/101933?v=3","gravatar_id":"","url":"https://api.github.com/users/glongo","html_url":"https://github.com/glongo","followers_url":"https://api.github.com/users/glongo/followers","following_url":"https://api.github.com/users/glongo/following{/other_user}","gists_url":"https://api.github.com/users/glongo/gists{/gist_id}","starred_url":"https://api.github.com/users/glongo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/glongo/subscriptions","organizations_url":"https://api.github.com/users/glongo/orgs","repos_url":"https://api.github.com/users/glongo/repos","events_url":"https://api.github.com/users/glongo/events{/privacy}","received_events_url":"https://api.github.com/users/glongo/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/glongo/suricata","description":"Mirror of the official OISF Suricata git repository","fork":true,"url":"https://api.github.com/repos/glongo/suricata","forks_url":"https://api.github.com/repos/glongo/suricata/forks","keys_url":"https://api.github.com/repos/glongo/suricata/keys{/key_id}","collaborators_url":"https://api.github.com/repos/glongo/suricata/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/glongo/suricata/teams","hooks_url":"https://api.github.com/repos/glongo/suricata/hooks","issue_events_url":"https://api.github.com/repos/glongo/suricata/issues/events{/number}","events_url":"https://api.github.com/repos/glongo/suricata/events","assignees_url":"https://api.github.com/repos/glongo/suricata/assignees{/user}","branches_url":"https://api.github.com/repos/glongo/suricata/branches{/branch}","tags_url":"https://api.github.com/repos/glongo/suricata/tags","blobs_url":"https://api.github.com/repos/glongo/suricata/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/glongo/suricata/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/glongo/suricata/git/refs{/sha}","trees_url":"https://api.github.com/repos/glongo/suricata/git/trees{/sha}","statuses_url":"https://api.github.com/repos/glongo/suricata/statuses/{sha}","languages_url":"https://api.github.com/repos/glongo/suricata/languages","stargazers_url":"https://api.github.com/repos/glongo/suricata/stargazers","contributors_url":"https://api.github.com/repos/glongo/suricata/contributors","subscribers_url":"https://api.github.com/repos/glongo/suricata/subscribers","subscription_url":"https://api.github.com/repos/glongo/suricata/subscription","commits_url":"https://api.github.com/repos/glongo/suricata/commits{/sha}","git_commits_url":"https://api.github.com/repos/glongo/suricata/git/commits{/sha}","comments_url":"https://api.github.com/repos/glongo/suricata/comments{/number}","issue_comment_url":"https://api.github.com/repos/glongo/suricata/issues/comments{/number}","contents_url":"https://api.github.com/repos/glongo/suricata/contents/{+path}","compare_url":"https://api.github.com/repos/glongo/suricata/compare/{base}...{head}","merges_url":"https://api.github.com/repos/glongo/suricata/merges","archive_url":"https://api.github.com/repos/glongo/suricata/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/glongo/suricata/downloads","issues_url":"https://api.github.com/repos/glongo/suricata/issues{/number}","pulls_url":"https://api.github.com/repos/glongo/suricata/pulls{/number}","milestones_url":"https://api.github.com/repos/glongo/suricata/milestones{/number}","notifications_url":"https://api.github.com/repos/glongo/suricata/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/glongo/suricata/labels{/name}","releases_url":"https://api.github.com/repos/glongo/suricata/releases{/id}","created_at":"2013-09-23T15:52:17Z","updated_at":"2015-05-02T21:15:07Z","pushed_at":"2015-05-29T12:11:05Z","git_url":"git://github.com/glongo/suricata.git","ssh_url":"git@github.com:glongo/suricata.git","clone_url":"https://github.com/glongo/suricata.git","svn_url":"https://github.com/glongo/suricata","homepage":"http://suricata-ids.org/","size":40960,"stargazers_count":0,"watchers_count":0,"language":"C","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"inliniac:master","ref":"master","sha":"07efec550dd249f3c4a1a1fb926cbbe4298245cc","user":{"login":"inliniac","id":719366,"avatar_url":"https://avatars.githubusercontent.com/u/719366?v=3","gravatar_id":"","url":"https://api.github.com/users/inliniac","html_url":"https://github.com/inliniac","followers_url":"https://api.github.com/users/inliniac/followers","following_url":"https://api.github.com/users/inliniac/following{/other_user}","gists_url":"https://api.github.com/users/inliniac/gists{/gist_id}","starred_url":"https://api.github.com/users/inliniac/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/inliniac/subscriptions","organizations_url":"https://api.github.com/users/inliniac/orgs","repos_url":"https://api.github.com/users/inliniac/repos","events_url":"https://api.github.com/users/inliniac/events{/privacy}","received_events_url":"https://api.github.com/users/inliniac/received_events","type":"User","site_admin":false},"repo":{"id":5410234,"name":"suricata","full_name":"inliniac/suricata","owner":{"login":"inliniac","id":719366,"avatar_url":"https://avatars.githubusercontent.com/u/719366?v=3","gravatar_id":"","url":"https://api.github.com/users/inliniac","html_url":"https://github.com/inliniac","followers_url":"https://api.github.com/users/inliniac/followers","following_url":"https://api.github.com/users/inliniac/following{/other_user}","gists_url":"https://api.github.com/users/inliniac/gists{/gist_id}","starred_url":"https://api.github.com/users/inliniac/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/inliniac/subscriptions","organizations_url":"https://api.github.com/users/inliniac/orgs","repos_url":"https://api.github.com/users/inliniac/repos","events_url":"https://api.github.com/users/inliniac/events{/privacy}","received_events_url":"https://api.github.com/users/inliniac/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/inliniac/suricata","description":"Mirror of the official OISF Suricata git repository","fork":false,"url":"https://api.github.com/repos/inliniac/suricata","forks_url":"https://api.github.com/repos/inliniac/suricata/forks","keys_url":"https://api.github.com/repos/inliniac/suricata/keys{/key_id}","collaborators_url":"https://api.github.com/repos/inliniac/suricata/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/inliniac/suricata/teams","hooks_url":"https://api.github.com/repos/inliniac/suricata/hooks","issue_events_url":"https://api.github.com/repos/inliniac/suricata/issues/events{/number}","events_url":"https://api.github.com/repos/inliniac/suricata/events","assignees_url":"https://api.github.com/repos/inliniac/suricata/assignees{/user}","branches_url":"https://api.github.com/repos/inliniac/suricata/branches{/branch}","tags_url":"https://api.github.com/repos/inliniac/suricata/tags","blobs_url":"https://api.github.com/repos/inliniac/suricata/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/inliniac/suricata/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/inliniac/suricata/git/refs{/sha}","trees_url":"https://api.github.com/repos/inliniac/suricata/git/trees{/sha}","statuses_url":"https://api.github.com/repos/inliniac/suricata/statuses/{sha}","languages_url":"https://api.github.com/repos/inliniac/suricata/languages","stargazers_url":"https://api.github.com/repos/inliniac/suricata/stargazers","contributors_url":"https://api.github.com/repos/inliniac/suricata/contributors","subscribers_url":"https://api.github.com/repos/inliniac/suricata/subscribers","subscription_url":"https://api.github.com/repos/inliniac/suricata/subscription","commits_url":"https://api.github.com/repos/inliniac/suricata/commits{/sha}","git_commits_url":"https://api.github.com/repos/inliniac/suricata/git/commits{/sha}","comments_url":"https://api.github.com/repos/inliniac/suricata/comments{/number}","issue_comment_url":"https://api.github.com/repos/inliniac/suricata/issues/comments{/number}","contents_url":"https://api.github.com/repos/inliniac/suricata/contents/{+path}","compare_url":"https://api.github.com/repos/inliniac/suricata/compare/{base}...{head}","merges_url":"https://api.github.com/repos/inliniac/suricata/merges","archive_url":"https://api.github.com/repos/inliniac/suricata/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/inliniac/suricata/downloads","issues_url":"https://api.github.com/repos/inliniac/suricata/issues{/number}","pulls_url":"https://api.github.com/repos/inliniac/suricata/pulls{/number}","milestones_url":"https://api.github.com/repos/inliniac/suricata/milestones{/number}","notifications_url":"https://api.github.com/repos/inliniac/suricata/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/inliniac/suricata/labels{/name}","releases_url":"https://api.github.com/repos/inliniac/suricata/releases{/id}","created_at":"2012-08-14T08:30:06Z","updated_at":"2015-05-29T00:42:47Z","pushed_at":"2015-05-29T12:29:58Z","git_url":"git://github.com/inliniac/suricata.git","ssh_url":"git@github.com:inliniac/suricata.git","clone_url":"https://github.com/inliniac/suricata.git","svn_url":"https://github.com/inliniac/suricata","homepage":"http://suricata-ids.org/","size":63494,"stargazers_count":175,"watchers_count":175,"language":"C","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":163,"mirror_url":null,"open_issues_count":28,"forks":163,"open_issues":28,"watchers":175,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/inliniac/suricata/pulls/1516"},"html":{"href":"https://github.com/inliniac/suricata/pull/1516"},"issue":{"href":"https://api.github.com/repos/inliniac/suricata/issues/1516"},"comments":{"href":"https://api.github.com/repos/inliniac/suricata/issues/1516/comments"},"review_comments":{"href":"https://api.github.com/repos/inliniac/suricata/pulls/1516/comments"},"review_comment":{"href":"https://api.github.com/repos/inliniac/suricata/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/inliniac/suricata/pulls/1516/commits"},"statuses":{"href":"https://api.github.com/repos/inliniac/suricata/statuses/79cba7b935e1e3fc7abddc7ef7fbd10a9ad442d5"}}}}