{"action":"created","comment":{"url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/pulls/comments/42271008","id":42271008,"diff_hunk":"@@ -0,0 +1,163 @@\n+/*##############################################################################\n+\n+    HPCC SYSTEMS software Copyright (C) 2015 HPCC Systems.\n+\n+    Licensed under the Apache License, Version 2.0 (the \"License\");\n+    you may not use this file except in compliance with the License.\n+    You may obtain a copy of the License at\n+\n+       http://www.apache.org/licenses/LICENSE-2.0\n+\n+    Unless required by applicable law or agreed to in writing, software\n+    distributed under the License is distributed on an \"AS IS\" BASIS,\n+    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n+    See the License for the specific language governing permissions and\n+    limitations under the License.\n+############################################################################## */\n+\n+#ifndef TRACESLAVE_HPP\n+#define TRACESLAVE_HPP\n+\n+#include \"slave.hpp\"\n+#include \"jqueue.tpp\"\n+#include \"jthread.hpp\"\n+#include \"limits.h\"\n+\n+\n+class CTracingThorDataLink :  implements CInterfaceOf<IThorDataLink>\n+{\n+private:\n+    class TraceQueue\n+    {\n+    private:\n+        void dump(MemoryBuffer &mb, IHThorArg *helper) const\n+        {\n+            // NOTE - we don't output every item in the list to be \n+            // consistent with the other dump() implementation\n+            // Note - whilst dumping, does not clear ring buffer\n+            // so that it is available later\n+            unsigned rowPos = rowHeadp;\n+            for (unsigned n=1; n<traceRowBufferSize;++n)\n+            {\n+                if (++rowPos==traceRowBufferSize) rowPos = 0;  // NOTE - skipping first item intentional\n+                OwnedConstThorRow row (rowBuffer[rowPos]);\n+                if (!row) continue;\n+\n+                CommonXmlWriter xmlwrite(XWFnoindent|XWFtrim);\n+                helper->queryOutputMeta()->toXML((const byte*)(row.get()), xmlwrite);\n+                VStringBuffer trace(\"<Row>%s</Row>\", xmlwrite.str());\n+                mb.append(trace.str());\n+            }\n+        }\n+    public:\n+        OwnedConstThorRow *rowBuffer;\n+        unsigned rowHeadp;\n+        TraceQueue()\n+        {\n+            rowBuffer = NULL;\n+            rowHeadp = 0;\n+        }\n+        ~TraceQueue()\n+        {\n+            delete [] rowBuffer;\n+        }\n+        void init()\n+        {\n+            if (!rowBuffer)\n+            {\n+                rowBuffer = new OwnedConstThorRow[traceRowBufferSize];\n+                rowHeadp = 0;\n+            }\n+        }\n+        void enqueue(const void *row)\n+        {\n+            // NOTE - updates then sets rowHeadp (atomically) so that the code\n+            // below (ignoring oldest record) is safe\n+            rowBuffer[rowHeadp].set(row);\n+            unsigned nextHead = rowHeadp + 1;\n+            if (nextHead==traceRowBufferSize)\n+                nextHead = 0;\n+            rowHeadp = nextHead;\n+        }\n+        // This dump() makes use of pass-through buffer to ensure if it is called again\n+        // then previous rows still available\n+        void dump(MemoryBuffer &mb, TraceQueue &rowLogCache, IHThorArg *helper)\n+        {\n+            // NOTE - we don't output every item in the list as one\n+            // COULD have been updated after we swapped the buffers","path":"thorlcr/slave/traceslave.hpp","position":87,"original_position":87,"commit_id":"9d429705902132eae7a22e6ac53119511d21bf8b","original_commit_id":"9d429705902132eae7a22e6ac53119511d21bf8b","user":{"login":"jakesmith","id":902700,"avatar_url":"https://avatars.githubusercontent.com/u/902700?v=3","gravatar_id":"","url":"https://api.github.com/users/jakesmith","html_url":"https://github.com/jakesmith","followers_url":"https://api.github.com/users/jakesmith/followers","following_url":"https://api.github.com/users/jakesmith/following{/other_user}","gists_url":"https://api.github.com/users/jakesmith/gists{/gist_id}","starred_url":"https://api.github.com/users/jakesmith/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jakesmith/subscriptions","organizations_url":"https://api.github.com/users/jakesmith/orgs","repos_url":"https://api.github.com/users/jakesmith/repos","events_url":"https://api.github.com/users/jakesmith/events{/privacy}","received_events_url":"https://api.github.com/users/jakesmith/received_events","type":"User","site_admin":false},"body":"Also (and maybe we discussed this(?), it does mean that if there's that race condition, that rows will appear out of sequence. e.g.:\r\n+ buffers[0] = [a],[b],[*c] (c about to be written)\r\n+ dump swaps, a,b added to tracing, buffers[1] now active\r\n+ buffers[1] = [d],[e],[*f] (f about to be written)\r\n+ dump swap, d,e added to tracing, buffers[0] now active\r\n+ buffers[0] = [],[],[c],[g],[h],[*i] (i about to be written)\r\n+ dump swap, c,g,h added to tracing, buffers[1] now active\r\n+ buffers[1] = [],[],[f],[j],[k],[*l] (l about to be written)\r\n+ dump swap, f,j,k added to tracing..\r\n\r\n\r\nShould we just accept that, and just put a comment to mention that this can happen?\r\n\r\nOr could we look at the other buffer (the one about to be swapped in) for the trailing row and add to the tracing circular ring, before swapping in ?\r\nI think that could mean there's still a window if 2 swaps happened very quickly, where it would still be active and not picked up and therefore still lead to out order rows, but in practice with the time taken between 2 swaps over 2 debugRequest() hops that isn't going to happen.\r\n","created_at":"2015-10-16T17:57:55Z","updated_at":"2015-10-16T17:57:55Z","html_url":"https://github.com/hpcc-systems/HPCC-Platform/pull/7862#discussion_r42271008","pull_request_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/pulls/7862","_links":{"self":{"href":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/pulls/comments/42271008"},"html":{"href":"https://github.com/hpcc-systems/HPCC-Platform/pull/7862#discussion_r42271008"},"pull_request":{"href":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/pulls/7862"}}},"pull_request":{"url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/pulls/7862","id":47804372,"html_url":"https://github.com/hpcc-systems/HPCC-Platform/pull/7862","diff_url":"https://github.com/hpcc-systems/HPCC-Platform/pull/7862.diff","patch_url":"https://github.com/hpcc-systems/HPCC-Platform/pull/7862.patch","issue_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/issues/7862","number":7862,"state":"open","locked":false,"title":"HPCC-13030 Add row tracing support to engines","user":{"login":"shamser","id":11410590,"avatar_url":"https://avatars.githubusercontent.com/u/11410590?v=3","gravatar_id":"","url":"https://api.github.com/users/shamser","html_url":"https://github.com/shamser","followers_url":"https://api.github.com/users/shamser/followers","following_url":"https://api.github.com/users/shamser/following{/other_user}","gists_url":"https://api.github.com/users/shamser/gists{/gist_id}","starred_url":"https://api.github.com/users/shamser/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shamser/subscriptions","organizations_url":"https://api.github.com/users/shamser/orgs","repos_url":"https://api.github.com/users/shamser/repos","events_url":"https://api.github.com/users/shamser/events{/privacy}","received_events_url":"https://api.github.com/users/shamser/received_events","type":"User","site_admin":false},"body":"Return last 10 rows for a given activity & edge, when xml submitted to port 9877 (or as configured in global) in the form:\r\n\r\n  \\<debug source=\"ACTIVITY_ID\" edge=\"EDGE_ID\"\\>\\</debug\\>\r\n","created_at":"2015-10-15T15:44:54Z","updated_at":"2015-10-16T17:57:55Z","closed_at":null,"merged_at":null,"merge_commit_sha":"6fcc842016e31911684fca44b6b07581429322fc","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/pulls/7862/commits","review_comments_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/pulls/7862/comments","review_comment_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/pulls/comments{/number}","comments_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/issues/7862/comments","statuses_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/statuses/9d429705902132eae7a22e6ac53119511d21bf8b","head":{"label":"shamser:issue13030","ref":"issue13030","sha":"9d429705902132eae7a22e6ac53119511d21bf8b","user":{"login":"shamser","id":11410590,"avatar_url":"https://avatars.githubusercontent.com/u/11410590?v=3","gravatar_id":"","url":"https://api.github.com/users/shamser","html_url":"https://github.com/shamser","followers_url":"https://api.github.com/users/shamser/followers","following_url":"https://api.github.com/users/shamser/following{/other_user}","gists_url":"https://api.github.com/users/shamser/gists{/gist_id}","starred_url":"https://api.github.com/users/shamser/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shamser/subscriptions","organizations_url":"https://api.github.com/users/shamser/orgs","repos_url":"https://api.github.com/users/shamser/repos","events_url":"https://api.github.com/users/shamser/events{/privacy}","received_events_url":"https://api.github.com/users/shamser/received_events","type":"User","site_admin":false},"repo":{"id":32939247,"name":"HPCC-Platform","full_name":"shamser/HPCC-Platform","owner":{"login":"shamser","id":11410590,"avatar_url":"https://avatars.githubusercontent.com/u/11410590?v=3","gravatar_id":"","url":"https://api.github.com/users/shamser","html_url":"https://github.com/shamser","followers_url":"https://api.github.com/users/shamser/followers","following_url":"https://api.github.com/users/shamser/following{/other_user}","gists_url":"https://api.github.com/users/shamser/gists{/gist_id}","starred_url":"https://api.github.com/users/shamser/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shamser/subscriptions","organizations_url":"https://api.github.com/users/shamser/orgs","repos_url":"https://api.github.com/users/shamser/repos","events_url":"https://api.github.com/users/shamser/events{/privacy}","received_events_url":"https://api.github.com/users/shamser/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/shamser/HPCC-Platform","description":"IMPORTANT - Issue tracking has been migrated to JIRA - use your github ID and password reset mechanism to log in","fork":true,"url":"https://api.github.com/repos/shamser/HPCC-Platform","forks_url":"https://api.github.com/repos/shamser/HPCC-Platform/forks","keys_url":"https://api.github.com/repos/shamser/HPCC-Platform/keys{/key_id}","collaborators_url":"https://api.github.com/repos/shamser/HPCC-Platform/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/shamser/HPCC-Platform/teams","hooks_url":"https://api.github.com/repos/shamser/HPCC-Platform/hooks","issue_events_url":"https://api.github.com/repos/shamser/HPCC-Platform/issues/events{/number}","events_url":"https://api.github.com/repos/shamser/HPCC-Platform/events","assignees_url":"https://api.github.com/repos/shamser/HPCC-Platform/assignees{/user}","branches_url":"https://api.github.com/repos/shamser/HPCC-Platform/branches{/branch}","tags_url":"https://api.github.com/repos/shamser/HPCC-Platform/tags","blobs_url":"https://api.github.com/repos/shamser/HPCC-Platform/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/shamser/HPCC-Platform/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/shamser/HPCC-Platform/git/refs{/sha}","trees_url":"https://api.github.com/repos/shamser/HPCC-Platform/git/trees{/sha}","statuses_url":"https://api.github.com/repos/shamser/HPCC-Platform/statuses/{sha}","languages_url":"https://api.github.com/repos/shamser/HPCC-Platform/languages","stargazers_url":"https://api.github.com/repos/shamser/HPCC-Platform/stargazers","contributors_url":"https://api.github.com/repos/shamser/HPCC-Platform/contributors","subscribers_url":"https://api.github.com/repos/shamser/HPCC-Platform/subscribers","subscription_url":"https://api.github.com/repos/shamser/HPCC-Platform/subscription","commits_url":"https://api.github.com/repos/shamser/HPCC-Platform/commits{/sha}","git_commits_url":"https://api.github.com/repos/shamser/HPCC-Platform/git/commits{/sha}","comments_url":"https://api.github.com/repos/shamser/HPCC-Platform/comments{/number}","issue_comment_url":"https://api.github.com/repos/shamser/HPCC-Platform/issues/comments{/number}","contents_url":"https://api.github.com/repos/shamser/HPCC-Platform/contents/{+path}","compare_url":"https://api.github.com/repos/shamser/HPCC-Platform/compare/{base}...{head}","merges_url":"https://api.github.com/repos/shamser/HPCC-Platform/merges","archive_url":"https://api.github.com/repos/shamser/HPCC-Platform/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/shamser/HPCC-Platform/downloads","issues_url":"https://api.github.com/repos/shamser/HPCC-Platform/issues{/number}","pulls_url":"https://api.github.com/repos/shamser/HPCC-Platform/pulls{/number}","milestones_url":"https://api.github.com/repos/shamser/HPCC-Platform/milestones{/number}","notifications_url":"https://api.github.com/repos/shamser/HPCC-Platform/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/shamser/HPCC-Platform/labels{/name}","releases_url":"https://api.github.com/repos/shamser/HPCC-Platform/releases{/id}","created_at":"2015-03-26T16:34:05Z","updated_at":"2015-03-26T16:34:22Z","pushed_at":"2015-10-16T14:06:03Z","git_url":"git://github.com/shamser/HPCC-Platform.git","ssh_url":"git@github.com:shamser/HPCC-Platform.git","clone_url":"https://github.com/shamser/HPCC-Platform.git","svn_url":"https://github.com/shamser/HPCC-Platform","homepage":"track.hpccsystems.com","size":301425,"stargazers_count":0,"watchers_count":0,"language":"C++","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"hpcc-systems:master","ref":"master","sha":"7b2c4a71c2cacfa808e225ac5d446fe22ccb66ae","user":{"login":"hpcc-systems","id":900867,"avatar_url":"https://avatars.githubusercontent.com/u/900867?v=3","gravatar_id":"","url":"https://api.github.com/users/hpcc-systems","html_url":"https://github.com/hpcc-systems","followers_url":"https://api.github.com/users/hpcc-systems/followers","following_url":"https://api.github.com/users/hpcc-systems/following{/other_user}","gists_url":"https://api.github.com/users/hpcc-systems/gists{/gist_id}","starred_url":"https://api.github.com/users/hpcc-systems/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hpcc-systems/subscriptions","organizations_url":"https://api.github.com/users/hpcc-systems/orgs","repos_url":"https://api.github.com/users/hpcc-systems/repos","events_url":"https://api.github.com/users/hpcc-systems/events{/privacy}","received_events_url":"https://api.github.com/users/hpcc-systems/received_events","type":"Organization","site_admin":false},"repo":{"id":2030681,"name":"HPCC-Platform","full_name":"hpcc-systems/HPCC-Platform","owner":{"login":"hpcc-systems","id":900867,"avatar_url":"https://avatars.githubusercontent.com/u/900867?v=3","gravatar_id":"","url":"https://api.github.com/users/hpcc-systems","html_url":"https://github.com/hpcc-systems","followers_url":"https://api.github.com/users/hpcc-systems/followers","following_url":"https://api.github.com/users/hpcc-systems/following{/other_user}","gists_url":"https://api.github.com/users/hpcc-systems/gists{/gist_id}","starred_url":"https://api.github.com/users/hpcc-systems/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hpcc-systems/subscriptions","organizations_url":"https://api.github.com/users/hpcc-systems/orgs","repos_url":"https://api.github.com/users/hpcc-systems/repos","events_url":"https://api.github.com/users/hpcc-systems/events{/privacy}","received_events_url":"https://api.github.com/users/hpcc-systems/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/hpcc-systems/HPCC-Platform","description":"IMPORTANT - Issue tracking has been migrated to JIRA - use your github ID and password reset mechanism to log in","fork":false,"url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform","forks_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/forks","keys_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/keys{/key_id}","collaborators_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/teams","hooks_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/hooks","issue_events_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/issues/events{/number}","events_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/events","assignees_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/assignees{/user}","branches_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/branches{/branch}","tags_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/tags","blobs_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/git/refs{/sha}","trees_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/git/trees{/sha}","statuses_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/statuses/{sha}","languages_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/languages","stargazers_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/stargazers","contributors_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/contributors","subscribers_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/subscribers","subscription_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/subscription","commits_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/commits{/sha}","git_commits_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/git/commits{/sha}","comments_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/comments{/number}","issue_comment_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/issues/comments{/number}","contents_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/contents/{+path}","compare_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/compare/{base}...{head}","merges_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/merges","archive_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/downloads","issues_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/issues{/number}","pulls_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/pulls{/number}","milestones_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/milestones{/number}","notifications_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/labels{/name}","releases_url":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/releases{/id}","created_at":"2011-07-11T14:29:51Z","updated_at":"2015-10-14T13:56:56Z","pushed_at":"2015-10-16T17:11:44Z","git_url":"git://github.com/hpcc-systems/HPCC-Platform.git","ssh_url":"git@github.com:hpcc-systems/HPCC-Platform.git","clone_url":"https://github.com/hpcc-systems/HPCC-Platform.git","svn_url":"https://github.com/hpcc-systems/HPCC-Platform","homepage":"track.hpccsystems.com","size":345815,"stargazers_count":257,"watchers_count":257,"language":"C++","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":122,"mirror_url":null,"open_issues_count":433,"forks":122,"open_issues":433,"watchers":257,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/pulls/7862"},"html":{"href":"https://github.com/hpcc-systems/HPCC-Platform/pull/7862"},"issue":{"href":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/issues/7862"},"comments":{"href":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/issues/7862/comments"},"review_comments":{"href":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/pulls/7862/comments"},"review_comment":{"href":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/pulls/7862/commits"},"statuses":{"href":"https://api.github.com/repos/hpcc-systems/HPCC-Platform/statuses/9d429705902132eae7a22e6ac53119511d21bf8b"}}}}