{"action":"created","comment":{"url":"https://api.github.com/repos/hazelcast/hazelcast/pulls/comments/31790944","id":31790944,"diff_hunk":"@@ -319,28 +333,50 @@ private void scanHandleOperationTimeout() {\n                     return;\n                 }\n \n-                try {\n-                    if (invocation.checkInvocationTimeout()) {\n-                        invocationTimeouts++;\n-                    }\n-                } catch (Throwable t) {\n-                    inspectOutputMemoryError(t);\n-                    logger.severe(\"Failed to handle operation timeout of invocation:\" + invocation, t);\n+                logSlowInvocation(now, invocation);\n+\n+                if (checkInvocationTimeout(invocation)) {\n+                    invocationTimeouts++;\n                 }\n \n-                try {\n-                    if (invocation.checkBackupTimeout(backupTimeoutMillis)) {\n-                        backupTimeouts++;\n-                    }\n-                } catch (Throwable t) {\n-                    inspectOutputMemoryError(t);\n-                    logger.severe(\"Failed to handle backup timeout of invocation:\" + invocation, t);\n+                if (checkBackupTimeout(invocation)) {\n+                    backupTimeouts++;\n                 }\n             }\n \n             log(backupTimeouts, invocationTimeouts);\n         }\n \n+        private boolean checkBackupTimeout(Invocation invocation) {\n+            try {\n+                return invocation.checkBackupTimeout(backupTimeoutMillis);\n+            } catch (Throwable t) {\n+                inspectOutputMemoryError(t);\n+                logger.severe(\"Failed to handle backup timeout of invocation:\" + invocation, t);\n+                return false;\n+            }\n+        }\n+\n+        private boolean checkInvocationTimeout(Invocation invocation) {\n+            try {\n+                return invocation.checkInvocationTimeout();\n+            } catch (Throwable t) {\n+                inspectOutputMemoryError(t);\n+                logger.severe(\"Failed to handle operation timeout of invocation:\" + invocation, t);\n+                return false;\n+            }\n+        }\n+\n+        private void logSlowInvocation(long now, Invocation invocation) {\n+            if (slowInvocationThresholdMs > 0) {\n+                long durationMs = now - invocation.op.getInvocationTime();","path":"hazelcast/src/main/java/com/hazelcast/spi/impl/operationservice/impl/InvocationRegistry.java","position":null,"original_position":103,"commit_id":"c3483a5766499756ca2e91e8962529ee870bfa59","original_commit_id":"68469c3f1ecb6f6c7c0ea0fe89c7e850fee30553","user":{"login":"jerrinot","id":158619,"avatar_url":"https://avatars.githubusercontent.com/u/158619?v=3","gravatar_id":"","url":"https://api.github.com/users/jerrinot","html_url":"https://github.com/jerrinot","followers_url":"https://api.github.com/users/jerrinot/followers","following_url":"https://api.github.com/users/jerrinot/following{/other_user}","gists_url":"https://api.github.com/users/jerrinot/gists{/gist_id}","starred_url":"https://api.github.com/users/jerrinot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jerrinot/subscriptions","organizations_url":"https://api.github.com/users/jerrinot/orgs","repos_url":"https://api.github.com/users/jerrinot/repos","events_url":"https://api.github.com/users/jerrinot/events{/privacy}","received_events_url":"https://api.github.com/users/jerrinot/received_events","type":"User","site_admin":false},"body":"that's a good finding!","created_at":"2015-06-05T06:51:06Z","updated_at":"2015-06-05T06:51:06Z","html_url":"https://github.com/hazelcast/hazelcast/pull/5452#discussion_r31790944","pull_request_url":"https://api.github.com/repos/hazelcast/hazelcast/pulls/5452","_links":{"self":{"href":"https://api.github.com/repos/hazelcast/hazelcast/pulls/comments/31790944"},"html":{"href":"https://github.com/hazelcast/hazelcast/pull/5452#discussion_r31790944"},"pull_request":{"href":"https://api.github.com/repos/hazelcast/hazelcast/pulls/5452"}}},"pull_request":{"url":"https://api.github.com/repos/hazelcast/hazelcast/pulls/5452","id":36999426,"html_url":"https://github.com/hazelcast/hazelcast/pull/5452","diff_url":"https://github.com/hazelcast/hazelcast/pull/5452.diff","patch_url":"https://github.com/hazelcast/hazelcast/pull/5452.patch","issue_url":"https://api.github.com/repos/hazelcast/hazelcast/issues/5452","number":5452,"state":"open","locked":false,"title":"Debug logging for slow invocations to deal with OperationTimeout","user":{"login":"pveentjer","id":105243,"avatar_url":"https://avatars.githubusercontent.com/u/105243?v=3","gravatar_id":"","url":"https://api.github.com/users/pveentjer","html_url":"https://github.com/pveentjer","followers_url":"https://api.github.com/users/pveentjer/followers","following_url":"https://api.github.com/users/pveentjer/following{/other_user}","gists_url":"https://api.github.com/users/pveentjer/gists{/gist_id}","starred_url":"https://api.github.com/users/pveentjer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pveentjer/subscriptions","organizations_url":"https://api.github.com/users/pveentjer/orgs","repos_url":"https://api.github.com/users/pveentjer/repos","events_url":"https://api.github.com/users/pveentjer/events{/privacy}","received_events_url":"https://api.github.com/users/pveentjer/received_events","type":"User","site_admin":false},"body":"Some logging to detect slow invocations for query op timeout (and anything else that takes time). It does this by lifting on top of the existing periodic scan of all pending invocations and adding some logging if an invocation takes more than a configurable amount of time.\r\n\r\nBy default it is disabled, but can be enabled by setting a group property. I added it as public property because it is useful for all kinds of problem detection.","created_at":"2015-06-05T04:04:14Z","updated_at":"2015-06-05T06:51:06Z","closed_at":null,"merged_at":null,"merge_commit_sha":"abb39c3119a0847faf669eb4c0f028cadbd5525e","assignee":null,"milestone":{"url":"https://api.github.com/repos/hazelcast/hazelcast/milestones/55","html_url":"https://github.com/hazelcast/hazelcast/milestones/3.5","labels_url":"https://api.github.com/repos/hazelcast/hazelcast/milestones/55/labels","id":819785,"number":55,"title":"3.5","description":null,"creator":{"login":"sancar","id":570462,"avatar_url":"https://avatars.githubusercontent.com/u/570462?v=3","gravatar_id":"","url":"https://api.github.com/users/sancar","html_url":"https://github.com/sancar","followers_url":"https://api.github.com/users/sancar/followers","following_url":"https://api.github.com/users/sancar/following{/other_user}","gists_url":"https://api.github.com/users/sancar/gists{/gist_id}","starred_url":"https://api.github.com/users/sancar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sancar/subscriptions","organizations_url":"https://api.github.com/users/sancar/orgs","repos_url":"https://api.github.com/users/sancar/repos","events_url":"https://api.github.com/users/sancar/events{/privacy}","received_events_url":"https://api.github.com/users/sancar/received_events","type":"User","site_admin":false},"open_issues":32,"closed_issues":691,"state":"open","created_at":"2014-10-09T13:04:53Z","updated_at":"2015-06-05T06:50:51Z","due_on":null,"closed_at":null},"commits_url":"https://api.github.com/repos/hazelcast/hazelcast/pulls/5452/commits","review_comments_url":"https://api.github.com/repos/hazelcast/hazelcast/pulls/5452/comments","review_comment_url":"https://api.github.com/repos/hazelcast/hazelcast/pulls/comments{/number}","comments_url":"https://api.github.com/repos/hazelcast/hazelcast/issues/5452/comments","statuses_url":"https://api.github.com/repos/hazelcast/hazelcast/statuses/c3483a5766499756ca2e91e8962529ee870bfa59","head":{"label":"pveentjer:feature/3.x/slow-invocation-detector","ref":"feature/3.x/slow-invocation-detector","sha":"c3483a5766499756ca2e91e8962529ee870bfa59","user":{"login":"pveentjer","id":105243,"avatar_url":"https://avatars.githubusercontent.com/u/105243?v=3","gravatar_id":"","url":"https://api.github.com/users/pveentjer","html_url":"https://github.com/pveentjer","followers_url":"https://api.github.com/users/pveentjer/followers","following_url":"https://api.github.com/users/pveentjer/following{/other_user}","gists_url":"https://api.github.com/users/pveentjer/gists{/gist_id}","starred_url":"https://api.github.com/users/pveentjer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pveentjer/subscriptions","organizations_url":"https://api.github.com/users/pveentjer/orgs","repos_url":"https://api.github.com/users/pveentjer/repos","events_url":"https://api.github.com/users/pveentjer/events{/privacy}","received_events_url":"https://api.github.com/users/pveentjer/received_events","type":"User","site_admin":false},"repo":{"id":10990111,"name":"hazelcast","full_name":"pveentjer/hazelcast","owner":{"login":"pveentjer","id":105243,"avatar_url":"https://avatars.githubusercontent.com/u/105243?v=3","gravatar_id":"","url":"https://api.github.com/users/pveentjer","html_url":"https://github.com/pveentjer","followers_url":"https://api.github.com/users/pveentjer/followers","following_url":"https://api.github.com/users/pveentjer/following{/other_user}","gists_url":"https://api.github.com/users/pveentjer/gists{/gist_id}","starred_url":"https://api.github.com/users/pveentjer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pveentjer/subscriptions","organizations_url":"https://api.github.com/users/pveentjer/orgs","repos_url":"https://api.github.com/users/pveentjer/repos","events_url":"https://api.github.com/users/pveentjer/events{/privacy}","received_events_url":"https://api.github.com/users/pveentjer/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/pveentjer/hazelcast","description":"Open Source In-Memory Data Grid","fork":true,"url":"https://api.github.com/repos/pveentjer/hazelcast","forks_url":"https://api.github.com/repos/pveentjer/hazelcast/forks","keys_url":"https://api.github.com/repos/pveentjer/hazelcast/keys{/key_id}","collaborators_url":"https://api.github.com/repos/pveentjer/hazelcast/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/pveentjer/hazelcast/teams","hooks_url":"https://api.github.com/repos/pveentjer/hazelcast/hooks","issue_events_url":"https://api.github.com/repos/pveentjer/hazelcast/issues/events{/number}","events_url":"https://api.github.com/repos/pveentjer/hazelcast/events","assignees_url":"https://api.github.com/repos/pveentjer/hazelcast/assignees{/user}","branches_url":"https://api.github.com/repos/pveentjer/hazelcast/branches{/branch}","tags_url":"https://api.github.com/repos/pveentjer/hazelcast/tags","blobs_url":"https://api.github.com/repos/pveentjer/hazelcast/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/pveentjer/hazelcast/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/pveentjer/hazelcast/git/refs{/sha}","trees_url":"https://api.github.com/repos/pveentjer/hazelcast/git/trees{/sha}","statuses_url":"https://api.github.com/repos/pveentjer/hazelcast/statuses/{sha}","languages_url":"https://api.github.com/repos/pveentjer/hazelcast/languages","stargazers_url":"https://api.github.com/repos/pveentjer/hazelcast/stargazers","contributors_url":"https://api.github.com/repos/pveentjer/hazelcast/contributors","subscribers_url":"https://api.github.com/repos/pveentjer/hazelcast/subscribers","subscription_url":"https://api.github.com/repos/pveentjer/hazelcast/subscription","commits_url":"https://api.github.com/repos/pveentjer/hazelcast/commits{/sha}","git_commits_url":"https://api.github.com/repos/pveentjer/hazelcast/git/commits{/sha}","comments_url":"https://api.github.com/repos/pveentjer/hazelcast/comments{/number}","issue_comment_url":"https://api.github.com/repos/pveentjer/hazelcast/issues/comments{/number}","contents_url":"https://api.github.com/repos/pveentjer/hazelcast/contents/{+path}","compare_url":"https://api.github.com/repos/pveentjer/hazelcast/compare/{base}...{head}","merges_url":"https://api.github.com/repos/pveentjer/hazelcast/merges","archive_url":"https://api.github.com/repos/pveentjer/hazelcast/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/pveentjer/hazelcast/downloads","issues_url":"https://api.github.com/repos/pveentjer/hazelcast/issues{/number}","pulls_url":"https://api.github.com/repos/pveentjer/hazelcast/pulls{/number}","milestones_url":"https://api.github.com/repos/pveentjer/hazelcast/milestones{/number}","notifications_url":"https://api.github.com/repos/pveentjer/hazelcast/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/pveentjer/hazelcast/labels{/name}","releases_url":"https://api.github.com/repos/pveentjer/hazelcast/releases{/id}","created_at":"2013-06-27T07:15:08Z","updated_at":"2015-04-22T04:00:47Z","pushed_at":"2015-06-05T06:49:55Z","git_url":"git://github.com/pveentjer/hazelcast.git","ssh_url":"git@github.com:pveentjer/hazelcast.git","clone_url":"https://github.com/pveentjer/hazelcast.git","svn_url":"https://github.com/pveentjer/hazelcast","homepage":"www.hazelcast.com","size":74435,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"hazelcast:master","ref":"master","sha":"82563c38685eeb9638e422ed96499a49ceab50b9","user":{"login":"hazelcast","id":1453152,"avatar_url":"https://avatars.githubusercontent.com/u/1453152?v=3","gravatar_id":"","url":"https://api.github.com/users/hazelcast","html_url":"https://github.com/hazelcast","followers_url":"https://api.github.com/users/hazelcast/followers","following_url":"https://api.github.com/users/hazelcast/following{/other_user}","gists_url":"https://api.github.com/users/hazelcast/gists{/gist_id}","starred_url":"https://api.github.com/users/hazelcast/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hazelcast/subscriptions","organizations_url":"https://api.github.com/users/hazelcast/orgs","repos_url":"https://api.github.com/users/hazelcast/repos","events_url":"https://api.github.com/users/hazelcast/events{/privacy}","received_events_url":"https://api.github.com/users/hazelcast/received_events","type":"Organization","site_admin":false},"repo":{"id":3786237,"name":"hazelcast","full_name":"hazelcast/hazelcast","owner":{"login":"hazelcast","id":1453152,"avatar_url":"https://avatars.githubusercontent.com/u/1453152?v=3","gravatar_id":"","url":"https://api.github.com/users/hazelcast","html_url":"https://github.com/hazelcast","followers_url":"https://api.github.com/users/hazelcast/followers","following_url":"https://api.github.com/users/hazelcast/following{/other_user}","gists_url":"https://api.github.com/users/hazelcast/gists{/gist_id}","starred_url":"https://api.github.com/users/hazelcast/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hazelcast/subscriptions","organizations_url":"https://api.github.com/users/hazelcast/orgs","repos_url":"https://api.github.com/users/hazelcast/repos","events_url":"https://api.github.com/users/hazelcast/events{/privacy}","received_events_url":"https://api.github.com/users/hazelcast/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/hazelcast/hazelcast","description":"Open Source In-Memory Data Grid","fork":false,"url":"https://api.github.com/repos/hazelcast/hazelcast","forks_url":"https://api.github.com/repos/hazelcast/hazelcast/forks","keys_url":"https://api.github.com/repos/hazelcast/hazelcast/keys{/key_id}","collaborators_url":"https://api.github.com/repos/hazelcast/hazelcast/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/hazelcast/hazelcast/teams","hooks_url":"https://api.github.com/repos/hazelcast/hazelcast/hooks","issue_events_url":"https://api.github.com/repos/hazelcast/hazelcast/issues/events{/number}","events_url":"https://api.github.com/repos/hazelcast/hazelcast/events","assignees_url":"https://api.github.com/repos/hazelcast/hazelcast/assignees{/user}","branches_url":"https://api.github.com/repos/hazelcast/hazelcast/branches{/branch}","tags_url":"https://api.github.com/repos/hazelcast/hazelcast/tags","blobs_url":"https://api.github.com/repos/hazelcast/hazelcast/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/hazelcast/hazelcast/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/hazelcast/hazelcast/git/refs{/sha}","trees_url":"https://api.github.com/repos/hazelcast/hazelcast/git/trees{/sha}","statuses_url":"https://api.github.com/repos/hazelcast/hazelcast/statuses/{sha}","languages_url":"https://api.github.com/repos/hazelcast/hazelcast/languages","stargazers_url":"https://api.github.com/repos/hazelcast/hazelcast/stargazers","contributors_url":"https://api.github.com/repos/hazelcast/hazelcast/contributors","subscribers_url":"https://api.github.com/repos/hazelcast/hazelcast/subscribers","subscription_url":"https://api.github.com/repos/hazelcast/hazelcast/subscription","commits_url":"https://api.github.com/repos/hazelcast/hazelcast/commits{/sha}","git_commits_url":"https://api.github.com/repos/hazelcast/hazelcast/git/commits{/sha}","comments_url":"https://api.github.com/repos/hazelcast/hazelcast/comments{/number}","issue_comment_url":"https://api.github.com/repos/hazelcast/hazelcast/issues/comments{/number}","contents_url":"https://api.github.com/repos/hazelcast/hazelcast/contents/{+path}","compare_url":"https://api.github.com/repos/hazelcast/hazelcast/compare/{base}...{head}","merges_url":"https://api.github.com/repos/hazelcast/hazelcast/merges","archive_url":"https://api.github.com/repos/hazelcast/hazelcast/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/hazelcast/hazelcast/downloads","issues_url":"https://api.github.com/repos/hazelcast/hazelcast/issues{/number}","pulls_url":"https://api.github.com/repos/hazelcast/hazelcast/pulls{/number}","milestones_url":"https://api.github.com/repos/hazelcast/hazelcast/milestones{/number}","notifications_url":"https://api.github.com/repos/hazelcast/hazelcast/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/hazelcast/hazelcast/labels{/name}","releases_url":"https://api.github.com/repos/hazelcast/hazelcast/releases{/id}","created_at":"2012-03-21T12:25:18Z","updated_at":"2015-06-04T03:27:30Z","pushed_at":"2015-06-05T06:50:52Z","git_url":"git://github.com/hazelcast/hazelcast.git","ssh_url":"git@github.com:hazelcast/hazelcast.git","clone_url":"https://github.com/hazelcast/hazelcast.git","svn_url":"https://github.com/hazelcast/hazelcast","homepage":"www.hazelcast.com","size":146007,"stargazers_count":1130,"watchers_count":1130,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":545,"mirror_url":null,"open_issues_count":424,"forks":545,"open_issues":424,"watchers":1130,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/hazelcast/hazelcast/pulls/5452"},"html":{"href":"https://github.com/hazelcast/hazelcast/pull/5452"},"issue":{"href":"https://api.github.com/repos/hazelcast/hazelcast/issues/5452"},"comments":{"href":"https://api.github.com/repos/hazelcast/hazelcast/issues/5452/comments"},"review_comments":{"href":"https://api.github.com/repos/hazelcast/hazelcast/pulls/5452/comments"},"review_comment":{"href":"https://api.github.com/repos/hazelcast/hazelcast/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/hazelcast/hazelcast/pulls/5452/commits"},"statuses":{"href":"https://api.github.com/repos/hazelcast/hazelcast/statuses/c3483a5766499756ca2e91e8962529ee870bfa59"}}}}