{"action":"created","issue":{"url":"https://api.github.com/repos/ukwa/webarchive-discovery/issues/46","labels_url":"https://api.github.com/repos/ukwa/webarchive-discovery/issues/46/labels{/name}","comments_url":"https://api.github.com/repos/ukwa/webarchive-discovery/issues/46/comments","events_url":"https://api.github.com/repos/ukwa/webarchive-discovery/issues/46/events","html_url":"https://github.com/ukwa/webarchive-discovery/pull/46","id":57469017,"number":46,"title":"Performance instrumentation","user":{"login":"tokee","id":199454,"avatar_url":"https://avatars.githubusercontent.com/u/199454?v=3","gravatar_id":"","url":"https://api.github.com/users/tokee","html_url":"https://github.com/tokee","followers_url":"https://api.github.com/users/tokee/followers","following_url":"https://api.github.com/users/tokee/following{/other_user}","gists_url":"https://api.github.com/users/tokee/gists{/gist_id}","starred_url":"https://api.github.com/users/tokee/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tokee/subscriptions","organizations_url":"https://api.github.com/users/tokee/orgs","repos_url":"https://api.github.com/users/tokee/repos","events_url":"https://api.github.com/users/tokee/events{/privacy}","received_events_url":"https://api.github.com/users/tokee/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-02-12T14:47:04Z","updated_at":"2015-02-16T11:29:33Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/ukwa/webarchive-discovery/pulls/46","html_url":"https://github.com/ukwa/webarchive-discovery/pull/46","diff_url":"https://github.com/ukwa/webarchive-discovery/pull/46.diff","patch_url":"https://github.com/ukwa/webarchive-discovery/pull/46.patch"},"body":"This pull is primarily about performance instrumentation: The class Instrument was added for lightweight tracking of where the time is spend (sample output below). The final report is currently only triggered by the `WARCIndexerCommand` with `Instrument.log(true)`.\r\n\r\nEye opener: Detecting language on our (admittedly tiny) sample took 1/5th of the total time.\r\n\r\n```\r\nWARC Indexer Finished in 180.572 seconds.\r\n2015-02-12 15:43:12 INFO  Instrument:166 - Performance statistics\r\nSolrRecord.removeControlCharacters#total(#=253778, time=3003.60ms, avg=84.49#/ms 0.01ms/#)\r\n  SolrRecord.sanitiseUTF8(#=253778, time=961.46ms, avg=263.95#/ms 0.00ms/#)\r\nWARCIndexerCommand.main#total(#=1, time=180577.93ms, avg=0.00#/ms 180577.93ms/#)\r\n  WARCIndexerCommand.parseWarcFiles#startup(#=1, time=4681.03ms, avg=0.00#/ms 4681.03ms/#)\r\n  WARCIndexerCommand.parseWarcFiles#solrdocCreation(#=2973, time=172194.94ms, avg=0.02#/ms 57.92ms/#)\r\n    WARCIndexer.extract#total(#=2973, time=171782.62ms, avg=0.02#/ms 57.78ms/#)\r\n      TextAnalyzers#total(#=2973, time=51045.44ms, avg=0.06#/ms 17.17ms/#)\r\n        LanguageAnalyzer#total(#=2762, time=49251.77ms, avg=0.06#/ms 17.83ms/#)\r\n          LanguageDetector.detectLanguage#li(#=2762, time=37463.25ms, avg=0.07#/ms 13.56ms/#)\r\n          LanguageDetector.detectLanguage#ld(#=2762, time=11740.58ms, avg=0.24#/ms 4.25ms/#)\r\n        FuzzyHashAnalyzer(#=2762, time=1411.92ms, avg=1.96#/ms 0.51ms/#)\r\n        PostcodeAnalyzer(#=2762, time=355.36ms, avg=7.77#/ms 0.13ms/#)\r\n      WARCIndexer.extract#hashstreamwrap(#=2973, time=6397.88ms, avg=0.46#/ms 2.15ms/#)\r\n      WARCIndexer.extract#archeaders(#=3377, time=971.33ms, avg=3.48#/ms 0.29ms/#)\r\n      WARCIndexer.extract#analyzetikainput(#=2973, time=113377.54ms, avg=0.03#/ms 38.14ms/#)\r\n        WARCPayloadAnalyzers.analyze#total(#=2973, time=113366.16ms, avg=0.03#/ms 38.13ms/#)\r\n          HTMLAnalyzer.analyze#total(#=2730, time=34759.37ms, avg=0.08#/ms 12.73ms/#)\r\n            HTMLAnalyzer.analyze#parser(#=2730, time=30107.75ms, avg=0.09#/ms 11.03ms/#)\r\n          WARCPayloadAnalyzers.analyze#droid(#=2973, time=16764.40ms, avg=0.18#/ms 5.64ms/#)\r\n          XMLAnalyzer.analyze(#=32, time=34.01ms, avg=0.94#/ms 1.06ms/#)\r\n          WARCPayloadAnalyzers.analyze#tikasolrextract(#=2973, time=54007.04ms, avg=0.06#/ms 18.17ms/#)\r\n          ImageAnalyzer.analyze#facesanddominant(#=13, time=7660.43ms, avg=0.00#/ms 589.26ms/#)\r\n          WARCPayloadAnalyzers.analyze#firstbytes(#=2973, time=125.02ms, avg=23.78#/ms 0.04ms/#)\r\n```"},"comment":{"url":"https://api.github.com/repos/ukwa/webarchive-discovery/issues/comments/74494803","html_url":"https://github.com/ukwa/webarchive-discovery/pull/46#issuecomment-74494803","issue_url":"https://api.github.com/repos/ukwa/webarchive-discovery/issues/46","id":74494803,"user":{"login":"tokee","id":199454,"avatar_url":"https://avatars.githubusercontent.com/u/199454?v=3","gravatar_id":"","url":"https://api.github.com/users/tokee","html_url":"https://github.com/tokee","followers_url":"https://api.github.com/users/tokee/followers","following_url":"https://api.github.com/users/tokee/following{/other_user}","gists_url":"https://api.github.com/users/tokee/gists{/gist_id}","starred_url":"https://api.github.com/users/tokee/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tokee/subscriptions","organizations_url":"https://api.github.com/users/tokee/orgs","repos_url":"https://api.github.com/users/tokee/repos","events_url":"https://api.github.com/users/tokee/events{/privacy}","received_events_url":"https://api.github.com/users/tokee/received_events","type":"User","site_admin":false},"created_at":"2015-02-16T11:29:33Z","updated_at":"2015-02-16T11:29:33Z","body":"Whoops. Yes, no problem. That's what I get for using class templates with license in my IDE."}}