{"action":"created","issue":{"url":"https://api.github.com/repos/OmniLayer/omnicore/issues/17","labels_url":"https://api.github.com/repos/OmniLayer/omnicore/issues/17/labels{/name}","comments_url":"https://api.github.com/repos/OmniLayer/omnicore/issues/17/comments","events_url":"https://api.github.com/repos/OmniLayer/omnicore/issues/17/events","html_url":"https://github.com/OmniLayer/omnicore/pull/17","id":68402044,"number":17,"title":"Logger and initialization refinement and overhaul","user":{"login":"dexX7","id":5836089,"avatar_url":"https://avatars.githubusercontent.com/u/5836089?v=3","gravatar_id":"","url":"https://api.github.com/users/dexX7","html_url":"https://github.com/dexX7","followers_url":"https://api.github.com/users/dexX7/followers","following_url":"https://api.github.com/users/dexX7/following{/other_user}","gists_url":"https://api.github.com/users/dexX7/gists{/gist_id}","starred_url":"https://api.github.com/users/dexX7/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dexX7/subscriptions","organizations_url":"https://api.github.com/users/dexX7/orgs","repos_url":"https://api.github.com/users/dexX7/repos","events_url":"https://api.github.com/users/dexX7/events{/privacy}","received_events_url":"https://api.github.com/users/dexX7/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/OmniLayer/omnicore/labels/ready%20for%20review","name":"ready for review","color":"009800"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":23,"created_at":"2015-04-14T14:58:04Z","updated_at":"2015-04-17T23:53:53Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/OmniLayer/omnicore/pulls/17","html_url":"https://github.com/OmniLayer/omnicore/pull/17","diff_url":"https://github.com/OmniLayer/omnicore/pull/17.diff","patch_url":"https://github.com/OmniLayer/omnicore/pull/17.patch"},"body":"**1) File logger and the loggers in general:**\r\n\r\nUsage:\r\n- Any entity that likes to use the logger has to include `mastercore_log.h`\r\n- To add a new category, an entry in `mastercore_log.h` and `mastercore_log.cpp` should be created\r\n- To enable or disable logging of a specific category, `mastercore_log.cpp` is the place to go\r\n\r\nNotes:\r\n- It provides a central place for anything log related\r\n- With `_log.h` included, there is no need for `external ...` anymore\r\n- `mp_log()` and `mp_error()` were never used, so I don't see any need to carry them around for now\r\n- The categories might be enabled and disabled via RPC or config file at some point later\r\n\r\n**2) Console or status logger:**\r\n\r\nAs subsitute for `printf()` another logger function `PrintToConsole()` was added:\r\n\r\n- `PrintToConsole()` has a similar behavior like `file_log()`, but instead of printing to `mastercore.log`, the logger prints directly to the console\r\n- `PrintToConsole()` is superior to `printf()`, as it comes with smarter tinyformat based formatting, and for example `.c_str()` can be omitted, when printing strings\r\n- The main motivation though, was to gain control over all those `printf()`, which were sort of annoying and difficult to tame.. ;)\r\n- In particular, encapsulating the console output allows to apply transformations, like using the option `-logtimestamps` to prepend the output with timestamps\r\n- It is thinkable to use categories in the future, or to add more configuration options, such as `-quiet`, to silence all regular console output\r\n\r\n**3) More useful and readable initializtion messages:**\r\n\r\nInstead of printing line numbers and file names, the status messages were updated.\r\n\r\nFurther, the dumping of balances after an initial scan was removed completely for a few reasons:\r\n\r\n- It was only printed after a scan, but not on every start\r\n- The information is not really relevant\r\n- The information is still available via other channels, e.g. via RPC `mscrpc`, `getallbalances..MP`\r\n\r\n**4) Progress reporting during the intial transaction scanning:**\r\n\r\nEvery 30 seconds the progress of the scanning is printed to the console, written to the debug log file, and the RPC status, as well as the splash screen progress label, are updated.\r\n\r\nIt was important for me to show that Omni Core hasn't stalled, or is broken and doesn't do anything  during the scanning, but in fact has already processed n of m blocks.\r\n\r\n**5) Shutdown request handling during the initial scanning:**\r\n\r\nUntil now there was no way to shutdown Omni Core during the initial scanning, whether the shutdown request was intentionally, or triggered by some other event, for example a restart of the OS. The only way was to forcefully kill the process, which is neither user friendly, nor does it properly close the databases, and in the worst case it could result in some corruption.\r\n\r\nClosing the QT splash screen window, or killing the process, triggers a global (Bitcoin Core based) shutdown request, which is honored by `msc_initial_scan()`.\r\n\r\n**6) Low-hanging fruits on the way:**\r\n\r\nAfter @zathras-crypto gave green light for the initial merge, I started to wonder how to address any potential merge conflicts, and I finally came to the conclusion that it's probably better to bundle somewhat related changes, to reduce the impact on other pull requests.\r\n\r\nWhile I generally think changes should come in smaller packets, the gain seemed to outweight the cost in this case, and in particular I'm referring to the last three commits [79749e9](https://github.com/dexX7/bitcoin/commit/79749e927b67c60105a86bcb7c634f43cbfc1d5f), [f07a23d](https://github.com/dexX7/bitcoin/commit/f07a23d2047ca2845e6adcacc43c96d9cf3cc773) and [93eae99](https://github.com/OmniLayer/omnicore/commit/93eae991967d1313411ff00c435cc33a2c6c31e9), which are hopefully easy to review and basically without any real impact.\r\n\r\n**7) No merge conflicts:**\r\n\r\nI actually got away with creating none for the pending pull requests.\r\n\r\nThe UI PR #5 can be merged directly, and #13 can be merged without conflicts, after merging https://github.com/zathras-crypto/mastercore/pull/63.\r\n\r\n**8) Let's see some results of the updated initialization and status reports... :)**\r\n\r\n```\r\n$ ./src/test/test_bitcoin \r\nInitializing Omni Core v0.0.9.2-dev [unittest]\r\nLoading trades database: OK\r\nLoading send-to-owners database: OK\r\nLoading transactions database: OK\r\nLoading smart property database: OK\r\nExodus balance: 0.00000000 MSC\r\nOmni Core initialization completed\r\nRunning 150 test cases...\r\nOmni Core shutdown completed\r\n\r\n*** No errors detected\r\n```\r\n```\r\n$ ./src/qt/bitcoin-qt -logtimestamps # or bitcoind\r\n2015-04-15 22:04:10 Initializing Omni Core v0.0.9.2-dev [main]\r\n2015-04-15 22:04:10 Loading trades database: OK\r\n2015-04-15 22:04:10 Loading send-to-owners database: OK\r\n2015-04-15 22:04:10 Loading transactions database: OK\r\n2015-04-15 22:04:10 Loading smart property database: OK\r\n2015-04-15 22:04:10 Scanning for transactions in block 249498 to block 352066..\r\n2015-04-15 22:04:25 Still scanning.. at block 253221 of 352066. Progress: 3.63 %\r\n2015-04-15 22:04:40 Still scanning.. at block 256863 of 352066. Progress: 7.18 %\r\n2015-04-15 22:04:55 Still scanning.. at block 258254 of 352066. Progress: 8.54 %\r\n2015-04-15 22:05:10 Still scanning.. at block 259397 of 352066. Progress: 9.65 %\r\n2015-04-15 22:05:25 Still scanning.. at block 260657 of 352066. Progress: 10.88 %\r\n2015-04-15 22:05:40 Still scanning.. at block 261969 of 352066. Progress: 12.16 %\r\n# ctrl+c, kill bitcoind, kill bitcoin-qt, or close bitcoin-qt\r\n2015-04-15 22:05:44 Scan stopped early at block 262346 of block 352066\r\n2015-04-15 22:05:44 3690943 transactions processed, 121 meta transactions found\r\n2015-04-15 22:05:44 Exodus balance: 3845.77658936 MSC\r\n2015-04-15 22:05:44 Omni Core initialization completed\r\n2015-04-15 22:05:44 Omni Core shutdown completed\r\n```\r\n```\r\n$ ./src/bitcoind -logtimestamps -startclean # or bitcoin-qt\r\n2015-04-15 22:18:37 Initializing Omni Core v0.0.9.2-dev [regtest]\r\n2015-04-15 22:18:37 Loading trades database: OK\r\n2015-04-15 22:18:37 Loading send-to-owners database: OK\r\n2015-04-15 22:18:37 Loading transactions database: OK\r\n2015-04-15 22:18:37 Loading smart property database: OK\r\n2015-04-15 22:18:37 Scanning for transactions in block 5 to block 749..\r\n2015-04-15 22:18:43 1591 transactions processed, 432 meta transactions found\r\n2015-04-15 22:18:43 Exodus balance: 38002.65700426 MSC\r\n2015-04-15 22:18:43 Omni Core initialization completed\r\n```"},"comment":{"url":"https://api.github.com/repos/OmniLayer/omnicore/issues/comments/94100899","html_url":"https://github.com/OmniLayer/omnicore/pull/17#issuecomment-94100899","issue_url":"https://api.github.com/repos/OmniLayer/omnicore/issues/17","id":94100899,"user":{"login":"zathras-crypto","id":5633351,"avatar_url":"https://avatars.githubusercontent.com/u/5633351?v=3","gravatar_id":"","url":"https://api.github.com/users/zathras-crypto","html_url":"https://github.com/zathras-crypto","followers_url":"https://api.github.com/users/zathras-crypto/followers","following_url":"https://api.github.com/users/zathras-crypto/following{/other_user}","gists_url":"https://api.github.com/users/zathras-crypto/gists{/gist_id}","starred_url":"https://api.github.com/users/zathras-crypto/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zathras-crypto/subscriptions","organizations_url":"https://api.github.com/users/zathras-crypto/orgs","repos_url":"https://api.github.com/users/zathras-crypto/repos","events_url":"https://api.github.com/users/zathras-crypto/events{/privacy}","received_events_url":"https://api.github.com/users/zathras-crypto/received_events","type":"User","site_admin":false},"created_at":"2015-04-17T23:53:53Z","updated_at":"2015-04-17T23:53:53Z","body":"> but I think it's a middleground\r\n\r\nAgreed - it's a fantastic improvement over what we had previously (an apparent hung process for 60 mins) :)\r\n\r\n> Can you describe how I can reproduce the issue?\r\n\r\nIf you just build and run ```0.0.10-Z-Auditor``` the auditor is enabled by default - you can play with the ```omni_debug_auditor``` and ```omni_debug_auditor_verbose``` settings in ```omnicore_auditor.cpp``` but I don't think you'll need to, you should just see it attempt to shutdown and fail when it detects that bad trade in the MetaDEx on testnet.\r\n\r\nOh, and run with --startclean so it reparses everything to pick up the issue :)\r\n\r\nThanks again bud"}}