{"action":"created","issue":{"url":"https://api.github.com/repos/Cognifide/CQ-Unix-Toolkit/issues/31","labels_url":"https://api.github.com/repos/Cognifide/CQ-Unix-Toolkit/issues/31/labels{/name}","comments_url":"https://api.github.com/repos/Cognifide/CQ-Unix-Toolkit/issues/31/comments","events_url":"https://api.github.com/repos/Cognifide/CQ-Unix-Toolkit/issues/31/events","html_url":"https://github.com/Cognifide/CQ-Unix-Toolkit/pull/31","id":66615518,"number":31,"title":"Incorrect sed usage during curl output clean up procedure","user":{"login":"jwadolowski","id":6334715,"avatar_url":"https://avatars.githubusercontent.com/u/6334715?v=3","gravatar_id":"","url":"https://api.github.com/users/jwadolowski","html_url":"https://github.com/jwadolowski","followers_url":"https://api.github.com/users/jwadolowski/followers","following_url":"https://api.github.com/users/jwadolowski/following{/other_user}","gists_url":"https://api.github.com/users/jwadolowski/gists{/gist_id}","starred_url":"https://api.github.com/users/jwadolowski/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jwadolowski/subscriptions","organizations_url":"https://api.github.com/users/jwadolowski/orgs","repos_url":"https://api.github.com/users/jwadolowski/repos","events_url":"https://api.github.com/users/jwadolowski/events{/privacy}","received_events_url":"https://api.github.com/users/jwadolowski/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-04-06T14:17:23Z","updated_at":"2015-04-07T09:34:45Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/Cognifide/CQ-Unix-Toolkit/pulls/31","html_url":"https://github.com/Cognifide/CQ-Unix-Toolkit/pull/31","diff_url":"https://github.com/Cognifide/CQ-Unix-Toolkit/pull/31.diff","patch_url":"https://github.com/Cognifide/CQ-Unix-Toolkit/pull/31.patch"},"body":"Hey Arek,\r\n\r\nit seems that `cqcfgls` has an issue that makes the output slightly wrong. \r\n\r\nThis bug was pretty hard to track, as at first glance all the entries look perfectly fine and in most cases it was close to impossible to realize that something is displayed incorrectly. To be even worse I thought that something has to be wrong with either my code (test suite for one of my Chef cookbooks, 9 out of 10 test runs ended successfully) or CQ/AEM itself. Because of that non-deterministic behaviour I stuck to this opinion for quite a long time, until I decided to check the code of `cqcfgls` and I think I found the root cause of the problem.\r\n\r\nSo, what is actually wrong with `cqcfgls`? In `_list_configurations()` function you merge the output of `/system/console/components.json` and `/system/console/config/Configurations.nfo`. \r\nThe first one is a JSON file that is returned as a single line of text. When `curl` is invoked you append HTTP status code to this single line, which eventually makes the output like this `<long_json_line><http_status_code>` (`STATUS` variable). Later on you pass this on to `sed` which removes the very last 3 digits (HTTP status code to be precise, `${SEDX} 's#[0-9]{3}$##'`). It works perfectly fine and I have absolutely no remarks when it comes to this part.\r\nThe second part though is slightly different. The output of GET request to `/system/console/config/Configurations.nfo` is not returned as a single line of text, but as a regular multi-line file. Regardless of that in current shape `cqcfgls` is processed the same way when it comes to `curl` and `sed` usage. The problem is `sed -r 's#[0-9]{3}$##'` removes last 3 digits from _every_ line that matches to this pattern.\r\n\r\nI decided to compare how `STATUS` variable looks like before and after `sed` processing and here's the diff (I intentionally removed some lines, as originally it was over 200 lines long):\r\n```\r\n35,36c35,36\r\n< &nbsp;&nbsp;cache.size&nbsp;=&nbsp;100\r\n< &nbsp;&nbsp;cache.ttl&nbsp;=&nbsp;100000\r\n---\r\n> &nbsp;&nbsp;cache.size&nbsp;=&nbsp;\r\n> &nbsp;&nbsp;cache.ttl&nbsp;=&nbsp;100\r\n49c49\r\n< &nbsp;&nbsp;maxTotalAttachmentSize&nbsp;=&nbsp;104857600\r\n---\r\n> &nbsp;&nbsp;maxTotalAttachmentSize&nbsp;=&nbsp;104857\r\n61c61\r\n< &nbsp;&nbsp;maxTotalAttachmentSize&nbsp;=&nbsp;104857600\r\n---\r\n> &nbsp;&nbsp;maxTotalAttachmentSize&nbsp;=&nbsp;104857\r\n84c84\r\n< &nbsp;&nbsp;oauth.client.id&nbsp;=&nbsp;324926940878880\r\n---\r\n> &nbsp;&nbsp;oauth.client.id&nbsp;=&nbsp;324926940878\r\n86c86\r\n< &nbsp;&nbsp;oauth.config.id&nbsp;=&nbsp;cb4da2dbc7a22ad11bacb8e39ef92929\r\n---\r\n> &nbsp;&nbsp;oauth.config.id&nbsp;=&nbsp;cb4da2dbc7a22ad11bacb8e39ef92\r\n153c153\r\n< &nbsp;&nbsp;com.adobe.granite.ocs.cloud-delete.sleep&nbsp;=&nbsp;5000\r\n---\r\n<removed/>\r\n---\r\n> PID&nbsp;=&nbsp;org.apache.sling.jcr.resourcesecurity.impl.ResourceAccessGateFactory.c65fdeea-de49-4931-b3e7-274c39f7e\r\n934c934\r\n< &nbsp;&nbsp;service.pid&nbsp;=&nbsp;org.apache.sling.jcr.resourcesecurity.impl.ResourceAccessGateFactory.c65fdeea-de49-4931-b3e7-274c39f7e579\r\n---\r\n> &nbsp;&nbsp;service.pid&nbsp;=&nbsp;org.apache.sling.jcr.resourcesecurity.impl.ResourceAccessGateFactory.c65fdeea-de49-4931-b3e7-274c39f7e\r\n983c983\r\n< &nbsp;&nbsp;json.maximumresults&nbsp;=&nbsp;1000\r\n---\r\n> &nbsp;&nbsp;json.maximumresults&nbsp;=&nbsp;1\r\n989c989\r\n< &nbsp;&nbsp;servletresolver.cacheSize&nbsp;=&nbsp;1000\r\n---\r\n> &nbsp;&nbsp;servletresolver.cacheSize&nbsp;=&nbsp;1\r\n995d994\r\n< 200\r\n\\ No newline at end of file\r\n```\r\nAs you can see a lot of things were modified. \r\n\r\nWhy I think it was hard to find this? `cqcfgls` cares only about PID lines, as the purpose of it is to list all available OSGi configurations. So despite of the fact there were amendments in over 200 lines in raw form, the final output contained only 12 incorrect PIDs on my test AEM instance (557 items in total were displayed). In my case only PIDs with UUID names were affected. \r\nMoreover this bug, as said in the beginning, used to occur in a pretty non-deterministic manner and majority (if not all) of these happened on my CI environment that spins up brand new VM, runs Chef inside followed by a number of test suites and destroys the VM, so I had no access to that environment when the process was over. When I click 'Build' after such failure the bug was gone. And, as you can imagine, I was not able to reproduce it on my laptop either.\r\n\r\nHow do I found that? One of my Chef cookbooks creates a couple of OSGi configurations. All the actions in the cookbook have a matching test suite. One of these configurations was created using factory OSGi configuration and I expected that after provisioning its state should be as follows:\r\n\r\n* only one factory config instance should be present\r\n* configuration of it should match certain criteria (i.e. `key1 == 100`, `key2 == 'string2'`, etc)\r\n\r\nTo verify first requirement I did the following thing:\r\n```ruby\r\nregex_suffix = '\\.[a-z0-9]{8}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{12}'\r\nlist.scan(/#{factory_pid}#{regex_suffix}/)\r\n```\r\nIn above code snippet `factory_pid` was the PID I was looking for and `list` is the output of `cqcfgls` command. All in all I was looking for the number of occurrences of a given factory PID.\r\n\r\nRecently I finally managed to get the same error locally. The output of my test was as follows:\r\n```\r\nFailures:\r\n       \r\n         1) AEM Author: <my_factory_config> is configured correctly\r\n            Failure/Error: )\r\n              \r\n              expected: \"1000\"\r\n                   got: \"\"\r\n              \r\n              (compared using ==)\r\n              \r\n            # /tmp/busser/suites/serverspec/author_conf_spec.rb:38:in `block (2 levels) in <top (required)>'\r\n       \r\n         2) AEM Author: <my_factory_config> has just a single instance\r\n            Failure/Error: ).length\r\n              \r\n              expected: 1\r\n                   got: 0\r\n              \r\n              (compared using ==)\r\n              \r\n            # /tmp/busser/suites/serverspec/author_conf_spec.rb:72:in `block (2 levels) in <top (required)>'\r\n```\r\n\r\nWhen I logged into OSGi console, the configuration was in place and had valid UUID. `cqcfgls` reported that there's a  `my.factory.config.43d23f6f-e5e7-4bc2-82da-72b2b1664` config (3 last characters of UUID are missing), but in fact it was `my.factory.config.43d23f6f-e5e7-4bc2-82da-72b2b1664787`.\r\n\r\nWhy this problem occurred only from time time? UUID is generated and appended to each OSGi config that is created from a factory PID. UUID is always a string that has 5 groups separated by hyphens (8-4-4-4-12 characters in each group). However there's no guarantee that the very last 3 characters of the last UUID group will always be numbers, hence the non-deterministic behaviour.\r\n\r\nIn this PR I modified the way `/system/console/config/Configurations.nfo` is processed. HTTP status code is appended as a new line to the raw output and later on, when this info is not needed any more, this line is simply removed. By the way, I've also updated how `sed` is referenced, previously it was just `sed` instead of `${SEDX}`.\r\n\r\nThat's a pretty long explanation as for the PR that changes just 2 lines :)"},"comment":{"url":"https://api.github.com/repos/Cognifide/CQ-Unix-Toolkit/issues/comments/90483829","html_url":"https://github.com/Cognifide/CQ-Unix-Toolkit/pull/31#issuecomment-90483829","issue_url":"https://api.github.com/repos/Cognifide/CQ-Unix-Toolkit/issues/31","id":90483829,"user":{"login":"kitarek","id":4390904,"avatar_url":"https://avatars.githubusercontent.com/u/4390904?v=3","gravatar_id":"","url":"https://api.github.com/users/kitarek","html_url":"https://github.com/kitarek","followers_url":"https://api.github.com/users/kitarek/followers","following_url":"https://api.github.com/users/kitarek/following{/other_user}","gists_url":"https://api.github.com/users/kitarek/gists{/gist_id}","starred_url":"https://api.github.com/users/kitarek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kitarek/subscriptions","organizations_url":"https://api.github.com/users/kitarek/orgs","repos_url":"https://api.github.com/users/kitarek/repos","events_url":"https://api.github.com/users/kitarek/events{/privacy}","received_events_url":"https://api.github.com/users/kitarek/received_events","type":"User","site_admin":false},"created_at":"2015-04-07T09:34:45Z","updated_at":"2015-04-07T09:34:45Z","body":"Hi Jakub,\r\n\r\nwow - thanks so much for such detailed explanation! sed -r is for sure not the best option - it's not portable.\r\nRegarding new status update it would be good to fix this for all different commands which we need to review in `1.2-dev` branch.\r\n\r\nAlso I need to check if `--write-out \"\\n%{http_code}\\n\"` is portable on xBSD systems.\r\n\r\nThanks once again,\r\nArek"}}